Oh yeah, our corporate website is a wordpress blog without comments, and with a static page. This is because three years ago my bosses hired the husband of the receptionist (he's a "web designer," and we like to stay in house for everything) to design a new page. He's a drag ~y~ drop coder, and the eventual design was a cut and paste collection of other bits glued by bad theme framework, all of it running under the wordpress thing.
So, I have gobs of badly made php on the server, and because I'm dealing with amateurs, the server had an old version of phpMyAdmin, an old version of an admin theme for the wordpress dashborard, and a shedload of useless bits. One of them was vulnerable.
I was OK with just nuking the site and restoring from backup, but then I saw the shell bot and decided that I didn't want to risk being rooted...my rootkit scan showed a handful of suspicious files. So, nuked the server from orbit and started over with bare metal.
Oh, and we don't have a website backup. We do have the old web server, which runs a Debian version actually written on papyrus leaves, and the oldest version of wordpress that would run on it. And a whole slew of other old crap.
Oh, also, our web server is our external facing dns server...authoritative, serving names for mail and web and etc.
So. We're down, I'm sanding the drive platters, installing redhat 5.5 64bit with patches, exporting the old web server's sql database, getting all the old theme + plugins, etc etc. Very long and brutal story short, in 9.5 hours we were back on the air. The backup web server was pressed into place as primary during the downtime (it is already our secondary name server) and held up well. Name services were mostly maintained, and the switchover was as painless as that can be.
New server: RedHat 5.5 updated yesterday. Runs SELinux, a firewall, and has MD5-summed pages. No PHPMyadmin. Directories locked down to the point of uselessness. Can not generate traffic to the outside world...can only respond to http and dns. All traffic to the site logged and dissected. Sophos antivirus for linux servers up and running. The core is secure, but the wordpress thing....gah. It isn't so much wordpress. I just don't know enough PHP. But you'd think my department...the guys who maintain the technology we use...would have a say in the website. You'd think.
That's a running theme here. There are two primary issues with this place. One is: there is no planning with regards to IT. Oh, the IT team plans...we have a gigantic budget, we have schedules, we're good at knowing what we need, what we have, and what we can do. What happens is, someone plans a test or an exercise, and months after planning it will come to us and say "OK, I need five laptops that have eSATA ports, Linux OSs, and support for every file system, plus I need thirteen 2TB drives with eSATA. I need them tomorrow."
The second major issue is that IT has no influence on the technology that is put in place in certain key areas. For instance, the first two years I was here, our mail server was a Dell consumer-grade desktop running mirrored IDE drives. Postfix + dovecot. It took me two years to convince the powers that be that we needed server hardware, regardless of the underlying OS or mail program. The fight to get Exchange (we wanted Exchange because it is bone simple to admin and if we died, anyone could support it, plus it has calendering) took 11 months. When the new website was developed, we were handed a wordpress tarball...our web server at the time couldn't run PHP, didn't meet any of the minimum requirements, and, well, we weren't included on any of the design or back end discussions.
This sort of thing happens all the time. Someone will show up with a set of servers wanting to get them installed and running....in a lab without any power (much less UPS) capacity, no cooling capacity, and no physical space. It makes the job pretty annoying, at best.
Now I have to tell the feebs, our upline security folks, the dss, and a handful of other people what happened to our server and what we think the intent was. There will be visits from feds and lots of logfile analysis. Things will be investigated in and out. I'll advise everyone who'll listen that we need to hire an actual web programmer type to create a static set of pages that look + feel like our current setup. I'll recommend that we have it hosted away from our DMZ. About the time the uproar calms and our ideas get ignored, we'll get hacked again.
It's an odd way to spend your time.
|< on celebrations | certified Afronauts, capable of funkitizing galaxies >|