Print Story I just haven't had a useful way of saying it.
Working life
By technician (Tue Aug 24, 2010 at 12:16:21 PM EST) (all tags)
I mean, there's only so much onomatopoeia that you can pack into a body of text before you completely lose the thread.

This is just me complaining about work. Because I'm dead tired, and can't do anything else.



Our website at work was hacked Saturday night / Sunday morning, with a failed php injection. The attackers tried to quietly place an iframe that would have loaded some malware, quietly....but they didn't read the instructions or their cut ~y~ paste skills weren't so good, so what they did was replace my index.php with a redirect to their malware site. They also tried 12 times to load a shellbot trojan, which almost worked. Lessons learned: lazy = bad...I'd left the server with capability to wget and resolve names, both for the sake of yum. I'd also not completely locked down the wordpress install....

Oh yeah, our corporate website is a wordpress blog without comments, and with a static page. This is because three years ago my bosses hired the husband of the receptionist (he's a "web designer," and we like to stay in house for everything) to design a new page. He's a drag ~y~ drop coder, and the eventual design was a cut and paste collection of other bits glued by bad theme framework, all of it running under the wordpress thing.

So, I have gobs of badly made php on the server, and because I'm dealing with amateurs, the server had an old version of phpMyAdmin, an old version of an admin theme for the wordpress dashborard, and a shedload of useless bits. One of them was vulnerable.

I was OK with just nuking the site and restoring from backup, but then I saw the shell bot and decided that I didn't want to risk being rooted...my rootkit scan showed a handful of suspicious files. So, nuked the server from orbit and started over with bare metal.

Oh, and we don't have a website backup. We do have the old web server, which runs a Debian version actually written on papyrus leaves, and the oldest version of wordpress that would run on it. And a whole slew of other old crap.

Oh, also, our web server is our external facing dns server...authoritative, serving names for mail and web and etc.

So. We're down, I'm sanding the drive platters, installing redhat 5.5 64bit with patches, exporting the old web server's sql database, getting all the old theme + plugins, etc etc. Very long and brutal story short, in 9.5 hours we were back on the air. The backup web server was pressed into place as primary during the downtime (it is already our secondary name server) and held up well. Name services were mostly maintained, and the switchover was as painless as that can be.

New server: RedHat 5.5 updated yesterday. Runs SELinux, a firewall, and has MD5-summed pages. No PHPMyadmin. Directories locked down to the point of uselessness. Can not generate traffic to the outside world...can only respond to http and dns. All traffic to the site logged and dissected. Sophos antivirus for linux servers up and running. The core is secure, but the wordpress thing....gah. It isn't so much wordpress. I just don't know enough PHP. But you'd think my department...the guys who maintain the technology we use...would have a say in the website. You'd think.

That's a running theme here. There are two primary issues with this place. One is: there is no planning with regards to IT. Oh, the IT team plans...we have a gigantic budget, we have schedules, we're good at knowing what we need, what we have, and what we can do. What happens is, someone plans a test or an exercise, and months after planning it will come to us and say "OK, I need five laptops that have eSATA ports, Linux OSs, and support for every file system, plus I need thirteen 2TB drives with eSATA. I need them tomorrow."

The second major issue is that IT has no influence on the technology that is put in place in certain key areas. For instance, the first two years I was here, our mail server was a Dell consumer-grade desktop running mirrored IDE drives. Postfix + dovecot. It took me two years to convince the powers that be that we needed server hardware, regardless of the underlying OS or mail program. The fight to get Exchange (we wanted Exchange because it is bone simple to admin and if we died, anyone could support it, plus it has calendering) took 11 months. When the new website was developed, we were handed a wordpress tarball...our web server at the time couldn't run PHP, didn't meet any of the minimum requirements, and, well, we weren't included on any of the design or back end discussions.

This sort of thing happens all the time. Someone will show up with a set of servers wanting to get them installed and running....in a lab without any power (much less UPS) capacity, no cooling capacity, and no physical space. It makes the job pretty annoying, at best.

Now I have to tell the feebs, our upline security folks, the dss, and a handful of other people what happened to our server and what we think the intent was. There will be visits from feds and lots of logfile analysis. Things will be investigated in and out. I'll advise everyone who'll listen that we need to hire an actual web programmer type to create a static set of pages that look + feel like our current setup. I'll recommend that we have it hosted away from our DMZ. About the time the uproar calms and our ideas get ignored, we'll get hacked again.

It's an odd way to spend your time.

< on celebrations | certified Afronauts, capable of funkitizing galaxies >
I just haven't had a useful way of saying it. | 33 comments (33 topical, 0 hidden)
You need by jimgon (4.00 / 1) #1 Tue Aug 24, 2010 at 12:41:26 PM EST
an IT manager with a project managment background. 




---------------
Technician - "We can't even get decent physical health care. Mental health is like witchcraft here."
you'd move to TX ? by sasquatchan (4.00 / 1) #3 Tue Aug 24, 2010 at 12:51:39 PM EST
that desperate ?

[ Parent ]
Define desperate by jimgon (4.00 / 1) #9 Tue Aug 24, 2010 at 04:58:22 PM EST
Not really interested in moving, but the opportunities that I thought existed in this neck of the woods have dried up.   Not much going on around here right now.




---------------
Technician - "We can't even get decent physical health care. Mental health is like witchcraft here."
[ Parent ]
Not much will be going on ANYWHERE soon enough. by lolwhat (2.00 / 0) #20 Wed Aug 25, 2010 at 06:53:44 AM EST
Best to get/stay close to family and friends. In fact, some would say it's utterly necessary. People have no idea what's coming...
--
If cigarette packs are required to have pictures of diseased lungs, college brochures should be required to have photos of grads working at Starbucks.
[ Parent ]
Well by jimgon (2.00 / 0) #22 Wed Aug 25, 2010 at 12:06:02 PM EST
If you're talking a repeat of the great depression or perhaps something a bit worse, yes, I am conscious that that is coming.




---------------
Technician - "We can't even get decent physical health care. Mental health is like witchcraft here."
[ Parent ]
Out of curiousity by theboz (2.00 / 0) #25 Wed Aug 25, 2010 at 07:23:15 PM EST
How do you know something bad like that is coming?  I mean, sure, the economy is looking bad and there are plenty of negative things, but wouldn't it be pretty easy to fix everything if everyone wanted things to get better?
- - - - -
That's what I always say about you, boz, you have a good memory for random facts about pussy. -- joh3n
[ Parent ]
Lots of reasons. by lolwhat (2.00 / 0) #26 Wed Aug 25, 2010 at 08:21:38 PM EST
Economic data. Unemployment numbers, CPI, PPI, housing, taxes, consumer debt, business debt, municipal debt, state debt, federal debt, global debt... The parallels to the Great Depression are there; you just need to look.
--
If cigarette packs are required to have pictures of diseased lungs, college brochures should be required to have photos of grads working at Starbucks.
[ Parent ]
I guess I'm playing devil's advocate by theboz (2.00 / 0) #27 Wed Aug 25, 2010 at 09:23:56 PM EST
There is lots of bad news, and it could go badly at some point, but why do you feel that it would be allowed to get that badly?  In dire straits, wouldn't the U.S. just consider canceling out debt or waging war on other countries to eliminate debt or steal their resources?  It seems to me that the problems we are encountering are, for lack of a better term, "paper" problems.

The core facts about the U.S. having a healthy, educated workforce; the U.S. having a tremendous amount of resources; we have access to an ability to produce more than enough food; we have a military capable of destroying any other nation we wish to destroy including China, etc. are all still there.  All those factors to me indicate that while things could get worse, we are too powerful to actually go into a full depression.  That's not to say that our nation will make good decisions to combat the problems, but I do think we can easily prevent ourselves from going into a depression.  I really feel like I should be very pessimistic about it and I am planning for the worst just in case, but for some reason I feel some optimism too.
- - - - -
That's what I always say about you, boz, you have a good memory for random facts about pussy. -- joh3n

[ Parent ]
well by lolwhat (2.00 / 0) #28 Wed Aug 25, 2010 at 10:05:08 PM EST
Wage war to grab resources? I guess the U.S. could try. However, it wouldn't be enough to soak up all the debt - and, of course, the rest of the world would be rather pissed off.

Default on debt? If the U.S. goes that route, crushing austerity would be the next step, because no one would lend to the U.S. for a very long time, if ever.

Also: We're already in a depression - if you don't count the enormously massive amount of money that the feds are throwing around. I - and many others - don't consider government spending as a productive part of the economy, especially when it goes to prop up banks and other favored "constituencies" that don't produce shit.
--
If cigarette packs are required to have pictures of diseased lungs, college brochures should be required to have photos of grads working at Starbucks.

[ Parent ]
I don't entirely disagree by theboz (2.00 / 0) #29 Wed Aug 25, 2010 at 10:47:21 PM EST
Again, I'm just trying to play devil's advocate somewhat to "test" my own point of view because something in the back of my mind is nagging me about being wrong.

Wage war to grab resources? I guess the U.S. could try. However, it wouldn't be enough to soak up all the debt - and, of course, the rest of the world would be rather pissed off.

I wouldn't say that waging war would be entirely to grab resources to pay off debt as much as destroy those that we owe the money to.  Hypothetically, a war with China and the U.S. having a "reason" to cancel all bonds owned by China could help U.S. solvency quite a bit.

Also: We're already in a depression - if you don't count the enormously massive amount of money that the feds are throwing around. I - and many others - don't consider government spending as a productive part of the economy, especially when it goes to prop up banks and other favored "constituencies" that don't produce shit.

I can't say that I fully agree here.  Most business do not keep a surplus of money, so the banks are required by our current system to keep companies running.  For example, if all the banks collapsed, take the top 50 most profitable companies in the U.S. -- they would likely collapse too, because they couldn't afford payroll because they run a lean ship and if they lost their banks, it would really screw up their ability to pay their employees, much less do business.

Personally, I wish that the Obama administration would have set up accounts for citizens that could be used for 1) buying a new house, 2) paying down your mortgage, or 3) performing maintenance on your house in order to sell it.  The people eligible for #1 would have to not currently own a house, #2 would be for people with mortgages, and #3 would be for people who own a house.  It's not perfect, but it would be sort of a "trickle up" effect to the banks where the actual assets could be more solvent, and they would still have to answer for their risky derivatives and whatnot.

Also, the core issue seems to be jobs.  If the U.S. started producing something again, whether it's factory work or some advanced research, we could easily create jobs that would make all the other problems with the economy go away.  However, as we've recently seen, America is too preoccupied with the location of a community center for moderate Muslims, and we can't do scientific research because it offends Jesus, and the rich need to keep robbing the rest of us while we face austerity measures such as 911 going away in some places and schools being closed down.
- - - - -
That's what I always say about you, boz, you have a good memory for random facts about pussy. -- joh3n

[ Parent ]
Hi. I'm a veteran by ammoniacal (2.00 / 0) #30 Thu Aug 26, 2010 at 06:39:53 AM EST
and I'm here to tell you there's no way we could militarily destroy China today. That day passed before Desert Storm kicked off. Not gonna happen.

"To this day that was the most bullshit caesar salad I have every experienced..." - triggerfinger

[ Parent ]
Really? by theboz (2.00 / 0) #31 Thu Aug 26, 2010 at 02:49:13 PM EST
They could stop us from bombing their major cities?  I'm not even talking about nukes or an occupation, just about destroying their economic power.
- - - - -
That's what I always say about you, boz, you have a good memory for random facts about pussy. -- joh3n
[ Parent ]
I did say militarily. by ammoniacal (2.00 / 0) #32 Thu Aug 26, 2010 at 07:15:45 PM EST
I'm not convinced that their military power depends on a few major cities. They're working on anti-ballistics now, so no, I don't think that way would be a resounding success in the near future.

"To this day that was the most bullshit caesar salad I have every experienced..." - triggerfinger

[ Parent ]
You don't need a war by jimgon (2.00 / 0) #33 Thu Aug 26, 2010 at 07:16:27 PM EST
The US could destroy China's economic foundation by simply implementing a tarriff.   The fact that China was allowed MFN status with the US is what powered their ascent.  If the US implemented a tarriff Chinese goods would no longer be so cheap in the US, the major consumer, and their economy would crash.  The Chinese people and the rest of the world can't take up the slack for the American consumer loss.  Incidentally if the Chinese devalued their currency a similar effect would take place.  It's the real reason the Chinese don't want to do a currency revaluation.




---------------
Technician - "We can't even get decent physical health care. Mental health is like witchcraft here."
[ Parent ]
(Comment Deleted) by aggressive cyclist (4.00 / 2) #2 Tue Aug 24, 2010 at 12:41:56 PM EST

This comment has been deleted by aggressive cyclist



Scoop's too web 1.0 by wiredog (4.00 / 2) #4 Tue Aug 24, 2010 at 01:35:09 PM EST
He needs something modern.

Earth First!
(We can strip mine the rest later.)

[ Parent ]
phpmyadmin by bobdole (4.00 / 1) #5 Tue Aug 24, 2010 at 03:32:06 PM EST
seems to be popular with your average portscanning hacker these days. The one server I actually care about (which has an outwards facing interface with about 40 odd IPv4 addresses active and a few IPv6 addresses) gets scanned from about 40 different hosts trying various combinations (more like a gazillion) of phpmyadmin every day.

While I feel for your pain, you more or less had it coming running out of date known vulnerabilities unprotected. Even wordpress has a self-updater these days...

-- The revolution will not be televised.
Its true. by technician (2.00 / 0) #6 Tue Aug 24, 2010 at 04:11:47 PM EST
Much of this would have been avoided entirely had we been more dedicated to updating it. Now, it's on the list.

By the way, this was the exploit: http://www.phpmyadmin.net/home_page/security/PMASA-2010-5.php

Disclosed 8/20.

[ Parent ]
(Comment Deleted) by aggressive cyclist (4.00 / 1) #8 Tue Aug 24, 2010 at 04:36:05 PM EST

This comment has been deleted by aggressive cyclist



[ Parent ]
Running should be giving you by ammoniacal (4.00 / 1) #7 Tue Aug 24, 2010 at 04:34:49 PM EST
megajoules of excess energy. What's the dillio?

"To this day that was the most bullshit caesar salad I have every experienced..." - triggerfinger

The opposite. by technician (4.00 / 1) #10 Tue Aug 24, 2010 at 06:36:37 PM EST
I'm dead like half the time. That ten miles on Saturday nearly killed me, and today's less than stellar 2.75 was like dragging an anvil. I get home and all I wanna do is sleep, but then I go to bed and can't sleep and GAH.

So, yeah. I may not be eating enough.

[ Parent ]
(Comment Deleted) by aggressive cyclist (4.00 / 1) #11 Tue Aug 24, 2010 at 06:51:14 PM EST

This comment has been deleted by aggressive cyclist



[ Parent ]
nuts to that by lm (2.50 / 2) #12 Tue Aug 24, 2010 at 08:15:51 PM EST
I almost never eat before I run.

There is no more degenerate kind of state than that in which the richest are supposed to be the best.
Cicero, The Republic
[ Parent ]
(Comment Deleted) by aggressive cyclist (2.00 / 0) #16 Wed Aug 25, 2010 at 02:44:43 AM EST

This comment has been deleted by aggressive cyclist



[ Parent ]
Sure there is by lm (4.00 / 1) #18 Wed Aug 25, 2010 at 05:43:31 AM EST
I tried it for a couple of months.

There is no more degenerate kind of state than that in which the richest are supposed to be the best.
Cicero, The Republic
[ Parent ]
(Comment Deleted) by aggressive cyclist (4.00 / 2) #21 Wed Aug 25, 2010 at 08:29:18 AM EST

This comment has been deleted by aggressive cyclist



[ Parent ]
Your supposition has some holes, by technician (2.00 / 0) #13 Tue Aug 24, 2010 at 08:16:02 PM EST
in that I do eat very well, but my running happens at 6am, right after I wake, so no time for anything.

My total diet, though, is less than 2000 calories a day. Need to work on that.

[ Parent ]
yeah, my first guess would be the distance by lm (4.00 / 1) #14 Tue Aug 24, 2010 at 09:11:39 PM EST
I know people that have had success shooting chiefly for distance, fast.

But in my experience, shooting for consitency first, and then slowly increasing distance works much better. That is to say, once you can run a week full of five miles, try throwing in a sixer one day a week. And when you get used to that, go for six twice a week. Eventually you'll get up to whatever distance it is that is your goal.

That said, keep in mind that not everyone is built like me. Different bodies and different sets of motivations mean that what works for me may not work for someone else.

But I still think going 10 is what's dragging you down all week. That's a hella distance. Running is a brutal exercise. Running  10 miles compounds that brutality and your body needs time to recover.


There is no more degenerate kind of state than that in which the richest are supposed to be the best.
Cicero, The Republic
[ Parent ]
(Comment Deleted) by aggressive cyclist (4.00 / 1) #17 Wed Aug 25, 2010 at 03:54:33 AM EST

This comment has been deleted by aggressive cyclist



[ Parent ]
Handful of nuts and dried frut. by Tonatiuh (4.00 / 2) #19 Wed Aug 25, 2010 at 06:51:36 AM EST
That should be enough for anything under 5km (3miles).

If you run longer you really need to eat more, otherwise you are going to pass out one day.

[ Parent ]
What's it going to take? by muchagecko (4.00 / 1) #15 Tue Aug 24, 2010 at 10:26:31 PM EST
I could probably take a few minutes to make an f'ing PowerPoint presentation that shows how much money they'll save if they recreate the website.

Would that get it through?

Or maybe you could pretend that 'hack and I are your sisters and we just happen to do web dev?


A purpose gives you a reason to wake up every morning.
So a purpose is like a box of powdered donut holes?
Exactly
My Name is Earl

Hmmmm. by technician (4.00 / 1) #23 Wed Aug 25, 2010 at 12:37:11 PM EST
I guess we'll see what happens after we meet with the FBI for the third time this year due to technical attacks. Maybe that will convince them.

[ Parent ]
I'm not a fan of DSS interaction by garlic (4.00 / 1) #24 Wed Aug 25, 2010 at 02:18:28 PM EST
mostly because it means we screwed something up. We got CRAD work on our IRAD network and are currently dealing with a fiasco getting forms filled out because of CRAD customer acknowledgement issues.


I just haven't had a useful way of saying it. | 33 comments (33 topical, 0 hidden)