Print Story A Day in the Life
Working life
By ReallyEvilCanine (Mon Mar 12, 2007 at 05:08:33 AM EST) A Day in the Life, cubicle, daylight savings time, fucktards, pie (all tags)
It's the End of the World! Oh noes!

In the run-up to Y2K there was no shortage of doomsayers. Systems would shut down. Banking software would go tits up and fortunes, if not lost, would be reduced to their 1900 or 1970 values. Planes would drop out of the sky. Software everywhere would crash.

And then came January 1, 2000. The world didn't end. It wasn't even thrown back into the stone age. Planes didn't fall from the sky. Systems didn't shut down. Banking software kept working. The worst problems most of us saw were nuisances, dates like "January 4, 19100" and old calendar apps mistakenly adding a February 29th to the year. The doomsayers disappeared and the IT industry was mocked by a flood of fuckwits chiding us for having made such a big deal about Y2K.

But for the alarms three years prior, it could've gone very differently.

x-posted to da brog.

Until the late 1990s, most of the public didn't realise that an awful lot of infrastructure software was as old as dirt, much of it written in ancient (to the point of obscure) languages, some of it loaded on old IBM 3430 tape drives to this day. Up until 1993, the FAA's TRACON was controlled by ARTS-III software written in Ultra for a 1960s UNIVAC processor. These days it's running on PowerPCs and the language is C, but the conversion from Ultra/Univac to C on LynxOS/Motorola 68000 was long and difficult, and with only nine months to go before Y2K, the GAO issued a rather harsh report on the FAA's status.

If you code COBOL, 1977-1999 were very lucrative. The now-defunct AccuCobol made a fortune selling their software to programmers trying to refresh their COBOL skills and to companies via the generated binaries and libraries for converted or modernised COBOL systems which were, truth be told, less than efficient. Thousands of programmers pored through tens of millions of lines of code, reaping as much as $0.50 per line for their efforts.

Banks and insurance companies were the first to realise the problem since they pay attention to what might happen with rates, values and tables seven to ten years in advance. The rest of us rarely look at things more than a few years ahead, and so we were unprepared. Microsoft didn't have Y2K updates for Windows 95 and NT until October, 1999. Their office applications weren't completely fixed for months after that.

UNIX was a little better since the OS keeps track of the number of seconds since 1970, but in the midst of all the finger-pointing at Microsoft, UNIX people realised that they had their own NTP time bomb set to blow in the year 2036. There were other bugs, like RFC 822 which used 2-digit years for mail headers.

The programmers worked furiously. And, as always happens, in the rush to repair the known problem, new bugs were introduced. In the rush to release the fixes not all of these new bugs were caught.

And we learned nothing. Here we are seven years later and we're going through the same shit.

On August 8, 2005, US Public Law 109–58 -- the Energy Policy Act of 2005 -- went into effect. Section 110a designated new, longer Daylight Savings Time (DST) periods. Worse, Section 110d allowed a right to reversion in as little as nine months. And no one did anything.

Granted, the problems caused by the one-hour change were relatively minor in comparison to some of the mess that could've happened after Y2K, and the Daily Show rightfully took the piss.

On the surface, yeah, it's a minor inconvenience. For many businesses, however, it's a bit more expensive. Many industries are required by law to keep extremely accurate audit trails. An hour makes a world of difference: either you're compliant or you're not. Scheduling, conferences, payments... businesses face a lot of problems if this problem isn't resolved. So while being late for a doctor's appointment isn't terribly drastic, $LargeTelco's inadvertently not running an entire day's reports is.

It wasn't until shortly before DST reverted back to Standard time in 2006 that anyone really started mentioning the upcoming changes, and with those changes the realisation that the complexity introduced by the changes which were more or less ignored in Israel and Australia in years past could no longer be ignored. DST problems were coming home to America.

Time zone information had normally been stored as a flat look-up. Check the zone, check the date look at the GMT offset and apply. No longer. Change a zone to the new value and all your historical data will be off: programs will apply the current DST info to dates prior, except that March 12, 2006 was Standard Time, not DST.

There's a full magnitude of complexity introduced in dealing with the problem. A flat file can't be used; a 3D table is necessary to account for years. Microsoft didn't have a fix until December, 2006. Companies including $MegaCorp came up with stopgap measures and temporary fixes, themselves buggy or quirky. And we'll ride out the storm for the three weeks between the old dates and new, hoping that most customers don't notice their jobs running an hour late or appearing to be scheduled two hours early.

A lot of customers tested. Those who tested complained. Those who complained kept me very busy. Every department involved, taking cues from upper management's decision regarding releasing the latest version with this time bomb bug in place, adopted a laissez faire attitude and little was done. Documentation was delayed, conferences ignored for "more important" tasks, and the few robe-wearing, long-bearded hippies carrying their signs reading "The End of the World is Nigh" were mostly ignored. That didn't stop us from trying.

We hounded our third-party suppliers for info and fixes. We tested as much as we could. I wrote documents. These documents had to be continually updated as we discovered new information, such as the fact that all version 3 systems were affected and not just 3c and 3g. Oh, and version 4b would be hit, too. A week before the expected changes it turned out that Sun's JREs which also had to be fixed, contained a major bug (ID 6530336) which broke the Eastern Time Zone functionality. The only fix: manual changes. That'll be fun for 20,000-seat call center admins.

Final documents with even more information weren't made public in a timely manner and required a lot of hounding and escalation to initiate their release to our Knowledge Base. Customers had been given incorrect information and had to be informed of our new discoveries. I escalated and called and screamed and bitched until the idea was finally accepted: every monkey had to go through every one of his own tickets to find those which had asked about DST, then send updated information. The cost of that? About 30-60 minutes per monkey on average.

Some of us did special weekend duty, ready for the onslaught of tickets from fuckwits who didn't read our Urgent Notices, pay attention to our direct mails, or who didn't follow our instructions.

It never came. Only a couple tickets related to the subject showed up and these dealt with unexpected problems which we hadn't had the time to test.

My manager Vera walked by this morning and asked how bad it had been. "Only two tickets for us and five for the US." "Well, you see? It wasn't a problem. You were so worried about this for nothing."

They said the same thing in January, 2000 as well. We have the testbeds. We proved that without our work things would've gone very differently, but no one wants to hear that. Shit's working now and that's all they care about. And because I didn't spend all day on the phone Sunday, instead of a few hundred in cash for being available Sunday, I got a T-shirt which is three sizes too small.

You're welcome, you fuckwits.

< summary | BBC White season: 'Rivers of Blood' >
A Day in the Life | 6 comments (6 topical, 0 hidden)
The change is working though by Herring (4.00 / 5) #1 Mon Mar 12, 2007 at 05:36:19 AM EST
I notice much more daylight than there was only a few weeks ago.

You can't inspire people with facts
- Small Gods

Cobol and Y2K by wiredog (2.00 / 0) #2 Mon Mar 12, 2007 at 05:44:58 AM EST
In the mid-90's my cousin in Seattle decided that she'd rather stay home and be a full time mother than go to work. Since her husband made more than she did (and she wrote software for the medical field) she could afford to.

In 1998 various former co-workers started mentioning her to their employers. She knows Cobol, and has programmed in it. Job offers started coming in. Large consulting fees were waved in her face.

She put the kids in day care, went back to work and, by January 2000, the kids college funds were fully funded. The retirement fund was in pretty good shape too.

Earth First!
(We can strip mine the rest later.)

Cobol by duxup (2.00 / 0) #6 Thu Mar 15, 2007 at 01:04:51 PM EST
When I was growing up a friend’s father said You wanna make money?  Learn Cobol, in a few years you’ll be able to name your price.  Had I known he was going to be as right as he was I would have done it.

I wasn’t interested in programming, but as soon as I entered the workforce full time I ended up in a job where it was explained that when the systems programmer talked everyone listened and helped him, when the systems programmer wanted to come to work he did, when he didn’t he didn’t, all because the people at the bank were terrified he might leave.


[ Parent ]
Yep by theboz (4.00 / 2) #3 Mon Mar 12, 2007 at 08:50:14 AM EST
I can see both sides I suppose. The bottom line though is that people don't seem to understand that when there is a risk of problem and you do something to avert the problem, that doesn't mean there was never a risk. This seems to be some sort of weird cultural fatalism where society prefers to sit back and let everything else happen to them instead of participating in their environment to shape it as they see fit.
- - - - -
That's what I always say about you, boz, you have a good memory for random facts about pussy. -- joh3n
Y2K by ucblockhead (4.00 / 2) #4 Mon Mar 12, 2007 at 09:08:15 AM EST
I spent time on alt.computers.y2k (or whatever it was) back in 1999. It was amusing, especially the guy who liked to tell people that their entrails would be torn out by wild dogs after civilization collapsed.

At the same time, I personally fixed a number of bugs, including one that would have had the large retail chain I worked for utterly unable to collect sales data from their stores. (Anything with a date of "00" would have been dumped.) I did miss a couple bugs...a couple minor reports said "1900" the day after. This was C code, not Cobol.
[ucblockhead is] useless and subhuman

The biggest problem by debacle (2.00 / 0) #5 Mon Mar 12, 2007 at 02:08:08 PM EST
Was that our payroll app doesn't support the pre-DST version of JRE. It runs fine on that version, they just disable functionality so you have to patch.

Vera sounds like a cunt.


A Day in the Life | 6 comments (6 topical, 0 hidden)