Print Story The best-laid plans...
By ana (Fri Jun 30, 2006 at 12:36:55 PM EST) Space, the final frontier. (all tags)
What happens when the shooting starts will astonish you.

--Dwight Eisenhower

I was involved in a project once that involved a payload on the Space Shuttle. Seemed appropriate to talk about it on this return-to-flight (yet again) weekend.

In some ways, the Challenger accident was good for the project, because we (they, in those days; it was before I started work there) were in no way ready for the pre-Challenger manifest. They got another couple of years to work on things, and I got a job, because I ran into the guy running the project at a conference.

"You need a pet theoretician," I told him. He seemed to like the idea.

I ended up doing all kinds of strange things. Sitting through endless meetings with engineers, who, it turns out, come in 2 or 3 varieties: mechanical, electrical (with subtypes digital and analog), and software. Whether software is an engineering discipline or not is the subject of another rant.

Turns out that mechanical engineers and electrical engineers can hardly talk to each other, and it showed on the project in question. The electronics boxes ran out of real estate, and had add-on boards which broke, physically, during vibration testing. The pressure transducers were susceptible to electromagnetic interference.

The PI's solution to all this: hire a physicist to translate. Which would have been me. It was a hoot. I've also been used as a translator between scientists and software people, which is a different kind of a hoot.

Anyway. A couple/three years went by, we repeated the testing program, more or less passed, fixed the "or less" parts, and arrived at the Cape. If I thought the engineering paperwork was nonlinear back home, it's an order of magnitude worse when you get anywhere near the manned space program. Tools. You check them out from the tool crib. You check them back in again. Anything that's lost becomes a class one fire drill. And still there have been a case or two of the Shuttle flying with a stray wrench loose someplace.

Payload integrated, we showed up at the control room, where we'd previously installed ground support equipment (GSE), some rather antique but real-time computers. The second set of GSE, for redundancy and a near-real-time look at the data, had to be shipped last minute from the Cape and installed.

Launch. It flew. Woh. I watched from a hotel room in Greenbelt, MD. The idea was we'd have 3 shifts: the PI, myself, and the cognizant EE would be shift leaders.

Well, except for Flight Day One; the PI had to be at the Cape to hold the hands of VIPs on launch day. So he spent his first shift on an airplane. And I had to use the launch time to select details of the operations strategy, once we knew the orbit. Even though I was supposedly evening shift, I had to do that at 10AM.

By the start of my 2nd shift, it was clear that all was not sweetness and light. Autonomous systems were behaving as designed, not as we'd intended. So the 3-shift thing was right out. The EE and I ramped up to 12-hour shifts, leaving the PI and the other brain trust to troubleshoot.

So I'd crawl out of bed for breakfast before the hotel restaurant closed at 10, go back for another few hours of sleep; show up at the control room late in the afternoon, and we'd run the damn thing for 12 hours until the EE came back to relieve me. I never really figured out when to eat.

Early one morning I was driving back down Greenbelt road, bound for the hotel and sleep. I turned the radio to NPR, and they were talking about some event on the Mall in Washington, DC, scheduled for Saturday Afternoon.

I thought about this for several minutes, and found that I had no idea what day it was, what time of day it was, or what the phrase "Saturday Afternoon" meant. My watch was set to Mission Elapsed Time, and that was pretty much all I needed to know.

We got the thing fixed, got some data, and (much later) published a paper and a PhD thesis based on it (not my thesis).

Woosh. What a ride. What a crazy way to make a living, though.

< There's Nothing New Under the Sun | BBC White season: 'Rivers of Blood' >
The best-laid plans... | 18 comments (18 topical, 0 hidden) | Trackback
software *ought* to be an engineering discipline by lm (2.00 / 0) #1 Fri Jun 30, 2006 at 01:01:39 PM EST
But as it is, there are precious few places where software is engineered rather than crafted. The software industry seems to be perpetually stuck in the artisan phase.

That's a nice anecdote about the space shuttle.

There is no more degenerate kind of state than that in which the richest are supposed to be the best.
Cicero, The Republic
Software culture by ucblockhead (2.00 / 0) #4 Fri Jun 30, 2006 at 01:31:04 PM EST
It's mostly because no one's willing to take the real time it takes to engineer software and exacerbated by the fact that to do real software engineering well, you need tools that are themselves well engineered.
[ucblockhead is] useless and subhuman
[ Parent ]
No by riceowlguy (2.00 / 0) #5 Fri Jun 30, 2006 at 01:40:19 PM EST
I mainly think it's because nobody's ever been sued for writing code that doesn't perform as it is spec'ed to perform.

[ Parent ]
IAWTP by MillMan (2.00 / 0) #16 Sat Jul 01, 2006 at 06:27:46 PM EST
As the tech media often says, consumer and business expectations for software are much lower than they are for other products.

Everybody still hates me in this city and I hate everybody.

[ Parent ]
I think the tool perspective is a cop out by lm (2.00 / 0) #14 Fri Jun 30, 2006 at 06:53:30 PM EST
But I wholeheartedly agree that few companies and few developers are willing to take the time to properly engineer software. And it's not just developers. Developers very seldom get proper requirements. Testers very seldom have meaningful test cases. The whole industry is rife with a ``good enough'' attitude.

There is no more degenerate kind of state than that in which the richest are supposed to be the best.
Cicero, The Republic
[ Parent ]
Are other disciplines different? by Gully Foyle (2.00 / 0) #15 Sat Jul 01, 2006 at 12:19:14 AM EST
I've certainly seen sparkies complaining about the same lack of proper requirements, and the development process for them seems pretty much a prototype-debug loop the same as us. Mechanical engineering seems just as bad from what I've seen of it (at the cheap end of the market). The only difference seems to be the amount of testing done, and that's hardly universal. Sure, aircraft manufacturers x-ray all their parts, but FADEC software is hardly less carefully engineered than that.

[ Parent ]
GASCAN shuttle project by jaxom green (2.00 / 0) #2 Fri Jun 30, 2006 at 01:15:06 PM EST
My senior project (for Bachelors in EE, digital) was to design and write the data acquasition and analysis programs for a shuttle GASCAN payload.  Given the years upon years of senior projects that went into that one simple payload I can't imagine what would be involved from start to finish.  I really should track down what happened to that project...

You should. by ana (2.00 / 0) #10 Fri Jun 30, 2006 at 03:29:32 PM EST
I wonder what all those engineers and techs think about the project, all these years later. Ours was kinda the next step up from the GAS cans (Get-Away Special).

Can you introspect out loud? --CRwM

[ Parent ]
Years later, likely they think of it very little by jaxom green (2.00 / 0) #17 Mon Jul 03, 2006 at 07:07:05 AM EST
Even with a fairly constant reminder of space exploration in my life (a very good lifelong friend of my father is very heavily involved with the Deep Impact mission) I don't often think about it.  For most people I still know from college it was just a way to fufill a degree requirement, a step on the way to a career in a more terestrial field.  It's really a shame since space is the next step in humankind's expansion, but I've got to count myself as one of the people who haven't kept track of it.  I think I owe my advisor an email howere.

[ Parent ]
Software engineering by miker2 (2.00 / 0) #3 Fri Jun 30, 2006 at 01:26:39 PM EST
After sitting through 2/3 of a Masters in it I can say that it does exist and should be used, but only on systems where human lives can be lost due to software malfunction.

Using Formal Methods in you average J2EE/.NET app is massive overkill and would cripple the project mainly due to the lack of actual CS/Math grads writing software these days.

Ah, sociopathy. How warm, how comforting, thy sweet embrace. - MNS
Some truth to that... by ana (4.00 / 1) #7 Fri Jun 30, 2006 at 01:49:29 PM EST
For example, the Space Shuttle operating system is widely believed to be bug-free. Think about it. Bug-free. It's a task whose difficulty basically requires human sacrifice.

Documentation of real engineering projects is a very serious thing. Just as a f'rinstance, the parts dude would get a dozen or two "alerts" per week, notifications to the effect that lot number of electronic part 972346B mark 4 with date codes between X and Y (you get the picture) is defective above 37 C. He could look at his database, and establish beyond a reasonable doubt that [a] we didn't have any of that stuff in the flight hardware, or [b] We have six of them, and then trigger the fire drill of verifying that we're not using them under such conditions, or what to do about it if we are.

All without being allowed to look at the source code (the payload; it's in the cleanroom down the hall and you can't mess with it).

Can you introspect out loud? --CRwM

[ Parent ]
They used formal methods by miker2 (2.00 / 0) #11 Fri Jun 30, 2006 at 04:23:49 PM EST
We did a case study of the NASA development programs in the Formal Methods class.  They mathematically prove every piece of that code before they write it.  That coupled with a ton of upfront design and exhaustive testing yielded a believed to be bug-free software system.  It's truly amazing that they were able to pull it off, but when you have to have perfection, they went the distance and got it done.

The software also took many engineers and mathematicians many many years to write, something that just can't be done in a business environment

Ah, sociopathy. How warm, how comforting, thy sweet embrace. - MNS
[ Parent ]
i don't know that it would be overkill. by aphrael (4.00 / 2) #8 Fri Jun 30, 2006 at 02:26:07 PM EST
the trouble is that producing software using engineering methodology is (a) more expensive and (b) more time-consuming than producing it using craftsmanship.

Very few people are willing to pay that for off-the-shelf software, and trying to do it would cause you to get creamed by your competition.

If television is a babysitter, the internet is a drunk librarian who won't shut up.

[ Parent ]
Software Engineering by miker2 (2.00 / 0) #12 Fri Jun 30, 2006 at 04:27:35 PM EST
I prefer the craftsman approach becuase you get something compilable and runnable quick, which gets you into the customer feedback loop quickly.  The SE practices and methodologies I've studied are all document heavy and front-loaded with design, docs, proofs and require a large staff for even mid-size projects just to keep everything in order.

I start a new job in a couple of weeks that's a complete Agile/XP with SCRUM shop and I'm very interested in seeing how that works on a ~70 developer project.

Ah, sociopathy. How warm, how comforting, thy sweet embrace. - MNS
[ Parent ]
Good story by cam (4.00 / 1) #6 Fri Jun 30, 2006 at 01:46:07 PM EST

Freedom, liberty, equity and an Australian Republic

+1 FP by Kellnerin (4.00 / 2) #9 Fri Jun 30, 2006 at 02:37:43 PM EST

"later" meant either "when you walk around the corner" or "oatmeal."
poll by theantix (4.00 / 1) #13 Fri Jun 30, 2006 at 05:31:31 PM EST
Yes I do!

Sometimes it just takes me a while though because I'm at work -- I only have so much time to spend reading diaries no matter how awesome the poll options are!

But eventually I get around to it.  :-)
Everything you wanted to know about Kansas City, and more.

Last place I worked by wumpus (2.00 / 0) #18 Wed Aug 16, 2006 at 06:47:00 PM EST
The EEs and the MEs often worked together, and worked pretty well together.  Oddly enough, we (the EEs) NEVER worked with anyone from software.  Not because of culture/attitudes, but because they programed "other stuff" where anything we did had to be sufficiently compatible with the last stuff that they couldn't change a line of code if they wanted to.


The best-laid plans... | 18 comments (18 topical, 0 hidden) | Trackback