Print Story Post.
Diary
By blixco (Wed Sep 26, 2007 at 01:17:31 PM EST) (all tags)
What we learned is:
*The data is never perfect.
*We're now in the data restoration business.
*If I work seven days a week for hours and hours day and night, I lose my goddamn mind.


Are we there yet?

Every time I started to think we were in the clear with our recovery, something new would happen.  The last thing was, none of the security descriptors for the windows files were copied.  I fixed roughly 80 percent of them by re-running one of three versions of robocopy (version 026 is the only one that will run on Windows 2k3 and includes a /secfix ooption).  Some of the files nevere had them to start with...in other words, they were restored without them.  So we're dealing with it now and forever onn a one-by-one basis.  The good thing is, they fail into a safe mode, more restrictive than not.

On Tuesday after being up all night fixing the files and re-copying a few directories, I emailed my sort-of boss (that's a term of endearment) who's been through this whole thing with me, and said, hey, I'm taking the rest of the week off.  Then I emailed the owner and said the same thing.  Haven't heard from him, but meh.

What I learned about myself and the world since August 30th:

  1. Data is fragile.
  2. The fastest systems and networks are still too slow for the volume of data that any of us typically use.
  3. Technology from the 1970s (magnetic tape) might not be better than technology from the 70s (hard drives).  Or vice versa.
  4. I can work for 28 hours solid and still write coherent emails, including one where I tried to resign.
  5. The thing is, someone had to be responsible for the whole goddamn thing.  And that someone was me: I made the decisions, I did the update that took down the array, I specified the technology, I was the admin on the servers, and I was ultimately the only person on sight who knew anything about the Linux or Windows specifics, the syntax of the storage array, and how to speak to Poland via skype.
  6. Point #5 needs to be fixed.  To that end we're interviewing.  My former co-worker Charlie would be a perfect fit: a storage engineer with an Air Force background.
  7. I've found that I can deal with working 7 days a week and having my house worked on.  "My" bathroom was gutted and work started right when this array failed.  Contractors and such have been in and out of my house for a month.
  8. In the end, these next five days will be fucking sweet, and it will be time for me to make a decision.  I plan on doing nothing.  Maybe driving to some other place in Texas and doing nothing.  No matter what, I will be doing nothing.
  9. However, my sort-of boss has asked me to be available.  That means no camping in Uvalde, no getting lost in the desert in west Texas, and no leaving my cell phone / email client.
< Books I've Read This Year 2007 | BBC White season: 'Rivers of Blood' >
Post. | 16 comments (16 topical, 0 hidden) | Trackback
Just don't forget to post a Friday Playlist. by greyrat (4.00 / 2) #1 Wed Sep 26, 2007 at 01:22:29 PM EST
No. Here. Have a tequila on me and -- No. No! Not a belly shot! Hey! Back off!!!

/sounds of a struggle ensue...




WIPO: NO! We ARE DEVO! by LinDze (4.00 / 2) #2 Wed Sep 26, 2007 at 01:49:57 PM EST
 It sounds like the Boss Man at least understands you really need someone(else) dedicated to storage. Probably about the best end result you could hope for all in all.

-Lin Dze
Arbeit Macht Frei


Available?? by ayrlander (4.00 / 4) #3 Wed Sep 26, 2007 at 02:17:13 PM EST
Haven't you been available enough by being there every day for a month?  How did you resist the urge to suppress violence or laughter?



My sort-of boss by blixco (4.00 / 1) #5 Wed Sep 26, 2007 at 02:29:15 PM EST
was with me the whole time, too.  He's very good at what he does, but if they have an emergency, he'll need me.

I understand that.  I'm OK with it, it just limits me for right now.  When we have someone else in place, I'll be more able to get away.  But for now, this is what we do.
---------------------------------
"You bring the weasel, I'll bring the whiskey." - kellnerin
[ Parent ]

Brave move, point 5 by Breaker (4.00 / 2) #4 Wed Sep 26, 2007 at 02:23:04 PM EST
For which I offer you /pint - many others would have have their shoulders go milk bottle shaped and tried to blame others.

Takes a rare amount of courage, that.




I am nothing by blixco (2.00 / 0) #6 Wed Sep 26, 2007 at 02:30:43 PM EST
if not responsible for my actions. 

It surprises me when people aren't, but I understand the impulse.

I'm having an ode to Atreides at the moment, sipping a caucasian.
---------------------------------
"You bring the weasel, I'll bring the whiskey." - kellnerin
[ Parent ]

Responsible? by thunderbee (4.00 / 2) #7 Wed Sep 26, 2007 at 02:51:53 PM EST
It's a threat now, a way of preventing you from doing something not yet forbidden but clearly undesirable:

"If you do that, you'll be responsible for whatever happens!"

You'd be surprised at the number of people that won't take responsibility for anything. Even if they did it. After being told not to. In front of witnesses. And the video is on YouTube.

Ah, ranting already. It really drives me crazy.

It's just that there is a slight difference between being responsible for something, and taking the blame for something that happened outside of your powers.

You did your work within the constraints you had (no extra backup), something failed that you could not anticipate (you trust the vendor with the firmware upgrade because you had no other choice); and you did everything you could to solve the problem.

Acknowledging and solving the problem is being responsible.
Trying to resign is very Samurai-like (and I definitely can appreciate that), but it's just taking the blame for something you had no power over.

IMHO anyway.

[ Parent ]

That was my take as well. by Breaker (4.00 / 1) #8 Wed Sep 26, 2007 at 03:01:42 PM EST
Could have been easy enough for blixco to try and blame others and could have easily made a case for that.


[ Parent ]

Next to my by blixco (2.00 / 0) #9 Wed Sep 26, 2007 at 03:28:24 PM EST
childhood bible was a copy of Hagakure.
---------------------------------
"You bring the weasel, I'll bring the whiskey." - kellnerin
[ Parent ]

Same for me by thunderbee (4.00 / 1) #16 Thu Sep 27, 2007 at 02:09:54 PM EST
without the bible ;)

And I did resign, losing 60K€, just because the management prevented me from providing good service to the clients (good service that I had sold).

As I said, I can understand you.

[ Parent ]

resignation by Merekat (4.00 / 1) #14 Thu Sep 27, 2007 at 04:15:05 AM EST
IME, the offer to do so can be the only thing that shocks others out of their complacency and into proper behaviour. Going through with the resignation is even more effective;)

[ Parent ]

However, my sort-of boss has asked me to be... by vorheesleatherface (4.00 / 3) #10 Wed Sep 26, 2007 at 04:26:46 PM EST
That's not vacation. That's being on call. You shouldn't even have to use up your vacation days for that. Enjoy your time off dude!

"Of course. I goatse my MP once a week!" - Hulver


And by "enjoy" I mean by vorheesleatherface (4.00 / 2) #11 Wed Sep 26, 2007 at 04:27:30 PM EST
have beerz for me too.

"Of course. I goatse my MP once a week!" - Hulver
[ Parent ]

One thing by jayhawk88 (4.00 / 3) #12 Wed Sep 26, 2007 at 04:41:12 PM EST
That I think most good bosses/people in charge know: Anyone can make a mistake, good employees take responsibility for it and do whatever necessary to sort it out.



hey! by LilFlightTest (4.00 / 2) #13 Wed Sep 26, 2007 at 06:56:38 PM EST
hey said available, not sober!
---------
if de-virgination results in me being able to birth hammerhead sharks, SIGN ME UP!!! --misslake


If there's any justice in the world by gpig (4.00 / 1) #15 Thu Sep 27, 2007 at 10:56:22 AM EST
You'll get a bonus the size of an elephant this year.

---
(,   ,') -- eep
"This option is deprecated, as it is conceptually flawed." -- man psql


Post. | 16 comments (16 topical, 0 hidden) | Trackback