Print Story My data is doomed
Help!
By ucblockhead (Fri Sep 18, 2009 at 12:40:20 PM EST) (all tags)
Please pray for my data.


I have a NAS.  It's a Synology cs407e, and it has served me quite well for a couple of years.  It had four 500 GB drives, giving me 1.5 GB of RAID 5 space.  It has provisions for automatic backup and I have a 750 GB drive attached, so important stuff, like family photos, etc. is backed up.  Stuff I can "recreate", like ripped CDs, movies from [redacted], etc. is not.

I have been pushing the limit, and drive space is cheap, so I recently purchased four 1.5 TB drives and a few days ago I started the slow upgrade process.

Monday: Swap the first drive.  Repair.  Everything AOK.
Tuesday: Swap the second drive.  Repair.  Everything AOK.
Wednesday: Swap the third drive.  Repair.

Thursday was my class, so I didn't get home until late, but wanted to do the last drive because, hey, I was looking forward to 4+ terabytes.  I am not sure exactly what happened, but my theory is this:  There was an error repairing drive three.  After swapping the fourth, the nice GUI said the following:

Drive 1: Initialized
Drive 2: Initialized
Drive 3: Hot Swappable
Drive 4: Uninitialized

Repair was obviously not an option.

With the "oh crap" feeling, I figured I'd better just put things back.  Now here was when I got to reflect on my own stupidity as I had not labeled the old drives and so I had two candidates for drive 4.  I replaced the drive and got:

Drive 1: Initialized
Drive 2: Initialized
Drive 3: Uninitialized
Drive 4: Uninitialized

Oh crap!  Try the other one:

Drive 1: Initialized
Drive 2: Initialized
Drive 3: Uninitialized
Drive 4: Uninitialized

Wuh...  I tried swapping 3 and 4:

Drive 1: Initialized
Drive 2: Uninitialized
Drive 3: Initialized
Drive 4: Uninitialized

Er...  Here I realized I'd cabled the drives wrong.  Stupid not the damn labels.  At this point, I decided to put the drives in an enclosure to try to figure out exactly which was which.  OSX just refused to recognize anything...fortunately, my old eee was able to show me the file system and I was at least able to separate the blank 750 from the one that had been drive 3.

I tried getting things back in their normal spot, but still got:

Drive 1: Initialized
Drive 2: Initialized
Drive 3: Uninitialized
Drive 4: Uninitialized

At this point, I went to bed.  I woke up very early the next morning, googling, and figured out how to ssh to the box and run mdadm.  I was able to query the volume, and it happily told me there were only two drives in the RAID array.  I used it to query drive 3, and got a cryptic message about there being a number collision.

I simply disconnected drive four and got:

Drive 1: Initialized
Drive 2: Initialized
Drive 3: Uninitialized
Drive 4: Not connected

Getting desperate, I connected it back up and decided to go for broke and did a "mdadm --add" to add drive three to the array.  I got:

Drive 1: Initialized
Drive 2: Initialized
Drive 3: Hot Swappable
Drive 4: Uninitialized

Success!  My theory is that once a new drive is connected, the NAS forgets the old drive ID and won't automatically pick it up.  So know I think I just have to figure out which of the two 500 GB drives was the original drive 4 and go through these steps.  I decided that while I was racing to get the kid ready for school was not the time to attempt this.

So the questions are:

  1. Can I actually get the drives into a good state?
  2. Did anything I did in all this actually change any data on the physical drives?
I am praying to that which I don't believe exists that #2 is the case.

Pray for me.

< on vacating | the Right Writing-Wright Rite >
My data is doomed | 21 comments (21 topical, 0 hidden) | Trackback
*hug* by duxup (2.00 / 0) #1 Fri Sep 18, 2009 at 12:53:34 PM EST
Good luck.  I keep meaning to setup a NAS device at home.

____
Wow. by CheeseburgerBrown (4.00 / 2) #2 Fri Sep 18, 2009 at 01:53:10 PM EST
I just keep all of my data under the mattress, in raw ones and zeroes zipped into sandwich bags.


I am from a small, unknown country in the north called Ca-na-da.
Yes, but... by ucblockhead (4.00 / 2) #3 Fri Sep 18, 2009 at 01:56:00 PM EST
You don't have important things, like rips of the first 11 seasons of Stargate SG1!
---
[ucblockhead is] useless and subhuman
[ Parent ]
My Data Treasure Trove... by CheeseburgerBrown (4.00 / 4) #4 Fri Sep 18, 2009 at 02:02:05 PM EST
...includes but is not limited to:

* About a zillion stolen Hollywood movies;
* About 10 stolen Bollywood movies;
* Rips of about half of my legitimately acquired CD/DVD library;
* Pictures of my children coming out of my wife;
* Pictures of my children stretching and deforming into bigger children (gradually);
* Digital masters of my student films;
* Everything I ever posted on the Internet;
* Candid snapshots of my wife's assets;
* At least two unfinished novels;
* Digital copies of just about everything I ever painted, drew or sculpted;
* "Trial versions" of software;
* Preproduction for an animated film I will never get around to making;
* My motion-graphics portfolio;
* DVD masters of anything original I've ever burned to DVD;
* Priceless, invaluable reams of crap I never bothered to sort.

If I lost one iota of it, I'd up and die on the spot.


I am from a small, unknown country in the north called Ca-na-da.
[ Parent ]
I am impressed by your discipline by infinitera (4.00 / 1) #6 Fri Sep 18, 2009 at 02:32:05 PM EST
I just lose data all the time and am resigned to it.

[…] a professional layabout. Which I aspire to be, but am not yet. — CheeseburgerBrown

[ Parent ]
If by Gedvondur (4.00 / 2) #5 Fri Sep 18, 2009 at 02:14:16 PM EST
If you didn't actually initialize or try to rebuild the array since then, it should be fine.  Unless there is a microcode issue.

What you are doing is HIGHLY risky.  I know it should work in theory, but swapping one drive at a time, with different sizes and firmware is NOT advised ever.  What you really needed to do was empty the array, swap all four drives, reinitialize the RAID and re-copy the data back.

You will also find that different RAID array controllers will handle this kind of abuse in different ways.  Yours must be fairly forgiving, many arrays I have worked with would have shit the bed long before then.


 

Gedvondur




"...I almost puked like a pregnant StackyMcRacky." --MillMan
Designed for it by ucblockhead (2.00 / 0) #7 Fri Sep 18, 2009 at 02:39:04 PM EST
This NAS is specifically designed to be upgraded this way, so I assumed I'd be fine.  I am still an idea for not backing everything up, of course.
---
[ucblockhead is] useless and subhuman
[ Parent ]
Wow by Gedvondur (2.00 / 0) #8 Fri Sep 18, 2009 at 02:45:36 PM EST
That's wild.  Designed for it, huh.

I would have never guessed.  Scary fuckin' hardware, that.



Gedvondur

"...I almost puked like a pregnant StackyMcRacky." --MillMan
[ Parent ]
Yeah by ucblockhead (2.00 / 0) #11 Fri Sep 18, 2009 at 03:10:46 PM EST
It's a really slick device...unfortunately, it failed the idiot-proof test.
---
[ucblockhead is] useless and subhuman
[ Parent ]
Is St Jude appropriate? by georgeha (4.00 / 1) #9 Fri Sep 18, 2009 at 03:03:26 PM EST



I sure as hell hope not (nt) by ucblockhead (2.00 / 0) #10 Fri Sep 18, 2009 at 03:10:18 PM EST

---
[ucblockhead is] useless and subhuman
[ Parent ]
Don't you mean by ad hoc (4.00 / 1) #12 Fri Sep 18, 2009 at 03:56:46 PM EST
At least that's hopeful by georgeha (2.00 / 0) #13 Fri Sep 18, 2009 at 03:58:10 PM EST



[ Parent ]
I cannot possibly by technician (2.00 / 0) #14 Fri Sep 18, 2009 at 04:06:21 PM EST
give you any advice on this.

Worse yet, I have had zero issues with mdadm, so I can't say for sure what sorts of trouble you're in.

Best of luck. Normally metadata, even if it is overwritten, is backed up, so there may be / should be many things you can do with mdadm to get to the right set of data....

My intuition by dark nowhere (2.00 / 0) #15 Fri Sep 18, 2009 at 04:56:47 PM EST
which is based on experience with many people being far too irresponsible with their data (and other things), says your guess is very likely to be exactly correct.

Label your drives and I'm sure the Loa will guide your data into a re-integrated state. You could also badger the vendor for tips.

Chill out, snowflake.

From where did you run mdadm? by chuckles (2.00 / 0) #16 Fri Sep 18, 2009 at 06:52:12 PM EST
Does the NAS give you a shell, or did you hook up the drives to some other computer?

"The one absolutely certain way of bringing this nation to ruin [...] would be to permit it to become a tangle of squabbling nationalities"
NAS by ucblockhead (2.00 / 0) #17 Fri Sep 18, 2009 at 09:26:50 PM EST
The NAS gives me a shell.

At this point, I've had no luck getting it working, so I am throwing myself on their support mercy.
---
[ucblockhead is] useless and subhuman

[ Parent ]
I would do whatever vendor support says by chuckles (2.00 / 0) #18 Sat Sep 19, 2009 at 06:31:06 AM EST
but if that doesn't work, and you have nothing left to lose, I have an idea.

"The one absolutely certain way of bringing this nation to ruin [...] would be to permit it to become a tangle of squabbling nationalities"
[ Parent ]
Why didn't you mark the original drives? by BadDoggie (2.00 / 0) #19 Sat Sep 19, 2009 at 05:41:27 PM EST
I deal with a lot of delicate HA shit at work and this is the best advice I can offer you: Spend a few hundred bucks on some 500GB drives you won't need very long. That's going to be a lot cheaper than data recovery from CBL.
  1. Mark each of the original NAS drives with a letter.
  2. Use GHOST or some other similar physical layer copier to do a 1:1 copy with live drives, each connected and running (USB or on the rails doesn't matter)
  3. NAS shouldn't care about the order as long as all drives are there, but if it does, then:
      i) Make a matrix (ABCD, ABDC, ACBD, etc) and start trying each order out.
      ii) Plug 'em all in and see what happens
      iii) Lather, rinse, repeat.

The root problem is most likely a duff 1.5T drive or a driver problem which isn't allowing the damned thing to see all the 1.5T drives because of their size (they cross the 1T threshold which is similar to the old 3.2G, 8G, 32G and other HD size thresholds of the past).

Good luck, and if you need I can PM you my phone number. Baby's sleeping so I can't, natch.

woof.

OMG WE'RE FUCKED! -- duxup ?

Why didn't I label? by ucblockhead (2.00 / 0) #20 Sun Sep 20, 2009 at 10:00:56 AM EST
Because I am an idiot, clearly.

My problem now seems to be that I don't know the right mdadm incantations.  Since drives 1 and 2 are definitely right, I know I've gotten the right combination.  It seems as if once a drive is removed, it gets marked "hot spare" if I put it back.  I don't know how to undo that.

I am sure it isn't a driver issue as this drive model is marked supported on their site.
---
[ucblockhead is] useless and subhuman

[ Parent ]
Let's see if we can reconstruct it by Herring (4.00 / 1) #21 Sun Sep 20, 2009 at 12:25:47 PM EST
What did you data being with? Was it 0x20?

christ, we're all old now - StackyMcRacky
My data is doomed | 21 comments (21 topical, 0 hidden) | Trackback