I've been meaning to blog about the importance of backing up your data, and how hard it is to do a good job, and how important it is to not rely on a single means of backup. And how I have managed to always have a backup when I need it, despite disasters galore.
This is not that post!
Back in 1986, I was laid off, from an interesting company called Symbolics.
Up until this point, I'd always used someone else's computer -- starting way back in 1970. And while I'd been responsible for duping my own decks of punch cards, backing up the hard drives had always been someone else's responsibility.
I'd always used someone else's computer, because computers cost millions -- or they were so tiny and slow and unreliable to be uninteresting to me.
But that had recently changed, with the release of Mac II. So the first thing I did when I got laid off was to go out and buy a computer. Initially it only had about 80 MB of space or so -- but I was soon able to buy a 300 MB hard drive for $1000.
And so the spiral began.
Backing up 80 MB onto floppies was very difficult. But backing up a 300MB hard drive onto 1.4 MB floppies might seem horribly tedious and a bit expensive.
The reality was quite different. Getting a full backup was nearly impossible. See, with that many floppies, the odds of nothing going wrong was nil. Something always went wrong. Bad floppies. Damaged floppies. The wrong floppy, or a lost floppy.
Or I just ran out of time and/or floppies.
And so, I bought my first removable-media disk drive, the Iomega Jaz drive. (1 GB, then 2 GB). I finally got rid of those a few weeks ago!
When I bought my first personally-owned PC, back around 1995, I bit the bullet and paid big bucks to get a tape drive to do backups. CDs were what most consumers backed up to those days, but you'd spend a fortune and a lifetime doing a backup that way. Result -- real backups were very rare, and I just made sure to have a second copy of truly critical stuff on another hard drive somewhere (such as an IOMEGA Jaz), or possibly a CD if it was really important.
I ended up buying more and more and more disk cartridges....
I used the tape drive even less as time went on -- and one day, I noticed it didn't work.
Fast forward 14 years. There are more working computers in the house than there are people -- not even counting such devices such as phones and TiVos -- the lowliest of which outshines any of my computers from even a decade ago.
Now, I type this on a laptop I just upgraded to 500 MB. That's a half terrabyte. Back in the late 1970's, there was a computer put on the ARPANET just to handle archival storage -- available to the entire ARPANET. It offered an AMAZING 1TB of available storage, via robotic tape libraries and arrays of staging hard drives. It had to cost millions.
It was excruciatingly slow, too -- but you could automate backups to it, like modern online backup!
Back to present day again... my local disk backup farm has grown. As of this morning, it had grown to include:
- A 160 MB USB external Maxtor drive (retired).
- A 120 GB hard drive in a desktop computer (computer & drive retired from service)
- A 500 GB hard drive in a desktop computer (retired from backup service)
- 2 500 GB external USB drives (Western Digital MyBook)
- 2 1 TB Network attached drives configured as Raid 1 (mirroring), Western Digital MyBook World Edition II)
- 1 1.5 TB Worldbook configured as Raid 1.
- 2 2 TB USB Western Digital My Book Mirror Edition (again, Raid 1 mirroring)
- About 6 retired laptop drives, holding the data that was there before upgrading to larger drives.
The problem is -- hard drives die. All too often. And if it's a backup, not only do you lose your backup, you also stop getting backed up. (And consumer backup software isn't all that good about telling you about that.)
In fact, out of the above farm, at least 4 drives were dead, representing 2.5 TB of storage.
The Drobo is like RAID in that it puts disks together for redundancy. Unlike RAID, however, it doesn't require that the disks be alike, or even similar.
The first thing I did was to stick in the new 2TB Hitachi drives and fire up the Drobo and Droboshare. Within a few minutes, I had 1.8 TB of storage available on the network. With just 2 drives, Drobo acts pretty much like mirrored drives, keeping two copies of everything. But things get better when you add more drives...
Last night, I took apart one of the 500 GB external MyBook drives. A rather challenging task, yielding me one raw 500 GB drive, which I stuck in the new USB enclosure. Success! The drive worked fine there; it was only the MyBook enclosure which was dead. I copied the data to the Drobo -- and then stuck the drive into the Drobo, bringing the total Drobo storage available to 2.26 TB.
I then turned my attention to the second 500 GB MyBook. This time, I had a little difficulty with the first step, a "simple" screw removal, but the screw was stuck, and painted, and it stripped. Rather than sweat it, since I had a spare cover (and no intention of reusing the enclosure anyway), I opted for destructive disassembly techniques, and quickly had it open with a minimum of aggravation.
This drive, when I powered it up, made a faint clicking sound, so I was not expecting it to work -- and it didn't. This one I disassemble to show the kids...
Next, I turned my attention to the dead MyBook World Edition, with two 500 GB drives inside. It didn't light up its light, and I figure either its software was corrupted, or the enclosure (which runs Linux, BTW) was dead -- so the drives were probably OK. Unlike the MyBook, opening this is a fairly straightforward task, and it is designed to make it fairly easy to replace drives. In this case, I just wanted to get the drives out.
Since I was using Raid 1, either drive should have the data I was looking for. I grabbed Drive A, and stuck it in the USB enclosure, and hooked it up. Windows 7 recognized the drive, and loaded the device drivers. But it didn't show me any filesystems. Window's Drive Manager didn't recognize the partition types. I had my suspicions, and installed Acronis Drive Director to take a good look.
[As I type this, I am informed that tonight's backup just failed....sigh...]
The drive had a Linux swap partition and three Linux ext3 partitions. Two of them are for the operation of the system (one would be the root filesystem). The big one is the data, and that's what I'm interested in.
But Windows doesn't understand ext3. So I fired up a Ubuntu (Linux) VMWare virtual machine, and allocated that USB drive to that virtual machine. I was able to quickly determine that I could see, but not mount the drive from Ubuntu. I then googled a bit, installed mdadm and the necessary raid support, found a nice post with instructions for recovering RAID 1 from a single drive, and found my data, intact!
Next step was to copy it to the Drobo! To do this, I had to mount it as a cifs share. I found Ubuntu doesn't come with cifs support installed, so I had to google and find what to install. Once I got past that hurdle, I mounted the share, and proceeded to do an rsync copy from my recovered drive to the Drobo.
That's still going on. One thing is that the Drobo is not fast. I wasn't expecting it to be. I bought it for its expandability and simple recovery. I expect to save money over the long run, by buying new internal drives, rather than larger RAID 1 drives. But the Drobo isn't going to be the sole home for my backup data.
The drive I just recovered will go into a drawer for now, as a backup. When I fill up the Drobo, I'll stick it in there and increase the space by 500 GB, for a total of about 2.7 TB. Then when that's full, I'll buy a bigger drive -- 2 TB or more, and replace one of the 500 GB drives. Etc.
The other drive -- drive B -- will go to replace a failed drive in my other 1 TB MyBook World Edition. That one hasn't lost data, but I've left it unused because I regard it as an unsafe location without the redundancy. This sort of failure is exactly why I use RAID 1!
But it doesn't really help with reliability -- just recoverability. Note the score:
- 1 dead non-RAID enclosure
- 1 dead non-RAID drive
- 1 dead RAID enclosure
- 1 dead RAID drive
A couple of problems remain. Some of the backups I've just recovered use software I no longer use, and which is no longer available. I have copies -- but I'll need to run them in an older OS in a virtual machine, to be able to recover anything from them!
And I didn't quite think think things out when I set up the Drobo, and I only configured it for 4 TB filesystems. I'm pretty sure I want to pay the startup penalty, and go for a larger filesystem size -- 16 TB. I can live with it taking 16 minutes to get started up after a reboot. It's a lot better than having to move things around between filesystems because one got filled up.
That means I need to find space to temporarily stick what I've already put on the Drobo, then reformat to 16 TB max, and restore. (I'll use the drive destined to replace the dead 500 GB in the WorldBook). That'll take a while...
Then, when I max that out, it'll be time to get a second, bigger Drobo (or other technology).
It never ends.