Skip to content

Storing Digital Memories (Part 3)

Over a year ago I started a multi-part series of postings about storing digital memories at home. In part 1 I wrote about the importance of frequently using archived files so they don’t get lost due to silent corruption. In part 2 I described how I moved my personal archive into my living room and to make it really easy for my family to view the archive.

This exercise already paid off big time: It turns out one of the most popular ways of using the archive was to simply run a slide show that displayed all archived photos (about 15,000 of them) in random order. During a particularly long run, I noticed that some of the photos had discolored stripes! When I interrupted the slide show and looked at the photos individually, I watched in horror how files would simply disappear! Very distraught, I unmounted the archive, switched off the disk shelf, and tried to figure out what to do next. After an hour or so of great upset I restarted the disk shelf and mounted it again. To my great relief, all photos looked normal, and the files that had previously disappeared were back again. I suspect it was a disk controller problem that only occurred after prolonged usage — maybe due to overheating. This would also explain why the corruption didn’t break the mirror, since I had connected the mirrored pair of disks to the same controller. As long as I didn’t let the disk shelf run too long, I seem to be fine. Of course, for the long term this was unacceptable. I also started to have my doubts whether my soft raid setup (using Apple’s software) would notify me if one of the disks goes bad. Recent research results show that the interconnect (controller and firewire cards in my case) is a dominant source of corruption (see here), and that mirroring (or any other RAID-based checksumming) is not enough to protect against important sources of data corruption (see here).

So last January I finally found a product that inspired confidence in me: Drobo. While probably not incorporating the latest research results (although I did send them pointers to the above papers — so maybe soon) it does put a lot of emphasis on expandability and — most importantly — immediate notification when something goes wrong. My tests confirmed that bright LEDs would quickly tell me which drive to replace, and long durations of using the archive did not introduce corruption. As a nice plus: the archive is now a lot more quiet — so I could run scrubbing programs on it without waking up any guests who usually get to sleep in our living room.