Camera Labs

Hey all,

Been doing photography commercially for a couple of months now, and I'm in need of a competent and price efficient backup system. My needs are as follows:

- Each commercial project typically uses 150-200GB of RAW and edited files (my camera is 24MP, so, that's why )
- I don't need to backup my computer per se, just each individual project, with its own lightroom catalog.
- I want to be able to delete the project from my main PC after I've delivered but still keep a reference catalog so that I can go back and view images offline at a later date.
- I want to be able to take my backups and keep them offsite.
- The media should be able to last undisturbed for many years without failure.

My constraints are:

- The price over the long term has to make sense since I will be building in the cost of backups into my pricing structure.
- Uploading to the web is not feasible due to the amount of data I generate on a monthly basis, even though I have a 10MB connection, I would still never catch up.
- I don't have access to a controlled environment, so I have no control over humidity, temp., etc.

I've been thinking of 3 potential solutions, namely:

- Tape: Investing in a LTO-2/SDLT/DAT320 drive. The drives are fairly costly, but tape media cost isn't so bad. Tapes are fairly easy to store as well, but they are slow and difficult to verify file integrity.
- Hard disk: Just have an old PC with 2 dockable hard drive bays, which would have 1 disk as the offsite backup, and the other as an onsite backup (or offsite at another site). I would do incremental backups of the project while I work, and at the end do one big backup of everything.
- Blu-ray: More expensive than tape and harddrives per media, but they do take up much less space, and since they aren't mechanical, nothing to really break (although I've had a lot of CDs go bad on me due to rapid deterioration).

As I said, I don't need any fancy raid system (which I can build anytime) since I only need to backup discrete projects in its entirety, maybe with revisions.

So, what would be the best option? I'm leaning towards Hard disks as I'm familiar with the technology, disks are easy to get here and capacity isnt an issue. Hard disks should also last 5yrs undisturbed which would be fine with me as well. Given how I work, I'll probably burn through about 20 300GB hard drives a year (given that I'm keep 2 copies of each project) and at around 50.00 per drive, that isn't all too bad.

_________________
I wish there was a way to randomly cycle through click-able thumbnails of my most recent images :'(

http://www.laroygreen.com (under construction)

Hi primitive,

I'm no expert but my guess is that hard disks are probably the way to go. I used to use tape as my primary backup media years ago but it was slow and random access was horribly slow. Another factor to consider is that five years from now it might be quite difficult to source a new tape drive to read your old media should the drive need replacing. That is probably not going to be an issue for hard disks as even as the world moves on it's still possible to buy controllers for the older standards as they are cheap enough to make and cheap enough to buy should your interface of choice look like it will be phased out.

Bob.

_________________
OM-D E-M5 + ED 12-50mm, H-PS14042E, H-F007014E, L-RS014150E, M.ZUIKO DIGITAL 45mm 1:1.8.
Gitzo GT1541T, Arca-Swiss Z1 DP ball-head.
Astrophotography: TEC 140 'scope, FLI ML16803 camera, ASA DDM60 Pro mount.

My gut feel is HD too. Manually sync to it as needed. I'd probably go USB to make it easier to move around. 1TB offers about the best cost/capacity point last time I looked, if you don't mind having multiple projects per disk.

Kept powered off they should last a long time. Even running 24/7 on average you'll get more than a couple years off one.

_________________
Canon DSLRs: 7D, 5D2, 1D, 600D, 450D full spectrum, 300D IR mod
Lenses: EF 35/2, 50/1.8, 85/1.8, 135/2+SF, 28-80 V, 70-300L, 100-400L, TS-E 24/3.5L, MP-E 65, EF-S 15-85 IS
3rd party: Zeiss 2/50 makro, Samyang 8mm fisheye, Sigma 150 macro, 120-300 f/2.8 OS, Celestron 1325/13
Tinies: Sony HX9V.

Looks like HD is the way to go. I actually used to use tapes quite a bit at work as they are basically the de-facto in sharing seismic data. Trouble is, you often times get tapes that are non-standard to your drive, even though the tapes are of the same form factor. Basically, tapes will probably never go out of use, but the market is so fragmented, and there are so many incompatibilities that what Bob describes is a real scenario.

I also thought about having multiple projects on 1 drive/tape, but because of how I am (lazy) I would be tempted to leave the drives home until they get filled, which is exactly what I don't want to do. Plus, it might make it easier to catalog everything as I can just write the name of the project on a disk label. Downsides is that I'll have 4 disks for 4 projects instead of 1 for 4 projects, so I'll take up more physical space, and probably spend more money as well. Still haven't decided yet.

So, the plan would be to have a drive that I use to backup only raw data and that would immediately be kept offsite (at a friends home ). Then I would leave two drives in the PC and have those mirroring each other, while having backups of revisions as I work. After about 3 months or so (which is usually how long I take to complete a project), I'll then make a final backup, pop out both drives and send them to two different locations...rinse and repeat.

So, if I lose my files before a full backup, I will have the raw files which were backed up already on the first day. I actually feel pretty good about this. Thanks guys!

_________________
I wish there was a way to randomly cycle through click-able thumbnails of my most recent images :'(

http://www.laroygreen.com (under construction)

I'm also going to go with hard drives. What I'd do - buy a 2x 250gb HDDs for each project (that's between $100-120 for the 2 drives), and also have 2 or 3 HDD hot swap bays installed in your work PC - something like two of these or this. Just pop the 2 drives in when you start the project, and have a single folder that you have auto sync to each drive every night. When you're done with the project you can just take both drives out and put them in storage (one at home, one on site or wherever you keep them.)

As long as you keep the drives in a safe spot, and they aren't ever used except when you need to access the backup for some reason, they should last a long time.

Blu-ray would get very pricey very fast. A pack of Dual Layer Blu-ray disks (50GB per disk, 25 disks per pack) is about $315. Assuming 200-250GB of data per project, that's 5 disks per backup copy, 10 disks total. That comes out to $126 per project for backups. Not to mention the higher cost of a dual layer blu-ray burner vs. HDD hotswap bays and the fact that a single scratch could wipe out a large part of your backup. Blu-ray just doesn't seem like it will be as effective in the long run. Oh, that doesn't take into account burn time as well, which will take much longer than just doing automatic backups once a day.

All that being said, the cheapest in the long run (and most space effective) solution would be Tape. Get an LTO Ultrium 2 (400GB tapes) for between $850-$1000, and spend $30 per tape. This gives you plenty of headroom in case you have a larger than normal project so you'd still be able to have 2 backups on 2 tapes (rather than 4 tapes for 2 backups on a 160gb tape). Tape will also be around for a LONG time (specifically because its fairly cheap once you have the drives, and its a fairly secure storage medium). And they don't take up much space (unlike hard drives which will get bulky very quickly). I might be changing my mind....

As long as you go into tape knowing what size you'll generally need (since each drive can only use one size/format of tape), and that you'll only ever buy one format/size of tape, its actually not a bad way to go.

_________________
Canon EOS T1i | Canon 18-55mm f/3.5-5.6 IS | Canon 50mm f/1.4 USM | Canon 70-200mm f/4L
Lowepro Slingshot 200

http://www.jessegamble.com

Thanks for your input Stig. I've pretty much eliminated Bluray. Its just too expensive and takes a lot of time and probably isn't that safe to begin with.

My biggest issue with tape is the restore time which is exceptionally long, even if you only want to recover one file (unless the file is at the start of the archive) and the fact that tape formats tend to go in and out of flux. For instance DAT320 drives will not read DAT160 or DAT72 tapes and those drives are getting harder to find new. So, if you had a bunch of DAT72 tapes, and your drive died, you'd be in a bind. This happens all the time with tapes. Also, I would need to import tapes as opposed to just going to the computer shop 5 mins from me to get a hard drive.

Thanks for the links, and thats pretty much what I was aiming for. Just have some SATA hotswap bays in a low spec computer (in a large tower) connected to my network, that way I can backup from my desktop and my laptop easily. I'll also probably be running linux on it as well, much cheaper and I'm actually an advanced Linux user , so no setup issues there.

_________________
I wish there was a way to randomly cycle through click-able thumbnails of my most recent images :'(

http://www.laroygreen.com (under construction)

I setup the following for a mate's graphic design business.

We have a small server that is used internally for testing web development projects. It also acts as a common file server for the office.

We have a 1.5TB RAID1 array setup simply using Linux software RAID. Mount that as /export and have www and samba shares stored on there.

Using cron jobs setup on their Macs, the two designers' home directories are rsynced to the server nightly.

Now we have a 1.5TB USB/removable hard disk used for offsite backup. Using udev rules, whenever the disk is plugged in (say, in the morning), the following happens:

The disk is mounted at /media/backup
A bash script is called that does an rsync to copy /export onto the removable hard disk.
Once the backup is complete, it unmounts the drive, and emails the guys upstairs telling them it is safe to remove.

They then take it home with them.

Now, whether you need all that automation is up to you. They are designers, and have exactly zero Linux knowledge. So, with a bit of experimentation they got a dead simple backup system that they don't have to do anything with, other than remembering to bring the drive to work.

(sorry to everyone else, but primitive said he was an advanced Linux user, so this post may be fairly technical).

What I was thinking of was to:

- Have 2 hot swap bays (not including an os disk, and other fileserver discs). The hot swap bays would be for backup (which would be mirrors of each other). I would simple have it mounted and shared automatically (although I don't ever expect the bays to be completely empty).

- On windows (which is where I do most of my work), I prefer to use rdiff-backup as I've had some very good results using it on Linux (and its saved me many times, plus I know python pretty well). My second option would be Cobian backup, which I also highly recommend.

- Before the files get backed up, they would be encrypted with my gpg key (or I can just encrypt the entire destination drive using truecrypt or something else), simple bacause these files aren't gonna be kept with me so I want to make sure there safe. This would be done incrementally, over the duration of the project, so that I can always access revisions of files (not so much for binary files, but even then, it can save you a bit of work in the long run).

- When the project is done and dusted, I would simply do a dump of the entire project directory to the backup drive. Once thats done, and I'm sure everything is on the up, i'll eject the drives and get them offsite (properly documented also, since it makes no sense if you can't identify your data).

- I'll then delete the project from my main pc and move on to the next thing.

Of course, there are many ways to do this, but, this is how I envision myself doing it. I don't need too much automation since this setup isn't very complex.

Thanks for you input and thanks for getting me thinking about implementation.

_________________
I wish there was a way to randomly cycle through click-able thumbnails of my most recent images :'(

http://www.laroygreen.com (under construction)

Just make sure you keep your GPG keys safe!

Agreed, you don't really need all the automation. To be honest, udev is kindof a pain to figure out and get working anyway.

At home I have work stuff in a git repository (really showing my geekiness now), and photos and other things simply stored on a RAID1. I just have a script that backs everything up to a Dreamhost virtual private server, because I'm too lazy to deal with physical offsite backups.

Wish my connection was fast enough to keep everything in the cloud, since I too am lazy . As it stands, I generate about 400 GB of data per month. I WOULD NEVER BE ABLE TO UPLOAD IT ALL!!!!

And even if I did upload it, my download speed is 10% of my upload speed, so it would take 10x longer to get my files back...as I say all the time "Mi caan badda wid dat"

As for my GPG keys, those might not be the way to go actually. I use it all the time, but you also have to update your keys ever so often (even though I have the same one from 1998). It's probably best to just encrypt the drive using an opensource software solution like EncFS, Truecrypt or something similar.

_________________
I wish there was a way to randomly cycle through click-able thumbnails of my most recent images :'(

http://www.laroygreen.com (under construction)

Yeah full drive encryption seems like the best route, considering you want all your projects to be encrypted anyway.

And trust me, if I were generating 400GB/month there's no way I could do online backups either!

_________________
Recent pics on flickr


Canon 500D, 18-55mm IS, 55-250mm IS, 50mm f1.8 II, 430exII

You could also consider going down the SSD (solid state drive) route. Whilst not the cheapest technology at the moment prices will drop and capacities increase over the years. The advantage of this route is that there are no mechanical parts to cease or fail. I have know HD units not spin up if left in cupboards for months. SSD capacities are up to 512GB per drive and they are physically smaller/lighter than standard drives as well as using less power.

If you purchase them in bulk I am sure you could get some discounted prices and you are looking at about 20 512GB drives a year to store two copies of your 400GB/ month.

_________________
Nikon D90
Nikkor AF-S DX; 18-105 f/3.5-5.6G VR, 55-300 f/4.5-5.6G VR, 35mm f/1.8G
Speedlight SB-700

http://keystrokesukimages.com

I think the MTBF (mean time between failures) on SSD's is actually lower than that of your average HDD, let alone an enterprise quality drive. Basically because of the way SSDs work, there is degradation in the memory they use every time you write/delete something on the drive. Some of the cheaper drives (not that any are "cheap" yet) have a demonstrated threshold of 200,000 read/write cycles before unrecoverable files start showing up because of read/write wear. A regular old hard drive, assuming its treated well, will last FAR longer than that - we're talking millions of read/write cycles. There are new "AV" hard drives (audio/video, built for constant use in a DVR or other media situation) that have a MTBF of 1 million+ HOURS of constant uptime.

And, for now, the cost of drives large enough to hold 200GB+ of backup files (twice, for redundant backups) is going to be in the $1500+ range per project.

_________________
Canon EOS T1i | Canon 18-55mm f/3.5-5.6 IS | Canon 50mm f/1.4 USM | Canon 70-200mm f/4L
Lowepro Slingshot 200

http://www.jessegamble.com

You really need to look at specific drives.

Yes, the read/write cycle of an SSD might be lower. These are calculated by rewriting the same address over and over. However, modern SSDs do wear levelling in the firmware. What this means is if the computer repeatedly writes to an address, the SSD actually remaps that to a different location on the disk each time. Practically speaking, this means the MTBF of the drive is a really long time.

The other advantage is an SSD will typically start failing with write errors, but the drive will still be readable. A regular old hard drive may catastrophically fail without warning at any time.

However, in this case I think SSDs would be both an inefficient and extremely expensive way to manage backups.

_________________
Recent pics on flickr


Canon 500D, 18-55mm IS, 55-250mm IS, 50mm f1.8 II, 430exII

Digging up an old thread, hope that its not a no-no here.

Why not buy a small form factor external (portable) hard drive for each project, or for 2 projects since the norm seems to be 500G right now. The WD Passport Essentials seem popular. Dunno what this would mean long term wise though.

Matt