My Online Photo and Media Backup Service

Recently, a little HW mishap had me think hard about my home backup policy. I think most people today have the similar nightmare that one day all of their data will disappear and that they wont be able to find their backup (insert your favorite backup media) to restore it, or at least all of it. Of course I am not talking about some mission critical enterprise data at some Corporation X but just your own personal stuff that is stored on one or few PCs at home.

Than I was thinking what would I miss the most if say, my house would burn down and with it all the data (and I did not have to go to such extreme situation when my own mistake has almost cost me my RAID 1 style “backup”.) Then I figured, the greatest tragedy would be for sure to lose all of the photos my family and I have taken over the last 10 years with the digital camera. Now if you are worried about some other kind of data e.g. your last tax return than that's OK too as far as you are worried because people who don't worry at all, usually just don't realize what they will lose until it happens. (The loudest cussing I heard in my life was when HD disaster struck my next door neighbor…. may his PC rest in peace now.)

This has made me look at a few solutions available today to see what are my options and how much time and money will they cost me. Being cheap and lazy as most people are, I did not opt for the enterprise version of (insert you favorite backup SW here) plus some off-site tape backup HW. Besides, I somehow never feel good about creating and storing some hundred gigabyte file even if its split into smaller but meaningless chunks. Thus I looked around for some online backup solution which seemed natural choice especially for my treasured photographs.

After looking around I decided that Mozy.com seems to meet most of my requirements. I was ready to sign up and dish out 4.95$/month/computer when I realized that I already pay 4.95$/month to my web hosting provider. So I looked at my hosting plan and found out that the SPACE and BANDWIDTH are unlimited and I do have FTP access there anyway so why not make use of this and just FTP all of my pictures there, and while all my pictures would be there I could look at them using some PHP photo gallery script my hosting provider offers preinstalled as a BONUS. What a novel idea? ;-)

So I clicked on “Cancel Payment” button on mozy sign up page, and started to think what is the best way to get my pics from my home computers to my web host via FTP. Sounds easy enough but I did want to have this fully automated and scheduled on regular basis so I would never have to think about this issue again. Using some simple shell script with TAR and scheduled by CRON (Win translation: using DOS batch script with ZIP or Windows Backup service ) should do it but it had two main problems:

1) Creating some huge file (possibly split into smaller chunks) and FTPed is out of question due to disk space requirements and FTP time outs due to some network issues such as slow upload speed of my ADSL.

2) Streaming the file during the TAR process is only possible on linux and still has problems with network issues and FTP not being able to resume upload if such problem occurs.

At the end I decided I could just upload each picture separately mirroring the complete directory structure on the host side (probably what Mozy does as well). Thus my photos could be also directly accessible by some PHP gallery script for direct viewing (and thumbnail creation). So I created simple specification of the SW that would need to do this job:

SPEC:

1) Copy each picture only once to FTP destination using same folder structure, and once backed up do not upload this picture anymore in the future unless rule number 6)

2) If there is error during upload, retry few times and if still not successful mark the picture as not backed up for possible retry during next scheduled backup.

3) Find all pictures in all subdirectories given one or more local folders e.g. C:/MyPictures

4) Don’t upload certain folders and files (filte thumbs.db, picasa.ini, etc.) - must have file/folder filter.

5) Keep track of what is backed up and where.

6) If the file has been modified detect this and update the backed up version too.

7) Must work on Windows and Linux since I use both at home.

8) Be simple enough to implement in few hours without buying any commercial SW.

And after few hours of work and some web searching I came up with the following: (to be continued)