February 25, 2007

Duplicity, encfs, sshfs.... off site storage

So now I have a good RAID 5 NAS device for my storage here, but that's not good enough - what about offsite backups? In the past I've played with a few things, tapes, cd's, dvd's and various other media and moving them offsite to my parent's house. It's works, and has the benefits of a historical archive but the problem is the backups are slow to create, don't seem to work quite well when I tried to automate the creation of the backups, most annoyingly, 4.7 GB on a single DVD isn't enough to store everything.

Then I thought about my hosting account at 1and1 that gives me 250GB of storage... DUH? Off site storage that I'm already paying for! Since I already mirror everything I have stored at 1and1 locally here which I update every night with rsync.... why I don't I do the same thing in the other direction... it's so obvious! Why did I think of this earlier!

I just need to put all my data that I want to be backed up in a folder (where it pretty much already is) and rsync it in the other direction over to my 1and1 account, essentially a simple 5 sec rsync command. The problem is that I have sensitive information that i don't want to be sitting wide open on untrusted machine somewhere.

So I did a bit of research and discovered duplicity, this a very cool idea... basically takes your data, and encrypts it on the fly and uploads it to an outside ftp, or scp (via ssh) machine. After the first run, it will do incremental updates only. Perfect!

However, as I played with this, I realized 45GB for an initial full backup will take a few weeks with my cruddy cable modem upload speeds, and if it dies (which it did once) then there was no resume. This is clearly a problem.... you cannot rely on a home cable connection to give you that kind of reliability, at least in this neck of the woods.... so back to researching other alternatives.

Next i discover something very cool, sshfs and encfs. sshfs mounts a remote filesystem locally over ssh, encfs provides an encrypted filesystem.... A simple apt-get install enfs and apt-get install sshfs got me both in a few seconds. They are 2 separate utilities, HOWEVER if you first setup a sshfs mount, and then create an encfs directory over the sshfs mount, you end up with what essentially is an filesystem on a remote machine that stores the data encrypted... perfect exactly what I need... of course, you can then run rsync to move your data over to!

Because this creates an encrypted copy of your data stored on a remote machine that can be updated with rsync, it has no problems with resumes and is smart enough to only transfer changed files after the initial transfer is done!

I ran into a problem while trying to create the encfs over the sshfs, I suspect it's a permission problem prevents me from writing to the encfs mount and I didn't want to spend time debugging, so I ended up instead just building a local encfs filesystem where I moved all the data I wanted to be backed up into. Now I schedule rsync to copy the encrypted directory over to the remote machine, and this gives me same identical net result.

It's not quite as elegant as my original plan, but it works the same way... in fact i'm actually able to use my original method to mount and read the remote directory using encfs over sshfs, just can't write to it.....
-Update: "This was easily fixed by updating to a new version of the fuse library, apparently it was a known issue... i tearing my hair out for nothing"...

Anyway, and so I finally have some peace of mind that my data is a little more safer.