John's Blog

I’ve started using duplicity combined with Amazon’s S3 to backup this server. Duplicity is an amazing application that makes backups simple.

Some of the features I like best about duplicity are: it encrypts the data, compresses it, splits it into manageable chunks, does incremental backups, and can backup to a variety of destinations. When you restore your data it takes care of applying the incremental backups to produce the final files. Also, duplicity can be used to restore previous versions of stored files.

Overall I’m very happy with using duplicity for pushing automatic backups to my S3 account. To make it easier to use and so that I can run it from a cron job I’ve written a simple bash script to handle calling duplicity with the correct options. The script also dumps my MySQL databases and pushes them separately to a different S3 bucket.

Following is a variant of the script I’m using on my server. This one is a bit more generic and allows for MySQL backups to be turned off as well as some basic dependency checking. It also allows for some simple option tuning. The version I use has most of the options hard coded in the appropriate place instead of putting them into variables. This script is mostly tested.

***Edit: minor changes to the script. It should have only been checking for MySQL commands when MySQL backup is enabled.

#!/bin/sh
 
### Duplicity Setup ###
PASSPHRASE="<your passphrase>"
AWS_ACCESS_KEY_ID="<your key id>"
AWS_SECRET_ACCESS_KEY="<your secret key>"
 
# This needs to be a newline separated list of files and directories to backup
INCLUDEFILES="/etc/duplicity/server-filelist.txt"
 
S3FILESYSLOCATION="s3+http://<your file bucket>"
S3MYSQLLOCATION="s3+http://<your mysql bucket>"
S3OPTIONS="--s3-use-new-style --s3-use-rrs"
 
EXTRADUPLICITYOPTIONS=
 
FULLDAYS="30D"
MAXFULL=3
 
### MySQL Setup ###
MUSER="<your mysql user>"
MPASS="<mysql user's password>"
MHOST="localhost"
 
### Disable MySQL ###
# Change to 0 to disable
BACKUPMYSQL=1
 
###### End Of Editable Parts ######
 
### Env Vars ###
PASSPHRASE_OLD="$(echo $PASSPHRASE)"
AWS_ACCESS_KEY_ID_OLD="$(echo $AWS_ACCESS_KEY_ID)"
AWS_SECRET_ACCESS_KEY_OLD="$(echo $AWS_SECRET_ACCESS_KEY)"
export PASSPHRASE=$PASSPHRASE
export AWS_ACCESS_KEY_ID=$AWS_ACCESS_KEY_ID
export AWS_SECRET_ACCESS_KEY=$AWS_SECRET_ACCESS_KEY
 
### Commands ###
if [[ -n "$BACKUPMYSQL" && "$BACKUPMYSQL" -gt 0 ]]; then
 MYSQLTMPDIR="$(mktemp -d)"
 MYSQL="$(which mysql)"
 MYSQLDUMP="$(which mysqldump)"
 GZIP="$(which gzip)"
fi
DUPLICITY="$(which duplicity)"
 
if [[ -n "$BACKUPMYSQL" && "$BACKUPMYSQL" -gt 0 ]]; then
 if [[ -n "$MYSQL" || -n "$MYSQL" || -n "$MYSQLDUMP" || -n "$GZIP" ]]; then
  echo "Not all MySQL commands found."
  exit 2
 fi
fi
if [ -n "$DUPLICITY"  ]; then
 echo "Duplicity not found."
 exit 2
fi
 
### Dump MySQL Databases ###
if [[ -n "$BACKUPMYSQL" && "$BACKUPMYSQL" -gt 0 ]]; then
 # Get all databases name
 DBS="$($MYSQL -u $MUSER -h $MHOST -p$MPASS -Bse 'show databases')"
 for db in $DBS
 do
  if [ "$db" != "information_schema" ]; then
   $MYSQLDUMP -u $MUSER -h $MHOST -p$MPASS $db | $GZIP -9 > $MYSQLTMPDIR/mysql-$db
  fi
 done
fi
 
### Backup files ###
if [ -n "$S3FILESYSLOCATION" ]; then
 $DUPLICITY --full-if-older-than $FULLDAYS $S3OPTIONS $EXTRADUPLICITYOPTIONS --include-globbing-filelist $INCLUDEFILES --exclude '**' / $S3FILESYSLOCATION
fi
if [[ -n "$BACKUPMYSQL" && "$BACKUPMYSQL" -gt 0 ]]; then
 if [ -n "$S3MYSQLLOCATION" ]; then
  $DUPLICITY --full-if-older-than $FULLDAYS $S3OPTIONS $EXTRADUPLICITYOPTIONS --allow-source-mismatch $MYSQLTMPDIR $S3MYSQLLOCATION
 fi
fi
 
### Cleanup ###
if [[ -n "$MAXFULL" && "$MAXFULL" -gt 0 ]]; then
 if [ -n "$S3FILESYSLOCATION" ]; then
  $DUPLICITY remove-all-but-n-full $MAXFULL $S3FILESYSLOCATION
 fi
 if [[ -n "$BACKUPMYSQL" && "$BACKUPMYSQL" -gt 0 ]]; then
  if [ -n "$S3MYSQLLOCATION" ]; then
   $DUPLICITY remove-all-but-n-full $MAXFULL $S3MYSQLLOCATION
  fi
 fi
fi
if [[ -n "$BACKUPMYSQL" && "$BACKUPMYSQL" -gt 0 ]]; then
 rm -rf $MYSQLTMPDIR
fi
export PASSPHRASE=$PASSPHRASE_OLD
export AWS_ACCESS_KEY_ID=$AWS_ACCESS_KEY_ID_OLD
export AWS_SECRET_ACCESS_KEY=$AWS_SECRET_ACCESS_KEY_OLD

Tags: backup, bash, duplicity, server.

For quite some time now Tati and I have both been using Dropbox to sync files between multiple computers. Recently Tati has been running low on space in her free Dropbox account and said she wanted to upgrade to the premium version that has more storage space. I have no problem with this and looked at Dropbox’s options. I remembered a friend telling me about Spideroak and I also went to their website to compare the two.

On paper Spideroak looks great. For $100 a year you get 100 GB instead of Dropbox’s 50 GB. Spideroak offers backup and sync modes. Dropbox only offers sync and with 40+ GB of data this can be a very long processes. Also, Spideroaks selective sync makes it a much better choice for installing on low storage space systems. Spideroaks backup also lets you backup multiple computers. Another really nice feature of Spideroak is how it handles privacy. It encrypts your files locally and send the encrypted data to their server to be stored. This way Sideroak never sees your data, they never have your password, and only you can decrypt it. Like Dropbox it also does versioning of your files so you can revert changes.

Just looking at the features Spideroak looks great. It does a lot more than Dropbox, you get more for what you pay and the privacy / security is beyond what the average person needs. However, Spideroak falls flat in execution.

Since Spideroak does so much more than Dropbox it is a little more involved with configuration. Dropbox is super simple. You tell Dropbox what folder you want to use as your Dropbox sync folder. Everything in that folder will be synced with the service and you will see files that were added on other computers appear in that folder. Spideroak has a desktop application you have to use for specifying what folders you want to backup and Sync.

The desktop application is where Spideroak starts to show its rough edges. Dropbox integrates right into Gnome and Windows Explorer giving you a very organic indication of what it’s doing. The tray icon tells you if it’s syncing. Each file in the file browser tells you its status. Spideroak does not have this integration. You have to rely on their desktop app to see what is happening. This isn’t terrible except the desktop app is a clunky, horribly complicated, confusing and ugly thing.

You have two options when backing up with the desktop application, basic and advanced. However, they are not well integrated into each other. Switching from one to the other will wipe out your settings. So if you have it set to backup your desktop, and pictures then switch to advances that is lost and you will have to reselect everything in the advanced mode. Even more frustrating is, as far as I could tell, advanced mode only let you select directories while basic only let you select media types. Combining the two would be helpful.

Another issue with the GUI is simply how complicated it is. There are tabs across the top for different and within each tab there are more tabs. Visually it’s very cluttered. Also, they are using their own custom “skin” for the application so it will never integrate with your desktop. Confusing and ugly. As Apple has shown the look of your application is just as important as how well it works.

I also took a peak at the web GUI and it is disappointing. Dropbox’s web interface is full featured. You can pull back previous versions of files, see how much storage space you’ve used, upload files, and share folders. Spideroak’s web GUI let me see a list of files and download them. I’m hoping I was accessing it wrong but it just wasn’t very useful.

Sync is what I really care about and backing up other files would have just been a very nice extra for me. Setting Spideroak to sync was not an easy process. First I went to the Sync tab and was told it can’t Sync. It wanted me to setup a backup first. So I selected a folder to backup. Going back to the Sync tab it would now let me create a Sync. Here is where it got confusing. Selecting a new sync gave me a set of input boxes labeled Folder 1 and Folder 2. Clicking the chooser next to them brought up a list of backups I had created. The instructions in the application said to select a folder on computer one and a folder on computer two. Not understanding how only letting me choose backup locations relates to folders on my computer I trued to the Spideroak help forums. After reading the first five or so threads on the forum I deleted my Spideroak account. Every thread had comments about sync not working. It wasn’t just people having issues setting it up either. One of the comments was by a developer basically saying, “We know it’s broken and we are working to fix it. Don’t worry it will work at some undetermined point in the future.” I see no reason to use, let alone buy, a service that advertise a feature that is unusable simply because they say it might work at some point down the road.

Overall Spideroak was not easy to use, and it did not work properly. The features it claims are exactly what I want but that isn’t enough. The features need to be polished and throughly tested before I can recommend Spideroak. I only used it for about an hour but that was enough time for me to decide to cancel my account and uninstall it. Even though I didn’t use it very long first impressions are important and an hour was more than enough time to waste on this piece of software.

Tags: backup, dropbox, review, spideroak, sync.