* Duplicity Backup Script

Posted on August 7th, 2010 by John. Filed under Linux, programming.


I’ve started using duplicity combined with Amazon’s S3 to backup this server. Duplicity is an amazing application that makes backups simple.

Some of the features I like best about duplicity are: it encrypts the data, compresses it, splits it into manageable chunks, does incremental backups, and can backup to a variety of destinations. When you restore your data it takes care of applying the incremental backups to produce the final files. Also, duplicity can be used to restore previous versions of stored files.

Overall I’m very happy with using duplicity for pushing automatic backups to my S3 account. To make it easier to use and so that I can run it from a cron job I’ve written a simple bash script to handle calling duplicity with the correct options. The script also dumps my MySQL databases and pushes them separately to a different S3 bucket.

Following is a variant of the script I’m using on my server. This one is a bit more generic and allows for MySQL backups to be turned off as well as some basic dependency checking. It also allows for some simple option tuning. The version I use has most of the options hard coded in the appropriate place instead of putting them into variables. This script is mostly tested.

***Edit: minor changes to the script. It should have only been checking for MySQL commands when MySQL backup is enabled.

#!/bin/sh

### Duplicity Setup ###
PASSPHRASE="<your passphrase>"
AWS_ACCESS_KEY_ID="<your key id>"
AWS_SECRET_ACCESS_KEY="<your secret key>"

# This needs to be a newline separated list of files and directories to backup
INCLUDEFILES="/etc/duplicity/server-filelist.txt"

S3FILESYSLOCATION="s3+http://<your file bucket>"
S3MYSQLLOCATION="s3+http://<your mysql bucket>"
S3OPTIONS="--s3-use-new-style --s3-use-rrs"

EXTRADUPLICITYOPTIONS=

FULLDAYS="30D"
MAXFULL=3

### MySQL Setup ###
MUSER="<your mysql user>"
MPASS="<mysql user's password>"
MHOST="localhost"

### Disable MySQL ###
# Change to 0 to disable
BACKUPMYSQL=1

###### End Of Editable Parts ######

### Env Vars ###
PASSPHRASE_OLD="$(echo $PASSPHRASE)"
AWS_ACCESS_KEY_ID_OLD="$(echo $AWS_ACCESS_KEY_ID)"
AWS_SECRET_ACCESS_KEY_OLD="$(echo $AWS_SECRET_ACCESS_KEY)"
export PASSPHRASE=$PASSPHRASE
export AWS_ACCESS_KEY_ID=$AWS_ACCESS_KEY_ID
export AWS_SECRET_ACCESS_KEY=$AWS_SECRET_ACCESS_KEY

### Commands ###
if [[ -n "$BACKUPMYSQL" && "$BACKUPMYSQL" -gt 0 ]]; then
 MYSQLTMPDIR="$(mktemp -d)"
 MYSQL="$(which mysql)"
 MYSQLDUMP="$(which mysqldump)"
 GZIP="$(which gzip)"
fi
DUPLICITY="$(which duplicity)"

if [[ -n "$BACKUPMYSQL" && "$BACKUPMYSQL" -gt 0 ]]; then
 if [[ -n "$MYSQL" || -n "$MYSQL" || -n "$MYSQLDUMP" || -n "$GZIP" ]]; then
  echo "Not all MySQL commands found."
  exit 2
 fi
fi
if [ -n "$DUPLICITY"  ]; then
 echo "Duplicity not found."
 exit 2
fi

### Dump MySQL Databases ###
if [[ -n "$BACKUPMYSQL" && "$BACKUPMYSQL" -gt 0 ]]; then
 # Get all databases name
 DBS="$($MYSQL -u $MUSER -h $MHOST -p$MPASS -Bse 'show databases')"
 for db in $DBS
 do
  if [ "$db" != "information_schema" ]; then
   $MYSQLDUMP -u $MUSER -h $MHOST -p$MPASS $db | $GZIP -9 > $MYSQLTMPDIR/mysql-$db
  fi
 done
fi

### Backup files ###
if [ -n "$S3FILESYSLOCATION" ]; then
 $DUPLICITY --full-if-older-than $FULLDAYS $S3OPTIONS $EXTRADUPLICITYOPTIONS --include-globbing-filelist $INCLUDEFILES --exclude '**' / $S3FILESYSLOCATION
fi
if [[ -n "$BACKUPMYSQL" && "$BACKUPMYSQL" -gt 0 ]]; then
 if [ -n "$S3MYSQLLOCATION" ]; then
  $DUPLICITY --full-if-older-than $FULLDAYS $S3OPTIONS $EXTRADUPLICITYOPTIONS --allow-source-mismatch $MYSQLTMPDIR $S3MYSQLLOCATION
 fi
fi

### Cleanup ###
if [[ -n "$MAXFULL" && "$MAXFULL" -gt 0 ]]; then
 if [ -n "$S3FILESYSLOCATION" ]; then
  $DUPLICITY remove-all-but-n-full $MAXFULL $S3FILESYSLOCATION
 fi
 if [[ -n "$BACKUPMYSQL" && "$BACKUPMYSQL" -gt 0 ]]; then
  if [ -n "$S3MYSQLLOCATION" ]; then
   $DUPLICITY remove-all-but-n-full $MAXFULL $S3MYSQLLOCATION
  fi
 fi
fi
if [[ -n "$BACKUPMYSQL" && "$BACKUPMYSQL" -gt 0 ]]; then
 rm -rf $MYSQLTMPDIR
fi
export PASSPHRASE=$PASSPHRASE_OLD
export AWS_ACCESS_KEY_ID=$AWS_ACCESS_KEY_ID_OLD
export AWS_SECRET_ACCESS_KEY=$AWS_SECRET_ACCESS_KEY_OLD

Tags: , , ,



8 Responses to “Duplicity Backup Script”

  1. Simon Says:

    Hi John

    I’m trying this script on my Debian box but am getting the following errors:

    ./duplicity-s3.sh: 45: [[: not found
    ./duplicity-s3.sh: 53: [[: not found
    Duplicity not found.

    Do you know what could cause this? If I type “which duplicity” at the command line it gives me “/usr/bin/duplicity” so I’m not sure why it’s failing.

    Any ideas?

    Many thanks
    Simon

  2. John Says:

    Try changing the shebang to #!/bin/bash. [[ is a keyword it appears the version of sh you have installed does not support. Running with bash >= 2.02 should be all you need to do to get the script working properly.

  3. Simon Says:

    thanks!

    all works as long as i remove the –s3-use-rrs option, for some reason duplicity doesn’t like that one…

    cheers
    simon

  4. John Says:

    Ah. The -s3-use-rrs option was added in 0.6.10 or 0.6.11 (I can’t remember which, might have been 0.6.9). That option isn’t required. It sets S3 to use Reduced Redundancy Storage. While not as safe it still provides 99.99% durability and availability. The main reason I use that option is RRS is charged at a lower rate.

  5. Peter Says:

    Forgive the basic question (Very new to duplicity), and need something to backup my wordpress databases. What do the following options do?

    FULLDAYS=”30D”
    MAXFULL=3

  6. John Says:

    FULLDAYS=ā€30Dā€ and MAXFULL=3 are easy to change constants used by the script. They are not part of duplicity itself.

    FULLDAYS set to 30D says that we want a full backup every 30 days. By default the script does incremental backups based upon the last full backup. I want to force a full backup every 30 days.

    MAXFULL is how many full backups to keep around. A full backup being set to 30 days means that I only store the last 3 months (or so) of backups.

    The idea is to do incremental backups most of the time but do a full backup every so often to make it easier to do a restore if necessary. Also only store the last 3 months of backups. Anything older will be automatically deleted.

  7. Peter Says:

    Thanks for that. And to restore, you just use duplicity?

  8. John Says:

    Restore is handled by duplicity. When a URL comes before a local directory it works in restore mode. See http://duplicity.nongnu.org/duplicity.1.html for details. However, my backup script does not handle restore. This would need to be done manually using duplicity directly.