* Duplicity Backup Script
Posted on August 7th, 2010 by John. Filed under Linux, programming.
I’ve started using duplicity combined with Amazon’s S3 to backup this server. Duplicity is an amazing application that makes backups simple.
Some of the features I like best about duplicity are: it encrypts the data, compresses it, splits it into manageable chunks, does incremental backups, and can backup to a variety of destinations. When you restore your data it takes care of applying the incremental backups to produce the final files. Also, duplicity can be used to restore previous versions of stored files.
Overall I’m very happy with using duplicity for pushing automatic backups to my S3 account. To make it easier to use and so that I can run it from a cron job I’ve written a simple bash script to handle calling duplicity with the correct options. The script also dumps my MySQL databases and pushes them separately to a different S3 bucket.
Following is a variant of the script I’m using on my server. This one is a bit more generic and allows for MySQL backups to be turned off as well as some basic dependency checking. It also allows for some simple option tuning. The version I use has most of the options hard coded in the appropriate place instead of putting them into variables. This script is mostly tested.
***Edit: minor changes to the script. It should have only been checking for MySQL commands when MySQL backup is enabled.
#!/bin/sh ### Duplicity Setup ### PASSPHRASE="<your passphrase>" AWS_ACCESS_KEY_ID="<your key id>" AWS_SECRET_ACCESS_KEY="<your secret key>" # This needs to be a newline separated list of files and directories to backup INCLUDEFILES="/etc/duplicity/server-filelist.txt" S3FILESYSLOCATION="s3+http://<your file bucket>" S3MYSQLLOCATION="s3+http://<your mysql bucket>" S3OPTIONS="--s3-use-new-style --s3-use-rrs" EXTRADUPLICITYOPTIONS= FULLDAYS="30D" MAXFULL=3 ### MySQL Setup ### MUSER="<your mysql user>" MPASS="<mysql user's password>" MHOST="localhost" ### Disable MySQL ### # Change to 0 to disable BACKUPMYSQL=1 ###### End Of Editable Parts ###### ### Env Vars ### PASSPHRASE_OLD="$(echo $PASSPHRASE)" AWS_ACCESS_KEY_ID_OLD="$(echo $AWS_ACCESS_KEY_ID)" AWS_SECRET_ACCESS_KEY_OLD="$(echo $AWS_SECRET_ACCESS_KEY)" export PASSPHRASE=$PASSPHRASE export AWS_ACCESS_KEY_ID=$AWS_ACCESS_KEY_ID export AWS_SECRET_ACCESS_KEY=$AWS_SECRET_ACCESS_KEY ### Commands ### if [[ -n "$BACKUPMYSQL" && "$BACKUPMYSQL" -gt 0 ]]; then MYSQLTMPDIR="$(mktemp -d)" MYSQL="$(which mysql)" MYSQLDUMP="$(which mysqldump)" GZIP="$(which gzip)" fi DUPLICITY="$(which duplicity)" if [[ -n "$BACKUPMYSQL" && "$BACKUPMYSQL" -gt 0 ]]; then if [[ -n "$MYSQL" || -n "$MYSQL" || -n "$MYSQLDUMP" || -n "$GZIP" ]]; then echo "Not all MySQL commands found." exit 2 fi fi if [ -n "$DUPLICITY" ]; then echo "Duplicity not found." exit 2 fi ### Dump MySQL Databases ### if [[ -n "$BACKUPMYSQL" && "$BACKUPMYSQL" -gt 0 ]]; then # Get all databases name DBS="$($MYSQL -u $MUSER -h $MHOST -p$MPASS -Bse 'show databases')" for db in $DBS do if [ "$db" != "information_schema" ]; then $MYSQLDUMP -u $MUSER -h $MHOST -p$MPASS $db | $GZIP -9 > $MYSQLTMPDIR/mysql-$db fi done fi ### Backup files ### if [ -n "$S3FILESYSLOCATION" ]; then $DUPLICITY --full-if-older-than $FULLDAYS $S3OPTIONS $EXTRADUPLICITYOPTIONS --include-globbing-filelist $INCLUDEFILES --exclude '**' / $S3FILESYSLOCATION fi if [[ -n "$BACKUPMYSQL" && "$BACKUPMYSQL" -gt 0 ]]; then if [ -n "$S3MYSQLLOCATION" ]; then $DUPLICITY --full-if-older-than $FULLDAYS $S3OPTIONS $EXTRADUPLICITYOPTIONS --allow-source-mismatch $MYSQLTMPDIR $S3MYSQLLOCATION fi fi ### Cleanup ### if [[ -n "$MAXFULL" && "$MAXFULL" -gt 0 ]]; then if [ -n "$S3FILESYSLOCATION" ]; then $DUPLICITY remove-all-but-n-full $MAXFULL $S3FILESYSLOCATION fi if [[ -n "$BACKUPMYSQL" && "$BACKUPMYSQL" -gt 0 ]]; then if [ -n "$S3MYSQLLOCATION" ]; then $DUPLICITY remove-all-but-n-full $MAXFULL $S3MYSQLLOCATION fi fi fi if [[ -n "$BACKUPMYSQL" && "$BACKUPMYSQL" -gt 0 ]]; then rm -rf $MYSQLTMPDIR fi export PASSPHRASE=$PASSPHRASE_OLD export AWS_ACCESS_KEY_ID=$AWS_ACCESS_KEY_ID_OLD export AWS_SECRET_ACCESS_KEY=$AWS_SECRET_ACCESS_KEY_OLD
8 Responses to “Duplicity Backup Script”
Tags
Archives
- January 2012 (3)
- December 2011 (2)
- November 2011 (1)
- October 2011 (3)
- September 2011 (9)
- August 2011 (15)
- July 2011 (5)
- June 2011 (3)
- May 2011 (4)
- April 2011 (2)
- March 2011 (2)
- February 2011 (4)
- January 2011 (4)
- December 2010 (2)
- November 2010 (1)
- October 2010 (1)
- August 2010 (3)
- July 2010 (4)
- June 2010 (1)
- May 2010 (2)
- March 2010 (1)
- January 2010 (8)
- December 2009 (5)
- November 2009 (6)
- October 2009 (4)
- September 2009 (2)
- August 2009 (6)
- July 2009 (6)
- June 2009 (4)
- May 2009 (6)
- April 2009 (4)
- March 2009 (2)
- February 2009 (4)
- January 2009 (4)
- December 2008 (7)
- November 2008 (2)
April 28th, 2011 at 6:04 am
Hi John
I’m trying this script on my Debian box but am getting the following errors:
./duplicity-s3.sh: 45: [[: not found
./duplicity-s3.sh: 53: [[: not found
Duplicity not found.
Do you know what could cause this? If I type “which duplicity” at the command line it gives me “/usr/bin/duplicity” so I’m not sure why it’s failing.
Any ideas?
Many thanks
Simon
April 28th, 2011 at 6:27 am
Try changing the shebang to #!/bin/bash. [[ is a keyword it appears the version of sh you have installed does not support. Running with bash >= 2.02 should be all you need to do to get the script working properly.
April 28th, 2011 at 7:11 am
thanks!
all works as long as i remove the –s3-use-rrs option, for some reason duplicity doesn’t like that one…
cheers
simon
April 28th, 2011 at 5:00 pm
Ah. The -s3-use-rrs option was added in 0.6.10 or 0.6.11 (I can’t remember which, might have been 0.6.9). That option isn’t required. It sets S3 to use Reduced Redundancy Storage. While not as safe it still provides 99.99% durability and availability. The main reason I use that option is RRS is charged at a lower rate.
December 6th, 2011 at 6:54 am
Forgive the basic question (Very new to duplicity), and need something to backup my wordpress databases. What do the following options do?
FULLDAYS=”30D”
MAXFULL=3
December 6th, 2011 at 7:03 am
FULLDAYS=ā30Dā and MAXFULL=3 are easy to change constants used by the script. They are not part of duplicity itself.
FULLDAYS set to 30D says that we want a full backup every 30 days. By default the script does incremental backups based upon the last full backup. I want to force a full backup every 30 days.
MAXFULL is how many full backups to keep around. A full backup being set to 30 days means that I only store the last 3 months (or so) of backups.
The idea is to do incremental backups most of the time but do a full backup every so often to make it easier to do a restore if necessary. Also only store the last 3 months of backups. Anything older will be automatically deleted.
December 6th, 2011 at 1:07 pm
Thanks for that. And to restore, you just use duplicity?
December 6th, 2011 at 8:48 pm
Restore is handled by duplicity. When a URL comes before a local directory it works in restore mode. See http://duplicity.nongnu.org/duplicity.1.html for details. However, my backup script does not handle restore. This would need to be done manually using duplicity directly.