Duplicity Backup Script

John — Sat, 07 Aug 2010 20:59:16 +0000

I’ve started using duplicity combined with Amazon’s S3 to backup this server. Duplicity is an amazing application that makes backups simple.

Some of the features I like best about duplicity are: it encrypts the data, compresses it, splits it into manageable chunks, does incremental backups, and can backup to a variety of destinations. When you restore your data it takes care of applying the incremental backups to produce the final files. Also, duplicity can be used to restore previous versions of stored files.

Overall I’m very happy with using duplicity for pushing automatic backups to my S3 account. To make it easier to use and so that I can run it from a cron job I’ve written a simple bash script to handle calling duplicity with the correct options. The script also dumps my MySQL databases and pushes them separately to a different S3 bucket.

Following is a variant of the script I’m using on my server. This one is a bit more generic and allows for MySQL backups to be turned off as well as some basic dependency checking. It also allows for some simple option tuning. The version I use has most of the options hard coded in the appropriate place instead of putting them into variables. This script is mostly tested.

***Edit: minor changes to the script. It should have only been checking for MySQL commands when MySQL backup is enabled.

#!/bin/sh
 
### Duplicity Setup ###
PASSPHRASE=""
AWS_ACCESS_KEY_ID=""
AWS_SECRET_ACCESS_KEY=""
 
# This needs to be a newline separated list of files and directories to backup
INCLUDEFILES="/etc/duplicity/server-filelist.txt"
 
S3FILESYSLOCATION="s3+http://"
S3MYSQLLOCATION="s3+http://"
S3OPTIONS="--s3-use-new-style --s3-use-rrs"
 
EXTRADUPLICITYOPTIONS=
 
FULLDAYS="30D"
MAXFULL=3
 
### MySQL Setup ###
MUSER=""
MPASS=""
MHOST="localhost"
 
### Disable MySQL ###
# Change to 0 to disable
BACKUPMYSQL=1
 
###### End Of Editable Parts ######
 
### Env Vars ###
PASSPHRASE_OLD="$(echo $PASSPHRASE)"
AWS_ACCESS_KEY_ID_OLD="$(echo $AWS_ACCESS_KEY_ID)"
AWS_SECRET_ACCESS_KEY_OLD="$(echo $AWS_SECRET_ACCESS_KEY)"
export PASSPHRASE=$PASSPHRASE
export AWS_ACCESS_KEY_ID=$AWS_ACCESS_KEY_ID
export AWS_SECRET_ACCESS_KEY=$AWS_SECRET_ACCESS_KEY
 
### Commands ###
if [[ -n "$BACKUPMYSQL" && "$BACKUPMYSQL" -gt 0 ]]; then
 MYSQLTMPDIR="$(mktemp -d)"
 MYSQL="$(which mysql)"
 MYSQLDUMP="$(which mysqldump)"
 GZIP="$(which gzip)"
fi
DUPLICITY="$(which duplicity)"
 
if [[ -n "$BACKUPMYSQL" && "$BACKUPMYSQL" -gt 0 ]]; then
 if [[ -n "$MYSQL" || -n "$MYSQL" || -n "$MYSQLDUMP" || -n "$GZIP" ]]; then
  echo "Not all MySQL commands found."
  exit 2
 fi
fi
if [ -n "$DUPLICITY"  ]; then
 echo "Duplicity not found."
 exit 2
fi
 
### Dump MySQL Databases ###
if [[ -n "$BACKUPMYSQL" && "$BACKUPMYSQL" -gt 0 ]]; then
 # Get all databases name
 DBS="$($MYSQL -u $MUSER -h $MHOST -p$MPASS -Bse 'show databases')"
 for db in $DBS
 do
  if [ "$db" != "information_schema" ]; then
   $MYSQLDUMP -u $MUSER -h $MHOST -p$MPASS $db | $GZIP -9 > $MYSQLTMPDIR/mysql-$db
  fi
 done
fi
 
### Backup files ###
if [ -n "$S3FILESYSLOCATION" ]; then
 $DUPLICITY --full-if-older-than $FULLDAYS $S3OPTIONS $EXTRADUPLICITYOPTIONS --include-globbing-filelist $INCLUDEFILES --exclude '**' / $S3FILESYSLOCATION
fi
if [[ -n "$BACKUPMYSQL" && "$BACKUPMYSQL" -gt 0 ]]; then
 if [ -n "$S3MYSQLLOCATION" ]; then
  $DUPLICITY --full-if-older-than $FULLDAYS $S3OPTIONS $EXTRADUPLICITYOPTIONS --allow-source-mismatch $MYSQLTMPDIR $S3MYSQLLOCATION
 fi
fi
 
### Cleanup ###
if [[ -n "$MAXFULL" && "$MAXFULL" -gt 0 ]]; then
 if [ -n "$S3FILESYSLOCATION" ]; then
  $DUPLICITY remove-all-but-n-full $MAXFULL $S3FILESYSLOCATION
 fi
 if [[ -n "$BACKUPMYSQL" && "$BACKUPMYSQL" -gt 0 ]]; then
  if [ -n "$S3MYSQLLOCATION" ]; then
   $DUPLICITY remove-all-but-n-full $MAXFULL $S3MYSQLLOCATION
  fi
 fi
fi
if [[ -n "$BACKUPMYSQL" && "$BACKUPMYSQL" -gt 0 ]]; then
 rm -rf $MYSQLTMPDIR
fi
export PASSPHRASE=$PASSPHRASE_OLD
export AWS_ACCESS_KEY_ID=$AWS_ACCESS_KEY_ID_OLD
export AWS_SECRET_ACCESS_KEY=$AWS_SECRET_ACCESS_KEY_OLD

Multiple Input with Single Input Apps

John — Wed, 24 Dec 2008 23:36:42 +0000

The few eBook formating tools I’ve posted all share one major flaw. They can only handle a single file as input. This is a problem when you want to run it on all or a number of different eBooks. If you are able to use Bash there is a simple way to run any single input command with multiple items.

This first method gets input from a single directory. This method will not go into subdirectories for input.

$ cd ~/Books/author
$ ls -1 *.txt | while read file; do echo $file; ./fix_paragraphs_ebook_txt "$file"; ./remove_extra_whitespace_ebook_txt "$file"; done

This second method gets all files matching the given pattern and will handle subdirectories.

$ find ~/books/ -iname "*.txt" | while read file; do echo "$file"; dos2unix -ad "$file"; ./fix_end_ebook_txt "$file"; done

John's Blog » bash

Duplicity Backup Script

Multiple Input with Single Input Apps