John's Blog

I’ve started using duplicity combined with Amazon’s S3 to backup this server. Duplicity is an amazing application that makes backups simple.

Some of the features I like best about duplicity are: it encrypts the data, compresses it, splits it into manageable chunks, does incremental backups, and can backup to a variety of destinations. When you restore your data it takes care of applying the incremental backups to produce the final files. Also, duplicity can be used to restore previous versions of stored files.

Overall I’m very happy with using duplicity for pushing automatic backups to my S3 account. To make it easier to use and so that I can run it from a cron job I’ve written a simple bash script to handle calling duplicity with the correct options. The script also dumps my MySQL databases and pushes them separately to a different S3 bucket.

Following is a variant of the script I’m using on my server. This one is a bit more generic and allows for MySQL backups to be turned off as well as some basic dependency checking. It also allows for some simple option tuning. The version I use has most of the options hard coded in the appropriate place instead of putting them into variables. This script is mostly tested.

***Edit: minor changes to the script. It should have only been checking for MySQL commands when MySQL backup is enabled.

#!/bin/sh
 
### Duplicity Setup ###
PASSPHRASE="<your passphrase>"
AWS_ACCESS_KEY_ID="<your key id>"
AWS_SECRET_ACCESS_KEY="<your secret key>"
 
# This needs to be a newline separated list of files and directories to backup
INCLUDEFILES="/etc/duplicity/server-filelist.txt"
 
S3FILESYSLOCATION="s3+http://<your file bucket>"
S3MYSQLLOCATION="s3+http://<your mysql bucket>"
S3OPTIONS="--s3-use-new-style --s3-use-rrs"
 
EXTRADUPLICITYOPTIONS=
 
FULLDAYS="30D"
MAXFULL=3
 
### MySQL Setup ###
MUSER="<your mysql user>"
MPASS="<mysql user's password>"
MHOST="localhost"
 
### Disable MySQL ###
# Change to 0 to disable
BACKUPMYSQL=1
 
###### End Of Editable Parts ######
 
### Env Vars ###
PASSPHRASE_OLD="$(echo $PASSPHRASE)"
AWS_ACCESS_KEY_ID_OLD="$(echo $AWS_ACCESS_KEY_ID)"
AWS_SECRET_ACCESS_KEY_OLD="$(echo $AWS_SECRET_ACCESS_KEY)"
export PASSPHRASE=$PASSPHRASE
export AWS_ACCESS_KEY_ID=$AWS_ACCESS_KEY_ID
export AWS_SECRET_ACCESS_KEY=$AWS_SECRET_ACCESS_KEY
 
### Commands ###
if [[ -n "$BACKUPMYSQL" && "$BACKUPMYSQL" -gt 0 ]]; then
 MYSQLTMPDIR="$(mktemp -d)"
 MYSQL="$(which mysql)"
 MYSQLDUMP="$(which mysqldump)"
 GZIP="$(which gzip)"
fi
DUPLICITY="$(which duplicity)"
 
if [[ -n "$BACKUPMYSQL" && "$BACKUPMYSQL" -gt 0 ]]; then
 if [[ -n "$MYSQL" || -n "$MYSQL" || -n "$MYSQLDUMP" || -n "$GZIP" ]]; then
  echo "Not all MySQL commands found."
  exit 2
 fi
fi
if [ -n "$DUPLICITY"  ]; then
 echo "Duplicity not found."
 exit 2
fi
 
### Dump MySQL Databases ###
if [[ -n "$BACKUPMYSQL" && "$BACKUPMYSQL" -gt 0 ]]; then
 # Get all databases name
 DBS="$($MYSQL -u $MUSER -h $MHOST -p$MPASS -Bse 'show databases')"
 for db in $DBS
 do
  if [ "$db" != "information_schema" ]; then
   $MYSQLDUMP -u $MUSER -h $MHOST -p$MPASS $db | $GZIP -9 > $MYSQLTMPDIR/mysql-$db
  fi
 done
fi
 
### Backup files ###
if [ -n "$S3FILESYSLOCATION" ]; then
 $DUPLICITY --full-if-older-than $FULLDAYS $S3OPTIONS $EXTRADUPLICITYOPTIONS --include-globbing-filelist $INCLUDEFILES --exclude '**' / $S3FILESYSLOCATION
fi
if [[ -n "$BACKUPMYSQL" && "$BACKUPMYSQL" -gt 0 ]]; then
 if [ -n "$S3MYSQLLOCATION" ]; then
  $DUPLICITY --full-if-older-than $FULLDAYS $S3OPTIONS $EXTRADUPLICITYOPTIONS --allow-source-mismatch $MYSQLTMPDIR $S3MYSQLLOCATION
 fi
fi
 
### Cleanup ###
if [[ -n "$MAXFULL" && "$MAXFULL" -gt 0 ]]; then
 if [ -n "$S3FILESYSLOCATION" ]; then
  $DUPLICITY remove-all-but-n-full $MAXFULL $S3FILESYSLOCATION
 fi
 if [[ -n "$BACKUPMYSQL" && "$BACKUPMYSQL" -gt 0 ]]; then
  if [ -n "$S3MYSQLLOCATION" ]; then
   $DUPLICITY remove-all-but-n-full $MAXFULL $S3MYSQLLOCATION
  fi
 fi
fi
if [[ -n "$BACKUPMYSQL" && "$BACKUPMYSQL" -gt 0 ]]; then
 rm -rf $MYSQLTMPDIR
fi
export PASSPHRASE=$PASSPHRASE_OLD
export AWS_ACCESS_KEY_ID=$AWS_ACCESS_KEY_ID_OLD
export AWS_SECRET_ACCESS_KEY=$AWS_SECRET_ACCESS_KEY_OLD

Tags: backup, bash, duplicity, server.

For quite a long time Intel has been the shining light of open source video drivers on Linux. Even though Intel integrated video doesn’t have the highest performance it was what you needed to have for full 3D support, and fancy splash screens when booting with (plymouth). By no means were the drivers ever perfect but they actually worked for the majority of people. That is until Kernel 2.6.34, Xorg 1.8 and the 2.12.0 Intel driver were released. There are really two different sets of issues I’ve experienced. One relates to Kernel 2.6.34 and the other is with Xorg 1.8 and video-intel 2.12.0.

With Kernel 2.6.34 suspend stopped working properly. At first I thought that the ACPI suspend patches for my notebook that were added years(?) ago had been reverted. The symptoms were the same as before Linux was able to suspend properly with my notebook. Instead of suspending the system would lockup, the screen would be blank and the fan would run at its maximum speed. The suspend code for my notebook was still present in 2.6.34 so I was at a bit of a loss as to why suspend was suddenly broken.

I ended up finding the solution to the suspend problem while looking for a the solution to the Xorg 1.8 issues. This bug report proposed setting i915.powersave=0 on the kernel line in grub’s menu.lst file. This worked for me and I can now suspend again. In the bug report not everyone is claiming success with this solution.

Xorg 1.8 and video-intel 2.12.0 are giving many people issues: screen not redrawing properly, GPU hangs and slow 3D performance. I have not experienced the GPU hangs but that could be because I haven’t had my computer running long enough with the 2.12.0 driver for it to happen.

Another ArchLinux bug has the solution to the screen redraw issue. I followed Alyssa Hung (Deciare) instructions and applied this patch to Mesa 7.8.2. Installing the intel-dri package based on a patched Mesa made the screen redraw issues disappear completely.

I also found a possible fix for the GPU hang issue. Applying this patch to the kernel is supposed to work. I have not tried it myself.

As for the slow 3D performance, I have not found a fix. Short of disabling compositing simple tasks like changing virtual desktops and cycling though open windows is noticeably slower with 2.12.0. On my laptop (2.0 GHz Core 2 Duo) it is so slow that the animation stutters. This is not caused by the new drivers fixing the 3D frame rate to the vsync refresh rate because the frame rate is much lower than the vsyc rate. Until a fix is found for this problem I’ve reverted to using Xort 1.7 and video-intel 2.10.0.

Tags: intel, kernel, video, xorg.

I recently purchased a Yubikey from Yubico. What got me to buy it was the discount they’re offering to Security Now! listeners. So far I’m liking it quite a bit and have been looking to use it any way I can. One of the uses I found was to have the presence of they Yubikey unlock and lock Gnome Screen Saver.

Toward the end of the forum thread there is a very nice set of udev rules that work perfect for me and are very clean. I put the following into /etc/udev/rules.d/85-yubikey.rules

ACTION=="add", ENV{ID_VENDOR}=="Yubico", RUN+="/usr/local/bin/gnome-screensaver-unlock"
ACTION=="remove", ENV{ID_VENDOR}=="Yubico", RUN+="/usr/local/bin/gnome-screensaver-lock"

I really dislike the scripts that are in the thread for locking and unlocking the computer. Gnome Screen Saver is a DBus enabled application so controlling it is very easy. Below are the unlock and lock scripts I’ve written. They use qdbus to send the dbus commands. This could be replaced with dbus-send but I use Qt and qdbus’s syntax is easier to work with.

gnome-screensaver-unlock

#!/bin/sh
 
user=`ps aux | grep gnome-screensaver | head -n 1 | awk '{print $1}'`
 
if [ -n $user ]; then
        GNOME_SCREENSAVER_PROC=`ps xa | grep gnome-screensaver | head -n 1 | awk '{print $1}'`
        export `grep -z DBUS_SESSION_BUS_ADDRESS /proc/$GNOME_SCREENSAVER_PROC/environ`
 
        su $user -c "qdbus org.gnome.ScreenSaver / SetActive false"
fi

gnome-screensaver-lock

#!/bin/sh
 
user=`ps aux | grep gnome-screensaver | head -n 1 | awk '{print $1}'`
 
if [ -n $user ]; then
	GNOME_SCREENSAVER_PROC=`ps xa | grep gnome-screensaver | head -n 1 | awk '{print $1}'`
	export `grep -z DBUS_SESSION_BUS_ADDRESS /proc/$GNOME_SCREENSAVER_PROC/environ`
 
	su $user -c "qdbus org.gnome.ScreenSaver / SetActive true"
fi

One thing that isn’t mentioned in the forum thread that is very important, this unlocking method is highly insecure. The locking portion is fine but unlocking shouldn’t actually be done in this way. The above udev rules only checks that a Yubikey is inserted. It does not which which Yubikey is inserted. Any Yubikey can bypass your password and unlock the computer. The unlocking script does not preform any additional checks against the yubikey. Due to this, I don’t have the unlock code enabled on my computer.

However, it is possible to make unlocking secure. You can use one of the two yubikey pam modules, Yubico’s yubico-pam and Securix Live’s yubipam. Yubico-pam requires internet access because it validates against Yubico’s servers. Yubipam does not need internet access but you will have to reprogram your Yubikey with a new AES key. The new key must be stored in the computer. Each has it’s advantages and disadvantages but using the pam module with Gnome Screen Saver (I haven’t actually tried so it might not work) will provide you with a secure unlock.

Thinking about secure unlocks there is a clever solution that allows for the Yubikey to be used with SSH without the need for the pam module. If I can find a way (I haven’t looked yet) to have an input that can capture the Yubikey’s output then it would be possible to handle the unlock in a secure manner without the need for the pam module…

Tags: gnome, screen saver, security, yubikey.

I unplugged my Cybook from the computer without first unmounting the volume. It was still in the process of deleting a few files. When I turned it on the files showed in the library but were unable to be viewed. After I plugged the Cybook back into the computer the file system was mounted as read only and dmesg spit out a large number of IO errors.

The Cybook’s file system was damaged. Thankfully the Cybook uses Fat32. All that was needed to fix the errors was to run the following:

sudo fsck.vfat /dev/sdb -artvVw

/dev/sdb is the device id for the Cybook’s memory. If you are unsure of what it is, plug in the Cybook, wait for it to be detected and run dmesg in the console. At the end of the output there will be information about the device being connected and what device id the system has assigned to it.

Tags: cybook, Linux.