Creating Incremental and Encrypted Backups with Duplicity

Overview

While Vultr does have a backup system for entire system images, this works at the block level, and requires that the backup is restored to a VM before the data can be restored. Additionally, backups are only stored for a short amount of time, and do not provide a true incremental backup solution.

Enter Duplicity - Duplicity builds upon the legendary rsync and provides the ability to store incremental backups along with encrypting the data at rest via GPG. It is Posix compliant, and only transfers deltas between backup runs, thus reducing the overall bandwidth requirement.

Setting up environment

###Terminology

• Source host - Server which will have its data backed up. For this tutorial, it has the IP address: 10.1.10.1
• Backup host - Destination server for backups. For this tutorial, it has the IP address: 10.1.10.2
• /backupdir - Source directory on source host used for backups in this tutorial. You can change this to match your environment.
• /destdir - Destination directory on backup host used for backups in this tutorial. You can change this to match your environment.
• Full backup - Entire copy of the source dataset.
• Incremental backup - Copy of all the changes made since the last backup.

###Installing Duplicity

#####Ubuntu 14.04:

sudo apt-get update
sudo apt-get install duplicity python-paramiko

#####CentOS (requires EPEL):

sudo yum install duplicity python-paramiko

#####Ubuntu 12.04/Debian 7:

sudo apt-get update 
sudo apt-get install ncftp python-paramiko python-pycryptopp lftp python-boto python-dev librsync-dev
wget https://launchpad.net/duplicity/0.7-series/0.7.02/+download/duplicity-0.7.02.tar.gz
tar xzvf duplicity*
cd duplicity*
sudo python setup.py install

We have to install from source as the Duplicity package within Debian 7 and Ubuntu 12.04 are broken due to a change in the backend SSH library.

Double check that Duplicity is installed by running:

duplicity -V

It should return the following output (version may be different):

duplicity 0.6.18

###Setting up key-less authentication for SSH

The next step is to setup certificate-based authentication for SSH between the backup host and the source host. This will enable the source server to SSH into the backup host without typing in a passphrase. Vultr has a great article which explains how to do this: How Do I Generate SSH Keys.

Backups

Running the first (full) backup

Let's run a full backup! This will send a full copy of data from the source server to the destination.

duplicity full -v --no-encryption --include="/sourcedir/" --exclude="**" /  ssh://user@backupserver:22/destdir/

You can add additional folders by including multiple --include="[dir]" statements.

The --no-encryption specifies that the data is not to be encrypted at the destination. Data will be encrypted during transport as it's passing through the SSH tunnel.

The --exclude="**" / option is a trick to backup everything that is only in the include list (and nothing else).

Running incremental backup

Running an incremental backup is very simple - just change the full flag to the incremental flag.

duplicity incremental -v --no-encryption --include="/sourcedir/" --exclude="**" /  ssh://user@backupserver:22/destdir/

Automation

Write automated scripts

Having to run these commands every time you need a backup is a drag - what if we had scripts to handle it for us?

Full backup script

Run the command.

nano /usr/local/bin/backup-full

Add the following content.

#!/bin/bash
duplicity full -v --no-encryption --include="/sourcedir/" --exclude="**" /  ssh://user@backupserver:22/destdir/

Incremental backup script

nano /usr/local/bin/backup-incremental

Add the following content.

#!/bin/bash
duplicity incremental -v --no-encryption --include="/sourcedir/" --exclude="**" /  ssh://user@backupserver:22/destdir/

Make scripts executable

To make the scripts executable, run the following command.

chmod +x /usr/local/bin/backup-*

Now you can perform a backup by running backup-full and backup-incremental from within the shell. Pretty cool!

Setup cron

Let's make the backups automatic! By setting up cron to run the above scripts at specified times, we can make sure that backups are performed at regular intervals.

Run the following command.

crontab -e

Add the following to the bottom of the file.

10 01 * * 1,2,3,4,5,6 backup-incremental
10 01 * * 7 backup-full

This will run a full backup every Sunday at 1:10 AM, and will run incremental backups every other day at 1:10 AM as well.

Restoration

Godzilla has destroyed Seattle and we need to be able to get the data back from the backup VPS in New York!

duplicity --no-encryption --file-to-restore / ssh://user@backupserver:22/destdir/

If we need to restore data from 3 days ago:

duplicity --no-encryption -t 3D --file-to-restore / ssh://user@backupserver:22/destdir/

The -t 3D option means restore a backup from three days ago. Similar options like -t 1M (for one month ago) or -t 5H (for 5 hours ago) also work.