diff options
| -rw-r--r-- | Makefile | 17 | ||||
| -rw-r--r-- | README.md | 96 | ||||
| -rw-r--r-- | debian/manpages | 1 | ||||
| -rwxr-xr-x | debian/rules | 7 | ||||
| -rw-r--r-- | man/byteback-backup.txt | 69 | ||||
| -rw-r--r-- | man/byteback-prune.txt | 47 | ||||
| -rw-r--r-- | man/byteback-restore.txt | 174 |
7 files changed, 323 insertions, 88 deletions
diff --git a/Makefile b/Makefile new file mode 100644 index 0000000..6a0cb38 --- /dev/null +++ b/Makefile @@ -0,0 +1,17 @@ + +all: docs + +man/%.man: ./man/%.txt + [ -d man ] || mkdir man + txt2man -s 1 -t $(basename $(notdir $<)) $< | sed -e 's/\\\\fB/\\fB/' > $@ + +docs: man/byteback-prune.man man/byteback-restore.man man/byteback-backup.man + +# To be written +# man/byteback-snapshot.man man/byteback-setup-client.man man/byteback-setup-client-receive.man man/byteback-receive.man + +clean: + $(RM) man/*.man + +.PHONY: clean docs all + @@ -68,65 +68,13 @@ client to start and watch the backup. Configuring byteback-backup --------------------------- -You can now set "byteback-backup" on a daily cron job to start backing up the -server on a regular basis. -Without any further options this will copy every file from the root downwards. - -It currently excludes /tmp, /var/tmp, /var/cache/apt/archives, /swap.file and -/var/backups/localhost which (on Bytemark systems) do not need to be part of -any backup. To specify which locations are excluded, add them to -/etc/byteback/excludes, one per line. The filesystems /dev, /proc, /run and -/sys are always excluded. - -It is possible to configure a full rsync filter by creating the file -/etc/byteback/rsync_filter, which is parsed to rsync via the --filter flag. -Note that excludes on the command line take precedence, unless the filter -starts with an exclamation mark, which resets everything. If you do this, -you'll need to specify /proc, /sys, etc manually. See the rsync manpage for -more information about filters. - -It is also possible to add extra rsync flags to the configuration. The -following flags are always set: - - --archive --numeric-ids --delete-during --inplace --relative --timeout 43200 - -If you wish to add to that (e.g. --xattrs --acl --hard-links) then simply add -them to /etc/byteback/rsync_flags. These extra flags get appended after the -default flags above, so they take precedence. - -When the backup has completed successfully, the server will take a snapshot -so that the client can't alter the backups, and then "prune" the backup -snapshots to ensure that the next backup is likely to run OK. - -If the backup is interrupted or dies unexpected, running "byteback backup" -will cause the backup to be resumed, with rsync saving the work of re-copying -any files that hadn't changed. By default this will happen automatically up to -5 times, with a 10 minute pause in between each attempt. +This is now documented in the manapge for byteback-backup(1). Viewing and restoring backups ----------------------------- -Backups can be viewed on the server filesystem, although the permissions will -be wrong. The rsync "fake-super" flag is used to store the permissions in a -user attribute list. To view this list on the server, run - - getfattr -d $filename - -This command is part of the "attr" package in Debian. - -To restore a file you may use the `byteback-restore` command, which allows you to find all files matching a pattern via a command-like: - - byteback-restore --file /etc/passwd - -To actually restore the file, to the current directory, run: - - byteback-restore --file /etc/passwd --revision=current - -Recursive restorations are supported too: - - byteback-restore --file /srv --revision=current - +This is now documented in the manapge for byteback-restore(1). The trust model --------------- @@ -151,43 +99,15 @@ over a long enough period of time: Pruning behaviour ----------------- -Unless you are backing up a very small amount of data, backups will always -need pruning, i.e. old backups must be deleted to make way for newer ones. - -There is a program on the server called byteback-prune which deals with this -operation. It deletes old backups until a certain amount of free space is -achieved, which is currently fixed at 10% free. This can be changed by setting -the --maxpercent option. - -It can choose which backups to delete by one of two methods: -1) the 'age' method simply deletes the oldest backup; - -2) the 'importance' method tries to retain a more spread-out backup pattern -by "scoring" each backup according to how close it is to a set of "target -times". These are: - - * midday today; - * midday on the previous 6 days; - * midday on the previous 4 Sundays; - * midday on every 4th Sunday before that. - -So when you ask the pruner to run, the backup closest to the present time will -be the last one to be deleted. The backup closes to "1 day ago" will be the -second-last, and so on. We score every backup in this way until we end up with -a "least important" snapshot to delete. - -The upshot of the second strategy should be that we retain closely-spaced -daily backups, but as they get too numerous, we make sure that we are reluctant -to delete our very oldest. - -The "4th Sunday" is calculated as every 4th Sunday in the Unix epoch, i.e. -every fourth Sunday since Sunday 4th Jan 1970. +This is now documented in byteback-prune(1). Acknowledgements ---------------- -For maximum portability, I've included two libraries. Thanks very much to +For maximum portability, I've included three libraries. Thanks very much to their authors: -sys-filesystem by Daniel J. Berger: https://github.com/djberg96/sys-filesystem -trollop by William Morgan: https://github.com/wjessop/trollop +* sys-filesystem by Daniel J. Berger: https://github.com/djberg96/sys-filesystem +* trollop by William Morgan: https://github.com/wjessop/trollop +* ffi-xattr by Jari Bakken: https://github.com/jarib/ffi-xattr + diff --git a/debian/manpages b/debian/manpages new file mode 100644 index 0000000..5054144 --- /dev/null +++ b/debian/manpages @@ -0,0 +1 @@ +man/*.man diff --git a/debian/rules b/debian/rules index 0587334..b361ae1 100755 --- a/debian/rules +++ b/debian/rules @@ -3,3 +3,10 @@ %: dh $@ + +override_dh_auto_build-indep: + $(MAKE) docs + +override_dh_auto_clean: + $(MAKE) clean + diff --git a/man/byteback-backup.txt b/man/byteback-backup.txt new file mode 100644 index 0000000..15c448f --- /dev/null +++ b/man/byteback-backup.txt @@ -0,0 +1,69 @@ +NAME + + byteback-backup - Back up this system to a byteback-enabled server + +OPTIONS + + --destination, -d <s> Backup destination (i.e. user@host:/path). + --source, -s <s+> Source paths (default: /) + --exclude, -x <s+> Paths to exclude + --retry-number, -r <i>: Number of retries on error (default: 3) + --io-timeout, -i <i>: Number of seconds to allow I/O timeout for (default: 10800) + --retry-delay, -e <i>: Number of seconds between retries after an error (default: 300) + --ssh-key, -k <s> SSH key filename (default: /etc/byteback/key) + --verbose, -v Show debugging messages + --help, -h Show this message + +Additional excludes can be specified using /etc/byteback/rsync_filter, which is +an rsync filter file. See the rsync man page for information on how this +works. + +DESCRIPTION + +You can set "byteback-backup" on a daily cron job to start backing up +the server on a regular basis. + +Without any further options this will copy every file from the root +downwards. + +It currently excludes /tmp, /var/tmp, /var/cache/apt/archives, +/swap.file and /var/backups/localhost which (on Bytemark systems) do +not need to be part of any backup. To specify which locations are +excluded, add them to /etc/byteback/excludes, one per line. The +filesystems /dev, /proc, /run and /sys are always excluded. + +It is possible to configure a full rsync filter by creating the file +/etc/byteback/rsync_filter, which is parsed to rsync via the --filter +flag. Note that excludes on the command line take precedence, unless +the filter starts with an exclamation mark, which resets everything. +If you do this, you'll need to specify /proc, /sys, etc manually. See +the rsync manpage for more information about filters. + +It is also possible to add extra rsync flags to the configuration. +The following flags are always set: + + --archive --numeric-ids --delete-excluded --delete-during --inplace --relative + +If you wish to add to that (e.g. --xattrs --acl --hard-links) then +simply add them to /etc/byteback/rsync_flags. These extra flags get +appended after the default flags above, so they take precedence. + +When the backup has completed successfully, the server will take a +snapshot so that the client can't alter the backups, and then "prune" +the backup snapshots to ensure that the next backup is likely to run +OK. + +If the backup is interrupted or dies unexpected, running +"byteback-backup" will cause the backup to be resumed, with rsync +saving the work of re-copying any files that hadn't changed. By +default this will happen automatically up to 5 times, with a 10 minute +pause in between each attempt. + +SEE ALSO + + byteback-restore(1), byteback-prune(1) + +AUTHOR + + Patrick J Cherry <patrick@bytemark.co.uk> + diff --git a/man/byteback-prune.txt b/man/byteback-prune.txt new file mode 100644 index 0000000..f8074e2 --- /dev/null +++ b/man/byteback-prune.txt @@ -0,0 +1,47 @@ +NAME + + byteback-prune - Prune old byteback backup directories to ensure there's enough space + +OPTIONS + + --minpercent, -m <i> Start prune when disk has less than this %age free (default: 5) + --maxpercent, -a <i> Stop prune when disk has more than this %age free (default: 10) + --list, -l List backups in pruning order, no other action + --prune, -p Prune the next backup if necessary + --prune-force, -r Prune the next backup regardless + --order, -o <s> Order backups by 'age' or 'importance' (default: importance) + --verbose, -v Show debugging messages + --help, -h Show this message + +DESCRIPTION + +Unless you are backing up a very small amount of data, backups will +always need pruning, i.e. old backups must be deleted to make way for +newer ones. + +This program deals with this operation by deleting old backups until a +certain amount of free space is achieved, which is currently fixed at +10% free. This can be changed by setting the --maxpercent option. + + * midday today; + * midday on the previous 6 days; + * midday on the previous 4 Sundays; + * midday on every fourth Sunday before that, where the "fourth + Sunday" is calculated as every fourth Sunday in the Unix epoch, + i.e. every fourth Sunday since Sunday 4th Jan 1970. + +So when you ask the pruner to run, the backup closest to the present +time will be the last one to be deleted. The backup closes to "1 day +ago" will be the second-last, and so on. We score every backup in +this way until we end up with a "least important" snapshot to delete. + +The pruner will not delete backups younger than 7 days. + +SEE ALSO + + byteback-backup(1), byteback-restore(1), byteback-receive(1) + +AUTHOR + + Patrick J Cherry <patrick@bytemark.co.uk> + diff --git a/man/byteback-restore.txt b/man/byteback-restore.txt new file mode 100644 index 0000000..d4ff01e --- /dev/null +++ b/man/byteback-restore.txt @@ -0,0 +1,174 @@ +NAME + +byteback-restore - Restore a file to this system from a byteback-enabled server + +SYNOPSIS + +byteback-restore [ --restore | --list ] [ --snapshot s ] + [ --destination d ] [ --io-timeout i ] [ --ssh-key k ] + [ --list-all ] [ --help ] [ --verbose ] pattern pattern.. + +OPTIONS + + --list, -l List or find files + --restore, -r Restore files + --snapshot, -s s The snaphot to use. (Default: *) + --destination, -d s Backup destination (i.e. user@host:/path). The default is read from /etc/bytebackde + --io-timeout, -i i Number of seconds to allow I/O timeout for (default: 300) + --ssh-key, -k s SSH key filename (default: /etc/byteback/key) + --all, -a List/restore all versrions of each file + --verbose, -v Show debugging messages. + --help, -h Show a brief usage message. + +DESCRIPTION + +Firstly the pattern argument. This is expanded relative to the current directory, +if it is not an absolute path, before being globbed. It is best to quote the +pattern to avoid premature shell expansion. Multiple patterns can be supplied. + +To list files that could be restored, just give a pattern. The --list flag can +be used for clairty, but it is entirely optional. + +To restore files, use the --restore flag. Restores are always done into the +current working directory. See EXAMPLES below. + +GLOBBING + +The following character are "globbed": + + - "*" matches any filename. + - "**" matches directories recursively. + - "?" matches any single character. + - "[set]" matches any set of characters, working in the same way as regular expressions. + - "{a,b}" matches either literal "a" or literal "b" + - "\\" escapes the next metacharacter. + +Note that to match "dotfiles", the pattern "{*,.*}" should be used. + +The full glob syntax is documented in the Ruby documentation for the Dir#glob +method. + +EXAMPLES + +To list the latest copies of the files available, just call the command with +no flags.. The --list flag is optional. + + # byteback-restore '/home/patch/.bash*' + snapshot modestring size uid gid mtime path + ---------------------------------------------------------------------------------------- + current -rw------- 47931 1000 1000 2015-11-29 09:26:10 +0000 /home/patch/.bash_history + current -rw-r--r-- 220 1000 1000 2014-03-10 10:13:33 +0000 /home/patch/.bash_logout + current -rw-r--r-- 3638 1000 1000 2015-07-30 16:33:30 +0100 /home/patch/.bashrc + +To list all different available versions of a file, use --all. + + # byteback-restore --all '/home/patch/.bashrc' + snapshot modestring size uid gid mtime path + ---------------------------------------------------------------------------------------------- + 2015-07-17T15:11+0100 -rw-r--r-- 3637 1000 1000 2014-03-10 10:13:33 +0000 /home/patch/.bashrc + current -rw-r--r-- 3638 1000 1000 2015-07-30 16:33:30 +0100 /home/patch/.bashrc + +To list every single copy of a file, use --verbose. Notice that a number of +these entries match on size and mtime. + + # byteback-restore --verbose '/home/patch/.bashrc' + ssh -o BatchMode=yes -o ConnectionAttempts=5 -o ConnectTimeout=30 -o ServerAliveInterval=60 -o TCPKeepAlive=yes -x -a -i /etc/byteback/key -l byteback backups.default.cherry.uk0.bigv.io byteback-receive --snapshot * --list --verbose /home/patch/.bashrc + snapshot modestring size uid gid mtime path + ---------------------------------------------------------------------------------------------- + 2015-04-30T11:44+0100 -rw-r--r-- 3637 1000 1000 2014-03-10 10:13:33 +0000 /home/patch/.bashrc + 2015-05-05T15:05+0100 -rw-r--r-- 3637 1000 1000 2014-03-10 10:13:33 +0000 /home/patch/.bashrc + 2015-07-17T15:11+0100 -rw-r--r-- 3637 1000 1000 2014-03-10 10:13:33 +0000 /home/patch/.bashrc + 2015-11-25T11:14+0000 -rw-r--r-- 3638 1000 1000 2015-07-30 16:33:30 +0100 /home/patch/.bashrc + 2015-11-26T13:38+0000 -rw-r--r-- 3638 1000 1000 2015-07-30 16:33:30 +0100 /home/patch/.bashrc + 2015-11-27T10:48+0000 -rw-r--r-- 3638 1000 1000 2015-07-30 16:33:30 +0100 /home/patch/.bashrc + 2015-11-30T15:37+0000 -rw-r--r-- 3638 1000 1000 2015-07-30 16:33:30 +0100 /home/patch/.bashrc + current -rw-r--r-- 3638 1000 1000 2015-07-30 16:33:30 +0100 /home/patch/.bashrc + +Restoring is always to the current directory, and into directories named after the snapshots. +This next command restores the last copies of the files shown in the first +listing above into the "current" directory, as they were all from the "current" +snapshot. + + # byteback-restore --restore '/home/patch/.bash*' + Restoring: + snapshot modestring size uid gid mtime path + ---------------------------------------------------------------------------------------- + current -rw------- 47931 1000 1000 2015-11-29 09:26:10 +0000 /home/patch/.bash_history + current -rw-r--r-- 220 1000 1000 2014-03-10 10:13:33 +0000 /home/patch/.bash_logout + current -rw-r--r-- 3638 1000 1000 2015-07-30 16:33:30 +0100 /home/patch/.bashrc + +Using find(1), we can see the files restored. Note that the permissions on the +parent directory have been set correctly. + + # find current/ -ls + 422049 4 drwxr-xr-x 3 root root 4096 Nov 16 16:04 current/ + 422050 4 drwxr-xr-x 3 root root 4096 Mar 12 2014 current/home + 422885 4 drwxr-xr-x 2 patch patch 4096 Nov 29 09:26 current/home/patch + 422886 4 -rw-r--r-- 1 patch patch 220 Mar 10 2014 current/home/patch/.bash_logout + 422887 4 -rw-r--r-- 1 patch patch 3638 Jul 30 16:33 current/home/patch/.bashrc + +To restore the files from a specific snapshot, use the --snapshot flag. + + # byteback-restore --restore --snapshot '2015-07-17T15:11+0100' '/home/patch/.bash*' + Restoring: + snapshot modestring size uid gid mtime path + ----------------------------------------------------------------------------------------------------- + 2015-07-17T15:11+0100 -rw------- 38504 1000 1000 2015-07-17 13:42:17 +0100 /home/patch/.bash_history + 2015-07-17T15:11+0100 -rw-r--r-- 220 1000 1000 2014-03-10 10:13:33 +0000 /home/patch/.bash_logout + 2015-07-17T15:11+0100 -rw-r--r-- 3637 1000 1000 2014-03-10 10:13:33 +0000 /home/patch/.bashrc + +These have been restored into the 2015-07-17T15:11+0100 directory. + + # find 2015-07-17T15:11+0100 -type f -ls + 422911 4 -rw-r--r-- 1 patch patch 220 Mar 10 2014 2015-07-17T15:11+0100/home/patch/.bash_logout + 422913 40 -rw------- 1 patch patch 38504 Jul 17 13:42 2015-07-17T15:11+0100/home/patch/.bash_history + 422912 4 -rw-r--r-- 1 patch patch 3637 Mar 10 2014 2015-07-17T15:11+0100/home/patch/.bashrc + +Finally, to restore all available copies of the files, use the --all flag. +This will restore all copies shown when --list is used with the --all flag. + + # byteback-restore --restore --all '/home/patch/.bashrc' + Restoring: + snapshot modestring size uid gid mtime path + ----------------------------------------------------------------------------------------------------- + 2015-07-17T15:11+0100 -rw-r--r-- 3637 1000 1000 2014-03-10 10:13:33 +0000 /home/patch/.bashrc + current -rw-r--r-- 3638 1000 1000 2015-07-30 16:33:30 +0100 /home/patch/.bashrc + +We now have two versions of the same file, in different restore directories. + + # find -type f -ls + 946020 4 -rw-r--r-- 1 patch patch 3637 Mar 10 2014 ./2015-07-17T15:11+0100/home/patch/.bashrc + 422893 4 -rw-r--r-- 1 patch patch 3638 Jul 30 16:33 ./current/home/patch/.bashrc + +CAVEATS + +If a pattern given matches a directory, then the whole directory will be +restored recursively, not just the directory itself. This can be shown by +adding the --verbose flag. + + # byteback-restore --verbose --restore '/home/patch' + rsync --archive --numeric-ids --inplace --relative --compress --timeout 300 --rsh ssh -o BatchMode=yes -x -a -i /etc/byteback/key -l byteback --verbose --rsync-path byteback-restore --fake-super --snapshot * byteback@backups.default.cherry.uk0.bigv.io:/home/patch . + Restoring: + snapshot modestring size uid gid mtime path + ------------------------------------------------------------------------- + current drwxr-xr-x 5152 1000 1000 2015-11-29 09:26:22 +0000 /home/patch + rsync --fake-super --server --sender -vlogDtprRze.iLsfx --timeout=300 --numeric-ids --inplace --exclude=. . ./current/home/patch + receiving incremental file list + current/home/patch/ + current/home/patch/.mrconfig + current/home/patch/.pam_environment + current/home/patch/.plan + current/home/patch/.profile + current/home/patch/.psql_history + etc... + + +SEE ALSO + +byteback-receive(1), byteback-prune(1), http://ruby-doc.org/core/Dir.html#method-c-glob + +AUTHOR + +Patrick J Cherry <patrick@bytemark.co.uk> + + |