First commit of working code, needs a lot more writing and testing

7 files changed, 675 insertions, 51 deletions

diff --git a/README.md b/README.md
index 6bb68b2..85926e1 100644
--- a/README.md
+++ b/README.md
@@ -6,82 +6,90 @@ with easy client and server setup.
 
 "Maintenance-free" means that we'd rather make full use of a fixed amount of
 disc space with simple & predictable rules.  Management of disc space must be
-completely automatic, so it never grinds to a halt.  Failed backups can be
-restarted in case of network problems.
+completely automatic, so the process never grinds to a halt for reasons that
+could be automatically resolved.  Failed backups can be restarted in case of
+network problems.
 
 We use the standard OpenSSH on the server for encrypted transport & access 
-control, btrfs for snapshots and rsync for efficient transfer.
+control, btrfs for simple snapshots and rsync for efficient data transfer
+across the network.
 
 Backups should require as little configuration as possible to be safe - just
-the server address will be enough.
+the server address should be enough.
 
 Setting up: server
 ------------------
 Install the 'byteback' package on the server, along with its dependencies
 (rsync, sudo).
 
-Create a UNIX user to receive the backups e.g. 'byteback', create a btrfs
-home directory with quotas enabled.
-
-# adduser byteback
-# echo 'byteback btrfs subvolume' XXX >>/etc/sudoers
-# lvcreate my_volume_group -n byteback -L1000GB
-# echo '/dev/my_volume_group /home/byteback btrfs compress 0 0' >>/etc/fstab
-# mount /home/byteback
-# btrfs quota enable /home/byteback
-
-The server is launched from the 'byteback' user with OpenSSH as the transport,
-so there is no special daemon to start, but you do need to set up the program's
-data directory which is done with 
-
-# su byteback
-$ byteback-server setup
-
-That's it!  You're now ready to start backing up your first client.
+You then need to perform the following local setup on the server, which can
+securely handle backups for multiple clients.  The following commands are
+appropriate for a Debian system, you might need to alter it for other Linux
+distributions:
+
+	# Create a dedicated UNIX user which will store everyone's backups, and
+	# allow logins
+	#
+	adduser --system byteback --home /byteback --shell /bin/bash
+
+	# Allow the backup user to run the snapshot command
+	#
+	# echo <<SUDOERS >/etc/sudoers.d/byteback
+	byteback ALL = (root) NOPASSWD: /usr/local/bin/byteback-snapshot
+	byteback ALL = (root) NOPASSWD: /usr/bin/byteback-snapshot
+	byteback ALL = (root) NOPASSWD: /sbin/btrfs subvolume create
+	Defaults:byteback !requiretty
+	SUDOERS
+
+	# Create a dedicated btrfs filesystem for the user, and add that as its home
+	#
+	lvcreate my_volume_group --name byteback --size 1000GB
+	mkfs.btrfs /dev/my_volume_group/byteback
+	echo '/dev/my_volume_group/byteback /byteback btrfs compress 0 0' >>/etc/fstab
+	mount /byteback
 
 Setting up: client
 ------------------
-Clients are machines that need to be backed up.  You need to tell each client
-where its server is using normal SSH user/host syntax:
-
-# byteback setup byteback@mybackuphost.net
-Your backup key is ssh-rsa AAAAAo ... w== root@host.to.back.up
-This will create keys for communication with the server, and put them into
-/etc/byteback.  
-
-You need to then log onto the server and inform it of this client, by using
-the "byteback-server new-client" command and supplying the SSH public key.
+Clients are machines that need to be backed up.  Assuming you can log into
+the remote 'byteback' user with a password or administrative key, you only
+need to type one command on the client to set things going:
 
-# su byteback
-$ byteback-server new-client ssh-rsa AAAAAo ... w== root@host.to.back.up
-Client setup for host.to.back.up done!
+	byteback-setup-client --destination byteback@mybackuphost.net:
 
-This will have created a new directory and subvolume for this host's backups.
-Then back on the client:
+If this goes OK, you are ready to start backing up.  I'd advise taking the
+first backup manually to make sure it goes as you expect.  Type this on the
+client to start and watch the backup.
 
-# byteback test
-Connecting to server mybackuphost.net...
-The authenticity of host 'mybackuphost.net (10.10.10.1)' can't be established.
-RSA key fingerprint is c8:f5:bf:75:1b:34:6f:08:24:04:ba:a2:71:9f:5d:22.
-Are you sure you want to continue connecting (yes/no)? yes
-Successfully connected and found backup space, ready to go.
+	byteback-backup --verbose
 
-This means the host is ready to start backing up, though you need to set
-a schedule.
-
-Setting a backup schedule
--------------------------
-You can then type "byteback backup" or put it on a daily cron job to start
-backing up the server.
+Configuring byteback-backup
+---------------------------
+You can now set "byteback backup"  on a daily cron job to start backing up the
+server on a regular basis.
 
 Without any further options this will copy every file from the root downwards,
 excluding kernel-based virtual filesystems (/proc, /sys etc.) network 
 filesystems (NFS, SMB) and tmpfs or loopback mounts.
 
+It currently excludes /swap.file and /var/backups/localhost which (on Bytemark
+systems) do not need to be part of any backup.
+
 When the backup has completed successfully, the server will take a snapshot
-so that the client can't alter the backups.
+so that the client can't alter the backups, and then "prune" the backup 
+snapshots to ensure that the next backup is likely to run OK.
 
 If the backup is interrupted or dies unexpected, running "byteback backup" 
 will cause the backup to be resumed, with rsync saving the work of re-copying
 any files that hadn't changed.  By default this will happen automatically up to 
 5 times, with a 10 minute pause in between each attempt.
+
+Pruning behaviour
+-----------------
+
+Features to come
+----------------
+* spotting a /var/lib/mysql directory and making a safe snapshot and re-copy
+  of a MySQL data directory (using FLUSH TABLES WITH READ LOCK)
+
+* (same for postgres using pg_start_backup() and pg_stop_backup())
+
diff --git a/byteback b/byteback
new file mode 100644
index 0000000..7418006
--- /dev/null
+++ b/byteback
@@ -0,0 +1,65 @@
+#!/usr/bin/ruby
+#
+# byteback backup script prototype
+#
+# (c) Bytemark Hosting 2013
+#
+# 
+VERSION='prototype'
+
+HOSTNAME=`hostname -f`
+
+mode = ARGV.shift
+case mode
+when 'backup'
+
+    @destination_host = HOSTNAME.split(".")[2..-1].join(".")
+
+	system *<<-CMD.split(/\s+/)
+
+rsync 
+
+  --rsync-path 
+    rsync --fake-super
+  --rsh 
+    ssh -i /etc/bytebackup/bytebackup.key
+  --delete
+  --one-file-system
+  --archive
+  --exclude
+    /swap.file
+
+  /
+
+  #{@destination_ssh}
+
+	CMD
+when 'backup-receive'
+
+else
+	print <<-SYNTAX
+byteback v#{VERSION}, a focused backup tool
+
+Usage: byteback <mode>
+
+Modes:
+    server-setup
+    client-setup
+    backup
+    backup-receive
+
+Type 'bytebackup help <mode>' for more information on a mode, or
+see the man page.
+	SYNTAX
+	exit 1
+end
+
+require 'trollop'
+opts = Trollop::options do
+	opt :mode, "Program mode to run", 
+		:default => :backup,
+		:required,
+		:type => String
+
+	opt 
+:end
\ No newline at end of file
diff --git a/byteback-backup b/byteback-backup
new file mode 100644
index 0000000..52a7d28
--- /dev/null
+++ b/byteback-backup
@@ -0,0 +1,187 @@
+#!/usr/bin/ruby
+#
+# Back up this system to a byteback-enabled server (just some command line
+# tools and SSH setup).  We aim to make sure this backups are easy, complete 
+# and safe for most types of hosting customer.
+#
+# See 'man byteback' for more information.
+
+require 'trollop'
+require 'resolv'
+
+HOSTNAME = `hostname -f`
+
+@sources = ["/"]
+@exclude = ["/swap.file", "/var/backups/localhost"]
+
+def error(message)
+	STDERR.print "*** #{message}\n"
+	exit 1
+end
+
+def verbose(message)
+	print "#{message}\n"
+end
+
+opts = Trollop::options do
+
+	opt :destination, "Backup destination (i.e. user@host:/path)", 
+	  :type => :string
+
+	opt :source, "Source paths",
+	  :type => :strings
+
+	opt :verbose, "Show rsync command and progress"
+
+	opt :retry_number, "Number of retries on error",
+	  :type => :integer,
+	  :default => 3
+
+	opt :retry_delay, "Wait number of seconds between retries",
+	  :type => :integer,
+	  :default => 1800
+
+	opt :ssh_key, "SSH key for connection",
+	  :type => :string,
+	  :default => "/etc/byteback/key"
+end
+
+@ssh_key = opts[:ssh_key]
+@verbose = opts[:verbose] ? "--verbose" : nil
+@sources = opts[:source] if opts[:source]
+@destination = opts[:destination]
+@retry_number = opts[:retry_number]
+@retry_delay = opts[:retry_delay]
+
+if !@destination && File.exists?("/etc/byteback/destination")
+	@destination = File.read("/etc/byteback/destination").chomp
+end
+error("Must suply --destination or put it into /etc/bytebackup/destination") unless @destination
+
+_dummy, @destination_user, @destination_host, colon, @destination_path = 
+  /^(.*)?(?:@)([^:]+)(:)(.*)?$/.match(@destination).to_a
+
+error("Must be a remote path") unless colon
+
+# Validate & normalise source directories
+#
+error("No sources specified") if @sources.empty?
+@sources = @sources.map do |s|
+	s = s.gsub(/\/+/,"/")
+	error("Can't read directory #{s}") unless File.readable?(s)
+	s
+end
+
+# Guess destination for backup
+#
+if !@destination
+	guesses = []
+	Resolv::DNS.open do |dns|
+		suffix = HOSTNAME.split(".")[2..-1].join(".")
+		["byteback." + suffix].each do |name|
+			[Resolv::DNS::Resource::IN::AAAA,
+			 Resolv::DNS::Resource::IN::A].each do |record_type|
+			 	next if !guesses.empty? # only care about first result
+				guesses += dns.getresources(name, record_type)
+			end
+		end
+	end
+
+	if guesses.empty?
+		error "Couldn't guess at backup host, please specify --destination"
+	end
+
+	# ick, do I really have to do this to get a string represnetation of
+	# the IP address?
+	#
+	guess = guesses.first.inspect
+	match = / (.*)>$/.match(guess)[1]
+	error "Result #{guesses} is not an IP" if !match
+	@destination = "byteback@#{match[1]}:#{HOSTNAME}/current/"
+
+	verbose "Guessed destination=#{@destination} from #{guess}"
+end
+
+# Test ssh connection is good before we start
+#
+error("Could not read ssh key #{@ssh_key}") unless File.readable?(@ssh_key)
+
+error("Could not connect to #{@destination}") unless system(
+	"ssh",
+	"-a",
+	"-i",
+	@ssh_key,
+	@destination.split(":")[0],
+	"byteback-receive",
+	"--ping",
+	"#{@verbose}"
+)
+
+# Call rsync to copy certain sources, returns exit status (see man rsync)
+#
+def rsync(*sources)
+	# Default options include --inplace because we only care about consistency
+	# at the end of the job, and rsync will do more work for big files without
+	# it.
+	#
+	args = [
+		"rsync",
+		"--archive",
+  		"--numeric-ids",
+  		"--delete",
+  		"--inplace",
+  		"--rsync-path",
+    		"rsync --fake-super",
+  		"--rsh",
+    		"ssh -a -i #{@ssh_key} -l #{@destination_user}",
+  		"--delete",
+  		"--one-file-system",
+  		"--relative"
+  	]
+
+  	args << "--verbose" if @verbose
+	args += @exclude.map { |x| ["--exclude", x] }.flatten
+	args += sources
+  	args << @destination
+
+	print args.map { |a| / /.match(a) ? "\"#{a}\"" : a }.join(" ")+"\n" if @verbose
+  	system(*args)
+
+  	return $?.exitstatus
+end
+
+RSYNC_EXIT_STATUSES_TO_RETRY_ON = [10,11,20,21,22,23,24,30]
+
+# Run the file copy, retrying if necessary
+# 
+loop do
+	status = rsync(*@sources)
+
+	if status === 0
+		break
+	elsif RSYNC_EXIT_STATUSES_TO_RETRY_ON.include?(status)
+		if @retry_number > 0
+			@retry_number -= 1
+			sleep @retry_delay
+			redo
+		else
+			error("Maximum number of rsync retries reached")
+		end
+	else
+		error("Fatal rsync error occurred (#{status})")
+	end
+end
+
+# Mark the backup as done on the other end
+#
+error("Backup could not be marked complete") unless system(
+	"ssh",
+	"-a",
+	"-i",
+	@ssh_key,
+	@destination.split(":")[0],
+	"sudo",
+	"byteback-snapshot",
+	"--snapshot",
+	"#{@verbose}"
+)
diff --git a/byteback-receive b/byteback-receive
new file mode 100644
index 0000000..1d87f6f
--- /dev/null
+++ b/byteback-receive
@@ -0,0 +1,52 @@
+#!/usr/bin/ruby
+#
+# Program to receive backups and run rsync in receive mode.  Must check that
+# user as authorised by SSH is allowed to access particular directory.
+
+require 'trollop'
+
+def error(message)
+	STDERR.print "*** #{message}\n"
+	exit 1
+end
+
+#STDERR.print "ARGV=#{ARGV.inspect}\nSSH_ORIGINAL_COMMAND=#{ENV['SSH_ORIGINAL_COMMAND']}\n"
+
+if ENV['SSH_ORIGINAL_COMMAND']
+	ARGV.concat(ENV['SSH_ORIGINAL_COMMAND'].split(" "))
+end
+
+#STDERR.print "after ARGV=#{ARGV.inspect}"
+
+byteback_host = ENV['BYTEBACK_HOST']
+error("BYTEBACK_HOST environment not set") unless byteback_host
+
+byteback_root = ENV['HOME'] + "/" + ENV["BYTEBACK_HOST"]
+error("#{byteback_root} does not exist") unless File.directory?(byteback_root)
+
+# force destination to be where we expect
+#
+if ARGV[0] == 'rsync'
+	ARGV[-1] = "#{byteback_root}/current"
+	exec(*ARGV)
+elsif ARGV[0] == 'byteback-snapshot' || (ARGV[0] == 'sudo' && ARGV[1] == 'byteback-snapshot')
+	ARGV.concat(["--root", "#{byteback_root}"])
+	exec(*ARGV)
+end
+
+opts = Trollop::options do
+	opt :verbose, "Print diagnostics"
+	opt :ping, "Check connection parameters and exit"
+	opt :complete, "Mark current backup as complete"
+end
+
+error("Please only choose one mode") if opts[:ping] && opts[:complete]
+if opts[:complete]
+	system("byteback-snapshot", byteback_root)
+elsif opts[:ping]
+	exit 0
+else
+	STDERR.print "byteback-receive failed\n"
+	exit 9
+end
+
diff --git a/byteback-setup-client b/byteback-setup-client
new file mode 100644
index 0000000..ddb6672
--- /dev/null
+++ b/byteback-setup-client
@@ -0,0 +1,54 @@
+#!/usr/bin/ruby
+#
+# Run on a client machine to set up backups for the first time
+
+require 'fileutils'
+require 'trollop'
+
+def error(message)
+	STDERR.print "*** #{message}\n"
+	exit 1
+end
+
+def verbose(message)
+	print "#{message}\n"
+end
+
+opts = Trollop::options do
+
+	opt :hostname, "Set host name for backups",
+	  :type => :string
+
+	opt :destination, "Backup destination (i.e. user@host:/path)", 
+	  :type => :string
+
+end
+
+@destination = opts[:destination]
+@hostname = opts[:hostname]
+
+_dummy, @destination_user, @destination_host, colon, @destination_path = 
+  /^(.*)?(?:@)([^:]+)(:)(.*)?$/.match(@destination).to_a
+
+error("Must be a remote path") unless colon
+if !@hostname
+	@hostname = `hostname -f`
+	print "No hostname set, using #{@hostname}"
+end
+
+error "This host already appears set up - you need to delete /etc/byteback if not" if
+  File.readable?("/etc/byteback/key")
+
+FileUtils.mkdir_p("/etc/byteback")
+
+error "Couldn't generate SSH key" unless 
+  system("ssh-keygen -q -t rsa -C \"byteback client key\" -N \"\" -f /etc/byteback/key")
+
+key_pub = File.read("/etc/byteback/key.pub")
+
+error "Setup didn't work" unless
+  system("ssh -i /etc/byteback/key -l #{@destination_user} #{@destination_host} byteback-setup-client-receive #{@hostname} #{key_pub}")
+
+File.open("/etc/byteback/destination", "w") { |f| f.print @destination }
+
+print "Setup worked!  To take your first backup run: byteback-backup --verbose\n"
diff --git a/byteback-setup-client-receive b/byteback-setup-client-receive
new file mode 100644
index 0000000..35a3b65
--- /dev/null
+++ b/byteback-setup-client-receive
@@ -0,0 +1,42 @@
+#!/usr/bin/ruby
+#
+# Called by byteback-setup-client to set up a new byteback-setup-client
+
+require 'fileutils'
+
+def error(message)
+	STDERR.print "*** #{message}\n"
+	exit 1
+end
+
+@hostname = ARGV.shift
+@pubkey = ARGV.join(" ")
+
+error("You must call this from byteback-setup-client on remote host") unless
+  @hostname && 
+  /^ssh/.match(@pubkey) &&
+  ENV['SSH_CONNECTION']
+
+@client_ip = ENV['SSH_CONNECTION'].split(" ").first
+
+Dir.chdir(ENV['HOME']) # don't know why we wouldn't be here
+
+Dir.mkdir(@hostname)
+
+error("Couldn't create btrfs subvolume (needs sudo)") unless 
+	system("sudo btrfs subvolume create #{@hostname}/current")
+
+FileUtils.mkdir_p(".ssh")
+
+error("This key already exists in .ssh/authorized_keys on server") if
+	File.exists?(".ssh/authorized_keys") &&
+	File.read(".ssh/authorized_keys").match(@pubkey.split(/\s+/)[1])
+
+File.open(".ssh/authorized_keys", "a+") do |fh|
+	fh.print <<-LINE.gsub(/\n/,"")
+command="byteback-receive",
+from="#{@client_ip}",
+environment="BYTEBACK_HOST=#{@hostname}"
+ #{@pubkey}
+	LINE
+end
diff --git a/byteback-snapshot b/byteback-snapshot
new file mode 100644
index 0000000..dfd216f
--- /dev/null
+++ b/byteback-snapshot
@@ -0,0 +1,216 @@
+#!/usr/bin/ruby
+#
+# Program to create a snapshot and/or rotate a directory of backup snapshots 
+# using btrfs subvolume commands.
+
+require 'trollop'
+require 'time'
+
+def error(message)
+	STDERR.print "*** #{message}\n"
+	exit 1
+end
+
+def verbose(message)
+	print "#{Time.now}: #{message}\n" if @verbose
+end
+
+# Icky way to find out free disc space on our mount
+#
+class DiskFree
+	def initialize(mount)
+		@mount = mount
+	end
+
+	def total
+		all[2]
+	end
+
+	def used
+		all[3]
+	end
+
+	def available
+		all[4]
+	end
+
+	def fraction_used
+		disk_device, disk_fs, disk_total, disk_used, disk_available, *rest = all
+		disk_used.to_f / disk_available
+	end
+
+	protected
+
+	def all
+		disk_device, disk_fs, disk_total, disk_used, disk_available, *rest = 
+			df.
+			split("\n")[1].
+			split(/\s+/).
+			map { |i| /^[0-9]+$/.match(i) ? i.to_i : i }
+	end
+
+	def df
+		`/bin/df -T -P -B1 #{@mount}`
+	end
+end
+
+# Represent a directory full of backups where "current"  is a subvolume
+# which is snapshotted to frozen backup directories called e.g. 
+# "yyyy-mm-ddThh:mm".
+#
+class BackupDirectory
+	attr_reader :dir
+
+	def initialize(dir)
+		@dir = Dir.new(dir)
+		current
+	end
+
+	# Return total amount of free space in backup directory (bytes)
+	#
+	def free
+		df = DiskFree.new(@dir.path)
+		df.total - df.used
+	end
+
+	# Return an array of Times representing the current list of 
+	# snapshots.
+	#
+	def snapshot_times
+		@dir.entries.map do |entry|
+			begin
+				Time.parse(entry)
+			rescue ArgumentError => error
+				nil
+			end
+		end.
+		compact.
+		sort
+	end
+
+	# What order to remove snapshots in to regain disk space?
+	#
+	# Order backups by their closeness to defined backup times, which are
+	# listed in a set order (i.e. today's backup is more important than yesterday's).
+	#
+	BACKUP_IMPORTANCE = [0, 1, 2, 3, 7, 14, 21, 28, 56, 112]
+	def snapshot_times_by_importance
+		now = Time.now
+		snapshot_times_unsorted = snapshot_times
+		snapshot_times_sorted = []
+		while !snapshot_times_unsorted.empty?
+			BACKUP_IMPORTANCE.each do |days|
+				target_time = now + (days*86400)
+				closest = snapshot_times_unsorted.inject(nil) do |best, time|
+					if best.nil? || (time-target_time).abs < (best-target_time).abs
+						time
+					else
+						best
+					end
+				end
+				break unless closest
+				snapshot_times_sorted << snapshot_times_unsorted.delete(closest)
+			end
+		end
+		snapshot_times_sorted
+	end
+
+	# Returns the size of the last completed snapshot (runs du, maybe slow)
+	#
+	# Would much prefer to take advantage of this feature:
+	#   http://dustymabe.com/2013/09/22/btrfs-how-big-are-my-snapshots/
+	# but it's not currently in Debian/wheezy.
+	#
+	def latest_snapshot_size
+		`du -s -b #{snapshot_path(snapshot_times.last)}`.to_i
+	end
+
+	# Create a new snapshot of 'current'
+	#
+	def new_snapshot!
+		system_no_error("btrfs subvolume snapshot -r #{current.path} #{snapshot_path}")
+	end
+
+	def delete_snapshot!(time)
+		system_no_error("btrfs subvolume delete #{snapshot_path(time)}")
+	end
+
+	def current
+		Dir.new("#{dir.path}/current")
+	end
+
+	def snapshot_path(time=Time.now)
+		"#{dir.path}/#{time.strftime("%Y-%m-%dT%H:%M")}"
+	end
+
+	protected
+
+	def system_no_error(*args)
+		raise RuntimeError.new("Command failed: "+args.join(" ")) unless
+		  system(*args)
+	end
+end
+
+opts = Trollop::options do
+
+    opt :root, "Backups directory (must be a btrfs subvolume)",
+    	:type => :string
+
+    opt :snapshot, "Take a new snapshot"
+
+    opt :prune, "Prune old backups"
+
+    opt :list, "List backups (by 'age' or 'importance')",
+    	:type => :string
+
+    opt :verbose, "Print diagnostics"
+
+end
+
+@root = opts[:root]
+@verbose = opts[:verbose]
+@do_snapshot = opts[:snapshot]
+@do_list = opts[:list]
+@do_prune = opts[:prune]
+
+error("Must specify snapshot, prune or list") unless @do_snapshot || @do_prune || @do_list
+
+error("--root not readable") unless File.directory?(@root)
+
+@backups = BackupDirectory.new(@root)
+
+
+if @do_snapshot
+	verbose "Making new snapshot"
+	@backups.new_snapshot! 
+end
+
+if @do_list
+	list = @do_list == 'age' ? 
+		@backups.snapshot_times : # oldest first
+		@backups.snapshot_times_by_importance.reverse # least important first
+	print "Backups in #{@root} by #{@do_list}:\n"
+	list.each_with_index do |time, index|
+		print "#{sprintf('% 3d',index)}: #{time}\n"
+	end
+end
+
+if @do_prune
+	target_free_space = @backups.latest_snapshot_size
+	verbose "Want to ensure we have #{target_free_space}"
+
+	if @backups.free >= target_free_space
+		verbose "(we have #{@backups.free} so no action needed)"
+	else
+		snapshot_times_by_importance = @backups.snapshot_times_by_importance
+
+		while @backups.free < target_free_space && !snapshot_times_by_importance.empty?
+			to_delete = snapshot_times_by_importance.pop
+			verbose "Deleting #{to_delete}"
+			@backups.delete_snapshot!(to_delete)
+			verbose "Leaves us with #{@backups.free}"
+		end
+	end
+end
+
+verbose "Finished"