From 68d26b80b823ab99b8cd6ee6e5f0b1265963b2bf Mon Sep 17 00:00:00 2001 From: Steve Kemp Date: Sat, 23 Apr 2016 07:23:00 +0300 Subject: Minor reformatting updates. --- README.md | 69 ++++++++++++++++++++++----------------------------------------- 1 file changed, 24 insertions(+), 45 deletions(-) diff --git a/README.md b/README.md index fb10d43..b02071d 100644 --- a/README.md +++ b/README.md @@ -1,42 +1,31 @@ byteback - maintenance-free client & server backup scripts for Linux ==================================================================== -byteback encapsulates Bytemark's "best practice" for maintenance-free backups +`byteback` encapsulates Bytemark's "best practice" for maintenance-free backups with easy client and server setup. -"Maintenance-free" means that we'd rather make full use of a fixed amount of -disc space. Management of disc space must be completely automatic, so the -process never grinds to a halt for reasons that could be automatically -resolved. Failed backups can be restarted in case of network problems. +"Maintenance-free" means that we'd rather make full use of a fixed amount of disc space. Management of disc space must be completely automatic, so the process never grinds to a halt for reasons that could be automatically resolved. Failed backups can be restarted in case of network problems. -We use the standard OpenSSH on the server for encrypted transport & access -control, btrfs for simple snapshots and rsync for efficient data transfer -across the network. +We use the standard OpenSSH on the server for encrypted transport & access control, `btrfs` for simple snapshots and `rsync` for efficient data transfer across the network. + +Backups should require as little configuration as possible to be safe - just the server address should be enough. -Backups should require as little configuration as possible to be safe - just -the server address should be enough. Setting up: server ------------------ -Install the 'byteback' package on the server, along with its dependencies -(rsync and ruby-ffi). +Install the '`byteback`' package on the server, along with its dependencies. -You then need to perform the following local setup on the server, which can -securely handle backups for multiple clients. You need a dedicated user -(which is usually called 'byteback') with a home directory on a btrfs -filesystem. You will need to mount the filesystem with the -'user_subvol_rm_allowed' flag to enable pruning to work (or run that part -as root). +You then need to perform the following local setup on the server, which can securely handle backups for multiple clients. You need a dedicated user (which is usually called 'byteback') with a home directory on a btrfs filesystem. You will need to mount the filesystem with the '`user_subvol_rm_allowed`' flag to enable pruning to work (or run that part as root). -The following commands are appropriate for a Debian system, you might need -to alter it for other Linux distributions, or if you are not using LVM -for your discs: +The following commands are appropriate for a Debian system, you might need to alter it for other Linux distributions, or if you are not using LVM for your discs: + # # Create a dedicated UNIX user which will store everyone's backups, and # allow logins # adduser --system byteback --home /byteback --shell /bin/bash + # # Create a dedicated btrfs filesystem for the user, and add that as its home # lvcreate my_volume_group --name byteback --size 1000GB @@ -46,62 +35,53 @@ for your discs: chown byteback /byteback chmod u+w /byteback -Finally, before setting up the client, add +Finally, before setting up the client you should add the following to `/etc/ssh/sshd_config`, and restart the ssh-service: PermitUserEnvironment yes -to /etc/ssh/sshd_config, and restart sshd. Setting up: client ------------------ -Clients are machines that need to be backed up. Assuming you can log into -the remote 'byteback' user with a password or administrative key, you only -need to type one command on the client to set things going: +Clients are machines that need to be backed up. Assuming you can log into the remote '`byteback`' user with a password or administrative key, you only need to type one command on the client to set things going: sudo byteback-setup-client --destination byteback@mybackuphost.net: -If this goes OK, you are ready to start backing up. I'd advise taking the -first backup manually to make sure it goes as you expect. Type this on the -client to start and watch the backup. +If this goes OK, you are ready to start backing up. I'd advise taking the first backup manually to make sure it goes as you expect. Type this on the client to start and watch the backup. sudo byteback-backup --verbose + Configuring byteback-backup --------------------------- -This is now documented in the manapge for byteback-backup(1). +This is now documented in the manpage for byteback-backup(1). + Viewing and restoring backups ----------------------------- This is now documented in the manapge for byteback-restore(1). + The trust model --------------- -Backups are intended to keep your data safe, and byteback makes the assumption -that the client may become hostile to the backup server. At Bytemark this -allows us to guard against rogue employees of our clients destroying the backup, -while ensuring that our clients can still access all their old backups. There -are several measures to guard against this, though they are all ineffective -over a long enough period of time: +Backups are intended to keep your data safe, and byteback makes the assumption that the client may become hostile to the backup server. At Bytemark this allows us to guard against rogue employees of our clients destroying the backup, while ensuring that our clients can still access all their old backups. There are several measures to guard against this, though they are all ineffective over a long enough period of time: -* the server uses SSH's command feature to ensure that clients can only - run rsync to the appropriate directory; +* the server uses SSH's command feature to ensure that clients can only run rsync to the appropriate directory; -* the server's snapshots are read-only, so the client can't just rsync an - empty directory over an old backup; +* the server's snapshots are read-only, so the client can't just rsync an empty directory over an old backup; -* the server will refuse to take snapshots "too often" to stop the client - from filling the disc with useless data; +* the server will refuse to take snapshots "too often" to stop the client from filling the disc with useless data; + +* the server will refuse to prune away space for a new backup that is suddenly larger than previous ones. -* the server will refuse to prune away space for a new backup that is - suddenly larger than previous ones. Pruning behaviour ----------------- This is now documented in byteback-prune(1). + Acknowledgements ---------------- For maximum portability, I've included three libraries. Thanks very much to @@ -110,4 +90,3 @@ their authors: * sys-filesystem by Daniel J. Berger: https://github.com/djberg96/sys-filesystem * trollop by William Morgan: https://github.com/wjessop/trollop * ffi-xattr by Jari Bakken: https://github.com/jarib/ffi-xattr - -- cgit v1.2.1 From 0f443fc795832b5d3a5b9bd134ce36799caea843 Mon Sep 17 00:00:00 2001 From: Steve Kemp Date: Sat, 23 Apr 2016 07:29:46 +0300 Subject: Updated public links to point to github --- debian/control | 4 ++-- debian/copyright | 2 +- 2 files changed, 3 insertions(+), 3 deletions(-) diff --git a/debian/control b/debian/control index 7c17d0e..5fa9d73 100644 --- a/debian/control +++ b/debian/control @@ -6,8 +6,8 @@ Uploaders: Steve Kemp Build-Depends: debhelper (>= 7.0.50~), txt2man Standards-Version: 3.9.6 # Vcs-Git: -Vcs-Browser: https://projects.bytemark.co.uk/projects/byteback/repository -Homepage: https://projects.bytemark.co.uk/projects/byteback +Vcs-Browser: https://github.com/BytemarkHosting/byteback.git +Homepage: https://github.com/BytemarkHosting/byteback Package: byteback Architecture: all diff --git a/debian/copyright b/debian/copyright index 9dbafb4..c9616c0 100644 --- a/debian/copyright +++ b/debian/copyright @@ -1,6 +1,6 @@ Format: http://www.debian.org/doc/packaging-manuals/copyright-format/1.0/ Upstream-Name: byteback -Source: +Source: Files: lib/trollop.rb Copyright:: 2007 William Morgan -- cgit v1.2.1 From a1069f8e4d8d061c1c74385a2efb0bd33be02521 Mon Sep 17 00:00:00 2001 From: Igor Vuk Date: Sat, 23 Apr 2016 22:14:59 +0200 Subject: Fix a typo in README.md --- README.md | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/README.md b/README.md index b02071d..d58ed46 100644 --- a/README.md +++ b/README.md @@ -60,7 +60,7 @@ This is now documented in the manpage for byteback-backup(1). Viewing and restoring backups ----------------------------- -This is now documented in the manapge for byteback-restore(1). +This is now documented in the manpage for byteback-restore(1). The trust model -- cgit v1.2.1 From a091630ee5c4851ef849b5f2642d156558ea22a2 Mon Sep 17 00:00:00 2001 From: Igor Vuk Date: Sat, 23 Apr 2016 22:17:39 +0200 Subject: Use plural when referring to 'commands' --- README.md | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/README.md b/README.md index d58ed46..5e43abf 100644 --- a/README.md +++ b/README.md @@ -17,7 +17,7 @@ Install the '`byteback`' package on the server, along with its dependencies. You then need to perform the following local setup on the server, which can securely handle backups for multiple clients. You need a dedicated user (which is usually called 'byteback') with a home directory on a btrfs filesystem. You will need to mount the filesystem with the '`user_subvol_rm_allowed`' flag to enable pruning to work (or run that part as root). -The following commands are appropriate for a Debian system, you might need to alter it for other Linux distributions, or if you are not using LVM for your discs: +The following commands are appropriate for a Debian system, you might need to alter them for other Linux distributions, or if you are not using LVM for your discs: # # Create a dedicated UNIX user which will store everyone's backups, and -- cgit v1.2.1 From 1df51bd2c3a3ee53834c89a074e56c361b051460 Mon Sep 17 00:00:00 2001 From: Jamie Nguyen Date: Wed, 11 May 2016 08:28:06 +0100 Subject: Prevent accumulator from being set to Nil --- bin/byteback-receive | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/bin/byteback-receive b/bin/byteback-receive index 0d7d025..1326bba 100755 --- a/bin/byteback-receive +++ b/bin/byteback-receive @@ -116,7 +116,7 @@ end # Make sure we don't get crazy option combinations. # n_modes = opts.keys.inject(0) do |s,m| - s += 1 if [:ping, :complete, :list].include?(m) + [:ping, :complete, :list].include?(m) ? s += 1 : s end error('Please only choose one mode') unless n_modes == 1 -- cgit v1.2.1 From acc0d316c9dd628e3e2d6be438de83b6d7cf6638 Mon Sep 17 00:00:00 2001 From: Jamie Nguyen Date: Wed, 11 May 2016 08:30:04 +0100 Subject: Avoid chaining over a newline --- lib/byteback/backup_directory.rb | 6 ++---- 1 file changed, 2 insertions(+), 4 deletions(-) diff --git a/lib/byteback/backup_directory.rb b/lib/byteback/backup_directory.rb index 14a55cf..89b0ae9 100644 --- a/lib/byteback/backup_directory.rb +++ b/lib/byteback/backup_directory.rb @@ -150,8 +150,7 @@ module Byteback next if entry[0] == '.' name = File.expand_path(ENV['HOME'] + '/' + entry) File.directory?(name + '/current') ? BackupDirectory.new(name) : nil - end - .compact + end.compact end # Returns every snapshot in every backup directory @@ -190,8 +189,7 @@ module Byteback # directory name must represent a parseable Time nil end - end - .compact + end.compact end # Create a new snapshot of 'current' -- cgit v1.2.1 From 1c660fb1f712ead94bf80fb1ccb1a5d0d50198cd Mon Sep 17 00:00:00 2001 From: Jamie Nguyen Date: Wed, 11 May 2016 09:58:29 +0100 Subject: Add rpm spec file --- rpmbuild/byteback.spec | 66 ++++++++++++++++++++++++++++++++++++++++++++++++++ 1 file changed, 66 insertions(+) create mode 100644 rpmbuild/byteback.spec diff --git a/rpmbuild/byteback.spec b/rpmbuild/byteback.spec new file mode 100644 index 0000000..f422c1d --- /dev/null +++ b/rpmbuild/byteback.spec @@ -0,0 +1,66 @@ +Name: byteback +Version: 0.4.2 +Release: 1%{?dist} +Summary: Maintenance-free client & server backup scripts for Linux + +Group: Applications/System +License: Ruby and GPLv2+ and ASL 2.0 and Artistic 2.0 +URL: https://github.com/BytemarkHosting/byteback +Source0: byteback.tar.gz + +BuildArch: noarch +BuildRequires: txt2man +Requires: openssh-clients +Requires: ruby +Requires: rubygem-ffi +Requires: rsync + +%description +byteback encapsulates Bytemark's "best practice" for maintenance-free backups +with easy client and server setup. + +"Maintenance-free" means that we'd rather make full use of a fixed amount of +disc space. Management of disc space must be completely automatic, so the +process never grinds to a halt for reasons that could be automatically +resolved. Failed backups can be restarted in case of network problems. + +We use the standard OpenSSH on the server for encrypted transport & access +control, btrfs for simple snapshots and rsync for efficient data transfer +across the network. + +Backups should require as little configuration as possible to be safe - just +the server address should be enough. + + +%prep +%setup -q -n byteback +# No binary files, so should go in /usr/share. +sed -i -e 's|/usr/lib/byteback|/usr/share/byteback|g' bin/* +# Required to load system gems. +sed -i -e "2irequire 'rubygems'" bin/* + + +%build +make docs + + +%install +mkdir -p %{buildroot}%{_bindir} +cp -a bin/* %{buildroot}%{_bindir}/ + +mkdir -p %{buildroot}%{_datadir}/byteback +cp -a lib/* %{buildroot}%{_datadir}/byteback/ + +mkdir -p %{buildroot}%{_mandir}/man1 +for i in man/*.man; do + mv $i ${i%%.man}.1 +done; +cp -a man/*.1 %{buildroot}%{_mandir}/man1/ + + +%files +%doc README.md +%{_bindir}/* +%{_datadir}/byteback +%{_mandir}/man1/*.1* + -- cgit v1.2.1 From 908a41eaf04aaa196f8670396b96d2ff180872b2 Mon Sep 17 00:00:00 2001 From: Jamie Nguyen Date: Wed, 11 May 2016 10:31:01 +0100 Subject: Use orig.tar.gz --- rpmbuild/byteback.spec | 4 ++-- 1 file changed, 2 insertions(+), 2 deletions(-) diff --git a/rpmbuild/byteback.spec b/rpmbuild/byteback.spec index f422c1d..58841be 100644 --- a/rpmbuild/byteback.spec +++ b/rpmbuild/byteback.spec @@ -6,7 +6,7 @@ Summary: Maintenance-free client & server backup scripts for Linux Group: Applications/System License: Ruby and GPLv2+ and ASL 2.0 and Artistic 2.0 URL: https://github.com/BytemarkHosting/byteback -Source0: byteback.tar.gz +Source0: byteback_%{version}.orig.tar.gz BuildArch: noarch BuildRequires: txt2man @@ -33,7 +33,7 @@ the server address should be enough. %prep -%setup -q -n byteback +%setup -q -c # No binary files, so should go in /usr/share. sed -i -e 's|/usr/lib/byteback|/usr/share/byteback|g' bin/* # Required to load system gems. -- cgit v1.2.1 From d7e7be0de90d0a028be00a05de25048f379a1b1b Mon Sep 17 00:00:00 2001 From: James Carter Date: Wed, 27 Jul 2016 10:01:50 +0100 Subject: fixed a typo --- bin/byteback-backup | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) mode change 100755 => 100644 bin/byteback-backup diff --git a/bin/byteback-backup b/bin/byteback-backup old mode 100755 new mode 100644 index 0a50909..3e0d5af --- a/bin/byteback-backup +++ b/bin/byteback-backup @@ -253,7 +253,7 @@ if __FILE__ == $PROGRAM_NAME e.gsub(/\/+/, '/') end - fatal('Must suply --destination or put it into /etc/bytebackup/destination') unless @destination + fatal('Must supply --destination or put it into /etc/bytebackup/destination') unless @destination # # Test that we have an SSH-key which we can read. -- cgit v1.2.1 From e0ec7c2ff72ba158bf088af65cad6f6bad61f2c3 Mon Sep 17 00:00:00 2001 From: James Lawrie Date: Wed, 27 Jul 2016 12:48:03 +0100 Subject: Fixes the essence of #2 --- bin/byteback-backup | 10 +++++----- 1 file changed, 5 insertions(+), 5 deletions(-) diff --git a/bin/byteback-backup b/bin/byteback-backup index 3e0d5af..0b81e79 100644 --- a/bin/byteback-backup +++ b/bin/byteback-backup @@ -230,11 +230,11 @@ if __FILE__ == $PROGRAM_NAME COMMON_JUNK = %w( /swap.file - /tmp - /var/backups/localhost - /var/cache/apt/archives - /var/lib/php5 - /var/tmp + /tmp/ + /var/backups/localhost/ + /var/cache/apt/archives/ + /var/lib/php5/ + /var/tmp/ ) MOUNT_HEADINGS = %w( spec file vfstype mntops freq passno ).map(&:to_sym) -- cgit v1.2.1 From 2e96f802b1aa2cd374ef727c5573b30ab38001ce Mon Sep 17 00:00:00 2001 From: James Lawrie Date: Wed, 27 Jul 2016 12:48:50 +0100 Subject: Exclude /var/lib/mysql/ --- bin/byteback-backup | 1 + 1 file changed, 1 insertion(+) diff --git a/bin/byteback-backup b/bin/byteback-backup index 0b81e79..8268353 100644 --- a/bin/byteback-backup +++ b/bin/byteback-backup @@ -235,6 +235,7 @@ if __FILE__ == $PROGRAM_NAME /var/cache/apt/archives/ /var/lib/php5/ /var/tmp/ + /var/lib/mysql/ ) MOUNT_HEADINGS = %w( spec file vfstype mntops freq passno ).map(&:to_sym) -- cgit v1.2.1 From bcfddd9c4bec12260abcdf105d918ff82a279794 Mon Sep 17 00:00:00 2001 From: Steve Kemp Date: Wed, 21 Sep 2016 14:25:58 +0300 Subject: Generate a new `byteback-mysql` package. --- debian/control | 7 +++++++ 1 file changed, 7 insertions(+) diff --git a/debian/control b/debian/control index 5fa9d73..8623f60 100644 --- a/debian/control +++ b/debian/control @@ -27,3 +27,10 @@ Description: Maintenance-free client & server backup scripts for Linux . Backups should require as little configuration as possible to be safe - just the server address should be enough. + +Package: byteback-mysql +Architecture: all +Depends: ${shlibs:Depends}, ${misc:Depends}, perl, libdbi-perl, +Description: Maintenance-free backups for MySQL + leveraging byteback this package allows simple and transparent + backups of MySQL databases. -- cgit v1.2.1 From f61507920beeffa925c1dd13d76939e0352e2ae7 Mon Sep 17 00:00:00 2001 From: Steve Kemp Date: Wed, 21 Sep 2016 14:26:09 +0300 Subject: Import byteback-mysql scripts/files from jlawrie. --- byteback-mysql/post-backup.d/00-mysql | 27 +++++++ byteback-mysql/pre-backup.d/00-mysql | 31 ++++++++ byteback-mysql/scripts.d/Bytemyback/ini.pm | 91 ++++++++++++++++++++++ byteback-mysql/scripts.d/Bytemyback/lvm | 80 +++++++++++++++++++ byteback-mysql/scripts.d/Bytemyback/mysqldump-full | 8 ++ .../scripts.d/Bytemyback/mysqldump-split-db | 44 +++++++++++ 6 files changed, 281 insertions(+) create mode 100755 byteback-mysql/post-backup.d/00-mysql create mode 100755 byteback-mysql/pre-backup.d/00-mysql create mode 100755 byteback-mysql/scripts.d/Bytemyback/ini.pm create mode 100755 byteback-mysql/scripts.d/Bytemyback/lvm create mode 100755 byteback-mysql/scripts.d/Bytemyback/mysqldump-full create mode 100755 byteback-mysql/scripts.d/Bytemyback/mysqldump-split-db diff --git a/byteback-mysql/post-backup.d/00-mysql b/byteback-mysql/post-backup.d/00-mysql new file mode 100755 index 0000000..72369bf --- /dev/null +++ b/byteback-mysql/post-backup.d/00-mysql @@ -0,0 +1,27 @@ +#!/usr/bin/perl + +use warnings; +use strict; + +use lib '/etc/byteback/scripts.d'; +use Bytemyback::ini qw(readconf writeconf generateconf); + +use Sys::Hostname; + +# Shim script to determine which MySQL backup(s) to run if any + +my %config = Bytemyback::ini::readconf(); + +my $backup_method = $config{'backup_method'}; + +foreach my $method (keys %$backup_method) { + next unless $backup_method->{$method}; + print "Running /etc/byteback/scripts.d/Bytemyback/${method} post\n"; + `/etc/byteback/scripts.d/Bytemyback/${method} post` if (-x "/etc/byteback/scripts.d/Bytemyback/${method}"); + print "Finished /etc/byteback/scripts.d/Bytemyback/${method} post\n"; + + # Each script should have a heartbeat alert. They may also have specific failure alerts on both client and + # server side. + my $host = hostname; + system("mauvesend -i $method-hb-${host}-low -c now -r +26h -s 'MySQL backup method $method on ${host} has not run for over a day' --detail='Please try running manually to see what the problem was, and check the mailing list'"); +} diff --git a/byteback-mysql/pre-backup.d/00-mysql b/byteback-mysql/pre-backup.d/00-mysql new file mode 100755 index 0000000..edfcd56 --- /dev/null +++ b/byteback-mysql/pre-backup.d/00-mysql @@ -0,0 +1,31 @@ +#!/usr/bin/perl + +use warnings; +use strict; + +use Data::Dumper; + +use lib '/etc/byteback/scripts.d'; +use Bytemyback::ini qw(readconf writeconf generateconf); + +# Shim script to determine which MySQL backup(s) to run if any + +my %config = Bytemyback::ini::readconf(); +if (!%config) { + %config = Bytemyback::ini::generateconf(); +} + +# Backup directory shouldn't be world-readable +system("mkdir -p /var/backups/byteback"); +system("chmod 700 /var/backups/byteback"); + +# List of each script to run (not mutually exclusive). Generated automatically if it doesn't exist + +my $backup_method = $config{'backup_method'}; + +foreach my $method (keys %$backup_method) { + next unless $backup_method->{$method}; + print "Running /etc/byteback/scripts.d/Bytemyback/${method} pre\n"; + `/etc/byteback/scripts.d/Bytemyback/${method} pre` if (-x "/etc/byteback/scripts.d/Bytemyback/${method}"); + print "Finished /etc/byteback/scripts.d/Bytemyback/${method} pre\n"; +} diff --git a/byteback-mysql/scripts.d/Bytemyback/ini.pm b/byteback-mysql/scripts.d/Bytemyback/ini.pm new file mode 100755 index 0000000..74b1fc7 --- /dev/null +++ b/byteback-mysql/scripts.d/Bytemyback/ini.pm @@ -0,0 +1,91 @@ +#!/usr/bin/perl + +package Bytemyback::ini; + +use warnings; +use strict; + +use Data::Dumper; + +use Exporter; +our @EXPORT_OK = qw(readconf writeconf generateconf); + +# Didn't want to have to install any libraries on all servers. +# Couldn't find anything default and lightweight to read ini files +# So wrote this. Sorry. + +sub readconf { + my $file = shift || '/etc/byteback/mysql.ini'; + my %config; + my $subsection = "default"; + return if !$file; + open my $fh, "<", $file or return; + foreach (<$fh>) { + if (/^\s*\[([^\]]+)\]/) { + $subsection = $1; + } + elsif (/^\s*(\S[^\s=]*)\s*=\s*(\S[^\s=]*)\s*$/) { + $config{$subsection}->{$1}=$2; + } + } + close $fh; + return %config; +} + +sub writeconf { + # Hate writing code like this. @_ is the list of arguments passed to + # this subroutine. It's either a file and a hash, or just a hash. A + # hash is passed as a list of pairs of arguments (and Perl puts them + # back into a hash). When used in scalar context, @_ returns the number + # of items in the array. @_ % 2 is the remainder when this is divided by + # 2 (modulo). So if it's either 1 if there are an odd number of elements + # or 0. Perl treats 1 as true, 0 as false. So this one line with 10 lines + # of explanation say if there are an odd number of elements, the first is + # the ini file and should be shifted (removed from the front), otherwise + # use a default. + my $config_file = @_ % 2 ? shift : '/etc/byteback/mysql.ini'; + my %config = @_; + open my $fh, ">", $config_file; + foreach my $subsection (keys %config) { + print $fh "[$subsection]\n"; + my $href = $config{$subsection}; + foreach my $key (keys %$href) { + print $fh $key, " = ", $href->{$key}, "\n"; + } + } + close $fh; +} + +sub generateconf { + # Wipes current config, tries to figure out a set of defaults. + # set LVM to 1 only if /etc/lvmbackup.conf exists + # If it does, also populate LV size and whether or not to lock from there + my %backup_method = ( "lvm", 0, "mysqldump-split-db", 0, "mysqldump-full", 1 ); + # LVM specific variables + my %lvm; + # By default, lock tables. May be overridden below. + $lvm{"lock"} = 1; + my $lvmconfig = '/etc/mylvmbackup.conf'; + if (-e $lvmconfig) { + my $failed = 0; + open my $conffh, "<", "/etc/mylvmbackup.conf" or $failed++; + if ($conffh) { + foreach (<$conffh>) { + if (/^\s*lvsize=(\S+)\s*$/) { + $lvm{"lvsize"} = $1; + } + if (/^\s*skip_flush_tables=1/) { + $lvm{"lock"} = 0; + } + } + close $conffh; + } + $backup_method{"lvm"} = 1; + } + else { + $backup_method{"mysqldump-full"} = 1; + } + my %config = ( "backup_method" => \%backup_method, "lvm" => \%lvm ); + writeconf(%config); + return(%config); +} diff --git a/byteback-mysql/scripts.d/Bytemyback/lvm b/byteback-mysql/scripts.d/Bytemyback/lvm new file mode 100755 index 0000000..c3ff4fb --- /dev/null +++ b/byteback-mysql/scripts.d/Bytemyback/lvm @@ -0,0 +1,80 @@ +#!/usr/bin/perl + +use DBI; +use Data::Dumper; + +use warnings; +use strict; + +my $mountpoint = "/var/backups/bytemyback/mnt/"; +my $snapshot_name = "byteback_data_snap"; +my $snapshot_size = "20G"; +my $dbh; + +die "Must specify either pre or post\n" unless @ARGV; + +if ($ARGV[0] eq 'pre') { + # Check snapshot doesn't already exist + die "Snapshot still exists\n" if `lvs 2>&1` =~ /$snapshot_name/; + my $lock_tables = 1; # This will be configurable. Whether or not it runs FLUSH TABLES WITH READ LOCK before taking the snapshot. + my $defaults_file = '/etc/mysql/debian.cnf'; + my $dsn = "DBI:mysql:;mysql_read_default_file=$defaults_file"; + $dbh = DBI->connect( + $dsn, + undef, + undef, + {RaiseError => 1} + ) or die "DBI::errstr: $DBI::errstr"; + + my $data_dir = ask_mysql_for_var('datadir'); + my $lvm_dir = get_LVM_dir(); + die "MySQL doesn't seem to be running from LVM\n" if !$lvm_dir; + $dbh->do("FLUSH TABLES WITH READ LOCK") if $lock_tables; + # Create snapshot + my $mounted_lvm = `df $lvm_dir | tail -n1 | cut -f1 -d" "`; + if (!`lvcreate -n $snapshot_name -L $snapshot_size --snapshot $mounted_lvm`) { + my $error = $!; + $dbh->do("UNLOCK TABLES") if $lock_tables; + die("Unable to create snapshot: $!\n"); + } + else { + $dbh->do("UNLOCK TABLES") if $lock_tables; + chomp (my $mapped_snapshot = "/dev/mapper/" . `dmsetup ls | grep $snapshot_name | grep -v cow | cut -f1 -d " "`); + `mkdir -p $mountpoint`; + `mount $mapped_snapshot $mountpoint`; + `touch ${mountpoint}.bytebacklvm`; + } +} +elsif ($ARGV[0] eq 'post') { + `umount $mountpoint`; + chomp (my $mapped_snapshot = "/dev/mapper/" . `dmsetup ls | grep $snapshot_name | grep -v cow | cut -f1 -d " "`); + `lvremove -f $mapped_snapshot`; +} + +sub ask_mysql_for_var { + my $var = '@@' . shift; + my $query = $dbh->prepare("SELECT ${var}"); + $query->execute; + return $query->fetchrow_hashref()->{$var}; +} + +sub get_LVM_dir { + # This is a bit hacky, it checks if lvs returns, + # then checks if the mountpoint is /dev/mapper/$firstbit-$secondbit + # Returns true only if above matches and lvs output has $secondbit $firstbit somewhere + my $datadir = ask_mysql_for_var('datadir'); + my $lvs = `lvs 2>&1`; + if (($?) || ($lvs =~ /No volume groups found/)) { + return 0; + } + my $output = `df $datadir | tail -n1`; + $output =~ s/--/:/g; + if ($output =~ m#/dev/mapper/([\w:]+)-([\w:]+)\s#) { + my ($vg, $lv) = ($1, $2); + $vg =~ s/:/-/; $lv =~ s/:/-/; + if ($lvs =~ /\s+$lv\s+$vg\s/) { + return $datadir; + } + } + return 0; +} diff --git a/byteback-mysql/scripts.d/Bytemyback/mysqldump-full b/byteback-mysql/scripts.d/Bytemyback/mysqldump-full new file mode 100755 index 0000000..b4bb01e --- /dev/null +++ b/byteback-mysql/scripts.d/Bytemyback/mysqldump-full @@ -0,0 +1,8 @@ +#!/bin/bash + +if [ "$1" == "pre" ]; then + HOST=`hostname` + echo "Dumping all databases into one backup file" + mkdir -p /var/backups/byteback/mysqldump-full + mysqldump --defaults-file=/etc/mysql/debian.cnf -A --events | gzip > /var/backups/byteback/mysqldump-full/dump.sql.gz && mauvesend -i mysqldump-split-${HOST}-low -c now -r +30h -s "mysqldump on ${HOST} has not run for over a day" --detail='Please try running manually to see what the problem was, and check the mailing list' +fi diff --git a/byteback-mysql/scripts.d/Bytemyback/mysqldump-split-db b/byteback-mysql/scripts.d/Bytemyback/mysqldump-split-db new file mode 100755 index 0000000..b484ef6 --- /dev/null +++ b/byteback-mysql/scripts.d/Bytemyback/mysqldump-split-db @@ -0,0 +1,44 @@ +#!/usr/bin/perl + +use warnings; +use strict; +use Sys::Hostname; + +if ((!@ARGV) or ($ARGV[0] ne 'pre')) { + exit; +} + +# Quick mysqldump wrapper to dump individual databases and alert if it fails at all +my $failed = 0; # only set to 0 once +my $failed_databases = ''; +my $backup_directory = '/var/backups/byteback/mysqldump-split-db/'; +`mkdir -p $backup_directory`; + +my @databases = `echo "SHOW DATABASES" | mysql --defaults-file=/etc/mysql/debian.cnf`; +shift @databases; # Get rid of 'Databases' title from the top +foreach my $database (@databases) { + chomp($database); + next if $database eq "lost+found"; + next if $database =~ /^#mysql..#lost\+found$/; + next if $database =~ /^information_schema$/; + next if $database =~ /^performance_schema$/; + next if $database =~ /^events$/; + next if $database =~ /^cond_instances$/; + print "Dumping $database\n"; + # Need to get rid of this -f once we've fixed errors with views (access denied etc.) + my $gzip = -x '/usr/bin/pigz' ? '/usr/bin/pigz' : 'gzip'; + my $error_code = system("mysqldump --defaults-file=/etc/mysql/debian.cnf --events -f $database | $gzip > ${backup_directory}/${database}.sql.gz\n"); + if ($error_code) { + $failed++; + $failed_databases .= " $database"; + } +} + +my $host = hostname; +if ($failed) { + $host = hostname; + system("mauvesend -i mysqldump-${host}-low -r now -s 'mysqldump on ${host} failed for $failed_databases' --detail='Please try running them manually to see what the problem was, and check the mailing list'"); +} +else { + system("mauvesend -i mysqldump-${host}-low -c now -s 'mysqldump on ${host} failed for $failed_databases' --detail='Please try running them manually to see what the problem was, and check the mailing list'"); +} -- cgit v1.2.1 From 7aa88abdacb7049f60fd681eb211fc4da37023df Mon Sep 17 00:00:00 2001 From: Steve Kemp Date: Wed, 21 Sep 2016 14:26:21 +0300 Subject: Ignore generated files. --- .gitignore | 14 ++++++++------ 1 file changed, 8 insertions(+), 6 deletions(-) diff --git a/.gitignore b/.gitignore index c39506c..3127636 100644 --- a/.gitignore +++ b/.gitignore @@ -1,10 +1,12 @@ -debian/byteback.substvars -debian/byteback -debian/patches -debian/files -debian/byteback.debhelper.log +*~ \.pc byteback.sublime-project byteback.sublime-workspace -*~ +debian/*.debhelper.log +debian/*.substvars +debian/byteback +debian/byteback-mysql +debian/debhelper-build-stamp +debian/files +debian/patches man/*.man -- cgit v1.2.1 From 10d3ddac7a625be28e9896f52bade5c99dc34462 Mon Sep 17 00:00:00 2001 From: Steve Kemp Date: Wed, 21 Sep 2016 14:26:33 +0300 Subject: Install the files in the right location. --- debian/byteback-mysql.install | 1 + 1 file changed, 1 insertion(+) create mode 100644 debian/byteback-mysql.install diff --git a/debian/byteback-mysql.install b/debian/byteback-mysql.install new file mode 100644 index 0000000..9cb72b5 --- /dev/null +++ b/debian/byteback-mysql.install @@ -0,0 +1 @@ +byteback-mysql/* etc/byteback/ -- cgit v1.2.1