Box Backup Build and Installation Guide
License
Copyright © 2003 - 2007, Ben Summers and contributors.
All rights reserved.
Redistribution and use in source and binary forms, with or without
modification, are permitted provided that the following conditions are
met:
Redistributions of source code must retain the above copyright
notice, this list of conditions and the following disclaimer.
Redistributions in binary form must reproduce the above
copyright notice, this list of conditions and the following disclaimer
in the documentation and/or other materials provided with the
distribution.
All use of this software and associated advertising materials
must display the following acknowledgement: This product includes
software developed by Ben Summers.
The names of the Authors may not be used to endorse or promote
products derived from this software without specific prior written
permission.
[Where legally impermissible the Authors do not disclaim liability
for direct physical injury or death caused solely by defects in the
software unless it is modified by a third party.]
THIS SOFTWARE IS PROVIDED BY THE AUTHORS "AS IS" AND ANY EXPRESS
OR IMPLIED WARRANTIES, INCLUDING, BUT NOT LIMITED TO, THE IMPLIED
WARRANTIES OF MERCHANTABILITY AND FITNESS FOR A PARTICULAR
PURPOSE ARE DISCLAIMED. IN NO EVENT SHALL THE AUTHORS BE LIABLE FOR
ANY DIRECT, INDIRECT, INCIDENTAL, SPECIAL, EXEMPLARY, OR CONSEQUENTIAL
DAMAGES (INCLUDING, BUT NOT LIMITED TO, PROCUREMENT OF SUBSTITUTE GOODS
OR SERVICES; LOSS OF USE, DATA, OR PROFITS; OR BUSINESS INTERRUPTION)
HOWEVER CAUSED AND ON ANY THEORY OF LIABILITY, WHETHER IN CONTRACT,
STRICT LIABILITY, OR TORT (INCLUDING NEGLIGENCE OR OTHERWISE) ARISING
IN ANY WAY OUT OF THE USE OF THIS SOFTWARE, EVEN IF ADVISED OF THE
POSSIBILITY OF SUCH DAMAGE.
Introduction
The backup daemon, bbackupd, runs on all machines to be backed up.
The store server daemon, bbstored runs on a central server. Data is sent
to the store server, which stores all data on local filesystems, that is,
only on local hard drives. Tape or other archive media is not used.
The system is designed to be easy to set up and run, and cheap to
use. Once set up, there should be no need for user or administrative
intervention, apart from usual system maintenance.
Client daemon
bbackupd is configured with a list of directories to back up. It
has a lazy approach to backing up data. Every so often, the directories
are scanned, and new data is uploaded to the server. This new data must
be over a set age before it is uploaded. This prevents rapid revisions
of a file resulting in many uploads of the same file in a short period
of time.
It can also operate in a snapshot mode, which behaves like
traditional backup software. When instructed by an external bbackupctl
program, it will upload all changed files to the server.
The daemon is always running, although sleeping most of the time.
In lazy mode, it is completely self contained -- scripts running under
cron jobs are not used. The objective is to keep files backed up, not to
make snapshots of the filesystem at particular points in time
available.
If an old version of the file is present on the server, a modified
version of the rsync algorithm is used to upload only the changed
portions of the file.
After a new version is uploaded, the old version is still
available (subject to disc space on the server). Similarly, a deleted
file is still available. The only limit to their availability is space
allocated to this account on the server
Future versions will add the ability to mark the current state of
files on the server, and restore from this mark. This will emulate the
changing of tapes in a tape backup system.
Restoration
Restoring files is performed using a query tool, bbackupquery.
This can be used to restore entire directories, or as an 'FTP-like'
tool to list and retrieve individual files. Old versions and deleted
files can be retrieved using this tool for as long as they are kept on
the server.
Client Resource Usage
bbackupd uses only a minimal amount of disc space to store
records on uploaded files -- less than 32 bytes per directory and file
over a set size threshold. However, it minimises the amount of queries
it must make to the server by storing, in memory, a data structure
which allows it to determine what data is new. It does not need to
store a record of all files, essentially just the directory names and
last modification times. This is not a huge amount of memory.
If there are no changes to the directories, then the client will
not even connect to the server.
Security
Box Backup is designed to be secure in several ways. The data
stored on the backup store server is encrypted using secret-key
cryptography. Additionally, the transport layer is encrypted using TLS,
to ensure that the communications can't be snooped.
Encryption
The files, directories, filenames and file attributes are all
encrypted. By examining the stored files on the server, it is only
possible to determine the approximate sizes of a files and the tree
structure of the disc (not names, just number of files and
subdirectories in a directory). By monitoring the actions performed by
a client, it is possible to determine the frequency and approximate
scope of changes to files and directories.
The connections between the server and client are encrypted
using TLS (latest version of SSL). Traffic analysis is possible to
some degree, but limited in usefulness.
An attacker will not be able to recover the backed up data
without the encryption keys. Of course, you won't be able to recover
your files without the keys either, so you must make a conventional,
secure, backup of these keys.
Authentication
SSL certificates are used to authenticate clients. UNIX user
accounts are not used to minimise the dependence on the configuration
of the operating system hosting the server.
A script is provided to run the necessary certification
authority with minimal effort.
Server daemon
The server daemon is designed to be simple to deploy, and run on
the cheapest hardware possible. To avoid the necessity to use expensive
hardware RAID or software RAID with complex setup, it (optionally)
stores files using RAID techniques.
It does not need to run as a privileged user.
Each account has a set amount of disc space allocated, with a soft
and a hard limit. If the account exceeds the soft limit, a housekeeping
process will start deleting old versions and deleted files to reduce the
space used to below the soft limit. If the backup client attempts to
upload a file which causes the store to exceed the hard limit, the
upload will be refused.
Building and installing
Before you start
Firstly, check that all the clocks on your clients, servers and
signing machines are accurate and in sync. A disagreement in time
between a client and a server is the biggest cause of installation
difficulties, as the times in the generated certificates will cause
login failures if the start date is in the future.
Box Backup compile
In the following instructions, replace 0.00 with the actual
version number of the archive you have downloaded.
For help building on Windows, see the Windows
Compile Appendix. And if you want to build a Linux RPM, look here.
You need the latest version of OpenSSL, as some of the extra APIs
it provides are required. You should have this anyway, as earlier
versions have security flaws. (If you have an earlier version installed,
the configuration script will give you instructions on enabling
experimental support for older versions.)
See OpenSSL notes for more information
on OpenSSL issues.
There are some notes in the archive about compiling on various
platforms within the boxbackup-0.00 directory -- read them first. For
example, if you are compiling under Linux, look for LINUX.txt as
boxbackup-0.00/LINUX.txt after untaring the archive.
Download the archive, then in that directory type
tar xvzf boxbackup-0.00.tgz
cd boxbackup-0.00
./configure
make
The server and client will be built and packaged up for
installation on this machine, or ready to be transferred as tar files to
another machine for installation.
This builds two parcels of binaries and scripts, 'backup-client'
and 'backup-server'. The generated installation scripts assumes you want
everything installed in /usr/local/bin
Optionally, type make test to run
all the tests.
Local installation
Type make install-backup-client
to install the backup client.
Type make install-backup-server
to install the backup server.
Remote installation
In the parcels directory, there are tar files for each parcel. The
name reflects the version and platform you have built it for.
Transfer this tar file to the remote server, and unpack it, then
run the install script. For example:
tar xvzf boxbackup-0.00-backup-server-OpenBSD.tgz
cd boxbackup-0.00-backup-server-OpenBSD
./install-backup-server
Configure options
You can use arguments to the configure script to adjust the
compile and link lines in the generated Makefiles, should this be
necessary for your platform. The configure script takes the usual GNU
autoconf arguments, a full list of which can be obtained with --help. Additional options for Box Backup
include:
--enable-gnu-readline
Use GNU readline if present. Linking Box Backup against
GNU readline may create licence implications if you then
distribute the binaries. libeditline is also supported as a safe
alternative, and is used by default if available.
--disable-largefile
Omit support for large files
--with-bdb-dir=DIR
Specify Berkeley DB location
--with-random=FILE
Use FILE as random number seed (normally
auto-detected)
--with-tmp-dir=DIR
Directory for temporary files (normally /tmp)
See OpenSSL notes for the OpenSSL
specific options.
Tests
There are a number of unit tests provided. To compile and run one
type:
./runtest.pl bbackupd release
./runtest.pl common debug
./runtest.pl ALL
The runtest.pl script will compile and run the test. The first
argument is the test name, and the second the type of build. Use ALL as
a test name to run all the tests.
The output from the tests is slightly muddled using this script.
If you're developing, porting or trying out new things, it might be
better to use the following scheme:
cd test/bbackupd
make
cd ../../debug/test/bbackupd
./t
or in release mode...
cd test/bbackupd
make -D RELEASE
cd ../../release/test/bbackupd
./t
(use RELEASE=1 with GNU make)
I tend to use two windows, one for compilation, and one for
running tests.
Box Backup and SSL
General notes
Ideally, you need to use version 0.9.7 or later of OpenSSL. If
this is installed on your system by default (and it is on most recent
releases of UNIX like OSes) then everything should just work.
However, if it isn't, you have a few options.
Upgrade your installation
The best option is to upgrade your installation to use 0.9.7.
Hopefully your package manager will make this easy for you. This may
require reinstallation of lots of software which depends on OpenSSL,
so may not be ideal.
(But as there have been a few security flaws in OpenSSL
recently, you probably want to upgrade it anyway.)
Install another OpenSSL
The second best option is to install another copy. If you
download and install from source, it will probably install into
/usr/local/ssl. You can then configure Box Backup to use it
using:
./configure --with-ssl-headers=/usr/local/ssl/include --with-ssl-lib=/usr/local/ssl/lib
which will set up the various includes and libraries for
you.
The configuration scripts may be a problem, depending on your
installation. See below for more information.
Use the old version of OpenSSL
If you have an old version installed, the configuration script
will give you instructions on how to enable support for older
versions. Read the warnings, and please, whatever you do, don't
release binary packages or ports which enable this option.
You may have issues with the configuration scripts, see
below.
If you have problems with the config scripts
If you get OpenSSL related errors with the configuration scripts,
there are two things to check:
The bin directory within your OpenSSL directory is in the path
(if you have installed another version)
You have an openssl.cnf file which works and can be
found.
OpenSSL config file
You need to have an openssl.cnf file. The default will generally
work well (see example at end). Make sure the openssl utility can find
it, either set the OPENSSL_CONF environment variable, or install it
into the location that is mentioned in the error messages.
Example OpenSSL config file:
#
# OpenSSL example configuration file.
# This is mostly being used for generation of certificate requests.
#
RANDFILE = /dev/arandom
####################################################################
[ req ]
default_bits = 1024
default_keyfile = privkey.pem
distinguished_name = req_distinguished_name
attributes = req_attributes
[ req_distinguished_name ]
countryName = Country Name (2 letter code)
#countryName_default = AU
countryName_min = 2
countryName_max = 2
stateOrProvinceName = State or Province Name (full name)
#stateOrProvinceName_default = Some-State
localityName = Locality Name (eg, city)
0.organizationName = Organization Name (eg, company)
#0.organizationName_default = Internet Widgits Pty Ltd
# we can do this but it is not needed normally :-)
#1.organizationName = Second Organization Name (eg, company)
#1.organizationName_default = CryptSoft Pty Ltd
organizationalUnitName = Organizational Unit Name (eg, section)
#organizationalUnitName_default =
commonName = Common Name (eg, fully qualified host name)
commonName_max = 64
emailAddress = Email Address
emailAddress_max = 64
[ req_attributes ]
challengePassword = A challenge password
challengePassword_min = 4
challengePassword_max = 20
unstructuredName = An optional company name
[ x509v3_extensions ]
nsCaRevocationUrl = http://www.cryptsoft.com/ca-crl.pem
nsComment = "This is a comment"
# under ASN.1, the 0 bit would be encoded as 80
nsCertType = 0x40
Compiling bbackupd on Windows using Visual C++
This Appendix explains how to build the bbackupd daemon for Windows
using the Visual C++ compiler.
If you have any problems following these instructions, please sign
up to the mailing
list and report them to us, or send an email to Chris Wilson. Thanks!
Note: bbstored will not be built
with this process. bbstored is not currently supported on Windows. There
are no plans for bbstored support on Windows.
Tools
You will need quite a bit of software to make this work. All of it
is available for free on the Internet, although Visual C++ Express has
license restrictions and a time limit.
Visual C++
Microsoft's Visual C++ compiler and development environment are
part of their commercial product Visual Studio. Visual
Studio 2005 is supported, and 2003 should work as well.
You can also download
a free copy of Visual C++ 2005 Express. This copy must be registered
(activated) within 30 days, and is free for one year.
You will need the Platform
SDK to allow you to compile Windows applications.
Perl
Download and install ActivePerl for
Windows, which you can probably find here.
Libraries
You will need to download and install several libraries. They
must all be built in the same directory, to be able to link
properly.
Choose a directory where you will unpack and compile OpenSSL,
Zlib and Box Backup. We will call this the base directory. An example
might be:
C:\Documents and Settings\Your Username\Desktop\Box
Make sure you know the full path to this directory.
OpenSSL
You will need to compile OpenSSL using Visual C++. The latest
release at this time, OpenSSL 0.9.8a, does not compile with Visual
C++ 2005 out of the box, so you need a
patched version. The original
source and patch
are also available.
To compile OpenSSL:
Use a Windows unzipper such as WinZip (free trial) to
extract the openssl-0.9.8a-vc2005.tar.gz archive,
which you just downloaded, into the base directory.
Rename the folder from openssl-0.9.8a-vc2005 to openssl
Open a command shell (run cmd.exe) and cd to the openssl directory
Run the following commands:
perl Configure VC-WIN32
ms\do_ms
"c:\program files\Microsoft Visual Studio 8\Common7\Tools\vsvars32.bat"
nmake -f ms\ntdll.mak
Zlib
You will need to download the Zlib compiled DLL.
Create a directory called zlib in
the base directory, and unzip the file you just downloaded into that
directory. You don't need to compile anything.
Download Box Backup
The first version of Box Backup that's known to compile and with
Visual C++ 2005 is available on the Subversion
server. However, this version has not been extensively tested
and may be out of date.
The changes are expected to be merged into the Subversion trunk
at some point, and this page should then be updated. If in doubt,
please sign up to the mailing
list and ask us.
To get the source code out of Subversion you will need a Subversion
client for Windows. After installing it, open a new command
prompt, go to the base directory, and type:
svn co http://bbdev.fluffy.co.uk/svn/box/chris/win32/vc2005-compile-fixes/ boxbackup
This should create a directory called boxbackup inside the base directory.
Configure Box Backup
Open a command prompt, change to the base directory then
boxbackup, and run win32.bat to configure the sources. Otherwise,
Visual C++ will complain about missing files whose names start with
autogen, and missing config.h.
Compile Box Backup
Open Visual C++. Choose "File/Open/Project", navigate to the
base directory, then to boxbackup\infrastructure\msvc\2005 (or
2003 if using Visual Studio 2003),
and open any project or solution file in that directory.
Press F7 to compile Box Backup. If the compilation is
successful, boxbackup\Debug\bbackupd.exe will be
created.
Install Box Backup
Create the destination directory, C:\Program Files\Box Backup\bbackupd.
Write a configuration file, keys and certificate on a Unix
machine, and copy them into the Box
Backup directory, together with the following files from
the base directory:
boxbackup\Debug\bbackupd.exe
openssl\out32dll\libeay32.dll
openssl\out32dll\ssleay32.dll
zlib\zlib1.dll
Ensure that the user running Box Backup can read from the
Box Backup directory, and write to
the bbackupd directory inside
it.
Run Box Backup by double-clicking on it, and check that it
connects to the server. If the window opens and closes immediately,
it's probably due to a problem with the configuration file - check the
Windows Event Viewer for details.
Windows Service
Box Backup can also run as a Windows service, in which case it
will be automatically started at boot time in the background. To
install this, open a command prompt, and run:
cd "C:\Program Files\Box Backup"
bbackupd.exe -i
This should output Box Backup service installed.
Compilation and installation by building an RPM on
Linux
It is very easy to build an RPM of Box Backup on Linux platforms.
This should work on all Red Hat distributions (including Fedora), SuSE,
and probably others too.
Given that you have the correct development packages installed
simply execute this command (replacing the version number):
rpmbuild -ta boxbackup-0.00.tgz
rpmbuild will report where the packages have been written to, and
these can be installed in the normal manner.
If you have never built an RPM before you should set up a convenient
build area as described in the RPM
book.
If you wish to customise the package you can find the spec file in
the contrib/rpm directory.