summaryrefslogtreecommitdiff
path: root/documentation
diff options
context:
space:
mode:
authorPer Reedtz Thomsen <pthomsen@reedtz.com>2006-03-14 06:08:20 +0000
committerPer Reedtz Thomsen <pthomsen@reedtz.com>2006-03-14 06:08:20 +0000
commit3de317a4d8dfa49a783a3dd1c34145c754599823 (patch)
treea66684466a46350318c604ed77a24ddcdced4009 /documentation
parentcb1b39e29e896d9fd4030174e7c73595e41c9f01 (diff)
Renamed instguide.sgml -> instguide.xml, to make XXE editing a little easier.
First cut of the admin guide. There are several TODO items there now.
Diffstat (limited to 'documentation')
-rw-r--r--documentation/boxbackup/adminguide.xml1230
-rw-r--r--documentation/boxbackup/instguide.xml (renamed from documentation/boxbackup/instguide.sgml)0
2 files changed, 1230 insertions, 0 deletions
diff --git a/documentation/boxbackup/adminguide.xml b/documentation/boxbackup/adminguide.xml
new file mode 100644
index 00000000..9620a823
--- /dev/null
+++ b/documentation/boxbackup/adminguide.xml
@@ -0,0 +1,1230 @@
+<?xml version="1.0" encoding="UTF-8"?>
+<!DOCTYPE book PUBLIC "-//OASIS//DTD DocBook XML V4.4//EN"
+"http://www.oasis-open.org/docbook/xml/4.4/docbookx.dtd">
+<book>
+ <title>Box Backup administrator's guide</title>
+
+ <preface>
+ <title>License</title>
+
+ <para>Copyright (c) &lt;YEAR&gt;, &lt;OWNER&gt;</para>
+
+ <para>All rights reserved.</para>
+
+ <para>Redistribution and use in source and binary forms, with or without
+ modification, are permitted provided that the following conditions are
+ met:</para>
+
+ <itemizedlist>
+ <listitem>
+ <para>Redistributions of source code must retain the above copyright
+ notice, this list of conditions and the following disclaimer.</para>
+ </listitem>
+
+ <listitem>
+ <para>Redistributions in binary form must reproduce the above
+ copyright notice, this list of conditions and the following disclaimer
+ in the documentation and/or other materials provided with the
+ distribution.</para>
+ </listitem>
+
+ <listitem>
+ <para>Neither the name of the &lt;ORGANIZATION&gt; nor the names of
+ its contributors may be used to endorse or promote products derived
+ from this software without specific prior written permission</para>
+ </listitem>
+ </itemizedlist>
+
+ <para>THIS SOFTWARE IS PROVIDED BY THE COPYRIGHT HOLDERS AND CONTRIBUTORS
+ "AS IS" AND ANY EXPRESS OR IMPLIED WARRANTIES, INCLUDING, BUT NOT LIMITED
+ TO, THE IMPLIED WARRANTIES OF MERCHANTABILITY AND FITNESS FOR A PARTICULAR
+ PURPOSE ARE DISCLAIMED. IN NO EVENT SHALL THE COPYRIGHT OWNER OR
+ CONTRIBUTORS BE LIABLE FOR ANY DIRECT, INDIRECT, INCIDENTAL, SPECIAL,
+ EXEMPLARY, OR CONSEQUENTIAL DAMAGES (INCLUDING, BUT NOT LIMITED TO,
+ PROCUREMENT OF SUBSTITUTE GOODS OR SERVICES; LOSS OF USE, DATA, OR
+ PROFITS; OR BUSINESS INTERRUPTION) HOWEVER CAUSED AND ON ANY THEORY OF
+ LIABILITY, WHETHER IN CONTRACT, STRICT LIABILITY, OR TORT (INCLUDING
+ NEGLIGENCE OR OTHERWISE) ARISING IN ANY WAY OUT OF THE USE OF THIS
+ SOFTWARE, EVEN IF ADVISED OF THE POSSIBILITY OF SUCH DAMAGE.</para>
+ </preface>
+
+ <chapter>
+ <title>Configuration</title>
+
+ <section>
+ <title>System configuration</title>
+
+ <section>
+ <title>Server</title>
+
+ <para>After you've downloaded and compiled the programs you need to
+ install the programs on your server. As root do the following:</para>
+
+ <programlisting>make install-backup-server</programlisting>
+
+ <para>This assumes that you are installing on the same server that you
+ compiled the software on. If not, copy the
+ boxbackup-x.xx-backup-server-OSNAME.tgz file to the server you want to
+ run on, and install there. For example (on Mac OS X):</para>
+
+ <programlisting>tar zxvf boxbackup-0.10-server-darwin8.5.0.tgz
+cd boxbackup-0.10-server-darwin8.5.0
+./install-backup-server</programlisting>
+
+ <para>Then create the user for the backup daemon on the server:</para>
+
+ <programlisting>useradd _bbstored</programlisting>
+
+ <para>BoxBackup supports Raid for the backup store. There are some
+ configuration options. Use the following command if you want to create
+ a simple server WITHOUT Raid protection:</para>
+
+ <programlisting>mkdir /tmp/boxbackupRepository # Create the directory
+chown _bbstored /tmp/boxbackupRepository/ # Change the owner to the new boxbackup daemon user
+
+/usr/local/bin/raidfile-config /etc/box/ 1024 /tmp/boxbackupRepository
+
+#substitute 1024 with the desired blocksize
+#substitute /tmp/boxbackupRepository with a directory that exists where you want the backup store located
+#/usr/local/bin/raidfile-config --help shows you the options</programlisting>
+
+ <para>Then create the configuration file /etc/box/bbstored.conf The
+ hostname is tricky as it is used for two things: The name of the
+ server in the certificate and the address the server is listening on.
+ Since you might be using NAT, might move the server around or the
+ domain name might change, choose a name that describes the server.
+ When the network address of the server changes, you need to update the
+ ListenAddresses directive in the /etc/box/bbstored.conf file.</para>
+
+ <programlisting>/usr/local/bin/bbstored-config /etc/box hostname _bbstored</programlisting>
+
+ <para>This last step outputs 5 instructions that you must execute to
+ the letter. A lot of questions are raised on the mailing list because
+ these steps have not been followed properly.</para>
+
+ <para>TODO: Expand on this. Explain the 5 steps in detail.</para>
+
+ <para>If you want to run the system as a non-root user, look <link
+ linkend="WORoot">here</link>.</para>
+ </section>
+
+ <section>
+ <title>Certificate Management</title>
+
+ <para>There are two steps involved to create an account. You need to
+ create the account on the server, and sign a certificate to give the
+ client permission to connect to the server.</para>
+
+ <para>Running a Certification Authority for TLS (SSL) connections is
+ not trivial. However, a script to does most of the work in a way which
+ should be good enough for most deployments.</para>
+
+ <para><emphasis role="bold">Important Note:</emphasis> The certificate
+ authority directory is intended to be stored on another server. It
+ should not be kept on the backup server to limit the impact of a
+ server compromise. The instructions and the script assume that it will
+ be kept elsewhere, so will ask you to copy files to and from the CA.
+ </para>
+
+ <para><emphasis role="bold">Clock time warning</emphasis>: SSL
+ certificates contain validity dates, including a "valid from" time. If
+ the clock on the machine which signs the certificates is not
+ syncronised to the clocks of the machines using these certificates,
+ you will probably get strange errors until the start time is reached
+ on all machines. If you get strange errors when attempting to use new
+ certificates, check the clocks! You will probably just need to wait a
+ while until the certificates become valid, rather than having to
+ regenerate them. </para>
+
+ <section>
+ <title>Set up a CA</title>
+
+ <para>It's best to do this on a machine other than your server,
+ probably without direct network access. The contents of this
+ directory control who can access your backup store server.</para>
+
+ <para>To setup the basic key structure, do the following:</para>
+
+ <programlisting>/usr/local/bin/bbstored-certs ca init</programlisting>
+
+ <para>(See <link linkend="instguide.sgml">OpenSSL notes</link> if
+ you get an OpenSSL error)</para>
+
+ <para>This creates the directory 'ca' in the current directory, and
+ initialises it with basic keys. </para>
+ </section>
+
+ <section>
+ <title>Sign a server certificate</title>
+
+ <para>When you use the bbstored-config script to set up a config
+ file for a server, it will generate a certificate request (CSR) for
+ you. Transfer it to the machine with your CA, then do:</para>
+
+ <programlisting>/usr/local/bin/bbstored-certs ca sign-server hostname-csr.pem</programlisting>
+
+ <para>This signs the certificate for the server. Follow the
+ instructions in the output on which files to install on the server.
+ The CSR file is now no longer needed. Make sure you run this command
+ from the directory above the directory 'ca'. </para>
+
+ <para>TODO: Explain instructions in output.</para>
+ </section>
+
+ <section>
+ <title>Set up an account</title>
+
+ <para>Choose an account number for the user. This must be unique on
+ the server, and is presented as a 31 bit number in hex greater than
+ 0, for example, 1 or 75AB23C. Then on the backup store server,
+ create the account with:</para>
+
+ <programlisting>/usr/local/bin/bbstoreaccounts create 75AB23C 0 4096M 4505M</programlisting>
+
+ <para>This looks complicated. The numbers are, in order:</para>
+
+ <itemizedlist>
+ <listitem>
+ <para>The account number allocated (hex)</para>
+ </listitem>
+
+ <listitem>
+ <para>The RAID disc set (0 if you use raidfile-config and don't
+ add a new set)</para>
+ </listitem>
+
+ <listitem>
+ <para>Soft limit (size)</para>
+ </listitem>
+
+ <listitem>
+ <para>Hard limit (size)</para>
+ </listitem>
+ </itemizedlist>
+
+ <para>The sizes are are specified in Mb, Gb, or blocks, depending on
+ the suffix. 1M specifies 1 Mb, 1G specifies 1 Gb, and 1B specifies 1
+ block, the size of which depends on how you have configured the
+ raidfile system with raidfile-config.</para>
+
+ <para>In this example, I have allocated 4Gb (assuming you use 2048
+ byte blocks as per my example) as the soft limit, and 4Gb + 10% as
+ the hard limit.</para>
+
+ <para>NOTE The sizes specified here are pre-RAID. So if you are
+ using userland RAID, you are actually allocating two-thirds of this
+ amount. This means that, when you take compression into account,
+ that if you allocate 2Gb on the server, it'll probably hold about
+ 2Gb of backed up files (depending on the compressability of those
+ files).</para>
+
+ <para>The backup client will (voluntarily) try not to upload more
+ data than is allowed by the soft limit. The store server will refuse
+ to accept a file if it would take it over the hard limit, and when
+ doing housekeeping for this account, try and delete old versions and
+ deleted files to reduce the space taken to below the soft
+ limit.</para>
+
+ <para>This command will create some files on disc in the raid file
+ directories (if you run as root, the utility will change to the user
+ specified in the bbstored.conf file to write them) and update the
+ accounts file. A server restart is not required.</para>
+
+ <para>NOTE If you get a message saying 'Exception: RaidFile (2/8)',
+ the directories you specified in the raidfile.conf are not writable
+ by the _bbstored user -- fix it, and try again.</para>
+
+ <para>Finally, tell the user their account number, and the hostname
+ of your server. They will use this to set up the backup client, and
+ send you a CSR. This has the account number embedded in it, and you
+ should be sure that it has the right account number in it.</para>
+
+ <para>Sign this CSR with this command:</para>
+
+ <programlisting>/usr/local/bin/bbstored-certs ca sign 75AB23C-csr.pem</programlisting>
+
+ <para>Don't forget to check that the embedded account number is
+ correct! Then send the two files back to the user, as instructed by
+ the script.</para>
+
+ <para>Please read the Troubleshooting page if you have problems.
+ </para>
+
+ <para>TODO: Link to troubleshooting...</para>
+ </section>
+ </section>
+
+ <section>
+ <title>Log Files</title>
+
+ <para>You may wish to see what's going on with the server. Edit
+ /etc/syslog.conf, and add:</para>
+
+ <programlisting>local6.info /var/log/box
+local5.info /var/log/raidfile</programlisting>
+
+ <para><emphasis role="bold">Note:</emphasis> Separators must be tabs,
+ otherwise these entries will be ignored.</para>
+
+ <programlisting>touch /var/log/box
+touch /var/log/raidfile</programlisting>
+
+ <para>Set up log rotation, by adding in /etc/newsyslog.conf:</para>
+
+ <programlisting>/var/log/box 644 7 2000 * Z
+/var/log/raidfile 644 7 2000 * Z</programlisting>
+
+ <para>Then restart syslogd.</para>
+ </section>
+
+ <section>
+ <title>Configuring a client</title>
+
+ <para>Before you can do any configuration, you need to know the
+ hostname of the server you will be using, and your account number on
+ that server.</para>
+
+ <para>Later in the process, you will need to send a certificate
+ request to the administrator of that server for it to be
+ signed.</para>
+
+ <para>Installation is covered in the compiling and installing section.
+ You only need the backup-client parcel.</para>
+
+ <para>It is important that you read all the output of the config
+ scripts. See the end of this page for an example.</para>
+
+ <para>The backup client has to be run as root, because it needs to
+ read all your files to back them up, although it is possible to back
+ up a single user's files by running it as that user. (Tip: specify a
+ directory other than /etc/box, and then give the alternate config file
+ as the first argument to bbackupd). However, it will fall over if you
+ don't give yourself read access to one of your files. </para>
+
+ <section>
+ <title id="BasicConfig">Basic configuration</title>
+
+ <para>Run the bbackupd-config script to generate the configuration
+ files and generate a private key and certificate request.</para>
+
+ <programlisting>/usr/local/bin/bbackupd-config /etc/box lazy <emphasis
+ role="bold">999 hostname</emphasis> /var/bbackupd <emphasis
+ role="bold">/home</emphasis></programlisting>
+
+ <para>(See <link linkend="instguide.sgml">OpenSSL notes</link> if
+ you get an OpenSSL error)</para>
+
+ <para>The items in bold need to be changed. In order, they are the
+ account number, the hostname of the server you're using, and
+ finally, the directories you want backed up. You can include as many
+ you want here.</para>
+
+ <para>However, the directories you specify must not contain other
+ mounted file systems within them at any depth. Specify them
+ separately, one per mount point. No checks are currently made to
+ catch bad configuration of this nature!</para>
+
+ <para>You may also want to consider changing the mode from lazy to
+ snapshot, depending on what your system is used for. </para>
+
+ <itemizedlist>
+ <listitem>
+ <para><emphasis role="bold">lazy</emphasis>. This mode regularly
+ scans the files, with only a rough schedule. It uploads files as
+ and when they are changed, if the latest version is more than a
+ set age. This is good for backing up user's documents stored on
+ a server, and spreads the load out over the day. </para>
+ </listitem>
+
+ <listitem>
+ <para><emphasis role="bold">snapshot</emphasis>. This mode
+ emulates the traditional backup behaviour of taking a snapshot
+ of the filesystem. The backup daemon does absolutely nothing
+ until it is instructed to make a backup using the bbackupctl
+ utility (probably as a cron job), at which point it uploads all
+ files which have been changed since the last time it uploaded.
+ </para>
+ </listitem>
+ </itemizedlist>
+
+ <para>When you run the config script, it will tell you what you need
+ to do next. Don't forget to read all the output. An example is shown
+ at the end of this page, but the instructions for your installation
+ may be different.</para>
+ </section>
+
+ <section>
+ <title>Certificates</title>
+
+ <para>After you have sent your certificate request off to the server
+ administrator and received your certificate and CA root back,
+ install them where instructed by the bbackupd-config script during
+ basic bbackupd configuration.</para>
+
+ <para>You can then run the daemon (as root) by typing
+ /usr/local/bin/bbackupd, and of course, adding it to your system's
+ startup scripts. The first time it's run it will upload everything.
+ Interrupting it and restarting it will only upload files which were
+ not uploaded before - it's very tolerant.</para>
+
+ <para>If you run in snapshot mode, you will need to add a cron job
+ to schedule backups. The config script will tell you the exact
+ command to use for your system.</para>
+
+ <para>Please read the Troubleshooting page if you have
+ problems.</para>
+
+ <para>Remember to make a traditional backup of the keys file, as
+ instructed. You cannot restore files without it.</para>
+
+ <para>It is recommended that you backup up all of /etc/box as it
+ will make things easier if you need to restore files. But only the
+ keys are absolutely essential.</para>
+
+ <para>If you want to see what it's doing in more detail (probably a
+ good idea), follow the instructions in the server setup to create
+ new log files with syslog. Adding and removing backed up
+ locations.</para>
+
+ <para>By editing the /etc/box/bbackupd.conf file, you can add and
+ remove directories to back up -- see comments in this file for help.
+ Send bbackupd a HUP signal after you modify it.</para>
+
+ <para>When you remove a location, it will not be marked as deleted
+ immediately. Instead, bbackupd waits about two days before doing so,
+ just in case you change your mind. After this, it will be eventually
+ removed from the store by the housekeeping process. Run as
+ root.</para>
+
+ <para>The backup client is designed to be run as root. It is
+ possible to run without root, but this is not recommended. Clock
+ synchronisation for file servers.</para>
+
+ <para>If you are using the backup client to backup a filesystem
+ served from a fileserver, you should ideally ensure that the
+ fileserver clocks are synchronised with the fileserver.</para>
+
+ <para>bbackupd will cope perfectly well if the clocks are not
+ synchronised. Errors up to about half an hour cause no problems.
+ Larger discrepancies cause a loss of efficiency and the potential to
+ back up a file during a write process.</para>
+
+ <para>There is a configuration parameter MaxFileTimeInFuture, which
+ specifies how far in the future a file must be for it to be uploaded
+ as soon as it is seen. You should not need to adjust this (default
+ is 2 days). Instead, get those clocks synchronised. Excluding files
+ and directories from the backup.</para>
+
+ <para>Within the bbackupd.conf file, there is a section named
+ BackupLocations which specifies which locations on disc should be
+ backed up. It has subsections, each of which is in the
+ format:</para>
+
+ <programlisting> name
+ {
+ Path = /path/of/directory
+ (optional exclude directives)
+ }</programlisting>
+
+ <para><emphasis role="bold">name</emphasis> is derived from the Path
+ by the config script, but should merely be unique.</para>
+
+ <para>The exclude directives are of the form:</para>
+
+ <programlisting>[Exclude|AlwaysInclude][File|Dir][|sRegex] = regex or full pathname</programlisting>
+
+ <para>(The regex suffix is shown as 'sRegex' to make File or Dir
+ plural)</para>
+
+ <para>For example:</para>
+
+ <programlisting> ExcludeDir = /home/guest-user
+ ExcludeFilesRegex = *.(mp3|MP3)\$
+ AlwaysIncludeFile = /home/username/veryimportant.mp3</programlisting>
+
+ <para>This excludes the directory /home/guest-user from the backup
+ along with all mp3 files, except one MP3 file in particular.</para>
+
+ <para>In general, Exclude excludes a file or directory, unless the
+ directory is explicitly mentioned in a AlwaysInclude
+ directive.</para>
+
+ <para>If a directive ends in Regex, then it is a regular expression
+ rather than a explicit full pathname. See </para>
+
+ <programlisting> man 7 re_format</programlisting>
+
+ <para>for the regex syntax on your platform.</para>
+ </section>
+
+ <section>
+ <title>Example configuration output</title>
+
+ <para>This is an example of output from the bbstored-config
+ script.</para>
+
+ <para><emphasis role="bold">Important</emphasis>: Follow the
+ instructions output by your script, not the ones here -- they may be
+ different for your system.</para>
+
+ <programlisting>/usr/local/bin/bbackupd-config /etc/box lazy 51 server.example.com /var/bbackupd /home /etc/samba
+
+Setup bbackupd config utility.
+
+Configuration:
+ Writing configuration file: /etc/box/bbackupd.conf
+ Account: 51
+ Server hostname: server.example.com
+ Directories to back up:
+ /home
+ /etc/samba
+
+Note: If other file systems are mounted inside these directories, then problems may occur
+with files on the store server being renamed incorrectly. This will cause efficiency
+problems, but not affect the integrity of the backups.
+
+WARNING: Directories not checked against mountpoints. Check mounted filesystems manually.
+
+Creating /etc/box...
+Creating /etc/box/bbackupd
+Generating private key...
+ [OpenSSL output omitted]
+
+Generating keys for file backup
+Writing notify script /etc/box/bbackupd/NotifyStoreFull.sh
+Writing configuration file /etc/box/bbackupd.conf
+
+===================================================================
+
+bbackupd basic configuration complete.
+
+What you need to do now...
+
+1) Make a backup of /etc/box/bbackupd/51-FileEncKeys.raw
+ This should be a secure offsite backup.
+ Without it, you cannot restore backups. Everything else can
+ be replaced. But this cannot.
+ KEEP IT IN A SAFE PLACE, OTHERWISE YOUR BACKUPS ARE USELESS.
+
+2) Send /etc/box/bbackupd/51-csr.pem
+ to the administrator of the backup server, and ask for it to
+ be signed.
+
+3) The administrator will send you two files. Install them as
+ /etc/box/bbackupd/51-cert.pem
+ /etc/box/bbackupd/serverCA.pem
+ after checking their authenticity.
+
+4) You may wish to read the configuration file
+ /etc/box/bbackupd.conf
+ and adjust as appropraite.
+
+ There are some notes in it on excluding files you do not
+ wish to be backed up.
+
+5) Review the script
+ /etc/box/bbackupd/NotifyStoreFull.sh
+ and check that it will email the right person when the store
+ becomes full. This is important -- when the store is full, no
+ more files will be backed up. You want to know about this.
+
+6) Start the backup daemon with the command
+ /usr/local/bin/bbackupd
+ in /etc/rc.local, or your local equivalent.
+ Note that bbackupd must run as root.
+
+===================================================================</programlisting>
+
+ <para>Remember to make a secure, offsite backup of your backup keys,
+ as described in <link linkend="BasicConfig">Basic
+ configuration</link> above. If you do not,, and that key is lost,
+ you have no backups.</para>
+ </section>
+ </section>
+ </section>
+ </chapter>
+
+ <chapter>
+ <title>Administration</title>
+
+ <para>This chapter deals with the dauily running and management of the Box
+ Backup system. It explains most day-to-day tasks.</para>
+
+ <section>
+ <title>Regular Maintenance</title>
+
+ <para>The steps involved in maintaining and keeping the backup sets
+ healthy are outlined in this section.</para>
+
+ <section>
+ <title>Controlling a backup client</title>
+
+ <para>The bbackupctl program sends control commands to the bbackupd
+ daemon. It must be run as the same user as the daemon, and there is no
+ exception for root.</para>
+
+ <para>The command line syntax is:</para>
+
+ <programlisting>/usr/local/bin/bbackupctl [-q] [-c config-file] command</programlisting>
+
+ <para>The -q option reduces the amount of output the program emits,
+ and -c allows an alternative configuration file to be
+ specified.</para>
+
+ <para>Valid commands are: </para>
+
+ <itemizedlist>
+ <listitem>
+ <para><emphasis role="bold">terminate</emphasis></para>
+
+ <para>Stop the bbackupd daemon now (equivalent to kill)</para>
+ </listitem>
+
+ <listitem>
+ <para><emphasis role="bold">reload</emphasis></para>
+
+ <para>Reload the configuration file (equivalent to kill
+ -HUP)</para>
+ </listitem>
+
+ <listitem>
+ <para><emphasis role="bold">sync</emphasis></para>
+
+ <para>Connect to the server and synchronise files now</para>
+ </listitem>
+ </itemizedlist>
+
+ <para><emphasis role="bold">bbackupctl</emphasis> communicates with
+ the server via a UNIX domain socket, specified in bbackupd.conf with
+ the CommandSocket directive. This does not need to be specified, and
+ <emphasis role="bold">bbackupd</emphasis> will run without the command
+ socket, but in this case bbackupctl will not be able to communicate
+ with the daemon.</para>
+
+ <para>Some platforms cannot check the user id of the connecting
+ process, so this command socket becomes a denial of service security
+ risk. <emphasis role="bold">bbackupd</emphasis> will warn you when it
+ starts up if this is the case on your platform, and you should
+ consider removing the CommandSocket directive on these
+ platforms.</para>
+ </section>
+
+ <section>
+ <title>Using bbackupctl to perform snapshots</title>
+
+ <para><emphasis role="bold">bbackupctl</emphasis>'s main purpose is to
+ implement snapshot based backups, emulating the behaviour of
+ traditional backup software.</para>
+
+ <para>Use bbackupd-config to write a configuration file in snapshot
+ mode, and then run the following command as a cron job. </para>
+
+ <programlisting>/usr/local/bin/bbackupctl -q sync</programlisting>
+
+ <para>This will cause the backup daemon to upload all changed files
+ immediately. <emphasis role="bold">bbackupctl</emphasis> will exit
+ almost immediately, and will not output anything unless there is an
+ error. </para>
+ </section>
+
+ <section>
+ <title>Checking storage space used on the server</title>
+
+ <section>
+ <title>From the client machine</title>
+
+ <para>bbackupquery can tell you how much space is used on the server
+ for this account. Either use the usage command in interactive mode,
+ or type:</para>
+
+ <programlisting>/usr/local/bin/bbackupquery -q usage quit</programlisting>
+
+ <para>to show the space used as a single command.</para>
+ </section>
+
+ <section>
+ <title>On the server</title>
+
+ <para>bbstoreaccounts allows you to query the space used, and change
+ the limits. To display the space used on the server for an account,
+ use:</para>
+
+ <programlisting>/usr/local/bin/bbstoreaccounts info 75AB23C</programlisting>
+
+ <para>To adjust the soft and hard limits on an account, use:</para>
+
+ <programlisting>/usr/local/bin/bbstoreaccounts setlimit 75AB23C new-soft-limit new-hard-limit</programlisting>
+
+ <para>You do not need to restart the server.</para>
+ </section>
+ </section>
+
+ <section>
+ <title>Verify and restore files</title>
+
+ <para>Backups are no use unless you can restore them. The bbackupquery
+ utility does this and more.</para>
+
+ <para>You don't provide any login information to it, as it just picks
+ up the data it needs from /etc/box/bbackupd.conf. You should run it as
+ root so it can find everything it needs.</para>
+
+ <para>Full documentation can be found in the <link
+ linkend="bbackupquery.xml">bbackupquery man page</link>. It follows
+ the model of a command line sftp client quite closely.</para>
+
+ <para>TODO: Link to bbackupquery man-page here.</para>
+
+ <para>On systems where GNU readline is available (by default) it uses
+ that for command line history and editing. Otherwise it falls back to
+ very basic UNIX text entry. </para>
+
+ <para>TODO: Did the readline dependency change to editline?</para>
+
+ <section>
+ <title>Using bbackupquery</title>
+
+ <para>bbackupquery is the tool you use to verify, restore and
+ investigate your backup files with. When invoked, it simply logs
+ into the server using the certificates you have listed in
+ bbackupd.conf. </para>
+
+ <para>After you run bbackupquery, you will see a prompt, allowing
+ you to execute commands. The list (or ls) command lets you view
+ files in the store. It works much like unix ls, but with different
+ options. An example:</para>
+
+ <programlisting>[pthomsen@host bbackupquery]$ bbackupquery
+Box Backup Query Tool v0.10, (c) Ben Summers and contributors 2003-2006
+Using configuration file /etc/box/bbackupd.conf
+Connecting to store...
+Handshake with store...
+Login to store...
+Login complete.
+
+Type "help" for a list of commands.
+
+query &gt; ls
+00000002 -d---- mp3
+00000003 -d---- video
+00000004 -d---- home-pthomsen
+00000005 -d---- root
+query &gt; </programlisting>
+
+ <para>The ls commands shows the directories that are backed up. Now
+ we'll take a closer look at the home-pthomsen directory:</para>
+
+ <programlisting>query &gt; cd home-pthomsen
+query &gt; ls
+00002809 f----- sample.tiff
+0000280a f----- s3.tiff
+0000280b f----- s4.tiff
+0000280d f----- s2.tiff
+0000280e f----- foo.pdf
+0000286c f----- core.28720
+0000339a -d---- .emacs.d
+0000339d -d---- bbackup-contrib
+00003437 f----- calnut.compare.txt
+0000345d f----- DSCN1783.jpg
+0000345e f----- DSCN1782.jpg
+query &gt;</programlisting>
+
+ <para>The ls command takes the following options;</para>
+
+ <itemizedlist>
+ <listitem>
+ <para><emphasis role="bold">-r </emphasis>-- recursively list
+ all files</para>
+ </listitem>
+
+ <listitem>
+ <para><emphasis role="bold">-d</emphasis> -- list deleted
+ files/directories</para>
+ </listitem>
+
+ <listitem>
+ <para><emphasis role="bold">-o</emphasis> -- list old versions
+ of files/directories</para>
+ </listitem>
+
+ <listitem>
+ <para><emphasis role="bold">-I</emphasis> -- don't display
+ object ID</para>
+ </listitem>
+
+ <listitem>
+ <para><emphasis role="bold">-F </emphasis>-- don't display
+ flags</para>
+ </listitem>
+
+ <listitem>
+ <para><emphasis role="bold">-t </emphasis>-- show file
+ modification time (and attr mod time if has the object has
+ attributes, ~ separated)</para>
+ </listitem>
+
+ <listitem>
+ <para><emphasis role="bold">-s</emphasis> -- show file size in
+ blocks used on server (only very approximate indication of size
+ locally)</para>
+ </listitem>
+ </itemizedlist>
+
+ <para>The flags displayed from the ls command are as follows:</para>
+
+ <simplelist>
+ <member>f = file</member>
+
+ <member>d = directory</member>
+
+ <member>X = deleted</member>
+
+ <member>o = old version</member>
+
+ <member>R = remove from server as soon as marked deleted or
+ old</member>
+
+ <member>a = has attributes stored in directory record which
+ override attributes in backup file</member>
+ </simplelist>
+ </section>
+
+ <section>
+ <title>Verify backups</title>
+
+ <para>Since this system is not yet 100% production-ready, you'll be
+ keen to verify that your backups are correct. This is easy:</para>
+
+ <programlisting>/usr/local/bin/bbackupquery "compare -a" quit</programlisting>
+
+ <para>It will report all the differences between the store and the
+ files on disc. It will download everything, so may take a while. You
+ should expect to see some differences on a typical compare, because
+ files which have recently changed are unlikely to have been uploaded
+ yet. Consider checking the timestamps on the files, or keeping a
+ record of these messages and comparing them with a future
+ verification.</para>
+
+ <para>If you would like to do a "quick" check which just downloads
+ file checksums and compares against that, then do:</para>
+
+ <programlisting>/usr/local/bin/bbackupquery "compare -aq" quit</programlisting>
+
+ <para>However, this does not check that the file attributes are
+ correct, and since the checksums are generated on the client they
+ may not reflect the data on the server if there is a problem -- the
+ server cannot check the encrypted contents. View this as a good
+ indication, rather than a definite check that your backup verifies
+ correctly.</para>
+
+ <para>You may wish to run either one as a cron job while testing
+ this system. </para>
+ </section>
+
+ <section>
+ <title>Restore backups</title>
+
+ <para>You will need the keys file created when you configured the
+ server. Without it, you cannot restore the files; this is the
+ downside of encrypted backups. However, by keeping the small keys
+ file safe, you indirectly keep your entire backup safe.</para>
+
+ <para>The first step is to recreate the configuration of the backup
+ client. It's probably best to have stored the /etc/box directory
+ with your keys. But if you're recreating it, all you really need is
+ to have got the login infomation correct (ie the certs and
+ keys).</para>
+
+ <para>Don't run bbackupd yet! It will mark all your files as deleted
+ if you do, which is not hugely bad in terms of losing data, just a
+ major inconvenience. (This assumes that you are working from a blank
+ slate. If you want to restore some files to a different location,
+ it's fine to restore while bbackupd is running, just do it outside a
+ backed up directory to make sure it doesn't start uploading the
+ restored files.)</para>
+
+ <para>Type: </para>
+
+ <programlisting>/usr/local/bin/bbackupquery</programlisting>
+
+ <para>to run it in interactive mode.</para>
+
+ <para>Type:</para>
+
+ <programlisting>list</programlisting>
+
+ <para>to see a list of the locations stored on the server.</para>
+
+ <para>For each location you want to restore, type:</para>
+
+ <programlisting>restore name-on-server local-dir-name</programlisting>
+
+ <para>The directory specified by local-dir-name must not exist yet.
+ If the restore is interrupted for any reason, repeat the above
+ steps, but add the <emphasis role="bold">-r</emphasis> flag to the
+ restore command to tell it to resume.</para>
+ </section>
+
+ <section>
+ <title>Retrieving deleted and old files</title>
+
+ <para>Box Backup makes old versions of files and files you have
+ deleted available, subject to there being enough disc space on the
+ server to hold them.</para>
+
+ <para>This is how to retrieve them using bbackupquery. Future
+ versions will make this far more user-friendly.</para>
+
+ <para>Firstly, run bbackupquery in interactive mode. It behaves in a
+ similar manner to a command line sftp client. </para>
+
+ <programlisting>/usr/local/bin/bbackupquery</programlisting>
+
+ <para>Then navigate to the directory containing the file you want,
+ using list, cd and pwd.</para>
+
+ <programlisting>query &gt; cd home/profiles/USERNAME</programlisting>
+
+ <para>List the directory, using the "o" option to list the files
+ available without filtering out everything apart from the current
+ version. (if you want to see deleted files as well, use list
+ -odt)</para>
+
+ <programlisting>query &gt; list -ot
+00000078 f--o- 2004-01-21T20:17:48 NTUSER.DAT
+00000079 f--o- 2004-01-21T20:17:48 ntuser.dat.LOG
+0000007a f--o- 2004-01-21T17:55:12 ntuser.ini
+0000007b f---- 2004-01-12T15:32:00 ntuser.pol
+0000007c -d--- 1970-01-01T00:00:00 Templates
+00000089 -d--- 1970-01-01T00:00:00 Start Menu
+000000a0 -d--- 1970-01-01T00:00:00 SendTo
+000000a6 -d--- 1970-01-01T00:00:00 Recent
+00000151 -d--- 1970-01-01T00:00:00 PrintHood
+00000152 -d--- 1970-01-01T00:00:00 NetHood
+00000156 -d--- 1970-01-01T00:00:00 My Documents
+0000018d -d--- 1970-01-01T00:00:00 Favorites
+00000215 -d--- 1970-01-01T00:00:00 Desktop
+00000219 -d--- 1970-01-01T00:00:00 Cookies
+0000048b -d--- 1970-01-01T00:00:00 Application Data
+000005da -d--- 1970-01-01T00:00:00 UserData
+0000437e f--o- 2004-01-24T02:45:43 NTUSER.DAT
+0000437f f--o- 2004-01-24T02:45:43 ntuser.dat.LOG
+00004380 f--o- 2004-01-23T17:01:29 ntuser.ini
+00004446 f--o- 2004-01-24T02:45:43 NTUSER.DAT
+00004447 f--o- 2004-01-24T02:45:43 ntuser.dat.LOG
+000045f4 f---- 2004-01-26T15:54:16 NTUSER.DAT
+000045f5 f---- 2004-01-26T15:54:16 ntuser.dat.LOG
+000045f6 f---- 2004-01-26T16:54:31 ntuser.ini</programlisting>
+
+ <para>(this is a listing from a server which is used as a Samba
+ server for a network of Windows clients.) You now need to fetch the
+ file using it's ID, rather than it's name. The ID is the hex number
+ in the first column. Fetch it like this: </para>
+
+ <programlisting>query &gt; get -i 0000437e NTUSER.DAT
+Object ID 0000437e fetched successfully.</programlisting>
+
+ <para>The object is now available on your local machine. You can use
+ lcd to move around, and sh ls to list directories on your local
+ machine.</para>
+ </section>
+ </section>
+ </section>
+
+ <section>
+ <title id="FixCorruptions">Fixing corruptions of store data</title>
+
+ <para>This section gives help on what to do if your server has suffered
+ corruption, for example, after an unclean shutdown or other OS or
+ hardware problem.</para>
+
+ <para>In general, as updates to the store are made in an atomic manner,
+ the most likely result is wasted disc space. However, if really bad
+ things happen, or you believe that there is a lot of wasted space, then
+ these instructions will help to restore your data.</para>
+
+ <para>You know you will need to do something if you get strange errors,
+ and bbackupd attempts to contact the server every 100 seconds or so. Or
+ if one of the discs in your RAID disc set has failed.</para>
+
+ <para>After following these instructions, the end result will be that
+ bbackupquery will be able to see all the files which were stored on your
+ server, and retrieve them. Some of them may be in lost+found directories
+ in the root of the store (or in their original position if they have
+ been moved) but they will all be able to be retrieved.</para>
+
+ <para>After you have retrieved the files you want, bbackupd will upload
+ new versions where necessary, and after about two days, mark any
+ lost+found directories as deleted. Finally, those directories will be
+ removed by the housekeeping process on the server.</para>
+
+ <para>These instructions assume you're working on account 1234,
+ subsitute this for whatever account you're actually working on. These
+ will need to be repeated for all affected accounts. </para>
+
+ <section>
+ <title>Stop bbackupd</title>
+
+ <para>First, make sure that bbackupd is not running on the client
+ machine for the account you are going to recover. Use kill to
+ terminate it. This step is not strictly necessary, but is recommended.
+ During any checks on the account, bbackupd will be unable to log in,
+ and after they are complete, the account is marked as changed on the
+ server so bbackupd will perform a complete scan.</para>
+ </section>
+
+ <section>
+ <title>Are you using RAID on the server?</title>
+
+ <para>At the moment, the raidfile recovery tools have not been
+ written. However, when two out of three files are available, the
+ server will run succesfully, even if it complains a lot in the logs.
+ So, your best bet here is to fix the accounts, if necessary, and
+ retrieve any files you need. Then move the old store directories aside
+ (in case you need them) and start afresh with new accounts, and let
+ the clients upload all their data again. These utilities will be
+ written shortly! </para>
+
+ <para>TODO: Is this true anymore???</para>
+ </section>
+
+ <section>
+ <title>Check and fix the account</title>
+
+ <para>First, run the check utility, and see what errors it
+ reports.</para>
+
+ <programlisting>/usr/local/bin/bbstoreaccounts check 1234</programlisting>
+
+ <para>This will take some time, and use a fair bit of memory (about 16
+ bytes per file and directory). If the output looks plausible and
+ reports errors which need fixing, run it again but with the fix
+ flag:</para>
+
+ <programlisting>/usr/local/bin/bbstoreaccounts check 1234 fix</programlisting>
+
+ <para>This will fix any errors, and remove unrecoverable files.
+ Directories will be recreated if necessary.</para>
+
+ <para><emphasis role="bold">NOTE</emphasis>: The utility may adjust
+ the soft and hard limits on the account to make sure that housekeeping
+ will not remove anything -- check these afterwards. </para>
+ </section>
+
+ <section>
+ <title>Grab any files you need with bbackupquery</title>
+
+ <para>At this point, you will have a working store. Every file which
+ was on the server, and wasn't corrupt, will be available.</para>
+
+ <para>On the client, use bbackupquery to log in and examine the store.
+ (type help at the prompt for instructions). Retrieve any files you
+ need, paying attention to any lost+found directories in the root
+ directory of the store.</para>
+
+ <para>You can skip this step if you are sure that the client machine
+ is fine -- in this case, bbackupd will bring the store up to date.
+ </para>
+ </section>
+
+ <section>
+ <title>Restart bbackupd</title>
+
+ <para>Restart bbackupd on the client machine. The store account will
+ be brought up to date, and files in the wrong place will be marked for
+ eventual deletion.</para>
+ </section>
+ </section>
+
+ <section>
+ <title>Troubleshooting</title>
+
+ <para>If you are trying to fix a store after your disc has been
+ corrupted, see <link linkend="FixCorruptions">Fixing corruptions of
+ store data</link>.</para>
+
+ <para>Unfortunately, the error messages are not particularly helpful at
+ the moment. This page lists some of the common errors, and the most
+ likely causes of them.</para>
+
+ <para>When an error occurs, you will see a message like 'Exception:
+ RaidFile/OSFileError (2/8)' either on the screen or in your log files.
+ (it is recommended you set up another log file as recommended in the
+ server setup instructions.)</para>
+
+ <para>This error may not be particularly helpful, although some do have
+ extra information about probable causes. To get further information,
+ check the ExceptionCodes.txt file in the root of the distribution. This
+ file is generated by the ./configure script, so you will need to have
+ run that first.</para>
+
+ <para>Some common causes of exceptions are listed below.</para>
+
+ <para>Please email me with any other codes you get, and I will let you
+ know what they mean, and add notes here. </para>
+
+ <section>
+ <title>RaidFile (2/8)</title>
+
+ <para>This is found either when running bbstoreaccounts or in the
+ bbstored logs.</para>
+
+ <para><emphasis role="bold">Problem</emphasis>: The directories you
+ specified in the raidfile.conf are not writable by the _bbstored
+ user.</para>
+
+ <para><emphasis role="bold">Resolution</emphasis>: Change permissions
+ appropriately.</para>
+ </section>
+
+ <section>
+ <title>Common (1/2)</title>
+
+ <para>This usually occurs when the configuration files can't be
+ opened.</para>
+
+ <para><emphasis role="bold">Problem</emphasis>: You created your
+ configurations in non-standard locations, and the programs cannot find
+ them.</para>
+
+ <para><emphasis role="bold">Resolution</emphasis>: Explicitly specify
+ configuration file locations to daemons and programs. For
+ example</para>
+
+ <programlisting>/usr/local/bin/bbstored /some/other/dir/bbstored.config /usr/local/bin/bbackupquery -c /some/other/dir/bbackupd.config</programlisting>
+
+ <para>(daemons specify the name as the first argument, utility
+ programs with the -c option).</para>
+
+ <para><emphasis role="bold">Problem</emphasis>: bbstored can't find
+ the raidfile.conf file specified in bbstored.conf.</para>
+
+ <para><emphasis role="bold">Resolution</emphasis>: Edit bbstored.conf
+ to point to the correct location of this additional configuration
+ file. </para>
+ </section>
+
+ <section>
+ <title>Server (3/16)</title>
+
+ <para>The server can't listen for connections on the IP address
+ specified when you configured it.</para>
+
+ <para><emphasis role="bold">Problem</emphasis>: This probably means
+ you've specified the wrong hostname to bbstored-config -- maybe your
+ server is behind a NAT firewall?</para>
+
+ <para><emphasis role="bold">Resolution</emphasis>: Edit bbstored.conf
+ and correct the ListenAddresses line. You should replace the server
+ address with the IP address of your machine. </para>
+ </section>
+
+ <section>
+ <title>Connection (7/x)</title>
+
+ <para>These errors all relate to connections failing -- you may see
+ them during operation if there are network failures or other problems
+ between the client and server. The backup system will recover from
+ them automatically.</para>
+
+ <section>
+ <title>Connection (7/30) - SSL problems</title>
+
+ <para>Log snippet from client side:</para>
+
+ <programlisting>bbackupd[1904]: Opening connection to server xxxx.xxx...
+bbackupd[1904]: SSL err during Connect: error:xxxxxxxx:rsa routines:RSA_padding_check_PKCS1_type_1:block type is not 01
+bbackupd[1904]: SSL err during Connect: error:xxxxxxxx:rsa routines:RSA_EAY_PUBLIC_DECRYPT:padding check failed
+bbackupd[1904]: SSL err during Connect: error:xxxxxxxx:asn1 encoding routines:ASN1_verify:EVP lib
+bbackupd[1904]: SSL err during Connect: error:xxxxxxxx:SSL routines:SSL3_GET_SERVER_CERTIFICATE:certificate verify failed
+bbackupd[1904]: TRACE: Exception thrown: ConnectionException(Conn_TLSHandshakeFailed) at SocketStreamTLS.cpp(237)
+bbackupd[1904]: Exception caught (7/30), reset state and waiting to retry...</programlisting>
+
+ <para>And from the server:</para>
+
+ <programlisting>bbstored[19291]: Incoming connection from xx.xxx.xx.xxx port xxxxx (handling in child xxxxx)
+bbstored[21588]: SSL err during Accept: error:xxxxxxxx:SSL routines:SSL3_READ_BYTES:tlsv1 alert decrypt error
+bbstored[21588]: in server child, exception Connection TLSHandshakeFailed (7/30) -- terminating child</programlisting>
+
+ <para><emphasis role="bold">Solution</emphasis>: Create a new CA on
+ the server side and re-generate the client certificate. Re-creating
+ the client certificate request is not necessary.</para>
+ </section>
+ </section>
+
+ <section>
+ <title>Advanced troubleshooting</title>
+
+ <para>If this really doesn't help, then using the DEBUG builds of the
+ system will give you much more information -- a more descriptive
+ exception message and the file and line number where the error
+ occurred.</para>
+
+ <para>For example, if you are having problems with bbstoreaccounts,
+ build the debug version with:</para>
+
+ <programlisting>cd boxbackup-0.0
+cd bin/bbstoreaccounts
+make</programlisting>
+
+ <para>Within the module directories, make defaults to building the
+ debug version. At the top level, it defaults to release.</para>
+
+ <para>This will build an executable in debug/bin/bbstoreaccounts which
+ you can then use instead of the release version. It will give far more
+ useful error messages.</para>
+
+ <para>When you get an error message, use the file and line number to
+ locate where the error occurs in the code. There will be comments
+ around that line to explain why the exception happened.</para>
+
+ <para>If you are using a debug version of a daemon, these extended
+ messages are found in the log files.</para>
+ </section>
+ </section>
+ </chapter>
+
+ <appendix>
+ <title id="WORoot">Running without root</title>
+
+ <para>It is possible to run both the server and client without root
+ privileges.</para>
+
+ <section>
+ <title>Server</title>
+
+ <para>The server, by default, runs as a non-root user. However, it
+ expects to be run as root and changes user to a specified user as soon
+ as it can, simply for administrative convenience. The server uses a port
+ greater than 1024, so it doesn't need root to start.</para>
+
+ <para>To run it entirely as a non-root user, edit the bbstored.conf
+ file, and remove the User directive from the Server section. Then simply
+ run the server as your desired user. </para>
+ </section>
+
+ <section>
+ <title>Client</title>
+
+ <para>The client requires root for normal operation, since it must be
+ able to access all files to back them up. However, it is possible to run
+ the client as a non-root user, with certain limitations.</para>
+
+ <para>Follow the installation instructions, but install the executable
+ files manually to somewhere in your home directory. Then use
+ bbackupd-config to configure the daemon, but use a directory other than
+ /etc/box, probably somewhere in your home directory.</para>
+
+ <para>All directories you specify to be backed up must be readable, and
+ all files must be owned by the user and readable to that user.</para>
+
+ <para>Important: If any file or directory is not readable by this user,
+ the backup process will skip that file or directory. Keep an eye on the
+ logs for reports of this failure.</para>
+
+ <para>Non-root operation of the backup client is recommended only for
+ testing, and should not be relied on in a production environment.
+ </para>
+ </section>
+ </appendix>
+</book> \ No newline at end of file
diff --git a/documentation/boxbackup/instguide.sgml b/documentation/boxbackup/instguide.xml
index 2a5bc3e3..2a5bc3e3 100644
--- a/documentation/boxbackup/instguide.sgml
+++ b/documentation/boxbackup/instguide.xml