]> Copyright © 2003 - 2007, Ben Summers and contributors. All rights reserved. Redistribution and use in source and binary forms, with or without modification, are permitted provided that the following conditions are met: Redistributions of source code must retain the above copyright notice, this list of conditions and the following disclaimer. Redistributions in binary form must reproduce the above copyright notice, this list of conditions and the following disclaimer in the documentation and/or other materials provided with the distribution. All use of this software and associated advertising materials must display the following acknowledgement: This product includes software developed by Ben Summers and contributors. The names of the Authors may not be used to endorse or promote products derived from this software without specific prior written permission. [Where legally impermissible the Authors do not disclaim liability for direct physical injury or death caused solely by defects in the software unless it is modified by a third party.] THIS SOFTWARE IS PROVIDED BY THE AUTHORS "AS IS" AND ANY EXPRESS OR IMPLIED WARRANTIES, INCLUDING, BUT NOT LIMITED TO, THE IMPLIED WARRANTIES OF MERCHANTABILITY AND FITNESS FOR A PARTICULAR PURPOSE ARE DISCLAIMED. IN NO EVENT SHALL THE AUTHORS BE LIABLE FOR ANY DIRECT, INDIRECT, INCIDENTAL, SPECIAL, EXEMPLARY, OR CONSEQUENTIAL DAMAGES (INCLUDING, BUT NOT LIMITED TO, PROCUREMENT OF SUBSTITUTE GOODS OR SERVICES; LOSS OF USE, DATA, OR PROFITS; OR BUSINESS INTERRUPTION) HOWEVER CAUSED AND ON ANY THEORY OF LIABILITY, WHETHER IN CONTRACT, STRICT LIABILITY, OR TORT (INCLUDING NEGLIGENCE OR OTHERWISE) ARISING IN ANY WAY OUT OF THE USE OF THIS SOFTWARE, EVEN IF ADVISED OF THE POSSIBILITY OF SUCH DAMAGE.

After you've downloaded and compiled the programs you need to install the programs on your server. As root do the following: make install-backup-server This assumes that you are installing on the same server that you compiled the software on. If not, copy the boxbackup-x.xx-backup-server-OSNAME.tgz file to the server you want to run on, and install there. For example (on Mac OS X): tar zxvf boxbackup-0.10-server-darwin8.5.0.tgz cd boxbackup-0.10-server-darwin8.5.0 ./install-backup-server Then create the user for the backup daemon on the server: useradd _bbstored Box Backup has a built-in software RAID facility (redundant array of inexpensive disks) for the backup store. This allows you to spread the store data over three disks, and recover from the loss of any one disk without losing data. However, this is now deprecated, and you are recommended to use the software or hardware RAID facilities of your operating system instead. Use the following command if you want to create a simple server without Box Backup RAID: mkdir /tmp/boxbackupRepository # Create the directory chown _bbstored /tmp/boxbackupRepository/ # Change the owner to the new boxbackup daemon user /usr/local/bin/raidfile-config /etc/box/ 1024 /tmp/boxbackupRepository #substitute 1024 with the desired blocksize #substitute /tmp/boxbackupRepository with a directory that exists where you want the backup store located #/usr/local/bin/raidfile-config --help shows you the options Then create the configuration file /etc/box/bbstored.conf The hostname is tricky as it is used for two things: The name of the server in the certificate and the address the server is listening on. Since you might be using NAT, might move the server around or the domain name might change, choose a name that describes the server. When the network address of the server changes, you need to update the ListenAddresses directive in the /etc/box/bbstored.conf file. /usr/local/bin/bbstored-config /etc/box hostname _bbstored This last step outputs 5 instructions that you must execute to the letter. A lot of questions are raised on the mailing list because these steps have not been followed properly. TODO: Expand on this. Explain the 5 steps in detail. If you want to run the server as a non-root user, look here.

There are two steps involved to create an account. You need to create the account on the server, and sign a certificate to give the client permission to connect to the server. Running a Certification Authority for TLS (SSL) connections is not trivial. However, a script to does most of the work in a way which should be good enough for most deployments. The certificate authority directory is intended to be stored on another server. It should not be kept on the backup server, in order to limit the impact of a server compromise. The instructions and the script assume that it will be kept elsewhere, so will ask you to copy files to and from the CA. SSL certificates contain validity dates, including a "valid from" time. If the clock on the machine which signs the certificates is not syncronised to the clocks of the machines using these certificates, you will probably get strange errors until the start time is reached on all machines. If you get strange errors when attempting to use new certificates, check the clocks on all machines (client, store and CA). You will probably just need to wait a while until the certificates become valid, rather than having to regenerate them.

It is recommended that you keep your Certificate Authority on a separate machine than either the client or the server, preferably without direct network access. The contents of this directory control who can access your backup store server. To setup the basic key structure, do the following: /usr/local/bin/bbstored-certs ca init (See OpenSSL notes if you get an OpenSSL error) This creates the directory called ca in the current directory, and initialises it with basic keys.

When you use the bbstored-config script to set up a config file for a server, it will generate a certificate request (CSR) for you. Transfer it to the machine with your CA, then do: /usr/local/bin/bbstored-certs ca sign-server hostname-csr.pem This signs the certificate for the server. Follow the instructions in the output on which files to install on the server. The CSR file is now no longer needed. Make sure you run this command from the directory above the directory 'ca'. TODO: Explain instructions in output.

Choose an account number for the user. This must be unique on the server, and is presented as a 31 bit number in hex greater than 0, for example, 1 or 75AB23C. Then on the backup store server, create the account with: /usr/local/bin/bbstoreaccounts create 75AB23C 0 4096M 4505M This looks complicated. The numbers are, in order: The account number allocated (hex) The RAID disc set (0 if you use raidfile-config and don't add a new set) Soft limit (size) Hard limit (size) The sizes are are specified in Mb, Gb, or blocks, depending on the suffix. 1M specifies 1 Mb, 1G specifies 1 Gb, and 1B specifies 1 block, the size of which depends on how you have configured the raidfile system with raidfile-config. In this example, I have allocated 4Gb (assuming you use 2048 byte blocks as per my example) as the soft limit, and 4Gb + 10% as the hard limit. NOTE The sizes specified here are pre-RAID. So if you are using userland RAID, you are actually allocating two-thirds of this amount. This means that, when you take compression into account, that if you allocate 2Gb on the server, it'll probably hold about 2Gb of backed up files (depending on the compressability of those files). The backup client will (voluntarily) try not to upload more data than is allowed by the soft limit. The store server will refuse to accept a file if it would take it over the hard limit, and when doing housekeeping for this account, try and delete old versions and deleted files to reduce the space taken to below the soft limit. This command will create some files on disc in the raid file directories (if you run as root, the utility will change to the user specified in the bbstored.conf file to write them) and update the accounts file. A server restart is not required. NOTE If you get a message saying 'Exception: RaidFile (2/8)', the directories you specified in the raidfile.conf are not writable by the _bbstored user -- fix it, and try again. Finally, tell the user their account number, and the hostname of your server. They will use this to set up the backup client, and send you a CSR. This has the account number embedded in it, and you should be sure that it has the right account number in it. Sign this CSR with this command: /usr/local/bin/bbstored-certs ca sign 75AB23C-csr.pem Don't forget to check that the embedded account number is correct! Then send the two files back to the user, as instructed by the script. Please read the Troubleshooting page if you have problems. TODO: Link to troubleshooting...

You may wish to see what's going on with the server. Edit /etc/syslog.conf, and add: local6.info /var/log/box local5.info /var/log/raidfile Note: Separators must be tabs, otherwise these entries will be ignored. touch /var/log/box touch /var/log/raidfile Set up log rotation for these new log files. For example, if you have /etc/newsyslog.conf, add the following lines to it: /var/log/box 644 7 2000 * Z /var/log/raidfile 644 7 2000 * Z If you have /etc/logrotate.d, create a new file in there (for example /etc/logrotate.d/boxbackup) containing the following: /var/log/box /var/log/raidfile { weekly create compress rotate 52 } Then restart syslogd, for example: /etc/init.d/syslogd restart

Before you can do any configuration, you need to know the hostname of the server you will be using, and your account number on that server. Later in the process, you will need to send a certificate request to the administrator of that server for it to be signed. Installation is covered in the compiling and installing section. You only need the backup-client parcel. It is important that you read all the output of the config scripts. See the end of this page for an example. The backup client has to be run as root, because it needs to read all your files to back them up, although it is possible to back up a single user's files by running it as that user. (Tip: specify a directory other than /etc/box, and then give the alternate config file as the first argument to bbackupd). However, it will fall over if you don't give yourself read access to one of your files.

Run the bbackupd-config script to generate the configuration files and generate a private key and certificate request. /usr/local/bin/bbackupd-config /etc/box lazy 999 hostname /var/bbackupd /home (See OpenSSL notes if you get an OpenSSL error) The items in bold need to be changed. In order, they are the account number, the hostname of the server you're using, and finally, the directories you want backed up. You can include as many you want here. However, the directories you specify must not contain other mounted file systems within them at any depth. Specify them separately, one per mount point. No checks are currently made to catch bad configuration of this nature! You may also want to consider changing the mode from lazy to snapshot, depending on what your system is used for: Lazy Mode This mode regularly scans the files, with only a rough schedule. It uploads files as and when they are changed, if the latest version is more than a set age. This is good for backing up user's documents stored on a server, and spreads the load out over the day. Snapshot Mode This mode emulates the traditional backup behaviour of taking a snapshot of the filesystem. The backup daemon does absolutely nothing until it is instructed to make a backup using the bbackupctl utility (probably as a cron job), at which point it uploads all files which have been changed since the last time it uploaded. When you run the config script, it will tell you what you need to do next. Don't forget to read all the output. An example is shown at the end of this page, but the instructions for your installation may be different.

After you have sent your certificate request off to the server administrator and received your certificate and CA root back, install them where instructed by the bbackupd-config script during basic bbackupd configuration. You can then run the daemon (as root) by running /usr/local/bin/bbackupd, and of course, adding it to your system's startup scripts. The first time it's run it will upload everything. Interrupting it and restarting it will only upload files which were not uploaded before - it's very tolerant. If you run in snapshot mode, you will need to add a cron job to schedule backups. The config script will tell you the exact command to use for your system. Please read the Troubleshooting page if you have problems. Remember to make a traditional backup of the keys file, as instructed. You cannot restore files without it. It is recommended that you backup up all of /etc/box as it will make things easier if you need to restore files. But only the keys are absolutely essential. If you want to see what it's doing in more detail (probably a good idea), follow the instructions in the server setup to create new log files with syslog.

By editing the /etc/box/bbackupd.conf file, you can add and remove directories to back up - see comments in this file for help. Send bbackupd a HUP signal after you modify it. When you remove a location, it will not be marked as deleted immediately. Instead, bbackupd waits about two days before doing so, just in case you change your mind. After this, it will be eventually removed from the store by the housekeeping process. Run as root. The backup client is designed to be run as root. It is possible to run without root, but this is not recommended. Clock synchronisation for file servers. If you are using the backup client to backup a filesystem served from a fileserver, you should ideally ensure that the fileserver clocks are synchronised with the fileserver. bbackupd will cope perfectly well if the clocks are not synchronised. Errors up to about half an hour cause no problems. Larger discrepancies cause a loss of efficiency and the potential to back up a file during a write process. There is a configuration parameter MaxFileTimeInFuture, which specifies how far in the future a file must be for it to be uploaded as soon as it is seen. You should not need to adjust this (default is 2 days). Instead, get those clocks synchronised. Excluding files and directories from the backup. Within the bbackupd.conf file, there is a section named BackupLocations which specifies which locations on disc should be backed up. It has subsections, each of which is in the format: name { Path = /path/of/directory (optional exclude directives) } name is derived from the Path by the config script, but should merely be unique. The exclude directives are of the form: [Exclude|AlwaysInclude][File|Dir][|sRegex] = regex or full pathname (The regex suffix is shown as 'sRegex' to make File or Dir plural) For example: ExcludeDir = /home/guest-user ExcludeFilesRegex = *.(mp3|MP3)\$ AlwaysIncludeFile = /home/username/veryimportant.mp3 This excludes the directory /home/guest-user from the backup along with all mp3 files, except one MP3 file in particular. In general, Exclude excludes a file or directory, unless the directory is explicitly mentioned in a AlwaysInclude directive. If a directive ends in Regex, then it is a regular expression rather than a explicit full pathname. See man 7 re_format for the regex syntax on your platform.

This is an example of output from the bbstored-config script. Follow the instructions output by your script, not the ones here -- they may be different for your system. /usr/local/bin/bbackupd-config /etc/box lazy 51 server.example.com /var/bbackupd /home /etc/samba Setup bbackupd config utility. Configuration: Writing configuration file: /etc/box/bbackupd.conf Account: 51 Server hostname: server.example.com Directories to back up: /home /etc/samba Note: If other file systems are mounted inside these directories, then problems may occur with files on the store server being renamed incorrectly. This will cause efficiency problems, but not affect the integrity of the backups. WARNING: Directories not checked against mountpoints. Check mounted filesystems manually. Creating /etc/box... Creating /etc/box/bbackupd Generating private key... [OpenSSL output omitted] Generating keys for file backup Writing notify script /etc/box/bbackupd/NotifyStoreFull.sh Writing configuration file /etc/box/bbackupd.conf =================================================================== bbackupd basic configuration complete. What you need to do now... 1) Make a backup of /etc/box/bbackupd/51-FileEncKeys.raw This should be a secure offsite backup. Without it, you cannot restore backups. Everything else can be replaced. But this cannot. KEEP IT IN A SAFE PLACE, OTHERWISE YOUR BACKUPS ARE USELESS. 2) Send /etc/box/bbackupd/51-csr.pem to the administrator of the backup server, and ask for it to be signed. 3) The administrator will send you two files. Install them as /etc/box/bbackupd/51-cert.pem /etc/box/bbackupd/serverCA.pem after checking their authenticity. 4) You may wish to read the configuration file /etc/box/bbackupd.conf and adjust as appropraite. There are some notes in it on excluding files you do not wish to be backed up. 5) Review the script /etc/box/bbackupd/NotifyStoreFull.sh and check that it will email the right person when the store becomes full. This is important -- when the store is full, no more files will be backed up. You want to know about this. 6) Start the backup daemon with the command /usr/local/bin/bbackupd in /etc/rc.local, or your local equivalent. Note that bbackupd must run as root. =================================================================== Remember to make a secure, offsite backup of your backup keys, as described in Basic configuration above. If you do not, and that key is lost, you have no backups.

Box Backup has many options in its configuration file. We will try to list them all here. First of all, here is an example configuration file, for reference: StoreHostname = localhost AccountNumber = 0x2 KeysFile = /etc/box/2-FileEncKeys.raw CertificateFile = /etc/box/2-cert.pem PrivateKeyFile = /etc/box/2-key.pem TrustedCAsFile = /etc/box/serverCA.pem DataDirectory = /var/run/boxbackup NotifyScript = /etc/box/NotifySysadmin.sh CommandSocket = /var/run/box/bbackupd.sock UpdateStoreInterval = 86400 MinimumFileAge = 3600 MaxUploadWait = 7200 FileTrackingSizeThreshold = 65536 DiffingUploadSizeThreshold = 65536 MaximumDiffingTime = 20 ExtendedLogging = no LogAllFileAccess = yes Server { PidFile = /var/run/bbackupd.pid } BackupLocations { etc { Path = /etc } home { Path = /home ExcludeDir = /home/shared ExcludeDir = /home/chris/.ccache ExcludeDir = /home/chris/.mozilla/firefox/vvvkq3vp.default/Cache } } As you can see from the example above, the configuration file has a number of subsections, enclosed in curly braces {}. Some options appear outside of any subsection, and we will refer to these as root options. The available options in each section are described below. Every option has the form name = value. Names are not case-sensitive, but values are. Depending on the option, the value may be: a path (to a file or directory); a number (usually in seconds or bytes); a boolean (the word Yes or No); a hostname (or IP address). Paths are specified in native format, i.e. a full Windows path with drive letter on Windows clients, or a full Unix path on Unix clients. StoreObjectInfoFile = /var/state/boxbackup/bbackupd.dat StoreObjectInfoFile = C:\Program Files\Box Backup\data\bbackupd.dat The use of relative paths (which do not start with a forward slash on Unix, or a drive specification on Windows) is possible but not recommended, since they are interpreted relative to the current working directory when bbackupd was started, which is liable to change unexpectedly over time. Numbers which start with "0x" are interpreted as hexadecimal. Numbers which do not start with "0x" are interpreted as decimal.

These options appear outside of any subsection. By convention they are at the beginning of the configuration file. Some options are required, and some are optional. StoreHostname (required) The Internet host name (DNS name) or IP address of the server. This is only used to connect to the server. AccountNumber (required) The number of the client's account on the server. This must be provided by the server operator, and must match the account number in the client's certificate, otherwise the client will not be able to log into the server. The account number may be specified in hexadecimal (starting with 0x, as in the example above) or in decimal, but since the server operator works in hexadecimal, that format is highly recommended and is the default. KeysFile (required) The path to the file containing the encryption key used for data encryption of client file data and filenames. This is the most important file to keep safe, since without it your backups cannot be decrypted and are useless. Likewise, if an attacker gets access to this key and to your encrypted backups, he can decrypt them and read all your data. Do not change the encryption key without deleting all files from the account on the server first. None of your old files on the store will be readable if you do so, and if you change it back, none of the files uploaded with the new key will be readable. CertificateFile (required) The path to the OpenSSL client certificate in PEM format. This is supplied by the server operator in response to the certificate request which you send to them. Together with the PrivateKeyFile, this provides access to the store server and the encrypted data stored there. It is not critical to protect this file or to back it up safely, since it can be regenerated by creating a new certificate request, and asking the server operator to sign it. You may wish to back it up, together with the PrivateKeyFile, to avoid this inconvenience if you lose all your data and need quick access to your backups. If you do back them up, you should keep them in a separate location to the KeysFile, since any person holding the KeysFile and the PrivateKeyFile can gain access to your encrypted data and decrypt it. PrivateKeyFile (required) The path to the OpenSSL private key in PEM format. This is generated at the same time as the certificate request, but there is no need to send it to the server operator, and you should not do so, in case the communication is intercepted by an attacker. Together with the CertificateFile, this provides access to the store server and the encrypted data stored there. See the notes under CertificateFile for information about backing up this file. TrustedCAsFile (required) The path to the OpenSSL certificate of the Client Certificate Authority (CCA), in PEM format. This is supplied by the server operator along with your account details, or along with your signed client certificate. This is used to verify that the server which you are connecting to is authorised by the person who signed your certificate. It protects you against DNS and ARP poisoning and IP spoofing attacks. DataDirectory (required) The path to a directory where bbackupd will keep local state information. This consists of timestamp files which identify the last backup start and end times, used by bbackupquery to determine whether files have changed, and optionally a database of inode numbers, which are used to check for files being renamed. The database is only saved if Box Backup is built with Berkeley Database (BDB) support. NotifyScript (optional) The path to the script or command to run when the Box Backup client detects an error during the backup process. This is normally used to notify the client system administrator by e-mail when a backup fails for any reason. The script or command is called with one of the following additional arguments to identify the cause of the problem: store-full The backup store is full. No new files are being uploaded. If some files are marked as deleted, they should be removed in due course by the server's housekeeping process. Otherwise, you need to remove some files from your backup set, or ask the store operator for more space. read-error One or more files which were supposed to be backed up could not be read. This could be due to: running the server as a non-root user; backing up a mounted filesystem such as NFS; access control lists being applied to some files; SELinux being enabled; trying to back up open files under Windows; strange directory permissions such as 0000 or 0400. Check the client logs, e.g. /var/log/bbackupd on Unix, or the Windows Event Viewer in Control Panel > Administrative Tools, for more information about which files are not being backed up and why. backup-error There was a communications error with the server, or an unexpected exception was encountered during a backup run. Check the client logs, e.g. /var/log/box on Unix, or the Windows Event Viewer in Control Panel > Administrative Tools, for more information about the problem. You may wish to check your Internet access to the server, check that the server is running, and ask your server operator to check your account on the server. CommandSocket (optional) The path to the Unix socket which bbackupd creates when running, and which bbackupctl uses to communicate with it, for example to force a sync or a configuration reload. If this option is omitted, no socket will be created, and bbackupctl will not function. Unix sockets appear within the filesystem on Unix, as a special type of file, and must be created in a directory which exists and to which bbackupd has write access, and bbackupctl has read access. On Windows, the path is ignored, and a named pipe is created instead. This does not currently have any security attached, so it can be accessed by any user. Unlike a Unix socket it can also be accessed remotely. Please use this option with extreme caution on Windows, and only on fully trusted networks. AutomaticBackup (optional) Enable or disable the client from connecting automatically to the store every UpdateStoreInterval seconds. When enabled (set to Yes), the client is in Lazy Mode. When disabled (set to No), it is in Snapshot Mode. This setting is optional, and the default value is Yes. UpdateStoreInterval (required) The approximate time between successive connections to the server, in seconds, when the client is in Lazy Mode. The actual time is randomised slightly to prevent "rush hour" traffic jams on the server, where many clients try to connect at the same time. This value is ignored if the client is in Snapshot Mode. However, it is still required. It can be set to zero in this case. You will probably need to experiment with the value of this option. A good value to start with is probably 86400 seconds, which is one day. MinimumFileAge (required) The number of seconds since a file was last modified before it will be backed up. The reason for this is to avoid repeatedly backing up files which are repeatedly changing. A good value is about 3600 seconds (one hour). If set to zero, files which have changed will always be backed up on the next backup run. The MaxUploadWait option overrides this option in some circumstances. MaxUploadWait (required) The number of seconds since a file was last uploaded before it will be uploaded again, even if it keeps changing. The reason for this is to ensure that files which are continuously modified are eventually uploaded anyway. This should be no less than the value of MinimumFileAge. A good value is about 14400 seconds (4 hours). MaxFileTimeInFuture (optional) The maximum time that a file's timestamp can be in the future, before it will be backed up anyway. Due to clock synchronisation problems, it is inevitable that you will occasionally see files timestamped in the future. Normally, for files which are dated only slightly in the future, you will want to wait until after the file's date before backing it up. However, for files whose dates are very wrong (more than a few hours) you will normally prefer to back them up immediately. A good value is about 7200 seconds (2 hours) to cope with potential problems when moving in and out of daylight saving time, if applicable in your timezone. The default value, if this setting is not provided, is 172800 seconds (2 days). FileTrackingSizeThreshold (required) The minimum size of files which will be tracked by inode number to detect renames. It is not worth detecting renames of small files, since they are quick to upload again in full, and keeping their inode numbers in memory increases the client's memory usage and slows down searches. Larger files should be tracked to avoid wasting space on the store and long uploads. A good value is about 65536 bytes (64 kilobytes). DiffingUploadSizeThreshold (required) The minimum size of files which will be compared to the old file on the server, and for which only changes will be uploaded. It is not worth comparing small files, since they are quick to upload again in full, and sending the entire file reduces the risk of data loss if the store is accidentally corrupted. Larger files should have only their differences uploaded to avoid wasting space on the store and long uploads. A good value is about 65536 bytes (64 kilobytes). MaximumDiffingTime (optional) The maximum time for which the client will attempt to find differences between the current version and the old version in the store, before giving up and uploading the entire file again. Very large files (several gigabytes) may take a very long time to scan for changes, but would also take a very long time to upload again and use a lot of space on the store, so it is normally worth omitting this value. Use this option only if, for some bizarre reason, you prefer to upload really large files in full rather than spend a long time scanning them for changes. KeepAliveTime (optional) The interval (in seconds) between sending Keep-Alive messages to the server while performing long operations such as finding differences in large files, or scanning large directories. These messages ensure that the SSL connection is not closed by the server, or an intervening firewall, due to lack of activity. The server will normally wait up to 15 minutes (900 seconds) before disconnecting the client, so the value should be given and should be less than 900. Some firewalls may time out inactive connections after 10 or 5 minutes. A good value is 300 seconds (5 minutes). You may need to reduce this if you frequently see TLSReadFailed or TLSWriteFailed errors on the client. StoreObjectInfoFile (optional) Enables the use of a state file, which stores the client's internal state when the client is not running. This is useful on clients machines which are frequently shut down, for example desktop and laptop computers, because it removes the need for the client to recontact the store and rescan all directories on the first backup run, which may take some time. This feature is somewhat experimental and not well tested. This is option is disabled by default, in which case the state is stored in memory only. The value is the path to the state file. ExtendedLogging (optional) Enables the connection debugging mode of the client, which writes all commands sent to or received from the server to the system logs. This generates a lot of output, so it should only be used when instructed, or when you suspect a connection problem or client-server protocol error (and you know how to interpret the output). This is a boolean value, which may be set to Yes or No. The default is of course No. ExtendedLogFile (optional, new in 0.11) Enables the same debugging output as ExtendedLogging, but written to a file instead of the system logs. This is useful if you need extended logging, but do not have access to the system logs, for example if you are not the administrator of the computer. The value is the path to the file where these logs will be written. If omitted, extended logs will not be written to a file. This is entirely independent of the ExtendedLogging option. It does not make much sense to use both at the same time. LogAllFileAccess (optional, new in 0.11) Enables logging of all local file and directory access, file uploads (full and differential), and excluded files. This may be useful if the client is failing to upload a particular file, or crashing while trying to upload it. The logs will be sent to the system log or Windows Event Viewer. This is a boolean value, which may be set to Yes or No. The default is of course No. SyncAllowScript (optional) The path to the script or command to run when the client is about to start an automatic backup run, and wishes to know whether it is safe to do so. This is useful for clients which do not always have access to the server, for example laptops and computers on dial-up Internet connections. The script should either output the word now if the backup should proceed, or else a number, in seconds, which indicates how long the client should wait before trying to connect again. Any other output will result in an error on the client, and the backup will not run. This value is optional, and by default no such script is used.

These options appear within the Server subsection, which is at the root level. PidFile This option enables the client to write its processs identifier (PID) to the specified file after starting. The file will be deleted when the client daemon exits for any reason. This is disabled by default, but is recommended whenever you run the client daemon as a daemon (in the background), which is usually the case. This file can be used by scripts to determine whether the daemon is still running, and to send it messages to reload its configuration or to terminate. Server { PidFile = /var/state/boxbackup/bbackupd.pid }

This section serves only as a container for all defined backup locations. BackupLocations { etc { Path = /etc } home { Path = /home ExcludeDir = /home/shared ExcludeDir = /home/chris/.ccache ExcludeDir = /home/chris/.mozilla/firefox/vvvkq3vp.default/Cache } } Each subsection is a backup location. The name of the subsection is the name that will be used on the server. The root directory of the account on the server contains one subdirectory per location. The name should be simple, not containing any spaces or special characters. If you do not define any locations, the client will not back up any files! It is currently not recommended to back up the root directory of the filesystem on Unix. Box Backup is designed to back up important data and configuration files, not full systems. Nevertheless, nothing prevents you from doing so if you desire. On Windows, it is currently not possible to back up files which are open (currently in use), such as open documents in Microsoft Office, and system files such as the registry and the paging file. You will get an error for each open file which the client attempts to back up. Once the file has been closed, it will be backed up normally. System files will always be open, and should be excluded from your backups.

This chapter deals with the dauily running and management of the Box Backup system. It explains most day-to-day tasks.

The steps involved in maintaining and keeping the backup sets healthy are outlined in this section.

The bbackupctl program sends control commands to the bbackupd daemon. It must be run as the same user as the daemon, and there is no exception for root. The command line syntax is: /usr/local/bin/bbackupctl [-q] [-c config-file] command The -q option reduces the amount of output the program emits, and -c allows an alternative configuration file to be specified. Valid commands are: terminate Stop the bbackupd daemon now (equivalent to kill) reload Reload the configuration file (equivalent to kill -HUP) sync Connect to the server and synchronise files now bbackupctl communicates with the server via a UNIX domain socket, specified in bbackupd.conf with the CommandSocket directive. This does not need to be specified, and bbackupd will run without the command socket, but in this case bbackupctl will not be able to communicate with the daemon. Some platforms cannot check the user id of the connecting process, so this command socket becomes a denial of service security risk. bbackupd will warn you when it starts up if this is the case on your platform, and you should consider removing the CommandSocket directive on these platforms.

bbackupctl's main purpose is to implement snapshot based backups, emulating the behaviour of traditional backup software. Use bbackupd-config to write a configuration file in snapshot mode, and then run the following command as a cron job. /usr/local/bin/bbackupctl -q sync This will cause the backup daemon to upload all changed files immediately. bbackupctl will exit almost immediately, and will not output anything unless there is an error.

bbackupquery can tell you how much space is used on the server for this account. Either use the usage command in interactive mode, or type: /usr/local/bin/bbackupquery -q usage quit to show the space used as a single command.

bbstoreaccounts allows you to query the space used, and change the limits. To display the space used on the server for an account, use: /usr/local/bin/bbstoreaccounts info 75AB23C To adjust the soft and hard limits on an account, use: /usr/local/bin/bbstoreaccounts setlimit 75AB23C new-soft-limit new-hard-limit You do not need to restart the server.

Backups are no use unless you can restore them. The bbackupquery utility does this and more. You don't provide any login information to it, as it just picks up the data it needs from /etc/box/bbackupd.conf. You should run it as root so it can find everything it needs. Full documentation can be found in the bbackupquery manual page. It follows the model of a command line sftp client quite closely. TODO: Link to bbackupquery man-page here. On systems where GNU readline is available (by default) it uses that for command line history and editing. Otherwise it falls back to very basic UNIX text entry. TODO: Did the readline dependency change to editline?

bbackupquery is the tool you use to verify, restore and investigate your backup files with. When invoked, it simply logs into the server using the certificates you have listed in bbackupd.conf. After you run bbackupquery, you will see a prompt, allowing you to execute commands. The list (or ls) command lets you view files in the store. It works much like unix ls, but with different options. An example: [pthomsen@host bbackupquery]$ bbackupquery Box Backup Query Tool v0.10, (c) Ben Summers and contributors 2003-2006 Using configuration file /etc/box/bbackupd.conf Connecting to store... Handshake with store... Login to store... Login complete. Type "help" for a list of commands. query > ls 00000002 -d---- mp3 00000003 -d---- video 00000004 -d---- home-pthomsen 00000005 -d---- root query > The ls commands shows the directories that are backed up. Now we'll take a closer look at the home-pthomsen directory: query > cd home-pthomsen query > ls 00002809 f----- sample.tiff 0000280a f----- s3.tiff 0000280b f----- s4.tiff 0000280d f----- s2.tiff 0000280e f----- foo.pdf 0000286c f----- core.28720 0000339a -d---- .emacs.d 0000339d -d---- bbackup-contrib 00003437 f----- calnut.compare.txt 0000345d f----- DSCN1783.jpg 0000345e f----- DSCN1782.jpg query > The ls command takes the following options; -r -- recursively list all files -d -- list deleted files/directories -o -- list old versions of files/directories -I -- don't display object ID -F -- don't display flags -t -- show file modification time (and attr mod time if has the object has attributes, ~ separated) -s -- show file size in blocks used on server (only very approximate indication of size locally) The flags displayed from the ls command are as follows: f = file d = directory X = deleted o = old version R = remove from server as soon as marked deleted or old a = has attributes stored in directory record which override attributes in backup file

As with any backup system, you should frequently check that your backups are working properly by comparing them. Box Backup makes this very easy and completely automatic. All you have to do is schedule the bbackupquery compare command to run regularly, and check its output. You can run the command manually as follows: /usr/local/bin/bbackupquery "compare -a" quit This command will report all the differences found between the store and the files on disc. It will download everything, so may take a while. You should expect to see some differences on a typical compare, because files which have recently changed are unlikely to have been uploaded yet. It will also tell you how many files have been modified since the last backup run, since these will normally have changed, and such failures are expected. You are strongly recommended to add this command as a cron job, at least once a month, and to check the output for anything suspicious, particularly a large number of compare failures, failures on files that have not been modified, or any error (anything except a compare mismatch) that occurs during the compare operation. Consider keeping a record of these messages and comparing them with a future verification. If you would like to do a "quick" check which just downloads file checksums and compares against that, then run: /usr/local/bin/bbackupquery "compare -aq" quit However, this does not check that the file attributes are correct, and since the checksums are generated on the client they may not reflect the data on the server if there is a problem -- the server cannot check the encrypted contents. View this as a quick indication, rather than a definite check that your backup verifies correctly.

You will need the keys file created when you configured the server. Without it, you cannot restore the files; this is the downside of encrypted backups. However, by keeping the small keys file safe, you indirectly keep your entire backup safe. The first step is to recreate the configuration of the backup client. It's probably best to have stored the /etc/box directory with your keys. But if you're recreating it, all you really need is to have got the login infomation correct (ie the certs and keys). Don't run bbackupd yet! It will mark all your files as deleted if you do, which is not hugely bad in terms of losing data, just a major inconvenience. (This assumes that you are working from a blank slate. If you want to restore some files to a different location, it's fine to restore while bbackupd is running, just do it outside a backed up directory to make sure it doesn't start uploading the restored files.) Type: /usr/local/bin/bbackupquery to run it in interactive mode. Type: list to see a list of the locations stored on the server. For each location you want to restore, type: restore name-on-server local-dir-name The directory specified by local-dir-name must not exist yet. If the restore is interrupted for any reason, repeat the above steps, but add the -r flag to the restore command to tell it to resume.

Box Backup makes old versions of files and files you have deleted available, subject to there being enough disc space on the server to hold them. This is how to retrieve them using bbackupquery. Future versions will make this far more user-friendly. Firstly, run bbackupquery in interactive mode. It behaves in a similar manner to a command line sftp client. /usr/local/bin/bbackupquery Then navigate to the directory containing the file you want, using list, cd and pwd. query > cd home/profiles/USERNAME List the directory, using the "o" option to list the files available without filtering out everything apart from the current version. (if you want to see deleted files as well, use list -odt) query > list -ot 00000078 f--o- 2004-01-21T20:17:48 NTUSER.DAT 00000079 f--o- 2004-01-21T20:17:48 ntuser.dat.LOG 0000007a f--o- 2004-01-21T17:55:12 ntuser.ini 0000007b f---- 2004-01-12T15:32:00 ntuser.pol 0000007c -d--- 1970-01-01T00:00:00 Templates 00000089 -d--- 1970-01-01T00:00:00 Start Menu 000000a0 -d--- 1970-01-01T00:00:00 SendTo 000000a6 -d--- 1970-01-01T00:00:00 Recent 00000151 -d--- 1970-01-01T00:00:00 PrintHood 00000152 -d--- 1970-01-01T00:00:00 NetHood 00000156 -d--- 1970-01-01T00:00:00 My Documents 0000018d -d--- 1970-01-01T00:00:00 Favorites 00000215 -d--- 1970-01-01T00:00:00 Desktop 00000219 -d--- 1970-01-01T00:00:00 Cookies 0000048b -d--- 1970-01-01T00:00:00 Application Data 000005da -d--- 1970-01-01T00:00:00 UserData 0000437e f--o- 2004-01-24T02:45:43 NTUSER.DAT 0000437f f--o- 2004-01-24T02:45:43 ntuser.dat.LOG 00004380 f--o- 2004-01-23T17:01:29 ntuser.ini 00004446 f--o- 2004-01-24T02:45:43 NTUSER.DAT 00004447 f--o- 2004-01-24T02:45:43 ntuser.dat.LOG 000045f4 f---- 2004-01-26T15:54:16 NTUSER.DAT 000045f5 f---- 2004-01-26T15:54:16 ntuser.dat.LOG 000045f6 f---- 2004-01-26T16:54:31 ntuser.ini (this is a listing from a server which is used as a Samba server for a network of Windows clients.) You now need to fetch the file using it's ID, rather than it's name. The ID is the hex number in the first column. Fetch it like this: query > get -i 0000437e NTUSER.DAT Object ID 0000437e fetched successfully. The object is now available on your local machine. You can use lcd to move around, and sh ls to list directories on your local machine.

This section gives help on what to do if your server has suffered corruption, for example, after an unclean shutdown or other operating system or hardware problem. In general, as updates to the store are made in an atomic manner, the most likely result is wasted disc space. However, if really bad things happen, or you believe that there is a lot of wasted space, then these instructions will help to restore your data. You know you will need to do something if you get strange errors, and bbackupd attempts to contact the server every 100 seconds or so. Or if one of the discs in your RAID disc set has failed. After following these instructions, the end result will be that bbackupquery will be able to see all the files which were stored on your server, and retrieve them. Some of them may be in lost+found directories in the root of the store (or in their original position if they have been moved) but they will all be able to be retrieved. After you have retrieved the files you want, bbackupd will upload new versions where necessary, and after about two days, mark any lost+found directories as deleted. Finally, those directories will be removed by the housekeeping process on the server. These instructions assume you're working on account 1234. Replace this with the account number that you actually want to check (the one that is experiencing errors). These steps will need to be repeated for all affected accounts.

First, make sure that bbackupd is not running on the client machine for the account you are going to recover. Use bbackupctl terminate to stop it. This step is not strictly necessary, but is recommended. During any checks on the account, bbackupd will be unable to log in, and after they are complete, the account is marked as changed on the server so bbackupd will perform a complete scan.

The raidfile recovery tools have not been written, and probably will not be, since Box Backup RAID is deprecated. However, when two out of three files are available, the server will successfully allow access to your data, even if it complains a lot in the logs. The best thing to do is to fix the accounts, if necessary, and retrieve any files you need. Then move the old store directories aside (in case you need them) and start afresh with new accounts, and let the clients upload all their data again.

First, run the check utility, and see what errors it reports. /usr/local/bin/bbstoreaccounts check 1234 This will take some time, and use a fair bit of memory (about 16 bytes per file and directory). If the output looks plausible and reports errors which need fixing, run it again but with the fix flag: /usr/local/bin/bbstoreaccounts check 1234 fix This will fix any errors, and remove unrecoverable files. Directories will be recreated if necessary. NOTE: The utility may adjust the soft and hard limits on the account to make sure that housekeeping will not remove anything -- check these afterwards.

At this point, you will have a working store. Every file which was on the server, and wasn't corrupt, will be available. On the client, use bbackupquery to log in and examine the store. (type help at the prompt for instructions). Retrieve any files you need, paying attention to any lost+found directories in the root directory of the store. You can skip this step if you are sure that the client machine is fine -- in this case, bbackupd will bring the store up to date.

Restart bbackupd on the client machine. The store account will be brought up to date, and files in the wrong place will be marked for eventual deletion.

If you are trying to fix a store after your disc has been corrupted, see Fixing corruptions of store data. Unfortunately, the error messages are not particularly helpful at the moment. This page lists some of the common errors, and the most likely causes of them. When an error occurs, you will see a message like 'Exception: RaidFile/OSFileError (2/8)' either on the screen or in your log files. (it is recommended you set up another log file as recommended in the server setup instructions.) This error may not be particularly helpful, although some do have extra information about probable causes. To get further information, check the ExceptionCodes.txt file in the root of the distribution. This file is generated by the ./configure script, so you will need to have run that first. Some common causes of exceptions are listed below. Please email me with any other codes you get, and I will let you know what they mean, and add notes here.

This is found either when running bbstoreaccounts or in the bbstored logs. Problem: The directories you specified in the raidfile.conf are not writable by the _bbstored user. Resolution: Change permissions appropriately.

This usually occurs when the configuration files can't be opened. Problem: You created your configurations in non-standard locations, and the programs cannot find them. Resolution: Explicitly specify configuration file locations to daemons and programs. For example /usr/local/bin/bbstored /some/other/dir/bbstored.config /usr/local/bin/bbackupquery -c /some/other/dir/bbackupd.config (daemons specify the name as the first argument, utility programs with the -c option). Problem: bbstored can't find the raidfile.conf file specified in bbstored.conf. Resolution: Edit bbstored.conf to point to the correct location of this additional configuration file.

The server can't listen for connections on the IP address specified when you configured it. Problem: This probably means you've specified the wrong hostname to bbstored-config -- maybe your server is behind a NAT firewall? Resolution: Edit bbstored.conf and correct the ListenAddresses line. You should replace the server address with the IP address of your machine.

These errors all relate to connections failing -- you may see them during operation if there are network failures or other problems between the client and server. The backup system will recover from them automatically.

Log snippet from client side: bbackupd[1904]: Opening connection to server xxxx.xxx... bbackupd[1904]: SSL err during Connect: error:xxxxxxxx:rsa routines:RSA_padding_check_PKCS1_type_1:block type is not 01 bbackupd[1904]: SSL err during Connect: error:xxxxxxxx:rsa routines:RSA_EAY_PUBLIC_DECRYPT:padding check failed bbackupd[1904]: SSL err during Connect: error:xxxxxxxx:asn1 encoding routines:ASN1_verify:EVP lib bbackupd[1904]: SSL err during Connect: error:xxxxxxxx:SSL routines:SSL3_GET_SERVER_CERTIFICATE:certificate verify failed bbackupd[1904]: TRACE: Exception thrown: ConnectionException(Conn_TLSHandshakeFailed) at SocketStreamTLS.cpp(237) bbackupd[1904]: Exception caught (7/30), reset state and waiting to retry... And from the server: bbstored[19291]: Incoming connection from xx.xxx.xx.xxx port xxxxx (handling in child xxxxx) bbstored[21588]: SSL err during Accept: error:xxxxxxxx:SSL routines:SSL3_READ_BYTES:tlsv1 alert decrypt error bbstored[21588]: in server child, exception Connection TLSHandshakeFailed (7/30) -- terminating child Solution: Create a new CA on the server side and re-generate the client certificate. Re-creating the client certificate request is not necessary.

If this really doesn't help, then using the DEBUG builds of the system will give you much more information -- a more descriptive exception message and the file and line number where the error occurred. For example, if you are having problems with bbstoreaccounts, build the debug version with: cd boxbackup-0.0 cd bin/bbstoreaccounts make Within the module directories, make defaults to building the debug version. At the top level, it defaults to release. This will build an executable in debug/bin/bbstoreaccounts which you can then use instead of the release version. It will give far more useful error messages. When you get an error message, use the file and line number to locate where the error occurs in the code. There will be comments around that line to explain why the exception happened. If you are using a debug version of a daemon, these extended messages are found in the log files.

&__ExceptionCodes__elfjz3fu; It is possible to run both the server and client without root privileges.

The server, by default, runs as a non-root user. However, it expects to be run as root and changes user to a specified user as soon as it can, simply for administrative convenience. The server uses a port greater than 1024, so it doesn't need root to start. To run it entirely as a non-root user, edit the bbstored.conf file, and remove the User directive from the Server section. Then simply run the server as your desired user.

The client requires root for normal operation, since it must be able to access all files to back them up. However, it is possible to run the client as a non-root user, with certain limitations. Follow the installation instructions, but install the executable files manually to somewhere in your home directory. Then use bbackupd-config to configure the daemon, but use a directory other than /etc/box, probably somewhere in your home directory. All directories you specify to be backed up must be readable, and all files must be owned by the user and readable to that user. Important: If any file or directory is not readable by this user, the backup process will skip that file or directory. Keep an eye on the logs for reports of this failure. Non-root operation of the backup client is recommended only for testing, and should not be relied on in a production environment.