diff options
Diffstat (limited to 'docs')
41 files changed, 0 insertions, 1698 deletions
diff --git a/docs/backup/INDEX.txt b/docs/backup/INDEX.txt deleted file mode 100644 index 50406cac..00000000 --- a/docs/backup/INDEX.txt +++ /dev/null @@ -1,61 +0,0 @@ -TITLE Programmers Notes for Box Backup - -This directory contains the programmers notes for the Box Backup system. They will be of interest only if you want to review or modify the code. - -These notes are intended to be run through a script to produce HTML at some later stage, hence the marks such as 'TITLE'. - - -SUBTITLE Organisation - -The project is split up into several modules. The modules within 'lib' are building blocks for creation of the actual executable programs within 'bin'. 'test' contains unit tests for lib and bin modules. - -The file modules.txt lists the modules which are to be built, and their dependencies. It also allows for platform differences. - - -SUBTITLE Documentation Organisation - -In this directory, the files correspond to modules or areas of interest. Sub directories of the same name contain files documenting specific classes within that module. - - -SUBTITLE Suggested reading order - -* lib_common -* lib_server -* bin_bbackupd -* backup_encryption.txt -* bin_bstored -* lib_raidfile - -and refer to other sections as required. - - -SUBTITLE Building - -The makefiles are generated by makebuildenv.pl. (The top level makefile is generated by makeparcels.pl, but this is for the end user to use, not a programmer.) - -To build a module, cd to it and type make. If the -DRELEASE option is specified (RELEASE=1 with GNU make) the release version will be built. The object files and exes are placed in a directory structure under 'release' or 'debug'. - -It is intended that a test will be written for everything, so in general make commands will be issued only within the test/* modules. Once it has been built, cd to debug/test/<testname> and run the test with ./t . - - -SUBTITLE Programming style - -The code is written to be easy to write. Ease of programming is the primary concern, as this should lead to fewer bugs. Efficiency improvements can be made later when the system as a whole works. - -Much use is made of the STL. - -There is no common base class. - -All errors are reported using exceptions. - -Some of the boring code is generated by perl scripts from description files. - -There are a lot of modules and classes which can easily be used to build other projects in the future -- there is a lot of "framework" code. - - -SUBTITLE Lots more documentation - -The files are extensively commented. Consider this notes as an overview, and then read the source files for detailed and definitive information. - -Each function and class has a very brief decsription of it's purpose in a standard header, and extensive efforts have been maed to comment the code itself. - diff --git a/docs/backup/Win32_Clients.txt b/docs/backup/Win32_Clients.txt deleted file mode 100644 index fad6c4af..00000000 --- a/docs/backup/Win32_Clients.txt +++ /dev/null @@ -1,13 +0,0 @@ -The basic client tools now run on Win32 natively. -The port was done by nick@omniis.com. - -* bbackupd -* bbackupquery -* bbackupctl - -Have been ported. bbackupd runs as a NT style service. - -Known limitations: - -* File attributes and permissions are not backed up. - diff --git a/docs/backup/backup_encryption.txt b/docs/backup/backup_encryption.txt deleted file mode 100644 index 36580581..00000000 --- a/docs/backup/backup_encryption.txt +++ /dev/null @@ -1,109 +0,0 @@ -TITLE Encryption in the backup system - -This document explains how everything is encrypted in the backup system, and points to the various functions which need reviewing to ensure they do actually follow this scheme. - - -SUBTITLE Security objectives - -The crpyto system is designed to keep the following things secret from an attacker who has full access to the server. - -* The names of the files and directories -* The contents of files and directories -* The exact size of files - -Things which are not secret are - -* Directory heirarchy and number of files in each directory -* How the files change over time -* Approximate size of files - - -SUBTITLE Keys - -There are four separate keys used: - -* Filename -* File attributes -* File block index -* File data - -and an additional secret for file attribute hashes. - -The Cipher is Blowfish in CBC mode in most cases, except for the file data. All keys are maximum length 448 bit keys, since the key size only affects the setup time and this is done very infrequently. - -The file data is encrypted with AES in CBC mode, with a 256 bit key (max length). Blowfish is used elsewhere because the larger block size of AES, while more secure, would be terribly space inefficient. Note that Blowfish may also be used when older versions of OpenSSL are in use, and for backwards compatibility with older versions. - -The keys are generated using "openssl rand", and a 1k file of key material is stored in /etc/box/bbackupd. The configuration scripts make this readable only by root. - -Code for review: BackupClientCryptoKeys_Setup() -in lib/backupclient/BackupClientCryptoKeys.cpp - - -SUBTITLE Filenames - -Filenames need to be secret from the attacker, but they need to be compared on the server so it can determine whether or not is it a new version of an old file. - -So, the same Initialisation Vector is used for every single filename, so the same filename encrypted twice will have the same binary representation. - -Filenames use standard PKCS padding implemented by OpenSSL. They are proceeded by two bytes of header which describe the length, and the encoding. - -Code for review: BackupStoreFilenameClear::EncryptClear() -in lib/backupclient/BackupStoreFilenameClear.cpp - - -SUBTITLE File attributes - -These are kept secret as well, since they reveal information. Especially as they contain the target name of symbolic links. - -To encrypt, a random Initialisation Vector is choosen. This is stored first, followed by the attribute data encrypted with PKCS padding. - -Code for review: BackupClientFileAttributes::EncryptAttr() -in lib/backupclient/BackupClientFileAttributes.cpp - - -SUBTITLE File attribute hashes - -To detect and update file attributes efficiently, the file status change time is not used, as this would give suprious results and result in unnecessary updates to the server. Instead, a hash of user id, group id, and mode is used. - -To avoid revealing details about attributes - -1) The filename is added to the hash, so that an attacker cannot determine whether or not two files have identical attributes - -2) A secret is added to the hash, so that an attacker cannot compare attributes between accounts. - -The hash used is the first 64 bits of an MD5 hash. - - -SUBTITLE File block index - -Files are encoded in blocks, so that the rsync algorithm can be used on them. The data is compressed first before encryption. These small blocks don't give the best possible compression, but there is no alternative because the server can't see their contents. - -The file contains a number of blocks, which contain among other things - -* Size of the block when it's not compressed -* MD5 checksum of the block -* RollingChecksum of the block - -We don't want the attacker to know the size, so the first is bad. (Because of compression and padding, there's uncertainty on the size.) - -When the block is only a few bytes long, the latter two reveal it's contents with only a moderate amount of work. So these need to be encrypted. - -In the header of the index, a 64 bit number is chosen. The sensitive parts of the block are then encrypted, without padding, with an Initialisation Vector of this 64 bit number + the block index. - -If a block from an previous file is included in a new version of a file, the same checksum data will be encrypted again, but with a different IV. An eavesdropper will be able to easily find out which data has been re-encrypted, but the plaintext is not revealed. - -Code for review: BackupStoreFileEncodeStream::Read() (IV base choosen about half-way through) -BackupStoreFileEncodeStream::EncodeCurrentBlock() (encrypt index entry) -in lib/backupclient/BackupStoreFileEncodeStream.cpp - - -SUBTITLE File data - -As above, the first is split into chunks and compressed. - -Then, a random initialisation vector is chosen, stored first, followed by the compressed file data encrypted using PKCS padding. - -Code for review: BackupStoreFileEncodeStream::EncodeCurrentBlock() -in lib/backupclient/BackupStoreFileEncodeStream.cpp - - diff --git a/docs/backup/bin_bbackupd.txt b/docs/backup/bin_bbackupd.txt deleted file mode 100644 index 67ad9267..00000000 --- a/docs/backup/bin_bbackupd.txt +++ /dev/null @@ -1,88 +0,0 @@ -TITLE bin/bbackupd - -The backup client daemon. - -This aims to maintain as little information as possible to record which files have been uploaded to the server, while minimising the amount of queries which have to be made to the server. - - -SUBTITLE Scanning - -The daemon is given a length of time, t, which files over this age should be uploaded to the server. This is to stop recently updated files being uploaded immediately to avoid uploading something repeatedly (on the assumption that if a file has been written, it is likely to be modified again shortly). - -It will scan the files at a configured interval, and connect to the server if it needs to upload files or make queries about files and directories. - -The scan interval is actually varied slightly between each run by adding a random number up to a 64th of the configured time. This is to reduce cyclic patterns of load on the backup servers -- otherwise if all the boxes are turned on at about 9am, every morning at 9am there will be a huge spike in load on the server. - -Each scan chooses a time interval, which ends at the current time - t. This will be from 0 to current time - t on the first run, then the next run takes the start time as the end time of the previous run. The scan is only performed if the difference between the start and end times is greater or equal to t. - -For each configured location, the client scans the directories on disc recursively. - -For each directory - -* If the directory has never been scanned before (in this invocation of the daemon) or the modified time on the directory is not that recorded, the listing on the server is downloaded. - -* For each file, if it's modified time is within the time period, it is uploaded. If the directory has been downloaded, it is compared against that, and only uploaded if it's changed. - -* Find all the new files, and upload them if they lie within the time interval. - -* Recurse to sub directories, creating them on the server if necessary. - -Hence, the first time it runs, it will download and compare the entries on the disc to those on the server, but in future runs it will use the file and directory modification times to work out if there is anything which needs uploading. - -If there aren't any changes, it won't even need to connect to the server. - -There are some extra details which allow this to work reliably, but they are documented in the source. - - -SUBTITLE File attributes - -The backup client will update the file attributes on files as soon as it notices they are changed. It records most of the details from stat(), but only a few can be restored. Attributes will only be considered changed if the user id, group id or mode is changed. Detection is by a 64 bit hash, so detection is strictly speaking probablistic. - - -SUBTITLE Encryption - -All the user data is encrypted. There is a separate file, backup_encryption.txt which describes this, and where in the code to look to verify it works as described. - - -SUBTITLE Tracking files and directories - -Renaming files is a difficult problem under this minimal data scanning scheme, because you don't really know whether a file has been renamed, or another file deleted and new one created. - -The solution is to keep (on disc) a map of inode numbers to server object IDs for all directories and files over a certain user configurable threshold. Then, when a new file is discovered, it is first checked to see if it's in this map. If so, a rename is considered, which will take place if the local object corresponding to the name of the tracked object doesn't exist any more. - -Because of the renaming requirement, deletions of objects from the server are recorded and delayed until the end of the scan. - - -SUBTITLE Running out of space - -If the store server indicates on login to the backup client, it will scan, but not upload anything nor adjust it's internal stored details of the local objects. However, deletions and renames happen. - -This is to allow deletions to still work and reduce the amount of storage space used on the server, in the hope that in the future there will be enough space. - -Just not doing anything would mean that one big file created and then deleted at the wrong time would stall the whole backup process. - - -SUBTITLE BackupDaemon - -This is the daemon class for the backup daemon. It handles setting up of all the objects, and implements calulcation of the time intervals for the scanning. - - -SUBTITLE BackupClientContext - -State information for the scans, including maintaining a connection to the store server if required. - - -SUBTITLE BackupClientDirectoryRecord - -A record of state of a directory on the local filesystem. Containing the recursive scanning function, which is long and entertaining, but very necessary. It contains lots of comments which explain the exact details of what's going on. - - -SUBTITLE BackupClientInodeToIDMap - -A implementation of a map of inode number to object ID on the server. If Berkeley DB is available on the platform, it is stored on disc, otherwise there is an in memory version which isn't so good. - - - - - - diff --git a/docs/backup/bin_bbstored.txt b/docs/backup/bin_bbstored.txt deleted file mode 100644 index c9c4e229..00000000 --- a/docs/backup/bin_bbstored.txt +++ /dev/null @@ -1,54 +0,0 @@ -TITLE bin/bbstored - -The backup store daemon. - -Maintains a store of encrypted files, and every so often goes through deleting unnecessary data. - -Uses an implementation of Protocol to communicate with the backup client daemon. See bin/bbstored/backupprotocol.txt for details. - - -SUBTITLE Data storage - -The data is arranged as a set of objects within a RaidFile disc set. Each object has a 64 bit object ID, which is turned into a filename in a mildly complex manner which ensure that directories don't have too many objects in them, but there is a minimal number of nested directories. See StoreStructure::MakeObjectFilename in lib/backupstore/StoreStructure.cpp for more details. - -An object can be a directory or a file. Directories contain files and other directories. - -Files in directories are supersceded by new versions when uploaded, but the old versions are flagged as such. A new version has a different object ID to the old version. - -Every so often, a housekeeping process works out what can be deleted, and deletes unnecessary files to take them below the storage limits set in the store info file. - - -SUBTITLE Note about file storage and downloading - -There's one slight entertainment to file storage, in that the format of the file streamed depends on whether it's being downloaded or uploaded. - -The problem is that it contains an index of all the blocks. For efficiency in managing these blocks, they all need to be in the same place. - -Files are encoded and decoded as they are streamed to and from the server. With encoding, the index is only completely known at the end of the process, so it's sent last, and lives in the filesystem last. - -When it's downloaded, it can't be decoded without knowing the index. So the index is sent first, followed by the data. - - -SUBTITLE BackupContext - -The context of the current connection, and the object which modifies the store. - -Maintains a cache of directories, to avoid reading them continuously, and keeps a track of a BackupStoreInfo object which is written back periodiocally. - - -SUBTITLE BackupStoreDaemon - -A ServerTLS daemon which forks off a separate housekeeping process as it starts up. - -Handling connections is delegated to a Protocol implementation. - - -SUBTITLE BackupCommands.cpp - -Implementation of all the commands. Work which requires writing is handled in the context, read only commands mainly in this file. - - -SUBTITLE HousekeepStoreAccount - -A class which performs housekeeping on a single account. - diff --git a/docs/backup/encrypt_rsync.txt b/docs/backup/encrypt_rsync.txt deleted file mode 100644 index a5db2df2..00000000 --- a/docs/backup/encrypt_rsync.txt +++ /dev/null @@ -1,66 +0,0 @@ -TITLE Encrypted rsync algorithm - -The backup system uses a modified version of the rsync algorithm. A description of the plain algorithm can be found here: - - http://samba.anu.edu.au/rsync/tech_report/ - -The algorithm is modified to allow the server side to be encrypted, yet still benefit from the reduced bandwidth usage. For a single file transfer, the result will be only slightly less efficient than plain rsync. For a backup of a large directory, the overall bandwidth may be less due to the way the backup client daemon detects changes. - -This document assumes you have read the rsync document. - -The code is in lib/backupclient/BackupStoreFile*.*. - - -SUBTITLE Blocks - -Each file is broken up into small blocks. These are individually compressed and encrypted, and have an entry in an index which contains, encrypted, it's weak and strong checksums and decoded plaintext size. This is all done on the client. - -Why not just encrypt the file, and use the standard rsync algorithm? - -1) Compression cannot be used, since encryption turns the file into essentially random data. This is not very compressible. - -2) Any modification to the file will result in all data after that in the file having different ciphertext (in any cipher mode we might want to use). Therefore the rsync algorithm will only be able to detect "same" blocks up until the first modification. This significantly reduces the effectiveness of the process. - -Note that blocks are not all the same size. The last block in the file is unlikely to be a full block, and if data is inserted which is not a integral multiple of the block size, odd sized blocks need to be created. This is because the server cannot reassemble the blocks, because the contents are opaque to the server. - - -SUBTITLE Modifed algorithm - -To produce a list of the changes to send the new version, the client requests the block index of the file. This is the same step as requesting the weak and strong checksums from the remote side with rsync. - -The client then decrypts the index, and builds a list of the 8 most used block sizes above a certain threshold size. - -The new version of the file is then scanned in exactly the same way as rsync for these 8 block sizes. If a block is found, then it is added to a list of found blocks, sorted by position in the file. If a block has already been found at that position, then the old entry is only replaced by the new entry if the new entry is a "better" (bigger) match. - -The block size covering the biggest file area is searched first, so that most of the file can be skipped over after the first pass without expensive checksumming. - -A "recipe" is then built from the found list, by trivially discarding overlapping blocks. Each entry consists of a number of bytes of "new" data, a block start number, and a number of blocks from the old file. The data is stored like this as a memory optimisation, assuming that files mostly stay the same rather than having all their blocks reordered. - -The file is then encoded, with new data being sent as blocks of data, and references to blocks in the old file. The new index is built completely, as the checksums and size need to be rencrypted to match their position in the index. - - -SUBTITLE Combination on server - -The "diff" which is sent from the client is assembled into a full file on the server, simply by adding in blocks from the old file where they are specified in the block index. - - -SUBTITLE Storage on server - -Given that the server will in general store several versions of a file, combining old and new files to form a new file is not terribly efficient on storage space. Particularly for large multi-Gb database files. - -An alternative scheme is outlined below, however, it is significantly more complex to implement, and so is not implemented in this version. - -1) In the block index of the files, store the file ID of the file which each block is source from. This allows a single file to reference blocks from many files. - -2) When the file is downloaded, the server combines the blocks from all the files into a new file as it is streamed to the client. (This is not particuarly complicated to do.) - -This all sounds fine, until housekeeping is considered. Old versions need to be deleted, without losing any blocks necessary for future versions. - -Instead of just deleting a file, the server works out which blocks are still required, and rebuilds the file omitting those blocks which aren't required. - -This complicates working out how much space a file will release when it is "deleted", and indeed, adds a whole new level of complexity to the housekeeping process. (And the tests!) - -The directory structure will need an additional flag, "Partial file", which specifies that the entry cannot be built as previous blocks are no longer available. Entries with this flag should never be sent to the client. - - - diff --git a/docs/backup/lib_backupclient.txt b/docs/backup/lib_backupclient.txt deleted file mode 100644 index 3e4a079b..00000000 --- a/docs/backup/lib_backupclient.txt +++ /dev/null @@ -1,46 +0,0 @@ -TITLE lib/backupclient - -Classes used on the store and on the server. - -See documentation in the files for more details. - - -SUBTITLE BackupStoreDirectory - -The directory listing class, containing a number of entries, representing files. - - -SUBTITLE BackupStoreFile - -Handles compressing and encrypting files, and decoding files downloaded from the server. - - -SUBTITLE BackupStoreFilename - -An encrypted filename. - - -SUBTITLE BackupStoreFilenameClear - -Derived from BackupStoreFilename, but with the ability to encrypt and decrypt filenames. Client side only. - - -SUBTITLE BackupClientFileAttributes - -Only used on the client -- the server treats attributes as blocks of opaque data. - -This reads attributes from files on discs, stores them, encrypts them, and applies them to new files. - -Also has a static function to generate filename attribute hashes given a struct stat and the filename. - - -SUBTITLE BackupClientRestore - -Routines to restore files from the server onto the client filesystem. - - -SUBTITLE BackupClientCryptoKeys - -This reads the key material from disc, and sets up the crypto for storing files, attributes and directories. - - diff --git a/docs/backup/lib_backupstore.txt b/docs/backup/lib_backupstore.txt deleted file mode 100644 index 8f24eb7b..00000000 --- a/docs/backup/lib_backupstore.txt +++ /dev/null @@ -1,30 +0,0 @@ -TITLE lib/backupstore - -Classes which are shared amongst the server side store utilities, bbstored and bbstoreaccounts. Note also depends on lib/backupclient, as a lot of code is shared between the client and server. - - -SUBTITLE BackupStoreAccountDatabase - -A simple implementation of an account database. This will be replaced with a more suitable implementation. - - -SUBTITLE BackupStoreAccounts - -An interface to the account database, and knowledge of how to initialise an account on disc. - - -SUBTITLE BackupStoreConfigVerify - -The same configuration file is used by all the utilities. This is the Configuration verification structure for this file. - - -SUBTITLE BackupStoreInfo - -The "header" information about an account, specifying current disc usage, space limits, etc. - - -SUBTITLE StoreStructure - -Functions specifying how the files are laid out on disc in the store. - - diff --git a/docs/backup/win32_build_on_cygwin_using_mingw.txt b/docs/backup/win32_build_on_cygwin_using_mingw.txt deleted file mode 100644 index 1caaf4ca..00000000 --- a/docs/backup/win32_build_on_cygwin_using_mingw.txt +++ /dev/null @@ -1,53 +0,0 @@ -How to build Box Backup on Win32 using Cygwin and MinGW -By Chris Wilson, 2007-05-26 - -(To read this document online with better formatting, browse to: -http://www.boxbackup.org/trac/wiki/CompileWithMinGW) - -Start by installing Cygwin on your Windows machine [http://www.cygwin.org]. -Make sure to select the following packages during installation: - -* Devel/gcc-mingw -* Devel/gcc-mingw-core -* Devel/gcc-mingw-g++ -* Mingw/mingw-zlib - -If you already have Cygwin installed, please re-run the installer and -ensure that those packages are installed. - -Download OpenSSL from -[http://www.openssl.org/source/openssl-0.9.7i.tar.gz] - -Open a Cygwin shell, and unpack OpenSSL: - - tar xzvf openssl-0.9.7i.tar.gz - -Configure OpenSSL for MinGW compilation, and build and install it: - - cd openssl-0.9.7i - ./Configure --prefix=/usr/i686-pc-mingw32/ mingw - make - make install - -Download PCRE from -[http://prdownloads.sourceforge.net/pcre/pcre-6.3.tar.bz2?download] - -Open a Cygwin shell, and unpack PCRE: - - tar xjvf pcre-6.3.tar.bz2 - -Configure PCRE for MinGW compilation, and build and install it: - - cd pcre-6.3 - export CFLAGS="-mno-cygwin" - ./configure - make winshared - cp .libs/libpcre.a .libs/libpcreposix.a /usr/lib/mingw - cp pcreposix.h /usr/include/mingw - -Now unpack the Box Backup sources, enter the source directory, -and configure like this: - - ./infrastructure/mingw/configure.sh - make - diff --git a/docs/backup/win32_build_on_linux_using_mingw.txt b/docs/backup/win32_build_on_linux_using_mingw.txt deleted file mode 100644 index 247d063d..00000000 --- a/docs/backup/win32_build_on_linux_using_mingw.txt +++ /dev/null @@ -1,73 +0,0 @@ -How to build Box Backup for Windows (Native) on Linux using MinGW -By Chris Wilson, 2005-12-07 - -Install the MinGW cross-compiler for Windows: - -- Debian users can "apt-get install mingw32" -- Fedora and SuSE users can download RPM packages from - [http://mirzam.it.vu.nl/mingw/] - -You will need to know the prefix used by the cross-compiler executables. -It will usually be something like "ix86-mingw32*-". All the binaries in the -cross-compiler package will start with this prefix. The documentation below -assumes that it is "i386-mingw32-". Adjust to taste. - -Download Zlib from [http://www.zlib.net/], unpack and enter source directory: - - export CC=i386-mingw32-gcc - export AR="i386-mingw32-ar rc" - export RANLIB="i386-mingw32-ranlib" - ./configure - make - make install prefix=/usr/local/i386-mingw32 - -Download OpenSSL 0.9.8b from -[http://www.openssl.org/source/openssl-0.9.8b.tar.gz] - -Unpack and configure: - - tar xzvf openssl-0.9.8b.tar.gz - cd openssl-0.9.8b - ./Configure --prefix=/usr/local/i386-mingw32 mingw - make makefile.one - wget http://www.boxbackup.org/svn/box/chris/win32/support/openssl-0.9.8b-mingw-cross.patch - patch -p1 < openssl-0.9.8b-mingw-cross.patch - make -f makefile.one - make -f makefile.one install - -Download PCRE from -[http://prdownloads.sourceforge.net/pcre/pcre-6.3.tar.bz2?download] - -Unpack: - - tar xjvf pcre-6.3.tar.bz2 - cd pcre-6.3 - -Configure and make: - - export AR=i386-mingw32-ar - ./configure --host=i386-mingw32 --prefix=/usr/local/i386-mingw32/ - make winshared - cp .libs/libpcreposix.a /usr/local/i386-pc-mingw32/lib - cp pcreposix.h /usr/local/i386-pc-mingw32/include - -Configure Box with: - - ./configure --host=i386-mingw32 \ - CXXFLAGS="-mthreads -I/usr/local/i386-mingw32/include" \ - LDFLAGS=" -mthreads -L/usr/local/i386-mingw32/lib" \ - LIBS="-lcrypto -lws2_32 -lgdi32" - make - -or, if that fails, try this: - - export CXX="i386-mingw32-g++" - export AR=i386-mingw32-ar - export RANLIB=i386-mingw32-ranlib - export CFLAGS="-mthreads" - export CXXFLAGS="-mthreads" - export LDFLAGS="-mthreads" - export LIBS="-lcrypto -lws2_32 -lgdi32" - (if you don't have a "configure" file, run "./bootstrap") - ./configure --target=i386-mingw32 - make CXX="$CXX" AR="$AR" RANLIB="$RANLIB" WINDRES="i386-mingw32-windres" diff --git a/docs/backup/windows_porting.txt b/docs/backup/windows_porting.txt deleted file mode 100644 index ada3b857..00000000 --- a/docs/backup/windows_porting.txt +++ /dev/null @@ -1,100 +0,0 @@ -TITLE Notes on porting the backup client to Windows - -It should be relatively easy to port the backup client (bbackupd) to Windows. However, the server relies on unlink() behaviour of the UNIX filesystem which is different on the Windows platform, so it will not be so easy to port. - -An installation of perl is required to build the system. The ActiveState port is the easiest to install. - - -SUBTITLE Build environment - -The build environment generation script, makebuildenv.pl, uses perl to scan all the files and generate makefiles. It's rather orientated towards UNIX systems with gcc. - -Probably the easiest way to handle this is to detect the Windows platform, set up a few variables appropriately (in BoxPlatform.pm) and then post-process the generated Makefiles to mould them into something more handy for the MS Windows compiler and toolchain. - -The script itself is a bit messy. It was fine at first, but then the multi-platform thing got in the way. I do intend to rewrite it at some point in the future. - -Make sure your new version defines PLATFORM_WIN32 on the compile line. - -All files #include "Box.h" as the first include file. Use this for pre-compiled headers. Edit BoxPlatform.h to include the Windows headers required, and include a PLATFORM_WIN32 section. The easiest start will be to leave this blank, apart from typedefs for the basic types and any "not supported" #defines you can find. - -Boring bits of the code, such as exceptions and protocol definitions, are autogenerated using perl scripts. This code should be portable without modification. - -I have tried to avoid the things I know won't work with the MS compiler, so hopefully the code will be fairly clean. However, it might be a little easier to use the MinGW compiler [ http://www.mingw.org/ ] just to be consistent with the UNIX version. But I hope this won't be necessary. - -You'll need the latest version of OpenSSL. This was slightly difficult to get to compile last time I tried -- especially if you're determined to use the optimised assembler version. The main difficulty was getting a version which would link properly with the options used in my project, the default libraries selected got in the way. - - -SUBTITLE Porting as UNIX emulation - -Since the daemon uses so few UNIX system calls and with a limited set of options, it seems to make sense to port it by writing emulations of these functions. It's probably nicest to create a lib/win32 directory, and populate this with header files corresponding to the UNIX header files used. These just contain inline functions which map the UNIX calls to Win32 calls. - -File/socket handles may have to be translated. -1 is used as a failure return value and by the code internally to mark an invalid socket handle. (0 is a valid socket handle) - -Of course, some bits of code aren't relevant, so will just be #ifdefed out, or replaced. But this should be minimal. (Only perhaps the small bit relating to filesystem structure -- there aren't really mount points as such.) - - -SUBTITLE File tracking - -The daemon uses the inode number of a file to keep track of files and directories, so when they're renamed they can be moved efficiently on the store. Some unique (per filesystem) number will have to be found and used instead. - -It uses the Berkeley DB to store these on disc. It's likely another storage system will need to be used. (It just has to map the file's unique number into to a 8 byte struct.) - -There is a in-memory implementation for platforms which don't support Berkeley DB, but this isn't so good when the daemon has to be restarted as all the tracking is lost. But it's an easy start. - - -SUBTITLE Expected filesystem behaviour - -File and directories have (at least) two modification times, for contents and attributes. - -For files, the contents modification time must change when the contents change, and the attributes time when the attributes change (and may change when the contents change too.) - -For directories, the contents modification time must change when files or directories are deleted or added. If it changes any more frequently than this, then the client will be slightly less efficient -- it will download the store's directory listing whenever this time changes. The attributes modification time is less important, as the actual attributes are compared and only uploaded if different. - - -SUBTITLE Attributes - -Attributes means file modification times, flags, and filesystem permissions. - -The BackupClientFileAttribute class will need to be extended. Allocate another "attribute type" for the Win32 attributes, and then serialise it in a compatible way -- put your new attribute type in the header, and then a serialised network byte order structure in the rest. The different size of block is handled for you, and the server never looks inside. - -Add code so that under UNIX, Win32 attributes are ignored, and UNIX attributes under Win32. - -It's probably not necessary to worry too much about these for the first version. Not many people seem to use these attributes anyway. - - -SUBTITLE Times - -The system uses it's own 64 bit time type -- see BoxTime.h. Everything is translated to this from the various different system time types, and calculated and stored internally in this form. - - -SUBTITLE Daemon as a Service - -The client is derived from the Daemon class, which implements a daemon. The interface is simple, and it shouldn't be hard to write a compatible class which implements a Windows Service instead. - -Or cheat and run it as a Win32 application. - -Note that the daemon expects to be able to read every file it wants, and will abort a scan and upload run if it gets an error. The daemon must therefore be run with sufficient privileges. It runs as root under UNIX. - - -SUBTITLE Command Socket - -The backup daemon accepts commands from bbackupctl through a UNIX domain socket. When a connection is made, the user ID of the connecting process is checked to see if it's the same user ID as the daemon is running under. - -This may not have any exact analogue under Win32, so another communications scheme may have to be devised. - -This is only actually necessary if the client is to be run in snapshot mode. It can be safely left unimplemented if snapshot mode is not required, or the prompts for it to sync with the server are implemented some other way. - - -SUBTITLE NTFS streams - -If you want to back up NTFS streams, then a generic solution should probably be defined, so that the Mac OS X resource forks can be backed up with the same mechanism. - - -SUBTITLE Source code - -I work on a slightly different version of the source files. A make distribution script adds the license header and removes private sections of code. This means submitted diffs need a slight bit of translation. - - - - - diff --git a/docs/common/lib_common.txt b/docs/common/lib_common.txt deleted file mode 100644 index 11d7b02d..00000000 --- a/docs/common/lib_common.txt +++ /dev/null @@ -1,52 +0,0 @@ -TITLE lib/common - -This module is the basic building block of the project. It is included implicitly as a dependency of all other modules. - -It provides basic services such as file and stream functionality, platform information, and various bits of utility code which doesn't fit naturally anywhere else but is useful to many other applications. - - -SUBTITLE Box.h - -The main include file for the project. It should be the first file included in every cpp file, followed by any system headers, then other project headers. This order has been chosen so that eventually it can be used as a target for pre-compiled headers. (This will be important for the Windows port) - - -SUBTITLE BoxPlatform.h - -This contains various bits of information on platform differences. The build scripts include a compile command line definition like PLATFORM_OPENBSD, which is then used to select the required options. - -Some of the stuff is not particularly pleasant, but it aims to produce a consistent compilation environment, and to have a abstracted way of setting other defines to tell the other files what things are available on this platform. - -GNU configure is not used, as it would be just another dependency. Although something does have to be done about compilation on Linux, which is just too different on each distribution to be caught by a common configuration here. - - -SUBTITLE Streams - -Much use is made of streams -- files, connections, buffers, all sorts of things are implemented using streams. - -The abstract base class IOStream defines the interface, see the class file for more details. - -Streams are lib/common's basic contribution to the project. A standard interface for handling data, and a number of utility classes which make performing common stream functions easy. - - -SUBTITLE Serialisation - -Note there is no "serialisable" class. Instead, objects which need to be serialised into Streams simply have two functions, ReadFromStream and WriteToStream. These perform as expected. - -As it turns out, there's no real need for a specific serialisation base class. If you do need to be able to serialise arbitary objects, there are two approaches. - -1) If you're serialising an arbitary object, you're going to know roughly what type it is. So it's likely to be a subclass of a known base class... in which case, you simply use virtual functions. - -2) If it really is an arbitary type, then you just write your function which accepts any type of object to serialise as a template. - - -SUBTITLE BoxException - -The exception base class for the project. All exceptions thrown by this code are dervied from BoxException. In general, each module which has unique errors has it's own exception class. - -CommonException errors are thrown throughout the project. - - -SUBTITLE Other bits - -There are some other extra useful bits which are documented only in the source files. - diff --git a/docs/common/lib_common/BoxTime.txt b/docs/common/lib_common/BoxTime.txt deleted file mode 100644 index 18bef5a5..00000000 --- a/docs/common/lib_common/BoxTime.txt +++ /dev/null @@ -1,7 +0,0 @@ -TITLE BoxTime - -Not strictly a class, but time is presented in the project as 64 bit integers as microseconds since the UNIX Epoch. - -There are a number of utility functions in the BoxTime*.h files, which are useful. - -To get BoxTime values from a struct stat, use the FileModificationTime.h functions.
\ No newline at end of file diff --git a/docs/common/lib_common/CollectInBufferStream.txt b/docs/common/lib_common/CollectInBufferStream.txt deleted file mode 100644 index 5f9556a3..00000000 --- a/docs/common/lib_common/CollectInBufferStream.txt +++ /dev/null @@ -1,26 +0,0 @@ -CLASS CollectInBufferStream - -This class is essentially a buffer. Data is written to it, and stored in memory. Then when all data is written which will be written, it can be read out again. - -Useful for building streams in memory. - - -FUNCTION CollectInBufferStream::SetForReading() - -After you've written all the data, call this to set it to read mode. - - -FUNCTION CollectInBufferStream::Reset() - -Reset the stream to the state where is has no data, and is about to have data written. - - -FUNCTION CollectInBufferStream::GetSize() - -Get the size of the buffered data -- the entire data. Use the interface function BytesLeftToRead() in most cases. - - -FUNCTION CollectInBufferStream::GetBuffer() - -Get the buffer, so you can do bad things with it if you're feeling naughty. - diff --git a/docs/common/lib_common/Configuration.txt b/docs/common/lib_common/Configuration.txt deleted file mode 100644 index ef5f38f6..00000000 --- a/docs/common/lib_common/Configuration.txt +++ /dev/null @@ -1,102 +0,0 @@ -CLASS Configuration - -Implements the reading of multi-level configuration files. This is intended to be a generic way of representing configuration implementation. - -Basic validation is performed on files as they are read, specified by a data structure. - -test/common has some examples of usage. - - -SUBTITLE Configuration file format - -The format is simple, a list of Key = Value pairs. For example - -Key1 = Value1 -Key2 = Value2 - -Optionally, multiple values for the same key are allowed. - -Lists of configurations can be nested to any level. These are called Sub-configurations: - -SubConfig -{ - SubKey1 = ValueX - SubKey2 = ValueY -} - - -SUBTITLE Verification - -The verification structure specifies what keys are required, what are optional, and what sub-configurations are expected. Default values can be specified. - -See Configuration.h for the structures. - -RaidFileController::Initialise has a good example of a simple verification layout. - -Wildcards can be used for SubConfigurations, by specifying a name of "*". This allows you to use multiple sections to configure any number of items. - -Each item has a number of flags, which are combined to say whether an item is required, should be an integer or boolean, and rather importantly, whether it's the last item in the list. - -Verification is limited, so you may wish to do more verification the file. There are unimplemented hooks for custom verification functions to be included in the verification definitions. Should be done at some point. - -Boolean keys have possible values "true", "yes", "false", "no" (case insensitive). - - -FUNCTION Configuration::LoadAndVerify() - -Loads the configuration from disc, and verifies it. If there are problems with the verification, it returns some text which can be used to tell the user the problems with the file. These are fairly basic error messages, but do say what should be done to get it to parse properly. - -This returns a Configuration object, which can then be queries for Keys and Values. Sub-configurations are implemented through an interface which returns a reference to another Configuration object. - - - -FUNCTION Configuration::KeyExists() - -Does a specified key exist? - -Use this for optional key values only -- specify them in the verification structure if they are required. - - -FUNCTION Configuration::GetKeyValue() - -Get the value of a key as a string. - -If ConfigTest_MultiValueAllowed is set in the relevant entry in the verify structure, this string may contain multiple values, separated by a single byte with value 0x01 (use Configuration::MultiValueSeparator in code). Use SplitString() defined in Utils.h to split it into components. - -This representation was chosen as multi-values are probably rare, and unlikely to justify writing a nicer (but more memory intensive and complex) solution. - - -FUNCTION Configuration::GetKeyValueInt() - -Get the value of a key as an integer. Make sure the AsInt property is requried in the verification structure. - - -FUNCTION Configuration::GetKeyValueBool() - -Get the value of a key as a boolean value. Make sure the AsBool property is requried in the verification structure. - -Default to "false", should this verification not be performed and an unknown value is specified. - - -FUNCTION Configuration::GetKeyNames() - -Return a list of all the keys in this configuration. - - -FUNCTION Configuration::SubConfigurationExists() - -Does a specified sub-configuration exist? - - -FUNCTION Configuration::GetSubConfiguration() - -Return another Configuration object representing the sub section. - - -FUNCTION Configuration::GetSubConfigurationNames() - -Get a list of all the sub configurations. - -As there isn't a particularly neat way that configurations can be iterated over, mSubConfigurations is public. (BAD!) - - diff --git a/docs/common/lib_common/Conversion.txt b/docs/common/lib_common/Conversion.txt deleted file mode 100644 index 62d1967a..00000000 --- a/docs/common/lib_common/Conversion.txt +++ /dev/null @@ -1,14 +0,0 @@ -TITLE Generic type conversion - -Conversion.h provides generic type conversion. Within the BoxConvert namespace, it defines the templated function - - ToType Convert<ToType, FromType>(FromType From) - -which converts the data type as expected. In general, from and to types have to be specified explicitly. - -Templates rather than overloaded functions are used, firstly to be absolutely explicit about the conversion required, and secondly because overloaded functions can't have differing return types for the same argument type. - -The function is specialised for various types, and the generic version uses C++ type conversion. - -Exceptions may be thrown if the conversion is not possible. These are all of the ConversionException type. - diff --git a/docs/common/lib_common/ExcludeList.txt b/docs/common/lib_common/ExcludeList.txt deleted file mode 100644 index 8a5bf36c..00000000 --- a/docs/common/lib_common/ExcludeList.txt +++ /dev/null @@ -1,21 +0,0 @@ -TITLE ExcludeList - -A class implementing a list of strings which are to be excluded in some way. Entries can be added as strings to be matched exactly, or as regular expressions. - -Multiple entries can be added in a single function call in a manner designed to work with the multi-value entries store in a Configuation object. - - -FUNCTION ExcludeList::AddDefiniteEntries() - -Definite entries are exact strings to match. - - -FUNCTION ExcludeList::AddRegexEntries() - -Regular expressions as defined by POSIX, and supported by the host platform. Will throw an exception if regular expressions are not supported by the platform. - - -FUNCTION ExcludeList::IsExcluded() - -Test a string for being excluded by definite or regex entries. - diff --git a/docs/common/lib_common/FdGetLine.txt b/docs/common/lib_common/FdGetLine.txt deleted file mode 100644 index d92ff94d..00000000 --- a/docs/common/lib_common/FdGetLine.txt +++ /dev/null @@ -1,11 +0,0 @@ -CLASS FdGetLine - -See IOStreamGetLine, difference is that it works on basic UNIX file descriptors rather than full blown streams. Follows basic interface, except... - - -FUNCTION FdGetLine::GetLine - -Returns a string containing the optionally preprocessed line. - -Do not call if IsEOF if true. - diff --git a/docs/common/lib_common/Guards.txt b/docs/common/lib_common/Guards.txt deleted file mode 100644 index 6174fea6..00000000 --- a/docs/common/lib_common/Guards.txt +++ /dev/null @@ -1,5 +0,0 @@ -TITLE Guards.h - -This file contains a couple of classes for using file and memory. When the class goes out of scope, the underlying object is closed or freed, respectively. - - diff --git a/docs/common/lib_common/IOStream.txt b/docs/common/lib_common/IOStream.txt deleted file mode 100644 index 09460656..00000000 --- a/docs/common/lib_common/IOStream.txt +++ /dev/null @@ -1,89 +0,0 @@ -CLASS IOStream - -The base class for streams of data. See IOStream.h for specific details on functions, but the interface is described here. - -Most streams only implement one direction, but some both. - - -SUBTITLE Reading - - -FUNCTION IOStream::Read() - -Reads data from the stream. Returns 0 if there is no data available within the timeout requested. - -Unlike the UNIX API, a return of 0 does not mean that there is no more data to read. Use IOStream::StreamDataLeft() to find this out. - - -FUNCTION IOStream::ReadFullBuffer() - -If you want to read a specific sized block of data from a stream, it is annoying ot use the Read function, because it might return less, so you have to loop. - -This implements that loop. Note that the timeout is the timeout for each individual read -- it might take a lot longer to complete. - -You must check the return value, which is whether or not it managed to get all that data. - - -FUNCTION IOStream::BytesLeftToRead() - -How many bytes are left to read in the stream treated as a whole. May return IOStream::SizeOfStreamUnknown if it is something like a socket which doesn't know how much data is in the stream. - - -FUNCTION IOStream::StreamDataLeft() - -Returns true if more data can be read. Or more specifically, if more data might be able to be read some time in the future. - - -SUBTITLE Writing - - -FUNCTION IOStream::Write() - -Write data to a stream. Writes the entire block, or exceptions. - - -FUNCTION IOStream::WriteAllBuffered() - -Writes any buffered data to the underlying object. - -Call at the end of a series of writes to make sure the data is actually written. (Buffering is not yet implemented anywhere, but it should be soon.) - - -FUNCTION IOStream::StreamClosed() - -Is the stream closed, and writing no longer possible? - - -FUNCTION IOStream::CopyStreamTo() - -A utility function to copy the contents of one stream to another, until the reading stream has no data left. - - -SUBTITLE Other interfaces - -These are slightly nasty interfaces, because they have to match the UNIX API to some extent. - - -FUNCTION IOStream::Close() - -Close the stream -- intended to indicate that nothing more will be written. - -However, also closes reading in most cases. Stream dependent. Probably best just to delete the object and let it sort itself out. - - -FUNCTION IOStream::Seek() - -Seeks within the stream -- mainly for using files within a stream interface, although some of the utility stream classes support it too. - -This is actually a bad interface, because it does not specify whether it applies to reading or writing. This matches the interface provided by files with a single file pointer, but does not map well onto other stream types. - -Should it be changed? Perhaps. But then it means that files would either have to have an inconsitent interface, or the class keep track of two separate file pointers. Which isn't nice. - -So use carefully, and remember whether you're using a file stream, a reading stream, or a writing stream. - - -FUNCTION IOStream::GetPosition() - -Get the current file pointer. Subject to same problems as Seek with regard to semantics. - - diff --git a/docs/common/lib_common/IOStreamGetLine.txt b/docs/common/lib_common/IOStreamGetLine.txt deleted file mode 100644 index 04c56b57..00000000 --- a/docs/common/lib_common/IOStreamGetLine.txt +++ /dev/null @@ -1,29 +0,0 @@ -CLASS IOStreamGetLine - -This class provides a convenient way to read text from a file, line by line. It also can preprocess the line to remove leading and trailing whitespace and comments. Comments are started by the character # and run to the end of the line. - -Create an instance by passing a reference to a stream into the constructor. - -Note the class does internal buffering, so you can only detach it later if the stream supports seeking backwards. - - -FUNCTION IOStreamGetLine::GetLine() - -Returns true if a line could be retreieved without a read timing out. - - -FUNCTION IOStreamGetLine::IsEOF() - -Whether the end of the stream has been reached. Do not call GetLine if this is true. - - -FUNCTION IOStreamGetLine::GetLineNumber() - -Returns the line number. - - -FUNCTION IOStreamGetLine::DetachFile() - -Detaches the stream from the GetLine class. Will seek backwards to "replace" data it's buffered back in the stream. - - diff --git a/docs/common/lib_common/MainHelper.txt b/docs/common/lib_common/MainHelper.txt deleted file mode 100644 index eb5b07f0..00000000 --- a/docs/common/lib_common/MainHelper.txt +++ /dev/null @@ -1,4 +0,0 @@ -TITLE MainHelper.h - -This header contains a couple of macros which add exception handling and reporting to main() functions for executable programs. - diff --git a/docs/common/lib_common/WaitForEvent.txt b/docs/common/lib_common/WaitForEvent.txt deleted file mode 100644 index 0bc55726..00000000 --- a/docs/common/lib_common/WaitForEvent.txt +++ /dev/null @@ -1,16 +0,0 @@ -CLASS WaitForEvent - -This class implements a way to efficiently wait on one or more system objects, for example sockets and file descriptors. Where kqueue() is available, this is used, otherwise poll(). The poll() implementation is less comprehensive and rather more limited. - -To add a compatible object, call Add(). To remove it, call Remove(). To wait for an object to become ready, call Wait(), which returns a pointer to the first ready object, or 0 for a timeout. - -The timeout is set either in the constructor, or using the SetTimout() method. It is specified in milliseconds. - -The kqueue based implementation will automatically remove objects when they are closed (actually, the OS does this), but the poll implementation requires that Remove() be called. - -The flags passed to Add() or Remove() are passed to the object, and are ignored by this class. - -For an object to be able to be added to the list, it should implement FillInKEvent() and FillInPoll() for kqueue and poll implementations respectively. Use the PLATFORM_KQUEUE_NOT_SUPPORTED define to test which is necessary for the platform. - -It does not have to be derived off any particular class, as it uses templates to be compatible with any class. - diff --git a/docs/common/lib_common/xStream.txt b/docs/common/lib_common/xStream.txt deleted file mode 100644 index 91e9c0ea..00000000 --- a/docs/common/lib_common/xStream.txt +++ /dev/null @@ -1,40 +0,0 @@ -TITLE Various stream types - -Some useful streams are implemented in lib/common. - - -SUBTITLE CollectInBufferStream - -Described in it's own page. - - -SUBTITLE FileStream - -Implements a stream from a file, allowing both reading and writing. - - -SUBTITLE MemBlockStream - -Turns a memory block into a readable stream. - -Can also be constructed using StreamableMemBlock, CollectInBufferStream and MemBlockStream as sources of the buffer. - - -SUBTITLE PartialReadStream - -Create a readable stream which will read a set number of bytes from another stream, and then declare itself as closed. - -Useful for extracting a small chunk of another stream to present to a function which expects to consume all of a stream. - - -SUBTITLE ReadGatherStream - -Create a readable stream out of blocks from many streams -- so various sections of any number of streams are composed into a single stream. - -To use, register each stream with AddComponent, then use the returned 'handle' with AddBlock to add blocks to the composed stream. - -Optionally, the object will take ownership of the streams added, and delete them when itself is deleted. - -See the comments in the function blocks in the cpp file for more info. - - diff --git a/docs/common/lib_compress.txt b/docs/common/lib_compress.txt deleted file mode 100644 index f3e26f20..00000000 --- a/docs/common/lib_compress.txt +++ /dev/null @@ -1,8 +0,0 @@ -TITLE lib/compress - -This is a horrid C++ interface to zlib. It is written this way to avoid having to buffer any data, and so be unnecessarily inefficient. But this does end up just being a wrapper to the zlib interface. - -See test/compress for an example of how to use it. But it should be very familiar. - -For an easier interface, use CompressStream. - diff --git a/docs/common/lib_compress/CompressStream.txt b/docs/common/lib_compress/CompressStream.txt deleted file mode 100644 index eca43eb6..00000000 --- a/docs/common/lib_compress/CompressStream.txt +++ /dev/null @@ -1,27 +0,0 @@ -TITLE CompressStream - -This implements a compressing stream class, which compresses and decompresses data sent to an underlying stream object. Data is likely to be buffered. - - -WARNING: Use WriteAllBuffered() (after objects which need to be sent in their entirity) and Close() (after you have finished writing to the stream) otherwise the compression buffering will lose the last bit of data for you. - - -The class works as a filter to an existing stream. Ownership can optionally be taken so that the source stream is deleted when this object is deleted. - -It can operate in one or two directions at any time. Data written to the stream is compressed, data read from the stream is decompressed. If a direction is not being (de)compressed, then it can act as a pass-through without touching the data. - -The constructor specifies the actions to be taken: - -CompressStream(IOStream *pStream, bool TakeOwnership, - bool DecompressRead, bool CompressWrite, - bool PassThroughWhenNotCompressed = false); - -pStream - stream to filter -TakeOwnership - if true, delete the stream when this object is deleted. -DecompressRead - reads pass through a decompress filter -CompressWrite - writes pass through a compression filter -PassThroughWhenNotCompressed - if not filtering a direction, pass through the data unaltered, otherwise exception if an attempt is made on that direction. - - -Note that the buffering and compression will result in different behaviour than the underlying stream: data may not be written in one go, and data being read may come out in different sized chunks. - diff --git a/docs/common/lib_crypto.txt b/docs/common/lib_crypto.txt deleted file mode 100644 index 9ddafe69..00000000 --- a/docs/common/lib_crypto.txt +++ /dev/null @@ -1,28 +0,0 @@ -TITLE lib/crypto - -Provides cryptographic primatives using the OpenSSL implementation. - - -SUBTITLE CipherContexts and CipherDescriptions - -A CipherContext is the interface to encryption and decryption, providing a generic interface. - -It is constructed with a decrypt or encrypt flag, and a CipherDescription. The CipherDescription specifies which cipher is to be used, the key, and all other cipher specific paramteres. - -The CipherContext has support for padding and initialisation vectors. - - -SUBTITLE Random numbers - -An interface to the OpenSSL psuedo random number generater is provided. - - -SUBTITLE Digests - -An interface to the MD5 digest is provided. - - -SUBTITLE RollingChecksum - -The rsync rolling checksum is implemented. The interface is a little tricky to be efficient as the caller must track the position and provide relevant bytes to advance the checksum. - diff --git a/docs/common/lib_crypto/CipherContext.txt b/docs/common/lib_crypto/CipherContext.txt deleted file mode 100644 index 30fb4608..00000000 --- a/docs/common/lib_crypto/CipherContext.txt +++ /dev/null @@ -1,28 +0,0 @@ -CLASS CipherContext - -Encryption and decryption using OpenSSL EVP interface. - -See the cpp file for more documentation in the function headers, and test/crypto for examples. - -General notes below. - - -SUBTITLE Construction - -Construct with the encryption direction, and a CipherDescription of the cipher and key required. - - -SUBTITLE Encrypting or decrypting - -Begin() and Transform() allow piece by piece transformation of a block. - -TransformBlock() transforms an entire block. - - -SUBTITLE Buffering - -All transforms expect to have enough space in the buffers for their entire output. Because of block boundaries and padding, it is necessary that the output buffer should be bigger than the input buffer. The amount of space depends on the cipher. - -InSizeForOutBufferSize() and MaxOutSizeForInBufferSize() perform these space calculations, returning the maximuim in size for a specified out size, and the reverse, respectively. - - diff --git a/docs/common/lib_crypto/RollingChecksum.txt b/docs/common/lib_crypto/RollingChecksum.txt deleted file mode 100644 index d871b3f2..00000000 --- a/docs/common/lib_crypto/RollingChecksum.txt +++ /dev/null @@ -1,36 +0,0 @@ -CLASS RollingChecksum - -Implementing the rsync rolling checksum algorithm. Read it's description first: - -http://samba.anu.edu.au/rsync/tech_report/node3.html - - -SUBTITLE Construction and initial checksum calculation - -The constructor takes a pointer to a block of data and a size, and calculates the checksum of this block. It can now be "rolled forward" to find the checksum of the block of the same size, one byte forward, with minimal calculation. - - -FUNCTION RollingChecksum::GetChecksum() - -Returns the checksum for the current block. - - -FUNCTION RollingChecksum::RollForward() - -This function takes the byte at the start of the current block, and the last byte of the block it's rolling forward to, and moves the checksum on. - -If the block is - - char *pBlock = <something>; - -with size s, then it should be called with - - RollForward(pBlock[0], pBlock[s]) - -and now GetChecksum will return the checksum of the block (pBlock+1) of size s. - - -FUNCTION RollingChecksum::RollForwardSeveral() - -Similar to RollForward(), but is more efficient for skipping several bytes at once. Takes pointers to the data buffer rather than the actual data values. - diff --git a/docs/common/lib_server.txt b/docs/common/lib_server.txt deleted file mode 100644 index 392f331d..00000000 --- a/docs/common/lib_server.txt +++ /dev/null @@ -1,9 +0,0 @@ -TITLE lib/server - -This provides various classes for implementing client/server applications (and UNIX daemons). - -The stars of the show are ServerTLS and Protocol, which allow client server applications which communicate using TLS (SSL successor) to be built with surprisingly few lines of code. - -All details in the respective class files. - -Look at test/basicserver for examples. diff --git a/docs/common/lib_server/Daemon.txt b/docs/common/lib_server/Daemon.txt deleted file mode 100644 index 6956ec2b..00000000 --- a/docs/common/lib_server/Daemon.txt +++ /dev/null @@ -1,96 +0,0 @@ -CLASS Daemon - -Implement a UNIX daemon which - -* Daemonises (detach from console, etc) -* Sets up signal handlers, and does useful things with them -* Reads configuration files -* Writes PID file -* Opens syslog - -all the usual UNIX basic daemon behaviour. - -The daemon exe takes optional arguments: The first is a configuration file filename which overrides the default. If the second argument is 'SINGLEPROCESS' the daemon will not daemonise. - -The configuration file must have a section like this - -Server -{ - PidFile = /var/run/daemon.pid - User = username -} - -Use DAEMON_VERIFY_SERVER_KEYS (defined in Daemon.h) to include the necessary keys in your configuration file's verify structure. - -The "User" line is optional, and if it is present the Daemon will change user to this username just before it daemonises. Note that unless the directory the PID file is written to has write permission for this user, the PID file will not be deleted on exit. - - -To implement a daemon, derive a class from Daemon, and override Run(). - -Then in your main file, do something like - -int main(int argc, const char *argv[]) -{ - MAINHELPER_START - - BackupDaemon daemon; - return daemon.Main(BOX_FILE_BBACKUPD_DEFAULT_CONFIG, argc, argv); - - MAINHELPER_END -} - -and that's it. Obviously it's worth doing a few more things as well, but that's it. - - -FUNCTION Daemon::Run() - -Override with the main daemon code. It should behave like this - -void SomethingDaemon::Run() -{ - // Read configuration file - - // Do lots of work until StopRun() returns true - while(!StopRun()) - { - ::sleep(10); - } - - // Clean up nicely -} - -which allows the base class to implement the standard start, terminate and -HUP behaviours correctly. - - -FUNCTION Daemon::DaemonName() - -Returns the name of the daemon, for use in syslog. - - -FUNCTION Daemon::DaemonBanner() - -Returns the banner to be displayed on startup, or 0 for no banner. - - -FUNCTION Daemon::GetConfigVerify() - -Returns a configuration verify structure for verifying the config file. Note that this configuration structure should include a sub-configuration called "Server, and have entries defined by DAEMON_VERIFY_SERVER_KEYS. See one of the bin/bbackupd for an example. - - -FUNCTION Daemon::StopRun() - -Returns true when the daemon needs to be terminated or restarted. Use IsReloadConfigWanted() and IsTerminateWanted() to find out which one, if you need to know. - - -FUNCTION Daemon::SetupInInitialProcess() - -Override to perform additional functions in the initial process, before forking and detachment happens. - - -FUNCTION Daemon::EnterChild() - -Called when a child is entered. If you override it, remember to call the base class. - - - - diff --git a/docs/common/lib_server/Protocol.txt b/docs/common/lib_server/Protocol.txt deleted file mode 100644 index 09d3c1f1..00000000 --- a/docs/common/lib_server/Protocol.txt +++ /dev/null @@ -1,120 +0,0 @@ -CLASS Protocol - -Protocol - -* serialises and deserialises data objects -* sends arbitary streams - -through a bi-directional IOStream object, usually a socket. - -These data objects are auto-generated by a perl script, along with the logic to implement a simple protocol where one end is a client, and the other a server. The client sends a command object (and optional streams) and the server returns a reply object (and optional streams). - -The perl script uses a description file which specifies the data inside these objects (which can be extended to include any type which implements the standard serialisation functions), the required responses to the commands, and whether streams are involved. - -It then implements a server object, which given a stream and a user defined context object, waits for commands, processes them, and sends the results back. All you need to do is implement a DoCommand() function for each command object you define. - -On the client side, a Query() function is implemented which takes objects as parameters, and returns the right type of reply object (or throws an exception if it doesn't get it.) Short cut Query<CommandName>() functions are also implemented which don't require objects to be created manually. - -Thus, implementing a server is as simple as deriving a daemon off ServerStream or ServerTLS, and implementing the Connection() function as - - void TestProtocolServer::Connection(SocketStream &rStream) - { - TestProtocolServer server(rStream); - TestContext context; - server.DoServer(context); - } - -and that's it. TestContext is a user defined class which keeps track of all the state of the connection, and is passed to all the DoCommand() functions, which look like this: - - std::auto_ptr<ProtocolObject> - TestProtocolServerSimple::DoCommand(TestProtocolServer &rProtocol, - TestContext &rContext) - { - return std::auto_ptr<ProtocolObject> - (new TestProtocolServerSimpleReply(mValue+1)); - } - -(taken from test/basicserver) - -The client code looks like this - - SocketStream conn; - conn.Open(Socket::TypeUNIX, "testfiles/srv4.sock"); - - TestProtocolClient protocol(conn); - - // Query - { - std::auto_ptr<TestProtocolClientSimpleReply> - reply(protocol.QuerySimple(41)); - TEST_THAT(reply->GetValuePlusOne() == 42); - } - - -Finally, debug logging can be generated which allows a list of all commands and their parameters to be logged to syslog or a file. - - -SUBTITLE Protocol Description File - -This file is passed to the lib/server/makeprotocol.pl script, which generates a h and cpp file to implement the protocol. - -It is in two sections, separated by a 'BEGIN_OBJECTS' on a line of it's own. - -In the top half, the following statements must be made. - -Name <name> - The name of the protocol, used in naming classes. - -IdentString <string> - The idenfitifaction string sent over the IOStream to confirm it it - is talking to another Protocol object speaking the same Protocol. - -ServerContextClass <class-name> <header-file> - The user defined context class used for the server, and the header - file it is defined in. - -Additionally, the following optional commands can be made. - -ClientType <description-typename> <C++ typename> <headerfile> -ServerType (similarly) - Extends the types used in the objects below. Server and client - can use different types for the same object type. - -ImplementLog (Client|Server) (syslog|file) - Implement command logging for client or server into syslog or a file. - -LogTypeToText (Client|Server) <description-typename> <printf-element> - <evaluate> - For extended types, optionally define how to convert them into printf - elements and the code to run to get the argument. Within the evaluate - parameter, VAR is replaced by the name of the variable to display. - If this is not specified for a given type, OPAQUE is output instead. - - -In the object section, an object is defined by a line - -<name> <id number> <attributes> - -followed by lines beginning with whitespace defining the data transmitted in the object. The type may be list<type>, which specifies a list (implemented as a std::vector) of entries of that type. - -The attributes specify exactly how that object is used in the defined protocol. - -Reply - The object is a reply object, sent from the server to the client. - -Command(Reply-Type) - The object is a command, send from the client to the server, and the server - will send back an object of type Reply-Type. - -IsError(Type-Field,SubType-Field) - The command is an error object, and the two files specify the data member - which describes the error type and sub type. - -EndsConversation - When this command is received, the connection is to be terminated. - (ie a logout command) - -StreamWithCommand - When this command is sent as a command, a stream follows it. - - diff --git a/docs/common/lib_server/ServerStream.txt b/docs/common/lib_server/ServerStream.txt deleted file mode 100644 index 6c5932a0..00000000 --- a/docs/common/lib_server/ServerStream.txt +++ /dev/null @@ -1,29 +0,0 @@ -CLASS ServerStream - -ServerStream implementes a Daemon which accepts stream connections over sockets, and forks into a child process to handle them. - -To implement a daemon, derive from - - ServerStream<SocketStream, SERVER_LISTEN_PORT> - -The type SocketStream specifies that incoming connections should be treated as normal sockets, and SERVER_LISTEN_PORT is the port to listen to (if it's a inet socket, and if not overridden in the config file). The actual addresses (or names) to bind to are specified in the configuration file by the user. - -Make sure SERVERSTREAM_VERIFY_SERVER_KEYS(0) is included in the configuration verification structure in the "Server" sub configuration. 0 could be replaced with a default address, for example "unix:/var/run/server.sock" to specific a default UNIX socket in the filesystem. - -See test/basicserver for a simple example. - -The ListenAddresses key in the Server subconfiguration is a comma separated list of addresses, specified as family:name. Internet sockets are family 'inet', for example 'inet:localhost' (or 'inet:localhost:1080' to specify a port number as well), and unix domain sockets are 'unix', example above. - - -Override Connection to handle the connection. - -Remember to override Daemon functions like the server name, and start it up, just like a generic Daemon. - - -FUNCTION ServerStream::Connection - -This function takes a connected stream as it's argument. It should then proceed to do whatever it needs to do to talk to the client. - -Using IOStreamGetLine or a Protocol class to communicate may be quick ways of implementing this functionality. - - diff --git a/docs/common/lib_server/ServerTLS.txt b/docs/common/lib_server/ServerTLS.txt deleted file mode 100644 index dbde500f..00000000 --- a/docs/common/lib_server/ServerTLS.txt +++ /dev/null @@ -1,6 +0,0 @@ -CLASS ServerTLS - -Implements a server which uses TLS (SSL) to encrypt and authenticate connections. - -Very similar to ServerStream, except it reads the certificate files for the TLSContext out of the Server sub-configuration to set up a TLSContext ("CertificateFile", "PrivateKeyFile" and "TrustedCAsFile"). Otherwise works exactly the same. - diff --git a/docs/common/lib_server/SocketStream.txt b/docs/common/lib_server/SocketStream.txt deleted file mode 100644 index 82813279..00000000 --- a/docs/common/lib_server/SocketStream.txt +++ /dev/null @@ -1,8 +0,0 @@ -CLASS SocketStream - -A implementation of IOStream which wraps a socket connection. - -It can either be created by attaching to an existing object, or use the Open() function to open a connection to a named host on a specific port (or a local UNIX socket in the filesystem). - -Follows stream interface. - diff --git a/docs/common/lib_server/SocketStreamTLS.txt b/docs/common/lib_server/SocketStreamTLS.txt deleted file mode 100644 index ebb3f233..00000000 --- a/docs/common/lib_server/SocketStreamTLS.txt +++ /dev/null @@ -1,11 +0,0 @@ -CLASS SocketStreamTLS - -An implementation of IOStream which wraps a TLS (SSL) connection over a socket. - -The Open function takes a TLSContext reference which specifies the parameters for the connection. - - -FUNCTION GetPeerCommonName() - -Returns the common name of the certificate presented by the remote end. - diff --git a/docs/common/lib_server/TLSContext.txt b/docs/common/lib_server/TLSContext.txt deleted file mode 100644 index ff50d3e6..00000000 --- a/docs/common/lib_server/TLSContext.txt +++ /dev/null @@ -1,16 +0,0 @@ -CLASS TLSContext - -A wrapper over the OpenSSL context object. - -Note: you need to call SSLLib::Initialise at the beginning of your program to use these functions. - - -SUBTITLE Construction - -The constuctor takes the following parameters - -* Boolean for whether this is acting as a server or a client -* The .pem file containing the certificate which will be used -* The .pem file containing the private key for this certificate -* The .pem file containing the certificates which will certify the other end of the connection. - diff --git a/docs/common/memory_leaks.txt b/docs/common/memory_leaks.txt deleted file mode 100644 index 9a9764ea..00000000 --- a/docs/common/memory_leaks.txt +++ /dev/null @@ -1,44 +0,0 @@ -TITLE Memory leak detection - -The build system contains a primative memory leak detection system in debug builds. - -It works by using #defines to replace calls to malloc, free, realloc, new and delete with debug versions which record their use. When a process ends, it dumps a list of all the blocks or objects which were allocated by not freed, and the file and line of the source where they are originally allocated. - -It's not perfect, but should catch most things and work on most platforms. - -If it doesn't work on your platform, define PLATFORM_DISABLE_MEM_LEAK_TESTING in BoxPlatform.h within the relevant section. - - -SUBTITLE Requirements in source - -It does require some extra lines in the source file. The last included file in each .cpp file must be - - #include "MemLeakFindOn.h" - -and if a .h file uses any of these functions, the contents of the .h file should be bounded with - - #include "MemLeakFindOn.h" - - // ... some code, but absolutely no #includes - - #include "MemLeakFindOff.h" - -The cleanupforcvs.pl script checks for correct usage. - - -SUBTITLE Use in tests - -Tests will automatically dump memory leaks and regard them as a failure for anything executing in the main test process. - -To test for memory leaks in programs run from the test, or daemons, use something like - - TestRemoteProcessMemLeaks("bbackupd.memleaks"); - -If memory leak detection is being used, it will check the specified file for memory leak reports (deleting it afterwards). Any leak is an error. - -The Daemon class automactically arranges for the memory leak reports to be written. Other exes should set this up themselves, preferably using the define in MainHelper.h, as the first thing in their main() function. - - MAINHELPER_SETUP_MEMORY_LEAK_EXIT_REPORT(file, name) - -File is the filename to write the report to, conventionally called "<programname>.memleaks", and name is the name of the exe. - diff --git a/docs/raidfile/lib_raidfile.txt b/docs/raidfile/lib_raidfile.txt deleted file mode 100644 index 0c4244ce..00000000 --- a/docs/raidfile/lib_raidfile.txt +++ /dev/null @@ -1,73 +0,0 @@ -TITLE lib/raidfile - -Implements RAID type abilities in files in userland, along with simple transaction like semantics for writing and reading files. - -IOStream interfaces are used to read and write files. - -Eventually a raid file daemon will be written to "raidify" files in the background. This will allow fast streaming of data to a single disc, followed by splitting into separate raid files later. - -There is an option to disable raidification for use on systems with only one hard disc, or a hardware RAID array. - - -SUBTITLE Controller - -The raid disc sets are managed by RaidFileController. This reads the configuration file, /etc/box/raidfile.conf, and then stores the sets of discs for use by the other classes. - -In the code, files are referenced using an integer set number, and a filename within that set. For example, (0, "dir/subdir/filename.ext"). The RAID file controller turns this into actual physcial filenames transparently. - -The files are devided into blocks, which should match the fragment/block size of the underlying filesystem for maximum space efficiency. - -If the directories specified in the configuration files are all equal, then userland RAID is disabled. The files will not be processed into the redundant parts. - - -SUBTITLE Writing - -Files are written (and modified) using the RaidFileWrite class. This behaves as a seekable IOStream for writing. - -When all data has been written, the file is committed. This makes it available to be read. Processing to split it into the three files can be delayed. - -If the data is not commited, the temporary write file will be deleted. - -RaidFileWrite will optionally refuse to overwrite another file. If it is being overwritten, a read to that file will get the old file until it has been committed. - - -SUBTITLE Reading - -File are opened and read with RaidFileRead::Open. This returns an IOStream (with some extras) which reads a file. - -If the file has been turned into the RAID representation, then if an EIO error occurs on any operation, it will switch transparently into recovery mode where the file will be regenerated from the remaining two files using the parity file. (and log an error to syslog) - - -SUBTITLE Layout on disc - -The directory structure is replicated onto each of the three directories within the disc set. - -Given a filename x, within the disc set, one of the three discs is chosen as the 0th disc for this file. This is done by hashing the filename. This ensures that an equal load is placed on each disc -- otherwise one disc would only be accessed when a RAID file was being written. - -When the file is being written, the actual physical file is "x.rfwX". This just a striaght normal file -- nothing is done to the data. - -When it is commited, it is renamed to "x.rfw". - -Then, when it is turned into the raid representation, it is split across three discs, each file named "x.rf". - -When trying to open a file, the first attempt is for "x.rfw". If this exists, it is used as is. - -If this fails, two of the "x.rf" files are attempted to be opened. If this succeeds, the file is opened in RAID mode. - -This procedure guarenettes that a file will be opened either as a normal file, or as the correct three RAID file elements, even if a new version is being commited at the same time, or another process is working to turn it into a raid file. - - -SUBTITLE Raid file representation - -The file is split into three files, stripe 1, stripe 2 and parity. - -Considering the file as a set of blocks of the size specified in the configuration, odd numbered blocks go in stripe 1, even blocks in stripe 2, and a XOR of the corresponding blocks in parity. - -Thus only two files are needed to recreate the original. - -The complexity comes because the size of the file is not stored inside the files at any point. This means if stripe 2 is missing, it is ambiguous as to how big the file is. - -Size is stored as a 64 bit integer. This is appended to the parity file, unless it can be stored by XORing it into the last 8 bytes of the parity file. - -This scheme is complex to implement (see the RaidFileRead.cpp!) but minimises the disc spaced wasted. - diff --git a/docs/raidfile/lib_raidfile/RaidFileRead.txt b/docs/raidfile/lib_raidfile/RaidFileRead.txt deleted file mode 100644 index 0a5efbf7..00000000 --- a/docs/raidfile/lib_raidfile/RaidFileRead.txt +++ /dev/null @@ -1,14 +0,0 @@ -CLASS RaidFileRead - -Read a raid file. - -IOStream interface, plus a few extras, including reading directories and checking that files exist. - - -FUNCTION RaidFileRead::Open - -Open a given raid file -- returns a pointer to a new RaidFileRead object. - -Note that one of two types could be returned, depending on the representation of the file. - - diff --git a/docs/raidfile/lib_raidfile/RaidFileWrite.txt b/docs/raidfile/lib_raidfile/RaidFileWrite.txt deleted file mode 100644 index 89500c37..00000000 --- a/docs/raidfile/lib_raidfile/RaidFileWrite.txt +++ /dev/null @@ -1,36 +0,0 @@ -CLASS RaidFileWrite - -Interface to writing raidfiles. - -See IOStream interface. - -Some other useful functions are available, see h and cpp files. - - -FUNCTION RaidFileWrite::RaidFileWrite() - -The constructor takes the disc set number and filename of the file you're interested. - - -FUNCTION RaidFileWrite::Open() - -Open() opens the file for writing, and takes an "allow overwrite" flag. - - -FUNCTION RaidFileWrite::Commit() - -Commmit the file, and make it visible to RaidFileRead. If ConvertToRaidNow == true, it will be converted to raid file representation immediately. - -Setting it to false is not a good idea. Later on, it will tell a daemon to convert it in the background, but for now it simply won't be converted. - - -FUNCTION RaidFileWrite::Discard() - -Abort the creation/update. Equivalent to just deleting the object without calling Commit(). - - -FUNCTION RaidFileWrite::Delete() - -Delete a file -- don't need to Open() it first. - - |