summaryrefslogtreecommitdiff
path: root/docs/common
diff options
context:
space:
mode:
Diffstat (limited to 'docs/common')
-rw-r--r--docs/common/lib_common.txt52
-rw-r--r--docs/common/lib_common/BoxTime.txt7
-rw-r--r--docs/common/lib_common/CollectInBufferStream.txt26
-rw-r--r--docs/common/lib_common/Configuration.txt102
-rw-r--r--docs/common/lib_common/Conversion.txt14
-rwxr-xr-xdocs/common/lib_common/ExcludeList.txt21
-rw-r--r--docs/common/lib_common/FdGetLine.txt11
-rw-r--r--docs/common/lib_common/Guards.txt5
-rw-r--r--docs/common/lib_common/IOStream.txt89
-rw-r--r--docs/common/lib_common/IOStreamGetLine.txt29
-rw-r--r--docs/common/lib_common/MainHelper.txt4
-rw-r--r--docs/common/lib_common/WaitForEvent.txt16
-rw-r--r--docs/common/lib_common/xStream.txt40
-rw-r--r--docs/common/lib_compress.txt8
-rw-r--r--docs/common/lib_compress/CompressStream.txt27
-rw-r--r--docs/common/lib_crypto.txt28
-rw-r--r--docs/common/lib_crypto/CipherContext.txt28
-rw-r--r--docs/common/lib_crypto/RollingChecksum.txt32
-rw-r--r--docs/common/lib_server.txt9
-rw-r--r--docs/common/lib_server/Daemon.txt96
-rw-r--r--docs/common/lib_server/Protocol.txt120
-rw-r--r--docs/common/lib_server/ServerStream.txt29
-rw-r--r--docs/common/lib_server/ServerTLS.txt6
-rw-r--r--docs/common/lib_server/SocketStream.txt8
-rw-r--r--docs/common/lib_server/SocketStreamTLS.txt11
-rw-r--r--docs/common/lib_server/TLSContext.txt16
-rwxr-xr-xdocs/common/memory_leaks.txt44
27 files changed, 878 insertions, 0 deletions
diff --git a/docs/common/lib_common.txt b/docs/common/lib_common.txt
new file mode 100644
index 00000000..11d7b02d
--- /dev/null
+++ b/docs/common/lib_common.txt
@@ -0,0 +1,52 @@
+TITLE lib/common
+
+This module is the basic building block of the project. It is included implicitly as a dependency of all other modules.
+
+It provides basic services such as file and stream functionality, platform information, and various bits of utility code which doesn't fit naturally anywhere else but is useful to many other applications.
+
+
+SUBTITLE Box.h
+
+The main include file for the project. It should be the first file included in every cpp file, followed by any system headers, then other project headers. This order has been chosen so that eventually it can be used as a target for pre-compiled headers. (This will be important for the Windows port)
+
+
+SUBTITLE BoxPlatform.h
+
+This contains various bits of information on platform differences. The build scripts include a compile command line definition like PLATFORM_OPENBSD, which is then used to select the required options.
+
+Some of the stuff is not particularly pleasant, but it aims to produce a consistent compilation environment, and to have a abstracted way of setting other defines to tell the other files what things are available on this platform.
+
+GNU configure is not used, as it would be just another dependency. Although something does have to be done about compilation on Linux, which is just too different on each distribution to be caught by a common configuration here.
+
+
+SUBTITLE Streams
+
+Much use is made of streams -- files, connections, buffers, all sorts of things are implemented using streams.
+
+The abstract base class IOStream defines the interface, see the class file for more details.
+
+Streams are lib/common's basic contribution to the project. A standard interface for handling data, and a number of utility classes which make performing common stream functions easy.
+
+
+SUBTITLE Serialisation
+
+Note there is no "serialisable" class. Instead, objects which need to be serialised into Streams simply have two functions, ReadFromStream and WriteToStream. These perform as expected.
+
+As it turns out, there's no real need for a specific serialisation base class. If you do need to be able to serialise arbitary objects, there are two approaches.
+
+1) If you're serialising an arbitary object, you're going to know roughly what type it is. So it's likely to be a subclass of a known base class... in which case, you simply use virtual functions.
+
+2) If it really is an arbitary type, then you just write your function which accepts any type of object to serialise as a template.
+
+
+SUBTITLE BoxException
+
+The exception base class for the project. All exceptions thrown by this code are dervied from BoxException. In general, each module which has unique errors has it's own exception class.
+
+CommonException errors are thrown throughout the project.
+
+
+SUBTITLE Other bits
+
+There are some other extra useful bits which are documented only in the source files.
+
diff --git a/docs/common/lib_common/BoxTime.txt b/docs/common/lib_common/BoxTime.txt
new file mode 100644
index 00000000..18bef5a5
--- /dev/null
+++ b/docs/common/lib_common/BoxTime.txt
@@ -0,0 +1,7 @@
+TITLE BoxTime
+
+Not strictly a class, but time is presented in the project as 64 bit integers as microseconds since the UNIX Epoch.
+
+There are a number of utility functions in the BoxTime*.h files, which are useful.
+
+To get BoxTime values from a struct stat, use the FileModificationTime.h functions. \ No newline at end of file
diff --git a/docs/common/lib_common/CollectInBufferStream.txt b/docs/common/lib_common/CollectInBufferStream.txt
new file mode 100644
index 00000000..5f9556a3
--- /dev/null
+++ b/docs/common/lib_common/CollectInBufferStream.txt
@@ -0,0 +1,26 @@
+CLASS CollectInBufferStream
+
+This class is essentially a buffer. Data is written to it, and stored in memory. Then when all data is written which will be written, it can be read out again.
+
+Useful for building streams in memory.
+
+
+FUNCTION CollectInBufferStream::SetForReading()
+
+After you've written all the data, call this to set it to read mode.
+
+
+FUNCTION CollectInBufferStream::Reset()
+
+Reset the stream to the state where is has no data, and is about to have data written.
+
+
+FUNCTION CollectInBufferStream::GetSize()
+
+Get the size of the buffered data -- the entire data. Use the interface function BytesLeftToRead() in most cases.
+
+
+FUNCTION CollectInBufferStream::GetBuffer()
+
+Get the buffer, so you can do bad things with it if you're feeling naughty.
+
diff --git a/docs/common/lib_common/Configuration.txt b/docs/common/lib_common/Configuration.txt
new file mode 100644
index 00000000..ef5f38f6
--- /dev/null
+++ b/docs/common/lib_common/Configuration.txt
@@ -0,0 +1,102 @@
+CLASS Configuration
+
+Implements the reading of multi-level configuration files. This is intended to be a generic way of representing configuration implementation.
+
+Basic validation is performed on files as they are read, specified by a data structure.
+
+test/common has some examples of usage.
+
+
+SUBTITLE Configuration file format
+
+The format is simple, a list of Key = Value pairs. For example
+
+Key1 = Value1
+Key2 = Value2
+
+Optionally, multiple values for the same key are allowed.
+
+Lists of configurations can be nested to any level. These are called Sub-configurations:
+
+SubConfig
+{
+ SubKey1 = ValueX
+ SubKey2 = ValueY
+}
+
+
+SUBTITLE Verification
+
+The verification structure specifies what keys are required, what are optional, and what sub-configurations are expected. Default values can be specified.
+
+See Configuration.h for the structures.
+
+RaidFileController::Initialise has a good example of a simple verification layout.
+
+Wildcards can be used for SubConfigurations, by specifying a name of "*". This allows you to use multiple sections to configure any number of items.
+
+Each item has a number of flags, which are combined to say whether an item is required, should be an integer or boolean, and rather importantly, whether it's the last item in the list.
+
+Verification is limited, so you may wish to do more verification the file. There are unimplemented hooks for custom verification functions to be included in the verification definitions. Should be done at some point.
+
+Boolean keys have possible values "true", "yes", "false", "no" (case insensitive).
+
+
+FUNCTION Configuration::LoadAndVerify()
+
+Loads the configuration from disc, and verifies it. If there are problems with the verification, it returns some text which can be used to tell the user the problems with the file. These are fairly basic error messages, but do say what should be done to get it to parse properly.
+
+This returns a Configuration object, which can then be queries for Keys and Values. Sub-configurations are implemented through an interface which returns a reference to another Configuration object.
+
+
+
+FUNCTION Configuration::KeyExists()
+
+Does a specified key exist?
+
+Use this for optional key values only -- specify them in the verification structure if they are required.
+
+
+FUNCTION Configuration::GetKeyValue()
+
+Get the value of a key as a string.
+
+If ConfigTest_MultiValueAllowed is set in the relevant entry in the verify structure, this string may contain multiple values, separated by a single byte with value 0x01 (use Configuration::MultiValueSeparator in code). Use SplitString() defined in Utils.h to split it into components.
+
+This representation was chosen as multi-values are probably rare, and unlikely to justify writing a nicer (but more memory intensive and complex) solution.
+
+
+FUNCTION Configuration::GetKeyValueInt()
+
+Get the value of a key as an integer. Make sure the AsInt property is requried in the verification structure.
+
+
+FUNCTION Configuration::GetKeyValueBool()
+
+Get the value of a key as a boolean value. Make sure the AsBool property is requried in the verification structure.
+
+Default to "false", should this verification not be performed and an unknown value is specified.
+
+
+FUNCTION Configuration::GetKeyNames()
+
+Return a list of all the keys in this configuration.
+
+
+FUNCTION Configuration::SubConfigurationExists()
+
+Does a specified sub-configuration exist?
+
+
+FUNCTION Configuration::GetSubConfiguration()
+
+Return another Configuration object representing the sub section.
+
+
+FUNCTION Configuration::GetSubConfigurationNames()
+
+Get a list of all the sub configurations.
+
+As there isn't a particularly neat way that configurations can be iterated over, mSubConfigurations is public. (BAD!)
+
+
diff --git a/docs/common/lib_common/Conversion.txt b/docs/common/lib_common/Conversion.txt
new file mode 100644
index 00000000..62d1967a
--- /dev/null
+++ b/docs/common/lib_common/Conversion.txt
@@ -0,0 +1,14 @@
+TITLE Generic type conversion
+
+Conversion.h provides generic type conversion. Within the BoxConvert namespace, it defines the templated function
+
+ ToType Convert<ToType, FromType>(FromType From)
+
+which converts the data type as expected. In general, from and to types have to be specified explicitly.
+
+Templates rather than overloaded functions are used, firstly to be absolutely explicit about the conversion required, and secondly because overloaded functions can't have differing return types for the same argument type.
+
+The function is specialised for various types, and the generic version uses C++ type conversion.
+
+Exceptions may be thrown if the conversion is not possible. These are all of the ConversionException type.
+
diff --git a/docs/common/lib_common/ExcludeList.txt b/docs/common/lib_common/ExcludeList.txt
new file mode 100755
index 00000000..8a5bf36c
--- /dev/null
+++ b/docs/common/lib_common/ExcludeList.txt
@@ -0,0 +1,21 @@
+TITLE ExcludeList
+
+A class implementing a list of strings which are to be excluded in some way. Entries can be added as strings to be matched exactly, or as regular expressions.
+
+Multiple entries can be added in a single function call in a manner designed to work with the multi-value entries store in a Configuation object.
+
+
+FUNCTION ExcludeList::AddDefiniteEntries()
+
+Definite entries are exact strings to match.
+
+
+FUNCTION ExcludeList::AddRegexEntries()
+
+Regular expressions as defined by POSIX, and supported by the host platform. Will throw an exception if regular expressions are not supported by the platform.
+
+
+FUNCTION ExcludeList::IsExcluded()
+
+Test a string for being excluded by definite or regex entries.
+
diff --git a/docs/common/lib_common/FdGetLine.txt b/docs/common/lib_common/FdGetLine.txt
new file mode 100644
index 00000000..d92ff94d
--- /dev/null
+++ b/docs/common/lib_common/FdGetLine.txt
@@ -0,0 +1,11 @@
+CLASS FdGetLine
+
+See IOStreamGetLine, difference is that it works on basic UNIX file descriptors rather than full blown streams. Follows basic interface, except...
+
+
+FUNCTION FdGetLine::GetLine
+
+Returns a string containing the optionally preprocessed line.
+
+Do not call if IsEOF if true.
+
diff --git a/docs/common/lib_common/Guards.txt b/docs/common/lib_common/Guards.txt
new file mode 100644
index 00000000..6174fea6
--- /dev/null
+++ b/docs/common/lib_common/Guards.txt
@@ -0,0 +1,5 @@
+TITLE Guards.h
+
+This file contains a couple of classes for using file and memory. When the class goes out of scope, the underlying object is closed or freed, respectively.
+
+
diff --git a/docs/common/lib_common/IOStream.txt b/docs/common/lib_common/IOStream.txt
new file mode 100644
index 00000000..09460656
--- /dev/null
+++ b/docs/common/lib_common/IOStream.txt
@@ -0,0 +1,89 @@
+CLASS IOStream
+
+The base class for streams of data. See IOStream.h for specific details on functions, but the interface is described here.
+
+Most streams only implement one direction, but some both.
+
+
+SUBTITLE Reading
+
+
+FUNCTION IOStream::Read()
+
+Reads data from the stream. Returns 0 if there is no data available within the timeout requested.
+
+Unlike the UNIX API, a return of 0 does not mean that there is no more data to read. Use IOStream::StreamDataLeft() to find this out.
+
+
+FUNCTION IOStream::ReadFullBuffer()
+
+If you want to read a specific sized block of data from a stream, it is annoying ot use the Read function, because it might return less, so you have to loop.
+
+This implements that loop. Note that the timeout is the timeout for each individual read -- it might take a lot longer to complete.
+
+You must check the return value, which is whether or not it managed to get all that data.
+
+
+FUNCTION IOStream::BytesLeftToRead()
+
+How many bytes are left to read in the stream treated as a whole. May return IOStream::SizeOfStreamUnknown if it is something like a socket which doesn't know how much data is in the stream.
+
+
+FUNCTION IOStream::StreamDataLeft()
+
+Returns true if more data can be read. Or more specifically, if more data might be able to be read some time in the future.
+
+
+SUBTITLE Writing
+
+
+FUNCTION IOStream::Write()
+
+Write data to a stream. Writes the entire block, or exceptions.
+
+
+FUNCTION IOStream::WriteAllBuffered()
+
+Writes any buffered data to the underlying object.
+
+Call at the end of a series of writes to make sure the data is actually written. (Buffering is not yet implemented anywhere, but it should be soon.)
+
+
+FUNCTION IOStream::StreamClosed()
+
+Is the stream closed, and writing no longer possible?
+
+
+FUNCTION IOStream::CopyStreamTo()
+
+A utility function to copy the contents of one stream to another, until the reading stream has no data left.
+
+
+SUBTITLE Other interfaces
+
+These are slightly nasty interfaces, because they have to match the UNIX API to some extent.
+
+
+FUNCTION IOStream::Close()
+
+Close the stream -- intended to indicate that nothing more will be written.
+
+However, also closes reading in most cases. Stream dependent. Probably best just to delete the object and let it sort itself out.
+
+
+FUNCTION IOStream::Seek()
+
+Seeks within the stream -- mainly for using files within a stream interface, although some of the utility stream classes support it too.
+
+This is actually a bad interface, because it does not specify whether it applies to reading or writing. This matches the interface provided by files with a single file pointer, but does not map well onto other stream types.
+
+Should it be changed? Perhaps. But then it means that files would either have to have an inconsitent interface, or the class keep track of two separate file pointers. Which isn't nice.
+
+So use carefully, and remember whether you're using a file stream, a reading stream, or a writing stream.
+
+
+FUNCTION IOStream::GetPosition()
+
+Get the current file pointer. Subject to same problems as Seek with regard to semantics.
+
+
diff --git a/docs/common/lib_common/IOStreamGetLine.txt b/docs/common/lib_common/IOStreamGetLine.txt
new file mode 100644
index 00000000..04c56b57
--- /dev/null
+++ b/docs/common/lib_common/IOStreamGetLine.txt
@@ -0,0 +1,29 @@
+CLASS IOStreamGetLine
+
+This class provides a convenient way to read text from a file, line by line. It also can preprocess the line to remove leading and trailing whitespace and comments. Comments are started by the character # and run to the end of the line.
+
+Create an instance by passing a reference to a stream into the constructor.
+
+Note the class does internal buffering, so you can only detach it later if the stream supports seeking backwards.
+
+
+FUNCTION IOStreamGetLine::GetLine()
+
+Returns true if a line could be retreieved without a read timing out.
+
+
+FUNCTION IOStreamGetLine::IsEOF()
+
+Whether the end of the stream has been reached. Do not call GetLine if this is true.
+
+
+FUNCTION IOStreamGetLine::GetLineNumber()
+
+Returns the line number.
+
+
+FUNCTION IOStreamGetLine::DetachFile()
+
+Detaches the stream from the GetLine class. Will seek backwards to "replace" data it's buffered back in the stream.
+
+
diff --git a/docs/common/lib_common/MainHelper.txt b/docs/common/lib_common/MainHelper.txt
new file mode 100644
index 00000000..eb5b07f0
--- /dev/null
+++ b/docs/common/lib_common/MainHelper.txt
@@ -0,0 +1,4 @@
+TITLE MainHelper.h
+
+This header contains a couple of macros which add exception handling and reporting to main() functions for executable programs.
+
diff --git a/docs/common/lib_common/WaitForEvent.txt b/docs/common/lib_common/WaitForEvent.txt
new file mode 100644
index 00000000..0bc55726
--- /dev/null
+++ b/docs/common/lib_common/WaitForEvent.txt
@@ -0,0 +1,16 @@
+CLASS WaitForEvent
+
+This class implements a way to efficiently wait on one or more system objects, for example sockets and file descriptors. Where kqueue() is available, this is used, otherwise poll(). The poll() implementation is less comprehensive and rather more limited.
+
+To add a compatible object, call Add(). To remove it, call Remove(). To wait for an object to become ready, call Wait(), which returns a pointer to the first ready object, or 0 for a timeout.
+
+The timeout is set either in the constructor, or using the SetTimout() method. It is specified in milliseconds.
+
+The kqueue based implementation will automatically remove objects when they are closed (actually, the OS does this), but the poll implementation requires that Remove() be called.
+
+The flags passed to Add() or Remove() are passed to the object, and are ignored by this class.
+
+For an object to be able to be added to the list, it should implement FillInKEvent() and FillInPoll() for kqueue and poll implementations respectively. Use the PLATFORM_KQUEUE_NOT_SUPPORTED define to test which is necessary for the platform.
+
+It does not have to be derived off any particular class, as it uses templates to be compatible with any class.
+
diff --git a/docs/common/lib_common/xStream.txt b/docs/common/lib_common/xStream.txt
new file mode 100644
index 00000000..91e9c0ea
--- /dev/null
+++ b/docs/common/lib_common/xStream.txt
@@ -0,0 +1,40 @@
+TITLE Various stream types
+
+Some useful streams are implemented in lib/common.
+
+
+SUBTITLE CollectInBufferStream
+
+Described in it's own page.
+
+
+SUBTITLE FileStream
+
+Implements a stream from a file, allowing both reading and writing.
+
+
+SUBTITLE MemBlockStream
+
+Turns a memory block into a readable stream.
+
+Can also be constructed using StreamableMemBlock, CollectInBufferStream and MemBlockStream as sources of the buffer.
+
+
+SUBTITLE PartialReadStream
+
+Create a readable stream which will read a set number of bytes from another stream, and then declare itself as closed.
+
+Useful for extracting a small chunk of another stream to present to a function which expects to consume all of a stream.
+
+
+SUBTITLE ReadGatherStream
+
+Create a readable stream out of blocks from many streams -- so various sections of any number of streams are composed into a single stream.
+
+To use, register each stream with AddComponent, then use the returned 'handle' with AddBlock to add blocks to the composed stream.
+
+Optionally, the object will take ownership of the streams added, and delete them when itself is deleted.
+
+See the comments in the function blocks in the cpp file for more info.
+
+
diff --git a/docs/common/lib_compress.txt b/docs/common/lib_compress.txt
new file mode 100644
index 00000000..f3e26f20
--- /dev/null
+++ b/docs/common/lib_compress.txt
@@ -0,0 +1,8 @@
+TITLE lib/compress
+
+This is a horrid C++ interface to zlib. It is written this way to avoid having to buffer any data, and so be unnecessarily inefficient. But this does end up just being a wrapper to the zlib interface.
+
+See test/compress for an example of how to use it. But it should be very familiar.
+
+For an easier interface, use CompressStream.
+
diff --git a/docs/common/lib_compress/CompressStream.txt b/docs/common/lib_compress/CompressStream.txt
new file mode 100644
index 00000000..eca43eb6
--- /dev/null
+++ b/docs/common/lib_compress/CompressStream.txt
@@ -0,0 +1,27 @@
+TITLE CompressStream
+
+This implements a compressing stream class, which compresses and decompresses data sent to an underlying stream object. Data is likely to be buffered.
+
+
+WARNING: Use WriteAllBuffered() (after objects which need to be sent in their entirity) and Close() (after you have finished writing to the stream) otherwise the compression buffering will lose the last bit of data for you.
+
+
+The class works as a filter to an existing stream. Ownership can optionally be taken so that the source stream is deleted when this object is deleted.
+
+It can operate in one or two directions at any time. Data written to the stream is compressed, data read from the stream is decompressed. If a direction is not being (de)compressed, then it can act as a pass-through without touching the data.
+
+The constructor specifies the actions to be taken:
+
+CompressStream(IOStream *pStream, bool TakeOwnership,
+ bool DecompressRead, bool CompressWrite,
+ bool PassThroughWhenNotCompressed = false);
+
+pStream - stream to filter
+TakeOwnership - if true, delete the stream when this object is deleted.
+DecompressRead - reads pass through a decompress filter
+CompressWrite - writes pass through a compression filter
+PassThroughWhenNotCompressed - if not filtering a direction, pass through the data unaltered, otherwise exception if an attempt is made on that direction.
+
+
+Note that the buffering and compression will result in different behaviour than the underlying stream: data may not be written in one go, and data being read may come out in different sized chunks.
+
diff --git a/docs/common/lib_crypto.txt b/docs/common/lib_crypto.txt
new file mode 100644
index 00000000..9ddafe69
--- /dev/null
+++ b/docs/common/lib_crypto.txt
@@ -0,0 +1,28 @@
+TITLE lib/crypto
+
+Provides cryptographic primatives using the OpenSSL implementation.
+
+
+SUBTITLE CipherContexts and CipherDescriptions
+
+A CipherContext is the interface to encryption and decryption, providing a generic interface.
+
+It is constructed with a decrypt or encrypt flag, and a CipherDescription. The CipherDescription specifies which cipher is to be used, the key, and all other cipher specific paramteres.
+
+The CipherContext has support for padding and initialisation vectors.
+
+
+SUBTITLE Random numbers
+
+An interface to the OpenSSL psuedo random number generater is provided.
+
+
+SUBTITLE Digests
+
+An interface to the MD5 digest is provided.
+
+
+SUBTITLE RollingChecksum
+
+The rsync rolling checksum is implemented. The interface is a little tricky to be efficient as the caller must track the position and provide relevant bytes to advance the checksum.
+
diff --git a/docs/common/lib_crypto/CipherContext.txt b/docs/common/lib_crypto/CipherContext.txt
new file mode 100644
index 00000000..30fb4608
--- /dev/null
+++ b/docs/common/lib_crypto/CipherContext.txt
@@ -0,0 +1,28 @@
+CLASS CipherContext
+
+Encryption and decryption using OpenSSL EVP interface.
+
+See the cpp file for more documentation in the function headers, and test/crypto for examples.
+
+General notes below.
+
+
+SUBTITLE Construction
+
+Construct with the encryption direction, and a CipherDescription of the cipher and key required.
+
+
+SUBTITLE Encrypting or decrypting
+
+Begin() and Transform() allow piece by piece transformation of a block.
+
+TransformBlock() transforms an entire block.
+
+
+SUBTITLE Buffering
+
+All transforms expect to have enough space in the buffers for their entire output. Because of block boundaries and padding, it is necessary that the output buffer should be bigger than the input buffer. The amount of space depends on the cipher.
+
+InSizeForOutBufferSize() and MaxOutSizeForInBufferSize() perform these space calculations, returning the maximuim in size for a specified out size, and the reverse, respectively.
+
+
diff --git a/docs/common/lib_crypto/RollingChecksum.txt b/docs/common/lib_crypto/RollingChecksum.txt
new file mode 100644
index 00000000..cbee1454
--- /dev/null
+++ b/docs/common/lib_crypto/RollingChecksum.txt
@@ -0,0 +1,32 @@
+CLASS RollingChecksum
+
+Implementing the rsync rolling checksum algorithm. Read it's description first:
+
+http://samba.anu.edu.au/rsync/tech_report/node3.html
+
+
+SUBTITLE Construction and initial checksum calculation
+
+The constructor takes a pointer to a block of data and a size, and calculates the checksum of this block. It can now be "rolled forward" to find the checksum of the block of the same size, one byte forward, with minimal calculation.
+
+
+FUNCTION RollingChecksum::GetChecksum()
+
+Returns the checksum for the current block.
+
+
+FUNCTION RollingChecksum::RollForward()
+
+This function takes the byte at the start of the current block, and the last byte of the block it's rolling forward to, and moves the checksum on.
+
+If the block is
+
+ char *pBlock = <something>;
+
+with size s, then it should be called with
+
+ RollForward(pBlock[0], pBlock[s])
+
+and now GetChecksum will return the checksum of the block (pBlock+1) of size s.
+
+
diff --git a/docs/common/lib_server.txt b/docs/common/lib_server.txt
new file mode 100644
index 00000000..392f331d
--- /dev/null
+++ b/docs/common/lib_server.txt
@@ -0,0 +1,9 @@
+TITLE lib/server
+
+This provides various classes for implementing client/server applications (and UNIX daemons).
+
+The stars of the show are ServerTLS and Protocol, which allow client server applications which communicate using TLS (SSL successor) to be built with surprisingly few lines of code.
+
+All details in the respective class files.
+
+Look at test/basicserver for examples.
diff --git a/docs/common/lib_server/Daemon.txt b/docs/common/lib_server/Daemon.txt
new file mode 100644
index 00000000..6956ec2b
--- /dev/null
+++ b/docs/common/lib_server/Daemon.txt
@@ -0,0 +1,96 @@
+CLASS Daemon
+
+Implement a UNIX daemon which
+
+* Daemonises (detach from console, etc)
+* Sets up signal handlers, and does useful things with them
+* Reads configuration files
+* Writes PID file
+* Opens syslog
+
+all the usual UNIX basic daemon behaviour.
+
+The daemon exe takes optional arguments: The first is a configuration file filename which overrides the default. If the second argument is 'SINGLEPROCESS' the daemon will not daemonise.
+
+The configuration file must have a section like this
+
+Server
+{
+ PidFile = /var/run/daemon.pid
+ User = username
+}
+
+Use DAEMON_VERIFY_SERVER_KEYS (defined in Daemon.h) to include the necessary keys in your configuration file's verify structure.
+
+The "User" line is optional, and if it is present the Daemon will change user to this username just before it daemonises. Note that unless the directory the PID file is written to has write permission for this user, the PID file will not be deleted on exit.
+
+
+To implement a daemon, derive a class from Daemon, and override Run().
+
+Then in your main file, do something like
+
+int main(int argc, const char *argv[])
+{
+ MAINHELPER_START
+
+ BackupDaemon daemon;
+ return daemon.Main(BOX_FILE_BBACKUPD_DEFAULT_CONFIG, argc, argv);
+
+ MAINHELPER_END
+}
+
+and that's it. Obviously it's worth doing a few more things as well, but that's it.
+
+
+FUNCTION Daemon::Run()
+
+Override with the main daemon code. It should behave like this
+
+void SomethingDaemon::Run()
+{
+ // Read configuration file
+
+ // Do lots of work until StopRun() returns true
+ while(!StopRun())
+ {
+ ::sleep(10);
+ }
+
+ // Clean up nicely
+}
+
+which allows the base class to implement the standard start, terminate and -HUP behaviours correctly.
+
+
+FUNCTION Daemon::DaemonName()
+
+Returns the name of the daemon, for use in syslog.
+
+
+FUNCTION Daemon::DaemonBanner()
+
+Returns the banner to be displayed on startup, or 0 for no banner.
+
+
+FUNCTION Daemon::GetConfigVerify()
+
+Returns a configuration verify structure for verifying the config file. Note that this configuration structure should include a sub-configuration called "Server, and have entries defined by DAEMON_VERIFY_SERVER_KEYS. See one of the bin/bbackupd for an example.
+
+
+FUNCTION Daemon::StopRun()
+
+Returns true when the daemon needs to be terminated or restarted. Use IsReloadConfigWanted() and IsTerminateWanted() to find out which one, if you need to know.
+
+
+FUNCTION Daemon::SetupInInitialProcess()
+
+Override to perform additional functions in the initial process, before forking and detachment happens.
+
+
+FUNCTION Daemon::EnterChild()
+
+Called when a child is entered. If you override it, remember to call the base class.
+
+
+
+
diff --git a/docs/common/lib_server/Protocol.txt b/docs/common/lib_server/Protocol.txt
new file mode 100644
index 00000000..09d3c1f1
--- /dev/null
+++ b/docs/common/lib_server/Protocol.txt
@@ -0,0 +1,120 @@
+CLASS Protocol
+
+Protocol
+
+* serialises and deserialises data objects
+* sends arbitary streams
+
+through a bi-directional IOStream object, usually a socket.
+
+These data objects are auto-generated by a perl script, along with the logic to implement a simple protocol where one end is a client, and the other a server. The client sends a command object (and optional streams) and the server returns a reply object (and optional streams).
+
+The perl script uses a description file which specifies the data inside these objects (which can be extended to include any type which implements the standard serialisation functions), the required responses to the commands, and whether streams are involved.
+
+It then implements a server object, which given a stream and a user defined context object, waits for commands, processes them, and sends the results back. All you need to do is implement a DoCommand() function for each command object you define.
+
+On the client side, a Query() function is implemented which takes objects as parameters, and returns the right type of reply object (or throws an exception if it doesn't get it.) Short cut Query<CommandName>() functions are also implemented which don't require objects to be created manually.
+
+Thus, implementing a server is as simple as deriving a daemon off ServerStream or ServerTLS, and implementing the Connection() function as
+
+ void TestProtocolServer::Connection(SocketStream &rStream)
+ {
+ TestProtocolServer server(rStream);
+ TestContext context;
+ server.DoServer(context);
+ }
+
+and that's it. TestContext is a user defined class which keeps track of all the state of the connection, and is passed to all the DoCommand() functions, which look like this:
+
+ std::auto_ptr<ProtocolObject>
+ TestProtocolServerSimple::DoCommand(TestProtocolServer &rProtocol,
+ TestContext &rContext)
+ {
+ return std::auto_ptr<ProtocolObject>
+ (new TestProtocolServerSimpleReply(mValue+1));
+ }
+
+(taken from test/basicserver)
+
+The client code looks like this
+
+ SocketStream conn;
+ conn.Open(Socket::TypeUNIX, "testfiles/srv4.sock");
+
+ TestProtocolClient protocol(conn);
+
+ // Query
+ {
+ std::auto_ptr<TestProtocolClientSimpleReply>
+ reply(protocol.QuerySimple(41));
+ TEST_THAT(reply->GetValuePlusOne() == 42);
+ }
+
+
+Finally, debug logging can be generated which allows a list of all commands and their parameters to be logged to syslog or a file.
+
+
+SUBTITLE Protocol Description File
+
+This file is passed to the lib/server/makeprotocol.pl script, which generates a h and cpp file to implement the protocol.
+
+It is in two sections, separated by a 'BEGIN_OBJECTS' on a line of it's own.
+
+In the top half, the following statements must be made.
+
+Name <name>
+ The name of the protocol, used in naming classes.
+
+IdentString <string>
+ The idenfitifaction string sent over the IOStream to confirm it it
+ is talking to another Protocol object speaking the same Protocol.
+
+ServerContextClass <class-name> <header-file>
+ The user defined context class used for the server, and the header
+ file it is defined in.
+
+Additionally, the following optional commands can be made.
+
+ClientType <description-typename> <C++ typename> <headerfile>
+ServerType (similarly)
+ Extends the types used in the objects below. Server and client
+ can use different types for the same object type.
+
+ImplementLog (Client|Server) (syslog|file)
+ Implement command logging for client or server into syslog or a file.
+
+LogTypeToText (Client|Server) <description-typename> <printf-element>
+ <evaluate>
+ For extended types, optionally define how to convert them into printf
+ elements and the code to run to get the argument. Within the evaluate
+ parameter, VAR is replaced by the name of the variable to display.
+ If this is not specified for a given type, OPAQUE is output instead.
+
+
+In the object section, an object is defined by a line
+
+<name> <id number> <attributes>
+
+followed by lines beginning with whitespace defining the data transmitted in the object. The type may be list<type>, which specifies a list (implemented as a std::vector) of entries of that type.
+
+The attributes specify exactly how that object is used in the defined protocol.
+
+Reply
+ The object is a reply object, sent from the server to the client.
+
+Command(Reply-Type)
+ The object is a command, send from the client to the server, and the server
+ will send back an object of type Reply-Type.
+
+IsError(Type-Field,SubType-Field)
+ The command is an error object, and the two files specify the data member
+ which describes the error type and sub type.
+
+EndsConversation
+ When this command is received, the connection is to be terminated.
+ (ie a logout command)
+
+StreamWithCommand
+ When this command is sent as a command, a stream follows it.
+
+
diff --git a/docs/common/lib_server/ServerStream.txt b/docs/common/lib_server/ServerStream.txt
new file mode 100644
index 00000000..6c5932a0
--- /dev/null
+++ b/docs/common/lib_server/ServerStream.txt
@@ -0,0 +1,29 @@
+CLASS ServerStream
+
+ServerStream implementes a Daemon which accepts stream connections over sockets, and forks into a child process to handle them.
+
+To implement a daemon, derive from
+
+ ServerStream<SocketStream, SERVER_LISTEN_PORT>
+
+The type SocketStream specifies that incoming connections should be treated as normal sockets, and SERVER_LISTEN_PORT is the port to listen to (if it's a inet socket, and if not overridden in the config file). The actual addresses (or names) to bind to are specified in the configuration file by the user.
+
+Make sure SERVERSTREAM_VERIFY_SERVER_KEYS(0) is included in the configuration verification structure in the "Server" sub configuration. 0 could be replaced with a default address, for example "unix:/var/run/server.sock" to specific a default UNIX socket in the filesystem.
+
+See test/basicserver for a simple example.
+
+The ListenAddresses key in the Server subconfiguration is a comma separated list of addresses, specified as family:name. Internet sockets are family 'inet', for example 'inet:localhost' (or 'inet:localhost:1080' to specify a port number as well), and unix domain sockets are 'unix', example above.
+
+
+Override Connection to handle the connection.
+
+Remember to override Daemon functions like the server name, and start it up, just like a generic Daemon.
+
+
+FUNCTION ServerStream::Connection
+
+This function takes a connected stream as it's argument. It should then proceed to do whatever it needs to do to talk to the client.
+
+Using IOStreamGetLine or a Protocol class to communicate may be quick ways of implementing this functionality.
+
+
diff --git a/docs/common/lib_server/ServerTLS.txt b/docs/common/lib_server/ServerTLS.txt
new file mode 100644
index 00000000..dbde500f
--- /dev/null
+++ b/docs/common/lib_server/ServerTLS.txt
@@ -0,0 +1,6 @@
+CLASS ServerTLS
+
+Implements a server which uses TLS (SSL) to encrypt and authenticate connections.
+
+Very similar to ServerStream, except it reads the certificate files for the TLSContext out of the Server sub-configuration to set up a TLSContext ("CertificateFile", "PrivateKeyFile" and "TrustedCAsFile"). Otherwise works exactly the same.
+
diff --git a/docs/common/lib_server/SocketStream.txt b/docs/common/lib_server/SocketStream.txt
new file mode 100644
index 00000000..82813279
--- /dev/null
+++ b/docs/common/lib_server/SocketStream.txt
@@ -0,0 +1,8 @@
+CLASS SocketStream
+
+A implementation of IOStream which wraps a socket connection.
+
+It can either be created by attaching to an existing object, or use the Open() function to open a connection to a named host on a specific port (or a local UNIX socket in the filesystem).
+
+Follows stream interface.
+
diff --git a/docs/common/lib_server/SocketStreamTLS.txt b/docs/common/lib_server/SocketStreamTLS.txt
new file mode 100644
index 00000000..ebb3f233
--- /dev/null
+++ b/docs/common/lib_server/SocketStreamTLS.txt
@@ -0,0 +1,11 @@
+CLASS SocketStreamTLS
+
+An implementation of IOStream which wraps a TLS (SSL) connection over a socket.
+
+The Open function takes a TLSContext reference which specifies the parameters for the connection.
+
+
+FUNCTION GetPeerCommonName()
+
+Returns the common name of the certificate presented by the remote end.
+
diff --git a/docs/common/lib_server/TLSContext.txt b/docs/common/lib_server/TLSContext.txt
new file mode 100644
index 00000000..ff50d3e6
--- /dev/null
+++ b/docs/common/lib_server/TLSContext.txt
@@ -0,0 +1,16 @@
+CLASS TLSContext
+
+A wrapper over the OpenSSL context object.
+
+Note: you need to call SSLLib::Initialise at the beginning of your program to use these functions.
+
+
+SUBTITLE Construction
+
+The constuctor takes the following parameters
+
+* Boolean for whether this is acting as a server or a client
+* The .pem file containing the certificate which will be used
+* The .pem file containing the private key for this certificate
+* The .pem file containing the certificates which will certify the other end of the connection.
+
diff --git a/docs/common/memory_leaks.txt b/docs/common/memory_leaks.txt
new file mode 100755
index 00000000..9a9764ea
--- /dev/null
+++ b/docs/common/memory_leaks.txt
@@ -0,0 +1,44 @@
+TITLE Memory leak detection
+
+The build system contains a primative memory leak detection system in debug builds.
+
+It works by using #defines to replace calls to malloc, free, realloc, new and delete with debug versions which record their use. When a process ends, it dumps a list of all the blocks or objects which were allocated by not freed, and the file and line of the source where they are originally allocated.
+
+It's not perfect, but should catch most things and work on most platforms.
+
+If it doesn't work on your platform, define PLATFORM_DISABLE_MEM_LEAK_TESTING in BoxPlatform.h within the relevant section.
+
+
+SUBTITLE Requirements in source
+
+It does require some extra lines in the source file. The last included file in each .cpp file must be
+
+ #include "MemLeakFindOn.h"
+
+and if a .h file uses any of these functions, the contents of the .h file should be bounded with
+
+ #include "MemLeakFindOn.h"
+
+ // ... some code, but absolutely no #includes
+
+ #include "MemLeakFindOff.h"
+
+The cleanupforcvs.pl script checks for correct usage.
+
+
+SUBTITLE Use in tests
+
+Tests will automatically dump memory leaks and regard them as a failure for anything executing in the main test process.
+
+To test for memory leaks in programs run from the test, or daemons, use something like
+
+ TestRemoteProcessMemLeaks("bbackupd.memleaks");
+
+If memory leak detection is being used, it will check the specified file for memory leak reports (deleting it afterwards). Any leak is an error.
+
+The Daemon class automactically arranges for the memory leak reports to be written. Other exes should set this up themselves, preferably using the define in MainHelper.h, as the first thing in their main() function.
+
+ MAINHELPER_SETUP_MEMORY_LEAK_EXIT_REPORT(file, name)
+
+File is the filename to write the report to, conventionally called "<programname>.memleaks", and name is the name of the exe.
+