diff options
Diffstat (limited to 'doc/latex/manual.tex')
| -rw-r--r-- | doc/latex/manual.tex | 131 |
1 files changed, 79 insertions, 52 deletions
diff --git a/doc/latex/manual.tex b/doc/latex/manual.tex index 8509c74..fdb46f7 100644 --- a/doc/latex/manual.tex +++ b/doc/latex/manual.tex @@ -12,8 +12,8 @@ \title{S3QL Documentation} -\date{December 06, 2011} -\release{1.8.1} +\date{January 21, 2012} +\release{1.9} \author{Nikolaus Rath} \newcommand{\sphinxlogo}{} \renewcommand{\releasename}{Release} @@ -237,20 +237,14 @@ so under Linux you should actually use \emph{2.6.26 or newer whenever possible}. \item {} -\href{http://www.python.org/}{Python}, version 2.6.6 or newer, but not -Python 3.x. Make sure to also install the development headers. +\href{http://www.python.org/}{Python} 2.7 (but not Python 3.x). Make +sure to also install the development headers. \item {} The \href{http://pypi.python.org/pypi/pycryptopp}{PyCrypto++ Python Module}. To check if this module is installed, try to execute \code{python -c 'import pycryptopp'}. \item {} -The \href{http://pypi.python.org/pypi/argparse}{argparse Python Module}. To check if this module is -installed, try to execute \code{python -c 'import argparse; print -argparse.\_\_version\_\_'}. If argparse is installed, this will print -the version number. You need version 1.1 or later. - -\item {} \href{http://www.sqlite.org/}{SQLite} version 3.7.0 or newer. SQLite has to be installed as a \emph{shared library} with development headers. @@ -430,8 +424,7 @@ On the plus side, if a backend looses or corrupts some of the stored data, S3QL \emph{will} detect the problem. Missing data will be detected when running \code{fsck.s3ql} or when attempting to access the data in the mounted file system. In the later case you will get an IO Error, and -on unmounting S3QL will warn you that the file system is damaged and -you need to run \code{fsck.s3ql}. +the S3QL mount point will become inaccessible. \code{fsck.s3ql} will report all the affected files and move them into the \code{/lost+found} directory of the file system. @@ -482,7 +475,8 @@ will ever be a concern. \chapter{Storage Backends} \label{backends:id1}\label{backends::doc}\label{backends:storage-backends} -The following backends are currently available in S3QL: +All storage backends respect the \code{http\_proxy} and \code{https\_proxy} +environment variables. \section{Google Storage} @@ -624,13 +618,57 @@ consistency guarantees that completely eliminate any of the described problems. +\section{OpenStack/Swift} +\label{backends:openstack-swift} +\href{http://www.openstack.org/}{OpenStack} is an open-source cloud server application suite. \href{http://openstack.org/projects/storage/}{Swift} is +the cloud storage module of OpenStack. Swift/OpenStack storage is +offered by many different companies. + +The storage URL for the OpenStack backend has the form + +\begin{Verbatim}[commandchars=\\\{\}] +\PYG{l}{swift://}\PYG{n+nv}{\textless{}hostname\textgreater{}}\PYG{g+ge}{[:\textless{}port\textgreater{}]}\PYG{l}{/}\PYG{n+nv}{\textless{}container\textgreater{}}\PYG{g+ge}{[/\textless{}prefix\textgreater{}]} +\end{Verbatim} + +Note that the storage container must already exist. Most OpenStack +providers offer a web frontend that you can use to create storage +containers. \emph{prefix} can be an arbitrary prefix that will be prepended +to all object names used by S3QL. This allows you to store several +S3QL file systems in the same container. + +The OpenStack backend always uses HTTPS connections. Note, however, +that at this point S3QL does not verify the server certificate (cf. +\href{http://code.google.com/p/s3ql/issues/detail?id=267}{issue 267}). + + +\section{RackSpace CloudFiles} +\label{backends:rackspace-cloudfiles}\label{backends:swift} +\href{http://www.rackspace.com/}{RackSpace} CloudFiles uses OpenStack internally, so you can use the +OpenStack/Swift backend (see above). The hostname for CloudFiles +containers is \code{auth.api.rackspacecloud.com}. Use your normal +RackSpace user name for the backend login, and your RackSpace API key +as the backend passphrase. You can create a storage container for S3QL +using the \href{https://manage.rackspacecloud.com/}{Control Panel} (go to +\emph{Cloud Files} under \emph{Hosting}). + +You should note that opinions about RackSpace differ widely among S3QL +users and developers. On the one hand, people praise RackSpace for +their backing of the (open source) OpenStack project. On the other +hand, their heavily advertised ``fanatical support'' is in practice +often not only \href{http://code.google.com/p/s3ql/issues/detail?id=243\#c5}{less than helpful}, but their +support agents also seem to be \href{http://code.google.com/p/s3ql/issues/detail?id=243\#c11}{downright incompetent}. However, +there are reports that the support quality increases dramatically once +you are a customer and use the ``Live Chat'' link when you are logged +into the control panel. + + \section{S3 compatible} -\label{backends:s3-compatible} -S3QL is also able to access other, S3 compatible storage services like -\href{http://www.openstack.org/}{OpenStack} for which no specific backend exists. Note that when -accessing such services, only the lowest common denominator of -available features can be used, so it is generally recommended to use -a service specific backend instead. +\label{backends:s3-compatible}\label{backends:rackspace} +S3QL is also able to access other, S3 compatible storage services for +which no specific backend exists. Note that when accessing such +services, only the lowest common denominator of available features can +be used, so it is generally recommended to use a service specific +backend instead. The storage URL for accessing an arbitrary S3 compatible storage service is @@ -650,7 +688,7 @@ not verify the server certificate (cf. \href{http://code.google.com/p/s3ql/issue \section{Local} -\label{backends:openstack}\label{backends:local} +\label{backends:local} S3QL is also able to store its data on the local file system. This can be used to backup data on external media, or to access external services that S3QL can not talk to directly (e.g., it is possible to @@ -943,36 +981,20 @@ For maximum file system performance, the best algorithm therefore depends on your network connection speed: the compression algorithm should be fast enough to saturate your network connection. -To find the optimal algorithm for your system, S3QL ships with a -program called \code{benchmark.py} in the \code{contrib} directory. You should -run this program on a file that has a size that is roughly equal to -the block size of your file system and has similar contents. It will -then determine the compression speeds for the different algorithms and -the upload speeds for the specified backend and recommend the best -algorithm that is fast enough to saturate your network connection. +To find the optimal algorithm and number of parallel compression +threads for your system, S3QL ships with a program called +\code{benchmark.py} in the \code{contrib} directory. You should run this program +on a file that has a size that is roughly equal to the block size of +your file system and has similar contents. It will then determine the +compression speeds for the different algorithms and the upload speeds +for the specified backend and recommend the best algorithm that is +fast enough to saturate your network connection. Obviously you should make sure that there is little other system load when you run \code{benchmark.py} (i.e., don't compile software or encode videos at the same time). -\section{Parallel Compression} -\label{mount:parallel-compression} -If you are running S3QL on a system with multiple cores, you might -want to set the \code{-{-}threads} value larger than one. This will -instruct S3QL to compress and encrypt several blocks at the same time. - -If you want to do this in combination with using the LZMA compression -algorithm, you should keep an eye on memory usage though. Every -LZMA compression threads requires about 200 MB of RAM. - -\begin{notice}{note}{Note:} -To determine the optimal compression algorithm for your network -connection when using multiple threads, you can pass the -\code{-{-}compression-threads} option to \code{contrib/benchmark.py}. -\end{notice} - - \section{Notes about Caching} \label{mount:notes-about-caching} S3QL maintains a local cache of the file system data to speed up @@ -1029,6 +1051,9 @@ as follows (and should be placed in \code{/etc/init/}): \PYG{l}{env BUCKET="s3://my-backup-bla"} \PYG{l}{env MOUNTPOINT="/mnt/backup"} +\PYG{l}{env USER="myusername"} +\PYG{l}{env AUTHFILE="/path/to/authinfo2"} + \PYG{l}{expect stop} \PYG{l}{script} @@ -1041,12 +1066,14 @@ as follows (and should be placed in \code{/etc/init/}): \PYG{l}{ rm -rf "\PYGZdl{}DIR"} \PYG{l}{ \PYGZsh{} Check and mount file system} -\PYG{l}{ fsck.s3ql --batch "\PYGZdl{}BUCKET"} -\PYG{l}{ exec mount.s3ql --upstart "\PYGZdl{}BUCKET" "\PYGZdl{}MOUNTPOINT"} +\PYG{l}{ su -s /bin/sh -c 'exec "\PYGZdl{}0" "\PYGZdl{}@"' "\PYGZdl{}USER" -- \PYGZbs{}} +\PYG{l}{ fsck.s3ql --batch --authfile "\PYGZdl{}AUTHFILE" "\PYGZdl{}BUCKET"} +\PYG{l}{ exec su -s /bin/sh -c 'exec "\PYGZdl{}0" "\PYGZdl{}@"' "\PYGZdl{}USER" -- \PYGZbs{}} +\PYG{l}{ mount.s3ql --upstart --authfile "\PYGZdl{}AUTHFILE" "\PYGZdl{}BUCKET" "\PYGZdl{}MOUNTPOINT"} \PYG{l}{end script} \PYG{l}{pre-stop script} -\PYG{l}{ umount.s3ql "\PYGZdl{}MOUNTPOINT"} +\PYG{l}{ su -s /bin/sh -c 'exec "\PYGZdl{}0" "\PYGZdl{}@"' "\PYGZdl{}USER" -- umount.s3ql "\PYGZdl{}MOUNTPOINT"} \PYG{l}{end script} \end{Verbatim} @@ -1502,11 +1529,11 @@ For a full list of available options, run \textbf{expire\_backups.py --help}. -\section{s3ql.conf} -\label{contrib:s3ql-conf} -\code{s3ql.conf} is an example upstart job definition file. It defines a -job that automatically mounts an S3QL file system on system start, and -properly unmounts it when the system is shut down. +\section{s3ql\_upstart.conf} +\label{contrib:s3ql-upstart-conf} +\code{s3ql\_upstart.conf} is an example upstart job definition file. It +defines a job that automatically mounts an S3QL file system on system +start, and properly unmounts it when the system is shut down. \chapter{Tips \& Tricks} |