path: root/doc/latex/manual.tex

%% Generated by Sphinx.
\def\sphinxdocclass{report}
\documentclass[letterpaper,10pt,english]{sphinxmanual}
\ifdefined\pdfpxdimen
   \let\sphinxpxdimen\pdfpxdimen\else\newdimen\sphinxpxdimen
\fi \sphinxpxdimen=.75bp\relax

\PassOptionsToPackage{warn}{textcomp}
\usepackage[utf8]{inputenc}
\ifdefined\DeclareUnicodeCharacter
% support both utf8 and utf8x syntaxes
\edef\sphinxdqmaybe{\ifdefined\DeclareUnicodeCharacterAsOptional\string"\fi}
  \DeclareUnicodeCharacter{\sphinxdqmaybe00A0}{\nobreakspace}
  \DeclareUnicodeCharacter{\sphinxdqmaybe2500}{\sphinxunichar{2500}}
  \DeclareUnicodeCharacter{\sphinxdqmaybe2502}{\sphinxunichar{2502}}
  \DeclareUnicodeCharacter{\sphinxdqmaybe2514}{\sphinxunichar{2514}}
  \DeclareUnicodeCharacter{\sphinxdqmaybe251C}{\sphinxunichar{251C}}
  \DeclareUnicodeCharacter{\sphinxdqmaybe2572}{\textbackslash}
\fi
\usepackage{cmap}
\usepackage[T1]{fontenc}
\usepackage{amsmath,amssymb,amstext}
\usepackage{babel}
\usepackage{times}
\usepackage[Bjarne]{fncychap}
\usepackage{sphinx}

\fvset{fontsize=\small}
\usepackage{geometry}

% Include hyperref last.
\usepackage{hyperref}
% Fix anchor placement for figures with captions.
\usepackage{hypcap}% it must be loaded after hyperref.
% Set up styles of URL: it should be placed after hyperref.
\urlstyle{same}

\addto\captionsenglish{\renewcommand{\figurename}{Fig.\@ }}
\makeatletter
\def\fnum@figure{\figurename\thefigure{}}
\makeatother
\addto\captionsenglish{\renewcommand{\tablename}{Table }}
\makeatletter
\def\fnum@table{\tablename\thetable{}}
\makeatother
\addto\captionsenglish{\renewcommand{\literalblockname}{Listing}}

\addto\captionsenglish{\renewcommand{\literalblockcontinuedname}{continued from previous page}}
\addto\captionsenglish{\renewcommand{\literalblockcontinuesname}{continues on next page}}
\addto\captionsenglish{\renewcommand{\sphinxnonalphabeticalgroupname}{Non-alphabetical}}
\addto\captionsenglish{\renewcommand{\sphinxsymbolsname}{Symbols}}
\addto\captionsenglish{\renewcommand{\sphinxnumbersname}{Numbers}}

\addto\extrasenglish{\def\pageautorefname{page}}

\setcounter{tocdepth}{1}



\title{S3QL Documentation}
\date{Sep 08, 2019}
\release{3.3}
\author{Nikolaus Rath}
\newcommand{\sphinxlogo}{\vbox{}}
\renewcommand{\releasename}{Release}
\makeindex
\begin{document}

\pagestyle{empty}
\sphinxmaketitle
\pagestyle{plain}
\sphinxtableofcontents
\pagestyle{normal}
\phantomsection\label{\detokenize{index::doc}}



\chapter{S3QL}
\label{\detokenize{about:s3ql}}\label{\detokenize{about::doc}}
S3QL is a file system that stores all its data online using storage
services like \sphinxhref{http://code.google.com/apis/storage/}{Google Storage}, \sphinxhref{http://aws.amazon.com/s3}{Amazon S3}, or \sphinxhref{http://openstack.org/projects/storage/}{OpenStack}. S3QL
effectively provides a hard disk of dynamic, infinite capacity that
can be accessed from any computer with internet access.

S3QL is a standard conforming, full featured UNIX file system that is
conceptually indistinguishable from any local file system.
Furthermore, S3QL has additional features like compression,
encryption, data de-duplication, immutable trees and snapshotting
which make it especially suitable for online backup and archival.

S3QL is designed to favor simplicity and elegance over performance and
feature-creep. Care has been taken to make the source code as
readable and serviceable as possible. Solid error detection and error
handling have been included from the very first line, and S3QL comes
with extensive automated test cases for all its components.


\section{Features}
\label{\detokenize{about:features}}\begin{itemize}
\item {} 
\sphinxstylestrong{Transparency.} Conceptually, S3QL is indistinguishable from a
local file system. For example, it supports hardlinks, symlinks,
standard unix permissions, extended attributes and file
sizes up to 2 TB.

\item {} 
\sphinxstylestrong{Dynamic Size.} The size of an S3QL file system grows and shrinks
dynamically as required.

\item {} 
\sphinxstylestrong{Compression.} Before storage, all data may compressed with the
LZMA, bzip2 or deflate (gzip) algorithm.

\item {} 
\sphinxstylestrong{Encryption.} After compression (but before upload), all data can be
AES encrypted with a 256 bit key. An additional SHA256 HMAC checksum
is used to protect the data against manipulation.

\item {} 
\sphinxstylestrong{Data De-duplication.} If several files have identical contents,
the redundant data will be stored only once. This works across all
files stored in the file system, and also if only some parts of the
files are identical while other parts differ.

\item {} 
\sphinxstylestrong{Immutable Trees.} Directory trees can be made immutable, so that
their contents can no longer be changed in any way whatsoever. This
can be used to ensure that backups can not be modified after they
have been made.

\item {} 
\sphinxstylestrong{Copy-on-Write/Snapshotting.} S3QL can replicate entire directory
trees without using any additional storage space. Only if one of the
copies is modified, the part of the data that has been modified will
take up additional storage space. This can be used to create
intelligent snapshots that preserve the state of a directory at
different points in time using a minimum amount of space.

\item {} 
\sphinxstylestrong{High Performance independent of network latency.} All operations
that do not write or read file contents (like creating directories
or moving, renaming, and changing permissions of files and
directories) are very fast because they are carried out without any
network transactions.

S3QL achieves this by saving the entire file and directory structure
in a database. This database is locally cached and the remote
copy updated asynchronously.

\item {} 
\sphinxstylestrong{Support for low bandwidth connections.} S3QL splits file contents
into smaller blocks and caches blocks locally. This minimizes both
the number of network transactions required for reading and writing
data, and the amount of data that has to be transferred when only
parts of a file are read or written.

\end{itemize}


\section{Development Status}
\label{\detokenize{about:development-status}}
S3QL is considered stable and suitable for production use.  Starting
with version 2.17.1, S3QL uses semantic versioning. This means that
backwards-incompatible versions (e.g., versions that require an
upgrade of the file system revision) will be reflected in an increase
of the major version number.


\section{Supported Platforms}
\label{\detokenize{about:supported-platforms}}
S3QL is developed and tested under Linux. Users have also reported
running S3QL successfully on OS-X, FreeBSD and NetBSD. We try to
maintain compatibility with these systems, but (due to lack of
pre-release testers) we cannot guarantee that every release will run
on all non-Linux systems. Please report any bugs you find, and we will
try to fix them.


\section{Contributing}
\label{\detokenize{about:contributing}}
The S3QL source code is available on \sphinxhref{https://github.com/s3ql/main}{GitHub}.


\chapter{Installation}
\label{\detokenize{installation:installation}}\label{\detokenize{installation::doc}}
S3QL depends on several other programs and libraries that have to be
installed first. The best method to satisfy these dependencies depends
on your distribution.


\section{Dependencies}
\label{\detokenize{installation:dependencies}}
The following is a list of the programs and libraries required for
running S3QL. Generally, you should first check if your distribution
already provides a suitable packages and only install from source if
that is not the case.
\begin{itemize}
\item {} 
Kernel: Linux 2.6.9 or newer or FreeBSD with \sphinxhref{http://www.freshports.org/sysutils/fusefs-kmod/}{FUSE4BSD}. Starting with
kernel 2.6.26 you will get significantly better write performance,
so under Linux you should actually use \sphinxstyleemphasis{2.6.26 or newer whenever
possible}.

\item {} 
The \sphinxhref{http://psmisc.sf.net/}{psmisc} utilities.

\item {} 
\sphinxhref{http://www.sqlite.org/}{SQLite} version 3.7.0 or newer. SQLite
has to be installed as a \sphinxstyleemphasis{shared library} with development headers.

\item {} 
\sphinxhref{http://www.python.org/}{Python} 3.5.0 or newer. Make sure to also
install the development headers.

\item {} 
The following Python modules:
\begin{itemize}
\item {} 
\sphinxhref{https://pypi.python.org/pypi/setuptools}{setuptools}, version 1.0 or newer.

\item {} 
\sphinxhref{https://cryptography.io/en/latest/installation/}{cryptography}

\item {} 
\sphinxhref{https://pypi.python.org/pypi/defusedxml/}{defusedxml}

\item {} 
\sphinxhref{https://github.com/rogerbinns/apsw}{apsw}, version 3.7.0 or
newer.

\item {} 
\sphinxhref{https://pypi.org/project/llfuse/}{llfuse}, any
version between 1.0 (inclusive) and 2.0 (exclusive)

\item {} 
\sphinxhref{https://pypi.org/project/dugong/}{dugong}, any
version between 3.4 (inclusive) and 4.0 (exclusive)

\item {} 
\sphinxhref{http://pytest.org/}{pytest}, version 3.7 or newer (optional, to run unit tests)

\item {} 
\sphinxhref{https://github.com/systemd/python-systemd}{systemd} (optional,
for enabling systemd support). Do \sphinxstyleemphasis{not} install the module from
PyPi, this is from a third-party developer and incompatible with
the official module from the systemd developers.

\item {} 
\sphinxhref{https://pypi.python.org/pypi/requests/}{requests} (optional,
required for OAuth2 authentication with Google Storage)

\item {} 
\sphinxhref{https://pypi.python.org/project/google-auth/}{google-auth}
(optional, required for ADC authentication with Google Storage)

\item {} 
\sphinxhref{https://pypi.python.org/project/google-auth-oauthlib/}{google-auth-oauthlib}
(optional, required for browser-based authentication with Google Storage)

\end{itemize}

To check if a specific module \sphinxcode{\sphinxupquote{\textless{}module\textgreater{}}} is installed, execute
\sphinxcode{\sphinxupquote{python3 -c 'import \sphinxstyleemphasis{\textless{}module\textgreater{}};
print(\sphinxstyleemphasis{\textless{}module\textgreater{}}.\_\_version\_\_)'}}. This will result in an
\sphinxcode{\sphinxupquote{ImportError}} if the module is not installed, and will print the
installed version if the module is installed.

\end{itemize}


\section{Installing S3QL}
\label{\detokenize{installation:installing-s3ql}}\label{\detokenize{installation:inst-s3ql}}
To build and install S3QL itself, proceed as follows:
\begin{enumerate}
\def\theenumi{\arabic{enumi}}
\def\labelenumi{\theenumi .}
\makeatletter\def\p@enumii{\p@enumi \theenumi .}\makeatother
\item {} 
Download S3QL from \sphinxurl{https://github.com/s3ql/s3ql/releases}

\item {} 
Unpack it into a folder of your choice

\item {} 
Run \sphinxcode{\sphinxupquote{python3 setup.py build\_ext -{-}inplace}} to build S3QL.

\item {} 
Run \sphinxcode{\sphinxupquote{python3 -m pytest tests/}} to run a self-test. If this fails, ask
for help on the \sphinxhref{http://groups.google.com/group/s3ql}{mailing list} or report a bug in the
\sphinxhref{https://github.com/s3ql/s3ql/issues}{issue tracker}.

\end{enumerate}

Now you have three options:
\begin{itemize}
\item {} 
You can run the S3QL commands from the \sphinxcode{\sphinxupquote{bin/}} directory.

\item {} 
You can install S3QL system-wide for all users. To do that, you
have to run \sphinxcode{\sphinxupquote{sudo python3 setup.py install}}.

\item {} 
You can install S3QL into \sphinxcode{\sphinxupquote{\textasciitilde{}/.local}} by executing \sphinxcode{\sphinxupquote{python3
setup.py install -{-}user}}. In this case you should make sure that
\sphinxcode{\sphinxupquote{\textasciitilde{}/.local/bin}} is in your \sphinxcode{\sphinxupquote{\$PATH}} environment variable.

\end{itemize}


\section{Development Version}
\label{\detokenize{installation:development-version}}
If you have checked out the unstable development version from the
Mercurial repository, a bit more effort is required. You’ll also need:
\begin{itemize}
\item {} 
Version 0.24 or newer of the \sphinxhref{http://www.cython.org/}{Cython} compiler.

\item {} 
Version 1.2b1 or newer of the \sphinxhref{http://sphinx.pocoo.org/}{Sphinx} document processor.

\end{itemize}

With these additional dependencies installed, S3QL can be build and
tested with

\begin{sphinxVerbatim}[commandchars=\\\{\}]
\PYG{l}{python3 setup.py build\PYGZus{}cython}
\PYG{l}{python3 setup.py build\PYGZus{}ext \PYGZhy{}\PYGZhy{}inplace}
\PYG{l}{python3 \PYGZhy{}m pytest test}
\end{sphinxVerbatim}

Note that when building from the Mercurial or Git repository, building
and testing is done with several additional checks. This may cause
compilation and/or tests to fail even though there are no problems
with functionality. For example, any use of functions that are
scheduled for deprecation in future Python version will cause tests to
fail. If you would rather just check for functionality, you can delete
the \sphinxcode{\sphinxupquote{MANIFEST.in}} file. In that case, the build system will
behave as it does for a regular release.

The HTML and PDF documentation can be generated with

\begin{sphinxVerbatim}[commandchars=\\\{\}]
\PYG{l}{python3 setup.py build\PYGZus{}sphinx}
\end{sphinxVerbatim}

and S3QL can be installed as usual with

\begin{sphinxVerbatim}[commandchars=\\\{\}]
\PYG{l}{python3 setup.py install }\PYG{g+ge}{[\PYGZhy{}\PYGZhy{}user]}
\end{sphinxVerbatim}


\section{Running tests requiring remote servers}
\label{\detokenize{installation:running-tests-requiring-remote-servers}}
By default, tests requiring a connection to a remote storage backend
are skipped. If you would like to run these tests too (which is always
a good idea), you have to create additional entries in your
\sphinxcode{\sphinxupquote{\textasciitilde{}/.s3ql/authinfo2}} file that tell S3QL what server and credentials to
use for these tests. These entries have the following form:

\begin{sphinxVerbatim}[commandchars=\\\{\}]
\PYG{g+ge}{[\PYGZlt{}BACKEND\PYGZgt{}\PYGZhy{}test]}
\PYG{l}{backend\PYGZhy{}login: }\PYG{n+nv}{\PYGZlt{}user\PYGZgt{}}
\PYG{l}{backend\PYGZhy{}password: }\PYG{n+nv}{\PYGZlt{}password\PYGZgt{}}
\PYG{l}{test\PYGZhy{}fs: }\PYG{n+nv}{\PYGZlt{}storage\PYGZhy{}url\PYGZgt{}}
\end{sphinxVerbatim}

Here \sphinxstyleemphasis{\textless{}BACKEND\textgreater{}} specifies the backend that you want to test
(e.g. \sphinxstyleemphasis{s3}, \sphinxstyleemphasis{s3c}, \sphinxstyleemphasis{gs}, or \sphinxstyleemphasis{swift}), \sphinxstyleemphasis{\textless{}user\textgreater{}} and \sphinxstyleemphasis{\textless{}password\textgreater{}} are
the backend authentication credentials, and \sphinxstyleemphasis{\textless{}storage-url\textgreater{}} specifies
the full storage URL that will be used for testing. \sphinxstylestrong{Any existing
S3QL file system in this storage URL will be destroyed during
testing}.

For example, to run tests that need connection to a Google Storage
server, you would add something like

\begin{sphinxVerbatim}[commandchars=\\\{\}]
\PYG{g+ge}{[gs\PYGZhy{}test]}
\PYG{l}{backend\PYGZhy{}login: GOOGIGWLONT238MD7HZ4}
\PYG{l}{backend\PYGZhy{}password: rmEbstjscoeunt1249oes1298gauidbs3hl}
\PYG{l}{test\PYGZhy{}fs: gs://joes\PYGZhy{}gs\PYGZhy{}bucket/s3ql\PYGZus{}tests/}
\end{sphinxVerbatim}

On the next run of \sphinxcode{\sphinxupquote{runtest.py}} (or \sphinxcode{\sphinxupquote{py.test}} when using the
development version), the additional tests will be run. If the tests
are still skipped, you can get more information about why tests are
being skipped by passing the \sphinxcode{\sphinxupquote{-rs}} argument to
\sphinxcode{\sphinxupquote{runtest.py}}/\sphinxcode{\sphinxupquote{py.test}}.


\chapter{Storage Backends}
\label{\detokenize{backends:storage-backends}}\label{\detokenize{backends:id1}}\label{\detokenize{backends::doc}}
S3QL supports different \sphinxstyleemphasis{backends} to store data at different service
providers and using different protocols. A \sphinxstyleemphasis{storage url} specifies a
backend together with some backend-specific information and uniquely
identifies an S3QL file system. The form of the storage url depends on
the backend and is described for every backend below.

Furthermore, every S3QL commands that accepts a storage url also
accepts a \sphinxcode{\sphinxupquote{-{-}backend-options}} parameter than can be used to
pass backend-specific options to the backend module. The available
options are documented with the respective backends below.

All storage backends respect the \sphinxcode{\sphinxupquote{http\_proxy}} (for plain HTTP
connections) and \sphinxcode{\sphinxupquote{https\_proxy}} (for SSL connections)
environment variables.

\begin{sphinxadmonition}{note}{Note:}
Storage backends are not necessarily compatible. Don’t expect that
you can e.g. copy the data stored by the local backend into Amazon
S3 using some non-S3QL tool and then access it with S3QL’s S3
backend. If you want to copy file systems from one backend to
another, you need to use the \sphinxcode{\sphinxupquote{clone\_fs.py}} script (from the
\sphinxcode{\sphinxupquote{contrib}} directory in the S3QL tarball).
\end{sphinxadmonition}


\section{Google Storage}
\label{\detokenize{backends:google-storage}}
\sphinxhref{https://cloud.google.com/storage/}{Google Storage} is an online
storage service offered by Google. In order to use it with S3QL, make
sure that you enable the JSON API in the \sphinxhref{https://console.cloud.google.com/apis/library/}{GCP Console API Library}

The Google Storage backend uses OAuth2 authentication or \sphinxhref{https://cloud.google.com/docs/authentication/production}{ADC}
(Application Default Credentials).

To use OAuth2 authentication, specify \sphinxcode{\sphinxupquote{oauth2}} as the backend login
and a valid OAuth2 refresh token as the backend password. To obtain a
refresh token, you can use the {\hyperref[\detokenize{man/oauth_client:oauth-client}]{\sphinxcrossref{\DUrole{std,std-ref}{s3ql\_oauth\_client}}}}
program. It will instruct you to open a specific URL in your browser,
enter a code and authenticate with your Google account. Once this
procedure is complete, {\hyperref[\detokenize{man/oauth_client:oauth-client}]{\sphinxcrossref{\DUrole{std,std-ref}{s3ql\_oauth\_client}}}} will
print out the refresh token. Note that you need to do this procedure
only once, the refresh token will remain valid until you explicitly
revoke it.

To use ADC, specify \sphinxcode{\sphinxupquote{adc}} as the backend login and use an arbitrary
value for the backend password.

To create a Google Storage bucket, you can use e.g. the \sphinxhref{https://console.cloud.google.com/storage/browser}{Google
Storage Manager}. The storage URL for accessing the bucket in S3QL is
then

\begin{sphinxVerbatim}[commandchars=\\\{\}]
\PYG{l}{gs://}\PYG{n+nv}{\PYGZlt{}bucketname\PYGZgt{}}\PYG{l}{/}\PYG{n+nv}{\PYGZlt{}prefix\PYGZgt{}}
\end{sphinxVerbatim}

Here \sphinxstyleemphasis{bucketname} is the name of the bucket, and \sphinxstyleemphasis{prefix} can be an
arbitrary prefix that will be prepended to all object names used by
S3QL. This allows you to store several S3QL file systems in the same
Google Storage bucket.

The Google Storage backend accepts the following backend options:
\index{gs\_backend command line option@\spxentry{gs\_backend command line option}!ssl-ca-path=\textless{}path\textgreater{}@\spxentry{ssl-ca-path=\textless{}path\textgreater{}}}\index{ssl-ca-path=\textless{}path\textgreater{}@\spxentry{ssl-ca-path=\textless{}path\textgreater{}}!gs\_backend command line option@\spxentry{gs\_backend command line option}}

\begin{fulllineitems}
\phantomsection\label{\detokenize{backends:cmdoption-gs-backend-arg-ssl-ca-path}}\pysigline{\sphinxbfcode{\sphinxupquote{ssl-ca-path}}\sphinxcode{\sphinxupquote{=\textless{}path\textgreater{}}}}
Instead of using the system’s default certificate store, validate
the server certificate against the specified CA
certificates. \sphinxcode{\sphinxupquote{\textless{}path\textgreater{}}} may be either a file containing
multiple certificates, or a directory containing one certificate
per file.

\end{fulllineitems}

\index{gs\_backend command line option@\spxentry{gs\_backend command line option}!tcp-timeout@\spxentry{tcp-timeout}}\index{tcp-timeout@\spxentry{tcp-timeout}!gs\_backend command line option@\spxentry{gs\_backend command line option}}

\begin{fulllineitems}
\phantomsection\label{\detokenize{backends:cmdoption-gs-backend-arg-tcp-timeout}}\pysigline{\sphinxbfcode{\sphinxupquote{tcp-timeout}}\sphinxcode{\sphinxupquote{}}}
Specifies the timeout used for TCP connections. If no data can be
exchanged with the remote server for longer than this period, the
TCP connection is closed and re-established (default: 20 seconds).

\end{fulllineitems}



\section{Amazon S3}
\label{\detokenize{backends:amazon-s3}}
\sphinxhref{http://aws.amazon.com/s3}{Amazon S3} is the online storage service
offered by \sphinxhref{http://aws.amazon.com/}{Amazon Web Services (AWS)}. To
use the S3 backend, you first need to sign up for an AWS account. The
account is free, you will pay only for the amount of storage and
traffic that you actually use. After that, you need to create a bucket
that will hold the S3QL file system, e.g. using the \sphinxhref{https://console.aws.amazon.com/s3/home}{AWS Management
Console}. For best
performance, it is recommend to create the bucket in the
geographically closest storage region, but not the US Standard region
(see {\hyperref[\detokenize{durability:durability}]{\sphinxcrossref{\DUrole{std,std-ref}{Important Rules to Avoid Losing Data}}}} for the reason).

The storage URL for accessing S3 buckets in S3QL has the form

\begin{sphinxVerbatim}[commandchars=\\\{\}]
\PYG{l}{s3://}\PYG{n+nv}{\PYGZlt{}region\PYGZgt{}}\PYG{l}{/}\PYG{n+nv}{\PYGZlt{}bucket\PYGZgt{}}\PYG{l}{/}\PYG{n+nv}{\PYGZlt{}prefix\PYGZgt{}}
\end{sphinxVerbatim}

\sphinxstyleemphasis{prefix} can be an arbitrary prefix that will be prepended to all
object names used by S3QL. This allows you to store several S3QL file
systems in the same S3 bucket. For example, the storage URL

\begin{sphinxVerbatim}[commandchars=\\\{\}]
\PYG{l}{s3://ap\PYGZhy{}south\PYGZhy{}1/foomart.net/data/s3ql\PYGZus{}backup/}
\end{sphinxVerbatim}

refers to the \sphinxstyleemphasis{foomart.net} bucket in the \sphinxstyleemphasis{ap-south-1} region. All
storage objects that S3QL stores in this bucket will be prefixed with
\sphinxstyleemphasis{data/s3ql\_backup/}.

Note that the backend login and password for accessing S3 are not the
user id and password that you use to log into the Amazon Webpage, but
the \sphinxstyleemphasis{AWS access key id} and \sphinxstyleemphasis{AWS secret access key} shown under \sphinxhref{https://aws-portal.amazon.com/gp/aws/developer/account/index.html?ie=UTF8\&action=access-key}{My
Account/Access Identifiers}.

The Amazon S3 backend accepts the following backend options:
\index{s3\_backend command line option@\spxentry{s3\_backend command line option}!no-ssl@\spxentry{no-ssl}}\index{no-ssl@\spxentry{no-ssl}!s3\_backend command line option@\spxentry{s3\_backend command line option}}

\begin{fulllineitems}
\phantomsection\label{\detokenize{backends:cmdoption-s3-backend-arg-no-ssl}}\pysigline{\sphinxbfcode{\sphinxupquote{no-ssl}}\sphinxcode{\sphinxupquote{}}}
Disable encrypted (https) connections and use plain HTTP instead.

\end{fulllineitems}

\index{s3\_backend command line option@\spxentry{s3\_backend command line option}!ssl-ca-path=\textless{}path\textgreater{}@\spxentry{ssl-ca-path=\textless{}path\textgreater{}}}\index{ssl-ca-path=\textless{}path\textgreater{}@\spxentry{ssl-ca-path=\textless{}path\textgreater{}}!s3\_backend command line option@\spxentry{s3\_backend command line option}}

\begin{fulllineitems}
\phantomsection\label{\detokenize{backends:cmdoption-s3-backend-arg-ssl-ca-path}}\pysigline{\sphinxbfcode{\sphinxupquote{ssl-ca-path}}\sphinxcode{\sphinxupquote{=\textless{}path\textgreater{}}}}
Instead of using the system’s default certificate store, validate
the server certificate against the specified CA
certificates. \sphinxcode{\sphinxupquote{\textless{}path\textgreater{}}} may be either a file containing
multiple certificates, or a directory containing one certificate
per file.

\end{fulllineitems}

\index{s3\_backend command line option@\spxentry{s3\_backend command line option}!tcp-timeout@\spxentry{tcp-timeout}}\index{tcp-timeout@\spxentry{tcp-timeout}!s3\_backend command line option@\spxentry{s3\_backend command line option}}

\begin{fulllineitems}
\phantomsection\label{\detokenize{backends:cmdoption-s3-backend-arg-tcp-timeout}}\pysigline{\sphinxbfcode{\sphinxupquote{tcp-timeout}}\sphinxcode{\sphinxupquote{}}}
Specifies the timeout used for TCP connections. If no data can be
exchanged with the remote server for longer than this period, the
TCP connection is closed and re-established (default: 20 seconds).

\end{fulllineitems}

\index{s3\_backend command line option@\spxentry{s3\_backend command line option}!sse@\spxentry{sse}}\index{sse@\spxentry{sse}!s3\_backend command line option@\spxentry{s3\_backend command line option}}

\begin{fulllineitems}
\phantomsection\label{\detokenize{backends:cmdoption-s3-backend-arg-sse}}\pysigline{\sphinxbfcode{\sphinxupquote{sse}}\sphinxcode{\sphinxupquote{}}}
Enable server side encryption. Both costs \& benefits of S3 server
side encryption are probably rather small, and this option does
\sphinxstyleemphasis{not} affect any client side encryption performed by S3QL itself.

\end{fulllineitems}

\index{s3\_backend command line option@\spxentry{s3\_backend command line option}!it@\spxentry{it}}\index{it@\spxentry{it}!s3\_backend command line option@\spxentry{s3\_backend command line option}}

\begin{fulllineitems}
\phantomsection\label{\detokenize{backends:cmdoption-s3-backend-arg-it}}\pysigline{\sphinxbfcode{\sphinxupquote{it}}\sphinxcode{\sphinxupquote{}}}
Use INTELLIGENT\_TIERING storage class for new objects.
See \sphinxhref{https://docs.aws.amazon.com/AmazonS3/latest/dev/storage-class-intro.html}{AWS S3 Storage classes}

\end{fulllineitems}

\index{s3\_backend command line option@\spxentry{s3\_backend command line option}!ia@\spxentry{ia}}\index{ia@\spxentry{ia}!s3\_backend command line option@\spxentry{s3\_backend command line option}}

\begin{fulllineitems}
\phantomsection\label{\detokenize{backends:cmdoption-s3-backend-arg-ia}}\pysigline{\sphinxbfcode{\sphinxupquote{ia}}\sphinxcode{\sphinxupquote{}}}
Use STANDARD\_IA (infrequent access) storage class for new objects.
See \sphinxhref{https://docs.aws.amazon.com/AmazonS3/latest/dev/storage-class-intro.html}{AWS S3 Storage classes}

\end{fulllineitems}

\index{s3\_backend command line option@\spxentry{s3\_backend command line option}!oia@\spxentry{oia}}\index{oia@\spxentry{oia}!s3\_backend command line option@\spxentry{s3\_backend command line option}}

\begin{fulllineitems}
\phantomsection\label{\detokenize{backends:cmdoption-s3-backend-arg-oia}}\pysigline{\sphinxbfcode{\sphinxupquote{oia}}\sphinxcode{\sphinxupquote{}}}
Use ONEZONE\_IA (infrequent access) storage class for new objects.
See \sphinxhref{https://docs.aws.amazon.com/AmazonS3/latest/dev/storage-class-intro.html}{AWS S3 Storage classes}

\end{fulllineitems}

\index{s3\_backend command line option@\spxentry{s3\_backend command line option}!rrs@\spxentry{rrs}}\index{rrs@\spxentry{rrs}!s3\_backend command line option@\spxentry{s3\_backend command line option}}

\begin{fulllineitems}
\phantomsection\label{\detokenize{backends:cmdoption-s3-backend-arg-rrs}}\pysigline{\sphinxbfcode{\sphinxupquote{rrs}}\sphinxcode{\sphinxupquote{}}}
Enable reduced redundancy storage for newly created objects
(overwrites the \sphinxstyleemphasis{ia} option).

When enabling this option, it is strongly recommended to
periodically run {\hyperref[\detokenize{fsck:s3ql-verify}]{\sphinxcrossref{\DUrole{std,std-ref}{s3ql\_verify}}}}, because objects
that are lost by the storage backend may cause subsequent data loss
even later in time due to the data de-duplication feature of S3QL (see
{\hyperref[\detokenize{durability:backend-reliability}]{\sphinxcrossref{\DUrole{std,std-ref}{Data Durability}}}} for details).

\end{fulllineitems}



\section{OpenStack/Swift}
\label{\detokenize{backends:openstack-swift}}\label{\detokenize{backends:openstack-backend}}
\sphinxhref{http://www.openstack.org/}{OpenStack} is an open-source cloud server application suite. \sphinxhref{http://openstack.org/projects/storage/}{Swift} is
the cloud storage module of OpenStack. Swift/OpenStack storage is
offered by many different companies.

There are two different storage URL for the OpenStack backend that
make use of different authentication APIs. For legacy (v1)
authentication, the storage URL is

\begin{sphinxVerbatim}[commandchars=\\\{\}]
\PYG{l}{swift://}\PYG{n+nv}{\PYGZlt{}hostname\PYGZgt{}}\PYG{g+ge}{[:\PYGZlt{}port\PYGZgt{}]}\PYG{l}{/}\PYG{n+nv}{\PYGZlt{}container\PYGZgt{}}\PYG{g+ge}{[/\PYGZlt{}prefix\PYGZgt{}]}
\end{sphinxVerbatim}

for Keystone (v2 and v3) authentication, the storage URL is

\begin{sphinxVerbatim}[commandchars=\\\{\}]
\PYG{l}{swiftks://}\PYG{n+nv}{\PYGZlt{}hostname\PYGZgt{}}\PYG{g+ge}{[:\PYGZlt{}port\PYGZgt{}]}\PYG{l}{/}\PYG{n+nv}{\PYGZlt{}region\PYGZgt{}}\PYG{l}{:}\PYG{n+nv}{\PYGZlt{}container\PYGZgt{}}\PYG{g+ge}{[/\PYGZlt{}prefix\PYGZgt{}]}
\end{sphinxVerbatim}

Note that when using Keystone authentication, you can (and have to)
specify the storage region of the container as well. Also note that when
using Keystone v3 authentication, the \sphinxcode{\sphinxupquote{domain}} option is required.

In both cases, \sphinxstyleemphasis{hostname} name should be the name of the
authentication server.  The storage container must already exist (most
OpenStack providers offer either a web frontend or a command line tool
for creating containers). \sphinxstyleemphasis{prefix} can be an arbitrary prefix that
will be prepended to all object names used by S3QL, which can be used
to store multiple S3QL file systems in the same container.

When using legacy authentication, the backend login and password
correspond to the OpenStack username and API Access Key. When using
Keystone authentication, the backend password is your regular
OpenStack password and the backend login combines you OpenStack
username and tenant name in the form \sphinxcode{\sphinxupquote{\textless{}tenant\textgreater{}:\textless{}user\textgreater{}}}. If no tenant
is required, the OpenStack username alone may be used as backend
login.

The OpenStack backend accepts the following backend options:
\index{swift\_backend command line option@\spxentry{swift\_backend command line option}!no-ssl@\spxentry{no-ssl}}\index{no-ssl@\spxentry{no-ssl}!swift\_backend command line option@\spxentry{swift\_backend command line option}}

\begin{fulllineitems}
\phantomsection\label{\detokenize{backends:cmdoption-swift-backend-arg-no-ssl}}\pysigline{\sphinxbfcode{\sphinxupquote{no-ssl}}\sphinxcode{\sphinxupquote{}}}
Use plain HTTP to connect to the authentication server. This option
does not directly affect the connection to the storage
server. Whether HTTPS or plain HTTP is used to connect to the
storage server is determined by the authentication server.

\end{fulllineitems}

\index{swift\_backend command line option@\spxentry{swift\_backend command line option}!ssl-ca-path=\textless{}path\textgreater{}@\spxentry{ssl-ca-path=\textless{}path\textgreater{}}}\index{ssl-ca-path=\textless{}path\textgreater{}@\spxentry{ssl-ca-path=\textless{}path\textgreater{}}!swift\_backend command line option@\spxentry{swift\_backend command line option}}

\begin{fulllineitems}
\phantomsection\label{\detokenize{backends:cmdoption-swift-backend-arg-ssl-ca-path}}\pysigline{\sphinxbfcode{\sphinxupquote{ssl-ca-path}}\sphinxcode{\sphinxupquote{=\textless{}path\textgreater{}}}}
Instead of using the system’s default certificate store, validate
the server certificate against the specified CA
certificates. \sphinxcode{\sphinxupquote{\textless{}path\textgreater{}}} may be either a file containing
multiple certificates, or a directory containing one certificate
per file.

\end{fulllineitems}

\index{swift\_backend command line option@\spxentry{swift\_backend command line option}!tcp-timeout@\spxentry{tcp-timeout}}\index{tcp-timeout@\spxentry{tcp-timeout}!swift\_backend command line option@\spxentry{swift\_backend command line option}}

\begin{fulllineitems}
\phantomsection\label{\detokenize{backends:cmdoption-swift-backend-arg-tcp-timeout}}\pysigline{\sphinxbfcode{\sphinxupquote{tcp-timeout}}\sphinxcode{\sphinxupquote{}}}
Specifies the timeout used for TCP connections. If no data can be
exchanged with the remote server for longer than this period, the
TCP connection is closed and re-established (default: 20 seconds).

\end{fulllineitems}

\index{swift\_backend command line option@\spxentry{swift\_backend command line option}!disable-expect100@\spxentry{disable-expect100}}\index{disable-expect100@\spxentry{disable-expect100}!swift\_backend command line option@\spxentry{swift\_backend command line option}}

\begin{fulllineitems}
\phantomsection\label{\detokenize{backends:cmdoption-swift-backend-arg-disable-expect100}}\pysigline{\sphinxbfcode{\sphinxupquote{disable-expect100}}\sphinxcode{\sphinxupquote{}}}
If this option is specified, S3QL does not use the \sphinxcode{\sphinxupquote{Expect:
continue}} header (cf. \sphinxhref{http://tools.ietf.org/html/rfc2616\#section-8.2.3}{RFC2616, section 8.2.3}) when uploading
data to the server. This can be used to work around broken storage
servers that don’t fully support HTTP 1.1, but may decrease
performance as object data will be transmitted to the server more
than once in some circumstances.

\end{fulllineitems}

\index{swift\_backend command line option@\spxentry{swift\_backend command line option}!no-feature-detection@\spxentry{no-feature-detection}}\index{no-feature-detection@\spxentry{no-feature-detection}!swift\_backend command line option@\spxentry{swift\_backend command line option}}

\begin{fulllineitems}
\phantomsection\label{\detokenize{backends:cmdoption-swift-backend-arg-no-feature-detection}}\pysigline{\sphinxbfcode{\sphinxupquote{no-feature-detection}}\sphinxcode{\sphinxupquote{}}}
If this option is specified, S3QL does not try to dynamically detect
advanced features of the Swift backend. In this case S3QL can only
use the least common denominator of supported Swift versions and
configurations.

\end{fulllineitems}

\index{swift\_backend command line option@\spxentry{swift\_backend command line option}!domain@\spxentry{domain}}\index{domain@\spxentry{domain}!swift\_backend command line option@\spxentry{swift\_backend command line option}}

\begin{fulllineitems}
\phantomsection\label{\detokenize{backends:cmdoption-swift-backend-arg-domain}}\pysigline{\sphinxbfcode{\sphinxupquote{domain}}\sphinxcode{\sphinxupquote{}}}
If this option is specified, S3QL will use the Keystone v3 API. The
default domain for OpenStack installations is \sphinxcode{\sphinxupquote{Default}}. If this
option is specified without setting the \sphinxcode{\sphinxupquote{project-domain}} option, this
will be used for both the project and the user domain. Note: some
instances of the Keystone v3 API prefer the use of UUIDs rather than
names for tenant (called project in newer OpenStack versions), as
well as domains.

\end{fulllineitems}

\index{swift\_backend command line option@\spxentry{swift\_backend command line option}!project-domain@\spxentry{project-domain}}\index{project-domain@\spxentry{project-domain}!swift\_backend command line option@\spxentry{swift\_backend command line option}}\index{swift\_backend command line option@\spxentry{swift\_backend command line option}!In simple cases, the project domain will be the same as the auth@\spxentry{In simple cases, the project domain will be the same as the auth}}\index{In simple cases, the project domain will be the same as the auth@\spxentry{In simple cases, the project domain will be the same as the auth}!swift\_backend command line option@\spxentry{swift\_backend command line option}}\index{swift\_backend command line option@\spxentry{swift\_backend command line option}!domain. If the {}`project-domain{}` option is not specified, it will be@\spxentry{domain. If the {}`project-domain{}` option is not specified, it will be}}\index{domain. If the {}`project-domain{}` option is not specified, it will be@\spxentry{domain. If the {}`project-domain{}` option is not specified, it will be}!swift\_backend command line option@\spxentry{swift\_backend command line option}}\index{swift\_backend command line option@\spxentry{swift\_backend command line option}!assumed to be the same as the user domain.@\spxentry{assumed to be the same as the user domain.}}\index{assumed to be the same as the user domain.@\spxentry{assumed to be the same as the user domain.}!swift\_backend command line option@\spxentry{swift\_backend command line option}}

\begin{fulllineitems}
\phantomsection\label{\detokenize{backends:cmdoption-swift-backend-arg-project-domain}}\pysigline{\sphinxbfcode{\sphinxupquote{project-domain}}\sphinxcode{\sphinxupquote{}}}\phantomsection\label{\detokenize{backends:cmdoption-swift-backend-arg-in}}\pysigline{\sphinxbfcode{\sphinxupquote{In}}\sphinxcode{\sphinxupquote{~simple~cases}}\sphinxcode{\sphinxupquote{,~}}\sphinxbfcode{\sphinxupquote{the}}\sphinxcode{\sphinxupquote{~project~domain~will~be~the~same~as~the~auth}}}\phantomsection\label{\detokenize{backends:cmdoption-swift-backend-arg-it}}\pysigline{\sphinxbfcode{\sphinxupquote{domain.}}\sphinxcode{\sphinxupquote{~If~the~{}`project-domain{}`~option~is~not~specified}}\sphinxcode{\sphinxupquote{,~}}\sphinxbfcode{\sphinxupquote{it}}\sphinxcode{\sphinxupquote{~will~be}}}\phantomsection\label{\detokenize{backends:cmdoption-swift-backend-arg-assumed}}\pysigline{\sphinxbfcode{\sphinxupquote{assumed}}\sphinxcode{\sphinxupquote{~to~be~the~same~as~the~user~domain.}}}
\end{fulllineitems}


\begin{sphinxadmonition}{note}{Note:}
The Swift API unfortunately lacks a number of features that S3QL
normally makes use of. S3QL works around these deficiencies as much
as possible. However, this means that storing data using the Swift
backend generally requires more network round-trips and transfer
volume than the other backends. Also, S3QL requires Swift storage
servers to provide immediate consistency for newly created objects.
\end{sphinxadmonition}


\section{Rackspace CloudFiles}
\label{\detokenize{backends:rackspace-cloudfiles}}
\sphinxhref{http://www.rackspace.com/}{Rackspace} CloudFiles uses \sphinxhref{http://www.openstack.org/}{OpenStack} internally, so it is possible to
just use the OpenStack/Swift backend (see above) with
\sphinxcode{\sphinxupquote{auth.api.rackspacecloud.com}} as the host name. For convenince,
there is also a special \sphinxcode{\sphinxupquote{rackspace}} backend that uses a storage URL
of the form

\begin{sphinxVerbatim}[commandchars=\\\{\}]
\PYG{l}{rackspace://}\PYG{n+nv}{\PYGZlt{}region\PYGZgt{}}\PYG{l}{/}\PYG{n+nv}{\PYGZlt{}container\PYGZgt{}}\PYG{g+ge}{[/\PYGZlt{}prefix\PYGZgt{}]}
\end{sphinxVerbatim}

The storage container must already exist in the selected
region. \sphinxstyleemphasis{prefix} can be an arbitrary prefix that will be prepended to
all object names used by S3QL and can be used to store several S3QL
file systems in the same container.

You can create a storage container for S3QL using the \sphinxhref{https://mycloud.rackspace.com/}{Cloud Control
Panel} (click on \sphinxstyleemphasis{Files} in the
topmost menu bar).

The Rackspace backend accepts the same backend options as the
{\hyperref[\detokenize{backends:openstack-backend}]{\sphinxcrossref{\DUrole{std,std-ref}{OpenStack backend}}}}.


\section{S3 compatible}
\label{\detokenize{backends:s3-compatible}}
The S3 compatible backend allows S3QL to access any storage service
that uses the same protocol as Amazon S3. The storage URL has the form

\begin{sphinxVerbatim}[commandchars=\\\{\}]
\PYG{l}{s3c://}\PYG{n+nv}{\PYGZlt{}hostname\PYGZgt{}}\PYG{l}{:}\PYG{n+nv}{\PYGZlt{}port\PYGZgt{}}\PYG{l}{/}\PYG{n+nv}{\PYGZlt{}bucketname\PYGZgt{}}\PYG{l}{/}\PYG{n+nv}{\PYGZlt{}prefix\PYGZgt{}}
\end{sphinxVerbatim}

Here \sphinxstyleemphasis{bucketname} is the name of an (existing) bucket, and \sphinxstyleemphasis{prefix}
can be an arbitrary prefix that will be prepended to all object names
used by S3QL. This allows you to store several S3QL file systems in
the same bucket.

The S3 compatible backend accepts the following backend options:
\index{s3c\_backend command line option@\spxentry{s3c\_backend command line option}!no-ssl@\spxentry{no-ssl}}\index{no-ssl@\spxentry{no-ssl}!s3c\_backend command line option@\spxentry{s3c\_backend command line option}}

\begin{fulllineitems}
\phantomsection\label{\detokenize{backends:cmdoption-s3c-backend-arg-no-ssl}}\pysigline{\sphinxbfcode{\sphinxupquote{no-ssl}}\sphinxcode{\sphinxupquote{}}}
Disable encrypted (https) connections and use plain HTTP instead.

\end{fulllineitems}

\index{s3c\_backend command line option@\spxentry{s3c\_backend command line option}!ssl-ca-path=\textless{}path\textgreater{}@\spxentry{ssl-ca-path=\textless{}path\textgreater{}}}\index{ssl-ca-path=\textless{}path\textgreater{}@\spxentry{ssl-ca-path=\textless{}path\textgreater{}}!s3c\_backend command line option@\spxentry{s3c\_backend command line option}}

\begin{fulllineitems}
\phantomsection\label{\detokenize{backends:cmdoption-s3c-backend-arg-ssl-ca-path}}\pysigline{\sphinxbfcode{\sphinxupquote{ssl-ca-path}}\sphinxcode{\sphinxupquote{=\textless{}path\textgreater{}}}}
Instead of using the system’s default certificate store, validate
the server certificate against the specified CA
certificates. \sphinxcode{\sphinxupquote{\textless{}path\textgreater{}}} may be either a file containing
multiple certificates, or a directory containing one certificate
per file.

\end{fulllineitems}

\index{s3c\_backend command line option@\spxentry{s3c\_backend command line option}!tcp-timeout@\spxentry{tcp-timeout}}\index{tcp-timeout@\spxentry{tcp-timeout}!s3c\_backend command line option@\spxentry{s3c\_backend command line option}}

\begin{fulllineitems}
\phantomsection\label{\detokenize{backends:cmdoption-s3c-backend-arg-tcp-timeout}}\pysigline{\sphinxbfcode{\sphinxupquote{tcp-timeout}}\sphinxcode{\sphinxupquote{}}}
Specifies the timeout used for TCP connections. If no data can be
exchanged with the remote server for longer than this period, the
TCP connection is closed and re-established (default: 20 seconds).

\end{fulllineitems}

\index{s3c\_backend command line option@\spxentry{s3c\_backend command line option}!disable-expect100@\spxentry{disable-expect100}}\index{disable-expect100@\spxentry{disable-expect100}!s3c\_backend command line option@\spxentry{s3c\_backend command line option}}

\begin{fulllineitems}
\phantomsection\label{\detokenize{backends:cmdoption-s3c-backend-arg-disable-expect100}}\pysigline{\sphinxbfcode{\sphinxupquote{disable-expect100}}\sphinxcode{\sphinxupquote{}}}
If this option is specified, S3QL does not use the \sphinxcode{\sphinxupquote{Expect:
continue}} header (cf. \sphinxhref{http://tools.ietf.org/html/rfc2616\#section-8.2.3}{RFC2616, section 8.2.3}) when uploading
data to the server. This can be used to work around broken storage
servers that don’t fully support HTTP 1.1, but may decrease
performance as object data will be transmitted to the server more
than once in some circumstances.

\end{fulllineitems}

\index{s3c\_backend command line option@\spxentry{s3c\_backend command line option}!dumb-copy@\spxentry{dumb-copy}}\index{dumb-copy@\spxentry{dumb-copy}!s3c\_backend command line option@\spxentry{s3c\_backend command line option}}

\begin{fulllineitems}
\phantomsection\label{\detokenize{backends:cmdoption-s3c-backend-arg-dumb-copy}}\pysigline{\sphinxbfcode{\sphinxupquote{dumb-copy}}\sphinxcode{\sphinxupquote{}}}
If this option is specified, S3QL assumes that a COPY request to
the storage server has succeeded as soon as the server returns a
\sphinxcode{\sphinxupquote{200 OK}} status. The \sphinxhref{http://docs.aws.amazon.com/AmazonS3/latest/API/RESTObjectCOPY.html}{S3 COPY API} specifies that the
storage server may still return an error in the request body (see
the \sphinxhref{https://doc.s3.amazonaws.com/proposals/copy.html}{copy proposal} for the rationale), so this
option should only be used if you are certain that your storage
server only returns \sphinxcode{\sphinxupquote{200 OK}} when the copy operation has been
completely and successfully carried out. Using this option may be
neccessary if your storage server does not return a valid response
body for a successful copy operation.

\end{fulllineitems}



\section{Local}
\label{\detokenize{backends:local}}
S3QL is also able to store its data on the local file system. This can
be used to backup data on external media, or to access external
services that S3QL can not talk to directly (e.g., it is possible to
store data over SSH by first mounting the remote system using \sphinxhref{http://fuse.sourceforge.net/sshfs.html}{sshfs}
and then using the local backend to store the data in the sshfs
mountpoint).

The storage URL for local storage is

\begin{sphinxVerbatim}[commandchars=\\\{\}]
\PYG{l}{local://}\PYG{n+nv}{\PYGZlt{}path\PYGZgt{}}
\end{sphinxVerbatim}

Note that you have to write three consecutive slashes to specify an
absolute path, e.g. \sphinxcode{\sphinxupquote{local:///var/archive}}. Also, relative paths will
automatically be converted to absolute paths before the authentication
file (see {\hyperref[\detokenize{authinfo:authinfo}]{\sphinxcrossref{\DUrole{std,std-ref}{Storing Backend Information and Credentials}}}}) is read, i.e. if you are in the
\sphinxcode{\sphinxupquote{/home/john}} directory and try to mount \sphinxcode{\sphinxupquote{local://s3ql}}, the
corresponding section in the authentication file must match the
storage url \sphinxcode{\sphinxupquote{local:///home/john/s3ql}}.

The local backend does not accept any backend options.


\chapter{Important Rules to Avoid Losing Data}
\label{\detokenize{durability:important-rules-to-avoid-losing-data}}\label{\detokenize{durability:durability}}\label{\detokenize{durability::doc}}
Most S3QL backends store data in distributed storage systems. These
systems differ from a traditional, local hard disk in several
important ways. In order to avoid losing data, this section should be
read very carefully.


\section{Rules in a Nutshell}
\label{\detokenize{durability:rules-in-a-nutshell}}
To avoid losing your data, obey the following rules:
\begin{enumerate}
\def\theenumi{\arabic{enumi}}
\def\labelenumi{\theenumi .}
\makeatletter\def\p@enumii{\p@enumi \theenumi .}\makeatother
\item {} 
Know what durability you can expect from your chosen storage
provider. The durability describes how likely it is that a stored
object becomes damaged over time. Such data corruption can never be
prevented completely, techniques like geographic replication and
RAID storage just reduce the likelihood of it to happen (i.e.,
increase the durability).

\item {} 
When choosing a backend and storage provider, keep in mind that
when using S3QL, the effective durability of the file system data
will be reduced because of S3QL’s data de-duplication feature.

\item {} 
Determine your storage service’s consistency window. The
consistency window that is important for S3QL is the smaller of the
times for which:
\begin{itemize}
\item {} 
a newly created object may not yet be included in the list of
stored objects

\item {} 
an attempt to read a newly created object may fail with the
storage service reporting that the object does not exist

\end{itemize}

If \sphinxstyleemphasis{one} of the above times is zero, we say that as far as S3QL is
concerned the storage service has \sphinxstyleemphasis{immediate} consistency.

If your storage provider claims that \sphinxstyleemphasis{neither} of the above can
ever happen, while at the same time promising high durability, you
should choose a respectable provider instead.

\item {} 
When mounting the same file system on different computers (or on
the same computer but with different \sphinxcode{\sphinxupquote{-{-}cachedir}} directories),
the time that passes between the first and second of invocation of
\sphinxstyleliteralstrong{\sphinxupquote{mount.s3ql}} must be at least as long as your storage
service’s consistency window. If your storage service offers
immediate consistency, you do not need to wait at all.

\item {} 
Before running \sphinxstyleliteralstrong{\sphinxupquote{fsck.s3ql}} or \sphinxstyleliteralstrong{\sphinxupquote{s3qladm}}, the file system
must have been left untouched for the length of the consistency
window. If your storage service offers immediate consistency, you
do not need to wait at all.

\end{enumerate}

The rest of this section explains the above rules and the reasons for
them in more detail. It also contains a list of the consistency
windows for a number of larger storage providers.


\section{Consistency Window List}
\label{\detokenize{durability:consistency-window-list}}
The following is a list of the consistency windows (as far as S3QL is
concerned) for a number of storage providers. This list doesn’t come
with any guarantees and may be outdated. If your storage provider is
not included, or if you need more reliable information, check with
your storage provider.


\begin{savenotes}\sphinxattablestart
\centering
\begin{tabulary}{\linewidth}[t]{|T|T|}
\hline
\sphinxstyletheadfamily 
Storage Provider
&\sphinxstyletheadfamily 
Consistency Window
\\
\hline
Amazon S3 in the US standard region
&
No guarantees
\\
\hline
Amazon S3 in other regions
&
Immediate
\\
\hline
Google Storage
&
Immediate
\\
\hline
\end{tabulary}
\par
\sphinxattableend\end{savenotes}


\section{Data Consistency}
\label{\detokenize{durability:data-consistency}}
In contrast to the typical hard disk, most storage providers do not
guarantee \sphinxstyleemphasis{immediate consistency} of written data. This means that:
\begin{itemize}
\item {} 
after an object has been stored, requests to read this object may
still fail or return the prior contents for a little while.

\item {} 
after an object has been deleted, attempts to read it may still
return the (old) data for some time, and it may still remain in the
list of stored objects for some time.

\item {} 
after a new object has been created, it may still not be included
when retrieving the list of stored objects for some time.

\end{itemize}

Of course, none of this is acceptable for a file system, and S3QL
generally handles any of the above situations internally so that it
always provides a fully consistent file system to the user. However,
there are some situations where an S3QL user nevertheless needs to be
aware of the peculiarities of his chosen storage service.

Suppose that you mount the file system, store some new data, delete
some old data and unmount it. If you then mount the file system again
right away on another computer, there is no guarantee that S3QL will
see any of the changes that the first S3QL process has made. At least
in theory it is therefore possible that on the second mount, S3QL does
not see any of the changes that you have done and presents you an “old
version” of the file system without them. Even worse, if you notice
the problem and unmount the file system, S3QL will upload the old
status (which S3QL necessarily has to consider as current) and thereby
permanently override the newer version (even though this change may
not become immediately visible either). S3QL uses several techniques
to reduce the likelihood of this to happen (see {\hyperref[\detokenize{impl_details:impl-details}]{\sphinxcrossref{\DUrole{std,std-ref}{Implementation Details}}}}
for more information on this), but without support from the storage
service, the possibility cannot be eliminated completely.

The same problem of course also applies when checking the file system.
If the storage service provides S3QL with only partially updated data,
S3QL has no way to find out if this a real consistency problem that
needs to be fixed or if it is only a temporary problem that will
resolve itself automatically (because there are still changes that
have not become visible yet).

This is where the so called \sphinxstyleemphasis{consistency window} comes in. The
consistency window is the maximum time (after writing or deleting the
object) for which any of the above “outdated responses” may be
received. If the consistency window is zero, i.e. all changes are
immediately effective, the storage service is said to have \sphinxstyleemphasis{immediate
consistency}. If the window is infinite, i.e. there is no upper bound
on the time it may take for changes to become effect, the storage
service is said to be \sphinxstyleemphasis{eventually consistent}. Note that often there
are different consistency windows for the different operations. For
example, Google Storage offers immediate consistency when reading
data, but only eventual consistency for the list of stored objects.

To prevent the problem of S3QL working with an outdated copy of the
file system data, it is therefore sufficient to simply wait for the
consistency window to pass before mounting the file system again (or
running a file system check). The length of the consistency window
changes from storage service to storage service, and if your service
is not included in the list below, you should check the web page or
ask the technical support of your storage provider. The window that is
important for S3QL is the smaller of the times for which
\begin{itemize}
\item {} 
a newly created object may not yet be included in the list of
stored objects

\item {} 
an attempt to read a newly created object may fail with the
storage service reporting that the object does not exist

\end{itemize}

Unfortunately, many storage providers are hesitant to guarantee
anything but eventual consistency, i.e. the length of the consistency
window is potentially infinite. In that case you simply have to pick a
length that you consider “safe enough”. For example, even though
Amazon is only guaranteeing eventual consistency, the ordinary
consistency window for data stored in S3 is just a few seconds, and
only in exceptional circumstances (i.e., core network outages) it may
rise up to hours (\sphinxhref{http://forums.aws.amazon.com/message.jspa?messageID=38471\#38471}{source}).


\section{Data Durability}
\label{\detokenize{durability:data-durability}}\label{\detokenize{durability:backend-reliability}}
The durability of a storage service a measure of the average
probability of a storage object to become corrupted over time. The
lower the chance of data loss, the higher the durability. Storage
services like Amazon S3 claim to achieve a durability of up to
99.999999999\% over a year, i.e. if you store 100000000 objects for 100
years, you can expect that at the end of that time one object will be
corrupted or lost.

S3QL is designed to reduce redundancy and store data in the smallest
possible form. Therefore, S3QL is generally not able to compensate for
any such losses, and when choosing a storage service you should
carefully review if the offered durability matches your requirements.
When doing this, there are two factors that should be kept in mind.

Firstly, even though S3QL is not able to compensate for storage
service failures, it is able to detect them: when trying to access
data that has been lost or corrupted by the storage service, an IO
error will be returned and the mount point will become inaccessible to
ensure that the problem is noticed.

Secondly, the consequences of a data loss by the storage service can
be significantly more severe than you may expect because of S3QL’s
data de-duplication feature: a data loss in the storage service at
time \sphinxstyleemphasis{x} may cause data that is written \sphinxstyleemphasis{after} time \sphinxstyleemphasis{x} to be lost as
well. Consider the following scenario:
\begin{enumerate}
\def\theenumi{\arabic{enumi}}
\def\labelenumi{\theenumi .}
\makeatletter\def\p@enumii{\p@enumi \theenumi .}\makeatother
\item {} 
You store an important file in the S3QL file system.

\item {} 
The storage service loses the data blocks of this file. As long as you
do not access the file or run \sphinxstyleliteralstrong{\sphinxupquote{fsck.s3ql}}, S3QL is not
aware that the data has been lost by the storage service.

\item {} 
You save an additional copy of the important file in a different
location on the same S3QL file system.

\item {} 
S3QL detects that the contents of the new file are identical to the
data blocks that have been stored earlier. Since at this point S3QL
is not aware that these blocks have been lost by the storage service, it
does not save another copy of the file contents in the storage service but
relies on the (presumably) existing blocks instead.

\item {} 
Therefore, even though you saved another copy, you still do not
have a backup of the important file (since both copies refer to the
same data blocks that have been lost by the storage service).

\end{enumerate}

For some storage services, \sphinxstyleliteralstrong{\sphinxupquote{fsck.s3ql}} can mitigate this
effect. When \sphinxstyleliteralstrong{\sphinxupquote{fsck.s3ql}} runs, it asks the storage service
for a list of all stored objects. If objects are missing, it can then
mark the damaged files and prevent the problem from spreading forwards
in time. Figuratively speaking, this establishes a “checkpoint”: data
loss that occurred before running \sphinxstyleliteralstrong{\sphinxupquote{fsck.s3ql}} can not affect
any file system operations that are performed after the check.
Unfortunately, many storage services only “discover” that objects are
missing or broken when the object actually needs to be retrieved. In
this case, \sphinxstyleliteralstrong{\sphinxupquote{fsck.s3ql}} will not learn anything by just
querying the list of objects.

This effect can be mitigated to some degree by using the
\sphinxstyleliteralstrong{\sphinxupquote{s3ql\_verify}} command in additon to
\sphinxstyleliteralstrong{\sphinxupquote{fsck.s3ql}}. \sphinxstyleliteralstrong{\sphinxupquote{s3ql\_verify}} asks the storage service
to look up every stored object and may therefore take much longer than
running \sphinxstyleliteralstrong{\sphinxupquote{fsck.s3ql}}, but can also offer a much stronger
assurance that no data has been lost by the storage service. To
“recover” from damaged storage objects in the backend, the damaged
objects found by \sphinxstyleliteralstrong{\sphinxupquote{s3ql\_verify}} have to be explicitly deleted
(so that a successive \sphinxstyleliteralstrong{\sphinxupquote{fsck.s3ql}} is able detect them as
missing, correct the file system metadata, and move any affected files
to \sphinxcode{\sphinxupquote{lost+found}}). This procedure is currently not automated, so
it is generally a good idea to choose a storage service where the
expected data durability is high enough so that the possibility of a
lost object (and thus the need to run any full checks) can be
neglected over long periods of time.


\chapter{File System Creation}
\label{\detokenize{mkfs:file-system-creation}}\label{\detokenize{mkfs::doc}}
A S3QL file system is created with the \sphinxstyleliteralstrong{\sphinxupquote{mkfs.s3ql}} command. It has the
following syntax:

\begin{sphinxVerbatim}[commandchars=\\\{\}]
\PYG{l}{mkfs.s3ql }\PYG{g+ge}{[options]}\PYG{l}{ }\PYG{n+nv}{\PYGZlt{}storage url\PYGZgt{}}
\end{sphinxVerbatim}

This command accepts the following options:
\begin{quote}
\begin{optionlist}{3cm}
\item [-{-}cachedir \textless{}path\textgreater{}]  
Store cached data in this directory (default:
\sphinxcode{\sphinxupquote{\textasciitilde{}/.s3ql)}}
\item [-{-}log \textless{}target\textgreater{}]  
Destination for log messages. Specify \sphinxcode{\sphinxupquote{none}} for
standard output or \sphinxcode{\sphinxupquote{syslog}} for the system logging
daemon. Anything else will be interpreted as a file
name. Log files will be rotated when they reach 1 MiB,
and at most 5 old log files will be kept. Default:
\sphinxcode{\sphinxupquote{None}}
\item [-{-}debug-modules \textless{}modules\textgreater{}]  
Activate debugging output from specified modules (use
commas to separate multiple modules). Debug messages
will be written to the target specified by the
\sphinxcode{\sphinxupquote{-{-}log}} option.
\item [-{-}debug]  
Activate debugging output from all S3QL modules. Debug
messages will be written to the target specified by
the \sphinxcode{\sphinxupquote{-{-}log}} option.
\item [-{-}quiet]  
be really quiet
\item [-{-}backend-options \textless{}options\textgreater{}]  
Backend specific options (separate by commas). See
backend documentation for available options.
\item [-{-}version]  
just print program version and exit
\item [-{-}authfile \textless{}path\textgreater{}]  
Read authentication credentials from this file
(default: \sphinxcode{\sphinxupquote{\textasciitilde{}/.s3ql/authinfo2)}}
\item [-L \textless{}name\textgreater{}]  
Filesystem label
\item [-{-}max-obj-size \textless{}size\textgreater{}]  
Maximum size of storage objects in KiB. Files bigger
than this will be spread over multiple objects in the
storage backend. Default: 10240 KiB.
\item [-{-}plain]  
Create unencrypted file system.
\end{optionlist}
\end{quote}

Unless you have specified the \sphinxcode{\sphinxupquote{-{-}plain}} option,
\sphinxstyleliteralstrong{\sphinxupquote{mkfs.s3ql}} will ask you to enter an encryption
password. This password will \sphinxstyleemphasis{not} be read from an authentication file
specified with the \sphinxcode{\sphinxupquote{-{-}authfile}} option to prevent accidental
creation of an encrypted file system.

Note that:
\begin{itemize}
\item {} 
All data that is stored under the given storage url is assumed to
managed exclusively by S3QL. Trying to manually save additional
objects (or remove or manipulate existing objects) will lead to file
system corruption, and \sphinxstyleliteralstrong{\sphinxupquote{fsck.s3ql}} may delete objects that
do not belong to the file system.

\item {} 
With most storage backends, slashes in the storage url prefix do not
have special meaning. For example, the storage urls
\sphinxcode{\sphinxupquote{s3://mybucket/myprefix/}} and \sphinxcode{\sphinxupquote{s3://mybucket/myprefix}} are
distinct. In the first case, the prefix is \sphinxcode{\sphinxupquote{myprefix/}}, while in
the second it is \sphinxcode{\sphinxupquote{myprefix}}.

\item {} 
S3QL file systems can not be “stacked”, i.e. you cannot have one
file system stored at \sphinxcode{\sphinxupquote{s3://bucketname/outerprefix}} and a second
one at \sphinxcode{\sphinxupquote{s3://bucketname/outerprefix/innerprefix}}.

\end{itemize}


\chapter{Managing File Systems}
\label{\detokenize{adm:managing-file-systems}}\label{\detokenize{adm::doc}}
The \sphinxcode{\sphinxupquote{s3qladm}} command performs various operations on \sphinxstyleemphasis{unmounted} S3QL
file systems. The file system \sphinxstyleemphasis{must not be mounted} when using
\sphinxcode{\sphinxupquote{s3qladm}} or things will go wrong badly.

The syntax is

\begin{sphinxVerbatim}[commandchars=\\\{\}]
\PYG{l}{s3qladm }\PYG{g+ge}{[options]}\PYG{l}{ }\PYG{n+nv}{\PYGZlt{}action\PYGZgt{}}\PYG{l}{ }\PYG{n+nv}{\PYGZlt{}storage\PYGZhy{}url\PYGZgt{}}
\end{sphinxVerbatim}

where \sphinxcode{\sphinxupquote{action}} may be either of \sphinxstyleliteralstrong{\sphinxupquote{passphrase}},
\sphinxstyleliteralstrong{\sphinxupquote{upgrade}}, \sphinxstyleliteralstrong{\sphinxupquote{clear}} or \sphinxstyleliteralstrong{\sphinxupquote{download-metadata}}.

The \sphinxstyleliteralstrong{\sphinxupquote{s3qladm}} accepts the following general options, no
matter what specific action is being invoked:
\begin{quote}
\begin{optionlist}{3cm}
\item [-{-}authfile \textless{}path\textgreater{}]  
Read authentication credentials from this file
(default: \sphinxcode{\sphinxupquote{\textasciitilde{}/.s3ql/authinfo2)}}
\item [-{-}debug-modules \textless{}modules\textgreater{}]  
Activate debugging output from specified modules (use
commas to separate multiple modules). Debug messages
will be written to the target specified by the
\sphinxcode{\sphinxupquote{-{-}log}} option.
\item [-{-}debug]  
Activate debugging output from all S3QL modules. Debug
messages will be written to the target specified by
the \sphinxcode{\sphinxupquote{-{-}log}} option.
\item [-{-}quiet]  
be really quiet
\item [-{-}log \textless{}target\textgreater{}]  
Destination for log messages. Specify \sphinxcode{\sphinxupquote{none}} for
standard output or \sphinxcode{\sphinxupquote{syslog}} for the system logging
daemon. Anything else will be interpreted as a file
name. Log files will be rotated when they reach 1 MiB,
and at most 5 old log files will be kept. Default:
\sphinxcode{\sphinxupquote{None}}
\item [-{-}backend-options \textless{}options\textgreater{}]  
Backend specific options (separate by commas). See
backend documentation for available options.
\item [-{-}cachedir \textless{}path\textgreater{}]  
Store cached data in this directory (default:
\sphinxcode{\sphinxupquote{\textasciitilde{}/.s3ql)}}
\item [-{-}version]  
just print program version and exit
\end{optionlist}
\end{quote}

Hint: run \sphinxcode{\sphinxupquote{s3qladm \textless{}action\textgreater{} -{-}help}} to get help on the additional arguments
that the different actions take.


\section{Changing the Passphrase}
\label{\detokenize{adm:changing-the-passphrase}}
To change the passphrase of a file system, use the \sphinxcode{\sphinxupquote{passphrase}}
subcommand:

\begin{sphinxVerbatim}[commandchars=\\\{\}]
\PYG{l}{s3qladm passphrase  }\PYG{n+nv}{\PYGZlt{}storage url\PYGZgt{}}
\end{sphinxVerbatim}

That this will not actually re-encrypt all the stored data with the
new passphrase. Rather, S3QL generates an internal “master key” when
the file system is created and uses this key to encrypt all stored
data. The user-supplied passphrase is used to encrypt/decrypt only the
master key, and can therefore be changed. This means, however, that if
a passphrase has been disclosed to someone untrusted, changing it
afterwards will \sphinxstylestrong{not revoke access to people who had access to the
old passphrase} (because they could have decrypted the master key and
retained a copy).


\section{Upgrading the file system}
\label{\detokenize{adm:upgrading-the-file-system}}
If you have installed a new version of S3QL, it may sometimes be
necessary to upgrade the file system metadata as well. Note that in
this case the file system can no longer be accessed with older
versions of S3QL after the upgrade.

During the upgrade you have to make sure that the command is not
interrupted, and that no one else tries to mount, check or upgrade the
file system at the same time.

To upgrade a file system from the previous to the current revision,
execute

\begin{sphinxVerbatim}[commandchars=\\\{\}]
\PYG{l}{s3qladm upgrade }\PYG{n+nv}{\PYGZlt{}storage url\PYGZgt{}}
\end{sphinxVerbatim}


\section{Deleting a file system}
\label{\detokenize{adm:deleting-a-file-system}}
A file system can be deleted with:

\begin{sphinxVerbatim}[commandchars=\\\{\}]
\PYG{l}{s3qladm clear }\PYG{n+nv}{\PYGZlt{}storage url\PYGZgt{}}
\end{sphinxVerbatim}

This physically deletes all the data and file system structures.


\section{Restoring Metadata Backups}
\label{\detokenize{adm:restoring-metadata-backups}}
If the most-recent copy of the file system metadata has been damaged
irreparably, it is possible to restore one of the automatically
created backup copies.

The command

\begin{sphinxVerbatim}[commandchars=\\\{\}]
\PYG{l}{s3qladm download\PYGZhy{}metadata }\PYG{n+nv}{\PYGZlt{}storage url\PYGZgt{}}
\end{sphinxVerbatim}

will give you a list of the available metadata backups and allow you
to download them. This will create two new files in the current
directory, ending in \sphinxcode{\sphinxupquote{.db}} and \sphinxcode{\sphinxupquote{.params}}. To actually use the
downloaded backup, you need to move these files into the \sphinxcode{\sphinxupquote{\textasciitilde{}/.s3ql/}}
directory and run \sphinxcode{\sphinxupquote{fsck.s3ql}}.

\begin{sphinxadmonition}{warning}{Warning:}
You should probably not use this functionality without having asked
for help on the mailing list first (see {\hyperref[\detokenize{resources:resources}]{\sphinxcrossref{\DUrole{std,std-ref}{Further Resources / Getting Help}}}}).
\end{sphinxadmonition}


\chapter{Mounting}
\label{\detokenize{mount:mounting}}\label{\detokenize{mount::doc}}
A S3QL file system is mounted with the \sphinxstyleliteralstrong{\sphinxupquote{mount.s3ql}}
command. It has the following syntax:

\begin{sphinxVerbatim}[commandchars=\\\{\}]
\PYG{l}{mount.s3ql }\PYG{g+ge}{[options]}\PYG{l}{ }\PYG{n+nv}{\PYGZlt{}storage url\PYGZgt{}}\PYG{l}{ }\PYG{n+nv}{\PYGZlt{}mountpoint\PYGZgt{}}
\end{sphinxVerbatim}

\begin{sphinxadmonition}{note}{Note:}
S3QL is not a network file system like \sphinxhref{http://en.wikipedia.org/wiki/Network\_File\_System\_\%28protocol\%29}{NFS}
or \sphinxhref{http://en.wikipedia.org/wiki/CIFS}{CIFS}. It can only be
mounted on one computer at a time.
\end{sphinxadmonition}

This command accepts the following options:
\begin{quote}
\begin{optionlist}{3cm}
\item [-{-}log \textless{}target\textgreater{}]  
Destination for log messages. Specify \sphinxcode{\sphinxupquote{none}} for
standard output or \sphinxcode{\sphinxupquote{syslog}} for the system logging
daemon. Anything else will be interpreted as a file
name. Log files will be rotated when they reach 1 MiB,
and at most 5 old log files will be kept. Default:
\sphinxcode{\sphinxupquote{\textasciitilde{}/.s3ql/mount.log}}
\item [-{-}cachedir \textless{}path\textgreater{}]  
Store cached data in this directory (default:
\sphinxcode{\sphinxupquote{\textasciitilde{}/.s3ql)}}
\item [-{-}debug-modules \textless{}modules\textgreater{}]  
Activate debugging output from specified modules (use
commas to separate multiple modules). Debug messages
will be written to the target specified by the
\sphinxcode{\sphinxupquote{-{-}log}} option.
\item [-{-}debug]  
Activate debugging output from all S3QL modules. Debug
messages will be written to the target specified by
the \sphinxcode{\sphinxupquote{-{-}log}} option.
\item [-{-}quiet]  
be really quiet
\item [-{-}backend-options \textless{}options\textgreater{}]  
Backend specific options (separate by commas). See
backend documentation for available options.
\item [-{-}version]  
just print program version and exit
\item [-{-}authfile \textless{}path\textgreater{}]  
Read authentication credentials from this file
(default: \sphinxcode{\sphinxupquote{\textasciitilde{}/.s3ql/authinfo2)}}
\item [-{-}compress \textless{}algorithm-lvl\textgreater{}]  
Compression algorithm and compression level to use
when storing new data. \sphinxstyleemphasis{algorithm} may be any of
\sphinxcode{\sphinxupquote{lzma}}, \sphinxcode{\sphinxupquote{bzip2}}, \sphinxcode{\sphinxupquote{zlib}}, or none. \sphinxstyleemphasis{lvl} may be any
integer from 0 (fastest) to 9 (slowest). Default:
\sphinxcode{\sphinxupquote{lzma-6}}
\item [-{-}cachesize \textless{}size\textgreater{}]  
Cache size in KiB (default: autodetect).
\item [-{-}max-cache-entries \textless{}num\textgreater{}]  
Maximum number of entries in cache (default:
autodetect). Each cache entry requires one file
descriptor, so if you increase this number you have to
make sure that your process file descriptor limit (as
set with \sphinxcode{\sphinxupquote{ulimit -n}}) is high enough (at least the
number of cache entries + 100).
\item [-{-}keep-cache]  
Do not purge locally cached files on exit.
\item [-{-}allow-other]  
Normally, only the user who called \sphinxcode{\sphinxupquote{mount.s3ql}} can
access the mount point. This user then also has full
access to it, independent of individual file
permissions. If the \sphinxcode{\sphinxupquote{-{-}allow-other}} option is
specified, other users can access the mount point as
well and individual file permissions are taken into
account for all users.
\item [-{-}allow-root]  
Like \sphinxcode{\sphinxupquote{-{-}allow-other}}, but restrict access to the
mounting user and the root user.
\item [-{-}fg]  
Do not daemonize, stay in foreground
\item [-{-}fs-name FS\_NAME]  
Mount name passed to fuse, the name will be shown in
the first column of the system mount command output.
If not specified your storage url is used.
\item [-{-}systemd]  
Run as systemd unit. Consider specifying \textendash{}log none as
well to make use of journald.
\item [-{-}metadata-upload-interval \textless{}seconds\textgreater{}]  
Interval in seconds between complete metadata uploads.
Set to 0 to disable. Default: 24h.
\item [-{-}threads \textless{}no\textgreater{}]  
Number of parallel upload threads to use (default:
auto).
\item [-{-}nfs]  
Enable some optimizations for exporting the file
system over NFS. (default: False)
\end{optionlist}
\end{quote}


\section{Permission Checking}
\label{\detokenize{mount:permission-checking}}
If the file system is mounted with neither the \sphinxcode{\sphinxupquote{allow-root}}
nor \sphinxcode{\sphinxupquote{allow-other}} option, the mounting user has full
permissions on the S3QL file system (he is effectively root). If one
(or both) of the options is used, standard unix permission checks
apply, i.e. only the real root user has full access and all other
users (including the mounting user) are subject to permission checks.


\section{Compression Algorithms}
\label{\detokenize{mount:compression-algorithms}}
S3QL supports three compression algorithms, LZMA, Bzip2 and zlib (with
LZMA being the default). The compression algorithm can be specified
freely whenever the file system is mounted, since it affects only the
compression of new data blocks.

Roughly speaking, LZMA is slower but achieves better compression
ratios than Bzip2, while Bzip2 in turn is slower but achieves better
compression ratios than zlib.

For maximum file system performance, the best algorithm therefore
depends on your network connection speed: the compression algorithm
should be fast enough to saturate your network connection.

To find the optimal algorithm and number of parallel compression
threads for your system, S3QL ships with a program called
\sphinxcode{\sphinxupquote{benchmark.py}} in the \sphinxcode{\sphinxupquote{contrib}} directory. You should run this program
on a file that has a size that is roughly equal to the block size of
your file system and has similar contents. It will then determine the
compression speeds for the different algorithms and the upload speeds
for the specified backend and recommend the best algorithm that is
fast enough to saturate your network connection.

Obviously you should make sure that there is little other system load
when you run \sphinxcode{\sphinxupquote{benchmark.py}} (i.e., don’t compile software or encode
videos at the same time).


\section{Notes about Caching}
\label{\detokenize{mount:notes-about-caching}}
S3QL maintains a local cache of the file system data to speed up
access. The cache is block based, so it is possible that only parts of
a file are in the cache.


\subsection{Maximum Number of Cache Entries}
\label{\detokenize{mount:maximum-number-of-cache-entries}}
The maximum size of the cache can be configured with the
\sphinxcode{\sphinxupquote{-{-}cachesize}} option. In addition to that, the maximum number
of objects in the cache is limited by the
\sphinxcode{\sphinxupquote{-{-}max-cache-entries}} option, so it is possible that the cache
does not grow up to the maximum cache size because the maximum number
of cache elements has been reached. The reason for this limit is that
each cache entry requires one open file descriptor, and Linux
distributions usually limit the total number of file descriptors per
process to about a thousand.

If you specify a value for \sphinxcode{\sphinxupquote{-{-}max-cache-entries}}, you should
therefore make sure to also configure your system to increase the
maximum number of open file handles. This can be done temporarily with
the \sphinxstyleliteralstrong{\sphinxupquote{ulimit -n}} command. The method to permanently change this limit
system-wide depends on your distribution.


\subsection{Cache Flushing and Expiration}
\label{\detokenize{mount:cache-flushing-and-expiration}}
S3QL flushes changed blocks in the cache to the backend whenever a block
has not been accessed for at least 10 seconds. Note that when a block is
flushed, it still remains in the cache.

Cache expiration (i.e., removal of blocks from the cache) is only done
when the maximum cache size is reached. S3QL always expires the least
recently used blocks first.


\section{NFS Support}
\label{\detokenize{mount:nfs-support}}
S3QL filesystems can be exported over NFS. The \sphinxcode{\sphinxupquote{-{-}nfs}} option
is recommended to improve performance when NFS is used, but no harm
will occur when it is not specified.

NFS supports persistence of client mounts across server restarts. This
means that if a client has mounted an S3QL file system over NFS, the
server may unmount and remount the S3QL filesystem (or even reboot)
without the client being affected beyond temporarily becoming
unavailable. This poses several challenges, but is supported by S3QL
as long as no \sphinxcode{\sphinxupquote{fsck.s3ql}} operation is run:

\begin{sphinxadmonition}{warning}{Warning:}
If \sphinxcode{\sphinxupquote{fsck.s3ql}} modifies a file system in any way, all NFS
clients must unmount and re-mount the NFS share before the
S3QL file system is re-mounted on the server.
\end{sphinxadmonition}


\section{Failure Modes}
\label{\detokenize{mount:failure-modes}}
Once an S3QL file system has been mounted, there is a multitude of
problems that can occur when communicating with the remote
server. Generally, \sphinxstyleliteralstrong{\sphinxupquote{mount.s3ql}} always tries to keep the file
system as accessible as possible under the circumstances. That means
that if network connectivity is lost, data can still be written as
long as there is space in the local cache. Attempts to read data not
already present in the cache, however, will block until connection is
re-established. If any sort of data corruption is detected, the file
system will switch to read-only mode. Attempting to read files that
are affected by the corruption will return an input/output error
(\sphinxstyleemphasis{errno} set to \sphinxcode{\sphinxupquote{EIO}}).

In case of other unexpected or fatal problems, \sphinxstyleliteralstrong{\sphinxupquote{mount.s3ql}}
terminates, but does not unmount the file system. Any attempt to
access the mountpoint will result in a “Transport endpoint not
connected” error (\sphinxstyleemphasis{errno} set to \sphinxcode{\sphinxupquote{ESHUTDOWN}}). This ensures that a
mountpoint whose \sphinxstyleliteralstrong{\sphinxupquote{mount.s3ql}} process has terminated can not
be confused with a mountpoint containing an empty file system (which
would be fatal if e.g. the mountpoint is automatically mirrored). When
this has happened, the mountpoint can be cleared by using the
\sphinxstyleliteralstrong{\sphinxupquote{fusermount}} command (provided by FUSE) with the \sphinxcode{\sphinxupquote{-u}}
parameter.

\sphinxstyleliteralstrong{\sphinxupquote{mount.s3ql}} will automatically try to re-establish the
connection to the server if network connectivity is lost, and retry
sending a request when the connection is established but the remote
server signals a temporary problem. These attempts will be made at
increasing intervals for a period up to 24 hours, with retry intervals
starting at 20 ms and increasing up to 5 minutes. After 24 hours,
\sphinxstyleliteralstrong{\sphinxupquote{mount.s3ql}} will give up and terminate, leaving the
mountpoint inaccessible as described above.

Generally, \sphinxstyleliteralstrong{\sphinxupquote{mount.s3ql}} will also emit log messages for any
unusual conditions that it encounters. The destination for these
messages can be set with the \sphinxcode{\sphinxupquote{-{-}log}} parameter. It is highly
recommended to periodically check these logs, for example with a tool
like \sphinxhref{http://sourceforge.net/projects/logcheck/}{logcheck}. Many potential issues that \sphinxstyleliteralstrong{\sphinxupquote{mount.s3ql}} may
encounter do not justify restricting access to the file system, but
should nevertheless be investigated if they occur. Checking the log
messages is the only way to find out about them.


\section{Automatic Mounting}
\label{\detokenize{mount:automatic-mounting}}
If you want to mount and umount an S3QL file system automatically at
system startup and shutdown, you should do so with a dedicated S3QL
init job (instead of using \sphinxcode{\sphinxupquote{/etc/fstab}}. When using systemd,
\sphinxstyleliteralstrong{\sphinxupquote{mount.s3ql}} can be started with \sphinxcode{\sphinxupquote{-{-}systemd}} to run
as a systemd service of type \sphinxcode{\sphinxupquote{notify}}.

\begin{sphinxadmonition}{note}{Note:}
In principle, it is also possible to automatically mount an S3QL
file system with an appropriate entry in \sphinxcode{\sphinxupquote{/etc/fstab}}. However,
this is not recommended for several reasons:
\begin{itemize}
\item {} 
file systems mounted in \sphinxcode{\sphinxupquote{/etc/fstab}} will be unmounted with the
\sphinxstyleliteralstrong{\sphinxupquote{umount}} command, so your system will not wait until all data has
been uploaded but shutdown (or restart) immediately (this is a
FUSE limitation, cf \sphinxurl{https://github.com/libfuse/libfuse/issues/1}).

\item {} 
There is no way to tell the system that mounting S3QL requires a
Python interpreter to be available, so it may attempt to run
\sphinxstyleliteralstrong{\sphinxupquote{mount.s3ql}} before it has mounted the volume containing
the Python interpreter.

\item {} 
There is no standard way to tell the system that internet
connection has to be up before the S3QL file system can be
mounted.

\end{itemize}
\end{sphinxadmonition}


\chapter{Advanced S3QL Features}
\label{\detokenize{special:advanced-s3ql-features}}\label{\detokenize{special::doc}}

\section{Snapshotting and Copy-on-Write}
\label{\detokenize{special:snapshotting-and-copy-on-write}}\label{\detokenize{special:s3qlcp}}
The command \sphinxcode{\sphinxupquote{s3qlcp}} can be used to duplicate a directory tree without
physically copying the file contents. This is made possible by the
data de-duplication feature of S3QL.

The syntax of \sphinxcode{\sphinxupquote{s3qlcp}} is:

\begin{sphinxVerbatim}[commandchars=\\\{\}]
\PYG{l}{s3qlcp }\PYG{g+ge}{[options]}\PYG{l}{ }\PYG{n+nv}{\PYGZlt{}src\PYGZgt{}}\PYG{l}{ }\PYG{n+nv}{\PYGZlt{}target\PYGZgt{}}
\end{sphinxVerbatim}

This will replicate the contents of the directory \sphinxcode{\sphinxupquote{\textless{}src\textgreater{}}} in the
directory \sphinxcode{\sphinxupquote{\textless{}target\textgreater{}}}. \sphinxcode{\sphinxupquote{\textless{}src\textgreater{}}} has to be an existing directory and
\sphinxcode{\sphinxupquote{\textless{}target\textgreater{}}} must not exist. Moreover, both directories have to be
within the same S3QL file system.

The replication will not take any additional space. Only if one of
directories is modified later on, the modified data will take
additional storage space.

\sphinxcode{\sphinxupquote{s3qlcp}} can only be called by the user that mounted the file system
and (if the file system was mounted with \sphinxcode{\sphinxupquote{-{-}allow-other}} or \sphinxcode{\sphinxupquote{-{-}allow-root}})
the root user.

Note that:
\begin{itemize}
\item {} 
After the replication, both source and target directory will still
be completely ordinary directories. You can regard \sphinxcode{\sphinxupquote{\textless{}src\textgreater{}}} as a
snapshot of \sphinxcode{\sphinxupquote{\textless{}target\textgreater{}}} or vice versa. However, the most common
usage of \sphinxcode{\sphinxupquote{s3qlcp}} is to regularly duplicate the same source
directory, say \sphinxcode{\sphinxupquote{documents}}, to different target directories. For a
e.g. monthly replication, the target directories would typically be
named something like \sphinxcode{\sphinxupquote{documents\_January}} for the replication in
January, \sphinxcode{\sphinxupquote{documents\_February}} for the replication in February etc.
In this case it is clear that the target directories should be
regarded as snapshots of the source directory.

\item {} 
Exactly the same effect could be achieved by an ordinary copy
program like \sphinxcode{\sphinxupquote{cp -a}}. However, this procedure would be orders of
magnitude slower, because \sphinxcode{\sphinxupquote{cp}} would have to read every file
completely (so that S3QL had to fetch all the data over the network
from the backend) before writing them into the destination folder.

\end{itemize}


\subsection{Snapshotting vs Hardlinking}
\label{\detokenize{special:snapshotting-vs-hardlinking}}
Snapshot support in S3QL is inspired by the hardlinking feature that
is offered by programs like \sphinxhref{http://www.samba.org/rsync}{rsync} or
\sphinxhref{http://savannah.nongnu.org/projects/storebackup}{storeBackup}.
These programs can create a hardlink instead of copying a file if an
identical file already exists in the backup. However, using hardlinks
has two large disadvantages:
\begin{itemize}
\item {} 
backups and restores always have to be made with a special program
that takes care of the hardlinking. The backup must not be touched
by any other programs (they may make changes that inadvertently
affect other hardlinked files)

\item {} 
special care needs to be taken to handle files which are already
hardlinked (the restore program needs to know that the hardlink was
not just introduced by the backup program to safe space)

\end{itemize}

S3QL snapshots do not have these problems, and they can be used with
any backup program.


\section{Getting Statistics}
\label{\detokenize{special:getting-statistics}}\label{\detokenize{special:s3qlstat}}
You can get more information about a mounted S3QL file system with the
\sphinxcode{\sphinxupquote{s3qlstat}} command. It has the following syntax:

\begin{sphinxVerbatim}[commandchars=\\\{\}]
\PYG{l}{s3qlstat }\PYG{g+ge}{[options]}\PYG{l}{ }\PYG{n+nv}{\PYGZlt{}mountpoint\PYGZgt{}}
\end{sphinxVerbatim}

This will print out something like this

\begin{sphinxVerbatim}[commandchars=\\\{\}]
\PYG{l}{Directory entries:    1488068}
\PYG{l}{Inodes:               1482991}
\PYG{l}{Data blocks:          87948}
\PYG{l}{Total data size:      400 GiB}
\PYG{l}{After de\PYGZhy{}duplication: 51 GiB (12.98\PYGZpc{} of total)}
\PYG{l}{After compression:    43 GiB (10.85\PYGZpc{} of total, 83.60\PYGZpc{} of de\PYGZhy{}duplicated)}
\PYG{l}{Database size:        172 MiB (uncompressed)}
\PYG{l}{(some values do not take into account not\PYGZhy{}yet\PYGZhy{}uploaded dirty blocks in cache)}
\end{sphinxVerbatim}

Probably the most interesting numbers are the total size of your data,
the total size after duplication, and the final size after
de-duplication and compression.

\sphinxcode{\sphinxupquote{s3qlstat}} can only be called by the user that mounted the file system
and (if the file system was mounted with \sphinxcode{\sphinxupquote{-{-}allow-other}} or \sphinxcode{\sphinxupquote{-{-}allow-root}})
the root user.

For a full list of available options, run \sphinxcode{\sphinxupquote{s3qlstat -{-}help}}.


\section{Immutable Trees}
\label{\detokenize{special:immutable-trees}}\label{\detokenize{special:s3qllock}}
The command \sphinxstyleliteralstrong{\sphinxupquote{s3qllock}} can be used to make a directory tree
immutable. Immutable trees can no longer be changed in any way
whatsoever. You can not add new files or directories and you can not
change or delete existing files and directories. The only way to get
rid of an immutable tree is to use the \sphinxstyleliteralstrong{\sphinxupquote{s3qlrm}} command (see
below).

For example, to make the directory tree beneath the directory
\sphinxcode{\sphinxupquote{2010-04-21}} immutable, execute

\begin{sphinxVerbatim}[commandchars=\\\{\}]
\PYG{l}{s3qllock 2010\PYGZhy{}04\PYGZhy{}21}
\end{sphinxVerbatim}

Immutability is a feature designed for backups. Traditionally, backups
have been made on external tape drives. Once a backup was made, the
tape drive was removed and locked somewhere in a shelf. This has the
great advantage that the contents of the backup are now permanently
fixed. Nothing (short of physical destruction) can change or delete
files in the backup.

In contrast, when backing up into an online storage system like S3QL,
all backups are available every time the file system is mounted.
Nothing prevents a file in an old backup from being changed again
later on. In the worst case, this may make your entire backup system
worthless. Imagine that your system gets infected by a nasty virus
that simply deletes all files it can find \textendash{} if the virus is active
while the backup file system is mounted, the virus will destroy all
your old backups as well!

Even if the possibility of a malicious virus or trojan horse is
excluded, being able to change a backup after it has been made is
generally not a good idea. A common S3QL use case is to keep the file
system mounted at all times and periodically create backups with
\sphinxstyleliteralstrong{\sphinxupquote{rsync -a}}. This allows every user to recover her files from a
backup without having to call the system administrator. However, this
also allows every user to accidentally change or delete files \sphinxstyleemphasis{in} one
of the old backups.

Making a backup immutable protects you against all these problems.
Unless you happen to run into a virus that was specifically programmed
to attack S3QL file systems, backups can be neither deleted nor
changed after they have been made immutable.


\section{Fast Recursive Removal}
\label{\detokenize{special:fast-recursive-removal}}\label{\detokenize{special:s3qlrm}}
The \sphinxcode{\sphinxupquote{s3qlrm}} command can be used to recursively delete files and
directories on an S3QL file system. Although \sphinxcode{\sphinxupquote{s3qlrm}} is faster than
using e.g. \sphinxcode{\sphinxupquote{rm -r}}, the main reason for its existence is that it
allows you to delete immutable trees as well. The syntax is rather
simple:

\begin{sphinxVerbatim}[commandchars=\\\{\}]
\PYG{l}{s3qlrm }\PYG{n+nv}{\PYGZlt{}directory\PYGZgt{}}
\end{sphinxVerbatim}

Be warned that there is no additional confirmation. The directory will
be removed entirely and immediately.


\section{Runtime Configuration}
\label{\detokenize{special:runtime-configuration}}\label{\detokenize{special:s3qlctrl}}
The \sphinxcode{\sphinxupquote{s3qlctrl}} can be used to control a mounted S3QL file system. Its
syntax is

\begin{sphinxVerbatim}[commandchars=\\\{\}]
\PYG{l}{s3qlctrl }\PYG{g+ge}{[options]}\PYG{l}{ }\PYG{n+nv}{\PYGZlt{}action\PYGZgt{}}\PYG{l}{ }\PYG{n+nv}{\PYGZlt{}mountpoint\PYGZgt{}}\PYG{l}{ ...}
\end{sphinxVerbatim}

\sphinxcode{\sphinxupquote{\textless{}mountpoint\textgreater{}}} must be the location of a mounted S3QL file system.
For a list of valid options, run \sphinxcode{\sphinxupquote{s3qlctrl -{-}help}}. \sphinxcode{\sphinxupquote{\textless{}action\textgreater{}}}
may be either of:
\begin{quote}
\begin{quote}\begin{description}
\item[{flushcache}] \leavevmode
Flush file system cache. The command blocks until the cache has
been flushed.

\item[{dropcache}] \leavevmode
Flush, and then drop file system cache. The command
blocks until the cache has been flushed and dropped.

\item[{log}] \leavevmode
Change log level.

\item[{cachesize}] \leavevmode
Change file system cache size.

\item[{upload-meta}] \leavevmode
Trigger a metadata upload.

\end{description}\end{quote}
\end{quote}


\chapter{Unmounting}
\label{\detokenize{umount:unmounting}}\label{\detokenize{umount::doc}}
To unmount an S3QL file system, use the command:

\begin{sphinxVerbatim}[commandchars=\\\{\}]
\PYG{l}{umount.s3ql }\PYG{g+ge}{[options]}\PYG{l}{ }\PYG{n+nv}{\PYGZlt{}mountpoint\PYGZgt{}}
\end{sphinxVerbatim}

This will block until all data has been written to the backend.

Only the user who mounted the file system with \sphinxstyleliteralstrong{\sphinxupquote{mount.s3ql}}
is able to unmount it again. If you are root and want to unmount an
S3QL file system mounted by an ordinary user, you have to use the
\sphinxstyleliteralstrong{\sphinxupquote{fusermount -u}} or \sphinxstyleliteralstrong{\sphinxupquote{umount}} command instead. Note
that these commands do not block until all data has been uploaded, so
if you use them instead of \sphinxcode{\sphinxupquote{umount.s3ql}} then you should manually wait
for the \sphinxcode{\sphinxupquote{mount.s3ql}} process to terminate before shutting down the
system.

The \sphinxstyleliteralstrong{\sphinxupquote{umount.s3ql}} command accepts the following options:
\begin{quote}
\begin{optionlist}{3cm}
\item [-{-}log \textless{}target\textgreater{}]  
Destination for log messages. Specify \sphinxcode{\sphinxupquote{none}} for
standard output or \sphinxcode{\sphinxupquote{syslog}} for the system logging
daemon. Anything else will be interpreted as a file
name. Log files will be rotated when they reach 1 MiB,
and at most 5 old log files will be kept. Default:
\sphinxcode{\sphinxupquote{None}}
\item [-{-}debug-modules \textless{}modules\textgreater{}]  
Activate debugging output from specified modules (use
commas to separate multiple modules). Debug messages
will be written to the target specified by the
\sphinxcode{\sphinxupquote{-{-}log}} option.
\item [-{-}debug]  
Activate debugging output from all S3QL modules. Debug
messages will be written to the target specified by
the \sphinxcode{\sphinxupquote{-{-}log}} option.
\item [-{-}quiet]  
be really quiet
\item [-{-}version]  
just print program version and exit
\item [-{-}lazy, -z]  
Lazy umount. Detaches the file system immediately,
even if there are still open files. The data will be
uploaded in the background once all open files have
been closed.
\end{optionlist}
\end{quote}

If, for some reason, the \sphinxcode{\sphinxupquote{umount.sql}} command does not work, the file
system can also be unmounted with \sphinxcode{\sphinxupquote{fusermount -u -z}}. Note that this
command will return immediately and the file system may continue to
upload data in the background for a while longer.


\chapter{Checking for Errors}
\label{\detokenize{fsck:checking-for-errors}}\label{\detokenize{fsck::doc}}
It is recommended to periodically run the \sphinxstyleliteralstrong{\sphinxupquote{fsck.s3ql}} and
\sphinxstyleliteralstrong{\sphinxupquote{s3ql\_verify}} commands (in this order) to ensure that the
file system is consistent, and that there has been no data corruption
or data loss in the storage backend.

\sphinxstyleliteralstrong{\sphinxupquote{fsck.s3ql}} is intended to detect and correct problems with
the internal file system structure, caused by e.g. a file system crash
or a bug in S3QL. It assumes that the storage backend can be fully
trusted, i.e. if the backend reports that a specific storage object
exists, \sphinxstyleliteralstrong{\sphinxupquote{fsck.s3ql}} takes that as proof that the data is
present and intact.

In contrast to that, the \sphinxstyleliteralstrong{\sphinxupquote{s3ql\_verify}} command is intended to
check the consistency of the storage backend. It assumes that the
internal file system data is correct, and verifies that all data can
actually be retrieved from the backend. Running \sphinxstyleliteralstrong{\sphinxupquote{s3ql\_verify}}
may therefore take much longer than running \sphinxstyleliteralstrong{\sphinxupquote{fsck.s3ql}}.


\section{Checking and repairing internal file system errors}
\label{\detokenize{fsck:checking-and-repairing-internal-file-system-errors}}
\sphinxstyleliteralstrong{\sphinxupquote{fsck.s3ql}} checks that the internal file system structure is
consistent and attempts to correct any problems it finds. If an S3QL
file system has not been unmounted correcly for any reason, you need
to run \sphinxstyleliteralstrong{\sphinxupquote{fsck.s3ql}} before you can mount the file system
again.

The \sphinxstyleliteralstrong{\sphinxupquote{fsck.s3ql}} command has the following syntax:

\begin{sphinxVerbatim}[commandchars=\\\{\}]
\PYG{l}{fsck.s3ql }\PYG{g+ge}{[options]}\PYG{l}{ }\PYG{n+nv}{\PYGZlt{}storage url\PYGZgt{}}
\end{sphinxVerbatim}

This command accepts the following options:
\begin{quote}
\begin{optionlist}{3cm}
\item [-{-}log \textless{}target\textgreater{}]  
Destination for log messages. Specify \sphinxcode{\sphinxupquote{none}} for
standard output or \sphinxcode{\sphinxupquote{syslog}} for the system logging
daemon. Anything else will be interpreted as a file
name. Log files will be rotated when they reach 1 MiB,
and at most 5 old log files will be kept. Default:
\sphinxcode{\sphinxupquote{\textasciitilde{}/.s3ql/fsck.log}}
\item [-{-}cachedir \textless{}path\textgreater{}]  
Store cached data in this directory (default:
\sphinxcode{\sphinxupquote{\textasciitilde{}/.s3ql)}}
\item [-{-}debug-modules \textless{}modules\textgreater{}]  
Activate debugging output from specified modules (use
commas to separate multiple modules). Debug messages
will be written to the target specified by the
\sphinxcode{\sphinxupquote{-{-}log}} option.
\item [-{-}debug]  
Activate debugging output from all S3QL modules. Debug
messages will be written to the target specified by
the \sphinxcode{\sphinxupquote{-{-}log}} option.
\item [-{-}quiet]  
be really quiet
\item [-{-}backend-options \textless{}options\textgreater{}]  
Backend specific options (separate by commas). See
backend documentation for available options.
\item [-{-}version]  
just print program version and exit
\item [-{-}authfile \textless{}path\textgreater{}]  
Read authentication credentials from this file
(default: \sphinxcode{\sphinxupquote{\textasciitilde{}/.s3ql/authinfo2)}}
\item [-{-}compress \textless{}algorithm-lvl\textgreater{}]  
Compression algorithm and compression level to use
when storing new data. \sphinxstyleemphasis{algorithm} may be any of
\sphinxcode{\sphinxupquote{lzma}}, \sphinxcode{\sphinxupquote{bzip2}}, \sphinxcode{\sphinxupquote{zlib}}, or none. \sphinxstyleemphasis{lvl} may be any
integer from 0 (fastest) to 9 (slowest). Default:
\sphinxcode{\sphinxupquote{lzma-6}}
\item [-{-}keep-cache]  
Do not purge locally cached files on exit.
\item [-{-}batch]  
If user input is required, exit without prompting.
\item [-{-}force]  
Force checking even if file system is marked clean.
\item [-{-}force-remote]  
Force use of remote metadata even when this would
likely result in data loss.
\end{optionlist}
\end{quote}


\section{Detecting and handling backend data corruption}
\label{\detokenize{fsck:detecting-and-handling-backend-data-corruption}}\label{\detokenize{fsck:s3ql-verify}}
The \sphinxstyleliteralstrong{\sphinxupquote{s3ql\_verify}} command verifies all data in the file
system.  In contrast to \sphinxstyleliteralstrong{\sphinxupquote{fsck.s3ql}}, \sphinxstyleliteralstrong{\sphinxupquote{s3ql\_verify}}
does not trust the object listing returned by the backend, but
actually attempts to retrieve every object. By default,
\sphinxstyleliteralstrong{\sphinxupquote{s3ql\_verify}} will attempt to retrieve just the metadata for
every object (for e.g. the S3-compatible or Google Storage backends
this corresponds to a \sphinxcode{\sphinxupquote{HEAD}} request for each object), which is
generally sufficient to determine if the object still exists. When
specifying the \sphinxcode{\sphinxupquote{-{-}data}} option, \sphinxstyleliteralstrong{\sphinxupquote{s3ql\_verify}} will
instead read every object entirely. To determine how much data will be
transmitted in total when using \sphinxcode{\sphinxupquote{-{-}data}}, look at the \sphinxstyleemphasis{After
compression} row in the {\hyperref[\detokenize{special:s3qlstat}]{\sphinxcrossref{\DUrole{std,std-ref}{s3qlstat}}}} output.

\sphinxstyleliteralstrong{\sphinxupquote{s3ql\_verify}} is not able to correct any data corruption that
it finds. Instead, a list of the corrupted and/or missing objects is
written to a file and the decision about the proper course of action
is left to the user. If you have administrative access to the backend
server, you may want to investigate the cause of the corruption or
check if the missing/corrupted objects can be restored from
backups. If you believe that the missing/corrupted objects are indeed
lost irrevocably, you can use the {\hyperref[\detokenize{contrib:remove-objects}]{\sphinxcrossref{\DUrole{std,std-ref}{remove\_objects.py}}}} script (from
the \sphinxcode{\sphinxupquote{contrib}} directory of the S3QL distribution) to explicitly
delete the objects from the storage backend. After that, you should
run \sphinxstyleliteralstrong{\sphinxupquote{fsck.s3ql}}. Since the (now explicitly deleted) objects
should now no longer be included in the object index reported by the
backend, \sphinxstyleliteralstrong{\sphinxupquote{fsck.s3ql}} will identify the objects as missing,
update the internal file system structures accordingly, and move the
affected files into the \sphinxcode{\sphinxupquote{lost+found}} directory.

The \sphinxstyleliteralstrong{\sphinxupquote{s3ql\_verify}} command has the following syntax:

\begin{sphinxVerbatim}[commandchars=\\\{\}]
\PYG{l}{s3ql\PYGZus{}verify }\PYG{g+ge}{[options]}\PYG{l}{ }\PYG{n+nv}{\PYGZlt{}storage url\PYGZgt{}}
\end{sphinxVerbatim}

This command accepts the following options:
\begin{quote}
\begin{optionlist}{3cm}
\item [-{-}log \textless{}target\textgreater{}]  
Destination for log messages. Specify \sphinxcode{\sphinxupquote{none}} for
standard output or \sphinxcode{\sphinxupquote{syslog}} for the system logging
daemon. Anything else will be interpreted as a file
name. Log files will be rotated when they reach 1 MiB,
and at most 5 old log files will be kept. Default:
\sphinxcode{\sphinxupquote{None}}
\item [-{-}debug-modules \textless{}modules\textgreater{}]  
Activate debugging output from specified modules (use
commas to separate multiple modules). Debug messages
will be written to the target specified by the
\sphinxcode{\sphinxupquote{-{-}log}} option.
\item [-{-}debug]  
Activate debugging output from all S3QL modules. Debug
messages will be written to the target specified by
the \sphinxcode{\sphinxupquote{-{-}log}} option.
\item [-{-}quiet]  
be really quiet
\item [-{-}version]  
just print program version and exit
\item [-{-}cachedir \textless{}path\textgreater{}]  
Store cached data in this directory (default:
\sphinxcode{\sphinxupquote{\textasciitilde{}/.s3ql)}}
\item [-{-}backend-options \textless{}options\textgreater{}]  
Backend specific options (separate by commas). See
backend documentation for available options.
\item [-{-}authfile \textless{}path\textgreater{}]  
Read authentication credentials from this file
(default: \sphinxcode{\sphinxupquote{\textasciitilde{}/.s3ql/authinfo2)}}
\item [-{-}missing-file \textless{}name\textgreater{}]  
File to store keys of missing objects.
\item [-{-}corrupted-file \textless{}name\textgreater{}]  
File to store keys of corrupted objects.
\item [-{-}data]  
Read every object completely, instead of checking just
the metadata.
\item [-{-}parallel PARALLEL]  
Number of connections to use in parallel.
\item [-{-}start-with \textless{}n\textgreater{}]  
Skip over first \textless{}n\textgreater{} objects and with verifying object
\textless{}n\textgreater{}+1.
\end{optionlist}
\end{quote}


\chapter{Storing Backend Information and Credentials}
\label{\detokenize{authinfo:storing-backend-information-and-credentials}}\label{\detokenize{authinfo:authinfo}}\label{\detokenize{authinfo::doc}}
Normally, S3QL reads username and password for the backend as well as
an encryption passphrase for the file system from the
terminal. Alternatively, these (and some other backend parameters) may
be specified in the \sphinxcode{\sphinxupquote{\textasciitilde{}/.s3ql/authinfo2}} file, or a file specified by the
\sphinxcode{\sphinxupquote{-{-}authfile}} parameter.

The authinfo file consists of sections, led by a \sphinxcode{\sphinxupquote{{[}section{]}}}
header and followed by \sphinxcode{\sphinxupquote{name: value}} entries. The section headers
themselves are not used by S3QL but have to be unique within the file.

In each section, the following entries can be defined:
\begin{quote}\begin{description}
\item[{storage-url}] \leavevmode
Specifies the storage url to which this section applies. If a
storage url starts with the value of this entry, the section is
considered applicable.

\item[{backend-login}] \leavevmode
Specifies the username to use for authentication with the backend.

\item[{backend-password}] \leavevmode
Specifies the password to use for authentication with the backend.

\item[{fs-passphrase}] \leavevmode
Specifies the passphrase to use to decrypt the file system (if
it is encrypted).

\end{description}\end{quote}

Furthermore, command line parameters that are shared between all S3QL
commands that work with storage URL may be specified in this file as
well by omitting the leading dashes, e.g. the \sphinxcode{\sphinxupquote{-{-}backend-options
notls}} parameter may be written into the authinfo file as
\sphinxcode{\sphinxupquote{backend-options: notls}}.

When reading the authinfo file, S3QL considers every applicable
section in order and uses the last value that it found for each entry.
For example, consider the following authentication file:

\begin{sphinxVerbatim}[commandchars=\\\{\}]
\PYG{g+ge}{[s3]}
\PYG{l}{storage\PYGZhy{}url: s3://}
\PYG{l}{backend\PYGZhy{}login: joe}
\PYG{l}{backend\PYGZhy{}password: notquitesecret}

\PYG{g+ge}{[fs1]}
\PYG{l}{storage\PYGZhy{}url: s3://joes\PYGZhy{}first\PYGZhy{}bucket}
\PYG{l}{fs\PYGZhy{}passphrase: neitheristhis}

\PYG{g+ge}{[fs2]}
\PYG{l}{storage\PYGZhy{}url: s3://joes\PYGZhy{}second\PYGZhy{}bucket}
\PYG{l}{fs\PYGZhy{}passphrase: swordfish}

\PYG{g+ge}{[fs3]}
\PYG{l}{storage\PYGZhy{}url: s3://joes\PYGZhy{}second\PYGZhy{}bucket/with\PYGZhy{}prefix}
\PYG{l}{backend\PYGZhy{}login: bill}
\PYG{l}{backend\PYGZhy{}password: bi23ll}
\PYG{l}{fs\PYGZhy{}passphrase: ll23bi}
\end{sphinxVerbatim}

With this authentication file, S3QL would try to log in as “joe”
whenever the s3 backend is used, except when accessing a storage url
that begins with “s3://joes-second-bucket/with-prefix”. In that case,
the last section becomes active and S3QL would use the “bill”
credentials. Furthermore, file system encryption passphrases will be used
for storage urls that start with “s3://joes-first-bucket” or
“s3://joes-second-bucket”.

The authentication file is parsed by the \sphinxhref{http://docs.python.org/library/configparser.html}{Python ConfigParser
module}.


\chapter{Contributed Programs}
\label{\detokenize{contrib:contributed-programs}}\label{\detokenize{contrib::doc}}
S3QL comes with a few contributed programs that are not part of the
core distribution (and are therefore not installed automatically by
default), but which may nevertheless be useful. These programs are in
the \sphinxcode{\sphinxupquote{contrib}} directory of the source distribution or in
\sphinxcode{\sphinxupquote{/usr/share/doc/s3ql/contrib}} if you installed S3QL from a package.


\section{benchmark.py}
\label{\detokenize{contrib:benchmark-py}}
This program measures S3QL write performance, uplink bandwidth and
compression speed to determine the limiting factor. It also gives
recommendation for compression algorithm and number of upload threads
to achieve maximum performance.


\section{clone\_fs.py}
\label{\detokenize{contrib:clone-fs-py}}
This program physically clones an S3QL file system from one backend
into another, without recompressing or reencrypting.  It can be used to
migrate S3 buckets to a different storage region or storage class
(standard or reduced redundancy).


\section{pcp.py}
\label{\detokenize{contrib:pcp-py}}\label{\detokenize{contrib:pcp}}
\sphinxcode{\sphinxupquote{pcp.py}} is a wrapper program that starts several rsync processes to
copy directory trees in parallel. This is important because
transferring files in parallel significantly enhances performance when
copying data from an S3QL file system (see {\hyperref[\detokenize{tips:copy-performance}]{\sphinxcrossref{\DUrole{std,std-ref}{Improving copy performance}}}} for
details).

To recursively copy the directory \sphinxcode{\sphinxupquote{/mnt/home-backup}} into
\sphinxcode{\sphinxupquote{/home/joe}} using 8 parallel processes and preserving permissions,
you would execute

\begin{sphinxVerbatim}[commandchars=\\\{\}]
\PYG{l}{pcp.py \PYGZhy{}a \PYGZhy{}\PYGZhy{}processes=8 /mnt/home\PYGZhy{}backup/ /home/joe}
\end{sphinxVerbatim}


\section{s3ql\_backup.sh}
\label{\detokenize{contrib:s3ql-backup-sh}}
This is an example script that demonstrates how to set up a simple but
powerful backup solution using S3QL and \sphinxhref{http://samba.org/rsync}{rsync}.

The \sphinxcode{\sphinxupquote{s3ql\_backup.sh}} script automates the following steps:
\begin{enumerate}
\def\theenumi{\arabic{enumi}}
\def\labelenumi{\theenumi .}
\makeatletter\def\p@enumii{\p@enumi \theenumi .}\makeatother
\item {} 
Mount the file system

\item {} 
Replicate the previous backup with {\hyperref[\detokenize{special:s3qlcp}]{\sphinxcrossref{\DUrole{std,std-ref}{s3qlcp}}}}

\item {} 
Update the new copy with the data from the backup source using rsync

\item {} 
Make the new backup immutable with {\hyperref[\detokenize{special:s3qllock}]{\sphinxcrossref{\DUrole{std,std-ref}{s3qllock}}}}

\item {} 
Delete old backups that are no longer needed

\item {} 
Unmount the file system

\end{enumerate}

The backups are stored in directories of the form
\sphinxcode{\sphinxupquote{YYYY-MM-DD\_HH:mm:SS}} and the {\hyperref[\detokenize{contrib:expire-backups-py}]{\sphinxcrossref{expire\_backups.py}}} command is used to
delete old backups.


\section{expire\_backups.py}
\label{\detokenize{contrib:expire-backups-py}}
\sphinxstyleliteralstrong{\sphinxupquote{expire\_backups.py}} is a program to intelligently remove old
backups that are no longer needed.

To define what backups you want to keep for how long, you define a
number of \sphinxstyleemphasis{age ranges}. \sphinxstyleliteralstrong{\sphinxupquote{expire\_backups}} ensures that you
will have at least one backup in each age range at all times. It will
keep exactly as many backups as are required for that and delete any
backups that become redundant.

Age ranges are specified by giving a list of range boundaries in terms
of backup cycles. Every time you create a new backup, the existing
backups age by one cycle.

Example: when \sphinxstyleliteralstrong{\sphinxupquote{expire\_backups}} is called with the age range
definition \sphinxcode{\sphinxupquote{1 3 7 14 31}}, it will guarantee that you always have the
following backups available:
\begin{enumerate}
\def\theenumi{\arabic{enumi}}
\def\labelenumi{\theenumi .}
\makeatletter\def\p@enumii{\p@enumi \theenumi .}\makeatother
\item {} 
A backup that is 0 to 1 cycles old (i.e, the most recent backup)

\item {} 
A backup that is 1 to 3 cycles old

\item {} 
A backup that is 3 to 7 cycles old

\item {} 
A backup that is 7 to 14 cycles old

\item {} 
A backup that is 14 to 31 cycles old

\end{enumerate}

\begin{sphinxadmonition}{note}{Note:}
If you do backups in fixed intervals, then one cycle will be
equivalent to the backup interval. The advantage of specifying the
age ranges in terms of backup cycles rather than days or weeks is
that it allows you to gracefully handle irregular backup intervals.
Imagine that for some reason you do not turn on your computer for
one month. Now all your backups are at least a month old, and if you
had specified the above backup strategy in terms of absolute ages,
they would all be deleted! Specifying age ranges in terms of backup
cycles avoids these sort of problems.
\end{sphinxadmonition}

\sphinxstyleliteralstrong{\sphinxupquote{expire\_backups}} usage is simple. It requires backups to be
stored in directories of the form \sphinxcode{\sphinxupquote{year-month-day\_hour:minute:seconds}}
(\sphinxcode{\sphinxupquote{YYYY-MM-DD\_HH:mm:ss}}) and works on all backups in the current
directory. So for the above backup strategy, the correct invocation
would be:

\begin{sphinxVerbatim}[commandchars=\\\{\}]
\PYG{l}{expire\PYGZus{}backups.py 1 3 7 14 31}
\end{sphinxVerbatim}

When storing your backups on an S3QL file system, you probably want to
specify the \sphinxcode{\sphinxupquote{-{-}use-s3qlrm}} option as well. This tells
\sphinxstyleliteralstrong{\sphinxupquote{expire\_backups}} to use the {\hyperref[\detokenize{special:s3qlrm}]{\sphinxcrossref{\DUrole{std,std-ref}{s3qlrm}}}} command to
delete directories.

\sphinxstyleliteralstrong{\sphinxupquote{expire\_backups}} uses a “state file” to keep track which
backups are how many cycles old (since this cannot be inferred from
the dates contained in the directory names). The standard name for
this state file is \sphinxcode{\sphinxupquote{.expire\_backups.dat}}. If this file gets
damaged or deleted, \sphinxstyleliteralstrong{\sphinxupquote{expire\_backups}} no longer knows the ages
of the backups and refuses to work. In this case you can use the
\sphinxcode{\sphinxupquote{-{-}reconstruct-state}} option to try to reconstruct the state
from the backup dates. However, the accuracy of this reconstruction
depends strongly on how rigorous you have been with making backups (it
is only completely correct if the time between subsequent backups has
always been exactly the same), so it’s generally a good idea not to
tamper with the state file.

For a full list of available options, run \sphinxstyleliteralstrong{\sphinxupquote{expire\_backups.py
-{-}help}}.


\section{remove\_objects.py}
\label{\detokenize{contrib:remove-objects-py}}\label{\detokenize{contrib:remove-objects}}
\sphinxstyleliteralstrong{\sphinxupquote{remove\_objects.py}} is a program to remove a list of objects
from a storage backend. Since it acts on the backend-level, the
backend need not contain an S3QL file system.


\chapter{Tips \& Tricks}
\label{\detokenize{tips:tips-tricks}}\label{\detokenize{tips::doc}}

\section{SSH Backend}
\label{\detokenize{tips:ssh-backend}}\label{\detokenize{tips:ssh-tipp}}
By combining S3QL’s local backend with \sphinxhref{http://fuse.sourceforge.net/sshfs.html}{sshfs}, it is possible to store an
S3QL file system on arbitrary SSH servers: first mount the remote
target directory into the local filesystem,

\begin{sphinxVerbatim}[commandchars=\\\{\}]
\PYG{l}{sshfs user@my.server.com:/mnt/s3ql /mnt/sshfs}
\end{sphinxVerbatim}

and then give the mountpoint to S3QL as a local destination:

\begin{sphinxVerbatim}[commandchars=\\\{\}]
\PYG{l}{mount.s3ql local:///mnt/sshfs/myfsdata /mnt/s3ql}
\end{sphinxVerbatim}


\section{Permanently mounted backup file system}
\label{\detokenize{tips:permanently-mounted-backup-file-system}}
If you use S3QL as a backup file system, it can be useful to mount the
file system permanently (rather than just mounting it for a backup and
unmounting it afterwards). Especially if your file system becomes
large, this saves you long mount- and unmount times if you only want
to restore a single file.

If you decide to do so, you should make sure to
\begin{itemize}
\item {} 
Use {\hyperref[\detokenize{special:s3qllock}]{\sphinxcrossref{\DUrole{std,std-ref}{s3qllock}}}} to ensure that backups are immutable
after they have been made.

\item {} 
Call {\hyperref[\detokenize{special:s3qlctrl}]{\sphinxcrossref{\DUrole{std,std-ref}{s3qlctrl upload-meta}}}} right after a every
backup to make sure that the newest metadata is stored safely (if
you do backups often enough, this may also allow you to set the
\sphinxcode{\sphinxupquote{-{-}metadata-upload-interval}} option of \sphinxstyleliteralstrong{\sphinxupquote{mount.s3ql}}
to zero).

\end{itemize}


\section{Improving copy performance}
\label{\detokenize{tips:improving-copy-performance}}\label{\detokenize{tips:copy-performance}}
\begin{sphinxadmonition}{note}{Note:}
The following applies only when copying data \sphinxstylestrong{from} an S3QL file
system, \sphinxstylestrong{not} when copying data \sphinxstylestrong{to} an S3QL file system.
\end{sphinxadmonition}

If you want to copy a lot of smaller files \sphinxstyleemphasis{from} an S3QL file system
(e.g. for a system restore) you will probably notice that the
performance is rather bad.

The reason for this is intrinsic to the way S3QL works. Whenever you
read a file, S3QL first has to retrieve this file over the network
from the backend. This takes a minimum amount of time (the network
latency), no matter how big or small the file is. So when you copy
lots of small files, 99\% of the time is actually spend waiting for
network data.

Theoretically, this problem is easy to solve: you just have to copy
several files at the same time. In practice, however, almost all unix
utilities (\sphinxcode{\sphinxupquote{cp}}, \sphinxcode{\sphinxupquote{rsync}}, \sphinxcode{\sphinxupquote{tar}} and friends) insist on copying
data one file at a time. This makes a lot of sense when copying data
on the local hard disk, but in case of S3QL this is really
unfortunate.

The best workaround that has been found so far is to copy files by
starting several rsync processes at once and use exclusion rules to
make sure that they work on different sets of files.

For example, the following script will start 3 rsync instances. The
first instance handles all filenames starting with a-f, the second the
filenames from g-l and the third covers the rest. The \sphinxcode{\sphinxupquote{+ */}} rule
ensures that every instance looks into all directories.

\begin{sphinxVerbatim}[commandchars=\\\{\}]
\PYG{c}{\PYGZsh{}!/bin/bash}

\PYG{l}{RSYNC\PYGZus{}ARGS=\PYGZdq{}\PYGZhy{}aHv /mnt/s3ql/ /home/restore/\PYGZdq{}}

\PYG{l}{rsync \PYGZhy{}f \PYGZdq{}+ */\PYGZdq{} \PYGZhy{}f \PYGZdq{}\PYGZhy{}! }\PYG{g+ge}{[a\PYGZhy{}f]}\PYG{l}{*\PYGZdq{} \PYGZdl{}RSYNC\PYGZus{}ARGS \PYGZam{}}
\PYG{l}{rsync \PYGZhy{}f \PYGZdq{}+ */\PYGZdq{} \PYGZhy{}f \PYGZdq{}\PYGZhy{}! }\PYG{g+ge}{[g\PYGZhy{}l]}\PYG{l}{*\PYGZdq{} \PYGZdl{}RSYNC\PYGZus{}ARGS \PYGZam{}}
\PYG{l}{rsync \PYGZhy{}f \PYGZdq{}+ */\PYGZdq{} \PYGZhy{}f \PYGZdq{}\PYGZhy{} }\PYG{g+ge}{[a\PYGZhy{}l]}\PYG{l}{*\PYGZdq{} \PYGZdl{}RSYNC\PYGZus{}ARGS \PYGZam{}}

\PYG{l}{wait}
\end{sphinxVerbatim}

The optimum number of parallel processes depends on your network
connection and the size of the files that you want to transfer.
However, starting about 10 processes seems to be a good compromise
that increases performance dramatically in almost all situations.

S3QL comes with a script named \sphinxcode{\sphinxupquote{pcp.py}} in the \sphinxcode{\sphinxupquote{contrib}} directory
that can be used to transfer files in parallel without having to write
an explicit script first. See the description of {\hyperref[\detokenize{contrib:pcp}]{\sphinxcrossref{\DUrole{std,std-ref}{pcp.py}}}} for
details.


\chapter{Frequently Asked Questions}
\label{\detokenize{faq:frequently-asked-questions}}\label{\detokenize{faq::doc}}

\section{What does “python-apsw must be linked dynamically to sqlite3” mean?}
\label{\detokenize{faq:what-does-python-apsw-must-be-linked-dynamically-to-sqlite3-mean}}
This error occurs if your python-apsw module is statically linked
against SQLite. You need to recompile and install the module with the
proper settings for dynamic linking.

If you are using Ubuntu or Debian, make sure that you are \sphinxstylestrong{not}
using the apsw PPA at
\sphinxurl{https://launchpad.net/~ubuntu-rogerbinns/+archive/apsw}, since this PPA
provides statically linked packages. Instead, use the \sphinxstyleemphasis{python-apsw}
packages shipped with Debian/Ubuntu. You can reinstall the package
(after removing the apsw PPA from \sphinxcode{\sphinxupquote{/etc/apt/sources.list}} and
\sphinxcode{\sphinxupquote{/etc/apt/sources.list.d/*}} with:

\begin{sphinxVerbatim}[commandchars=\\\{\}]
\PYG{c}{\PYGZsh{} dpkg \PYGZhy{}\PYGZhy{}purge \PYGZhy{}\PYGZhy{}force\PYGZhy{}depends python\PYGZhy{}apsw}
\PYG{c}{\PYGZsh{} apt\PYGZhy{}get install python\PYGZhy{}apsw}
\end{sphinxVerbatim}


\section{How can I improve the file system throughput?}
\label{\detokenize{faq:how-can-i-improve-the-file-system-throughput}}
There are three possible limiting factors for file system throughput:
\begin{enumerate}
\def\theenumi{\arabic{enumi}}
\def\labelenumi{\theenumi .}
\makeatletter\def\p@enumii{\p@enumi \theenumi .}\makeatother
\item {} 
Throughput from process through kernel to S3QL

\item {} 
Compression speed

\item {} 
Upload speed

\end{enumerate}

To figure out what’s the limiting factor in your case, you can use the
script \sphinxcode{\sphinxupquote{contrib/benchmark.py}} in the S3QL tarball. It will
determine the individual throughputs, and also give a recommendation
for compression algorithm and number of upload threads.


\section{Why does \sphinxstyleliteralintitle{\sphinxupquote{df}} show 1 TB of free space?}
\label{\detokenize{faq:why-does-df-show-1-tb-of-free-space}}
The file system size of S3QL is unlimited. However, most Unix
utilities (including \sphinxcode{\sphinxupquote{df}}) have no concept of an infinite size, so
S3QL has to provide some numerical value. Therefore, S3QL always
reports that the file system is 50\% free, but it also reports at least
1 TB. This means that if you store 1.5 TB you will have another 1.5 TB
free, but if you store just 2 MB, you will still have TB free rather
than just another 2 MB.


\section{Which operating systems are supported?}
\label{\detokenize{faq:which-operating-systems-are-supported}}
S3QL is developed on Linux, but should in principle work on all FUSE
supported operating systems that have the required Python modules. The
\sphinxstyleliteralstrong{\sphinxupquote{\{umount.s3ql}} program does some tricks which may not work on
non-Linux systems, but you can always umount with the \sphinxcode{\sphinxupquote{fusermount
-u}} command provided by FUSE itself (the only difference is that it
will not block until all data has been uploaded to S3 but return
immediately). Please report your success (or failure) with other
operating systems on the mailing list, so that this information can be
extended.


\section{Is there a file size limit?}
\label{\detokenize{faq:is-there-a-file-size-limit}}
No, S3QL puts no limits on the size of your files. Any limits on
maximum object size imposed by the storage service do not matter for
S3QL, since files are automatically and transparently split into
smaller blocks.


\section{Suppose I want to make a small change in a very large file. Will S3QL download and re-upload the entire file?}
\label{\detokenize{faq:suppose-i-want-to-make-a-small-change-in-a-very-large-file-will-s3ql-download-and-re-upload-the-entire-file}}
No, S3QL splits files in blocks of a configurable size (default: 10 MB). So to make a small change in a big file, only one block needs to be transferred and not the entire file.


\section{I don’t quite understand this de-duplication feature…}
\label{\detokenize{faq:i-don-t-quite-understand-this-de-duplication-feature}}
The great thing about data de-duplication is that you don’t need to know anything about it and will still get all the benefits. Towards you, S3QL will behave and look exactly like a file system without data duplication. When talking to the storage backend, however, the S3QL will try to store identical data only once and thus occupy less space than an ordinary file system would.


\section{What does the “Transport endpoint not connected” error mean?}
\label{\detokenize{faq:what-does-the-transport-endpoint-not-connected-error-mean}}
It means that the file system has crashed. Please check
\sphinxcode{\sphinxupquote{mount.log}} for a more useful error message and report a bug if
appropriate. If you can’t find any errors in \sphinxcode{\sphinxupquote{mount.log}}, the
mount process may have “segfaulted”. To confirm this, look for a
corresponding message in the \sphinxstyleliteralstrong{\sphinxupquote{dmesg}} output. If the mount process
segfaulted, please try to obtain a C backtrace (see {[}{[}Providing
Debugging Info{]}{]}) of the crash and file a bug report.

To make the mountpoint available again (i.e., unmount the crashed file
system), use the \sphinxcode{\sphinxupquote{fusermount -u}} command.

Before reporting a bug, please make sure that you’re not just using
the most recent S3QL version, but also the most-recent version of the
most important dependencies (python-llfuse, python-apsw, python-lzma).


\section{What does “Backend reports that fs is still mounted elsewhere, aborting” mean?}
\label{\detokenize{faq:what-does-backend-reports-that-fs-is-still-mounted-elsewhere-aborting-mean}}
It means that either the file systems has not been unmounted cleanly,
or the data from the most-recent mount is not yet available from the
backend. In the former case you should try to run fsck on the computer
where the file system has been mounted most recently. In the later
case, waiting may fix the problem: many storage providers have a
so-called “consistency window” for which recently uploaded data may
not yet be available for download. If you know that your backend is
fully consistent, or the consistency window has passed, you can also
run fsck in this situation. S3QL will then assume that whatever data
is missing at this point is not going to show up again in the
future. Affected files will be moved to /lost+found.


\section{Can I access an S3QL file system on multiple computers simultaneously?}
\label{\detokenize{faq:can-i-access-an-s3ql-file-system-on-multiple-computers-simultaneously}}
To share a file system between several computers, you need to have a
method for these computers to communicate, so that they can tell each
other when they are changing data. However, S3QL is designed to store
data in cloud storage services which can not be used for this kind of
communication, because different computers may end up talking to
different servers in the cloud (and there is no way for S3QL to force
the cloud servers to synchronize quickly and often enough).

Therefore, the only way to share an S3QL file system among different
computers is to have one “master” computer that runs \sphinxstyleemphasis{mount.s3ql},
and then shares the mountpoint over a network file system like NFS,
CIFS or sshfs. If the participating computers are connected over an
insecure network, NFS and CIFS should be combined with VPN software
like \sphinxhref{http://www.tinc-vpn.org/}{Tinc} (which is very easy to set up
for a few computers) or OpenVPN (which is a lot more complicated).

In principle, both VPN and NFS/CIFS-alike functionality could be
integrated into S3QL to allow simultaneous mounting using
\sphinxstyleemphasis{mount.s3ql} directly. However, consensus among the S3QL developers
is that this is not worth the increased complexity.


\section{What maximum object size should I use?}
\label{\detokenize{faq:what-maximum-object-size-should-i-use}}
The \sphinxcode{\sphinxupquote{-{-}max-obj-size}} option of the \sphinxstyleliteralstrong{\sphinxupquote{mkfs.s3ql}}
command determines the maximum size of the data chunks that S3QL
exchanges with the backend server.

For files smaller than the maximum object size, this option has no
effect. Files larger than the maximum object size are split into
multiple chunks. Whenever you upload or download data from the
backend, this is done in complete chunks. So if you have configured a
maximum object size of 10 MB, and want to read (or write) 5 bytes in a
100 MB file, you will still download (or upload) 10 MB of data. If you
decreased the maximum object size to 5 MB, you’d download/upload 5 MB.

On the other hand, if you want to read the whole 100 MB, and have
configured an object size of 10 MB, S3QL will have to send 10 separate
requests to the backend. With a maximum object size of 1 MB, there’d
be 100 separate requests. The larger the number of requests, the more
inefficient this becomes, because there is a fixed time associated
with the processing of each request. Also, many storage providers
charge a fixed amount for each request, so downloading 100 MB in one
request of 100 MB is cheaper than downloading it with 100 requests of
1 MB.

When choosing a maximum object size, you have to find a balance
between these two effects. The bigger the size, the less requests will
be used, but the more data is transfered uselessly. The smaller the
size, the more requests will be used, but less traffic will be wasted.

Generally you should go with the default unless you have a good reason
to change it. When adjusting the value, keep in mind that it only
affects files larger than the maximum object size. So if most of your
files are less than 1 MB, decreasing the maximum object size from the
default 10 MB to 1 MB will have almost no effect.


\section{Is there a way to make the cache persistent / access the file system offline?}
\label{\detokenize{faq:is-there-a-way-to-make-the-cache-persistent-access-the-file-system-offline}}
No, there is no way to do this. However, for most use-cases this
feature is actually not required. If you want to ensure that all data
in a directory is stored in the cloud \sphinxstyleemphasis{and} available offline, all you
need to do is store your data in a local directory that you
periodically synchronize to an S3QL mountpoint. For example, if you
would like to have an S3QL mountpoint with persistent cache at
\sphinxcode{\sphinxupquote{/mnt/data}}, you get can this as follows:
\begin{enumerate}
\def\theenumi{\arabic{enumi}}
\def\labelenumi{\theenumi .}
\makeatletter\def\p@enumii{\p@enumi \theenumi .}\makeatother
\item {} 
Mount a local block device at \sphinxcode{\sphinxupquote{/mnt/data}}, e.g. using ext4 or btrfs

\item {} 
Mount an S3QL file system at \sphinxcode{\sphinxupquote{/mnt/data\_online}} using a small
cache size and number of threads (eg. \sphinxcode{\sphinxupquote{-{-}threads
2 -{-}max-cache-entries 10}}).

\item {} 
Keep \sphinxcode{\sphinxupquote{/mnt/data\_online}} synchronized with \sphinxcode{\sphinxupquote{/mnt/data}}
by either
\begin{itemize}
\item {} 
Periodically running rsync, e.g. by calling
\sphinxcode{\sphinxupquote{rsync -aHA -{-}delete-during -{-}partial}} from cron, or

\item {} 
Continously synchronizing using e.g. \sphinxhref{https://github.com/drunomics/syncd/blob/master/watch.sh}{watch.sh} or
\sphinxhref{https://code.google.com/p/lsyncd/}{lsyncd} (these programs use
\sphinxstyleemphasis{inotify} to constantly monitor the source directory and
immediately copy over any changes)

\end{itemize}

\item {} 
Now use \sphinxcode{\sphinxupquote{/mnt/data}} in the same way as you would use an S3QL
file system with a persistent cache.

\end{enumerate}


\section{I would like to use S3QL with Hubic, but…}
\label{\detokenize{faq:i-would-like-to-use-s3ql-with-hubic-but}}
\sphinxhref{http://www.hubic.com/}{HubiC} has a terribly designed API,
effectively no customer service, and the servers are unreliable and
produce sporadic weird errors. Don’t expect any help if you encounter
problems with S3QL and hubiC.


\section{What’s a reasonable metadata upload interval?}
\label{\detokenize{faq:what-s-a-reasonable-metadata-upload-interval}}
The metadata upload interval can be specified using the
\sphinxcode{\sphinxupquote{-{-}metadata-upload-interval}} option of \sphinxstyleliteralstrong{\sphinxupquote{mount.s3ql}}
and a reasonable value to use is (surprise!) the default of 24 hours.

To understand why intervals smaller than a few hours do not make
sense, consider what happens during a metadata upload:
\begin{enumerate}
\def\theenumi{\arabic{enumi}}
\def\labelenumi{\theenumi .}
\makeatletter\def\p@enumii{\p@enumi \theenumi .}\makeatother
\item {} 
The file system is frozen (all requests will block)

\item {} 
The entire SQLite database holding the metadata is dumped into a
more space efficient format.

\item {} 
The file system is unfrozen

\item {} 
The dumped database is uploaded

\item {} 
The previous metadata is backed up, and backups are rotated

\end{enumerate}

In other words, uploading metadata is a very expensive operation that
typically takes several seconds to several minutes.

On the other hand, consider the situation that the periodic upload is
intended to prevent:
\begin{enumerate}
\def\theenumi{\arabic{enumi}}
\def\labelenumi{\theenumi .}
\makeatletter\def\p@enumii{\p@enumi \theenumi .}\makeatother
\item {} 
Your computer crashes, and the SQLite database in your S3QL cache
directory is \sphinxstylestrong{irreparably} corrupted or lost.

\end{enumerate}

This is a \sphinxstyleemphasis{very} unlikely situation. When \sphinxstyleliteralstrong{\sphinxupquote{mount.s3ql}} crashes
(but your computer keeps running), the local metadata will not get
corrupted. If your entire computer crashes (e.g. because of a power
outage), \sphinxstyleliteralstrong{\sphinxupquote{fsck.s3ql}} is almost always still able to recover the
local metadata. You have to be exceedingly unlucky to depend on the
periodically uploaded metadata.

In other words, if you set the metadata upload interval to \sphinxstyleemphasis{x}
hours, what you are saying is that you consider your system so
unstable that there is a good chance that your local hard disk will be
irreparably corrupted sometime in the next \sphinxstyleemphasis{x} hours. In such a
situation, you should thus be \sphinxstylestrong{backing up your entire system every
*x* hours} (not just the S3QL metadata). If the metadata upload
interval coincides with the interval at which you’re doing full system
backups (not necessarily using S3QL), you are using the right
value. If the metadata upload interval is smaller (or if you’re not
doing full system backups at all), you are probably uploading metadata
too often.


\chapter{Known Issues}
\label{\detokenize{issues:known-issues}}\label{\detokenize{issues::doc}}\begin{itemize}
\item {} 
S3QL de-duplicates data blocks based solely only on SHA256
checksums, without doing a byte-by-byte comparison of the blocks.
Since it is possible for two data blocks to have the same checksum
despite having different contents, this can lead to problems. If two
such blocks are stored in an S3QL file system, the data in one block
will be lost and replaced by the data in the other block. However,
the chances of this occuring for any two blocks are about 1 in 10\textasciicircum{}77
(2\textasciicircum{}256). For a file system that holds a total of 10\textasciicircum{}34 blocks, the
chances of a collision increase to about 1 in 10\textasciicircum{}9. Storing more
than 10\textasciicircum{}34 blocks (or about 10\textasciicircum{}25 TB with an (extremely small) block
size of 4 kB) is therefore not recommended. Being exceptionally
unlucky may also be a disadvantage.

\item {} 
S3QL does not support Access Control Lists (ACLs). There is nothing
fundamentally that prevents this, someone just has to write the
necessary code.

\item {} 
As of Linux kernel 3.5 S3QL file systems do not implement the “write
protect” bit on directories. In other words, even if a directory has
the write protect bit set, the owner of the directory can delete any
files and (empty) subdirectories inside it. This is a bug in the
FUSE kernel module
(cf. \sphinxurl{https://github.com/libfuse/libfuse/issues/23}) and needs to be
fixed in the kernel.  Unfortunately it does not look as if this is
going to be fixed anytime soon (as of 2016/2/28).

\item {} 
S3QL is rather slow when an application tries to write data in
unreasonably small chunks. If a 1 MiB file is copied in chunks of 1
KB, this will take more than 10 times as long as when it’s copied
with the (recommended) chunk size of 128 KiB.

This is a limitation of the FUSE library (which does not yet support
write caching) which will hopefully be addressed in some future FUSE
version.

Most applications, including e.g. GNU \sphinxcode{\sphinxupquote{cp}} and \sphinxcode{\sphinxupquote{rsync}}, use
reasonably large buffers and are therefore not affected by this
problem and perform very efficient on S3QL file systems.

However, if you encounter unexpectedly slow performance with a
specific program, this might be due to the program using very small
write buffers. Although this is not really a bug in the program,
it might be worth to ask the program’s authors for help.

\item {} 
S3QL always updates file and directory access times as if the \sphinxcode{\sphinxupquote{relatime}}
mount option has been specified: the access time (“atime”) is only updated
if it is currently earlier than either the status change time
(“ctime”) or modification time (“mtime”).

\item {} 
S3QL directories always have an \sphinxcode{\sphinxupquote{st\_nlink}} value of 1. This may confuse
programs that rely on directories having \sphinxcode{\sphinxupquote{st\_nlink}} values of \sphinxstyleemphasis{(2 +
number of sub directories)}.

Note that this is not a bug in S3QL. Including sub directories in
the \sphinxcode{\sphinxupquote{st\_nlink}} value is a Unix convention, but by no means a
requirement. If an application blindly relies on this convention
being followed, then this is a bug in the application.

A prominent example are early versions of GNU find, which required
the \sphinxcode{\sphinxupquote{-{-}noleaf}} option to work correctly on S3QL file systems. This
bug has already been fixed in recent find versions.

\item {} 
The \sphinxcode{\sphinxupquote{umount}} and \sphinxcode{\sphinxupquote{fusermount -u}} commands will \sphinxstyleemphasis{not} block until all
data has been uploaded to the backend. (this is a FUSE limitation,
cf. \sphinxurl{https://github.com/libfuse/libfuse/issues/1}). If you use either
command to unmount an S3QL file system, you have to take care to
explicitly wait for the \sphinxcode{\sphinxupquote{mount.s3ql}} process to terminate before you
shut down or restart the system. Therefore it is generally not a
good idea to mount an S3QL file system in \sphinxcode{\sphinxupquote{/etc/fstab}} (you should
use a dedicated init script instead).

\item {} 
S3QL relies on the backends not to run out of space. This is a given
for big storage providers like Amazon S3 or Google Storage, but you
may stumble upon this if you use your own server or smaller providers.

If there is no space left in the backend, attempts to write more
data into the S3QL file system will fail and the file system will be
in an inconsistent state and require a file system check (and you
should make sure to make space available in the backend before
running the check).

Unfortunately, there is no way to handle insufficient space in the
backend without leaving the file system inconsistent. Since
S3QL first writes data into the cache, it can no longer return an
error when it later turns out that the cache can not be committed to
the backend.

\item {} 
When using python-dugong versions 3.3 or earlier, S3QL supports only
CONNECT-style proxying, which may cause issues with some proxy
servers when using plain HTTP. Upgrading to python-dugong 3.4 or
newer removes this limitation.

\end{itemize}


\chapter{Manpages}
\label{\detokenize{man/index:manpages}}\label{\detokenize{man/index::doc}}
The man pages are installed with S3QL on your system and can be viewed
with the \sphinxstyleliteralstrong{\sphinxupquote{man}} command. For reference, they are also included
here in the User’s Guide.


\section{The \sphinxstyleliteralstrong{\sphinxupquote{mkfs.s3ql}} command}
\label{\detokenize{man/mkfs:the-command-command}}\label{\detokenize{man/mkfs::doc}}

\subsection{Synopsis}
\label{\detokenize{man/mkfs:synopsis}}
\begin{sphinxVerbatim}[commandchars=\\\{\}]
\PYG{l}{mkfs.s3ql }\PYG{g+ge}{[options]}\PYG{l}{ }\PYG{n+nv}{\PYGZlt{}storage url\PYGZgt{}}
\end{sphinxVerbatim}


\subsection{Description}
\label{\detokenize{man/mkfs:description}}
The \sphinxstyleliteralstrong{\sphinxupquote{mkfs.s3ql}} command creates a new file system in the location
specified by \sphinxstyleemphasis{storage url}. The storage url depends on the backend
that is used. The S3QL User’s Guide should be consulted for a
description of the available backends.

Unless you have specified the \sphinxcode{\sphinxupquote{-{-}plain}} option, \sphinxcode{\sphinxupquote{mkfs.s3ql}} will ask
you to enter an encryption password. This password will \sphinxstyleemphasis{not} be read
from an authentication file specified with the \sphinxcode{\sphinxupquote{-{-}authfile}}
option to prevent accidental creation of an encrypted file system.


\subsection{Options}
\label{\detokenize{man/mkfs:options}}
The \sphinxstyleliteralstrong{\sphinxupquote{mkfs.s3ql}} command accepts the following options.
\begin{quote}
\begin{optionlist}{3cm}
\item [-{-}cachedir \textless{}path\textgreater{}]  
Store cached data in this directory (default:
\sphinxcode{\sphinxupquote{\textasciitilde{}/.s3ql)}}
\item [-{-}log \textless{}target\textgreater{}]  
Destination for log messages. Specify \sphinxcode{\sphinxupquote{none}} for
standard output or \sphinxcode{\sphinxupquote{syslog}} for the system logging
daemon. Anything else will be interpreted as a file
name. Log files will be rotated when they reach 1 MiB,
and at most 5 old log files will be kept. Default:
\sphinxcode{\sphinxupquote{None}}
\item [-{-}debug-modules \textless{}modules\textgreater{}]  
Activate debugging output from specified modules (use
commas to separate multiple modules). Debug messages
will be written to the target specified by the
\sphinxcode{\sphinxupquote{-{-}log}} option.
\item [-{-}debug]  
Activate debugging output from all S3QL modules. Debug
messages will be written to the target specified by
the \sphinxcode{\sphinxupquote{-{-}log}} option.
\item [-{-}quiet]  
be really quiet
\item [-{-}backend-options \textless{}options\textgreater{}]  
Backend specific options (separate by commas). See
backend documentation for available options.
\item [-{-}version]  
just print program version and exit
\item [-{-}authfile \textless{}path\textgreater{}]  
Read authentication credentials from this file
(default: \sphinxcode{\sphinxupquote{\textasciitilde{}/.s3ql/authinfo2)}}
\item [-L \textless{}name\textgreater{}]  
Filesystem label
\item [-{-}max-obj-size \textless{}size\textgreater{}]  
Maximum size of storage objects in KiB. Files bigger
than this will be spread over multiple objects in the
storage backend. Default: 10240 KiB.
\item [-{-}plain]  
Create unencrypted file system.
\end{optionlist}
\end{quote}


\subsection{Exit Codes}
\label{\detokenize{man/mkfs:exit-codes}}
\sphinxstyleliteralstrong{\sphinxupquote{mkfs.s3ql}} may terminate with the following exit codes:
\begin{quote}\begin{description}
\item[{0}] \leavevmode
Everything went well.

\item[{1}] \leavevmode
An unexpected error occured. This may indicate a bug in the
program.

\item[{2}] \leavevmode
Invalid command line argument or configuration file key.

\item[{3}] \leavevmode
Invalid backend option.

\item[{11}] \leavevmode
No such backend.

\item[{12}] \leavevmode
Authentication file has insecure permissions.

\item[{13}] \leavevmode
Unable to parse proxy settings.

\item[{14}] \leavevmode
Invalid credentials (Authentication failed).

\item[{15}] \leavevmode
No permission to access backend (Authorization denied).

\item[{16}] \leavevmode
Invalid storage URL, specified location does not exist in backend.

\item[{19}] \leavevmode
Unable to connect to backend, can’t resolve hostname.

\item[{45}] \leavevmode
Unable to access cache directory.

\end{description}\end{quote}


\subsection{See Also}
\label{\detokenize{man/mkfs:see-also}}
The S3QL homepage is at \sphinxurl{https://github.com/s3ql/s3ql/}.

The full S3QL documentation should also be installed somewhere on your
system, common locations are \sphinxcode{\sphinxupquote{/usr/share/doc/s3ql}} or
\sphinxcode{\sphinxupquote{/usr/local/doc/s3ql}}.


\section{The \sphinxstyleliteralstrong{\sphinxupquote{s3qladm}} command}
\label{\detokenize{man/adm:the-command-command}}\label{\detokenize{man/adm::doc}}

\subsection{Synopsis}
\label{\detokenize{man/adm:synopsis}}
\begin{sphinxVerbatim}[commandchars=\\\{\}]
\PYG{l}{s3qladm }\PYG{g+ge}{[options]}\PYG{l}{ }\PYG{n+nv}{\PYGZlt{}action\PYGZgt{}}\PYG{l}{ }\PYG{n+nv}{\PYGZlt{}storage url\PYGZgt{}}
\end{sphinxVerbatim}

where \sphinxcode{\sphinxupquote{action}} may be either of \sphinxstyleliteralstrong{\sphinxupquote{passphrase}},
\sphinxstyleliteralstrong{\sphinxupquote{upgrade}}, \sphinxstyleliteralstrong{\sphinxupquote{delete}} or \sphinxstyleliteralstrong{\sphinxupquote{download-metadata}}.


\subsection{Description}
\label{\detokenize{man/adm:description}}
The \sphinxstyleliteralstrong{\sphinxupquote{s3qladm}} command performs various operations on \sphinxstyleemphasis{unmounted} S3QL
file systems. The file system \sphinxstyleemphasis{must not be mounted} when using
\sphinxstyleliteralstrong{\sphinxupquote{s3qladm}} or things will go wrong badly.

The storage url depends on the backend that is used. The S3QL User’s
Guide should be consulted for a description of the available backends.


\subsection{Options}
\label{\detokenize{man/adm:options}}
The \sphinxstyleliteralstrong{\sphinxupquote{s3qladm}} command accepts the following options.
\begin{quote}
\begin{optionlist}{3cm}
\item [-{-}authfile \textless{}path\textgreater{}]  
Read authentication credentials from this file
(default: \sphinxcode{\sphinxupquote{\textasciitilde{}/.s3ql/authinfo2)}}
\item [-{-}debug-modules \textless{}modules\textgreater{}]  
Activate debugging output from specified modules (use
commas to separate multiple modules). Debug messages
will be written to the target specified by the
\sphinxcode{\sphinxupquote{-{-}log}} option.
\item [-{-}debug]  
Activate debugging output from all S3QL modules. Debug
messages will be written to the target specified by
the \sphinxcode{\sphinxupquote{-{-}log}} option.
\item [-{-}quiet]  
be really quiet
\item [-{-}log \textless{}target\textgreater{}]  
Destination for log messages. Specify \sphinxcode{\sphinxupquote{none}} for
standard output or \sphinxcode{\sphinxupquote{syslog}} for the system logging
daemon. Anything else will be interpreted as a file
name. Log files will be rotated when they reach 1 MiB,
and at most 5 old log files will be kept. Default:
\sphinxcode{\sphinxupquote{None}}
\item [-{-}backend-options \textless{}options\textgreater{}]  
Backend specific options (separate by commas). See
backend documentation for available options.
\item [-{-}cachedir \textless{}path\textgreater{}]  
Store cached data in this directory (default:
\sphinxcode{\sphinxupquote{\textasciitilde{}/.s3ql)}}
\item [-{-}version]  
just print program version and exit
\end{optionlist}
\end{quote}

Hint: run \sphinxcode{\sphinxupquote{s3qladm \textless{}action\textgreater{} -{-}help}} to get help on the additional arguments
that the different actions take.


\subsection{Actions}
\label{\detokenize{man/adm:actions}}
The following actions may be specified:
\begin{description}
\item[{passphrase}] \leavevmode
Changes the encryption passphrase of the file system.

\item[{upgrade}] \leavevmode
Upgrade the file system to the newest revision.

\item[{clear}] \leavevmode
Delete the file system with all the stored data.

\item[{download-metadata}] \leavevmode
Interactively download backups of the file system metadata.

\end{description}


\subsection{Exit Codes}
\label{\detokenize{man/adm:exit-codes}}
\sphinxstyleliteralstrong{\sphinxupquote{s3qladm}} may terminate with the following exit codes:
\begin{quote}\begin{description}
\item[{0}] \leavevmode
Everything went well.

\item[{1}] \leavevmode
An unexpected error occured. This may indicate a bug in the
program.

\item[{2}] \leavevmode
Invalid command line argument or configuration file key.

\item[{3}] \leavevmode
Invalid backend option.

\item[{10}] \leavevmode
Could not open log file for writing.

\item[{11}] \leavevmode
No such backend.

\item[{12}] \leavevmode
Authentication file has insecure permissions.

\item[{13}] \leavevmode
Unable to parse proxy settings.

\item[{14}] \leavevmode
Invalid credentials (Authentication failed).

\item[{15}] \leavevmode
No permission to access backend (Authorization denied).

\item[{16}] \leavevmode
Invalid storage URL, specified location does not exist in backend.

\item[{17}] \leavevmode
Wrong file system passphrase.

\item[{18}] \leavevmode
No S3QL file system found at given storage URL.

\item[{19}] \leavevmode
Unable to connect to backend, can’t resolve hostname.

\item[{45}] \leavevmode
Unable to access cache directory.

\end{description}\end{quote}


\subsection{See Also}
\label{\detokenize{man/adm:see-also}}
The S3QL homepage is at \sphinxurl{https://github.com/s3ql/s3ql/}.

The full S3QL documentation should also be installed somewhere on your
system, common locations are \sphinxcode{\sphinxupquote{/usr/share/doc/s3ql}} or
\sphinxcode{\sphinxupquote{/usr/local/doc/s3ql}}.


\section{The \sphinxstyleliteralstrong{\sphinxupquote{mount.s3ql}} command}
\label{\detokenize{man/mount:the-command-command}}\label{\detokenize{man/mount::doc}}

\subsection{Synopsis}
\label{\detokenize{man/mount:synopsis}}
\begin{sphinxVerbatim}[commandchars=\\\{\}]
\PYG{l}{mount.s3ql }\PYG{g+ge}{[options]}\PYG{l}{ }\PYG{n+nv}{\PYGZlt{}storage url\PYGZgt{}}\PYG{l}{ }\PYG{n+nv}{\PYGZlt{}mount point\PYGZgt{}}
\end{sphinxVerbatim}


\subsection{Description}
\label{\detokenize{man/mount:description}}
The \sphinxstyleliteralstrong{\sphinxupquote{mount.s3ql}} command mounts the S3QL file system stored in \sphinxstyleemphasis{storage
url} in the directory \sphinxstyleemphasis{mount point}. The storage url depends on the
backend that is used. The S3QL User’s Guide should be consulted for a
description of the available backends.


\subsection{Options}
\label{\detokenize{man/mount:options}}
The \sphinxstyleliteralstrong{\sphinxupquote{mount.s3ql}} command accepts the following options.
\begin{quote}
\begin{optionlist}{3cm}
\item [-{-}log \textless{}target\textgreater{}]  
Destination for log messages. Specify \sphinxcode{\sphinxupquote{none}} for
standard output or \sphinxcode{\sphinxupquote{syslog}} for the system logging
daemon. Anything else will be interpreted as a file
name. Log files will be rotated when they reach 1 MiB,
and at most 5 old log files will be kept. Default:
\sphinxcode{\sphinxupquote{\textasciitilde{}/.s3ql/mount.log}}
\item [-{-}cachedir \textless{}path\textgreater{}]  
Store cached data in this directory (default:
\sphinxcode{\sphinxupquote{\textasciitilde{}/.s3ql)}}
\item [-{-}debug-modules \textless{}modules\textgreater{}]  
Activate debugging output from specified modules (use
commas to separate multiple modules). Debug messages
will be written to the target specified by the
\sphinxcode{\sphinxupquote{-{-}log}} option.
\item [-{-}debug]  
Activate debugging output from all S3QL modules. Debug
messages will be written to the target specified by
the \sphinxcode{\sphinxupquote{-{-}log}} option.
\item [-{-}quiet]  
be really quiet
\item [-{-}backend-options \textless{}options\textgreater{}]  
Backend specific options (separate by commas). See
backend documentation for available options.
\item [-{-}version]  
just print program version and exit
\item [-{-}authfile \textless{}path\textgreater{}]  
Read authentication credentials from this file
(default: \sphinxcode{\sphinxupquote{\textasciitilde{}/.s3ql/authinfo2)}}
\item [-{-}compress \textless{}algorithm-lvl\textgreater{}]  
Compression algorithm and compression level to use
when storing new data. \sphinxstyleemphasis{algorithm} may be any of
\sphinxcode{\sphinxupquote{lzma}}, \sphinxcode{\sphinxupquote{bzip2}}, \sphinxcode{\sphinxupquote{zlib}}, or none. \sphinxstyleemphasis{lvl} may be any
integer from 0 (fastest) to 9 (slowest). Default:
\sphinxcode{\sphinxupquote{lzma-6}}
\item [-{-}cachesize \textless{}size\textgreater{}]  
Cache size in KiB (default: autodetect).
\item [-{-}max-cache-entries \textless{}num\textgreater{}]  
Maximum number of entries in cache (default:
autodetect). Each cache entry requires one file
descriptor, so if you increase this number you have to
make sure that your process file descriptor limit (as
set with \sphinxcode{\sphinxupquote{ulimit -n}}) is high enough (at least the
number of cache entries + 100).
\item [-{-}keep-cache]  
Do not purge locally cached files on exit.
\item [-{-}allow-other]  
Normally, only the user who called \sphinxcode{\sphinxupquote{mount.s3ql}} can
access the mount point. This user then also has full
access to it, independent of individual file
permissions. If the \sphinxcode{\sphinxupquote{-{-}allow-other}} option is
specified, other users can access the mount point as
well and individual file permissions are taken into
account for all users.
\item [-{-}allow-root]  
Like \sphinxcode{\sphinxupquote{-{-}allow-other}}, but restrict access to the
mounting user and the root user.
\item [-{-}fg]  
Do not daemonize, stay in foreground
\item [-{-}fs-name FS\_NAME]  
Mount name passed to fuse, the name will be shown in
the first column of the system mount command output.
If not specified your storage url is used.
\item [-{-}systemd]  
Run as systemd unit. Consider specifying \textendash{}log none as
well to make use of journald.
\item [-{-}metadata-upload-interval \textless{}seconds\textgreater{}]  
Interval in seconds between complete metadata uploads.
Set to 0 to disable. Default: 24h.
\item [-{-}threads \textless{}no\textgreater{}]  
Number of parallel upload threads to use (default:
auto).
\item [-{-}nfs]  
Enable some optimizations for exporting the file
system over NFS. (default: False)
\end{optionlist}
\end{quote}


\subsection{Exit Codes}
\label{\detokenize{man/mount:exit-codes}}
\sphinxstyleliteralstrong{\sphinxupquote{mount.s3ql}} may terminate with the following exit codes:
\begin{quote}\begin{description}
\item[{0}] \leavevmode
Everything went well.

\item[{1}] \leavevmode
An unexpected error occured. This may indicate a bug in the
program.

\item[{2}] \leavevmode
Invalid command line argument or configuration file key.

\item[{3}] \leavevmode
Invalid backend option.

\item[{10}] \leavevmode
Could not open log file for writing.

\item[{11}] \leavevmode
No such backend.

\item[{12}] \leavevmode
Authentication file has insecure permissions.

\item[{13}] \leavevmode
Unable to parse proxy settings.

\item[{14}] \leavevmode
Invalid credentials (Authentication failed).

\item[{15}] \leavevmode
No permission to access backend (Authorization denied).

\item[{16}] \leavevmode
Invalid storage URL, specified location does not exist in backend.

\item[{17}] \leavevmode
Wrong file system passphrase.

\item[{18}] \leavevmode
No S3QL file system found at given storage URL.

\item[{19}] \leavevmode
Unable to connect to backend, can’t resolve hostname.

\item[{30}] \leavevmode
File system was not unmounted cleanly.

\item[{31}] \leavevmode
File system appears to be mounted elsewhere.

\item[{32}] \leavevmode
Unsupported file system revision (too old).

\item[{33}] \leavevmode
Unsupported file system revision (too new).

\item[{34}] \leavevmode
Insufficient free nodes, need to run \sphinxstyleliteralstrong{\sphinxupquote{fsck.s3ql}}.

\item[{35}] \leavevmode
Attempted to mount read-only, this is not supported.

\item[{36}] \leavevmode
Mountpoint does not exist.

\item[{37}] \leavevmode
Not enough available file descriptors.

\item[{39}] \leavevmode
Unable to bind file system to mountpoint.

\item[{45}] \leavevmode
Unable to access cache directory.

\end{description}\end{quote}


\subsection{See Also}
\label{\detokenize{man/mount:see-also}}
The S3QL homepage is at \sphinxurl{https://github.com/s3ql/s3ql/}.

The full S3QL documentation should also be installed somewhere on your
system, common locations are \sphinxcode{\sphinxupquote{/usr/share/doc/s3ql}} or
\sphinxcode{\sphinxupquote{/usr/local/doc/s3ql}}.


\section{The \sphinxstyleliteralstrong{\sphinxupquote{s3qlstat}} command}
\label{\detokenize{man/stat:the-command-command}}\label{\detokenize{man/stat::doc}}

\subsection{Synopsis}
\label{\detokenize{man/stat:synopsis}}
\begin{sphinxVerbatim}[commandchars=\\\{\}]
\PYG{l}{s3qlstat }\PYG{g+ge}{[options]}\PYG{l}{ }\PYG{n+nv}{\PYGZlt{}mountpoint\PYGZgt{}}
\end{sphinxVerbatim}


\subsection{Description}
\label{\detokenize{man/stat:description}}
The \sphinxstyleliteralstrong{\sphinxupquote{s3qlstat}} command prints statistics about the S3QL file system mounted
at \sphinxcode{\sphinxupquote{mountpoint}}.

\sphinxstyleliteralstrong{\sphinxupquote{s3qlstat}} can only be called by the user that mounted the file system
and (if the file system was mounted with \sphinxcode{\sphinxupquote{-{-}allow-other}} or
\sphinxcode{\sphinxupquote{-{-}allow-root}}) the root user.


\subsection{Options}
\label{\detokenize{man/stat:options}}
The \sphinxstyleliteralstrong{\sphinxupquote{s3qlstat}} command accepts the following options:
\begin{quote}
\begin{optionlist}{3cm}
\item [-{-}log \textless{}target\textgreater{}]  
Destination for log messages. Specify \sphinxcode{\sphinxupquote{none}} for
standard output or \sphinxcode{\sphinxupquote{syslog}} for the system logging
daemon. Anything else will be interpreted as a file
name. Log files will be rotated when they reach 1 MiB,
and at most 5 old log files will be kept. Default:
\sphinxcode{\sphinxupquote{None}}
\item [-{-}debug-modules \textless{}modules\textgreater{}]  
Activate debugging output from specified modules (use
commas to separate multiple modules). Debug messages
will be written to the target specified by the
\sphinxcode{\sphinxupquote{-{-}log}} option.
\item [-{-}debug]  
Activate debugging output from all S3QL modules. Debug
messages will be written to the target specified by
the \sphinxcode{\sphinxupquote{-{-}log}} option.
\item [-{-}quiet]  
be really quiet
\item [-{-}version]  
just print program version and exit
\item [-{-}raw]  
Do not pretty-print numbers
\end{optionlist}
\end{quote}


\subsection{Exit Codes}
\label{\detokenize{man/stat:exit-codes}}
\sphinxstyleliteralstrong{\sphinxupquote{s3qlstat}} may terminate with the following exit codes:
\begin{quote}\begin{description}
\item[{0}] \leavevmode
Everything went well.

\item[{1}] \leavevmode
An unexpected error occured. This may indicate a bug in the
program.

\item[{2}] \leavevmode
Invalid command line argument or configuration file key.

\end{description}\end{quote}


\subsection{See Also}
\label{\detokenize{man/stat:see-also}}
The S3QL homepage is at \sphinxurl{https://github.com/s3ql/s3ql/}.

The full S3QL documentation should also be installed somewhere on your
system, common locations are \sphinxcode{\sphinxupquote{/usr/share/doc/s3ql}} or
\sphinxcode{\sphinxupquote{/usr/local/doc/s3ql}}.


\section{The \sphinxstyleliteralstrong{\sphinxupquote{s3qlctrl}} command}
\label{\detokenize{man/ctrl:the-command-command}}\label{\detokenize{man/ctrl::doc}}

\subsection{Synopsis}
\label{\detokenize{man/ctrl:synopsis}}
\begin{sphinxVerbatim}[commandchars=\\\{\}]
\PYG{l}{s3qlctrl }\PYG{g+ge}{[options]}\PYG{l}{ }\PYG{n+nv}{\PYGZlt{}action\PYGZgt{}}\PYG{l}{ }\PYG{n+nv}{\PYGZlt{}mountpoint\PYGZgt{}}\PYG{l}{ ...}
\end{sphinxVerbatim}

where \sphinxcode{\sphinxupquote{action}} may be either of \sphinxstyleliteralstrong{\sphinxupquote{flushcache}},
\sphinxstyleliteralstrong{\sphinxupquote{upload-meta}}, \sphinxstyleliteralstrong{\sphinxupquote{cachesize}} or
\sphinxstyleliteralstrong{\sphinxupquote{log-metadata}}.


\subsection{Description}
\label{\detokenize{man/ctrl:description}}
The \sphinxstyleliteralstrong{\sphinxupquote{s3qlctrl}} command performs various actions on the S3QL file system mounted
in \sphinxcode{\sphinxupquote{mountpoint}}.

\sphinxstyleliteralstrong{\sphinxupquote{s3qlctrl}} can only be called by the user that mounted the file system
and (if the file system was mounted with \sphinxcode{\sphinxupquote{-{-}allow-other}} or
\sphinxcode{\sphinxupquote{-{-}allow-root}}) the root user.

The following actions may be specified:
\begin{description}
\item[{flushcache}] \leavevmode
Flush file system cache, i.e. upload all changed file data
to the backend.  The command blocks until the cache has been
flushed.

\item[{dropcache}] \leavevmode
Flush, and then drop file system cache. The command
blocks until the cache has been flushed and dropped.

\item[{upload-meta}] \leavevmode
Upload metadata to the backend. All file system operations will
block while a snapshot of the metadata is prepared for upload.

\item[{cachesize}] \leavevmode
Changes the cache size of the file system. This action requires an
additional argument that specifies the new cache size in KiB, so the
complete command line is:

\begin{sphinxVerbatim}[commandchars=\\\{\}]
\PYG{l}{s3qlctrl }\PYG{g+ge}{[options]}\PYG{l}{ cachesize }\PYG{n+nv}{\PYGZlt{}mountpoint\PYGZgt{}}\PYG{l}{ }\PYG{n+nv}{\PYGZlt{}new\PYGZhy{}cache\PYGZhy{}size\PYGZgt{}}
\end{sphinxVerbatim}

\item[{log}] \leavevmode
Change the amount of information that is logged into
\sphinxcode{\sphinxupquote{\textasciitilde{}/.s3ql/mount.log}} file. The complete syntax is:

\begin{sphinxVerbatim}[commandchars=\\\{\}]
s3qlctrl [options] log \PYGZlt{}mountpoint\PYGZgt{} \PYGZlt{}level\PYGZgt{} [\PYGZlt{}module\PYGZgt{} [\PYGZlt{}module\PYGZgt{} ...]]
\end{sphinxVerbatim}

here \sphinxcode{\sphinxupquote{level}} is the desired new log level and may be either of
\sphinxstyleemphasis{debug}, \sphinxstyleemphasis{info} or \sphinxstyleemphasis{warn}. One or more \sphinxcode{\sphinxupquote{module}} may only be
specified with the \sphinxstyleemphasis{debug} level and allow to restrict the debug
output to just the listed modules.

\end{description}


\subsection{Options}
\label{\detokenize{man/ctrl:options}}
The \sphinxstyleliteralstrong{\sphinxupquote{s3qlctrl}} command also accepts the following options, no matter
what specific action is being invoked:
\begin{quote}
\begin{optionlist}{3cm}
\item [-{-}log \textless{}target\textgreater{}]  
Destination for log messages. Specify \sphinxcode{\sphinxupquote{none}} for
standard output or \sphinxcode{\sphinxupquote{syslog}} for the system logging
daemon. Anything else will be interpreted as a file
name. Log files will be rotated when they reach 1 MiB,
and at most 5 old log files will be kept. Default:
\sphinxcode{\sphinxupquote{None}}
\item [-{-}debug-modules \textless{}modules\textgreater{}]  
Activate debugging output from specified modules (use
commas to separate multiple modules). Debug messages
will be written to the target specified by the
\sphinxcode{\sphinxupquote{-{-}log}} option.
\item [-{-}debug]  
Activate debugging output from all S3QL modules. Debug
messages will be written to the target specified by
the \sphinxcode{\sphinxupquote{-{-}log}} option.
\item [-{-}quiet]  
be really quiet
\item [-{-}version]  
just print program version and exit
\end{optionlist}
\end{quote}

Hint: run \sphinxcode{\sphinxupquote{s3qlctrl \textless{}action\textgreater{} -{-}help}} to get help on the additional arguments
that the different actions take.


\subsection{Exit Codes}
\label{\detokenize{man/ctrl:exit-codes}}
\sphinxstyleliteralstrong{\sphinxupquote{s3qlctrl}} may terminate with the following exit codes:
\begin{quote}\begin{description}
\item[{0}] \leavevmode
Everything went well.

\item[{1}] \leavevmode
An unexpected error occured. This may indicate a bug in the
program.

\item[{2}] \leavevmode
Invalid command line argument or configuration file key.

\end{description}\end{quote}


\subsection{See Also}
\label{\detokenize{man/ctrl:see-also}}
The S3QL homepage is at \sphinxurl{https://github.com/s3ql/s3ql/}.

The full S3QL documentation should also be installed somewhere on your
system, common locations are \sphinxcode{\sphinxupquote{/usr/share/doc/s3ql}} or
\sphinxcode{\sphinxupquote{/usr/local/doc/s3ql}}.


\section{The \sphinxstyleliteralstrong{\sphinxupquote{s3qlcp}} command}
\label{\detokenize{man/cp:the-command-command}}\label{\detokenize{man/cp::doc}}

\subsection{Synopsis}
\label{\detokenize{man/cp:synopsis}}
\begin{sphinxVerbatim}[commandchars=\\\{\}]
\PYG{l}{s3qlcp }\PYG{g+ge}{[options]}\PYG{l}{ }\PYG{n+nv}{\PYGZlt{}source\PYGZhy{}dir\PYGZgt{}}\PYG{l}{ }\PYG{n+nv}{\PYGZlt{}dest\PYGZhy{}dir\PYGZgt{}}
\end{sphinxVerbatim}


\subsection{Description}
\label{\detokenize{man/cp:description}}
The \sphinxstyleliteralstrong{\sphinxupquote{s3qlcp}} command duplicates the directory tree \sphinxcode{\sphinxupquote{source-dir}}
into \sphinxcode{\sphinxupquote{dest-dir}} without physically copying the file contents.
Both source and destination must lie inside the same S3QL file system.

The replication will not take any additional space. Only if one of
directories is modified later on, the modified data will take
additional storage space.

\sphinxcode{\sphinxupquote{s3qlcp}} can only be called by the user that mounted the file system
and (if the file system was mounted with \sphinxcode{\sphinxupquote{-{-}allow-other}} or \sphinxcode{\sphinxupquote{-{-}allow-root}})
the root user.

Note that:
\begin{itemize}
\item {} 
After the replication, both source and target directory will still
be completely ordinary directories. You can regard \sphinxcode{\sphinxupquote{\textless{}src\textgreater{}}} as a
snapshot of \sphinxcode{\sphinxupquote{\textless{}target\textgreater{}}} or vice versa. However, the most common
usage of \sphinxcode{\sphinxupquote{s3qlcp}} is to regularly duplicate the same source
directory, say \sphinxcode{\sphinxupquote{documents}}, to different target directories. For a
e.g. monthly replication, the target directories would typically be
named something like \sphinxcode{\sphinxupquote{documents\_January}} for the replication in
January, \sphinxcode{\sphinxupquote{documents\_February}} for the replication in February etc.
In this case it is clear that the target directories should be
regarded as snapshots of the source directory.

\item {} 
Exactly the same effect could be achieved by an ordinary copy
program like \sphinxcode{\sphinxupquote{cp -a}}. However, this procedure would be orders of
magnitude slower, because \sphinxcode{\sphinxupquote{cp}} would have to read every file
completely (so that S3QL had to fetch all the data over the network
from the backend) before writing them into the destination folder.

\end{itemize}


\subsubsection{Snapshotting vs Hardlinking}
\label{\detokenize{man/cp:snapshotting-vs-hardlinking}}
Snapshot support in S3QL is inspired by the hardlinking feature that
is offered by programs like \sphinxhref{http://www.samba.org/rsync}{rsync} or
\sphinxhref{http://savannah.nongnu.org/projects/storebackup}{storeBackup}.
These programs can create a hardlink instead of copying a file if an
identical file already exists in the backup. However, using hardlinks
has two large disadvantages:
\begin{itemize}
\item {} 
backups and restores always have to be made with a special program
that takes care of the hardlinking. The backup must not be touched
by any other programs (they may make changes that inadvertently
affect other hardlinked files)

\item {} 
special care needs to be taken to handle files which are already
hardlinked (the restore program needs to know that the hardlink was
not just introduced by the backup program to safe space)

\end{itemize}

S3QL snapshots do not have these problems, and they can be used with
any backup program.


\subsection{Options}
\label{\detokenize{man/cp:options}}
The \sphinxstyleliteralstrong{\sphinxupquote{s3qlcp}} command accepts the following options:
\begin{quote}
\begin{optionlist}{3cm}
\item [-{-}log \textless{}target\textgreater{}]  
Destination for log messages. Specify \sphinxcode{\sphinxupquote{none}} for
standard output or \sphinxcode{\sphinxupquote{syslog}} for the system logging
daemon. Anything else will be interpreted as a file
name. Log files will be rotated when they reach 1 MiB,
and at most 5 old log files will be kept. Default:
\sphinxcode{\sphinxupquote{None}}
\item [-{-}debug-modules \textless{}modules\textgreater{}]  
Activate debugging output from specified modules (use
commas to separate multiple modules). Debug messages
will be written to the target specified by the
\sphinxcode{\sphinxupquote{-{-}log}} option.
\item [-{-}debug]  
Activate debugging output from all S3QL modules. Debug
messages will be written to the target specified by
the \sphinxcode{\sphinxupquote{-{-}log}} option.
\item [-{-}quiet]  
be really quiet
\item [-{-}version]  
just print program version and exit
\end{optionlist}
\end{quote}


\subsection{Exit Codes}
\label{\detokenize{man/cp:exit-codes}}
\sphinxstyleliteralstrong{\sphinxupquote{s3qlcp}} may terminate with the following exit codes:
\begin{quote}\begin{description}
\item[{0}] \leavevmode
Everything went well.

\item[{1}] \leavevmode
An unexpected error occured. This may indicate a bug in the
program.

\item[{2}] \leavevmode
Invalid command line argument or configuration file key.

\end{description}\end{quote}


\subsection{See Also}
\label{\detokenize{man/cp:see-also}}
The S3QL homepage is at \sphinxurl{https://github.com/s3ql/s3ql/}.

The full S3QL documentation should also be installed somewhere on your
system, common locations are \sphinxcode{\sphinxupquote{/usr/share/doc/s3ql}} or
\sphinxcode{\sphinxupquote{/usr/local/doc/s3ql}}.


\section{The \sphinxstyleliteralstrong{\sphinxupquote{s3qlrm}} command}
\label{\detokenize{man/rm:the-command-command}}\label{\detokenize{man/rm::doc}}

\subsection{Synopsis}
\label{\detokenize{man/rm:synopsis}}
\begin{sphinxVerbatim}[commandchars=\\\{\}]
\PYG{l}{s3qlrm }\PYG{g+ge}{[options]}\PYG{l}{ }\PYG{n+nv}{\PYGZlt{}directory\PYGZgt{}}
\end{sphinxVerbatim}


\subsection{Description}
\label{\detokenize{man/rm:description}}
The \sphinxstyleliteralstrong{\sphinxupquote{s3qlrm}} command recursively deletes files and directories on an
S3QL file system. Although \sphinxstyleliteralstrong{\sphinxupquote{s3qlrm}} is faster than using e.g.
\sphinxstyleliteralstrong{\sphinxupquote{rm -r{}`}}, the main reason for its existence is that it allows
you to delete immutable trees (which can be created with
\sphinxstyleliteralstrong{\sphinxupquote{s3qllock}}) as well.

Be warned that there is no additional confirmation. The directory will
be removed entirely and immediately.

\sphinxstyleliteralstrong{\sphinxupquote{s3qlrm}} can only be called by the user that mounted the file system
and (if the file system was mounted with \sphinxcode{\sphinxupquote{-{-}allow-other}} or
\sphinxcode{\sphinxupquote{-{-}allow-root}}) the root user.


\subsection{Options}
\label{\detokenize{man/rm:options}}
The \sphinxstyleliteralstrong{\sphinxupquote{s3qlrm}} command accepts the following options:
\begin{quote}
\begin{optionlist}{3cm}
\item [-{-}log \textless{}target\textgreater{}]  
Destination for log messages. Specify \sphinxcode{\sphinxupquote{none}} for
standard output or \sphinxcode{\sphinxupquote{syslog}} for the system logging
daemon. Anything else will be interpreted as a file
name. Log files will be rotated when they reach 1 MiB,
and at most 5 old log files will be kept. Default:
\sphinxcode{\sphinxupquote{None}}
\item [-{-}debug-modules \textless{}modules\textgreater{}]  
Activate debugging output from specified modules (use
commas to separate multiple modules). Debug messages
will be written to the target specified by the
\sphinxcode{\sphinxupquote{-{-}log}} option.
\item [-{-}debug]  
Activate debugging output from all S3QL modules. Debug
messages will be written to the target specified by
the \sphinxcode{\sphinxupquote{-{-}log}} option.
\item [-{-}quiet]  
be really quiet
\item [-{-}version]  
just print program version and exit
\end{optionlist}
\end{quote}


\subsection{Exit Codes}
\label{\detokenize{man/rm:exit-codes}}
\sphinxstyleliteralstrong{\sphinxupquote{s3qlrm}} may terminate with the following exit codes:
\begin{quote}\begin{description}
\item[{0}] \leavevmode
Everything went well.

\item[{1}] \leavevmode
An unexpected error occured. This may indicate a bug in the
program.

\item[{2}] \leavevmode
Invalid command line argument or configuration file key.

\end{description}\end{quote}


\subsection{See Also}
\label{\detokenize{man/rm:see-also}}
The S3QL homepage is at \sphinxurl{https://github.com/s3ql/s3ql/}.

The full S3QL documentation should also be installed somewhere on your
system, common locations are \sphinxcode{\sphinxupquote{/usr/share/doc/s3ql}} or
\sphinxcode{\sphinxupquote{/usr/local/doc/s3ql}}.


\section{The \sphinxstyleliteralstrong{\sphinxupquote{s3qllock}} command}
\label{\detokenize{man/lock:the-command-command}}\label{\detokenize{man/lock::doc}}

\subsection{Synopsis}
\label{\detokenize{man/lock:synopsis}}
\begin{sphinxVerbatim}[commandchars=\\\{\}]
\PYG{l}{s3qllock }\PYG{g+ge}{[options]}\PYG{l}{ }\PYG{n+nv}{\PYGZlt{}directory\PYGZgt{}}
\end{sphinxVerbatim}


\subsection{Description}
\label{\detokenize{man/lock:description}}
The \sphinxstyleliteralstrong{\sphinxupquote{s3qllock}} command makes a directory tree in an S3QL file
system immutable. Immutable trees can no longer be changed in any way
whatsoever. You can not add new files or directories and you can not
change or delete existing files and directories. The only way to get
rid of an immutable tree is to use the \sphinxstyleliteralstrong{\sphinxupquote{s3qlrm}} command.

\sphinxstyleliteralstrong{\sphinxupquote{s3qllock}} can only be called by the user that mounted the file system
and (if the file system was mounted with \sphinxcode{\sphinxupquote{-{-}allow-other}} or
\sphinxcode{\sphinxupquote{-{-}allow-root}}) the root user.


\subsection{Rationale}
\label{\detokenize{man/lock:rationale}}
Immutability is a feature designed for backups. Traditionally, backups
have been made on external tape drives. Once a backup was made, the
tape drive was removed and locked somewhere in a shelf. This has the
great advantage that the contents of the backup are now permanently
fixed. Nothing (short of physical destruction) can change or delete
files in the backup.

In contrast, when backing up into an online storage system like S3QL,
all backups are available every time the file system is mounted.
Nothing prevents a file in an old backup from being changed again
later on. In the worst case, this may make your entire backup system
worthless. Imagine that your system gets infected by a nasty virus
that simply deletes all files it can find \textendash{} if the virus is active
while the backup file system is mounted, the virus will destroy all
your old backups as well!

Even if the possibility of a malicious virus or trojan horse is
excluded, being able to change a backup after it has been made is
generally not a good idea. A common S3QL use case is to keep the file
system mounted at all times and periodically create backups with
\sphinxstyleliteralstrong{\sphinxupquote{rsync -a}}. This allows every user to recover her files from a
backup without having to call the system administrator. However, this
also allows every user to accidentally change or delete files \sphinxstyleemphasis{in} one
of the old backups.

Making a backup immutable protects you against all these problems.
Unless you happen to run into a virus that was specifically programmed
to attack S3QL file systems, backups can be neither deleted nor
changed after they have been made immutable.


\subsection{Options}
\label{\detokenize{man/lock:options}}
The \sphinxstyleliteralstrong{\sphinxupquote{s3qllock}} command accepts the following options:
\begin{quote}
\begin{optionlist}{3cm}
\item [-{-}log \textless{}target\textgreater{}]  
Destination for log messages. Specify \sphinxcode{\sphinxupquote{none}} for
standard output or \sphinxcode{\sphinxupquote{syslog}} for the system logging
daemon. Anything else will be interpreted as a file
name. Log files will be rotated when they reach 1 MiB,
and at most 5 old log files will be kept. Default:
\sphinxcode{\sphinxupquote{None}}
\item [-{-}debug-modules \textless{}modules\textgreater{}]  
Activate debugging output from specified modules (use
commas to separate multiple modules). Debug messages
will be written to the target specified by the
\sphinxcode{\sphinxupquote{-{-}log}} option.
\item [-{-}debug]  
Activate debugging output from all S3QL modules. Debug
messages will be written to the target specified by
the \sphinxcode{\sphinxupquote{-{-}log}} option.
\item [-{-}quiet]  
be really quiet
\item [-{-}version]  
just print program version and exit
\end{optionlist}
\end{quote}


\subsection{Exit Codes}
\label{\detokenize{man/lock:exit-codes}}
\sphinxstyleliteralstrong{\sphinxupquote{s3qllock}} may terminate with the following exit codes:
\begin{quote}\begin{description}
\item[{0}] \leavevmode
Everything went well.

\item[{1}] \leavevmode
An unexpected error occured. This may indicate a bug in the
program.

\item[{2}] \leavevmode
Invalid command line argument or configuration file key.

\end{description}\end{quote}


\subsection{See Also}
\label{\detokenize{man/lock:see-also}}
The S3QL homepage is at \sphinxurl{https://github.com/s3ql/s3ql/}.

The full S3QL documentation should also be installed somewhere on your
system, common locations are \sphinxcode{\sphinxupquote{/usr/share/doc/s3ql}} or
\sphinxcode{\sphinxupquote{/usr/local/doc/s3ql}}.


\section{The \sphinxstyleliteralstrong{\sphinxupquote{umount.s3ql}} command}
\label{\detokenize{man/umount:the-command-command}}\label{\detokenize{man/umount::doc}}

\subsection{Synopsis}
\label{\detokenize{man/umount:synopsis}}
\begin{sphinxVerbatim}[commandchars=\\\{\}]
\PYG{l}{umount.s3ql }\PYG{g+ge}{[options]}\PYG{l}{ }\PYG{n+nv}{\PYGZlt{}mount point\PYGZgt{}}
\end{sphinxVerbatim}


\subsection{Description}
\label{\detokenize{man/umount:description}}
The \sphinxstyleliteralstrong{\sphinxupquote{umount.s3ql}} command unmounts the S3QL file system mounted in the
directory \sphinxstyleemphasis{mount point} and blocks until all data has been uploaded to
the storage backend.

Only the user who mounted the file system with \sphinxstyleliteralstrong{\sphinxupquote{mount.s3ql}}
is able to unmount it with \sphinxstyleliteralstrong{\sphinxupquote{umount.s3ql}}. If you are root and want to
unmount an S3QL file system mounted by an ordinary user, you have to
use the \sphinxstyleliteralstrong{\sphinxupquote{fusermount -u}} or \sphinxstyleliteralstrong{\sphinxupquote{umount}} command instead.
Note that these commands do not block until all data has been
uploaded, so if you use them instead of \sphinxstyleliteralstrong{\sphinxupquote{umount.s3ql}} then
you should manually wait for the \sphinxstyleliteralstrong{\sphinxupquote{mount.s3ql}} process to
terminate before shutting down the system.


\subsection{Options}
\label{\detokenize{man/umount:options}}
The \sphinxstyleliteralstrong{\sphinxupquote{umount.s3ql}} command accepts the following options.
\begin{quote}
\begin{optionlist}{3cm}
\item [-{-}log \textless{}target\textgreater{}]  
Destination for log messages. Specify \sphinxcode{\sphinxupquote{none}} for
standard output or \sphinxcode{\sphinxupquote{syslog}} for the system logging
daemon. Anything else will be interpreted as a file
name. Log files will be rotated when they reach 1 MiB,
and at most 5 old log files will be kept. Default:
\sphinxcode{\sphinxupquote{None}}
\item [-{-}debug-modules \textless{}modules\textgreater{}]  
Activate debugging output from specified modules (use
commas to separate multiple modules). Debug messages
will be written to the target specified by the
\sphinxcode{\sphinxupquote{-{-}log}} option.
\item [-{-}debug]  
Activate debugging output from all S3QL modules. Debug
messages will be written to the target specified by
the \sphinxcode{\sphinxupquote{-{-}log}} option.
\item [-{-}quiet]  
be really quiet
\item [-{-}version]  
just print program version and exit
\item [-{-}lazy, -z]  
Lazy umount. Detaches the file system immediately,
even if there are still open files. The data will be
uploaded in the background once all open files have
been closed.
\end{optionlist}
\end{quote}


\subsection{Exit Codes}
\label{\detokenize{man/umount:exit-codes}}
\sphinxstyleliteralstrong{\sphinxupquote{umount.s3ql}} may terminate with the following exit codes:
\begin{quote}\begin{description}
\item[{0}] \leavevmode
Everything went well.

\item[{1}] \leavevmode
An unexpected error occured. This may indicate a bug in the
program.

\item[{2}] \leavevmode
Invalid command line argument or configuration file key.

\end{description}\end{quote}


\subsection{See Also}
\label{\detokenize{man/umount:see-also}}
The S3QL homepage is at \sphinxurl{https://github.com/s3ql/s3ql/}.

The full S3QL documentation should also be installed somewhere on your
system, common locations are \sphinxcode{\sphinxupquote{/usr/share/doc/s3ql}} or
\sphinxcode{\sphinxupquote{/usr/local/doc/s3ql}}.


\section{The \sphinxstyleliteralstrong{\sphinxupquote{fsck.s3ql}} command}
\label{\detokenize{man/fsck:the-command-command}}\label{\detokenize{man/fsck::doc}}

\subsection{Synopsis}
\label{\detokenize{man/fsck:synopsis}}
\begin{sphinxVerbatim}[commandchars=\\\{\}]
\PYG{l}{fsck.s3ql }\PYG{g+ge}{[options]}\PYG{l}{ }\PYG{n+nv}{\PYGZlt{}storage url\PYGZgt{}}
\end{sphinxVerbatim}


\subsection{Description}
\label{\detokenize{man/fsck:description}}
The \sphinxstyleliteralstrong{\sphinxupquote{fsck.s3ql}} command checks the file system in the location specified
by \sphinxstyleemphasis{storage url} for errors and attempts to repair any problems. The
storage url depends on the backend that is used. The S3QL User’s Guide
should be consulted for a description of the available backends.


\subsection{Options}
\label{\detokenize{man/fsck:options}}
The \sphinxstyleliteralstrong{\sphinxupquote{fsck.s3ql}} command accepts the following options.
\begin{quote}
\begin{optionlist}{3cm}
\item [-{-}log \textless{}target\textgreater{}]  
Destination for log messages. Specify \sphinxcode{\sphinxupquote{none}} for
standard output or \sphinxcode{\sphinxupquote{syslog}} for the system logging
daemon. Anything else will be interpreted as a file
name. Log files will be rotated when they reach 1 MiB,
and at most 5 old log files will be kept. Default:
\sphinxcode{\sphinxupquote{\textasciitilde{}/.s3ql/fsck.log}}
\item [-{-}cachedir \textless{}path\textgreater{}]  
Store cached data in this directory (default:
\sphinxcode{\sphinxupquote{\textasciitilde{}/.s3ql)}}
\item [-{-}debug-modules \textless{}modules\textgreater{}]  
Activate debugging output from specified modules (use
commas to separate multiple modules). Debug messages
will be written to the target specified by the
\sphinxcode{\sphinxupquote{-{-}log}} option.
\item [-{-}debug]  
Activate debugging output from all S3QL modules. Debug
messages will be written to the target specified by
the \sphinxcode{\sphinxupquote{-{-}log}} option.
\item [-{-}quiet]  
be really quiet
\item [-{-}backend-options \textless{}options\textgreater{}]  
Backend specific options (separate by commas). See
backend documentation for available options.
\item [-{-}version]  
just print program version and exit
\item [-{-}authfile \textless{}path\textgreater{}]  
Read authentication credentials from this file
(default: \sphinxcode{\sphinxupquote{\textasciitilde{}/.s3ql/authinfo2)}}
\item [-{-}compress \textless{}algorithm-lvl\textgreater{}]  
Compression algorithm and compression level to use
when storing new data. \sphinxstyleemphasis{algorithm} may be any of
\sphinxcode{\sphinxupquote{lzma}}, \sphinxcode{\sphinxupquote{bzip2}}, \sphinxcode{\sphinxupquote{zlib}}, or none. \sphinxstyleemphasis{lvl} may be any
integer from 0 (fastest) to 9 (slowest). Default:
\sphinxcode{\sphinxupquote{lzma-6}}
\item [-{-}keep-cache]  
Do not purge locally cached files on exit.
\item [-{-}batch]  
If user input is required, exit without prompting.
\item [-{-}force]  
Force checking even if file system is marked clean.
\item [-{-}force-remote]  
Force use of remote metadata even when this would
likely result in data loss.
\end{optionlist}
\end{quote}


\subsection{Exit Codes}
\label{\detokenize{man/fsck:exit-codes}}
If \sphinxstyleliteralstrong{\sphinxupquote{fsck.s3ql}} found any file system errors (no matter if they were
corrected or not), the exit code will be 128 plus one of the codes
listed below. If no errors were found, the following exit codes are
used as-is:
\begin{quote}\begin{description}
\item[{0}] \leavevmode
Everything went well.

\item[{1}] \leavevmode
An unexpected error occured. This may indicate a bug in the
program.

\item[{2}] \leavevmode
Invalid command line argument or configuration file key.

\item[{3}] \leavevmode
Invalid backend option.

\item[{10}] \leavevmode
Could not open log file for writing.

\item[{11}] \leavevmode
No such backend.

\item[{12}] \leavevmode
Authentication file has insecure permissions.

\item[{13}] \leavevmode
Unable to parse proxy settings.

\item[{14}] \leavevmode
Invalid credentials (Authentication failed).

\item[{15}] \leavevmode
No permission to access backend (Authorization denied).

\item[{16}] \leavevmode
Invalid storage URL, specified location does not exist in backend.

\item[{17}] \leavevmode
Wrong file system passphrase.

\item[{18}] \leavevmode
No S3QL file system found at given storage URL.

\item[{19}] \leavevmode
Unable to connect to backend, can’t resolve hostname.

\item[{32}] \leavevmode
Unsupported file system revision (too old).

\item[{33}] \leavevmode
Unsupported file system revision (too new).

\item[{40}] \leavevmode
Cannot check mounted file system.

\item[{41}] \leavevmode
User input required, but running in batch mode.

\item[{42}] \leavevmode
File system check aborted by user.

\item[{43}] \leavevmode
Local metadata is corrupted.

\item[{44}] \leavevmode
Uncorrectable errors found.

\item[{45}] \leavevmode
Unable to access cache directory.

\item[{128}] \leavevmode
This error code will be \sphinxstyleemphasis{added} to one of the codes above if any
file system errors have been found (no matter if they were
corrected or not).

\end{description}\end{quote}


\subsection{See Also}
\label{\detokenize{man/fsck:see-also}}
The S3QL homepage is at \sphinxurl{https://github.com/s3ql/s3ql/}.

The full S3QL documentation should also be installed somewhere on your
system, common locations are \sphinxcode{\sphinxupquote{/usr/share/doc/s3ql}} or
\sphinxcode{\sphinxupquote{/usr/local/doc/s3ql}}.


\section{The \sphinxstyleliteralstrong{\sphinxupquote{s3ql\_oauth\_client}} command}
\label{\detokenize{man/oauth_client:the-command-command}}\label{\detokenize{man/oauth_client:oauth-client}}\label{\detokenize{man/oauth_client::doc}}

\subsection{Synopsis}
\label{\detokenize{man/oauth_client:synopsis}}
\begin{sphinxVerbatim}[commandchars=\\\{\}]
\PYG{l}{s3ql\PYGZus{}oauth\PYGZus{}client }\PYG{g+ge}{[options]}
\end{sphinxVerbatim}


\subsection{Description}
\label{\detokenize{man/oauth_client:description}}
The \sphinxstyleliteralstrong{\sphinxupquote{s3ql\_oauth\_client}} command may be used to obtain OAuth2 authentication
tokens for use with Google Storage. It requests “user code” from
Google which has to be pasted into the browser to complete the
authentication process interactively. Once authentication in the
browser has been completed, \sphinxstyleliteralstrong{\sphinxupquote{s3ql\_oauth\_client}} displays the OAuth2 refresh
token.

When combined with the special username \sphinxcode{\sphinxupquote{oauth2}}, the refresh token
can be used as a backend passphrase when using the Google Storage S3QL
backend.


\subsection{Options}
\label{\detokenize{man/oauth_client:options}}
The \sphinxstyleliteralstrong{\sphinxupquote{s3ql\_oauth\_client}} command accepts the following options:


\subsection{Exit Codes}
\label{\detokenize{man/oauth_client:exit-codes}}
\sphinxstyleliteralstrong{\sphinxupquote{s3ql\_oauth\_client}} may terminate with the following exit codes:
\begin{quote}\begin{description}
\item[{0}] \leavevmode
Everything went well.

\item[{1}] \leavevmode
An unexpected error occured. This may indicate a bug in the
program.

\item[{2}] \leavevmode
Invalid command line argument or configuration file key.

\end{description}\end{quote}


\subsection{See Also}
\label{\detokenize{man/oauth_client:see-also}}
The S3QL homepage is at \sphinxurl{https://github.com/s3ql/s3ql/}.

The full S3QL documentation should also be installed somewhere on your
system, common locations are \sphinxcode{\sphinxupquote{/usr/share/doc/s3ql}} or
\sphinxcode{\sphinxupquote{/usr/local/doc/s3ql}}.


\section{The \sphinxstyleliteralstrong{\sphinxupquote{s3ql\_verify}} command}
\label{\detokenize{man/verify:the-command-command}}\label{\detokenize{man/verify::doc}}

\subsection{Synopsis}
\label{\detokenize{man/verify:synopsis}}
\begin{sphinxVerbatim}[commandchars=\\\{\}]
\PYG{l}{s3ql\PYGZus{}verify }\PYG{g+ge}{[options]}\PYG{l}{ }\PYG{n+nv}{\PYGZlt{}storage url\PYGZgt{}}
\end{sphinxVerbatim}


\subsection{Description}
\label{\detokenize{man/verify:description}}
The \sphinxstyleliteralstrong{\sphinxupquote{s3ql\_verify}} command verifies all data in the file system.  In
contrast to \sphinxstyleliteralstrong{\sphinxupquote{fsck.s3ql}}, \sphinxstyleliteralstrong{\sphinxupquote{s3ql\_verify}} does not trust the object
listing returned by the backend, but actually attempts to retrieve
every object. It therefore takes a lot longer.

The format of \sphinxcode{\sphinxupquote{\textless{}storage url\textgreater{}}} depends on the backend that is
used. The S3QL User’s Guide should be consulted for a description of
the available backends.


\subsection{Options}
\label{\detokenize{man/verify:options}}
The \sphinxstyleliteralstrong{\sphinxupquote{s3ql\_verify}} command accepts the following options.
\begin{quote}
\begin{optionlist}{3cm}
\item [-{-}log \textless{}target\textgreater{}]  
Destination for log messages. Specify \sphinxcode{\sphinxupquote{none}} for
standard output or \sphinxcode{\sphinxupquote{syslog}} for the system logging
daemon. Anything else will be interpreted as a file
name. Log files will be rotated when they reach 1 MiB,
and at most 5 old log files will be kept. Default:
\sphinxcode{\sphinxupquote{None}}
\item [-{-}debug-modules \textless{}modules\textgreater{}]  
Activate debugging output from specified modules (use
commas to separate multiple modules). Debug messages
will be written to the target specified by the
\sphinxcode{\sphinxupquote{-{-}log}} option.
\item [-{-}debug]  
Activate debugging output from all S3QL modules. Debug
messages will be written to the target specified by
the \sphinxcode{\sphinxupquote{-{-}log}} option.
\item [-{-}quiet]  
be really quiet
\item [-{-}version]  
just print program version and exit
\item [-{-}cachedir \textless{}path\textgreater{}]  
Store cached data in this directory (default:
\sphinxcode{\sphinxupquote{\textasciitilde{}/.s3ql)}}
\item [-{-}backend-options \textless{}options\textgreater{}]  
Backend specific options (separate by commas). See
backend documentation for available options.
\item [-{-}authfile \textless{}path\textgreater{}]  
Read authentication credentials from this file
(default: \sphinxcode{\sphinxupquote{\textasciitilde{}/.s3ql/authinfo2)}}
\item [-{-}missing-file \textless{}name\textgreater{}]  
File to store keys of missing objects.
\item [-{-}corrupted-file \textless{}name\textgreater{}]  
File to store keys of corrupted objects.
\item [-{-}data]  
Read every object completely, instead of checking just
the metadata.
\item [-{-}parallel PARALLEL]  
Number of connections to use in parallel.
\item [-{-}start-with \textless{}n\textgreater{}]  
Skip over first \textless{}n\textgreater{} objects and with verifying object
\textless{}n\textgreater{}+1.
\end{optionlist}
\end{quote}


\subsection{Exit Codes}
\label{\detokenize{man/verify:exit-codes}}
\sphinxstyleliteralstrong{\sphinxupquote{s3ql\_verify}} may terminate with the following exit codes:
\begin{quote}\begin{description}
\item[{0}] \leavevmode
Everything went well.

\item[{1}] \leavevmode
An unexpected error occured. This may indicate a bug in the
program.

\item[{2}] \leavevmode
Invalid command line argument or configuration file key.

\item[{3}] \leavevmode
Invalid backend option.

\item[{10}] \leavevmode
Could not open log file for writing.

\item[{11}] \leavevmode
No such backend.

\item[{12}] \leavevmode
Authentication file has insecure permissions.

\item[{13}] \leavevmode
Unable to parse proxy settings.

\item[{14}] \leavevmode
Invalid credentials (Authentication failed).

\item[{15}] \leavevmode
No permission to access backend (Authorization denied).

\item[{16}] \leavevmode
Invalid storage URL, specified location does not exist in backend.

\item[{17}] \leavevmode
Wrong file system passphrase.

\item[{18}] \leavevmode
No S3QL file system found at given storage URL.

\item[{19}] \leavevmode
Unable to connect to backend, can’t resolve hostname.

\item[{32}] \leavevmode
Unsupported file system revision (too old).

\item[{33}] \leavevmode
Unsupported file system revision (too new).

\item[{45}] \leavevmode
Unable to access cache directory.

\item[{46}] \leavevmode
The file system data was verified, and some objects were found
to be missing or corrupted.

\end{description}\end{quote}


\subsection{See Also}
\label{\detokenize{man/verify:see-also}}
The S3QL homepage is at \sphinxurl{https://github.com/s3ql/s3ql/}.

The full S3QL documentation should also be installed somewhere on your
system, common locations are \sphinxcode{\sphinxupquote{/usr/share/doc/s3ql}} or
\sphinxcode{\sphinxupquote{/usr/local/doc/s3ql}}.


\section{The \sphinxstyleliteralstrong{\sphinxupquote{pcp}} command}
\label{\detokenize{man/pcp:the-command-command}}\label{\detokenize{man/pcp::doc}}

\subsection{Synopsis}
\label{\detokenize{man/pcp:synopsis}}
\begin{sphinxVerbatim}[commandchars=\\\{\}]
\PYG{l}{pcp }\PYG{g+ge}{[options]}\PYG{l}{ }\PYG{n+nv}{\PYGZlt{}source\PYGZgt{}}\PYG{l}{ }\PYG{g+ge}{[\PYGZlt{}source\PYGZgt{} ...]}\PYG{l}{ }\PYG{n+nv}{\PYGZlt{}destination\PYGZgt{}}
\end{sphinxVerbatim}


\subsection{Description}
\label{\detokenize{man/pcp:description}}
The \sphinxstyleliteralstrong{\sphinxupquote{pcp}} command is a is a wrapper that starts several
\sphinxstyleliteralstrong{\sphinxupquote{sync}} processes to copy directory trees in parallel. This is
allows much better copying performance on file system that have
relatively high latency when retrieving individual files like S3QL.

\sphinxstylestrong{Note}: Using this program only improves performance when copying
\sphinxstyleemphasis{from} an S3QL file system. When copying \sphinxstyleemphasis{to} an S3QL file system,
using \sphinxstyleliteralstrong{\sphinxupquote{pcp}} is more likely to \sphinxstyleemphasis{decrease} performance.


\subsection{Options}
\label{\detokenize{man/pcp:options}}
The \sphinxstyleliteralstrong{\sphinxupquote{pcp}} command accepts the following options:
\begin{quote}
\begin{optionlist}{3cm}
\item [-{-}log \textless{}target\textgreater{}]  
Destination for log messages. Specify \sphinxcode{\sphinxupquote{none}} for
standard output or \sphinxcode{\sphinxupquote{syslog}} for the system logging
daemon. Anything else will be interpreted as a file
name. Log files will be rotated when they reach 1 MiB,
and at most 5 old log files will be kept. Default:
\sphinxcode{\sphinxupquote{None}}
\item [-{-}quiet]  
be really quiet
\item [-{-}debug-modules \textless{}modules\textgreater{}]  
Activate debugging output from specified modules (use
commas to separate multiple modules). Debug messages
will be written to the target specified by the
\sphinxcode{\sphinxupquote{-{-}log}} option.
\item [-{-}debug]  
Activate debugging output from all S3QL modules. Debug
messages will be written to the target specified by
the \sphinxcode{\sphinxupquote{-{-}log}} option.
\item [-{-}version]  
just print program version and exit
\item [-a]  
Pass -aHAX option to rsync.
\item [-{-}processes \textless{}no\textgreater{}]  
Number of rsync processes to use (default: 10).
\end{optionlist}
\end{quote}


\subsection{Exit Codes}
\label{\detokenize{man/pcp:exit-codes}}
\sphinxstyleliteralstrong{\sphinxupquote{pcp}} may terminate with the following exit codes:
\begin{quote}\begin{description}
\item[{0}] \leavevmode
Everything went well.

\item[{1}] \leavevmode
An unexpected error occured. This may indicate a bug in the
program.

\item[{2}] \leavevmode
Invalid command line argument or configuration file key.

\end{description}\end{quote}


\subsection{See Also}
\label{\detokenize{man/pcp:see-also}}
\sphinxstyleliteralstrong{\sphinxupquote{pcp}} is shipped as part of S3QL, \sphinxurl{https://github.com/s3ql/s3ql/}.


\section{The \sphinxstyleliteralstrong{\sphinxupquote{expire\_backups}} command}
\label{\detokenize{man/expire_backups:the-command-command}}\label{\detokenize{man/expire_backups::doc}}

\subsection{Synopsis}
\label{\detokenize{man/expire_backups:synopsis}}
\begin{sphinxVerbatim}[commandchars=\\\{\}]
\PYG{l}{expire\PYGZus{}backups }\PYG{g+ge}{[options]}\PYG{l}{ }\PYG{n+nv}{\PYGZlt{}age\PYGZgt{}}\PYG{l}{ }\PYG{g+ge}{[\PYGZlt{}age\PYGZgt{} ...]}
\end{sphinxVerbatim}


\subsection{Description}
\label{\detokenize{man/expire_backups:description}}
The \sphinxstyleliteralstrong{\sphinxupquote{expire\_backups}} command intelligently remove old backups that are no
longer needed.

To define what backups you want to keep for how long, you define a
number of \sphinxstyleemphasis{age ranges}. \sphinxstyleliteralstrong{\sphinxupquote{expire\_backups}} ensures that you
will have at least one backup in each age range at all times. It will
keep exactly as many backups as are required for that and delete any
backups that become redundant.

Age ranges are specified by giving a list of range boundaries in terms
of backup cycles. Every time you create a new backup, the existing
backups age by one cycle.

Example: when \sphinxstyleliteralstrong{\sphinxupquote{expire\_backups}} is called with the age range
definition \sphinxcode{\sphinxupquote{1 3 7 14 31}}, it will guarantee that you always have the
following backups available:
\begin{enumerate}
\def\theenumi{\arabic{enumi}}
\def\labelenumi{\theenumi .}
\makeatletter\def\p@enumii{\p@enumi \theenumi .}\makeatother
\item {} 
A backup that is 0 to 1 cycles old (i.e, the most recent backup)

\item {} 
A backup that is 1 to 3 cycles old

\item {} 
A backup that is 3 to 7 cycles old

\item {} 
A backup that is 7 to 14 cycles old

\item {} 
A backup that is 14 to 31 cycles old

\end{enumerate}

\begin{sphinxadmonition}{note}{Note:}
If you do backups in fixed intervals, then one cycle will be
equivalent to the backup interval. The advantage of specifying the
age ranges in terms of backup cycles rather than days or weeks is
that it allows you to gracefully handle irregular backup intervals.
Imagine that for some reason you do not turn on your computer for
one month. Now all your backups are at least a month old, and if you
had specified the above backup strategy in terms of absolute ages,
they would all be deleted! Specifying age ranges in terms of backup
cycles avoids these sort of problems.
\end{sphinxadmonition}

\sphinxstyleliteralstrong{\sphinxupquote{expire\_backups}} usage is simple. It requires backups to be
stored in directories of the form \sphinxcode{\sphinxupquote{year-month-day\_hour:minute:seconds}}
(\sphinxcode{\sphinxupquote{YYYY-MM-DD\_HH:mm:ss}}) and works on all backups in the current
directory. So for the above backup strategy, the correct invocation
would be:

\begin{sphinxVerbatim}[commandchars=\\\{\}]
\PYG{l}{expire\PYGZus{}backups.py 1 3 7 14 31}
\end{sphinxVerbatim}

When storing your backups on an S3QL file system, you probably want to
specify the \sphinxcode{\sphinxupquote{-{-}use-s3qlrm}} option as well. This tells
\sphinxstyleliteralstrong{\sphinxupquote{expire\_backups}} to use the {\hyperref[\detokenize{special:s3qlrm}]{\sphinxcrossref{\DUrole{std,std-ref}{s3qlrm}}}} command to
delete directories.

\sphinxstyleliteralstrong{\sphinxupquote{expire\_backups}} uses a “state file” to keep track which
backups are how many cycles old (since this cannot be inferred from
the dates contained in the directory names). The standard name for
this state file is \sphinxcode{\sphinxupquote{.expire\_backups.dat}}. If this file gets
damaged or deleted, \sphinxstyleliteralstrong{\sphinxupquote{expire\_backups}} no longer knows the ages
of the backups and refuses to work. In this case you can use the
\sphinxcode{\sphinxupquote{-{-}reconstruct-state}} option to try to reconstruct the state
from the backup dates. However, the accuracy of this reconstruction
depends strongly on how rigorous you have been with making backups (it
is only completely correct if the time between subsequent backups has
always been exactly the same), so it’s generally a good idea not to
tamper with the state file.


\subsection{Options}
\label{\detokenize{man/expire_backups:options}}
The \sphinxstyleliteralstrong{\sphinxupquote{expire\_backups}} command accepts the following options:
\begin{quote}
\begin{optionlist}{3cm}
\item [-{-}quiet]  
be really quiet
\item [-{-}log \textless{}target\textgreater{}]  
Destination for log messages. Specify \sphinxcode{\sphinxupquote{none}} for
standard output or \sphinxcode{\sphinxupquote{syslog}} for the system logging
daemon. Anything else will be interpreted as a file
name. Log files will be rotated when they reach 1 MiB,
and at most 5 old log files will be kept. Default:
\sphinxcode{\sphinxupquote{None}}
\item [-{-}debug-modules \textless{}modules\textgreater{}]  
Activate debugging output from specified modules (use
commas to separate multiple modules). Debug messages
will be written to the target specified by the
\sphinxcode{\sphinxupquote{-{-}log}} option.
\item [-{-}debug]  
Activate debugging output from all S3QL modules. Debug
messages will be written to the target specified by
the \sphinxcode{\sphinxupquote{-{-}log}} option.
\item [-{-}version]  
just print program version and exit
\item [-{-}state \textless{}file\textgreater{}]  
File to save state information in (default:
“.expire\_backups.dat”)
\item [-n]  
Dry run. Just show which backups would be deleted.
\item [-{-}reconstruct-state]  
Try to reconstruct a missing state file from backup
dates.
\item [-{-}use-s3qlrm]  
Use \sphinxcode{\sphinxupquote{s3qlrm}} command to delete backups.
\end{optionlist}
\end{quote}


\subsection{Exit Codes}
\label{\detokenize{man/expire_backups:exit-codes}}
\sphinxstyleliteralstrong{\sphinxupquote{expire\_backups}} may terminate with the following exit codes:
\begin{quote}\begin{description}
\item[{0}] \leavevmode
Everything went well.

\item[{1}] \leavevmode
An unexpected error occured. This may indicate a bug in the
program.

\item[{2}] \leavevmode
Invalid command line argument or configuration file key.

\end{description}\end{quote}


\subsection{See Also}
\label{\detokenize{man/expire_backups:see-also}}
\sphinxstyleliteralstrong{\sphinxupquote{expire\_backups}} is shipped as part of S3QL, \sphinxurl{https://github.com/s3ql/s3ql/}.


\chapter{Further Resources / Getting Help}
\label{\detokenize{resources:further-resources-getting-help}}\label{\detokenize{resources:resources}}\label{\detokenize{resources::doc}}
If you have questions or problems with S3QL that you weren’t able to
resolve with this manual, you might want to consider the following other resources:
\begin{itemize}
\item {} 
The \sphinxhref{https://github.com/s3ql/s3ql/wiki}{S3QL Wiki}

\item {} 
The \sphinxhref{http://groups.google.com/group/s3ql}{S3QL Mailing List}. You
can subscribe by sending a mail to
\sphinxhref{mailto:s3ql+subscribe@googlegroups.com}{s3ql+subscribe@googlegroups.com}.

\end{itemize}

Please report any bugs you may encounter in the \sphinxhref{https://github.com/s3ql/s3ql/issues}{Issue Tracker}.


\chapter{Implementation Details}
\label{\detokenize{impl_details:implementation-details}}\label{\detokenize{impl_details:impl-details}}\label{\detokenize{impl_details::doc}}
This section provides some background information on how S3QL works
internally. Reading this section is not necessary to use S3QL.


\section{Metadata Storage}
\label{\detokenize{impl_details:metadata-storage}}
Like most unix filesystems, S3QL has a concept of inodes.

The contents of directory inodes (aka the names and inodes of the
files and sub directories contained in a directory) are stored
directly in an \sphinxhref{http://www.sqlite.org/}{SQLite} database. This database
is stored in a special storage object that is downloaded when the file
system is mounted and uploaded periodically in the background and when
the file system is unmounted. This has two implications:
\begin{enumerate}
\def\theenumi{\arabic{enumi}}
\def\labelenumi{\theenumi .}
\makeatletter\def\p@enumii{\p@enumi \theenumi .}\makeatother
\item {} 
The entire file system tree can be read from the
database. Fetching/storing storage objects from/in the storage
backend is only required to access the contents of files (or, more
precisely, inodes). This makes most file system operations very
fast because no data has to be send over the network.

\item {} 
An S3QL filesystem can only be mounted on one computer at a time,
using a single \sphinxstyleliteralstrong{\sphinxupquote{mount.s3ql}} process. Otherwise changes made in
one mountpoint will invariably be overwritten when the second mount
point is unmounted.

\end{enumerate}

Sockets, FIFOs and character devices do not need any additional
storage, all information about them is contained in the database.


\section{Data Storage}
\label{\detokenize{impl_details:data-storage}}
The contents of file inodes are split into individual blocks. The
maximum size of a block is specified when the file system is created
and cannot be changed afterwards. Every block is stored as an
individual object in the backend, and the mapping from inodes to
blocks and from blocks to objects is stored in the database.

While the file system is mounted, blocks are cached locally.

Blocks can also be compressed and encrypted before they are stored in
the storage backend. This happens during upload, i.e. the cached data
is unencrypted and uncompressed.

If some files have blocks with identical contents, the blocks will be
stored in the same backend object (i.e., the data is only stored
once).


\section{Data De-Duplication}
\label{\detokenize{impl_details:data-de-duplication}}
Instead of uploading every block, S3QL first computes a checksum (a
SHA256 hash) to check if an identical blocks has already been stored
in an backend object. If that is the case, the new block will be
linked to the existing object instead of being uploaded.

This procedure is invisible for the user and the contents of the block
can still be changed. If several blocks share a backend object and one
of the blocks is changed, the changed block is automatically stored in
a new object (so that the contents of the other block remain
unchanged).


\section{Caching}
\label{\detokenize{impl_details:caching}}
When an application tries to read or write from a file, S3QL
determines the block that contains the required part of the file and
retrieves it from the backend or creates it if it does not yet exist.
The block is then held in the cache directory. It is committed to S3
when it has not been accessed for more than a few seconds. Blocks are
removed from the cache only when the maximum cache size is reached.

When the file system is unmounted, all modified blocks are written to
the backend and the cache is cleaned.


\section{Eventual Consistency Handling}
\label{\detokenize{impl_details:eventual-consistency-handling}}
S3QL has to take into account that with some storage providers,
changes in objects do not propagate immediately. For example, when an
Amazon S3 object is uploaded and immediately downloaded again, the
downloaded data might not yet reflect the changes done in the upload
(see also
\sphinxurl{http://developer.amazonwebservices.com/connect/message.jspa?messageID=38538})

For the data blocks this is not a problem because a data blocks always
get a new object ID when they are updated.

For the metadata however, S3QL has to make sure that it always
downloads the most recent copy of the database when mounting the file
system.

To that end, metadata versions are numbered, and the most recent
version number is stored as part of the object id of a very small
“marker” object. When S3QL has downloaded the metadata it checks the
version number against the marker object and, if the two do not agree,
waits for the most recent metadata to become available. Once the
current metadata is available, the version number is increased and the
marker object updated.


\section{Encryption}
\label{\detokenize{impl_details:encryption}}
When the file system is created, \sphinxstyleliteralstrong{\sphinxupquote{mkfs.s3ql}} generates a 256 bit
master key by reading from \sphinxcode{\sphinxupquote{/dev/random}}. The master key is
encrypted with the passphrase that is entered by the user, and then
stored with the rest of the file system data. Since the passphrase is
only used to access the master key (which is used to encrypt the
actual file system data), the passphrase can easily be changed.

Data is encrypted with a new session key for each object and each
upload. The session key is generated by appending a nonce to the
master key and then calculating the SHA256 hash. The nonce is
generated by concatenating the object id and the current UTC time as a
32 bit float. The precision of the time is given by the Python \sphinxhref{http://docs.python.org/library/time.html\#time.time}{time()} function and
usually at least 1 millisecond. The SHA256 implementation is included
in the Python standard library.

Once the session key has been calculated, a SHA256 HMAC is calculated
over the data that is to be uploaded. Afterwards, the data is
compressed (unless \sphinxcode{\sphinxupquote{-{-}compress none}} was passed to
\sphinxstyleliteralstrong{\sphinxupquote{mount.s3ql}}) and the HMAC inserted at the beginning. Both HMAC
and compressed data are then encrypted using 256 bit AES in CTR
mode using \sphinxhref{http://www.pycrypto.org/}{PyCrypto}.  Finally, the nonce is
inserted in front of the encrypted data and HMAC, and the packet is
send to the backend as a new S3 object.



\renewcommand{\indexname}{Index}
\printindex
\end{document}