1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
35
36
37
38
39
40
41
42
43
44
45
46
47
48
49
50
51
52
53
54
55
56
57
58
59
60
61
62
63
64
65
66
67
68
69
70
71
72
73
74
75
76
77
78
79
80
81
82
83
84
85
86
87
88
89
90
91
92
93
94
95
96
97
98
99
100
101
102
103
104
105
106
107
108
109
110
111
112
113
114
115
116
117
118
119
120
121
122
123
124
125
126
127
128
129
130
131
132
133
134
135
136
137
138
139
140
141
142
143
144
145
146
147
148
149
150
151
152
153
154
155
156
157
158
159
160
161
162
163
164
165
166
167
168
169
170
171
172
173
174
175
176
177
178
179
180
181
182
183
184
185
186
187
188
189
190
191
192
193
194
195
196
197
198
199
200
201
202
203
204
205
206
207
208
209
210
211
212
213
214
215
216
217
218
219
220
221
222
223
224
225
226
227
228
229
230
231
232
233
234
235
236
237
238
239
240
241
242
|
<!DOCTYPE html PUBLIC "-//W3C//DTD XHTML 1.0 Transitional//EN"
"http://www.w3.org/TR/xhtml1/DTD/xhtml1-transitional.dtd">
<html xmlns="http://www.w3.org/1999/xhtml">
<head>
<meta http-equiv="Content-Type" content="text/html; charset=utf-8" />
<title>Contributed Programs — S3QL 1.16 documentation</title>
<link rel="stylesheet" href="_static/sphinxdoc.css" type="text/css" />
<link rel="stylesheet" href="_static/pygments.css" type="text/css" />
<script type="text/javascript">
var DOCUMENTATION_OPTIONS = {
URL_ROOT: '',
VERSION: '1.16',
COLLAPSE_INDEX: false,
FILE_SUFFIX: '.html',
HAS_SOURCE: true
};
</script>
<script type="text/javascript" src="_static/jquery.js"></script>
<script type="text/javascript" src="_static/underscore.js"></script>
<script type="text/javascript" src="_static/doctools.js"></script>
<link rel="author" title="About these documents" href="about.html" />
<link rel="top" title="S3QL 1.16 documentation" href="index.html" />
<link rel="next" title="Tips & Tricks" href="tips.html" />
<link rel="prev" title="Storing Authentication Information" href="authinfo.html" />
</head>
<body>
<div class="related">
<h3>Navigation</h3>
<ul>
<li class="right" style="margin-right: 10px">
<a href="tips.html" title="Tips & Tricks"
accesskey="N">next</a></li>
<li class="right" >
<a href="authinfo.html" title="Storing Authentication Information"
accesskey="P">previous</a> |</li>
<li><a href="index.html">S3QL 1.16 documentation</a> »</li>
</ul>
</div>
<div class="sphinxsidebar">
<div class="sphinxsidebarwrapper">
<h3><a href="index.html">Table Of Contents</a></h3>
<ul class="current">
<li class="toctree-l1"><a class="reference internal" href="about.html">About S3QL</a></li>
<li class="toctree-l1"><a class="reference internal" href="installation.html">Installation</a></li>
<li class="toctree-l1"><a class="reference internal" href="backends.html">Storage Backends</a></li>
<li class="toctree-l1"><a class="reference internal" href="durability.html">Important Rules to Avoid Losing Data</a></li>
<li class="toctree-l1"><a class="reference internal" href="mkfs.html">File System Creation</a></li>
<li class="toctree-l1"><a class="reference internal" href="adm.html">Managing File Systems</a></li>
<li class="toctree-l1"><a class="reference internal" href="mount.html">Mounting</a></li>
<li class="toctree-l1"><a class="reference internal" href="special.html">Advanced S3QL Features</a></li>
<li class="toctree-l1"><a class="reference internal" href="umount.html">Unmounting</a></li>
<li class="toctree-l1"><a class="reference internal" href="fsck.html">Checking for Errors</a></li>
<li class="toctree-l1"><a class="reference internal" href="authinfo.html">Storing Authentication Information</a></li>
<li class="toctree-l1 current"><a class="current reference internal" href="">Contributed Programs</a><ul>
<li class="toctree-l2"><a class="reference internal" href="#benchmark-py">benchmark.py</a></li>
<li class="toctree-l2"><a class="reference internal" href="#s3-copy-py">s3_copy.py</a></li>
<li class="toctree-l2"><a class="reference internal" href="#pcp-py">pcp.py</a></li>
<li class="toctree-l2"><a class="reference internal" href="#s3-backup-sh">s3_backup.sh</a></li>
<li class="toctree-l2"><a class="reference internal" href="#expire-backups-py">expire_backups.py</a></li>
<li class="toctree-l2"><a class="reference internal" href="#s3ql-upstart-conf">s3ql_upstart.conf</a></li>
</ul>
</li>
<li class="toctree-l1"><a class="reference internal" href="tips.html">Tips & Tricks</a></li>
<li class="toctree-l1"><a class="reference internal" href="issues.html">Known Issues</a></li>
<li class="toctree-l1"><a class="reference internal" href="man/index.html">Manpages</a></li>
<li class="toctree-l1"><a class="reference internal" href="resources.html">Further Resources / Getting Help</a></li>
<li class="toctree-l1"><a class="reference internal" href="impl_details.html">Implementation Details</a></li>
</ul>
<div id="searchbox" style="display: none">
<h3>Quick search</h3>
<form class="search" action="search.html" method="get">
<input type="text" name="q" size="18" />
<input type="submit" value="Go" />
<input type="hidden" name="check_keywords" value="yes" />
<input type="hidden" name="area" value="default" />
</form>
<p class="searchtip" style="font-size: 90%">
Enter search terms.
</p>
</div>
<script type="text/javascript">$('#searchbox').show(0);</script>
</div>
</div>
<div class="document">
<div class="documentwrapper">
<div class="bodywrapper">
<div class="body">
<div class="section" id="contributed-programs">
<h1>Contributed Programs<a class="headerlink" href="#contributed-programs" title="Permalink to this headline">¶</a></h1>
<p>S3QL comes with a few contributed programs that are not part of the
core distribution (and are therefore not installed automatically by
default), but which may nevertheless be useful. These programs are in
the <tt class="docutils literal"><span class="pre">contrib</span></tt> directory of the source distribution or in
<tt class="docutils literal"><span class="pre">/usr/share/doc/s3ql/contrib</span></tt> if you installed S3QL from a package.</p>
<div class="section" id="benchmark-py">
<h2>benchmark.py<a class="headerlink" href="#benchmark-py" title="Permalink to this headline">¶</a></h2>
<p>This program measures S3QL write performance, uplink bandwidth and
compression speed to determine the limiting factor. It also gives
recommendation for compression algorithm and number of upload threads
to achieve maximum performance.</p>
</div>
<div class="section" id="s3-copy-py">
<h2>s3_copy.py<a class="headerlink" href="#s3-copy-py" title="Permalink to this headline">¶</a></h2>
<p>This program physically duplicates Amazon S3 bucket. It can be used to
migrate buckets to a different storage region or storage class
(standard or reduced redundancy).</p>
</div>
<div class="section" id="pcp-py">
<span id="pcp"></span><h2>pcp.py<a class="headerlink" href="#pcp-py" title="Permalink to this headline">¶</a></h2>
<p><tt class="docutils literal"><span class="pre">pcp.py</span></tt> is a wrapper program that starts several rsync processes to
copy directory trees in parallel. This is important because
transferring files in parallel significantly enhances performance when
copying data from an S3QL file system (see <a class="reference internal" href="tips.html#copy-performance"><em>Improving copy performance</em></a> for
details).</p>
<p>To recursively copy the directory <tt class="docutils literal"><span class="pre">/mnt/home-backup</span></tt> into
<tt class="docutils literal"><span class="pre">/home/joe</span></tt> using 8 parallel processes and preserving permissions,
you would execute</p>
<div class="highlight-commandline"><div class="highlight"><pre><span class="l">pcp.py -a --processes=8 /mnt/home-backup/ /home/joe</span>
</pre></div>
</div>
</div>
<div class="section" id="s3-backup-sh">
<h2>s3_backup.sh<a class="headerlink" href="#s3-backup-sh" title="Permalink to this headline">¶</a></h2>
<p>This is an example script that demonstrates how to set up a simple but
powerful backup solution using S3QL and <a class="reference external" href="http://samba.org/rsync">rsync</a>.</p>
<p>The <tt class="docutils literal"><span class="pre">s3_backup.sh</span></tt> script automates the following steps:</p>
<ol class="arabic simple">
<li>Mount the file system</li>
<li>Replicate the previous backup with <a class="reference internal" href="special.html#s3qlcp"><em>s3qlcp</em></a></li>
<li>Update the new copy with the data from the backup source using rsync</li>
<li>Make the new backup immutable with <a class="reference internal" href="special.html#s3qllock"><em>s3qllock</em></a></li>
<li>Delete old backups that are no longer needed</li>
<li>Unmount the file system</li>
</ol>
<p>The backups are stored in directories of the form
<tt class="docutils literal"><span class="pre">YYYY-MM-DD_HH:mm:SS</span></tt> and the <a class="reference internal" href="#expire-backups-py">expire_backups.py</a> command is used to
delete old backups.</p>
</div>
<div class="section" id="expire-backups-py">
<h2>expire_backups.py<a class="headerlink" href="#expire-backups-py" title="Permalink to this headline">¶</a></h2>
<p><strong class="program">expire_backups.py</strong> is a program to intelligently remove old
backups that are no longer needed.</p>
<p>To define what backups you want to keep for how long, you define a
number of <em>age ranges</em>. <strong class="program">expire_backups</strong> ensures that you
will have at least one backup in each age range at all times. It will
keep exactly as many backups as are required for that and delete any
backups that become redundant.</p>
<p>Age ranges are specified by giving a list of range boundaries in terms
of backup cycles. Every time you create a new backup, the existing
backups age by one cycle.</p>
<p>Example: when <strong class="program">expire_backups</strong> is called with the age range
definition <tt class="docutils literal"><span class="pre">1</span> <span class="pre">3</span> <span class="pre">7</span> <span class="pre">14</span> <span class="pre">31</span></tt>, it will guarantee that you always have the
following backups available:</p>
<ol class="arabic simple">
<li>A backup that is 0 to 1 cycles old (i.e, the most recent backup)</li>
<li>A backup that is 1 to 3 cycles old</li>
<li>A backup that is 3 to 7 cycles old</li>
<li>A backup that is 7 to 14 cycles old</li>
<li>A backup that is 14 to 31 cycles old</li>
</ol>
<div class="admonition note">
<p class="first admonition-title">Note</p>
<p class="last">If you do backups in fixed intervals, then one cycle will be
equivalent to the backup interval. The advantage of specifying the
age ranges in terms of backup cycles rather than days or weeks is
that it allows you to gracefully handle irregular backup intervals.
Imagine that for some reason you do not turn on your computer for
one month. Now all your backups are at least a month old, and if you
had specified the above backup strategy in terms of absolute ages,
they would all be deleted! Specifying age ranges in terms of backup
cycles avoids these sort of problems.</p>
</div>
<p><strong class="program">expire_backups</strong> usage is simple. It requires backups to have
names of the forms <tt class="docutils literal"><span class="pre">year-month-day_hour:minute:seconds</span></tt>
(<tt class="docutils literal"><span class="pre">YYYY-MM-DD_HH:mm:ss</span></tt>) and works on all backups in the current
directory. So for the above backup strategy, the correct invocation
would be:</p>
<div class="highlight-commandline"><div class="highlight"><pre><span class="l">expire_backups.py 1 3 7 14 31</span>
</pre></div>
</div>
<p>When storing your backups on an S3QL file system, you probably want to
specify the <tt class="docutils literal"><span class="pre">--use-s3qlrm</span></tt> option as well. This tells
<strong class="program">expire_backups</strong> to use the <a class="reference internal" href="special.html#s3qlrm"><em>s3qlrm</em></a> command to
delete directories.</p>
<p><strong class="program">expire_backups</strong> uses a “state file” to keep track which
backups are how many cycles old (since this cannot be inferred from
the dates contained in the directory names). The standard name for
this state file is <tt class="file docutils literal"><span class="pre">.expire_backups.dat</span></tt>. If this file gets
damaged or deleted, <strong class="program">expire_backups</strong> no longer knows the ages
of the backups and refuses to work. In this case you can use the
<tt class="cmdopt docutils literal"><span class="pre">--reconstruct-state</span></tt> option to try to reconstruct the state
from the backup dates. However, the accuracy of this reconstruction
depends strongly on how rigorous you have been with making backups (it
is only completely correct if the time between subsequent backups has
always been exactly the same), so it’s generally a good idea not to
tamper with the state file.</p>
<p>For a full list of available options, run <strong class="program">expire_backups.py
–help</strong>.</p>
</div>
<div class="section" id="s3ql-upstart-conf">
<h2>s3ql_upstart.conf<a class="headerlink" href="#s3ql-upstart-conf" title="Permalink to this headline">¶</a></h2>
<p><tt class="docutils literal"><span class="pre">s3ql_upstart.conf</span></tt> is an example upstart job definition file. It
defines a job that automatically mounts an S3QL file system on system
start, and properly unmounts it when the system is shut down.</p>
</div>
</div>
</div>
</div>
</div>
<div class="clearer"></div>
</div>
<div class="related">
<h3>Navigation</h3>
<ul>
<li class="right" style="margin-right: 10px">
<a href="tips.html" title="Tips & Tricks"
>next</a></li>
<li class="right" >
<a href="authinfo.html" title="Storing Authentication Information"
>previous</a> |</li>
<li><a href="index.html">S3QL 1.16 documentation</a> »</li>
</ul>
</div>
<div class="footer">
© Copyright 2008-2011, Nikolaus Rath.
Created using <a href="http://sphinx.pocoo.org/">Sphinx</a> 1.1.3.
</div>
</body>
</html>
|
