1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
35
36
37
38
39
40
41
42
43
44
45
46
47
48
49
50
51
52
53
54
55
56
57
58
59
60
61
62
63
64
65
66
67
68
69
70
71
72
73
74
75
76
77
78
79
80
81
82
83
84
85
86
87
88
89
90
91
92
93
94
95
96
97
98
99
100
101
102
103
104
105
106
107
108
109
110
111
112
113
114
115
116
117
118
119
120
121
122
123
124
125
126
127
128
129
130
131
132
133
134
135
136
137
138
139
140
141
142
143
144
145
146
147
148
149
150
151
152
153
154
155
156
157
158
159
160
161
162
163
164
165
166
167
168
169
170
171
172
173
174
175
176
177
178
179
180
181
182
183
184
185
186
187
188
189
190
191
192
193
194
195
196
197
198
199
200
201
202
203
204
205
206
207
208
209
210
211
212
213
214
215
216
217
218
219
220
221
222
223
224
225
226
227
228
229
230
231
232
233
234
235
236
237
238
239
240
241
242
243
244
245
246
247
248
249
250
251
252
253
254
255
256
257
258
259
260
261
262
263
264
265
266
267
268
269
270
271
272
273
274
275
276
277
278
279
280
281
282
283
|
<!DOCTYPE html PUBLIC "-//W3C//DTD XHTML 1.0 Transitional//EN"
"http://www.w3.org/TR/xhtml1/DTD/xhtml1-transitional.dtd">
<html xmlns="http://www.w3.org/1999/xhtml">
<head>
<meta http-equiv="Content-Type" content="text/html; charset=utf-8" />
<title>Storage Backends — S3QL 1.14 documentation</title>
<link rel="stylesheet" href="_static/sphinxdoc.css" type="text/css" />
<link rel="stylesheet" href="_static/pygments.css" type="text/css" />
<script type="text/javascript">
var DOCUMENTATION_OPTIONS = {
URL_ROOT: '',
VERSION: '1.14',
COLLAPSE_INDEX: false,
FILE_SUFFIX: '.html',
HAS_SOURCE: true
};
</script>
<script type="text/javascript" src="_static/jquery.js"></script>
<script type="text/javascript" src="_static/underscore.js"></script>
<script type="text/javascript" src="_static/doctools.js"></script>
<link rel="author" title="About these documents" href="about.html" />
<link rel="top" title="S3QL 1.14 documentation" href="index.html" />
<link rel="next" title="Important Rules to Avoid Losing Data" href="durability.html" />
<link rel="prev" title="Installation" href="installation.html" />
</head>
<body>
<div class="related">
<h3>Navigation</h3>
<ul>
<li class="right" style="margin-right: 10px">
<a href="durability.html" title="Important Rules to Avoid Losing Data"
accesskey="N">next</a></li>
<li class="right" >
<a href="installation.html" title="Installation"
accesskey="P">previous</a> |</li>
<li><a href="index.html">S3QL 1.14 documentation</a> »</li>
</ul>
</div>
<div class="sphinxsidebar">
<div class="sphinxsidebarwrapper">
<h3><a href="index.html">Table Of Contents</a></h3>
<ul class="current">
<li class="toctree-l1"><a class="reference internal" href="about.html">About S3QL</a></li>
<li class="toctree-l1"><a class="reference internal" href="installation.html">Installation</a></li>
<li class="toctree-l1 current"><a class="current reference internal" href="">Storage Backends</a><ul>
<li class="toctree-l2"><a class="reference internal" href="#google-storage">Google Storage</a></li>
<li class="toctree-l2"><a class="reference internal" href="#amazon-s3">Amazon S3</a></li>
<li class="toctree-l2"><a class="reference internal" href="#openstack-swift">OpenStack/Swift</a></li>
<li class="toctree-l2"><a class="reference internal" href="#rackspace-cloudfiles">RackSpace CloudFiles</a></li>
<li class="toctree-l2"><a class="reference internal" href="#s3-compatible">S3 compatible</a></li>
<li class="toctree-l2"><a class="reference internal" href="#local">Local</a></li>
</ul>
</li>
<li class="toctree-l1"><a class="reference internal" href="durability.html">Important Rules to Avoid Losing Data</a></li>
<li class="toctree-l1"><a class="reference internal" href="mkfs.html">File System Creation</a></li>
<li class="toctree-l1"><a class="reference internal" href="adm.html">Managing File Systems</a></li>
<li class="toctree-l1"><a class="reference internal" href="mount.html">Mounting</a></li>
<li class="toctree-l1"><a class="reference internal" href="special.html">Advanced S3QL Features</a></li>
<li class="toctree-l1"><a class="reference internal" href="umount.html">Unmounting</a></li>
<li class="toctree-l1"><a class="reference internal" href="fsck.html">Checking for Errors</a></li>
<li class="toctree-l1"><a class="reference internal" href="authinfo.html">Storing Authentication Information</a></li>
<li class="toctree-l1"><a class="reference internal" href="contrib.html">Contributed Programs</a></li>
<li class="toctree-l1"><a class="reference internal" href="tips.html">Tips & Tricks</a></li>
<li class="toctree-l1"><a class="reference internal" href="issues.html">Known Issues</a></li>
<li class="toctree-l1"><a class="reference internal" href="man/index.html">Manpages</a></li>
<li class="toctree-l1"><a class="reference internal" href="resources.html">Further Resources / Getting Help</a></li>
<li class="toctree-l1"><a class="reference internal" href="impl_details.html">Implementation Details</a></li>
</ul>
<div id="searchbox" style="display: none">
<h3>Quick search</h3>
<form class="search" action="search.html" method="get">
<input type="text" name="q" size="18" />
<input type="submit" value="Go" />
<input type="hidden" name="check_keywords" value="yes" />
<input type="hidden" name="area" value="default" />
</form>
<p class="searchtip" style="font-size: 90%">
Enter search terms.
</p>
</div>
<script type="text/javascript">$('#searchbox').show(0);</script>
</div>
</div>
<div class="document">
<div class="documentwrapper">
<div class="bodywrapper">
<div class="body">
<div class="section" id="storage-backends">
<span id="id1"></span><h1>Storage Backends<a class="headerlink" href="#storage-backends" title="Permalink to this headline">¶</a></h1>
<p>S3QL supports different <em>backends</em> to store data at different service
providers and using different protocols. A <em>storage url</em> specifies a
backend together with some backend-specific information and uniquely
identifies an S3QL file system. The form of the storage url depends on
the backend and is described for every backend below.</p>
<p>All storage backends respect the <tt class="docutils literal"><span class="pre">http_proxy</span></tt> and <tt class="docutils literal"><span class="pre">https_proxy</span></tt>
environment variables.</p>
<div class="section" id="google-storage">
<h2>Google Storage<a class="headerlink" href="#google-storage" title="Permalink to this headline">¶</a></h2>
<p><a class="reference external" href="http://code.google.com/apis/storage/">Google Storage</a> is an online
storage service offered by Google. To use the Google Storage backend,
you need to have (or sign up for) a Google account, and then <a class="reference external" href="http://code.google.com/apis/storage/docs/signup.html">activate
Google Storage</a>
for your account. The account is free, you will pay only for the
amount of storage and traffic that you actually use. Once you have
created the account, make sure to <a class="reference external" href="http://code.google.com/apis/storage/docs/reference/v1/apiversion1.html#enabling">activate legacy access</a>.</p>
<p>To create a Google Storage bucket, you can use e.g. the <a class="reference external" href="https://sandbox.google.com/storage/">Google
Storage Manager</a>. The storage
URL for accessing the bucket in S3QL is then</p>
<div class="highlight-commandline"><div class="highlight"><pre><span class="l">gs://</span><span class="nv"><bucketname></span><span class="l">/</span><span class="nv"><prefix></span><span class="l"></span>
</pre></div>
</div>
<p>Here <em>bucketname</em> is the name of the bucket, and <em>prefix</em> can be
an arbitrary prefix that will be prepended to all object names used by
S3QL. This allows you to store several S3QL file systems in the same
Google Storage bucket.</p>
<p>Note that the backend login and password for accessing your Google
Storage bucket are not your Google account name and password, but the
<em>Google Storage developer access key</em> and <em>Google Storage developer
secret</em> that you can manage with the <a class="reference external" href="https://code.google.com/apis/console/#:storage:legacy">Google Storage key management
tool</a>.</p>
</div>
<div class="section" id="amazon-s3">
<h2>Amazon S3<a class="headerlink" href="#amazon-s3" title="Permalink to this headline">¶</a></h2>
<p><a class="reference external" href="http://aws.amazon.com/s3">Amazon S3</a> is the online storage service
offered by <a class="reference external" href="http://aws.amazon.com/">Amazon Web Services (AWS)</a>. To
use the S3 backend, you first need to sign up for an AWS account. The
account is free, you will pay only for the amount of storage and
traffic that you actually use. After that, you need to create a bucket
that will hold the S3QL file system, e.g. using the <a class="reference external" href="https://console.aws.amazon.com/s3/home">AWS Management
Console</a>. For best
performance, it is recommend to create the bucket in the
geographically closest storage region, but not the US Standard region
(see below).</p>
<p>The storage URL for accessing S3 buckets in S3QL has the form</p>
<div class="highlight-commandline"><div class="highlight"><pre><span class="l">s3://</span><span class="nv"><bucketname></span><span class="l">/</span><span class="nv"><prefix></span><span class="l"></span>
</pre></div>
</div>
<p>Here <em>bucketname</em> is the name of the bucket, and <em>prefix</em> can be an
arbitrary prefix that will be prepended to all object names used by
S3QL. This allows you to store several S3QL file systems in the same
S3 bucket.</p>
<p>Note that the backend login and password for accessing S3 are not the
user id and password that you use to log into the Amazon Webpage, but
the <em>AWS access key id</em> and <em>AWS secret access key</em> shown under <a class="reference external" href="https://aws-portal.amazon.com/gp/aws/developer/account/index.html?ie=UTF8&action=access-key">My
Account/Access Identifiers</a>.</p>
<div class="section" id="reduced-redundancy-storage-rrs">
<h3>Reduced Redundancy Storage (RRS)<a class="headerlink" href="#reduced-redundancy-storage-rrs" title="Permalink to this headline">¶</a></h3>
<p>S3QL does not allow the use of <a class="reference external" href="http://aws.amazon.com/s3/#protecting">reduced redundancy storage</a>. The reason for that is a
combination of three factors:</p>
<ul class="simple">
<li>RRS has a relatively low reliability, on average you lose one
out of every ten-thousand objects a year. So you can expect to
occasionally lose some data.</li>
<li>When <tt class="docutils literal"><span class="pre">fsck.s3ql</span></tt> asks S3 for a list of the stored objects, this list
includes even those objects that have been lost. Therefore
<tt class="docutils literal"><span class="pre">fsck.s3ql</span></tt> <em>can not detect lost objects</em> and lost data will only
become apparent when you try to actually read from a file whose data
has been lost. This is a (very unfortunate) peculiarity of Amazon
S3.</li>
<li>Due to the data de-duplication feature of S3QL, unnoticed lost
objects may cause subsequent data loss later in time (see
<a class="reference internal" href="durability.html#backend-reliability"><em>Data Durability</em></a> for details).</li>
</ul>
</div>
</div>
<div class="section" id="openstack-swift">
<h2>OpenStack/Swift<a class="headerlink" href="#openstack-swift" title="Permalink to this headline">¶</a></h2>
<p><a class="reference external" href="http://www.openstack.org/">OpenStack</a> is an open-source cloud server application suite. <a class="reference external" href="http://openstack.org/projects/storage/">Swift</a> is
the cloud storage module of OpenStack. Swift/OpenStack storage is
offered by many different companies.</p>
<p>The storage URL for the OpenStack backend has the form</p>
<div class="highlight-commandline"><div class="highlight"><pre><span class="l">swift://</span><span class="nv"><hostname></span><span class="ge">[:<port>]</span><span class="l">/</span><span class="nv"><container></span><span class="ge">[/<prefix>]</span><span class="l"></span>
</pre></div>
</div>
<p>Note that the storage container must already exist. Most OpenStack
providers offer a web frontend that you can use to create storage
containers. <em>prefix</em> can be an arbitrary prefix that will be prepended
to all object names used by S3QL. This allows you to store several
S3QL file systems in the same container.</p>
<p>The OpenStack backend always uses HTTPS connections. Note, however,
that at this point S3QL does not verify the server certificate (cf.
<a class="reference external" href="http://code.google.com/p/s3ql/issues/detail?id=267">issue 267</a>).</p>
</div>
<div class="section" id="rackspace-cloudfiles">
<h2>RackSpace CloudFiles<a class="headerlink" href="#rackspace-cloudfiles" title="Permalink to this headline">¶</a></h2>
<p><a class="reference external" href="http://www.rackspace.com/">RackSpace</a> CloudFiles uses OpenStack internally, so you can use the
OpenStack/Swift backend (see above). The hostname for CloudFiles
containers is <tt class="docutils literal"><span class="pre">auth.api.rackspacecloud.com</span></tt>. Use your normal
RackSpace user name for the backend login, and your RackSpace API key
as the backend passphrase. You can create a storage container for S3QL
using the <a class="reference external" href="https://manage.rackspacecloud.com/">Control Panel</a> (go to
<em>Cloud Files</em> under <em>Hosting</em>).</p>
<div class="admonition warning">
<p class="first admonition-title">Warning</p>
<p class="last">As of January 2012, RackSpace does not give any information about
data consistency or data durability on their web page. However,
RackSpace support agents (especially in the live chat) often claim
very high guarantees. Any such statement is wrong. As of 01/2012,
RackSpace CloudFiles does <em>not</em> give <em>any</em> durability or
consistency guarantees (see <a class="reference internal" href="durability.html#durability"><em>Important Rules to Avoid Losing Data</em></a> for why this is
important). Why this fact is only acknowledged RackSpace’s
technical engineers, and/or not communicated to their sales agents
is not known.</p>
</div>
<p>You should note that opinions about RackSpace differ widely among S3QL
users and developers. On one hand, people praise RackSpace for their
backing of the (open source) OpenStack project. On the other hand,
their heavily advertised “fanatical support” is in practice often not
only <a class="reference external" href="http://code.google.com/p/s3ql/issues/detail?id=243#c5">less than helpful</a>, but their
support agents also seem to be <a class="reference external" href="http://code.google.com/p/s3ql/issues/detail?id=243#c11">downright incompetent</a>. However,
there are reports that the support quality increases dramatically once
you are a customer and use the “Live Chat” link when you are logged
into the control panel.</p>
</div>
<div class="section" id="s3-compatible">
<h2>S3 compatible<a class="headerlink" href="#s3-compatible" title="Permalink to this headline">¶</a></h2>
<p>The S3 compatible backend allows S3QL to access any storage service
that uses the same protocol as Amazon S3. The storage URL has the form</p>
<div class="highlight-commandline"><div class="highlight"><pre><span class="l">s3c://</span><span class="nv"><hostname></span><span class="l">:</span><span class="nv"><port></span><span class="l">/</span><span class="nv"><bucketname></span><span class="l">/</span><span class="nv"><prefix></span><span class="l"></span>
</pre></div>
</div>
<p>Here <em>bucketname</em> is the name of an (existing) bucket, and <em>prefix</em>
can be an arbitrary prefix that will be prepended to all object names
used by S3QL. This allows you to store several S3QL file systems in
the same bucket.</p>
</div>
<div class="section" id="local">
<h2>Local<a class="headerlink" href="#local" title="Permalink to this headline">¶</a></h2>
<p>S3QL is also able to store its data on the local file system. This can
be used to backup data on external media, or to access external
services that S3QL can not talk to directly (e.g., it is possible to
store data over SSH by first mounting the remote system using <a href="#id4"><span class="problematic" id="id5">`sshfs`_</span></a>
and then using the local backend to store the data in the sshfs
mountpoint).</p>
<p>The storage URL for local storage is</p>
<div class="highlight-commandline"><div class="highlight"><pre><span class="l">local://</span><span class="nv"><path></span><span class="l"></span>
</pre></div>
</div>
<p>Note that you have to write three consecutive slashes to specify an
absolute path, e.g. <tt class="docutils literal"><span class="pre">local:///var/archive</span></tt>. Also, relative paths will
automatically be converted to absolute paths before the authentication
file (see <a class="reference internal" href="authinfo.html#authinfo"><em>Storing Authentication Information</em></a>) is read, i.e. if you are in the
<tt class="docutils literal"><span class="pre">/home/john</span></tt> directory and try to mount <tt class="docutils literal"><span class="pre">local://s3ql</span></tt>, the
corresponding section in the authentication file must match the
storage url <tt class="docutils literal"><span class="pre">local:///home/john/s3ql</span></tt>.</p>
</div>
</div>
</div>
</div>
</div>
<div class="clearer"></div>
</div>
<div class="related">
<h3>Navigation</h3>
<ul>
<li class="right" style="margin-right: 10px">
<a href="durability.html" title="Important Rules to Avoid Losing Data"
>next</a></li>
<li class="right" >
<a href="installation.html" title="Installation"
>previous</a> |</li>
<li><a href="index.html">S3QL 1.14 documentation</a> »</li>
</ul>
</div>
<div class="footer">
© Copyright 2008-2011, Nikolaus Rath.
Created using <a href="http://sphinx.pocoo.org/">Sphinx</a> 1.1.3.
</div>
</body>
</html>
|
