Introduction

For those who don’t know, Amazon S3 is a remote backup service. The advantages are:

  • the price
  • the bandwidth
  • the storage

I’ll let you look at the price list to see for yourself.

There isn’t really any provided software, but rather an API. And with its current popularity, some guys have worked on it and created some pretty nice tools :-)

I’ll describe here how I’ve set up something quite elegant and cost-effective.

Installation

First, let’s install the basic commands (just in case):

  apt-get install s3cmd

Then we’ll install what’s needed for Fuse, which will allow us to navigate on Amazon S3 as if it were a mounted filesystem. Here we install the development libraries for compilation:

  apt-get install libcurl4-openssl-dev libxml2-dev libfuse-dev make g++ fuse-utils

Note: If you already have the s3fs binary (so no compilation needed), just install these packages:

  apt-get install libcurl3 libfuse2 fuse-utils

Compilation

Now we need to download the sources for Fuse Over Amazon, then compile them:

  wget http://s3fs.googlecode.com/files/s3fs-1.40.tar.gz
tar -xzf s3fs-1.40.tar.gz
cd s3fs-1.40
./configure --bindir /usr/bin
make
make install

Configuration

We’ll first use s3cmd to create our bucket (personal folder for amazon s3), then use it with Fuse.

s3cmd

Let’s first configure our account by running the following command:

  s3cmd --configure

Fill in all the fields using the information from your account. The most important are the access key and the secret key. Once done, we’ll create our bucket:

  s3cmd mb monbucket_with_a_unique_name

It’s important to create a bucket with a specific name because everyone is basically on the same filesystem. So if the name already exists, you won’t be able to create this bucket.

FuseOverAmazon

Let’s create the /etc/passwd-s3fs file and fill in the info as follows (/etc/passwd-s3fs):

  access_key:secret_key

Enter your access key, followed by ‘:’, then your secret key.

Then we’ll apply the appropriate permissions:

  chmod 600 /etc/passwd-s3fs

Finally, we can mount all this in /mnt for example (adapt according to your needs):

  s3fs monbucket_with_a_unique_name /mnt -ouse_cache=/tmp

And there you go :-). You can do ls, mkdir, cp, rm etc… in /mnt and it will hit your Amazon S3 backup :-).

Fstab

And if we want to put all this in fstab? This is of course optional, but very practical:

  s3fs#monbucket_with_a_unique_name /mnt   fuse    ouse_cache=/tmp,noatime,allow_other 0 0

Here’s the line to add.

AutoFS

The best setup is with autofs! I invite you to read this documentation first to not be too confused, then add this line in a file auto.as3 for example:

  amazons3        -fstype=fuse,ouse_cache=/tmp,noatime,allow_other:s3fs\#monbucket_with_a_unique_name

Then modify the auto.master file:

  /mnt /etc/auto.as3 --timeout=60

Restart the autofs service for this to work.

Backups with specific software

With some software, you need to be clever to bypass the 5GB file size limitation imposed by Amazon S3 or multiple files in too large a number which will cost us dearly.

Sbackup

For Sbackup, it’s quite simple, you need to check if the file size is larger than 5 GB and split the files.tgz file.

BackupPc

For BackupPc, it is recommended to compress existing data before uploading it because there are often too many small files.

Resources

Edit this page

Last updated 15 Feb 2011, 08:19 CET. history