FAQ Content


What is the difference between S3rsync and other S3 synchronization utilities?

As known, S3 storage protocol is "all or nothing", i.e. if you want to make even a minor change to your file you'll have to upload (S3 PUT) the entire file again.

S3rsync resolves this limitation, Rsync bandwidth efficient algorithm enables you to upload only the partial file that has been changed.

Other S3 synchronization utilities do not provide any bandwidth efficient algorithm thus the entire file is uploaded each time, not just the parts that were changed. So you are wasting unnecessary bandwidth and the backup process is significantly slower.

What's wrong about mounting my S3 bucket as a local disk drive and then using rsync?

You can fully benefit form the Rsync power only when you connect to Rsync service on the remote side (our Ec2 servers at Amazon). Rsync to a local drive is a bad idea since it uploads the entire file, as mentioned above. It means you are not using Rsync bandwidth efficient algorithm and again, you're wasting lot of bandwidth and time since the backup process takes much longer.

What is the Rsync command syntax?

 
rsync -e "ssh -i ssh-key" -r /your/local/dir
user@farm.s3rsync.com:remote-dir%%your-bucket-name%%your-s3-id%%your-s3-key

The following suffix should be added to the standard Rsync command: %%your-bucket-name%%your-s3-id%%your-s3-key

Upon your subscription, we will provide you with an unique user name and related key.

There is no need to install any additional software beside standard Rsync and Ssh.

Which software is needed to connect?

You only need to install standard Rsync and Ssh which is free and available on all popular operating systems. In case you are unable to install Ssh, we will suggest a workaround to overcome this limitation.

How the data is stored on my Amazon S3 buckets?

Your data is stored in a standard multi tar format.

How can I restore my backup?

Through our Rsync service. It is also possible to download your backup and restore it directly form your S3 bucket. Amazon provides an https access. You can also use any 3rd party tool that enables access to your S3 data. You need a standard tar utility to open the archive.

How can I restore single file/s or directory/s?

You can do it through Rsync by using Rsync include/exclude flags.

For example, to restore all the directories which start with "site" under /home/customers/ directory and

if your backup command is: 

rsync -a /home \
user@farm.s3rsync.com:home%%your-bucket-name%%your-s3-id%%your-s3-keyThe

the restore command would be: 

rsync -a  \
--include "home/" \
--include "home/customers/" \
--include "home/customers/site*/" \
--exclude "*" \
user@farm.s3rsync.com:home%%your-bucket-name%%your-s3-id%%your-s3-key \
/some-local-dir/

For more information read Rsync man page.

What is the storage limitation?

Your bucket size is limited to 10 Gb. There is no limit to the number of buckets you can Rsync to.

If you would like to increase this limit, please Contact Us with a request that explains your needs.

Other alternative to bypass this limit is described in the next question.

How can I bypass this limit?

Your backup can be split to several Rsync commands. Each command related to a different source directory can be managed in the different target s3 bucket.

For example:

rsync /var to var-bucket
rsync /home to home-bucket
rsync /usr to usr-bucket

etc...

Other option is to take advantage of the Rsync powerful include/exclude filter functionality using regular expressions. For more information have a look at Rsync documentation of "filter rules"

For example, lets say your directory tree is organized by dates: 

C:\
2006-03-10
2006-04-17
2007-03-12
2007-05-01
2007-11-13
2008-01-17
2008-03-18 ...

These Rsync commands (on MS-Windows) will backup every year to the different bucket: 

rsync.exe -r --include "2007-??-??/" --exclude="*/" /cygdrive/C/ user@farm.s3rsync.com:%%2007-bucket...
rsync.exe -r --include "2008-??-??/" --exclude="*/" /cygdrive/C/ user@farm.s3rsync.com:%%2008-bucket...

These Rsync commands will backup every month (during the year) to the different bucket: 

rsync.exe -r --include "????-03-??/" --exclude="*/" /cygdrive/C/ user@farm.s3rsync.com:%%March-bucket...
rsync.exe -r --include "????-04-??/" --exclude="*/" /cygdrive/C/ user@farm.s3rsync.com:%%April-bucket...

These Rsync commands will backup every quarter to the different bucket: 

rsync.exe -r --include "2006-0[123]-??/" --exclude="*/" /cygdrive/C/ user@farm.s3rsync.com:%%2006Q1-bucket...
rsync.exe -r --include "2006-0[456]-??/" --exclude="*/" /cygdrive/C/ user@farm.s3rsync.com:%%2006Q2-bucket...
rsync.exe -r --include "2006-0[789]-??/" --exclude="*/" /cygdrive/C/ user@farm.s3rsync.com:%%2006Q3-bucket...
rsync.exe -r --include "2006-1[012]-??/" --exclude="*/" /cygdrive/C/ user@farm.s3rsync.com:%%2006Q4-bucket...
...

What type of bucket are supported?

Only buckets located in US are supported.

Are there any other limitations?

Each user can run one Rsync session at a time.

Is there any overhead?

We copy your bucket content to and from the Ec2 server. It takes some time which depends on the bucket size. Normally, the speed is dependent on the physical distance from our Ec2 server to your S3 bucket within Amazon data center. As much as our server uses your bucket, the bucket will become closer and the access will be faster.

Can I bypass Amazon S3 5Gb file limit?

Yes, as long as the total size of your directory is in the bucket boundary (10Gb for now).

Can I Rsync to my Europe Amazon S3 buckets?

No, Rsync to buckets in Europe is not allowed. We may remove this limitation in the future.

Is it secure?

The whole session (yours and ours) is encrypted via SSL. Even if our servers will be hacked, there is nothing to steal. We don't store your AWS Secret Key at any time. We don't cache it or write it to any local file.

For added protection, consider encrypting your data before sending it to any 3rd party online storage solution, S3 or other. Here is a link to Rsync friendly encryption utility's.

What if your service is down?

Your data is stored in the S3 buckets in native GNU tar format. Since there are no proprietary formats, in case of emergency, data could be recovered with native utilities, regardless of whether or not S3rsync.com is available.

Do I have to sign up for Amazon S3?

No. Rsync to S3rsync can work without it. You can store your data on our S3 bucket for additional fee without ever signing up directly with S3. Your data is stored securely and enjoys all the benefits of S3 by means of our S3 account.

What about the bandwidth costs?

We charge the standard Amazon bandwidth fee for any data uploaded to or downloaded from s3rsync servers.

You will pay ONLY for the bandwidth from your rsync client to our S3rsync servers. With use of our bandwidth efficient algorithm these costs should be minimal.

You will NOT pay for any bandwidth from our S3rsync servers to S3 storage! Bandwidth charges will be added to your s3rsync costs and are not included in your monthly S3 bill.

You also save bandwidth costs by using rsync flag "--compress" to compress file data during the transfer.

How do I sign up for Amazon S3?

Our service allows you to use your own S3 account. To do that, you must sign up for Amazon Web Services and activate the Amazon Simple Storage Service (S3) option.

  1. Go to Amazon Web Services home page.
  2. Sign up for a Web Services account.
  3. Go to My Web Services Account and Edit Payment Method. Enter your credit card there.
  4. Return to main Amazon Web Services home page.
  5. Go to Amazon Simple Storage Service (on left-hand side).
  6. Click Sign up for this service.
  7. Your account will be activated.

s3rsync ©    Privacy Policy