FAQ
From S3rsync
What is the difference between S3rsync to other S3 synchronization utilities?
As known, S3 storage protocol is "all or nothing", i.e. if you want to change a file you have to upload (S3 PUT) the whole file again.
S3rsync resolve this limitation and allows you to use rsync bandwidth efficient algorithm that enables you to upload only the partial file that has changed.
Other S3 synchronization utilities do not provide any bandwidth efficient algorithm and upload the whole new file, not just what was changed. So you are wasting unnecessary bandwidth and the backup process is significantly slower!
What's wrong abut mounting my S3 bucket as local disk drive and then use rsync?
Only when you connect to Rsync service on the remote side (our Ec2 servers at Amazon), you can fully benefit form Rsync power. Rsync to "local" drive is a bad idea since it uploads the whole file, as mentioned above. It means you are not using Rsync bandwidth efficient algorithm and again you're using lot of bandwidth and the backup process is much more slower.
What is the Rsync command syntax?
rsync -e "ssh -i ssh-key" -r /your/local/dir user@farm.s3rsync.com:remote-dir%%your-bucket-name%%your-s3-id%%your-s3-key
The following suffix should be added to the standard Rsync command: %%your-bucket-name%%your-s3-id%%your-s3-key
Upon your subscription, we will provide you with an unique user name and related public key.
There is no need to install any additional software beside standard Rsync and Ssh.
What software my PC/Server need in order to connect to your Rsync server
You only need to install standard Rsync and Ssh. Those software are free and available on all popular operating systems. If you can't install Ssh we can workaround this limitation.
How the data is store on my Amazon S3 buckets?
Your data is stored in a standart multi tar format.
How can I restore my backup?
Through our Rsync service. You also can download your backup and restore it directly form your S3 bucket. Amazon provides an https access. Any 3rd party tool that let you access your S3 data is good for this purpose. You need standard tar utility to open the archive.
How can restore single file/s or directory/s
You can do through Rsync . Use Rsync include/exclude flags. For example to restore all the directory which start with "site" under /home/customers/
If you backup command is:
rsync -a /home \ user@farm.s3rsync.com:home%%your-bucket-name%%your-s3-id%%your-s3-key \
The restore command is:
rsync -a \ --include "home/" \ --include "home/customers/" \ --include "home/customers/site*/" \ --exclude "*" \ user@farm.s3rsync.com:home%%your-bucket-name%%your-s3-id%%your-s3-key \ /some-local-dir/
For more information read Rsync man page.
What are the storage limitation?
Your bucket size is limited to 10 Gb. There is no limit to the number of buckets you can Rsync to.
We may remove this limit in the future.
How can I bypass this limit?
Your backup can be split to several Rsync command. Each command form different source directory to different target s3 bucket.
For example:
rsync /var to var-bucket rsync /home to home-bucket rsync /usr to usr-bucket
etc...
Other option is to use Rsync powerful include/exclude filter rules base on regular expression. For more information that a look at Rsync documentation of "filter rules"
For example: Let say your directory tree organize by dates:
C:\ 2006-03-10 2006-04-17 2007-03-12 2007-05-01 2007-11-13 2008-01-17 2008-03-18 ...
This Rsync commands (on MS-Windows) will backup every year to different bucket:
rsync.exe -r --include "2007-??-??/" --exclude="*/" /cygdrive/C/ user@farm.s3rsync.com:%%2007-bucket... rsync.exe -r --include "2008-??-??/" --exclude="*/" /cygdrive/C/ user@farm.s3rsync.com:%%2008-bucket...
This rsync commands will backup every month (from all the years) to different bucket
rsync.exe -r --include "????-03-??/" --exclude="*/" /cygdrive/C/ user@farm.s3rsync.com:%%March-bucket... rsync.exe -r --include "????-04-??/" --exclude="*/" /cygdrive/C/ user@farm.s3rsync.com:%%April-bucket...
This rsync commands will backup every quarter to different bucket
rsync.exe -r --include "2006-0[123]-??/" --exclude="*/" /cygdrive/C/ user@farm.s3rsync.com:%%2006Q1-bucket... rsync.exe -r --include "2006-0[456]-??/" --exclude="*/" /cygdrive/C/ user@farm.s3rsync.com:%%2006Q2-bucket... rsync.exe -r --include "2006-0[789]-??/" --exclude="*/" /cygdrive/C/ user@farm.s3rsync.com:%%2006Q3-bucket... rsync.exe -r --include "2006-1[012]-??/" --exclude="*/" /cygdrive/C/ user@farm.s3rsync.com:%%2006Q4-bucket... ...
Are there any other limitations?
Each user can run one Rsync session at a time.
Are there any overhead?
Yes, we copy your bucket content to and from our Ec2 server, it takes some time which depends on the bucket size. Normally, The speed is depend on the physical distance from our Ec2 server to your S3 bucket within Amazon data center. As more as our server use your bucket, the bucket will became closer and the access will be faster.
Can I bypass Amazon S3 5Gb file limit?
Yes, as long as the total size of your directory is in the bucket boundary (10Gb for now).
Can I Rsync to my Europe Amazon S3 buckets?
No, Rsync to buckets in Europe is not allowed. We may remove this limitation in the future.
Is it secure?
The whole session (yours and ours) is encrypted via SSL. Even if our servers will be hacked, there is nothing to steal. We don't store your AWS Secret Key at any time. We don't cache it or write it to any local file.
For added protection, consider encrypting your data before sending it to any 3rd party online storage solution, S3 or other. Here is a link to Rsync friendly encryption utility's: Web links#Rsync_encryption
Do I have to sign up for Amazon S3?
No. Rsync to S3rsync can work without it. You can store your data on our S3 bucket for additional fee. Without ever signing up directly with S3. Your data is stored securely and enjoys all the benefits of S3 by means of our S3 account.
What abut the bandwidth cost?
We will charge you for the standard Amazon bandwidth fee for any data uploaded to or downloaded from s3rsync servers.
You will pay ONLY for the bandwidth from your rsync client to our S3rsync servers. Base on rsync algorithm this bandwidth should be minimal.
You will NOT pay for any bandwidth from our S3rsync servers to S3 storage!
Bandwidth charges will be added to your s3rsync cost and are not included in your monthly S3 bill.
You can save bandwidth by using rsync flag "--compress" to compress file data during the transfer.
How do I sign up for Amazon S3?
Our service allows you to optionally use your own S3 account. To do that, you must sign up for Amazon Web Services and have activated the Amazon Simple Storage Service (S3) option.
- Go to Amazon Web Services home page.
- Sign up for a web services account.
- Go to My Web Services Account and Edit Payment Method. Enter your credit card there.
- Return to main Amazon Web Services home page.
- Go to Amazon Simple Storage Service (on left-hand side).
- Click Sign up for this service.
- Your account is now activated.