Amazon S3 is a reasonably priced data storage service. Ideal for off-site backups, archiving and other data storage needs. It is generally more reliable than your regular web hosting for storing your files and images. Check out About Amazon S3 section to find out more.
S3cmd is a command line tool for uploading, retrieving and managing data in Amazon S3 and other Cloud Storage Service Providers that use the S3 protocol such as DreamHost DreamObjects. It is best suited for power users who don't fear command line. It is also ideal for scripts, automated backups triggered from cron, etc.
S3cmd is an open source project available under GNU Public License v2 (GPLv2) and is free for both commercial and private use. You will only have to pay Amazon for using their storage. None of these money go to S3cmd developers.
Unless, of course, you decide to donate some money to us ;-)
S3cmd source code and packages for major linux distributions can be downloaded on our Download page
Simple S3cmd HowTo
The following example demonstrates just the the basic features. However there is much more s3cmd can do. The most popular feature seems to be rsync-like sync command. Check out our s3cmd sync HowTo for more details. Also don't miss out s3cmd encryption HowTo if you're after privacy.
- Register for Amazon AWS / S3
Go to Amazon S3 homepage, click on the "Sign up for web service" button in the right column and work through the registration. You will have to supply your Credit Card details in order to allow Amazon charge you for S3 usage. At the end you should posses your Access and Secret Keys.
You will be asked for the two keys - copy and paste them from your confirmation email or from your Amazon account page. Be careful when copying them! They are case sensitive and must be entered accurately or you'll keep getting errors about invalid signatures or similar.
You can optionally enter a GPG encryption key that will be used for encrypting your files before sending them to Amazon. Using GPG encryption will protect your data against reading by Amazon staff or anyone who may get access to your them while they're stored at Amazon S3.
Another option to decide about is whether to use HTTPS or HTTP transport for communication with Amazon. HTTPS is an encrypted version of HTTP, protecting your data against eavesdroppers while they're in transit to and from Amazon S3.
Please note: - both the above mentioned forms of encryption are independent on each other and serve a different purpose. While GPG encryption is protects your data against reading while they are stored in Amazon S3, HTTPS protects them only while they're being uploaded to Amazon S3 (or downloaded from). There are pros and cons for each and you are free to select either, or, both or none. Refer to s3cmd encryption HowTo for more details.
s3cmd lsto list all your buckets.
As you have just started using S3 there are no buckets owned by you as of now. So the output will be empty.
- Make a bucket with
s3cmd mb s3://my-new-bucket-name
As mentioned above bucket names must be unique amongst _all_ users of S3. That means the simple names like "test" or "asdf" are already taken and you must make up something more original. I often prefix my bucket names with my e-mail domain name (logix.cz) leading to a bucket name, for instance, 'logix.cz-test':
~$ s3cmd mb s3://logix.cz-test Bucket 'logix.cz-test' created
- List your buckets again with
Now you should see your freshly created bucket
~$ s3cmd ls 2007-01-19 01:41 s3://logix.cz-test
- List the contents of the bucket
~$ s3cmd ls s3://logix.cz-test Bucket 'logix.cz-test': ~$
It's empty, indeed.
- Upload a file into the bucket
~$ s3cmd put addressbook.xml s3://logix.cz-test/addrbook.xml File 'addressbook.xml' stored as s3://logix.cz-test/addrbook.xml (123456 bytes)
Note about ACL (Access control lists) — a file uploaded to Amazon S3 bucket can either be private, that is readable only by you, possessor of the access and secret keys, or public, readable by anyone. Each file uploaded as public is not only accessible using s3cmd but also has a HTTP address, URL, that can be used just like any other URL and accessed for instance by web browsers.
~$ s3cmd put --acl-public --guess-mime-type storage.jpg s3://logix.cz-test/storage.jpg File 'storage.jpg' stored as s3://logix.cz-test/storage.jpg (33045 bytes) Public URL of the object is: http://logix.cz-test.s3.amazonaws.com/storage.jpg
Now anyone can display the storage.jpg file in their browser. Cool, eh?
- Now we can list the bucket contents again
~$ s3cmd ls s3://logix.cz-test Bucket 'logix.cz-test': 2008-01-19 01:46 120k s3://logix.cz-test/addrbook.xml 2008-11-14 01:46 32k s3://logix.cz-test/storage.jpg
- Retrieve the file back and verify that its hasn't been corrupted
~$ s3cmd get s3://logix.cz-test/addrbook.xml addressbook-2.xml Object s3://logix.cz-test/addrbook.xml saved as 'addressbook-2.xml' (123456 bytes) ~$ md5sum addressbook.xml addressbook-2.xml 39bcb6992e461b269b95b3bda303addf addressbook.xml 39bcb6992e461b269b95b3bda303addf addressbook-2.xml
Checksums of the original file matches the one of the retrieved one. Looks like it worked :-)
- Clean up: delete the object and remove the bucket
~$ s3cmd rb s3://logix.cz-test ERROR: S3 error: 409 (Conflict): BucketNotEmpty
Ouch, we can only remove empty buckets!
~$ s3cmd del s3://logix.cz-test/addrbook.xml s3://logix.cz-test/storage.jpg Object s3://logix.cz-test/addrbook.xml deleted Object s3://logix.cz-test/storage.jpg deleted ~$ s3cmd rb s3://logix.cz-test Bucket 'logix.cz-test' removed
Ok, ok, I'll put that storage.jpg back up for you ;-)
- Other features
Check out our advanced tutorials:
- The basic usage is as simple as described in the previous section.
- You can increase the level of verbosity with -v option and if you're really keen to know what the program does under its bonet run it with -d to see all 'debugging' output.
- After configuring it with
--configureall available options are spitted into your
~/.s3cfgfile. It's a text file ready to be modified in your favourite text editor.
- Multiple local files may be specified for
s3cmd putoperation. In that case the S3 URI should only include the bucket name, not the object part:
~$ s3cmd put file-* s3://logix.cz-test/ File 'file-one.txt' stored as s3://logix.cz-test/file-one.txt (4 bytes) File 'file-two.txt' stored as s3://logix.cz-test/file-two.txt (4 bytes)
- Alternatively if you specify the object part as well it
will be treated as a prefix and all filenames given on the
command line will be appended to the prefix making up
the object name. However
--forceoption is required in this case:
~$ s3cmd put --force file-* s3://logix.cz-test/prefixed: File 'file-one.txt' stored as s3://logix.cz-test/prefixed:file-one.txt (4 bytes) File 'file-two.txt' stored as s3://logix.cz-test/prefixed:file-two.txt (4 bytes)