Using AWS CLI in Linux
There are three methods to upload and download data to Amazon Web Services. You can use the command line (CLI), AWS SDK, or the S3 REST API. In this article, we will explore the command Line interface, and the most common commands to manage an S3 bucket.
The maximum size of a file that you can upload by using the Amazon S3 console is 160 GB. The maximum bucket size is 5TB. You can not use s3api on files uploads larger than 5GB. Command line tools can achieve upload speeds greater than 7 MB’s. But, you can go even faster if you turn on acceleration. It is not recommended because an additional cost will be incurred.
Common Switches
- –dryrun = test what files would be uploaded, prior to running command.
- — summarize = include a total at the bottom of the output.
- — human-readable = show files sizes in Gb and not Bytes.
- –output text = format the output on separate lines
- –content-type=text/plain = Tell aws the upload data is text data (not video or other).
- –recursive = show full file path
- –exclude – leave out certain files.
- –include = include certain files.
- –delete = this flag is needed to remove any files.
List Contents of a Bucket
aws s3 ls s3://bucket1/directory1/directory2/ --summarize --human-readable
Copy a Single File
If the file is large, the cp command will automatically handle a multi-part upload dynamically. If the full path is not present, it will create it automatically in the s3 bucket.
aws s3 cp ~/test.txt s3://bucket1/dir1/dir2/ --human-readable Copy Multiple Files in a Local Directory
There are two commands that can be used to copy multiple files. Use sync or cp with the –recursive switch.
aws s3 sync ./test/May s3://bucket1/May/ --human-readable --summarizeOR
aws cp ./test/May s3://bucket1/May/ --recursive --content-type=text/plainCopy only Files with .sum Extension
aws s3 sync ~/dir1/ s3://bucket1/dir1/ --exclude '*' --include '*.sum' --dryrun Copy Directory and Exclude Two Files
aws s3 sync ~/dir1/ s3://bucket1/dir1/ --exclude 'file1.txt' --exclude 'file2.txt'
