S3

Frequently used AWS CLI commands for managing S3 buckets and objects.

Sync between buckets (same or different accounts)

# Sync with bucket-owner ACL (required when copying across accounts)
aws s3 sync s3://source-bucket/ s3://destination-bucket/ --acl bucket-owner-full-control

# Sync with a named profile (for cross-account)
aws s3 sync s3://source-bucket/ s3://destination-bucket/   --acl bucket-owner-full-control   --profile source-account-profile

Copy / move objects

# Copy a single file
aws s3 cp s3://bucket/file.txt s3://other-bucket/file.txt

# Move (copy then delete)
aws s3 mv s3://bucket/file.txt s3://bucket/archive/file.txt

# Copy all objects matching a prefix
aws s3 cp s3://bucket/logs/ s3://archive-bucket/logs/ --recursive

List and inspect

# List buckets
aws s3 ls

# List objects in a bucket
aws s3 ls s3://my-bucket --recursive --human-readable

# Get object metadata
aws s3api head-object --bucket my-bucket --key path/to/file.txt

Bucket operations

# Create a bucket
aws s3 mb s3://my-new-bucket --region us-east-1

# Delete all objects in a bucket (empty it)
aws s3 rm s3://my-bucket --recursive

# Delete the bucket (must be empty)
aws s3 rb s3://my-bucket

Presigned URLs (temporary access)

# Generate a URL valid for 1 hour (3600 seconds)
aws s3 presign s3://my-bucket/private-file.pdf --expires-in 3600

s3cmd is a free, open-source command-line tool for managing Amazon S3 and other S3-compatible object stores (DigitalOcean Spaces, Google Cloud Storage, etc.).

Installation

# Ubuntu / Debian
sudo apt-get install -y s3cmd

# macOS
brew install s3cmd

# pip
pip install s3cmd

Configuration

s3cmd --configure

This creates ~/.s3cfg. For non-AWS S3 endpoints (e.g., DigitalOcean Spaces), set host_base and host_bucket in the config file.

Common commands

# List buckets
s3cmd ls

# List objects in a bucket
s3cmd ls s3://my-bucket

# Upload a file
s3cmd put local-file.zip s3://my-bucket/backups/

# Upload and make public
s3cmd put --acl-public image.png s3://my-bucket/

# Download a file
s3cmd get s3://my-bucket/file.zip ./

# Sync a directory to S3
s3cmd sync ./local-dir/ s3://my-bucket/dir/

# Delete an object
s3cmd del s3://my-bucket/old-file.txt

# Show disk usage of a bucket
s3cmd du s3://my-bucket

Automated backups with cron

0 3 * * * s3cmd sync /var/www/html/ s3://my-backup-bucket/www/ >> /var/log/s3-backup.log 2>&1

S3 bucket policies control who can access your buckets and objects. The following are commonly used policy templates.

Allow public read access to all objects in a bucket

{
  "Version": "2012-10-17",
  "Statement": [{
    "Sid": "PublicReadGetObject",
    "Effect": "Allow",
    "Principal": "*",
    "Action": "s3:GetObject",
    "Resource": "arn:aws:s3:::YOUR-BUCKET-NAME/*"
  }]
}

Allow a specific IAM user to read and write

{
  "Version": "2012-10-17",
  "Statement": [{
    "Effect": "Allow",
    "Principal": {
      "AWS": "arn:aws:iam::ACCOUNT_ID:user/USERNAME"
    },
    "Action": ["s3:GetObject", "s3:PutObject", "s3:DeleteObject", "s3:ListBucket"],
    "Resource": [
      "arn:aws:s3:::YOUR-BUCKET-NAME",
      "arn:aws:s3:::YOUR-BUCKET-NAME/*"
    ]
  }]
}

Allow another AWS account to put objects

{
  "Version": "2012-10-17",
  "Statement": [{
    "Effect": "Allow",
    "Principal": { "AWS": "arn:aws:iam::OTHER_ACCOUNT_ID:root" },
    "Action": ["s3:PutObject", "s3:PutObjectAcl"],
    "Resource": "arn:aws:s3:::YOUR-BUCKET-NAME/*"
  }]
}

Enforce HTTPS only (deny HTTP)

{
  "Version": "2012-10-17",
  "Statement": [{
    "Sid": "DenyHTTP",
    "Effect": "Deny",
    "Principal": "*",
    "Action": "s3:*",
    "Resource": [
      "arn:aws:s3:::YOUR-BUCKET-NAME",
      "arn:aws:s3:::YOUR-BUCKET-NAME/*"
    ],
    "Condition": {
      "Bool": { "aws:SecureTransport": "false" }
    }
  }]
}

For buckets with millions of objects, deleting everything individually is slow and costly. A lifecycle rule that expires all objects is the most efficient way to empty a large bucket.

Method 1 – Lifecycle rule (recommended for large buckets)

1. In the S3 console, select your bucket → Management tab → Create lifecycle rule.
2. Rule name: empty-bucket
3. Scope: Apply to all objects in the bucket.
4. Under Lifecycle rule actions, check:
   • Expire current versions of objects — set 0 days after object creation.
   • Permanently delete noncurrent versions — set 0 days after objects become noncurrent (if versioning is enabled).
   • Delete expired object delete markers or incomplete multipart uploads.
5. Save. AWS will delete objects in the background (may take hours for very large buckets).

Method 2 – AWS CLI (for smaller buckets or immediate action)

# Delete all objects (current versions)
aws s3 rm s3://my-bucket --recursive

# If versioning is enabled, delete all versions too
aws s3api delete-objects   --bucket my-bucket   --delete "$(aws s3api list-object-versions     --bucket my-bucket     --query '{Objects: Versions[].{Key:Key,VersionId:VersionId}}'     --output json)"

After emptying — delete the bucket

aws s3 rb s3://my-bucket