Calculate AWS S3 Bucket Size

Getting S3 Bucket Size

Long story short you can use one of the following methods:

aws s3api --profile PROFILE_NAME list-objects --bucket BUCKET_NAME --output json \
   --query "[sum(Contents[].Size), length(Contents[])]"

or some existing tool such as

s3cmd du s3://bucket-name

But it may be instructive to study in detail how to work with the AWS api:

Listing the Buckets

For getting the bucket size we need two S3 CLI commands

s3 ls or s3api list-buckets

and

s3api list-objects

we also need two related permissions:
s3:ListAllMyBuckets - for seeing the bucket names
s3:ListBucket - for listing the objects (this permission does not allow us to read the content)

A new user without any permissions returns:

petr@petr-VirtualBox:~$ aws --profile demo s3 ls

An error occurred (AccessDenied) when calling the ListBuckets operation: Access Denied

with the policy following IAM policy:

{
    "Version": "2012-10-17",
    "Statement": [
        {
            "Sid": "awss3ls",
            "Effect": "Allow",
            "Action": "s3:ListAllMyBuckets",
            "Resource": "*"
        }
    ]
}

we can get the list of our S3 buckets:

2018-11-04 21:06:24 cf-templates-1he2i9kkenjwe-eu-central-1
2018-01-20 13:49:55 elasticbeanstalk-us-east-2-588956854787
2018-11-17 23:13:08 test-userfiles-mobilehub-927286690

This output is fine, but if we need only the bucket name we need to play with it a little bit more:

We can filter the third column with AWK:

aws --profile demo s3 ls | awk '{print $3}'

or we can call directly s3api list-buckets:

aws --profile demo s3api list-buckets --query "Buckets[].Name"
[
    "cf-templates-1he2i9kkenjwe-eu-central-1",
    "elasticbeanstalk-us-east-2-588956854787",
    "test-userfiles-mobilehub-927286690"
]

For using this in a bash loop it is more useful to work with the text output:

aws --profile demo s3api list-buckets --query "Buckets[].Name" --output text

Listing the Objects

petr@petr-VirtualBox:~$ aws --profile demo s3api list-objects --bucket cf-templates-1he2i9kkenjwe-eu-central-1 --output json
An error occurred (AccessDenied) when calling the ListObjects operation: Access Denied

If we modify the 'Action' of our policy to:

"Action": ["s3:ListAllMyBuckets","s3:ListBucket"]

we would now be able to see the the bucket content:

petr@petr-VirtualBox:~$ aws --profile demo s3api list-objects --bucket cf-templates-1he2i9kkenjwe-eu-central-1 --output json
{
    "Contents": [
        {
            "ETag": "\"7b6127c13b5d833d1632c081e46ad5f2\"",
            "StorageClass": "STANDARD",
            "Size": 2859,
            "LastModified": "2018-11-04T20:06:24.000Z",
            "Key": "2018308Opv-rds-and-lambda.template"
        },
        {
            "ETag": "\"4b20b9c1dc2ae1522c51cdad5c6009ce\"",
            "StorageClass": "STANDARD",
            "Size": 2857,
            "LastModified": "2018-11-04T20:08:29.000Z",
            "Key": "2018308Ugb-rds-and-lambda.template"
        },

Now we can modify the output with a query parameter. The query parameter supports the

http://jmespath.org/ syntax.

petr@petr-VirtualBox:~$ aws --profile demo s3api list-objects 
            --bucket cf-templates-1he2i9kkenjwe-eu-central-1 --output json  
            --query "[sum(Contents[].Size), length(Contents[])]"
[
    22833,
    8
]

Finally, we can put it all together:

for I in `aws --profile demo s3api list-buckets --query "Buckets[].Name" --output text`;
  do aws s3api --profile demo list-objects --bucket $I --output json 
        --query "{\"$I\":{\"size\":sum(Contents[].Size), \"count\":length(Contents[])}}"; 
done
{
    "cf-templates-1he2i9kkenjwe-eu-central-1": {
      "count": 22833,
      "size": 8
   }
}{
    "elasticbeanstalk-us-east-2-588956854787": {
        "count": 6,
        "size": 84688741
    }
}
{
    "test-userfiles-mobilehub-927286690": {
        "size": 211857,
        "count": 4
    }
}

Maybe the last tweak can be to use the jq -s command to create a valid json output:

[
  {
    "cf-templates-1he2i9kkenjwe-eu-central-1": [
      22833,
      8
    ]
  },
  {
    "elasticbeanstalk-us-east-2-588956854787": [
      84688741,
      6
    ]
  },
  {
    "test-userfiles-mobilehub-927286690": [
      211857,
      4
    ]
  }
]

The result is a valid json object which can be easily processed further.