In my last post, we talked about how to increase the speed of copying files from an S3 bucket to the local machine. Today let's talk about how to get the object count in an AWS S3 bucket.
Background
AWS S3 does not provide a direct API to get the number of objects in a bucket or within a specific directory.
An indirect way to achieve this is by using ListObjectsV2 to paginate through all the objects and then count them.
ListObjectsV2
The REST API syntax for ListObjectsV2 is as follows:
GET /?list-type=2&continuation-token=ContinuationToken&delimiter=Delimiter&encoding-type=EncodingType&fetch-owner=FetchOwner&max-keys=MaxKeys&prefix=Prefix&start-after=StartAfter HTTP/1.1 Host: Bucket.s3.amazonaws.com x-amz-request-payer: RequestPayer x-amz-expected-bucket-owner: ExpectedBucketOwner
There are a few key parameters to pay attention to.
●Max-keys: Each request returns some or all of the objects (up to 1000) in the bucket. You can use max-keys to specify the number of objects retrieved per request, with valid values ranging from 1 to 1000.
●Prefix: If you want to get the object count in a specific directory of the bucket, you can use prefix to specify the object prefix.
●Continuation-token: ListObjectsV2 uses deep pagination to retrieve objects, so a token like NextToken is used to paginate through results. The continuation-token can be obtained from the response of the previous request.
In the response, there is one key field to focus on:
IsTruncated: This indicates whether the list is truncated. If it’s true, it means not all objects have been retrieved; if it’s false, it means the entire list has been retrieved.
Golang SDK Example
Below is an implementation example using the S3 Golang SDK.
input := &s3.ListObjectsV2Input{
Bucket: aws.String("your-bucket-name"),
Prefix: aws.String("path/to/folder/"),
MaxKeys: aws.Int64(1000),
}
objectCount := 0
for {
resp, err := clt.ListObjectsV2(input)
if err != nil {
log.Fatal(err)
}
objectCount += len(resp.Contents)
fmt.Println("Objects Count:", objectCount)
if !*resp.IsTruncated {
break // Reach the end of the object listing
}
input.ContinuationToken = resp.NextContinuationToken
}
In the above example, we first set MaxKeys to 1000, then use a loop to call the ListObjectsV2 API until the returned IsTruncated field is false, indicating we’ve reached the end of the object list. In each iteration, we accumulate the number of objects returned and update the ContinuationToken to proceed with the next page of results.
By using the pagination mechanism, you can retrieve more than 1000 objects incrementally and handle the results of each page in the loop. This way, you can handle any number of objects without exceeding AWS S3 API limits.
You can refer to the following code for initializing the S3 client.
package main
import (
"log"
"github.com/aws/aws-sdk-go/aws"
"github.com/aws/aws-sdk-go/aws/session"
"github.com/aws/aws-sdk-go/service/s3"
)
func main() {
// Create a new AWS session
sess, err := session.NewSession(&aws.Config{
Region: aws.String("your-region"),
// Optional: Provide credential information, or the SDK will attempt to use the default credentials.
// Credentials: credentials.NewStaticCredentials("your-access-key", "your-secret-key", ""),
})
if err != nil {
log.Fatal(err)
}
// Create a new S3 service client
svc := s3.New(sess)
// Use svc to perform S3 operations, such as uploading, downloading, and deleting objects.
// ...
// Close the session
sess.Close()
}
In the example above, you need to replace the following:
●"your-region": Replace with your AWS region code, such as “us-west-1”.
●"your-access-key" and "your-secret-key": If you haven’t set your credential information as environment variables or in the configuration file, you can uncomment and replace them with your AWS Access Key ID and Secret Access Key.
By creating an AWS session and using the session to create an S3 service client, you can use the Go client to perform various S3 operations, such as uploading objects, downloading objects, deleting objects, etc. After completing the operations, you can close the session.
Make sure you have installed the github.com/aws/aws-sdk-go dependency using the go get command so you can use the AWS S3 Go client in your code.
