In my last post, we talked about the reasons for slow file copying speed on EC2. Today, let's discuss how to speed up file transfers when using AWS S3. You can refer to this AWS document for the underlying principles.
How to increase the speed of copying files from an S3 bucket to the local system?
When I need to copy a bucket to my local machine, I can use the following code:
aws s3 cp --recursive s3://aft-vbi-pds/bin-images/ .
The speed is approximately 240KB/s.
s3 = max_concurrent_requests = 150 max_queue_size = 10000
I modified the configuration file with vim ~/.aws/config, adding the following code at the end.
Parameter explanation:
With this modification, the download speed can reach approximately 3.2MB/s, achieving the effect of speeding up the process.
Additionally, you can try using the sync command:
aws s3 sync s3://aft-vbi-pds/bin-images/ .
