Incomplete Multipart Upload occurs when you try to upload an object via Multipart upload process and this specifically occurs only when the complete request is not sent (Even if you have uploaded all the parts but if complete request is not sent then all the parts become Incomplete Multipart Uploads). This can also occur when all the parts are not uploaded successfully as the complete request is sent only after all the parts are uploaded). Let us see in this blog post how to avoid Incomplete Multipart Uploads and save costs by configuring a lifecycle policy to delete the Incomplete Multipart Uploads
S3 Multipart Upload
Amazon S3’s multipart upload feature is a powerful tool for efficiently uploading large objects (between 5 MB and 5 TB) to your S3 buckets. This feature divides your object into smaller parts, which are then uploaded individually. By doing so, it offers several advantages:
- Improved Throughput: By uploading parts concurrently, you can significantly increase the overall upload speed.
- Enhanced Reliability: If a network issue interrupts the upload process, you can resume the transfer from the last successful part, minimizing data loss and reducing the need for complete re-uploads.
When to Use Multipart Uploads:
- Object Size: Generally, if your object size exceeds 100 MB, using multipart upload is recommended.
- Tool and Application Defaults: Many tools and applications, such as the AWS Management Console, AWS CLI, and S3 Replication, automatically utilize multipart uploads for large files.
The Multipart Upload Process:
- Initiation: The upload process begins with an initiation request, which sets up the multipart upload and generates a unique upload ID.
- Part Uploads: Individual parts of the object are uploaded sequentially or concurrently, each identified by a part number.
- Completion: Once all parts are successfully uploaded, a completion request is sent to S3, which assembles the parts into a single object.
Incomplete Multipart Uploads:
If the completion request is not sent successfully, S3 will not assemble the object, and the uploaded parts will remain in your account until they are either completed or explicitly aborted. It’s important to note that you will be charged storage fees for these parts based on the specified storage class.
Cost Savings due to Deletion of InComplete Multipart Uploads
- S3 Data may cost you significantly as each GB stored is charged at $0.021 GB (For storage over $500 TB)
- If your incomplete multipart uploads is more than 50% then implement a lifecycle policy such that it deletes incomplete multipart upload greater than 7 days ore more
- Though lifecycle policy helps you delete multipart upload, Finding the root cause may save our more dollars