Last Updated on July 19, 2021
I have always been wary of EC2 Images and EBS Snapshots since they are cheap and are usually the last place where people would look to lower their AWS bill.
When I saw one of my clients that has a high cost on EBS Snapshots, I checked if there are EBS Snapshots that are unattached to EC2 Images.
To my surprise, the Python script that I was using threw a Read timeout error.
When I checked where the error was triggered, it turns out that it is the describe_images
part of boto3 EC2.
I quickly checked other places where the EC2 Images / AMIs are listed and saw that the listing of the AMI during EC2 Instance launch and in the AWS Console was also not working.
When I investigated the issue, it turns out describe_images
does not support pagination. So calling the AWS API with EC2 DescribeImages will return all the EC2 Images that are currently accessible in the account.
I was able to create a workaround script to list all of the EC2 Images and it turns out it was already more than 200,000 EC2 Images. Most of them were around 3 years old already.
Now imagine there was an issue with an EC2 Instance and we had to launch it from a backup EC2 Image. We will not be able to do so because Describe Images was not working.
In the end, we decided to delete most of the EC2 Images and it went back to normal.
We also deleted the unattached EBS Snapshots and costs.
2 responses to “What happens when you have too many EC2 Images / AMI?”
Your experience sounds terrible. But I have some good news.
Pagination is now supported in the API (as of December 2022)
https://aws.amazon.com/about-aws/whats-new/2022/12/amazon-ec2-describeimages-supports-pagination/
And it seems that boto3 supports it, too:
https://github.com/boto/boto3/issues/1991
This is super good news! Thanks Jason!