Nowadays, cloud computing is a mature, widely established model to efficiently develop and operate applications at scale. Despite this, the limitations that lurk beneath cloud vendor abstractions can still easily take engineers by surprise. Learn from my experience and avoid the pitfalls I recently encountered working with the Amazon EC2 burstable instance model.
I joined a product development team to help them increase the capacity and observability of their B2B e-commerce application. The aim was to make the system resilient against an anticipated peak traffic season looming ever closer.
The tech stack comprised two PHP web applications running on a small cluster of Amazon EC2 instances distributed across two environments, namely:
We had just added several alerts and dashboards to Datadog, our centralised application monitoring platform. The configuration of the alerts was mirrored across both environments to obtain early warning of concerning trends in Development before they had the chance to emerge in Production.
The team set up an alarm to monitor CPU utilisation for all the EC2 instances in the platform. The alarm would trigger if CPU consumption remained higher than 60% on average for a few minutes. Sometime later, this notification appeared in our non-production alerts channel.
There are an array of reasons why available CPU capacity can dip dramatically for a short period in a development server, even if the server is typically quiet:
The first time this happened we didn’t put much effort into discerning the underlying cause. We used the above mundane reasons to explain the problem away and moved on to more pressing matters. However, this wasn’t the last we’d see of this alert.
Following this first instance, it wasn’t long before the high CPU utilisation alarms for dev-2 became a regular occurrence.
We could no longer dismiss this as a transient issue so we decided to look closer at the problem. Sure, testing in the environment could increase the workload on the instance but not to the extreme levels our alerts were showing. What was even weirder is that dev-1 or dev-3 were not suffering from the same problem! Nonetheless, there were a few obvious explanations we could rule out to narrow things down.
Our system had recently become the target of a few low-scale DDoS attacks. Crawlers and bots, both benign and malicious, probe our public endpoints routinely for weaknesses so this explanation didn’t sound far-fetched at all. However, neither our quiet traffic gauges nor the uninteresting makeup of our incoming HTTP requests backed up this hypothesis.
The metric system.cpu.stolen (in yellow) immediately drew my attention as it was shown as the single biggest contributor to the high CPU figures we were registering. But what is system.cpu.stolen exactly? Who’s stealing our CPUs from us, and why would they do that?
The two graphs above depict an EC2 instance on a Standard credit specification, running out of credits and being throttled as a result. These graphs established the direct link between the CPU resource starvation alerts we had been regularly seeing in dev-2. At this stage, the remaining balance for dev-2 was nearly 0. It didn’t accrue any credits as it rarely remained under its baseline 10% CPU utilisation target.
Solving this problem was far easier for us than identifying it, as all we had to do was flip the credit specification of dev-2 to Unlimited. This is something you can do on a running EC2 instance. We might incur slightly higher EC2 costs as a result of doing this if we expect the baseline to be surpassed regularly but we found it was worth it as the price difference was negligible in a tiny compute cluster of t3.micro instances.
Even more so than the cost of our Development environment bill, we were concerned about the possibility that our Production EC2s were subject to the same limitation, as they are also t3.micro. Fortunately, this was not the case. The default credit specification for new burstable EC2 instances for our account/region combination had been previously configured to be Unlimited. dev-2 might have been set up during a time when the default account-wide setting was Standard.
One of the most surprising things I learned from this was how conservative the EC2 burstable CPU capacity baselines are and how easy it is to breach them if you’re not careful. This was something that had been pretty far down in my list of potential reasons why an EC2 could misbehave up to this point.
Provisioning burstable EC2 instances on a Standard credit specification is the cheapest way to run EC2 on-demand instances but it is important to familiarise ourselves with the intricate rules that underpin its billing mechanism to avoid wasting time troubleshooting easily preventable resource saturation issues in the cloud.
Regular performance testing and routine monitoring of our services help us develop a mental model of what normal usage looks like in our systems. This allows us to plan for capacity and minimise costs by ensuring our infrastructure can operate smoothly without breaching vendor-imposed restrictions.
Let’s connect and see how we can help you achieve your vision.
Your privacy is important to us. It is T5 Digital’s policy to respect your privacy and comply with any applicable law and regulation regarding any personal information we may collect about you, including across our website, https://t5.digital/, and other sites we own and operate.
This policy is effective as of 27 May 2022 and was last updated on 27 May 2022.
Information We Collect
Information we collect includes both information you knowingly and actively provide us when using or participating in any of our services and promotions, and any information automatically sent by your devices in the course of accessing our products and services.
Log Data
When you visit our website, our servers may automatically log the standard data provided by your web browser. It may include your device’s Internet Protocol (IP) address, your browser type and version, the pages you visit, the time and date of your visit, the time spent on each page, other details about your visit, and technical details that occur in conjunction with any errors you may encounter.
Please be aware that while this information may not be personally identifying by itself, it may be possible to combine it with other data to personally identify individual persons.
Collection and Use of Information
We may collect personal information from you when you do any of the following on our website:
Use a mobile device or web browser to access our content
Contact us via email, social media, or on any similar technologies
When you mention us on social media
We may collect, hold, use, and disclose information for the following purposes, and personal information will not be further processed in a manner that is incompatible with these purposes:
Please be aware that we may combine information we collect about you with general information or research data we receive from other trusted sources.
Security of Your Personal Information
When we collect and process personal information, and while we retain this information, we will protect it within commercially acceptable means to prevent loss and theft, as well as unauthorized access, disclosure, copying, use, or modification.
Although we will do our best to protect the personal information you provide to us, we advise that no method of electronic transmission or storage is 100% secure, and no one can guarantee absolute data security. We will comply with laws applicable to us in respect of any data breach.
You are responsible for selecting any password and its overall security strength, ensuring the security of your own information within the bounds of our services.
How Long We Keep Your Personal Information
We keep your personal information only for as long as we need to. This time period may depend on what we are using your information for, in accordance with this privacy policy. If your personal information is no longer required, we will delete it or make it anonymous by removing all details that identify you.
However, if necessary, we may retain your personal information for our compliance with a legal, accounting, or reporting obligation or for archiving purposes in the public interest, scientific, or historical research purposes or statistical purposes.
Children’s Privacy
We do not aim any of our products or services directly at children under the age of 13, and we do not knowingly collect personal information about children under 13.
International Transfers of Personal Information
The personal information we collect is stored and/or processed where we or our partners, affiliates, and third-party providers maintain facilities. Please be aware that the locations to which we store, process, or transfer your personal information may not have the same data protection laws as the country in which you initially provided the information. If we transfer your personal information to third parties in other countries: (i) we will perform those transfers in accordance with the requirements of applicable law; and (ii) we will protect the transferred personal information in accordance with this privacy policy.
Your Rights and Controlling Your Personal Information
You always retain the right to withhold personal information from us, with the understanding that your experience of our website may be affected. We will not discriminate against you for exercising any of your rights over your personal information. If you do provide us with personal information you understand that we will collect, hold, use and disclose it in accordance with this privacy policy. You retain the right to request details of any personal information we hold about you.
If we receive personal information about you from a third party, we will protect it as set out in this privacy policy. If you are a third party providing personal information about somebody else, you represent and warrant that you have such person’s consent to provide the personal information to us.
If you have previously agreed to us using your personal information for direct marketing purposes, you may change your mind at any time. We will provide you with the ability to unsubscribe from our email-database or opt out of communications. Please be aware we may need to request specific information from you to help us confirm your identity.
If you believe that any information we hold about you is inaccurate, out of date, incomplete, irrelevant, or misleading, please contact us using the details provided in this privacy policy. We will take reasonable steps to correct any information found to be inaccurate, incomplete, misleading, or out of date.
If you believe that we have breached a relevant data protection law and wish to make a complaint, please contact us using the details below and provide us with full details of the alleged breach. We will promptly investigate your complaint and respond to you, in writing, setting out the outcome of our investigation and the steps we will take to deal with your complaint. You also have the right to contact a regulatory body or data protection authority in relation to your complaint.
Use of Cookies
We use “cookies” to collect information about you and your activity across our site. A cookie is a small piece of data that our website stores on your computer, and accesses each time you visit, so we can understand how you use our site. This helps us serve you content based on preferences you have specified.
Limits of Our Policy
Our website may link to external sites that are not operated by us. Please be aware that we have no control over the content and policies of those sites, and cannot accept responsibility or liability for their respective privacy practices.
Changes to This Policy
At our discretion, we may change our privacy policy to reflect updates to our business processes, current acceptable practices, or legislative or regulatory changes. If we decide to change this privacy policy, we will post the changes here at the same link by which you are accessing this privacy policy.
If required by law, we will get your permission or give you the opportunity to opt in to or opt out of, as applicable, any new uses of your personal information.
Contact Us
For any questions or concerns regarding your privacy, you may contact us using the following details:
Anna Broadhurst
info@t5.digital
Introduction
By participating in the competition, you are agreeing to these competition terms and conditions.The competition is being run by T5 Digital.
Eligibility to Enter
The competition is open to entrants who are at least 18 years of age or older. T5 Digital employees and associates are excluded from the draw.
By entering the competition, you confirm that you are eligible to do so and that you are eligible to receive any prizes that may be awarded to you.
There is a limit of one entry per person and the competition is completely free to enter.
The Prize
The winning prize will be Bespoke Food and Drink Hamper.
The use of specific brands as prizes by T5 Digital does not imply any affiliation with or endorsement of such brands.
The prize is non-transferable and non-exchangeable, and no cash alternatives will be provided.
We reserve the right to substitute prizes of equal or greater value if circumstances beyond our control require doing so.
T5 Digital’s decision on any aspect of the competition is final and binding, and no correspondence will be entered into about it.
Winner Announcement
The winner will be chosen at random and notified via the email address provided on 14th March 2025.
T5 Digital will make two attempts to contact the winner via email.
If the winner does not respond to the emails informing them of their win within 14 days of the second email, they forfeit their right to the prize, and T5 Digital reserves the right to select and notify a new winner.
Delivery of the Prize
The winner will allow 14 days for the prize to be delivered; otherwise, alternative collection or delivery arrangements can be made through mutual agreement.
Data Protection and Publicity
You agree that any personal information that you provide when entering the competition will be used by T5 Digital for the purposes of administering the competition and for other purposes as specified in our Privacy Policy.
All entrants may request information on the winning participant by emailing info@t5.digital.
If requested by T5 Digital, the winner agrees to release their first name and place of employment to other competition participants.
The winner’s first name and country of residence will be announced on T5 Digital’s website and social media channels.
Limitation of Liability
T5 Digital accepts no liability for any damage, loss, injury, or disappointment suffered by entrants as a result of participating in the competition or being selected for a prize.
General
T5 Digital reserves the right, at any time and without prior notice, to cancel the competition or amend these terms and conditions.
