Starting on Tuesday, Dec. 7, 2021 at 10:32 AM EST, our services were impacted by a widespread outage to Amazon Web Services (AWS) in the US-EAST-1 Region. This outage caused Public’s app, in addition to more than 11,000 other platforms, apps, and other AWS customers, to be temporarily and intermittently unavailable beginning at 10:32 AM EST, with restoration of crypto trading services at 4:19 PM EST, and full service restoration at 8:45 PM EST.
Although access to our app was limited during this window, our members were still able to submit trades by contacting our Customer Support team at email@example.com. During the outage, our customers’ accounts, information and funds were continuously secure.
We apologize for any inconvenience caused to our members. This post explains what happened, how we responded, and what’s being done to improve our processes in the future.
Amazon Web Services and Public
Amazon Web Services, or AWS, is a cloud platform that provides data servers and storage for millions of companies, from startups to some of the largest platforms in the world, like Disney, Netflix, Pinterest, and NASA. Many companies use AWS, but you might not realize it until the rare instance of an outage.
Companies like Public choose cloud service partners carefully, and AWS is among the most stable and reliable services in the market today. While hearing about external outages can be frustrating, the reason many apps and platforms use services like AWS is due to their overall stability the vast majority of the time.
Recent AWS outage and its impact on Public
On Dec. 7, 2021 at approximately 10:32 AM EST, we detected a disruption to our AWS service. Thousands of AWS customers reported similar issues, thus indicating that this was a global outage and not isolated to Public.
AWS attributed the outage to an issue with several of its network devices, which were receiving more traffic than usual during the time frame. In an effort to reduce the load in the problem area, AWS re-routed traffic to alternate services, which led to widespread Domain Name System (DNS) issues. AWS then needed to work through these issues to ultimately restore service to Public, and more than 11,000 others.
We always plan for the unexpected and have protocols in place for these moments, rare as they may be. As such, we urgently proceeded with the following action plan:
Immediate contact with our partners at AWS to gather key information, expectations, and plan for working through the outage
Direct communication to impacted members via Twitter, in-app messaging, and our Customer Support team
Our Customer Support team quickly escalated to prioritize manual trade requests, which enabled our licensed team members to facilitate trades requested by members via our Customer Support channels
Constant real-time monitoring of AWS status and communication with AWS team service was 100% restored for our app and our members
Follow-up communication to our members to notify them of restored app access
As of early morning on Dec. 8, AWS reported that they resolved the core issues that led to the outage. As stated above, full service of our app was restored as of 8:45 PM EST on Dec. 7.
We understand that disruptions to service are frustrating and stressful for our members. Member safety and app stability are the two most important things we optimize for at Public, which is why we have protocols in place for rare moments of service disruption (in this case due to our external cloud partner, AWS).
With that, we are actively looking into ways to mitigate similar situations in the future, including exploring improvements to our contingency plan and working with Amazon Web Services to identify how we may reduce the impact of future outages.
We’re building Public for you and are always open to hearing from our members. Please reach out to our Customer Support team at firstname.lastname@example.org.