Amazon Web Services (AWS) reported Tuesday, December 7, 2021 that “many” of its web services had recovered Tuesday after an outage disrupted major streaming platforms and other widely used websites. Reports of problems with AWS on Downdetector.com peaked about 11:00 a.m. CST Tuesday, December 7, 2021. Around that time, AWS reported, “We are seeing impact to multiple AWS APIs in the US-EAST-1 Region. This issue is also affecting some of our monitoring and incident response tooling, which is delaying our ability to provide updates. We have identified the root cause and are actively working towards recovery.”
At 12:34 p.m. PST, AWS reported the root cause of the issue was an impairment of several network devices, which are generally defined to include core network components known as gateways, routers, switches, bridges, repeaters, repeater hubs, wireless access points, and other non-core devices that are located further down the line toward users and consumers.
Cryptocurrency exchange site Coinbase, Disney+, Netflix, Robinhood (stock trading app), Slack, and other services were among the sites reporting issues.
In an update at 3:03 p.m. PST, Amazon Web Services said that it was still working toward a full recovery, but services like SSO and Eventbridge were still down.”Engineers are actively working on resolving impact to these services,” the 3:03 p.m. PST update said.
Sellers on Amazon were also having issues accessing Seller Center, a site used to manage orders. Earlier, Amazon said that network devices in its US-EAST-1 region, hosted in Northern Virginia, had caused the outage.
“This issue is also affecting some of our monitoring and incident response tooling, which is delaying our ability to provide updates.”
— Amazon Web Services
The AWS outage also affected airlines, DoorDash, Google, Ring, Spotify, and Venmo. Reports of problems with Google on Downdetector.com peaked about 12:12 p.m. CST Tuesday, December 7, 2021. Two Google users reported home automation devices were not working.
Gray Television stations, such as WFSB CBS affiliate in Hartford, Connecticut and WOWT NBC affiliate in Omaha, Nebraska, reported that the AWS outage affected the content management system used by Gray Television that updates content for its television stations. According to a Gray Television official web page retrieved on December 7, 2021, Gray owns and/or operates television stations and leading digital properties in 94 television markets that collectively reach approximately 24% of U.S. television households. According to Gray, the business is expecting to own TV stations in 102 markets that reach 25.4% of US television households.
CNBC indicated there were Reddit posts from Amazon Flex warehouse, and delivery workers who reported the apps that keep track of packages, and provide delivery instructions were down.
A Reddit post with a photo in a warehouse and responsive comments indicated CLE2 Amazon (a warehouse in North Randall, Ohio) was down, along with CLE3 (Euclid, Ohio), CLE5 (Twinsburg, Ohio)
Amazon Web Service Status Report December 7, 2021
API Error Rates in US-EAST-1
[9:37 AM PST] We are seeing impact to multiple AWS APIs in the US-EAST-1 Region. This issue is also affecting some of our monitoring and incident response tooling, which is delaying our ability to provide updates. We have identified the root cause and are actively working towards recovery.
[10:12 AM PST] We are seeing impact to multiple AWS APIs in the US-EAST-1 Region. This issue is also affecting some of our monitoring and incident response tooling, which is delaying our ability to provide updates. We have identified root cause of the issue causing service API and console issues in the US-EAST-1 Region, and are starting to see some signs of recovery. We do not have an ETA for full recovery at this time.
[11:26 AM PST] We are seeing impact to multiple AWS APIs in the US-EAST-1 Region. This issue is also affecting some of our monitoring and incident response tooling, which is delaying our ability to provide updates. Services impacted include: EC2, Connect, DynamoDB, Glue, Athena, Timestream, and Chime and other AWS Services in US-EAST-1. The root cause of this issue is an impairment of several network devices in the US-EAST-1 Region. We are pursuing multiple mitigation paths in parallel, and have seen some signs of recovery, but we do not have an ETA for full recovery at this time. Root logins for consoles in all AWS regions are affected by this issue, however customers can login to consoles other than US-EAST-1 by using an IAM role for authentication.
[12:34 PM PST] We continue to experience increased API error rates for multiple AWS Services in the US-EAST-1 Region. The root cause of this issue is an impairment of several network devices. We continue to work toward mitigation, and are actively working on a number of different mitigation and resolution actions. While we have observed some early signs of recovery, we do not have an ETA for full recovery. For customers experiencing issues signing-in to the AWS Management Console in US-EAST-1, we recommend retrying using a separate Management Console endpoint (such as https://us-west-2.console.aws.amazon.com/). Additionally, if you are attempting to login using root login credentials you may be unable to do so, even via console endpoints not in US-EAST-1. If you are impacted by this, we recommend using IAM Users or Roles for authentication. We will continue to provide updates here as we have more information to share.
[2:04 PM PST] We have executed a mitigation which is showing significant recovery in the US-EAST-1 Region. We are continuing to closely monitor the health of the network devices and we expect to continue to make progress towards full recovery. We still do not have an ETA for full recovery at this time.
[2:43 PM PST] We have mitigated the underlying issue that caused some network devices in the US-EAST-1 Region to be impaired. We are seeing improvement in availability across most AWS services. All services are now independently working through service-by-service recovery. We continue to work toward full recovery for all impacted AWS Services and API operations. In order to expedite overall recovery, we have temporarily disabled Event Deliveries for Amazon EventBridge in the US-EAST-1 Region. These events will still be received & accepted, and queued for later delivery.
[3:03 PM PST] Many services have already recovered, however we are working towards full recovery across services. Services like SSO, Connect, API Gateway, ECS/Fargate, and EventBridge are still experiencing impact. Engineers are actively working on resolving impact to these services.
[4:35 PM PST] With the network device issues resolved, we are now working towards recovery of any impaired services. We will provide additional updates for impaired services within the appropriate entry in the Service Health Dashboard.
^^ MOBILE? USE VOICE MIC ^^
facebook …
Please ‘LIKE’ the ‘Arlington Cardinal Page. See all of The Cardinal Facebook fan pages at Arlingtoncardinal.com/about/facebook …
Help fund The Cardinal Arlingtoncardinal.com/sponsor
20240105-1435future
THANKS FOR READING CARDINAL NEWS