AWS SSM Outage

Incident Report for Cavo

Resolved

This incident has been resolved.
Posted Sep 10, 2022 - 16:19 CDT

Update

We are in the final stages of getting the queues caught up. Closing this issue.
Posted Sep 10, 2022 - 16:19 CDT

Update

We have completed triage work associated with our OCR queues. We are monitoring Friday evening and expect to close this issue out on Saturday.
Posted Sep 09, 2022 - 23:28 CDT

Update

We are continuing to troubleshoot some issues related to the AWS outage. We will continue to provide updates.
Posted Sep 09, 2022 - 15:53 CDT

Monitoring

The AWS outage has been resolved and CAVO document processing has been resumed. Because there is a large backlog, customers may see documents appear to take longer to finish processing than normal
Posted Sep 09, 2022 - 13:58 CDT

Identified

From AWS:

We continue to work on the issue causing increased API error rates and latencies for the Systems Manager Parameter Store in the US-EAST-1 Region. Customers using the Parameter Store for the storage of configurations may be affected by the event as they are unable to retrieve the data. Customers invoking Lambda functions that use Environment Variables will also experience increased error rates and latencies. The subsystem responsible for the Parameter Store is experiencing resource contention, which is leading to the increased error rates and latencies. We continue to work toward the resolving the issue, but do not have an ETA on recovery at this stage. We will keep you updated on our progress.
Posted Sep 09, 2022 - 12:08 CDT
This incident affected: AWS ec2-us-east-1.