Cavo Outage

Incident Report for Cavo

Postmortem

Prior to release, this code was tested on our hotfix server. We were able to view pages in Cavo. This is the first time we have seen a test succeed on the hotfix server and yet fail once promoting to production. In hindsight, we plan to mitigate the potential for this as follows:

  1. Release hotfix code in the evening rather than in the morning. This code was approved for release in the evening but was not pushed until morning. Releasing in the evening gives us a greater amount of time to resolve the issue while affecting fewer customers.
  2. Post deployment testing. We are establishing a manual test suite to run on newly promoted code for both beta and production environments. These tests will be performed by an individual immediately following the deployment.
Posted Feb 14, 2020 - 09:46 CST

Resolved

We have resolved the issue by rolling back the latest hotfix. All systems are operational.
Posted Feb 14, 2020 - 08:37 CST

Identified

We have identified the issue and are building the fix now. We expect this to be resolved by no later than 8:45 am.
Posted Feb 14, 2020 - 08:25 CST

Investigating

We are investigating an issue with Cavo.
Posted Feb 14, 2020 - 08:08 CST
This incident affected: CAVO Application - Production and CAVO Application - Beta.