Certificate Authority

Incident Report for Cavo

Postmortem

Several events led to this problem.

Notifications - Our notifications for admin, beta and production were set properly to notify support@cavo.io for unreachable instances as well as for certificate expiration.

Mail Server - Our mail server was set to properly forward these notifications to Intercom, our ticketing system.

Intercom - Intercom changed the target email address in charge of receiving emails forwarded to support@cavo.io. As such, our support team was not notified that the certificates were an issue.

*.cavo.io certificate - The new certificate was renewed last month.

*.apello.io certificate - This new “branding” certificate was obtained at the same time.

In order to minimize confusion, we implemented the apello.io certificate and failed to update the cavo.io certificate. This oversight coupled with the lack of notifications led to the issue.

Steps Taken to mitigate:

  1. Intercom receipt of notifications fixed and confirmed.
  2. apello.io and cavo.io certificates verified as updated.
Posted Mar 29, 2021 - 13:31 CDT

Resolved

We pushed the updated CA to the Cavo application servers and tested.
Posted Mar 29, 2021 - 10:24 CDT

Investigating

While updating the certificate resolved the issue for most users, we have also discovered that the CA needed to be updated for API access and in some edge cases. We are performing this maintenance now.
Posted Mar 29, 2021 - 10:19 CDT
This incident affected: CAVO Application - Production and CAVO Application - Beta.