Email service interruption and restoration July 12, 2022

Posted on: 2022-07-13 16:41:38+00:00

July 13, 2022

At around 09:11 UTC on Tuesday, July 12th 2022, the primary mailing list server (colloquially known as Hermes) at The Apache Software Foundation suffered a fatal breakdown and became unresponsive.

The Infrastructure team (Infra) was immediately notified and, in cooperation with our data center provider, attempted to restore services and notify the Foundation of the outage.

As restoring the machine to a useful state proved more difficult than we had hoped, and due to the importance of this service to the Foundation, Infra decided to "fail forward" at approximately 14:40 UTC, and migrate all affected mailing lists and accounts to the new replacement mailing list server for the Foundation (mailgw). We had announced the start of this migration on June 15, 2022.

At approximately 17:33 UTC the bulk of our migration operations had completed, and mail was flowing again. The team continued to address and monitor issues arising as a result of the migration, and the mailing list services were deemed fully operational at approximately 20:00 UTC.