Buildbot master currently off-line

Posted on: 2015-06-29 21:17:45+00:00

Update (2015-06-30 ~12.00 UTC):

The replacement buildbot master is now live. The CMS service and the ci.apache.org  website have been restored. The project CI builds are mostly working but builds that upload docs, snapshots etc. to the buildmaster for publishing are likely to fail at the upload stage while we ensure all the necessary directory structures are in place to receive the uploads. Work to resolve these final few issues is ongoing.

We continue to try and contact the owner of the account where the IRC proxy was running. In case their account has been compromised, it remains locked. In addition, all their commits have been reviewed by other project committers and that review has confirmed that no malicious commits have been made by the account in question.

The review of aegis.apache.org  is ongoing. No evidence of compromise beyond the possible compromise of the single, non-privileged user account has been found.

Original post (2015-06-29 ~21.00 UTC):

As per the e-mails to committers@ earlier today, aegis.apache.org is currently offline after a report was received that suspicious network traffic had been observed from that host. This blog post will be updated as more information becomes known.

What we know:

• At ~16.00 UTC 28 June 2015 a report of suspicious network activity from a buildbot host was reported to the Apache security team.
• Further information was requested and at ~18.00 UTC 28 June 2015 the Apache Infrastructure team received a copy of network logs that showed a number of suspicious IRC connections originating from aegis.apache.org
• These IRC connections were traced to a non-privileged user account on aegis.apache.org  running an open IRC proxy
• At ~20.00 UTC 28 June 2015 the user account concerned was locked for all ASF services and the proxy process terminated.
• At ~10.00 UTC 29 June 2015, after further discussion within the infrastructure team, aegis.apache.org was taken off-line as a precaution.

It remains unclear whether the open IRC proxy was installed by the user that owned the account or whether their account was compromised and the IRC proxy was installed by an unauthorized user.

It is worth stressing that no further information came to light between 20.00 UTC 28 June 2015 and 10.00 UTC 29 June 2015 that triggered the decision to take the host off-line. The host was taken off-line purely as a precaution while we reviewed the available information. That process is ongoing. So far we have found no evidence to even suggest anything more than a user account being used to run an IRC proxy and plenty of evidence that suggests that this was the only activity this account was used for.

Risks:

There is no risk to released source or binaries for any ASF project. There are multiple reasons for this:

• buildbot is a CI system used to build snapshots, not releases
• no builds are performed on aegis.apache.org

Buildbot is used to build some project web sites and / or project documentation. The risk of compromise here is viewed as very low for the following reasons:

• the builds do not take place on aegis.apache.org
• diffs of every change are sent to the relevant project team's mailing list for review and an unexpected / malicious change would be spotted

Project impact:

The following services are currently off-line and will remain so until the buildbot master is restored

• All buildbot builds
• Projects that use the CMS will be unable to update their web sites (the CMS uses buildbot to build web site updates)
  
• the ci.apache.org  website
  

Work in progress:

Analyzing aegis.apache.org  is going to take time and, while we view the chances of a wider compromise of this host as very, very small, we are not willing to bring the host back on line at this point. This host was due for replacement so the decision has been taken to pull this work forward and rebuild the buildbot master on a new host now. We have taken this decision not because we believe aegis.apache.org  to be compromised, but because it is possible to complete this work far more quickly than it is possible to confirm our view that aegis.apache.org is not compromised.  We currently estimate that the rebuild of the new buildbot master host will be completed by 1 July 2015.

We continue to analyze the information we have obtained from aegis.apache.org  and from other sources and will update this blog post as more information becomes available.

Questions:

Questions, concerns, comments etc. should be directed to infrastructure@apache.org