apache.org incident report for 05292012

Posted on: 2012-05-29 16:59:09+00:00

Last week, internal audit activity discovered that the access logs of some committer-only Apache services contained passwords but had been available to every Apache committer.

The problem

The httpd logs of several ASF services are aggregated and archived on minotaur.apache.org.  Minotaur is also people.apache.org, the shell host for committers, and committers were encouraged to analyse the logs and produce aggregated data.

However, for two services, the archived logs included forensic logs, which are extra-verbose logs that include all HTTP request headers.  (The logs are never encrypted, even if the HTTP connection was wrapped by SSL encryption.)  Both of these services http://s.apache.org and http://svn.apache.org allow anyone to use them in a read-only manner anonymously, and allow further operations (such as creating shortlinks) to LDAP-authenticated committers.  Authentication is usually done by embedding the username and password, encoded in base64, in the "Authorization:" HTTP header, under SSL encryption.

Base64 is a reversible transform.  (It is an encoding, not a cipher.)

Consequently, any Apache committer could learn the passwords of any other committer by reading the log files and reversing the base64 encoding.

Shutting the barn door

The logs archive directory was made readable by the root user only.  Forensic logging was disabled, and past forensic logs deleted.  ZFS snapshots containing those logs were destroyed, too.

Finding the horse

We know that several committers had on one occasion or another copied the logs in order to analyse them, so we operated on the assumption that copies of the sensitive forensic logs were circulating on hardware we do not control.  We therefore opted to have all passwords changed, or reset.

Several Apache committers whose passwords grant very high access were advised privately to change their passwords.  The root@ team ensured the follow-through and, before announcing the vulnerability any further, changed the passwords of those whom had not done so themselves.  The root@ team also changed the passwords of all non-human (role) accounts on those services.

The vulnerability was then announced to all Apache committers with the same instructions: 'Your passwords may be compromised; change them "now"; we will explain the problem later.'.  This notice was authenticated via a PGP signature and via acknowledging it in a root-owned file on people.apache.org.

Finally, passwords that have not been changed after forensic logs had been disabled and, therefore, were presumed to be contained in compromised forensic logs were changed by the root@ team to random strings.

Implications

Were some committer to have compromised another Apache account using this vulnerability prior to these steps being taken, note that root access to all apache.org hosts is only available using one-time-passwords (otp) for certain privileged sudo users.  Such account holders have been instructed not to use the same password for otp as for LDAP, so this would not have resulted in an attacker gaining root privileges without our knowledge.  All of our commit activity is peer-reviewed and logged to various commit lists, and no reports of unusual commit activity have been received during the time frame in which this exposure was effective.  In fact no unusual activity has ever been reported regarding any of our LDAP-based services, so there is no reason for us to suspect malicious activity has occurred as a result of this vulnerability.

Preventing recurrence

No code changes were needed to the software that s.apache.org and svn.apache.org run; the software was behaving correctly according to its configuration, but the configuration itself and the in-house log archiving scripts were incorrect.

A member of the infrastructure team will be approaching the Apache HTTPD PMC with a documentation patch for mod_log_forensic.

Epilogue

There were no malicious parties involved here (to our knowledge); we just made a configuration error.  The nature of the error meant we had to assume all passwords were compromised, and that was costly to fix.

We hope our disclosure has been as open as possible and true to the ASF spirit.  Hopefully others can learn from our mistakes.  See our prior incident reports from the Apache Infrastructure Team.

Committers please address questions to root@apache.org only.

Queries from the press should be sent to press@apache.org.

Happy hacking!