On-demand workers from Rackspace added to builds.apache.org

A couple of weeks ago, Apache's Infrastructure team added a new feature to our Jenkins server, builds.apache.org to help deal with the at times overwhelming queues of builds waiting for an executor. While this has been improved dramatically by the increase in workers generously provided by Yahoo! on physical hosts, we're always trying to look forward and be prepared for increased usage in the future. 

To that end, we've set up worker images on Rackspace, generated using the fantastic tool Packer. Using the Apache jclouds plugin for Jenkins, Ubuntu workers will be spun up dynamically on Rackspace using those images when there's a queue of pending builds that are able to run on the “ubuntu” label. Up to five of these workers can be running at a time, and they're automatically removed from Jenkins and destroyed on Rackspace once they've been idle a set period of time. This burst capacity will help us prevent a long wait for builds to run on builds.apache.org.

We're able to do this thanks to Rackspace generously donating resources to the Apache Software Foundation. We're extremely grateful for this, and if any other public cloud providers are also interested in donating compute cycles to the Foundation, please contact the Infrastructure team.

One thing to note - the worker image we're using is still new and may have bugs in it. If you see your build suddenly failing for mysterious reasons, please take a look at the worker it ran on - if it's a worker named something like “jenkins-ubuntu-1404-4gb-abc”, please open a BUILDS JIRA at issues.apache.org with a link to the failing build and we'll investigate.

We've got more improvements for builds.apache.org planned for the future, and we're looking forward to sharing them with all of you - there'll be a talk at ApacheCon EU this November on the current status of Jenkins at the ASF, what we've done to stabilize and improve the developer experience on builds.apache.org this year, and what's planned for the future - hope to see you there!