Optimizing GitHub Actions workflows

GitHub-hosted runners are a shared resource: every ASF project draws on the same pool, so the minutes one project saves shorten the queue for everyone else. The GitHub Actions Policy sets the limits every project must stay within, and How your project can help lists the quick wins. This page lists the optimizations a PMC can implement in its own workflows, and gives an example for each. To check your project's workflows against them, you can audit them with Apache Magpie.

The examples come from Apache Airflow, whose CI is one of the largest at the ASF and which has tried most of these techniques. Your project's results may differ.

Some of these techniques are experiments that Airflow is exploring. Treat them as recipes to evaluate, not as recommended defaults.

Summary¶

Technique	Status	Effect in Airflow
Start with the basics	Recommended for every project	Frees runners held by hung, duplicate and superseded runs
Run only the tests a change needs	In production	A typical PR runs 30-40 jobs instead of the full 239 (Java projects: Develocity)
Replace post-merge runs with scheduled canary builds	In production	main gets a full run four times a day instead of after every merge
Use smaller runners for small jobs	In production	Bookkeeping jobs moved off the standard runner queue
Cap open pull requests per contributor	In production	210 PRs from 21 contributors closed in the first pass
Run external contributors' CI in their forks	Experimental	Expected to cut shared-runner use by about 40%
Self-hosted runners with RunsOn	Experimental	Expected to move at least 40% of the heaviest jobs off shared runners

Start with the basics¶

These changes take minutes and often recover a lot of capacity on their own. They come from Infra's GitHub Actions Recommended Practices. The aim is that each workflow runs for as little time as possible, and that as few of your project's workflows as possible run at the same time.

• Set timeouts. A hung job holds its runner for GitHub's default of 6 hours. Set timeout-minutes on every job, a little above how long it normally takes, and on long steps where useful.
• Cancel superseded pull request runs. When a contributor pushes a new commit, runs for the earlier commits keep going although nobody needs their results. A concurrency group per ref cancels them. Limit cancelling to pull requests, so builds of release branches always complete:

concurrency:
  group: ${{ github.workflow }}-${{ github.ref }}
  cancel-in-progress: ${{ github.event_name == 'pull_request' }}

• Restrict the push trigger to long-lived branches. If push has no branch filter and pull_request is also configured, every push to a branch with an open pull request runs the workflow twice. List the branches that need push builds. If your contributors rely on push builds of other branches, at least exclude Dependabot's branches, which are already covered by their pull requests:

on:
  push:
    branches:
      - main
      - 2.x
    # or, if you build every branch on push:
    # branches-ignore:
    #   - dependabot/**

• Fail fast in matrices. Use fail-fast: true so that one failing combination cancels the rest of the matrix, and keep max-parallel low.
• Stagger scheduled workflows. Most schedule triggers fire at the top of the hour, when the shared queue is busiest. Pick an odd minute, and add path filters to avoid redundant runs.
• Cache dependencies. Use dependency caching (see also the actions/cache tips) so that jobs do not download and build the same dependencies on every run.
• Lint your workflows. actionlint catches mistakes in workflow files before they cost a failed run.

Run only the tests a change needs¶

The largest saving usually comes from not running work a change cannot affect. A full Airflow CI run is 239 jobs, but most pull requests run 30-40 jobs or fewer. A first job, "selective checks", looks at which files the pull request changes (plus the event, the target branch and the labels) and computes the smallest set of jobs that still gives confidence in the change. Later jobs read their matrix from that job's outputs. A documentation fix gets the docs build, a change to one provider gets that provider's tests, and a change to core code or to CI itself still gets the full run.

How to start, from simplest to most capable:

• Use paths and paths-ignore filters so that whole workflows only run when relevant files change.
• Add a first job that inspects the changed files (git diff --name-only against the merge base) and writes a JSON matrix to its outputs. Later jobs use fromJSON() on it, and skip themselves when it is empty.
• Make the rules conservative: when in doubt, run more. Run the full suite in scheduled "canary" builds of the default branch, so anything a pull request skipped is still caught soon after merge.
• Let maintainers force a full run with a label (Airflow uses full tests needed).

Airflow reference: selective checks documentation, including the rules it applies and worked examples.

Java projects: use Develocity instead of custom rules¶

Maven and Gradle projects do not need to write their own selection logic. The ASF has a sponsored instance of Gradle Develocity at develocity.apache.org, available to all projects, and it works with both Maven and Gradle builds:

• Predictive Test Selection uses the history of your project's builds to pick the tests a change is likely to affect, and skips the rest. Clay Johnson of Gradle presented it at the July 2026 Infra Roundtable (notes and a link to slides on the page). A presentation at the March 2024 Roundtable covered Develocity's test features more broadly.
• The remote build cache lets a build reuse the outputs, including test results, of modules whose inputs have not changed since an earlier build, so CI only builds and tests what actually changed.
• Build scans show where a build spends its time and which tests are failing or flaky, so you know what to fix first.

Apache Ozone recently made its Maven test goals cacheable and enabled the Develocity remote build cache (HDDS-16940). To get started, follow the Project Onboarding Instructions for Develocity, and open an Infra Jira ticket if you run into problems.

Maven projects that do not use Develocity can still skip unchanged modules with the Apache Maven Build Cache Extension. Gradle's own build cache does the same for Gradle builds.

For other build systems, Apache Yetus can work out which modules a patch touches and test only those.

Replace post-merge runs with scheduled canary builds¶

Many workflows run the full CI suite again on every push to main, after the pull request has already been tested. On a busy repository that is one full run per merged pull request, often within minutes of each other, and each run tests code that its pull request run has already covered.

Instead, you can drop main from the push trigger and run the full suite on a schedule: a "canary" build. However many pull requests are merged in between, the canary runs at a fixed rate, and it is still the safety net for anything that selective tests skipped on a pull request. Add workflow_dispatch, so a maintainer can start a canary by hand, for example to check a fix.

on:
  schedule:
    # Full run on main four times a day, off the top of the hour.
    - cron: '58 1,7,13,19 * * *'
  pull_request:
  push:
    # Release branches change rarely, so they can still build on every push.
    branches:
      - 'v[0-9]+-[0-9]+-test'
  workflow_dispatch:

Things to plan for:

• Report canary failures. Nobody watches a scheduled run, so send its failures somewhere people will see them: a mailing list, a chat channel, or an issue. Airflow posts them to Slack.
• Finding the commit that broke main takes longer. A failing canary covers every pull request merged since the last green one. Run the canary often enough that this stays a short list, and use workflow_dispatch to test a suspect commit.
• Move post-merge jobs to the canary. Jobs that must run after a merge, such as publishing documentation, refreshing caches or updating constraint files, move to the scheduled run too. Run them only for schedule and workflow_dispatch events in the upstream repository, not in forks.
• With a merge queue, the push run is a duplicate. If your repository uses GitHub's merge queue, the queue has already tested the exact tree that lands on main, so a run on push repeats it.

Airflow does not build main on push at all. Its full-matrix canary runs four times a day, alternating between AMD and ARM runners, so each architecture runs twice a day, and release branches build on every push. Airflow reference: the on: triggers and comments in ci-amd.yml and ci-arm.yml, and apache/airflow#66348, which split the canary between the two architectures.

Use smaller runners for small jobs¶

Many jobs do very little work: computing a matrix, posting a notification, labelling or closing issues, checking a file name. They do not need a full ubuntu-latest (or ubuntu-24.04, ubuntu-22.04) virtual machine. Run them on runs-on: ubuntu-slim, a much smaller runner type that GitHub schedules from a separate pool, so these jobs start faster at peak times and leave the standard runners for jobs that need them.

ubuntu-slim has significant limits, so check a job against them before you move it:

• 1 vCPU and 5 GB of memory, and a small disk.
• A 15-minute maximum run time.
• It is an unprivileged container, not a VM: there is no Docker daemon, so the job cannot build images, start containers or use Docker container actions.
• A minimal toolset. actions/setup-* actions work, but the system Python is externally managed, so a bare pip install fails.

A good rule: if a job only moves metadata around, it belongs on ubuntu-slim. If it builds, runs containers, or runs a real test suite, keep it on a standard runner. See GitHub's GitHub-hosted runners reference for the current specification.

Airflow reference: apache/airflow#72852 moved its short CI bookkeeping jobs to ubuntu-slim. The runners section of its CI documentation explains which jobs go where.

Cap open pull requests per contributor¶

Every pull request costs a CI run on every push. Some contributors, often using AI tools, open dozens of low-value pull requests at once, and each one keeps drawing on the shared runners.

You can cap the number of pull requests a user without write access may have open in a repository with the creation_cap setting in .asf.yaml. This does not affect committers and other users with write access. See Pull request creation cap in the .asf.yaml reference for details.

github:
  pull_requests:
    creation_cap:
      enabled: true
      max_open_pull_requests: 5

Airflow agreed a cap of 5 by lazy consensus on its dev list. When it enabled the cap it closed the excess pull requests in a one-time pass: 210 pull requests from 21 contributors. Each closed pull request got a comment asking its author to choose which ones matter most and bring those back. Tell your contributors before you enable the cap, and document what they should do when they reach it.

Airflow references: apache/airflow#73469 (the cap), the contributor guide for the limit, and the announcement on the dev list.

Run external contributors' CI in their forks (experimental)¶

Status: experimental. Airflow has accepted this design and is starting an opt-in pilot. Findings and reusable recipes will be published here when the pilot ends.

A fork has its own GitHub Actions quota. When a pull request comes from a contributor without write access, its CI can run in the contributor's fork instead of on the ASF's shared runners. In Airflow's design ("your PR, your CI"):

• CI for a pull request from an external contributor runs in their fork. The pull request shows the fork run's status and links to it.
• The pull request stays in draft until its fork run passes, so maintainers mostly review pull requests that already pass.
• Committers and collaborators' pull requests keep using project CI as before.
• A passing fork run is trusted only when the pull request does not change what CI runs (.github/, the CI tooling and scripts). Such pull requests stay in draft until a maintainer reviews them and runs project CI with a label. Maintainers can use the same label on any pull request they want verified in project CI before merging.
• The gate never falls back to project CI when it cannot decide, and a pull request that edits the workflows to get around the gate is detected and closed.

Airflow expects this to cut its use of shared runners by about 40%.

Airflow references: AIP-120: Run CI for external contributions in contributor forks, the design and ADR in apache/airflow#73378, and the implementation, which starts with apache/airflow#74819 (the gate), #74820 (fork-mode selective checks) and #74822 (reconciler).

Self-hosted runners with RunsOn (experimental)¶

Status: experimental.

If your project has cloud credits, for example from a targeted donation, you can move its heaviest jobs off the shared runners. RunsOn starts an ephemeral EC2 instance in your project's own AWS account for each job, while GitHub Actions remains the orchestrator. A workflow requests one with a runs-on: label that names the size it needs, for example runs-on=${{ github.run_id }}/cpu=8/ram=32. There is no cluster to operate, both AMD and ARM instances are available, and RunsOn offers a free license to open source projects in exchange for a public acknowledgement. Apache Arrow, Apache DataFusion and ASF Infra already use it.

Self-hosted runners need Infra's involvement: read Self-hosted runners and open an Infra Jira ticket before you start.

In Airflow's design:

• Scheduled canary runs, and pushes to and pull requests from apache/airflow branches, run on RunsOn.
• Pull requests from forks stay on GitHub-hosted runners, with a read-only token and no secrets, as before. A committer can opt a fork pull request in with a label.
• When RunsOn or the budget is unavailable, a pull request can be sent back to GitHub-hosted runners with a label, and a repository variable moves every run back without a code change.
• Windows jobs stay on GitHub-hosted runners.
• Self-hosted runners do not become the default for all pull requests until the budget is known to support it.

Airflow expects this to move at least 40% of its heaviest jobs (the complete and full test runs) off the shared runners.

Securing self-hosted runners¶

A self-hosted runner gives every job that lands on it compute in your cloud account and whatever that account lets the instance reach. Address each item before you let pull requests run on self-hosted runners.

• The workflow's choice of runner is not a security boundary. A pull_request run uses the workflow files and scripts from the pull request itself. A pull request from a fork can change the logic your workflow uses to decide which jobs go to self-hosted runners, or put a RunsOn label in runs-on: directly and ask for any instance size. Treat that logic as a routing preference. Enforce the restriction outside the repository, in the RunsOn stack and the cloud account.
• Approval for fork workflows does not close the gap. Even "Require approval for all external contributors" exempts members of the apache GitHub organization (committers of any ASF project who linked their GitHub account) and the repository's collaborators. Ask Infra to confirm your repository's approval setting, but do not rely on it.
• Restrict what the instance can reach. Limit the instance role to the cache bucket prefix the jobs need, with no access to RunsOn's configuration bucket. Deny network traffic between runners in the security group, because CI jobs often bind unauthenticated local services on all interfaces. Disable SSH access, and isolate sticky disks between jobs (EnableStickyDiskIsolation).
• Cap cost in the stack, not just in alerts. Limit the instance types jobs can request and their run time (RunnerMaxRuntime). RunsOn has no instance cap of its own, and its daily budget setting (AppBudgetDailyUsd) only alerts, so use the account's EC2 vCPU quotas as the hard ceiling.
• Consider a separate stack for untrusted code. Running pull requests from forks on their own stack (in their own account, or with a role that can reach nothing beyond its own cache bucket) is stronger than sharing a stack with the jobs that hold secrets.
• Bind an opt-in label to the reviewed commit. A label that sends a fork pull request to self-hosted runners would otherwise also cover every later push. Airflow honours the label only while a committer's latest review approves the pull request's current head commit and no committer's request for changes stands. A re-run reuses the labels of its original event, so check the pull request's current labels through the API rather than the event payload.
• Make shared caches tamper-evident. Every job on the stack can write to the cache bucket, so a pull request's job could overwrite a cached artifact that a later job with secrets restores. Name cached blobs by their SHA-256 digest, store the digest through a store the job cannot forge (Airflow uses the ASF stash action, whose restore accepts only artifacts from the same run or branch), and verify the digest before extracting. A tampered blob then only causes a cache miss.
• Keep a way back. A repository variable that sends every run back to GitHub-hosted runners lets you switch off self-hosted runners quickly if the stack or the credits lapse.

RunsOn offers two modes. Fleet adds a GitHub runner group restricted to selected workflows, but GitHub matches those workflows by ref, so runs of a pull request (refs/pull/N/merge) never qualify, and Fleet then covers only canary and push runs. Airflow chose Flex on a hardened stack, as described above, which is also how Infra already runs RunsOn for pull requests in other ASF repositories.

Airflow references: AIP-118: Self-hosted runners for Apache Airflow CI and its appendix with evidence and implementation details, the AIP-119 discussion on cost guardrails, the implementation and its security review in apache/airflow#74889, and the earlier full-CI experiment in apache/airflow#74714. See also RunsOn's own documentation on security for public repositories.

Audit your workflows with Apache Magpie¶

Apache Magpie is a set of skills for the AI agent a maintainer already uses, built at the ASF for the work of running a project. Its repo-health skills let a PMC audit its repositories without writing anything to them: each skill reads the repository, reports what it found, and proposes changes that the maintainers decide on.

The workflow-optimization skill uses this page as its source. It reads one repository's workflows, .asf.yaml and Dependabot configuration, and a week of its GitHub Actions run history, and checks them against the techniques above. The result is a prioritised plan: what is broken, what runs more jobs than it needs to, what is quick hygiene, and what needs the community to agree first. Each finding names the file and line it is about and the run numbers behind it, and the plan suggests how to split the changes into pull requests.

Other repo-health skills cover related checks:

• workflow-security-audit scans workflows with zizmor for injection risks, excessive permissions, unpinned actions and unsafe triggers.
• ci-runner-audit finds retired runner labels, which leave jobs waiting for a runner that never comes.
• flaky-test-triage separates flaky tests from real failures in the run history, so re-runs stop eating runner time.
• Dependency and licence audits that check for vulnerable dependencies and licence compliance.

To try it, follow the installation instructions, install the repo-health skills with /plugin install magpie-repo-health@apache-magpie, and ask your agent to review your project's workflows, or run /magpie-repo-health:workflow-optimization.

Getting help¶

If you want advice on which of these fits your project, or you have optimizations of your own that worked, contact Infra via a Jira ticket or by email to users@infra.apache.org. We will add all proven recipes to this page.