We've recovered a caller measurement to velocity up parallel Golang continuous integration workflows by taking advantage of the Golang build cache. Replacing GitHub's charismatic actions/setup-go action pinch a drop-in balanced trim our trial occupation runtimes by 69%. We're open-sourcing cloudx-io/setup-go (opens successful a caller tab) truthful you tin do the same.
GitHub's charismatic actions/setup-go measurement makes parallel jobs interfere pinch each other's performance, and it continuously loads old cache values. Backtesting successful our monorepo, which has the communal business of a fewer parallel Golang trial jobs (one for lints, 1 for tests, 1 for builds), suggests that 86% of the activity the default action does is wholly unnecessary.
If you negociate a moderately analyzable Go project, you tin expect akin capacity improvements; spot our CI measurement methodology aliases conscionable effort it for yourself.
We Care About Fast CI
We've been shipping a batch of caller products and features, and the gait astatine which we do it is really expanding complete time. This is nary mishap — we put heavy successful the devices and processes required to make this possible. At the halfway of each "software factory" are the trial suites and Continuous Integration (CI) workflows that guarantee codification changes won't break successful production. If our tests tally reliably, and quickly, connected each change, we tin build astatine awesome velocity without worrying astir breaking things for our customers.
This is important to us, truthful we measurement and put successful the velocity of our CI jobs. If you push codification to a CloudX repository, our extremity is that you get a clear reply arsenic to its acceptability — whether it builds, its tests pass, and it abides by our linter rules — wrong 90 seconds.
Speed tin beryllium achieved successful a number of ways, but astatine the extremity of the time if you want things to beryllium accelerated you person to make algorithmic improvements. We're already utilizing Warp Build (opens successful a caller tab) to tally our CI jobs connected fast, cost-efficient machines. As our trial suite has scaled pinch our merchandise aboveground area, we realized that actions/setup-go was not mounting america up for success.
How actions/setup-go fails for parallel jobs
GitHub's actions/setup-go (opens successful a caller tab) is the GitHub-encouraged measurement to instal and tally Go successful GitHub Actions. It uses actions/cache internally to prevention and reconstruct the section Go module cache and build cache directories. In principle, that should make downloaded module root codification and build/test artifacts from 1 occupation tally disposable to each the consequent occupation runs successful your repo. Here's the default actions/setup-go cache cardinal construction:
This cache cardinal is woefully incomplete: successful a emblematic merchandise nether progressive development, only a mini number of codification changes modify the target operating system, architecture, Go version, aliases go.mod files.
The first clip a occupation computes this hash key, it persists the final cache authorities to the GitHub cache service. Until the adjacent alteration that modifies 1 of those cardinal elements, every azygous CI run will load that first value. As you alteration your application, the restored spell build module archives from this first tally weaken — each consequent build does much activity from scratch. The restored spell trial outputs spell old too, truthful each consequent occupation reruns much tests. CI degrades until you update go.mod!
Moreover, aggregate parallel jobs moving actions/setup-go title to constitute different section cache states to the GitHub cache work — different because the last Go cache authorities connected a runner depends some connected the root codification and connected the commands run. For example, you mightiness tally abstracted lint and trial jobs successful parallel:
Both jobs resoluteness the aforesaid default cache key, past title to constitute its value. Suppose the lint occupation finishes first: it saves a worth without an updated trial cache state. Subsequent trial jobs will support utilizing that old worth until the cache cardinal changes, and truthful re-run tests unnecessarily.
Linting, building, and testing a codebase are perfect candidates for memoization: their outputs (linter messages, built binaries, and trial results respectively) should beryllium axenic functions of the root code. You tin shop outputs and reuse them alternatively than recomputing them, truthful agelong arsenic the inputs haven't changed.
Several parts of the modular Go toolchain prevention their outputs to the filesystem and cheque if they tin reuse an existing output alternatively of recomputing a caller 1 from scratch:
| Module cache | GOMODCACHE | $GOPATH/pkg/mod |
| Build cache | GOCACHE | ~/.cache/go-build |
| Test cache | GOCACHE | ~/.cache/go-build |
Go's module cache saves clip spent downloading root codification for your module dependencies, which you tin trigger explicitly pinch spell mod download but besides implicitly pinch spell build. There's thing mysterious here, conscionable root codification organized by the package identifiers successful your go.mod:
You trigger caller downloads erstwhile you alteration your go.mod, e.g. to adhd a caller dependency aliases upgrade an existing one.
Go's build cache and test cache are really located together successful the GOCACHE directory and stock a wide structure. Both build and trial processes hash their afloat inputs for usage arsenic a cache key. Those hashes are organized into subdirectories by prefix, and utilized arsenic filenames for the reusable process outputs:
Files pinch the suffix -d are information payloads, and the -a-suffixed files service arsenic indexes. Of course, build and trial processes output different information payloads:
- go build stores package archives, intermediates that are linked into a last binary.
- go trial stores stdout, stderr, and the last exit codification of the trial execution.
The Go trial runner spies connected the trial process, automatically detects what files it reads, and incorporates their contents arsenic inputs to the cache key.
The principles underlying these instrumentality caches are the same: they maximize deed rates by making keys of complete but minimal sets of dependencies, truthful misses only hap erstwhile perfectly necessary. Whenever there's a miss, the caller consequence is always persisted to the cache truthful early processes tin reuse it.
This useful brilliantly successful a azygous persistent filesystem, but CI runners don't person the use of a azygous persistent filesystem. In GitHub Actions, these toolchain caches are smuggled from 1 ephemeral runner to the adjacent by stowing them successful yet different cache — 1 pinch very different creation priorities.
The GitHub Actions cache
GitHub's guidelines actions/cache (opens successful a caller tab) conscionable knows keys and filepaths. You springiness GitHub's cache work a cardinal of your ain design. If the cache work recognizes the key, it loads the corresponding cached files into your runner; otherwise, it loads nothing. If and only if this primary-key lookup missed, actions/cache saves these files to the cache work aft your CI occupation completes.
Once you constitute an entity to the GitHub cache work nether a definite key, that key-value brace is immutable. Any consequent calls that would persist a different worth for that cardinal are rejected.
There is thing incorrect pinch immoderate of that. Indeed, actions/cache is indispensable, and it uses GitHub's cache entree restrictions (opens successful a caller tab) to forestall cache poisoning.
The difficult portion is picking bully keys.
Improving setup-go
cloudx-io/setup-go is efficaciously a drop-in replacement for actions/setup-go; here's why we really for illustration it for our web monorepo:
- We're happy to salary a premium to support engineers and coding agents unblocked. That intends parallelizable activity must tally successful parallel (even if this increases billed runner clip by repeating setup work), and we gladly salary a fewer bucks per period for other cache space.
- Our build, test, and lint workloads are overmuch faster erstwhile they tin reuse anterior cached values. If each our tests were wicked accelerated (maybe 1 time they will be!) aliases each our lint rules wimpy, we wouldn't sweat our GOCACHE deed rates.
Our main penetration is conscionable that the Go toolchain is really, really good; a bully CI caching strategy has to sphere that toolchain's astir important properties crossed tons of ephemeral runner instances, while moving wrong GitHub's constraints — i.e. still adapting actions/cache.
Let's revisit the important properties 1 by one.
They maximize deed rates by making keys of complete but minimal sets of dependencies, truthful misses only hap erstwhile perfectly necessary.
GitHub's default setup-go keying is incomplete because it doesn't seizure what a fixed occupation really does. That's why the trial and lint jobs successful the illustration supra title to constitute a single, partial cache entry.
cloudx-io/setup-go solves this by making the occupation personality (or immoderate arbitrary cache-key-prefix input) portion of the Actions cache key. The lint occupation and trial occupation prevention and reconstruct abstracted caches without conflicts.
Whenever there's a miss, the caller consequence is ever persisted to the cache truthful early processes tin reuse it.
GitHub's default setup-go only saves a caller cache introduction erstwhile go.mod changes, moreover though there's caller information written to the runner's section cache directories each clip you build aliases trial a caller type of your root code.
Instead of discarding that incremental effort, cloudx-io/setup-go writes a cache introduction every azygous time: the last constituent successful its cardinal is the GitHub Actions tally ID. The fully-qualified cache cardinal includes respective different elements to promote prefix-matching successful a git-aware way:
Measuring performance
Late past year, while we still utilized the default action, we encountered precisely the title information discussed above: our parallel lint occupation saved a Go cache without trial results, which slowed our trial jobs from a 76-second median runtime to an unacceptable 180-second median. Remember, this slowdown represents precisely zero value: the jobs slowed down to re-test logic wholly unchanged from the tally before.
Eliminating the title by separating caches for our various jobs instantly solved this problem: we introduced cloudx-io/setup-go, trial jobs resumed loading due caches, and the median occupation runtime fell to 41 seconds, a 69% improvement.
cloudx-io/setup-go instantly trim trial occupation runtimes by 69%.
GitHub Actions trial occupation durations from the CloudX monorepo main and characteristic branches.
The parallel lint occupation wins the shared cache-key race, redeeming a build-cache authorities that is very old for tests. Subsequent trial runs many times reconstruct that lint-shaped cache.
Median trial runtime falls from 131 seconds to 41 seconds, a 69% reduction.
Even if we lint and trial successful series, cloudx-io/setup-go would outperform the default because it saves an updated cache authorities aft each run. Using the GitHub default, the loaded cache grows progressively staler betwixt cardinal changes (go.mod changes). With our caller strategy, the loaded cache is always caller from the tally before; the trial tally for a perpetrate only exercises trial packages genuinely modified by that commit.
In aggregate, we hold for 86% less trial packages to tally now that we load a fresher cache. To tally the counterfactual comparison connected existent data, we took a series of 4,000 existent commits, calculated the action IDs for each snapshot's trial packages, and modeled cache-hit rates nether the aged and caller cardinal constructions.
86% of actions/setup-go trial runs are unnecessary.
Count of trial package runs for commits connected the CloudX monorepo main branch.
Show percent of full trial packages
Count of trial package runs complete consecutive commits0400Total trial package countCommit index
| 526,166 |
| −86%71,928 |
Of course, your mileage will alteration (according to really often you alteration go.mod). To beryllium transparent, we've seen 2 downsides to the switch, some because we prevention truthful galore much cache objects:
- Initially our cache blobs grew linearly pinch each run; yet they grew truthful ample that cache-load times became a awesome facet successful our wide CI time. This is an rumor coming successful the actions/setup-go default behaviour too: Go's cache doesn't prune itself; it grows until you clear it. We prevention the cache much often, truthful it grows faster. We solved this pinch automatic pruning.
- You whitethorn request an expanded GitHub Actions cache capacity. This is offset by making your jobs faster — runners measure by the infinitesimal — but locating the basal settings successful GitHub is simply a pain.
cloudx-io/setup-go (opens successful a caller tab) has been unchangeable internally since November of past year. We dream it saves your squad immoderate time, and we look guardant to proceeding what you think!
(opens successful a caller tab)Did you publication this while waiting for your CI to finish?
Explore careers astatine CloudX!
English (US) · 
Indonesian (ID) · 
