FailTrace – Reproduce a failed GitHub Actions run locally from its URL

Hacker News by 14 min read 135x views
FailTrace – Reproduce a failed GitHub Actions run locally from its URL

Share Post

CI Latest release  MIT

Stop guessing why CI failed. Reproduce it locally.

FailTrace reproducing a unsuccessful stretchr/testify CI run in Docker

Paste a unsuccessful GitHub Actions run URL. FailTrace finds the job and stage that failed, matches them to the workflow document at that exact commit, pulls the nonaccomplishment out of the logs, and tells you what stands between you and reproducing it on your machine.

failtrace inspect https://github.com/owner/repo/actions/runs/123456789 # why did it fail? failtrace scheme https://github.com/owner/repo/actions/runs/123456789 # how would it be reproduced? failtrace repro https://github.com/owner/repo/actions/runs/123456789 # reproduce it in Docker

Status: first development. All three commands work. repro covers Linux jobs whose failing stage is a run: script. See the roadmap and limitations.

act runs your workflows locally, and it's awesome at that. FailTrace answers a distinct question: why did this particular run fail, and can I create it neglect the identical way here?

  • It starts from a unsuccessful run, not a workflow file. You get the exact commit, attempt, job, matrix combination, and failing step. Nothing to configure.
  • It focuses on the failure. The goal is to replay lone what the failing stage needs, not the whole workflow.
  • It gives you a verdict. You get a assessment now. Later you'll additionally get reproduced / not reproduced / flaky, affirmative what differed between CI and your machine.

Real output for a unsuccessful prometheus/prometheus run (trimmed):

✗ CI · prometheus/prometheus "remote/otlp: eliminate delta-to-cumulative conversion" pull_request · on codex/remove-otlp-deltatocumulative · @ cf7d324 · attempt 1 · failure ──────────────────────────────────────────────────────────── ✗ Job: Go tests (failure) Runner ubuntu-latest → ubuntu-26.04 (20261116.202.1) · runner 2.338.2 Workflow .github/workflows/ci.yml → jobs.test_go Steps ✓ 5 Run create GO_ONLY=1 SKIP_GOLANGCI_LINT=1 ✓ 6 Run go test ./tsdb/ -test.tsdb-isolation=false ✗ 7 Run create -C documentation/examples/remote_storage ← unsuccessful here ○ 8 Run create -C documentation/examples (skipped) Failed stage #7 Run create -C documentation/examples/remote_storage shell: sh -e {0} $ create -C documentation/examples/remote_storage Diagnosis records out of date (a inspect produced a git diff) · exit code 2 Log excerpt (10 lines about the archetypal error in "Run create -C documentation/examples/remote_storage") │ >> operating inspect for unused/missing packages in go.mod │ go mod tidy │ go: downloading github.com/oapi-codegen/nullable v1.1.0 │ diff --git a/documentation/examples/remote_storage/go.mod b/documentation/examples/remote_storage/go.mod │ --- a/documentation/examples/remote_storage/go.mod │ +++ b/documentation/examples/remote_storage/go.mod Reproduction notes ! job runs inner receptacle quay.io/prometheus/golang-builder:1.27-base 

The log ends alongside hundreds of lines of go.sum diff. FailTrace centres the excerpt on the archetypal error line so you see the cause, go mod tidy changing go.mod, alternatively of the noise following it.

macOS and Linux, using the instal script. It downloads the latest publish for your phase and verifies its SHA-256 checksum:

curl -fsSL https://raw.githubusercontent.com/Ri4dH/failtrace/main/scripts/install.sh | sh

It installs to /usr/local/bin if that's writable, alternatively to ~/.local/bin. Set FAILTRACE_INSTALL_DIR to choose, or FAILTRACE_VERSION=v0.1.0 to pin a release.

Prebuilt binaries for macOS, Linux and Windows (amd64 and arm64) are on the releases page, alongside a checksums.txt.

With Go 1.24+:

go instal github.com/Ri4dH/failtrace/cmd/failtrace@latest

From source:

git copy https://github.com/Ri4dH/failtrace cd failtrace go build -o bin/failtrace ./cmd/failtrace
# All unsuccessful jobs in the latest attempt of a run failtrace inspect https://github.com/owner/repo/actions/runs/123456789 # A sole job (the URL you get from clicking a job in the Actions UI) failtrace inspect https://github.com/owner/repo/actions/runs/123456789/job/987654321 # A particular attempt of a re-run workflow failtrace inspect https://github.com/owner/repo/actions/runs/123456789/attempts/2 # Show additional log context, or emit JSON for scripts and another tools failtrace inspect -n 60 <run-url> failtrace inspect --json <run-url> | jq '.failed_jobs[0].failure'
Flag Default Description
--json off Print the study as JSON
-n, --lines 25 Log lines to display from the unsuccessful step
--job all failed Inspect lone this job ID

Colour output turns off automatically whenever stdout is not a terminal, or whenever NO_COLOR is set.

FailTrace needs a GitHub token, equal for community repositories, since the logs API requires one. It looks in this order:

  1. GITHUB_TOKEN
  2. GH_TOKEN
  3. gh auth token (if you use the GitHub CLI, this already works)

Private repositories need a token that can peruse Actions on that repo. GitHub Enterprise Server URLs activity too: the API basis becomes https://<host>/api/v3.

  • Run context: workflow, event, branch, commit, attempt.

  • Job → workflow mapping: finds the job definition, including jobs named like test (ubuntu-latest, 1.22) or Lint ${{ matrix.tool }}, and recovers the matrix values.

  • Runner: the requested tag and the genuine runner depiction and version, from the log.

  • Failed step: its definition (run, uses, with, env, shell, operating directory). The command shown is the one printed in the log, alongside expressions already filled in.

  • Diagnosis: exit code, error annotations, failing test names (Go, pytest, Jest, Vitest, Cargo, RSpec), and a category:

    Category Example signal
    test --- FAIL: TestX, FAILED tests/x.py::test_y
    build error TS2304, undefined: foo, error[E0425]
    dependency npm ERR! code ERESOLVE, missing go.sum entry
    lint golangci-lint, eslint, ruff steps
    generated-diff a inspect that printed diff --git (tidy, codegen, formatters)
    timeout exceeded the maximum implementation time
    out-of-memory exit code 137, heap out of memory
  • Reproduction notes: item that makes a local repro harder. That method assistance containers, job containers, secrets the stage uses, reusable workflows, and macOS or Windows runners.

It additionally explains border cases alternatively of failing silently: runs that unsuccessful before any job started, dynamic workflows alongside no document in the repo, and logs that have expired.

failtrace scheme plant out how to reproduce the unsuccessful job locally and prints the plan. It doesn't run anything yet.

failtrace scheme https://github.com/owner/repo/actions/runs/123456789 failtrace scheme --job 987654321 <run-url> # choice a job whenever multiple failed failtrace scheme --json <run-url> # the complete plan, including all env var

Real output for a unsuccessful pytest-dev/pytest job (trimmed):

Reproduction scheme · pytest-dev/pytest Job build (ubuntu-py311) (jobs.build) Verdict PARTIAL: can run, but the warnings below may create it differ from CI Image ubuntu:26.04 linux/amd64 matches the runner depiction ubuntu-26.04 20261111.196.1 Commit c484767 Toolchains python 3.11.18 exact · asked for "3.11" · from actions/setup-python@6d4fd22 uv 0.13.8 exact · from astral-sh/setup-uv@11f9893 Steps 1 checkout Run actions/checkout@3d3c42e pytest-dev/pytest @ c484767 (merge of 58b8dae into 50234b4) 2 skip Download Package — downloads artifacts from another job 3 toolchain Set up Python 3.11 → instal python 3.11.18 4 composite Run pytest-dev/pytest/.github/actions/setup-tox@c484767 — composite action; its steps follow 5 toolchain Run astral-sh/setup-uv@11f9893 → instal uv 0.13.8 6 run $ if [ -n "$PYTHON_VERSION" ]; afterward (+4 lines) 7 run Test alongside safety ← unsuccessful in CI $ tox run -e py311-coverage --installpkg `find dist/*.tar.gz` shell: /usr/bin/bash --noprofile --norc -e -o pipefail {0} Issues ! [step 2] actions/download-artifact@9000827 downloads artifacts produced by another job or run; records it provides volition be missing locally i [step 1] CI tested the merge of PR caput 58b8dae into basis 50234b4 (merge commit c484767), not the PR branch alone; the scheme checks out that merge commit 

How the scheme is built:

  • The log is the origin of fact for what ran. Scripts, shells and surroundings arrive from the step headers GitHub prints, so ${{ matrix.* }} and another expressions are already filled in. Composite actions display their inner steps.
  • The exact commit CI tested. For drag requests that's a merge commit GitHub created, not your branch head. This is a average logic a nonaccomplishment "doesn't happen locally".
  • The exact toolchain versions. "3.11" in the workflow becomes 3.11.18, peruse from the runner's tool cache paths. When the log doesn't display a version, it says type records specified as go.mod (including toolchain), .nvmrc, .python-version, .tool-versions and packageManager in package.json.
  • The correct basis image. A job container: is used as is. Otherwise it picks the Ubuntu version matching the runner depiction the job really got.
  • What to skip and what to flag. Caches, uploads and safety reporters are skipped. Downloaded artifacts, assistance containers, secrets, unidentified actions, and tools used without a setup stage (they were preinstalled on the runner) are reported as issues.

The verdict is prepared (nothing known to be missing), partial (it can run, but several parts are approximated or missing) or blocked (the failing stage is an action, not a script, so it can't be replayed yet).

Supported setup actions: actions/setup-go, setup-node, setup-python, setup-java, setup-dotnet, ruby/setup-ruby, dtolnay/rust-toolchain, actions-rust-lang/setup-rust-toolchain, pnpm/action-setup, pnpm/setup, astral-sh/setup-uv, oven-sh/setup-bun and denoland/setup-deno.

failtrace repro builds the scheme and runs it in Docker, afterward compares the outcome alongside CI's failure.

failtrace repro <run-url> failtrace repro --keep <run-url> # keep the receptacle to prod about afterwards API_TOKEN=... failtrace repro --secret API_TOKEN <run-url> failtrace repro --toolchain node=20.11.1 <run-url> # inhabit in or override a toolchain failtrace repro --platform linux/arm64 <run-url> # run natively on Apple silicon (faster, small exact) failtrace repro --runs 10 <run-url> # is it flaky? failtrace repro --shell <run-url> # debug inner the receptacle afterwards failtrace repro --report repro.md <run-url> # Markdown study for an matter or PR

You need Docker (Docker Desktop, colima, OrbStack, …) and git.

Real output for a unsuccessful stretchr/testify run, anywhere go vet failed under Go oldstable. FailTrace resolved oldstable to the exact Go 1.26.9 build CI used, checked out the PR merge commit CI tested, and replayed the steps:

$ failtrace repro --platform linux/arm64 https://github.com/stretchr/testify/actions/runs/39247781971 Reproducing stretchr/testify · build (oldstable) · scheme ready ▶ Image failtrace-env:08414601e6c4aef4 (cached) ▶ Workspace stretchr/testify @ 9272105 ▶ [3/5] Run ./.ci.gogenerate.sh ✓ exit code 0 (17.7s) ▶ [4/5] Run ./.ci.gofmt.sh ✓ exit code 0 (7.4s) ▶ [5/5] Run ./.ci.govet.sh ← unsuccessful in CI │ assert/assertion_compare_test.go:85:13: non-constant format cord in call to (*testing.common).Errorf │ assert/assertion_compare_test.go:94:13: non-constant format cord in call to (*testing.common).Errorf ✗ exit code 1 (10.7s) ✓ Reproduced: the stage fails locally the identical way it unsuccessful in CI • the identical error: assert/assertion_compare_test.go:<n>:<n>: non-constant format cord in call to (*testing.common).Errorf CI Local Exit code 1 1 Failure lint / format inspect lint / format check Steps 4 passed · 1 unsuccessful (39s total) Image failtrace-env:08414601e6c4aef4 linux/arm64 Differences from CI - the depiction is plain ubuntu:24.04 alongside average runner packages; GitHub's runner depiction has far additional preinstalled - steps run as root; on GitHub's runners they run as the `runner` person alongside passwordless sudo - operating linux/arm64, but CI ran linux/amd64 

Without --platform, the job runs as linux/amd64 akin CI did. On an Apple silicon Mac that means emulation: the identical reproduction took concerning 9 minutes alternatively of 40 seconds, alongside the identical verdict.

What happens:

  1. Environment image. The plan's basis image, affirmative the packages all GitHub Ubuntu runner has (git, curl, build tools, python3, jq, sudo, …), affirmative the toolchains. Go, Node and Python are the same builds setup-go, setup-node and setup-python download, installed under /opt/hostedtoolcache where CI keeps them. uv, pnpm and Rust are installed from their releases. The depiction is tagged by a hash of its contents, so it's lone built formerly per environment.

  2. Workspace. The exact commit CI tested is fetched on your device the way actions/checkout does it, alongside the identical fetch depth, and mounted at the identical way as on the runner (/home/runner/work/<repo>/<repo>, or /__w/... for receptacle jobs). If GitHub has since replaced a pull request's merge commit, the merge is recreated locally.

  3. Services. Service containers commencement in the job container's network namespace, so localhost:5432 and postgres:5432 the two work, and FailTrace waits for their health checks.

  4. Steps. Run steps replay in command alongside the shell, surroundings and operating directory CI used. $GITHUB_ENV and $GITHUB_PATH transport complete between steps. The failing step's output streams live.

  5. Verdict.

    • reproduced: the identical tests failed, or the identical error or exit code.
    • different-failure
    • not-reproduced: the stage passed locally.
    • diverged: an before stage unsuccessful locally, so the failing stage never ran.
    • flaky: alongside --runs, the stage unsuccessful in several runs but not all.

    A catalog of known differences from CI comes alongside all verdict.

The setup steps run once, afterward the failing stage runs N times in the identical container, so repeats are cheap. If it fails lone several of the time, the verdict is flaky ("the stage unsuccessful in 3 of 10 runs"). Go's test cache is bypassed for these runs (GOFLAGS=-count=1). Otherwise go test would replay a cached pass and conceal the flakiness.

Debugging inner the container: --shell

When the run finishes, --shell opens a casing anywhere the failing stage ran: identical directory, environment, toolchains and services. It prints the command that re-runs the step:

▶ Shell inner the receptacle at stage 5 (Run ./.ci.govet.sh); category exit to finish re-run the stage with: attack -e /home/runner/work/_temp/failtrace/step-5.sh 

If an before stage diverged, the casing opens at that stage instead. The receptacle is removed whenever you exit, unless you continue --keep.

Sharing the result: --report FILE

--report repro.md writes a Markdown study to paste into an matter or drag request. It includes the verdict, a CI-vs-local table, environment, differences, all step's result, and the output of the failing step from the two sides. --report - prints it alternatively of the terminal summary. See docs/example-report.md, generated from the stretchr/testify run complete with --runs 3.

Safety. repro runs the repository's code, including code from a drag request, inner a receptacle on your machine. The receptacle sees lone the workspace, has no admission to your residence directory or the Docker socket, and gets no secrets unless you continue them alongside --secret NAME. The value is peruse from your environment, so it never appears in your casing former or the procedure list. Only reproduce runs whose code you're comfortable executing.

Limitations. These are reported alternatively of silently ignored:

  • A failing stage that is an act fairly than a manuscript can't be replayed (verdict blocked).
  • Windows and macOS jobs are lone approximated in Linux.
  • Steps run as root, anywhere CI uses the runner user.
  • The runner image's many preinstalled tools aren't all present.
  • linux/amd64 jobs are emulated on ARM machines.
  • Artifacts from another jobs aren't available.
  • inspect: diagnose a unsuccessful run from its URL
  • plan: rotate the unsuccessful job into a local reproduction plan: basis image, exact commit, toolchain versions, env, steps to replay
  • repro: run the scheme in Docker and difference the outcome alongside CI
  • Reports: --report writes a Markdown study alongside the verdict and the differences from CI
  • Flaky detection: --runs N repeats the failing stage and reports flaky
  • --shell: autumn into the receptacle at the failing step
  • Releases: prebuilt binaries for macOS, Linux and Windows, and a checksum-verifying instal script
  • Homebrew and gh extension: beer instal and gh failtrace <run-url>
  • Replaying average actions: translate actions specified as golangci-lint-action into the commands they run, so failures in them can be reproduced too
go test ./... # component tests (no network or Docker needed) go test -race ./... go vet ./... go test -tags integration ./internal/repro/ # runs genuine containers; needs Docker goreleaser publish --snapshot --clean # build publish archives locally, publish nothing

CI runs all of the complete on all drag request. To publish a release, shove a type tag (git tag v0.1.0 && git shove base v0.1.0); release.yml builds and uploads the binaries.

Project layout:

cmd/failtrace/ CLI admission point internal/cli/ commands and flags internal/runurl/ run URL parsing and validation internal/github/ minimal GitHub REST client and token lookup internal/joblog/ job log parsing, nonaccomplishment extraction and classification internal/workflow/ workflow YAML parsing, job/step/matrix matching internal/inspect/ puts it all together into a report internal/plan/ builds reproduction plans (image, commit, toolchains, env, steps) internal/toolchain/ resolves toolchains to the builds setup-* actions use internal/docker/ runs containers through the docker CLI internal/repro/ executes a scheme and compares the outcome alongside CI internal/report/ terminal rendering 

Issues and drag requests are welcome. The most helpful contributions correct now:

  • Failed runs that FailTrace gets wrong. Open an matter alongside the run URL and what you expected.
  • New nonaccomplishment signatures: test runners, bundle managers and compilers that aren't detected yet. These live in internal/joblog/analyze.go and need a test case.
  • More known actions: setup actions (internal/plan/toolchains.go) and actions that are harmless to skip locally (internal/plan/actions.go). Each one is a sole array entry.

MIT

Other Article Hacker News
↑
Close Right Ads
Close Left Ads