Baseline

Compare your CI to a baseline you name

Two sets of timings, matched job by job. The baseline is written out above every number.

Baseline compare
illustrative example
Baselinename it
Candidatename it

Baseline: GitHub-hosted ubuntu-latest. Compared with Larger runner, same commit, job by job by name, on 3 matched workloads, from the timings pasted. Not a prediction for other workloads.

GitHub-hosted ubuntu-latest
19m 27s
Larger runner, same commit
10m 26s
Difference
−46%
WorkloadPhases, baseline over candidateGitHub-hosted ubuntu-latestLarger runner, same commitDifference
build
5m 51s3m 04s−2m 47s −48%
test
10m 55s5m 43s−5m 12s −48%
lint
2m 41s1m 39s−1m 02s −39%
Phase, all matched jobsGitHub-hosted ubuntu-latestLarger runner, same commitDifference
Queue1m 03s28s−35s
Setup6s4s−2s
Cache restore1m 38s44s−54s
Install4m 30s2m 37s−1m 53s
Build3m 10s1m 40s−1m 30s
Test8m 05s4m 12s−3m 53s
Other55s41s−14s
Runs in your browser. Nothing is uploaded.

Make it like for like

A comparison is only as fair as its inputs. Keep everything but the runner the same.

Hold constantWhyHow to check
The commitDifferent code is a different workload.Same SHA in both runs' headers.
Cache stateA warm cache on one side decides the result.Both runs log a hit, or both a miss.
The verdictA faster run that tests less isn't faster.Same conclusion and test count.
Several runsOne run includes noise from a busy network or queue.Three runs each; compare medians.

Jobs are matched by name, case-insensitive. Jobs on one side only are listed and left out of the totals. See the method.