Release experiments
These experiments run release tools against disposable Git repositories and a real local npm registry. They show what remains after a partial publication and what each tool does on retry. The separate coverage, test results, and benchmarks pages describe the automated suite.
Measured on 2026-09-25 at commit 374e6469e43c, by the release that published this site.
This release ran 25 cells against yohimik/dispat-alpine:1.11.0-rc.6: 3 experiments and 4 release tools, the same fixture each time. Of dispat's 10 cells, 10 hold every expectation. The other tools' cells are records rather than expectations, and their counts describe those tools. Of dispat's 8 measured catch-ups, 8 finish in one run of one command with no manual step.
The protocols cover three failures: a registry rejects one package, a colleague pushes while the release is running, or a consumer fails after its provider has published. Each experiment records the local checkout, remote Git refs, and registry versions before making a recovery claim.
The fixture has six npm packages. cli, ui, and api depend on core; theme and docs depend on ui.
All six start at version 1.0.0, tagged and published. The original publication and push-race protocols use a minor
change to core with tilde dependency ranges. The propagation protocol uses a patch-compatible correction so it can
separate explicit propagation intent from a dependency range that needs updating.
The base fixture gives every package one build script. The propagation cells run it where each tool documents a
build: dispat's build stage, nx's release.version.preVersionCommand, and the prepack script that lerna publish
and changeset publish run for every package they pack. A failed build is therefore one fault seen through four tools.
The publication and push-race cells run no build under lerna, nx or changesets. The baseline tags are annotated,
because git describe ignores a lightweight tag and a tool that reads its previous release that way would otherwise
find no release at all and treat every package as changed.
By default, each cell runs the binary from yohimik/dispat-alpine:<version>. A prepublication run can supply a candidate
executable and an explicit identity, which is recorded in its verdict. A local Verdaccio registry sits behind a proxy
that can reject uploads for one named package while forwarding other requests. These runs use no public registry,
and every cell has its own registry state.
Records first, verdicts second
A cell records what a tool left behind in the three places a release writes: the local clone, the origin, and the registry. After every step of the protocol, those three are read and joined into one state per package.
For dispat, some of those readings are expectations, and a cell that misses one fails the release that ran it. For lerna, nx and changesets the same readings are recorded either way. A row below that reads 3/5 is a description of that tool under this fault, not a failure of the run and not a claim that the tool is broken: the expectations are dispat's own, written from what dispat promises, and holding another tool to a promise it never made would be a comparison of nothing.
A cell whose injected fault never fired recorded no answer to that fault, so the harness fails the cell rather than publishing its row.
By experiment
midrelease (clean)
| Tool | Steps | Expectations | Catch-up | Outcome | Final state |
|---|---|---|---|---|---|
changesets | version=0 commit=0 publish=0 push=1 rebase=0 push2=0 status=0 | 3/5 | not measured | the release commit is on origin/main after recovery; every released package is consistent after recovery | api=1.0.1/dangling cli=1.0.1/dangling core=1.1.0/dangling ui=1.0.1/dangling |
dispat | release=0 status=0 rerun=0 | 12/12 | not measured | every expectation holds | api=1.0.2/consistent cli=1.0.1/consistent core=1.1.0/consistent ui=1.0.1/consistent |
lerna | version=1 publish=0 rebase=0 push=0 changed=0 | 1/5 | not measured | version exited 0; the release commit is on origin/main after recovery; every released package is consistent after recovery; the next plan is the colleague's change and its dependents alone | api=1.0.1/dangling cli=1.0.1/dangling core=1.1.0/dangling docs=1.0.1/dangling theme=1.0.1/dangling ui=1.0.1/dangling |
nx | release=1 recover=0 release2=0 publish=0 push=0 next=0 | 2/6 | not measured | release exited 0; the release run left a commit and tags behind; the release commit was pushed by the tool; the next plan is the colleague's change and its dependents alone | api=1.0.1/consistent cli=1.0.1/consistent core=1.1.0/consistent docs=1.0.1/consistent theme=1.0.1/consistent ui=1.0.1/consistent |
midrelease (conflict)
| Tool | Steps | Expectations | Catch-up | Outcome | Final state |
|---|---|---|---|---|---|
changesets | version=0 commit=0 publish=0 push=1 rebase=1 resolve=0 push2=0 status=0 | 3/5 | not measured | the release commit is on origin/main after recovery; every released package is consistent after recovery | api=1.0.1/dangling cli=1.0.1/dangling core=1.1.0/dangling ui=1.0.1/dangling |
dispat | release=0 status=0 rerun=0 | 14/14 | not measured | every expectation holds | api=1.0.1/consistent cli=1.0.1/consistent core=1.1.1/consistent ui=1.0.1/consistent |
lerna | version=1 publish=0 rebase=1 resolve=0 push=0 changed=0 | 2/5 | not measured | version exited 0; the release commit is on origin/main after recovery; every released package is consistent after recovery | api=1.0.1/dangling cli=1.0.1/dangling core=1.1.0/dangling docs=1.0.1/dangling theme=1.0.1/dangling ui=1.0.1/dangling |
nx | release=1 recover=0 release2=0 publish=0 push=0 next=0 | 2/6 | not measured | release exited 0; the release run left a commit and tags behind; the release commit was pushed by the tool; the next plan is the colleague's change and its dependents alone | api=1.0.1/consistent cli=1.0.1/consistent core=1.1.0/consistent docs=1.0.1/consistent theme=1.0.1/consistent ui=1.0.1/consistent |
orphan
| Tool | Steps | Expectations | Catch-up | Outcome | Final state |
|---|---|---|---|---|---|
changesets | version=0 commit=0 publish=1 recovery=0 push=0 status=0 | 5/5 | 1 run, 2 commands, 1 manual | every expectation holds | api=1.0.1/consistent cli=1.0.1/consistent core=1.1.0/consistent ui=1.0.1/consistent |
dispat | release1=1 release2=0 status=3 | 8/8 | 1 run, 1 command | every expectation holds | api=1.0.1/consistent cli=1.0.1/consistent core=1.1.0/consistent ui=1.0.1/consistent |
lerna | version=0 publish=1 recovery=1 cleanup=0 recovery2=0 changed=1 | 3/5 | 2 runs, 3 commands, 1 manual | the refused package carries no tag for an unpublished version; the recovery needed no manual cleanup | api=1.0.1/consistent cli=1.0.1/consistent core=1.1.0/consistent docs=1.0.1/consistent theme=1.0.1/consistent ui=1.0.1/consistent |
nx | version=0 publish=1 recovery=0 push=0 next=0 | 4/5 | 1 run, 2 commands, 1 manual | the refused package carries no tag for an unpublished version | api=1.0.1/consistent cli=1.0.1/consistent core=1.1.0/consistent docs=1.0.1/consistent theme=1.0.1/consistent ui=1.0.1/consistent |
propagation (build)
| Tool | Steps | Expectations | Catch-up | Outcome | Final state |
|---|---|---|---|---|---|
changesets | version=0 commit=0 publish=1 retry-plan=0 recovery=0 push=0 final-plan=0 | 7/9 | 1 run, 2 commands, 1 manual | the plan before the catch-up named the owed consumer; the catch-up took no manual step | api=1.0.1/consistent cli=1.0.1/consistent core=1.0.1/consistent ui=1.0.1/consistent |
dispat | release1=1 retry-plan=0 release2=0 final-plan=3 | 18/18 | 1 run, 1 command | every expectation holds | api=1.0.1/consistent cli=1.0.1/consistent core=1.0.1/consistent ui=1.0.1/consistent |
lerna | version=0 publish=42 retry-plan=1 recovery=1 cleanup=0 recovery2=0 final-plan=1 | 5/9 | 2 runs, 3 commands, 1 manual | the provider shipped in the first run; the plan before the catch-up named the owed consumer; the catch-up took one run; the catch-up took no manual step | api=1.0.1/consistent cli=1.0.1/consistent core=1.0.1/consistent docs=1.0.1/consistent theme=1.0.1/consistent ui=1.0.1/consistent |
nx | release1=1 retry-plan=0 recovery=0 push=0 final-plan=0 | 7/9 | 1 run, 2 commands, 1 manual | the provider shipped in the first run; the catch-up took no manual step | api=1.0.1/consistent cli=1.0.1/consistent core=1.0.1/consistent docs=1.0.1/consistent theme=1.0.1/consistent ui=1.0.1/consistent |
propagation (build-distributed)
| Tool | Steps | Expectations | Catch-up | Outcome | Final state |
|---|---|---|---|---|---|
dispat | release1=1 retry-plan=0 release2=0 final-plan=3 | 19/19 | 1 run, 1 command | every expectation holds | api=1.0.1/consistent cli=1.0.1/consistent core=1.0.1/consistent ui=1.0.1/consistent |
propagation (deferred-build)
| Tool | Steps | Expectations | Catch-up | Outcome | Final state |
|---|---|---|---|---|---|
dispat | release1=1 cleanup=0 provider-plan=0 release2=0 retry-plan=0 release3=0 final-plan=3 | 14/14 | 1 run, 1 command | every expectation holds | api=1.0.1/consistent cli=1.0.1/consistent core=1.0.1/consistent ui=1.0.1/consistent |
propagation (deferred)
| Tool | Steps | Expectations | Catch-up | Outcome | Final state |
|---|---|---|---|---|---|
dispat | release1=1 cleanup=0 provider-plan=0 provider-commit=0 release2=0 retry-plan=0 release3=0 final-plan=3 | 15/15 | 1 run, 1 command | every expectation holds | api=1.0.1/consistent cli=1.0.2/consistent core=1.0.1/consistent ui=1.0.1/consistent |
propagation (deselected)
| Tool | Steps | Expectations | Catch-up | Outcome | Final state |
|---|---|---|---|---|---|
dispat | release1=0 retry-plan=0 release2=0 final-plan=3 | 14/14 | 1 run, 1 command | every expectation holds | api=1.0.1/consistent cli=1.0.1/consistent core=1.0.1/consistent ui=1.0.1/consistent |
propagation (held)
| Tool | Steps | Expectations | Catch-up | Outcome | Final state |
|---|---|---|---|---|---|
dispat | release1=0 resume=0 retry-plan=0 release2=0 final-plan=3 | 15/15 | 1 run, 1 command | every expectation holds | api=1.0.1/consistent cli=1.0.1/consistent core=1.0.1/consistent ui=1.0.1/consistent |
propagation (publish)
| Tool | Steps | Expectations | Catch-up | Outcome | Final state |
|---|---|---|---|---|---|
changesets | version=0 commit=0 publish=1 retry-plan=0 recovery=0 push=0 final-plan=0 | 7/9 | 1 run, 2 commands, 1 manual | the plan before the catch-up named the owed consumer; the catch-up took no manual step | api=1.0.1/consistent cli=1.0.1/consistent core=1.0.1/consistent ui=1.0.1/consistent |
dispat | release1=1 retry-plan=0 release2=0 final-plan=3 | 18/18 | 1 run, 1 command | every expectation holds | api=1.0.1/consistent cli=1.0.1/consistent core=1.0.1/consistent ui=1.0.1/consistent |
lerna | version=0 publish=1 retry-plan=1 recovery=1 cleanup=0 recovery2=0 final-plan=1 | 6/9 | 2 runs, 3 commands, 1 manual | the plan before the catch-up named the owed consumer; the catch-up took one run; the catch-up took no manual step | api=1.0.1/consistent cli=1.0.1/consistent core=1.0.1/consistent docs=1.0.1/consistent theme=1.0.1/consistent ui=1.0.1/consistent |
nx | release1=1 retry-plan=0 recovery=0 push=0 final-plan=0 | 7/9 | 1 run, 2 commands, 1 manual | the plan before the catch-up named the owed consumer; the catch-up took no manual step | api=1.0.1/consistent cli=1.0.1/consistent core=1.0.1/consistent docs=1.0.1/consistent theme=1.0.1/consistent ui=1.0.1/consistent |
The final state column is the answer per package at the end of the run, in this vocabulary:
| State | Meaning |
|---|---|
consistent | the registry's version is tagged on origin and the tag is reachable from main |
orphan | a tag names a version the registry does not hold |
unpushed | the tag exists in the local clone only |
dangling | the tag is on origin but outside main's ancestry |
unrecorded | the registry serves a version no tag names |
baseline | nothing beyond the fixture's 1.0.0, so the row is left out |
dangling is what a rebase after a refused push produces: the release commit is rewritten, the tags stay on the
original, and the next plan of a tag-driven tool sees everything as changed again. orphan is what tagging before
publishing produces: the tag says a version shipped that nobody can install.
Each reading also records the manifest the checkout holds at that moment, its version and its dependency ranges. The registry says what shipped and the tags say what was recorded; a range a release reconciled, or a version rewrite it rolled back, appears in neither.
The build and publish propagation cells start with a patch to core that remains inside the consumers' ~1.0.0
dependency range, so semver alone asks for no consumer release. dispat receives fix(core)^: correct reader, whose
caret is explicit propagation intent. lerna and nx receive the plain fix(core): correct reader and decide for
themselves: nothing is forced on their side, and what their version step selects is part of the record. changesets
receives explicit patch entries for core and its three direct consumers, the closest way to request the same release
set with that tool.
The release sets differ. dispat releases core and its three declared consumers, writing each consumer's range up to
the provider it shipped with. lerna and nx release all six, because they bump every transitive dependent of a changed
package, theme and docs included, although ~1.0.0 already accepted core@1.0.1.
The fault is armed before the first run, and each tool runs its own documented release with nothing published by hand:
dispat release; lerna version and then lerna publish from-git; nx release; changeset version, a commit and
then changeset publish. The build scenario fails cli's build script, and the publish scenario lets the build
succeed and has the registry refuse cli's upload. Where each tool builds decides what its first run publishes.
dispat's fixture sets isBuildWaitingPublish: true on core, which holds every consumer's build until the provider is
published, and the protocol reads that order back out of the run's own log. lerna runs every package's prepack
before it publishes any, and nx runs its pre-version command before it versions, so under the build fault neither
publishes anything in its first run. changesets publishes the missing packages side by side, so core, ui and api
publish while cli fails.
After the fault is removed, no commit is made and no new release intent is supplied, and each tool takes its own
documented recovery. dispat runs the same command again. lerna runs lerna publish from-package, which publishes
whatever the registry is missing; it first refuses the working tree its own failed publish left rewritten, so the
protocol checks the manifests out by hand and runs it again. nx runs nx release publish for what its first run
versioned, or nx release again when it versioned nothing, and the release commit nx leaves on the clone is pushed by
hand. changesets runs changeset publish again, and the operator pushes what it tagged.
Five more cells are dispat's alone. build-distributed repeats the build fault with every build on a worker node,
which the release reaches through the repository being released: each command names it with --worker build-a and no
endpoint. Its record matches the local cell's, and its log names the worker that ran cli's build. In deselected the
first run is dispat release --package core,ui,api, so cli sits the provider's release out. In held, cli is held
by a Release-As: none commit while core ships, and the operator resumes it with Release-As: auto before the
catch-up. In each, the next full run has to find what cli is owed from the provider's release alone.
The deferred cell tests a further edge: cli has no build command and publishes its own fix after core fails, then
the operator commits once and releases core, ui and api while leaving cli out. With no new source commit, the
next full plan must catch up cli from the provider's durable publication, and the run after that must be empty. The
earlier cli tag already contains the propagation commit, so the pending commit window alone cannot reveal this debt.
The deferred-build companion keeps cli's actual build, and the fault proxy holds the provider's refusal until
cli's build output exists, so cli has always built over the provider's planned version when the provider fails.
dispat withholds that artifact with W194, and the catch-up rebuilds and publishes the consumer's own pending fix
against the version that really shipped.
The colleague's push lands at a point in the tool's own sequence rather than on a timer. A shim placed first on PATH
for the duration of the release command records every git call the tool makes and fires the colleague's commit and push
once, right before the first git push that is neither the release lock's nor a commit. The interleaving is therefore
the same every time and the same for every tool, and where in the release it falls is part of the record: lerna commits
and tags and then pushes both, nx pushes before it commits or tags, changesets makes no push of its own so the push is
the operator's, and dispat publishes first and pushes last.
The comparison this measures against the documented behaviour of each tool is in beside other release tools.
Reading the Catch-up column
The Catch-up column says what finishing a release took once its fault was removed, counted from the protocol's own
steps. Every step is typed: one of the tool's own release commands, a step done by hand such as a checkout, a commit
or a push, or a read-only question such as dispat status or lerna changed, which is never counted. A run is an
uninterrupted pass of the tool's own commands, and a step done by hand or a command that failed ends it.
| Reading | Meaning |
|---|---|
1 run, 1 command | one of the tool's own commands finished the release |
1 run, 2 commands, 1 manual | one pass of the tool's commands and one step done by hand |
2 runs, 3 commands, 1 manual | a failed command or a step done by hand interrupted the tool, and it ran again |
none needed | the catch-up ran nothing, because the release had already converged |
not measured | the protocol measures no catch-up, as the push-race cells do |
A reading ends in did not converge when the release did not end settled. A release converged when every package is
consistent or at its baseline, the local clone is clean and level with its origin, and the tool's own next plan is
empty: dispat status --require-release finds nothing to release, lerna changed lists no package,
nx release --dry-run writes no new version, and changeset status lists no package to bump.
The propagation cells ask every tool the same questions about it: whether the plan before the catch-up named the owed consumer, whether the catch-up took one run and no manual step, whether the consumer shipped at the version first planned, whether the provider was published once, and whether the release converged.
Reproducing
Every cell is one container, and the whole campaign is one command from the repository root:
EXPERIMENTS_DISPAT_VERSION=1.7.1 dispat exec experiments --for pkg:experiments --in pkg:experiments
dispat exec summary --for pkg:experiments --in pkg:experiments
One cell at a time, for re-recording a single protocol:
EXPERIMENT=midrelease TOOL=dispat SCENARIO=conflict \
dispat exec experiment --for pkg:experiments --in pkg:experiments
| Variable | What it does |
|---|---|
EXPERIMENTS_DISPAT_VERSION | the released version to run against; the newest published image when unset |
EXPERIMENTS_DISPAT_ARTIFACT | an unpublished candidate executable to copy into the experiment image |
EXPERIMENTS_DISPAT_ID | required immutable identity recorded when a candidate executable is supplied |
EXPERIMENTS_RESULTS | where the records land; coverage/experiments at the repository root by default |
EXPERIMENTS_JOBS | how many cells run at once, four by default |
EXPERIMENTS_EXPECT | 0 records a dispat cell without gating on its verdict |
Each cell leaves a folder with the transcript, every step's output, one observation per step, every failed
expectation's own output, the registry's logs, the git calls the tool made, and verdict.json. The harness itself, the
fixture, the observer and the fault proxy are documented in
tests/experiments/README.md.
The release workflow carries forward a completed campaign when the published native image and harness are identical.
The docs package's beforeBuild hook checks the image id, exact cell list, every verdict and observation stream, and
every dispat verdict before reuse. A new binary or protocol, or a missing, corrupt, partial or failed record, reruns
every campaign cell. This page is built from the validated records. The
Experiments workflow uses the same generated cell list on
demand, for any released version.
What this is for
These are not benchmarks and they are not a scoreboard. They exist because the claims dispat makes about recovery are claims about failure, and a claim about failure that has only ever been tested against a stub is a claim about the stub. Publishing before tagging, re-running rather than querying a registry, joining a branch that moved instead of refusing it: each of those is a sentence in the documentation, and each of these cells is that sentence executed against a registry and a remote that really do refuse.
The compared tools are here for the same reason a control is: a result that only ever ran against one tool cannot say whether what it measured was the tool or the fixture. Running four tools through one fixture with one fault says which differences are the tools'.