floot/endo

Fork 0

Deploy workflow never builds the Endo release, so a cold revision always fails its first apply #4

New issue

Closed

opened 2026-08-25 16:51:04 +09:00 by floot · 2 comments

floot commented

2026-08-25 16:51:04 +09:00

Owner

The endo-release chart gates apply behind a build stage, which reads as
"the release has been built and validated". It has not been. That stage runs a
nixos-rebuild DRY RUN of the system closure, which validates the NixOS config
only — the Endo release is not built by Nix. endo-deploy builds it at
ACTIVATION time (git fetch, yarn install, package builds), inside the window
the applier gives the daemon to come back healthy.

wait_healthy in modules/endo-nixos-admin.nix allows 45 tries at 2s = 90s.
A first-time revision spends most of that window building.

Observed

Run r-6f886ae6d099, pinning 206e5d24, approved and applied:

[16:37:25] switch flake=…#endo-tokyo rev=4d00023
[16:39:03] pinned revision 206e5d24…
[16:39:03] revision 206e5d24… not in the local clone; fetching
[16:39:07] building release 206e5d24…
[16:39:08] yarn install
[16:39:21] building @endo/chat
[16:39:55] release 206e5d24… built
[16:39:55] switch failed; rolling back to be safe

The release finished building in the same second the health check gave up. The
auto-rollback worked correctly and restored generation 11 — the mechanism is
fine, the ordering is not.

Every earlier deploy passed because those revisions logged
already built; reusing. This was the first cold build, and it failed. Retrying
the same commit now succeeds precisely because the failed attempt left the
release cached — so the workflow is only reliable for revisions a previous
FAILED run happened to warm.

Expected

By the time the operator is asked to approve, the release exists and the apply
is an activation, not a build.

Actual

The operator approves a release that has never been built, and the build happens
during activation, competing with the health check that decides whether to roll
it back.

Suggested fix

Add a prebuild step and run it in the chart before await-approval.
endo-deploy-run has only bootstrap and deploy modes today; a prebuild
mode that fetches and builds the pinned revision WITHOUT activating it would
make the chart's build stage mean what its name says. Then apply reduces to
a reuse of an already-built release and comfortably fits the health window.

Widening wait_healthy would paper over it, but the honest fix is to stop doing
unbounded build work inside the rollback window.

Related: floot/endo#3 (the approval form has no way to answer its boolean).

The `endo-release` chart gates `apply` behind a `build` stage, which reads as "the release has been built and validated". It has not been. That stage runs a `nixos-rebuild` DRY RUN of the system closure, which validates the NixOS config only — the Endo release is not built by Nix. `endo-deploy` builds it at ACTIVATION time (`git fetch`, `yarn install`, package builds), inside the window the applier gives the daemon to come back healthy. `wait_healthy` in `modules/endo-nixos-admin.nix` allows 45 tries at 2s = 90s. A first-time revision spends most of that window building. ## Observed Run `r-6f886ae6d099`, pinning `206e5d24`, approved and applied: ``` [16:37:25] switch flake=…#endo-tokyo rev=4d00023 [16:39:03] pinned revision 206e5d24… [16:39:03] revision 206e5d24… not in the local clone; fetching [16:39:07] building release 206e5d24… [16:39:08] yarn install [16:39:21] building @endo/chat [16:39:55] release 206e5d24… built [16:39:55] switch failed; rolling back to be safe ``` The release finished building in the same second the health check gave up. The auto-rollback worked correctly and restored generation 11 — the mechanism is fine, the ordering is not. Every earlier deploy passed because those revisions logged `already built; reusing`. This was the first cold build, and it failed. Retrying the same commit now succeeds precisely because the failed attempt left the release cached — so the workflow is only reliable for revisions a previous FAILED run happened to warm. ## Expected By the time the operator is asked to approve, the release exists and the apply is an activation, not a build. ## Actual The operator approves a release that has never been built, and the build happens during activation, competing with the health check that decides whether to roll it back. ## Suggested fix Add a prebuild step and run it in the chart before `await-approval`. `endo-deploy-run` has only `bootstrap` and `deploy` modes today; a `prebuild` mode that fetches and builds the pinned revision WITHOUT activating it would make the chart's `build` stage mean what its name says. Then `apply` reduces to a reuse of an already-built release and comfortably fits the health window. Widening `wait_healthy` would paper over it, but the honest fix is to stop doing unbounded build work inside the rollback window. Related: floot/endo#3 (the approval form has no way to answer its boolean).

floot referenced this issue

2026-08-25 16:57:49 +09:00

Host services on slug subdomains, with the Chat preview as the first customer #5

floot referenced this issue

2026-08-25 17:07:45 +09:00

Rollback leaves endo.rev owned by root, wedging every later agent deploy #6

floot commented

2026-08-25 18:07:32 +09:00

Author

Owner

Correction: the cause in the original report is wrong.

I attributed the failure to wait_healthy running out. It did not. The applier
logs two different lines, and the one both failures produced was:

switch failed; rolling back to be safe

which endo-nixos-admin.nix only reaches when nixos-rebuild switch itself
returns non-zero. A health-check failure takes the other branch and logs
health check failed after switch; rolling back. The near-simultaneous
timestamps that suggested a photo finish were a coincidence of the build
finishing as activation gave up, not evidence of the window expiring.

Widening the health window to 5 minutes (generation 13) was therefore not the
fix. It is harmless and still reasonable, but it did not address this.

What still holds

The correlation with a cold build is real and now has three data points:

rev	release state at apply	result
`206e5d24` (r-6f886ae6d099)	cold — built during activation	switch failed, rolled back
`206e5d24` (r-2bace3d08cfa)	cached	applied
`f6506885` (r-bc5da1aa5ded)	cold — built during activation	switch failed, rolled back
`f6506885` (r-2e39c8f7b8d6)	cached	applied

Same revision, same config, opposite outcomes — the only difference is whether
the release was already built. So building the release BEFORE the apply, which
is what this issue asks for, remains the right fix. It is now the difference
between a deploy that works and one that always fails on its first attempt.

What is still unknown

WHY the switch returns non-zero. endo-deploy-bootstrap logs success and exits
0, and endo-daemon shows Result=success, so the failing unit is not obvious
from outside. Reading the systemd journal needs root and the daemon runs as its
own user, so the actual error is not visible from the agent side at all — a
failed deploy currently leaves no readable reason for anyone without root.
Capturing the nixos-rebuild output into apply.log is a prerequisite for
diagnosing this properly.

**Correction: the cause in the original report is wrong.** I attributed the failure to `wait_healthy` running out. It did not. The applier logs two different lines, and the one both failures produced was: ``` switch failed; rolling back to be safe ``` which `endo-nixos-admin.nix` only reaches when `nixos-rebuild switch` itself returns non-zero. A health-check failure takes the other branch and logs `health check failed after switch; rolling back`. The near-simultaneous timestamps that suggested a photo finish were a coincidence of the build finishing as activation gave up, not evidence of the window expiring. Widening the health window to 5 minutes (generation 13) was therefore not the fix. It is harmless and still reasonable, but it did not address this. ## What still holds The correlation with a cold build is real and now has three data points: | rev | release state at apply | result | | --- | --- | --- | | `206e5d24` (r-6f886ae6d099) | cold — built during activation | switch failed, rolled back | | `206e5d24` (r-2bace3d08cfa) | cached | applied | | `f6506885` (r-bc5da1aa5ded) | cold — built during activation | switch failed, rolled back | | `f6506885` (r-2e39c8f7b8d6) | cached | applied | Same revision, same config, opposite outcomes — the only difference is whether the release was already built. So building the release BEFORE the apply, which is what this issue asks for, remains the right fix. It is now the difference between a deploy that works and one that always fails on its first attempt. ## What is still unknown WHY the switch returns non-zero. `endo-deploy-bootstrap` logs success and exits 0, and `endo-daemon` shows `Result=success`, so the failing unit is not obvious from outside. Reading the systemd journal needs root and the daemon runs as its own user, so the actual error is not visible from the agent side at all — a failed deploy currently leaves no readable reason for anyone without root. Capturing the `nixos-rebuild` output into `apply.log` is a prerequisite for diagnosing this properly.

floot changed title from ~~Deploy workflow never builds the Endo release, so a cold revision loses the health-check race~~ to Deploy workflow never builds the Endo release, so a cold revision always fails its first apply

2026-08-25 18:07:33 +09:00

floot referenced this issue from a commit

2026-08-25 18:47:58 +09:00

feat(nixos-admin,floot): build the release before asking for approval

floot commented

2026-08-25 20:11:12 +09:00

Author

Owner

Fixed, deployed, and exercised end to end.

Run r-6a7ef1069be3 was the first through the v2 chart, on a revision that had
never been built:

[20:02:5x] Prebuild requested: 856714fa…      <- prebuild stage, no generation touched
[20:03:57] Prebuilt release 856714fa…
[20:09:45] pinned revision 856714fa…          <- apply, after approval
[20:09:45] release 856714fa… already built; reusing
[20:09:45] Pinned release 856714fa… is current

The build happened in the prebuild stage with current untouched, and the apply
was a reuse in the same second. Generation 21, applied first try.

What landed

Host side (generation 19): a prebuild action on the deploy spool — fetch and
build a revision without flipping current or restarting the daemon.
Endo side (7fa3be2f): prebuildRev on the nixos-admin caplet, and the
endo-release chart at v2 — pin → prebuild → build → await-approval.

A revision that will not build now costs an unpin, not an operator's attention
and not a rolled-back generation. Before this, a new revision could only be
deployed by proposing it twice and letting the first attempt fail.

Correction carried forward

The original report blamed wait_healthy; that was wrong (see the correction
comment above) — the switch itself was failing. Widening the health window to 5
minutes in generation 13 stays, but it was not the fix. WHY a cold-build
activation made nixos-rebuild switch return non-zero is still unknown, and now
largely moot: the build no longer happens during activation. Generations 17 and
18 added tee-ing of the switch and build output into apply.log, so if it
resurfaces there will be something to read.

Fixed, deployed, and exercised end to end. Run `r-6a7ef1069be3` was the first through the v2 chart, on a revision that had never been built: ``` [20:02:5x] Prebuild requested: 856714fa… <- prebuild stage, no generation touched [20:03:57] Prebuilt release 856714fa… [20:09:45] pinned revision 856714fa… <- apply, after approval [20:09:45] release 856714fa… already built; reusing [20:09:45] Pinned release 856714fa… is current ``` The build happened in the prebuild stage with `current` untouched, and the apply was a reuse in the same second. Generation 21, applied first try. ## What landed - Host side (generation 19): a `prebuild` action on the deploy spool — fetch and build a revision without flipping `current` or restarting the daemon. - Endo side (`7fa3be2f`): `prebuildRev` on the nixos-admin caplet, and the `endo-release` chart at v2 — `pin → prebuild → build → await-approval`. A revision that will not build now costs an unpin, not an operator's attention and not a rolled-back generation. Before this, a new revision could only be deployed by proposing it twice and letting the first attempt fail. ## Correction carried forward The original report blamed `wait_healthy`; that was wrong (see the correction comment above) — the switch itself was failing. Widening the health window to 5 minutes in generation 13 stays, but it was not the fix. WHY a cold-build activation made `nixos-rebuild switch` return non-zero is still unknown, and now largely moot: the build no longer happens during activation. Generations 17 and 18 added tee-ing of the switch and build output into `apply.log`, so if it resurfaces there will be something to read.

floot closed this issue

2026-08-25 20:11:12 +09:00

floot referenced this issue

2026-08-26 12:27:30 +09:00

Host services on slug subdomains, with the Chat preview as the first customer #5

floot referenced this issue from a commit

2026-09-08 22:41:41 +09:00

design(agents): agent-assigned pet name as the monitor handle (per #234 review)

floot referenced this issue from a commit

2026-09-08 22:41:41 +09:00

feat(nixos-admin,floot): build the release before asking for approval