Rollback leaves endo.rev owned by root, wedging every later agent deploy #6

New issue

Closed

opened 2026-08-25 17:06:25 +09:00 by floot · 1 comment

floot commented

2026-08-25 17:06:25 +09:00

Owner

After an apply fails its health check, the applier rewinds the NixOS config
checkout as root. The rewind recreates endo.rev as a ROOT-owned file inside a
checkout that is otherwise owned by the endo daemon user, and the daemon can
never write it again.

Every subsequent deploy then dies at its very first step:

stage-failed: EACCES: permission denied, open '/var/lib/endo/nixos-config/endo.rev'

Observed

Run r-6f886ae6d099 rolled back at 16:40:00 (+09:00). Immediately after:

/var/lib/endo/nixos-config          755 uid 999 gid 999
/var/lib/endo/nixos-config/endo.rev 644 uid 0   gid 0    mtime 07:40:00Z
/var/lib/endo/nixos-config/flake.nix 644 uid 999 gid 999

Only endo.rev changed hands, and its mtime is the exact second of the rewind.
The next run, r-4dfaca4613b8, failed at pin with the EACCES above.

Why it matters

This is self-inflicted and permanent. The rollback exists to leave the host in a
working state, and it does — for the running system. But it silently disables the
agent deploy path, so the recovery mechanism guarantees the next attempt cannot
even start. It needs a human with root to unwedge.

Same shape as the spool jam fixed earlier: a root-run recovery path leaving a
daemon-owned artifact behind as root's.

Expected

After any root-run rewind, the config checkout is owned by the daemon user, so
the next stageRev works.

Fix

Restore ownership at the end of the rewind, alongside the existing rollback
bookkeeping, rather than leaving it to whoever notices next.

After an apply fails its health check, the applier rewinds the NixOS config checkout as root. The rewind recreates `endo.rev` as a ROOT-owned file inside a checkout that is otherwise owned by the endo daemon user, and the daemon can never write it again. Every subsequent deploy then dies at its very first step: ``` stage-failed: EACCES: permission denied, open '/var/lib/endo/nixos-config/endo.rev' ``` ## Observed Run `r-6f886ae6d099` rolled back at 16:40:00 (+09:00). Immediately after: ``` /var/lib/endo/nixos-config 755 uid 999 gid 999 /var/lib/endo/nixos-config/endo.rev 644 uid 0 gid 0 mtime 07:40:00Z /var/lib/endo/nixos-config/flake.nix 644 uid 999 gid 999 ``` Only `endo.rev` changed hands, and its mtime is the exact second of the rewind. The next run, `r-4dfaca4613b8`, failed at `pin` with the EACCES above. ## Why it matters This is self-inflicted and permanent. The rollback exists to leave the host in a working state, and it does — for the running system. But it silently disables the agent deploy path, so the recovery mechanism guarantees the next attempt cannot even start. It needs a human with root to unwedge. Same shape as the spool jam fixed earlier: a root-run recovery path leaving a daemon-owned artifact behind as root's. ## Expected After any root-run rewind, the config checkout is owned by the daemon user, so the next `stageRev` works. ## Fix Restore ownership at the end of the rewind, alongside the existing rollback bookkeeping, rather than leaving it to whoever notices next.

floot commented

2026-08-25 17:07:45 +09:00

Author

Owner

Fixed and applied — generation 14, run r-75f2f77ad259.

rewind_checkout now hands the tree back after a successful rewind:

chown -R ${cfg.user}:${cfg.user} "$CONFIG_DIR"

The already-broken endo.rev was repaired separately (unlink + recreate as the
daemon user; content unchanged at 23f9b698…), since the running system still
carried the root-owned file from the earlier rollback.

Note this fixes the ownership leak, not the rollback itself — the rollback
behaved correctly. The remaining deploy gaps are floot/endo#3 (approval form has
no boolean control) and floot/endo#4 (release built during activation rather
than before approval).

Fixed and applied — generation 14, run `r-75f2f77ad259`. `rewind_checkout` now hands the tree back after a successful rewind: ```sh chown -R ${cfg.user}:${cfg.user} "$CONFIG_DIR" ``` The already-broken `endo.rev` was repaired separately (unlink + recreate as the daemon user; content unchanged at `23f9b698…`), since the running system still carried the root-owned file from the earlier rollback. Note this fixes the ownership leak, not the rollback itself — the rollback behaved correctly. The remaining deploy gaps are floot/endo#3 (approval form has no boolean control) and floot/endo#4 (release built during activation rather than before approval).

floot closed this issue

2026-08-25 17:07:45 +09:00

No labels

No milestone

No project

No assignees

1 participant

Notifications

Due date

The due date is invalid or out of range. Please use the format "yyyy-mm-dd".

No due date set.

Dependencies

No dependencies set.

Reference

floot/endo#6

No description provided.

Rows
Columns