floot/endo

Fork 0

An ENDO_EXTRA setup can fail silently forever: the forgejo credential seeder has never run #29

New issue

Closed

opened 2026-09-01 13:35:49 +09:00 by floot · 2 comments

floot commented

2026-09-01 13:35:49 +09:00

Owner

A setup listed in ENDO_EXTRA that fails leaves no trace anyone on this host can read, and at least one of them has been failing silently for as long as the records go back.

What was observed

packages/space-nixos-admin/setup-forgejo-credential.js is in the daemon's ENDO_EXTRA and should provision forgejo-credential on every start. That name has never been bound in the host inventory. After deploying 046bfeb (which changed the setup to rotate an existing credential in place instead of re-minting it) and applying it as generation 42, the restart still left both forgejo-credential and forgejo-credential-https at { available: false, revoked: true }. No rotation happened.

Everything the setup needs is present:

ENDO_FORGEJO_FLOOT_PW and ENDO_FORGEJO_URL are both in the daemon process environment (read from /proc/<MainPID>/environ; the password is 21 bytes).
systemctl show -p Environment confirms the setup is in ENDO_EXTRA, and the file is present in the current release.
The deployed code really is the new revision: the slash-path change from the same commit is live.

So the setup either never runs or throws before reaching the credential, and there is no way to tell which.

Why it is invisible

packages/daemon/src/manager-node.js runs each extra in a try/catch and reports failure with console.error:

try {
  const namespace = await import(specifier);
  await namespace.main(host);
} catch (error) {
  console.error(`Endo extra: ${specifier} failed:`, error);
}

That goes to the journal, and the endo user is not in adm, wheel or systemd-journal, so journalctl -u endo-daemon.service answers "No journal files were opened due to insufficient permissions". An agent administering this host cannot read the one place the answer exists.

Wanted

Record each extra's outcome where a capability holder can read it: a status file under the state dir, or a name in the inventory holding { specifier, ok, error, at } per start. A setup that silently does nothing for weeks should not be possible.
Separately, give the daemon user journal read access, or tee daemon output to a file under the state dir.

Related: #21 (a revoked git credential only surfaces when a push fails). This is why the re-provisioning meant to prevent that never happened.

A setup listed in `ENDO_EXTRA` that fails leaves no trace anyone on this host can read, and at least one of them has been failing silently for as long as the records go back. ## What was observed `packages/space-nixos-admin/setup-forgejo-credential.js` is in the daemon's `ENDO_EXTRA` and should provision `forgejo-credential` on every start. That name has never been bound in the host inventory. After deploying 046bfeb (which changed the setup to rotate an existing credential in place instead of re-minting it) and applying it as generation 42, the restart still left both `forgejo-credential` and `forgejo-credential-https` at `{ available: false, revoked: true }`. No rotation happened. Everything the setup needs is present: - `ENDO_FORGEJO_FLOOT_PW` and `ENDO_FORGEJO_URL` are both in the daemon process environment (read from `/proc/<MainPID>/environ`; the password is 21 bytes). - `systemctl show -p Environment` confirms the setup is in `ENDO_EXTRA`, and the file is present in the current release. - The deployed code really is the new revision: the slash-path change from the same commit is live. So the setup either never runs or throws before reaching the credential, and there is no way to tell which. ## Why it is invisible `packages/daemon/src/manager-node.js` runs each extra in a try/catch and reports failure with `console.error`: ```js try { const namespace = await import(specifier); await namespace.main(host); } catch (error) { console.error(`Endo extra: ${specifier} failed:`, error); } ``` That goes to the journal, and the `endo` user is not in `adm`, `wheel` or `systemd-journal`, so `journalctl -u endo-daemon.service` answers "No journal files were opened due to insufficient permissions". An agent administering this host cannot read the one place the answer exists. ## Wanted - Record each extra's outcome where a capability holder can read it: a status file under the state dir, or a name in the inventory holding `{ specifier, ok, error, at }` per start. A setup that silently does nothing for weeks should not be possible. - Separately, give the daemon user journal read access, or tee daemon output to a file under the state dir. Related: #21 (a revoked git credential only surfaces when a push fails). This is why the re-provisioning meant to prevent that never happened.

floot referenced this issue from a commit

2026-09-01 13:43:03 +09:00

feat(daemon): record what each ENDO_EXTRA setup did

floot commented

2026-09-01 13:54:44 +09:00

Author

Owner

Answered, by the record this issue asked for

Generation 43 wrote the first extras.json, and the seeder is not failing at all:

{ "specifier": "file:///var/lib/endo/current/packages/space-nixos-admin/setup-forgejo-credential.js",
  "ok": true, "at": "2026-09-01T04:47:37.428Z" }

All fifteen extras report ok: true. Yet immediately after that boot, forgejo-credential in the machine-admin inventory is { available: false, revoked: true }, and a fetch through a remote holding it fails with "has been revoked".

Two different inventories

ENDO_EXTRA setups run against the daemon's ROOT host - E(endoBootstrap).host() in manager-node.js. The machine-admin session's endo is a different host:

root agent (statePath/root):        a9696f13ebf0b212...
machine-admin endo, identify(@self): b19f6f4ec984c118...

So the seeder provisions and rotates forgejo-credential in the root inventory, where nothing that pushes can see it, while the credentials the machine-admin host actually holds are never touched and stay revoked from the restart onward. That also explains the older half of the mystery: forgejo-credential has never appeared in the machine-admin inventory because it was never written there.

The rotate-in-place change in 046bfeb is correct but inert for the same reason - it rotates the root's copy.

Proposed fix

Grant the credential the way nixos-admin is already granted. floot-factory-setup.js has grantNixosAdmin(agent, factoryProfilePath), which locates a name in the root inventory and storeLocators it onto the factory profile. The same three lines for forgejo-credential would put a live, boot-rotated credential in the inventory the machine-admin host reads, and because a GitRemote holds the credential cap, remotes wired to it would then survive every restart without being rebuilt.

Worth noting the diagnostic worked exactly as intended: the record turned a four-hour guess into one file read.

## Answered, by the record this issue asked for Generation 43 wrote the first `extras.json`, and the seeder is not failing at all: ```json { "specifier": "file:///var/lib/endo/current/packages/space-nixos-admin/setup-forgejo-credential.js", "ok": true, "at": "2026-09-01T04:47:37.428Z" } ``` All fifteen extras report `ok: true`. Yet immediately after that boot, `forgejo-credential` in the machine-admin inventory is `{ available: false, revoked: true }`, and a fetch through a remote holding it fails with "has been revoked". ## Two different inventories ENDO_EXTRA setups run against the daemon's ROOT host - `E(endoBootstrap).host()` in `manager-node.js`. The machine-admin session's `endo` is a different host: ``` root agent (statePath/root): a9696f13ebf0b212... machine-admin endo, identify(@self): b19f6f4ec984c118... ``` So the seeder provisions and rotates `forgejo-credential` in the root inventory, where nothing that pushes can see it, while the credentials the machine-admin host actually holds are never touched and stay revoked from the restart onward. That also explains the older half of the mystery: `forgejo-credential` has never appeared in the machine-admin inventory because it was never written there. The rotate-in-place change in 046bfeb is correct but inert for the same reason - it rotates the root's copy. ## Proposed fix Grant the credential the way `nixos-admin` is already granted. `floot-factory-setup.js` has `grantNixosAdmin(agent, factoryProfilePath)`, which locates a name in the root inventory and `storeLocator`s it onto the factory profile. The same three lines for `forgejo-credential` would put a live, boot-rotated credential in the inventory the machine-admin host reads, and because a GitRemote holds the credential cap, remotes wired to it would then survive every restart without being rebuilt. Worth noting the diagnostic worked exactly as intended: the record turned a four-hour guess into one file read.

floot referenced this issue from a commit

2026-09-01 13:57:35 +09:00

fix(floot): grant the Forgejo credential to the factory host

floot commented

2026-09-01 14:05:22 +09:00

Author

Owner

Fixed in 949c3af (generation 43) and it paid for itself immediately. <stateDir>/extras.json now records every ENDO_EXTRA setup per start - specifier, ok, timestamp, error message on failure - and failing to write it never fails the daemon.

The first record answered the question this issue was opened to answer: the seeder was not failing, it was writing to the root inventory while the machine-admin session read its own. Fixed in 9dd1371 by granting the credential to the factory profile; generation 44 came up with a live credential and no manual rotation.

The second bullet - journal read access for the daemon user - is not done and is a separate concern; reopen or split it out if it is still wanted.

Fixed in 949c3af (generation 43) and it paid for itself immediately. `<stateDir>/extras.json` now records every ENDO_EXTRA setup per start - specifier, ok, timestamp, error message on failure - and failing to write it never fails the daemon. The first record answered the question this issue was opened to answer: the seeder was not failing, it was writing to the root inventory while the machine-admin session read its own. Fixed in 9dd1371 by granting the credential to the factory profile; generation 44 came up with a live credential and no manual rotation. The second bullet - journal read access for the daemon user - is not done and is a separate concern; reopen or split it out if it is still wanted.

floot closed this issue

2026-09-01 14:05:22 +09:00

floot referenced this issue from a commit

2026-09-08 22:41:41 +09:00

feat(daemon): record what each ENDO_EXTRA setup did

No labels

No milestone

No project

No assignees

1 participant

Notifications

Due date

The due date is invalid or out of range. Please use the format "yyyy-mm-dd".

No due date set.

Dependencies

No dependencies set.

Reference

floot/endo#29

No description provided.

Rows
Columns