Decommission legacy single-user containers

arne commented

2026-05-09 23:24:59 +02:00

Owner

What to build

After ~1 week stability window post-migration (#3): permanently delete the legacy single-user containers from fismen.

ssh fismen 'incus delete arne-msg marcus-msg tarald-msg'

This is HITL — destructive operation, operator decision on timing.

Acceptance criteria

At least 7 days have passed since cutover
No issues observed in logs of the new daemon during the stability window
All three legacy containers removed via incus delete
Caddy config has no remaining references to legacy container IPs (sanity check; should already be true from #3)
Closeout note added to DEPLOY.md (or wherever the migration runbook lives)

Blocked by

#3

## What to build After ~1 week stability window post-migration (#3): permanently delete the legacy single-user containers from fismen. ```sh ssh fismen 'incus delete arne-msg marcus-msg tarald-msg' ``` This is HITL — destructive operation, operator decision on timing. ## Acceptance criteria - [ ] At least 7 days have passed since cutover - [ ] No issues observed in logs of the new daemon during the stability window - [ ] All three legacy containers removed via `incus delete` - [ ] Caddy config has no remaining references to legacy container IPs (sanity check; should already be true from #3) - [ ] Closeout note added to DEPLOY.md (or wherever the migration runbook lives) ## Blocked by - #3

arne added the

needs-triage

label

2026-05-09 23:24:59 +02:00

arne referenced this issue

2026-05-09 23:33:00 +02:00

Production migration: arne/marcus/tarald → one container #3

arne commented

2026-05-09 23:38:55 +02:00

Author

Owner

This was generated by AI during triage.

Ready-for-Human Brief

Category: enhancement
Summary: After a stability window of at least 7 days following the migration in #3, permanently delete the legacy single-user containers (arne-msg, marcus-msg, tarald-msg) from fismen.

Why this is human-only:
The operation is destructive and irreversible — incus delete removes the containers and their disks. The "stability window" criterion is a human judgment call: the operator reads daemon logs over the preceding week and decides whether the new multi-tenant deployment has been stable enough to give up the rollback artifact. There is no automated signal that's safe to delegate; an AFK agent cannot meaningfully assess "anything weird in the logs?" The cost of getting this wrong is high (no quick rollback path) and the cost of waiting longer is essentially zero (stopped containers consume only disk).

Blocked by: #3 (cutover must have happened, with the legacy containers stopped-but-not-deleted as rollback artifacts).

Procedure:

Confirm at least 7 days have passed since the cutover in #3.
Spot-check journalctl -u posta (or equivalent) on the new posta container for the past week. Look for unexpected restarts, panics, or per-identity errors. If anything looks off, postpone — there is no time pressure.
Confirm the Caddyfile on fismen has no remaining references to legacy container IPs (this should already be the case from #3 step 8 — this is a sanity check, not a fix).

Delete the legacy containers:

ssh fismen 'incus delete arne-msg marcus-msg tarald-msg'

Add a closeout note to DEPLOY.md (or wherever the migration runbook from #3 ended up) recording the deletion date and confirming the rollback artifacts are gone.

Acceptance criteria:

At least 7 days have passed since the cutover.
A spot-check of new-daemon logs over the stability window surfaced no concerning issues.
All three legacy containers (arne-msg, marcus-msg, tarald-msg) are removed via incus delete.
The Caddyfile on fismen contains no remaining references to legacy container IPs.
The migration runbook (DEPLOY.md or equivalent) has a closeout note recording the deletion.

Out of scope:

Any changes to posta-server code. This is purely a deploy-side cleanup.
Removing other legacy resources beyond the three named containers (e.g. old Incus profiles, snapshots) — handle as needed but not as part of closing this issue.
Building automation around the stability check. The decision is judgment-based and one-off.
Adjusting the stability window. 7 days is the minimum; the operator may wait longer at their discretion.

> *This was generated by AI during triage.* ## Ready-for-Human Brief **Category:** enhancement **Summary:** After a stability window of at least 7 days following the migration in #3, permanently delete the legacy single-user containers (`arne-msg`, `marcus-msg`, `tarald-msg`) from fismen. **Why this is human-only:** The operation is destructive and irreversible — `incus delete` removes the containers and their disks. The "stability window" criterion is a human judgment call: the operator reads daemon logs over the preceding week and decides whether the new multi-tenant deployment has been stable enough to give up the rollback artifact. There is no automated signal that's safe to delegate; an AFK agent cannot meaningfully assess "anything weird in the logs?" The cost of getting this wrong is high (no quick rollback path) and the cost of waiting longer is essentially zero (stopped containers consume only disk). **Blocked by:** #3 (cutover must have happened, with the legacy containers stopped-but-not-deleted as rollback artifacts). **Procedure:** 1. Confirm at least 7 days have passed since the cutover in #3. 2. Spot-check `journalctl -u posta` (or equivalent) on the new `posta` container for the past week. Look for unexpected restarts, panics, or per-identity errors. If anything looks off, postpone — there is no time pressure. 3. Confirm the Caddyfile on fismen has no remaining references to legacy container IPs (this should already be the case from #3 step 8 — this is a sanity check, not a fix). 4. Delete the legacy containers: ```sh ssh fismen 'incus delete arne-msg marcus-msg tarald-msg' ``` 5. Add a closeout note to `DEPLOY.md` (or wherever the migration runbook from #3 ended up) recording the deletion date and confirming the rollback artifacts are gone. **Acceptance criteria:** - [ ] At least 7 days have passed since the cutover. - [ ] A spot-check of new-daemon logs over the stability window surfaced no concerning issues. - [ ] All three legacy containers (`arne-msg`, `marcus-msg`, `tarald-msg`) are removed via `incus delete`. - [ ] The Caddyfile on fismen contains no remaining references to legacy container IPs. - [ ] The migration runbook (DEPLOY.md or equivalent) has a closeout note recording the deletion. **Out of scope:** - Any changes to `posta-server` code. This is purely a deploy-side cleanup. - Removing other legacy resources beyond the three named containers (e.g. old Incus profiles, snapshots) — handle as needed but not as part of closing this issue. - Building automation around the stability check. The decision is judgment-based and one-off. - Adjusting the stability window. 7 days is the minimum; the operator may wait longer at their discretion.

arne added

enhancement

ready-for-human

and removed

needs-triage

labels

2026-05-09 23:38:55 +02:00

arne referenced this issue from a commit

2026-05-09 23:58:02 +02:00

Multi-tenant rollout: daemon, identity CLI, invite flow, setup page

arne referenced this issue

2026-05-09 23:58:42 +02:00

Multi-tenant rollout: daemon, identity CLI, invite flow, setup page #7

arne referenced this issue

2026-05-10 00:47:55 +02:00

Production migration: arne/marcus/tarald → one container #3

arne commented

2026-05-10 00:50:41 +02:00

Author

Owner

This was generated by AI during cleanup assistance.

Cleanup complete — 2026-05-10

The three legacy containers (arne-msg, marcus-msg, tarald-msg) were incus delete'd on fismen, same day as the cutover in #3. The 7-day stability window in the original brief was waived — operator decision after verifying the new multi-tenant daemon end-to-end and confirming all per-identity data already lives in the new posta container.

State after cleanup:

incus list shows no *-msg posta containers; only posta (.168), posta-web, and unrelated msg-web (the historical web frontend container) remain.
Caddyfile has no remaining references to legacy IPs (already true since the cutover; pre-cutover backup preserved at /etc/caddy/Caddyfile.bak.posta-multitenant-cutover-20260510-004551).
All three public URLs still 200 with their original keys after the delete.
DEPLOY.md rollback section updated: no live rollback target — recovery path now requires rebuilding from the legacy binary at commit 793e9ed plus the per-identity data still in posta:/var/lib/posta/<slug>/.

Acceptance criteria:

All three legacy containers removed via incus delete (forced, since they had been auto-restarted by Incus default autostart between cutover and cleanup)
Caddy config has no remaining references to legacy container IPs
Closeout note added to DEPLOY.md
[~] At least 7 days have passed since cutover — waived by operator
[~] No issues observed in logs of the new daemon during the stability window — waived; same-day cleanup

Closing.

> *This was generated by AI during cleanup assistance.* ## Cleanup complete — 2026-05-10 The three legacy containers (`arne-msg`, `marcus-msg`, `tarald-msg`) were `incus delete`'d on fismen, same day as the cutover in #3. The 7-day stability window in the original brief was waived — operator decision after verifying the new multi-tenant daemon end-to-end and confirming all per-identity data already lives in the new `posta` container. **State after cleanup:** - `incus list` shows no `*-msg` posta containers; only `posta` (.168), `posta-web`, and unrelated `msg-web` (the historical web frontend container) remain. - Caddyfile has no remaining references to legacy IPs (already true since the cutover; pre-cutover backup preserved at `/etc/caddy/Caddyfile.bak.posta-multitenant-cutover-20260510-004551`). - All three public URLs still 200 with their original keys after the delete. - `DEPLOY.md` rollback section updated: no live rollback target — recovery path now requires rebuilding from the legacy binary at commit `793e9ed` plus the per-identity data still in `posta:/var/lib/posta/<slug>/`. **Acceptance criteria:** - [x] All three legacy containers removed via `incus delete` (forced, since they had been auto-restarted by Incus default autostart between cutover and cleanup) - [x] Caddy config has no remaining references to legacy container IPs - [x] Closeout note added to DEPLOY.md - [~] At least 7 days have passed since cutover — **waived by operator** - [~] No issues observed in logs of the new daemon during the stability window — **waived; same-day cleanup** Closing.

arne closed this issue

2026-05-10 00:50:41 +02:00

Rows
Columns

Decommission legacy single-user containers #6

What to build

Acceptance criteria

Blocked by

Ready-for-Human Brief

Cleanup complete — 2026-05-10