Open Records #

What is left before each open record can close: each Proposed ADR accepted, each planned or in-progress change closed, each open incident closed. Each bar counts the items its record lists as left and as done, so it measures what the record says, and it falls when an item is added. Each line links to its items further down this page; each heading there links to the record.

The items are read from the records themselves. An ADR lists them under To accept and a change under To close, as a checklist; an incident’s are its action and follow-up rows. A record that is open but lists nothing is marked ⚠ no list.

Legend: ✅ Done | 🔄 Under way | ⏳ Left

Decisions to accept

Changes to close

Incidents to close

Every outstanding item

ADR-0042: The Mobile Factory’s OpenBao Is Sealed by Shamir Shares, Opened for a Work Session, and Rebuilt From the Vault

• ⏳ V12, rebuild from the vault, passed: not run.
• ✅ §1 and §2 built, the Shamir seal and work sessions, by CHG-0053, 2026-11-08.
• ✅ §3 built, by CHG-0052.
• ✅ §4 built, by CHG-0036.
• ✅ V13, sealed, passed: the API’s half in CHG-0052, the plays’ in CHG-0053, 2026-11-08.
• ✅ V14, shares and custody, passed: every pair proven, no unseal material on the kit, CHG-0053, 2026-11-08.

↑ Decisions to accept

ADR-0039: The Mobile Factory’s Tenant CAs Serve Tenants at Every Site, and Each Site Keeps Only Its Own Substrate CAs

• ⏳ §4 and §5 built, by CHG-0034.
• ⏳ V1, ESP32 trust, passed: in part, three ESP32s holding only the Root connected by name to a Deevnet Pi.
• ⏳ V2, one CA for every site, passed: in part, on EdS’s home Pi and its workload console.
• ⏳ V3, imported keys, passed: in part, the key-identifier check is not recorded.
• ⏳ V6, device admission, passed: not run.
• ✅ §2 built: the tenant CAs re-keyed under the Mobile Site CA, by CHG-0036.
• ✅ §3 built: devices hold only the Deevnet Root, shown by three ESP32s in CHG-0054.

↑ Decisions to accept

ADR-0033: Code Is the State, for the Substrate and for Every Tenant

• ⏳ Device secrets moved into tenant code, by CHG-0040.
• ⏳ A tenant rebuilt from its repository with no restore.
• ⏳ Open question 1 answered: re-admission at scale.
• ⏳ Open question 2 closed: the Tenant Device CA’s key, built by CHG-0036, once its rebuild drill (V12) has run.

↑ Decisions to accept

ADR-0026: The State Store Becomes Object Storage, Defined by a Contract, With State and Tenant Buckets as Two Classes

• ⏳ An acceptance test: none is stated yet (no What would settle it).
• ⏳ Built by a change record: none planned yet.
• ⏳ Open question 1 answered: the size of the state reserve.
• ⏳ Open question 3 answered: bucket lifecycle rules.
• ⏳ Open question 4 answered: whether a tenant bucket can be read publicly.
• ✅ Open question 2 answered: where the replica lives, by ADR-0033 (no replica).

↑ Decisions to accept

ADR-0025: Keycloak Is the Identity Provider, With a Realm per Tenant and the Rebuild Path Kept Clear of It

• ⏳ An acceptance test: none is stated yet (no What would settle it).
• ⏳ Built by a change record: none planned yet.
• ⏳ Open question 1 answered: Keycloak Organizations instead of realms.
• ⏳ Open question 2 answered: SSH certificate lifetime and principals.
• ⏳ Open question 4 answered: tenant realm limits.
• ✅ Open question 3 answered: backup target, by ADR-0033 (no replica).

↑ Decisions to accept

ADR-0023: Metrics Are Pulled From Management, Stored Beside the Logs, and Alerted On Per Tenant

• ⏳ An acceptance test: none is stated yet (no What would settle it).
• ⏳ Built by a change record: none planned yet.
• ⏳ Open question 1 answered: who watches the watcher.
• ⏳ Open question 2 answered: delivering the VMID map to vmagent.
• ⏳ Open question 3 answered: off-site notifications.
• ⏳ Open question 4 answered: tenant limits.
• ⏳ Open question 5 answered: whether vmalert writes recording rules back to (index, 0).

↑ Decisions to accept

ADR-0021: Tenants Keep Runtime Secrets in Their Own OpenBao Namespace

• ⏳ An acceptance test: none is stated yet (no What would settle it).
• ⏳ Built by a change record: none planned yet.
• ⏳ Open question 1 answered: namespace naming and depth.
• ⏳ Open question 2 answered: re-delivering workload identity after a rebuild.
• ⏳ Open question 3 answered: paths per workload or per tenant.
• ⏳ Open question 4 answered: the read client.
• ⏳ Open question 5 answered: whether the tenant’s age recipients become an API concern.

↑ Decisions to accept

CHG-0066: The Management Hypervisor Moves to Proxmox VE 9

• ⏳ Step 1: latest 8.4, systemd-boot meta-package removed, grub-efi-amd64 installed
• ⏳ Step 2: VMs 100 and 104 backed up; the node’s configuration on the Builder
• ⏳ Step 3: Trixie repositories; packages downloaded, the router up throughout
• ⏳ Step 4: guests stopped; apt dist-upgrade done
• ⏳ Step 5: rebooted onto 9; the network and the guests back
• ⏳ Step 6: DeevnetAutomation holds VM.GuestAgent.Audit, not VM.Monitor
• ⏳ Step 7: sources modernized
• ⏳ Verification: a template build and a proxmox_vm run pass
• ⏳ The Software Catalog lists Proxmox VE 9 on Debian 13 for the node

↑ Changes to close

CHG-0064: The Inventory Vault Under a New Password

• ⏳ Step 2: rekey merged (inventory pull request #100)
• ⏳ Verification: a device backup sealed with the new password
• ✅ Step 1: all 14 files rekeyed

↑ Changes to close

CHG-0057: Build the Home Site

• ⏳ Step 1: a Home Site CA and a Home Substrate CA
• ⏳ Step 2: the home inventory: hosts, VLANs, firewall policy and Wi-Fi
• ⏳ Step 3: the core router’s DNS, DHCP and certificate
• ⏳ Step 4: the Pi’s VLAN, firewall, Wi-Fi and names in home’s DNS
• ⏳ Verification: a Deevnet Pi is served at home and its devices connect to it

↑ Changes to close

CHG-0043: A Local Package Mirror

• ⏳ Step 1: updates and the hypervisors’ repositories mirrored to the artifact server
• ⏳ Step 2: the base roles point each host at the local mirror
• ⏳ Step 3: a management-plane VM rebuilt with the internet link down
• ⏳ Step 4: the standard’s §5.4 restored to cover provisioning
• ⏳ Verification: the offline rebuild passes

↑ Changes to close

CHG-0042: Two Undeclared VMs Moved Off the Management Hypervisor

• ⏳ Step 1: VM 100 checked for anything the substrate still depends on
• ⏳ Step 2: both VMs backed up, off the host, and each restore checked
• ⏳ Step 3: both brought up at their destination
• ⏳ Step 4: both removed from dv02hyp001p01, and Build a Hypervisor updated
• ⏳ Verification: both run, or restore, at their destination, and neither is on the management hypervisor

↑ Changes to close

CHG-0041: The Artifact Server Over HTTPS

• ⏳ Step 1: HTTPS on the artifact server, from a Substrate CA certificate
• ⏳ Step 2: installs write the Root CA into the trust store, then fetch over HTTPS
• ⏳ Step 3: the hypervisor bootstrap checks the script’s hash before running it
• ⏳ Step 4: plain HTTP serves the network-boot paths only
• ⏳ Verification: a fresh template build and a network-boot install complete with plain HTTP limited to the boot payload

↑ Changes to close

CHG-0040: Device Secrets in Tenant Code

• ⏳ Step 1: the API and provider accept a supplied secret on create and restore
• ⏳ Step 2: tdemo, the reference tenant, supplies its secrets from an encrypted file
• ⏳ Step 3: eds and mabell migrated, with no secret changed and devices connected throughout
• ⏳ Step 4: the rebuild drill: a tenant rebuilt from its repository, its devices connecting unchanged
• ⏳ Verification: the recovery chart no longer lists “devices need re-provisioning” for a lost provisioning VM

↑ Changes to close

CHG-0039: Backup to an Attached USB Drive

• ⏳ The other three tenants’ plans, run from where their credentials are
• ⏳ The second drive, a USB flash drive
• ✅ Step 1: the drives, on one USB flash drive
• ✅ Step 2: the backup role writes encrypted files to the drive
• ✅ Step 3: a restore rehearsed into a rebuilt provisioning VM: the registry, the state bucket and one tenant, 2026-10-07
• ✅ Step 4: the docs
• ✅ The job takes backups unattended: 29 by the timer alone, 2026-10-08 to 2026-11-08

↑ Changes to close

CHG-0038: The Reconcile Restores Everything

• ⏳ Step 1: the API’s reconcile steps, built and tested off the site
• ⏳ Step 2: the provider expects a changed dashboard password, tagged and published
• ⏳ Step 3: the API’s own OPNsense and Grafana credentials, vaulted
• ⏳ Step 4: deployed, and make reconcile --all reconciles every tenant ok
• ⏳ Step 5: the recovery docs updated
• ⏳ Verification: after a runtime service is rebuilt, one make reconcile --all restores every tenant’s infrastructure

↑ Changes to close

CHG-0037: Substrate Builds Off OpenBao

• ⏳ Step 5: the ISO published for every built release, and an empty-shell mode in proxmox_vm
• ⏳ Verification: builds succeed with OpenBao stopped
• ✅ Step 1: pve-creds reads the vault only, with the same output, and a template builds with it, 2026-11-08
• ✅ Step 2: the build path removed from OpenBao, 2026-11-12
• ✅ Step 3: the automation tokens narrowed to custom Proxmox roles per node, 2026-11-12
• ✅ Step 4: packer-prov@pve and the TerraformProv role removed, 2026-11-13

↑ Changes to close

CHG-0034: Device Certificates Through the Deevnet API

• ⏳ Step 1: the API signs a registered device’s request with the Tenant Device CA and records its key fingerprint
• ⏳ Step 2: a device certificate resource in the provider
• ⏳ Step 3: the broker accepts Tenant Device CA certificates, checked against registrations and fingerprints (V6)
• ⏳ Step 4: one real device on a certificate, its key made on the device
• ⏳ Verification: a device’s certificate can’t reach another tenant’s topics, and V1 and V6 of the validation plan pass

↑ Changes to close

CHG-0009: Access Switch Omada Adoption

• ⏳ Step 1: the missing networks created in the controller, VLAN only
• ⏳ Step 2: the switch’s Inform URL pointed at the controller, and the switch pending there
• ⏳ Step 3: adopted, and every port’s VLANs re-applied and verified against inventory
• ⏳ Step 4: ownership handed over: switch_management: omada, and switch-vlans.yml refuses
• ⏳ Step 5: what adoption kept recorded, port by port

↑ Changes to close

INC-0005: Core Router Hard Hang During a Provisioning VM Rebuild

• ⏳ Install Realtek’s vendor driver on the router, as its own change record (INC-0004, Follow-up 3 and corrective action 4). Two hard hangs now rest on it (Open)
• ⏳ Enable a crash dump device on the router before the next bulk transfer across it, so a third hang leaves evidence (INC-0004, preventive action 2) (Open)
• ⏳ Check the router’s firewall rules, reservations and resolver overrides against inventory after the power cycle (Open)
• ⏳ Until corrective action 1 is done and measured, treat any image push to a Platform or IoT Backend VM as a change that can take the router down: console attached, operator present, and a way back that does not cross the router (Open)
• ⏳ Keep a change’s working files on disk, not in memory, when losing them would cost the change its evidence (Open)
• ⏳ Decide whether every substrate VM is built without a first-boot upgrade, and how images reach Platform and IoT Backend VMs as a matter of course, not only in this rehearsal (Open)
• ⏳ Rotate the automation’s Proxmox API token on the management hypervisor. The rehearsal’s new USB pass-through tasks wrote it into the Builder’s journal; the tasks are corrected (Open)
• 🔄 Give the Builder’s operator account a console password that survives a rebuild (In Progress)
• ✅ Add the risk to CHG-0039, and decide how its rehearsal rebuilds the provisioning VM without a bulk transfer across the router, or after the driver change. Done: images on an ISO from the VM’s hypervisor, first-boot upgrade off, and the rehearsal passed that way (Done)
• ✅ Restore the provisioning VM and confirm the registry, the state bucket and a tenant’s plan are as they were (Done)
• ✅ Set a console password for the operator on the Builder (Done)
• ✅ Remove the dumps of VM 201 from the management hypervisor (one from each rehearsal attempt) once the rehearsal has passed. They held the registry and tenants’ state unencrypted (Done)

↑ Incidents to close

INC-0004: Core Router Lost: a Hard Hang, Then Recurring re0 Watchdog Timeouts

• ⏳ Establish why the router hung. The console and local logs had nothing, so this now depends on catching a recurrence with the preventive actions below in place. (Open)
• ⏳ Test both hypotheses by reproducing deliberately. In a window where losing the router is acceptable, with its console attached, push sustained traffic across VLANs (iperf3 from a management host to a Platform host, so it crosses re0 both ways). Record the temperature every few seconds, to somewhere that survives a hang. Then compare with the same test after a cool-down. A hang that tracks temperature points to heat; a hang at low temperature under load points to the driver or the hardware. (Open)
• ⏳ Expose per-core temperatures (load coretemp), so a thermal reading means the CPU rather than one ACPI zone (Open)
• ⏳ Fix or replace the re0 path. First, Realtek’s vendor driver in place of the in-tree re(4) (Follow-up 3), which OPNsense ships for exactly this symptom. If timeouts persist after that, replace the router with hardware whose NICs are not Realtek, and that has more than two ports, so the trunk isn’t the only LAN link (the operator’s preference). Disabling hardware offloads on re0 is a further candidate, unverified. (Open)
• ⏳ Capture the next watchdog timeout with context: the timestamp, netstat -I re0 counters before and after, and the temperature. This comes from the console until off-box syslog is safe to enable. (Open)
• ⏳ Enable a crash dump device (dumpdev) on the router so a kernel panic leaves a dump. Now the main way the next event leaves evidence, since off-box syslog was declined, so do it before or with the vendor-driver change. (Open)
• ⏳ Monitor the router’s reachability, so a hang is detected by alert rather than by the operator losing a session (Open)
• ⏳ Move the router from the in-tree re(4) driver to Realtek’s vendor driver, as its own change record, because it changes a kernel module on the site gateway and needs a reboot. Details below the table. (Open)
• ⏳ The hard hang recurred on 2026-10-07, again during a container image push to a Platform VM: INC-0005. Follow-up 3 was still open (Open)
• ⏳ re0 stopped receiving on 2026-10-10 with the router alive and no watchdog message, at about 08:09 EDT, during a 946 MB download from a Platform VM to the Builder ( CHG-0046). The console answered and the WAN interface, re1, passed traffic. With the Builder sending one ARP request a second for 10.20.99.1, netstat -I re0 -w 1 showed almost no input. dmesg was read at the console and not captured. A reboot restored it before 08:34; the resolver’s records were as before, and a zone policy report afterwards showed no drift beyond that change’s own two rules. Follow-up 3 was still open (Open)
• ✅ Send the router’s syslog off-box so the last lines before a hang survive it. Declined by the operator on 2026-09-22: the central store is for tenants only ( ADR-0027), and no separate router log path is wanted. Evidence of the next hang is to come from the console and from crash dumps (Preventive action 2). (Declined)
• ✅ After recovery, confirm the router’s configuration survived: the zone policy (57 rules), Kea reservations and Unbound overrides (Done)
• ✅ Resume CHG-0018 Step 4 only after follow-up 1; dv02obs001v01 is left half-deployed (Done)
• ✅ Only once re0 is stable, send the router’s syslog to the central store. Declined on 2026-09-22 with Preventive action 1 (ADR-0027: tenants only). (Declined)

↑ Incidents to close

INC-0002: Controller VM Silent — Running but Off the Network

• ⏳ Determine why the freshly provisioned guest lost its network stack — next occurrence, capture the noVNC console before rebooting (panic trace / emergency shell / journal since boot) (Open)
• ⏳ Monitor the controller’s health (/api/info reachable and configured: true) on a schedule, so a silent controller raises an alert instead of waiting for the next change window (Open)
• ⏳ Establish whether a Terraform re-create of the domain VMs preserves the Omada data disk, and if not, protect the controller’s state (out-of-band backup, or a lifecycle guard) so a rebuild cannot wipe it (Open)
• ✅ Confirm qemu-guest-agent is enabled and started on the Fedora VM template, so qm agent is a reliable probe rather than another thing that is “not running” (Done)

↑ Incidents to close