2026 on Deevnet Infrastructure Platform

INC-0001: Firewall Policy Deleted, Total Connectivity Loss

Mon, 01 Jan 0001 00:00:00 +0000

INC-0001: Firewall Policy Deleted, Total Connectivity Loss #


Date	2026-09-07
Site	mobile (`dvntm`)
Systems	Core router `dv02cor002p01` (OPNsense); the `opnsense_firewall` role in `ansible-collection-deevnet.net`
Severity	Total site outage; physical console access required to recover
Status	Root cause confirmed. Service restored from config backup. Corrective and preventive actions 1–8 done as of 2026-09-08. Of five open items, two were settled on 2026-09-14 and one narrowed; three remain, planned as CHG-0007.
Times	UTC (local is UTC−4), as recorded in the session transcript

**The guards are in, but have not yet met the live router.** Since 2026-09-08, `opnsense_firewall` refuses to reconcile from a broken discovery, withholds deletions by default, protects the operator's path, and applies behind a rollback savepoint ([Corrective actions](#corrective-actions), [Preventive actions](#preventive-actions)). They were verified offline against this incident's own input. The zone policy has still never been applied, so its first real application remains a watched change with the console open.

Summary #

Four ad-hoc runs of the opnsense_firewall role against the core router deleted every ansible:-prefixed filter rule on it: 18 zone policies, the internet rules for 8 zones, the per-zone conntrack rules, and both anti-lockout rules. That is the entire inter-VLAN policy. The role reported success on every run.

INC-0002: Controller VM Silent — Running but Off the Network

Mon, 01 Jan 0001 00:00:00 +0000

INC-0002: Controller VM Silent — Running but Off the Network #


Date	2026-09-15
Site	mobile (`dvntm`)
Systems	Network-management VM `dv02nms001v01` (VMID 204 on `dv02hyp001p01`), running the Omada controller container
Severity	Management-plane only. The controller was unreachable, which blocked CHG-0005. No client-facing outage: the AP was still standalone on its old firmware, so wireless kept serving.
Status	Service restored by a guest reboot; root cause not established. The controller and its Omada data (local Owner, Open API client) came back intact. Corrective and preventive actions are open.
Times	Local (EDT, UTC−4), as observed from the builder during the CHG-0005 window

Summary #

At the start of the CHG-0005 window, the controller VM dv02nms001v01 reported status: running to Proxmox but had no presence on the network at all — no ARP reply, no QEMU guest-agent response, and nothing answering on 10.20.99.40:8043. The guest was wedged in a way that Proxmox’s own “running” status did not reflect.

INC-0003: OpenBao's Recovery Key and AppRole Destroyed by a Git Reset

Mon, 01 Jan 0001 00:00:00 +0000

INC-0003: OpenBao’s Recovery Key and AppRole Destroyed by a Git Reset #


Date	2026-09-17
Site	mobile (`dvntm`)
Systems	OpenBao on `dv02idn001v01`; the Deevnet API on `dv02prv001v01`; the inventory repository `ansible-inventory-deevnet`
Severity	Provisioning only. OpenBao kept running and self-unsealing, the Deevnet API kept serving, and both live tenants — `tdemo` and `eds` — were unaffected throughout, including during the rebuild. No client-facing outage.
Status	Resolved. OpenBao was rebuilt, both tenants resupplied their secrets, and the practice that allowed it is now written down and enforced by a checklist. Three code defects the rebuild exposed are fixed.
Cause	`git reset --hard` run in a repository whose vault files were decrypted, eight minutes after an initialisation wrote once-only credentials into one of them

Summary #

During CHG-0010, step 3 initialised OpenBao. That run produces three values that exist nowhere else: the recovery key, and the role ID and secret ID of Ansible’s AppRole. It writes them to .openbao/<host>-init.json on the control node, and the change record instructs the operator to move them into the inventory vault and delete that file.