2026 on Deevnet Infrastructure Platform

CHG-0002: Authority Transition Rework

Mon, 01 Jan 0001 00:00:00 +0000

CHG-0002: Authority Transition Rework #

**Retrospective change record.** This was rebuilt from the gap analysis written at the time, which is kept below as captured, and from the commits that delivered the fixes. The analysis was first published in the runbook as "Authority Transition Gap Analysis". Host names follow ADR-0008. At the time the builder was `provisioner-ph01`, the sites were `dvntm` and `dvnt`, and the site zones were `dvntm.deevnet.net` and `dvnt.deevnet.net`.


Date	2026-03-26
Change type	Configuration
Classification	Structural — playbook, role and inventory changes
Status	Complete. Analysed and fixed on 2026-03-26.
Site	mobile, and the builder’s multi-site design
Systems	Builder `dv00bld001p01`: the `bootstrap` role and `bootstrap-authoritative.yml` / `core-authoritative.yml` in `ansible-collection-deevnet.builder`; `disable-` / `enable-opnsense-services.yml` in `ansible-collection-deevnet.net`; `group_vars/bootstrap_nodes.yml`
Automation	Builder `f5d38d6`, net `d37a95f`, inventory `9600744`, docs `9e0b967`
Risk	Low to the running site. Nothing changes until an authority transition is run, and then the builder’s own IP swap is the risky step.
Related changes	CHG-0001: Flat Network → VLANs, which left these playbooks non-functional
Related incidents	None
Related runbooks	Authority Transition; Configure PXE (`make bootstrap-auth`); Build Network (`make core-auth`)

Summary #

The authority transition is how the builder takes over DNS, DHCP and the gateway from the core router to bootstrap a site (make bootstrap-auth), and hands them back afterwards (make core-auth). The VLAN migration moved the site onto the segmented 10-space, and nothing in it touched this procedure.

CHG-0003: Host Rename (ADR-0008)

Mon, 01 Jan 0001 00:00:00 +0000

CHG-0003: Host Rename (ADR-0008) #

**Retrospective change record.** This was rebuilt from the runbook page that tracked the rename as it ran (marked done on 2026-09-08), and from the commits and pull requests across the seven repos it touched. Times are local (UTC−4), from commit timestamps.


Date	2026-09-05
Change type	Migration
Classification	Disruptive
Status	Complete. Executed 2026-09-05, 08:41–13:55; recorded done 2026-09-08. Four follow-ups open.
Site	mobile; home received its site identity only, having no hosts
Systems	Every host in the mobile inventory; DNS and DHCP on `dv02cor002p01`; both Proxmox nodes; the site inventories and every repo that names a host, site or zone
Automation	Pull requests across `deevnet-docs`, `ansible-inventory-deevnet`, the three collections, `deevnet-image-factory`, `deevnet-tenant-factory` and `deevnet-tenant-tdemo` — see Procedure
Risk	High. The first live change was DNS and DHCP (phase 8), and a Proxmox node name has no supported rename (phase 13).
Related changes	None
Related incidents	None
Related runbooks	Renaming Hosts; Rename a Proxmox Node

Summary #

The migration that brought the estate onto the naming scheme in ADR-0008: fixed-width hostnames (dv02hyp001p01), site codes, and the home / mobile DNS zones. The site inventories were renamed with it, from dvnt/ and dvntm/ to home/ and mobile/. The ADR was proposed on 2026-09-04 and accepted as the first step of the change.

CHG-0004: Omada Controller Upgrade

Mon, 01 Jan 0001 00:00:00 +0000

CHG-0004: Omada Controller Upgrade #


Date	2026-09-10
Change type	Upgrade
Classification	Structural — a controller outage, but nothing on site depended on the controller
Status	Complete 2026-09-10. Re-scoped 2026-09-11: see Scope change.
Window	2026-09-10, 05:16–05:30 local
Site	mobile
Systems	Omada controller on `dv00bld001p01`
Automation	Controller pinned by inventory #24 and builder #13; runbook in docs #30
Risk	A newer controller’s database cannot be opened by an older one
Related changes	CHG-0005 (the AP) and CHG-0006 (the switch), both split out of this record
Related incidents	None
Related runbooks	Omada Controller Upgrade; Omada Controller Recovery

Summary #

The Omada controller ran 6.1. This change upgraded it in place to 6.3.0.45, the current release, and staged 6.2.14.11 as a rehearsed fallback. The aim was to bring the controller current before it is used to manage the site’s Omada devices. The switch’s current firmware recommends controller 6.2.0.

CHG-0005: Wireless AP Firmware and Omada Adoption

Mon, 01 Jan 0001 00:00:00 +0000

CHG-0005: Wireless AP Firmware and Omada Adoption #


Date	2026-09-15
Change type	Migration — the AP moves from standalone to controller-managed
Classification	Disruptive — wireless is down during each firmware hop and during adoption, and one hop cannot be undone
Status	Complete, 2026-09-15. The AP runs 1.3.11, is adopted and Connected, and serves the three controller-provisioned SSIDs; `DVNTM` was verified on a client landing on 10.20.10.x. PPSK with per-key VLAN binding was proven on the AP (phase 6) and then torn down, leaving `DVNTM-IOT` for a follow-up that provisions per-device keys from inventory.
Window	2026-09-15, one operator window. Operator on site at the AP; the AP was reached and adopted through the builder on VLAN 99, so no laptop on `gi1/0/2` was needed.
Site	mobile
Systems	AP `dv02wap001p01` (EAP650-Outdoor v1); the Omada controller, now a container in the network management VM `dv02nms001v01` on `dv02hyp001p01` (moved off `dv00bld001p01`) ( ADR-0013). Not the access switch.
Automation	Firmware mirrored by inventory #22. Networks, SSIDs and AP settings by `ansible-collection-deevnet.net` `playbooks/omada-wireless.yml`, through the documented Open API ( ADR-0009).
Risk	High. The AP’s 1.3.3 hop cannot be undone, and adoption replaces the AP’s own configuration with the controller’s.
Related changes	CHG-0001 — the hand-set SSIDs this retires; CHG-0004 — the controller upgrade this was split from; CHG-0006 — the switch, deliberately separate
Related incidents	INC-0002 — the controller VM was silent at the start of this window; Phase 0 recovered it before the change could run
Related runbooks	Wireless AP (the reset-and-recover path); Omada Controller Recovery; Important URLs

Summary #

The AP ran firmware 1.0.4, from April 2023. Omada could not push VLAN-tagged SSIDs to it, so CHG-0001 left the four SSIDs configured by hand in the AP’s own web UI — the one piece of the site’s network configuration that lived nowhere but on the device.

CHG-0006: Access Switch Firmware Upgrade

Mon, 01 Jan 0001 00:00:00 +0000

CHG-0006: Access Switch Firmware Upgrade #


Date	2026-09-16
Change type	Upgrade
Classification	Disruptive — the reboot drops every wired path on site
Status	Complete, 2026-09-16. The switch runs 1.20.24 from `image1.bin`, with 1.20.1 kept in `image2.bin` as the rollback. The configuration came through unchanged and matches inventory, and Ansible connects. STP is still disabled. LLDP came up enabled under the new default and was kept on deliberately; it is now declared in inventory.
Window	2026-09-16, one operator window. The operator was on site and reached the builder through the travel router. The switch was down from 08:15:32 to 08:17:11 EDT, about 100 seconds.
Site	mobile
Systems	Access switch `dv02acc001p01` (SG2218, hardware 1.20)
Automation	Firmware mirrored by inventory #23; procedure in docs #29 and #70. LLDP declared by net #21 (`switch_lldp`) and inventory #33.
Risk	High. The switch reboot takes the builder, the controller and Ansible offline with everything else.
Related changes	CHG-0004 — this was its phase 2 until 2026-09-11
Related incidents	None
Related runbooks	Access Switch → Firmware upgrade; Operator Access; Important URLs

Summary #

The access switch ran firmware 1.20.1, from January 2024. Current for its hardware is 1.20.24, from May 2026: one hop, with no minimum prior version and nothing irreversible. This change is the firmware only. The switch stays standalone, configured by switch_vlans. Adopting it into the controller is a later change under ADR-0009.

CHG-0007: Core Router Zone Policy, First Application

Mon, 01 Jan 0001 00:00:00 +0000

CHG-0007: Core Router Zone Policy, First Application #


Date	Not yet scheduled
Change type	Configuration
Classification	Disruptive — it changes what every segment on site can reach, the operator’s own path included
Status	Planned, after the `opnsense_firewall` fixes and the mobile reachability targets (see Prerequisites). Pre-change state read on 2026-09-14 (see Pre-change state). Allow-all removal decided on 2026-09-14: Option A (see Decision).
Window	To be scheduled, with the operator at the rack and the router’s console connected
Site	mobile
Systems	Core router `dv02cor002p01` (OPNsense 26.7.3); control host `dv00bld001p01`
Automation	`ansible-collection-deevnet.net` role `opnsense_firewall`, run by `playbooks/migration/07-opnsense-firewall.yml` (`make migration-opnsense-firewall`) against `ansible-inventory-deevnet/mobile`
Risk	High. The policy has never been applied as a whole, and the last unguarded run of this role took the site down.
Related changes	CHG-0001 — the segmentation that declared this policy
Related incidents	INC-0001 — its open items are this change
Related decisions	ADR-0011 — attachment by trust class depends on this policy; its Validation is where the pre-change state was first read
Related runbooks	Change Management; Console Recovery → Core Router

Summary #

mobile/group_vars/all/firewall.yml declares a default-deny zone policy: 18 inter-zone allows and internet egress for 8 zones. INC-0001 records that this policy “has still never been applied”. The router is running whatever a config backup of unknown vintage put back.

CHG-0008: Management Domain VMs, First Build

Mon, 01 Jan 0001 00:00:00 +0000

CHG-0008: Management Domain VMs, First Build #


Date	2026-09-15
Change type	Deployment · Decommission
Classification	Structural. It replaces every VM on the management hypervisor except the provisioners, and re-applies the access switch’s trunks.
Status	Complete, 2026-09-16. Done on 2026-09-15: Steps 1–2 and 5–6; Step 7 for `dv02nms001v01` and `dv02sob001v01`, both rebooted after their first-boot upgrade; Step 8 for the Omada controller on `dv02nms001v01`, with its firewall ports corrected at 22:05Z. Step 9’s manual floor is done: the wizard, a local Owner, the automation account and the Open API client. Step 10 stopped the Builder’s controller at 23:14Z, and Step 3 carried VLAN 25 to the management hypervisor’s port at 23:28Z with the operator at the rack. Step 4 made the hv01 bridge VLAN-aware at 23:38Z, and Step 7 finished at 00:07Z with the four Platform and IoT Backend VMs built, verified and rebooted. Step 8 finished at 00:17Z with MinIO and the Deevnet API on `dv02prv001v01`. Every step is done; what is left is the follow-ups below. Steps 3–4 wait for someone at the rack, and the Platform and IoT Backend VMs in Steps 7–8 wait on those.
Window	Started 2026-09-15 about 04:10Z, vault decrypted for each working window. Steps 3 and 4 need hands at the hardware. The SG2218 has no console port, so recovery is a laptop on `gi1/0/2` or the reset button; `dv02hyp001p01` needs its monitor and keyboard.
Site	mobile
Systems	Management hypervisor `dv02hyp001p01`; six new VMs: `dv02nms001v01`, `dv02sob001v01`, `dv02prv001v01`, `dv02idn001v01`, `dv02tob001v01`, `dv02msg001v01`; retired: `dv02tdn001v01`, `dv02tst001v01`, `dv02mqt001v01`; core router `dv02cor002p01` (Unbound, Kea); access switch `dv02acc001p01`; control host `dv00bld001p01`
Automation	`ansible-collection-deevnet.mgmt` (`podman_service`, `powerdns`, `minio`, `deevnet_api`, `omada_controller`, `proxmox_vm`, `vm_identity`); `ansible-collection-deevnet.net` (`dns.yml`, `dhcp.yml`, `switch-vlans.yml`, `proxmox-node-network.yml`); `ansible-collection-deevnet.builder` (`artifacts`); the new `deevnet-provisioning-api` repository; all against `ansible-inventory-deevnet/mobile`
Risk	Medium. The step most likely to go wrong is the switch trunk re-apply, which rewrites every trunk on the switch, including the ports the control host and the management hypervisor sit behind.
Related decisions	ADR-0013 — the domain VMs this builds, and §6’s fold-in of tenant DNS and state, which this record carries out; ADR-0012 — the API whose shell is deployed; ADR-0009 — the controller’s manual floor
Related changes	CHG-0005 — waits for the network management VM this builds; CHG-0007 — the zone policy these VMs are placed to survive
Related incidents	INC-0001 — why the router record prune is previewed and gated
Related runbooks	VM Identity; Console Recovery → Access Switch

Summary #

ADR-0013 groups the management hypervisor’s services into domain VMs, each on exactly one segment, with every function a container. Today the hypervisor still runs the earlier one-service VMs: tenant DNS (dv02tdn001v01), the tenant state store (dv02tst001v01) and an MQTT broker that never answered (dv02mqt001v01). The Omada controller runs on the Builder.

CHG-0009: Access Switch Omada Adoption

Mon, 01 Jan 0001 00:00:00 +0000

CHG-0009: Access Switch Omada Adoption #


Date	Not yet scheduled
Change type	Migration — the switch moves from standalone to controller-managed
Classification	Disruptive — adoption rewrites port VLANs, and the uplink is expected to drop until they are re-applied
Status	Planned, on hold since 2026-09-16. Deprioritised behind the Deevnet API and the first working tenant; it is not on that path. The research below settles the route, so the work resumes from here without repeating it.
Window	To be scheduled, with the operator on site and connected through the travel router ( Operator Access)
Site	mobile
Systems	Access switch `dv02acc001p01` (SG2218 hardware 1.20, firmware 1.20.24); the Omada controller (6.3.0.45) in `dv02nms001v01`
Automation	To be written: `ansible-collection-deevnet.net` `playbooks/omada-switch.yml`, modelled on `playbooks/omada-wireless.yml`. `switch_vlans` becomes break-glass once the switch declares `switch_management: omada`.
Risk	High. Every path to the controller crosses this switch; if adoption does not keep the management VLAN, the controller loses the switch mid-provision and the way back is a factory reset.
Related changes	CHG-0005 — the AP, adopted the same way; CHG-0006 — the firmware this relies on
Related incidents	None
Related runbooks	Access Switch (the reset path, and the rollback); Operator Access; Important URLs

Summary #

The access switch is standalone. Its configuration is declared in inventory and applied over its CLI by switch_vlans. ADR-0009 decides that the controller applies that inventory through its documented Open API instead, as it already does for the AP since CHG-0005. This change adopts the switch and moves its port configuration to the controller.

CHG-0010: Deploy the Deevnet API and Cut Tenants Over

Mon, 01 Jan 0001 00:00:00 +0000

CHG-0010: Deploy the Deevnet API and Cut Tenants Over #


Date	2026-09-17
Change type	Build-out and migration — OpenBao and the tenant API are deployed, and tenants stop being built from inventory
Classification	Disruptive in one step only: the tenant DNS server restarts when its HTTP API is turned on. Everything else adds services or changes objects nothing depends on yet.
Status	Complete. All eleven steps are done. OpenBao, the PowerDNS API, the API and the egress agent are deployed; tdemo and eds are both live and built entirely through the API; and the inventory tenant registry and the role tasks that read it are gone. One working-tree edit awaits `make vault`: the two now-unread tenant secret dicts.
Window	No operator on site needed: every step is a control-node run or an API call. The one restart affects tenant name resolution for seconds, and no tenant is live.
Site	mobile
Systems	`dv02idn001v01` (OpenBao, tenant DNS), `dv02prv001v01` (the API, its database, the state store), `dv02hyp002p02` (the tenant hypervisor and exit node), `dv02cor002p01` (the resolver’s delegations)
Automation	`deevnet.mgmt` `openbao`, `powerdns`, `minio`, `deevnet_api`; `deevnet.net` `tenant_egress_agent`; `deevnet-provisioning-api` v0.2.4 (v0.2.0 as planned, then four defects the run and the rebuild found); `terraform-provider-deevnet` v0.1.0; the tenant repositories `deevnet-tenant-tdemo` and `eds`
Risk	Medium. OpenBao becomes a service everything else needs to start, and its seal key is the root of the whole arrangement. The API gains write access to tenant DNS, the resolver, the state store and the tenant hypervisor.
Related decisions	ADR-0015 — what the API builds and what a tenant holds; ADR-0016 — where the credentials live; ADR-0012 §5 and §9; ADR-0014
Related changes	CHG-0008 — built the VMs this deploys into, and created eds’s zones, key and state credential; CHG-0007 — the zone policy this adds two rules to, still unapplied
Related runbooks	Provisioning a Tenant — replaced by this change’s step 9

Summary #

ADR-0015 moves everything about a tenant behind the Deevnet API, and ADR-0016 puts the credentials that needs into OpenBao. This change deploys both and cuts the two tenants over.

CHG-0011: Tenant Workloads Get a Resolver

Mon, 01 Jan 0001 00:00:00 +0000

CHG-0011: Tenant Workloads Get a Resolver #


Date	2026-09-17
Change type	Configuration · Deployment
Classification	Structural
Status	Complete. All three steps are done and verified. Step 3 was run as a full destroy-and-rebuild at the operator’s request, which proved the fix on the create path as well as the repair path.
Window	Started 2026-09-17; Steps 1–2 ran 2026-09-17/18
Site	mobile
Systems	`dv02prv001v01` (the Deevnet API); tenant workloads 2040 `tdemo-app` and 2080 `eds-services` on `dv02hyp002p02`
Automation	`deevnet.mgmt` `playbooks/site.yml --limit deevnet_api`, against `ansible-inventory-deevnet/mobile`; then `terraform apply` in each tenant repo
Risk	Medium — both tenant workloads are replaced, so anything on their disks is lost. Today both are empty.
Related changes	CHG-0010 introduced the defect
Related incidents	None
Related runbooks	Tenant DNS

Summary #

Every tenant workload the Deevnet API has ever built was handed an authoritative DNS server as its resolver. dv02idn001v01 (10.20.25.21) runs PowerDNS authoritative: it answers the tenant zones it is master for and returns REFUSED for everything else. A workload pointed at it can resolve its own tenant’s names and nothing else — not a public name, not a substrate name, not even api.mobile.deevnet.net, the Deevnet API the tenant is meant to talk to.

CHG-0012: Operator Access to Tenant Workloads

Mon, 01 Jan 0001 00:00:00 +0000

CHG-0012: Operator Access to Tenant Workloads #


Date	2026-09-18
Change type	Configuration
Classification	Structural
Status	Complete. The route is applied and verified from both operator zones.
Window	2026-09-18
Site	mobile
Systems	`dv02cor002p01` (core router, static route and zone policy); `ansible-inventory-deevnet` firewall policy
Automation	`deevnet.net` role `opnsense_routes`, run by `playbooks/routes.yml` against `ansible-inventory-deevnet/mobile`. The zone-policy rules are inventory, applied by `opnsense_firewall` when CHG-0007 runs.
Risk	Low — it adds reachability and removes none. The one thing most likely to go wrong is the route being written to the wrong gateway, which fails closed.
Related changes	CHG-0007, which must not undo it
Related incidents	None
Related runbooks	None

Summary #

A tenant workload cannot be reached from anywhere today. Building, checking or debugging one means the hypervisor console, or ip vrf exec from the exit node as root — the most privileged access the substrate has, used for ordinary work.