CHG-0008: Management Domain VMs, First Build #

Summary #

ADR-0013 groups the management hypervisor’s services into domain VMs, each on exactly one segment, with every function a container. Today the hypervisor still runs the earlier one-service VMs: tenant DNS (dv02tdn001v01), the tenant state store (dv02tst001v01) and an MQTT broker that never answered (dv02mqt001v01). The Omada controller runs on the Builder.

Nothing depends on those three VMs yet:

• The only registered tenant, eds, has never applied its Terraform ( ADR-0011).
• tdemo was destroyed on 2026-09-05.
• The broker was never reachable.

So the VMs are rebuilt from the current design rather than migrated. The operator deletes the old VMs by hand. This record covers everything else.

After this change:

Container images are pushed, not pulled. The shared podman_service role copies each image tarball from the control host over SSH (management -> platform and management -> iot_backend are both declared), so no VM needs a path back to the artifact server. That keeps the build working once CHG-0007 enforces the policy.

The Builder’s controller becomes a cold fallback. Its role stays in deevnet.builder, and its play now excludes management-plane hosts. The Builder leaves network_controllers in the inventory change itself, not after verification: it keeps running untouched until Step 10 stops it.

Goal #

• The six VMs exist on dv02hyp001p01 with VMID-derived MACs, each on its declared VLAN and address, and answer SSH by name.
• The core router holds no record or reservation for dv02tdn001v01 or dv02tst001v01. dv02mqt001v01 never had one.
• tdns, tfstate, api and omada resolve to the new hosts.
• The eds tenant zones answer authoritatively from dv02idn001v01 and, through the core router, from any segment that resolves there.
• http://api.mobile.deevnet.net:8080/readyz returns 200, and /version reports v0.1.0.
• The controller on dv02nms001v01 has completed its manual floor. The Builder’s controller is stopped and disabled, with its data kept.

Scope #

In scope:

• inventory and router records for the three retired hosts
• VLAN 25 on the management hypervisor’s trunk
• making the management hypervisor’s bridge VLAN-aware
• staging the PostgreSQL and API images
• identity, addressing and DNS for the six VMs
• creating them
• PowerDNS, MinIO, the API shell and the Omada controller on their VMs
• the controller’s manual floor
• stopping the Builder’s controller

Out of scope:

• deleting the old VMs (done by the operator, before Step 1)
• the VerneMQ broker and its auth database
• observability tooling
• any API functionality
• the platform -> management rule the API will need to reach the controller
• adopting devices ( CHG-0005)
• enforcing the zone policy ( CHG-0007)

Risk and impact #

Prerequisites #

• The old VMs (VMIDs 200, 201, 204) are deleted from dv02hyp001p01
• The branches for this change are merged, or checked out on the control host: inventory, deevnet.mgmt, deevnet.builder, deevnet-tenant-factory, deevnet-provisioning-api
• Vault decrypted, including mobile/group_vars/deevnet_api/vault.yml
• Console access to dv02acc001p01, for Step 3, and to dv02hyp001p01, for Step 4
• Each collection installed: make install-dev in each. The commands below run from the collection’s directory with export ANSIBLE_COLLECTIONS_PATH=./.ansible/collections:$HOME/.ansible/collections.

Procedure #

Step 1: Preflight #

Read-only.

Run:

ssh a_autoprov@dv02hyp001p01 'sudo qm list; free -g; sudo pvesm status; grep -A8 "iface vmbr0" /etc/network/interfaces'
cd ansible-collection-deevnet.mgmt && make vm-identity

Verify:

1. VMIDs 200, 201 and 204 are gone, and a fedora-server-* template is listed.
2. At least 16 GB of memory is free.
3. vmbr0’s stanza in /etc/network/interfaces has no bridge-vlan-aware line. Read on 2026-09-14: bridge-ports enp0s31f6, bridge-stp off, bridge-fd 0, nothing else. Step 4 changes that.
4. make vm-identity reports the six new hosts as unallocated, and no collision.

Undo: Nothing to undo.

Step 2: Remove the retired hosts’ router records #

This deletes records on the core router, and nothing else. The inventory no longer declares the three hosts, so the roles list them as undeclared.

Run the preview first:

cd ansible-collection-deevnet.net
ansible-playbook playbooks/dns.yml --check --diff
ansible-playbook playbooks/dhcp.yml --check --diff

Gate: both “LEFT IN PLACE” reports list exactly what the preview showed on 2026-09-14, and nothing else:

• DNS: host overrides dv02tdn001v01 (10.20.99.30) and dv02tst001v01 (10.20.99.31); aliases none
• DHCP: reservations dv02tdn001v01 (02:DE:20:00:00:C8) and dv02tst001v01 (02:DE:20:00:00:C9)

dv02mqt001v01 has nothing to remove:

• It never had a reservation, because it declared dhcp_reservation: false.
• The router holds no override and no mqtt alias for it, which fits the broker never resolving in ADR-0011’s validation.

tdns and tfstate are still declared, now by the new hosts, so the role moves them rather than deleting them. Anything else in either list stops the step.

Then run:

ansible-playbook playbooks/dns.yml -e dns_delete_unmanaged=true
ansible-playbook playbooks/dhcp.yml -e dhcp_delete_unmanaged=true

Verify:

1. The preview, run again, has no “LEFT IN PLACE” report.
2. dig +short @10.20.99.1 dv02tdn001v01.mobile.deevnet.net and dv02tst001v01.mobile.deevnet.net return nothing.

Undo: Undo Step 2

Step 3: Carry VLAN 25 to the management hypervisor #

This re-applies every trunk on the switch and writes flash. The --check flag cannot preview this role, so the change is reviewed as the inventory diff: gigabitEthernet 1/0/15 goes from allowed_vlans: [35, 99] to [25, 35, 99], and native 99 is unchanged.

Run:

# Baseline, from the switch CLI:  show running-config interface gigabitEthernet 1/0/15
cd ansible-collection-deevnet.net
ansible-playbook playbooks/switch-vlans.yml --tags trunk

Verify:

1. The switch’s running config for gigabitEthernet 1/0/15 carries 25 tagged, and 99 untagged.
2. dv02hyp001p01 (10.20.99.21) and the control host’s session stay up.

Undo: Undo Step 3

Step 4: Make the management hypervisor’s bridge VLAN-aware #

vmbr0 on dv02hyp001p01 is a plain bridge, so the tagged NICs of the Platform and IoT Backend VMs have no VLAN-aware bridge to join. This step sets it to the configuration Proxmox documents and dv02hyp002p02 already runs: bridge-vlan-aware yes, bridge-vids 2-4094. It is declared in host_vars/dv02hyp001p01/vars.yml as proxmox_node_network, bridge only. There are no VLAN sub-interfaces, and management keeps its untagged address and default route.

This touches the node’s only NIC, so run it with the console available. Proxmox’s documentation does not say whether the change takes effect live.

Run:

cd ansible-collection-deevnet.net
ansible-playbook playbooks/proxmox-node-network.yml --tags interfaces -e target=dv02hyp001p01 --check
ansible-playbook playbooks/proxmox-node-network.yml --tags interfaces -e target=dv02hyp001p01

Verify:

1. dv02hyp001p01 (10.20.99.21) still answers, and so does dv02bld001v01 on it.
2. On the node, cat /sys/class/net/vmbr0/bridge/vlan_filtering prints 1, and /etc/network/interfaces shows bridge-vlan-aware yes and bridge-vids 2-4094 under vmbr0.

Undo: Undo Step 4

Step 5: Stage the images #

These writes are local to the control host.

Run:

cd ansible-collection-deevnet.builder
ansible-playbook playbooks/site.yml --limit dv00bld001p01 --tags container-images
cd ../deevnet-provisioning-api && make stage   # tag v0.1.0 checked out

Verify:

1. /srv/deevnet-http/container-images/postgres/postgres-17.11.tar exists.
2. /srv/deevnet-http/container-images/deevnet-api/deevnet-api-v0.1.0.tar exists.

Undo: Nothing to undo; the tarballs are inert.

Step 6: Allocate identity, then publish addresses and names #

Run:

cd ansible-collection-deevnet.mgmt && make vm-identity-assign
# Commit the six generated host_vars/<host>/identity.yml files in the inventory.
cd ../ansible-collection-deevnet.net
ansible-playbook playbooks/dhcp.yml --check --diff && ansible-playbook playbooks/dhcp.yml
ansible-playbook playbooks/dns.yml --check --diff && ansible-playbook playbooks/dns.yml

Verify:

1. make vm-identity reports no drift.
2. Kea holds reservations for 10.20.99.40 and 10.20.99.41 on their derived MACs.
3. dig @10.20.99.1 resolves all six host names, plus omada, tdns, tfstate and api.

Undo: Remove the identity.yml files, then rerun Step 2’s apply.

Step 7: Create the VMs #

Run:

cd ansible-collection-deevnet.mgmt
ansible-playbook playbooks/site.yml --tags vms --limit 'management_plane:!dv02bld001v01:!dv02bld002v01'

Verify:

1. proxmox_vm passes its MAC and address assertions for all six.
2. qm config <vmid> shows net0 tagged 25, 35 or none, as declared.
3. Each VM answers ssh a_autoprov@<host>.mobile.deevnet.net.
4. ip route on each VM shows its segment’s gateway.

Undo: Undo Step 7

Step 8: Configure the services #

Run:

cd ansible-collection-deevnet.mgmt
ansible-playbook playbooks/site.yml --skip-tags vms --limit dv02idn001v01
( cd ../ansible-collection-deevnet.net && ansible-playbook playbooks/dns.yml --tags tenant-dns )
ansible-playbook playbooks/site.yml --skip-tags vms --limit dv02prv001v01
ansible-playbook playbooks/site.yml --skip-tags vms --limit dv02nms001v01

The tenant-dns run moves the core router’s eds forwarding rows from 10.20.99.30 to 10.20.25.21.

Verify:

1. dig @10.20.25.21 eds.mobile.deevnet.net SOA answers authoritatively, and dig @10.20.99.1 eds.mobile.deevnet.net SOA gives the same answer.
2. curl -s http://api.mobile.deevnet.net:8080/readyz returns 200, and /version reports v0.1.0.
3. /v1/anything returns 401 without the token.
4. From the control host, the mc alias for http://tfstate.mobile.deevnet.net:9000 lists tf-state.
5. https://omada.mobile.deevnet.net:8043 loads the controller’s setup wizard.

Undo: Stop and disable the service units on the VM (pdns-auth, minio, deevnet-api, deevnet-api-db, omada-controller). The VMs themselves are undone by Step 7’s undo.

Step 9: The controller’s manual floor #

This is manual, per ADR-0009 §5.

1. In the setup wizard, create the Owner account, and decide whether to cloud-register it (ADR-0013’s open question 2). Make the vault’s vault_omada_owner_* match the account.

2. Create the automation account:

   cd ansible-collection-deevnet.builder && ansible-playbook playbooks/omada-automation-user.yml
   

3. As the Owner, create the Open API client, and record its id and secret in the vault.

Verify: A client-credentials token request from the control host succeeds. The request only reads.

Undo: Reset the controller: stop omada-controller, clear /opt/omada-controller/data, and rerun Step 8 for dv02nms001v01.

Step 10: Stop the Builder’s controller #

Run on dv00bld001p01:

sudo systemctl disable --now omada-controller

Verify:

1. systemctl is-enabled omada-controller reports disabled.
2. /opt/omada-controller is intact.

Undo: sudo systemctl enable --now omada-controller

Verification #

• Every point under Goal holds.
• Two reboot checks:
  • qm reboot for dv02idn001v01 and dv02prv001v01; every service comes back without a manual step.
  • systemctl restart deevnet-api-db on dv02prv001v01 makes /readyz return 503, and it recovers by itself.
• The empty VMs (sob, tob, msg) answer SSH, and nothing but sshd is listening on them.

Undo #

Steps are backed out in reverse order.

No practical undo point: the old VMs are deleted, so there’s nothing to return to. Undo restores the router and the switch to their pre-change state, and removes the new VMs.

Undo Step 2 #

Revert the inventory change, then rerun dns.yml and dhcp.yml without the delete flags. This recreates the records and reservations. They point at VMs that no longer exist.

Undo Step 3 #

Set allowed_vlans back to [35, 99], and rerun switch-vlans.yml --tags trunk. If the run cuts the path, restore the baseline from the console.

Undo Step 4 #

Remove the proxmox_node_network block, and set the bridge back through the API or the node’s network page: VLAN aware off. If management was lost, restore the stanza read in Step 1 from the console, then run ifreload -a.

Undo Step 7 #

qm stop <vmid> && qm destroy <vmid> for each new VM. Then remove its identity.yml and rerun Step 2’s apply.

Outcome #

Departures from the plan #

• Step 2 also did work planned for Steps 6 and 8. The DNS run applies the whole role, not only the prune. So it also:

  • added host overrides for the six new VMs
  • added the omada and api aliases
  • moved tdns and tfstate to the new hosts
  • added the core router’s forwarding rows for the eds tenant zones (eds.mobile.deevnet.net, 129.20.10.in-addr.arpa), pointing at dv02idn001v01 (10.20.25.21)

  Queries for those zones through the router timed out until the identity VM existed. Resolved at 00:10Z, once PowerDNS came up on dv02idn001v01: both zones now answer through the router. eds had never applied its Terraform, so nothing depended on them in the meantime.

• The DHCP run rewrote all 13 existing reservations with the values they already had. This is a defect in opnsense_dhcp, not a change:

  • its update set is every declared reservation whose MAC already exists, with no field comparison
  • the task is changed_when: true

  The API read above confirms nothing moved.

• Steps 5 and 6 ran before Steps 3 and 4. The operator was remote, reaching the Builder through its WAN port. Steps 3 and 4 can cut the Builder’s management path and need hands at the hardware, while Steps 5 and 6 touch only the Builder and the router’s API. Neither depends on VLAN 25 or the bridge.

• Step 6’s DNS run was skipped. Step 2’s DNS run had already published every record Step 6 would write. The check-mode preview had nothing to add, the override update only fires on an address change, and dig through the router confirmed every name.

• The Step 6 DHCP run again rewrote the 13 existing reservations with unchanged values: the same opnsense_dhcp defect as in Step 2, already a follow-up.

• Step 7 ran for the two management-segment VMs only. dv02prv001v01, dv02idn001v01 and dv02tob001v01 (Platform) and dv02msg001v01 (IoT Backend) need VLANs 25 and 35 on the hv01 trunk and a VLAN-aware bridge first, which are Steps 3 and 4.

• dv02bld001v01 and dv02bld002v01 were stopped by the operator during the window, along with the two VMs that are not in inventory. Nothing in this change needs them: images are pushed from dv00bld001p01, which is untouched.

• Every new VM upgrades itself on first boot. Proxmox’s ciupgrade option defaults to on: “cloud-init: do an automatic package upgrade after the first boot.” (qm manual, default = 1). proxmox_vm doesn’t set it. From the Fedora 44 1.7 template, that meant:

  • a 628 MiB download from the internet
  • several extra minutes on every build
  • a VM that needs a reboot
  • a failure on a site without internet access

  The VMs weren’t rebooted as part of Step 7.

• The controller role’s firewall port list doesn’t match 6.3.0.45. The list was carried over from the Builder’s role. Omada’s port reference, “Which Ports do Omada SDN Controller and Omada Discovery Utility Use (above Controller 5.0.15)” (updated 08-28-2026), is the source for what follows.

  • Listening but blocked. The controller listens on TCP 29815, 29816, 29817 and 8044, and on UDP 19810. firewalld opens none of them. The reference lists the first three as TCP:

    • 29815: “Starting from v5.9, Omada Controller receives Device Info, Packet Capture Files.”
    • 29816: “Starting from v5.9, Omada Controller establishes the remote control terminal session.”
    • 29817: “To ensure proper communication between devices and the Controller.”

    8044 isn’t in the reference; the controller’s omada.properties names it upgrade.es.https.port. UDP 19810 is OLT discovery, and this site has no OLT.

  • Open but unused. firewalld opens UDP 29814, which the reference lists as TCP only, and TCP 27002, which isn’t in the reference and nothing binds.

  Nothing is adopted yet, so nothing is affected today. It matters before CHG-0005 adopts the AP.

• Making the bridge VLAN-aware dropped its address from Proxmox’s record. proxmox_node_network PUTs only the bridge fields, and a PUT replaces the interface definition, so PVE rewrote vmbr0’s method from static to manual.

  • Effect: none observed. The address and gateway lines stayed in /etc/network/interfaces, ifupdown2 read them, and hv01 kept its address, its route and its guests.
  • Why it still mattered: hv01 has one NIC, and a method that no longer claims its address is not a state to leave a node in.
  • Repair: a PUT carrying the address and gateway restored inet static at 23:41:23, with VLAN awareness unchanged. The role now carries cidr and gateway through ( deevnet.net #19).

• Re-running Step 8 took two data directories away from the controller.

  • Cause: omada_controller forced root:root 0755 on its data directories on every run. The image’s entrypoint (fix_permissions) gives data and logs to its omada account (UID 508) at each start, so the firewall-fix run reset both to root.
  • Effect: only the top-level directories changed, and nothing was logged. But until a restart, the controller couldn’t create new files directly in either directory.
  • Repair: ownership was restored at 22:07:30 without a restart. The role now creates missing directories and leaves existing ones alone (mgmt #16, builder #17).

Follow-ups #

The change is complete; these are carried separately.

• Reboot dv02nms001v01 and dv02sob001v01 onto the upgraded kernel before Step 8 (found in Step 7; done 21:31Z)
• omada_controller, in both deevnet.mgmt and deevnet.builder: align the firewalld port lists with Omada’s port reference for the running version (found in Step 8). Done in mgmt #15 and builder #16, applied to dv02nms001v01 at 22:05Z; 8044 left closed.
• After mgmt #16 merges, re-run Step 8 on dv02nms001v01 and confirm the data-directory task reports no change (found in Step 8; done 22:22Z, changed=0)
• The vault holds one controller’s credentials - Owner, automation account and Open API client. The Builder fallback controller’s own credentials are now only in the vault file’s git history. Decide whether that is acceptable for a fallback whose database is derived state (found in Step 9).
• proxmox_vm: decide on first-boot upgrades. Either set ciupgrade off and keep the template current through image-factory rebuilds, or keep it on and reboot the guest when dnf needs-restarting -r asks (found in Step 7)
• opnsense_dhcp: compare IP, hostname and description before updating a reservation, so a run with no drift reports no change (found in Step 2)
• The VerneMQ broker and its auth database in dv02msg001v01, with the mqtt name and mqtt_brokers membership ( ADR-0012 §8)
• Observability tooling for dv02sob001v01 and dv02tob001v01
• API functionality, and the platform -> management rule to the controller’s Open API port ( ADR-0013 §10)
• Review powerdns_dnsupdate_from, which still admits only the management subnet, now that the server sits on Platform. Done: mgmt #19 admits management, trusted and tenant_transit, applied to dv02idn001v01 on 2026-09-17 (--tags tenant-dns, changed=3). The eds zones read back ALLOW-DNSUPDATE-FROM = 10.20.99.0/24, 10.20.10.0/24, 10.20.50.0/24, and an update signed with eds’s key from the Builder added and removed a TXT record.
• deevnet-tenant-tdemo still names tfstate.mobile.deevnet.net and 10.20.99.30. The tenant is destroyed; update or archive the repository. Not archived: a do-not-apply banner went in (tdemo #3), and ADR-0015 makes tdemo the reference tenant, rewritten to create itself through the API.
• deevnet-tenant-factory TENANTS.md shows index 1 as free, but the inventory gives it to eds. Done in factory #13; ADR-0015 retires the file at cutover.
• Update the pages that describe current state: tenant DNS platform page, management hypervisor page, Important URLs, the VM identity and MAC naming worked examples, and the Omada recovery and upgrade runbooks
• ADR-0013: status Accepted; record the answer to open question 1 (the Builder’s controller is a cold fallback) and open question 2 (as decided in Step 9)