Verify Site #

Validation after the site is built or rebuilt, once every component is up.

Overview #

Each build phase checks itself: the Ansible roles assert what they deployed, and each change record carries its own verification. This page is the end-to-end pass: a short list of checks that, together, say the site is up.

• Run it from the Builder, on the management segment, which reaches every zone. Tenant-facing segments are checked from a real client instead: see From the client segments.

• Use the Deevnet Root CA for every TLS check. The Builder trusts it at the OS level, so --cacert is optional there; naming it keeps the check honest on any other machine. The inventory’s copy is ansible-inventory-deevnet/pki/deevnet-root-ca.pem:

  CA=/srv/dvnt/ansible-inventory-deevnet/pki/deevnet-root-ca.pem
  

  OpenBao’s listener is no exception: its certificate comes from the Substrate CA like every other (ADR-0031 §4).

• Addresses are from inventory. Names are in the mobile.deevnet.net zone and resolve through the core router, 10.20.99.1.

Rewritten 2026-09-26 from inventory and the change records, for the domain-VM layout of ADR-0013. Expected results marked “recorded” were measured in the change record named. The others are the service’s standard health endpoint and have not yet been run as part of this page.

Network #

# Core router answers on management. Not ping: management reaches the router itself on 443 and 22 only
curl -sk -o /dev/null -w '%{http_code}\n' https://10.20.99.1/   # 200

# DNS: the router answers for site names, and forwards tenant zones
dig +short @10.20.99.1 dv02hyp001p01.mobile.deevnet.net      # 10.20.99.21
dig +short @10.20.99.1 api.mobile.deevnet.net                # 10.20.25.20
dig +short @10.20.99.1 tdemo-1.tdemo.mobile.deevnet.net      # 10.20.129.10, if tdemo is applied

• DHCP is checked from a client: a lease in the right subnet is the first thing Segment Check tests.
• Zone policy is checked by the firewall role’s own plan (no drift), and from clients by Segment Check. Pinging across segments from the Builder proves little, since management reaches every zone.

Hypervisors #

ping -c 3 dv02hyp001p01.mobile.deevnet.net    # management hypervisor, 10.20.99.21
ping -c 3 dv02hyp002p02.mobile.deevnet.net    # tenant hypervisor, 10.20.99.22

# Proxmox answers, verified against the Deevnet Root CA (401: no token, which is expected)
curl --cacert $CA -s -o /dev/null -w '%{http_code}\n' https://dv02hyp001p01.mobile.deevnet.net:8006/api2/json/version
curl --cacert $CA -s -o /dev/null -w '%{http_code}\n' https://dv02hyp002p02.mobile.deevnet.net:8006/api2/json/version

# SSH as the automation user
ssh a_autoprov@dv02hyp001p01.mobile.deevnet.net hostname

A 401 from /api2/json/version means the API is up and wants a token: that passes. A timeout fails.

Management-plane services #

The substrate service VMs, and one check each.

Then reconcile one tenant. It exercises the backends a reconcile re-ensures (DNS and its forwarding, the state store, the tenant network, the log store and Grafana) in one call, and a failing backend is named in the response. Broker accounts, Wi-Fi keys and workloads are not part of a reconcile:

# OPERATOR_TOKEN is vault_deevnet_api_token, in the deevnet_api group vault
curl --cacert $CA -X POST -H "Authorization: Bearer $OPERATOR_TOKEN" \
  https://api.mobile.deevnet.net/v1/tenants/tdemo/reconcile

The tenant ends ready, and every step succeeded. A 502 names the backend that is down (recorded: CHG-0020).

Tenant DNS #

The table row above proves the server answers. To look further:

# through the resolver, which is what clients actually do
dig @10.20.99.1 tdemo.mobile.deevnet.net SOA

# the apex names the server, never a.misconfigured.dns.server.invalid
dig +short @10.20.25.21 tdemo.mobile.deevnet.net NS

# what the server actually holds (on dv02idn001v01)
sudo podman exec pdns-auth pdnsutil list-all-zones
sudo podman exec pdns-auth pdnsutil list-zone tdemo.mobile.deevnet.net

The namespace boundary is worth testing rather than assuming: an update signed with one tenant’s key and aimed at another tenant’s zone must be REFUSED by the server.

PXE infrastructure #

Only while the Builder is serving PXE, in bootstrap mode ( Authority Transition). On the Builder:

systemctl status tftp.socket
ls /srv/tftp/pxelinux.cfg/

From the client segments #

Run Segment Check from a laptop on each SSID. It proves what the Builder can’t: a lease in the right subnet, names answered by the segment’s own gateway, and that everything the zone policy denies is really dropped.

bash segment-check.sh DVNTM-TD     # tenant dev
bash segment-check.sh DVNTM-IOT    # IoT

Recorded results for comparison: DVNTM-TD 29/29 and DVNTM-IOT 15/15 ( CHG-0024, CHG-0025).

Automated verification #

Not built. One playbook that runs the checks above and produces a single pass/fail report is on the Extended Management Plane roadmap, under Build Verification.