CHG-0031: The Site Root and Its Intermediate #

Summary #

Builds ADR-0030 §1–§2 and §4–§7, everything but the appliances.

Today OpenBao’s pki/ mount holds a root it generated itself, CN=Deevnet mobile internal CA, and six Platform services serve 90-day certificates from it. Tenants and scripts trust it as site-ca.pem. Rebuilding OpenBao without its data re-roots the site (INC-0003).

After this change:

• The root is offline. CN=Deevnet mobile root CA, P-256, valid to 2046-10-02, key in the inventory vault. It is the only trust anchor, named deevnet-mobile-root-ca.pem everywhere.
• OpenBao issues from an intermediate under it, CN=Deevnet mobile intermediate CA, whose key OpenBao generated and keeps. A rebuilt OpenBao gets a new intermediate; nobody gets a new CA.
• Services serve one-year certificates as leaf + intermediate, laid down by the roles that build them and renewed by certs.yml.
• The Builder, the hypervisors, every management-plane VM and the VM template trust the root at the OS level.
• A bootstrap intermediate exists for CHG-0032 to sign the core router and the hypervisors with.

Goal #

• curl https://api.mobile.deevnet.net:8080/ from the Builder verifies with no --cacert.
• Every service presents two certificates, leaf then CN=Deevnet mobile intermediate CA, and openssl verify -CAfile deevnet-mobile-root-ca.pem accepts each: API :8080, state store :9000, broker :8883, log store :8427, Grafana :3000, downloads :8443.
• Each leaf is valid for a year.
• https://downloads.mobile.deevnet.net:8443/deevnet-mobile-root-ca.pem serves the root, SHA-256 68:D5:C9:8E:3D:2E:B2:DF:B6:1B:99:E4:F3:4D:F9:D3:B4:65:C3:66:34:97:30:97:36:B7:7B:60:C4:15:2C:6B.
• tenant-check.sh passes for each tenant, and terraform plan works from tdemo, eds and mabell.
• Device logs still reach the log store through the bridge, and Grafana’s data sources answer for each tenant.
• A second run of certs.yml changes nothing.

Scope #

In scope:

• The root and bootstrap intermediate (site-root-ca.yml).
• OpenBao’s intermediate (openbao role).
• The six services’ certificates (site_cert role), the log bridge’s and egress agent’s root.
• The root in the OS trust store (site_trust role) and in the Fedora template.
• The rename to deevnet-mobile-root-ca.pem: tenant repositories, scripts, downloads.
• On the Pi, deevnet-kit exports its own CA as deevnet-kit-ca.pem.

Out of scope:

• Proxmox, the core router and the Omada controller (CHG-0032).
• OpenBao’s own listener, PowerDNS’s API, the artifact server (ADR-0030 open question 3).
• Automatic renewal and expiry monitoring (ADR-0030 open questions 1–2).

Risk and impact #

Prerequisites #

• ADR-0030 merged (#236, #237)
• Step 1 done
• PRs merged (all chg-0031-*): ansible-collection-deevnet.mgmt, .builder, .net, ansible-inventory-deevnet, terraform-provider-deevnet, deevnet-tenant-tdemo, deevnet-provisioning-api (deevnet-kit), deevnet-image-factory; eds and mabell’s own
• Vault decrypted on the control node

Procedure #

Step 1: The site root and bootstrap intermediate #

Done 2026-10-02. site-root-ca.yml made both on the control node; the inventory was encrypted, committed and pushed (ansible-inventory-deevnet b0e18bc). Nothing live changed.

Run (in ansible-collection-deevnet.mgmt, inventory decrypted):

ansible-playbook playbooks/site-root-ca.yml
cd ../ansible-inventory-deevnet && make vault   # then commit and push

Verify: openssl verify -CAfile pki/mobile/deevnet-mobile-root-ca.pem pki/mobile/deevnet-mobile-bootstrap-ca.pem is OK, and the pushed vault.yml starts $ANSIBLE_VAULT.

Undo: remove both .pem files and both vault_site_* keys, then commit.

Step 2: Publish the collections #

Run:

cd ansible-collection-deevnet.builder && make publish   # deevnet.builder.site_trust, used by mgmt
cd ../ansible-collection-deevnet.mgmt && make install-dev

Verify: ansible-galaxy collection list deevnet.builder shows the new build, and ansible-playbook playbooks/certs.yml --syntax-check passes.

Undo: publish the previous main.

Done 2026-10-02. Merged mgmt #59, inventory #63, builder #21, image-factory #19 and this record (docs #238); make publish installed deevnet.builder with site_trust, and certs.yml, site.yml and openbao.yml pass --syntax-check. Before step 3, OpenBao’s default issuer is a5679c07-dbe3-9ac2-a0eb-bfe7f31d7412 (drill-rotated, the internal root the 2026-09-17 drill made): Undo Step 3 sets it back.

Step 3: OpenBao issues from the intermediate #

Run:

ansible-playbook playbooks/openbao.yml

Verify:

1. The play’s own check passes: pki/ca/pem verifies against the root.
2. pki/config/issuers names the new issuer as default; the old root’s issuer is still listed.
3. Services are untouched: each still serves its old certificate, so nothing has broken yet.

Undo: Undo Step 3

Step 4: The OS trust stores #

Run:

ansible-playbook playbooks/certs.yml --tags trust --limit '!dv02bld002v01'

Verify: on each host, openssl verify /etc/pki/ca-trust/source/anchors/deevnet-mobile-root-ca.pem (Proxmox: /usr/local/share/ca-certificates/deevnet-mobile-root-ca.crt) is OK. The play checks it.

Undo: remove the anchor file and run update-ca-trust extract / update-ca-certificates. Nothing depends on it yet.

Step 5: Reissue every service #

The rename changes container configuration (VerneMQ’s cafile, MinIO’s SSL_CERT_FILE, the API’s CA paths, the bridge’s), so this is site.yml, not certs.yml. Each role finds its certificate no longer chains to the root, reissues, and restarts.

Run, back to back:

ansible-playbook playbooks/site.yml --skip-tags vms --limit dv02prv001v01   # API, state store
ansible-playbook playbooks/site.yml --skip-tags vms --limit dv02msg001v01   # broker, bridge
ansible-playbook playbooks/site.yml --skip-tags vms --limit dv02obs001v01   # log store, Grafana, downloads
cd ../ansible-collection-deevnet.net && ansible-playbook playbooks/tenant-egress-agent.yml
cd ../ansible-collection-deevnet.mgmt && make reconcile NAME=--all

Verify:

1. For each service, openssl s_client -connect <name>:<port> -showcerts shows two certificates, and Verify return code: 0 (ok) with -CAfile the root. Broker and state store first: see Risk.
2. curl -fsS https://api.mobile.deevnet.net:8080/v1/version from the Builder with no --cacert.
3. The downloads role’s check passes: every CA name it serves is the inventory’s root.
4. The bridge’s health endpoint answers, and a device log line reaches the store.
5. make reconcile reports every tenant reconciled; each tenant’s Grafana data sources pass Save & test.
6. The egress agent’s next run succeeds (journalctl -u deevnet-egress-agent).

Undo: Undo Step 5

Step 6: The tenants #

Each tenant replaces site-ca.pem with the root, under its new name, once.

Run, for tdemo, eds and mabell (in each tenant directory, after its PR merges):

curl -fsSLk -O https://downloads.mobile.deevnet.net:8443/deevnet-mobile-root-ca.pem
openssl x509 -in deevnet-mobile-root-ca.pem -noout -fingerprint -sha256   # 68:D5:C9:8E:...:2C:6B
rm -f site-ca.pem
terraform init -reconfigure     # tdemo and eds: the backend's custom_ca_bundle changed
terraform plan                  # no changes

Then eds’s workload: regenerate deploy/kit.env from the outputs and make units in deploy/, so lightd mounts the root. cdeever is the operator’s own tenant on the Mac: the same steps there.

Verify: tenant-check.sh passes from DVNTM-TD; terraform plan shows no changes; eds’s lights respond.

Undo: keep the old site-ca.pem beside the new file until step 7 verifies.

Step 7: Scripts, docs, downloads #

Run:

1. Stage the provider’s install-provider.sh and tenant-check.sh onto the downloads mirror.
2. Merge the docs PR that renames the CA across the tenant and runbook pages.
3. Run certs.yml once more.

Verify: certs.yml reports changed=0; segment-check.sh DVNTM-TD passes from a computer on DVNTM-TD.

Verification #

The Goal, as a whole: from the Builder without --cacert, and from a laptop on DVNTM-TD with only deevnet-mobile-root-ca.pem.

Undo #

Undo Step 3 #

The old root is still an issuer on the mount. Set it back as the default:

# POST pki/config/issuers as Ansible's AppRole, with the id it listed before step 3
curl --cacert .openbao/dv02idn001v01-listener.pem -H "X-Vault-Token: $T" \
  -d '{"default":"a5679c07-dbe3-9ac2-a0eb-bfe7f31d7412"}' https://bao.mobile.deevnet.net:8200/v1/pki/config/issuers

OpenBao issues from the old root again. Nothing else changed in step 3.

Undo Step 5 #

Undo step 3, then deploy each service from the commit before the CHG-0031 merges. The old roles compare the host’s CA with pki/cert/ca, which is the old root again, so they reissue from it. Tenants keep their old site-ca.pem.

Outcome #

Steps 1–7 run on 2026-10-02 from the Builder, steps 3–7 between 12:39 and 13:10 UTC.

Step 3, OpenBao. The play signed CN=Deevnet mobile intermediate CA (P-256, CA:TRUE, pathlen:0, keyCertSign, cRLSign, valid to 2031-10-02) under the root and made it the default issuer, e706e3de-7bee-94c3-b407-ee87324ca988; the old default, a5679c07-… (drill-rotated), is still on the mount. The platform role’s max_ttl is one year. A second run: changed=0. Services still verified against the old CA afterwards.

Step 4, trust stores. The root is an OS anchor on all 11 hosts: the Builder, both hypervisors, the eight management-plane VMs including the test VM. A second run: changed=0.

Step 5, services. Every service presents two certificates, leaf then the intermediate, and verifies against the root alone:

Both open questions in the Risk table closed: MinIO serves the chain from public.crt, and VerneMQ from certfile, so the cafile fallback was not needed. Downloads serves the root under /deevnet-mobile-root-ca.pem and both old names, fingerprint 68:D5:C9:8E:…:2C:6B each. The log bridge resubscribed; the egress agent’s run succeeded over the new chain; make reconcile NAME=--all reconciled tdemo, eds, mabell and cdeever, and all twelve of their Grafana data sources pass Grafana’s health check.

Step 6, tenants. tdemo, eds and mabell fetched the root, checked its fingerprint and dropped site-ca.pem. eds and tdemo reconfigured their backends against the state store with the root.

• eds: the plan changes outputs only (broker.ca, kit_env). make units put the root on the workload; lightd reconnected to the broker over TLS.
• mabell: the plan changes outputs only. It also reports state_endpoint http → https, read from the API: CHG-0030’s change, not applied to mabell’s local state since.
• tdemo: its state reads over the new chain, but its plan is not empty, for a reason outside this change: the live tenant predates the tdemo #7 rework (workload.app, dns_record.service), and the plan is that rework (5 to add, 2 to destroy). Not applied.
• cdeever: not done; it lives on the operator’s computer.

Step 7. install-provider.sh and tenant-check.sh, rendered with the versions already staged (0.5.0, Grafana 4.46.0) rather than by make stage, which would have rebuilt the provider zips and broken every tenant’s lock-file hashes, are on the downloads server with the new root embedded. The runbook rename merged (docs #239). A full certs.yml run: changed=0 on all 11 hosts.

Departures from the plan #

• The PKI certificates moved out of the inventory directory (step 2). Step 1 wrote them to mobile/pki/, and Ansible parses every file there as an inventory source: every run warned five times that it could not parse them (hosts still loaded). They are at pki/mobile/ at the inventory repository’s root (inventory #64, mgmt #60, image-factory #20).
• The broker could not read its new root, and the log bridge was down 12:46–12:49 (step 5). The broker’s tls directory is mounted :Z, and podman applies the container’s private SELinux label only when it creates the container. The new root file got the host default, var_t; the broker, only restarted, listed its listener as running and reset every handshake (errno=104). Repaired by hand with chcon --reference, then fixed in site_cert, which now gives its files their directory’s label, and in vernemq, whose readiness is now a verified handshake (mgmt #61). Proven by setting the label back to var_t: the role relabelled, restarted and passed. On obs, the fixed role relabelled vmauth’s, Grafana’s and downloads’ new root files, which would otherwise have broken the same way.
• site_trust’s own check needed openssl (step 4), which dv02bld002v01 lacks, and the first replacement read /etc/pki/tls/certs/ca-bundle.crt, which Fedora 44 no longer has. It now reads the bundle each OS builds (builder #22); the template’s check uses trust list (image-factory #21).
• The test VM was included in step 4, not limited out: it was up, and an anchor is harmless.

Follow-ups #

• CHG-0032: Proxmox, the core router and the Omada controller on site certificates, signed by the bootstrap intermediate (Proxmox, the router) and OpenBao (Omada); clients stop skipping verification.
• Rebuild the Fedora template so new clones carry the root (the template change is merged; the build is not part of this window).
• Release deevnet-kit with the deevnet-kit-ca.pem export, and rebuild the Pi image.
• Retire the old root’s issuer from the pki/ mount once nothing it signed is in service.
• Remove the old download names (/deevnet-mobile-ca.pem, /site-ca.pem) after one release.
• Apply eds and mabell once, so the state holds the new output values (broker.ca, kit_env); and decide tdemo’s pending #7 rework.
• Remove Grafana’s deleted-* organizations, which tenant deletions leave behind with no data sources.