Substrate on Deevnet IoTaaS

ADR-0037: Tenant Service Certificates

Mon, 01 Jan 0001 00:00:00 +0000

ADR-0037: A Tenant Service CA Issues Server Certificates for Names in a Tenant’s Own Zone #


Status	Accepted (2026-10-11): built by CHG-0047
Date	2026-10-11
Scope	Where a tenant workload’s own service gets a server certificate that chains to the Deevnet Root CA: which CA signs it, who may ask, for which names, and what the certificate says. Not device identity ( ADR-0031 §6, unchanged), not who may reach a tenant’s service, and not revocation, which stays open.
Extends	ADR-0031: answers its open question 2, and adds a third issuing CA to §3’s two.
Amends	ADR-0031 §3 (“Two issuing CAs per site”) and §4 (“ADR-0016 §3’s `pki/` row becomes: issues tenant device certificates only”): OpenBao also issues tenant service certificates. Certificates standard 1.2 and the tables of §1, §2, §4, §5 and §6.
Related	ADR-0012 (the Deevnet API), ADR-0015 §13 (a tenant’s names), ADR-0016 (OpenBao), ADR-0030 §5–§6 (lifetimes and renewal), ADR-0036 (the substrate’s own services on 443)

Context #

A tenant workload cannot serve HTTPS that the site trusts. The two issuing CAs each have one job, and neither is this one:

ADR-0031: Deevnet PKI

Mon, 01 Jan 0001 00:00:00 +0000

ADR-0031: An Offline Deevnet Root, Offline Site CAs, and Separate Substrate and Tenant Device Identity #


Status	Accepted (2026-10-05): built by CHG-0033. §6, device identity (mTLS), is decided and not yet built: CHG-0034.
Date	2026-10-03
Scope	The certificate hierarchy for every Deevnet site: the root and each site’s CA and who holds them, the issuing CAs and what each may issue, subject names, and the target for tenant device identity (mTLS). Not automatic renewal or expiry alerting ( ADR-0030 open questions 1–2, unchanged).
Supersedes	ADR-0030 §1–§4
Amends	ADR-0016 §3 (the `pki/` row)
Extended by	ADR-0037: a third issuing CA per site, the Tenant Service CA, for server certificates in a tenant’s own zone. It answers open question 2 and amends §3 and §4.
Related	ADR-0011 §4, ADR-0012 §8, ADR-0020 §2, ADR-0021 §8, CHG-0031, CHG-0032

Context #

ADR-0030 gave the site a certificate hierarchy, and CHG-0031 and CHG-0032 built it:

ADR-0030: Site Certificate Hierarchy

Mon, 01 Jan 0001 00:00:00 +0000

ADR-0030: An Offline Site Root, an OpenBao Intermediate, and Certificates Laid Down by the Build #


Status	Accepted (2026-10-05) for §5–§8, built by CHG-0033 with “issued by OpenBao” read as “issued by the Substrate CA” ( ADR-0031 §7). §1–§4 superseded by ADR-0031 (an offline Deevnet root and Site CAs; the substrate issued by Ansible, not OpenBao).
Date	2026-10-02
Scope	Where the site’s root of trust lives, what issues TLS certificates, how long they last, how a build or repave puts certificates and trust on substrate hosts, appliances, images and the operator’s computer, and how tenants keep trusting it. Not automatic renewal, not expiry alerting, not client certificates.
Extends	ADR-0016 §3 (the `pki/` row) and §6 (rebuild order); answers its open question 1
Related	INC-0003, CHG-0030, ADR-0023

Context #

OpenBao’s pki/ mount is the site’s certificate authority (ADR-0016 §3). It holds a self-generated root, CN=Deevnet mobile internal CA, valid ten years, and one role, platform, that issues 90-day leaves. Ansible issues six of them as it builds each service: the Deevnet API, the state store, the broker, the log store’s proxy, Grafana and tenant downloads. Tenants, the provider and the scripts that check the site trust that root as site-ca.pem, passed to each tool by file.

ADR-0016: Substrate Secrets in OpenBao

Mon, 01 Jan 0001 00:00:00 +0000

ADR-0016: The Substrate Keeps Its Runtime Secrets in OpenBao #


Status	Accepted
In conflict with	ADR-0033: Code Is the State (Proposed): OpenBao’s data as kept data (§1, Consequences). To be revisited; the open part is the Tenant Device CA’s key
Accepted	2026-09-19, recording what CHG-0010 built on 2026-09-17. The record’s own Current state already read “Proposed, and deployed”; the status field had simply not caught up.
Date	2026-09-17
Scope	Where the credentials substrate services use at runtime are kept, how tenant secrets are protected at rest and in transit, and where the internal certificate authority lives
Extends	ADR-0015: Tenants Are Built Through the Deevnet API, whose API holds credentials for PowerDNS, the core router, the state store and the tenant hypervisor, and stores tenant secrets
Extended by	ADR-0021: Tenant Secrets (Proposed), the record §8 asked for on tenant namespaces
Related	ADR-0012: IoT Platform Services Through a Deevnet API and Terraform Provider §9 (credential delivery), ADR-0013: Management-Hypervisor Services Run as Containers on Domain VMs, ADR-0014: Tenant State Durability

Context #

How the API’s credentials are handled in ADR-0015’s first slice #

Credential	Kept in	Reaches the API as	Can do
Operator token, database password	inventory vault	a root-only env file, then container environment	the whole API; its database
PowerDNS HTTP API key	inventory vault; also in `pdns.conf` on the DNS host	the same env file	every zone and key on the server
Core router API key	the router automation user’s vault entry	the same env file	everything that user may do on the router
State-store admin	inventory vault	the same env file	user and policy administration
Proxmox token	inventory vault	the same env file	what ADR-0015 grants it on the tenant hypervisor

And for what is handed to tenants:

ADR-0013: Management-Hypervisor Services Run as Containers on Domain VMs

Mon, 01 Jan 0001 00:00:00 +0000

ADR-0013: Management-Hypervisor Services Run as Containers on Domain VMs #


Status	Accepted
Accepted	2026-09-16, once CHG-0008 built all six domain VMs and moved the controller, tenant DNS and the tenant state store onto them
Date	2026-09-14
Scope	How services on the management hypervisor are grouped into VMs and named, where the site’s Omada controller officially runs, and the order a rebuild uses them in
Extends	ADR-0009: Network Device Configuration Is Inventory-Owned and Controller-Applied, which decided that the controller is the actuator for switch and AP configuration, but not where it runs
Extended by	ADR-0014: Tenant State Durability — the provisioning VM holds both the state store and the API’s database, so their data needs a data disk and a copy on separate hardware (Superseded by ADR-0033); ADR-0025: Identity Directory — Keycloak in the identity VM, with a realm per tenant (Proposed)
Superseded in part by	ADR-0022: Central Logging — §5, for logs only: one log store on Platform instead of one per audience (Accepted). The two observability VMs are replaced as a result, `sob` by `dv02col001v01` (collector) and `tob` by `dv02obs001v01` (store), in CHG-0018 (Planned)
Related	ADR-0004: Tenant DNS Publication, ADR-0008: Host Naming and Site Codes, ADR-0012: IoT Platform Services Through a Deevnet API and Terraform Provider (its Open questions 2, 5 and 6), CHG-0005

Context #

One VM per function is where things were heading #

The management hypervisor, dv02hyp001p01, runs one VM per service today:

ADR-0009: Network Device Configuration Is Inventory-Owned and Controller-Applied

Mon, 01 Jan 0001 00:00:00 +0000

ADR-0009: Network Device Configuration Is Inventory-Owned and Controller-Applied #


Status	Accepted
Date	2026-09-10
Accepted	2026-09-10
Scope	Who owns, and who applies, the configuration of controller-managed network devices — the access switch and the AP — and how automation talks to the Omada controller
Extended by	ADR-0013: Management-Hypervisor Services Run as Containers on Domain VMs — where the controller runs (a container in the network management VM, `nms`), and the order a rebuild uses it in
Related	CHG-0004 — brings the controller current; CHG-0005 — the AP, the first device adopted under this record; CHG-0006 — the switch’s firmware; INC-0001 — the guards any reconciling role now carries

Context #

The mobile site’s network devices are configured three different ways today: