CHG-0030: The State Store Over TLS

CHG-0030: The State Store Over TLS #

Date2026-09-27
Change typeConfiguration
ClassificationStructural
StatusComplete, 2026-09-27. The state store serves TLS only, from the site CA; every client verifies it. Proven from DVNTM-TD (29/29).
Window2026-09-27, 12:41 to 13:06
Sitemobile
Systemsdv02prv001v01 (MinIO, the Deevnet API), dv02idn001v01 (OpenBao issues one certificate), the tdemo, eds and mabell backends
Automationdeevnet.mgmt site.yml --tags tenant-state and --tags deevnet-api, --tags tenant-downloads; deevnet-provisioning-api make stage; terraform-provider-deevnet make stage. Inventory ansible-inventory-deevnet/mobile
RiskMedium — every tenant backend and the API’s admin client stop working at the moment MinIO switches, until each is pointed at https
Related changesCHG-0029, whose isolation needs this
Related incidentsINC-0003 (a rebuilt OpenBao’s new root, which the reissue check covers)
Related runbooksState Store

Summary #

The state store, MinIO on dv02prv001v01, is the last service a tenant reaches that is plain HTTP. A tenant’s Terraform state holds every secret the substrate issued it, and on DVNTM-TD it crosses a segment whose client isolation stops IP but not ARP spoofing ( ADR-0029 §4). ADR-0016 already says it should be TLS.

After this change MinIO serves TLS on :9000 with a certificate from the site CA:

  • Tenants reach it at https://tfstate.mobile.deevnet.net:9000, verified with the site-ca.pem they already hold.
  • The Deevnet API reaches it by address, verified with the CA it already mounts.
  • The role’s own checks reach it at 127.0.0.1.

No client can skip verification.

Goal #

  • curl --cacert site-ca.pem https://tfstate.mobile.deevnet.net:9000/minio/health/live answers 200 from DVNTM-TD. A plain http:// request gets only Go’s 400 Client sent an HTTP request to an HTTPS server, and no data.
  • The certificate names tfstate.mobile.deevnet.net, dv02prv001v01.mobile.deevnet.net, the host address and 127.0.0.1, and is issued by the site CA.
  • The API reports v0.9.0 or later. A tenant create and delete (throwaway tprobe) succeed, which proves the admin client works over TLS.
  • terraform plan works from tdemo, eds and mabell with the https endpoint and custom_ca_bundle.
  • deevnet_tenant.state_endpoint reads https://… for every tenant.
  • tenant-check.sh and segment-check.sh report the state store as TLS verified.

Scope #

In scope:

  • MinIO’s certificate and listener.
  • The API’s admin client (MINIO_ADMIN_TLS, MINIO_ADMIN_CACERT).
  • The endpoint tenants are told.
  • The three existing backends.
  • The two check scripts, and the tenant state-store page.

Out of scope:

  • MinIO’s console on :9001. It is not reachable from any tenant segment. It gets TLS with the same certificate as a side effect, but is not verified here.
  • Replacing MinIO ( ADR-0026).
  • A second copy of the state ( ADR-0014).

Risk and impact #

RiskWhereGuard
Every backend and the API break together at the switchprvSteps 2 and 3 run back to back. The tenants’ backends switch in step 4, and nothing is lost meanwhile: state stays in the bucket, and a failed init changes nothing
mc inside the container cannot verify the listenerprvThe container’s SSL_CERT_FILE is the site CA, and podman exec inherits it. If it does not, the bucket step fails loudly and the role stops. That this holds is inferred from how podman behaves, not documented
The certificate goes stale when OpenBao is rebuiltprvThe role compares the host’s CA with OpenBao’s on every run and reissues on a change (the INC-0003 lesson)
A tenant runs init without the CAlaptopIt fails with an x509 error, not silently; the tenant page says what to add

Prerequisites #

  • PRs merged: deevnet-provisioning-api, terraform-provider-deevnet, ansible-collection-deevnet.mgmt, ansible-collection-deevnet.net and ansible-inventory-deevnet (all chg-0030-state-store-tls). tdemo’s backend.tf already names https and the site CA (the reference-tenant rework, deevnet-tenant-tdemo #7)
  • Vault decrypted, collections built

Procedure #

Step 1: Stage the API and provider #

One release carries all three changes. The API and provider on main hold the code for CHG-0028, CHG-0029 and CHG-0030 together, and each behavior is off until that change’s own role or inventory switch turns it on. So the first of the three to run tags and stages API v0.9.0 and provider 0.5.0, and the others deploy that same release. If it is already staged, skip the tag.

Run:

cd deevnet-provisioning-api && git tag v0.9.0 && git push origin v0.9.0 && make stage   # unless staged
cd ../terraform-provider-deevnet && git tag v0.5.0 && git push origin v0.5.0 && make stage   # unless staged

Verify: both artifacts are on the Builder.

Undo: nothing to undo.

Step 2: TLS on MinIO #

Run (in ansible-collection-deevnet.mgmt):

ansible-playbook playbooks/site.yml --tags tenant-state --limit dv02prv001v01

Verify:

  1. The role’s own health check passes over https://127.0.0.1:9000, verified against the site CA.
  2. openssl s_client -connect tfstate.mobile.deevnet.net:9000 -CAfile site-ca.pem gives Verify return code: 0 (ok), and the SANs are the four in the Goal.
  3. The bucket step’s mc alias set local https://127.0.0.1:9000 succeeds, which proves mc inside the container verifies.

Undo: Undo Step 2

Step 3: The API #

Run:

ansible-playbook playbooks/site.yml --tags deevnet-api --limit dv02prv001v01

Verify: the API reports v0.9.0 or later, and its environment has MINIO_ADMIN_TLS=true, MINIO_ADMIN_CACERT and DEEVNET_STATE_ENDPOINT=https://…. Admit, create and delete tprobe.

Undo: Undo Step 3

Step 4: The tenants’ backends #

Run: in the eds and mabell repositories, change the backend’s endpoint to https://tfstate.mobile.deevnet.net:9000 and add custom_ca_bundle = "site-ca.pem". tdemo’s backend.tf already has both. Then, in all three:

terraform init -reconfigure
terraform plan

Verify: init succeeds against the same bucket and key, and plan shows only state_endpoint changing to https.

Undo: none needed. A backend that has not switched just fails to init until it does.

Step 5: Scripts, downloads, docs #

Run: ansible-playbook playbooks/site.yml --tags tenant-downloads --limit dv02obs001v01 (the updated tenant-check.sh), then merge the docs PR for the tenant State Store page.

Verify: from DVNTM-TD, tenant-check.sh and segment-check.sh report the state store as TLS verified.

Verification #

The Goal, as a whole, from a laptop on DVNTM-TD.

Undo #

Undo Step 2 #

Set minio_tls_enabled: false in group_vars/tenant_state/vars.yml and run the role again. MinIO serves plain HTTP again; the certificate files are left in place and ignored.

Undo Step 3 #

Set deevnet_api_minio_tls: false, and set deevnet_api_state_endpoint back to http://…, then deploy the API again.

Outcome #

Run from the Builder on 2026-09-27, and verified last from a computer on DVNTM-TD.

WhenStepsWhat happened
12:41StartRecord In progress. /home on the Builder was 92% full; pruned one dangling image (2.4 GB free)
12:421API tagged v0.9.0 (75fdf40) and staged; provider tagged 0.5.0 and staged with the scripts. Both carry the CHG-0028 and CHG-0029 code too, off until their own switches
~12:502inventory #62 merged. site.yml --tags tenant-state: ok=45 changed=11 failed=0. The role’s health check passed over https://127.0.0.1:9000 against the site CA, and mc alias set local https://127.0.0.1:9000 passed, so podman exec does inherit SSL_CERT_FILE. From the Builder: Verify return code: 0 (ok); SANs tfstate.mobile.deevnet.net, dv02prv001v01.mobile.deevnet.net, 10.20.25.20, 127.0.0.1; issuer the site CA; expires 2026-12-26. https health 200; plain http gets only 400 Client sent an HTTP request to an HTTPS server
~13:003mgmt #53 merged (version bump). site.yml --tags deevnet-api: ok=77 changed=9 failed=0. /version is v0.9.0, /readyz 200; env has MINIO_ADMIN_TLS=true, MINIO_ADMIN_CACERT, DEEVNET_STATE_ENDPOINT=https://…; CHG-0028/0029 settings unchanged. Operator-created tprobe (index 4): 201, and its own state credentials put, read and deleted an object in tenants/tprobe/ over TLS and got 403 in tenants/eds/. Deleted: 204, then 404
~13:104eds and tdemo backends to https + custom_ca_bundle (eds and tdemo PRs merged). terraform init -reconfigure succeeded against the store over TLS for both; plan shows only state_endpoint http → https, no resource changes
~13:205net #39 merged. site.yml --tags tenant-downloads: ok=34 changed=1 failed=0. Downloads serve provider 0.4.1 and 0.5.0; the served tenant-check.sh, run from the Builder, reports the API, the state store, the broker, the log store, Grafana and downloads all TLS verified

| 13:06 | Verification | From a computer on DVNTM-TD (10.20.45.50): segment-check.sh DVNTM-TD 29 passed, 0 failed, the state store REACH https://tfstate.mobile.deevnet.net:9000 - HTTP 403 (TLS verified) among them. Log below |

segment-check.sh DVNTM-TD, 2026-09-27 13:06
```text Segment check: DVNTM-TD (from 10.20.45.50) == Address and DNS PASS lease 10.20.45.50 PASS api.mobile.deevnet.net -> 10.20.25.20 PASS tfstate.mobile.deevnet.net -> 10.20.25.20 PASS mqtt.mobile.deevnet.net -> 10.20.35.20 PASS downloads.mobile.deevnet.net -> 10.20.25.22 PASS REACH https://api.mobile.deevnet.net:8080 - HTTP 404 (TLS verified) PASS REACH https://tfstate.mobile.deevnet.net:9000 - HTTP 403 (TLS verified) PASS mqtt.mobile.deevnet.net:8883 TLS verified PASS dv02obs001v01.mobile.deevnet.net:8427 TLS verified PASS REACH https://dv02obs001v01.mobile.deevnet.net:3000 - HTTP 302 (TLS verified) PASS REACH https://downloads.mobile.deevnet.net:8443 - HTTP 200 (TLS verified) PASS internet (https://example.com 200) PASS BLOCK obs-ssh(platform) 10.20.25.22:22 PASS BLOCK Builder-ssh(management) 10.20.99.95:22 PASS BLOCK router-GUI(management) 10.20.99.1:443 PASS BLOCK hypervisor-PVE(management) 10.20.99.21:8006 PASS BLOCK router-GUI-on-own-gateway 10.20.45.1:443 PASS BLOCK router-ssh-on-own-gateway 10.20.45.1:22 PASS BLOCK router-on-trusted 10.20.10.1:443 PASS BLOCK prv-ssh(platform) 10.20.25.20:22 PASS BLOCK prv-other-port(platform) 10.20.25.20:8200 PASS BLOCK msg-ssh(iot_backend) 10.20.35.20:22 PASS BLOCK broker-plaintext(iot_backend) 10.20.35.20:1883 PASS BLOCK tenant-workload 10.20.130.10:22 PASS BLOCK pi(iot,if-on) 10.20.30.11:22 PASS BLOCK pi(iot,if-on) 10.20.30.12:22 PASS BLOCK pi(iot,if-on) 10.20.30.13:22 PASS BLOCK pi(iot,if-on) 10.20.30.14:22 PASS BLOCK edge-router-admin(CHG-0023) 192.168.8.1:80 RESULT: DVNTM-TD - 29 passed, 0 failed ```

Departures from the plan #

  • The API version bump was missing from the procedure. The deevnet_api role pins deevnet_api_version (it was v0.8.0), so step 3 as written would have redeployed the old API without the TLS client. Added as mgmt #53 before step 3.
  • mabell was not affected. Its state was never moved into the store (its backend block is commented out), so step 4 was eds and tdemo only.
  • tdemo needed terraform init -upgrade. Its local lock file pinned the provider build 0.1.0, which was no longer installed. Nothing to do with TLS.
  • The Builder’s own Terraform plugin mirror holds provider 0.1.0 to 0.4.0 only; the tenants install from the downloads site, so it was not updated here.

Follow-ups #

  • The console on :9001: decide whether it should be reachable at all.
  • segment-check.sh expects DVNTM-TD → tenant workload :22 to be blocked; CHG-0028 opens it, and must flip that check in the same change.
Page last modified: September 27, 2026