CHG-0047: Server Certificates for Tenants’ Services #

Summary #

A tenant workload cannot serve HTTPS that the site trusts: the Substrate CA certifies only the substrate, and the Tenant Device CA only devices. The eds tenant serves a console over plain HTTP for that reason.

ADR-0037 decides a third issuing CA, the Tenant Service CA, in OpenBao. After this change a tenant sends a signing request through the Deevnet API for a name in its own zone and receives a one-year server certificate that chains to the Deevnet Root CA. The eds console is the first to use one.

Goal #

• OpenBao has a mount pki-tenant-service whose CA is the Mobile Tenant Service CA, chaining to the Deevnet Root CA through the Mobile Site CA.
• The mount has one signing role. It signs a request for console.eds.mobile.deevnet.net, and refuses one for grafana.mobile.deevnet.net, for mobile.deevnet.net, for a wildcard and for a name outside deevnet.net.
• The API’s OpenBao policy allows signing on that role and nothing else on the mount.
• A tenant’s deevnet_certificate applies and returns a leaf that meets the Certificates standard: the subject, names, usages and lifetime of a tenant service certificate, whatever the signing request asked for.
• One tenant cannot obtain a certificate for a name in another tenant’s zone.
• https://console.eds.mobile.deevnet.net/ opens in a browser that trusts only the Deevnet Root CA.
• The Tenant Device CA, its mount and every existing certificate are unchanged.

Scope #

In scope: the mount, the CA and its ceremony, the signing role and the API’s policy; the API’s certificate route and the provider’s resource; the API’s release and deployment; one tenant service moved to HTTPS.

Out of scope: revocation; renewal by anything but a tenant’s own apply; expiry alerting; device certificates (CHG-0034); where OpenBao’s CA keys are kept (CHG-0036); the tenant runbook and the architecture pages, which are written when this is built.

Risk and impact #

Prerequisites #

• Pull requests merged: deevnet-docs (ADR-0037, this record, the standard) and deevnet.mgmt
• Pull request merged: the inventory. Not done before the change
• The inventory’s vaults re-encrypted and its working branch closed. Not done before the change
• Vault decrypted, collections built
• The ceremony kit to hand: the offline Pi, the pi-pki media, the key media, the transfer media and the paper record
• deevnet-provisioning-api and terraform-provider-deevnet merged with the certificate route and resource. Released during step 5, not before

Procedure #

Step 1: Read what is there #

Read-only.

Run:

curl -s --cacert pki/deevnet-root-ca.pem https://<openbao>:8200/v1/pki-tenant-device/ca/pem | openssl x509 -noout -subject -fingerprint -sha256
curl -s -o /dev/null -w '%{http_code}\n' --cacert pki/deevnet-root-ca.pem https://<openbao>:8200/v1/pki-tenant-service/ca/pem

Verify:

1. The Tenant Device CA is the certificate in the inventory.
2. pki-tenant-service does not exist.

Undo: nothing to undo.

Step 2: The mount, the key and the request #

Not disruptive. Creates the mount and, inside OpenBao, the Tenant Service CA’s key.

Run:

cd ansible-collection-deevnet.mgmt
ansible-playbook playbooks/tenant-service-ca.yml

Verify:

1. pki/mobile/deevnet-mobile-tenant-service-ca.csr is written, with subject O=Deevnet, OU=Mobile Site, CN=Deevnet Mobile Tenant Service CA and an RSA 3072 key.
2. The mount exists and has no CA.
3. The Tenant Device CA reads as it did in step 1.

Undo: Undo Step 2

Step 3: The ceremony #

Offline, by the operator: Issuing CA, with --ca tenant-service.

Verify:

1. deevnet-pki-transfer.sh accept passes and writes pki/mobile/deevnet-mobile-tenant-service-ca.pem.
2. Its fingerprint is the one on paper.

Undo: none. A signed CA certificate cannot be withdrawn; an unused one is left uninstalled.

Step 4: Install the CA, and the signing role #

Not disruptive. Nothing can call the role yet.

Run:

ansible-playbook playbooks/tenant-service-ca.yml
ansible-playbook playbooks/tenant-service-ca.yml      # a second run changes nothing

Verify:

1. GET pki-tenant-service/ca/pem is the certificate in the inventory, and chains to the root.
2. With the automation’s own token, the role signs a request for probe.eds.mobile.deevnet.net. The leaf has the subject, names and usages of a tenant service certificate, a CA:FALSE it did not take from the request, and one year.
3. The role refuses grafana.mobile.deevnet.net, mobile.deevnet.net, *.eds.mobile.deevnet.net, example.com and an IP address.
4. A request that asks for CA:TRUE, clientAuth and a second name gets none of them.

Undo: Undo Step 4

Step 5: The API #

Restarts the API.

Run:

ansible-playbook playbooks/site.yml --tags openbao --limit openbao      # the API's policy
ansible-playbook playbooks/site.yml --tags deevnet-api --limit deevnet_api

Verify:

1. GET /version is the new version.
2. A tenant is read through the API as before.
3. With a tenant’s token, a certificate is issued for a label; with another tenant’s token, the same path answers 404.

Undo: Undo Step 5

Step 6: The first tenant service #

The eds tenant’s own change, recorded in its own records. Its console moves to HTTPS.

Verify:

1. openssl s_client -connect console.eds.mobile.deevnet.net:443 -CAfile deevnet-root-ca.pem -verify_return_error verifies with the root alone.
2. The page opens in a browser on the trusted network with no warning.

Verification #

The Goal’s seven facts, each read after step 6.

Undo #

Undo Step 5 #

Set the API’s version back and run the deevnet-api play; remove the policy path and run the openbao play. Certificates already issued stay valid until they expire.

Undo Step 4 #

Delete the role: DELETE pki-tenant-service/roles/<role>. The CA stays installed and can sign nothing.

Undo Step 2 #

Disable the mount: DELETE sys/mounts/pki-tenant-service, which destroys the key. Remove the request from the inventory. Practical until step 3; after it, the Site CA has signed a certificate for a key that no longer exists, which is harmless and stays on the paper record.

Outcome #

All seven of the Goal’s facts hold. One was read with curl and openssl, not by a person looking at a browser.

Departures from the plan #

• The API’s signing permission was written at step 2, not step 5. tenant-service-ca.yml writes every policy before it enables the mount, the API’s among them. The risk table had the role tested “before the API is given the policy”. The API then deployed was v0.10.0, which has no code that signs, so nothing could use the permission until step 5, and the role was tested before that. The order the plan wanted held in effect, not in the letter.
• Step 5’s first command was not run. The policy was already written, so there was nothing for an openbao play to add. OpenBao is also administered from playbooks/openbao.yml, not from site.yml with a limit, which the plan had wrong.
• Step 4’s verification became a playbook. The plan described the checks; they were written as playbooks/tenant-service-ca-verify.yml during the change, and that is what ran.
• The offline Pi’s clock was four hours behind. The certificate’s validity starts at 14:40:41 UTC on the day, about four hours before it was signed: local time, entered as UTC. It is valid now and expires four hours early in 2031. The signing tool refuses a date before its image was built, and has no way to notice one that is merely a few hours off.
• The Builder’s /home was full. make stage for the API failed with no space left on device: 20 GB of 20 GB, mostly container build layers and Go’s build cache, with this change’s own builds on top. The Go build cache and dangling images were cleared, which freed about 3.7 GB, and the second attempt staged. The tag had already been pushed.
• The API and provider were merged before they were tagged. Neither could be tested against a CA that had not been signed. Each was tagged only after step 4 had shown what the role really returns.
• The pages that describe what exists were written after, not before. Prerequisite 1 named the documentation; ADR-0037, this record and the standard were merged first, and Trust and Identity, the runbooks and the tenant’s page with this record’s completion.
• The inventory was not merged first. Its working tree held decrypted vaults on another branch when the change began, so the request and the certificate were written into it and left uncommitted. See Follow-ups.
• The offline ceremony’s menu still read “Substrate or Tenant Device”. The image was not rebuilt for a label: nothing offline is specific to which issuing CA is signed.

Follow-ups #

• Commit the Tenant Service CA’s request and certificate to the inventory, with site_tenant_service_ca_file, once its vaults are re-encrypted. Done 2026-10-11, about an hour after the change: inventory pull request 83, with the vaults re-encrypted.
• How a tenant service certificate is revoked (ADR-0037 open question 1).
• Nothing tells a tenant a certificate is close to expiring (ADR-0037 open question 2). The first one expires 2027-10-11.
• Where OpenBao’s CA keys are kept now covers two CAs ( CHG-0036).
• The Builder’s /home is 20 GB and holds every container build. Move the container store to /srv, or prune on a schedule. The API’s image build also copies bin/, 369 MB of old tarballs, into its build context.
• The pi-pki image’s menu and banner name two issuing CAs. The collection’s script is corrected; the next image built for another reason carries it.
• The signing role admits any name two or more labels below the site’s zone, so a.b.<tenant>.<site>.deevnet.net would be signed if asked. Only the API asks, and it sends one label. A role per tenant would close it (ADR-0037, alternatives).
• The provider’s TestAccCertificate has not been run. It needs a throwaway tenant.
• A check that the offline machine’s clock is within reason of the request’s own date.