ADR-0021: Tenants Keep Runtime Secrets in Their Own OpenBao Namespace #

Context #

A tenant has secrets of its own, and nowhere to put them #

A tenant’s backend service calls out: a weather API, a notification service, a payment provider. Each call carries a key the tenant got from that provider. The substrate did not issue it and cannot re-derive it. The service needs it at runtime, on a workload the substrate built.

Today there is no path for it:

• A workload receives only what the API sets on it: a user, SSH keys, a resolver and its network interface. There is no user-data and no secret delivery.
• The first real case is already waiting. EdS’s lightd reads its broker credentials from environment variables, and its own code says how they get there is still open. That credential is substrate-issued (ADR-0012 §4), but it has the same problem: it lives in tenant state, and nothing carries it from there to the running service.
• ADR-0017 says how code and configuration will arrive, not secrets. Its leading mechanism is preferred because “secrets never sit on substrate storage”, but it doesn’t say where a secret is kept before it is fetched, or how one is rotated without a new workload.

Two kinds of tenant secret #

The first kind has a custody model and no delivery. The second has neither. Both have to reach a workload at runtime, so one mechanism should serve both.

What is already true #

• OpenBao was chosen partly for this. ADR-0016 rejected Vault Community Edition in part because its namespaces are Enterprise-only, “which closes off per-tenant isolation later”. OpenBao has them free.
• OpenBao is already reachable from tenant transit. ADR-0016 §1: “The control is its authentication and policies, not the network.”
• The substrate already hosts tenant secrets. The offered state store holds every opted-in tenant’s state, which ADR-0014 §4 treats as a secret store. A tenant secrets service would not be the first. What matters is that it is never the only copy (ADR-0010 §4).
• The copy split is what made INC-0003 recoverable. OpenBao was rebuilt with a new Transit key, and the API lost the ability to read its stored copies. Each tenant resupplied from its own state and nothing was lost. Any design here has to keep that property.

Options considered #

A — Nothing from the substrate: the tenant encrypts in its repository and delivers its own #

The tenant keeps secrets age-encrypted in its repository and ships them to the workload with its code.

• For:
  • No substrate involvement.
  • The same tool ADR-0012 §9 already uses.
• Against:
  • The workload needs the decryption key, so that key is the secret to deliver. Nothing is solved.
  • One key opens every secret the tenant has.
  • Rotating a secret means redeploying every workload that holds it.
  • Nothing records which workload read what.
• Verdict: Remains the fallback for a tenant that declines the service (§7). Not the platform answer.

B — Secrets ride in the ADR-0017 workload configuration #

The tenant declares secrets beside its other configuration, and the API stores them with it.

• For: one declaration, one fetch.
• Against:
  • The API’s database becomes the runtime store for tenant secrets.
  • Secrets get configuration’s lifecycle: cached on the workload, fetched once, and rotated only by a replace.
  • It mixes something to inspect with something to protect.
• Verdict: Rejected. The configuration may name a secret, but never carry its value.

C — Each tenant runs its own secrets manager in its fabric #

• For: the substrate holds nothing.
• Against:
  • It moves the problem rather than solving it. The tenant’s manager starts on an empty workload that has to be filled, unsealed and backed up.
  • Every tenant would operate a seal key.
  • ADR-0017 rejected 2d as the base case for the same reason.
• Verdict: Rejected as the base case. A tenant may still do it.

D — A per-tenant namespace in the substrate’s OpenBao, as a runtime copy (chosen) #

• The API creates a namespace for each tenant.

• The tenant’s Terraform writes its secrets there from an authoritative copy the tenant keeps.

• Workloads read them directly, each under its own identity.

• For:

  • One mechanism for both kinds of secret.
  • Isolation enforced by the server.
  • Rotation without a replace.
  • Revocation per workload.
  • An audit trail.
  • The substrate stays re-derivable: losing OpenBao costs every tenant one apply.

• Against:

  • OpenBao becomes something tenant services need in order to start (§6).
  • Its blast radius grows from substrate secrets to every opted-in tenant’s secrets.
  • Workload identity still has to be delivered once (§4).

E — As D, with OpenBao as the only copy #

The tenant writes a secret once, and OpenBao holds it.

• For: the simplest thing for a tenant to use.
• Against:
  • It breaks ADR-0010 §4 outright, because the substrate would hold the only copy of tenant content.
  • A rebuild like INC-0003 would destroy tenant secrets instead of costing an apply.
  • Recovery would rest on a Raft snapshot restore, which ADR-0016 still lists as unconfirmed.
• Verdict: Rejected.

F — As D, on a separate tenant-facing OpenBao instance #

• For:
  • Substrate secrets and tenant secrets never share a seal key or a Raft store.
  • A tenant-facing fault can’t reach the API’s backend credentials.
• Against:
  • One more critical service to run, seal, snapshot and drill.
  • A second seal key held in ansible-vault, so the root of trust is the same.
  • The isolation namespaces already give is enforced by policy, and policy is what protects the substrate’s own paths today.
• Verdict: Not chosen now. See What would reopen this.

Decision #

Option D. Tenants keep runtime secrets in a namespace of the substrate’s OpenBao. The tenant holds the authoritative copy, the API provisions the namespace and scopes every write to one tenant, and workloads read directly under their own identity.

1. One namespace per tenant, made by the API #

• The API creates the namespace when it creates the tenant, as it creates the tenant’s zone, state credential and network (ADR-0015). No substrate commit is involved, as ADR-0010 requires.

• Inside it:

  • one KV v2 mount for the tenant’s secrets
  • one AppRole auth mount for its workloads
  • the policies the API writes for them (§4)

  In v1 the tenant enables nothing else there.

• Destroying the tenant deletes the namespace.

• The API’s own policy grows to include creating and deleting namespaces and administering inside them. That is a real increase in what the API can reach; see Consequences.

2. The tenant writes through the Deevnet API, not OpenBao directly #

• The provider gains a secret resource. The API writes the value into the tenant’s KV mount with its own credential, and every call is scoped to the caller’s own tenant, as for every other resource.
• The tenant holds no OpenBao credential. Its one credential stays its API token (ADR-0015).
• Recovery is the provider’s existing resupply pattern. After an OpenBao rebuild, a read reports the secret absent and the next apply writes it again, as secrets_stored already works for tenants.
• The API is in the write path only. It stores nothing itself and returns no value. A read reports presence and version only.

Writing through OpenBao directly, with a namespace token issued to the tenant, was considered. It would give the tenant a second credential that dies with every OpenBao rebuild and has to be reissued, the problem ADR-0015 closed for the operator token.

3. The tenant’s repository holds the authoritative copy #

• Tenant-authored secrets live age-encrypted in the tenant’s repository, with the recipients ADR-0012 §9 already registers per consumer. The tenant decrypts at plan time and passes each value to the secret resource.
• The value is write-only, not stored in state. ADR-0012 §4 rejected Terraform’s write-only arguments because there, state is the authoritative copy and must be kept. Here the repository is the authoritative copy, so keeping the value in state as well would make one more copy for no benefit. A version attribute, which is not secret, tells the provider when to write again.
• Substrate-issued secrets need no second copy. A tenant that wants its workload to read its broker password passes that attribute from state into a secret resource. Its authoritative copy stays where ADR-0012 §4 put it.
• What the substrate holds is always a copy. Losing all of OpenBao costs each opted-in tenant one apply, and never costs a secret (ADR-0010 §4).

4. Workloads read directly, each under its own identity #

• Each workload gets its own AppRole in the tenant’s namespace. Its policy grants read on the paths the tenant names for that workload, and nothing else. A workload never reads what its tenant did not assign to it.
• The workload’s credential is delivered once, through ADR-0017’s channel.
  • The role ID is not secret.
  • The secret ID arrives response-wrapped: single-use and short-lived, the property ADR-0017’s open question 2 already needs from a bootstrap token.
  • This record names the credential. ADR-0017 decides how it travels.
• The API is never in the read path. A workload talks to OpenBao and nothing else, so ADR-0012 §8’s provisioning-only stance is untouched.
• The ADR-0017 configuration may name secrets, but never contain them. It says which paths the workload’s policy grants and where the workload should put each value.

5. Rotation and revocation #

• Rotating a tenant-authored secret is a new value in the repository and an apply. Workloads pick it up on their next read, with no replace.
• Revoking a workload deletes its AppRole. It loses access at once, whatever it still has cached.
• Destroying a workload revokes its role as part of the destroy.

6. OpenBao is needed to start, not to keep running #

• A workload reads its secrets when its service starts, and holds them in memory only, never on its disk.
• A running service is unaffected by an OpenBao outage. A service that starts during one waits until OpenBao is back.
• This amends a consequence of ADR-0016, which could say “Nothing at runtime depends on it.” For tenants that opt in, that sentence becomes: nothing that is already running depends on it.
• The trade is deliberate. A disk cache would let a workload restart through an outage, but it puts secrets on a disk the substrate hosts, which is exactly what ADR-0017 prefers 2b for avoiding.

7. A tenant may decline #

As ADR-0007 allows for state, a tenant may decline the service and fall back to Option A. The cost of declining is stated plainly: no rotation without a redeploy, no revocation per workload, no audit.

8. Not now #

• Per-tenant PKI. A PKI mount in each namespace could issue certificates to the tenant’s services and devices. It is the obvious answer to ADR-0020 §2, which requires devices to prove a credential but leaves the mechanism open. It needs its own record, because device identity crosses ADR-0011 §4 and ADR-0012 §4.
• Dynamic secrets such as database credentials for a tenant’s own database.
• Tenant-enabled engines. The namespace carries only what §1 lists.

Consequences #

A tenant has one place for every runtime secret, whether it wrote the secret or the substrate issued it, and one pattern to read it.

The substrate stays re-derivable. Every value in a tenant namespace is a copy. A rebuild like INC-0003 costs each tenant an apply, and costs no secret.

OpenBao’s blast radius becomes every opted-in tenant’s secrets. ADR-0016’s warning, “Anyone with the seal key and a copy of the Raft data reads every secret”, now covers them too. So does a compromised API, whose policy can administer every tenant namespace. ADR-0012 already accepts that “a compromised API is a site-wide compromise of those services”. This record extends that to tenant runtime secrets, and says so rather than implying namespaces protect against it. Namespaces isolate tenants from each other, not from the substrate.

Two of ADR-0016’s open items become preconditions for accepting this record:

• An audit device. A tenant-facing secrets service without an audit log gives up half its value.
• The Raft snapshot restore. It is less critical here, because tenants can resupply. Workload identities, however, can’t be resupplied (below).

An OpenBao rebuild costs every opted-in workload its identity. Tenants rewrite their secrets by applying. The AppRole secret IDs delivered to workloads are gone, though, and each workload needs a new one through ADR-0017’s channel. Under ADR-0017 §3 as written, that means a replace. That is acceptable for stateless workloads and costly for others. Open question 2.

This record depends on ADR-0017 being built. Until a workload can receive a single credential, the namespace can be written and never read. The two are the same piece of work from different ends.

The API gains a resource and a wider policy. The provider gains a secret resource. The API’s OpenBao policy gains namespace administration. Its OpenBao client adds the namespace header.

Open questions #

1. Namespace naming and depth. tenants/<name> under a parent namespace, or flat at the root? A parent lets one policy grant the API everything below it and nothing above.
2. Re-delivering workload identity after a rebuild. Should a workload be able to re-bootstrap with no replace, for example through a fresh single-use token, or is a replace the answer?
3. Paths per workload or per tenant? Per-workload policies are the least privilege, but a tenant with many identical workloads may want a shared path.
4. The read client. A small agent in the base image that renders secrets into the service’s environment, OpenBao Agent, or the application’s own client? If the ADR-0017 agent exists, this may be the same agent.
5. Should the tenant’s age recipients (ADR-0012 §9) become an API concern rather than a Makefile target, now that the API issues everything else?

What would reopen this #

• A reason to separate the instances (Option F): a tenant-facing incident on the shared OpenBao, a need to upgrade or restart it on a tenant’s schedule, or a tenant whose secrets need custody the substrate can’t give.
• Per-namespace sealing (OpenBao 2.6.0+) making separation cheap enough to be worth it without running a second instance.

To confirm when building #

• That namespaces in OpenBao 2.6.2 isolate KV, AppRole and policies as described: a token from one namespace is refused in another, and the API’s policy scoped to a parent namespace reaches its children and not the root’s paths.
• That the provider’s write-only argument works as the plan relies on: nothing in plan or state, and a new version triggers a write. The site runs Terraform 1.14.3, and write-only arguments need 1.11 or later.
• That deleting a namespace removes everything under it, including AppRoles and their secret IDs, in one call.

Current state #

• Proposed. Nothing is built.
• OpenBao has no namespaces.
• The API has no secret resource.
• Workloads receive no secrets.
• lightd’s broker credential is still handled by hand.