C1 Design: The Broker Account Writer #

The design C1 was accepted on at review, 2026-09-20, and built the same day. It is the mechanism by which the Deevnet API gets an MQTT account onto the messaging VM without making the broker’s auth database reachable from the network.

This page is the design as reviewed, kept because the constraints it fixes are binding on anything that touches the writer later. What was actually built, the two defects found deploying it, and the tests that prove it are in CHG-0016 itself. Where the two disagree, the change record wins.

What it is #

A single program on dv02msg001v01, invoked by sshd as a forced command. The API opens an SSH connection with a key pinned to that program, writes one request as JSON on stdin, and reads one response as JSON on stdout plus an exit status.

No new daemon and no new listening port. sshd is already running, already hardened, and already patched by the normal update path.

It does add a capability, and that should be said plainly. A narrowly constrained one, reached through an existing surface: anyone holding the key can cause this program to run. The claim is not that C1 is free, but that it adds no second listener to configure, expose, patch and remember. What it does add is the program, and the rest of this document exists to bound it.

Why not a direct database connection #

CHG-0016 has the full account. In short: publishing PostgreSQL on the segment cannot be constrained to the API, because a firewalld rich rule does not filter a published podman port — publishing is DNAT plus forward and never reaches the INPUT chain the rule sits on. A host listener is filterable, which is the whole reason this mechanism exists.

The wire contract #

One request, one response, one connection. No streaming, no session, no second round trip.

Request — JSON on stdin #

{
  "version": 1,
  "op": "put",
  "tenant": "eds",
  "account": "lightd",
  "password_hash": "$2a$12$…",
  "publish": ["eds/lightstand/+/scene"],
  "subscribe": ["eds/lightstand/+/status", "eds/lightstand/+/state"]
}

{ "version": 1, "op": "delete", "tenant": "eds", "account": "lightd" }

Response — JSON on stdout, plus exit status #

{ "version": 1, "ok": true, "username": "eds-lightd", "existed": false }

{ "version": 1, "ok": false, "error": "publish pattern 2 is not under the tenant prefix" }

Exit 0 only when ok is true. Any non-zero exit, and any unparseable response, is a failure — see Failure and retry.

What the program derives rather than accepts #

This is the part that matters most, and it is a deliberate inversion: the caller supplies as little as possible.

The tenant trust boundary, precisely #

The writer trusts the authenticated API to name the correct tenant. It has no independent way to know which tenant a request is really for, and it is deliberately not given one.

What it enforces is internal consistency with the claimed tenant: the username it derives, and every pattern it accepts, must fall within <tenant>/. A request claiming eds cannot smuggle a tdemo/ pattern past it.

What it cannot do — and this is the honest limit — is stop a compromised API from claiming tdemo outright and writing entirely self-consistent tdemo ACLs. The API is the component that knows which tenant authenticated; if it is compromised, tenant scoping is already lost upstream and no check in this program recovers it.

That is accepted, not solved. A second tenant-identity mechanism here — per-tenant keys, signed requests — would duplicate authorization the API already performs, and site the duplicate where there is less context to perform it well. The boundary belongs at the API, and this program is not the place to rebuild it.

So the check’s value is narrower than “defends against a tampered API”, and still real: it catches API bugs that produce inconsistent requests, and it makes a malformed or half-constructed request fail closed rather than write a cross-prefix grant.

The program also refuses any pattern that uses # other than as the whole final level, uses + other than as a whole level, or contains % — the same rules as §10, re-applied rather than trusted. Same reasoning, same limit: this catches malformed requests, not a lying caller.

The password never leaves the API #

The API computes the bcrypt hash and sends the hash. The plaintext is generated by the API, returned once to the tenant, and never crosses this boundary. The program cannot learn a device’s password because it is never sent one.

The hash format is $2a$, which pgcrypto’s crypt() verifies — established against the live database, not assumed (CHG-0016).

Failure and retry #

Every operation is an upsert. put writes ON CONFLICT (mountpoint, client_id, username) DO UPDATE; delete is a DELETE that treats a missing row as success. Calling either twice is the same as calling it once.

The API must persist the hash before it calls. A retry that regenerates the password would rotate a credential that devices are already flashed with — the failure CHG-0013 records for Wi-Fi keys, in a new place. The API stores the hash, then calls; a retry sends the same hash and converges.

Every wait is bounded. Nothing here may hang, because a tenant’s terraform apply is holding the other end.

Each expiry is an ambiguous failure, not a distinct outcome: the API does not know whether the row was written, and takes the retry path below with the hash it already persisted. A timeout is therefore indistinguishable from a dropped connection, deliberately — one path, already proven safe, rather than a second one that gets less testing.

An ambiguous failure is safe. If the connection drops after the request is written and before the response is read, the API does not know whether the row was written. Because the operation is idempotent and the hash is stable, retrying is correct either way. This is the property that lets the API answer a tenant honestly instead of guessing.

What the API does with each outcome:

The last row matters for a tenant’s terraform apply: it fails with a named step rather than hanging, which is what the rest of the API already does.

Privilege #

Each layer holds the least it can.

The authorized_keys entry #

Each of these is a requirement. A key installed without them is a general login to the messaging VM.

command="/usr/local/bin/deevnet-broker-account",restrict,from="10.20.25.20" ssh-ed25519 AAAA…

• command= pins the key. The program must not execute or interpret SSH_ORIGINAL_COMMAND.
• restrict is default-deny: it denies pty, agent, port, X11 and user-rc, and keeps denying options OpenSSH has not added yet. Preferred to listing no-* options, which only enumerate what was known when they were written.
• from= binds the key to the API’s address, so a stolen key is not usable from elsewhere. It is belt to the firewalld rule’s braces and survives a firewall mistake.

The program itself #

A real program with a strict schema. Not a shell script. A shell script reading JSON from a network peer is where the injection bug will be. It parses into typed fields, rejects unknown keys, bounds every length, and builds SQL only through parameterized queries — it never interpolates a caller’s string into a statement.

How it reaches the database #

The database container has no port on the segment and must not gain one. The program runs on the host, so it needs a host-local path to a container that deliberately has none.

Selected: the published port is bound to loopback — 127.0.0.1:5432:5432. Decided at review, not left open.

PostgreSQL remains network-unpublished. The host-side listener exists only on 127.0.0.1, so nothing off this host has a route to it: there is no segment exposure to filter, and podman’s DNAT behavior never enters the picture. The security property CHG-0016 exists to protect — not reachable from the network — holds exactly.

The wording matters only because podman would describe this as a published port. It is, in podman’s sense, and it is not network-published in any sense that affects the threat this change addresses. Both things are true and the record says both, so nobody later reads -p in the role and concludes the database was exposed after all.

podman exec was considered and rejected. It needs no port at all, and needs the program’s account to have podman access — which on this host is root or close to it. That trades a loopback listener for a large privilege increase in precisely the component this design exists to keep small.

Deployment #

Ansible deploys the program, the authorized_keys entry, the account and the credential file, as part of the vernemq role. Ansible is not in the request path. The API answers a tenant synchronously; Ansible is not a request-response channel and must not become one.

The API’s private key is stored in its OpenBao KV secret, beside the credentials it already reads there (ADR-0016).

Restricting sshd on the messaging VM #

Two rules, and this is an improvement on the current state rather than an exposure C1 creates:

• A narrow platform -> iot_backend rule for port 22, from the API’s address only.
• A firewalld rich rule on the messaging VM restricting port 22 to the API host and the control node, because sshd is host INPUT traffic — the one path where a rich rule is proven to work on this host, as against a published container port where it does nothing (CHG-0016).

sshd there is reachable from VLAN 30 today, because iot -> iot_backend is a zone-level pass. Every device on that segment can already reach port 22 on the messaging VM. So this rule closes an exposure that predates C1.

The control node must be in that rule, and it must go in before the restriction bites. Ansible reaches this host over exactly the port being restricted; omitting it, or ordering it wrong, costs console access to put right. The rule is written with both sources present from the first apply — never API-only first and control node second.

Permanent reachability tests #

The negative tests that assert the property this whole mechanism turns on — PostgreSQL is not reachable from the network, from anywhere, including the host that uses it — are recorded with their results in CHG-0016, not duplicated here. The row most worth keeping is the least obvious one: the API host is the legitimate consumer and is still supposed to fail, because its access is a forced command over SSH and never a database connection.

What it must never do #

• Execute or interpret anything the caller supplies, including SSH_ORIGINAL_COMMAND
• Accept a username, mountpoint or client_id from the caller
• Write an ACL pattern that does not begin with the tenant’s own prefix
• Return a password, a hash, or any database error text to the caller — errors name the problem, not the internals
• Hold database privileges beyond vmq_auth_acl
• Become a general-purpose administrative interface. One request type, two operations

Review disposition, 2026-09-20 #

All three open questions were closed at review. C1 was accepted, then built, deployed and verified the same day as API v0.5.1.

Non-negotiable #

Carried from review, and binding on any later change to the writer. None of these is a preference:

• A dedicated key pair, used for nothing else
• command=, restrict, and from= on the key entry
• No interpretation of SSH_ORIGINAL_COMMAND — not executed, not parsed, not logged as a command
• No PTY, no forwarding of any kind, no general shell
• A dedicated unprivileged account, not root and not a_autoprov
• A strict typed input schema, unknown keys rejected, every field length-bounded
• Parameterized SQL only — no caller string is ever interpolated into a statement
• Only the existing least-privilege deevnet_api database role, which holds SELECT, INSERT, UPDATE, DELETE on vmq_auth_acl and nothing else

Left to implementation, and where it landed #

These were deliberately decided in code rather than here. The build settled them as cmd/deevnet-broker-account and internal/backend/brokerwriter in deevnet-provisioning-api, installed by the deevnet.mgmt vernemq role — see CHG-0016 for the shipped components and the two defects found deploying them.

Still genuinely open: whether the writer logs a request digest for audit without recording any hash or pattern that would make the log a secondary copy of the ACL table.