CHG-0046: Tenant-Facing Services on 443, Through a Service Proxy #

Summary #

Each tenant-facing service answers on its own port: Grafana :3000, the log store :8427 and the tenant downloads :8443 on the observability store VM; the Deevnet API :8080 and the state store :9000 on the provisioning VM. A tenant has to know all five.

This change builds ADR-0036. Each of the two VMs gets a service proxy that listens on 443 and forwards by name to the service, which moves to the host’s loopback address. Tenants dial a name and no port. The old ports keep answering, through the proxy, so nothing a tenant holds stops working.

Goal #

• Each of these returns its service, with a certificate that verifies against the Deevnet Root CA, from an operator network and from DVNTM-TD: https://grafana.mobile.deevnet.net, https://logs.mobile.deevnet.net, https://downloads.mobile.deevnet.net, https://api.mobile.deevnet.net, https://tfstate.mobile.deevnet.net.
• Each old address still returns its service: grafana…:3000, dv02obs001v01…:8427, downloads…:8443, api…:8080, tfstate…:9000.
• On both hosts, each service listens on 127.0.0.1 only, and the proxy on the host’s segment address.
• A name a host does not serve gets no TLS handshake on 443.
• The state store’s console on :9001 is not reachable from any other host.
• The API hands a tenant dashboard_url, log_endpoint and the state endpoint with no port.
• From DVNTM-IOT, both hosts are still blocked on 443.

Scope #

In scope: the service_proxy role and its certificate; Grafana, vmauth, the tenant downloads and MinIO on loopback over plain HTTP; the API on loopback, keeping its certificate; the logs alias; two zone rules for 443; the addresses the API hands out; the log bridge’s store address; the tenant guide and the operator’s check scripts.

Out of scope:

• retiring the old ports and their six zone rules
• the broker, which is not HTTP and is on another segment
• OpenBao, the tenant DNS server’s API, the Omada controller and the artifact server, none of which tenants reach
• the take-home Pi image, which keeps the old port numbers
• rewriting the address held in existing tenants’ Terraform backends

Risk and impact #

Prerequisites #

• Pull requests merged: deevnet.mgmt and the inventory
• Pull request open and not merged: deevnet.net’s segment-check.sh with the 443 checks. Tenants fetch that script from main, where it must keep matching the site until step 5 passes
• Vault decrypted, collections built
• A backup taken ( CHG-0039), before the state store changes how it listens
• Tenants told that the ports are going away and that the old addresses keep working. Not done before the change

Procedure #

Step 1: Read what is there #

Read-only.

Run:

dig @10.20.10.1 logs.mobile.deevnet.net

ssh dv02obs001v01 'sudo ss -tlnp'
ssh dv02prv001v01 'sudo ss -tlnp'

cd ansible-collection-deevnet.net
ansible-playbook playbooks/dns.yml --check
ansible-playbook playbooks/opnsense.yml
bash scripts/segment-check.sh DVNTM-TD      # from a client on DVNTM-TD

Verify:

1. logs does not resolve.
2. Each service listens on every address, on its own port; nothing listens on 443 on either host.
3. The DNS check run reports one alias to add.
4. The zone policy report shows two rules to add and no other drift.
5. The segment check passes as it stands.

Result, 2026-10-10: passed, with item 4 read at step 5 and item 5 not run.

• logs.mobile.deevnet.net answered NXDOMAIN.
• On the observability store, Grafana listened on every address at 3000, vmauth at 8427 and the downloads server at 8443. On the provisioning VM, the API was at 8080 and MinIO at 9000 and 9001. Nothing listened on 443 on either.
• The DNS check run reported one alias to add, with 15 unchanged.
• The zone policy report was not run here. It was read at step 5, before the apply.
• The segment check was not run before the change.

Undo: nothing to undo.

Step 2: The alias #

Not disruptive.

Run:

ansible-playbook playbooks/dns.yml

Verify:

1. dig @10.20.10.1 logs.mobile.deevnet.net answers NOERROR with 10.20.25.22.
2. A second run reports no change.

Result, 2026-10-10: passed.

• logs.mobile.deevnet.net answers NOERROR with 10.20.25.22.
• A second run added nothing and reported 16 aliases unchanged.
• The alias was still published after the router’s reboot later in the change.

Undo: remove the alias from inventory and run the playbook with -e dns_delete_unmanaged=true.

Step 3: The observability store #

Interrupts Grafana, the log store and the downloads for about a minute.

Run:

cd ../ansible-collection-deevnet.mgmt
ansible-playbook playbooks/site.yml --limit dv02obs001v01 \
  --tags log-store,dashboards,tenant-downloads,service-proxy

Verify:

1. ss -tlnp on the host: Grafana, vmauth and the downloads server on 127.0.0.1; the proxy on 10.20.25.22 at 443, 3000, 8427 and 8443.
2. The proxy’s certificate names the host, grafana, logs, downloads and the address.
3. curl --cacert <root> returns each service at its name on 443, and at its old address.
4. A request to https://dv02obs001v01.mobile.deevnet.net/ fails in the handshake.
5. A browser at https://grafana.mobile.deevnet.net logs in as a tenant, opens a dashboard that shows data, and its links carry no port. The same at :3000.
6. With a tenant’s write token, a log line pushed to https://logs.mobile.deevnet.net is read back with its read token. A request with no token is refused with 401.
7. A large download completes with the right checksum.
8. The retired certificate directories are gone from the host.
9. A second run changes nothing and restarts nothing.
10. Stop the proxy: items 3 and 5 fail. Start it: they pass.

Result, 2026-10-10: passed, except item 6, which passed in part.

• Grafana, vmauth and the downloads server listen on 127.0.0.1 only. The proxy listens on 10.20.25.22 at 443, 3000, 8427 and 8443.
• The certificate names the host, downloads, grafana, logs and 10.20.25.22.
• Grafana and the downloads return 200 on 443 and on their old ports. The log store returns 401 with no token at logs on 443 and at the host name on 8427. Each chain verifies.
• A request for the bare host name on 443, and one for the address, failed in the handshake.
• The operator logged in at https://grafana.mobile.deevnet.net in a browser and opened a dashboard.
• Item 6: mabell’s Grafana read its device log lines from the log store at logs on 443, with the tenant’s read token. A push with a tenant’s write token was not run.
• The first megabyte of the largest download, and the whole of a smaller one, matched the files on the host. A full download of the largest file was not completed: see Departures from the plan.
• The three retired certificate directories are gone.
• A second run changed nothing. No image was pushed in either run: every image was already on the host.
• With the proxy stopped, Grafana and the downloads failed at both addresses. Started, they passed.

Undo: revert the deevnet.mgmt change and run the same tags without service-proxy; each role reissues its certificate and listens on its port again. Then stop and disable service-proxy.

Step 4: The provisioning VM #

Interrupts the API and the state store for about a minute.

Run:

ansible-playbook playbooks/site.yml --limit dv02prv001v01 \
  --tags tenant-state,deevnet-api,backup,service-proxy

Verify:

1. ss -tlnp on the host: MinIO on 127.0.0.1:9000 and :9001, the API on 127.0.0.1:8080; the proxy on 10.20.25.20 at 443, 8080 and 9000.
2. The proxy’s certificate names the host, api, tfstate and the address.
3. curl --cacert <root> https://api.mobile.deevnet.net/readyz returns 200, and so does the same at :8080.
4. As a tenant, GET of its own record shows dashboard_url, log_endpoint and the state endpoint with no port.
5. tdemo’s terraform init and terraform plan succeed against its unchanged :9000 backend and plan nothing but the changed addresses. With the port removed from the backend, terraform init -reconfigure and a plan succeed again.
6. A reconcile of tdemo through the API reports every step done, the state store’s and the dashboards’ included.
7. :9001 is refused from the Builder.
8. sudo deevnet-backup on the host completes, and the archive verifies.
9. A second run changes nothing and restarts nothing.

Result, 2026-10-10: passed, except item 5, which passed in part.

• MinIO listens on 127.0.0.1 at 9000 and 9001, and the API on 127.0.0.1:8080. The proxy listens on 10.20.25.20 at 443, 8080 and 9000.
• The certificate names the host, tfstate, api and 10.20.25.20.
• readyz returns 200 at api on 443, at api on 8080 and at the host name on 8080.
• tdemo’s record shows its state endpoint, log_endpoint and dashboard_url with no port.
• tdemo’s terraform init succeeded against its unchanged :9000 backend, and again with the port removed. Its plan read the state both ways and showed state_endpoint losing its port. The plan then failed in the tenant’s own outputs, the same way at both addresses, in a checkout two commits behind its main. The cause was not looked into.
• A reconcile of tdemo returned reconciled, with the state store’s and the dashboards’ steps.
• :9001 is refused from the Builder.
• A backup taken with make backup-now completed, and the archive verified against its manifest.
• A second run restarted nothing. Two of MinIO’s tasks report a change on every run, as they did before this change.

Undo: revert the inventory’s minio_* and deevnet_api_* values and run the same tags without service-proxy; MinIO reissues its certificate and both publish on every address again. Then stop and disable service-proxy.

Step 5: The zone policy #

Not disruptive: two rules are added, none changed.

Run:

cd ../ansible-collection-deevnet.net
ansible-playbook playbooks/opnsense.yml                         # the report
ansible-playbook playbooks/opnsense.yml -e firewall_apply=true

Verify:

1. The report shows two rules to add, tenant dev -> provisioning services (https) and tenant dev -> observability services (https), and nothing else.
2. After the apply, a second report shows no drift.
3. From DVNTM-TD: segment-check.sh DVNTM-TD, taken from its branch, passes with the five names on 443.
4. From DVNTM-TD: a name neither host serves is refused on 443, and :9001 times out.
5. From DVNTM-IOT: the branch’s segment-check.sh DVNTM-IOT passes, with both hosts blocked on 443.
6. Merge the segment-check.sh pull request.

Result, 2026-10-10: passed.

• The report showed the two rules to add, none to update of 69 present, and none to delete.
• The apply reported Added 2, updated 0, deleted 0, and all six of its required paths still answered. The router holds 71 managed rules, the number the inventory declares. A second report was not run.
• From DVNTM-TD and from DVNTM-IOT, the branch’s segment-check.sh passed.
• The first run from DVNTM-TD, of the script on main, failed two checks: the broker and the log store’s old port. The script carried the root that CHG-0033 retired. The branch was corrected to carry the Deevnet Root CA before it merged.
• The pull request is merged.

Undo: remove the two rules from inventory and apply.

Step 6: The log bridge #

Restarts the bridge. Device log lines published in that moment are lost, as on any restart.

Run:

cd ../ansible-collection-deevnet.mgmt
ansible-playbook playbooks/site.yml --tags log-bridge

Verify:

1. The bridge’s store address is https://logs.mobile.deevnet.net.
2. A device log line published to the broker appears in its tenant’s device partition.

Result, 2026-10-10: passed.

• The bridge restarted with https://logs.mobile.deevnet.net as its store, and subscribed.
• From the bridge’s host, the log store answers at logs on 443.
• With a mabell device connected afterwards, its log lines appeared in the tenant’s Grafana, read from its device partition. The bridge logged no error.

Undo: revert log_bridge_store_url and run the tag again.

Step 7: Tenants are told #

Not disruptive.

Run: merge the tenant guide’s port-free addresses and the provider’s scripts, stage the scripts to the downloads, and tell each tenant. None of these merges before step 5 passes: until then the published guide and the scripts tenants fetch describe the ports that are deployed. A tenant does nothing unless it wants the shorter addresses in its own code: its next plan reads the new dashboard_url and log_endpoint, and its backend keeps working as written.

Verify:

1. The tenant guide shows no port for any of the five services.
2. tenant-check.sh passes from DVNTM-TD.
3. The documentation site builds without warnings.

Result, 2026-10-10: the guide and the scripts are changed; the tenants are not told.

• The tenant guide, the provider’s scripts and guide, and the reference tenant are merged.
• The downloads serve the new tenant-check.sh and install-provider.sh. They were rendered from main at the provider version already staged, 0.6.0, without a release.
• tenant-check.sh, fetched from the downloads, passed from the Builder: 14 ok, with every site service verified at its name and no port. It was not run from DVNTM-TD.

Undo: revert the guide and the scripts.

Verification #

Steps 2 to 6 pass, a zone policy report shows no drift, and Build Verification’s service checks pass at the new addresses.

Undo #

Each step’s own undo, in reverse. Steps 3 and 4 are independent: one host can be put back while the other stays. Nothing a tenant holds is removed at any point, because every old address answers before, during a host’s minute, and after.

To discover #

• Whether a container on the provisioning VM’s podman network reaches the proxy at the host’s own address on 443. Found 2026-10-10: it does. From the API container’s network, the state store, Grafana and the log store each answered at its name on 443 with a chain that verified.
• Whether Grafana’s data sources, which hold the log store’s host name and :8427, are rewritten to the service name by a reconcile or only on create. Found 2026-10-10: a reconcile rewrites them. tdemo’s three held https://logs.mobile.deevnet.net after its reconcile, and the other three tenants’ did after a reconcile of all tenants the same day.
• Whether anything else dials these services by an address this record does not list. The exit node’s egress agent dials the API at :8080.

Outcome #

All five services answer on 443 at their names, and every old address answers through the proxy.

Departures from the plan #

• A check took the site’s routing down for about 25 minutes. Step 3’s seventh item was run as a full download of the largest file, 946 MB, from Platform to the Builder across the core router. The router’s LAN interface stopped receiving part-way. Its console was alive and showed no watchdog message; the operator rebooted it. The services on both hosts were unaffected, but nothing routed between segments and the site resolver was unreachable. The check was repeated with a ranged request and a small file. INC-0004 records the event.
• The API keeps its certificate. The proxy re-encrypts to it and verifies it under the API’s name. The other four services serve plain HTTP on loopback.
• Step 1’s zone policy report and segment check were not run before the change. The report was read at step 5.
• opnsense.yml is not write-free as a report. Its DHCP role rewrote the 14 existing reservations on each run, adding and removing none.
• Step 4’s Terraform check did not end in a clean plan. The state store accepted signed requests at both addresses; the plan failed later in the tenant’s own outputs.
• The scripts were staged without make stage, which builds a release. Only the two scripts changed.
• The exit node’s egress agent was not changed. It dials the API at :8080, through the proxy’s legacy listener.
• The playbooks ran from the Builder’s inventory checkout on an unmerged branch, with main merged into it.
• Tenants were not told, before or after.

Follow-ups #

• Retire the old ports: the proxy’s legacy listeners, the six per-port tenant dev rules, and the egress agent’s API address
• Decide whether the take-home Pi image moves to 443 ( ADR-0036 open question 2)
• Decide whether the identity VM’s services get the same treatment (open question 3)
• Accept ADR-0036 when this change completes
• Publish a device log line and find it in its tenant’s partition (step 6, item 2). Done 2026-10-10 with a mabell device
• Push a log line with a tenant’s write token to https://logs.mobile.deevnet.net (step 3, item 6)
• Tell each tenant that the ports are gone from the addresses and that the old ones keep working
• Reconcile cdeever, eds and mabell, so their data sources hold the log store’s service name. Done 2026-10-10: every tenant’s three data sources hold https://logs.mobile.deevnet.net
• Find why tdemo’s plan fails in its outputs
• Make the DHCP role report no change when the router’s reservations match
• Make MinIO’s bucket and policy tasks report no change when nothing changed