Platform Services on Deevnet IoTaaS

ADR-0036: A Service Proxy on Each Service VM

Mon, 01 Jan 0001 00:00:00 +0000

ADR-0036: Tenant-Facing Services Answer on 443, Through a Proxy on Each Service VM #


Status	Accepted (2026-10-10): built and deployed by CHG-0046
Date	2026-10-10
Scope	Where a tenant-facing HTTPS service listens, what name and port a tenant dials, and which process holds the certificate. Not who may call a service, which each service still decides ( ADR-0020 §5), and not the broker, which is not HTTP.
Extends	ADR-0013: a service VM gains one more container, and a new service on it no longer opens a port to its segment.
Amends	ADR-0024 As built §1 (“port 3000”), ADR-0022 §3 (“vmauth is the only listener tenants can reach”) and ADR-0026 §5 (“the store serves TLS”): each service is on loopback, and the proxy is the listener. What each decided about authentication stands.
Related	ADR-0031 (the Substrate CA issues the proxy’s certificate), ADR-0008 (service names are aliases of a host), ADR-0023 and ADR-0025 (the services that arrive next)

Context #

Every tenant-facing service has its own port. Grafana is on 3000, the tenant downloads on 8443 and the log store on 8427, all on the observability store VM. The Deevnet API is on 8080 and the state store on 9000, both on the provisioning VM. A tenant has to know five port numbers, and each is written into tenant code, scripts and bookmarks.

ADR-0027: Tenant Log Store

Mon, 01 Jan 0001 00:00:00 +0000

ADR-0027: The Log Store Is for Tenants Only, and Device Logs Arrive Over MQTT #


Status	Proposed
Date	2026-09-22
Scope	Who the central log store is for, what writes into each tenant’s partitions, how edge devices’ logs reach it over MQTT, and what would justify moving it.
Supersedes, in part	ADR-0022: Central Logging. Its scope, which included substrate logs, and its §5 shipping of substrate hosts and network devices, including the syslog listener, are replaced. ADR-0022’s store, proxy, partition scheme, tokens and “publish, don’t filter” rule stand.
Related	ADR-0012: IoT Platform Services Through a Deevnet API and Terraform Provider §8, §10, ADR-0013: Management-Hypervisor Services Run as Containers on Domain VMs, ADR-0018: Operator Access to Tenant Workloads, CHG-0018: The Central Log Store, INC-0004

Context #

What exists #

CHG-0018 built ADR-0022’s store on dv02obs001v01, a Platform VM on the management hypervisor: VictoriaLogs behind vmauth, with tenants partitioned by index. It was verified, and it was completed with nothing shipping to it. ADR-0022 stayed Proposed until a real writer exists.

ADR-0026: Object Storage

Mon, 01 Jan 0001 00:00:00 +0000

ADR-0026: The State Store Becomes Object Storage, Defined by a Contract, With State and Tenant Buckets as Two Classes #


Status	Proposed
Amended by	ADR-0036: A Service Proxy on Each Service VM: the store serves plain HTTP on loopback, and the service proxy holds the certificate (§5)
Date	2026-09-21
Scope	What the substrate’s S3 service must guarantee, which engine provides it, and how it serves both Terraform state and tenants’ own buckets. Covers isolation, quotas, locking, versioning, transport and replication.
Extends	ADR-0007: Terraform State Custody, which offered a state store and chose MinIO as a detail; ADR-0014: Tenant State Durability, whose data disk and replica this store needs before it holds anything more
Amended by	ADR-0033: Code Is the State (Proposed): no replica. State is re-derivable, and tenant buckets are the tenant’s to copy; the `replicated` option is withdrawn (§3, §4)
Related	ADR-0010: Tenants Consume Platform Services §4, ADR-0013: Management-Hypervisor Services Run as Containers on Domain VMs §6, ADR-0015: Tenants Are Built Through the Deevnet API, ADR-0016: Substrate Secrets in OpenBao

Context #

The engine under the state store is abandoned #

The state store is MinIO’s community edition, RELEASE.2025-09-07, on dv02prv001v01.
The API confines each tenant through MinIO’s own admin API. It creates a user per tenant with a canned policy limiting it to that tenant’s prefix of one bucket, tf-state.
Tenants lock state with use_lockfile = true.

MinIO’s community repository was archived on 2026-04-25:

ADR-0025: Identity Directory

Mon, 01 Jan 0001 00:00:00 +0000

ADR-0025: Keycloak Is the Identity Provider, With a Realm per Tenant and the Rebuild Path Kept Clear of It #


Status	Proposed
Date	2026-09-21
Scope	Who can sign in to what, and how. This covers the operator on substrate UIs and hosts, each tenant’s people on the UIs the substrate offers them, and a tenant application’s own end users. It also covers how tenants are separated and what stays reachable when the directory is down. Not machine credentials, which the API and OpenBao issue.
Extends	ADR-0013: Management-Hypervisor Services Run as Containers on Domain VMs, which placed “later an LDAP or Active Directory–compatible directory” in the identity VM without choosing one; ADR-0024: Dashboards, whose local Grafana logins it replaces for people
In conflict with	ADR-0033: Code Is the State (Proposed): §7 makes the directory’s database kept data, which that record’s principle forbids. To be revisited before Keycloak is built
Related	ADR-0010: Tenants Consume Platform Services §4, ADR-0014: Tenant State Durability, ADR-0015: Tenants Are Built Through the Deevnet API, ADR-0016: Substrate Secrets in OpenBao, ADR-0022: Central Logging, ADR-0023: Metrics and Alerting

Context #

Every system has its own accounts #

Today, each system keeps its own accounts:

ADR-0024: Dashboards

Mon, 01 Jan 0001 00:00:00 +0000

ADR-0024: Dashboards Are Grafana, One Organization per Tenant, Declared as Code #


Status	Accepted (2026-10-05): built and deployed by CHG-0024, which amends it (see As built)
Amended by	ADR-0036: A Service Proxy on Each Service VM: Grafana listens on loopback over plain HTTP, and the service proxy serves it at `grafana` on 443 (As built §1)
Date	2026-09-21
Scope	How the operator and each tenant view their metrics and logs as dashboards, how people log in to do it, how dashboards are declared as code, and what a tenant may and may not configure. Not alerting, which ADR-0023 decides.
Extends	ADR-0023: Metrics and Alerting, whose §5 deferred saved dashboards and named Grafana with one organization per tenant as the likely shape
Extended by	ADR-0025: Identity Directory: people sign in to Grafana through the directory, and the per-tenant login becomes machine-only (Proposed)
Related	ADR-0010: Tenants Consume Platform Services §4, ADR-0012: IoT Platform Services Through a Deevnet API and Terraform Provider §4, §7, ADR-0015: Tenants Are Built Through the Deevnet API, ADR-0016: Substrate Secrets in OpenBao, ADR-0022: Central Logging

Context #

What exists to read data #

ADR-0022 and ADR-0023 store logs and metrics, partitioned per tenant behind vmauth. The only reading surfaces they give are the store’s own UIs:

ADR-0023: Metrics and Alerting

Mon, 01 Jan 0001 00:00:00 +0000

ADR-0023: Metrics Are Pulled From Management, Stored Beside the Logs, and Alerted On Per Tenant #


Status	Proposed
Date	2026-09-21
Scope	How substrate and tenant metrics are collected and stored, who may read them, how alert rules are declared and evaluated for the substrate and for each tenant, and where notifications go. Dashboards only as far as needed to read what is stored.
Extends	ADR-0022: Central Logging. Metrics use its store host, its authenticating proxy, its partition scheme and its tenant credentials.
Extended by	ADR-0024: Dashboards: saved dashboards in Grafana, one organization per tenant (Accepted)
Related	ADR-0010: Tenants Consume Platform Services, ADR-0013: Management-Hypervisor Services Run as Containers on Domain VMs §5, ADR-0015: Tenants Are Built Through the Deevnet API, ADR-0016: Substrate Secrets in OpenBao, ADR-0018: Operator Access to Tenant Workloads

Context #

Nothing is measured and nothing alerts #

The substrate has no metrics and no alerts. The operator learns about a fault when a tenant’s apply times out or a page stops loading.
Both observability VMs are still empty. dv02sob001v01 is on management and dv02tob001v01 is on Platform. Both are being replaced: tob by the store dv02obs001v01, and sob by the collector dv02col001v01 ( CHG-0018, naming §3.4). The rest of this record uses the new names.
Tenants were promised more than they have. The tenant pages promise “VM resource utilization”, “Per-tenant thresholds” and “Tenant-specific notification channels”. None of it exists.

What is wanted #

Substrate metrics: every host, hypervisor and service, plus the router, switch and access point.
Tenant metrics: what a tenant’s workloads emit, and the resource use of those workloads as the hypervisor sees it.
Operator alerts on the substrate.
Tenant alerts: a tenant declares its own rules and where they notify, and nobody edits substrate code for it.
The same partitioning as the logs. A tenant reads and alerts on its own data only.

Where ADR-0022 left collection #

ADR-0022 moved log storage to Platform because every zone with something to log can push there. For metrics, pull is the norm: the collector scrapes each target. ADR-0013 §5’s reasoning applies to that directly. Management reaches every zone, so a scraper on management reaches every target. A scraper on Platform would need new rules into management and IoT Backend.

ADR-0022: Central Logging

Mon, 01 Jan 0001 00:00:00 +0000

ADR-0022: One Central Log Store on Platform, Partitioned by Tenant #


Status	Accepted, as narrowed by ADR-0027
Amended by	ADR-0036: A Service Proxy on Each Service VM: vmauth listens on loopback, and the service proxy is the listener tenants reach, at `logs` on 443 (§3)
Accepted	2026-09-26, by the operator. What stands after ADR-0027 is built and in use: the store, vmauth, the partitions and the API-issued tenant tokens ( CHG-0018 to CHG-0021). Tenants read it through Grafana since CHG-0024. The v1 deferrals are on the Extended Management Plane roadmap.
Date	2026-09-21
Scope	Where substrate and tenant logs are sent and kept, how they are partitioned, who may read which partition, and which substrate events a tenant sees. Logs only: metrics and alerting are left for their own records.
Supersedes, in part	ADR-0013: Management-Hypervisor Services Run as Containers on Domain VMs §5, for logs only. That section splits observability into a substrate store on management and a tenant store on Platform. For logs there is now one store, on Platform. Everything else in ADR-0013 stands, including both VMs.
Extended by	ADR-0023: Metrics and Alerting: metrics use the same store host, proxy, partitions and tenant tokens (Proposed). ADR-0024: Dashboards: tenants read their logs in Grafana, one organization per tenant (Accepted)
Superseded in part by	ADR-0027: Tenant Log Store: the scope and §5. The store is for tenants only, the substrate and network devices don’t ship to it, and the syslog listener goes. Device logs arrive over MQTT into a new `(index, 2)`. The store, proxy, partitions, tokens and §4 stand. (Proposed)
Built by	CHG-0018: The Central Log Store: the store, Complete 2026-09-22. CHG-0020: tenant tokens. CHG-0021: device logs over MQTT. Substrate shipping was withdrawn by ADR-0027.
Related	ADR-0002: Tenant Fabric Numbering, ADR-0010: Tenants Consume Platform Services, ADR-0012: IoT Platform Services Through a Deevnet API and Terraform Provider, ADR-0015: Tenants Are Built Through the Deevnet API, ADR-0016: Substrate Secrets in OpenBao, ADR-0021: Tenant Secrets

Context #

Nothing collects logs today #

CHG-0008 built both observability VMs empty. dv02sob001v01, on management, is for the substrate. dv02tob001v01, on Platform, is for tenants. Each runs only sshd, and CHG-0008’s follow-up “Observability tooling for dv02sob001v01 and dv02tob001v01” is still open.
Both are replaced, not reused. This record puts every log in one store, so the audience split their names describe no longer exists. CHG-0018 replaces tob with dv02obs001v01, the observability store, and sob with dv02col001v01, the collector (naming §3.4). Both are still empty, so replacing them costs nothing. The rest of this record uses the new names.
Every host keeps its own journal. Reading across the site means an SSH session per host, and the core router’s log buffer holds about fifty seconds.
Tenants have nothing at all. The tenant pages promise logs “Tagged with tenant identifier”, “Queryable by tenant scope” and “Retained per tenant policy”, and the shared-services table lists tenant observability as “To be defined”.

What is wanted #

One place for all logs. Tenants ship their workloads’ logs there, and so do substrate hosts and services.
Partitioned by tenant. Each tenant reads its own logs and never another tenant’s.
A tenant also sees the substrate events that concern it. Examples: the API creating its workload, a DNS update to its zone, a broker refusing one of its accounts. It should not have to ask the operator.
The operator reads everything.

What the zone policy allows #

ADR-0013 §5 put substrate observability on management. That followed from the zone policy: IoT Backend and Platform can’t reach management, so a store there has to pull. That is still true. What ADR-0013 didn’t weigh is the reverse.