CHG-0024: Tenant Dashboards #

Summary #

Tenants can read their logs only by query. ADR-0024 decided on Grafana OSS with one organization per tenant, created by the API, and nothing was built.

The take-home Pi image needs dashboards before its first meetup. The Pi copies what Deevnet offers rather than inventing its own, so Deevnet gets them first. That makes this change the contract the Pi then reproduces: the organization, the login, and three log data sources with fixed UIDs, so a tenant’s dashboard code applies unchanged to both.

There are no metrics yet. The dashboards read logs, and a numeric field in a device’s JSON log line can already be graphed.

Goal #

• Grafana 13.2.2 serves HTTPS on dv02obs001v01:3000 with a site-CA certificate. The VictoriaLogs plugin 0.32.0 is loaded with a valid signature, and an unauthenticated request gets 401.
• The API is v0.8.0. tdemo, eds and mabell each have an organization, an Editor login in it and nowhere else, and the three data sources deevnet-logs-workloads, deevnet-logs-platform and deevnet-logs-devices.
• Logged in as mabell, deevnet-logs-devices shows the lines already in (3, 2). The login cannot see another organization or create a data source.
• From DVNTM-TD: :8427 answers with a read token, and :3000 loads. The router has 67 rules and no drift.

Scope #

In scope:

• the Grafana server
• the API step and the provider’s attributes
• the two tenant_dev rules
• the tenant guide

Out of scope:

• metrics, and the metrics data sources (ADR-0023’s change)
• substrate dashboards in organization 1, since the store is tenants-only (ADR-0027)
• the Pi’s Grafana, which is an image build and touches no substrate
• SSO

Risk and impact #

Prerequisites #

• vault_grafana_admin_password (group_vars/all) and vault_grafana_secret_key (group_vars/observability_store) vaulted, committed and pushed. Ciphertext was checked on origin (inventory chg0024-dashboards).
• grafana/grafana:13.2.2 and the plugin zip mirrored on the Builder: deevnet.builder site.yml --limit dv00bld001p01 --tags container-images,fetched-artifacts. The zip matched upstream’s published sha1 and is pinned by sha256.
• PRs merged: inventory #54, deevnet.mgmt #46 and #47, deevnet.builder #19, API #16, provider #8, image factory #9
• Vault decrypted, collections built

Procedure #

Step 1: Build and stage the API #

Run (in deevnet-provisioning-api, on main after the merge):

git tag v0.8.0 && git push origin v0.8.0
make stage

Verify: /srv/deevnet-http/container-images/deevnet-api/deevnet-api-v0.8.0.tar exists.

Undo: nothing to undo; the running API is untouched.

Step 2: Grafana on the observability store #

Run (in ansible-collection-deevnet.mgmt):

ansible-playbook playbooks/site.yml --tags dashboards --limit dv02obs001v01

Verify:

1. The play’s own checks pass: /api/health, an unauthenticated /api/org gets 401, the admin password works, and the plugin reports version 0.32.0 with signature valid.
2. A second run reports changed=0.
3. podman ps on obs shows grafana, victorialogs and vmauth all up, and the log store is unaffected.

Undo: Undo Step 2

Step 3: The API #

Run:

ansible-playbook playbooks/site.yml --tags deevnet-api --limit dv02prv001v01

Verify: the API reports v0.8.0 and is ready. Its environment has DEEVNET_GRAFANA_URL.

Undo: Undo Step 3

Step 4: Give the existing tenants their organizations #

Reconcile tdemo, eds and mabell with the operator token (POST /v1/tenants/<t>/reconcile). Each response carries dashboard.password. Hand each one to its tenant as you handed over the log tokens.

Verify: each tenant shows a dashboards step, ok, and dashboard.org_id ≥ 2. In Grafana (Administration → Organizations) there are three tenant organizations, each with one Editor.

Step 5: tenant_dev reaches the log store and Grafana #

Run (in ansible-collection-deevnet.net):

make migration-opnsense-firewall                                  # plan: 2 to add, nothing else
make migration-opnsense-firewall EXTRA_ARGS="-e firewall_apply=true"

Verify: the plan shows exactly the two new rules. After the apply there are 67 managed rules and no drift.

Undo: revert the two rules in group_vars/all/firewall.yml and apply again.

Verification #

1. As mabell (username and password from Step 4):
   • Explore on deevnet-logs-devices shows the ma-bell-gw-01 lines already in (3, 2).
   • GET /api/user/orgs lists one organization, its own, as Editor.
   • POST /api/datasources gets 403.
2. Tenant Terraform from a DVNTM-TD laptop: the Dashboards example applies a folder and the starter dashboard, and the next terraform plan is clean. This is also CHG-0022’s untested terraform plan from that segment.
3. From DVNTM-TD: curl to :8427 with a read token works, and :3000 loads. From IoT: neither is reachable.
4. Rebuild drill: stop Grafana, move /srv/grafana/data aside, re-run Step 2, and reconcile. Organizations, logins with the same passwords, and data sources come back, and the tenant’s apply restores its dashboards.
5. Delete: a throwaway tenant’s login is gone, and its organization is renamed deleted-<tenant>-<id> with no data sources (see below).

Undo #

Undo Step 3 #

Pin deevnet_api_version: "v0.7.0", remove deevnet_api_grafana_url’s value (set it to ""), and re-run --tags deevnet-api. Migration 0007 only adds columns, so v0.7.0 runs on the migrated database. The tenants’ organizations stay in Grafana until Step 2 is undone.

Undo Step 2 #

systemctl disable --now grafana on obs, remove its firewalld port, and delete /srv/grafana. The log store is untouched.

Outcome #

Live, 2026-09-24 (EDT):

What was done, and tested off the site:

Departures from the plan, and what was found #

• The role’s plugin marker broke the plugin’s signature. The role recorded the unpacked version in a file inside the plugin’s directory. Grafana checks that directory against the plugin’s signed manifest, so an extra file makes the signature “modified” and the plugin refuses to load. The off-site test had unpacked the plugin without the marker, so it passed. The marker now sits beside the plugins directory (mgmt #47).

• Verification 3 (from DVNTM-TD and IoT) and 5 (deleting a throwaway tenant) were not run live. Verification 3 needs step 5. The delete path is covered by the API’s integration test against Grafana 13.2.2.

• Grafana 13 ignores OrgId on user create when users.auto_assign_org is off. It then makes every new user a personal organization named for its login, which is the tenant’s own name. With it on (the default), the named organization wins and the login lands only there, as Viewer, before the API sets Editor. So ADR-0024 §3’s “new users are not added to the main organization” holds through the API naming the organization, not through that setting.

• Grafana 13 cannot delete an organization. DELETE /api/orgs/<id> answers 500 on every 12.x and 13.x release with unified storage ( grafana/grafana#127386; the fix, #127404, is unmerged). A tenant delete removes the data sources and the login, tries the delete, and on a 500 renames the organization deleted-<tenant>-<id>, which frees the name. When the fix ships, the delete succeeds and nothing needs changing.

• The grafana provider ignores provider-level org_id under basic auth. It sends organization 1, where the tenant is not a member, and gets 403. Every resource must carry org_id, so the contract adds TF_VAR_grafana_org_id.

• Grafana 13 tries to download plugins at start. GF_PLUGINS_PREINSTALL_DISABLED, the update checks and the news feed are off, since obs has no internet.

• Port 3000, not 443. The image runs as an unprivileged uid.

• The tenant guide’s ingest example stored nothing. Reproduced on the site’s VictoriaLogs version: a jsonline POST without Content-Type: application/stream+json answers 200 and keeps nothing. The Logs page is fixed.

• The API’s store tests had not run cleanly since CHG-0016. Their reset never dropped tenant_broker_accounts. That hid a step count left stale by CHG-0020. Both are fixed.

Follow-ups #

• Step 5 (tenant_dev rules), applied by the operator
• Verification 3 from DVNTM-TD: 29/29
• Verification 3 from IoT: 15/15
• Hand tdemo, eds and mabell their dashboard passwords (in the operator’s reconcile output of 2026-09-24)
• Remove /srv/grafana/data.drill-20260924 on obs once no one needs it
• When grafana/grafana#127404 ships: upgrade, and delete the deleted-* organizations
• ADR-0024 → Accepted once this is Complete