Troubleshooting #
Is it the network? #
If Terraform or an MQTT client times out, check the network before anything else. Tenants connect over Wi-Fi only, so from your computer on
DVNTM-TD:
curl -fsSLO https://raw.githubusercontent.com/deevnet/ansible-collection-deevnet.net/main/scripts/segment-check.sh
bash segment-check.sh DVNTM-TD
It checks your address and DNS, the provisioning API, the state store and the broker (each verified against the Deevnet Root CA), the internet, and that the rest of the site is correctly out of reach. Turn off any VPN or iCloud Private Relay first. All passing means the network is fine and the problem is on the Terraform or client side (below). Anything failing: send the whole output to the operator; the fix is on the substrate, not in your repo.
segment-check.sh is bash: on Windows, run it under WSL2. Natively, tenant-check.ps1 without
-Offline checks the same services, but not what should be out of reach.
Can’t join DVNTM-TD at all, or lost its password? The password you were handed at admission is
the only copy: nothing shows it again. Ask the operator to rotate your key. You get a new password, and
the old one stops working on every device that had it. A key bound to one computer’s MAC address
also refuses any other computer, including the same one with a new private Wi-Fi address.
Terraform side #
| Symptom | What it means |
|---|---|
No API token | DEEVNET_API_TOKEN is not exported in this shell |
Failed to query available provider packages for deevnet/deevnet | the provider is not in your filesystem mirror, or not at a version your constraint allows. See Before you start |
Connection timed out to api.mobile.deevnet.net or tfstate.mobile.deevnet.net | you are not on DVNTM-TD (or a trusted seat), or your laptop is also on ethernet and mobile traffic is routing out the wrong interface —
where to sit |
api.mobile.deevnet.net does not resolve on DVNTM-TD | a VPN, Private Relay or hard-coded DNS is bypassing the site’s resolver 10.20.45.1 |
x509: certificate signed by unknown authority | DEEVNET_API_CACERT does not point at deevnet-root-ca.pem |
401 on the first apply | the enrollment token was for a different name, and is now spent. Ask for a new admission |
401 later | you are presenting the enrollment token (spent) instead of terraform output -raw api_token |
503, OpenBao is sealed or unreachable; tenant work waits until it is opened | the factory is closed between work sessions, and nothing was changed. Ask the operator to open one, then apply again |
400 on a broker account | a topic pattern breaks the rules: a leading /, $ or %, a # not at the end, or a device reaching into log/ (
rules) |
400 on a DNS record | the address is outside your own subnet |
Saved plan is stale | the state moved between plan and apply. Re-plan |
Permission denied (publickey) logging in to a workload | the user is tenant, not your own; and one key your computer offers (ssh -v … 2>&1 | grep Offering) must be in the workload’s ssh_keys. Compare fingerprints, not comments (
Deploy Your App) |
deevnet/deevnet not found on Windows, though install-provider.ps1 ran | the provider went into %APPDATA%\terraform.d\plugins for the CPU it saw. An amd64 Terraform on an ARM laptop wants windows_amd64; install Terraform first, then rerun the script, which asks Terraform |
$'\r': command not found, missing separator, or a Makefile that fails on its first line, under WSL2 or Git Bash | the repository was checked out with Windows line endings. Clone inside WSL; or, with a .gitattributes that says eol=lf, git add --renormalize . |
WSL2: api.mobile.deevnet.net does not resolve, though Windows resolves it | WSL is in NAT mode: networkingMode=mirrored and dnsTunneling=true (
WSL2) |
curl: (35) schannel: ... revocation on Windows | add --ssl-revoke-best-effort: Windows’ curl refuses a chain whose revocation it cannot check |
REMOTE HOST IDENTIFICATION HAS CHANGED for a workload | expected after you replaced it: ssh-keygen -R <fqdn>. If you did not replace it, stop and ask the operator |
terraform destroy times out partway through | expected at the end of a destroy: deleting the tenant removes its Wi-Fi key and its state-store login, so Terraform cannot save its state or release its lock. Ask the operator to remove your leftover state. errored.tfstate in your directory should hold no resources; delete it once they have |
Device side #
In rough order of how often each is the cause:
| Symptom | Check |
|---|---|
| Never joins Wi-Fi | the PSK is the one from terraform output, not an older one; the SSID is the output’s (DVNTM-IOT on the mobile kit); the board is 2.4 GHz-capable and in range |
| Joins, but the broker connection fails at TLS | the CA is deevnet-root-ca.pem (DER for MicroPython); you connect by name, mqtt.mobile.deevnet.net, not by IP — the certificate is for the name; the ESP32 has a sane clock |
TLS works, CONNACK refused (not authorized) | username is <tenant>-<name>; the password is from the current state — a -replace issued a new one |
| Connected, but publishes vanish | the topic must be the granted one, <tenant>/…, exactly. An unauthorized publish is dropped or disconnects the client — it is not an error you will see |
| Connected, subscribed, but nothing arrives | a refused subscription is also silent. Compare your topic with granted_subscribe |
| Keeps reconnecting | two devices with the same client id — the broker drops the older one each time |
| Log lines missing | the device publishes to <tenant>/log/<its own device name> and nothing else under log/; read partition <index>-2, not the default |