https://deevnet.github.io/deevnet-docs/docs/runbook/substrate/recovery/Recent content in Recovery on Deevnet IoTaaSHugoen-ushttps://deevnet.github.io/deevnet-docs/docs/runbook/substrate/recovery/omada-controller-recovery/Mon, 01 Jan 0001 00:00:00 +0000https://deevnet.github.io/deevnet-docs/docs/runbook/substrate/recovery/omada-controller-recovery/<h1 id="omada-controller-recovery"> Omada Controller Recovery <a class="anchor" href="#omada-controller-recovery">#</a> </h1> <p>Getting the Omada controller on the builder back when an upgrade goes wrong or its data is damaged. Everything here restores from a data snapshot. How snapshots are taken, and how upgrades are done, is in <a href="https://deevnet.github.io/deevnet-docs/deevnet-docs/docs/runbook/substrate/lifecycle/omada-controller-upgrade/">Omada Controller Upgrade</a>.</p> <table> <thead> <tr> <th></th> <th></th> </tr> </thead> <tbody> <tr> <td>Runs as</td> <td><code>mbentley/omada-controller</code> under podman, <code>omada-controller.service</code>, on <code>dv00bld001p01</code></td> </tr> <tr> <td>Data</td> <td><code>/opt/omada-controller</code> (<code>data</code>, <code>work</code>, <code>logs</code>)</td> </tr> <tr> <td>Current</td> <td><code>6.3.0.45</code> since 2026-09-10. TP-Link released it on 2026-09-04.</td> </tr> <tr> <td>Fallback</td> <td><code>6.2.14.11</code>, staged and rehearsed</td> </tr> <tr> <td>Last resort</td> <td><code>6.1.0.19</code>, what ran before</td> </tr> <tr> <td>Snapshot</td> <td><code>/opt/omada-controller-backup/omada-6.1.0.19-data-2026-09-10.tar.gz</code></td> </tr> <tr> <td>Web UI</td> <td><code>https://10.20.99.95:8043/independent/index.html#login</code> (6.3 moved it from <code>/login</code>)</td> </tr> </tbody> </table> <blockquote class="book-hint danger"> <p><strong>A controller cannot open a database that a newer one has upgraded.</strong> Starting an older image on the current data is therefore not a way back. Every path here starts from a snapshot, and <strong>anything configured since that snapshot was taken is lost</strong>: adoptions, networks, SSIDs, accounts.</p>https://deevnet.github.io/deevnet-docs/docs/runbook/substrate/recovery/substrate-secrets-drills/Mon, 01 Jan 0001 00:00:00 +0000https://deevnet.github.io/deevnet-docs/docs/runbook/substrate/recovery/substrate-secrets-drills/<h1 id="openbao-drills"> OpenBao Drills <a class="anchor" href="#openbao-drills">#</a> </h1> <p>Two exercises, and they prove different things. Both were written from a run against the live mobile site on 2026-09-17; every command below is one that was actually issued.</p> <table> <thead> <tr> <th>Drill</th> <th>What changes</th> <th>What it proves</th> <th>Destructive?</th> </tr> </thead> <tbody> <tr> <td><strong>Key change</strong></td> <td>a new PKI issuer and a new Transit key version</td> <td>the site survives keys it has never seen before</td> <td>No — every step reverses</td> </tr> <tr> <td><strong>Snapshot restore</strong></td> <td>the instance is replaced and its data restored</td> <td>the data survives at all ( <a href="https://deevnet.github.io/deevnet-docs/deevnet-docs/docs/architecture/decisions/0014-tenant-state-durability/">ADR-0014</a>)</td> <td>Yes, on a scratch instance</td> </tr> </tbody> </table> <p><strong>Why both.</strong> A snapshot restore brings back the <em>same</em> issuer and the <em>same</em> Transit key, so nothing about a changed key is exercised — a certificate still verifies and a stored secret still decrypts. The three defects <a href="https://deevnet.github.io/deevnet-docs/deevnet-docs/docs/incidents/2026/0003-openbao-credential-loss/">INC-0003</a> found were all in the <em>key change</em> path, and none of them would have appeared in a restore.</p>