<?xml version="1.0" encoding="utf-8" standalone="yes"?><rss version="2.0" xmlns:atom="http://www.w3.org/2005/Atom"><channel><title>Recovery on Deevnet Infrastructure Platform</title><link>https://deevnet.github.io/deevnet-docs/docs/runbook/recovery/</link><description>Recent content in Recovery on Deevnet Infrastructure Platform</description><generator>Hugo</generator><language>en-us</language><atom:link href="https://deevnet.github.io/deevnet-docs/docs/runbook/recovery/index.xml" rel="self" type="application/rss+xml"/><item><title>Omada Controller Recovery</title><link>https://deevnet.github.io/deevnet-docs/docs/runbook/recovery/omada-controller-recovery/</link><pubDate>Mon, 01 Jan 0001 00:00:00 +0000</pubDate><guid>https://deevnet.github.io/deevnet-docs/docs/runbook/recovery/omada-controller-recovery/</guid><description>&lt;h1 id="omada-controller-recovery">
 Omada Controller Recovery
 &lt;a class="anchor" href="#omada-controller-recovery">#&lt;/a>
&lt;/h1>
&lt;p>Getting the Omada controller on the builder back when an upgrade goes wrong or its data is
damaged. Everything here restores from a data snapshot. How snapshots are taken, and how
upgrades are done, is in 
 &lt;a href="https://deevnet.github.io/deevnet-docs/deevnet-docs/docs/runbook/lifecycle/omada-controller-upgrade/">Omada Controller Upgrade&lt;/a>.&lt;/p>
&lt;table>
 &lt;thead>
 &lt;tr>
 &lt;th>&lt;/th>
 &lt;th>&lt;/th>
 &lt;/tr>
 &lt;/thead>
 &lt;tbody>
 &lt;tr>
 &lt;td>Runs as&lt;/td>
 &lt;td>&lt;code>mbentley/omada-controller&lt;/code> under podman, &lt;code>omada-controller.service&lt;/code>, on &lt;code>dv00bld001p01&lt;/code>&lt;/td>
 &lt;/tr>
 &lt;tr>
 &lt;td>Data&lt;/td>
 &lt;td>&lt;code>/opt/omada-controller&lt;/code> (&lt;code>data&lt;/code>, &lt;code>work&lt;/code>, &lt;code>logs&lt;/code>)&lt;/td>
 &lt;/tr>
 &lt;tr>
 &lt;td>Current&lt;/td>
 &lt;td>&lt;code>6.3.0.45&lt;/code> since 2026-09-10. TP-Link released it on 2026-09-04.&lt;/td>
 &lt;/tr>
 &lt;tr>
 &lt;td>Fallback&lt;/td>
 &lt;td>&lt;code>6.2.14.11&lt;/code>, staged and rehearsed&lt;/td>
 &lt;/tr>
 &lt;tr>
 &lt;td>Last resort&lt;/td>
 &lt;td>&lt;code>6.1.0.19&lt;/code>, what ran before&lt;/td>
 &lt;/tr>
 &lt;tr>
 &lt;td>Snapshot&lt;/td>
 &lt;td>&lt;code>/opt/omada-controller-backup/omada-6.1.0.19-data-2026-09-10.tar.gz&lt;/code>&lt;/td>
 &lt;/tr>
 &lt;tr>
 &lt;td>Web UI&lt;/td>
 &lt;td>&lt;code>https://10.20.99.95:8043/independent/index.html#login&lt;/code> (6.3 moved it from &lt;code>/login&lt;/code>)&lt;/td>
 &lt;/tr>
 &lt;/tbody>
&lt;/table>
&lt;blockquote class="book-hint danger">
 
**A controller cannot open a database that a newer one has upgraded.** Starting an older image
on the current data is therefore not a way back. Every path here starts from a snapshot, and
**anything configured since that snapshot was taken is lost**: adoptions, networks, SSIDs,
accounts.

Today's only snapshot predates the 6.3 upgrade. Falling back is cheap while 6.3 is new and has
had little configured on it, and it gets more expensive with every change. If 6.3 is going to be
abandoned, decide early — ideally during the first adoptions, not months later.

&lt;/blockquote>

&lt;p>Do &lt;strong>not&lt;/strong> use &lt;code>playbooks/upgrade-omada.yml&lt;/code> in the builder collection for any of this. It
hard-codes a fresh install and deletes the data directories.&lt;/p></description></item><item><title>OpenBao Drills</title><link>https://deevnet.github.io/deevnet-docs/docs/runbook/recovery/substrate-secrets-drills/</link><pubDate>Mon, 01 Jan 0001 00:00:00 +0000</pubDate><guid>https://deevnet.github.io/deevnet-docs/docs/runbook/recovery/substrate-secrets-drills/</guid><description>&lt;h1 id="openbao-drills">
 OpenBao Drills
 &lt;a class="anchor" href="#openbao-drills">#&lt;/a>
&lt;/h1>
&lt;p>Two exercises, and they prove different things. Both were written from a run against the live mobile
site on 2026-09-17; every command below is one that was actually issued.&lt;/p>
&lt;table>
 &lt;thead>
 &lt;tr>
 &lt;th>Drill&lt;/th>
 &lt;th>What changes&lt;/th>
 &lt;th>What it proves&lt;/th>
 &lt;th>Destructive?&lt;/th>
 &lt;/tr>
 &lt;/thead>
 &lt;tbody>
 &lt;tr>
 &lt;td>&lt;strong>Key change&lt;/strong>&lt;/td>
 &lt;td>a new PKI issuer and a new Transit key version&lt;/td>
 &lt;td>the site survives keys it has never seen before&lt;/td>
 &lt;td>No — every step reverses&lt;/td>
 &lt;/tr>
 &lt;tr>
 &lt;td>&lt;strong>Snapshot restore&lt;/strong>&lt;/td>
 &lt;td>the instance is replaced and its data restored&lt;/td>
 &lt;td>the data survives at all (
 &lt;a href="https://deevnet.github.io/deevnet-docs/deevnet-docs/docs/architecture/decisions/0014-tenant-state-durability/">ADR-0014&lt;/a>)&lt;/td>
 &lt;td>Yes, on a scratch instance&lt;/td>
 &lt;/tr>
 &lt;/tbody>
&lt;/table>
&lt;p>&lt;strong>Why both.&lt;/strong> A snapshot restore brings back the &lt;em>same&lt;/em> issuer and the &lt;em>same&lt;/em> Transit key, so nothing
about a changed key is exercised — a certificate still verifies and a stored secret still decrypts.
The three defects 
 &lt;a href="https://deevnet.github.io/deevnet-docs/deevnet-docs/docs/incidents/2026/0003-openbao-credential-loss/">INC-0003&lt;/a> found were all in the
&lt;em>key change&lt;/em> path, and none of them would have appeared in a restore.&lt;/p></description></item></channel></rss>