<?xml version="1.0" encoding="utf-8" standalone="yes"?><rss version="2.0" xmlns:atom="http://www.w3.org/2005/Atom"><channel><title>Incident Management on Deevnet Infrastructure Platform</title><link>https://deevnet.github.io/deevnet-docs/docs/runbook/incident-management/</link><description>Recent content in Incident Management on Deevnet Infrastructure Platform</description><generator>Hugo</generator><language>en-us</language><atom:link href="https://deevnet.github.io/deevnet-docs/docs/runbook/incident-management/index.xml" rel="self" type="application/rss+xml"/><item><title>Incident Record Template (INC)</title><link>https://deevnet.github.io/deevnet-docs/docs/runbook/incident-management/incident-record-template/</link><pubDate>Mon, 01 Jan 0001 00:00:00 +0000</pubDate><guid>https://deevnet.github.io/deevnet-docs/docs/runbook/incident-management/incident-record-template/</guid><description>&lt;h1 id="incident-record-template-inc">
 Incident Record Template (INC)
 &lt;a class="anchor" href="#incident-record-template-inc">#&lt;/a>
&lt;/h1>
&lt;p>The shape every 
 &lt;a href="https://deevnet.github.io/deevnet-docs/deevnet-docs/docs/incidents/">incident record&lt;/a> takes. Copy the skeleton below into
&lt;code>content/docs/incidents/&amp;lt;YYYY&amp;gt;/&amp;lt;NNNN&amp;gt;-&amp;lt;slug&amp;gt;.md&lt;/code>, numbered with the next unused &lt;code>INC-NNNN&lt;/code> —
global across years and never reused, like ADRs.

 &lt;a href="https://deevnet.github.io/deevnet-docs/deevnet-docs/docs/incidents/2026/0001-firewall-policy-deletion/">INC-0001: Firewall Policy Deleted, Total Connectivity Loss&lt;/a>
is a worked example.&lt;/p>
&lt;p>Start the record while the incident is fresh, even with most sections empty. The timeline
and the wrong turns are the parts memory loses first.&lt;/p>
&lt;hr>
&lt;h2 id="skeleton">
 Skeleton
 &lt;a class="anchor" href="#skeleton">#&lt;/a>
&lt;/h2>
&lt;div class="highlight">&lt;pre tabindex="0" style="color:#f8f8f2;background-color:#272822;-moz-tab-size:4;-o-tab-size:4;tab-size:4;">&lt;code class="language-markdown" data-lang="markdown">&lt;span style="display:flex;">&lt;span>---
&lt;/span>&lt;/span>&lt;span style="display:flex;">&lt;span>title: &amp;#34;INC-NNNN: &amp;lt;&lt;span style="color:#f92672">What&lt;/span> &lt;span style="color:#a6e22e">broke&lt;/span>&amp;gt;&amp;#34;
&lt;/span>&lt;/span>&lt;span style="display:flex;">&lt;span>weight: NNNN
&lt;/span>&lt;/span>&lt;span style="display:flex;">&lt;span>---
&lt;/span>&lt;/span>&lt;span style="display:flex;">&lt;span>
&lt;/span>&lt;/span>&lt;span style="display:flex;">&lt;span># INC-NNNN: &amp;lt;What broke&amp;gt;
&lt;/span>&lt;/span>&lt;span style="display:flex;">&lt;span>
&lt;/span>&lt;/span>&lt;span style="display:flex;">&lt;span>| | |
&lt;/span>&lt;/span>&lt;span style="display:flex;">&lt;span>|---|---|
&lt;/span>&lt;/span>&lt;span style="display:flex;">&lt;span>| &lt;span style="font-weight:bold">**Date**&lt;/span> | YYYY-MM-DD — the day it began |
&lt;/span>&lt;/span>&lt;span style="display:flex;">&lt;span>| &lt;span style="font-weight:bold">**Site**&lt;/span> | |
&lt;/span>&lt;/span>&lt;span style="display:flex;">&lt;span>| &lt;span style="font-weight:bold">**Systems**&lt;/span> | Devices, hosts and roles involved, by inventory name |
&lt;/span>&lt;/span>&lt;span style="display:flex;">&lt;span>| &lt;span style="font-weight:bold">**Severity**&lt;/span> | What was lost, and what recovery required |
&lt;/span>&lt;/span>&lt;span style="display:flex;">&lt;span>| &lt;span style="font-weight:bold">**Status**&lt;/span> | Open · Service restored · Resolved (actions done) |
&lt;/span>&lt;/span>&lt;span style="display:flex;">&lt;span>| &lt;span style="font-weight:bold">**Times**&lt;/span> | Timezone used in this record |
&lt;/span>&lt;/span>&lt;span style="display:flex;">&lt;span>
&lt;/span>&lt;/span>&lt;span style="display:flex;">&lt;span>---
&lt;/span>&lt;/span>&lt;span style="display:flex;">&lt;span>
&lt;/span>&lt;/span>&lt;span style="display:flex;">&lt;span>&lt;span style="color:#75715e">## Summary
&lt;/span>&lt;/span>&lt;/span>&lt;span style="display:flex;">&lt;span>&lt;span style="color:#75715e">&lt;/span>
&lt;/span>&lt;/span>&lt;span style="display:flex;">&lt;span>What happened and why, in two short paragraphs.
&lt;/span>&lt;/span>&lt;span style="display:flex;">&lt;span>
&lt;/span>&lt;/span>&lt;span style="display:flex;">&lt;span>&lt;span style="color:#75715e">## Impact
&lt;/span>&lt;/span>&lt;/span>&lt;span style="display:flex;">&lt;span>&lt;span style="color:#75715e">&lt;/span>
&lt;/span>&lt;/span>&lt;span style="display:flex;">&lt;span>What stopped working, for whom, and for how long.
&lt;/span>&lt;/span>&lt;span style="display:flex;">&lt;span>
&lt;/span>&lt;/span>&lt;span style="display:flex;">&lt;span>-
&lt;/span>&lt;/span>&lt;span style="display:flex;">&lt;span>
&lt;/span>&lt;/span>&lt;span style="display:flex;">&lt;span>&lt;span style="color:#75715e">## Detection
&lt;/span>&lt;/span>&lt;/span>&lt;span style="display:flex;">&lt;span>&lt;span style="color:#75715e">&lt;/span>
&lt;/span>&lt;/span>&lt;span style="display:flex;">&lt;span>How the incident was noticed, by what or by whom, and how long that took. If it was not
&lt;/span>&lt;/span>&lt;span style="display:flex;">&lt;span>detected, say why not.
&lt;/span>&lt;/span>&lt;span style="display:flex;">&lt;span>
&lt;/span>&lt;/span>&lt;span style="display:flex;">&lt;span>&lt;span style="color:#75715e">## Timeline
&lt;/span>&lt;/span>&lt;/span>&lt;span style="display:flex;">&lt;span>&lt;span style="color:#75715e">&lt;/span>
&lt;/span>&lt;/span>&lt;span style="display:flex;">&lt;span>| Time | Event |
&lt;/span>&lt;/span>&lt;span style="display:flex;">&lt;span>|---|---|
&lt;/span>&lt;/span>&lt;span style="display:flex;">&lt;span>| | |
&lt;/span>&lt;/span>&lt;span style="display:flex;">&lt;span>
&lt;/span>&lt;/span>&lt;span style="display:flex;">&lt;span>&lt;span style="color:#75715e">## Symptoms
&lt;/span>&lt;/span>&lt;/span>&lt;span style="display:flex;">&lt;span>&lt;span style="color:#75715e">&lt;/span>
&lt;/span>&lt;/span>&lt;span style="display:flex;">&lt;span>What was observable, in the order it appeared: output, errors, what answered and what did not.
&lt;/span>&lt;/span>&lt;span style="display:flex;">&lt;span>
&lt;/span>&lt;/span>&lt;span style="display:flex;">&lt;span>-
&lt;/span>&lt;/span>&lt;span style="display:flex;">&lt;span>
&lt;/span>&lt;/span>&lt;span style="display:flex;">&lt;span>&lt;span style="color:#75715e">## Investigation
&lt;/span>&lt;/span>&lt;/span>&lt;span style="display:flex;">&lt;span>&lt;span style="color:#75715e">&lt;/span>
&lt;/span>&lt;/span>&lt;span style="display:flex;">&lt;span>How the cause was found, including the wrong conclusions and what overturned them.
&lt;/span>&lt;/span>&lt;span style="display:flex;">&lt;span>
&lt;/span>&lt;/span>&lt;span style="display:flex;">&lt;span>&lt;span style="color:#75715e">## Root cause
&lt;/span>&lt;/span>&lt;/span>&lt;span style="display:flex;">&lt;span>&lt;span style="color:#75715e">&lt;/span>
&lt;/span>&lt;/span>&lt;span style="display:flex;">&lt;span>The fault, or the faults that combined. Cite the file and line.
&lt;/span>&lt;/span>&lt;span style="display:flex;">&lt;span>
&lt;/span>&lt;/span>&lt;span style="display:flex;">&lt;span>&lt;span style="color:#75715e">## Recovery
&lt;/span>&lt;/span>&lt;/span>&lt;span style="display:flex;">&lt;span>&lt;span style="color:#75715e">&lt;/span>
&lt;/span>&lt;/span>&lt;span style="display:flex;">&lt;span>How service came back, and the state things were left in.
&lt;/span>&lt;/span>&lt;span style="display:flex;">&lt;span>
&lt;/span>&lt;/span>&lt;span style="display:flex;">&lt;span>&lt;span style="color:#75715e">## Contributing factors
&lt;/span>&lt;/span>&lt;/span>&lt;span style="display:flex;">&lt;span>&lt;span style="color:#75715e">&lt;/span>
&lt;/span>&lt;/span>&lt;span style="display:flex;">&lt;span>What made it possible, worse, or harder to see — without being the cause.
&lt;/span>&lt;/span>&lt;span style="display:flex;">&lt;span>
&lt;/span>&lt;/span>&lt;span style="display:flex;">&lt;span>-
&lt;/span>&lt;/span>&lt;span style="display:flex;">&lt;span>
&lt;/span>&lt;/span>&lt;span style="display:flex;">&lt;span>&lt;span style="color:#75715e">## Corrective actions
&lt;/span>&lt;/span>&lt;/span>&lt;span style="display:flex;">&lt;span>&lt;span style="color:#75715e">&lt;/span>
&lt;/span>&lt;/span>&lt;span style="display:flex;">&lt;span>Fix the faults behind this incident.
&lt;/span>&lt;/span>&lt;span style="display:flex;">&lt;span>
&lt;/span>&lt;/span>&lt;span style="display:flex;">&lt;span>| # | Action | Where | Status |
&lt;/span>&lt;/span>&lt;span style="display:flex;">&lt;span>|---|--------|-------|--------|
&lt;/span>&lt;/span>&lt;span style="display:flex;">&lt;span>| 1 | | | Open |
&lt;/span>&lt;/span>&lt;span style="display:flex;">&lt;span>
&lt;/span>&lt;/span>&lt;span style="display:flex;">&lt;span>&lt;span style="color:#75715e">## Preventive actions
&lt;/span>&lt;/span>&lt;/span>&lt;span style="display:flex;">&lt;span>&lt;span style="color:#75715e">&lt;/span>
&lt;/span>&lt;/span>&lt;span style="display:flex;">&lt;span>Stop this class of failure recurring, here or elsewhere, or make surviving it unnecessary.
&lt;/span>&lt;/span>&lt;span style="display:flex;">&lt;span>
&lt;/span>&lt;/span>&lt;span style="display:flex;">&lt;span>| # | Action | Where | Status |
&lt;/span>&lt;/span>&lt;span style="display:flex;">&lt;span>|---|--------|-------|--------|
&lt;/span>&lt;/span>&lt;span style="display:flex;">&lt;span>| | | | Open |
&lt;/span>&lt;/span>&lt;span style="display:flex;">&lt;span>
&lt;/span>&lt;/span>&lt;span style="display:flex;">&lt;span>&lt;span style="color:#75715e">## Lessons learned
&lt;/span>&lt;/span>&lt;/span>&lt;span style="display:flex;">&lt;span>&lt;span style="color:#75715e">&lt;/span>
&lt;/span>&lt;/span>&lt;span style="display:flex;">&lt;span>What to carry into the next change, stated so it applies beyond this one incident.
&lt;/span>&lt;/span>&lt;span style="display:flex;">&lt;span>
&lt;/span>&lt;/span>&lt;span style="display:flex;">&lt;span>-
&lt;/span>&lt;/span>&lt;span style="display:flex;">&lt;span>
&lt;/span>&lt;/span>&lt;span style="display:flex;">&lt;span>&lt;span style="color:#75715e">## Related changes
&lt;/span>&lt;/span>&lt;/span>&lt;span style="display:flex;">&lt;span>&lt;span style="color:#75715e">&lt;/span>
&lt;/span>&lt;/span>&lt;span style="display:flex;">&lt;span>The change that caused it, if any; changes made in the same window; the changes that fixed it.
&lt;/span>&lt;/span>&lt;span style="display:flex;">&lt;span>
&lt;/span>&lt;/span>&lt;span style="display:flex;">&lt;span>-
&lt;/span>&lt;/span>&lt;span style="display:flex;">&lt;span>
&lt;/span>&lt;/span>&lt;span style="display:flex;">&lt;span>&lt;span style="color:#75715e">## Related runbooks
&lt;/span>&lt;/span>&lt;/span>&lt;span style="display:flex;">&lt;span>&lt;span style="color:#75715e">&lt;/span>
&lt;/span>&lt;/span>&lt;span style="display:flex;">&lt;span>Procedures used during recovery, and procedures written or changed because of it.
&lt;/span>&lt;/span>&lt;span style="display:flex;">&lt;span>
&lt;/span>&lt;/span>&lt;span style="display:flex;">&lt;span>-
&lt;/span>&lt;/span>&lt;/code>&lt;/pre>&lt;/div>&lt;hr>
&lt;h2 id="filling-it-in">
 Filling it in
 &lt;a class="anchor" href="#filling-it-in">#&lt;/a>
&lt;/h2>
&lt;ul>
&lt;li>&lt;strong>Detection and symptoms are different things.&lt;/strong> Detection is &lt;em>how the incident came to
light&lt;/em>, and how long that took. Symptoms are &lt;em>what could be seen&lt;/em>, whether or not anyone read
it correctly at the time. An incident that was visible but misread is recorded in both.&lt;/li>
&lt;li>&lt;strong>Corrective actions and preventive actions are different things.&lt;/strong> Corrective actions fix
the faults that caused this incident. Preventive actions stop the same class of failure
elsewhere, or make surviving it unnecessary. Number actions across both tables, so that &amp;ldquo;action
4&amp;rdquo; means one thing wherever it is cited.&lt;/li>
&lt;li>&lt;strong>Keep action status current.&lt;/strong> Mark an action done with the commit or PR that did it. Until
every action is done, the record&amp;rsquo;s status says so.&lt;/li>
&lt;li>&lt;strong>Keep the record, even when it is embarrassing.&lt;/strong> A wrong conclusion committed to git during
the incident belongs under Investigation, quoted, with what overturned it.&lt;/li>
&lt;li>&lt;strong>Evidence over recollection:&lt;/strong> command output, recap counts, timestamps. Cite files and
lines so the analysis stays checkable after the code moves on.&lt;/li>
&lt;li>&lt;strong>Link both ways.&lt;/strong> If a change caused the incident, name it under Related Changes, and add
the incident to that change record&amp;rsquo;s header table.&lt;/li>
&lt;/ul></description></item></channel></rss>