Incident Records #
One record per incident on Deevnet infrastructure: what broke, how it was found, why it
happened, how service came back, and what was done so it does not happen again. Records are
numbered INC-NNNN in the order they are opened, like
ADRs,
and grouped by year.
Change Management states that “manual changes without validation are considered defects.” Most incidents are what happened when a change reached a live site without the validation that would have caught it. Each record is written so the next person meets the failure mode in a document rather than in the dark.
Like Change Records, these are retained information: evidence, written once, updated only as their actions close. How to write one, and the template, are maintained in the runbook under Incident Management.
Records #
| ID | Date | Incident | Site | Root cause | Status | Substatus |
|---|---|---|---|---|---|---|
| INC-0005 | 2026-10-07 | Core Router Hard Hang During a Provisioning VM Rebuild | mobile | Not established: a second hard hang during a bulk image push across the router, recurring INC-0004 | Open | Mitigated |
| INC-0004 | 2026-09-21 | Core Router Lost: a Hard Hang, Then Recurring re0 Watchdog Timeouts | mobile | re0 (Realtek) watchdog timeouts, recurring; the first event’s hard hang is not yet explained | Open | Investigating |
| INC-0003 | 2026-09-17 | OpenBao’s Recovery Key and AppRole Destroyed by a Git Reset | mobile | Once-only secrets not locked in (pushed) before git reset --hard in a decrypted repository | Closed | Completed 2026-09-21 |
| INC-0002 | 2026-09-15 | Controller VM Silent — Running but Off the Network | mobile | Not established; restored by a guest reboot | Open | Mitigated |
| INC-0001 | 2026-09-07 | Firewall Policy Deleted, Total Connectivity Loss | mobile | Empty desired set treated as authoritative by an ungated reconcile | Closed | Completed 2026-09-19 |
Status and substatus are defined in Incident Management.