Recovering and Rebuilding the Nerve Center

There is a special kind of confidence you feel when a decommissioning plan says everything important has moved. There is a different feeling when you stop the old server and your favorite dashboard disappears with it.

That is what happened to the Nerve Center.

The Nerve Center is the page I use to see the state of the Spartina Landing homelab in one place. Solar production, battery condition, house load, host health, active alerts, packages, cameras, and the overnight charge review all live there. It started as an experiment on an OpenCode LXC, then quietly became part of how I operate the house.

During a larger consolidation, I retired that OpenCode container. The archive passed its checksum. Its replacement tools worked. The old DNS record was removed. The container was stopped and set not to start at boot.

Then I tried to open the Nerve Center. Nothing.

This was not an exotic distributed-systems failure. We had decommissioned the machine before treating the dashboard on it as a production service. During the host migration, Brock, my operations agent, had checked the main boxes and treated the old LXC as retired before accounting for the Nerve Center. I gave him the blunt version: This application is extremely important to me. Do not stop until it is recovered and confirmed working in totality.

The data still existed, but recovering it meant rebuilding the Nerve Center as a real service instead of leaving it attached to an experimental host.

First, bring the old host back without pretending it is fixed

The first recovery step was temporary and deliberate. We started the retired LXC long enough to inventory everything behind the dashboard. This was a recovery boot, not a reversal of the retirement decision. After migration and verification, LXC 109 would be stopped again and left disabled at boot.

Simply getting the old page to answer again would have been a poor finish. It would restore the symptom while preserving the dependency that caused the outage.

The inventory found more than an HTML file:

• An nginx container serving the status page
• Prometheus, Grafana, node-exporter, and cAdvisor
• Python and shell collectors for Home Assistant telemetry, fleet health, history, alerts, package state, and charge-review data
• Prometheus time-series history and Grafana state in named Docker volumes
• Cron entries that refreshed some data while other files were only historical leftovers
• Hard-coded paths, host labels, links, and host-specific configuration tied to the OpenCode machine
• A read-only Home Assistant credential that needed to move without appearing in chat, logs, or documentation

That inventory changed the job from "copy the web page" to "move a small monitoring platform." Copying only the visible files would have produced a convincing page backed by stale JSON.

Choosing the new home

I moved the Nerve Center to the Hermes LXC. Hermes had become the surviving operations host during the same consolidation, so it was the logical place for the private operations console. Public web services, solar control, and Home Assistant remained separate. The Nerve Center would observe them without becoming another controller.

Docker was not installed on the destination, so the rebuild began there. Even that had a small trap: the package name I expected for the Compose plugin was not available in the host's repository. The correct package was Compose V2 under a different name. Not dramatic, but exactly the sort of detail a migration checklist misses when it says only "install Docker."

The recovered application now lives under a neutral operations path instead of a user's home directory:

/opt/nerve-center/
├── status/       # HTML and generated JSON
├── automation/   # read-only collectors
├── monitoring/   # Compose, Prometheus, Grafana, nginx
├── systemd/      # source copies of installed units
└── docs/          # recovery and operating notes

The private credential moved through the local credential-handling path into a root-only file. The recovery process verified only that the file existed, had mode 0600, contained the expected key name, and worked. The value never needed to cross the conversation.

Preserving history instead of starting over

Grafana without its dashboards and Prometheus without its history would have been a new installation wearing the old application's name. I wanted the real thing back.

We briefly paused the old Prometheus and Grafana containers, archived their Docker volumes, and restored those archives into new named volumes on Hermes. The destination came up with the existing metrics and dashboards intact.

The restored Grafana instance reported four dashboards, including the fleet overview and dedicated application and Mac performance views. Prometheus loaded the restored history and returned valid query results. That was the point where the migration began to feel like recovery rather than reconstruction.

The hidden dependencies showed themselves

The stack started, but "the containers are up" is not the same as "the application works." Several problems appeared only after the services were running.

cAdvisor expected devices the LXC did not expose

The original Compose file mounted /dev/kmsg and /dev/disk. Those paths were not available inside the destination LXC. cAdvisor can still provide useful container metrics without them, so we removed the unsupported optional mounts rather than changing the LXC's security profile just to satisfy an old configuration.

Prometheus accused five healthy containers of being dead

The first health summary showed five critical failures: Prometheus, Grafana, node-exporter, cAdvisor, and the status page. All five were visibly running.

The alert expression was matching every historical container name in the restored time series. The old containers had names such as prometheus and grafana; the rebuilt stack used names prefixed with nerve-center-. Prometheus was correctly remembering that the old names had stopped reporting and incorrectly turning that history into a current outage.

We scoped the rule to the new container namespace:

time() - max by (name) (
  container_last_seen{name=~"nerve-center-.+"}
) > 60

The false alerts should have cleared after a configuration reload, but they did not. The container still saw the old rule. Individual file bind mounts had preserved the old inode when the host file was replaced. We changed Prometheus to mount the configuration directory, recreated the container, then queried its rules API to prove the active expression had changed. The alert list dropped to zero.

The camera cards needed a route, not a new camera system

The Nerve Center expected Frigate under a relative /frigate/ path. Frigate actually lived on the media host. The old environment had made that relationship easy to overlook.

We added an nginx read-only proxy to the existing Frigate API. Both configured camera names appeared in the stats response, and each snapshot endpoint returned a real JPEG. No second Frigate instance, no camera reconfiguration, and no control path were added.

The page still believed it lived on OpenCode

The HTML displayed the old hostname, old filesystem paths, old private links, and other host-specific configuration. Those details were not fatal, but leaving them would turn the dashboard into a source of bad instructions. We updated the system-reference card, internal links, collector paths, host identity, and private console URLs, then searched the migrated tree for stale OpenCode references.

Making it survive the next reboot

The old stack depended on a mix of Docker restart policies and cron. The recovered version has explicit systemd ownership.

nerve-center-stack.service
    └── starts the five-container Docker Compose stack

nerve-center-collectors.timer
    └── refreshes dashboard data every five minutes

The collector runner uses a lock so two refreshes cannot overlap. Each collector reports its own result. A failure in one source is visible instead of silently preventing every other JSON file from updating. The service and timer are enabled at boot, and the source unit files live beside the application for disaster recovery.

We also wrote an actual recovery document. It lists the application tree, both Docker volumes, the protected credential location, systemd units, restore sequence, and concrete health probes. I would rather test a boring restore recipe later than repeat an exciting archaeology project.

Migration verification covered the whole path

Before stopping OpenCode again, we tested the replacement from several angles:

• All five Nerve Center containers were running; the containers with health checks reported healthy.
• The status page, Grafana health API, Prometheus readiness endpoint, node-exporter metrics, and cAdvisor health endpoint all returned HTTP 200.
• All eight Prometheus scrape targets reported up, including the local stack and the remote homelab hosts.
• Grafana listed all four restored dashboards.
• The Home Assistant collector produced fresh solar, battery, house, weather, vehicle, and vacuum telemetry.
• The Frigate proxy returned both camera names and valid JPEG snapshots.
• The page loaded through its private network address in a real browser with no JavaScript errors.
• A manual collector run advanced the telemetry timestamp, proving the page was not merely serving copied data.

Only after those checks passed did we stop the OpenCode LXC again and verify that it remained disabled at boot. The Nerve Center continued running on Hermes with the old host offline.

The restored page still has one known piece of technical debt: it loads Tailwind through the CDN, which produces a production-use warning in the browser console. That is not an outage, and it did not justify delaying recovery, but it is documented rather than hidden.

The recovery turned into a rebuild

Once the Nerve Center had a durable home, it became the obvious place for the diagnostics that do not belong in a normal Home Assistant dashboard.

The best example is the new EG4 inverter page. My solar system has three EG4 6000XP inverters in parallel. I wanted per-inverter and split-leg detail, including L1/L2 voltage, active and apparent power, daily energy, grid voltage, temperatures, faults, warnings, imbalance, and freshness.

The easy but wrong design would have been to let the dashboard open another Modbus connection. Instead, the production poller remains the sole reader of the inverters. It writes one atomic snapshot. A read-only Nerve Center collector validates and copies that snapshot every ten seconds. The browser never talks to Modbus, and the dashboard has no path to the inverter actuator.

EG4 inverters
      ↓ read-only FC04
sole production poller
      ↓ atomic JSON snapshot
Nerve Center collector
      ↓ validated dashboard JSON
three-inverter diagnostics page

The page uses three side-by-side device cards, a comparison table, explicit data provenance, and visible range failures instead of turning missing values into zeros. After the later verification pass, the work finished with 107 production poller tests, eight collector tests, and 46 final end-to-end Nerve Center checks passing. A hamburger menu now connects the overview, inverter diagnostics, Grafana, Prometheus, and the other operations tools without filling every header with icons.

What I changed in the decommissioning rule

The mistake was not that the Nerve Center lacked a backup. The files and data were recoverable. The mistake was treating a host inventory as a service inventory.

A machine can be safe to retire only after every service on it has one of four outcomes: migrate it, replace it, archive it intentionally, or delete it intentionally. "The main application moved" is not enough. Neither is "the container has a backup." If people depend on a page, script, scheduled job, data volume, DNS name, or credential, it belongs in the retirement plan.

I also reinforced a boundary that matters around home energy systems. The Nerve Center is an observation surface. It can show what the solar controller plans, what the inverter poller sees, and whether the overnight result matched the plan. It cannot call Home Assistant services, change an inverter setting, command a vehicle, or repair a host. Those actions belong to separate, reviewed paths.

Today the Nerve Center is no longer an accidental passenger on an experimental coding server. It has a durable host, explicit startup ownership, preserved history, protected credentials, tested collectors, recovery documentation, and an expanding diagnostics layer.

The practical rule is now written down: retire services, not just hosts.