Most of the interesting failures in my lab happen on servers I never touch. This one happened on the laptop I use every day, in a desktop environment I had just rebuilt from scratch, and it managed to stay invisible for a day and a half while quietly eating a CPU core.
I run Omarchy on my daily driver: an Arch-based, Hyprland-powered desktop that comes pre-wired with sensible defaults and a real plugin system. Omarchy replaced a pile of scripts I had accumulated over the years, and it gave me something I actually wanted — a place to hang my own homelab widgets.
It also gave me three lessons I keep re-learning: a keybind that does nothing is not a keybind, a reload is not an apply, and if you want to know whether a process is working, look at what it is producing, not how busy it looks.
The bar is a plugin host, not a status tray
The part of Omarchy I care about most is the bar. It is not a collection of hardcoded indicators; it hosts Quickshell plugins, and plugins can register widgets, overlays, panels and services. That is why I moved: I could write a widget that answers a question about my house in one keypress.
The Spartina plugin is the one I use constantly. Press a key and an overlay reports the state of the house and the lab: solar and house load, battery state of charge and stored energy, today's solar and grid import, the car, host health, camera count, and current alerts. Press escape and it is gone. No browser, no dashboard tab, no second status page to keep in sync.
Two of my other plugins are smaller versions of the same idea. One puts solar production in the bar, one shows Tempest weather station readings. A third, the plugin updater, checks every installed plugin for updates every four hours and lists what is behind — including third-party plugins I did not write, since it just scans every directory with a manifest.
I also use a community Home Assistant plugin (the hass plugin by Konrad Kruk) which talks to my HA instance over the local API. Its README documents fewer capabilities than the panel actually has — the QML exposes open, close, toggle, refresh and entity-state methods that the docs never mention. I learned that by reading the source instead of trusting the documentation, and that habit paid for itself later the same week.
The keybind that did nothing
I wanted one keypress to open the Home Assistant panel. Omarchy generates its bindings from a Lua config, and the helper is a one-liner:
o.bind("SUPER + H", "Home Assistant", "omarchy-shell hass toggle")My first version did nothing at all. No error, no log line, no flash of a window. It was also registered — hyprctl binds listed it with the correct key combination and description. From the compositor's point of view, that binding existed.
The cause was a habit from every other Linux desktop configuration I have ever written. In Hyprland's native config language, a keybind is a dispatcher and arguments, so muscle memory says to write exec, command. The Omarchy helper instead passes a bare command string. Give it exec, foot -e something and it cheerfully launches a program named exec, with the rest as arguments — which fails silently and instantly.
The wrong form still shows up in the binding list. The right form is one string with no dispatcher prefix. The single strangest thing about this bug is how quiet it is; there is no failure to read anywhere unless you deliberately test the helper in isolation.
The reload that disabled my keyboard
Having edited the bindings file, the obvious next step was to apply it. On a Lua-config Hyprland session in the version I was running, hyprctl reload does not do that. It removes all Lua bindings — including the defaults the desktop shipped with — until the compositor is restarted.
So the reload I used to "apply" one new binding silently disabled the launcher, the terminal key, and everything else, while the config on disk looked perfect. Only bindings registered at startup actually fire.
That turned into a standing rule in this house: editing configuration files is free, but restarting the session — or the machine — requires asking first. The one exception worth knowing is the keybind menu. Each time you open it, it spawns a fresh interpreter that re-reads the config from disk, so it repairs itself with no restart at all.
The process that had been spinning for 34 hours
Then the laptop started feeling wrong. Fans, sluggishness, nothing dramatic. I looked at the process table sorted by CPU and found a Lua process at about a third of a core. Then another. Then four of them, the oldest running for 34 hours. Load average was over 15 on a four-core laptop.
They were not leftovers from my own scripts. They were the keybind menu's scan processes, wedged forever, one leaked per invocation. Opening that menu had been costing me a core since the previous morning.
What the scanner actually does
To print a keybinding list, Omarchy runs an inline Lua program that loads your personal config under deliberately permissive stub globals. Anything you ask the compositor for — monitors, clients, configuration values — returns a placeholder object that answers to every name and every call. That design is intentional: it lets the scanner read bindings out of a config that normally expects a real graphical session.
My personal config had a line that walked the monitor list with ipairs. Under the stub, "the monitor list" is that placeholder object, and iterating it never terminates: every index returns something truthy, so the loop never sees the nil that would end it. The scan spins forever, and every open of the menu leaves another process behind.
I bisected it the way you bisect anything you do not understand yet. First I extracted the scanner's own inline Lua from the system script and ran it under a timeout — that reproduced the hang and gave me the exit code to watch for. Then I copied my personal configuration files into a scratch home directory one at a time, requiring each in turn, which named the offending file in three steps.
# reproduce the scan standalone — exit 124 means the loop
awk "/lua <<'LUA'/{f=1;next} f&&/^LUA$/{exit} f{print}" \
/usr/share/omarchy/bin/omarchy-menu-keybindings > /tmp/scan.lua
timeout 6 lua /tmp/scan.lua; echo "exit=$?"The fix is a loop bound
The repair is not to avoid the compositor's getters, which is the useful part of the config. It is to never iterate their result directly. Count what you have, and only loop when the count is a real number:
-- WRONG: hangs the scan, spins a process forever
for _, monitor in ipairs(hl.get_monitors()) do
local is_internal = monitor.name:match("^eDP%-")
...
end
-- RIGHT: bounded iteration, identical behavior in a live session
local monitors = hl.get_monitors()
local monitor_count = (type(monitors) == "table") and #monitors or 0
for index = 1, monitor_count do
local monitor = monitors[index]
local name = monitor and monitor.name
if type(name) == "string" then ... end
endUnder the stub, the count is zero, so the loop body never runs and the scan finishes. In a live session the count is real and the loop behaves exactly as before. After the fix the scan returned more than 200 binding records and exited cleanly — and it needed no session restart, because that scanner is a fresh process every time.
The leaked processes were safe to kill, too: they were wedged scan children, not the compositor. Clearing them dropped load average back to normal within a minute.
The dock was never going to give me 60 Hz
A smaller mystery from the same machine: my travel dock drove a 4K monitor at 50 Hz, and every attempt to ask for 60 Hz was ignored. Runtime monitor changes on a Lua session also rejected the syntax I knew — the older keyword command is not accepted, and the compositor's newer evaluation call is, but it silently clamps requests to the modes the kernel actually offered.
The answer was not in the compositor at all. Comparing the monitor's decoded capabilities against the kernel's list of offered modes showed the 60 Hz mode had been pruned before the desktop ever saw it. The dock's DisplayPort lanes simply could not carry 4K at 60 Hz over a single stream; the hardware ceiling was 50 Hz. The desktop had been telling the truth the whole time, and the "bug" was an unstated assumption about what the dock could do.
The agent pane that was working and not working
One more useful distinction came out of the same week. I run coding agents on the laptop inside a terminal multiplexer, and one pane went quiet while the process stayed healthy and busy — a familiar shape to anyone who has watched an agent "think".
Two very different failures wear the same face. The operating system can be wedged — uninterruptible I/O, zombie state — which is rare. More often the process is perfectly alive and the work is wedged: it is burning CPU and completing nothing.
CPU percentage does not distinguish them. The session log does. Agents write a record when a message completes, so the file's modification time is the progress metric: frozen timestamp plus no live network connection to a model API equals a client-side spin, not a slow response. The correct repair is to interrupt the turn rather than kill the process, which returned the pane to idle with all its work intact.
What I changed
None of this was about Linux being hard. It was about my assumptions being invisible, and I wrote down the four that cost me the most time:
- Pixels beat the compositor's opinion. When a window is reported as mapped, visible and correctly sized while the screen shows nothing, screenshot the framebuffer and read it. The system's self-report is a claim, not evidence.
- A status surface must carry its own freshness. This is now a hard rule in my house-status widget. Any row that has not updated within twice its expected cadence renders dimmed with the age attached, and an entity that is unavailable renders as missing — never as a number. A comfortable-looking value from thirty hours ago is worse than no value at all, because it hides the exact failure the display exists to catch.
- Never apply by reloading. Know which changes need a restart, and say so out loud instead of calling an edit "live".
- A silent failure is a specification problem. Both the keybind and the scan hang were invisible by design. If a tool cannot tell you it failed, test it in isolation once so you know what its success shape looks like.
There is also a fifth, less technical one, and it is the reason all of the above got fixed in a morning instead of a weekend: an operations agent that treats "the service is up" and "the thing works" as different claims, and refuses to tell me the first when it means the second.
Where it stands
The laptop is quiet again. The status overlay answers in one keypress. The plugin updater keeps everything current without me thinking about it. The smallest of my own plugins — the one that puts solar production in the bar — has been running untouched since the day I enabled it, which is the highest compliment I can pay a piece of software I wrote myself.
The largest one is still in progress: a Tidal client as a native panel, with local playback as the next milestone. It follows the same theme as everything on this desktop, and I have no interest in shipping it until it works.
Omarchy has been worth the detour. A desktop you can extend in the same language as the rest of your infrastructure stops being something you maintain and starts being another surface your lab can speak through — as long as you keep checking the pixels.