This is the story of a crash investigation — and more usefully, of the tools and thinking that solved it. It began when my 2011 MacBook Air (Fedora 44 KDE) developed an alarming habit: launching a newly installed app made the screen go dark, spawned a flurry of crash reporters, left the task bar missing icons, and occasionally took Firefox down minutes later.
The obvious suspect was the new app. The evidence, as it turned out, had other ideas — and following it taught me more about Linux diagnostics than a year of trouble-free computing ever could. Along the way I was confidently wrong three times, which is rather the point of this article.
The first principle: many crashes, one cause
When several unrelated programs crash simultaneously, it’s almost never several bugs. It’s one shared resource failing underneath all of them. Desktop widgets, tray icons and Firefox don’t share code — but they do share the GPU, the memory, and (on Wayland) the compositor. So the question was never “what’s wrong with this app?” but “what do all the victims have in common?” That reframing decides where you look first.
Tool 1: journalctl — and the art of reading an acquittal
My first theory was a GPU hang: the Air’s Intel HD 3000 is ancient, and a kernel-level GPU reset kills every OpenGL client at once — which fits the symptoms perfectly. The kernel logs such resets loudly, so I went looking:
journalctl -k -b | grep -iE "oom|hang|reset|gpu"The results looked promising — until I actually read them. Every hit was an innocent bystander that merely contained the letters: ACPI announcing the graphics device at boot, the OOM daemon starting up (not killing anything), USB hubs “reset” during an ordinary suspend/resume cycle. Grep finds strings, not guilt. The log was in fact spotless — no GPU hang, no OOM kill — and a clean kernel log is itself a finding: it exonerates the kernel and moves the investigation up into userspace. Theory one, dead. (Bonus tool: journalctl --list-boots shows the journal’s table of contents, one line per boot, and lets you replay earlier boots with -b -1, -b -2 and so on. Mine even revealed sessions from a Fedora install I’d forgotten this machine ever had — the journal remembers what you don’t.)
Tool 2: coredumpctl — the morgue ledger
Here’s the tool I wish I’d met years earlier. systemd quietly catalogues every process crash on your system — timestamped, with the signal that killed it and the full core dump filed away:
coredumpctl listOne command turned my vague “the desktop freaks out sometimes” into hard data: kwin_wayland dying of SIGBUS at every single incident, with plasmashell following minutes later, timestamps matching my crashes to the minute. That pattern explains the entire spectacle, because on Wayland the compositor is a single point of failure: when kwin dies, every window and widget dies with it. Dark screen (compositor gone), crash reporters (the casualties), missing tray icons (plasmashell’s wounded restart), fragile Firefox (a survivor reconnecting to a restarted compositor). One death, many bodies — exactly as the first principle predicted, just one layer higher than I’d guessed.
The signal itself narrows things: SIGBUS is rarer than the everyday SIGSEGV, and its classic meaning is “you touched memory-mapped file territory that doesn’t really exist” — a truncated or unreadable file behind an mmap. That single word pointed the whole investigation at files, not code. My second theory (a misbehaving client resizing a shared-memory window buffer while kwin read it) was built on exactly this — right mechanism, and still wrong, as the next tool proved.
Tool 3: reading the stack trace — the part that outranks all theories
coredumpctl info kwin_waylandTwo reading tips, because the output is intimidating. First, the long “Module … from rpm …” wall is just an inventory of everything the process had loaded — the suspects present in the building, not the culprit. Second, the actual verdict is the stack trace further down, and you read it bottom-up as a story: main loop → paint the screen → paint a window → an animation effect → compile a shader → and then, at the very top, the exact function where the bus error struck:
#4 disk_cache_has_key (libgallium-26.1.4.so)
#5 can_skip_compile.part.0 (libgallium-26.1.4.so)
#6 _mesa_glsl_compile_shader (libgallium-26.1.4.so)disk_cache_has_key is Mesa’s on-disk shader cache — a memory-mapped file under ~/.cache/ where compiled shaders are stored so they needn’t be recompiled. The new app was never special: its window animation simply forced kwin to compile a shader, Mesa checked its cache, and the mmap’d cache file bit back. That also explained why the crash survived reboots and even a kwin version update — the cache lives in the home directory, untouched by both. Theory three wrote itself: corrupted cache file. Delete it, done.
When the fix doesn’t fix: the most informative failure
I deleted the cache. The crash returned — with a byte-identical stack trace, in a freshly created cache. A brand-new file producing the same failure isn’t damage; it’s behaviour. Then two more facts landed: booting into GNOME instead of KDE produced the same crash in mutter (so it’s below both desktops), and a targeted disk interrogation — journalctl -k | grep -iE "csum|i/o error" for Btrfs checksum failures and SATA errors — came back completely clean, acquitting the fifteen-year-old SSD. Every local suspect was now eliminated: not the app, not KDE, not GNOME, not the disk, not one corrupt file. What remains when everything local is innocent? A bug in the shared code itself.
The final tool: someone else’s crash
A stack trace is also a search query. Searching for the function names turned up Launchpad bug #2126903: gnome-shell on Ubuntu, dying of SIGBUS in the identical call chain — disk_cache_has_key, can_skip_compile, _mesa_glsl_compile_shader — confirmed against Mesa, with Ubuntu’s automated error tracker collecting reports en masse. A known upstream bug, hitting strangers on entirely different hardware. Better still: the Ubuntu folks had tagged it [amdgpu], assuming it was AMD-driver-specific. My reproduction on an Intel HD 3000 disproves that theory — which is exactly the kind of data point worth contributing back. If you file or comment on bugs with your distro’s slider set to “share everything”, this is why: your weird old laptop can redirect an entire investigation.
The workaround, until Mesa ships a fix, is to route around the buggy cache entirely:
echo 'MESA_SHADER_CACHE_DISABLE=1' | sudo tee -a /etc/environmentReboot, verify with echo $MESA_SHADER_CACHE_DISABLE, and enjoy the silence. The cost is a barely-perceptible delay the first time each shader compiles. The machine that crashed daily then ran a nine-hour writing session on battery without a squeak.
What the investigation actually taught
The stack trace outranks the story. I had three theories that each explained every symptom beautifully — GPU hang, hostile shared-memory buffer, corrupted cache file — and each was killed by the next piece of evidence. That’s not embarrassing; that’s the method working. A plausible narrative is a hypothesis, not a conclusion, and the discipline is deleting it the moment better evidence outranks it.
The machine keeps receipts. Between the journal (per-boot kernel and system logs), coredumpctl (every crash, catalogued with its backtrace) and rpm -qf (which package owns this file?), a Linux system is astonishingly well documented about its own misfortunes. Most mysteries aren’t unsolvable — they’re just unread.
Preserve history. This machine was nearly wiped for a fresh install a few weeks earlier; I upgraded instead. That decision preserved the journal archive and the coredump ledger that solved the case in an afternoon. An undocumented system can only be reinstalled; a documented one can be healed. And when your crash matches a stranger’s on the other side of the internet, the receipts become a contribution — the bug gets fixed a little faster for everyone, including the next owner of a stubborn old laptop that refuses, magnificently, to die.