Copy the panic text before you touch anything. Photograph the console, scroll back through the serial console or the hypervisor console, or pull it from the BMC/IPMI log — whatever the box gives you, capture it first. The panic message names the function that failed, the CPU it failed on, and usually the module involved. That text is the only record you have if kdump was never configured, and it is the starting point even if it was.
Note what was running. What changed recently — a kernel update, a new module, a driver update, a storage path change, a backup job starting, a load spike. Panics cluster around changes, and “nothing changed” usually means something changed and nobody wrote it down.
Do not power-cycle blindly if kdump is configured. On a kdump-enabled box the panic triggers the kdump kernel, which writes the vmcore and then reboots on its own. Power-cycling mid-dump destroys the vmcore you were about to get. Wait for the dump to finish — watch the console for the kdump progress — and only intervene if the dump itself has hung.

The kernel parameter panic= sets how long the kernel waits after a panic before rebooting, in seconds. panic=0 means never reboot — the box sits at the panic message until someone power-cycles it. panic=10 means reboot after ten seconds. Neither is wrong; they are different trade-offs. panic=0 is the right choice when you need the panic text to stay on the console for a human to read it, which is common on a box with no serial console and no kdump. panic=10 (or 30, or 60) is the right choice for a server that has to come back on its own — a VM in a fleet, a box in a remote datacentre, anything where a human cannot reach the power button quickly.
Set it in the boot loader, not at runtime. Add panic=10 to the kernel command line in /etc/default/grub (GRUB_CMDLINE_LINUX) and regenerate the grub config — grub2-mkconfig -o /boot/grub2/grub.cfg on BIOS systems, /boot/efi/EFI/redhat/grub.cfg on UEFI — then reboot once, deliberately, to confirm it took. A panic= value set in a running system via sysctl does not exist; it is a boot-time parameter, and setting it anywhere else is setting nothing.
SysRq is the manual override. Alt+SysRq+c triggers a crashdump on demand — the test you use to prove kdump works — and Alt+SysRq+b reboots immediately. On a VM the hypervisor console usually has a “send SysRq” option, and on a physical box the BMC does. Know where it is before the panic, not during it.

What kdump is. A second, small kernel — the capture kernel — that boots when the first one panics, with one job: write the crashed kernel’s memory (the vmcore) to disk, then reboot the box. The capture kernel runs from memory reserved at boot by the crashkernel= parameter, which is why it has to be set before the panic. No reservation, no capture kernel, no vmcore.
Install and enable it:
dnf install kexec-tools crash — RHEL 8/9/10. On RHEL 6/7 the package is the same (kexec-tools), installed with yum.
crashkernel=auto — the boot parameter; auto picks a reservation size from the box’s RAM, which is fine for most boxes. On a box with very large RAM, or one where auto reserves too little for the capture kernel to load, set it explicitly — crashkernel=512M is a common explicit value, and the kdump service will warn you at start if the reservation is too small.
systemctl enable --now kdump — and check it: systemctl status kdump should report active, and kdumpctl status should say the kdump kernel is loaded. If it says the crash kernel is not loaded, the reservation did not take — usually because the boot loader config was not regenerated after adding crashkernel=, or because the box was not rebooted after the change.
Where the vmcore goes — /etc/kdump.conf. Local disk is the default (path /var/crash), and it is the simplest: the vmcore lands in /var/crash/<date>/vmcore, and you need that filesystem to have room — a vmcore is roughly the size of RAM, minus what the capture kernel excludes. NFS (nfs server:/export) and SSH (ssh user@server) are the alternatives for boxes whose local disk cannot hold a vmcore, or for fleets where the cores are collected centrally. The trade-off is the obvious one: local is fast and needs disk space; network needs the network to be up during a panic, which is usually true but is exactly the thing you cannot guarantee.
Test it before you need it — on a throwaway box, never first on production. echo c > /proc/sysrq-trigger as root crashes the kernel on purpose and fires kdump. The box reboots, a vmcore appears in /var/crash, and you have proven the whole chain works. Do this once, on a VM you can afford to lose, and then never again on that box. Testing kdump on production is how you cause the outage you were preparing for.
Version differences, because a procedure post names every version’s path. RHEL 6/7: service kdump start and chkconfig kdump on, crashkernel=auto in /boot/grub/grub.conf (RHEL 6) or /etc/default/grub (RHEL 7), grub2-mkconfig on RHEL 7. RHEL 8/9/10: systemctl, /etc/default/grub with grub2-mkconfig, and on RHEL 9/10 the kdump service also manages the crashkernel= reservation through kdumpctl — kdumpctl estimate will tell you what the reservation should be for the box’s RAM. The crash tool itself is the same across all of them, which is the one constant in this post.

Open it:
crash /var/crash/<date>/vmcore /usr/lib/debug/lib/modules/<kernel-version>/vmlinux
The vmlinux with debug symbols is what makes the output readable — without it, crash shows addresses instead of function names, and you are guessing. Install it with debuginfo-install kernel (RHEL 6/7) or from the debug repos on RHEL 8/9/10. If the vmlinux version does not match the crashed kernel exactly, crash will refuse — and that is correct behaviour, because a mismatched vmlinux would lie to you.
bt — what was the kernel doing when it died? The backtrace of the panicking CPU. Read it bottom to top: the functions at the bottom are the call chain, the one at the top is where it failed. A backtrace that ends in a driver function names the driver. A backtrace that ends in memory management or the scheduler names the subsystem, and the driver that called into it is the suspect.
log — what did the kernel say before it died? The kernel log buffer from the crashed kernel, which is the panic text plus everything leading up to it — the warnings, the errors, the first fault before the fatal one. The panic message is the last line; the cause is usually several lines earlier.
ps — what was running? The process list at the time of the panic. A process stuck in uninterruptible sleep (D state) on a storage path that died is a different story from a process in a driver call. ps with the crashing task highlighted tells you whether the panic was triggered by a workload or by something the kernel was doing on its own.
vm — was memory the problem? The virtual memory summary: free memory, slab usage, and whether the box was under memory pressure. A panic on a box with no free memory and a slab cache grown without bound is an out-of-memory story wearing a panic’s clothes, and the fix is in the workload, not in the kernel.

1. Third-party and out-of-tree modules. The most common cause, by a distance. A vendor driver, a storage agent, a monitoring module — anything not shipped with the kernel, and anything tainted (cat /proc/sys/kernel/tainted — a non-zero value means a third-party module is loaded, and the panic message will say so). The fix is usually a module update, a module removal, or a kernel update that the module was not built for. A panic that starts the day a module was installed is not a coincidence.
2. Failing RAM. A bit flip in kernel memory panics the box, and it panics in a different place every time — which is the signature. Run memtest86+ from a boot, or check the BMC/IPMI for ECC errors: a box with ECC logging correctable errors for weeks before the panic was warning you. Non-ECC RAM gives you no warning at all, which is why servers use ECC.
3. Failing disk or a storage path that died underneath. The disk the root filesystem sits on goes away — a SAN path fails, a multipath device loses all paths, an HBA resets — and the kernel panics when it cannot read what it needs. The backtrace ends in the block layer or the filesystem, and log shows I/O errors before the panic. The fix is in the storage, not in the kernel, and the panic is the symptom.
4. A genuine kernel bug. The least common cause, and the one everyone assumes first. A kernel bug panics the same way, on the same workload, reproducibly — which is also how you prove it is one. If it is not reproducible, it is probably one of the three above.

Write it down while it is fresh: the panic text (verbatim), the vmcore location and date, the kernel version (uname -r), the loaded module list (lsmod output, or cat /proc/modules), and what changed recently. That is the whole after-action record, and it is the difference between a panic that gets diagnosed and a panic that gets forgotten until it happens again — at which point the module list and the change history are gone, and the second panic is as mysterious as the first.
Keep the vmcore. A vmcore is large and tempting to delete. Keep it until the cause is found and fixed, because a second opinion — a vendor, a colleague, a mailing list — needs the core, not your summary of it. Archive it with the panic text and the module list, in one place, and note the kernel version on the archive: a vmcore without its matching vmlinux is unreadable, and a vmcore whose kernel version nobody recorded is a file you cannot use.

First 60 seconds: capture the panic text, note what changed, don’t power-cycle mid-dump if kdump is configured.
panic=: panic=0 never reboots, panic=10 reboots after 10s — set it in grub, not at runtime.
kdump: crashkernel=auto + kexec-tools + systemctl enable --now kdump; test with sysrq-c on a throwaway box, never first on production.
crash: bt what was it doing, log what did it say, ps what was running, vm was memory the problem.
Causes: third-party modules first, then RAM, then storage — a genuine kernel bug is the least common and the most assumed.
After-action: panic text + vmcore + kernel version + module list, kept together until the cause is found.