root@rhel:~# journalctl -b -p err # every error since this boot, nothing else
journalctl Beyond -u and –since
The Log Investigation Guide
Every RHEL admin uses journalctl. Almost nobody uses it well. The usual pattern is journalctl -u nginx, then a pipe to grep, then scrolling, then giving up and opening the log file by hand — which defeats the entire point of a structured journal. The journal already knows the unit, the priority, the boot, the process, and the timestamp of every message. The trick is asking it properly, and knowing where it actually keeps its data.
# the investigation order — narrow before you read
1. Start with the boot, not the service.
2. Then the unit, with a time window.
3. Then the previous boot.
4. Then follow it live, narrowly.
journalctl -b -p err is the first command after any reboot: every error-priority message since this boot, across all units. If the box just came up wrong, the answer is usually in that list, and it is a short list.2. Then the unit, with a time window.
journalctl -u nginx --since "1 hour ago" — unit and time together, not one or the other. Add --until "30 minutes ago" when the incident has a known end and you want to exclude everything after it.3. Then the previous boot.
journalctl -b -1 -p err reads the boot before this one. A service that fails on every boot but works when you start it by hand lives here, not in the current boot’s logs. journalctl --list-boots shows which boots the journal still has, with their IDs and dates.4. Then follow it live, narrowly.
journalctl -u nginx -f follows one unit. Follow the unit, not the whole journal — journalctl -f on a busy box is a firehose, and a firehose is not an investigation.Narrow first, read second. Every filter you add before reading is time you do not spend scrolling past messages that were never relevant.
# filtering by field — grep is the last resort, not the first
The journal is structured. Every message carries fields — unit, priority, process, executable, boot ID — and journalctl can filter on any of them directly, without grep.
Combine fields freely:
When grep is the right tool:
Output formats, for when the answer leaves the terminal:
journalctl _SYSTEMD_UNIT=nginx.service is the field form of -u nginx. journalctl _PID=1234 follows one process across restarts of its unit. journalctl _COMM=sshd catches a binary by name, whatever unit launched it.Combine fields freely:
journalctl _SYSTEMD_UNIT=nginx.service -p err --since today is unit, priority, and day in one query, and it runs against the journal’s indexes instead of scanning text. That is why it is fast on a large journal and grep is not.When grep is the right tool:
journalctl -u nginx --grep "upstream timed out" searches message text — and --grep with -i for case-insensitive, or a real pattern when the string alone is not enough. Use it for the message content, not for the unit, the time, or the priority. Those already have flags, and the flags are faster and more precise.Output formats, for when the answer leaves the terminal:
-o cat gives plain messages with no metadata, for pasting into a ticket. -o json gives one JSON object per message, with every field, for feeding into a script or a log pipeline. -o verbose shows all fields of a single message, which is how you find out what a field is actually called before you filter on it.# where the logs actually live — volatile vs persistent, and the reboot that eats them
The default storage mode is
Reading a journal that is not the running one:
The disk will fill, eventually.
RHEL 7 vs 9/10, briefly: the commands above work on all of them. What changed is the ecosystem around them — RHEL 7 still leans on rsyslog for text logs in
auto. With Storage=auto in /etc/systemd/journald.conf, the journal is persistent only if /var/log/journal exists — otherwise it lives in /run/log/journal and vanishes on reboot. On a stock RHEL install that directory usually exists, so logs normally persist. Usually is not always: check with ls -ld /var/log/journal before you assume the previous boot’s logs are still there. If the directory is missing, create it and restart systemd-journald — or set Storage=persistent explicitly, which creates it for you.Reading a journal that is not the running one:
journalctl --directory=/mnt/sysimage/var/log/journal reads the journal of a mounted system — the rescue-mode path, and the reason the initramfs guide’s chroot step matters. journalctl --file=/var/log/journal/<id>/system.journal reads one journal file directly. Both work without booting the system the logs belong to.The disk will fill, eventually.
journalctl --disk-usage shows how much the journal is using. journalctl --vacuum-size=500M and journalctl --vacuum-time=2weeks trim it to a size or an age, oldest first. Set the limits permanently in journald.conf (SystemMaxUse=, MaxRetentionSec=) instead of vacuuming by hand every month — a manual vacuum is a habit, a configured limit is a policy. And run journalctl --verify occasionally: it checks the journal files for corruption, which is the kind of thing you want to learn about on a quiet afternoon, not during an incident.RHEL 7 vs 9/10, briefly: the commands above work on all of them. What changed is the ecosystem around them — RHEL 7 still leans on rsyslog for text logs in
/var/log, while RHEL 9 and 10 lean harder on the journal itself, and the journal’s defaults are more generous. If you are still grepping /var/log/messages on a RHEL 9 box out of habit, check whether the message is in the journal first. It usually is, with more fields attached.If the logs you need are gone after a reboot, that is not a journal failure. That is a storage-mode configuration you never checked. Check it now, on a quiet day.
# journal + rsyslog — what goes where, on purpose
On RHEL, the journal and rsyslog are not competitors. journald collects structured logs from systemd units and the kernel; rsyslog forwards them onward — to text files in
The forwarding habit that matters: forward to a central host with the structured fields intact where your collector supports it, and keep local retention short enough that a runaway service cannot fill the disk before anyone notices. A central log host is also the answer to the previous section’s worst case — if the journal on a dead box is gone, the copy on the log host is not.
One gotcha, seen in the field: a service that logs to its own file in
/var/log, and, where configured, to a central log host. The forwarding is configured in /etc/rsyslog.conf and /etc/rsyslog.d/, and the rule to remember is that rsyslog receives what journald sends it, not the other way around.The forwarding habit that matters: forward to a central host with the structured fields intact where your collector supports it, and keep local retention short enough that a runaway service cannot fill the disk before anyone notices. A central log host is also the answer to the previous section’s worst case — if the journal on a dead box is gone, the copy on the log host is not.
One gotcha, seen in the field: a service that logs to its own file in
/var/log does not automatically appear in the journal, and a service that logs to the journal does not automatically appear in a text file. When a message is missing from one place, check the other before concluding it was never logged.# three investigations, end to end — the patterns, not the commands
A service fails on boot, works by hand.
A service is slow, intermittently.
The disk filled overnight.
journalctl -b -p err, then journalctl -b -1 -u <service> to see the previous boot’s attempt in full. The usual answer is ordering — the service started before its dependency was ready — and the journal shows the timestamps that prove it.A service is slow, intermittently.
journalctl -u <service> --since "2 hours ago" --grep -i "timeout|retry|refused", then the same window with -o json and the fields that tell you which upstream, which peer, which connection. Intermittent slowness is almost always a peer, and the journal is where the peer’s name shows up.The disk filled overnight.
journalctl --disk-usage first — if the journal itself is the problem, vacuum it and set the limits. If it is not, the journal is still the witness: journalctl --since yesterday -p warning shows which service started complaining first, and that service is usually the one writing the files.# the cheat sheet — keep this, not the man page
journalctl -b -p errjournalctl -b -1 (and --list-boots to see what is kept)journalctl -u nginx --since "1 hour ago"journalctl -u nginx -fjournalctl _SYSTEMD_UNIT=nginx.service _PID=1234journalctl -u nginx --grep "upstream timed out"journalctl --disk-usage, then --vacuum-size=500M or --vacuum-time=2weeksjournalctl --directory=/mnt/sysimage/var/log/journaljournalctl -o json (and -o cat for tickets, -o verbose to learn the fields)Commands verified against systemd’s journalctl documentation and RHEL behaviour, November 7, 2026. journald.conf defaults and rsyslog integration vary by RHEL minor release — check /etc/systemd/journald.conf on the box in front of you before assuming a default.
