Infrastructure & Operations › Linux & Servers

Reading System Logs

journalctl, /var/log, and finding out what went wrong.

Also known as: system logs, journalctl, syslog

When something breaks on a server, the evidence is usually in a log. Kernels, services and applications all write messages, and a mid-level engineer is expected to find the right one quickly.

On most modern distributions, systemd collects logs in the journal, read with journalctl:

journalctl -u nginx         # one service
journalctl -f               # follow live, like tail -f
journalctl -b               # since the last boot
journalctl -p err -b        # only errors since boot
journalctl --since "1 hour ago"

Older or extra logs live as plain files under /var/log — syslog or messages, auth.log, kern.log, and per-service folders like /var/log/nginx/. These are rotated by logrotate so they don’t fill the disk; rotated files end in .1, .2 or .gz.

The classic mistake is grepping /var/log/syslog and finding nothing, because the distro logs only to the journal — or the reverse. Check both, and remember container output often goes elsewhere; see container logs.

Two more traps:

• Volume. A misbehaving service can write gigabytes and fill the disk (see disk usage). Set rotation and size limits.
• Missing context. Plain text lines are hard to search at scale. Structured logging (events with named fields) and a log aggregation system turn “find every request for user 42” into a query instead of a grep.

Read logs by timestamp and by severity (log levels), and correlate services around the same moment. The first error is usually the cause; the rest are fallout.