Back to Blog
LinuxUbuntuLogsSysAdminTutorial

System logs with journalctl: query, persist and diagnose

Complete tutorial on journalctl in Ubuntu. Learn journalctl -u/-f, time ranges, persistent storage, priority filters.

Daniel Urbano
November 18, 2025
7 min read
System logs with journalctl: query, persist and diagnose

System Logs with journalctl: Query, Persist and Diagnose

Introduction

In the realm of system administration, managing and analyzing system logs is crucial for troubleshooting and maintaining system health. In Linux, systemd has redefined logging with its logging service called journald, which collects and manages logs from various sources. The journalctl command is the primary tool for querying and displaying logs from the journal, allowing system administrators to efficiently diagnose issues and monitor system performance.

Understanding how to use journalctl effectively is vital because it provides insights into system events, service statuses, and application-level logs. Use cases include troubleshooting boot issues, monitoring service failures, and analyzing security-related events. With its advanced filtering capabilities, journalctl empowers users to sift through potentially vast amounts of log data, making it a cornerstone tool for any Linux administrator.

Prerequisites

  • Ubuntu Version: This guide is applicable for Ubuntu 16.04 and later versions, as they come with systemd and journalctl pre-installed.
  • Required Packages: Ensure your system uses systemd (which is default in modern Ubuntu versions).
  • Permissions: You may need sudo privileges to access certain logs, especially those pertaining to other users or system processes.
  • Risks: Improper use of log queries or cleaning logs can lead to loss of critical information. Always ensure you understand the commands before executing them.

Core Concepts

Terminology

  • Journal: A binary log format used by systemd to store log entries.
  • journalctl: The command-line utility to query the journal logs.
  • Persistent Storage: Configuring systemd to store logs permanently instead of only in memory.

Architecture

  • The journal stores logs in a binary format, which allows for more efficient querying than plain text logs. Logs can originate from various sources, including the kernel, system services, and user applications.

When to Use

You would typically use journalctl in scenarios such as:

  • Diagnosing service failures.
  • Analyzing system boot and shutdown logs.
  • Monitoring application logs for debugging.

Limits

While journalctl is powerful, users should note that logs can grow in size and consume disk space if not managed properly. Additionally, filtering through extensive logs may require familiarity with its command-line options.

Syntax/Commands

Command Description
journalctl -u <unit> Show logs for a specific systemd unit.
journalctl -f Follow logs in real-time (similar to tail -f).
journalctl --since <time> Show logs since a specific time.
journalctl --until <time> Show logs until a specific time.
journalctl -p <priority> Filter logs by priority level (e.g., err, warning).
journalctl --no-pager Output all logs without paging.
journalctl --disk-usage Show disk space used by the journal.
journalctl --vacuum-size=<size> Reduce journal size to specified size.

Practical Examples

1. Display All Logs

# Display all logs stored in the journal.
journalctl

2. Real-Time Log Monitoring

# Follow logs in real-time, similar to tail -f.
journalctl -f

3. Filter Logs by Systemd Unit

# Show logs for the nginx service.
journalctl -u nginx.service

4. Time-Based Log Queries

# Show logs since yesterday.
journalctl --since "yesterday"

5. Show Logs Until a Specific Time

# Show logs until a specific date and time.
journalctl --until "2023-10-01 12:00:00"

6. Filter Logs by Priority

# Display only error messages.
journalctl -p err

7. Check Disk Usage of the Journal

# Show the current disk usage of the journal.
journalctl --disk-usage

8. Vacuum Journal Logs by Size

# Reduce the journal size to 100MB.
journalctl --vacuum-size=100M

Real-World Scenarios

Scenario 1: Troubleshooting a Failed Service

When a service fails, you can quickly find relevant logs to diagnose the issue:

# Check logs for a hypothetical web service.
journalctl -u web_service.service --since "1 hour ago"

Context: This command helps identify issues that occurred within the last hour, allowing you to pinpoint the root cause of the failure.

Scenario 2: Analyzing Boot Issues

If your system encounters boot problems, you can examine logs from the previous boot:

# Show logs from the previous boot.
journalctl -b -1

Context: This command facilitates diagnosis by allowing you to view logs related to the system’s last boot, which can reveal critical errors or warnings.

Scenario 3: Monitoring System Events

To keep track of ongoing system events, use the following command:

# Monitor system logs related to user logins.
journalctl -f _UID=1000

Context: This command displays real-time logs for a specific user (with UID 1000), helping monitor user activity and system interactions.

Best Practices

  1. Regularly Monitor Logs: Make it a habit to regularly check logs to catch issues early.
  2. Use Persistent Storage: Configure journald for persistent storage to ensure logs survive reboots.
  3. Limit Log Size: Use --vacuum-size or time-based limits to manage disk usage effectively.
  4. Employ Filters: Leverage filtering options to reduce noise and focus on relevant log entries.
  5. Secure Logs: Restrict access to log files to prevent unauthorized access to sensitive information.

Common Errors

1. Error Message: "Failed to open journal files: Permission denied"

Cause: Insufficient permissions to read the logs.

Fix: Use sudo to access the logs.

sudo journalctl

2. Error Message: "No journal files found"

Cause: The journal is not persistent, and logs have been cleared after a reboot.

Fix: Enable persistent logging by creating /var/log/journal directory.

sudo mkdir -p /var/log/journal
sudo systemctl restart systemd-journald

3. Error Message: "No matching messages found"

Cause: The filter criteria do not match any log entries.

Fix: Check the criteria and modify the command accordingly.

4. Error Message: "Invalid argument"

Cause: An invalid option was provided to the command.

Fix: Verify the command syntax and available options.

Related Commands

Command Description
dmesg Show kernel-related logs.
tail Display the last part of files (logs).
grep Search for specific terms in logs.
logger Add messages to the system log.

Automation Script

Here’s a simple script that checks journal logs, filters for errors, and sends an email if any are found:

#!/bin/bash

# Define log file and email recipient
LOG_FILE="/var/log/error_logs.log"
EMAIL="admin@example.com"

# Check for error logs in the last 24 hours
ERROR_LOGS=$(journalctl -p err --since "24 hours ago")

# If errors are found, log them and send an email
if [[ ! -z "$ERROR_LOGS" ]]; then
    echo "$ERROR_LOGS" >> "$LOG_FILE"
    echo "Errors found in journal logs. Check $LOG_FILE for details." | mail -s "Error Logs Alert" "$EMAIL"
fi

Notes:

  • Make sure to give execute permissions to the script using chmod +x scriptname.sh.
  • Set up a cron job to run the script periodically.

Conclusion

In this tutorial, we explored journalctl, the powerful command-line tool for querying system logs in Linux. We discussed its core concepts, syntax, practical usage, and best practices for effective log management. By understanding how to leverage journalctl, you can enhance your troubleshooting capabilities, maintain system health, and respond swiftly to issues.

As a next step, consider integrating journalctl log monitoring into your daily routine, explore advanced filtering options, or automate logging tasks using scripts.

References

Happy logging! 🚀