How to View and Manage System Logs Using rsyslog in Linux

Linux systems generate a continuous stream of logs from the kernel, services, and user applications. Managing these logs is essential for diagnosing issues, tracking system activity, and maintaining operational visibility.

rsyslog is a widely adopted logging service that collects, processes, stores, and optionally forwards these logs. It supports configurable inputs, filtering rules, and output destinations, making it suitable for both simple setups and complex logging infrastructures.

This article explains how to use rsyslog to view and manage system logs, define custom rules, and forward logs to remote servers for centralized logging.

What is `rsyslog` and Why It Matters?

rsyslog is a logging daemon used on most modern Linux distributions to manage system and application logs. It originated as an extended version of the traditional syslog service, adding more configuration options and better performance. Unlike basic syslog tools, rsyslog supports multiple input sources, advanced filtering, rule-based message routing, and remote logging capabilities. These features allow administrators to collect and organize logs efficiently from different parts of the system or even from multiple servers.

This makes rsyslog useful not only for local log management but also for building centralized logging infrastructures where logs from many systems are sent to a single log server for analysis and archiving.

Key Features of `rsyslog`

Modular architecture: Uses a plugin based system for inputs, filters, and outputs, allowing flexible extension.
Flexible configuration: Supports rules based on log priority, facility, program name, or message content.
Remote log forwarding: Can send logs over TCP or UDP to remote servers for centralized log management.
Structured data support: Handles logs in formats like JSON and RFC 5424 for easier parsing and analysis.
High performance: Designed with multi-threading to process large volumes of logs efficiently.
Custom logging rules: Allows precise control over which messages are logged, where, and how they are stored.
Multiple input sources: Collects logs from the kernel, system services, applications, and custom scripts.
Structured Output: Unlike traditional text logs, rsyslog offers the ability to format log output, making it more structured and easier to analyze.

How `rsyslog` Works: The Input, Filter, and Output Model

rsyslog processes log messages through a structured pipeline composed of three core stages: Input, Filter, and Output. This model allows you to control how logs are collected, processed, and stored or forwarded.

Input: rsyslog collects logs from various sources such as system sockets (imuxsock), kernel messages (imklog), plain-text files (imfile), or network protocols like TCP/UDP (imtcp, imudp). Each input module defines how and where logs enter the system.
Filter: After logs are received, rsyslog evaluates them using filters based on facility, severity, or custom expression rules. This stage determines which logs should be processed further and how they should be handled.
Output: Filtered logs are routed to output destinations such as local log files, SQL databases, remote syslog servers, or message queues. Each destination is managed by a specific module and can be configured with custom formatting and delivery rules.

This flexible model allows you to tailor your logging setup to fit your needs, whether you're managing a single server or a large network of systems.

Common Log File Locations in Linux

On Linux systems, log files are typically stored under the /var/log/ directory. These files record system events, application activity, kernel operations, and security information. While file names and purposes can vary slightly across distributions, the following are among the most commonly used log files:

/var/log/syslog: Captures general system activity, including messages from system services and applications. Commonly used in Debian and Ubuntu-based systems for broad troubleshooting.
/var/log/messages: A general-purpose log file used on Red Hat-based systems (like CentOS and Fedora). It logs system messages that are not related to authentication or kernel.
/var/log/auth.log: Records authentication attempts, including user logins, sudo actions, and SSH access. This file is essential for auditing and security monitoring.
/var/log/kern.log: Contains kernel-generated messages that aren't handled by other facilities. Useful for diagnosing hardware issues or kernel module problems.
/var/log/dmesg: Stores the kernel ring buffer from the most recent boot, showing hardware detection and initialization messages. Often used when diagnosing device or boot-related issues.
/var/log/cron.log: Logs the execution of scheduled cron jobs, which can be useful when debugging missed or failed automated tasks.

These logs are often the first point of reference when diagnosing system problems, investigating user activity, or monitoring server health.

How to View Logs with Basic Linux Commands

Before configuring rsyslog, it's useful to understand how to inspect and analyze log files using common Linux utilities. These tools let you check the contents of logs for troubleshooting and monitoring.

cat: Outputs the entire contents of a file to the terminal. Best used for small log files.
console
```
$ cat <LOGFILE_PATH>
```
more: Displays the contents of a file one screen at a time. Use Space to scroll forward, Q to quit.
console
```
$ more <LOGFILE_PATH>
```
less: Opens the file in a scrollable viewer, allowing you to navigate forward and backward through large log files.
console
```
$ less <LOGFILE_PATH>
```
tail: Displays the last 10 lines of a file by default. Useful for checking recent log entries.
console
```
$ tail <LOGFILE_PATH>
```
tail -f: Continuously streams new log entries as they are written, ideal for real-time monitoring.
console
```
$ tail -f <LOGFILE_PATH>
```
zcat, zless, zgrep – Use these to read or search compressed .gz log files created by log rotation. They allow access without decompressing the file first.
console
```
$ zless <GZ_LOGFILE_PATH>
```

These tools form the foundation for interacting with log files and are especially useful for troubleshooting system and application behavior.

Filter Logs Using `grep`, `awk`, and `sed`

Linux provides great text processing tools that make it possible to search, extract, and analyze logs. These utilities are essential for filtering large log files and identifying relevant events.

grep: Searches for specific keywords or regular expressions within log files. Use it to locate entries containing errors, warnings, or other patterns.
console
```
$ grep "error" /var/log/syslog
$ grep -i "failed" /var/log/auth.log
```
awk: Processes log entries by splitting them into fields. It's useful for extracting timestamps, usernames, or specific columns from structured logs.
console
```
$ awk '{print $1, $2, $5}' /var/log/syslog
```
sed: A stream editor that can filter or transform lines of text. You can use it to extract a block of logs between two keywords, substitute text, or remove unwanted entries.
console
```
$ sed -n '/Start/,/End/p' /var/log/syslog
$ sed 's/error/ERROR/g' /var/log/syslog
```

By combining these tools, you can create effective pipelines to sift through logs and uncover the specific information you need, whether it’s a failed login try or a particular service error.

Configure `rsyslog` Using `/etc/rsyslog.conf`

The behavior of rsyslog is primarily controlled through its main configuration file located at /etc/rsyslog.conf. This file defines how incoming log messages are processed, filtered, and directed based on their source (facility), importance (severity level), and desired destination (action).

Basic Rule Format

The general format for a rule in rsyslog.conf is:

FACILITY.LEVEL    ACTION

FACILITY: The source of the log message (for example, auth, cron, daemon, kern, mail, user).
LEVEL: The severity of the log message (for example, debug, info, notice, warning, err, crit, alert, emerg).
ACTION: The destination for the log (for example, a file, remote server, or program).

Common Configuration Examples

Send Auth Logs to a Separate File.
ini
```
auth.*    /var/log/auth.log
```
Logs from the auth facility (for example, SSH, sudo) go to /var/log/auth.log.
Store Kernel Logs in a Separate File.
ini
```
kern.*    /var/log/kern.log
```
All kernel messages are redirected to /var/log/kern.log.
Log Only Critical Mail Errors.
ini
```
mail.crit    /var/log/mail_critical.log
```
Only critical-level messages from the mail facility are logged.
Log Everything Except Mail and Auth to a Common File.
ini
```
*.info;mail.none;authpriv.none    /var/log/general.log
```
Sends all info-level or higher messages to /var/log/general.log, except those from mail and authpriv.
Discard Unwanted Logs.
ini
```
*.debug    ~
```
The tilde (~) discards all debug-level logs to reduce noise.
Log User Messages to Console.
ini
```
user.*    /dev/console
```

This rule specifies that logs of level info or higher, from all facilities except mail, authpriv, and cron, should be written to /var/log/messages.

Add Custom Logging Rules

rsyslog provides flexibility to define custom rules for logging events based on specific programs or services. This is useful when you want to isolate logs for easier analysis, monitor critical activity like SSH access, or keep application logs organized.

Log SSH Activity

Let’s say you want to keep track of all authentication-related events, specifically SSH logins. To do this, you can add a custom rule.

Create a new configuration file in the /etc/rsyslog.d/ directory.
console
```
$ sudo nano /etc/rsyslog.d/10-ssh.conf
```
Add the following rule to capture all messages from the authpriv facility.
ini
```
authpriv.*    /var/log/ssh.log
```
Restart the rsyslog service to apply the new configuration.
console
```
$ systemctl restart rsyslog.service
```

SSH-related log entries will now be written to /var/log/ssh.log, making it more convenient to monitor authentication activity independently of other system logs.

Log Activity for Custom Application

If you're running a custom application that logs messages via syslog, you can configure rsyslog to capture those logs. This helps keep application logs organized and isolated from system messages.

Create a new rsyslog configuration file in /etc/rsyslog.d/.
console
```
$ sudo nano /etc/rsyslog.d/20-myapp.conf
```
Add the following rule to match logs from your application.
ini
```
if ($programname == 'myapp') then /var/log/myapp.log
& stop
```
Restart the rsyslog service to apply the changes.
console
```
$ systemctl restart rsyslog.service
```

Now, all syslog messages from myapp will be written to /var/log/myapp.log, keeping your application logs organized and separate from system logs.

Forward Logs to a Remote Server

In larger environments, centralizing logs from multiple systems on a single server helps streamline monitoring and troubleshooting. rsyslog can forward logs from a client to a remote server over TCP or UDP.

Configure the Client and Server

Follow these steps to set up log forwarding.

Create a configuration file on the client to forward all logs to the remote server.
console
```
$ sudo nano /etc/rsyslog.d/60-remote.conf
```
Add the forwarding rule to the file.
ini
```
*.* @@192.2.0.3:514
```
In the above configuration:
- Replace 192.2.0.3 with your remote rsyslog server’s IP address.
- Use a single @ for UDP, or double @@ for TCP.
On the remote rsyslog server, enable TCP log reception by editing the main config file.
console
```
$ sudo nano /etc/rsyslog.conf
```
Uncomment or add the following lines to enable TCP input.
ini
```
module(load="imtcp")
input(type="imtcp" port="514")
```
(Optional) To organize incoming logs by hostname and program name on the server. Add the following to the remote server’s rsyslog configuration.
ini
```
$template RemoteLogs,"/var/log/remote/%HOSTNAME%/%PROGRAMNAME%.log"
*.* ?RemoteLogs
```
This will store logs in structured folders like /var/log/remote/client1/myapp.log.
Restart rsyslog on both systems to apply the changes.
console
```
$ sudo systemctl restart rsyslog
```

After setup, the client system will send logs to the remote server, which will organize and store them based on the defined templates.

Test the Configuration

You can verify your rsyslog setup by sending a test log message.

                            console
                            
$ logger -p local0.info "Test message from rsyslog"

Then, check the appropriate log file (for example, /var/log/syslog or your custom-defined log path) to confirm that the message was recorded.

Best Practices for Using `rsyslog`

To ensure your system logs are organized, secure, and straightforward to manage, follow these recommended practices:

Use separate files: For better management, store logs from different services or applications in separate files.
Secure your logs: Protect log files with proper file permissions to prevent unauthorized access.
Use log rotation: Set up log rotation to avoid log files growing too large.
Centralized logging: Use remote logging to collect logs from multiple systems in one place.
Monitor your logs: Check log files often to identify and address issues promptly.

Conclusion

In this article, you have learned how to configure and manage system logging using rsyslog on Linux. You explored the structure of the rsyslog configuration file, created custom logging rules for services like SSH and custom applications, and configured remote logging to centralize log data across multiple systems. You tested your configuration with the logger command and learned best practices for organizing, securing, and monitoring logs. These steps helped you build a reliable logging system to better manage your Linux environment.

Tags:

Comments

No comments yet.

How to View and Manage System Logs Using rsyslog in Linux