How to Build Real-Time Log Streams with Rsyslog and FIFOs

Beyond Static Logs: The Need for Streaming

In a standard Linux environment, system logging is usually a passive affair. Daemons like rsyslog or journald write events to flat files in /var/log, and administrators occasionally query them using grep, awk, or tail -f. While this is sufficient for post-mortem analysis, it falls short when you need real-time reactions—such as triggering an immediate firewall block during a brute-force attack or updating a live dashboard without the overhead of a heavy log aggregator like ElasticSearch.

As engineers at SiberFX, we often encounter scenarios where lightweight, real-time log processing is required on edge servers where resources are too tight for a full Java-based logging stack. The solution? Named Pipes (FIFOs) integrated directly into the rsyslog pipeline.

Understanding the FIFO (First-In, First-Out)

A Named Pipe, or FIFO, is a special type of file on Linux that acts as a temporary buffer between two processes. Unlike a standard file, data written to a FIFO is not stored on the disk; it resides in kernel memory. One process writes to it, and another reads from it in a synchronous fashion. If the reader isn't ready, the writer blocks (or vice versa), ensuring a controlled flow of data.

Creating the Pipe

Before configuring our logging daemon, we must create the pipe in a secure location. We generally recommend /run or a dedicated subdirectory in /var/lib to ensure it persists correctly across service restarts but doesn't clutter the root filesystem.

sudo mkfifo /var/log/log_stream.pipe
sudo chown root:adm /var/log/log_stream.pipe
sudo chmod 660 /var/log/log_stream.pipe

Configuring Rsyslog to Stream Data

Rsyslog is incredibly modular. It uses the ompipe (Output Module for Pipes) to send data to a FIFO. To avoid modifying the main configuration file, we create a drop-in script in /etc/rsyslog.d/. Suppose we want to stream all SSH authentication attempts to our pipe for real-time analysis.

Example: Filtered Pipe Output

Create a file named /etc/rsyslog.d/30-auth-stream.conf:

# Load the pipe module if not already loaded
module(load="ompipe")

# Filter for sshd events and send to our pipe
if $programname == 'sshd' then {
    action(type="ompipe" pipe="/var/log/log_stream.pipe")
}

After saving, restart the service to apply changes:

sudo systemctl restart rsyslog

Building the Consumer Script

Now that rsyslog is pumping data into the pipe, we need a consumer. A common mistake for mid-level developers is using a simple cat in a loop. However, pipes can be tricky: if the consumer closes the file descriptor, the writer (rsyslog) might encounter errors. We need a robust script that keeps the pipe open.

The Bash Consumer Pattern

This script reads from the pipe line-by-line. Notice the use of a file descriptor to keep the pipe open and prevent the loop from terminating when the pipe is momentarily empty.

#!/bin/bash

PIPE=/var/log/log_stream.pipe

# Open the pipe for reading using file descriptor 3
exec 3<"$PIPE"

while true; do
    if read line <&3; then
        # Process the log line
        if [[ "$line" == *"Failed password"* ]]; then
            IP_ADDR=$(echo "$line" | grep -oE '[0-9]+\.[0-9]+\.[0-9]+\.[0-9]+' | head -n1)
            echo "[ALERT] Brute force attempt detected from $IP_ADDR"
            # Here you could call an API or update an IPset
        fi
    else
        # Sleep briefly if no data to prevent CPU spiking
        sleep 0.1
    fi
done

Operational Reliability with Systemd

Running a bash script in a screen session is not production-grade. To ensure our log processor is always running, we should wrap it in a Systemd unit file. This provides automatic restarts and logging for the processor itself.

Sample Unit File: /etc/systemd/system/log-processor.service

[Unit]
Description=Real-time Log Processor
After=rsyslog.service

[Service]
Type=simple
ExecStart=/usr/local/bin/log-processor.sh
Restart=always
RestartSec=5
User=root

[Install]
WantedBy=multi-user.target

Performance and Buffer Management

One critical aspect of using pipes is the kernel buffer size. By default, a Linux pipe buffer is usually 64KB. In high-traffic environments, if your consumer script is slow (e.g., performing a slow DNS lookup or a blocking API call), the pipe will fill up. When the pipe is full, rsyslog may block, potentially slowing down the entire system or dropping logs depending on your rsyslog queue configuration.

Optimization Strategies:

• Increase Pipe Capacity: You can increase the pipe buffer size using the fcntl system call in languages like Python or C, though it's rarely necessary for standard text logs.
• Asynchronous Processing: Ensure your consumer script handles heavy lifting (like database writes) in the background or via a secondary internal queue.
• Rsyslog Queuing: Configure rsyslog to use a disk-assisted memory queue so that if the pipe blocks, logs are cached safely.

Security Considerations

Since the data in the pipe can contain sensitive information (IP addresses, usernames, system paths), file permissions are paramount. Never leave a pipe with 777 permissions. The consumer script should run with the minimum privileges necessary. If your script doesn't need to modify firewall rules, run it as a non-privileged user that belongs to the adm group.

Conclusion

Architecting a real-time log pipeline using Rsyslog and Named Pipes is a powerful, low-overhead strategy for system administrators. It bridges the gap between legacy logging and modern reactive infrastructure without the complexity of external message brokers. By leveraging the native strengths of the Linux kernel and the modularity of Rsyslog, you can build highly responsive systems that act on events the millisecond they occur.