Exposing Docker Compose Logs for IDS/IPS

Overview

If you've worked with Docker containers in a production environment, you've likely faced a common challenge: how do you monitor container logs for security threats? While Docker Compose makes deploying multi-container applications simple, it doesn't automatically expose logs to the host's filesystem where Intrusion Detection Systems (IDS) and Intrusion Prevention Systems (IPS) like fail2ban can monitor them.

TL;DR

Learn how to expose Docker Compose container logs to your host system for IDS/IPS tools like fail2ban. This post walks you through configuring syslog for Docker services, routing logs to files with rsyslog, configuring fail2ban to use these logs, and setting up log rotation to prevent disk space issues.

What is fail2ban?

fail2ban is a log-parsing application that scans log files for patterns for suspicious activity and failed login attempts (aka Intrusion Detection). When it detects such patterns, it can take actions like blocking the offending IP addresses for a period of time (aka Intrusion Prevention).

"Won't the attacker just come back, or try with a different IP?" you might ask. Yes, probably! The purpose of fail2ban isn't to completely eliminate the threat, but to make it very annoying and frustrating for the attacker. Hopefully they will give up and move on to easier targets. This is just one layer security in our Defense in Depth strategy.

Motivation / Problem Statement

I recently spent a day unraveling this exact problem while setting up my self-hosted Mailu email server. It was frustrating to discover that after deploying my shiny new mail server with Docker Compose, fail2ban couldn't monitor authentication attempts because the logs were not available. By default, Docker runs headless and outputs to STDOUT and STDERR, which are not accessible to the host system.

There are several ways to get this wired up, and I've tried a few. In the end, this was - believe it or not - the simplest and most effective solution I found. It works well for my use case, and I think it would for most use cases.

Since this project is a public-facing mail server, we really need to make sure we are effectively protecting it as much as we reasonably can. The last thing I want is to wake up one morning to find my server has been compromised, or worse, used as a spam relay.

The solution involves:

1. Configuring Docker services to send logs to syslog
2. Setting up rsyslog to route these messages to physical files on the Docker host
3. Pointing fail2ban at these physical files
4. Setting up log rotation to manage disk space

Let's dive in!

Step 1: Configure Docker Compose Services to Use Syslog

The first step is to modify your Docker Compose configuration to use the syslog logging driver. This tells Docker to send container logs to the host's syslog service.

For each service in your docker-compose.yml file, add the following configuration:

services:
  your-service:
    image: your-image
    # ...existing configuration...
    logging:
      driver: syslog
      options:
        syslog-address: "unixgram:///dev/log"
        tag: "your-service-tag"
    volumes:
      - /dev/log:/dev/log:ro

Let's examine this configuration:

• driver: syslog tells Docker to use the syslog logging driver
• syslog-address: "unixgram:///dev/log" specifies the UNIX socket where syslog listens on most Linux systems
• tag: "your-service-tag" adds a prefix to all log messages to identify which service they came from
• The volume mount - /dev/log:/dev/log:ro gives the container read-only access to the host's syslog socket

Naming Convention Tip: Choose a consistent prefix for all service tags to make filtering easier. For example, if you're running Mailu, you might use tags like mailu-smtp, mailu-imap, mailu-web, etc.

Let's look at a real-world example from a Mailu setup:

services:
  front:
    image: ${DOCKER_ORG:-mailu}/${DOCKER_PREFIX:-}nginx:${MAILU_VERSION:-2.0}
    restart: always
    # ...existing configuration...
    logging:
      driver: syslog
      options:
        syslog-address: "unixgram:///dev/log"
        tag: "mailu-front"
    volumes:
      - /dev/log:/dev/log:ro
      # ...other volume mounts...
      
  admin:
    image: ${DOCKER_ORG:-mailu}/${DOCKER_PREFIX:-}admin:${MAILU_VERSION:-2.0}
    restart: always
    # ...existing configuration...
    logging:
      driver: syslog
      options:
        syslog-address: "unixgram:///dev/log"
        tag: "mailu-admin"
    volumes:
      - /dev/log:/dev/log:ro
      # ...other volume mounts...

After adding this configuration to all services, your container logs will be forwarded to the host's syslog system. However, by default, these will just end up in your system's main log files mixed with everything else. Let's fix that next.

Step 2: Create a Syslog Router to Direct Log Messages to Files

Now we need to configure the syslog daemon (rsyslog on most Linux systems) to route these tagged messages to separate files.

Create a new configuration file in the /etc/rsyslog.d/ directory:

sudo nano /etc/rsyslog.d/<YOUR-SERVICE>.conf

Here's a template that dynamically routes logs based on the tag prefix:

###############################################################################
# /etc/rsyslog.d/<YOUR-SERVICE>.conf
#
# Dynamically route any SYSLOG_IDENTIFIER starting with "your-prefix-" into
# /path/to/logs/your-prefix-<service>.log
###############################################################################

# Use the built-in file format (optional, but recommended)
$ActionFileDefaultTemplate RSYSLOG_FileFormat

# Define a template that writes to /path/to/logs/<programname>.log
$template ServiceLogFile,"/path/to/logs/%programname%.log"

# If the program name begins with "your-prefix-", write to the dynamic file
if ($programname startswith 'your-prefix-') then -?ServiceLogFile
& stop

For our Mailu example, we would create /etc/rsyslog.d/mailu.conf that looks something like this, where /opt/mailu/logs/ is the directory where we want to store our logs:

###############################################################################
# /etc/rsyslog.d/mailu.conf
#
# Dynamically route any SYSLOG_IDENTIFIER starting with "mailu-" into
# /opt/mailu/logs/mailu-<service>.log
###############################################################################

# Use the built-in file format (optional, but recommended)
$ActionFileDefaultTemplate RSYSLOG_FileFormat

# Define a template that writes to /opt/mailu/logs/<programname>.log
$template MailuLogFile,"/opt/mailu/logs/%programname%.log"

# If the program name begins with "mailu-", write to the dynamic file
if ($programname startswith 'mailu-') then -?MailuLogFile
& stop

Make sure the target directory exists:

sudo mkdir -p /opt/mailu/logs
sudo chmod 755 /opt/mailu/logs

Then restart rsyslog to apply the changes:

sudo systemctl restart rsyslog.service

Now, restart your Docker Compose services:

cd /path/to/your/compose/directory
sudo docker compose down
sudo docker compose up -d

After these changes, each Docker service's logs will be written to a separate file in your specified directory, named according to the tag you configured.

Check Your Logs: Verify the logs are being created correctly by checking the target directory:

ls -l /opt/mailu/logs/

You should see files named after your service tags, like mailu-front.log, mailu-admin.log, etc.

For example, you might see:

-rw-r----- 1 syslog adm  1103 May 10 14:37 mailu-admin.log
-rw-r----- 1 syslog adm 20521 May 10 14:39 mailu-antispam.log
-rw-r----- 1 syslog adm   363 May 10 14:30 mailu-antivirus.log
-rw-r----- 1 syslog adm 11978 May 10 14:37 mailu-front.log
-rw-r----- 1 syslog adm     0 May 10 14:06 mailu-imap.log
-rw-r----- 1 syslog adm     0 May 10 14:06 mailu-oletools.log
-rw-r----- 1 syslog adm     0 May 10 14:06 mailu-redis.log
-rw-r----- 1 syslog adm     0 May 10 14:06 mailu-resolver.log
-rw-r----- 1 syslog adm     0 May 10 14:06 mailu-smtp.log
-rw-r----- 1 syslog adm     0 May 10 14:06 mailu-tika.log
-rw-r----- 1 syslog adm     0 May 10 14:06 mailu-webmail.log

Step 3: Configure fail2ban to Use the Log Files

Now that our logs are accessible on the filesystem, we can configure fail2ban to monitor them for suspicious activity.

Creating a Custom Filter

First, we need to create a filter that defines what patterns to look for in the logs. For example, let's create a filter for failed login attempts, for Mailu for example:

sudo nano /etc/fail2ban/filter.d/mailu-web.conf

Here's a simple example:

[Definition]
failregex = ^.*Failed login for .* from <HOST>.*$
            ^.*Login attempt for: .* from: <HOST>(?:/\d+)?: failed: badauth:.*$
ignoreregex =

This filter will match log lines that contain both "Failed login" or "Login attempt… failed" and an IP address (which fail2ban captures with the <HOST> tag).

How It Works: The idea is that Fail2Ban watches these log files for specific patterns like "Failed login" or "Login attempt… failed", etc. These "filter" files define those Regular Expressions (regex) patterns.

What's really great is that /etc/fail2ban/filter.d/ already contains a lot of built-in filters for common services like SSH, Apache, Nginx, etc. You can use these as a starting point and modify them to suit your needs. So in the case of Mailu, I made invalid login attempts, looked in the logs to see what kinds of messages showed up, and then created a filter to match those messages. Again, you only have to do that for custom services typically. Look in the filter.d directory first to see if there is already a filter for your program/service.

Creating a Jail Configuration

Next, create a jail configuration file to tell fail2ban which services to monitor. Note that the filter on most of these just uses the built-in filters, but you can also use your custom filter if you want:

sudo nano /etc/fail2ban/jail.d/mailu.conf

Here's an example for our Mailu services:

[DEFAULT]
banaction = iptables-multiport
# IMPORTANT: This will add a DROP rule to the DOCKER-USER chain in iptables
action    = %(banaction)s[chain=DOCKER-USER]

[mailu-smtp]
enabled = true
port = smtp,submission,465
filter = postfix
logpath = /opt/mailu/logs/mailu-smtp.log
maxretry = 5
findtime = 600
bantime = 3600

[mailu-imap]
enabled = true
port = imap,imaps,pop3,pop3s
filter = dovecot
logpath = /opt/mailu/logs/mailu-imap.log
maxretry = 5
findtime = 600
bantime = 3600

[mailu-webmail]
enabled = true
port = http,https
filter = roundcube-auth
logpath = /opt/mailu/logs/mailu-webmail.log
maxretry = 5
findtime = 600
bantime = 3600

[mailu-admin]
enabled = true
filter = docker-service-auth
port = http,https
logpath = /opt/mailu/logs/mailu-admin.log
maxretry = 5
findtime = 600
bantime = 3600

[mailu-front]
enabled = true
filter = docker-service-auth
port = http,https
logpath = /opt/mailu/logs/mailu-front.log
maxretry = 5
findtime = 600
bantime = 3600

Let's understand this configuration:

• enabled = true: Activates this jail
• port: The ports this service uses (fail2ban will ban these ports for offending IPs)
• filter: The name of the filter to use (we created mailu-web.conf above, but you can also use built-in filters)
• logpath: The path to the log file for this service
• maxretry: Number of failures before a ban
• findtime: Time window in seconds to count failures (10 minutes in this case)
• bantime: How long to ban an IP in seconds (1 hour in this case)

The DOCKER-USER Chain: Note that the [DEFAULT] section above has the banaction is set to iptables-multiport. This will add a DROP rule to the DOCKER-USER chain in iptables. This is important because Docker uses its own iptables rules, and we need to make sure our fail2ban rules are applied everywhere.

In other words, if we've found a bad actor, we want to make sure they can't access any of our resources, not just the ones that are using the ports we specified.

Restart fail2ban to apply the changes:

sudo systemctl restart fail2ban

Verifying fail2ban is Working

You can check the status of your jails with this command:

# Status of the jails
sudo fail2ban-client status

# Status of the service
sudo systemctl status fail2ban

WARNING: Fail2ban is kind of all-or-nothing. If a monitored log file is missing for even ONE jail, the service will fail and not start. Meaning, the IDS/IPS protections are not in-place for any jails. So, if a file is missing, you can just create an empty file with the same name, and then restart fail2ban. This will allow the service to start, but it won't be able to monitor that jail until you fix the missing file.

In other words, ALWAYS check the status of the Fail2Ban service after making changes and make sure it didn't fail upon startup.

To see details about a specific jail:

sudo fail2ban-client status mailu-front

Here's a helpful script to check all your jails at once:

#!/bin/bash

JAILS=$(fail2ban-client status | grep "Jail list" | sed -E 's/^[^:]+:[ \t]+//' | sed 's/,//g')
INDEX=1
for JAIL in $JAILS
do
  echo ""
  echo -n "${INDEX}) "
  fail2ban-client status $JAIL
  ((INDEX++))
done
echo ""
echo "$(find /var/log/syslog -type f -mtime -1 -exec grep "UFW BLOCK" {} \; | wc -l) blocks in the past 24 hours."
echo ""

Save this as all-jails.sh, make it executable, and run it:

chmod +x all-jails.sh
sudo ./all-jails.sh

That should output something like this:

1) Status for the jail: dovecot
|- Filter
|  |- Currently failed: 0
|  |- Total failed:     0
|  `- File list:        /opt/mailu/logs/mailu-imap.log
`- Actions
   |- Currently banned: 0
   |- Total banned:     0
   `- Banned IP list:

2) Status for the jail: mailu-admin
|- Filter
|  |- Currently failed: 1
|  |- Total failed:     3
|  `- File list:        /opt/mailu/logs/mailu-admin.log
`- Actions
   |- Currently banned: 0
   |- Total banned:     0
   `- Banned IP list:

3) Status for the jail: mailu-front
|- Filter
|  |- Currently failed: 0
|  |- Total failed:     0
|  `- File list:        /opt/mailu/logs/mailu-front.log
`- Actions
   |- Currently banned: 0
   |- Total banned:     0
   `- Banned IP list:

4) Status for the jail: postfix
|- Filter
|  |- Currently failed: 0
|  |- Total failed:     0
|  `- File list:        /opt/mailu/logs/mailu-smtp.log
`- Actions
   |- Currently banned: 0
   |- Total banned:     0
   `- Banned IP list:

5) Status for the jail: roundcube-auth
|- Filter
|  |- Currently failed: 0
|  |- Total failed:     0
|  `- File list:        /opt/mailu/logs/mailu-webmail.log
`- Actions
   |- Currently banned: 0
   |- Total banned:     0
   `- Banned IP list:

6) Status for the jail: sshd
|- Filter
|  |- Currently failed: 3
|  |- Total failed:     19
|  `- File list:        /var/log/auth.log
`- Actions
   |- Currently banned: 0
   |- Total banned:     1
   `- Banned IP list:

30721 blocks in the past 24 hours.

Step 4: Set Up Log Rotation

The final step is to set up log rotation to prevent your log files from growing indefinitely and filling up your disk.

What is logrotate? logrotate is a system utility that manages the automatic rotation and compression of log files. It helps to keep log files from consuming too much disk space by rotating them out and compressing old logs.

For example, it will finish off access.log and rename it to access.log.1, then create a new access.log file. The old log files can be compressed to save space, and you can configure how many old logs to keep. You can then configure how many copies or for how long you want to keep old copies of the logs. After that expiration period, the old logs will be deleted. This is a built-in mechanism that we can take advantage of to keep our logs from filling up the disk.

Without this, over time the logs will grow and grow, and eventually fill up the disk. This can cause all sorts of problems, including making the system unresponsive or even crashing it.

Create a new log rotation configuration. Again, I'll use Mailu as an example:

sudo nano /etc/logrotate.d/mailu-logs

Here's an example configuration for the logs in /opt/mailu/logs/:

/opt/mailu/logs/mailu-*.log {
    # Rotate the log files every day
    daily

    # Keep 14 rotated logs before deleting the oldest
    rotate 14

    # Compress old log files to save space
    compress

    # Skip compression for the most recent rotated log; compress it on the next run
    delaycompress

    # If a log file is missing, go on without error
    missingok

    # Do not rotate the file if it is empty
    notifempty

    # After copying the current log, truncate it in place so the service can continue writing
    copytruncate

    # After rotation, create a new log file with these permissions and ownership
    create 640 syslog adm
}

Restart the logrotate service:

sudo systemctl restart logrotate.service

You can test the configuration by forcing a log rotation:

sudo logrotate --force /etc/logrotate.d/docker-services

What the test does is it will rotate the logs one time. So, if you had an access.log file before, now you should see an access.log.1 file, and a new access.log file should be created.

Monitor Disk Usage: Even with log rotation in place, it's a good idea to monitor your log directory's size periodically with something like:

du -sh /opt/mailu/logs/

Summary

Setting up proper logging for containerized services is a crucial but often overlooked part of deploying applications with Docker Compose. This additional setup time pays dividends in security and monitoring capabilities because it allows you to use battle-hardened tools like Fail2Ban, Crowdsec, etc.

The approach outlined here solved several problems:

1. It makes container logs accessible to host-based tools like fail2ban
2. It organizes logs from different services into separate files
3. It prevents disk space issues with log rotation
4. It enables proper security monitoring for public-facing services

For a real-world implementation of this approach, check out my Self-Hosted Domain Email with Mailu series, where I apply these techniques to secure a mail server stack. This became so involved that I had to break this part out into its own post.

Further Reading / References

• Docker Logging Documentation
• RSyslog Documentation
• Fail2Ban Documentation
• Logrotate Documentation
• My Self-Hosted Email Guide Part 2: Mailu Installation