Docker Compose Mail Server with Rspamd and ClamAV 2026

Why Build Your Own Mail Server with Rspamd and ClamAV?

You’re not just checking email — you’re trusting a third party with your messages, your calendar, even your passwords. Most hosted services scan your content, not to protect you, but to serve ads. What if you could own that data entirely?

With Docker Compose, you build a mail server stack in one file — Rspamd for intelligent spam filtering, ClamAV for real-time malware scans — all under your control, portable, and reproducible.

Key takeaways

• Self-hosting with Docker Compose gives you full, auditable control over your email data.
• Rspamd uses machine learning and collaborative filtering to reduce spam without over-blocking legitimate mail.
• ClamAV scans every attachment in real time, blocking malware before it reaches your inbox.

What Does a Secure, Self-Hosted Mail Server Actually Need?

You need a solid foundation: a trusted MTA like Postfix to handle mail routing, a reliable MDA like Dovecot for storing messages, spam filtering with Rspamd, malware scanning via ClamAV, TLS encryption using Let’s Encrypt certificates, and correctly configured DNS records—MX, SPF, DKIM, and DMARC—for deliverability and sender reputation. That’s the baseline. Anything less and you’re leaving gaps.

Core Components You Can’t Skip

• Mail Transfer Agent (MTA): Use Postfix or Exim. It’s responsible for sending and receiving mail on the internet. Without it, your server can’t communicate with others. RFC 5321 defines SMTP, the standard Postfix implements.
• Mail Delivery Agent (MDA): Dovecot stores messages in IMAP or POP3 format. It manages inbox access, user authentication, and mailbox integrity. Without it, you can’t read or organize email locally.
• Spam filtering: Rspamd uses machine learning and reputation checks to filter junk. It’s fast, modular, and integrates directly with Postfix. It’s not perfect, but it’s among the most reliable open-source options.
• Malware detection: ClamAV scans attachments in real time. Enable it in your MTA and MDA stack to block infected files before they reach users. Real-world threats evolve—keep ClamAV’s malware database updated.
• TLS encryption: Use Let’s Encrypt certificates (via Certbot or Traefik) to encrypt mail in transit. Always enforce TLS 1.2+ and prefer encrypted connections via STARTTLS. This isn’t optional—it’s the baseline for privacy.

DNS Records: The Foundation of Inbox Trust

Even with good tech, poor DNS setups get you flagged as spam. These records are not afterthoughts—they’re gatekeepers.

Use MXToolbox to validate your DNS setup. It’s free, real-time, and trusted by administrators worldwide.

Let’s be honest: self-hosting email isn’t about perfection. It’s about control. If you want privacy and sovereignty—not just security—start here. You can automate much of this with docker-compose, but don’t skip the fundamentals. For a ready-to-use, secure alternative, learn how Unifiedesk offers end-to-end encrypted email, calendar, and docs—with full self-hosting support and domain control at self-hosting.

Can You Use Rspamd and ClamAV in a Docker Setup? Yes — Here’s How

You absolutely can run Rspamd and ClamAV in a Docker setup. Both tools are designed for modular, container-friendly operation: Rspamd offers a built-in web UI for real-time monitoring and configuration, while ClamAV uses a dedicated scanning daemon and freshclam for updates. Together, they integrate seamlessly with Postfix via content filters, running in separate containers with shared volumes—no root access needed, and they scale reliably in production environments.

How Rspamd and ClamAV Fit Into a Containerized Mail Stack

With Rspamd, you get a lightweight, high-performance spam filter that runs in its own container and exposes a responsive web interface on port 11334. That interface lets you inspect scores, adjust rules, and review logs—all without touching the server’s filesystem. It also supports real-time Bayesian filtering and integrates with external DNSBLs, giving you fine-grained control over what gets through. The design makes it ideal for self-hosted setups where visibility into spam decisions matters.

ClamAV runs as a daemon, scanning attachments and inline content as mail flows through Postfix. The freshclam process, running in a separate container or alongside it, keeps virus definitions updated automatically—critical for staying ahead of new threats. Because ClamAV is designed for Unix-like systems, it works cleanly in container environments with proper volume mounting for scan queues and update logs.

Integrating with Postfix and Scaling in Production

Both services integrate with Postfix using content filter hooks. You define them in the main.cf file, pointing to the Rspamd and ClamAV container IP addresses and listen ports. They process messages before delivery, blocking clearly spammy or infected mail without burdening the primary MTA. This separation of concerns means you can independently scale or restart either service without downtime.

For production use, running Rspamd and ClamAV in isolated containers—each with its own volume mounts for configuration and state—improves resilience. You can even shard scanning load across multiple ClamAV instances with a load balancer, or adjust Rspamd's threading based on your CPU resources. This architecture is proven in enterprise email systems and documented in RFC 5322, which defines the core email format these tools process.

You can deploy this entire stack in a self-hosted Unifiedesk instance, where end-to-end encryption, per-account keys, and granular access controls give you full ownership of your data—no third-party exposure. No matter your use case, whether securing your team’s inbox or running a private email server, this setup is both reliable and transparent.

Set Up Your Docker Compose File with Rspamd and ClamAV

You create a docker-compose.yml file defining Postfix, Dovecot, Rspamd, ClamAV, and Nginx, with dependencies ensuring ClamAV starts before Postfix, and map ports 25, 587, 143, 993, and 443 for SMTP, submission, IMAP, IMAPS, and HTTPS. Mount configuration directories for each service to persist settings and maintain control over your email stack.

Define the Services and Dependencies

1. Create a file named docker-compose.yml in your project directory. This is your single source of truth for the entire email stack.
2. Define five services: postfix, dovecot, rspamd, clamav, and nginx. Each handles a specific aspect: mail transfer, storage, spam filtering, virus scanning, and HTTPS termination.
3. Add depends_on to ensure ClamAV starts before Postfix and Rspamd. This prevents mail delivery failures during startup, as Postfix relies on ClamAV’s scan daemon and Rspamd needs access to the message queue early.
4. Use restart: unless-stopped for resilience. If a container crashes, Docker will restart it automatically, maintaining uptime.

Map Ports and Mount Configurations

5. Expose ports 25 (SMTP), 587 (submission), 143 (IMAP), 993 (IMAPS), and 443 (HTTPS) on the host. These are standard for email clients and web access.
6. Mount configuration directories: /etc/postfix, /etc/dovecot, /var/lib/rspamd, and /var/lib/clamav from the host to the containers. This preserves settings across container restarts and enables easy management.
7. Use volumes to link local directories like ./config/postfix:/etc/postfix. This keeps your configurations version-controlled and editable via your editor of choice.
8. For HTTPS, use Nginx with a reverse proxy. You can generate certificates via Let’s Encrypt using Let’s Encrypt’s official guide or a tool like SSL.com’s documentation to ensure traffic is encrypted in transit.
9. Test connectivity after starting with docker-compose up -d. Check logs via docker-compose logs -f to catch early issues like failed DNS lookups or missing certificates.

Once deployed, your system will filter spam with Rspamd, scan for malware via ClamAV, and deliver encrypted mail through proper TLS. You gain full visibility and control—no data sold, no backdoors. For a managed alternative with the same privacy principles, consider a self-hosted setup like Unifiedesk’s on-premise suite, which includes full end-to-end encryption, JMAP, and native AI assistant support—all built from open-source foundations with no telemetry.

Configure Rspamd Container — Spam Detection Made Simple

You can set up Rspamd in your Docker compose mail server with spam detection, sender identity verification, and custom filtering by enabling the web UI, activating SPF/DKIM checks, allowing manual review of flagged messages, mounting local rules, and enabling Bayesian learning. Once configured, you’ll have a smart, self-improving spam filter without relying on third-party services.

Enable Core Features for Real-World Spam Protection

• Set RSPAMD_ENABLE_WEBUI=1 to expose the Rspamd web interface, allowing you to monitor scores, review flagged messages, and adjust rules in real time — essential for tuning without restarting services.
• Use SPF_ENABLED=1 and DKIM_ENABLED=1 to validate sender identity, reducing spoofing and improving inbox placement — a common industry practice backed by RFC 7208 and RFC 6376.
• Set CONTENT_SCORE_LIMIT=0 to disable automatic rejection of high-scoring messages, letting you manually review flagged emails before acting. This is critical during onboarding and tuning.

Personalize & Improve Over Time

• Mount custom rules into /etc/rspamd/local.d from your local config directory to override defaults or add domain-specific filters — this way, your server learns your unique threat profile.
• Enable the bayes plugin so Rspamd learns from your feedback over time, improving accuracy without relying on cloud-based training or anonymized data.

Let’s not overcomplicate this: Rspamd is designed for scale and accuracy. By exposing the UI, validating sender identity, and allowing manual review, you keep control — not a black box. Over time, the bayes filter adapts, meaning fewer false positives. If you're managing a team or building for privacy, consider how your email stack handles user feedback: self-hosting Unifiedesk gives you the same control, with end-to-end encryption across mail, files, and meetings — all under your domain, in your control.

Enable ClamAV Scanning — Protect Against Malware in Attachments

You can protect your Docker-based mail server from malware by running ClamAV with fresh signatures, streaming inbound mail through a tcp socket, and using a persistent volume to store the database. Set ClamAV to update every 6 hours via freshclam, and route all incoming email through it using Postfix’s content_filter. This stops malicious attachments before they reach users.

Set up ClamAV with Fresh Updates and Persistence

1. Use the official clamav/clamd image, which bundles both the daemon and freshclam for real-time signature updates.
2. Set the environment variable CLAMAV_FRESHCLAM_PERIOD=6h to ensure signatures are updated every six hours — this balances timely protection with network efficiency.
3. Mount /var/lib/clamav as a named volume to persist the virus database across container restarts and to avoid re-downloading from scratch.
4. Enable freshclam with CLAMAV_FRESHCLAM_ENABLED=true to keep updates automated, and use CLAMAV_FRESHCLAM_LOG to monitor sync health.

Integrate ClamAV with Postfix

5. Configure Postfix to use ClamAV by setting content_filter = clamav:local in /etc/postfix/main.cf.
6. Define the filter in /etc/postfix/master.cf using clamav unix - - n - 10 smtp with o=clamav as the transport.
7. Use clamd over TCP (not Unix socket) by specifying 127.0.0.1:3310 in the filter configuration — TCP ensures consistent access in container setups.
8. Set max-threads=10 in the ClamAV daemon to handle multiple concurrent scans without starving other services.
9. Test the integration with a malware simulation: send an email with a known test file from MalwareTips to verify detection.

ClamAV scans only attachments and is not CPU-heavy when scaled right. By running it in a dedicated service within your docker-compose.yml and enabling regular updates, you keep your mail server's defenses current. For a full-featured, self-hosted mailbox suite with built-in antivirus and threat protection, explore how Unifiedesk handles security by default: secure email and workspace.

Integrate DNS and Security Records for Deliverability

You must set up MX, SPF, DKIM, DMARC, and MTA-STS records to ensure your Docker Compose mail server delivers securely and reliably. Without them, mail providers will flag your server as untrustworthy, likely sending your messages to spam or rejecting them outright. This is how major platforms like Gmail and Outlook verify domain authenticity — it’s not optional.

Configure DNS Records for Authentication

• Point an MX record at your server's public IP address (e.g., mail.yourdomain.com IN MX 10 203.0.113.42). This tells other servers where to deliver incoming mail.
• Add an SPF record using v=spf1 include:_spf.yourdomain.com ~all. This authorizes specific servers to send mail on your domain's behalf. You can validate it with MXToolbox’s SPF checker.
• Generate DKIM keys with opendkim-genkey and publish the public key as a TXT record in DNS under a selector (e.g., mail._domainkey.yourdomain.com IN TXT "v=DKIM1; k=rsa; p=MIGfMA0GCSqGSIb3DQEBAQUAA4GNADCBiQ...).
• Set a DMARC policy: v=DMARC1; p=none; rua=mailto:[email protected]. This tells receivers how to handle mail that fails authentication, and reports back to you. Start with p=none to monitor without blocking.
• Enable MTA-STS by publishing a policy in DNS as a TXT record at _mta-sts.yourdomain.com. This enforces encrypted SMTP connections, which improves security and compliance. See the MTA-STS RFC for specifications.

Verify and Maintain Your Setup

Use tools like dmarcian.com’s DMARC analyzer to monitor reports and validate your configuration. Even when your mail flows, records can drift or expire — schedule quarterly reviews.

Let’s not forget: security isn’t a one-time setup. Your domain’s trustworthiness depends on consistent DNS hygiene. If you’re managing this alongside calendars, documents, and video meetings, consider a unified platform like Unifiedesk, which handles DNS integration, encryption, and compliance out of the box across mail, calendar, Meet, Drive, and Docs.

How to Test and Verify Your Mail Server Setup

Let's verify your Docker Compose mail server with Rspamd and ClamAV is working and secure: send a test email from Gmail to see spam scores in Rspamd’s web UI, attach the EICAR test file to confirm ClamAV blocks it, check TLS with OpenSSL, review mail logs for delivery issues, and validate MX, SPF, and DKIM records via mxtoolbox.com. These steps ensure your server is both functional and hardened against spam and malware.

Test Spam and Virus Protection

• Send a test email from Gmail to your domain and check the Rspamd web UI at http://yourdomain.com:11111 to see the spam score and inspection results.
• Attach the EICAR test file (a harmless virus signature) to a message—ClamAV should immediately block and quarantine it; verify this in the server logs.
• Use openssl s_client -connect yourdomain.com:443 to confirm TLS 1.2 or higher is enforced and the certificate chain is valid. A successful connection shows the server is properly configured.

Validate DNS and Logs

• Check /var/log/mail.log (or /var/log/mail/ on some systems) for delivery failures, authentication rejects, or connection errors—common causes include misconfigured SPF or MX records.
• Use mxtoolbox.com to verify your domain’s MX, SPF, and DKIM records are published correctly. It checks real-time DNS and warns of common issues like missing or malformed records.
• Use RFC 5321 as a reference to confirm your SMTP server complies with standard email transport behavior—especially important when troubleshooting authentication or relaying.

If you’re setting up your own domain email system, consider tools like Unifiedesk’s self-hosted suite—it includes built-in spam filtering, virus scanning, and end-to-end encryption, all without managing Docker or configuration files manually.

Never rely on a single test. A robust email system must be validated from end to end: delivery, spam detection, virus scanning, and encryption.

Can You Run This Stack with Unifiedesk Instead?

Yes — you can run a mail server with Rspamd and ClamAV using Unifiedesk, but without the complexity of Docker, DNS management, or certificate handling. Unifiedesk provides a self-hosted, open-source suite with Rspamd and ClamAV built in, including JMAP, IMAP, SMTP, full encryption at rest, and admin controls — all managed through a clean UI. You keep your data private with AES-256-GCM encryption under per-account keys.

Why Unifiedesk Simplifies the Stack

Instead of wrestling with Docker Compose files, managing certificate renewal, or configuring SPF/DKIM records by hand, you set up your domain and start syncing mail, calendar, Drive, and AI in minutes. The platform handles TLS for transit, enforces inbound spam filtering via Rspamd, and signs outbound messages with DKIM — all through a dashboard.

Let’s say you want to move from a self-hosted mail stack. With Unifiedesk, you’re not just replacing Postfix or Dovecot — you’re switching to a full workspace that treats email, files, schedules, and conversation as a unified system, not a patchwork.

What You Still Control (and Why It Matters)

Your data never leaves your control. Unlike cloud services that store mail in plaintext or scan content for ads, Unifiedesk encrypts every message and file at rest with per-account keys using AES-256-GCM — a standard trusted by financial and government systems. This means even if someone gains access to your server storage, your data remains unreadable.

You maintain full ownership of your domain, with automated, verified SPF, DKIM, and DMARC setup via the admin UI. As the RFC 7208 standard notes, proper DMARC enforcement is essential for deliverability and sender reputation — and Unifiedesk implements it correctly out of the box.

From your mobile device or desktop, you’ll use JMAP — the modern, efficient successor to IMAP — which syncs your mail, calendar, contacts, and Drive seamlessly. You’ll see real-time updates, manage team calendars, share files with expiring links, and use a private AI assistant that never trains on your inputs. All this without writing a line of Docker config.

For full control, you can deploy Unifiedesk on-premise or in your own cloud — no hosting dependency. You choose the location, the hardware, and the access policy. If you ever want to move, the open-source engine makes it straightforward.

Whether you're managing your own domain, protecting sensitive communications, or simply tired of juggling tools, Unifiedesk gives you a secure, private, and usable alternative. No tradeoffs. Just a clean stack with real privacy.

Explore the full suite: email, calendar, Meet, Drive, Documents, contacts, and AI assistant. Ready to run it? Self-host with confidence.

When Should You Host Your Own vs. Use Unifiedesk?

You should host your own mail server with Docker Compose if you want full control over every component—like tweaking Rspamd rules or managing ClamAV updates yourself. But if you value simplicity, built-in security, and time savings, Unifiedesk handles encryption, spam filtering, and maintenance for you—without the overhead. You get the same strong privacy foundation, just without the daily grind.

Control vs. Convenience: What You Really Gain

With Docker Compose, you see every layer: how Rspamd evaluates a message, how ClamAV scans attachments, and how your mail flows through each step. That visibility is powerful if you’re a system administrator or security enthusiast. But it also means you’re responsible for updates, backups, log monitoring, and security patching. One missed update can open a vulnerability window.

Unifiedesk gives you that same security posture—E2E encryption, per-account keys, and DKIM/SPF/DKIM enforcement—without any of the chores. Your mail is always protected, and the platform handles everything from OS updates to TLS cipher suite rotation. The open-source engine powers it all, and you can audit it, but you don’t need to run it. TLS for transit is enforced everywhere, just as industry standards require.

When You Need More Than Just Mail

Mail is just the start. If you’re in a team, you’ll want shared mailboxes, calendar sync, document collaboration, real-time video meetings, and a smart AI assistant. Docker Compose can’t deliver that—all those pieces must be wired together manually, often with incompatible tools.

Unifiedesk bundles it all: email, calendar, Meet, Drive, Documents, contacts, and an AI assistant with no training data leaks. All with custom domains, live DNS records, and a single sign-on. You don’t need to manage five separate services or worry about compatibility.

For most users, especially teams, the trade-off isn’t worth it: you gain minor control but lose hours every week. Self-hosting is viable if you have engineers or an IT team. But for everyone else, Unifiedesk delivers sovereignty without the burden. Self-hosted versions still encrypt data at rest with AES-256-GCM and keep keys per account—so your privacy remains intact, just no need to manage it.

Conclusion: You Don’t Need to Choose Between Security and Simplicity

Setting up a mail server with Rspamd and ClamAV gives you strong protection against spam and malware. But maintaining it—DNS records, TLS certificates, rule updates, performance tuning—requires constant attention.

Unifiedesk delivers the same level of privacy and security without the operational burden. You get full control over your data, domain, and infrastructure, with zero maintenance on your part.

Ready to put this into practice? Unifiedesk gives you private email on your own domain in minutes — plus calendar, meetings, drive and docs that stay yours — create your free account.

Frequently asked questions

Can I run Rspamd and ClamAV without Docker?

Yes, but it increases complexity. Docker simplifies deployment, updates, and isolation.

Is ClamAV enough to block all malware?

No — it detects known threats. For zero-day protection, pair it with sandboxing and email filtering tools.

How does Rspamd prevent false positives?

It uses Bayesian learning, reputation scoring, and collaborative filtering to reduce false spam reports.

Do I need a static IP for my mail server?

Yes — most ISPs block port 25, and email servers require a fixed IP for DNS records and reputation.

Can I use Rspamd with Gmail?

Yes — but only if you’re sending mail via your own server, not receiving from Gmail.

Is self-hosting email compliant with GDPR?

Yes, if you control data location and consent — data residency and encryption at rest are key.

How often should I update ClamAV signatures?

Daily updates are standard. Use `freshclam` with a 6-hour refresh interval for best protection.

What happens if my mail server is blocked by spam filters?

Check if your IP is blacklisted. Use tools like Spamhaus or MXToolbox to diagnose and request delisting.

Does Rspamd work with DKIM signing?

Yes. Rspamd can verify outgoing DKIM signatures and help enforce them in outbound mail.

Can I use JMAP with my Docker mail setup?

Yes — but only if you add a JMAP server like Sieve or use a tool like mailcow-jmap. Unifiedesk includes JMAP natively.

Is Unifiedesk end-to-end encrypted?

Yes — the hosted platform uses end-to-end encryption by default. Self-hosted deployments use AES-256-GCM with per-account keys for all messages and files.

Can I use Unifiedesk with my existing domain?

Yes — you can add any custom domain and generate MX, SPF, DKIM, and DMARC records in minutes.