Post

Self-Hosted Domain Email Part 1: Why is Email So Hard?

Discover why self-hosted domain email is so complex despite its 1970s origins, and how SPF, DKIM, and DMARC transformed the landscape. This guide demystifies email protocols and prepares you for a cost-effective multi-domain solution using Mailu, saving you $5 per month per domain compared to commercial providers.

Posted
Preview Image
By robertsinfosec
17 min read
Self-Hosted Domain Email Part 1: Why is Email So Hard?

Overview

If you're an IT hobbyist like me, you probably have a few domain names lying around for various projects or just for fun. Getting custom email addresses like [email protected] is a nice touch, right? But if you've looked into it, you'll find most hosting providers charge around $5 per month, per user, per domain. Got 3 domains and want a couple of addresses on each? That starts to add up!

If this is not you: If this isn't you and you just have 1 or 2 domains, then the far simpler solution is just to pay for a hosted email solution. This blog series is really if you have 5, 10 or more domains and you simply want basic email services for each domain.

If this is you: Consider that this is not a simple project to take on, and it's also not a "set it and forget it" solution. Even following this script, this could take a day or maybe two (some of that waiting for DNS changes to propagate). You will need to be comfortable with the command line, Docker, and some basic networking concepts. If you are not, then I would recommend sticking with a hosted solution.

This is the first part of a series where we'll explore a more cost-effective solution: self-hosting your email. But before we dive into the "how" with an amazing tool called Mailu, we need to understand the "why." Specifically, why is email, a technology that's been around since the 1970s, so notoriously complex to manage yourself? This post will help unravel that mystery.

TL;DR

Email seems simple on the surface, but decades of fighting spam and ensuring security have layered on immense complexity. This post explores the core email protocols and the essential anti-spam technologies (SPF, DKIM, DMARC) that make self-hosting a challenge, yet a rewarding endeavor. We'll set the stage for a practical solution in upcoming posts.

Heads up: This post contains some affiliate links. If you buy something through them, I might earn a small commission — at no extra cost to you. I only recommend tools I’ve actually used or vetted.

Motivation / Problem Statement

It seems natural and intuitive to want a [email protected] "domain email address" when you buy an example.com domain, even for your personal projects. It should be such a simple idea, but the reality is you either need to pay an endless monthly fee to a provider, or you need to self-host. Self-hosting though, especially if you were to piecemeal it together with a few different services was always absurdly complex. It was a nightmare of configuration, DNS records, and constant maintenance.

Recently, I looked into a few modern options and ran across Mailu, a Dockerized mail server suite that makes self-hosting pretty easy. This is a huge step in the right direction since this absorbs a majority of the complexity. However, there are still a few things you need to know before digging in. I don't know if email can ever be "turn key".

In this case, we'll get an inexpensive VPS. For example, you can spin up your own mail server on an inexpensive Virtual Private Server (VPS) – perhaps from a provider like Racknerd (I use their 4.5GB RAM, 3vCPU, 100GB SSD plan with 8.5TB monthly bandwidth, which comes out to about $3.32/month when paid annually at $39.88). You gain control, learn a ton, solve your "email problem" for good, and potentially save a good chunk of change.

Email is a different beast compared to, say, hosting a website. The protocols might be old, but the ecosystem around them is a minefield. Let's dig in.

The Deceptive Simplicity of Email: 1970s Roots

At its core, email relies on a few fundamental protocols, some of which date back to the early days of the internet when the network was a much smaller, more trusting place.

  • SMTP (Simple Mail Transfer Protocol): Originally defined in RFC 821 (1982) and updated several times since (currently RFC 5321), SMTP is the workhorse for sending email. It dictates how mail servers transfer messages to each other.
  • POP3 (Post Office Protocol version 3): Defined in RFC 1939 (1996), POP3 is one way your email client (like Thunderbird or Outlook) retrieves messages from the server. It typically downloads emails to your local device, often removing them from the server.
  • IMAP (Internet Message Access Protocol): Currently defined in RFC 3501 (2003), IMAP is another protocol for retrieving mail. Unlike POP3, IMAP generally leaves messages on the server, allowing you to synchronize your mail across multiple devices. This is what most of us use today.

In those early days, systems were often "open relays" – they'd happily accept email from anyone and send it anywhere. Trust was implicit. Imagine that!?

Then Came Spam… The Great Complicator

The open and trusting nature of early email protocols was a paradise for spammers. As the internet grew, so did unsolicited bulk email – spam. This single factor is arguably the primary reason why managing a mail server today is so complex. An endless arms race began between spammers and those trying to stop them, leading to layers upon layers of checks, verifications, and reputation systems.

The critical consequence for anyone wanting to self-host email is this: if you attempt to send email from an IP address or domain without the proper configurations and established reputation, your messages are highly likely to be flagged as spam, silently dropped, or outright rejected by receiving mail servers.

This means that you can't just simply stand up an SMTP and IMAP server, set your DNS MX records and be done with it.

The Alphabet Soup of Modern Email Deliverability

To combat spam and ensure some level of trust and authenticity, a host of new technologies and DNS record types were developed. Getting these right is non-negotiable for successful email delivery.

DNS Records: The Foundation

Your Domain Name System (DNS) records are critical. Beyond the basic A (address) record for your server, email relies heavily on others:

  • MX (Mail Exchange) Record: This fundamental DNS record tells the world which server(s) are responsible for receiving email for your domain. When someone sends an email to [email protected], their mail server looks up the MX record for yourdomain.com to know where to deliver it.
  • TXT (Text) Record: A versatile DNS record type that can hold arbitrary text. It has become the go-to for various email authentication mechanisms.
  • SRV (Service) Record: While less critical for basic sending/receiving, SRV records can be used for autodiscovery features, helping email clients automatically find server settings for IMAP, SMTP, CalDAV, CardDAV, etc.
  • PTR (Pointer) Record / Reverse DNS: This isn't a record you set in your domain's DNS zone, but rather one your VPS provider sets for your server's IP address. It should resolve your IP address back to your mail server's hostname (e.g., mail.yourdomain.com). Many receiving servers check this, and a missing or mismatched PTR record is a big red flag.

Just to give you an idea, here is what is required for DNS entries if you were configuring Mailu for a domain called example.com:

Mailu Configuration

As you can see though, there is a "Download Zonefile" button in the top-right corner. This is significant because DNS providers like Cloudflare let you Import a "zonefile" in this same Bind format. This means you can easily copy and paste the entire configuration into your DNS provider after you generate it. This is a huge time saver.

Fighting Forgery and Ensuring Integrity

These anti-spam technologies are primarily implemented as TXT records in your DNS:

  • SPF (Sender Policy Framework): Defined in RFC 7208, SPF allows domain owners to specify which mail servers are authorized to send email on behalf of their domains. A receiving server can check the SPF record to verify that the email came from an approved source.
    • Example: An SPF record might say, "Only IPs 1.2.3.4 and servers listed in the MX records for anotherdomain.com are allowed to send mail for mydomain.com."

  • DKIM (DomainKeys Identified Mail): Described in RFC 6376, DKIM provides a way to cryptographically sign outgoing emails. A public key is published in a DNS TXT record, and the corresponding private key is used by the sending mail server to sign messages. Receiving servers can then use the public key to verify the signature, ensuring the email hasn't been tampered with and genuinely originated from the claimed domain.

  • DMARC (Domain-based Message Authentication, Reporting, and Conformance): Specified in RFC 7489, DMARC builds on SPF and DKIM. It allows domain owners to publish a policy stating what receiving mail servers should do if an email fails SPF and/or DKIM checks (e.g., do nothing, quarantine it, or reject it). DMARC also provides a mechanism for receiving servers to send reports back to the domain owner about email authentication results, which is invaluable for troubleshooting.
    • A DMARC record is also a TXT record, typically at _dmarc.yourdomain.com.

Other Critical Considerations

  • IP Reputation & Blacklists (RBLs/DNSBLs): Your mail server's IP address has a reputation. If it's a new IP, it has no reputation (which can be bad). If it's an IP that was previously used for sending spam, it has a bad reputation. Numerous Real-time Blackhole Lists (RBLs) or DNS Blacklists (DNSBLs) track spammy IPs, and mail servers worldwide query these lists. Getting listed can cripple your email delivery.
  • TLS/SSL for Encryption: All modern mail servers expect encrypted connections. This means configuring TLS/SSL certificates (often via Let's Encrypt) for SMTPS (SMTP over SSL/TLS), IMAPS (IMAP over SSL/TLS), POP3S (POP3 over SSL/TLS), and for opportunistic encryption via STARTTLS on standard ports.
  • Server-Side Measures: Beyond DNS, your mail server itself will need things like greylisting (temporarily rejecting mail from unknown senders, which trips up many simple spam bots), rate limiting (preventing too many messages too quickly), and content filtering (spam/virus scanning).

Here are some examples of RBLs/DNSBLs where you can check if your IP is blacklisted:

Port Requirements and ISP Challenges

Another aspect of self-hosting email is that most residential ISPs and some VPS providers block the standard email ports:

  • Port 25 (SMTP): The standard port for server-to-server email transfer. Many ISPs and cloud providers (including AWS and some Digital Ocean plans) block outgoing traffic on port 25 to prevent spam. Without this port, your server cannot send mail to other mail servers.
  • Port 465 (SMTPS): Used for secure SMTP submissions.
  • Port 587 (Submission): The standard port for email clients to submit messages to a mail server.
  • Port 993 (IMAPS): The standard port for secure IMAP connections.
  • Port 995 (POP3S): The standard port for secure POP3 connections.

This is one reason why you usually can't self-host email on a home internet connection. Most ISPs block port 25 to prevent residential customers from running mail servers.

Before committing to a VPS provider, explicitly confirm they allow outbound traffic on port 25. Many don't mention this limitation until after you've paid.

Additionally, some major email providers (particularly Gmail, Yahoo, and Outlook.com) are increasingly stringent about accepting mail from new, low-volume mail servers. This is another layer of the anti-spam ecosystem but creates a very real "chicken and egg" problem for new self-hosters.

Time & Maintenance Commitment

Self-hosted email isn't a "set it and forget it" service:

  • Regular Updates: Email software, like any internet-facing service, requires regular security updates.
  • Certificate Renewals: Your TLS certificates will need periodic renewal (though Let's Encrypt and tools like Certbot can automate this).
  • Monitoring Deliverability: You'll need to occasionally check that your emails are still being delivered and not hitting spam folders.
  • Resource Management: Disk space, backups, and handling storage growth over time all require attention.
  • Backups: Yes, I'll say it again: backups! Even for low-volume [email protected] mailboxes, it must be part of your email strategy to make sure you have reliable backup/restore processes in place.

This ongoing maintenance is another hidden "cost" of self-hosting that's often overlooked in the initial financial calculation.

Tips, Warnings, and Gotchas

Self-hosting email is a significant responsibility. Misconfiguration can lead to your emails not being delivered, your domain gaining a poor reputation, or, in a worst-case scenario, your server being compromised and used to send spam, landing you on multiple blacklists.

Given these potential pitfalls, it's wise to approach self-hosting with caution and a methodical plan.

Start with a non-critical domain for testing your setup. Don't switch your primary personal or business email to a new self-hosted server until you're confident it's working reliably and securely.

This careful, iterative approach is essential because the alternative, relying on established providers, means they are the ones managing these complexities for you.

This overwhelming complexity is precisely why services like Gmail, Outlook.com, Fastmail, ProtonMail, and various business email providers are so popular. They abstract away all these headaches, managing the infrastructure, reputation, and constant updates for you. And again, if you only have 1 or 2 domains, this is the far simpler solution. Just pay the $5/month and be done with it!

Email Verification and Testing Tools

Before launching your email server for real-world use, you'll want to verify your configuration. Several valuable tools can help:

  • MX Toolbox: A comprehensive suite of email testing tools. The "SuperTool" can check MX records, blacklists, SPF, DKIM, DMARC, and more.
  • mail-tester.com: Send an email to a specially generated address, and get a detailed score and report on your email's deliverability factors.
  • DKIMValidator: Test if your DKIM configuration is working correctly.
  • DMARC Analyzer: Analyze DMARC reports to improve email deliverability.
  • SPF Record Testing Tools: Validate your SPF record syntax.

Before making your email server your primary mail solution, spend a week or two sending test emails to various major providers (Gmail, Outlook, Yahoo, etc.) and to your own accounts on these services. Check if they arrive correctly and don't end up in spam folders.

Summary

Email, born in an era of digital innocence, has had to mature in a world rife with abuse. What started as a set of simple protocols has become a complex ecosystem of verifications, authentications, and reputation management systems. Understanding these layers – SMTP, IMAP/POP3, MX, SPF, DKIM, DMARC, PTR records, IP reputation, and TLS – is the first crucial step before starting a project like this.

It's a lot to take in, and it might seem daunting. But don't be discouraged! With the right tools and a methodical approach, self-hosting your domain email is achievable.

Coming Up in This Series

This post has laid the groundwork by explaining why email is complex and what challenges you'll face. Let's look at what's coming in the rest of this series:

Part 2: Mailu Installation and Hardening

In the next post, we'll roll up our sleeves and get to work:

  • Selecting the right VPS provider and plan for your needs
  • Setting up a basic Debian/Ubuntu server with Docker and Docker Compose following this Setting up a Public VPS guide
  • Using the Mailu setup wizard to generate your configuration
  • Deploying Mailu with Docker Compose
  • Configuring proper TLS/SSL certificates with Let's Encrypt
  • Hardening your mail server against common threats
  • Setting up backups for your mail data

Part 3: Domain Configuration and Management

The final post will focus on the day-to-day management and practical aspects:

  • Adding and managing multiple domains in a single Mailu instance
  • Creating mailboxes, aliases, and catch-all addresses
  • Configuring webmail access and mobile app setup
  • Setting up proper DNS records for all your domains
  • Testing email deliverability
  • Monitoring logs and troubleshooting common issues
  • Advanced configurations like forwarding and automatic replies

By the end of this series, you'll have a fully functional, multi-domain mail server that's secure, reliable, and costs a fraction of what you'd pay for hosted email services.

Appendix

Here are some resources that can help you understand the concepts discussed in this post and assist you in setting up your self-hosted email server:

Resources and References

Email Acronyms and Terms Demystified

A quick reference for the terms used in this post and commonly encountered when dealing with email systems:

  • SMTP (Simple Mail Transfer Protocol): The standard protocol for sending emails between servers and from email clients to servers. (Defined in RFC 5321)
  • IMAP (Internet Message Access Protocol): A protocol for retrieving emails from a server, designed to keep messages on the server and synchronize across multiple clients. (Defined in RFC 3501)
  • POP3 (Post Office Protocol version 3): An older protocol for retrieving emails, typically downloading them to a single client and often removing them from the server. (Defined in RFC 1939)
  • DNS (Domain Name System): The internet's distributed directory that translates human-readable domain names (like robertsinfosec.github.io) into numerical IP addresses (like 192.0.2.1) and provides other domain-related information.
  • MX Record (Mail Exchange): A DNS record type that specifies the mail server(s) responsible for accepting email messages on behalf of a domain name.
  • TXT Record (Text): A versatile DNS record type that holds plain text information. Used for SPF, DKIM, DMARC, domain ownership verification, and other purposes.
  • SRV Record (Service): A DNS record type that specifies the location (hostname and port number) of servers for specific services (e.g., _autodiscover._tcp.example.com for email client autoconfiguration).
  • SPF (Sender Policy Framework): An email authentication method that allows a domain owner to specify which mail servers (IP addresses) are authorized to send email for that domain. Helps prevent sender address forgery. (Defined in RFC 7208)
  • DKIM (DomainKeys Identified Mail): An email authentication method that uses cryptographic signatures to verify that an email message was sent by an authorized owner of the sending domain and that the message has not been altered in transit. (Defined in RFC 6376)
  • DMARC (Domain-based Message Authentication, Reporting, and Conformance): An email authentication, policy, and reporting protocol that builds on SPF and DKIM. It allows domain owners to specify how receiving mail servers should handle messages that fail SPF or DKIM checks and to receive reports on such messages. (Defined in RFC 7489)
  • PTR Record (Pointer Record / Reverse DNS): A DNS record that maps an IP address back to a hostname. Mail servers often check this to verify the sender's identity.
  • RBL/DNSBL (Real-time Blackhole List / DNS Blacklist): Lists of IP addresses that are known or suspected sources of spam. Mail servers use these lists to filter incoming email.
  • TLS/SSL (Transport Layer Security/Secure Sockets Layer): Cryptographic protocols designed to provide secure communication over a computer network. Used to encrypt email connections (e.g., SMTPS, IMAPS, POP3S, and via the STARTTLS command).
  • MTA (Mail Transfer Agent): Software that transfers email messages from one computer to another using SMTP (e.g., Postfix, Exim, Sendmail). This is your mail server software.
  • MDA (Mail Delivery Agent): Software that receives emails from an MTA and delivers them to a specific user's mailbox (e.g., Dovecot, procmail).
  • MUA (Mail User Agent): An email client application used by end-users to compose, send, and receive email (e.g., Mozilla Thunderbird, Microsoft Outlook, Apple Mail, Gmail web interface).
Series, Self-Hosted Domain Email
email self-hosting domain email email protocols spam dkim dmarc spf mailu smtp imap pop3 rfc series:self-hosted-domain-email
This post is licensed under CC BY 4.0 by the author.