Install Your Own Mail Server

Email is a critical part of communication, both for individuals and businesses. While most people rely on third-party email providers like Gmail, Outlook, or Yahoo, setting up your own mail server on a GNU/Linux system offers several key advantages—greater control, privacy, and customization.

One of the biggest concerns with using commercial email providers is confidentiality. Your emails pass through third-party servers, where they may be scanned, stored, or even shared for advertising and data analysis purposes. When you run your own mail server, you maintain full control over your messages, ensuring that sensitive communications remain private and secure. This is especially crucial for businesses handling confidential client data, or privacy-conscious individuals who do not want their emails monitored.

Beyond privacy, owning a mail server allows for advanced customization, better security configurations, and avoiding potential service outages or censorship from big providers. While setting up and maintaining a mail server requires effort and technical knowledge, the benefits of full control over your email far outweigh the challenges.

In this guide, I’ll walk you through the step-by-step process of setting up your own mail server on a GNU/Linux system, helping you take charge of your communication security.

Managing mails involves several software component (IMAP, SMTP, Anti-spam), that is why we call it as mail infrastructure instead of a simple mail server.

How Do Your Users Fetch Emails ?

Mail User Agent

The user usually has a mail client or MUA — Mail User Agent (Thunderbird, Mutt, etc…), that can use the POP3 or IMAP protocol to fetch emails from the server.

That mail client connects to the POP3 (TCP 110) or IMAP (TCP 143) port on the server and may or may not send the STARTTLS command that initiates an encrypted connection. It then sends the user's username (which is equal to the email address in our case) and their password. POP3 is less used nowadays and lacks of support for multiple folders on the server.

The client may as well use the secure TLS-encrypted ports directly — 995 for POPs or 993 for IMAPs.

IMAP Server

Dovecot sends a query to the MySQL database and verifies that the username and password belong to a known user. If the password is wrong then Dovecot will refuse the login.

As defined in the Devecot configuration file, each user has a directory where emails are stored. So if the user is named vselvarajah@atomicl.net, Dovecot will look for the mail directory in /var/vmail/atomicl.net/vselvarajah/Maildir/… and save the email in the corresponding mail directory.

NOTE: We can also integrate a webmail service so users can send/read mails using the browser but we are not going to do that here.

How Do Users Send Emails ?

./img/smtp.jpg

Mail User Agent

The user writes the email using a mail user agent (MUA) such as Thunderbird, Mutt, etc. And clicks on "Send". The mail client establishes an SMTP connection to your Postfix server on port 25 or 587. In most cases, the user's internet service provider blocks port 25 to prevent spam, so users use port 587.

To ensure that the user is allowed to send emails to other SMTP servers through your system, a username (or email address) and password are required.

In the guide we instruct Postfix to use encryption for authentication for security reasons, but this is completely optional.

SMTP Verification

Your Postfix server checks the email address and password in the database. Or it can delegate to another service like Dovecot to handle authentication.

Then Dovecot sends a query to the MariaDB database to check if the email address and password are correct and tells Postfix the result.

SMTP Response

Postftix knows now that it is authorized to send the email on behalf of the user. It tells the user that it successfully accepted the email. The email is put into Postfix’s mail queue for further processing. Postfix will now query a DNS server to determine mail server of recipient. As the recipient has an ...@recipient-domain.com email address it checks the MX record of the recipient-domain.com domain and then gets the respective IP address.

SMTP Establish Connection

Postfix now knows which mail server to send the email to. It opens an SMTP connection and delivers the email.

How Emails Are Transferred ?

./img/Mail_Infra.jpg

Les's see how mails get transferred to vselvarajah@atomicl.net recipient from some else.

Query DNS

  1. SENDER's SMTP: Query DNS server for email server of the domain atomicl.net.
  2. DNS: Checks MX record of atomicl.net domain for mails servers and sends the IP address.

Establish Connection

  1. SENDER's SMTP: Connects to that IP on TCP port 25 which is by default used by SMTP server (Simple Mail Transport Protocol)
  2. MY POSTFIX: Welcome, I am Postfix server. Who is there ? (220 mail.atomicl.net ESMTP Postfix)
  3. SENDER's SMTP: Hi, I am a remote server. (EHLO mail.sender-domain.com)
  4. MY POSTFIX: Nice to meet you. I can offer you a few features like pipelining and encryption… (STARTTLS, PIPELINING, SIZE 4000000, …)
  5. SENDER's SMTP: OK, then we switch to an encrypted connection. (STARTTLS) The connection is now using TLS encryption.
  6. SENDER's SMTP: I have an email from user@sender-domain.com (MAIL FROM:<user@sender-domain.com>)
  7. MY POSTFIX: I see. (Ok)
  8. SENDER's SMTP: This email is for vselvarajah@atomicl.net (RCPT TO:<vselvarajah@atomicl.com>)

Query DB

  1. MY POSTFIX: Hi database. (Connects to TCP port 3306 on the local host to talk to MariaDB.) Could you check if atomicl.net is one of our mail domains ? (SELECT … from virtual_domains …)
  2. MY MariaDB: Yes, I have a that domain in my database.
  3. MY POSTFIX: Nice. And do you have a mailbox or a forwarding rule for vselvarajah@atomicl.net ? (SELECT … from virtual_aliases/virtual_users …)
  4. MY MARIADB: Yes, there is a mailbox for that address.
  5. MY POSTIFX: Remote server, the recipient looks good. (Ok)
  6. SENDER'S SMTP: Good. Then here’s the actual email. (DATA) The remote server sends the email header and body.

Check for Spam

  1. MY POSTFIX: Connects to port 11332 on the local host to reach the rspamd.
  2. My Postfix: Hi, rspamd. I have a new email. Could you spam check ?
  3. MY RSPAMD: Sure. Well, there are a few minor issues. But generally the mail looks good. I suggest you accept it.

SMTP Response

  1. MY POSTFIX: Hey, remote server. Your email is fine.

Mail Delivery

  1. MY POSTFIX: Uses a socket file at /var/spool/postfix/private/dovecot-lmtp to talk to Dovecot.
  2. MY POSTFIX: Hi, Dovecot. Here is a new email for vselvarajah@atomicl.net
  3. MY DOVECOT: Got it.

Check Sieve Rules (Optional)

  1. MY DOVECOT: Checks for additional Sieve rules and then stores the email on disk at /var/vmail/atomicl.net/vselvarajah/Maildir/INBOX

Prerequisites

  1. GNU/Linux operating system (This guides is uses Debian 12)
  2. Postfix receives incoming emails from the internet and sends out outgoing emails to other mail servers. It is the software that speaks SMTP.
  3. rspamd runs sanity checks on an incoming email to determine whether it is spam.
  4. Dovecot stores emails on your hard disk, applies filters and lets your users fetch their emails using the POP3 and IMAP protocols
  5. MariaDB is a database that stores information about your domains, email aliases and email accounts

Let's install all required packages:

  sudo apt install mariadb-server postfix postfix-mysql dovecot-mysql dovecot-imapd dovecot-lmtpd mutt certbot ca-certificates telnet swaks

Installation

Welcome to the guide on setting up your own mail server. Below are the steps to follow:

  1. SMTP server — Postfix
  2. IMAP server — Dovecot
  3. Obtain TLS certificate
  4. Configure DB
  5. Configure Postfix
  6. Configure Dovecot
  7. Postfix and Dovecot communication
  8. Optional — Quotas
  9. Test IMAP
  10. SMTP authentication
  11. Postfix submission server
  12. Deny forged sender addresses
  13. Sender Policy Framework (SPF)
  14. DomainKeys Identified Mail (DKIM)
  15. Domain-based Message Authentication, Reporting and Conformance (DMARC)
  16. SPF and DKIM check
  17. Optional — Block spam
  18. Tips

Follow each step carefully to have a fully functional mail server.