Big picture

The mail server that you are about to set up uses several software components. Let me first explain briefly what the purpose of each software is:

  • Debian “Buster” – the operating system
  • Postfix received incoming emails from the internet and sends out outgoing emails to other mail servers. The component that speaks SMTP.
  • rspamd runs sanity checks on an incoming email to determine how likely it is to be spam
  • Dovecot stores emails on your hard disk, applies filters and lets your users fetch their emails using the POP3 and IMAP protocols
  • Roundcube is a webmail interface so users can read their emails using a web browser, manage their email rules and change their password
  • MariaDB (formerly known as MySQL) is a database that stores information about your domains, email aliases and email accounts

What happens when someone sends you an email?

Let us assume that you are responsible for the email domain example.org and someone on the internet sends an email to john@example.org. This is what happens step by step:

Receiving an email from another mail server
  1. REMOTE: Hey, DNS server. I have an email for someone in the example.org domain. Can you tell me the name of the responsible mail server?
    DNS: According to the DNS zone of example.org it is mx.example.org.
  2. REMOTE: Nice. Do you have an an IP address for mx.example.org?
    DNS: I have an IPv4 address here. It is 85.25.72.76.
  3. REMOTE connects to that IP address on TCP port 25 which is by definition used for SMTP – the simple mail transport protocol.
    POSTFIX: Welcome. I am Postfix. Who is there? (“220 mx.example.org ESMTP Postfix”)
    REMOTE: Hi, I am remote server. (“EHLO remoteserver”)
    POSTFIX: Nice to meet you. I can offer a few features like pipelining and encryption by the way. (“STARTTLS, PIPELINING, SIZE 4000000, …”)
    REMOTE: Let’s switch to an encrypted connection. (“STARTTLS”)
    (The connection is now using TLS encryption.)
    REMOTE: I have an email from donald@duck.com here. (“MAIL FROM:<donald@duck.com>”)
    POSTFIX: I see. (“Ok”)
    REMOTE: The email is meant for john@example.org. (“RCPT TO:<john@example.org>”)
  4. POSTFIX: Hey database. (Connects to TCP port 3306 on the local host to talk to MariaDB.) Could you check if example.org is one of our mail domains? (“SELECT … from virtual_domains …”)
    MariaDB: Yes, I have a domain like that.
    POSTFIX: Oh, good. And do you have a mailbox or forwarding for someone called john@example.org? (“SELECT … from virtual_aliases/virtual_users …”)
    MariaDB: Yes, there is a mailbox for that address.
    POSTIFX: Hey, remote server. The recipient looks good. (“Ok”)
    REMOTE: Roger. Then here’s the actual email. (“DATA”)
    (The remote server sends the email header and body.)
  5. (POSTFIX connects to port 11332 on the local host to reach the rspamd.)
    POSTFIX: Hey, rspamd. I have a new email here. Could you give it a look for, you know, spam and stuff?
    RSPAMD: Sure. Well, there are a few minor issues. But generally the mail looks good. I suggest you accept it.
    POSTFIX: Hey, remote server. Your email is fine.
  6. (Postfix uses a socket file at /var/spool/postfix/private/dovecot-lmtp to talk to Dovecot.)
    POSTFIX: Hey, Dovecot. Here is a new email for john@example.org.
    DOVECOT: Got it.
  7. (Dovecot checks for additional Sieve rules and then stores the email on disk at /var/vmail/example.org/john/Maildir/INBOX)

Phew, that was a lot. I hope it helps to understand though how the different components are involved until an email gets delivered.

A lot of things are happening when your users get an email. The next situation I would like to explain is…

What happens if a user fetches their email using IMAP/POP3?

This process is way simpler. It looks like this:

User connects through IMAP
  1. The user usually has a mail client (also called a mail user agent) that can use the POP3 or IMAP protocol to fetch emails from the mail server. I prefer Thunderbird for example. That mail client connects to the POP3 (TCP 110) or IMAP (TCP 143) port, sends the STARTTLS command that initiates an encrypted connection and sends the user’s username (which equals the email address in our case) and their password. If possible users should use IMAP. POP3 is pretty much deprecated nowadays and lacks support for multiple folders on the server.
  2. Dovecot sends a query to the MySQL database and verifies that the username and password belong to a known user. The password is not stored in plain text in the database. Instead it computes the password hash and compares it to the hash in the database. If the password is wrong then Dovecot will refuse the login.
  3. We have a fixed scheme how emails are stored on disk. So if the user is called jane@example.org then Dovecot will look for email files in /var/vmail/example.org/jane/Maildir/… and send the user the requested emails.

Nowadays many users seem to prefer webmail to a mail client installed on their computers. As an email power user I honestly do not understand that trend. But who am I to judge. So let’s take a look how webmail works technically:

What happens if a user reads their email using web mail?

The Roundcube software that provides the web mail interface is basically a PHP software that is a gateway between HTML pages and a built-in IMAP client. So when a user uses their browser to connect to the web mail interface…

User connects through webmail
  1. The users points the browser to the HTTPS URL of the webmail interface. Apache web server receives the connection and runs the PHP scripts of Roundcube. Roundcube shows a login form for the username and password. The user enters the username (that equals the email address) and the password and submits the login form.
  2. Roundcube connects to Dovecot using IMAP and forwards the username and password to check if they are valid.
  3. Dovecot treats this connection similar to a mail client connection. It queries the MariaDB database to verify the username (=email address) and password.
  4. If the authentication was successful Dovecot fetches the mail files from disk and sends them through IMAP to Roundcube. Roundcube then renders the emails as HTML and the user can read it.

So you see that the web mail access also works through IMAP. The user does not realize that though.

Okay, the final scenario I would like to explain is…

What happens if the user wants to send an email to the internet?

Of course your users also want to send emails to other internet users. But they cannot send the email directly to the destination mail server. First their mail client does not know which destination server is responsible for the recipient (hint: DNS) – that functionality just is not built in. And second the user is likely assigned a dynamic IP address which is blocked by most mail servers because they tend to get abused by infected Windows PCs that send out spam. So the correct way to send an email to the internet is through your mail server. This is called relaying because your mail server acts as a relay. In this example your user wants to send an email to fred@example.net.

Relaying/sending email via the mail server
  1. The user writes the email in their mail client and clicks on “Send”. The mail client establishes an SMTP connection to Postfix. To make sure that the user is allowed to send email through your system it requires a username and password along with the email. This information is sent in an encrypted way.
  2. Postfix could now check the password in the database directly. But as Dovecot already knows how to handle authentication it is easier to ask Dovecot to verify the username and password. (SMTP authentication in Postfix is surprisingly ugly.)
  3. Dovecot now sends a query to the MariaDB database to check if the username and password (hash) are correct and tells Postfix the result.
  4. Postix knows now that it is authorized to send the email on behalf of the user. It tells the user that it successfully accepted the email. The email is put into Postfix’s mail queue for further processing. Postfix will now query a DNS (name server) to determine the responsible destination mail server. As the recipient has an “…@example.net” address it checks the MX record of the “example.net” domain and then gets the respective IP address.
  5. Postfix now knows which mail server to send the email to. It opens an SMTP connection and delivers the email.

12 thoughts on “Big picture”

    1. Christoph Haas

      Thanks for reporting that. Seems that a WordPress plugin is behaving strangely. I’ll disable it and open a bug report.

    2. Christoph Haas

      The old plugin seemed to be dead anyways. Whoa… PHP is bad enough. But they are using SVN and Trac to manage plugins. Let me check the calendar again if it’s really 2020. 🙂

      Anyway… I have installed a different plugin and it seems to be better without Javascript now.

  1. Just a thought: have you ever considered using SOGo instead of Roundcube? I think that one would also give a calendar and nice “push-sync” to Android and Outlook clients.

    1. Christoph Haas

      Yes, I even spent two days working with it. SOGo doesn’t plug in nicely with existing solutions. Maybe there is a very hacky solution but every detail looks like you should throw away everything and just use SOGo. 🙁

  2. Thank you for this overview and the nice, simple visualizations! Being nitpicky, obviously a DNS server that is asked for a type A record and returns 2001:616:c8:something is broken – the mailserver would ask for AAAA instead to fix this little inconsistency. 😉

  3. Minor typos…

    In “What happens when someone sends you an email?”.

    You write that you’ll show step by step what happens when someone sends an email to john@example.org, but the rest of the text says max@example.org.

    Furthermore in the first image, the remote site asks for the MX record of example.com, not example.org.

    In “What happens if a user fetches their email using IMAP/POP3?”, step 2, you talk about MySQL. You mean MariaDB.

  4. Nice guide!

    FYI – In your first diagram you show the first interaction referencing the “.com” TLD (“MX of example.com?” “mx.example.com”).
    Other text and diagrammed interactions reference a “.org” TLD.

  5. Excellent guide!

    Small typo..

    In “What happens if the user wants to send an email to the internet?” at number 4, Postfix isn’t spelled correctly.

  6. Hi Christoph,

    Thanks a lot for this great guide, it clears up and answers a lot of questions I had about this matter.

    Any remarks, pro/cons with regard to using LDAP instead of a RDBMS for information storage?

  7. Hi Christoph,

    I am grateful to you for producing such a marvelous guide. I can’t thank you enough. It has been a pleasure going through the steps…I got to learn a lot and yet, I have a lot of ground to cover before I would be in any position to run my own mail server.

    I have several questions and I don’t know which one to ask first–but I guess the bottom line is with regards to the big picture “Relaying/sending email via the mail server” — the item 5 here (i.e. Postfix)– what port does it use? 25 or 587 or 465?

    In the email log (var/log/mail.log) I could see it trying to connect on port 25 and then timeout.
    [I guess, I totally misunderstood that I would be able to send mail to external email addresses via port 587].

    It would be great if you could clarify this.

    If it is port 25, I would have come to an end of the road (after several weeks of hitting my head against the brick wall in a cloud) with my current provider because I don’t think I will follow their advice of using an authenticated relay like sendgrid etc.

    And then, I am pretty excited by your opening remarks “the monthly costs of a cheap virtual server (starting at around 5€) ” — Christoph, are you able to recommend some providers that meet the requirements of running ISPMail and who also happen to satisfy your high quality standards?

    Thank you ever so much.

    With warm regards

  8. Thank you so much ! I had terrible memories of setting up postfix, dovecot and roundcube on my debian machine.
    Thanks to this guide I spent some good time, understanding much better how everything connects.
    Great job, really !

Leave a Reply

Your email address will not be published. Required fields are marked *

Scroll to Top