The big picture

The mail server that you are about to set up uses several software components. Let me first explain briefly what the purpose of each software is:

  • Debian “Jessie” – the operating system
  • Postfix received incoming emails from the internet and sends out outgoing emails to other mail servers
  • SpamAssassin runs sanity checks on an incoming email to determine how likely it is to be spam
  • Dovecot stores emails on your hard disk, applies filters and lets your users fetch their emails using the POP3 and IMAP protocols
  • Roundcube is a webmail interface so users can read their emails using a web browser, manage their email rules and change their password
  • MySQL is a database that stores information about your domains, email aliases and email accounts

What happens when someone sends you an email?

Let us assume that you are responsible for the email domain example.org and someone on the internet sends an email to john@example.org. This is what happens step by step:

ispmail-jessie-bigpicture-incoming

  1. The remote mail server asks its nearest DNS server for the MX (Mail eXchanger) of the recipient’s domain. The domain is example.org so it asks for the “MX” record of the example.org domain. The DNS server replies that the host name of the reponsible destintaion mail server is mx.example.org. (In case there is no MX record it will try to use the A record as a fallback.)
  2. The mail server asks the DNS server again to find out the IP address of the mail server mx.example.org. The DNS server responds with the IP address (or multiple IP addresses).
  3. Now the mail server establishes a TCP connection on port 25 (which is by definition used for SMTP – the simple mail transport protocol) to that IP address. It tells the receiving server about the sender’s and receiver’s email address.
  4. The Postfix software on the receiving server does a query to the MySQL database to find out if it is responsible for that recipient. If it would not find any information about the recipient in the database it would now respond with an error message and terminate the SMTP connection. The sending server would generate a non-delivery receipt (also called bounce) so that the sender knows that the recipient’s email address was not deliverable.
  5. The recipient seems to be valid. So Postfix accepts the email, puts it into its incoming queue and sends a copy to the SpamAssassin software using the milter protocol. SpamAssassin will run various checks to compute the probability that the incoming email is spam. It adds an entry to the mail header (that contains control information) telling the score (=spam probability).
  6. Postfix forwards the email to the Dovecot software which is responsible for delivering the email to disk.
  7. Dovecot knows that the user directory of john@example.com is /var/vmail/example.com/john and checks if there are filter rules defines. It also has a global filter rule that stores spam emails into a folder called “Junk” if it reaches a spam score of at least 5. Finally it saves the email as a file to disk at /var/vmail/example.com/john/Maildir/…

A lot of things are happening when your users get an email. The next situation I would like to explain is…

What happens if a user fetches their email using IMAP/POP3?

This process is way simpler. It looks like this:

ispmail-jessie-bigpicture-fetching

  1. The user usually has a mail client (also called a mail user agent) that can use the POP3 or IMAP protocol to fetch emails from the mail server. I prefer Thunderbird for example. That mail client connects to the POP3 (TCP 110) or IMAP (TCP 143) port, sends the STARTTLS command that initiates an encrypted connection and sends the user’s username (which equals the email address in our case) and their password. If possible users should use IMAP which always leaves the emails on the server and which also supports multiple folders. POP3 is very limited and pretty much deprecated nowadays.
  2. Dovecot does a query to the MySQL database and verifies that the username and password belong to a known user. The password is not stored in plain text in the database. Instead it computes the password hash and compares it to the hash in the database. If the password is wrong then Dovecot will refuse the login.
  3. We have a fixed scheme how emails are stored on disk. So if the user is called jane@example.org then Dovecot will look for email files in /var/vmail/example.org/jane/Maildir/… and send the user the requested emails.

Some users are do not like to install and configure a mail client. They prefer to read email in their browser. So…

What happens if a user reads their email using web mail?

The Roundcube software that provides the web mail interface is basically a PHP software that is a gateway between HTML pages and a built-in IMAP client. So when a user uses their browser to connect to the web mail interface…

ispmail-jessie-bigpicture-webmail

  1. The users points the browser to the HTTPS URL of the webmail interface. Apache web server receives the connection and runs the PHP scripts of Roundcube. Roundcube shows a login form for the username and password. The user enters the username (that equals the email address) and the password and submits the login form.
  2. Roundcube connects to Dovecot using IMAP and forwards the username and password to check if they are valid.
  3. Dovecot treats this connection similar to a mail client connection. It queries the MySQL database to verify the username (=email address) and password.
  4. If the authentication was successful Dovecot fetches the mail files from disk and sends them through IMAP to Roundcube. Roundcube then renders the emails as HTML and the user can read it.

So you see that the web mail access also works through IMAP. The user does not realize that though.

Okay, the final scenario I would like to explain is…

What happens if the user wants to send an email to the internet?

Of course your users want to be able to send email to other internet users. But they cannot send the email directly to the destination mail server. First their mail client does not know which destination server is responsible for the recipient – that functionality just is not built in. And second the user is likely assigned a dynamic IP address which is blocked by most mail servers because they tend to abused by infected Windows PCs that send out spam. So the correct way to send an email to the internet is through your mail servers. This is called relaying because your mail server acts as a relay. In this example your user wants to send an email to fred@example.net.

ispmail-jessie-bigpicture-relaying

  1. The user writes the email in their mail client and clicks on “Send”. The mail client establishes an SMTP connection to your Postfix. To make sure that the user is allowed to send email through your system it requires a username and password along with the email. This information is sent in an encrypted way.
  2. Postfix could now check the password in the database directly. But as Dovecot already knows how to handle authentication it is easier to ask Dovecot to verify the username and password. (SMTP authentication in Postfix is surprisingly ugly otherwise.)
  3. Dovecot now sends a query to the MySQL database to check if the username and password (hash) are correct and tells Postfix the result.
  4. Postix knows now that it is authorized to send the email on behalf of the user. It tells the user that it successfully accepted the email. The email is put into Postfix’s mail queue for further processing. Postfix will now query its nearest DNS (name server) to determine the responsible destination mail server. As the recipient has an “…@example.net” address it checks the MX record of the “example.net” domain and then gets the respective IP address.
  5. Postfix now knows which mail server to send the email to. It opens an SMTP connection and delivers the email.

9 thoughts on “The big picture

  • 2016-01-16 at 15:07
    Permalink

    Hey Christoph,

    where should one report typos? (I’ll do it here if you do not mind.. ;-))

    “Postix knows now” -> “Postfix knows now”

    Reply
  • 2016-02-10 at 14:53
    Permalink

    Thanks for that great description of how e-mail works

    Reply
  • 2016-03-22 at 19:42
    Permalink

    “of the reponsible destintaion mail server is” -> “of the _responsible_ _destination_ mail server is”.

    Great tutorial so far 😉

    Reply
  • 2016-05-04 at 00:43
    Permalink

    Thanks for the great tutorial Christoph 🙂 I was successfully able to setup a mail server! One small (or big) problem is that the SMTP relaying doesn’t seem to be using TLS to encrypt the data in transit. Hence the reason why whenever I send to Gmail the red padlock is shown meaning that my mail server did not encrypt the message using TLS. Is there any way to enable this?

    Reply
    • 2016-05-04 at 00:48
      Permalink

      But the red padlock does not appear when sending (composing) from Gmail to ispmail server and that makes sense because ispmail server has STARTTLS support. But why does it not work the other way around – as in when sending (relaying) from ispmail server to Gmail? Could it be just that Gmail doesn’t support STARTTLS and only traditional TLS?

      Reply
      • 2016-05-04 at 06:38
        Permalink

        In hindsight it was foolish to dismiss Gmail for not having STARTTLS; a simple telnet showed that it does support STARTTLS. I wonder what could be the problem. Is it because of the way ispmail server is set up. I mean we only use TLS certs for each mail server and not for each domain like in some hosting panels.

        Reply
        • 2016-05-04 at 08:41
          Permalink

          Solved the TLS relay problem with this:

          smtp_use_tls = yes
          smtp_tls_security_level = may
          smtp_tls_note_starttls_offer = yes

          Thanks again for the best mail server tut on the internet 🙂

          Reply
  • 2016-10-19 at 21:25
    Permalink

    “The mail server responds with the IP address” -> “The DNS server responds with the IP address”

    Reply
    • 2016-10-19 at 21:30
      Permalink

      Whoops. Thanks for the hint. Fixed.

      Reply

Leave a Reply

Your email address will not be published. Required fields are marked *

By continuing to use the site, you agree to the use of cookies. more information

The cookie settings on this website are set to "allow cookies" to give you the best browsing experience possible. If you continue to use this website without changing your cookie settings or you click "Accept" below then you are consenting to this.

Close