TLS certificate

The internet is not a friendly place where you can trust people. If you send data over the internet there is a pretty good chance someone intercepts it. You don’t want that. Our best weapon against that is transport encryption. All you need is to create an encryption key and a certificate for Postfix (SMTP), Dovecot (IMAP/POP3) and Apache (HTTPS). If you are confused why that requires a key and certificate then please consider reading the Wikipedia about public-key cryptography. The single most important detail here: nobody but you must have access to the private key.

What makes mail clients trust a certificate?

Your client’s operating system contains a list of “trusted” certificate authorities. Sometimes that list is located in a central location for all applications. And sometimes a browser comes with its own list. A certificate signed by any of those organization is fully trusted. Yes, your browser trusts companies you probably have never heard of. And so does your mail client if you use software like Thunderbird instead of using web mail. The tricky part is not the technology. Everyone with some knowledge about certificates can create their own authority. Mathematically your certificates are just as good as anyone else’s. The tricky part is to convince browser manufacturers to trust them and add your certificates to their trust list.

Those organization claim to do checks (in exchange for money) whether you are the righteous owner of a certain domain. These checks have sometimes failed so badly that they popped up in the news. Ten years ago a nerd requested to get trusted by Firefox with his “Honest Achmed’s Used Cars and Certificates” authority. It was fun to watch the developers struggle with arguments why Achmed would be any worse than authorities like Türktrust or Verisign (who failed so hard one must wonder why they are still in business).

In 2012 two Mozilla employees decided to do something about that. They came up with the idea of a non-profit certificate authority. The world needed encryption everywhere and people should no longer shy away from it because getting certificates cost money. Two years later the service was ready and everyone could get a free certificate valid for 90 days. An automated process renewed the certificate in time so there was no hassle. (They surely deserve a donation for the many years of service to the internet community.)

Where is the certificate used?

You guessed it, we will need our own certificate. It will be used in three places:

1. the webmail interface (driven by the Apache web server)
2. Postfix (to encrypt SMTP communication where possible)
3. Dovecot (to encrypt IMAP/POP3 communication and deny any unencrypted traffic)

A single certificate (per domain) can and should be used by all three pieces of software.

Choosing a hostname for your mail server

Your mail server will be able to serve emails in many domains. But a certificate is usually issued for a single fully-qualified domain name like “mail.example.org”. The common name is an important part of the certificate. That attribute must correspond to the host name you use during communication or else the certificate is rejected. Hmmm, many domains but only one name is allowed on a certificate? How do we deal with that?

Option 1: a generic name and a single certificate

The simple solution is to take a generic name. The German ISP Hetzner for example uses the name “mail.your-server.de” for all customers and their domains. And there are many examples of such an approach. Experience shows that most users do not care about the hostname as long as they get their emails. So I would generally recommend this approach due to its simplicity.

Option 2: multiple names/certificates & SNI

Some people however want to give their mail server as many names as they have domains. Suppose that you have three domains: example.org, example.net and example.com. Users of example.org can use mail.example.org while users of example.net use mail.example.net. As you can imagine this would not work if you had a single certificate. After all which name would you use as a common name there? If you used “mail.example.org” then talking to your server by the name “mail.example.net” would lead to a certificate error.

So your mail server needs multiple certificates – one for each host name. And depending on how a user connects to your server you need to send the matching certificate. If the user wants to speak to “mail.example.org” then your server needs to send the certificate for “mail.example.org”. Fortunately that problem has been addressed by a technique called Server Name Indication (SNI). A client talks to the server and even before the encrypted connection is established it tells the server that it expects a certificate for the intended host name. The server can then load the matching certificate and initiate the encryption process.

SNI has long been a problem for mail servers. The mail user agent (e.g. Thunderbird) needs to support it as well as Postfix and Dovecot. Postfix has finally added SNI in version 3.4 so that we can use it.

While adding multiple host names needs extra work, it also has a benefit. If you want to move domains to other servers (e.g. when upgrading your server) you can move one domain at a time.

Preparing the Apache web server for HTTP

Let’s start with the web server. As an example I will assume that you want to offer a host name webmail.example.org to your users. Of course your server will have another name in a domain that you control. I will use that example throughout the tutorial though and keep that name printed in bold letters to remind you that you have to use your own host name.

Do you just want to play around with your new server and not use any real domain yet? No problem. Then use nip.io. If your IP address is 1.2.3.4 then you can use 1.2.3.4.nip.io as a domain name that points to your server. (There used to be another domain xip.io for that purpose but it died in mid-2021.) Another service like that which also supports IPv6 is sslip.io.

If you have an actual domain then set up a DNS “A” and “AAAA” (if you use IPv6) record for that host name pointing to your server’s IP address.

First you need a web root directory for that host name:

mkdir /var/www/webmail.example.org
chown www-data:www-data /var/www/webmail.example.org

Next you need to create a virtual host configuration file. Apache on Debian uses a neat system to manage virtual hosts:

• /etc/apache2/sites-available/*.conf contains the actual configuration files for each virtual host. Putting a file here does not enable that host though. That’s done in the next step. There are two configuration files by default. “000-default.conf” is a HTTP virtual host and “default-ssl.conf” is a HTTPS virtual host.
• /etc/apache2/sites-enabled/*.conf contains symbolic links (“symlinks”) pointing to configuration files in the /etc/apache2/sites-available directory. Only *.conf links in this directory will be loaded by Apache.

This technique allows you to enable and disable virtual hosts without having to destroy any configuration. Debian ships with the “a2ensite” (short for “apache2 enable site”) and “a2dissite” commands. In addition to some sanity checks those commands essentially create or remove symlinks between “sites-available” and “sites-enabled”.

You may remove the default symlinks in /etc/apache2/sites-enabled/* unless you use them already.

Create a new virtual host configuration file /etc/apache2/sites-available/webmail.example.org-http.conf and make it contain:

<VirtualHost *:80>
  ServerName webmail.example.org
  DocumentRoot /var/www/webmail.example.org
</VirtualHost>

The simple configuration makes Apache handle HTTP requests (on the standard TCP port 80) if a certain line in the request header from the browser reads “Host: webmail.example.org”. So the browser actually tells your Apache web server which server name it is looking for. That allows for multiple web sites on a single IP address. (Thanks to Server Name Indication as explained earlier this works well for HTTPS, too.)

Enable the site:

a2ensite webmail.example.org-http

You will be told:

To activate the new configuration, you need to run:
 systemctl reload apache2

Do that.

Let’s check if the configuration works. Put a test file into your web root directory:

echo "Just a test" > /var/www/webmail.example.org/test

Now when you open the URL http://webmail.example.org/test in your browser you should see the text “Just a test”.

This is enough setup to make LetsEncrypt issue a certificate for you.

Getting a LetsEncrypt certificate

Now you can use the certbot tool to request an encryption certificate from LetsEncrypt. What will happen?

• certbot creates a private key and a certificate request. It sends the certificate request to the LetsEncrypt server.
• the LetsEncrypt server replies with a challenge/token.
• certbot puts that token into a file in the /var/www/webmail.example.org/.well-known/acme-challenge directory.
• the LetsEncrypt server does an HTTP connection to http://webmail.example.org/.well-known/acme-challenge/… and expects to find that token. This verifies that you are in charge of the domain and the web server.
• If all works well the LetsEncrypt server signs your certificate request and thus creates the actual certificate.
• certbot receives the certificate and puts it into /etc/letsencrypt/archive/webmail.example.org/

To get a certificate for your domain run:

certbot certonly --webroot --webroot-path /var/www/webmail.example.org -d webmail.example.org

You can use multiple occurences of “-d” here to get a certificate valid for multiple domains. For example: “-d webmail.example.org -d something-else.example.org”. (See also: https://eff-certbot.readthedocs.io/en/stable/using.html#webroot)

The first time you do that you will get asked for your email address so LetsEncrypt can send you reminders if your certificate would expire. You will also have to agree to their terms of service.

If everything worked well you should get output like:

Requesting a certificate for webmail.example.org

Successfully received certificate.
Certificate is saved at: /etc/letsencrypt/live/webmail.example.org/fullchain.pem
Key is saved at:         /etc/letsencrypt/live/webmail.example.org/privkey.pem
This certificate expires on 2024-01-02.
These files will be updated when the certificate renews.
Certbot has set up a scheduled task to automatically renew this certificate in the background.

In /etc/letsencrypt/live/webmail.example.org you will find a couple of files now:

• cert.pem: the certificate file
• chain.pem: the chaining or intermediate certificate. This certificate provides information how the LetsEncrypt certificates are linked to other known certificate authorities. It is generally a good idea to always send this certificate along with your own for clients who may not know LetsEncrypt properly yet.
• fullchain.pem: this file contains a concatenation of the cert.pem and the chain.pem. This is the preferred file to use when a piece of software asks where to find the certificate.
• privkey.pem: the private key file. Keep it secret.

Add HTTPS

Now that you have a valid certificate you can finally enable HTTPS for your web server. Create a new file /etc/apache2/sites-available/webmail.example.org-https.conf containing:

<VirtualHost *:443>
 ServerName webmail.example.org
 DocumentRoot /var/www/webmail.example.org
 SSLEngine on
 SSLCertificateFile /etc/letsencrypt/live/webmail.example.org/fullchain.pem
 SSLCertificateKeyFile /etc/letsencrypt/live/webmail.example.org/privkey.pem
</VirtualHost>

This virtual host configuration looks suspiciously similar to the HTTP virtual host above. It just listens on port 443 (standard port for HTTPS) instead of port 80. And it uses the “SSLEngine” that handles encryption and gets information about the certificate for your web server (that is shown to your users) and the private key (that the web servers uses to decrypt the user’s communication).

Enable the SSL module in Apache:

a2enmod ssl

Then enable the virtual host for HTTPS:

a2ensite webmail.example.org-https

And restart the web server. A reload is not sufficient this time because you added a module.

systemctl restart apache2

Now when you point your web browser to https://webmail.example.org your browser should tell you that it trusts the web site’s certificate:

(Yes, sorry, this is not webmail.example.org. But I do not own the example.org domain and thus cannot get a valid certificate for it. This is my own site.)

So should you keep the HTTP virtual host? Yes. First for the HTTP->HTTPS redirection. And second to keep certbot working.

Redirect HTTP to HTTPS

Sometimes users forget to enter https://… when accessing your webmail service. So they access the HTTP web site. We obviously don’t want them to send their password over HTTP. So we should redirect all HTTP connections to HTTPS.

One exception though. Let’s Encrypt will use HTTP to verify your challenge token. So we need to serve files at http://webmail.example.org/.well-known/acme-challenge/… directly while redirecting all other requests to HTTPS. You can accomplish that by putting these lines inside the <VirtualHost> section of your /etc/apache2/sites-available/webmail.example.org-http.conf file:

RewriteEngine On
RewriteCond %{REQUEST_URI} !.well-known/acme-challenge
RewriteRule ^(.*)$ https://%{SERVER_NAME}$1 [R=301,L]

This requires the rewrite module to be enabled in Apache. That is simple though:

a2enmod rewrite
systemctl restart apache2

So now entering http://webmail.example.org will redirect you to https://webmail.example.org.

Automatic certificate renewal

The certbot package automatically adds a timed job that runs twice a day at random times. The random part is important to avoid millions of server hammering the LetsEncrypt service at the same second.

So the renewal already happens automatically. Should it fail then LetsEncrypt start sending you reminder emails that your certificate should be renewed. That’s a clear sign that something went wrong with the automatic renewal.

There is one puzzle piece missing though. Even if the renewal worked it will only update the certificate files. But the software components – Postfix, Dovecot and Apache – will not notice the change. So we need to add a so called post-hook to certbot that triggers a restart of all processes thereafter.

For that purpose edit the /etc/letsencrypt/cli.ini file and add:

post-hook = systemctl restart postfix dovecot apache2

Well done. You have implemented Let’s Encrypt for all your services now. Let’s go on.