[2019-11-23]

Creating a TLS encryption key and certificate

TLS is SSL

If you are unfamiliar with the abbreviation “TLS“: it is the successor to SSL but works one the same principle. Every new version of TLS deprecates unsafe encryption methods (ciphers) of previous versions.

The internet is the best invention since sliced bread but it has become an evil place more than ever. We are dealing with script kiddies, out-of-control secret services and organized crime on a large scale. It is completely out of the question to run any unencrypted services over the internet. To use encryption you need to create an encryption key and a certificate for Postfix (SMTP), Dovecot (IMAP/POP3) and web mail (HTTPS).

This is called transport encryption. It encrypts data sent over the internet. That regards both the connection between users and mail servers and the communication between mail servers. You cannot rely on it for confidential correspondence because the on the mail server the email is still stored in plain text. But it’s better than nothing.

Fun fact: a while ago a couple of large German freemail providers founded an “email made in Germany” alliance. Guess what cool thing they did to make the world a better place… they enabled transport encryption. Something that should have been best practice anyway. πŸ™‚

Caveat

Do not use different certificates for Postfix and Dovecot. At least the MacOS mail client would refuse your connection with confusing error messages. If you follow this guide then you will be safe.

What makes mail clients trust a certificate?

Just like with your favorite web browser there is a list of “trusted” certificate authorities in the world. Yes, your browser trusts companies you probably have never heard of. And so does your mail client if you use software like Thunderbird instead of using web mail. Companies called certificate authorities managed to convince browser developers to make their browser trust all certificates that were cryptographically signed by them. Those certificates were mathematically as secure as self-signed certificates. They just had a cryptographic signature that was created by a computer of that certificate authority. How does that make such certificates more trusted? Well, those companies promise to verify the identity of the certificate requestor. But let’s face it: they just take your money, maybe check if you own the domain and then sign your certificate. There are even cases of such companies who managed to create certificates incorrectly.

Luckily we live in the future and have LetsEncrypt. They managed to become trusted by all major browsers and operating systems. LetsEncrypt gives you working certificates for your servers at no cost. Don’t take their great service for granted. They are making the world a much better place. Donate already. πŸ™‚

The certificate will be used in three places:

  1. the webmail interface (driven by Apache)
  2. Postfix (for authentication during SMTP communication with your users)
  3. Dovecot (for authentication during IMAP communication)

Choosing a hostname for your mail server

Frequently fellow sysadmins ask how to name the mail server. After all you will be receiving emails on many domains. Do you give the server multiple names?

Okay, okay, I confused you. Sorry. Let’s take an example. You want to receive emails for these domains:

  • gadgets.example.org
  • clothing.example.org
  • pizza.example.org

You could name your mail server gadgets.example.org. But then the users of pizza.example.org may wonder why they talk to a mail server at a completely different domain. If you are thinking of customers they would surely not be happy talking to a mail server of another company.

If you have worked with name-based virtual hosts in a web server then you know that one server can indeed be responsible for many domains. That’s what common web hosting providers do. They put hundreds of domains on one server with a single IP address. That works flawlessly even with encrypted HTTPS. So why don’t we just do it the same way here?

Well, the reason it works for web sites is that both web servers (like Apache) and all modern browsers support a feature called SNI. That is short for server name indication. If your browser connects to a web server using HTTPS it sends (“indicates”) the desired host name even before the encrypted connection has been established. So the web server can pull out the matching certificate for that host name and send it to your browser for the encryption.

In the world of mail servers however we have not yet arrived in the future. Neither Postfix, nor Dovecot, nor your mail clients support SNI. Bummer. Say you use Thunderbird to connect to pizza.example.com but get a certificate for gadgets.example.org. It will fail. Thunderbird will warn you that the certificate does not match the name of the server you entered.

So what is my recommendation? Think of a neutral name that fits all. What about mailservice.example.org? Nobody would mind using such a name.

Keep in mind though that using one common host name for multiple domains makes migrating domains harder. If your users are used to connect to mailservice.example.org then you cannot just migrate one domain to another server because all domains hide behind this host name. This may not be an issue for small installations. But if you are thinking big please keep this in mind.

More security?

If you wish to further enhance security against man-in-the-middle attacks you should get acquainted with DNSSEC and DANE. Also take a look at the smtp_dns_support_level parameter in Postfix.

Preparing the Apache web server for HTTP

Let’s start with the web server. As an example I will assume that you want to offer a host name webmail.example.org to your users. Of course your server will have another name in a domain that you control. I will use that example throughout the tutorial though and keep that name printed in bold letters to remind you that you have to use your own host name everywhere.

Oh, wait. You have set up a DNS “A” and “AAAA” (if you use IPv6) record for that host name pointing to your server’s IP address, right?? Good. Just wanted to make sure.

First you need a web root directory for that host name:

mkdir /var/www/webmail.example.org
chown www-data:www-data /var/www/webmail.example.org

Next you will have to create a virtual host configuration file. Apache on Debian uses a neat system to manage virtual hosts:

  • /etc/apache2/sites-available/*.conf contains two configuration files by default. “000-default.conf” is a HTTP virtual host and “default-ssl.conf” is a HTTPS virtual host configuration. You can create arbitrary files here. Apache will not consider them automatically.
  • /etc/apache2/sites-enabled/*.conf contains symbolic links (“symlinks”) pointing to configuration files in the /etc/apache2/sites-available directory. Only *.conf links in this directory will be loaded by Apache.

This system allows you to enable and disable virtual hosts without having to destroy any configuration. Debian ships with the “a2ensite” (short for “apache2 enable site”) and “a2dissite” commands. In addition to some sanity checks those commands essentially create or remove symlinks between “sites-available” and “sites-enabled”.

*.conf

Apache configuration files must have a “.conf” suffix or else they will get ignored

You may remove the default symlinks in /etc/apache2/sites-enabled/* unless you use them already.

Create a new virtual host configuration file /etc/apache2/sites-available/webmail.example.org-http.conf and make it contain:

<VirtualHost *:80>
  ServerName webmail.example.org
  DocumentRoot /var/www/webmail.example.org
</VirtualHost>

The simple configuration makes Apache handle HTTP requests (on the standard TCP port 80) if a certain line in the request header from the browser reads “Host: webmail.example.org”. So the browser actually tells your Apache web server which server name it is looking for. That allows for multiple web sites on a single IP address. (By the way: this works on HTTPS, too, nowadays, thanks to Server Name Indication as explained earlier.)

Enable the site:

a2ensite webmail.example.org-http

You will be told:

To activate the new configuration, you need to run:
 systemctl reload apache2

Do that.

Let’s check if the configuration works. Put a test file into your web root directory:

echo "Just a test" > /var/www/webmail.example.org/test

Now when you open the URL http://webmail.example.org/test in your browser you should see the text “Just a test”.

This is enough setup to make LetsEncrypt issue a certificate for you.

Getting a LetsEncrypt certificate

Now you can use the certbot tool to request an encryption certificate from LetsEncrypt. What will happen?

  • certbot creates a private key and a certificate request. It sends the certificate request to the LetsEncrypt server.
  • the LetsEncrypt server replies with a challenge/token.
  • certbot puts that token into a file in the /var/www/webmail.example.org/.well-known/acme-challenge directory.
  • the LetsEncrypt server does an HTTP connection to http://webmail.example.org/.well-known/acme-challenge/… and expects to find that token. This verifies that you are in charge of the domain and the web server.
  • If all works well the LetsEncrypt server signs your certificate request and thus creates the actual certificate.
  • certbot receives the certificate and puts it into /etc/letsencrypt/archive/webmail.example.org/

To get a certificate for your domain run:

certbot certonly --webroot --webroot-path /var/www/webmail.example.org -d webmail.example.org

The first time you do that you will get asked for your email address so LetsEncrypt can send you reminders if your certificate would expire. You will also have to agree to their terms of service.

If everything worked well you should get output like:

Saving debug log to /var/log/letsencrypt/letsencrypt.log
Obtaining a new certificate
Performing the following challenges:
http-01 challenge for webmail.example.org
Using the webroot path /var/www/webmail.example.org for all unmatched domains.
Waiting for verification...
Cleaning up challenges
Generating key (2048 bits): /etc/letsencrypt/keys/0049_key-certbot.pem
Creating CSR: /etc/letsencrypt/csr/0049_csr-certbot.pem

IMPORTANT NOTES:
- Congratulations! Your certificate and chain have been saved at
  /etc/letsencrypt/live/webmail.example.org/fullchain.pem. Your
  cert will expire on 2018-04-01. To obtain a new or tweaked version
  of this certificate in the future, simply run certbot again. To
  non-interactively renew *all* of your certificates, run "certbot
  renew"
- If you like Certbot, please consider supporting our work by:

  Donating to ISRG / Let's Encrypt: https://letsencrypt.org/donate
  Donating to EFF: https://eff.org/donate-le

In /etc/letsencrypt/archive/webmail.example.org you will find a couple of files now:

  • cert.pem: the certificate file
  • chain.pem: the chaining or intermediate certificate. This certificate provides information how the LetsEncrypt certificates are linked to other known certificate authorities. It is generally a good idea to always send this certificate along with your own for clients who may not know LetsEncrypt properly yet.
  • fullchain.pem: this file contains a concatenation of the cert.pem and the chain.pem. This is the preferred file to use when a piece of software asks where to find the certificate.
  • privkey.pem: the private key file. Keep it secret.

Expires in 3 months?

LetsEncrypt issues certificates that are valid for 3 months. This is by design to get rid of old certificates quickly. The renewal feature of the certbot will automatically get a fresh certificate after 2 months. So you don’t have to worry about that. And if a renewal failed you will get an email from them reminding you to take a look.

Add HTTPS

Now that you have a valid certificate you can finally enable HTTPS for your web server. Create a new file /etc/apache2/sites-available/webmail.example.org-https.conf containing:

<VirtualHost *:443>
 ServerName webmail.example.org
 DocumentRoot /var/www/webmail.example.org
 SSLEngine on
 SSLCertificateFile /etc/letsencrypt/live/webmail.example.org/fullchain.pem
 SSLCertificateKeyFile /etc/letsencrypt/live/webmail.example.org/privkey.pem
</VirtualHost>

This virtual host configuration looks suspiciously similar to the HTTP virtual host above. It just listens on port 443 (standard port for HTTPS) instead of port 80. It enables the “SSLEngine” that handles encryption and gets information about the certificate for your web server (that is shown to your users) and the private key (that the web servers uses to decrypt the user’s communication).

Enable the SSL module in Apache:

a2enmod ssl

Then enable the virtual host for HTTPS:

a2ensite webmail.example.org-https

And restart the web server. A reload is not sufficient this time because you added a module.

systemctl restart apache2

Now when you point your web browser to https://webmail.example.org your browser should tell you that it trusts the web site’s certificate:

(Yes, sorry, this is not webmail.example.org. But I do not own the example.org domain and thus cannot get a valid certificate for it. This is my own site.)

So should you keep the HTTP virtual host? Yes. Use it to redirect your users to the HTTPS site if they forget to put “https://” in front of the URL. And HTTP is always required for the…

Redirect HTTP to HTTPS

Sometimes users forget to enter https://… when accessing your webmail service. So they access the HTTP web site. We obviously don’t want them to send their password over HTTP. So we should redirect all HTTP connections to HTTPS.

However Let’s Encrypt will always use HTTP to verify your challenge token. So we need to serve files at http://webmail.example.org/.well-known/acme-challenge/… directly while redirecting all other requests to HTTPS. You can accomplish that by putting these lines inside the <VirtualHost> section of your /etc/apache2/sites-available/webmail.example.org-http.conf file:

RewriteEngine On
RewriteCond %{REQUEST_URI} !.well-known/acme-challenge
RewriteRule ^(.*)$ https://%{SERVER_NAME}$1 [R=301,L]

This requires the rewrite module to be enabled in Apache. That is fortunately simple though:

a2enmod rewrite
systemctl restart apache2

Automatic certificate renewal

The certbot package adds an automatic timer job that runs twice a day at random times. The random part is important to avoid millions of server hammering the LetsEncrypt service at the same second.

Systemd timer instead of a Cron job

This job is not a classic Cron job but instead latches into systemd. You can find the timer definition in the /lib/systemd/system/certbot.timer file. That timer triggers the renewal service defined in /lib/systemd/system/certbot.service.

There is also a Cron file at /etc/cron.d/certbot. Don’t be confused. This job will not do anything due to the “-d /run/systemd/system” condition in it that checks if systemd is installed.

So the renewal already happens automatically. Should it fail then LetsEncrypt start sending you reminder emails that your certificate should be renewed. That’s a clear sign that something went wrong with the automatic renewal.

There is one puzzle piece missing though. Even when the renewal worked it will only update the certificate files. But the software components – Postfix, Dovecot and Apache – will not notice the change. So we need to add a so called post-hook to certbot that triggers a restart of all processes thereafter.

For that purpose create/edit the /etc/letsencrypt/cli.ini file and add:

post-hook = systemctl restart postfix ; systemctl restart dovecot ; systemctl restart apache2

Well done. You have implemented LetsEncrypt for all your services now. Let’s go on.

40 thoughts on “Creating a TLS encryption key and certificate

  • 2019-12-27 at 23:52
    Permalink

    Preparing the Apache web server for HTTP
    […]
    OOPS chown www-data.www-data mkdir /var/www/webmail.example.org
    I think you meant
    chown www-data:www-data /var/www/webmail.example.org

    Reply
    • 2019-12-28 at 00:40
      Permalink

      Thanks. Oh, my. You can proof-read your pages a dozen times and still don’t see such issues.

      Reply
      • 2019-12-28 at 19:54
        Permalink

        More than happy to help! Just realized getting certbot going that you probably want to chown -R as well?

        Reply
      • 2020-01-28 at 19:29
        Permalink

        The chown command still contains the dot.

        Reply
  • 2019-12-29 at 01:20
    Permalink

    Found another error in the Redirect to HTTPS section. The rewrite code needs to go in the webmail.example.org-http.conf file.

    Reply
    • 2019-12-29 at 11:30
      Permalink

      That’s right, thanks for the hint.

      Reply
  • 2019-12-30 at 17:22
    Permalink

    Hi Christoph,

    Why not using modern commands, such as “systemctl restart xxxx” ? (well, let’s get ready for the future !! πŸ™‚ )

    Reply
    • 2019-12-31 at 11:13
      Permalink

      Ah, snap. I changed it everywhere but apparently forgot that one spot. πŸ™‚

      Reply
  • 2019-12-30 at 20:21
    Permalink

    Thanks so much for your precious work, you rock! πŸ™‚
    I spotted a small mistake on the page – it should be

    RewriteRule ^(.*)$ https://%{SERVER_NAME}/$1 [R=301,L]

    so all URLs are redirected to their HTTPS variant – otherwise the redirect always ends up at the main site (i.e. no subdirectories are redirected).

    Reply
    • 2019-12-31 at 11:31
      Permalink

      Thanks, you are right. Fixed.

      Reply
      • 2019-12-31 at 14:27
        Permalink

        I think I made a small mistake myself – it should be

        RewriteRule ^(.*)$ https://%{SERVER_NAME}$1 [R=301,L]

        (without the “/” before “$1”).

        Reply
        • 2020-01-01 at 11:12
          Permalink

          True. The $1 contains the leading slash already.

          Reply
  • 2019-12-31 at 00:53
    Permalink

    I thought both Postfix and Dovecot support SNI already. Postfix since 3.4.0 and Dovecot since, I don’t know. And Thunderbird for example supports SNI too.

    Reply
      • 2019-12-31 at 14:31
        Permalink

        http://www.postfix.org/announcements/postfix-3.4.0.html claims that

        “SNI (server name indication) support in the Postfix SMTP server, the Postfix SMTP client, and in the tlsproxy(8) daemon (both server and client roles).”

        but didn’t try myself yet. Another option, although that means revealing all other hosted domains and does not scale well for too many domains, is to create a certificate with several domains. acme.sh supports that at least, see https://github.com/Neilpang/acme.sh/wiki/How-to-issue-a-cert at item #2

        Reply
        • 2020-01-01 at 11:14
          Permalink

          Interesting. I wonder how it’s supposed to work. The documentation still doesn’t seem to describe that.

          Reply
          • 2020-03-11 at 11:51
            Permalink

            For dovecot:

            # default
            ssl_cert = </etc/letsencrypt/live/example.com/fullchain.pem
            ssl_key = </etc/letsencrypt/live/example.com/privkey.pem

            # per domain
            local_name example.com {
            ssl_cert = </etc/letsencrypt/live/example.com/fullchain.pem
            ssl_key = </etc/letsencrypt/live/example.com/privkey.pem
            }

            local_name example.org {
            ssl_cert = </etc/letsencrypt/live/example.org/fullchain.pem
            ssl_key = postmap -F hash:/etc/postfix/vmail_ssl.map

            And enable it in main.cf:
            smtpd_tls_chain_files =
            /etc/letsencrypt/live/example.com/privkey.pem,
            /etc/letsencrypt/live/example.com/fullchain.pem

            tls_server_sni_maps = hash:/etc/postfix/vmail_ssl.map

            That’s all, now both dovecot and postfix will deliver the right certificates when you connect, depending on the domain you’re using.

          • 2020-03-11 at 11:58
            Permalink

            Sorry, I don’t know what’s happened. A part of my comment is missing before the “postmap”.
            The contents of the mapping file for postfix (/etc/postfix/vmail_ssl.map) should look like this:

            example.com /etc/letsencrypt/live/example.com/privkey.pem /etc/letsencrypt/live/example.com/fullchain.pem
            example.org /etc/letsencrypt/live/example.org/privkey.pem /etc/letsencrypt/live/example.org/fullchain.pem

            One line per domain. You need to compile it:
            postmap -F hash:/etc/postfix/vmail_ssl.map
            And can then use it in your main.cf as shown.

  • 2020-01-03 at 13:52
    Permalink

    Is there a reason to use mod_rewrite to force the https connection?
    I just use
    Redirect permanent / https://webmail.example.org
    in the section.
    That seems to work fine for me, and as there are often many ways to the same goal, I wonder if one way is to prefer over the other

    Reply
  • 2020-01-06 at 16:38
    Permalink

    Should the .well-known/acme-challenge be created with group (www-data) permission? Or how exactly certbot should be able to write it on web-root?

    Reply
    • 2020-01-07 at 08:01
      Permalink

      By default the certbot service runs as root with read permissions for www-data.

      Reply
      • 2020-01-07 at 10:25
        Permalink

        Certbot did not create the acme-challenge. What might have gone wrong? Any ideas?

        Reply
        • 2020-01-07 at 10:29
          Permalink

          What do you get if you run “certbot renew” as root?

          Reply
          • 2020-01-07 at 21:30
            Permalink

            I hit the limit. I guess I have to wait wait a week (or 6 days).

  • 2020-01-13 at 17:41
    Permalink

    I got it working. I think my misunderstanding…

    I was accessing https with IP address, not with FQDN. Now it is working.

    Reply
  • 2020-01-13 at 17:53
    Permalink

    You write: After renewal Postfix, Dovecot and Apache – will not notice the change.

    For that purpose edit the /etc/letsencrypt/cli.ini file

    There is no cli.ini there. There are directories like renewal-hooks and more, but should I create a cli.ini file and place it under /etc/letsencrypt/?

    Reply
    • 2020-01-14 at 12:25
      Permalink

      Yes, please create the file from scratch. You are right, the file is not necessarily there by default.

      Reply
  • 2020-01-13 at 18:21
    Permalink

    I’m using your method for an internal (intranet) mail server. AFAIK, LetsEncrypt won’t generate keys for a non-public server. So I’ve resorted to a self-signed certificate from your Jessie version. https://workaround.org/ispmail/jessie/create-certificate

    Is this still the accepted way to enable SSL/TLS for a non-public mail server? Might still be worth mentioning for non-public servers.

    Thanks for the great tutorial.

    Reply
    • 2020-01-14 at 12:30
      Permalink

      That’s a bit tricky for internal servers.

      Option 1: Create an internal CA and distribute its certificate to all workstations.

      Option 2: Set a DNS entry for your intranet domain (if you own the domain) and pass through HTTP requests for /.well-known/acme-challenge/… to the internal server. I have done that once with Nginx as a reverse proxy. But it’s an advanced hack. πŸ™‚

      Reply
      • 2020-05-18 at 20:52
        Permalink

        If you own the DNS entry for the Intranet domain, the Intranet domain exists on the Internet, and the DNS server accepts dynamic updates, you could also use DNS-01 challenge mode. Let’s Encrypt doesn’t care if the host has an Internet accessible A record as long as it can validate _acme-challenge.hostname.example.org.

        I’ve got all my internal servers configured with Let’s Encrypt certificates but none of them have valid Internet A records. I used dynamic updates with BIND, but acme.sh (and, I assume, Certbot) have a long list of plugins that they can use against DNS service providers.

        Reply
  • 2020-01-15 at 17:27
    Permalink

    Hi Christoph,

    I have a Host mail service.example.org. And I would like to name the webmail service (roundcube) as webmail.example.org.

    Should I sign for service.example.org or webmail.example.org? Or should I have two different certs for both services?

    Reply
    • 2020-01-17 at 06:54
      Permalink

      Hmm,

      Is it autocorrect or is it just me writing badly…

      I have a host named mailservice.example.org since it does not look bad and You started with that naming philosophy on Your guide. I would like to have a webmail user interface and would like to use virtual host name webmail.example.org.

      Letsencrypt has signed me a certificate with the name mailservice.example.org and it is working. But, could You please tell me a little bit more why did You used different name for the cert than the initial mailservice? And how should this work? Having a hostname signed and a virtual host signed with different host name?

      Reply
      • 2020-01-20 at 12:13
        Permalink

        Bear with me, english is not my native language either. πŸ™‚ Did I mix up anything? I tried to use “webmail.example.org” everywhere in the guide.

        Technically you could use several names. The only exception being that the IMAP (Dovecot) and SMTP (Postfix) certificates should be the same.

        Reply
        • 2020-01-20 at 12:55
          Permalink

          You are writing on this document that “Nobody would mind using such a name” and You were referring to mailservice.example.org. This make sense for me too. And until this current page You were referring to mailservice.example.org elsewhere.

          How should I state the certbot command? Is there a typo or something? Currently it says that webmail.example.org and if using name like mailservice.example.org for postfix and dovecot and a second name for webmail.example.org I do run into problems.

          If I understood correctly it should be a wildcard certificate…

          Reply
          • 2020-01-20 at 21:14
            Permalink

            I think I might solved my problem:

            certbot certonly –webroot –webroot-path /var/www/mailservice.example.org -d mailservice.example.org -d webmail.example.org

            But one question still arise: Could it make sense to have hostname as smtp.example.org and let that be FQDN and a CNAME for Dovecot like mail.example.org and finally a CNAME webmail.example.org for the Roundcube?

            Am I just an old school guy… How do You see it? Should a SMTP service be named smtp.example.org and not anything else?

          • 2020-01-22 at 08:58
            Permalink

            Yes, you can use different FQDNs for the different services. Might be helpful if you later plan to seperate the servers for each service. I just learned painfully that MacOS gets bitchy if the TLS certificates for SMTP and IMAP are different. Just keep that in mind and test it before going into production.

  • 2020-05-15 at 12:30
    Permalink

    Can I perhaps circumvent the whole SNI problem by generating a certificiate that contains all the domains? Something like this:
    certbot –webroot /var/www/default -d mydomain.com -d otherdomain.com …

    That would solve all the problems, as far as I understand.

    Reply
  • 2020-05-22 at 12:32
    Permalink

    As always great guide! I move to Buster exactly 2 months (and a day ago) and today found my mail clients complaining about expired certificates. A simple dovecot restart solved this. Digging further, trying a certbot renew –dry-run I noticed that only postfix was being restarting. Looking on the net i found that changing the line in /etc/letsencrypt/cli.ini from post-hook = systemctl restart postfix ; systemctl restart dovecot ; systemctl restart apache2 to post-hook = systemctl restart postfix dovecot apache2 solved the problem (at least –dry-run indicates it will) for me.

    Reply

Leave a Reply to John Bergquist Cancel reply

Your email address will not be published. Required fields are marked *