TLS is SSL
The internet is not a friendly place where you can trust people. If you send data over the internet there is a pretty good chance someone intercepts it. You don’t want that. Our best weapon against that is transport encryption. All you need is to create an encryption key and a certificate for Postfix (SMTP), Dovecot (IMAP/POP3) and Apache (HTTPS). If you are confused why that requires a key and certificate then please consider reading the Wikipedia about public-key cryptography. The single most important detail here: nobody but you must have access to the private key. Ever.
Fun fact: German freemail providers founded an “email made in Germany” alliance. Guess what cool thing they did to make the world a better place… they enabled transport encryption. Something that is practice anyway. Don’t we all love marketing bullshit.
Caveat
What makes mail clients trust a certificate?
Your web browser contains a list of “trusted” certificate authorities. Any certificate signed by any of those organization is fully trusted. Yes, your browser trusts companies you probably have never heard of. And so does your mail client if you use software like Thunderbird instead of using web mail. The tricky part is not the technology. Everyone with some knowledge about certificates can create their own authority. Mathematically your certificates are just as good as anyone else’s. The tricky part is to convince browser manufacturers to trust them and add you to their trust list. That is something those organization managed to accomplish. Does it make you suspicious that those organizations are profit-oriented? What is more important to them: actual security or money?
Those organization claim to do checks whether you are the righteous owner of a certain domain. These checks have sometimes failed so badly that they popped up in the news. Ten years ago a nerd requested to get trusted by Firefox with his “Honest Achmed’s Used Cars and Certificates” authority. It was fun to watch the developers struggle with arguments why Achmed would be any worse than authorities like Türktrust or Verisign (who failed so hard one must wonder why they are still in business).
Also ten years ago two Mozilla employees decided to do something about that. They came up with the idea of a non-profit certificate authority. The world needed encryption everywhere and people should no longer shy away from it because getting certificates cost money. Two years later the service was ready and everyone could get a free certificate valid for 90 days. An automated process renewed the certificate in time so there was no hassle. (They surely deserve a donation for the many years of service to the internet community.)
Where is the certificate used?
The certificate will be used in three places:
- the webmail interface (driven by the Apache web server)
- Postfix (to encrypt SMTP communication where possible)
- Dovecot (to encrypt IMAP/POP3 communication and deny any unencrypted attempts)
One certificate (per domain) can and should be used by all three pieces of software.
Choosing a hostname for your mail server
Your mail server will be able to serve emails in many domains. But a certificate is usually issued for a single fully-qualified domain name like “mail.example.org”. The common name is an important part of the certificate. That attribute must correspond to the host name you use during communication or else the certificate is rejected. Hmmm, many domains but only one name on the certificate. How do we fix that?
Option 1: a generic name and a single certificate
The simple solution is to take a generic name. The German ISP Hetzner for example uses the name “mail.your-server.de” for all customers and their domains. And there are many examples of such an approach. Experience shows that most users do not care about the hostname as long as they get their emails. So I would generally recommend this approach.
Option 2: multiple names/certificates & SNI
Some people however want to give their mail server as many names as they have domains. Suppose that you have three domains: example.org, example.net and example.com. Users of example.org can use mail.example.org while users of example.net use mail.example.net. As you can imagine this would not work if you had a single certificate. After all which name would you use as a common name there? If you used “mail.example.org” then talking to your server by the name “mail.example.net” would lead to a certificate error.
So your mail server needs multiple certificates – one for each host name. And depending on how a user connects to your server you need to send the matching certificate. If the user wants to speak to “mail.example.org” then your server needs to send the certificate for “mail.example.org”. Fortunately that problem has been addressed by a technique called Server Name Indication (SNI). A client talks to the server and even before the encrypted connection is established it tells the server that it expects a certificate for the intended host name. The server can then load the matching certificate and initiate the encryption process.
SNI has long been a problem for mail servers. The mail user agent (e.g. Thunderbird) needs to support it as well as Postfix and Dovecot. Postfix has finally added SNI in version 3.4 so that we can use it.
While adding multiple host names needs extra work, it also has a benefit. If you want to move domains to other servers (e.g. when upgrading your server) you can move one domain at a time.
More security?
Preparing the Apache web server for HTTP
Let’s start with the web server. As an example I will assume that you want to offer a host name webmail.example.org to your users. Of course your server will have another name in a domain that you control. I will use that example throughout the tutorial though and keep that name printed in bold letters to remind you that you have to use your own host name.
Do you just want to play around with your new server and not use any real domain yet? No problem. Then use nip.io. If your IP address is 1.2.3.4 then you can use 1.2.3.4.nip.io as a domain name that points to your server. (There used to be another domain xip.io for that purpose but it died in mid-2021.) Another service like that which also supports IPv6 is sslip.io.
If you have an actual domain then set up a DNS “A” and “AAAA” (if you use IPv6) record for that host name pointing to your server’s IP address.
First you need a web root directory for that host name:
mkdir /var/www/webmail.example.org chown www-data:www-data /var/www/webmail.example.org
Next you need to create a virtual host configuration file. Apache on Debian uses a neat system to manage virtual hosts:
/etc/apache2/sites-available/*.confcontains the actual configuration files for each virtual host. Putting a file here does not enable that host though. That’s done in the next step. There are two configuration files by default. “000-default.conf” is a HTTP virtual host and “default-ssl.conf” is a HTTPS virtual host./etc/apache2/sites-enabled/*.confcontains symbolic links (“symlinks”) pointing to configuration files in the /etc/apache2/sites-available directory. Only *.conf links in this directory will be loaded by Apache.
This technique allows you to enable and disable virtual hosts without having to destroy any configuration. Debian ships with the “a2ensite” (short for “apache2 enable site”) and “a2dissite” commands. In addition to some sanity checks those commands essentially create or remove symlinks between “sites-available” and “sites-enabled”.
*.conf
You may remove the default symlinks in /etc/apache2/sites-enabled/* unless you use them already.
Create a new virtual host configuration file /etc/apache2/sites-available/webmail.example.org-http.conf and make it contain:
<VirtualHost *:80> ServerName webmail.example.org DocumentRoot /var/www/webmail.example.org </VirtualHost>
The simple configuration makes Apache handle HTTP requests (on the standard TCP port 80) if a certain line in the request header from the browser reads “Host: webmail.example.org”. So the browser actually tells your Apache web server which server name it is looking for. That allows for multiple web sites on a single IP address. (Thanks to Server Name Indication as explained earlier this works well for HTTPS, too.)
Enable the site:
a2ensite webmail.example.org-http
You will be told:
To activate the new configuration, you need to run: systemctl reload apache2
Do that.
Let’s check if the configuration works. Put a test file into your web root directory:
echo "Just a test" > /var/www/webmail.example.org/test
Now when you open the URL http://webmail.example.org/test in your browser you should see the text “Just a test”.
This is enough setup to make LetsEncrypt issue a certificate for you.
Getting a LetsEncrypt certificate
Now you can use the certbot tool to request an encryption certificate from LetsEncrypt. What will happen?
- certbot creates a private key and a certificate request. It sends the certificate request to the LetsEncrypt server.
- the LetsEncrypt server replies with a challenge/token.
- certbot puts that token into a file in the /var/www/webmail.example.org/.well-known/acme-challenge directory.
- the LetsEncrypt server does an HTTP connection to http://webmail.example.org/.well-known/acme-challenge/… and expects to find that token. This verifies that you are in charge of the domain and the web server.
- If all works well the LetsEncrypt server signs your certificate request and thus creates the actual certificate.
- certbot receives the certificate and puts it into /etc/letsencrypt/archive/webmail.example.org/
To get a certificate for your domain run:
certbot certonly --webroot --webroot-path /var/www/webmail.example.org -d webmail.example.org
The first time you do that you will get asked for your email address so LetsEncrypt can send you reminders if your certificate would expire. You will also have to agree to their terms of service.
If everything worked well you should get output like:
Saving debug log to /var/log/letsencrypt/letsencrypt.log Obtaining a new certificate Performing the following challenges: http-01 challenge for webmail.example.org Using the webroot path /var/www/webmail.example.org for all unmatched domains. Waiting for verification... Cleaning up challenges Generating key (2048 bits): /etc/letsencrypt/keys/0049_key-certbot.pem Creating CSR: /etc/letsencrypt/csr/0049_csr-certbot.pem IMPORTANT NOTES: - Congratulations! Your certificate and chain have been saved at /etc/letsencrypt/live/webmail.example.org/fullchain.pem. Your cert will expire on 2022-01-22. To obtain a new or tweaked version of this certificate in the future, simply run certbot again. To non-interactively renew *all* of your certificates, run "certbot renew" - If you like Certbot, please consider supporting our work by: Donating to ISRG / Let's Encrypt: https://letsencrypt.org/donate Donating to EFF: https://eff.org/donate-le
In /etc/letsencrypt/live/webmail.example.org you will find a couple of files now:
- cert.pem: the certificate file
- chain.pem: the chaining or intermediate certificate. This certificate provides information how the LetsEncrypt certificates are linked to other known certificate authorities. It is generally a good idea to always send this certificate along with your own for clients who may not know LetsEncrypt properly yet.
- fullchain.pem: this file contains a concatenation of the cert.pem and the chain.pem. This is the preferred file to use when a piece of software asks where to find the certificate.
- privkey.pem: the private key file. Keep it secret.
Expires in 3 months?
Add HTTPS
Now that you have a valid certificate you can finally enable HTTPS for your web server. Create a new file /etc/apache2/sites-available/webmail.example.org-https.conf containing:
<VirtualHost *:443> ServerName webmail.example.org DocumentRoot /var/www/webmail.example.org SSLEngine on SSLCertificateFile /etc/letsencrypt/live/webmail.example.org/fullchain.pem SSLCertificateKeyFile /etc/letsencrypt/live/webmail.example.org/privkey.pem </VirtualHost>
This virtual host configuration looks suspiciously similar to the HTTP virtual host above. It just listens on port 443 (standard port for HTTPS) instead of port 80. And it uses the “SSLEngine” that handles encryption and gets information about the certificate for your web server (that is shown to your users) and the private key (that the web servers uses to decrypt the user’s communication).
Enable the SSL module in Apache:
a2enmod ssl
Then enable the virtual host for HTTPS:
a2ensite webmail.example.org-https
And restart the web server. A reload is not sufficient this time because you added a module.
systemctl restart apache2
Now when you point your web browser to https://webmail.example.org your browser should tell you that it trusts the web site’s certificate:
(Yes, sorry, this is not webmail.example.org. But I do not own the example.org domain and thus cannot get a valid certificate for it. This is my own site.)
So should you keep the HTTP virtual host? Yes. First for the HTTP->HTTPS redirection. And second to keep certbot working.
Redirect HTTP to HTTPS
Sometimes users forget to enter https://… when accessing your webmail service. So they access the HTTP web site. We obviously don’t want them to send their password over HTTP. So we should redirect all HTTP connections to HTTPS.
One exception though. Let’s Encrypt will use HTTP to verify your challenge token. So we need to serve files at http://webmail.example.org/.well-known/acme-challenge/… directly while redirecting all other requests to HTTPS. You can accomplish that by putting these lines inside the <VirtualHost> section of your /etc/apache2/sites-available/webmail.example.org-http.conf file:
RewriteEngine On
RewriteCond %{REQUEST_URI} !.well-known/acme-challenge
RewriteRule ^(.*)$ https://%{SERVER_NAME}$1 [R=301,L]
This requires the rewrite module to be enabled in Apache. That is simple though:
a2enmod rewrite systemctl restart apache2
So now entering http://webmail.example.org will redirect you to https://webmail.example.org.
Automatic certificate renewal
The certbot package automatically adds a timed job that runs twice a day at random times. The random part is important to avoid millions of server hammering the LetsEncrypt service at the same second.
Systemd timer instead of a Cron job
/lib/systemd/system/certbot.timer file. That timer triggers the renewal service defined in /lib/systemd/system/certbot.service.There is also a Cron file at /etc/cron.d/certbot. Don’t be confused. This job will not do anything due to the “
-d /run/systemd/system” condition that checks if systemd is installed. And Debian nowadays uses systemd.So the renewal already happens automatically. Should it fail then LetsEncrypt start sending you reminder emails that your certificate should be renewed. That’s a clear sign that something went wrong with the automatic renewal.
There is one puzzle piece missing though. Even when the renewal worked it will only update the certificate files. But the software components – Postfix, Dovecot and Apache – will not notice the change. So we need to add a so called post-hook to certbot that triggers a restart of all processes thereafter.
For that purpose create/edit the /etc/letsencrypt/cli.ini file and add:
post-hook = systemctl restart postfix dovecot apache2
Well done. You have implemented Let’s Encrypt for all your services now. Let’s go on.
Hi Christoph – in response to:
“Do you just want to play around with your new server and not use any real domain yet? No problem. Then use nip.io. If your IP address is 1.2.3.4 then you can use 1.2.3.4.nip.io as a domain name that points to your server.”
I cannot get this to work with letsencrypt – where you have webmail.example.org, I’ve been using webmail.1.2.3.4.nip.io. When I use that, letsencrypt returns an error: no valid A records for webmail.1.2.3.4.nip.io, which isn’t surprising. So I can use webmail.1.2.3.4.nip.io for all unencrypted pieces and testing, but success stops there, and the deeper I get into your model (I’ve completed “Testing email delivery”), the more this restrictive this limitation becomes.
As always, thanks for putting this together, and if you have any suggestions for getting the parts requiring encryption to work without having a public domain to use, all the more thanks! 🙂
Hi Brad. That’s very unusual. Just tried it:
» host webmail.1.2.3.4.nip.io
webmail.1.2.3.4.nip.io has address 1.2.3.4
I’m stunned why Let’s Encrypt gives you that error. Did they perhaps have a temporary DNS problem?
Just tried that:
# certbot certonly -d hansolo.49.12.224.230.nip.io
How would you like to authenticate with the ACME CA?
– – – – – – – – – – – – – – – – – – – – – – – – – – – – – – – – – – – – – – – –
1: Obtain certificates using a DNS TXT record (if you are using Hetzner for
DNS). (dns-hetzner)
2: Spin up a temporary webserver (standalone)
3: Place files in webroot directory (webroot)
– – – – – – – – – – – – – – – – – – – – – – – – – – – – – – – – – – – – – – – –
Select the appropriate number [1-3] then [enter] (press ‘c’ to cancel): 2
Plugins selected: Authenticator standalone, Installer None
Requesting a certificate for hansolo.49.12.224.230.nip.io
Performing the following challenges:
http-01 challenge for hansolo.49.12.224.230.nip.io
Waiting for verification…
Cleaning up challenges
IMPORTANT NOTES:
– Congratulations! Your certificate and chain have been saved at:
/etc/letsencrypt/live/hansolo.49.12.224.230.nip.io/fullchain.pem
Thanks for the quick response! I get the same as you with the host command – it works fine. The certbot command in your reply…
# certbot certonly -d hansolo.49.12.224.230.nip.io
… doesn’t include the ‘–webroot’ or ‘–webroot path’ portions as seen in the tutorial…
# certbot certonly –webroot –webroot-path /var/www/webmail.example.org -d webmail.example.org
All of my attempts have been using:
# certbot certonly –webroot –webroot-path /var/www/webmail.1.2.3.4.nip.io -d webmail.1.2.3.4.nip.io
Should I try without one or both of those in the command?
There are different ways to prove to Let’s Encrypt that you are in charge of that DNS record and IP address. The way I used requires you to stop Apache and verify the new certificates. I personally prefer the way with –webroot.
However both ways should work. And it proves that webmail.1.2.3.4.nip.io should work. So either Let’s Encrypt had a hickup or you may have made a mistake in the process. Have you tried again? If the problem is the same can you please elaborate what you entered and what error you got exactly?
I’ve tried many times over many days. 🙂 Here is the entire exchange for the certbot command I am trying to use:
# certbot certonly –webroot –webroot-path /var/www/webmail.10.0.0.151.nip.io -d webmail.10.0.0.151.nip.io
Saving debug log to /var/log/letsencrypt/letsencrypt.log
Plugins selected: Authenticator webroot, Installer None
Requesting a certificate for webmail.10.0.0.151.nip.io
Performing the following challenges:
http-01 challenge for webmail.10.0.0.151.nip.io
Using the webroot path /var/www/webmail.10.0.0.151.nip.io for all unmatched domains.
Waiting for verification…
Challenge failed for domain webmail.10.0.0.151.nip.io
http-01 challenge for webmail.10.0.0.151.nip.io
Cleaning up challenges
Some challenges have failed.
IMPORTANT NOTES:
– The following errors were reported by the server:
Domain: webmail.10.0.0.151.nip.io
Type: dns
Detail: no valid A records found for webmail.10.0.0.151.nip.io; no
valid AAAA records found for webmail.10.0.0.151.nip.io
There’s your problem. The IP address 10.x.x.x is a private IP address that is not routable in the public internet. Let’s Encrypt can you reach you on that address.
Referencing an earlier conversation for which the Reply button stopped appearing…
Christoph said:
“There’s your problem. The IP address 10.x.x.x is a private IP address that is not routable in the public internet. Let’s Encrypt can you reach you on that address.”
And that makes sense, but is that where we leave it? For those of us just “playing around” with the concepts here and trying to learn, is there a functional alternative to enabling and testing encrypted communication on the server? I suppose I could build my own CA and generate my own certs, good for internal testing only. Are there any other ideas out there?
Well, no public CA can/will give you a valid certificate for a system on a private IP address. So, yes, your own CA would work. But if you are just playing around then any self-signed certificate should work.
I was able to use the server’s self-signed certificate for this. I’ve also reconfigured my setup to use a “normal” site name without the nip.io address involved. Without a valid DNS entry, it was necessary to add webmail and webmail.example.org to the server’s hosts file.
This has been a great learning experience. Your efforts here in building the site, and taking time to check in here and reply are very much appreciated!
I’ve been unable to create a virtual host for the webmail subdomain. It just refers back to my main site. It may have something to do with me hosting a WordPress site on the same server, but I’ve tried all day to get it working.
Hello Christoph,
why not using a wildcard-certificate from Let’s Encrypt?
Only thing to do is to attach the requested domains to the -d parm
like so: … -d example.com, *.example.com (mind the order of entries, first gives name of cert)
With this you get an allrounder that works with subdomains and the domain standing alone.
For me it worked with certonly and omitting the webroot parms.
Then one has to implement the certificate stuff into the various vhosts/nginxs.
Here a vhost as an example:
ServerName sub.example.com
…
SSLEngine On
SSLCertificateFile /etc/letsencrypt/live/example.com/fullchain.pem
SSLCertificateKeyFile /etc/letsencrypt/live/example.com/privkey.pem
…
Regards
Güner
small addition:
not sure but I think I got it from here (…alles nur geklaut 🙂 … )
https://websiteforstudents.com/generate-free-wildcard-certificates-using-lets-encrypt-certbot-on-ubuntu-18-04/
This is probably fine if the top-level domain is your’s alone *and* the entire domain is hosted on a single machine, but I can foresee trouble if that isn’t the case.
Hey, wildcard can work(does work) however you should be aware that certbot cannot autorenew wildcard certificates easily. while there are plugins for automatic renewal for say apache etc, those will not work on a wild-card cert(at least they didnt a year ago, if things have changed in the mean time someone please add your input here)
Sure many will know this, but it stumped me for a LONG time!!
If you “su” in ssh, to run a2ensite you need to “su -” instead
I was having an issue where the renewed certificate was not being served by Postfix even after a restart. (Dovecot and Apache were working correctly.)
This manifested in Roundcube and Thunderbird being unable to send mail, and log messages like this:
Sep 15 10:17:55 mx postfix/submission/smtpd[2255]: SSL_accept error from unknown[x.x.x.x]: -1
Sep 15 10:17:55 mx postfix/submission/smtpd[2255]: warning: TLS library problem: error:0A000415:SSL routines::sslv3 alert certificate expired:../ssl/record/rec_layer_s3.c:1584:SSL alert number 45:
S
Essentially certbot had successfully renewed the certificates but the old certificates were still being used by Postfix. The old certificates would continue working for another few weeks until they expired, at which point anyone verifying the TLS certificates (pretty much every client by default) would refuse to connect.
The fix was to modify the post-hook to run postmap on the sni.map file:
post-hook = postmap -F hash:/etc/postfix/sni.map; systemctl restart postfix dovecot apache2
Presumably this is only required for systems that are using SNI to serve multiple certificates.
If you want your hook to run only after a successful renewal, use –deploy-hook
question: any reason you go with manual verification instead of using certbot –apache (after installing the apache plugin for it in the previous step)?
This will not only do the verification automatically by reading in configured ServerName/ServerAliases from enabled configurations in apache2 but also configure their SSL versions and the ssl redirect from non-ssl connection attempts automatically, thus greatly speeding up the process.
Also, how do you get certbot to automagically add the cronjob? it does not do that for me, I had to add the 0 0 1 * * certbot renew && systemctl restart dovecot && systemctl restart postfix
to my crontab manually
Hey Christoph,
amazing tutorial as always, thank you so much for all the hard work! I’ve been following it for years and am just setting up a new Debian Bullseye machine and go through everything step by step to make sure I’m up to date with the latest and greatest. 😉
In the HTTP to HTTPS Redirect chapter, I’m pretty sure you don’t need to exclude the .well-known directories from the redirect. I have multiple hosts with an unconditional HTTP to HTTPS redirect and Let’s Encrypt/Certbot has been renewing them fine for years.
So I guess you could omit the entire paragraph here:
Begin <<<<>>>> End
And then just omit the RewriteCond:
Begin <<<<<
You can accomplish that by putting these lines inside the section of your /etc/apache2/sites-available/webmail.example.org-http.conf file:
RewriteEngine On
RewriteRule ^(.*)$ https://%{SERVER_NAME}$1 [R=301,L]
>>>>> End
Huh, the comment system ate one of my remarks. 😉
So, just to be clear, this paragraph be omitted, as far as I’m aware:
—–
One exception though. Let’s Encrypt will use HTTP to verify your challenge token. So we need to serve files at http://webmail.example.org/.well-known/acme-challenge/… directly while redirecting all other requests to HTTPS.
—–
So that leaves us with:
—–
You can accomplish that by putting these lines inside the section of your /etc/apache2/sites-available/webmail.example.org-http.conf file:
RewriteEngine On
RewriteRule ^(.*)$ https://%{SERVER_NAME}$1 [R=301,L]
—–
Hi Christoph,
I keep getting the below error when trying to get a certificate for my domain (XYZ.com) by running the below command:
sudo certbot certonly –webroot –webroot-path /var/www/XYZ.com -d XYZ.com
Saving debug log to /var/log/letsencrypt/letsencrypt.log
Plugins selected: Authenticator webroot, Installer None
Requesting a certificate for XYZ.com
Performing the following challenges:
http-01 challenge for XYZ.com
Using the webroot path /var/www/XYZ.com for all unmatched domains.
Waiting for verification…
Challenge failed for domain XYZ.com
http-01 challenge for XYZ.com
Cleaning up challenges
Some challenges have failed.
IMPORTANT NOTES:
– The following errors were reported by the server:
Domain: XYZ.com
Type: unauthorized
Detail: X.X.X.X: Invalid response from
http://XYZ.com/.well-known/acme-challenge/5z_II_aZ3XgbftBGqL0mVVmebKBiLOQhLzLS_AHCny4:
404
To fix these errors, please make sure that your domain name was
entered correctly and the DNS A/AAAA record(s) for that domain
contain(s) the right IP address.
I have checked the DNS records and they are pointing to the right IP address, so am not sure what the issue could be.
Thanks in advance for you help.
AJ
Try the complete URL in your web browser. Does it show a hexadecimal string? Or do you get a 404 as well?
Thanks for your response, I get a 404 not found error when I try the url in the web browser.
Hi Christoph, just wondering if you had any further thoughts on what could be the issue.
Solved, it was an issue with the DNS records not being correctly updated.