Success stories

I would appreciate if you leave a comment here telling others if your mail server is running. Perhaps even a word about the purpose of the mail server and the number of users you have. That allows others to see that their time is well-spent and indeed leads to a working mail server. Thanks for your feedback.

(Have I mentioned the donation and Flattr buttons at the bottom? ūüôā )

Managing users, aliases and domains

Maybe you already know what you have to do to create mail domains and mail users. After all I tried to explain the database schema on the page that dealt with preparing the database. But if that wasn’t clear enough let me explain what you need to do to manage your mail accounts.

Web interfaces

If you don’t like using SQL queries to manage your mail server you may like to install a web-based management software. Several developers contributed web interfaces for the previous Wheezy tutorial and they will probably still work but only create weak password hashes. With a little experience in web development you should be able to develop your own management system. I have documented a few common SQL queries further down.

These web interfaces were specifically made for this Jessie guide:

ISPmail Admin

Homepage: http://ima.jungclaussen.com/
Demo: http://ima.jungclaussen.com/demo/

ima_screenshot

GRSoft Virtual Mail Manager

Peter Gutwein has updated his PHP-based web interface to create strong password hashes as recommended in this Jessie guide. It supports english, german, spanish, french, italian, russian and swedish.

Homepage -> http://www.grs-service.ch/pub/grs_mminstallation.html

Here are screenshots of the working application:

grsoft-vmm-screenshot1grsoft-vmm-screenshot2grsoft-vmm-screenshot3grsoft-vmm-screenshot4grsoft-vmm-screenshot5grsoft-vmm-screenshot6grsoft-vmm-screenshot7grsoft-vmm-screenshot8grsoft-vmm-screenshot9

Managing the database directly

Common tasks / PHPMyAdmin

This table explains what changes are required in the database for your everyday tasks. You can click your way through PHPMyAdmin following the instructions in this table:

Create a mail domainInsert a new row into the virtual_domains table and set the “name” to the name of the new domain.
Delete a mail domainDelete the row from the virtual_domains table that has the right “name”. All aliases and users will automatically be deleted, too. However the mailboxes will stay on disk at /var/vmail/… and you need to delete them manually.
Create a mail userFind out the “id” of the right domain from the virtual_domains table. The insert a new row into the virtual_users table. Set the domain_id to the value you just looked up in the virtual_domains table. Set the “email” field to the complete email address of the new user. Create a new password in a shell using the “dovecot pw -s SHA256-CRYPT” command and insert the result into the “password” field.
Change the password of a userFind the row in the virtual_users table by looking for the right “email” field. Create a new password in a shell using the “dovecot pw -s SHA256-CRYPT” command and insert the result into the “password” field.
Delete a mail userFind the row in the virtual_users table by looking for the right “email” field and delete it. The mailbox will stay on disk at /var/vmail/… and you need to delete it manually
Create a mail forwardingYou can forward emails from one (source) email to other addresses (destinations) – even outside of your mail server. Find out the “id” of the right domain (the part after the “@” of the source email address) from the virtual_domains table. Create a new row in the virtual_aliases table for each destination (if you have multiple destination addresses). Set the “source” field to the complete source email address. And set the “destination” field to the respective complete destination email address.
Delete a mail forwarddingFind all rows in the virtual_aliases table by looking for the right “source” email address. Remove all rows that you lead to “destination” addresses you don’t want to forward email to.

SQL queries

Create a mail domainINSERT INTO virtual_domains (name) VALUES (“example.org”);
Delete a mail domainDELETE FROM virtual_domains where name=’example.org’;
Create a mail userINSERT INTO virtual_users (domain_id, email, password) VALUES ( (SELECT id FROM virtual_domains WHERE name=’example.org’), ‘john@example.org’,CONCAT(‘{SHA256-CRYPT}’, ENCRYPT (‘new password’, CONCAT(‘$5$’, SUBSTRING(SHA(RAND()), -16)))));
Change the password of a userUPDATE virtual_users SET password=CONCAT(‘{SHA256-CRYPT}’, ENCRYPT (‘new password’, CONCAT(‘$5$’, SUBSTRING(SHA(RAND()), -16)))) WHERE email=’email@address’;
Delete a mail userDELETE FROM virtual_users WHERE email=’john@example.org’;
Create a mail forwardingINSERT INTO virtual_aliases (domain_id, source, destination) VALUES ( (SELECT id FROM virtual_domains WHERE name=’example.org’), ‘melissa@example.org’, ‘juila@example.net’);
Delete a mail forwardingDELETE FROM virtual_aliases WHERE source=’melissa@example.org’;

Deploying a mail server with Ansible

Pangs of conscience

I refused to provide automatic installation scripts since I started the first ISPmail guide for Debian. Why? Because my intention is to help you understand the complexity of setting up your own mail server. The protocols. The software components. The caveats. The features. There are countless copy’n’paste tutorials on the internet that you can follow if you don’t care learning all that. Unfortunately even if those tutorials help you set up a mail server (which quite often does not work) you will not understand how it works nor be able to troubleshoot a faulty installation. I purposely make you do all the tiny steps manually and explain what they mean. Not to torture you but to¬†help and¬†educate you.

But I have to be honest with you. As I’m dealing with mail servers nearly every day I have automated everything that I describe in this guide. So let me share it with you. Just promise me that you won’t take the easy route. Don’t just use the playbook without reading the entire tutorial. I will help you a bit with Ansible and Vagrant but bear with me if I refuse to expain topics that are actually part of the ISPmail guide.

Ansible

The repository is currently in the process of migration for the Buster guide. Sorry about that.

There are various tools to help you install and configure a server ‚Äď they are called¬†configuration management systems. In their special syntax you describe how you want your system to look like and then the tools checks what has to be done and does it. You can use them to automatically install a whole system or just deploy your own application. Famous pieces of software like that are Puppet and Chef. However both are painful on different levels. More modern alternatives are Ansible and Salt. I have been using Ansible very successfully in the last two years so obviously I chose it for setting up ISPmail servers.

You haven’t tried Ansible? You should. Head over to their introduction page and walk through the first chapter. From now on I expect you to know the basics of playbooks, roles and variables.

Make sure you have Ansible¬†(at least version 1.9) installed. That version is contained in Debian testing or Ubuntu 15.10. Otherwise you can install it as a Debian package from an Ubuntu PPA or ‚Äď if you don’t mind spoiling your workstation ‚Äď running “pip install ansible”. Now you can clone my Git repository¬†that contains everything you need. Use Git to clone it to somewhere on the disk of your workstation. It does not belong onto the mail server.

git clone https://git.workaround.org/chaas/ansible-ispmail-jessie.git

Vagrant

This is another very valuable tool. It is mainly used by developers but is also useful for system administrators. Have you ever thought‚Ķ “Heck, I would just need a quick Debian Jessie installation to try something.”? Vagrant is the answer. It has a repository of prepared installation images that you can download and run as a virtual machine in VirtualBox.

The reason I am mentioning it is that it allows you to run all the tasks necessary to create an ISPmail server in a virtual machine. All the work that is explained in this tutorial is contained in the Ansible playbook and roles in machine-readable form. So with Ansible and Vagrant you can easily simulate a mail server installation in a virtual machine without the need of an actual server. You could use it to play around with it and don’t have to fear breaking it. Should it break you just throw it away and rebuild it.

Install Vagrant and VirtualBox:

apt-get install vagrant virtualbox

My Git repository contains a Vagrantfile that defines what Vagrant is supposed to do. It is very simple indeed:

Vagrant.configure(2) do |config|
  config.vm.box = "debian/jessie64"

  config.vm.provision "ansible" do |ansible|
    ansible.playbook = "ispmail.yml"
    ansible.sudo = true
  end

config.vm.provider "virtualbox" do |vb|
  config.vm.network "private_network", ip: "10.0.0.100"
  end
end

When you start Vagrant it downloads the image “debian/jessie64” from the image repository¬†and boots it. You could already log into a working Jessie installation and administer it with¬†sudo. Then it takes the¬†ispmail.yml file and runs it as an Ansible playbook. That installs the mail server. And it sets up a virtual network on your workstation and makes the virtual server available as 10.0.0.100.

Make sure you are in the directory that contains the Vagrantfile and go create a mail server:

vagrant up

Be patient while the server is provisioned. Downloading the Jessie image takes a bit depending on your internet connection. And the first task “ispmail-packages” downloads and installs quite a lot of software – so give a few minutes. At the end you should just see green and yellow result message. Let me know if any red messages appear because that probably means a bug in the playbook.

After that you can log into the server using

vagrant ssh

And become root…

sudo -i

Of course everything else is set up already. Point your browser to https://10.0.0.100/ and you will get the RoundCube webmail interface. Go to https://10.0.0.100/phpmyadmin to manage the database via PHPmyAdmin. IMAP, POP3 and SMTP work as expected. And the example data described in this guide is already there, too.

To destroy the virtual server again run…

vagrant destroy

Customizing the installation

I am using various variable to allow you to customize the installation. See group_vars/all. Their meaning:

  • ispmail_populate_test_data: true
    Populates the database with sample data (e.g. john@example.org)
  • ispmail_mysql_mailuser_password: ChangeMe
    MySQL password for the ‘mailuser’ user
  • ispmail_mysql_root_password: ChangeMeRoot
    MySQL password for the ‘root’ user
  • ispmail_mysql_roundcube_password: ChangeMeRound
    MySQL password for the ’roundcube’ user
  • ispmail_dovecot_auth_mechanisms: plain login
    Authentication mechanisms for Dovecot. ‘plain’ is the common mechanism. ‘login’ is needed for old Microsoft clients.
  • ispmail_postmaster_address: postmaster@example.org
    Email address of the person who is responsible for the server
  • ispmail_webmail_hostname: 10.0.0.100
    Host name to be used for the web server. Should be set to the FQN.
  • ispmail_certificate_country: DE
    Entries starting with ‘ispmail_certificate‚Ķ’ are strings passed to OpenSSL when generating the self-signed certificate.
  • ispmail_certificate_state: Hamburg
  • ispmail_certificate_location: Hamburg
  • ispmail_certificate_organisation: workaround.org
  • ispmail_certificate_orgunit: IT-Crowd
  • ispmail_certificate_domains:
    – example.org
    You can use multiple hostnames/domains here if your mail server should be used for them. Additional entries get added as “alternate names” in the certificate. The names must exactly match the name that users enter to access the mail server.
  • ispmail_certificate_email: postmaster@example.org
  • ispmail_certificate_days_valid: 3650
    Number of days that the certificate will be valid

Using Ansible to deploy a real-life server

Have you played with your virtual Vagrant mail server enough? Would you like to deploy a real server with Ansible? That’s easy. After a basic installation of Debian Jessie make sure that you can access the server without authentication ‚Äď use SSH public key authentication.

Edit the group_vars/all. Choose proper secure passwords (‘pwgen’ is your friend) and set the email addresses, domains and certificate information to values that fit your bill.

Tell Ansible where your mail server is by editing the ‘hosts’ file and changing the ‘default’ host to the FQDN of your server.¬†Then run

ansible-playbook ispmail.yml

You are invited to add a comment if this page helped you. Even though it’s a pretty advanced topic.

Testing email delivery

So far you have spent considerable time with theory and configuration. Are you worried whether all you did actually leads to a working mail server? Before we do the final steps let’s take a break and verify that all you did works so far.

At this point the /var/vmail directory should be empty or maybe contain an “example.org” directory if you played with the john@example.org account previously. You can get a list of all files and directories within by running:

find /var/vmail

There may be nothing except perhaps a “lost+found” directory if /var/vmail is on a separate partition. And perhaps skeleton files like “.profile” or “.bashrc”.

If you added test data to the database in the section about database preparation as you were told then you have “example.org” configured as a virtual domain and “john@example.org” as a user in that domain.

Send a test email

Open a new terminal window and run

tail -f /var/log/mail.log

to see what the mail server is doing. Now let’s send an email to John. A simple way to create and send an email is to pipe text to the “mail” command. Do that in another terminal window:

echo test | mail john@example.org

If all works as expected your mail.log will show a lot of technical information about the email delivery. Like this:

Nov 8 22:29:40 jessie postfix/pickup[26781]: D663322AF9: uid=0 from=<root>
Nov 8 22:29:40 jessie postfix/cleanup[26854]: D663322AF9: message-id=<20151108222940.D663322AF9@debian-jessie.vagrantup.com>
Nov 8 22:29:40 jessie postfix/qmgr[21866]: D663322AF9: from=<root@debian-jessie.vagrantup.com>, size=313, nrcpt=1 (queue active)
Nov 8 22:29:40 jessie dovecot: lmtp(26859): Connect from local
Nov 8 22:29:40 jessie dovecot: lmtp(26859, john@example.org): 4xrmNdTMP1braAAACYsm4Q: sieve: msgid=<20151108222940.D663322AF9@debian-jessie.vagrantup.com>: stored mail into mailbox 'INBOX'
Nov 8 22:29:40 jessie postfix/lmtp[26858]: D663322AF9: to=<john@example.org>, relay=debian-jessie.vagrantup.com[private/dovecot-lmtp], delay=0.06, delays=0.02/0/0.01/0.02, dsn=2.0.0, status=sent (250 2.0.0 <john@example.org> 4xrmNdTMP1braAAACYsm4Q Saved)
Nov 8 22:29:40 jessie dovecot: lmtp(26859): Disconnect from local: Successful quit
Nov 8 22:29:40 jessie postfix/qmgr[21866]: D663322AF9: removed

Your output will look slightly different. But focus on the parts that are printed in bold letters. They should be exactly the same in your log file. If everything worked as expected Postfix has accepted the email and forwarded it to Dovecot which in turn wrote the email in John’s maildir. If you get any errors in the log file then try to understand the error message and find the cause of the problem before you proceed.

Look again:

find /var/vmail

Dovecot has now created a directory structure for John and created several directories and extra files:

/var/vmail/
/var/vmail/example.org
/var/vmail/example.org/john
/var/vmail/example.org/john/Maildir
/var/vmail/example.org/john/Maildir/dovecot-uidvalidity
/var/vmail/example.org/john/Maildir/dovecot.index.cache
/var/vmail/example.org/john/Maildir/tmp
/var/vmail/example.org/john/Maildir/dovecot.index.log
/var/vmail/example.org/john/Maildir/cur
/var/vmail/example.org/john/Maildir/dovecot-uidvalidity.562762ef
/var/vmail/example.org/john/Maildir/dovecot-uidlist
/var/vmail/example.org/john/Maildir/new
/var/vmail/example.org/john/Maildir/new/1445423017.M410998P32117.ispmail-jessie,S=376,W=386

Your files may have slightly different numbers. That’s okay. The line printed in bold letter depicts the actual email file. It is the only file in the “new” folder.

Access the email as a file

So John finally has finally received a new email into his inbox (that I have highlighted in bold letters). Just open that file and take a look – it is just a standard email with headers and a body. On my system it reads:

Return-Path: <root@ispmail-jessie.pmd.local>
Delivered-To: john@example.org
Received: by ispmail-jessie.pmd.local (Postfix, from userid 0)
 id 54FF0260EB4; Wed, 21 Oct 2015 12:23:37 +0200 (CEST)
To: john@example.org
Message-Id: <20151021102337.54FF0260EB4@ispmail-jessie.pmd.local>
Date: Wed, 21 Oct 2015 12:23:37 +0200 (CEST)
From: root@ispmail-jessie.pmd.local (root)
test

If anything went wrong then carefully check the last lines of your /var/log/mail.log. It will very likely point you to the problem. Or read the troubleshooting section. Or just add a comment to this page and ask other readers for help.

Access the email from your disk using mutt

You can also use a slightly more comfortable tool to access Maildirs that will come handy for you as a mail server administrator: “mutt”.

mutt -f /var/vmail/example.org/john/Maildir

(You may get asked to create /root/Mail – this is standard procedure. Just press Enter.)

What you see now are the contents of John’s mailbox:

ispmail-jessie-mutt-index

Press enter and you can read the email number 1:

ispmail-jessie-mutt-email

So there is your test email. Press “q” to quit “mutt”.

This is a simple method to access mail directories on disk to trace problems.

Access the email through IMAP using mutt

Actually we just cheated a little as we have accessed John’s inbox directly on disk. A better test is to use POP3 or IMAP. And fortunately “mutt” supports IMAP:

mutt -f imap://john@example.org@localhost

You may be prompted to confirm that you are connecting to a mail server with an untrusted SSL certificate. That’s okay if you generated a self-signed certificate. In the end you should see the index and email just like in the screenshots above. (The password is “summersun” – remember?) That worked? Great. Otherwise check your /var/log/mail.log for error messages.

The parameter after the “-f” looks a bit weird, doesn’t it. Let me explain it:

  • imap://
    The protocol. You can also use “imaps” but mutt automatically switches to imaps here.
  • john@example.org
    The login name (as we use the email address as the user name, too)
  • @
    The seperator between the user name and the host name of the mail server
  • localhost
    The mail server

Access the email through IMAP using a desktop mail client

Your users are not likely to use “mutt” to read their email – unless they are immortal console nerds. So of course you can use a graphical mail client to access John’s emails. If you use Thunderbird (or “icedove” on Debian) then feel free to configure a new email account using IMAP and connect to the server. Remember that the user name and email address are both “john@example.org” and the password is “summersun”.

Access the email through Roundcube webmail

And last but not least you have a very easy way to access John’s emails by just using your web browser. After all you installed Roundcube earlier. So go to https://YOUR.MAIL.SERVER/ and login as “john@example.org” with password “summersun”. That should take you to the inbox:

ispmail-jessie-roundcube-john-inbox

POP3 versus IMAP

If you wonder what the difference between POP3 and IMAP is:

  • POP3 (Post Office Protocol) is a simple protocol that lets you fetch email from a single mailbox. It is usually used to collect all emails, though you can also leave them on the server but this is a bit of a hack and you can’t create multiple folders on the server to sort your mail. It saves space on the mail server because the email gets moved to the user’s hard disk on their computer. But they won’t be able to access the same email from another computer. Besides you cannot create multiple folders on the server to sort your mail. There is just the inbox. This variant is antiquated and not exactly user-friendly.
  • IMAP (Internet Messaging Application Protocol) is predominantly focused upon leaving your mail on the server but you can also collect it like POP3. The inbox is where your incoming emails are stored but users can also maintain folders and move emails to them. But users can move emails to different directories. IMAP is useful when you want to access your email from different locations without losing mail because you fetched it from another location. The drawback is that lazy users leave their mail on the server thus filling up your server’s hard disk (unless you use quotas).

Troubleshooting your mail server

General troubleshooting tips

  • Run “postfix check” to make Postfix look for obvious configuration errors. If it returns no output then no problem was found.
  • Read your /var/log/mail.log and look for warnings and errors.

Postfix keeps unwanted emails in the queue

Check that you do not keep soft_bounce enabled. If “postconf soft_bounce” shows “yes” then run “postconf soft_bounce=no”.

I get “Permission denied” from Dovecot in the mail.log file

You have the permissions wrong. Run:

chown -R vmail:vmail /var/vmail

Postfix says: “Mail loops back to myself”

Postfix tried to send an email to the internet because it did not feel responsible for the recipient’s domain. However DNS records pointed it back to itself. So either your DNS record for a certain domain is wrong (unlikely) or your Postfix refuses to receive email for the given domain (very likely). Check that the recipient’s domain is listed in the “virtual_domains” table in the MySQL database. And verify that your CF files are working correctly

Setting up Roundcube webmail

Now that you have a valid key and certificate and installed the Roundcube packages we are ready to set up your webmail service. The Apache web server installation on Debian stores all virtual host configurations in /etc/apache2/sites-available/. There are two files after a fresh Apache installation:

  • 000-default.conf serves the /var/www directory via HTTP and is enabled by default. We will edit this file so that if a user accesses your web server via HTTP they will be redirected to the HTTPS site.
  • default-ssl.conf serves the /var/www directory via HTTPS. It uses an automatically created self-signed “snake oil” certificate. We will enable this virtual host and configure the Roundcube webmail interface.

If you want to enable such a virtual host configuration you can use the “a2ensite” (Apache version 2 enable site) command. It will create a symbolik link from /etc/apache2/sites-available/FILE to /etc/apache2/sites-enabled/FILE.

Note: Debian Jessie ships with Apache version 2.4. This requires that all configuration files end with “.conf”. Expect confusing errors if you omit that suffix.

Virtual host for HTTPS

Edit the file /etc/apache2/sites-available/default-ssl. Change these two lines to make Apache use the key and  certificate you created earlier:

SSLCertificateFile /etc/ssl/certs/mailserver.pem
SSLCertificateKeyFile /etc/ssl/private/mailserver.pem

Also add these two lines to include Roundcube anywhere in that file between the <VirtualHost…> and the </Virtualhost> tag:

Include /etc/roundcube/apache.conf
Alias / /var/lib/roundcube/

Enable this virtual host:

a2ensite default-ssl

Also enable the “mod_ssl” module and make Apache listen to port 443:

a2enmod ssl
service apache2 reload

If everything works fine you will see an “[ ok ]” message. If not please run “apache2ctl configtest” to see if Apache detects any obvious problems. Or check out the /var/log/apache2/error.log log file for further hints.

Now if you point your web browser to http://YOURSERVER/ it will show the webmail interface.

ispmail-jessie-install-packages-roundcube-loginform

You could already log in by entering “localhost” as a server. But let’s improve the configuration a litte. The “Server” is always “localhost”. So edit the /etc/roundcube/config.inc.php file and set:

$config['default_host'] = 'localhost';

 

Now when you reload the login form the “Server” field should be gone.

Set up HTTP to HTTPS redirection

For those users who forget to type “https” instead of “http” let us also set up an automatic redirection so that they will be forwarded to the secure URL. Edit the /etc/apache2/sites-available/000-default file and insert

Redirect permanent / https://YOUR.MAIL.SERVER/

anywhere within the VirtualHost section. Of course you have to replace YOUR.MAIL.SERVER with the fully-qualified domain name of your mail server. This configuration is enabled by default so you just need to reload the web server to make your change work:

service apache2 reload

Plugins

Further down in /etc/roundcube/config.inc.php there is a list of plugins that Roundcube loads. The “archive” and “zipdownload” plugins are proably already enabled. Add the “managesieve” and “password” plugins so that the setting looks like this:

$config['plugins'] = array(	 	 
 'archive',	 	 
 'zipdownload',	 	 
 'managesieve',	 	 
 'password',	 	 
);

Next an optional setting. The default session lifetime in Roundcube is 10 minutes. That means if a user is not using the webmail interface for more than 10 minutes they will be logged out. I found that annoying and increased that timeout to one hour. To do that at the end of the config file add:

$config['session_lifetime'] = 60;

And if you would like to change the default logo of Roundcube that can be done by setting:

$config['skin_logo'] = './ispmail-logo.png';
ispmail-logo
ISPmail logo for Roundcube

You just need to copy that image file by that name to /var/lib/roundcube/ispmail-logo.png. The logo should be 177×49 pixels large. Feel free to take this nifty ISPmail logo I crafted. ūüôā
If the logo appears to be broken then make sure that the permissions are correct:

chmod a+r /var/lib/roundcube/ispmail-logo.png

When you reload the login form in the browser it will then look like this instead:
ispmail-jessie-roundcube-pimped

Configuring the managesieve plugin

The “managesieve” password will allow your users to manage automatic rules to manage their email. These rules are stored on the server and will be run automatically. You need to configure this plugin though. A default configuration can be found at /usr/share/roundcube/plugins/managesieve/config.inc.php.dist on your system. Copy it to the location where Roundcube will look for it:

cp /usr/share/roundcube/plugins/managesieve/config.inc.php.dist /etc/roundcube/plugins/managesieve/config.inc.php

No further changes are required.

Configuring the password plugin

We urge our users to change their passwords frequently. So we need to give them a chance to actually do that. Copy the default configuration file /usr/share/roundcube/plugins/password/config.inc.php to the right place:

cp /usr/share/roundcube/plugins/password/config.inc.php.dist /etc/roundcube/plugins/password/config.inc.php

The configuration file at /etc/roundcube/plugins/password/config.inc.php requires a couple of changes though. We need to tell it how our database works and what to do when a user wants to change their password. The first setting deals with the minimal length of the password. I recommend to enforce at least 10 characters. In fact the complexity of the password is not that important. Consider XKCD as food for thought on password security. So set:

$config['password_minimum_length'] = 10;

We should allow the user to use the old password as the new password. It may sound stupid but as we are upgrading the password scheme from the weak unsalted MD5 to the better SHA2 algorithm we should allow that:

$config['password_force_save'] = true;

Next the password plugin needs to know how to access your database:

$config['password_db_dsn'] = 'mysql://mailuser:ChangeMe@127.0.0.1/mailserver';

Replace “ChangeMe” by the randomly generated password you created earlier for the “mailuser” MySQL user.
Now tell the plugin how to actually write the new password hash into the database:

$config['password_query'] = "UPDATE virtual_users SET password=CONCAT('{SHA256-CRYPT}', ENCRYPT (%p, CONCAT('$5$', SUBSTRING(SHA(RAND()), -16)))) WHERE email=%u;";

Whoa, that looks weird, doesn’t it? That SQL query generates a password hash from these parts:

  • The string “{SHA256-CRYPT}”. It tells Dovecot explicitly that this password is a salted SHA256 hash. You may have different kinds of encrypted passwords in your database so this makes it clear.
  • The actual encrypted password that is generated using your operating system’s crypt() function (MySQL calls it when you use the ENCRYPT SQL function) using‚Ķ
    • The new plaintext password (Roundcube replaces “%p” by it).
    • “$5$” – that stands for using the SHA-256 algorithm. Check “man crypt” to see which algorithms are supported by crypt().
    • And SUBSTRING(SHA(RAND()), -16) is used as a salt. A salt is just some additional randomness that makes it much much harder to reverse-engineer the actual password from the encrypted string.
  • And of course we just want to change the password of the current email user. So “WHERE email=%u” makes sure we choose the right row in the database. Roundcube replaces “%u” with the user name – which is the same as the email address in our case.

(I used to recommend just using “dovecot pw -s SHA256-CRYPT” to generate passwords. You can do that, too. But in fact MySQL is capable of generating salted SHA256 hashes without calling any shell command. Thanks for the hint, Martin.)

Alright. Roundcube is now set up. But to be able to login and use it we need to install and configure Dovecot first. So don’t worry if the login does not work yet.

Trying it out

That was a lot of theory and configuration. Let’s finally try it. Go to https://YOUR.MAIL.SERVER/ and login as‚Ķ

ispmail-jessie-roundcube-testlogin
If everything works as intended you will be logged in and see…
ispmail-jessie-roundcube-testlogin-successful
If that didn’t work then check your /var/log/mail.log and /var/log/roundcube/errors files for errors.

Making Postfix get its information from the MySQL database

In the¬†previous chapter¬†you have created the MySQL database schema and inserted¬†some data to play with. The entry point for all email on your system is Postfix. So we need to tell Postfix how to get the information from the database. First let’s tell it how to find out if a certain domain is a vaild email domain.

virtual_mailbox_domains

As described earlier a mapping in Postfix is just a table that contains a left-hand side (LHS) and a right-hand side (RHS). To make Postfix use MySQL we need to¬†create¬†a ‘cf’ file (configuration file). Start by creating a file called¬†/etc/postfix/mysql-virtual-mailbox-domains.cf¬†for the virtual_mailbox_domains mapping that contains:

user = mailuser
password = fLxsWdf5ABLqwhZr‚Äč ¬† ¬†<- use your own database password here
hosts = 127.0.0.1
dbname = mailserver
query = SELECT 1 FROM virtual_domains WHERE name='%s'

Imagine that Postfix received an email for somebody@example.org and wants to find out if¬†example.org¬†is a virtual mailbox domain. It will run the above SQL query and replace ‘%s’ by ‘example.org’. If it finds such a row¬†in the virtual_domains table it will return a ‘1’. Actually it does not matter what exactly is returned as long as there is a result.

Note: You may be tempted to write “localhost” instead of “127.0.0.1”. Don’t do that because there is indeed a difference in this context. “localhost” will make Postfix look for the MySQL socket file and it can’t find it within it’s chroot jail at /var/spool/postfix because it is at /var/run/mysqld/mysqld.sock by default. But if you tell Postfix to use 127.0.0.1 as described here you make Postfix use a TCP connection to port 3306 on localhost which is working even if Postfix is jailed.

Now you need to make Postfix use this database mapping:

postconf virtual_mailbox_domains=mysql:/etc/postfix/mysql-virtual-mailbox-domains.cf

The “postconf"¬†command conveniently adds configuration lines to your¬†/etc/postfix/main.cf¬†file. It also activates the new setting instantly so you do not have to reload the Postfix process.

The test data you created earlier added the domain “example.org” as one of your mailbox domains. Let’s ask Postfix if it recognizes that domain:

postmap -q example.org mysql:/etc/postfix/mysql-virtual-mailbox-domains.cf

You should get ‘1’ as a result. Your first mapping is working. Great.¬†On to the second mapping.

virtual_mailbox_maps

You will now define the¬†virtual_mailbox_maps¬†which is mapping email addresses (left-hand side) to the location of the user’s mailbox on your harddisk (right-hand side). Postfix has a built-in transport service called “virtual” that can receive the email and put it into that directory. But we will not use it that way. Instead we we use the “dovecot” transport service to make our lives a bit easier.

Now when we use “dovecot” Postfix will still check if the destination email address is defined in the virtual_mailbox_maps mapping. But it will ignore the right-hand side.¬†Postfix just needs to check if a certain email address is valid. Similar to the above you need an SQL query that searches for an email address and returns “1”.

To accomplish that create another configuration file at /etc/postfix/mysql-virtual-mailbox-maps.cf:

user = mailuser
password = fLxsWdf5ABLqwhZr    <- use your own database password here
hosts = 127.0.0.1
dbname = mailserver
query = SELECT 1 FROM virtual_users WHERE email='%s'

Tell Postfix that this mapping file is supposed to be used for the virtual_mailbox_maps mapping:

postconf virtual_mailbox_maps=mysql:/etc/postfix/mysql-virtual-mailbox-maps.cf

Test if Postfix is happy with this mapping by asking it where the mailbox directory of our john@example.org user would be:

postmap -q john@example.org mysql:/etc/postfix/mysql-virtual-mailbox-maps.cf

You should get¬†“1"¬†back which means that¬†john@example.org¬†is an existing virtual mailbox user on your server. Very good. Later when we deal with the Dovecot configuration we will also use the password¬†field but Postfix does not need it right here. On to the next mapping:

virtual_alias_maps

The virtual_alias_maps mapping is used for forwarding emails from one email address to others. It is possible to name multiple destinations. In the database this is achieved by using different rows. See the page on virtual domains if you need details.

Create another “.cf” file at¬†/etc/postfix/mysql-virtual-alias-maps.cf:

user = mailuser
password = fLxsWdf5ABLqwhZr    <-- use your own database password here
hosts = 127.0.0.1
dbname = mailserver
query = SELECT destination FROM virtual_aliases WHERE source='%s'

Make Postfix use this database mapping:

postconf virtual_alias_maps=mysql:/etc/postfix/mysql-virtual-alias-maps.cf

Test if the mapping file works as expected:

postmap -q jack@example.org mysql:/etc/postfix/mysql-virtual-alias-maps.cf

You should see the expected destination:

john@example.org

So if Postfix would receive an email for jack@example.org it would forward it to john@example.org.

Catch-all aliases

As explained earlier in the tutorial there is way to forward¬†all email addresses in a domain to a certain destination email address. This is called a “catchall” alias. Catchalls catch all emails for a domain if there is no specific virtual user for that email address. Catchalls are evil – seriously. It is tempting to generally forward all email addresses to one person if e.g. your marketing department requests a new email aliases every week. But the drawback is that you will get even more insane amounts of spam because spammers will send their stuff¬†to any address of your domain. Or perhaps a sender mixed up the proper spelling of a recipient but the mail server will forward the email instead of rejecting it for a good reason. So think twice before using catchalls.

I could not convince you to keep your hands off that evilness? Well, okay. A catchall alias looks like “@example.org” and forwards email for the whole domain to other addresses. We have created the ‘john@example.org’ user and would like to forward all¬†other¬†email on the domain to ‘kerstin@example.com’. So we would add a catchall alias like:

sourcedestination
@example.orgkerstin@example.com

But there is a catch. Imagine what happens when Postfix receives an email for ‘john@example.org’. Postfix will first check the aliases in the¬†virtual_alias_maps¬†table. (It does not look at the¬†virtual_mailbox_maps¬†table yet.) It finds the catchall entry as above and since there is no more specific alias the catchall account matches and the email is redirected to ‘kerstin@example.com’. John will never get any email. This is not what you wanted.

So you need to make the table look like this instead:

emaildestination
@example.orgkerstin@example.com
john@example.orgjohn@example.org

More specific aliases have precedence over general catchall aliases. Postfix will find an entry for ‘john@example.org’ first and sees that email should be “forwarded” to ‘john@example.org’ – the same email address. This trickery may sound weird but it is needed if you plan to use catchall accounts.

Postfix will lookup all these mappings for each of:

  • john@example.org (most specific)
  • john (only works if “example.org” is the $myorigin domain)
  • @example.org (catchall – least specific)

This is outlined in the virtual(5) man page in the TABLE SEARCH ORDER section.

For that “john-to-himself” mapping you need to create a “.cf” file¬†/etc/postfix/mysql-email2email.cf¬†for the latter mapping:

user = mailuser
password = fLxsWdf5ABLqwhZr    <- use your own database password here
hosts = 127.0.0.1
dbname = mailserver
query = SELECT email FROM virtual_users WHERE email='%s'

Check that you get John’s email address back when you ask Postfix if there are any aliases for him:

postmap -q john@example.org mysql:/etc/postfix/mysql-email2email.cf

The result should be the same address:

john@example.org

Now you need to tell Postfix that it should check¬†both the aliases and the “john-to-himself”:

postconf virtual_alias_maps=mysql:/etc/postfix/mysql-virtual-alias-maps.cf,mysql:/etc/postfix/mysql-email2email.cf

The order of the two mappings is not important here.

You did it! All mappings are set up and the database is generally ready to be filled with domains and users. Make sure that only ‘root’ and the ‘postfix’ user can read the “.cf” files – after all your database password is stored there:

chgrp postfix /etc/postfix/mysql-*.cf
chmod u=rw,g=r,o= /etc/postfix/mysql-*.cf

Setting up Dovecot

This chapter of our journey leads us to Dovecot Рthe software that…

  • gets emails¬†from Postfix and saves them to disk
  • executes user-based “sieve” filter rules (can be used to e.g. move emails to different folders based on certain criteria or send automated vacation responses)
  • allows the user to fetch emails using POP3 or IMAP

Before we get to the actual configuration for security reasons I recommend that you create a new system user that will own all virtual mailboxes. The following shell commands will create a system group “vmail” with GID (group ID) 5000 and a system user “vmail” with UID (user ID) 5000. (Make sure that UID and GID are not yet used or choose another – the number can be anything between 1000 and 65000 that is not yet used):

groupadd -g 5000 vmail
useradd -g vmail -u 5000 vmail -d /var/vmail -m

If the /var/vmail directory was already there because you assigned it a dedicated mount point then you should make sure that the permissions are set correctly:

chown -R vmail.vmail /var/vmail

The configuration files for Dovecot are found in /etc/dovecot/conf.d/. All these files are loaded by Dovecot. This is done by this magical line in the dovecot.conf file:

!include conf.d/*.conf

It loads all files in /etc/dovecot/conf.d/ that end on “.conf” in sorted order. So “10-auth.conf” is loaded first and “90-quota.conf” is loaded last. The big advantage is that you can edit or replace parts of the configuration without having to overwrite the entire configuration. The main /etc/dovecot/dovecot.conf file does not require any changes. Those¬†other files in conf.d/ however need to be edited‚Ķ

conf.d/

10-auth.conf

If your users are still using Outlook Express (Windows XP) or Microsoft Mail (Windows Vista) then you need to add the “LOGIN” authentication mechanism in addition to the standard “PLAIN” mechanism:

auth_mechanisms = plain login

These are plaintext (unencrypted) ways to transmit a mail user’s password. But don’t worry. By default Dovecot sets “disable_plaintext_auth = yes” which ensures that authentication is only accepted over TLS-encrypted connections.

At the end of this file you will find various authentication backends that Dovecot uses. By default it will use system users (that are listed in /etc/passwd). But we want to use the MySQL database backend so go ahead and change this block to:

#!include auth-system.conf.ext
!include auth-sql.conf.ext
#!include auth-ldap.conf.ext
#!include auth-passwdfile.conf.ext
#!include auth-checkpassword.conf.ext
#!include auth-vpopmail.conf.ext
#!include auth-static.conf.ext

Now edit the SQL configuration file:

auth-sql.conf.ext

Change the¬†“userdb” section to:

userdb {
  driver = static
  args = uid=vmail gid=vmail home=/var/vmail/%d/%n
}

The placeholder %d stands for the domain part and %n for the user part if you have an email address like “user@domain”. Dovecot’s variables are documented on their website.

10-mail.conf

Change the mail_location setting to:

mail_location = maildir:/var/vmail/%d/%n/Maildir

This is the directory where Dovecot will look for the emails of a specific user. So for john@example.org this leads to “/var/vmail/example.org/john/Maildir”.

Further down you will find sections defining the namespaces. These namespaces are folder structures that your email program sees when connecting to the mail server. If you use POP3 you can only access the “inbox” – which is where all incoming email is stores. Using the IMAP protocol you get access to a hierarchy of folders and subfolders. And you can even use folders shared between users. Or public folder that could be accessed by anyone – even anonymously.

Look for the “namespace inbox” section. If you already have emails stored on your server from previous versions of this ISPmail guide you need to change:

separator = .

here. By default the seperator is “/” which creates a directory structure like “/var/vmail/example.org/john/Maildir/INBOX/staff/marketing/simon”. This is perfectly fine. But previous ISPmail guides used “.” as a seperator so the mentioned folder would rather have been “/var/vmail/example.org/john/Maildir/.INBOX.staff.marketing.simon”.

10-master.conf

This configuration file deals with services that allow communication with other processes. For example it enables or disables POP3 or IMAP. Don’t worry about the standard unencrypted TCP ports 110 (for POP3) and 143 (for IMAP). They can be kept accessible. If a user connects to these ports they will have to issue a STARTTLS command to switch into encrypted mode before they are allowed to send their password. There is basically no difference between using an plaintext port like 110 for POP3 and then using STARTTLS – or connecting to the encrypted 995 port for POP3S (=secure). See the Dovecot documentation for another explanation.

So most settings are sane here and do not have to be changed. However one change is required in the “service auth” section because we want Postfix to allow Dovecot as an authentication service. This is what has to be entered:

# Postfix smtp-auth
unix_listener /var/spool/postfix/private/auth {
  mode = 0660
  user = postfix
  group = postfix
}

Why that strange path? Well, Postfix runs in a chroot environment located at /var/spool/postfix. It can’t access anything outside of that directory. So to allow communication with Postfix we tell Dovecot to use the communication socket file in that chroot area.

10-ssl.conf

Earlier in this guide you created both a key and a certificate file to encrypt the communication with POP3, IMAPs and HTTPS between the users and your mail server. You need to tell Dovecot where to find these files:

ssl_cert = </etc/ssl/certs/mailserver.pem
ssl_key = </etc/ssl/private/mailserver.pem

And enable TLS/SSL encryption by setting:

ssl = yes

You could also set “ssl=required” but as Dovecot disallows sending plaintext passwords over unencrypted connections anyway we don’t actually need it. See the Dovecot documentation on SSL encryption¬†for more information.

15-mailboxes.conf

You should also add these lines within your “namespace inbox” section:

 mailbox INBOX.Junk {
  auto = subscribe
  special_use = \Junk
 }
 mailbox INBOX.Trash {
  auto = subscribe
  special_use = \Trash
 }

It ensures that a “Junk” and a “Trash” folder are created within the inbox when a user logs in. The user is also automatically subscribed to these folders. (When using IMAP you can choose which folder you want to see by subscribing them.) The “Junk” folder is required so that we can move spam emails to it – we will configure that later. And the “Trash” folder is required so that users using the Roundcube web mailer can actually delete emails.

The names used in “special_use” refer to the¬†special-use mailboxes as described in the RFC 6154.

/etc/dovecot/dovecot-sql.conf.ext

This file is referred to by /etc/dovecot/conf.d/auth-sql.conf.ext. It tells Dovecot how to access the MySQL database and where to find the information about email users/accounts. You will find it well documented although all configuration directives are commented out. Add these lines at the bottom of the file:

driver = mysql
connect = host=127.0.0.1 dbname=mailserver user=mailuser password=ChangeMe  <- use your database access password instead
default_pass_scheme = SHA256-CRYPT
password_query = SELECT email as user, password FROM virtual_users WHERE email='%u';

What these lines mean:

  • driver: the kind of database
  • connect: where to find the MySQL database and how to use it (username, password)
  • default_pass_scheme: the format in which the passwords are stored (we use strong salted SHA256 hashes)
  • password_query: an SQL statement that returns the user (=the email address) and the password (SHA256¬†hash) from the database where “%u” is the login user name (=we use the email address as the user name of an email account).¬†Whenever Dovecot needs to check an email user’s password if will run the above query and verify the password against the hash in the database.

Finishing move

You should also make sure that only root can access the SQL configuration file so nobody else is reading your database access passwords:

chown root:root /etc/dovecot/dovecot-sql.conf.ext
chmod go= /etc/dovecot/dovecot-sql.conf.ext

Restart Dovecot from the shell:

service dovecot restart

Look at your /var/log/mail.log logfile. You should see:

... dovecot: master: Dovecot v2.2.13 starting up for imap, sieve, pop3 (core dumps disabled)

If you get any error messages please double-check your configuration files.

Relaying with SMTP authentication

Relaying

Your mail server is almost ready for use. But one puzzle piece is missing. Your users can already fetch emails but they need to be able to send emails, too. Mail server know how to send an email to another mail server by looking up MX records. Mail clients however are expected to send email to their mail server first which then forwards email to the internet. When Postfix receives an email from a user forwards it to another server this is called relaying. Even if the mail client would know how to deal with MX records it would not be able to send the email. Users are often sitting a home at a DSL line. And such IP networks are usually blocked off from sending mail by so called real-time blacklists.

Am I going too fast? Okay, let’s walk through the different scenarios one by one.

Incoming email

When someone on the internet sends an email to john@example.org some other mail server will deliver the email using SMTP to your mail server. Postfix will determine that it’s responsible for email addresses in the example.org domain and accept the email. John can then use POP3 or IMAP to fetch the email from your server.

Outgoing email (without authentication)

John is on the internet somewhere and wants to send an email to lisa@example.com. Your mail server is not responsible for the “example.com” domain¬†so¬†it receives John’s email and forwards (relays) it to the mail server that is responsible for …@example.com email addresses. This may seem like a harmless scenario but¬†your mail server¬†must¬†deny that:

Why? Because anyone can claim to be John and make your mail server forward mail. If an attacker (like a spammer) would send millions of spam emails in John’s name through your server then other organisations will accuse you of spamming. Your mail server would be what people call an “open relay”. This is not what you want because your mail server would get blacklisted and you will not be able to¬†send out mail to most other servers. So without any proof that John is actually John your server will reject the email.

Outgoing email (with authentication)

So how does John prove his identity? He needs to use authenticated SMTP. This is similar to the previous case but John’s email program will also send his username and password.

Of course we are making sure that his authentication happens over an encrypted connection.

mynetworks

In addition to using SMTP authentication you can tell Postfix to always relay email for certain IP addresses. The mynetworks setting contains the list of IP networks or IP addresses that you trust. Usually you define your own local network here. The reason John had to authenticate in the above example is because he is not sending the email from your local network.

If your users are using the webmail interface running on your mail server then they will not need to authenticate. Roundcube sends email to localhost which is trusted according to the mynetworks setting.

Make Postfix use Dovecot for authentication

Authenticated SMTP with Postfix has been a hassle in the past. It was done through the SASL (Simple Authentication and Security Layer) library that was once part of the Cyrus mail server. And it wasn’t “simple” at all. It was nearly impossible to debug and threw error messages that were gibberish and misleading. Fortunately nowadays we can make Postfix ask the Dovecot server to verify the username and password. And as you already configured Dovecot’s authentication part this¬†you just need to make Postfix talk to Dovecot. Postfix just needs some extra configuration:

postconf smtpd_sasl_type=dovecot
postconf smtpd_sasl_path=private/auth
postconf smtpd_sasl_auth_enable=yes

This enables SMTP authentication and tells Postfix that it can talk to Dovecot through a socket file located at /var/spool/postfix/private/auth. Do you remember that Postfix runs in a sandboxed¬†chroot directory? That’s at /var/spool/postfix. So we configured Dovecot to put a socket into that directory to allow communication with Postfix.

Enable encryption

The following settings enable encryption, define the key and certificate and enforce that authentication must only occur over encrypted connections:

postconf smtpd_tls_security_level=may
postconf smtpd_tls_auth_only=yes
postconf smtpd_tls_cert_file=/etc/ssl/certs/mailserver.pem
postconf smtpd_tls_key_file=/etc/ssl/private/mailserver.pem

In previous versions of the ISPmail guide you also needed to set¬†the smtpd_recipient_restrictions to restrict relaying to¬†authenticated users. Fortunately the new version of Postfix in Jessie (2.11) has better defaults. Postfix offers a new setting called “smtpd_relay_restrictions” that deals with relaying requests in the “RCPT TO” phase of the SMTP dialog. So essentially it works like the “smtpd_recipient_restrictions” but is checked first. And as the smtpd_relay_restrictions have a reasonable default we don’t have to add anything to make authenticated relaying work.

By the way: Postfix 2.10 has changed the behavior of the smtpd_*_restrictions altogether. If a restriction makes Postfix reject an email it will wait until¬†after the “RCPT TO” line. Read the Postfix documentation for a description of the reasons for that change.

The¬†smtpd_tls_security_level is set to “may” to allow¬†encrypted communication when Postfix receives an email. This not only applies to users sending emails but also remote mail servers that send email to you. It may sound tempting to change that to “encrypt” which would enforce encryption and reject any attempts to create an unencrypted SMTP connection. But unfortunately there are still mail servers in the world that can’t encrypt and would not be able to deliver email to you. So “may” is the proper setting.

Although emails may still get delivered unencrypted from other servers we should make sure that users don’t accidentally (or out of ignorance) send their passwords unencrypted over the internet. That’s what we enforce using “smtpd_tls_auth_only=yes”. Now we should be safe.

How SMTP authentication works

Are you curious how SMTP authentication looks on a protocol level? Let’s go through that. In previous versions of this guide we¬†used “telnet” to connect to TCP port 25 and speak SMTP. But now we enforce encryption and can’t do SMTP authentication unencrypted. Let’s try the unencrypted part:

telnet localhost smtp

The server will let you in:

Trying 127.0.0.1...
Connected to localhost.
Escape character is '^]'.
220 mailtest ESMTP Postfix (Debian/GNU)

Say hello:

ehlo example.com

Postfix will present a list of features that are available during the SMTP dialog:

250-mailtest
250-PIPELINING
250-SIZE 10240000
250-VRFY
250-ETRN
250-STARTTLS
250-ENHANCEDSTATUSCODES
250-8BITMIME
250 DSN

Let me briefly explain what these lines mean:

  • PIPELINING
    This is a feature to speed up SMTP communication. Usually the remote system has to wait for a response to every command it sends. Pipelining allows the remote server to send bulks of commands without waiting for a response. Postfix will just store these commands and execute them one by one. If you told Postfix to forbid pipelining it would disconnect the remote server when it tries to send bulks of commands without waiting for the proper reply. It is mainly a feature against spamming programs that don’t behave.
  • SIZE 10240000
    The remote server is allowed to send emails up to 10 MB large. This has long been a common maximum size for emails. However nowadays 40 MB or even more are more common sizes because emails have grown larger.
  • VRFY
    Allows remote servers to verify a given name or email address. For example the remote server could send “VRFY john” and your server might respond “250 John Doe <john@example.org>”.¬†It can be used to verify that a certain recipient email address is deliverable
  • ETRN
    A command that a remote system can send to flush the Postfix queue of mails for a certain domain. It can be used if the remote system had technical problems and failed to receive email for a while. Then it could send an ETRN command to make your server start sending outstanding emails for that domain. It is rarely used.
  • STARTTLS
    This tells the remote system that it might start switching from this unencrypted to an encrypted connection by sending the “STARTTLS” command. It will then start negotiating a TLS-encrypted connection. You could compare it to an HTTP connection that suddenly switches over to an encrypted HTTPS connection. The advantage is that you can start talking SMTP on TCP port 25 and don’t have to open up a second TCP port like 465 which is the “SSMTP” (secure SMTP) port and only accepts encrypted connections.
  • ENHANCEDSTATUSCODES
    This enables more three-digit return codes for various conditions. See the RFC2034 if you are curious.
  • 8BITMIME
    In ancient times SMTP only processed 7-bit characters. You couldn’t transfer special characters like “√Ą” or “√ü” without special encoding. 8BITMIME allows a transmission of emails using¬†8-bit characters. Still many emails are specially encoded using ISO8859-1 or UTF-8.
  • DSN
    It enables DSNs (delivery status notofications) that allows the sender to control the messages that Postfix creates when an email could not be delivered as intended

However one important line is missing here that would allow us to send our username and password:

250-AUTH PLAIN LOGIN

We told Postfix to only allow authentication when the connection is encrypted. So we are not offered authentication over this plaintext connection.

Are you still connected? Okay, good. So we need an encrypted connection using TLS. You could enter STARTTLS:

STARTTLS

And the server would reply:

220 2.0.0 Ready to start TLS

However now it’s getting complicated because you would have to speak TLS encryption which is not a language that humans speak. But we can use OpenSSL to help us with the decryption. Terminate the SMTP connection. Now¬†run:

openssl s_client -connect localhost:25 -starttls smtp

You will see a lot of output. OpenSSL has connected to TCP port 25 and issued a STARTTLS command to switch to an encrypted connection. So whatever you type now will get encrypted. Enter:

ehlo example.com

And Postfix will send a list of capabilities that will look like this:

250-PIPELINING
250-SIZE 10240000
250-VRFY
250-ETRN
250-AUTH PLAIN LOGIN
250-ENHANCEDSTATUSCODES
250-8BITMIME
250 DSN

And now that we are using an encrypted connection Postfix offers us to authenticate. So let us send the authentication string with a Base64-encoded password:

AUTH PLAIN am9obkBleGFtcGxlLm9yZwBqb2huQGV4YW1wbGUub3JnAHN1bW1lcnN1bg==

The server should accept that authentication:

235 2.7.0 Authentication successful

Disconnect from Postfix:

QUIT

Goodbye:

221 2.0.0 Bye

Authentication works. Well done.

Base64-encoded passwords

You may wonder how I got to the long cryptic string that apparently contained John’s email address and his password. It is just a Base64-encoded (not encrypted!) version of the string “john@example.org<NULL-BYTE>john@example.org<NULL-BYTE>summersun”.

One way to create that string is using Perl:

perl -MMIME::Base64 -e \
'print encode_base64("john\@example.org\0john\@example.org\0summersun")';

The submission port

Although I have been talking about SMTP on port 25 to relay mails there is actually a better way: using the submission port on TCP port 587 (as described in RFC 4409). The idea is to use port 25 for transporting email (MTA) from server to server and port 587 for submitting (MSA) email from a user to a mail server. As many users still use SMTP by default you will likely have to run both ports.

Edit your /etc/postfix/master.cf file and add a service for the¬†submission port like this (mind the indentation from the second line on). Don’t use the suggested and commented-out submission service in the master.cf – just ignore it.

submission inet n - - - - smtpd
 -o syslog_name=postfix/submission
 -o smtpd_tls_security_level=encrypt
 -o smtpd_sasl_auth_enable=yes
 -o smtpd_sasl_type=dovecot
 -o smtpd_sasl_path=private/auth
 -o smtpd_sasl_security_options=noanonymous
 -o smtpd_sender_login_maps=mysql:/etc/postfix/mysql-email2email.cf
 -o smtpd_sender_restrictions=reject_sender_login_mismatch
 -o smtpd_sasl_local_domain=$myhostname
 -o smtpd_client_restrictions=permit_sasl_authenticated,reject
 -o smtpd_recipient_restrictions=reject_non_fqdn_recipient,reject_unknown_recipient_domain,permit_sasl_authenticated,reject

Basically this new service uses the “smtpd” daemon which is the piece of software that responds if you open an SMTP connection on TCP port 25. But it gets a few extra options set‚Ķ

  • in the /var/log/mail.log mark the connections to the submission port as “postfix/submission/smtpd” (syslog)
  • enforce encryption on this port (security level)
  • enforce user authentication (sasl)
  • make sure that an authenticated user can only send emails in their own name (sender_login_maps)

Restart the Postfix server:

service postfix restart

Your users can now use the submission port to send email.

Protecting against forged sender addresses

Wait a minute. We can make sure that a user can only send emails using their own sender address? Cool. How does that work? Take a look at this line we just used above when we added the submission service:

smtpd_sender_login_maps=mysql:/etc/postfix/mysql-email2email.cf

The smtpd_sender_login_maps is a Postfix mapping with two columns:

  1. the email address the user wants to use as a sender address
  2. the username that was used when authenticating at the mail server

The reason I am just using the¬†/etc/postfix/mysql-email2email.cf mapping is that it fits perfectly here. That mapping maps the email address to itself if it exists. Confused? Look at the¬†SQL query that the “email2email” mapping runs:

SELECT email FROM virtual_users WHERE email='%s'

So when a user wants to relay an email Postfix checks the virtual_users table looking for the sender address. Say that John wants to send out an email from his email address “john@example.org”. Postfix will check the¬†virtual_users table if there are any rows with an email field like that. It will find one and return the¬†email field – which is just again the email address. Postfix expects to get back the username that was used to login. We are using the email address as a login username – so that mapping works. Only if John would try to send an email with a forged sender address then Postfix will see that it does not match his account and reject the email. Your mail.log would read something like:

NOQUEUE: reject: RCPT from foo.bar[…]: 553 5.7.1 <forged@email.address>: Sender address rejected: not owned by user john@example.org; from=<forged@email.address> to=<…>

It is your own decision whether you want to use the smtpd_sender_restrictions like that.

ISPmail Jessie – Postfix SMTPd restrictions

Postfix provides the ability to apply filters during the SMTP session. When an email arrives at your mail server the following steps are usually run through:

  • CLIENT:
    Incoming TCP connection on port 25
  • HELO
    The sending mail server tells you its name.
  • MAIL FROM
    The sender’s name and email address of the email
  • RCPT TO
    The recipient the mail is intended for
  • DATA
    In this phase of the SMTP session the actual email is transmitted
  • QUIT
    End of SMTP session

With the appropriate settings of SMTPd restrictions you can control what checks Postfix will run when an email is received. Those restrictions are defined in the /etc/postfix/main.cf configuration file and are called smtpd_…_restrictions. You can for example restrict certain IP addresses to even open an SMTP (TCP/25) connection to your mail server during the “client” phase. Or you can filter what recipients are allowed during the “rcpt to” phase. According to the above list of phases these restrictions apply:

  • smtpd_client_restrictions (default: empty)
  • smtpd_helo_restrictions (default: empty)
  • smtpd_sender_restrictions (default: empty)
  • smtpd_recipient_restrictions (default: permit_mynetworks, reject_unauth_destination)
  • smtpd_data_restrictions (default: empty)

So by default Postfix only applies checks during the “RCPT TO” phase to check if the email is either coming from your local network (the sending server’s IP address is part of the “mynetworks” setting) or if the recipient has a valid accounton your mail server. Otherwise the email would get rejected. It is important that you understand that the restrictions are checked in the respective phase. So it’s pointless to apply checks on the sender’s email address (MAIL FROM) in the HELO phase because that information is not yet known at that phase. Many mail server administrators just put all the checks into the smtpd_recipient_restrictions. The following sections give you some advice on how to configure your restrictions properly.

Realtime blacklists (RBL)

The most important technique nowadays to fight spammers are IP blacklists. There exist dozens of these lists that you can use freely. The operators of such lists have different policies regarding who is getting blacklisted. But most of them have spam probes on the internet that spammers fall for. The technique of spam traps means that otherwise unused email addresses are put on a web site in a hidden place or use them on mailing lists. Spammers then automatically spider (=scan) the internet for email addresses and then send their spam to the found addresses. If such a trap account gets email then that information is used by the operators of an IP blacklist to build a list of server that send spam. It is potentially dangerous to use IP blacklists though because you might block email from a legit server if that server was inadvertently blacklisted. My favorite blacklist “SORBS” for example has let me down several times by blacklistting entire internet providers for a period of hours or even days. So it’s a good idea to check their policies and see if you agree with it. In the end the risk is yours. Once you reject an email you cannot get it back. But I still strongly recommend you use blacklists or you will literaly drown in spam.

IP blacklists are specified by a domain name like “bl.spamcop.net”. To check if a certain IP address is listed in that zone you need to check if an entry for the reversed IP address exists in that zone. Assume that a mail server with the IP address 217.217.34.231 wants to send you an email. If you told Postfix to use the “bl.spamcop.net” blacklist then Postfix will reverse this IP address and check the DNS entry of 231.34.217.217.bl.spamcop.net. I just did that and got a result:

$> host 231.34.217.217.bl.spamcop.net
231.34.217.217.bl.spamcop.net has address 127.0.0.2

The fact that a result (127.0.0.2) was found means that the IP is blacklisted and that I should not accept emails from it. Postfix will then tell the sender that it triggered a blacklist and cancels the connection. The result returned almost always is 127.0.0.x where the X stands for the reason of the blacklisting. You will have to read the policy of the respective blacklist to see the reason. Let’s take another IP address that is not blacklisted – for example 85.214.93.191:

$> host 191.93.214.85.bl.spamcop.net
Host 191.93.214.85.bl.spamcop.net not found: 3(NXDOMAIN)

I did not get a result. So the IP is not blacklisted and I should accept email from it.

There are a few major blacklists that you should check out:

  • SORBS (dnsbl.sorbs.net)
    SORBS is very efficient but every few months tends to blacklist even large ISPs accidentally. I currently do not use it as I had too many user complaints.
  • SpamCop (bl.spamcop.net)
    Pretty reliable. On a sample day it blocked 29 mails on my server.
  • SpamHaus (zen.spamhaus.org)
    Also pretty reliable. On a sample day it blocked 147 mails on my server.
  • UCEprotect (dnsbl-1.uceprotect.net)
    Careful – the Class-2 and Class-3 lists are too strict and block even legit senders. On a sample day it blocked 33 mails on my server.

If you used these blacklists then the restrictions in your /etc/postfix/main.cf would be these:

smtpd_recipient_restrictions =
permit_mynetworks
reject_rbl_client dnsbl.sorbs.net
reject_rbl_client bl.spamcop.net
reject_rbl_client zen.spamhaus.org
reject_rbl_client dnsbl-1.uceprotect.net
reject_unauth_destination

(Note: You can also list the restrictions all in one line and seperated by commas. But I prefer the way to use multiple lines and indent the subsequent lines. It makes the main.cf file more readable.)

With these restrictions in place Postfix will run each of these checks after the “RCP TO” phase during the SMTP dialog.

  1. Is the sending IP address in your network range?
  2. Is it blacklisted by SORBS?
  3. Is it blacklisted by SpamHaus?
  4. Is it blacklisted by UCEprotect?
  5. Is your mail server responsible for the recipient’s domain? Or is the sender using SMTP authentication for relaying?

Further restrictions to consider

Postfix offers quite a lot of checks you can use in your smtpd_…_restrictions. Just to name a few that I use on my mail servers:

  • reject_unknown_client_hostname
    Runs a two-way DNS check of the IP address and hostname of the sending server. This can be potentially dangerous and reject legit email from misconfigured mail servers but nowadays every decent mail server is supposed to take care of their DNS settings.
  • reject_unknown_sender_domain
    Checks if the domain used in the sender’s email address actually exists. This runs a check for an MX and A record of the sending domain.
  • reject_unauth_pipelining
    Some spam bots just try to deliver emails as fast as possible. But formally the sending server must wait for the recipient server’s response in each SMTP step. If the recipient server allows pipelining then fast sending is allowed. This setting drops the connection if pipelining is attempted before it was offered by the destination server.

Further settings can be found in the Postfix documentation.

Are you blacklisted?

Of course you want to send out emails through your mail server. So you need to make sure that your server IP address is not blacklisted either. I suggest you check your server’s IP address through one of the blacklist test services:

(These services also give you an idea what other blacklists are available.)

If you are listed on any of these lists then you will want to get off the lists. How you do that depends on each blacklist’s policy. Some don’t even support manual removal from these lists. If your ISP assigned you a dynamic IP address then you are in a so called “dialup IP range” and are most likely blacklisted and it’s pointless to get off the list. A lot of spam is coming from dynamic IP addresses. Either spammers send email from their ISP account. Or the large amount of virus-infested Windows PCs in the world is forwarding spam. So dynamic IP addresses are frowned upon by mail server administrators.

Source spoof prevention

The above section on IP blacklists helped you fight spam by rejecting it. Now let’s add another security measure to prevent source address spoofing. Spoofing means that someone fakes information. And the source address is the email address that appears to be the sender of an email. So it means that somebody pretends to be someone else by faking the email address of the sender. Maybe you haven’t given it much thought but basically everyone can send an email and pretend to be someone else. Perhaps you are even using multiple sender addresses yourself. What prevents someone from pretending he is you? There are two common options:

  1. SPF (Sender Policy Framework): Using SPF anyone can find out if an email was sent from an IP address (=mail server) that is allowed to act as a mail server for that sender domain. Technically with SPF you create a DNS entry for your domain and list the valid IP addresses there that are allowed to send email on your domain’s behalf.
  2. DKIM (DomainKeys Identified Mail): DKIM uses a private/public key pair to sign all outgoing emails automatically. By making your domain’s public key available through DNS every mail server on the internet can check if the signature is valid for your domain and nobody spoofed the email. Although SPF is more widespread than DKIM it has a major design flaw because it fails when you forward email (as the sending IP address changes and may not be a valid address listed in the SPF entry any more).

Setting your own SPF entry

The content of the SPF entry follows a certain syntax. To make things easy you can use the service at OpenSPF to set up your SPF entry. Enter your domain name there and fill out the fields. The web site will tell you how your SPF entry will have to look like. (OpenSPF vanished in 02/2019.) Add this to your DNS zone and you will both help others to fight spam as well as help yourself to prevent that anyone is sending unwanted emails on your behalf. Formerly a TXT resource record in the DNS zone was used but nowadays there is a distict SPF record type used for that purpose. After quite some controversy the “SPF” type of DNS records has been discontinued. So Using a TXT records is the way to go.

Checking other SPF entries

Many domains already have SPF entries. So you are encouraged to use that information for your own spam and phishing protection. Read the next section on greylisting to learn how to use the “tumgreyspf” service to use both the power of SPF and greylisting.

Forwarding breaks SPF

SPF has a major design flaw. It fails if someone forwards email to another server. Then the forwarding server is not listed in the SPF entry and the destination mail server will (correctly) reject the email. Less technically-savy users will not understand this limitation. So many mail administrators just take the SPF checks into account when looking for spam but they don’t ultimately reject the email. It’s up to you. SPF reliably fights spoofing and phishing for major domains like PayPal, eBay, Amazon or other large organisations that are often a target of phishers. In some cases it will just be overly careful.

Checking DKIM signatures of incoming email

DKIM was invented for a similar reason as SPF. It provides email authentication – to allow a receiving mail server to check that the email is authentic for a given sender domain. SPF just tells which IP are allowed to send email for a sender domain. But DKIM is working way more reliably because it adds a cryptographic signature to every outgoing email. To verify if an email is authentic a mail server can use the email’s signature and match it with the sender domain’s public key that can be obtained by DNS. Sounds too complicated? Actually it’s pretty simple if you understand the basics of public-key cryptography. Have you ever used PGP? Then you know that you had to create a keypair consisting of a private key you can use to sign emails and a public key that you can give others to encrypt emails for you or check your signatures. DKIM uses exactly that technique. The sending mail server has a keypair for each domain. When it sends out an email it will use the private key for the sender domain, take the email that is to be sent out and sign it with a cryptographic signature. That signature is added to the email (as an additional mail header) and the receiving mail server can take the email and the signature from the mail header and the public key of the sender domain to determine whether the signature was valid.

Adding automatic verification of other domains’ DKIM signatures is pretty easy. Just install the DKIM filter software:

apt-get install dkim-filter

Actually this software is called a “milter”. Milters are “mail filters” that were introduced as additional pieces of software for the ancient Sendmail mail server software. Milters speak a certain protocol that mail server can use to communicate with it and Postfix fortunately knows how to use milters, too. Right after the installation the “dkim-filter” process is already started and running in the background. But as Postfix is running in a chroot jail in /var/spool/postfix it cannot use the DKIM filter’s communication socket at its default location in /var/run/dkim-filter/dkim-filter.sock. You can fix the permissions and make DKIM run a socket in /var/spool/postfix instead but I prefer to rather run dkim-filter as a TCP service rather than a socket. Edit your /etc/default/dkim-filter file that contains its default settings and add

SOCKET=”inet:54321@localhost”

to it. Restart the dkim-filter process:

/etc/init.d/dkim-filter restart

Now dkim-filter is listening to TCP port 54321 on the localhost interface. To make Postfix use this milter run these commands to add the appropriate milter definitions to your /etc/postfix/main.cf file:

postconf -e smtpd_milters=inet:127.0.0.1:54321
postconf -e non_smtpd_milters=inet:127.0.0.1:54321

Now Postfix will run all emails (both incoming and outgoing) through your DKIM filter and check DKIM signatures (if found) on incoming emails. It will not sign outgoing emails though – this is covered in the next section.

A side note: you can decide what should happen if there is a problem with the DKIM filter. By default Postfix will temporarily reject emails until you fixed the issue. It’s up to you. But I prefer to just accept emails if there is a milter problem. So I set:

postconf -e milter_default_action=accept

For more information on milters please consult the Postfix documentation on milters.

Setting up DKIM for sending emails

The actual idea of DKIM is to make a domain’s mail server sign outgoing emails automatically. To do that you need to create a cryptographic keypair first. Fortunately you do not have to buy a certificate because publishing the key in your DNS zone is enough. The tool you need is “dkim-genkey” and it requires the bit-length of the key and the domain name. You can optionally use a selector to use multiple keys. By default the selector is called “default”. I suggest you use the domain name as your selector which will name your key files correctly automatically.

It’s best to create a seperate directory to store the keys. I keep mine in /etc/postfix/dkim:

mkdir /etc/postfix/dkim
cd /etc/postfix/dkim

Then create the keypair there:

dkim-genkey -b 1024 -d example.org -s example.org

In your current directory you will now find two files.

  • “example.org.private” is the private key that the DKIM milter will use to sign all outgoing emails
  • “example.org.txt” is the public key that you are supposed to add to your domain’s DNS zone

The configuration file for the DKIM milter is /etc/dkim-filter.conf. Edit that file. The only change you need is to add a line for the key list file like:

KeyList     /etc/dkim-keys.conf

Create a new file /etc/dkim-keys.conf that you just pointed to. Its contents define which key is used for which sender address. Or to be specific the syntax is:

sender-pattern:signing-domain:keypath

as pointed out in the “man dkim-filter.conf” man page. So if your domain were example.org you would write:

*@example.org:example.org:/etc/postfix/dkim/example.org

Add one line per (sender) domain that you wish to sign outgoing emails for.

Caveat: dkim-filter adds “.private” to the keypath you specify. So the above line is correct. Do not specify /etc/postfix/dkim/example.org.private or signing will fail.

Restart your DKIM milter:

/etc/init.d/dkim-filter restart

Now outgoing emails should get a DKIM signature automatically. Send an email to another mail server and check the headers of the email. You should see a header like this:

DKIM-Signature: v=1; a=rsa-sha256; c=simple/simple; d=example.org; s=example.org t=1307571770; bh=OtZOIOSkkhL1t+kR5KOE6HvvjUjLkDE+70agcKmcjJg=; h=Message-ID:Date:From:MIME-Version:To:Subject:Content-Type: Content-Transfer-Encoding; b=RlxE9IRiOJrOzM4JbJFIp/YmK4XBRJFti79rL0vzGkppAC1AJi2CdtwGFT/sTovt/iKikrdyQ7M7JhdlC9u3bh8rrvhLA53HZCmu6WvHvv059ysdpjUhrktEqZFrFpgds2wqCkzF4ar/ly3TzYZmoZJO+C8j2uU5L3cKzESfGw4=

The meaning of each parameter can be found on the Wikipedia page. or in the less human-friendly format definition in the RFC 4871.

However the recipient cannot tell if this is a correct signature because they don’t have the public key to verify this signature with. Edit your DNS zone for your domain and just add the TXT record that “dkim-genkey” created for you in /etc/postfix/dkim/example.org.txt. On my test server it looked like this:

example.org._domainkey IN TXT “v=DKIM1; g=*; k=rsa; p=MIGfMA0GCSqGSIb3DQEBAQUAA4GNADCBiQKBgQDKPFhCADCGcxDchGHFqvSoQYuW0epBT0gJuCwDMkX0uPejeeMdBNDcaGo8AyZdXJTFKqGK5kao2OvpDsH6XeKMBjPt/CnyBm4PNwlwNrkJTBc15xjD6Swmlk457+Ioz/tbpBA3b3RnC8NsqiknLQ3JxDkE7fXfji7Uds5+swWwJQIDAQAB” ; —– DKIM example.org for example.org

Verify that your DNS zone for your domain really contains the right entry by querying for a TXT record with your domain name and the selector you used:

dig +short example.org._domainkey.example.org txt

And it should return your public key:

“v=DKIM1\; g=*\; k=rsa\; p=MIGfMA0GCSqGSIb3DQEBAQUAA4GNADCBiQKBgQDKPFhCADCGcxDchGHFqvSoQYuW0epBT0gJuCwDMkX0uPejeeMdBNDcaGo8AyZdXJTFKqGK5kao2OvpDsH6XeKMBjPt/CnyBm4PNwlwNrkJTBc15xjD6Swmlk457+Ioz/tbpBA3b3RnC8NsqiknLQ3JxDkE7fXfji7Uds5+swWwJQIDAQAB”

Now all outgoing emails for …@example.org will get a DKIM signature added automatically and recipients can verify them. Any mail server relying on DKIM signature checks can now check whether emails from your domain are legit.

I assume that you have set up the DKIM milter in Postfix as described in the previous section. Setting “smtpd_milters” and “non_smtpd_milters” in your /etc/postfix/main.cf will now do both the verification of DKIM signatures as well as the signing.

Greylisting

Another way to help you reduce the amount of spam you receive is using greylisting. Whitelisting means that a certain kind of email is always accepted (delivery status  2.x.x). Blacklisting is the opposite and always blocks certain emails (delivery status 5.x.x). Greylisting is something in between and temporarily refuses to accept the email for a couple of minutes. The idea is to tell virus infested computers trying to spam you from actual mail servers. Viruses (or rather: spam worms) just try to deliver a spam email once. But real mail servers have a queue of outgoing emails and upon a temporary error they will try to re-deliver the email. If the sending mail server has never sent us an email then greylisting will send back a temporary error code (delivery status 4.x.x) which signals the sending server to try delivering the email again in a few minutes. Your server will take a note of the IP address of the sending server and after a set period (a common value is 10 minutes) the blockade is lifted and upon the next delivery attempt the email will be accepted. Also the greylisting service will take a note in its cache and further delivery attempts will be allowed instantly.

The obvious drawback is that emails are often delayed by several minutes – especially if you just start using greylisting and the cache of trusted sending servers is empty. It is your decision whether your users will accept that. In my personal experience greylisting is not as effective nowadays as it has been in the beginning. It appears like malware now just tries to deliver spam emails several times. But it’s one measure to maybe block some spam emails.

There are several ways to implement greylisting. The way I do it is using the “tumgreyspf” software. Start by installing it:

apt-get install tumgreyspf

To make Postfix use tumgreyspf you have to define a service for it in the /etc/postfix/master.cf:

tumgreyspf unix¬† –¬†¬†¬†¬†¬† n¬†¬†¬†¬†¬†¬† n¬†¬†¬†¬†¬†¬† –¬†¬†¬†¬†¬†¬† –¬†¬†¬†¬†¬†¬† spawn
user=tumgreyspf argv=/usr/bin/tumgreyspf

You also need to add it to your smtpd_recipient_restrictions of your /etc/postfix/main.cf file. Example:

smtpd_recipient_restrictions =
permit_mynetworks
permit_sasl_authenticated
reject_rbl_client bl.spamcop.net
check_policy_service unix:private/tumgreyspf
reject_unauth_destination

These are example restrictions. Do not just copy/paste it. The example is just meant to show you a good place to add the policy check in your smtpd_recipient_restrictions.

Finally restart Postfix:

postfix reload

To customize the behavior of tumgreyspf you can alter the files in /etc/tumgreyspf.

(This procedure is also documented in the /usr/share/doc/tumgreyspf/README.Debian file.)

© 2021 workaround.org - Proudly powered by theme Octo