ISPmail tutorial for Debian Lenny

A spanish translation of this tutorial is also available - courtesy of José Ramón Magán Iglesias.

What this tutorial is about

You surely know the internet service providers that allow you to rent a domain and use it to receive emails. If you have a computer running Debian which is connected to the internet permanently you can do that yourself. You do not even need to have a fixed IP address thanks to dynamic DNS services like dyndns.org. All you need is this document, a cup of tea and a little time. When you are done your server will be able to...

License/Copyright

This tutorial book is copyrighted 2009 Christoph Haas (email@christoph-haas.de). It can be used freely under the terms of the GNU General Public License. Don't forget to refer to this URL when using it. Thank you.

Changelog

Things you will need

The server setup described here is totally standard work for a professional system administrator. Depending on your level of knowledge and experience you may walk through this document easily or curse the author and fail miserably. You will need to know or learn about different topics regarding basic system administration (e.g. how do I edit a text file, where are my log files), DNS, SMTP, MySQL and POP3/IMAP. You also need root access to an existing Debian Lenny server or be able to install one. Also you will likely have to change DNS records for your domain. If your server is protected by the firewall make sure you can change its rules. Going through this setup takes you between 2 hours (for a professional) and a week (if you are a beginner who will inevitably make mistakes). So make sure you are not in a hurry.

About this document

Many years ago I wanted to turn my Debian server into a mail server with virus scanning, spam detection, email forwarding ("aliasing"), POP3 and IMAP access and a webmail service. All the components were there but it took a while until they worked properly together. So I summed up my desk full of scrawly notes into a tutorial that has become pretty famous. According to my web server statistics it's the main reason why people visit workaround.org.

This document is not a simple copy-and-paste tutorial where you just copy the commands from the web site and run it on your server. Instead it will make you understand the different components that you are setting up. In the end you will be skilled enough to debug problems yourself. If you feel you need help with your setup then try the hints in the Troubleshooting section or ask on the mailing list. The setup in this tutorial has been tested very thoroughly by many readers. Unlike many other Postfix tutorials on the internet this is already the fifth edition. Writing this tutorial took a lot of work so these are not just quick draft notes thrown together but a consistent document guiding you.

The whole tutorial is split into several chapters. Please use the links on the right side or below to navigate through the tutorial. If you prefer all content on a single page (e.g. for printing the tutorial) then use the "printer-friendly" link below. You are also invited to comment on the pages - just click on the "Add new comment" link at the bottom.

If you like the tutorial then a tiny donation is appreciated to run the test server and to pay for the internet connection.

The Big Picture

The Software we will use

The configuration described here uses these software components (the versions are Debian Lenny's default versions):

The wonderous Ways of an Email

Before going into the details let's see the big picture:

  1. An email is sent to your server via the SMTP protocol on TCP port 25. Postfix accepts the connection, reads the email and does some basic checks. Is the sender blacklisted on a realtime blacklist? Is the email from an authenticated user so we bypass relay checks? Or is the recipient of the email a valid user on our system? If we don't trust the remote system yet we apply greylisting. At this stage Postfix can reject the email or accept it.
  2. Postfix forwards the email via the SMTP protocol on the TCP port 10024 to AMaViS for content checking. Notice that at this stage the email can't be rejected any more. So AMaVis can either accept it or throw it away. Commonly AMaViS is configured to add a certain email header so the user can see that AMaViS thinks it is spam.
  3. AMaViS lets SpamAssassin check the email for spam. SpamAssassin will be taught which emails are spam to increase its detection accuracy.
  4. AMaViS also runs the email through ClamAV to see if it contains any viruses.
  5. After these checks AMaViS returns the email to the Postfix process but on TCP port 10025. Postfix is configured to trust emails sent to this port so further content checks are skipped.
  6. Postfix forwards the email to Dovecot. Dovecot can optionally apply per-user filters so that emails can be stored in certain email folders automatically if desired.
  7. Dovecot writes the email to the hard disk in maildir format.
  8. The user's email client can now fetch the new emails from Dovecot using the POP3 or IMAP protocol.

Migrating from the Etch tutorial

The ISPmail tutorial is maintained since 2002. You may have followed former versions of the tutorial and now want to know how to upgrade your mail server to Lenny properly. It is hard to provide exact instructions on what steps to go through. Here are some issues:

Database schema change

I know you'll hate me for that. But the normalized database layout used in the Etch tutorial was overshooting. So this tutorial uses a more human-readable layout which isn't that much different actually. And it's lighter on the database because the former queries used string operations on a view which might get slow on mail servers with very many users. To migrate your database please first make a backup of it (you never know). Then issue these SQL queries which should migrate your database painlessly to the new schema:

-- Create an additional 'email' column in the virtual_users table
ALTER TABLE virtual_users ADD email VARCHAR(100) NOT NULL;

-- Fill the 'email' column with the complete email address.
UPDATE virtual_users LEFT JOIN virtual_domains ON virtual_users.domain_id=virtual_domains.id SET email=concat(virtual_users.user,'@',virtual_domains.name);

-- Remove the 'user' column
ALTER TABLE virtual_users DROP user;

-- Drop the 'view_users' view
DROP VIEW view_users;

-- Enlarge the email fields in the virtual_aliases view
ALTER TABLE virtual_aliases CHANGE source source VARCHAR(100);
ALTER TABLE virtual_aliases CHANGE destination destination VARCHAR(100);

-- Rewrite the source side of the virtual_aliases to the full email address
UPDATE virtual_aliases LEFT JOIN virtual_domains ON virtual_aliases.domain_id=virtual_domains.id SET source=concat(virtual_aliases.source,'@',virtual_domains.name);

-- Drop the 'view_aliases' view
DROP VIEW view_aliases;

You will also need to adjust the ".cf" configuration files as described in the respective chapter.

Mail directories now under /var/vmail

The FHS (file hierarchy standard) suggests to put mails under /var/mail. Previously the tutorial expected to put all mails under /home/vmail but the mail directories do not really belong there. So the new tutorial uses /var/vmail as a location. This issue is a bit nasty as you need to move your Maildirs. This tutorial introduces server-side filtering to allow users to apply pre-defined filters when an email arrives. Dovecot stores these filter files in the user's mail directory so the actual mails need to be moved one level deeper in your directory structure.

It's just cosmetics so you don't have to move your mails there. But if you do it then change the following files:

AMaViS configuration file

Previously the tutorial recommended to put your custom AMaViS configuration into the /etc/amavis/conf.d/20-debian_defaults file. This will make this file risk getting changed during an update. Please move your settings to the /etc/amavis/conf.d/50-user file instead.

Global Dovecot/Sieve config file

In recent Dovecot versions the configuration directory for the global sieve filter file has changed. Previously it was configured as the "global_script_path" in the "protocol lda" section. Now it's the "sieve_global_path" setting in the /etc/dovecot.conf "plugin" section. See also.

Dovecot now creates maildirs automatically

The Dovecot version used in Debian Etch threw errors if a user accessed a mailbox which had not received any email yet. The Debian Lenny version does not have this issue. So you don't have to send the user a "welcome mail" or even create the maildir manually.

Dovecot's configuration file

In the /etc/dovecot/dovecot-sql.conf file you need to change the line

password_query = SELECT email as user, password FROM view_users WHERE email='%u';

to

password_query = SELECT email,password FROM virtual_users WHERE email='%u';

Debian's release notes

Also you should read Debian Lenny's release notes before attempting to upgrade your system from Etch to Lenny.

The Basics: Virtual Domains in a Database

Before going into detail about virtual domains let's first understand the concept of...

Local domains

Postfix is the component that receives emails from the internet. Typically Postfix knows about local domains (configured in the "mydestination" setting) and local users (those who can log into the system and are listed in the /etc/passwd file). This means that all system users will get emails for any local domain. As an example you may have set...

mydestination = example1.com, example2.com, example3.com

Let's say you created a system user "johndoe" (e.g. using the "adduser" command). This simple setup will make Postfix receive emails for

You can't make johndoe's account just work in one domain. So this is not feasible for different users in different domains. Neither will it work well with many users as you had to create system accounts for each of them.

Virtual domains

Instead of defining your email domains in a text file and creating many system accounts there is a better choice for non-trivial Postfix-based mail servers. Domain names and user accounts can be stored in a directory like LDAP or a database like MySQL or PostgreSQL. Postfix will just need to know how to access the database to get this control information. In such a configuration email domains that Postfix will be responsible for are called virtual domains. Similarly the user accounts are called virtual users. They just live in a database.

There are two basic types of virtual domains that Postfix knows. "Virtual alias domains" can be used for forwarding ("aliasing") email from an email address to another email address (or multiple addresses). Virtual alias domains do not receive email for any users. They only forward mail somewhere else. The virtual_alias_maps mapping contains forwardings (source, destination) of users or domains to other email addresses or whole domains. Incidentally virtual_alias_maps also works for local email addresses, too. So you do not really need virtual alias domains as you can declare all domains as virtual mailbox domains and use virtual alias maps for aliases.

The other type of virtual domains are "virtual mailbox domains" which are more important here. They define domains which are used to actually receive emails.

 

All this starts to sound fancy? Don't panic. Let's see an example of such a database table that tells which user accounts can receive emails and where those should be stored on your system:

Example for virtual_mailbox_maps
Virtual user Virtual mailbox location
john@doe.org /var/mail/doe.org/john/Maildir
jack@doe.org /var/mail/doe.org/jack/Maildir
jeff@foo.org /var/mail/foo.org/jeff/Maildir

The above is a perfectly valid example for what Postfix expects as a mapping called "virtual_mailbox_maps".

Although you may argue that the above table contains all necessary information there is one more thing that Postfix expects: a list of (virtual) domains. In the above example that list would consist of doe.org and foo.org. That mapping table would be used as "virtual_mailbox_domains" and looked like this:

Example for virtual_mailbox_domains
Virtual domain Just some dummy string
doe.org banana daiquiri
foo.org tequila sunrise

You will probably wonder why there is a second column with seemingly useless data. The reason is that Postfix always expects two columns in a mapping. The left column (or "left-hand side"=LHS) is usually the key and contains what Postfix is looking for. The right column (or "right-hand side"=RHS) is what tells Postfix what to do. For the list of virtual domains Postfix just looks for any non-empty result in a line where the domain is listed on the left. Some people just write "OK" in there - it doesn't matter.

You have now seen that a mapping assigns one value to another. If you query a database you need to tell Postfix which two columns you mean. This is done through '.cf' configuration files as documented at http://www.postfix.org/MYSQL_README.html or through "man 5 mysql_table".

Example file:

# Information on how to connect to your MySQL server
user = someone
password = some_password
hosts = 127.0.0.1

# The database name on the servers.
dbname = mailserver

# The SQL query template.
query = SELECT destination FROM virtual_aliases WHERE source='%s'

This file defines the way that Postfix can get data from your database. It would be suitable for a virtual_alias_maps mapping (it is used for email forwarding - we will get to that later). Imagine you saved the above lines into a configration file /etc/postfix/mysql-virtual-alias-maps.cf. Then the following line in your /etc/postfix/main.cf would make Postfix query the database:

virtual_alias_maps = mysql:/etc/postfix/mysql-virtual_alias_maps.cf

How does this work? Imagine that Postfix is about to send an email to john@doe.net and wants to check the virtual alias map. Postfix then opens up a connection to the MySQL server at the IP address 127.0.0.1 and authenticates to the MySQL server with the username someone and the password some_password. It selects the database mailserver and finally runs a query replacing '%s' by what it's looking for:

SELECT destination FROM virtual_aliases WHERE source='john@doe.net'

Assume this query returned these rows:

That would be as if you had a text file with an alias like this:

john@doe.net -> jack@example.com, jeff@example.com, kerstin@example.com

So much for a quick introduction on how mappings are used with databases.

Basic Debian Installation

Unless you have a pre-installed Debian server you should instead do the installation yourself. This gives you a better choice on the disk partitioning and choice of file systems. Just skip this chapter if you have no console access or (re)installation from scratch is not an option.

As a general guideline your system should have at least 2 GB of RAM and lots of disk space living on a hardware-based RAID controller. Users will collect a lot of mails with a lot of useless attachments. And don't expect them to clean up their inbox - especially when offering IMAP. Project a 500 GB disk for 100 users. Regarding the CPU load there is mainly spam and virus scanning which is using up computing power. A quad core system will probably handle around 5-10 emails per second. If you don't need spam checking then a smaller CPU will also do.

Btw, I have created a quick screencast covering the first steps and especially the partitioning of the server with the logical volume manager. If you get stuck somewhere please watch this partitioning video screencast. (OGG video format)

Insert a boot medium (e.g. CD 1, Netinst-CD or DVD 1) and boot the system. You should see the splash screen:

Just choose "Install".

You will next get asked to choose the default language used for your system.

Always use "English" here even if your native language is something else. Otherwise system processes may write non-english log messages which may cause problems in automatic scripts. Besides you may look for help on the internet and will get better help with the english messages.

Continue with sensible choices until you get to the partitioning menu:

Choose "Manual" here so you get the option to use the logical volume manager. The logical... what?

Logical volume manager (LVM) in a nutshell

It is strongly suggested that you choose to use the LVM (logical volume manager) here. It scares many system administrators but it's not half the black magic you might expect. Besides Debian does the initial setup for you anyway. These are some of the advantages you would gain from using LVM:

This is LVM vocabulary you will stumble upon:

To sum up: you take a few classical partitions (PVs), combine them into something larger (VGs) and finally take parts (PEs) of that large space to get logical partitions (LVs) where you put your file systems on. A quick diagram may help visualize this. First a typical mail server partitioning without LVM:

As you can see the partitions and their sizes are fixed. If you need more space on "/" later you are screwed and have to copy everything to a new disk thus creating a major downtime of your mail server. With LVM this could rather look like this:

In the above example you have two physical hard disks /dev/sda and /dev/sdb. Booting from an LVM partition is in theory possible but it's recommended you have a boot partition on its own. So /dev/sda1 is a classical small ext2 or ext3 partition. /dev/sda2 and /dev/sdb1 are partitions of type 8e (Linux LVM). These two partitions are the physical volumes that provide the actual physical space where data is stored. They are aggregated into a volume group that is called "vgmain" here - it's just an abitrary name. Now this volume group is divided up into logical volumes that are partitions that you can put file systems on and mount as usual. The logical volume called "lvvar" will later appear in your system as /dev/mapper/vgmain-lvvar and can be used just like a physical partition. And you still have 537 GB left for later use in this example to extend either / or /var if you lack space there.

(If you are still not yet sure if you understood the LVM then there is another nice explanation at tuxradar.)

Setting up the partitions

Anyway. You chose "Manual" before and should now see a screen similar to this (your hard disk will hopefully be larger than 8 GB):

Go down to the "FREE SPACE" line and press Enter. Let's start with /boot. Choose to create a new partition. Make it at least 100 MB. You will now be asked how to use this new partition:

Set the options as shown above. It should be ext2 or ext3, mounted at /boot and be bootable. Then choose "Done setting up the partition". You will be sent back to the overview of your partitions and can see your new /boot partition:

Set up all remaining space as "physical volume for LVM" as in:

Now that you created a physical volume there is a new option "Configure the Logical Volume Manager" in the overview:

Select this option and after a quick confirmation dialog (choose "Yes" there) you will enter the LVM configurator:

Create a volume group

Create a (new) volume group and give it a name. For example call it "vgmailserver" (the "vg" makes it clearer later that this is a volume group). You will get asked which physical volumes this volume group is supposed to consist of - select them.

Create the logical volume for / (root)

Now use this volume group to "Create a logical volume" and select your volume group (vgmailserver) to contain it. Give the new logical volume an explanatory name like "lvroot" (the "lv" makes it clear that it's a logical volume). Make it 2 GB large. This partition will later hold your basic operating system binaries.

Create the logical volume for swap

Even if you have a lot of RAM you should create a swap partition. It will be used to swap out parts of your memory that are not actively needed. The freed RAM can automatically be used as disk cache then. Create another logical volume "lvswap" as a swap partition and make it 1 GB. (If your system starts to swap like crazy then either you have too little RAM or something is going wrong and more swap wouldn't save you anyway. Note: a decent mail server has at least 2 GB RAM.)

Create the logical volume for /var

This partition is where your database (/var/lib/mysql) and mail directories (/var/vmail) will be stored. You will likely need to extend this partition later. So create another logical volume "lvvar". Unless you already know how much space you need just start with 2 GB.

(If you want to earn extra points here you should create seperate logical volumes for /var/lib/mysql and /var/vmail. This allows quick MySQL backups by locking the tables, making an LVM snapshot and backing up the raw files safely. This is much faster than the old-school way by using mysqldump. You can as well set the "noatime" mount flag on /var/mail to save an extra write access on every read access to an email. And you can decide whether you want to create a logical volume for /tmp or - if you have enough RAM - use a tmpfs for it. Keep in mind that virus scanning takes place in /tmp so don't go penny-pinching here.)

LVM configuration is done

In the "Partition disks" dialog you can choose "Display configuration details" and verify that your LVM configuration looks like this:

Finish the LVM configuration and the logical volumes will be shown in the overview:

All there is left to do is format the logical volumes. Format "lvroot" using an XFS file system mounted at "/". And format "lvvar" as an XFS file system mounted at "/var". Finally set up "lvswap" as a swap area.

XFS, ReiserFS or ext3?

There are different file systems available and many people seem to use ext3 nowadays. XFS, ReiserFS and Ext3 are  journaling file systems. It's just that ext3 does not seem to trust its journaling much because after a number of mounts or a certain time span and a reboot it does a full file system check! That doesn't sound too bad? In real life that means that if after half a year of running your mail server if you need to reboot it then get a pot of coffee because this check can easily take hours or - in worse cases - days. This is obviously totally unacceptable. That automatic file system check can be disabled but it is discouraged.

Be aware that XFS has disadvantages, too. One disadvantage of XFS though is that existing partitions can only be extended but not shrunk. XFS is also dangerous if your server dies suddenly due to its write caching. So battery-backed write-cache or an uninterruptible power supply (UPS) is strongly recommended. If these disadvantages scare you then an alternative to XFS may be ReiserFS.

Done with the partitioning

If all is done the partitioning should look like this:

Admittedly this was a lot of work. But this time was well-invested. If later you need more space in any of these partitions you don't even have downtime. For example if you need 10GB more space in /var you would issue these two commands:

(Btw, you may want to create a rescue CD like with "grml" in case something goes wrong later and you need to get your data back from an LVM partition. Boot the CD with "grml lvm" and your logical volumes are automatically activated so you can mount them.)

Preparing the system

If your server was upgraded from "Etch"

If your server has been upgraded from a former Debian 4.0 "Etch" installation then you may need to fix two files. First make sure that your /etc/hostname contains the host name without the domain part. It does that on Lenny by default but if you are upgrading from a previous "Etch" installation then this may be wrong. The file /etc/mailname is supposed to contain the fully-qualified host name with the domain part.

Your /etc/hosts may also need to be fixed. Run hostname --fqdn and see if you get the fully-qualified hostname. If you just get the hostname without the domain please check that your /etc/hosts file has the fully-qualified hostname first in the list.

Wrong:

20.30.40.50   mailserver42 mailserver42.example.com

Right:

20.30.40.50   mailserver42.example.com mailserver42

Installing packages

Then begin by installing Postfix with MySQL backend support:

$> aptitude install postfix-mysql

This will also install the postfix package automatically. Exim (the default mail service installed with a fresh installation) will be removed. When asked for the general type of configuration choose "Internet Site". Answer the question for the "System mail name" by entering the fully qualified hostname of your system.

Run this line on the system to install the MySQL server:

$> aptitude install mysql-server

Note: You can run MySQL server on the same system as your actual mail service - but don't have to. The mail server can communicate with the MySQL server via TCP networking. During the installation you will get asked for the password of a newly created "root" user. This is not your system administrators login account but a special user to access the MySQL server. Choose any password you want and write it down. And another warning: during the installation a MySQL user account called "debian-sys-maint" will be created with a random password. The password is stored in the /etc/mysql/debian.cnf file. Do not touch this file or change the user's password in the database or you won't be able to stop or start the MySQL service any more.

You will want to offer POP3 and IMAP services to your users so you need to install Dovecot's services:

$> aptitude install dovecot-pop3d dovecot-imapd

Some of the packages needed to scan for virus-infected attachments are not "free enough" to be included in Debian's main section (e.g. "unrar" or "lha"). If you want to install them you first need to add the non-free section to your Debian mirrors in the /etc/apt/sources.list file. You usually just need to add "non-free" to your existing mirror lines for the Debian archive like this:

deb http://ftp.debian.org/debian/ lenny main non-free

and run

$> aptitude update

to update your package cache with the list of "non-free" packages.

It's a good service to your users to filter out spam and viruses for them. AMaViS is doing a good job here to detect unwanted emails:

$> aptitude install amavisd-new spamassassin clamav-daemon lha arj \
unrar zoo nomarch cpio lzop cabextract

AMaViS is now installed with a number of suggested packages to scan attachments for viruses and detect spam.

If you intend to offer a webmail service I can recommend the Squirrelmail package. It will automatically install the Apache web server if you do not yet have one installed. Type:

$> aptitude install squirrelmail

As your control information for Postfix will be stored in a MySQL database you may want to install the PhpMyAdmin software that allows you to manage the database and its data in your web browser:

$> aptitude install phpmyadmin

(If you get asked which web server to configure you will probably want to choose "apache2".)

The console-based mutt email client lets you read mail from mailboxes directly from the hard disk. It will be helpful for testing the configuration. And it's even a very powerful IMAP email client that many people use as their main mail program. Maybe you'll start to like it, too. You should install it at least for testing:

$> aptitude install mutt

Now all basic packages are installed and it's time to prepare the database in the next chapter.

Preparing the database

Your Debian server should now be installed. So it's time to prepare the MySQL database that will store information that controls your mail server. In the process you will have to enter SQL queries. You can enter them on the 'mysql' command line. But if you are less experienced with MySQL I suggest you start easy with "phpMyAdmin" by pointing your web browser at this URL: http://YOUR-MAIL-SERVER/phpmyadmin. You should see a web page like:

Create the database

Your first task is to create a new database in MySQL. Let's call it 'mailserver'. In a root shell enter this command:

$> mysqladmin -p create mailserver

You will be asked for the MySQL "root" password that you entered when you installed the MySQL package.

(In phpMyAdmin this can be done by entering "mailserver" in the "Create new database" field and clicking on "Create".)

Add a less privileged MySQL user

For security reasons you should create another MySQL user account with fewer privileges. Postfix just needs to read from the database so it does not need write access.

Connect to your database:

$> mysql -p

When you see the mysql> prompt enter the following SQL statement (your input is shown in bold letters) to grant the appropriate privileges:

mysql> GRANT SELECT ON mailserver.*
TO 'mailuser'@'127.0.0.1'
IDENTIFIED BY 'mailuser2009';
Query OK, 0 rows affected (0.00 sec) mysql> flush privileges;
Query OK, 0 rows affected (0.00 sec) mysql> exit Bye

(It can also be done in phpMyAdmin by following these steps: Click on "Privileges". Select "Add a new User" As "User" enter "mailuser". As "Host" choose "local". Enter "mailuser2009" as the password twice. Click on "Go". Next look for "Database-specific privileges". Choose "mailserver" as the database. On the next page tick the "SELECT" checkbox and click on "Go".)

This will create a user called 'mailuser' that has only the privilege to select/read data from the database but not to alter it. If you want to add or alter data in the database either use the 'root' account or create another account for that purpose. The password 'mailuser2009' is just an example. Please replace it by a more decent password. If you lack creativity use "pwgen" or "apg" to create good passwords.

Create the database tables

Inside the newly created database you will have to create tables that store information about domains, forwardings and the users' mailboxes. It's easier here to create the database tables using SQL instead of clicking your way through phpMyAdmin.

Connect to MySQL again and choose the 'mailserver' database:

$> mysql -p mailserver

You will see the mysql> prompt again. First create a table for the list of virtual domains that you want to host:

mysql>
CREATE TABLE `virtual_domains` (
`id` int(11) NOT NULL auto_increment,
`name` varchar(50) NOT NULL,
PRIMARY KEY (`id`)
) ENGINE=InnoDB DEFAULT CHARSET=utf8;

The next table contains information on the actual user accounts. Every user has a username and password. It is used for accessing the mailbox by POP3 or IMAP, logging into the webmail service or to send mail ("relay") if they are not in your local network. As users tend to easily forget things the user's email address is also used as the login username. Let's create the users table:

mysql>
CREATE TABLE `virtual_users` (
`id` int(11) NOT NULL auto_increment,
`domain_id` int(11) NOT NULL,
`password` varchar(32) NOT NULL,
`email` varchar(100) NOT NULL,
PRIMARY KEY (`id`),
UNIQUE KEY `email` (`email`),
FOREIGN KEY (domain_id) REFERENCES virtual_domains(id) ON DELETE CASCADE
) ENGINE=InnoDB DEFAULT CHARSET=utf8;

The email field will contain the email address/username. And the password field will contain an MD5 hash of the user's password. The unique key on the email field makes sure that there are no two users in a domain accidentally.

And finally a table is needed for aliases (email forwardings from one account to another):

mysql>
CREATE TABLE IF NOT EXISTS `virtual_aliases` (
`id` int(11) NOT NULL auto_increment,
`domain_id` int(11) NOT NULL,
`source` varchar(100) NOT NULL,
`destination` varchar(100) NOT NULL,
PRIMARY KEY (`id`),
FOREIGN KEY (domain_id) REFERENCES virtual_domains(id) ON DELETE CASCADE
) ENGINE=InnoDB DEFAULT CHARSET=utf8;

Here the source column contains the email address of the user who wants to forward their mail. In case of catchall addresses the source looks like "@domain". The destination column contains the target email address. As described in the section on virtual domains there can be several rows for a source address designating multiple destinations who will get copies of an email.

You wonder about the foreign keys? They express that entries in the virtual_aliases and virtual_users tables are connected to entries in the virtual_domains table. This will keep the data in your database consistent because you cannot create virtual aliases or virtual users that are not connected to a virtual domain. The suffix 'ON DELETE CASCADE' means that if you delete a row from the referenced table that the deletion will also be done on the current table automatically. So you do not leave orphaned entries accidentally. Imagine that you do not host a certain domain any longer. You can remove the domain entry from the virtual_domains table and all dependent/referenced entries in the other tables will also be removed. (Note however that this would not remove the physical mail directories from the hard disk automatically.)

An example of the data in the tables:

virtual_domains
id name
1 example.com
2 foobar.org
virtual_users
id domain_id email password
1 1 john@example.com 14cbfb845af1f030e372b1cb9275e6dd
2 1 steve@example.com a57d8c77e922bf756ed80141fc77a658
3 2 kerstin@foobar.org 5d6423c4ccddcbbdf0fcfaf9234a72d0

Let us add a simple alias:

virtual_aliases  
id domain_id source destination
1 1 steve@example.com devnull@workaround.org
2 2 kerstin@foobar.org kerstin42@yahoo.com
3 2 kerstin@foobar.org kerstin@mycompany.com

This will make the mail for steve@example.com be redirected to devnull@workaround.org. And the mail for kerstin@foobar.org is redirected to both kerstin42@yahoo.com and kerstin@mycompany.com. Neither Steve nor Kerstin receive a copy of the email.

Postfix to Database Mappings

Now the database is ready to be filled with information on user accounts. The entry point for all email on your system is Postfix. So we need to tell Postfix how to get to the database-stored information. Let's start by telling it which virtual domains you have.

virtual_mailbox_domains

As described earlier a mapping in Postfix is just a table that contains a left-hand side (LHS) and a right-hand side (RHS). To make Postfix use MySQL to define a mapping we need a 'cf' file (configuration file). Start by creating a file called /etc/postfix/mysql-virtual-mailbox-domains.cf for the virtual_mailbox_domains mapping that contains:

	user = mailuser
password = mailuser2009
hosts = 127.0.0.1
dbname = mailserver
query = SELECT 1 FROM virtual_domains WHERE name='%s'

Imagine that Postfix received an email for somebody@example.com and wants to find out if example.com is a virtual mailbox domain. It will run the above SQL query and replace '%s' by 'example.com'. If it finds such an entry in the virtual_domains table it will return a '1'. Actually it does not matter what exactly is returned as long as there is a result.

And you need to make Postfix use this database mapping:

	$> postconf -e virtual_mailbox_domains=mysql:/etc/postfix/mysql-virtual-mailbox-domains.cf

(The postconf -e command conveniently adds configuration lines to your /etc/postfix/main.cf file. It also activates the new setting instantly so you do not have to reload the Postfix process.)

Postfix will now search your virtual_domains table to find out if "example.com" is a virtual mailbox domain. Let us see if this works. Create a new row in the virtual_domains table with one domain. Either do it in phpmyadmin through the "Insert" tab or connect to your database by running:

	$> mysql -p mailserver

and execute this query:

	mysql> INSERT INTO virtual_domains (id, name) VALUES (1, 'example.com');
exit

Back at your root shell you can now check if the 'example.com' domain is known as a virtual mailbox domain:

	$> postmap -q example.com mysql:/etc/postfix/mysql-virtual-mailbox-domains.cf

You should get '1' as a result. Your first mapping is working. Great. Get straight to the second one.

Note

If you get an error message telling you postmap: warning: connect to mysql server 127.0.0.1: Access denied... then you have a problem with the user account "mailuser" that you use to connect to the database. Check the MySQL privileges again.

If you get an error message reading postmap: warning: connect to mysql server 127.0.0.1: Can't connect to MySQL server on '127.0.0.1' then your MySQL server is either not running or at least not listening on 127.0.0.1. Check your MySQL server configuration (/etc/mysql/my.cnf).

virtual_mailbox_maps

You will now define the virtual_mailbox_maps which is mapping email addresses (left-hand side) to the location of the user's mailbox on your harddisk (right-hand side). If you saved incoming email to the hard disk using Postfix's built-in virtual delivery agent then it would be queried to find out the mailbox path. But in our setup the actual delivery is done by Dovecot's LDA (local delivery agent) so Postfix does not really care about the path. Postfix just needs to check if a certain email address is valid. Similar to the above you need an SQL query that searches for an email address and returns "1".

Go create an entry in the virtual_users table for a test user john@example.com:

	mysql>
INSERT INTO virtual_users (id, domain_id, email, password)
VALUES (1, 1, 'john@example.com', MD5('summersun'));

Next you will need to create a ".cf" file to tell Postfix about the SQL query for this table. In addition to the email address it is also important to get the user's password later on. As the path of the user's mailbox is fixed it is not important to get that information from the database. The directory structure will always be /var/vmail/$DOMAIN/$USER. So for John's example it would be /var/vmail/example.com/john.

Now things are a bit simpler and you can finally create a ".cf" file at /etc/postfix/mysql-virtual-mailbox-maps.cf that is as simple as:

	user = mailuser
password = mailuser2009
hosts = 127.0.0.1
dbname = mailserver
query = SELECT 1 FROM virtual_users WHERE email='%s'

Tell Postfix that this mapping file is supposed to be used for the virtual_mailbox_maps mapping:

	$> postconf -e virtual_mailbox_maps=mysql:/etc/postfix/mysql-virtual-mailbox-maps.cf

Test if Postfix is happy with this mapping by asking it where the mailbox directory of our john@example.com user would be:

	$> postmap -q john@example.com mysql:/etc/postfix/mysql-virtual-mailbox-maps.cf

You should get "1" back which means that john@example.com is an existing virtual mailbox user on your server. Later in the Dovecot configuration part you will also use the email and password fields but Postfix does not need them here. Great, there is just one mapping left to define:

virtual_alias_maps

The virtual_alias_maps mapping is used for forwarding emails from one email address to another. Examples of how entries in this mapping may look:




source (LHS) destination (RHS) Meaning
john@example.com devnull@workaround.org Forward John's mail to another address
john@example.com

john@example.com
devnull@workaround.org

Deliver one copy to the original account john@example.com but also send a copy to devnull@workaround.org (yes, Postfix understands this - it does not create a loop)
@example.com john@example.com Deliver all email for the domain to john@example.com unless there is a specific virtual user account defined in the virtual alias map. If kerstin@example.com does not exist as a specific virtual user then her mail would be sent to john. If kerstin@example.com is a valid user then she would get the mail. This is called a catchall account.

(See "man 5 virtual" for a more formal definition.)

Beware: Catch-all aliases catch spam. Lots of spam. They may look comfortable because they forward all email to one person without the need for creating aliases. But spammers often try to guess email addresses at a known domain. And with a catch-all alias you will receive spam for any of those guessed email addresses. Try to avoid them and rather define the existing email addresses. Even if it seems to be more work.

As you see it is possible to name multiple destinations. In the database this is achieved by using different rows. For example the second line above should be split into two rows:



source (LHS) destination (RHS)
john@example.com john@example.com
john@example.com devnull@workaround.org

Let us add this example to the database:

	mysql>
INSERT INTO virtual_aliases (id, domain_id, source, destination)
VALUES (1, 1, 'john@example.com', 'john@example.com'),
       (2, 1, 'john@example.com', 'devnull@workaround.org');

Create another ".cf" file at /etc/postfix/mysql-virtual-alias-maps.cf:

	user = mailuser
password = mailuser2009
hosts = 127.0.0.1
dbname = mailserver
query = SELECT destination FROM virtual_aliases WHERE source='%s'

Test if the mapping file works as expected:

	$> postmap -q john@example.com mysql:/etc/postfix/mysql-virtual-alias-maps.cf

You should see the two expected destinations:

	john@example.com,devnull@workaround.org

The black magic "email-to-email" mapping

Before you define the virtual_alias_maps setting there is a small quirk you need to take care of. There is a special kind of forwarding: the "catchall" alias. Catchalls catch all emails for a domain if there is no specific user account. A catchall alias looks like "@example.com" and forwards email for the whole domain to one account. We have created the 'john@example.com' user and would like to forward all other email on the domain to 'kerstin@gmail.com'. So we would add a catchall alias like:



source destination
@example.com kerstin@gmail.com

Now imagine what happens when Postfix receives an email for 'john@example.com'. Postfix will first check if there are any aliases in the virtual_alias_maps table. (It does not look at the virtual_mailbox_maps table at the moment.) It finds the catchall entry as above and since there is no more specific alias the catchall account matches and the email is redirected to 'kerstin@gmail.com'. This is probably not what you wanted. So you need to make the table rather look like this:



email destination
@example.com kerstin@gmail.com
john@example.com john@example.com

More specific aliases have precedence over general catchall aliases. Postfix will lookup all these mappings for each of:

  • john@example.com (most specific)
  • john (only works if "example.com" is the $myorigin domain)
  • @example.com (catchall - least specific)

Postfix will find an entry for 'john@example.com' first and sees that email should be "forwarded" to 'john@example.com' - the same email address. This trickery may sound weird but it is needed if you plan to use catchall accounts. So the virtual_alias_maps mapping must obey both the "view_aliases" view and this "john-to-himself" mapping. This is outlined in the virtual(5) man page in the TABLE SEARCH ORDER section.

Create a ".cf" file /etc/postfix/mysql-email2email.cf for the latter mapping:

	user = mailuser
password = mailuser2009
hosts = 127.0.0.1
dbname = mailserver
query = SELECT email FROM virtual_users WHERE email='%s'

Check that you get John's email address back when you ask Postfix if there are any aliases for him:

	$> postmap -q john@example.com mysql:/etc/postfix/mysql-email2email.cf

The result should be the same address:

	john@example.com

Now you need to tell Postfix that these two mappings should be searched by adding this line to your main.cf:

	$> postconf -e virtual_alias_maps=mysql:/etc/postfix/mysql-virtual-alias-maps.cf,mysql:/etc/postfix/mysql-email2email.cf

The order of the two mappings is not important here.

You did it! All mappings are set up and the database is generally ready to be filled with domains and users. Make sure that only 'root' and the 'postfix' user can read the ".cf" files - after all your database password is stored there:

	$> chgrp postfix /etc/postfix/mysql-*.cf
$> chmod u=rw,g=r,o= /etc/postfix/mysql-*.cf

Make Postfix send mails to Dovecot

In the previous chapter we made sure that Postfix knows which emails it's allowed to receive. Now what to do with the email? It has to be stored to disk. Usually that's done by Postfix itself which comes with a very basic mail delivery agent (MDA) called "virtual" that you can use to save emails to virtual mailboxes on your hard disk. But as we will use Dovecot (for IMAP and POP3 access) anyway we can use its more featureful "local delivery agent" (also known as "Dovecot LDA"). To make Postfix use that agent you will have to add a service to your /etc/postfix/master.cf:

dovecot   unix  -       n       n       -       -       pipe
    flags=DRhu user=vmail:vmail argv=/usr/lib/dovecot/deliver -d ${recipient}

(Note: the second line has to be indented by spaces!)

Restart Postfix:

$> postfix reload

Also make Postfix use that service for virtual delivery by adding these lines to your /etc/postfix/main.cf:

$> postconf -e virtual_transport=dovecot
$> postconf -e dovecot_destination_recipient_limit=1

So far this will make Postfix pass on incoming emails to virtual users to the /usr/lib/dovecot/deliver program. Now it is time to configure Dovecot in the next chapter.

Set up Dovecot

Let us now configure Dovecot which will do several things for us:

Before we get to the actual configuration for security reasons it is suggested you create a new system user that will own all virtual mailboxes. The following shell commands will create a system group "vmail" with GID (group ID) 5000 and a system user "vmail" with UID (user ID) 5000. (Make sure that UID and GID are not yet used or choose another - the number can be anything between 1000 and 65000 that is not yet used):

$> groupadd -g 5000 vmail
$> useradd -g vmail -u 5000 vmail -d /var/vmail -m

The configuration files for Dovecot are found under /etc/dovecot. Start with the main file...

/etc/dovecot/dovecot.conf

See the line protocols and define the protocols you want to offer. By default this line reads:

	protocols = imap imaps pop3 pop3s

so that Dovecot starts the IMAP and POP3 services and also its equivalents that work over an encrypted SSL (secure socket layer) connection.

Although this is a less secure setting you will probably still need it:

	disable_plaintext_auth = no

This will allow plaintext passwords over an unsecured (non-SSL) connection. By default it is set to 'yes' for security reasons. Setting it to 'no' will mean less security but may help users of a "certain" Microsoft email software that is broken in many ways.

A more important setting is:

	mail_location = maildir:/var/vmail/%d/%n/Maildir

which will tell that the users' mailboxes are always found at /var/vmail/DOMAIN/USER/Maildir and that it should be in maildir format.

There is already a section "namespace private" in your dovecot.conf which is commented out by "#" characters. The "private" namespace is the personal mailbox this is only available for a certain user. You can leave this section disabled and get a maildir directory schema like:

/var/vmail/christoph.haas/email/Maildir/.spam

If you followed previous ISPmail tutorials then your directories may be different. If you rather have:

/var/vmail/christoph.haas/email/Maildir/.INBOX.spam

then you need to declare that in the "namespace private" section as follows. Enable this section and make sure these variables are set:

	namespace private {
    separator = .
    inbox = yes
}

Next look for a section called "auth default". First define the allowed authentication mechanisms:

	mechanisms = plain login

As you browse through the section you see many backends that Dovecot can access to get the email users' data. Inside this section you need to set:

	passdb sql {
    args = /etc/dovecot/dovecot-sql.conf
}

which tells Dovecot that the passwords are stored in an SQL database and:

	userdb static {
    args = uid=5000 gid=5000 home=/var/vmail/%d/%n allow_all_users=yes
}

to tell Dovecot where the mailboxes are located. This is similar to the mail_location setting. The user gets authenticated in the "passdb sql" section. So the "userdb static" section defined where the mail folders are located. Using "userdb sql" is not needed as all mailboxes follow a fixed directory schema.

You will want to comment out the section called "passdb pam that deals with system users. Otherwise Dovecot will also look for system users when someone fetches emails which leads to warnings in your log file.

Now look for another section called socket listen. Here you define socket files that are used to interact with Dovecot's authentication mechanism. Make the section read:

	socket listen {
    master {
        path = /var/run/dovecot/auth-master
        mode = 0600
        user = vmail
    }

    client {
        path = /var/spool/postfix/private/auth
        mode = 0660
        user = postfix
        group = postfix
    }
}

The "master" section is needed to give Dovecot's delivery agent (the program that saves a new mail to the user's mailbox) access to the userdb information. The "client" section creates a socket inside the "chroot" directory of Postfix. chroot means that parts of Postfix are jailed into /var/spool/postfix and can only access files in that directory or its subdirectories. It is a good security measure so that even if Postfix had bugs and were hacked then the attacker would not be able to access /etc/passwd for example because it's outside of /var/spool/postfix.

And finally the "protocol lda" section needs to be customized. The LDA (local delivery agent) is more capable than Postfix's built-in virtual delivery agent. It allows for quotas and Sieve (ships with the dovecot-common package) filtering. Let the section be:

	protocol lda {
    log_path = /var/vmail/dovecot-deliver.log
    auth_socket_path = /var/run/dovecot/auth-master
    postmaster_address = postmaster@example.com
    mail_plugins = cmusieve
}

Please change the above postmaster email address to a valid address where the administrator can be reached.

The log_path setting is optional but may help you figure out why a certain server-side filter is not doing what you expect. As the dovecot-deliver.log may grow pretty fast you should create a logrotate configuration file /etc/logrotate.d/dovecot-deliver for it:

	/var/vmail/dovecot-deliver.log {
        weekly
        rotate 14
        compress
}

Edit the /etc/dovecot/dovecot-sql.conf and change these settings:

	driver = mysql
connect = host=127.0.0.1 dbname=mailserver user=mailuser password=mailuser2009
default_pass_scheme = PLAIN-MD5
password_query = SELECT email as user, password FROM virtual_users WHERE email='%u';

Restart Dovecot:

	$> /etc/init.d/dovecot restart

Now look at your /var/log/mail.log logfile. You should see:

	dovecot: Dovecot v1.0.rc15 starting up
dovecot: auth-worker(default): mysql: Connected to 127.0.0.1 (mymailserver)

Before you send a first test email you will need to fix file system permissions for the /etc/dovecot/dovecot.conf file so that the vmail user can access the Dovecot configuration. The reason is that Postfix starts the delivery agent with vmail permissions:

	$> chgrp vmail /etc/dovecot/dovecot.conf
$> chmod g+r /etc/dovecot/dovecot.conf

Send a test mail using Telnet

Emulating an SMTP session with Telnet

Let us try to send an email to the user now. As the "example.com" domain does not really existing your DNS settings will likely not point to the right server. So we are simulating an SMTP session with the telnet command. Install the telnet package if you haven't already:

$> aptitude install telnet

Then establish a TCP connection to the SMTP port:

$> telnet localhost smtp

The server should reply:

Trying 127.0.0.1...
Connected to localhost.
Escape character is '^]'.
220 mailtest ESMTP Postfix (Debian/GNU)

Great. Postfix is listening and wants us to speak SMTP. First we need to be friendly:

ehlo example.com

Postfix appreciates our friendliness and tells us which features it provides:

250-my-new-mailserver
250-PIPELINING
250-SIZE 10240000
250-VRFY
250-ETRN
250-ENHANCEDSTATUSCODES
250-8BITMIME
250 DSN

Hey, Postfix, we have a mail from steve@example.com:

mail from:<steve@example.com>

Looks like Postfix is happy with that because return codes that start with a '2' are good news:

250 2.1.0 Ok

Tell Postfix who the recipient of the mail is:

rcpt to:<john@example.com>

Postfix accepts that:

250 2.1.5 Ok

Then we are ready to send the actual mail:

data

Postfix agrees and tells us we can send the actual mail now and end our input with a dot on an empty line:

354 End data with <CR><LF>.<CR><LF>

Okay, type in the mail:

Hi John,

just wanted to drop you a note.
.

Postfix tells us it has received the mail and queued under a queue ID:

250 2.0.0 Ok: queued as A9D64379C4

Thanks, Postfix, we are done:

quit

Checking the logs

Take a look at the /var/log/mail.log file now. You should see something similar to:

postfix/smtpd[...]: connect from localhost[127.0.0.1]
postfix/smtpd[...]: 5FF712A6: client=localhost[127.0.0.1]
postfix/cleanup[...]: 5FF712A6: message-id=<...>
postfix/qmgr[...]: 5FF712A6: from=<steve@example.com>, size=364, nrcpt=1 (queue active)
postfix/pipe[...]: 5FF712A6: to=<john@example.com>, relay=dovecot, ..., status=sent (delivered via dovecot service)
postfix/qmgr[...]: 5FF712A6: removed
postfix/smtpd[...]: disconnect from localhost[127.0.0.1]

The delivery has worked. Postfix has correctly determined that the destination domain is a virtual domain and forwarded the email to the 'dovecot' service.

Checking the user's mailbox

The email should now be somewhere under /var/vmail/example.com/john. Let us take a look:

$> cd /var/vmail/example.com/john/Maildir
$> find
.
./cur
./new
./new/1179521979.V801I2bbf7M15352.mailtest
./tmp

There sits the email. Try to read the mail with the "mutt" program:

$> mutt -f .

The new email is shown:

q:Quit  d:Del  u:Undel  s:Save  m:Mail  r:Reply  g:Group  ?:Help
   1 N   May 18 steve@example.c (0.1K)

Press ENTER to read the email:

From: steve@example.com
To: undisclosed-recipients: ;

Hi John,

just wanted to drop you a note.

So the email arrived at John's account. Perfect. Choose 'q' twice to exit mutt again.

Test fetching emails with IMAP and POP3

John will surely prefer to read his mail in a comfortable mail program. So he needs a way to get access to his mailbox. Two protocols come into play here:

Testing POP3

Let us establish a POP3 connection and retrieve John's email:

$> telnet localhost pop3

The server replies:

Trying 127.0.0.1...
Connected to localhost.
Escape character is '^]'.
+OK Dovecot ready.

Login as John:

user john@example.com

The server should accept that:

+OK

Send the password:

pass summersun

The server should recognize the correct password:

+OK Logged in.

Get a list of John's emails:

list

Dovecot will tell you that one email is in the mailbox:

+OK 1 messages:
1 474
.

Fetch that email number 1:

retr 1

Dovecot sends you the email:

+OK 474 octets
Return-Path: <steve@example.com>
X-Original-To: john@example.com
Delivered-To: john@example.com
Received: from example.com (localhost [127.0.0.1])
    by ... (Postfix) with ESMTP id 692DF379C7
    for <john@example.com>; Fri, 18 May 2007 22:59:31 +0200 (CEST)
Message-Id: <...>
Date: Fri, 18 May 2007 22:59:31 +0200 (CEST)
From: steve@example.com
To: undisclosed-recipients:;

Hi John,

just wanted to drop you a note.
.

Close the connection to the POP3 server:

quit

The server will disconnect you:

+OK Logging out.
Connection closed by foreign host.

Of course users won't use TELNET to read their mail. He will use a more comfortable email client. This is just to show you how POP basically works and to make sure Dovecot is behaving correctly.

Test IMAP

Instead of going through the following procedure (IMAP is rather complicated) you may as well just use mutt to create an IMAP connection:

$> mutt -f imap://john@example.com@localhost

Alternatively you can open up a raw IMAP connection to the server and enter the IMAP commands yourself:

$> telnet localhost imap2

You should get a connection:

Trying 127.0.0.1...
Connected to localhost.
Escape character is '^]'.
* OK Dovecot ready.

IMAP commands always start with a number and reply to that command with the same number. So the following commands must be entered with the number at the beginning of each line. Login with username and password:

1 login john@example.com summersun

Dovecot logs you in:

1 OK Logged in.

Ask Dovecot for a list of John's mail folders:

2 list "" "*"

Here comes the list:

* LIST (\HasNoChildren) "." "INBOX"
2 OK List completed.

Select your inbox:

3 select "INBOX"

Dovecot gives you all kinds of information about that folder:

* FLAGS (\Answered \Flagged \Deleted \Seen \Draft)
* OK [PERMANENTFLAGS (\Answered \Flagged \Deleted \Seen \Draft \*)] Flags permitted.
* 1 EXISTS
* 0 RECENT
* OK [UIDVALIDITY 1180039205] UIDs valid
* OK [UIDNEXT 3] Predicted next UID
3 OK [READ-WRITE] Select completed.

You see that one email exists. Fetch it:

4 fetch 1 all

IMAP will just give you basic information on the email:

* 1 FETCH (FLAGS (\Seen) INTERNALDATE .........
4 OK Fetch completed.

To read the actual mail body you need to fetch it explicitly:

5 fetch 1 body[]

Here it comes:

* 1 FETCH (BODY[] {474}
Return-Path: <steve@example.com>
X-Original-To: john@example.com
Delivered-To: john@example.com
Received: from example.com (localhost [127.0.0.1])
        by ... (Postfix) with ESMTP id 692DF379C7
        for <john@example.com>; Fri, 18 May 2007 22:59:31 +0200 (CEST)
Message-Id: <...>
Date: Fri, 18 May 2007 22:59:31 +0200 (CEST)
From: steve@example.com
To: undisclosed-recipients:;

Hi John,

just wanted to drop you a note.
)
5 OK Fetch completed.

Disconnect from the server:

6 logout

Dovecot logs you out:

* BYE Logging out
6 OK Logout completed.
Connection closed by foreign host.

POP3 and IMAP appear to work. You could now use any email program like Kmail, Evolution or Thunderbird/Icedove and set up a POP3 or IMAP email account. The quickest way to check encrypted connections is using mutt again:

$> mutt -f imaps://john@example.com@localhost

If you use other mail programs note that the username will be the email address 'john@example.com' and the password is 'summersun'. You can try these kinds of connections:

  • POP3
  • IMAP
  • POP3 with TLS/SSL enabled
  • IMAP with TLS/SSL enabled

When using TLS/SSL you will probably get a warning that the certificate of the server cannot be trusted. Dovecot creates a self-signed certificate. If you are not happy with the certificate then create your own:

$> openssl req -new -x509 -days 3650 -nodes -out /etc/ssl/certs/dovecot.pem \
-keyout /etc/ssl/private/dovecot.pem

The certificate and key will be created while you get asked a few questions:

Generating a 1024 bit RSA private key
.........++++++
............................++++++
writing new private key to '/etc/ssl/certs/dovecot.pem'
-----
You are about to be asked to enter information that will be incorporated
into your certificate request.
What you are about to enter is what is called a Distinguished Name or a DN.
There are quite a few fields but you can leave some blank
For some fields there will be a default value,
If you enter '.', the field will be left blank.
-----
Country Name (2 letter code) [AU]:DE
State or Province Name (full name) [Some-State]:Hamburg
Locality Name (eg, city) []:Hamburg
Organization Name (eg, company) [Internet Widgits Pty Ltd]:workaround email service
Organizational Unit Name (eg, section) []:
Common Name (eg, YOUR name) []:mailtest.workaround.org
Email Address []:postmaster@workaround.org

Of course you should fill in your own information here. The most important setting is the Common Name which must contain the fully-qualified name of your mail server. Oh, and this certificate will be valid for 10 years (3650 days) - adjust that period as you want.

Do not forget to set the permissions on the private key so that no unauthorized people can read it:

$> chmod o= /etc/ssl/private/dovecot.pem

And you will have to restart Dovecot to make it read your new certificate:

$> /etc/init.d/dovecot restart

 

Authenticated SMTP

It is important that you understand what "relaying" is and why "open relays" are a big problem on the internet. Usually Postfix accepts an email only if one of these criteria match:

For security reasons the following must not be allowed:

Otherwise a spammer could abuse your system to send millions of spam emails.This would waste your bandwidth, annoy many people and get your server blacklisted quickly. Such a relay that is not checking which emails to accept is called an "open relay". Fortunately Postfix makes it hard to become an open relay. Setting the smtpd_recipient_restrictions right (as described below) is very important.

Generally you can define the networks that are allowed to relay through your mail server by setting the mynetworks parameter in your main.cf. Usually you set it to your local network so local users do not need to authenticate:

$> postconf -e mynetworks=192.168.50.0/24

But one case of relaying from outside of your network is important in real-life. Imagine you have a user who is currently not connected to your local network but wants to send out an email to the internet through your mail server. According to the above rules the user would not be able to do that because neither are they on your network nor are they sending an email to one of your domains. You need to find a way to make your remote user be trusted by your mail server. The users will need to send their username and password so you know relaying should be permitted. This is what authenticated SMTP is about. And all decent email clients have that feature built in.

Authenticated SMTP with Postfix has always been a pain. It was done through the SASL (Simple Authentication and Security Layer) library that was part of the Cyrus mail server. It was nearly impossible to debug and threw error messages that were gibberish and misleading. Fortunately starting with Postfix 2.3 we can make Postfix ask the Dovecot server to verify the username and password. And since you already configured Dovecot this is really easy now. Postfix just needs some extra configuration:

$> postconf -e smtpd_sasl_type=dovecot
$> postconf -e smtpd_sasl_path=private/auth
$> postconf -e smtpd_sasl_auth_enable=yes
$> postconf -e smtpd_recipient_restrictions=permit_mynetworks,permit_sasl_authenticated,reject_unauth_destination

smtpd_sasl_auth_enable enables SMTP authentication altogether. And the smtpd_recipient_restrictions define rules that are checked after the remote user sends the RCPT TO: line during the SMTP dialog. In this case relaying is allowed if:

There are further restrictions (smtpd_client_restrictions, smtpd_helo_restrictions, smtpd_sender_restrictions) that get checked during the different states of the SMTP dialog (IP connection, HELO/EHLO command, MAIL FROM command) but for now you should put all restrictions into the smtpd_recipient_restrictions.

Try to authenticate during an SMTP session:

$> telnet localhost smtp

The server will let you in:

Trying 127.0.0.1...
Connected to localhost.
Escape character is '^]'.
220 mailtest ESMTP Postfix (Debian/GNU)

Say hello:

ehlo example.com

Postfix will present a list of features that are available during the SMTP dialog:

250-mailtest
250-PIPELINING
250-SIZE 10240000
250-VRFY
250-ETRN
250-ENHANCEDSTATUSCODES
250-8BITMIME
250 DSN

Send the authentication string with a Base64-encoded password:

auth plain am9obkBleGFtcGxlLmNvbQBqb2huQGV4YW1wbGUuY29tAHN1bW1lcnN1bg==

The server should accept that authentication:

235 2.0.0 Authentication successful

Disconnect from Postfix:

quit

Goodbye:

221 2.0.0 Bye

Authentication works. Well done.

Note

If you have set John's password to something other than 'summersun' you need Base64-encode it yourself. Use:

$> perl -MMIME::Base64 -e \
    'print encode_base64("john\@example.com\0john\@example.com\0password")';

Now you can test sending email with authentication enabled. To make even your local network untrusted temporarily you can set:

$> postconf -e mynetworks=
$> postfix reload

Restart Postfix (/etc/init.d/postfix restart). Fire up your mail program and watch your mail.log (tail -f /var/log/mail.log) while you send an email to a domain on the internet. I recommend you send a test email to devnull@workaround.org which is an email address that will just discard your email. If everything worked well your logfile will show:

postfix/smtpd[4032]: 1234567890: client=..., sasl_method=PLAIN, sasl_username=john@example.com
postfix/cleanup[4040]: 2EAE8379CB: message-id=<...>
postfix/qmgr[3963]: 1234567890: from=<john@example.com>, size=830, nrcpt=1 (queue active)
postfix/smtpd[4032]: disconnect from ...
postfix/smtp[4041]: 1234567890: to=<devnull@workaround.org>,
    relay=torf.workaround.org[212.12.58.129]:25, delay=6,
    delays=0.09/0.08/5.6/0.23, dsn=2.0.0, status=sent
    (250 OK id=1HsPC3-0008UJ-O5)
postfix/qmgr[3963]: 2EAE8379CB: removed

Otherwise in case of an error your logfile might look like:

postfix/smtpd[4032]: connect from ...[10.20.30.40]
postfix/smtpd[4032]: warning: ...[10.20.30.40]: SASL PLAIN authentication failed:
postfix/smtpd[4032]: lost connection after AUTH from ...[10.20.30.40]
postfix/smtpd[4032]: disconnect from ...[10.20.30.40]

Don't forget to set mynetworks back to your network definition:

$> postconf -e mynetworks=192.168.50.0/24
$> postfix reload

Your email program may have warned you that the mail server uses an untrusted SSL certificate. When you installed Postfix Debian created a self-signed SSL certificate for you automatically. The default certificate is sufficient for testing but just like you did for Dovecot you can create a custom SSL certificate. The default certificate is stored at /etc/ssl/certs/ssl-cert-snakeoil.pem and the default private key is stored at /etc/ssl/private/ssl-cert-snakeoil.key. This will create a new certificate/key pair:

$> openssl req -new -x509 -days 3650 -nodes -out /etc/ssl/certs/postfix.pem \
-keyout /etc/ssl/private/postfix.pem

Same procedure as above when you created a certificate for Dovecot. Just remember to set the "Common Name" to the fully-qualified hostname. You could as well use the same certificate you created for Dovecot if the server name is the same. In that case just use the files /etc/ssl/certs/dovecot.pem and /etc/ssl/private/dovecot.pem below.

Do not forget to set the permissions on the private key so that no unauthorized people can read it:

$> chmod o= /etc/ssl/private/postfix.pem

You will have to tell Postfix where to find your certificate and private key:

$> postconf -e smtpd_tls_cert_file=/etc/ssl/certs/postfix.pem
$> postconf -e smtpd_tls_key_file=/etc/ssl/private/postfix.pem

When you relay through Postfix again you should not get that certificate warning any longer.

By default Postfix will allow that the login data for SMTP authentication is sent in plain text. You better only allow encrypted transmission of the credentials by setting these parameters:

$> postconf -e smtpd_use_tls=yes
$> postconf -e smtpd_tls_auth_only=no

What these parameters do is offer (but not require) encryption for SASL authentication to be used so that if you're using the PLAIN or LOGIN SASL methods your passwords aren't transmitted in the clear. When the client initially connects to the server, the AUTH command isn't offered by the server. If the client issues the STARTTLS command and successfully negotiates the TLS connection, the client sends the EHLO command a second time and this time the server offers the AUTH command. This won't require any kind of encryption from clients who don't need to authenticate (i.e. servers that connect to send mail to the domains our server is a final destination for, as opposed to our end users attempting to relay mail through it to external domains).

If you truly do want to forbid unencrypted SMTP connections (I do this on ports 587 and 465), you'd want to use either "smtpd_tls_security_level = encrypt" (for STARTTLS, generally on port 587) or "smtpd_tls_wrappermode = yes" (for SSL encryption from the initial connection on, generally on port 465). You could also set "smtpd_tls_security_level = may" so that TLS encryption is offered but it's not mandatory (so-called opportunistic TLS).

Once you are done you should test if your mail server is safely configured to prevent illegal relaying attempts. On a root shell enter:

$> telnet relay-test.mail-abuse.org

This contacts a very convenient service on the internet which tries to relay emails through your mail server. Give it a moment while it checks your server. If at the end you see something like "System appeared to reject relay attempts" then you are fine.

AMaViS: Filtering spam and viruses

Two annoying issues when using email are spam and viruses. Fortunately you can fight both using AMaViS (A Mail Virus Scanner). AMaViS is an interface between Postfix, SpamAssassin (famous for its Bayesian spam filtering capabilities) and optionally a virus scanner. AMaViS contains the spam filtering part but has no virus scanner built in. You have installed ClamAV for that purpose. It is a high quality and free virus scanner that gets updated frequently.

The AMaViS configuration is spread across several files in the /etc/amavis/conf.d directory. Fortunately the virus scanner "ClamAV" is already configured by default, though it will need enabling in the file /etc/amavis/conf.d/15-content_filter_mode by removing the '#' from the @bypass_... lines so that spam and virus filtering gets enabled. (It's confusing that enabling the bypass option actually enables spam and virus scanning.)

I suggest you also take a closer look at /etc/amavis/conf.d/20-debian_defaults. Don't change this file though. If you want to change settings then put them into the /etc/amavis/conf.d/50-user file.

So a sensible "50-user" file would be:

$sa_spam_subject_tag = undef;
$spam_quarantine_to  = undef;
$sa_tag_level_deflt  = undef;
$final_spam_destiny  = D_PASS;
1;  # ensure a defined return

Restart AMaViS if you have made changes to the config files:

$> /etc/init.d/amavis restart

Make sure that AMaViS is listening on TCP port 10024:

$> netstat -nap | grep 10024

You should get this output:

tcp  0   0 127.0.0.1:10024     0.0.0.0:*    LISTEN   12345/amavisd

If you get such a line then AMaViS is running and waiting for incoming SMTP sessions. Otherwise check your /var/log/mail.log file - perhaps you have made a mistake in the configuration files.

A few words on how AMaViS will be plugged into Postfix. If a person sends you an email from the internet it will be received by Postfix (the smtpd process) on TCP port 25 (SMTP). If Postfix accepts the email it will be forwarded to AMaViS on TCP port 10024 (SMTP). And if AMaViS is happy with the email's contents it will be sent back to Postfix on TCP port 10025 (SMTP). Postfix finally delivers the email to the actual recipient. See the chapter "The Big Picture" for an illustration.

Making Postfix forward emails to AMaViS is done by setting the content_filter setting. Also set the "receive_override_options" setting that will be explained later:

$> postconf -e content_filter=smtp-amavis:[127.0.0.1]:10024
$> postconf -e receive_override_options=no_address_mappings

We need to define the smtp-amavis service first in the /etc/postfix/master.cf. And we also need Postfix to listen on TCP port 10025 for emails that get sent back from AMaViS. So add these sections to your /etc/postfix/master.cf:

smtp-amavis unix -      -       n     -       2  smtp
    -o smtp_data_done_timeout=1200
    -o smtp_send_xforward_command=yes
    -o disable_dns_lookups=yes
    -o max_use=20

127.0.0.1:10025 inet n  -       -     -       -  smtpd
    -o content_filter=
    -o local_recipient_maps=
    -o relay_recipient_maps=
    -o smtpd_restriction_classes=
    -o smtpd_delay_reject=no
    -o smtpd_client_restrictions=permit_mynetworks,reject
    -o smtpd_helo_restrictions=
    -o smtpd_sender_restrictions=
    -o smtpd_recipient_restrictions=permit_mynetworks,reject
    -o smtpd_data_restrictions=reject_unauth_pipelining
    -o smtpd_end_of_data_restrictions=
    -o mynetworks=127.0.0.0/8
    -o smtpd_error_sleep_time=0
    -o smtpd_soft_error_limit=1001
    -o smtpd_hard_error_limit=1000
    -o smtpd_client_connection_count_limit=0
    -o smtpd_client_connection_rate_limit=0
    -o receive_override_options=no_header_body_checks,no_unknown_recipient_checks
    -o local_header_rewrite_clients=

Restart Postfix first:

$> postfix reload

Now two settings here deserve an explanation. First the receive_override_options are set to no_address_mappings. This disables all address mappings. Your virtual aliases for example are not considered at first. Then the email is sent to smtp-amavis and in the end returned to the 127.0.0.1:10025 service that sets a lot of options. One of those options is the receive_override_options again. But this time the no_address_mappings setting is left out. So at this stage Postfix finally checks your virtual aliases. Sounds complicated? Well, it has to be done like this or otherwise your aliases would be evaluated twice which leads to mails sent twice. The other options are used to disable checks that Postfix has already done during the first stage.

Note: receive_override_options=no_address_mappings makes Postfix not consider aliases any more. So if you wish to disable AMaViS as a content filter then you must disable this parameter, too!

Another tiny caveat is that the user "clamav" must be a member of the system group "amavis" so that the two services are allowed to talk to each other:

$> adduser clamav amavis
$> /etc/init.d/clamav-daemon restart

And another issue to take care of: AMaViS tries to find out whether a certain email is incoming (sent from the internet to your domains) or outgoing (sent from your system to the internet) by looking at the @acl_local_domains setting. You need to tell AMaVis where to check if a certain domain is one of your destination domains. The reason is that you usually don't want to scan your outgoing emails. Imagine that an email is accidentally deemed to be spam and your customer gets warned of your emails. You don't need that.

So edit the /etc/amavis/conf.d/50-user file and before the "1;" enter these lines:

@lookup_sql_dsn = (
    ['DBI:mysql:database=mailserver;host=127.0.0.1;port=3306',
     'mailuser',
     'mailuser2009']);

$sql_select_policy = 'SELECT name FROM virtual_domains WHERE CONCAT("@",name) IN (%k)';

The @lookup_sql_dsn defines how AMaVis can access your database. And the $sql_select_policy sets the SQL query that is run when AMaVis wants to determine if the destination domain of the currently scanned email is one of your virtual domains. The %k is a list of strings that AMaVis expects to find. The actual query will look like this:

SELECT name
FROM virtual_domains
WHERE CONCAT("@",name)
IN (
    'john@example.com',
    'john',
    '@example.com',
    '@.example.com',
    '@.com',
    '@.')

This may look a bit weird. But in the end the string '@example.com' is searched for. Restart AMaVis:

$> /etc/init.d/amavis restart

Now try sending an email to john@example.com. If you examine the email headers of that mail you should find lines that got added by AMaVis:

X-Virus-Scanned: Debian amavisd-new at mymailserver
X-Spam-Score: 0
X-Spam-Level:
X-Spam-Status: No, score=0 tagged_above=-9999 required=6.31 tests=[none]

Your users will be able to filter out spam depending on this information. The X-Spam-Status line will be set to "Yes" if the spam score is above the required score ($sa_tag2_level_deflt). The X-Spam-Level line contains a number of "*" that desc the spam score. If your email program looked for a X-Spam-Level: ***** header then it would catch spam with at least a score of 5. A real-life example of caught spam:

X-Spam-Status: Yes, hits=16.0 tagged_above=-9999.0 required=6.31
    tests=BAYES_99, FORGED_MUA_OUTLOOK, MSGID_FROM_MTA_ID,
    RCVD_IN_BL_SPAMCOP_NET, UNDISC_RECIPS, URIBL_OB_SURBL, WORK_AT_HOME
    X-Spam-Level: ***************
    X-Spam-Flag: YES

All incoming emails should now be tested for viruses and spam. Try that. SpamAssassin comes with a spam sample containing the GTUBE (Generic Test for Unsolicited Bulk Email) that contains a signature that is supposed to trigger spam filters - just like EICAR.COM for virus scanners. Send John that spam email:

$> sendmail john@example.com < /usr/share/doc/spamassassin/examples/sample-spam.txt

Among other information your /var/log/mail.log will now contain a line being written by AMaViS:

amavis[13001]: (13001-02) Passed SPAM, <...> -> <john@example.com>, ...

So the email was detected as spam and passed through to John. Very well. Finally, fix the permissions of the AMaViS configuration files so that no unauthorized user can read the database password:

$> chmod o= /etc/amavis/conf.d/50-user

 

Manage your email accounts

ISPwebAdmin (web interface)

I have created a simple web tool (although the installation is not trivial - sorry for that) for administering domains, users and aliases called ISPwebAdmin. Further interesting projects like this are:

Screenshots

Just a few impressions how the ISPwebAdmin's interface looks like:

Download

Scroll down to the attachments section for the download link.

Installation instructions

User authentication

In addition to the SQL tables as described in the ISPmail tutorial you will need to create an additional table to store usernames and passwords for the administrators that are allowed to manage email accounts. So please run this SQL query to create the table:

		CREATE TABLE IF NOT EXISTS `admins` (
  `id` int(11) NOT NULL auto_increment,
  `username` varchar(40) NOT NULL,
  `pwhash` varchar(32) NOT NULL,
  PRIMARY KEY  (`id`)
) ENGINE=MyISAM  DEFAULT CHARSET=utf8;

Create an admin user here who can login to the web interface. The pwhash field stores an MD5 hash of the password to avoid storing it as plain text.

Example SQL:

		INSERT INTO mailserver.admins (username,pwhash) VALUES ('john', MD5('doe'));

Note that you will need to create a MySQL user who has write access to the 'mailserver' database.

Installation on the destination (mail) server

As user root:

  • aptitude install python-virtualenv
  • apt-get build-dep python-mysqldb

Become the user vmail:

  • su -s /bin/bash vmail

Prepare the application:

  • cd /var/vmail
  • mkdir ispwebadmin
  • cd ispwebadmin
  • virtualenv .
  • . bin/activate
  • easy_install ispwebadmin*.tar.gz
  • paster make-config ispwebadmin run.ini (this creates a 'run.ini' file that contains setting that are specific to your installation - see below for settings you need to customize)
  • vi run.ini
  • paster serve run.ini

As a quick-and-dirty solution this can be used to run the web application. But to keep the service running in the background and start up automatically when the system boots the daemontools mechanism is recommended:

Starting automatically using runit (Debian Lenny)

As user root:

  • aptitude install runit
  • mkdir /etc/sv/ispwebadmin
  • cd /etc/sv/ispwebadmin

Create a file "/etc/sv/ispwebadmin/run" there containing:

		#!/bin/sh
exec 2>&1
echo 'ISPwebAdmin starting.'
cd /var/vmail/ispwebadmin
sudo -u vmail PYTHON_EGG_CACHE=/var/vmail/ispwebadmin/.python-eggs bin/paster serve run.ini

Make this script executable:

  • chmod u+x /etc/sv/ispwebadmin/run

Create a symlink to tell runit you want to use this service:

  • ln -s /etc/sv/ispwebadmin /etc/service/

The service should start up automatically. To see if it works:

		$> sv status /etc/service/ispwebadmin
run: ispwebadmin: (pid 3335) 0s; run: log: (pid 3326) 4s

To stop it:

		$> sv stop /etc/service/ispwebadmin
ok: down: ispwebadmin: 1s, normally up

To start it:

		$> sv start /etc/service/ispwebadmin
ok: run: ispwebadmin: (pid 3956) 1s

If you want to get a logfile for this web service you can create a "/etc/sv/ispwebadmin/log/run" file containing:

		#!/bin/sh -e
LOG=/var/log/runit-ispwebadmin
test -d "$LOG" || mkdir -p -m2750 "$LOG"
exec svlogd -tt "$LOG"

Then you should get output in the "/var/log/runit-ispwebadmin" file.

INI file settings

The ini file that you created with "paster make-config something.ini" contains settings that are specific to your mail server installation. Before you run the web interface please check these settings:

[server:main] -> host

The IP address the web server listens on. "0.0.0.0" can be used for all interfaces

[server:main] -> port

The TCP port the web server listens on.

[app:main] -> ispwebadmin.maildir_location

The path to your directory where the users' virtual mailboxes are stored. If you followed the Lenny ISPmail tutorial then this will be "/var/vmail".

[app:main] -> ispwebadmin.postmaster

The email address that is used as an abuse@ and postmaster@ address for newly created domains. You are required to have those accounts for each domain.

This setting is optional.

[app:main] -> sqlalchemy.url

The access string that defines how to access your database. Example:

mysql://root:seoroct3@mailserver.example.com/mailserver?charset=utf8

AttachmentSize
ispwebadmin-1.0.tar.gz21.34 KB
ispwebadmin-1.0.1.tar.gz21.38 KB
ispwebadmin-1.0.2.tar.gz21.38 KB

DNS

DNS is short for "domain name system" and is the magic on the internet that turns names like "www.workaround.org" into IP addresses like "212.12.58.129". If you want to receive email for a certain domain you need to be in charge of its "zone" which contains the actual "records". The zone file for "workaround.org" contains an A (address) record so that you can view this website. And among other records it contains an MX (mail exchanger) record telling which servers should be used when someone sends an email to devnull@workaround.org. This is a simplified extract of the zone file:

IN  TXT  "v=spf1 ip4:212.12.58.128/27 -all"
IN  MX      10 mx1.workaround.org
IN  MX      20 mx2.workaround.org
IN  MX      30 mx3.workaround.org
IN  MX      40 mx4.workaround.org
IN  A    212.12.58.129

The "IN" is the class of the records. On the internet it's always "IN".  The second column designates the record type:

How other mail servers find your mail server

Imagine that some mail server on the internet has an email for your domain in its queue. Now it needs to find out which IP address to connect and send the email via the SMTP protocol. First it does a DNS lookup for an MX record of the domain used in the email address. If it gets several addresses back it will try the servers mentioned in the response beginning with the servers having the highest priority (=smallest number) assigned. If it doesn't reach it then it will try the next mail server in order. If the mail server did not find any MX entries then it will try another DNS lookup for an A record and send the email there.

Let's try an example. You want to send an email to devnull@workaround.org. So the domain is apparently the "workaround.org" part. Is there an MX entry? You can try that yourself using the dig tool. ("host" or "nslookup" will work equally well. Install dig using "aptitude install dnsutils" if you haven't already.)

$> dig +short workaround.org mx
30 mx3.workaround.org.
40 mx4.workaround.org.
10 mx1.workaround.org.
20 mx2.workaround.org.

So there are four mail servers in question. The one with the highest priority (10) is mx1.workaround.org. Let's find out its IP address by asking for an A record of this hostname:

$> dig +short mx1.workaround.org
212.12.58.158

The mail server will now try to reach the mail server on this IP address on TCP port 25 (SMTP) and attempt to deliver the email. Incidentally (or maybe not) the mail server does not appear to be reachable. There are no more mail servers with priority 10 so it tries the next best mail server mx2.workaround.org with priority 20. Which IP address does it have?

$> dig +short mx2.workaround.org
212.12.58.129

This mail server works better and the SMTP connection is established and the email delivered.

So in order to receive email for a domain you need to set at least one MX entry in its zone to point to your mail server. Note that an MX entry points to a hostname - never an IP address. If you can't add MX entries then an A record will do, too.

SPF to avoid spoofing

If you are tired of faked emails from popular domains like eBay or Paypal then there is hope. And you can even protect your own domain from spammers sending in your name. It is about SPF which is short for sender policy framework. Basically  the owner of a domain defines which IP addresses are allowed to send email. Let's take an example. If you have the dig tool around then run…

$> dig +short workaround.org txt

and you should see something like

"v=spf1 ip4:85.214.93.191 ip4:85.214.149.150 -all"

This entry means that if someone sends you an email with a sender email address of …@workaround.org then you should only accept it if it was sent from one of these two IP addresses. The "-all" at the end tells you that no other IP addresses should be accepted (FAIL). Another definition like "~all" means a SOFTFAIL - you should at least be suspicious and perhaps increase the spam score. (Actually "~all" is widely used but pretty useless. Either you know what you are doing and use "-all" or leave it off entirely. If you want to test if SPF workd then you could use "?all".) So if someone sent you an email from …@workaround.org from another IP address then you should drop it - it's spam.

Many organisations already have SPF records that you can use to reduce the amount of spam you receive. So your duties as a mail server administrator are:

Set up an SPF entry

Obviously you need to have full control of your DNS zone to add a TXT record. If you don't then speak to your ISP and let them add the TXT entry. The SPF record needs to be properly machine readable. I suggest you go to the OpenSPF web site and use their wizard to create a proper SPF entry. Add this string as a TXT record for your domain. Just like above you should get the SPF entry back if you run:

$> dig +short mydomain.com txt

You need to be aware of one caveat though which is also SPFs biggest problem: if your users forward the email somewhere else then it might get rejected. By all means you need to be sure that your users always send email just through your mail server.

Check other mail servers' SPF entries

Fortunately there is a Debian package for the tumgreyspf software which makes checking SPF entries easy. Just install it:

$> apt-get install tumgreyspf

tumgreyspf is a policy daemon written in Python that does both greylisting and SPF checking of incoming emails. Using it is detailed in the /usr/share/doc/tumgreyspf/README.Debian file. Basically it boils down to adding one line to your smtpd_sender_restrictions (or smtpd_recipient_restrictions if you put everything in there) making use of it. Example:

smtpd_sender_restrictions =
    permit_mynetworks,
    permit_sasl_authenticated,
    [ ... ]
    check_policy_service unix:private/tumgreyspf
    [ ... ]
    permit

And to define the program that is called when using this policy service you need to add two lines to your /etc/postfix/master.cf:

tumgreyspf unix  -      n       n       -       -       spawn
    user=tumgreyspf argv=/usr/bin/tumgreyspf

Now reload your Postfix and that's it:

$> postfix reload

Case: SPF okay

Watch your /var/log/mail.log logfile. Every incoming email should now log an additional line from tumgreyspf. If the SPF check was positive then you will get:

tumgreyspf[24672]: sender SPF authorized: QUEUE_ID=""; identity=mailfrom;
   client-ip=26.21.244.31; helo=squedge2.squ.edu.om;
   envelope-from=…@squ.edu.om;
   receiver=…@workaround.org;

This means that the sender …@squ.edu.om was allowed in after checking the SPF entry.

Case: SPF fail

If the SPF check fails then you will see something like:

tumgreyspf[24672]: SPF fail - not authorized: QUEUE_ID=""; identity=mailfrom;
   client-ip=41.234.18.141; helo=gmx.de;
   envelope-from=…gmx.de;
   receiver=…christoph-haas.de;

The email got rejected. One spam mail less in the world. :)

Case: SPF softfail

The third case is when the SPF entry does not enforce a FAIL (-all) but just uses SOFTFAIL (~all). In your log file it will look like:

tumgreyspf[20408]: domain owner discourages use of this host: QUEUE_ID="";
   identity=mailfrom; client-ip=220.245.2.67; helo=220-245-2-67.static.tpgi.com.au;
   envelope-from=…@rollouts.com; receiver=…@workaround.org

Unfortunately the SOFTFAIL does not actually reject the email. But the sender still believes that this IP address should not be sending email in their name. Luckily tumgreyspf adds this information to a "Received-SPF" header into the email. Just like:

Received-SPF: Softfail (domain owner discourages use of this host) identity=mailfrom;
   client-ip=61.146.93.243; helo=mail.163gd.com;
   envelope-from=…@cantv.net; receiver=…@christoph-haas.de;

So you still have a chance to mark such emails in spam in your email client by filtering out emails having a "Received-SPF: Softfail" header.

(Unfortunately tumgreyspf does not have a configuration option to treat SOFTFAIL as FAIL.)

Case: No SPF information

If the remote domain is ignorant and stupid and does not have any SPF entries yet then your log file will read:

Received-SPF: Neutral (access neither permitted nor denied) identity=mailfrom;
   client-ip=80.65.65.222; helo=mail.unze.ba;
   envelope-from=…@gmail.com; receiver=…@christoph-haas.de; 

Quotas

Your mail server doesn't have infinite space. Especially if you use IMAP then users appreciate the comfort of leaving their emails on the server. And there are even IMAP users who are not aware of emails that are just marked for deletion but are still occupying space. So unless you have very few or just highly disciplined users you may want to limit the space a user can occupy. Dovecot can store the size of a user's mailbox and the number of mails therein along with the virtual folder.

Enabling the quota plugin in Dovecot

There are three locations in the /etc/dovecot/dovecot.conf file where you need to enable the quota plugin (taken from the Dovecot documentation):

protocol imap {
  mail_plugins = quota imap_quota
}
protocol pop3 {
  mail_plugins = quota
}
protocol lda {
  mail_plugins = quota
} 

Setting global quota

The simplest case is a general quota limit for all users. Say you grant every user 1 GB of space with no more than 1000 emails. This would be the configuration in your /etc/dovecot/dovecot.conf file:

plugin {
  quota = maildir:storage=1000000:messages=1000
} 

Keep in mind that the storage parameter refers to KB. You can even omit the messages parameter if you don't intend to limit the number of messages.

Setting per-user quota

If you have certain users who have different quotas from your global quota setting then you can store the quota settings in your virtual_users table. Use this SQL statement to add two columns to the virtual_users table:

mysql>
ALTER TABLE `virtual_users` ADD `quota_kb` INT NOT NULL,
ADD `quota_messages` INT NOT NULL ;

And you will have to enable the user_query in your /etc/dovecot/dovecot-sql.conf. Throughout this tutorial it was suggested you use the "userdb static" approach in your /etc/dovecot/dovecot.conf. But this assumes that all users have equal settings. So in this per-user quota situation you will need to disable "userdb static" again and enable "userdb sql" to make Dovecot make an extra query to fetch the user's data from the database:

#userdb static {
#  args = uid=5000 gid=5000 home=/var/vmail/%d/%n allow_all_users=yes
#

userdb sql {
    args = /etc/dovecot/dovecot-sql.conf
}

And in your /etc/dovecot/dovecot-sql.conf add this line:

user_query = SELECT CONCAT('/var/vmail/',CONCAT(SUBSTRING_INDEX(email,'@',-1),'/',SUBSTRING_INDEX(email,'@',1))) AS home, 5000 AS uid, 5000 AS gid, CONCAT('maildir:storage=',quota_kb,':messages=',quota_messages) AS quota FROM virtual_users WHERE email='%u';

This query may look a bit weird but it does the same as the above

args = uid=5000 gid=5000 home=/var/vmail/%d/%n allow_all_users=yes

line and in addition gets the quota_kb and quota_messages information.

What happens if a user is over quota

Quotas in Dovecot aren't especially user-friendly. For example the recipient does not get a warning at a soft quota limit. Just the sender gets a bounce email with the subject reading "Automatically rejected mail" saying "Your message to <john@example.com> was automatically rejected: Quota exceeded"

Offering webmail access

This step is optional but most email service providers offer their users a way to read emails in a browser. Luckily this is easy. The package you need is called "squirrelmail" - a web mail system that can use any IMAP server - just like the Dovecot IMAP server we use here. To set it up you first need to add its configuration to your Apache configuration:

$> ln -s /etc/squirrelmail/apache.conf /etc/apache2/conf.d/squirrelmail.conf
$> apache2ctl restart

You need to change one configuration option at least so please run the configuration utility:

$> squirrelmail-configure

Select option 3 (Folder Defaults) and set option 1 (Default Folder Prefix) to 'none'. You may also want to browse through the other menu options to e.g. set your organisation's name right.

Point your browser at http://YOUR-MAIL-SERVER/squirrelmail and you should see the webmail interface. Login as 'john@example.com' with password 'summersun' and you should be able to read John's email. It couldn't be simpler.

In case you are unhappy with the http://my.server/squirrelmail URL and prefer to use http://my.server instead you should read the users will prefer a simple URL section in the /etc/squirrelmail/apache.conf file.

Server-side filters: Sieve

What is server-based filtering?

You already have a fully functioning mail server that even tags spam. But so far you left the actual task of sorting out the tagged spam to the user. We can do better by setting up server-side filters. The task ahead is to move all spam emails to a user's "Spam" folder. We are using the Sieve feature of Dovecot which is a mail filter like procmail (which does not work for virtual mailboxes). Sieve has a simple scripting language that allows us to forward all emails that are tagged by SpamAssassin to a "Spam" folder automatically. At the end of the /etc/dovecot/dovecot.conf file you will find a "plugin" section. Within there you can define a "sieve_global_path" which points to a default filter file that takes effect if the recipient user hasn't defined any per-user filtering rules.

By the way the term "server-side filtering" means that the rules are executed on the server automatically. As opposed to "client-side filtering" which is configured in the mail program on the user's computer. Apparently server-side filtering is preferred as it happens even if the user is offline. And it gives you interesting options such as a vacation autoresponder or filtering based on header fields.

The global sieve filter script

A basic recipe will do the "Spam" sorting. Create a /var/vmail/globalsieverc file containing these lines:

require ["fileinto"];
# Move spam to spam folder
if header :contains "X-Spam-Flag" ["YES"] {
  fileinto "spam";
  stop;
}

Make sure that the globalsieverc file is readable by the vmail user:

$> chown vmail /var/vmail/globalsieverc

Define this file as a global filtering file within the "plugin { }" section of your /etc/dovecot/dovecot.conf file:

sieve_global_path = /var/vmail/globalsieverc

and restart Dovecot:

$> /etc/init.d/dovecot restart

Send John another spam email:

$> sendmail john@example.com < /usr/share/doc/spamassassin/examples/sample-spam.txt

Then watch the /var/vmail/dovecot-deliver.log file. The last line should look like:

deliver(john@example.com): 2009-07-01 01:00:22 Info: msgid=<GTUBE1.1010101@example.net>: saved mail to spam

This says that the email was saved into the "spam" folder. If you fetch email you should see the email show up in that folder automatically.

Note that different folders only work when IMAP is used to fetch emails. POP3 doesn't have a notion of folders and will only show emails in the inbox. And the user will have to subscribe to the "Spam" folder manually. Alternatively you can add "spam" to the "/var/vmail/example.com/john/Maildir/subscriptions" file which contains a list of folders that John wants to see in his mail program.

Managesieve server

We were talking about per-user filter scripts. How does a user manage such a script? This is done by a low-level interface called "managesieve". Previously additional software like "pysieved" was needed to offer "managesieve". In Lenny's Dovecot this is already supported out of the box.

To enable it you'll have to add "managesieve" to the "protocols" line in your /etc/dovecot/dovecot.conf file. In its "protocol managesieve" section the line "sieve=~/.dovecot.sieve" should already be set. This means that a user's sieve filtering script will be stored in his virtual "home directory". John's sieve script would end up in "/var/mail/example.com/john/.dovecot.sieve".

Managesieve client

Your users probably don't want to learn the sieve filtering language. So they need a comfortable way to administer their filtering rules. This can be accomplished by the "avelsieve" plugin to the "squirrelmail" web mail software. It's assumed that you have set up squirrelmail as decribed in its chapter. Install the "avelsieve" plugin:

$> aptitude install avelsieve

Fortunately that's all that needs to be done. If you login via squirrelmail you will see a new link at the top called "Filters". Click on it. You can now add rules of all kinds.

Note that avelsieve in Debian Lenny currently has a bug (Debian #516198 , Avelsieve #247) that can make existing rules disappear when using Dovecot and TLS. I have prepared a patched avelsieve Debian package that you can install with "dpkg -i" if you encounter the same problem.

Optional features

By now you will have a fully functional email server. There are further features that you may want to use. Pick what you need.

Fetching and filtering using fetchmail and sieve

Mathias Geat wrote an article on integrating fetchmail and sieve that fits this setup.

Removing old deleted mails

With IMAP you can mark emails as deleted and some email clients will not even show them any more. But the emails are still there and occupy space. Usually there is an option to purge all marked emails but many users do not care. So Michael Weisgerber suggests to run this command frequently via crontab to remove such emails:

$> find /var/vmail -type f -ctime +7 -name '*,ST' -delete

Dovecot renames all deleted emails so that they get a ,ST added at the end of the filename. Adjust the parameter to -ctime as you like. In this example deleted mails older than 7 days are purged.

Troubleshooting

If you are having trouble receiving or sending email you may try these general steps:
Check your /var/log/mail.log. 93.7% of all problems cause a more or less clear error message there. :)

Sending an email directly to the author asking for help is the worst choice. Sending an email directly to the author giving suggestions or feedback is a good choice. :)

Contributing to this document

Many parts of this tutorial were tried, researched and documented by helpful readers. If you would like to contribute to this tutorial you are invited to. Just use the "Add new comment" link at the bottom of each page. If possible your contributions will be added to the tutorial in the right places.

Thanks

This tutorial has been maintained since the age of Debian Woody. And a lot of people translated it, contributed suggestions and helped finding bugs. Thank you:

You can find additional contributions that did not (yet) make it into the tutorial at the appropriate contributions wiki page. Also thanks to all readers who sent feedback on the tutorial - both technical and emotional. Perhaps you are curious to read what others think. If you like to support the author you may help improve the tutorial further or consider donating. :)