This chapter of our journey leads us to Dovecot – the software that…
- gets emails from Postfix and saves them to disk
- executes user-based sieve filter rules (can be used to e.g. move emails to different folders based on certain criteria or to send automated vacation responses)
- allows the user to fetch emails using POP3 or IMAP
Before we get to the actual configuration for security reasons I recommend that you create a new system user that will own all virtual mailboxes. The following shell commands will create a system group “vmail” with GID (group ID) 5000 and a system user “vmail” with UID (user ID) 5000. (Make sure that UID and GID are not yet used or choose another – the number can be anything between 1000 and 65000 that is not yet used):
groupadd -g 5000 vmail useradd -g vmail -u 5000 vmail -d /var/vmail -m
If the /var/vmail directory was already there because you assigned it a dedicated mount point then you should make sure that the permissions are set correctly:
chown -R vmail:vmail /var/vmail
The configuration files for Dovecot are found in /etc/dovecot/conf.d/. All these files are loaded by Dovecot. This is done by this magical line in the /etc/dovecot/dovecot.conf file:
!include conf.d/*.conf
It loads all files in /etc/dovecot/conf.d/ that end in “.conf” in alphanumerical order. So “10-auth.conf” is loaded first and “90-sieve-extprograms.conf” is loaded last. The big advantage is that you can edit or replace parts of the configuration without having to overwrite the entire configuration. The main /etc/dovecot/dovecot.conf file does not require any changes. Those other files in conf.d/ however need a few changes…
conf.d/
10-auth.conf
The most common authentication mechanism is called PLAIN. However if you have Outl**k users then you need to add the LOGIN mechanism, too.:
auth_mechanisms = plain login
These two mechanisms would ask for a password without enforcing encryption to secure the password. But don’t worry. By default Dovecot sets “disable_plaintext_auth = yes” which ensures that authentication is only accepted over TLS-encrypted connections.
At the end of this file you will find various authentication backends that Dovecot ships with. By default it will use system users (those from the /etc/passwd). But we want to use the MariaDB database backend so go ahead and change this block to:
#!include auth-system.conf.ext !include auth-sql.conf.ext #!include auth-ldap.conf.ext #!include auth-passwdfile.conf.ext #!include auth-checkpassword.conf.ext #!include auth-static.conf.ext
10-mail.conf
Change the mail_location setting to:
mail_location = maildir:~/Maildir
This is the directory where Dovecot will look for the emails of a specific user. The tilde character (~) means the user’s home directory as defined in the previous section.
Further down you will find sections defining the namespaces. Those are folder structures that your email program sees when connecting to the mail server. If you use POP3 you can only access the “inbox” – which is where all incoming email is stored. Using the IMAP protocol you get access to a hierarchy of folders and subfolders. And you can even share folders between users. Or use a public folder that can be accessed by anyone – even anonymously. So IMAP is generally to be preferred.
Look for the “namespace inbox” section. If you already have emails stored on your server from previous versions of this ISPmail guide you need to change:
separator = .
here. By default the separator is “/” which creates a directory structure like “/var/vmail/example.org/john/Maildir/INBOX/staff/marketing/simon”. This is perfectly fine. But previous ISPmail guides used “.” as a separator so the mentioned folder would rather have been “/var/vmail/example.org/john/Maildir/.INBOX.staff.marketing.simon”.
Also edit the “mail_plugins” line to enable the quota plugin we will configure later and turn it into:
mail_plugins = quota
10-master.conf
This configuration file deals with services that allow communication with other processes. For example it enables or disables POP3 or IMAP. Don’t worry about the standard unencrypted TCP ports 110 (for POP3) and 143 (for IMAP). They can be kept accessible. If a user connects to these ports they will have to issue a STARTTLS command to switch into encrypted mode before they are allowed to send their password. There is basically no difference between using an plaintext port like 110 for POP3 and then using STARTTLS – or connecting to the encrypted 995 port for POP3S (=secure). See the Dovecot documentation for another explanation.
So most settings are sane here and do not have to be changed. However one change is required in the “service auth” section because we want Postfix to allow Dovecot as an authentication service. This is what has to be entered:
# Postfix smtp-auth
unix_listener /var/spool/postfix/private/auth {
mode = 0660
user = postfix
group = postfix
}
Why that strange path? Well, Postfix runs in a chroot environment located at /var/spool/postfix. It can’t access anything outside of that directory. So to allow communication with Postfix we tell Dovecot to place a communication socket into that chroot.
10-ssl.conf
Earlier in this guide you created both a key and a certificate file to encrypt the communication with POP3, IMAPs and HTTPS between the users and your mail server. You need to tell Dovecot where to find these files:
ssl_cert = </etc/letsencrypt/live/webmail.example.org/fullchain.pem ssl_key = </etc/letsencrypt/live/webmail.example.org/privkey.pem
And enforce TLS encryption by setting:
ssl = required
See the Dovecot documentation on SSL encryption for more information.
Next let’s take a look at how Dovecot knows about users and their passwords:
auth-sql.conf.ext
Dovecot reads the auth-sql.conf.ext which defines how to find user information in your database. Open the file. There are two sections:
- userdb: where to find a user’s mailbox in the file system
- passdb: where to find the user’s hashed password
By default Dovecot will run two queries at your database. One for the userdb that gets information like the user ID, group ID, home directory and quota. And another for the passdb that gets the hashed password.
The “userdb” section already reads:
userdb {
driver = sql
args = /etc/dovecot/dovecot-sql.conf.ext
}
Apparently Dovecot uses an SQL database lookup to get that information. And it refers to the dovecot-sql.conf.ext file for more information. Let’s see…
/etc/dovecot/dovecot-sql.conf.ext
You will find this file well documented although all configuration directives are commented out. Add these lines at the bottom of the file:
driver = mysql
connect = \
host=127.0.0.1 \
dbname=mailserver \
user=mailserver \
password=x893dNj4stkHy1MKQq0USWBaX4ZZdq
user_query = SELECT email as user, \
concat('*:bytes=', quota) AS quota_rule, \
'/var/vmail/%d/%n' AS home, \
5000 AS uid, 5000 AS gid \
FROM virtual_users WHERE email='%u'
password_query = SELECT password FROM virtual_users WHERE email='%u'
iterate_query = SELECT email AS user FROM virtual_users
Backslashes
What these lines mean:
- driver: the kind of database. MariaDB is the same kind as MySQL.
- connect: where to find the MySQL database and how to access it (username, password)
- user_query: an SQL query that returns the user name (=the email address), the quota, the home directory, user ID and group ID.
- password_query: this SQL query just gets the password hash from the database
- iterate_query: ‘doveadm’ uses this query to get a list of all users. That allows you to use the “doveadm user ‘*'” command later.
The user_query gets several pieces of information from the database. Let’s look at it one by one:
- email AS user
It gets the the email field from the database which corresponds to the user name. Dovecot expects it in the user field so we set an alias to “user”. - userdb_quota_rule
This is the user’s quota in bytes. Think of it as the maximum possible space on disk that the user can occupy. As documented Dovecot expects the quota in a special format like “*:bytes=10000” if the user should not be able to store more than 10,000 bytes. That’s why we begin the string with ‘*:bytes=’. - userdb_home
This leads to the directory where all emails and various control files for this user are located. The placeholder ‘%d’ replaces the domain and ‘%n’ the user part. So for John that makes it “/var/vmail/example.org/john”. - userdb_uid and userdb_gid
Those are the user ID and group ID of vmail user – 5000 for both. Dovecot uses it to set the permissions of files it creates. As all users share the same system user “vmail” this is just a static number.s
Fix permissions
Make sure that only root can access the SQL configuration file so nobody else is reading your database access passwords:
chown root:root /etc/dovecot/dovecot-sql.conf.ext chmod go= /etc/dovecot/dovecot-sql.conf.ext
Restart Dovecot from the shell:
systemctl restart dovecot
Look at your /var/log/mail.log logfile. You should see:
... Dovecot v2.3.13 (f79e8e7e4) starting up for imap, lmtp, sieve, pop3 (core dumps disabled)
If you get any error messages please double-check your configuration files.
A tiny note:
“userdb_quota_rule […]
userdb_home[…]
userdb_uid and userdb_gid[…]”
It is possible to get password an user information in just one query using userdb prefetch (dovecot.conf). In this case the query has to return the described userdb fields.
But the above method uses two queries (password_query and user_query), in that case the fields are just:
quota_rule, home, uid, gid
without any userdb prefix (anyway, the above query ist right).
Thanks for the comment. In fact I used to recommend prefetching several years ago. However there was a specific reason that made it fail. But I can’t find it in my notes any more. Have you tried prefetching and still got a working setup?
Hello!
Thank you so much for that great tutorial!
I noticed some of the following errors in my logs:
Dec 20 15:40:28 XXX postfix/lmtp[183352]: XXX: to=, relay=XXX[private/dovecot-lmtp], delay=XXX, delays=XXX/XXX/XXX/XXX, dsn=4.4.2, status=deferred (lost connection with XXX[private/dovecot-lmtp] while sending end of data — message may be sent more than once)
It seems that these errors appear when I receive emails with lots of images attached.
Any idea what could help?
I have read that increasing the vsz_limit parameter for the service lmtp might help. How can I do that (if that is really the reason for these errors)?
Hi Matthias. I haven’t observed that issue yet. How large has the email been that you tried to send? The default_vsz_limit is defined in /etc/dovecot/conf.d/10-master.conf and defaults to 256 MB. Postfix by default accepts emails up to 10 MiB. I usually set it to 40 MiB in /etc/postfix/main.cf (message_size_limit = 41943040). Some mail servers may allow up to 100 MB but I rather play it safe.
Hello Christoph!
Thanks for your answer.
I had this problem with 2 emails I have received from a friend:
First Mail had 27 pictures with an image-size of 27 MB together.
Second Mail had 25 pictures with an image-size of 26 MB together.
Normally I get pictures that are uploaded to a picture host (so there is only a link in the mail).
But the person that sent me these 2 mails had all pictures attached to the mail.
By the way: he also sent me some other mails with images under 20 MB size. The all came through without problems.
My Postfix message_size_limit is 104.857.600.
Please check the output of “doveconf” and look for “service lmtp”. That’s what I get:
service lmtp {
chroot =
client_limit = 1
drop_priv_before_exec = no
executable = lmtp
extra_groups = $default_internal_group
group =
idle_kill = 0
privileged_group =
process_limit = 0
process_min_avail = 0
protocol = lmtp
service_count = 0
type =
unix_listener /var/spool/postfix/private/dovecot-lmtp {
group = postfix
mode = 0600
user = postfix
}
unix_listener lmtp {
group =
mode = 0666
user =
}
user =
vsz_limit = 18446744073709551615 B
}
Is your server short on RAM? Do you see anything with “memory” in /var/log/syslog?
My Server has 64GB RAM, used are ~24GB (37%). You can find more information about the server here:
https://www.hetzner.com/de/dedicated-rootserver/ax41-nvme
There are no problems with the memory in the logs.
My doveconf output:
service lmtp {
chroot =
client_limit = 1
drop_priv_before_exec = no
executable = lmtp
extra_groups = $default_internal_group
group =
idle_kill = 0
privileged_group =
process_limit = 0
process_min_avail = 0
protocol = lmtp
service_count = 0
type =
unix_listener /var/spool/postfix/private/dovecot-lmtp {
group = postfix
mode = 0600
user = postfix
}
unix_listener lmtp {
group =
mode = 0666
user =
}
user =
vsz_limit = 18446744073709551615 B
}
Here is some information I found about the problem:
http://www.ronny-mueller.com/2019/05/06/howto-solve-lost-connection-with-private-dovecot-lmtp-while-sending-end-of-data-error/
https://community.keyhelp.de/viewtopic.php?t=10233
Changing the vsz_limit to 512MB seems to work (I see no error when I send myself a email with 30MB of images attached).
Hi Christoph,
on this version of ispmail I couldn’t find how to disable pop3 protocol on dovecot; in the old version (2.3.4.1) of dovecot in dovecot.conf I get the “protocols” parameters; this parameter seems to be disappeared in this version (2.3.13) config files. Looking on the former version config file in the /usr/share/dovecot/protocols.d/ all files terminating with .protocol are loaded; so it’s appeared logic to me to append a .disabled to pop3d.protocol file to get the pop3 disabled; it worked, pop3 port is closed. Maybe worth of mentioning on your tutorial: Who use pop3 anymore? (and is this the right way to get this thing done?)
Another issue I’m struggling with is I’m unable to save emails in “Draft” and “Sent” folder with Thunderbird; emails are correctly received and stored by dovecot so I don’t think a /var/vmail permission issue; logfiles don’t contains anything interesting to point me at the issue, also configuring 10-logging.conf to get a verbose logging. Any help / suggestion is appreciated here.
Grüezi, Eidgenosse. 🙂 Basically don’t install the “dovecot-pop3d” package and the protocol should be gone.
In Thunderbird you can choose which folder to use for drafts and sent emails. What happens when you leave it to the default settings? Are the folder not shown? Do you get an error message when you try to save a draft?
Grüezi! 😉
Thanks for your answer. It seems that is something weird in Thunderbird: As said nothing is logged on server side and on Thunderbird only a generic “Network error” popup appears… nothing really useful for debug. All configurations are left as default (IT are lazy of nature…) on folders and also recreating a new setup on Thunderbird doesn’t help.
If you have a suggestion on alternate email client than Thunderbird to suggest (MacOS / Linux) it’s the right time!
Very strange. Do you get the same problems when connecting through Roundcube/webmail? For Linux and MacOS you could try Sylpheed, Claws or Trojitá. But for Linux I’d probably rather use Evolution, Geary or KMail. For testing “mutt” is very nice. However it must work with Thunderbird nonetheless. Have you tried with a fresh Thunderbird profile (thunderbird –ProfileManager)? I’d probably use tcpdump/tshark to see how far the communication goes.
Great Tutorial! I Had issues with special characters in password… mine was #…
Solved by making these lines
connect = \
host=127.0.0.1 \
dbname=mailserver \
user=mailserver \
password=x893dNj4stkHy1MKQq0USWBaX4ZZdq
Into…
Connect=”host=127 dbname= … password=pw#$@”
all on 1 line.
First off, thanks for this guide, both current updated version and all the versions over the years.
It’s especially great that you make sure it is up to date and relevant all the time, so much of my time is “find guide, find the bits that don’t work any more, mash bits in for other newer guides, make offering to techgods/desses” so it’s always quite refreshing to come back here for updates and see working examples!
I would like to ask, is there a reason you don’t use a username separate from an email address?
I’ve noticed in the years since first setting up a server based on this, more and more of my email aliases are showing up in my logs for failed login attempts. Which is interesting, to see which companies own up to their userDBs being leaks and which do not, since part of the reason for me making this was to use unique emails for everything, as well as unique passwords.
But I am thinking it might be worth getting logins via map, smtp and submission happening with an actual username, and removing the “@” symbol from legal characters in the usernames (which in turn will cut down sql lookups a LOT for me).
Before I go too far down that rabbit hole is there anything particular that stops you from publishing this with username and email separate, or is it purely a matter of “one less thing to do and teach”?
Again, thank you so much for making and maintaining this guide!
My take: ISPmail (and the like) is designed to seamlessly handle multiple top-level domains so it isn’t reasonable to use “usernames” without a domain. The whole system is built around e-mail addresses (as are the user manager tools).
Assigning a domain if only the username is present is certainly not trivial and would surely splatter custom conditional PHP code all over the place.
Is there a way to use Caddy certificates with postfix/dovecot?
Great guide!
Is there any specific reason not to use the Debian way of adding a system user (in the system UID range)?
adduser –system –group –home /var/vmail vmail
Instead of:
groupadd -g 5000 vmail
useradd -g vmail -u 5000 vmail -d /var/vmail -m
cookie monster ate some of my dashes, I try again:
adduser –system –group –home /var/vmail vmail
Chris,
I am having two problems:
SSL and hostname resolve
/etc/dovecot/conf.d/10-ssl.conf does not like:
ssl_cert = </etc/letsencrypt/live/webmail.example.org/fullchain.pem
ssl_key = </etc/letsencrypt/live/webmail.example.org/privkey.pem
and /etc/mysql/my.cnf does not like: its not compatible with mariadb but works as a work around because the mysql will only logon as localhost and not 127.0.0.1
i have been using 'skip-name-resolve'
but what is a permanent fix? then how do i get the let's encrypt! certificates to work?
thanks
Hi Cristoph,
as always: Great evolution of the tutorial/workshop – many thanks for that & happy new year indeed 😉
I was a bit confused why after creating a test mailbox the usual folders like Sent, Drafts, Trash etc. were not created. The solution is quite simple:
Just add “auto = create” (alternatively “auto = subscribe”) to the corresponding namespaces in /etc/dovecot/conf.d/15-mailboxes.conf.
After restarting Dovecot, this works perfectly (tested with MS “Outlook”, Ritlab’s “The Bat” and with “FairMail” on Android).
– – – Config Snippet – – –
# NOTE: Assumes “namespace inbox” has been defined in 10-mail.conf.
namespace inbox {
# These mailboxes are widely used and could perhaps be created automatically:
mailbox Drafts {
special_use = \Drafts
auto = subscribe
}
mailbox Junk {
special_use = \Junk
auto = subscribe
}
mailbox Trash {
special_use = \Trash
auto = subscribe
}
mailbox Sent {
special_use = \Sent
auto = subscribe
}
# archive folder, not in Config by default
mailbox Important {
special_use = \Sent
auto = subscribe
}
– – – Config Snippet – – –
Hi Christoph,
Thanks for your brilliant tutorial. I regularly check your page to improve my configuration. My current problem is that I still have a relatively large number of passwords with an outdated hash. Now I have found this page, but unfortunately I can’t get a working configuration. https://wiki.dovecot.org/HowTo/ConvertPasswordSchemes
It seems to me that it is the:
userdb {
driver = prefetch
}
Have you ever worked with this?
Thanks
Tobias
Translated with http://www.DeepL.com/Translator (free version)
10-mail.conf
mail_location = maildir:~/Maildir
postfix won’t create Maildir