This chapter of our journey leads us to Dovecot – the software that…
- gets emails from Postfix and saves them to disk
- executes user-based sieve filter rules (can be used to e.g. move emails to different folders based on certain criteria or to send automated vacation responses)
- allows the user to fetch emails using POP3 or IMAP
Before we get to the actual configuration for security reasons I recommend that you create a new system user that will own all virtual mailboxes. The following shell commands will create a system group “vmail” with GID (group ID) 5000 and a system user “vmail” with UID (user ID) 5000. (Make sure that UID and GID are not yet used or choose another – the number can be anything between 1000 and 65000 that is not yet used):
groupadd -g 5000 vmail useradd -g vmail -u 5000 vmail -d /var/vmail -m
If the /var/vmail directory was already there because you assigned it a dedicated mount point then you should make sure that the permissions are set correctly:
chown -R vmail:vmail /var/vmail
The configuration files for Dovecot are found in /etc/dovecot/conf.d/. All these files are loaded by Dovecot. This is done by this magical line in the dovecot.conf file:
!include conf.d/*.conf
It loads all files in /etc/dovecot/conf.d/ that end on “.conf” in alphanumerical order. So “10-auth.conf” is loaded first and “90-sieve-extprograms.conf” is loaded last. The big advantage is that you can edit or replace parts of the configuration without having to overwrite the entire configuration. The main /etc/dovecot/dovecot.conf file does not require any changes. Those other files in conf.d/ however need a few changes…
conf.d/
10-auth.conf
The most common authentication mechanism is called PLAIN. However if you have Outl**k users then you need to add the LOGIN mechanism, too.:
auth_mechanisms = plain login
These are plaintext (unencrypted) ways to transmit a mail user’s password. But don’t worry. By default Dovecot sets “disable_plaintext_auth = yes” which ensures that authentication is only accepted over TLS-encrypted connections.
At the end of this file you will find various authentication backends that Dovecot ships with. By default it will use system users (that are listed in /etc/passwd). But we want to use the MariaDB database backend so go ahead and change this block to:
#!include auth-system.conf.ext !include auth-sql.conf.ext #!include auth-ldap.conf.ext #!include auth-passwdfile.conf.ext #!include auth-checkpassword.conf.ext #!include auth-vpopmail.conf.ext #!include auth-static.conf.ext
Now edit the SQL configuration file:
auth-sql.conf.ext
Now Dovecot reads the auth-sql.conf.ext which defines how to find user information in your database. Open the file. You will find two sections:
- userdb: where to find a user’s mailbox in the file system
- passdb: which password hash the user has
By default Dovecot will run two queries at your database. One for the userdb that gets information like the user ID, group ID, home directory and quota. And another for the passdb that gets the hashed password.
So please change the “userdb” section to:
userdb {
driver = sql
args = /etc/dovecot/dovecot-sql.conf.ext
}
Leave the passdb section as it is. We will deal with the configuration file it refers to (/etc/dovecot/dovecot-sql.conf.ext) later.
10-mail.conf
Change the mail_location setting to:
mail_location = maildir:~/Maildir
This is the directory where Dovecot will look for the emails of a specific user. The tilde character (~) means the user’s home directory as defined in the previous section.
Further down you will find sections defining the namespaces. Those are folder structures that your email program sees when connecting to the mail server. If you use POP3 you can only access the “inbox” – which is where all incoming email is stored. Using the IMAP protocol you get access to a hierarchy of folders and subfolders. And you can even share folders between users. Or use a public folder that can be accessed by anyone – even anonymously.
Look for the “namespace inbox” section. If you already have emails stored on your server from previous versions of this ISPmail guide you need to change:
separator = .
here. By default the separator is “/” which creates a directory structure like “/var/vmail/example.org/john/Maildir/INBOX/staff/marketing/simon”. This is perfectly fine. But previous ISPmail guides used “.” as a separator so the mentioned folder would rather have been “/var/vmail/example.org/john/Maildir/.INBOX.staff.marketing.simon”.
Also edit the “mail_plugins” line to enable the quota plugin we will configure later and turn it into:
mail_plugins = quota
10-master.conf
This configuration file deals with services that allow communication with other processes. For example it enables or disables POP3 or IMAP. Don’t worry about the standard unencrypted TCP ports 110 (for POP3) and 143 (for IMAP). They can be kept accessible. If a user connects to these ports they will have to issue a STARTTLS command to switch into encrypted mode before they are allowed to send their password. There is basically no difference between using an plaintext port like 110 for POP3 and then using STARTTLS – or connecting to the encrypted 995 port for POP3S (=secure). See the Dovecot documentation for another explanation.
So most settings are sane here and do not have to be changed. However one change is required in the “service auth” section because we want Postfix to allow Dovecot as an authentication service. This is what has to be entered:
# Postfix smtp-auth
unix_listener /var/spool/postfix/private/auth {
mode = 0660
user = postfix
group = postfix
}
Why that strange path? Well, Postfix runs in a chroot environment located at /var/spool/postfix. It can’t access anything outside of that directory. So to allow communication with Postfix we tell Dovecot to place a communication socket into that chroot.
10-ssl.conf
Earlier in this guide you created both a key and a certificate file to encrypt the communication with POP3, IMAPs and HTTPS between the users and your mail server. You need to tell Dovecot where to find these files:
ssl_cert = </etc/letsencrypt/live/webmail.example.org/fullchain.pem ssl_key = </etc/letsencrypt/live/webmail.example.org/privkey.pem
And enforce TLS encryption by setting:
ssl = required
See the Dovecot documentation on SSL encryption for more information.
/etc/dovecot/dovecot-sql.conf.ext
This file is referred to by /etc/dovecot/conf.d/auth-sql.conf.ext. Do you remember from earlier? It tries to get passdb information (the user’s password hash) from there. You will find it well documented although all configuration directives are commented out. Add these lines at the bottom of the file:
driver = mysql
connect = host=127.0.0.1 \
dbname=mailserver \
user=mailserver \
password=x893dNj4stkHy1MKQq0USWBaX4ZZdq
user_query = SELECT email as user, \
concat('*:bytes=', quota) AS quota_rule, \
'/var/vmail/%d/%n' AS home, \
5000 AS uid, 5000 AS gid \
FROM virtual_users WHERE email='%u'
password_query = SELECT password FROM virtual_users WHERE email='%u'
iterate_query = SELECT email AS user FROM virtual_users
What these lines mean:
- driver: the kind of database. MariaDB is the same kind as MySQL.
- connect: where to find the MySQL database and how to use it (username, password)
- user_query: an SQL query that returns the user name (=the email address), the quota, the home directory, user ID and group ID.
- password_query: this SQL query just gets the password hash from the database
- iterate_query: ‘doveadm’ uses this query to get a list of all users. Without this query “doveadm user ‘*'” would not work.
So the user_query gets several pieces of information from the database. Let’s look at it one by one:
- email AS user
It gets the the email field from the database which corresponds to the user name. Dovecot expects it in the user field so we set an alias to “user”. - userdb_quota_rule
This the user’s quota in bytes. Think of it as the maximum possible space on disk that the user must occupy. As documented Dovecot expects the quota in a special format like “*:bytes=10000” if the user should not be able to store more than 10,000 bytes. - userdb_home
This leads to the directory where all emails and various control file of this user are located. The placeholder ‘%d’ replaces the domain and ‘%n’ the user part. So for John that makes it “/var/vmail/example.org/john”. - userdb_uid and userdb_gid
Those are the user ID and group ID of vmail user – 5000 for both. Dovecot uses it to set the permissions of files it creates. As all users share the same system user “vmail” this is just a static number.
Fix permissions
Make sure that only root can access the SQL configuration file so nobody else is reading your database access passwords:
chown root:root /etc/dovecot/dovecot-sql.conf.ext chmod go= /etc/dovecot/dovecot-sql.conf.ext
Restart Dovecot from the shell:
systemctl restart dovecot
Look at your /var/log/mail.log logfile. You should see:
... Dovecot v2.3.4.1 (f79e8e7e4) starting up for imap, lmtp, sieve, pop3 (core dumps disabled)
If you get any error messages please double-check your configuration files.
Hallo Christoph, please look at: /etc/dovecot/dovecot-sql.conf.ext you make a double connection to the database. Think one is a copy paste from the stretch guide 😉
Ouch, thanks. 🙂
Hello Christoph,
For auth-sql.conf.ext, in fact, I left the entire file free of changes. It was in the state you mentionned.
Okay, thanks. I’ll check that on a fresh installation.
I can confirm that on a fresh debian buster installation no edits where necessary to auth-sql.conf.ext as the file was in that exact state.
Same for me today 😉
Take a look at the /var/spool/postfix/private directory. If all went as intended you will find a socket file called quota-status there. Otherwise please check the /var/log/mail.log file for errors.
Well, i followed your guide with attention but i don’t see quota-status.
root@xxx:/var/spool/postfix/private# cat /var/log/mail.log
Dec 29 22:27:37 xxx postfix/postfix-script[3369]: starting the Postfix mail system
Dec 29 22:27:37 xxx postfix/master[3371]: daemon started — version 3.4.7, configuration /etc/postfix
Dec 29 22:31:53 xxx dovecot: master: Dovecot v2.3.4.1 (f79e8e7e4) starting up without any protocols (core dumps disabled)
Dec 29 22:31:58 xxx dovecot: master: Warning: Killed with signal 15 (by pid=15924 uid=0 code=kill)
Dec 29 22:31:59 xxx dovecot: master: Dovecot v2.3.4.1 (f79e8e7e4) starting up for imap, lmtp, sieve, pop3 (core dumps disabled)
Dec 30 00:33:46 xxx dovecot: master: Warning: Killed with signal 15 (by pid=18307 uid=0 code=kill)
I had the same problem as Gompali. After adding the “service quota-status …” section in 90-quota.conf the dovecot service did no more start.
A look at “service dovecot status” shows:
doveconf: Fatal: Error in configuration file /etc/dovecot/conf.d/90-quota.conf line 95: Expecting ‘{‘
I could get it running when I removed the newline between these two lines:
unix_listener /var/spool/postfix/private/quota-status
{
Regards, Thomas
i am following your guide (more or less) on ubuntu 18.04 and experienced a problem with the “service quota-status” section as written above. dovecot expects the opening squirly bracket after the “unix_listener” line seems to be expected to be on the same line as the unix_listener directive.. i get this error when i restart dovecot in my syslog:
doveconf: Fatal: Error in configuration file /etc/dovecot/conf.d/90-quota.conf line 95: Expecting ‘{‘
i moved the { to the end of the the unix_listener line and it now seems to work.
Had the same problem as Pascal, his bugfix worked for me. 🙂
Another question: Is it intentional to edit the existing /etc/dovecot/conf.d files instead of creating an override file with all custom changes inside? The latter would be helpful in case the defaults change with a package update, and gives a more comprehensive overview of the own settings. However, not sure if there are drawbacks to that approach.
I considered that, too. In fact I’m a huge fan of not changing the upstream files. In this case I found the documentation in the source files very helpful and wanted to avoid a mess in an additional file. But your solution sounds sane as well. 🙂
Thanks! 🙂 I will try to setup with override files – if it works, happy to share the documentation with you!
For quota it says Bytes but if I want to set “2147483648” Bytes = “2048” Megabytes this does not work… what do I do wrong?
Usually space on disk is measured in MB while RAM is measured in MiB. So if you want to allow 2 GB of quota that should be 2000000000.
I’m having some permission issues with the quota-warning function. when a warning should be sent out, i get this error message:
Dec 30 23:04:24 mail dovecot: lmtp(mymail@mydomain.ch): Error: quota: net_connect_unix(/var/run/dovecot/quota-warning) failed: Permission denied (euid=5000(vmail) egid=5000(vmail) missing +r perm: /var/run/dovecot/quota-warning, we’re not in group 116(dovecot), dir owned by 0:0 mode=0755)
again, i’m trying this on ubuntu 18.04, so maybe this is only an issue in my setup. i’ve also tried to add the vmail user to the dovecot group but even though i restarted both dovecot and postfix i still get the same errors.
adding global read and write permissions to /var/run/dovecot/quota-warning solved the issue, however i don’t feel good about this, not knowing the security implications this may have.
..and a small change request: after the title “Test It” you suggest one should test quotas now using swaks. this will fail at this state because we haven’t configured the lmtp connection between postfix and dovecot yet (see next section).. so if someone follows the advise and tests this right here, he will see errors from postfix saing:
“fatal: bad string length 0 < 1: virtual_mailbox_base ="
this is quite misleading (at least i fell for it ;)) before finally realizing, that one piece of the puzzle is still missing 🙂
so i suggest you mention that one can test this LATER ON using swaks.
Oops… sounds like I shouldn’t move sections around without testing everything. 🙂
Okay, quotas now have a seperate page later on.
It appears the quota section randomly vanished? Was it moved? Is it coming back?
Sorry for the confusion. I moved the quotas page to a seperate and later position in the guide because there were dependencies as pointed out by Pascal above.
Why do you still use the Maildir format?
I see a lot better performance with mdbox. What do you think about the different formats like Maildir and sdbox and mdbox?
best regards
wittiko
I considered that, too. The reason to stay with Maildir was mainly due to laziness (change as little as possible compared to previous versions) and to lack of time to actually try it out. But I see how some users have tens of thousands of emails in a single folder (mainly their inbox… I wonder how those people survive) and reading those folder is very slow.
It seems that according to https://wiki2.dovecot.org/MailboxFormat/dbox#Alternate_storage you can keep the old format and receive new emails in the mdbox format. Although I would prefer to have a clear migration path from Maildir to mdbox. Have you come across anything like that?
Alt storage is for mixing capacity and performance disks for the mailstore.
Altmove moves the old mails to the altstorage. This is not we need when converting maildir to (m)dbox.
I did the migration some years ago. I had no problems and use hourly snpshots for backup with replication to another datacenter.
I would do following:
One Server:
1) Create a snapshot of the existing data (just to be sure)
2) Doing initial dsync
3) Stop delivering Mails and kicking all users
4) Doing another Dsync
5) Update database fields or edit dovecot file
6) Reactivate mail dilivery
If you have a lot of users you can do this on per user – but then you need to query the location from the database.
PS: I would use views for dovecot and postfix and add a boolean field enabled. So you can also disable domains/mailboxes/aliases without deleting them.
Thanks for your great tutorial.
In 10-mail.conf, if you use the “”/”” seperator, with :
mail_location = maildir:~/Maildir
You will have :
Maildir/.folder/
Maildir/.folder.subfolder/
With mail_location = maildir:~/Maildir:LAYOUT=fs
You will have :
Maildir/folder/
Maildir/folder/subfolder/
Isn’t it better ?
https://wiki2.dovecot.org/MailLocation/Maildir
I think there should be no difference.
The performance should be more or less the same. Maybe some Mailboxes with a lot of subfolders could be faster with the 2nd method.
However Maildir is slow per design so i don’t think this is doing a big difference.
One of the chown commands on this page use a dot (“.”) instead of a colon (“:”) to separate user and group. This should probably be changed to a colon, which is the new syntax and would be consistent with (most) of the chown commands in the tutorial (some other pages also use dots, others use colons).
See: https://www.gnu.org/software/coreutils/manual/html_node/chown-invocation.html
Thanks. I was so used to the dot. 🙂
On my setup I prohibit TLSv1.1 with (10-ssl.conf):
ssl_min_protocol = TLSv1.2
Also I am using the imp_quota plugin (20-imap.conf):
protocol imap { mail_plugins = $mail_plugins imap_quota }
I propose to add an iterate query to the file /etc/dovecot/dovecot-sql.conf.ext :
# Query to get a list of all usernames
iterate_query = SELECT email AS user FROM virtual_users
This is useful for doveadm. Without it the command
doveadm user ‘*’
returns an error instead of a list of all users.
Good idea. Thanks. I have added that.
I wonder if it makes sense to limit the vmail user a bit more, by
1. making it a system user
2. giving nologin as shell
3. explicitly have a disabled login
For additional security, /var/vmail could be made chmod 700, I think?
For the service auth section, does it make sense to set mode = 0600 for additional security too?
A system user sounds reasonable. You mean with a UID below 1000, right? I’m taking a note at least for the next version of the guide.
Nologin and disabled login sounds good. Will add that.
I have 0700 on /var/vmail on my server. So that seems to be fine. I can’t think of a service that needs to access the directory using group permissions.
Regarding the settings of the auth socket… I think there was a reason that I didn’t set it to 0600. But I can’t quite remember what it was. Currently my test server is down and I don’t want to try it on a production system. Have you tried it by any chance?
Something like:
useradd –home-dir /var/vmail –create-home –system –shell /usr/sbin/nologin –user-group vmail
At the end of “/etc/dovecot/dovecot-sql.conf.ext” you talk about “userdb_uid and user_gid”. I think those should be “uid and gid”. That’s what you use in the SQL.
You are right of course. Thanks for the hint.
Is there a specific reason to use “host=127.0.0.1” instead of “host=localhost” in “/etc/dovecot/dovecot-sql.conf.ext”? In couldn’t think of one and the documentation tells nothing about that.
Yes, same reason as with Postfix. 🙂 See the “prepare the database” page for a detailed explanation.
I have a question. According to the dovecot documentation, we could use the prefetch driver instead of SQL; this way a single lookup can return information necessary for both userdb and passdb. Is there any specific reason why you chose the SQL driver instead of prefetch? Does it perhaps have any side effect?
Sometimes separator is mis-spelled as seperator
Thanks. There is a buggy wiring in my brain regarding that word because my english teacher has already been correcting that 35 years ago. 🙂
In the 10-ssl.conf section, the “Earlier in this guide” link is to the stretch page
Thanks for spotting that. Surely a copy/paste error.
Hello – first off, these are great guides and I wanted to THANK YOU for publishing these for all of us.
Next, it appears that the “dovecot-sql.conf.ext” is changed quite a bit from the previous guides and that the mail location and quota have been added to this file. My questions are:
1) Do I need to use quota if not desired?
2) Is there any reason I cannot just use the previous versions of the “dovecot-sql.conf.ext” file and define my mail location as in previous guides?
3) If the mail location is fetched from DB via the “dovecot-sql.conf.ext” file, then what effect does the “mail_location = maildir:~/Maildir” statement have?
Thanks in advance for your time on these questions.
Hey Tony.
1) No.
2) To be honest I don’t remember the exact settings of those many guides. But you should be fine if you stay with the way the previous guide did it.
3) You get the “home” from the database. That is the “~” part of the mail_location.
dovecot-sql.conf.ext says that password_query needs to return password and (user or (username and domain)), yet you return just the password. Why? It even gives example queries:
# Example:
# password_query = SELECT userid AS user, pw AS password \
# FROM users WHERE userid = ‘%u’ AND active = ‘Y’
#
#password_query = \
# SELECT username, domain, password \
# FROM users WHERE username = ‘%n’ AND domain = ‘%d’
That file also reads:
# None of these are strictly required. If you use a single UID and GID, and
# home or mail directory fits to a template string, you could use userdb static
# instead. For a list of all fields that can be returned, see
# http://wiki2.dovecot.org/UserDatabase/ExtraFields
You have quoted the wrong section. It says that for the userdb section, yet I’m talking about the passdb section.
You are right. Still the username and domain do not need to be fetched from the database because they are derived from the email address.
It’s better not to hardcode 5000 as uid/gid in the SQL query but set the default uid/gid in 10-mail.conf using the user and group names:
mail_uid = vmail
mail_gid = vmail
and not return any uid/gid overrides in the SQL.
Correction.
Turns out there are rare cases (bugs?) when dovecot needs uid/gid but doesn’t fallback to using mail_uid/mail_gid. So do hardcode uid/gid in the SQL query, however not as a number but as a name, e.g.:
‘vmail’ as uid,
‘vmail’ as gid
Also set:
mail_uid = vmail
mail_gid = vmail
in 10-mail.conf too.
Quota rule support b/k/M/G/T/% suffixes,It’s better not to using “b”as default,try this:
user_query = SELECT email as user, \
concat(‘*:bytes=’, quota, ‘M’) AS quota_rule, \
‘/var/vmail/%d/%n’ AS home, \
5000 AS uid, 5000 AS gid \
FROM virtual_users WHERE email=’%u’
Hi Christoph,
Is there a reason why you haven’t included editing 15-mailboxes.conf to include the
auto = subscribeoption to the mail folders (as per previous guides)? I was at the rspamd section and suddenly I realized I had no Junk folder (or any other folders) in roundcube, only inbox.They do get created automatically when adding a mail client but it may be confusing for someone that is following this guide for the first time not to seeing any mail folders in roundcube.
Hello,
my name is hans-werner and I have a question. What are the correct file and folder permissions in /var/vmail/ ?
I have 0755 for folders and 0644 for files, but should it not be 750 for folders and 640 for files?
cu
Hawe
I’m using the guide step by step. This is the furthest I’ve ever been! I am getting the following error when I test iMAP:
Sep 20 18:04:27 raspberrypi dovecot: imap-login: Disconnected (auth failed, 1 attempts in 6 secs): user=, method=PLAIN, rip=86.176.179.xx, lip=192.168.1.xxx, TLS, session=
I replaced some details if x’s for privacy. The certificates was fine. I tired with my own email address and john@example.org. Both came back with ‘Login failed’.
Any suggestions?
Just ran into problems after upgrading dovecot to 2.3.7.2. I now get the error `dh key too small`.
It looks like since v2.3 you need a pem size of at least 4096.
For anyone with the same issue. I fixed it by generating a dh.pem file: `openssl dhparam 4096 > dh.pem` (will take very long!).
Then I added it in the `10-ssl.conf` as `ssl_dh = <[path]/dh.pem`.
Hello,
I’m using two dovecot instances with dsync – how do i delete the users mail data (maildir) properly with doveadm when a virtual user gets removed from the db?
thanks,
Alex
Great tutorial Christoph.
I encountered an issue not documented in your guide, found when testing with roundcube.
In dovecot-sql-conf.ext there is a default password scheme parameter as follows:
default_pass_scheme = MD5
This default setting caused problem when I changed test user password through Roundcube. I had to change this to:
default_pass_scheme = CRYPT
Thought I would pass along in case other readers run into this. Thanks!
>In dovecot-sql-conf.ext there is a default password scheme parameter as follows:
>default_pass_scheme = MD5
Where in the Buster tutorial do you see that?
I don’t think MD5 (PLAIN-MD5 to be more precise) has been used since Wheezy.
@harshness – It is not in the tutorial, that is why I added the comment 🙂
I did a fresh install of Buster yesterday, all default packages from Deb repositories. In my dovecot-sql-conf.ext file the offensive parameter is on line 81. Note that IT IS commented out! But the value is MD5. I can only assume this is the default value if you make no changes. I should have been more clear that the parameter is commented out by default.
I discovered this when testing Roundcube in the “Webmail using Roundcube” section. The paragraph right above ‘sieve plugin’ section talks about changing password from within Roundcube, logging out, then logging back in again. This is when I found out it was using MD5 (stated clearly in /var/log/mail.log).
In the codeblock for dovecot-sql.conf.ext, it looks like the password=[example password] is on a new line, when in fact it belongs to the connect line.
driver = mysql
connect = host=127.0.0.1 dbname=mailserver user=mailserver password=x893dNj4stkHy1MKQq0USWBaX4ZZdq
user_query = SELECT email as user, \
…
Indeed. The line wraps badly here. Thanks for the hint. Fixed.
Hello,
i have an error when im trying to send an email :
Feb 16 19:41:27 mail postfix/smtps/smtpd[10263]: warning: SASL PLAIN authentication failed:
Feb 16 19:41:33 mail postfix/smtps/smtpd[10263]: warning: SASL LOGIN authentication failed: GFzc3dvcmQ6
Thx.
Best regards
So, after beating my head against an error, I finally figured it out:
Everything went swimmingly, right up until I tried to log in, when I got a server error. Checking the log gave me: dovecot: imap(foo@bar.com): Error: Namespace ”:
No home directory for system user. Can’t expand ~Maildir for mail root dir in: ~Maildir
Turns out that in 10-mail.conf, I needed to change ‘mail_priviledged_group = mail’ to ‘… = vmail’, so dovecot had the permissions to create the mailbox
Hello,
Is it possibleto configure pop3 port on Microsoft Outlook 2019 client with option starttls. When I choose pop3, the only option is SSL/TLS. Thanks