[2019-11-26]

Testing email delivery

So far you have spent considerable time with theory and configuration. Are you worried whether all you did actually leads to a working mail server? Before we do the final steps let’s take a break and verify that everything you did now works as expected.

At this point the /var/vmail directory should be empty or maybe contain an “example.org” directory if you played with the john@example.org account previously. You can get a list of all files and directories within by running:

find /var/vmail

Although there are not actually any emails on the server yet, you may still get somethinng along the lines of:

/var/vmail
/var/vmail/example.org
/var/vmail/example.org/john
/var/vmail/example.org/john/.dovecot.sieve
/var/vmail/example.org/john/sieve
/var/vmail/example.org/john/sieve/roundcube.sieve
/var/vmail/example.org/john/sieve/tmp
/var/vmail/example.org/john/Maildir
/var/vmail/example.org/john/Maildir/subscriptions
/var/vmail/example.org/john/Maildir/maildirfolder
/var/vmail/example.org/john/Maildir/new
/var/vmail/example.org/john/Maildir/cur
/var/vmail/example.org/john/Maildir/dovecot-uidlist
/var/vmail/example.org/john/Maildir/dovecot-uidvalidity
/var/vmail/example.org/john/Maildir/.INBOX.test
/var/vmail/example.org/john/Maildir/.INBOX.test/maildirfolder
/var/vmail/example.org/john/Maildir/.INBOX.test/new
/var/vmail/example.org/john/Maildir/.INBOX.test/dovecot.index.log
/var/vmail/example.org/john/Maildir/.INBOX.test/cur
/var/vmail/example.org/john/Maildir/.INBOX.test/dovecot-uidlist
/var/vmail/example.org/john/Maildir/.INBOX.test/tmp
/var/vmail/example.org/john/Maildir/dovecot-uidvalidity.5ddc842b
/var/vmail/example.org/john/Maildir/tmp

Basically the schema you see here is /var/vmail/DOMAIN/USER/Maildir/…

Every IMAP mail folder then has three subdirectories:

  • new – every file here is an email that was stored in this mail folder but not yet read
  • cur – the same but for email that has been read already
  • tmp – for temporary actions by the mail server

According to our Dovecot configuration the mail folders are nested like this:

  • …/Maildir/new/… – the main inbox
  • …/Maildir/.INBOX.reddit/new/… – the “reddit” mail folder below the inbox
  • …/Maildir/.INBOX.servers.inga/new/… – the “servers”/”inga” mail folder below the inbox

If you do not use the legacy scheme with “separator = .” your directories will rather look like:

  • …/Maildir/new/… – the main inbox
  • …/Maildir/INBOX/reddit/new/… – the “reddit” mail folder below the inbox
  • …/Maildir/INBOX/servers/inga/new/… – the “servers”/”inga” mail folder below the inbox

Check Postfix

To check for obvious configuration error in Postfix please run:

postfix check

Did you get the error…
error: open database /etc/aliases.db: No such file or directory“? Don’t worry. Just run the “newaliases” command to create a new machine-readable file from what aliases were defined in the /etc/aliases.

Very likely you will get this error: “postfix/postfix-script: warning: symlink leaves directory: /etc/postfix/./makedefs.out“. That is a harmless bug of the Debian package that can safely be ignored.

If you get errors about missing values in smtpd_recipient_restrictions and compatibility levels then please do as the log says: “postconf compatibility_level=2” and restart Postfix.

Send a test email

It is time to send a new email into the system. Open a new terminal window and run

tail -f /var/log/mail.log

to see what the mail server is doing. Now let’s send an email to John. My favorite tool for mail tests is swaks that you installed earlier. In a second terminal run:

swaks --to john@example.org --server localhost

If all works as expected your mail.log will show a lot of technical information about the email delivery. Let me explain what happens at each stage.

  • postfix/smtpd[29225]: connect from localhost.localdomain[127.0.0.1]
    Postfix receives an incoming SMTP connection.
  • postfix/smtpd[29225]: 8BA46A0A3A: client=localhost.localdomain[127.0.0.1]
    Postfix assigns a unique ID (8BA46A0A3A) to this connection so that you see which log lines belong together. This is especially important with busy mail servers where multiple mails are handled in parallel.
  • postfix/cleanup[29233]: 8BA46A0A3A: message-id=20191126153053.029243@webmail.example.org
    swaks created a unique message id to the email which helps you identify specific mails in the log file.
  • postfix/qmgr[13667]: 8BA46A0A3A: from=root@webmail.example.org, size=485, nrcpt=1 (queue active)
    The sender was root@webmail.example.org. This is logged after swaks sent the “MAIL FROM” line during the SMTP dialog.
  • postfix/smtpd[29225]: disconnect from localhost.localdomain[127.0.0.1] ehlo=1 mail=1 rcpt=1 data=1 quit=1 commands=5
    The SMTP communication ends. Postfix has now received and queued the email.
  • dovecot: lmtp(29237): Connect from local
    Postfix connects to Dovecot to hand over the email via the LMTP interface.
  • dovecot: lmtp(john@example.org)<29237><2PJTIh033V01cgAARGEcaw>: sieve: msgid=20191126153053.029243@webmail.example.org: stored mail into mailbox 'INBOX.test'
    Dovecot received the email and even evaluated John’s sieve rule which made the email get stored to the ‘test’ folder of his mailbox.
  • dovecot: lmtp(29237): Disconnect from local: Client has quit the connection (state=READY)
    The LMTP connection between Postfix and Dovecot is closed.
  • postfix/lmtp[29236]: 8BA46A0A3A: to=john@example.org, relay=webmail.example.org[private/dovecot-lmtp], delay=0.01, delays=0/0/0/0.01, dsn=2.0.0, status=sent (250 2.0.0 john@example.org 2PJTIh033V01cgAARGEcaw Saved)
    This is the one of the most interesting lines in your mail log. It tells you what happened with a certain email. In this case it says that it was handed over to dovecot-lmtp and that the delivery was successful (status=sent). The status codes like 2.0.0 are defined in RFC 3463 and work similar to status codes in HTTP. Codes beginning with ‘2’ are good. Those with ‘4’ are temporary errors. And ‘5’ stands for a permanent failure.

Your output may look slightly differently. Just focus on the parts that are printed in bold letters. If everything worked as expected Postfix has accepted the email and forwarded it to Dovecot which in turn wrote the email in John’s maildir. If you get any errors in the log file then try to understand the error message and find the cause of the problem before you proceed.

Look again:

find /var/vmail

Dovecot has now created a directory structure for John and created a new file:

/var/vmail/
[…]
/var/vmail/example.org/john/Maildir/new/1515485447.M404984P2636.mail,S=510,W=522
[…]

The file may have a different name on your system – that’s okay. The file name printed in bold letter depicts the actual email file. It is the only file in the “new” folder.

Accessing the email as a file on disk

The file just contains the email:

Return-Path: <root@mail.example.org>
Delivered-To: john@example.org
Received: from mail.example.org
 by mail.example.org (Dovecot) with LMTP id iHHHEwd5VFpMCgAA3BOsLQ
 for <john@example.org>; Tue, 09 Jan 2018 09:10:47 +0100
Received: bymail.example.org (Postfix, from userid 0)
 id 30B4B41A98; Tue, 9 Jan 2018 09:10:47 +0100 (CET)
Message-Id: <20180109081047.30B4B41A98@mail.example.org>
Date: Tue, 9 Jan 2018 09:10:47 +0100 (CET)
From: root@mail.example.org (root)

Di 9. Jan 09:10:47 CET 2018

If anything went wrong then carefully check the last lines of your /var/log/mail.log. It will very likely point you to the problem. Or read the troubleshooting section. Or just add a comment to this page and ask other readers for help.

You can also use a slightly more comfortable tool to access Maildirs that will come handy for you as a mail server administrator: “mutt”.

mutt -f /var/vmail/example.org/john/Maildir

(You may get asked to create /root/Mail – this is standard procedure. Just press Enter.)

What you see now are the contents of John’s mailbox:

ispmail-jessie-mutt-index

Using mutt is a nice way to check mailboxes while you are logged in to the mail server.

To reiterate what happens when you receive an email:

  1. Postfix receives the email (using the “sendmail” command in this example – but usually through the network using the SMTP protocol from other servers)
  2. Postfix talks to Dovecot via LMTP and hands over the email
  3. Dovecot write the email file to disk

Accessing the email via IMAP (Roundcube)

Now that the email has been delivered you can talk to Dovecot using the IMAP protocol to retrieve your email again. Are you still logged in via the Roundcube webmail interface? Then just reload and you will see the email:

Accessing the email via IMAP (mutt)

In the last section we used mutt to access the Maildir directly on disk. But remember that mutt also speaks IMAP:

mutt -f imaps://john@example.org@localhost

Does it show you the email now? Great.

Access the email through IMAP (desktop client)

Your users are not likely to use “mutt” to read their email – unless they are immortal console nerds. So of course you can use a graphical mail client to access John’s emails. If you use Thunderbird, Evolution, A**le Mail or some other IMAP client then feel free to configure a new IMAP account and connect to the server. Remember that the user name and email address are both “john@example.org” and the password is “summersun”.

POP3 versus IMAP

Aside from talking IMAP Dovecot also knows how to speak POP3. But let’s face it – POP3 is dead. I do not offer it to my users any more. But as your users may ask let’s quickly review the differences of the two protocols:

  • POP3 (Post Office Protocol) is a simple protocol that lets you fetch email from a single mailbox. It is usually used to collect all emails, though you can also leave them on the server but this is a bit of a hack and you can’t create multiple folders on the server to sort your mail. It saves space on the mail server because the email gets moved to the user’s hard disk on their computer. But they won’t be able to access the same email from another computer. Besides you cannot create multiple folders on the server to sort your mail. There is just the inbox. This variant is antiquated and not exactly user-friendly.
  • IMAP (Internet Messaging Application Protocol) is predominantly focused upon leaving your mail on the server but you can also collect it like POP3. The inbox is where your incoming emails are stored but users can also maintain folders and move emails to them. But users can move emails to different directories. IMAP is useful when you want to access your email from different locations without losing mail because you fetched it from another location. The drawback is that lazy users leave their mail on the server thus filling up your server’s hard disk (unless you use quotas).

24 thoughts on “Testing email delivery

  • 2019-12-30 at 16:15
    Permalink

    Hiya,

    Me again. Working through this while I have the time. So I tried the swaks test and I was getting this.

    ** 451 4.3.5 : Recipient address rejected: Server configuration problem

    I know this error well so I checked my postfix configuration

    I had this

    smtpd_recipient_restrictions = reject_unauth_destination check_policy_service unix:private/quota-status

    I had to modify it to include “permit_mynetworks”

    smtpd_recipient_restrictions = permit_mynetworks reject_unauth_destination check_policy_service unix:private/quota-status

    I am pretty sure this is OK since its this way on my other postfix servers

    Thanks Again!!!

    Francesca

    Reply
    • 2019-12-30 at 21:31
      Permalink

      Seems to work for me without permit_mynetworks

      Reply
    • 2019-12-31 at 12:09
      Permalink

      I suspect the cause somewhere else. Try “postfix check”. You probably changed more than one thing.

      Reply
      • 2019-12-31 at 17:26
        Permalink

        Hiya,

        I get this

        /usr/sbin/postconf: warning: /etc/postfix/master.cf: undefined parameter: mua_sender_restrictions
        /usr/sbin/postconf: warning: /etc/postfix/master.cf: undefined parameter: mua_client_restrictions
        /usr/sbin/postconf: warning: /etc/postfix/master.cf: undefined parameter: mua_helo_restrictions

        Interesting but I don’t recall anywhere in the guide where we were setting this up ??

        I pretty much to follow this faithfully started with a clean debian 10 system and only cut and pasted any postconf’s from the guide.

        Looking at what the heck is up with the above issues.

        Thanks Again!!

        Francesca

        Reply
  • 2019-12-31 at 17:36
    Permalink

    Hello,

    I think the key part in master.cf that is causing this is::

    submission inet n – y – – smtpd
    -o syslog_name=postfix/submission
    -o smtpd_tls_security_level=encrypt
    -o smtpd_sasl_auth_enable=yes
    -o smtpd_tls_auth_only=yes
    -o smtpd_reject_unlisted_recipient=no
    -o smtpd_client_restrictions=$mua_client_restrictions <— Here
    -o smtpd_helo_restrictions=$mua_helo_restrictions <— Here
    -o smtpd_sender_restrictions=$mua_sender_restrictions <— Here
    -o smtpd_recipient_restrictions=
    -o smtpd_relay_restrictions=permit_sasl_authenticated,reject
    -o milter_macro_daemon_name=ORIGINATING

    I followed this part faithfully from the guide

    But of course I did not read this part 🙂

    "the restrictions point to $mua… configuration variables (MUA = mail user agent) that you can set in your main.cf. The defaults for these four (client, helo, sender, recipient) restrictions are just empty. So unless you want to enforce extra restrictions I suggest you comment out or delete these lines."

    So I commented those out for now and will come back to them later on.

    However I am still getting this. (Unless I have permit_mynetworks)

    <** 451 4.3.5 : Recipient address rejected: Server configuration problem”

    Since its working as is for you guys let me mess with this some more.

    Thanks!!

    Francesca

    PS: Told you I am learning some cool things from this.

    Reply
    • 2020-01-01 at 11:41
      Permalink

      First run “postfix check” to look for obvious configuration problems.

      Next check your /var/log/mail.log. I best you will find the issue for the “server configuration problem” there. Maybe a permissions problem on the “mysq-…cf” file or a database access error. It will be easy to resolve.

      Reply
      • 2020-01-01 at 13:37
        Permalink

        Hiya,

        Well I did fix the postfix check issues which were because I did not carefully read the guide 🙂

        Here is what I am getting. (Removed any personal data)

        “Jan 1 08:30:38 vps1 postfix/smtpd[26845]: connect from localhost[127.0.0.1]
        Jan 1 08:30:38 vps1 dovecot: script: Fatal: Client not compatible with this binary (connecting to wrong socket?)
        Jan 1 08:30:39 vps1 dovecot: script: Fatal: Client not compatible with this binary (connecting to wrong socket?)
        Jan 1 08:30:39 vps1 postfix/smtpd[26845]: warning: problem talking to server private/quota-status: Success
        Jan 1 08:30:39 vps1 postfix/smtpd[26845]: NOQUEUE: reject: RCPT from localhost[127.0.0.1]: 451 4.3.5 : Recipient address rejected: Server configuration problem; from= to= proto=ESMTP helo=
        Jan 1 08:30:39 vps1 postfix/smtpd[26845]: disconnect from localhost[127.0.0.1] ehlo=1 mail=1 rcpt=0/1 quit=1 commands=3/4”

        Maybe I need to look at the quota part of the guide again as I note it was moved and maybe revised some. (Never used quotas like this in the past)

        Also not sure what the heck this error is ??

        “dovecot: script: Fatal: Client not compatible with this binary (connecting to wrong socket?”

        Going to start with the quota settings as even though I got a “Success” it still fired off a error.

        Thanks Again,

        Francesca

        Reply
  • 2020-01-01 at 13:54
    Permalink

    Ok,

    Progress 🙂

    It seems to be the quota section (I followed your guide before the modifications and I think I have an error in there)

    Reason I say is that when I remove this

    “check_policy_service unix:private/quota-status”

    Leaving this.

    smtpd_recipient_restrictions = reject_unauth_destination

    Things now work as expected.

    So its something with quotas I think.

    Let me go back and triple check my conf files. I know I had a pickle of a time with the “{” issue and I modified that so maybe I missed a bracket or something.

    As I related I am totally not familiar with the quota setup. Even though its elegant and right.

    So its not you its me. 🙂

    Thanks And Happy New Year!!!

    Francesca

    Reply
    • 2020-01-01 at 14:15
      Permalink

      Yep,

      For whatever reason it was the placement of the “{“,s in the quota conf file. (I had implemented quotas before you moved things and fixed some of the syntax.

      I had it working (No Dovecot Errors) but the quota status service section was not properly delimited so it was not working. That was the ” dovecot: script: Fatal: Client not compatible with this binary (connecting to wrong socket?)” which I was thinking was something with swaks.

      So now things work properly.

      Onward and Upwards!!! 🙂

      Thanks Again!!!

      Francesca

      Reply
      • 2020-01-01 at 15:35
        Permalink

        Oh, good to hear you found it. 🙂

        Reply
  • 2020-01-02 at 21:19
    Permalink

    Hi Christoph!

    Well, this question might become quite embarrassing for me 🙂 However, I’m still going to ask it, because I’m curious whether I got something wrong or what point I’m missing.

    After sending test e-mails to one of the test e-mail addresses on the new mailserver I’m currently setting up, I logged on to the mailbox using Roundcube. I then did an “ls -la” on the corresponding Maildir on disk. Just as expected, most of the initial files and directories you mentioned above as well as the files containing the e-mails I sent, were created in /var/vmail///Maildir.

    I then created a new folder “Test” using Roundcube’s folder interface. Since I had set “separator = /” in 10-mail.conf (and, of course, restarted Dovecot afterwards 🙂 ), I expected a new subdirectory of the root Maildir’s “INBOX” directory to be created (just as you described above). To my surprise, however, the new folder was created directly in the user’s Maildir as a new directory with name “.Test”. Again, using Roundcube, I then created a subfolder “Subtest” in the newly created folder. This, in turn, resulted in the new directory “.Test.Subtest” being created in the user’s root Maildir again.

    To check, whether the folders were actually created correctly and accessible, I then tried to log into the user’s root mailbox as well as the (sub)folder using mutt. It worked — both for “separator = .” as well as “separator = /”. In fact, this was the first time I “saw” the chosen separator “influencing something”. More precisely, it was used to separate the subfolders from the rest of the mailbox path displayed in mutt’s bottom status bar.

    Having a look at Dovecot’s documentation at [1], I learned that “[…] changing the separator doesn’t change the on-disk “layout separator”” and that one has to change the “LAYOUT” if one wants to change the entire directory structure [1] [2]. I noticed that there was a similar discussion going on in the Stretch guide [3].

    Now, my question is, did you really manage to achieve the hierarchical folder structure without changing the “LAYOUT” setting on the mailbox definition? If yes, what am I missing?
    Oh and by the way, what is you recommendation for a new installation without any legacy constraints? Should one go for the hierarchical on-disk structure?

    Thanks in advance!
    Gabe

    [1] https://doc.dovecot.org/configuration_manual/namespace/
    [2] https://wiki.dovecot.org/MailLocation/Maildir
    [3] https://workaround.org/ispmail/stretch/setting-up-dovecot

    Reply
    • 2020-01-03 at 09:42
      Permalink

      I have an awkward confession to make, too. None of the productive servers I run use the “/” separator layout because they have been in use for about ten years with the same users and I never bothered to change the layout upon upgrades. 🙂

      What the documentation in [1] probably means that nothing on disk changes magically just because you re-define the “separator” setting. If you have a .INBOX.projects.ispmail structure you will need to create the directories yourself and move it into INBOX/projects/ispmail.

      I’m working on the Ansible playbooks. Let me check that once the playbook ran through. In the meantime maybe someone else can share their experiences.

      Reply
      • 2020-01-03 at 16:14
        Permalink

        Same or at least similar situation here. I first set up the mail server I’m currently migrating back in 2014 on Wheezy. Then, in 2015, I upgraded it to Jessie leaving the directory structure untouched. Currently, I’m upgrading the same installation to Buster and thankfully, your guide was released just on time 🙂

        Actually, I don’t need/want to change the folder structure. I was just being curious “what happens, if”… 😀 That’s why I set the “LAYOUT=fs” option on the “mail_location” definition in 10-mail.conf just as described in [2] today. Just for the fun of it. And et voila, the maildir folders were organized hierarchically 🙂

        Reply
  • 2020-02-18 at 20:21
    Permalink

    Hi Christoph! For an unknown reason I’ve found on my clean buster installation postfix configured with:

    inet_interfaces = loopback-only

    Obviously with this parameter all tests you suggested passed successfully but incoming emails are always refused.

    Now the parameter has been changed like:

    inet_interfaces = all

    Everything works now; I don’t found any place where you suggest to set this parameter as “loopback-only” so I think it’s something related to my buster installation.

    Note: my installation came from a linux container provided by Proxmox…

    Reply
    • 2020-03-05 at 10:46
      Permalink

      I just checked the /var/lib/dpkg/postfix.postinst which is the shell script run after the Postfix Debian package is installed. Apparently when the configuration is set to “Satellite system” it will set the inter_interfaces to loopback-only. Maybe that happened?

      Reply
  • 2020-03-26 at 19:36
    Permalink

    Hello,

    I have identified an issue which was raised a warning during the swaks test:

    postfix/trivial-rewrite[1292]: warning: do not list domain example.net in BOTH mydestination and virtual_mailbox_domains

    I have commented “example.net” in mydestination line in /etc/postfix/main.cf

    and it is working !

    Reply
    • 2020-04-05 at 10:50
      Permalink

      Thanks! I can confirm I had this issue too.

      In that situation, the lmtp delivery did not trigger (instead postfix was doing local delivery apparently) and I couldn’t understand what I forgot to do to let postfix use LMTP with dovecot.

      It was because I had the domain both in mydestination and coming from the virtual_mailbox_domains.

      After removing it from mydestination, and sending another test email, postfix stopped complaining and used the LMTP delivery as expected and I cloud see the email being delivered in the correct mailbox by checking through IMAP.

      Reply
  • 2020-05-10 at 09:40
    Permalink

    For some reason this line didn’t work for me:
    postconf virtual_mailbox_domains=mysql:/etc/postfix/mysql-virtual-mailbox-domains.cf
    It worked on a box I have on Strech but not in with Buster, I don’t know why and so the email was al delivered to /var/mail/user
    I had to change this to mailbox_transport = lmtp:unix:private/dovecot-lmtp for it to work

    Reply
  • 2020-05-12 at 15:45
    Permalink

    Am I mistaken to assume, that in this setup one can only manage their milters with the RoundCube UI?

    Reply
    • 2020-05-12 at 15:49
      Permalink

      You probably mean the Sieve filters, right? In fact you can manage it with any software that can speak “managesieve”. I have seen a Thunderbird plugin to manage them for example.

      Reply
  • 2020-07-19 at 16:14
    Permalink

    Hi,
    I am blocked here, can’t send e-mail with the swaks command.
    I get : Recipient address rejected: User unknown in local recipient table
    However, by rechecking the postfix and mysql settings, the command :
    postmap -q john@sorpi.fr mysql:/etc/postfix/mysql-virtual-mailbox-maps.cf returns 1

    And it is even worse, in the logs, I get this :
    warning: do not list domain sorpi.fr in BOTH mydestination and virtual_mailbox_domains
    but again the
    postmap -q example.org mysql:/etc/postfix/mysql-virtual-mailbox-domains.cf
    command returns 1 too.

    I double checked and reran postconf commands but it doesn’t change the result.
    Hope you have some ideas 🙂

    Reply
    • 2020-07-19 at 16:15
      Permalink

      I forgot to remove my DNS everywhere in my message, but don’t worry I didn’t put example.org everywhere 🙂

      Reply

Leave a Reply

Your email address will not be published. Required fields are marked *