You have come a long way. Glad you made it. So many weird things complicated the journey that evolved from the very beginning of using email roughly 35 years ago. But before you take your much deserved break from these adventures let’s take care of some final tasks.
Monitoring
For me there is nothing more awkward than having your users tell me that my systems are not working properly. As a good sysadmin I should notice outages in advance. The most frequent cause for mail server issues is a disk that is running full. Users are pretty bad with cleaning their mess up. For them the space of their mailbox is infinite. So one way is to use quotas where possible. But at least you should get notified of error conditions on your mail server.
I will not dive into monitoring deeply here. Setting up proper monitoring will easily fill a guide similar to the size of the one you are currently reading. But you should either have your own monitoring system set up or at least use an internet service to check if emails are getting through.
Things I suggest you check frequently:
- Is there enough space on disk? Especially /var/vmail?
- Are the SMTP, Submission, POP and IMAP ports responding from other servers?
- Is your mail server’s IP address blacklisted? There are several services you can use.
- Are any emails coming through? (I just do a simple count like “
grep -c relay= /var/log/mail.log“) - Numer of emails in the mail queue? (mailq)
- Are your backups restoreable? A propos backups…
Backup
No matter if you are running your family mail server or are in charge for an organisation – nobody likes to lose email. There are many different backup tools for different purposes. Just to name my open-source favorites that ship with Debian:
- rsnapshot. A simple solution if you want to send backups to other servers using SSH/SCP. You can also keep backups for different points in time on your local server. But if your disk crashes then all would be lost. Also rsnapshot can become slow with many emails because every file is copied as a link. But you can access the files directly without any restore procedure. It’s basically a frontend for rsync which is very reliable.
- borg. This tools needs to be installed both on the client and the server. Setting up an archive (where backups live) on the server feels a bit weird. But you are rewarded by compression, encryption and deduplication. Also data is stored in chunks which takes less I/O than rsnapshot. It is very efficient.
- restic. Similar to Borg in a way. It also offers deduplication, compression and encryption. It’s main advantage is to be able to store backups in S3 buckets. This makes it my favorite tool at the moment because I like off-site backups. And I’m not talking about holy-shit-S3-is-expensive backups that you get at Amaz*n, Micr*s*ft or G**gle. I use Wasabi which costs me $6 per TB per month. (No, they do not sponsor me. I just like them.) That makes them 5-10 times cheaper than other providers.
- Bareos. Formerly known as Bacula until its author started to have crazy ideas and these guys forked it. This is a full-fledged system to handle many servers in parallel and store things on different media – for example LTO magnetic tapes in a big tape library.
If you cannot backup your entire disk then at least include these directories:
- /etc (configuration)
- /var/vmail (the actual emails)
- /var/log (in case the system breaks you may be able to figure out what happened last)
- /var/backups (for general backups)
- /var/lib/rspamd/dkim/ (private part of the DKIM keys)
Also make sure to exclude:
- /var/lib/mysql
Instead run…
mysqldump mailserver > /var/backups/mailserver.sql
…daily to get a proper database backup that can be restored later.
I also like to backup the output of these files because they help me figure out the specific layout of a server:
- fdisk -l
- lshw
- lsb_release -a
- dpkg –get-selections
If one of your users just accidentally deleted an email and wants it back then simply get the files from /var/vmail/domain/user/Maildir/… from the backup and put them into the original location. Dovecot will handle everything else. You don’t need to restart anything.
Hallo,
Thanks for this Buster guide. It’s really cool stuff.
In my case, I booked a 5 Eur box (2 vcpu and 4 Gb ram) to start with a short list of users with a 40 GB disk.
If I have to expand disk space, what would be correct ?
1. Get a volume on my virtual server, move the /var/vmail to the bigger volume and make a symbolic link ?
2. Mount a bucket from aws or google, transfert files and make a symlink ?
3. Nothing, there’s a better procedure ?
I would not use an S3 bucket. You would probably mount it using fuse/S3 and that’s slow. Especially as each email is a seperate file.
If your ISP allows you to add disks then I would suggest to turn the new volume into a PV (physical volume) in terms of LVM (logical volume manager). Say it’s /dev/vdb then you would issue:
pvcreate /dev/vdb
Then create a new volume group (unless you have one already) from this disk:
vgcreate gompali0 /dev/vdb
Then create a logical volume (LV) which is the counterpart to a classical block device volume. Pro tip: don’t use all the space from the physical volume yet… extend as needed:
lvcreate –name lvmail –size 20G gompali0
And finally put a file system onto that:
mkfs.ext4 /dev/gompali0/lvmail
This may seem like overkill but later you can enlarge the volume if needed (lvextend…), move the whole stuff to another (maybe larger) disk (pvmove…) or add another disk to get more space (vgextend). You could even create a snapshot from your logical volume to create and have a microsend-exact read-only copy of your /var/vmail to make backups.
Yes, yes, LVM seems weird at first. But I have enjoyed it so many times that I really appreciate its help. 🙂
Hi,
I settled the server on a machine that does not support LVM, the provider says I have to install it myself, or provide an iso image that they’ll run for me. But my guess is that I don’t know their network parameters, so I’m not able to install a debian fully with LVM.
Now I have several choice : I get attach “virtual volume” they sell and move my /var/vmail on this volume, which is by itself expandable. Or I can change to a provider that let choose for LVM. I’m no sysadmin 🙂 i may opt out for buying their expensive storage
If there is Debian on the system you will be able to use LVM. It’s not something that the data center has to activate or that the underlying storage system knows about. It’s just software within Debian. So the commands I mentioned will still work.
But if you feel less confident with LVM then feel free to mount /dev/vdb (or whatever the new disk is called) to /var/vmail and move stuff over. Sure.
ok, i’ll dive the LVM doc
anyway it’s a good news i will not need to reformat actual work
pflogsumm(https://tracker.debian.org/pkg/pflogsumm) is a nice tool to monitor postfix, e.g. via a daily systemd timer:
/etc/systemd/system/pflogsumm.timer:
[Unit]
Description=Daily pflogsumm report
[Timer]
OnCalendar=*-*-* 00:00:20
AccuracySec=10s
Persistent=true
[Install]
WantedBy=postfix.service
/etc/systemd/system/pflogsumm.service:
[Unit]
Description=Daily pflogsumm report
[Service]
Type=oneshot
ExecStart=/bin/sh -c “/usr/sbin/pflogsumm -d yesterday /var/log/mail.log /var/log/mail.log.1 | mail -s ‘pflogsumm daily report’ root”
KillMode=process
# hardening options
# details: https://www.freedesktop.org/software/systemd/man/systemd.exec.html
LockPersonality=yes
MemoryDenyWriteExecute=yes
NoNewPrivileges=yes
PrivateDevices=yes
PrivateNetwork=yes
PrivateTmp=yes
ProtectControlGroups=yes
ProtectHome=yes
ProtectKernelLogs=yes
ProtectKernelModules=yes
ProtectKernelTunables=yes
ProtectSystem=strict
ReadWritePaths=/var/spool/postfix/
RestrictNamespaces=yes
RestrictRealtime=yes
This is nice! Thank you!
Thanks for the guide. Got it working like a charm (took a few days, though, but I’m not an IT pro), migrated our small server fom a very ancient configuration (qmail/vpopmail) – imapsync is a lifesaver in such situations. Also got mailman3 working (from Debian packages.
A tiny issue: dpkg requires two hyphens in front of the get-selections option.
This is a great guide, really. In this section about backups, there’s a tiny comment I need to drop in:
I’d suggest to add “/var/lib/rspamd/dkim/” to the list of directories to be backed up, to not lose DKIM private keys.
Good point. Thanks.
Thank you so much for providing such a good guide! I’m in Linux land for quite some time now, but I can’t remember the last time I found a guide so well written, one where all the steps turned out to be working and all the test results on my side match up with the ones provided. It guided me from 0 to a working system in no time (… well … it took me around two days but I’m using a separate webserver with a proxy and didn’t want to open any ports on the mailserver except the absolute necessary ones, so my setup does not match 100% the one described 😉 )
Amazing guide, thank you so much! can’t wait for the ansible update :D.. before this, I’d been mucking around with the idea of my own smtp server for a while, and made reasonably good progress towards it. I got one going in plesk, but the UI was kinda clunky and it still took ages to set up. This guide take things to a whole new level with best practices that have much broader application. Very cool and thank you 🙂
I have a newbie question on debian buster, I converted to linux desktop (mainly ubuntu) a few years ago and haven’t looked back! Simply love it (this guide is a case in point for one of many reasons why!). My questions is why did you chose it for a SMTP server over alternatives? Is it just that its small and reliable, and rate of change is slower (change breaks stuff) than the alternatives? I’ve had it running on vps for a while now and I’m impressed as it’s been rock solid. I’d love to hear your thoughts.
restic does not (yet) feature compression (restic bug #21 tracks progress on this feature).
Yet another backup software package which does support compression (as well as public key encryption) is one called “rdedup”.
Christoph ,
thanks a lot!
I searched a lot of tutorial on this subject and they are either old or with huge mistakes (and btw, al lot of them is just copy&paster all over again…)
Your tutorial is spot on!
I thank you from the bottom of my heart.
Best regards,
Aleksandar
Video link above about Monitoring you did previously is not valid, https://www.packtpub.com/networking-and-servers/zabbix-network-monitoring-essentials-video, please update or post alternative.
The publisher told me that the interest in the video series has dropped. So they do not offer it any longer. I assume it’s because I features the 2.x version. If only they knew that the only major thing that changed since is minor improvements in the UI. 🙂