[2019-12-27]

Monitoring and backup

You have come a long way. Glad you made it. So many weird things complicated the journey that evolved from the very beginning of using email roughly 35 years ago. But before you take your much deserved break from these adventures let’s take care of some final tasks.

Monitoring

For me there is nothing more awkward than having your users tell me that my systems are not working properly. As a good sysadmin I should notice outages in advance. The most frequent cause for mail server issues is a disk that is running full. Users are pretty bad with cleaning their mess up. For them the space of their mailbox is infinite. So one way is to use quotas where possible. But at least you should get notified of error conditions on your mail server.

I will not dive into monitoring deeply here. Setting up proper monitoring will easily fill a guide similar to the size of the one you are currently reading. (Trust me, I have done a video series on it.) But you should either have your own monitoring system set up or at least use an internet service to check if emails are getting through.

Things I suggest you check frequently:

  • Is there enough space on disk? Especially /var/vmail?
  • Are the SMTP, Submission, POP and IMAP ports responding from other servers?
  • Is your mail server’s IP address blacklisted? There are several services you can use.
  • Are any emails coming through? (I just do a simple count like “grep -c relay= /var/log/mail.log“)
  • Numer of emails in the mail queue? (mailq)
  • Are your backups restoreable? A propos backups…

Backup

No matter if you are running your family mail server or are in charge for an organisation – nobody likes to lose email. There are many different backup tools for different purposes. Just to name my open-source favorites that ship with Debian:

  • rsnapshot. A simple solution if you want to send backups to other servers using SSH/SCP. You can also keep backups for different points in time on your local server. But if your disk crashes then all would be lost. Also rsnapshot can become slow with many emails because every file is copied as a link. But you can access the files directly without any restore procedure. It’s basically a frontend for rsync which is very reliable.
  • borg. This tools needs to be installed both on the client and the server. Setting up an archive (where backups live) on the server feels a bit weird. But you are rewarded by compression, encryption and deduplication. Also data is stored in chunks which takes less I/O than rsnapshot. It is very efficient.
  • restic. Similar to Borg in a way. It also offers deduplication, compression and encryption. It’s main advantage is to be able to store backups in S3 buckets. This makes it my favorite tool at the moment because I like off-site backups. And I’m not talking about holy-shit-S3-is-expensive backups that you get at Amaz*n, Micr*s*ft or G**gle. I use Wasabi which costs me $6 per TB per month. (No, they do not sponsor me. I just like them.) That makes them 5-10 times cheaper than other providers.
  • Bareos. Formerly known as Bacula until its author started to have crazy ideas and these guys forked it. This is a full-fledged system to handle many servers in parallel and store things on different media – for example LTO magnetic tapes in a big tape library.

If you cannot backup your entire disk then at least include these directories:

  • /etc
  • /var/vmail
  • /var/log
  • /var/backups

Also make sure to exclude:

  • /var/lib/mysql

Instead run…

mysqldump mailserver > /var/backups/mailserver.sql

…daily to get a proper database backup that can be restored later.

I also like to backup the output of these files because they help me figure out the specific layout of a server:

  • fdisk -l
  • lshw
  • lsb_release -a
  • dpkg –get-selections

If one of your users just accidentally deleted an email and wants it back then simply get the files from /var/vmail/domain/user/Maildir/… from the backup and put them into the original location. Dovecot will handle everything else. You don’t need to restart anything.

5 thoughts on “Monitoring and backup

  • 2019-12-31 at 17:23
    Permalink

    Hallo,

    Thanks for this Buster guide. It’s really cool stuff.

    In my case, I booked a 5 Eur box (2 vcpu and 4 Gb ram) to start with a short list of users with a 40 GB disk.

    If I have to expand disk space, what would be correct ?

    1. Get a volume on my virtual server, move the /var/vmail to the bigger volume and make a symbolic link ?

    2. Mount a bucket from aws or google, transfert files and make a symlink ?

    3. Nothing, there’s a better procedure ?

    Reply
    • 2020-01-01 at 11:39
      Permalink

      I would not use an S3 bucket. You would probably mount it using fuse/S3 and that’s slow. Especially as each email is a seperate file.

      If your ISP allows you to add disks then I would suggest to turn the new volume into a PV (physical volume) in terms of LVM (logical volume manager). Say it’s /dev/vdb then you would issue:

      pvcreate /dev/vdb

      Then create a new volume group (unless you have one already) from this disk:

      vgcreate gompali0 /dev/vdb

      Then create a logical volume (LV) which is the counterpart to a classical block device volume. Pro tip: don’t use all the space from the physical volume yet… extend as needed:

      lvcreate –name lvmail –size 20G gompali0

      And finally put a file system onto that:

      mkfs.ext4 /dev/gompali0/lvmail

      This may seem like overkill but later you can enlarge the volume if needed (lvextend…), move the whole stuff to another (maybe larger) disk (pvmove…) or add another disk to get more space (vgextend). You could even create a snapshot from your logical volume to create and have a microsend-exact read-only copy of your /var/vmail to make backups.

      Yes, yes, LVM seems weird at first. But I have enjoyed it so many times that I really appreciate its help. 🙂

      Reply
      • 2020-01-02 at 10:42
        Permalink

        Hi,

        I settled the server on a machine that does not support LVM, the provider says I have to install it myself, or provide an iso image that they’ll run for me. But my guess is that I don’t know their network parameters, so I’m not able to install a debian fully with LVM.

        Now I have several choice : I get attach “virtual volume” they sell and move my /var/vmail on this volume, which is by itself expandable. Or I can change to a provider that let choose for LVM. I’m no sysadmin 🙂 i may opt out for buying their expensive storage

        Reply
        • 2020-01-02 at 12:01
          Permalink

          If there is Debian on the system you will be able to use LVM. It’s not something that the data center has to activate or that the underlying storage system knows about. It’s just software within Debian. So the commands I mentioned will still work.

          But if you feel less confident with LVM then feel free to mount /dev/vdb (or whatever the new disk is called) to /var/vmail and move stuff over. Sure.

          Reply
          • 2020-01-02 at 13:17
            Permalink

            ok, i’ll dive the LVM doc
            anyway it’s a good news i will not need to reformat actual work

Leave a Reply

Your email address will not be published. Required fields are marked *