So many bad guys everywhere. Paranoia you say? Well, take a look at your mail log. While I was preparing this guide I could barely follow it. The mail server I used for testing was by far not ready and I already got visits by a bunch of chinese IP addresses trying to guess accounts of email users. Scoundrels constantly scan the internet for mail servers and do not tire to find a way to abuse them. Let’s do something about that before they actually guess email accounts successfully.
Attack patterns
There was one pattern that caught my eye:
Jan 24 23:59:34 jen postfix/smtpd[31879]: lost connection after AUTH from unknown[122.231.43.242]
Jan 24 23:59:35 jen postfix/smtpd[31879]: lost connection after AUTH from unknown[122.231.43.242]
Jan 24 23:59:36 jen postfix/smtpd[31879]: lost connection after AUTH from unknown[122.231.43.242]
Jan 24 23:59:37 jen postfix/smtpd[31879]: lost connection after AUTH from unknown[122.231.43.242]
Jan 24 23:59:38 jen postfix/smtpd[31879]: lost connection after AUTH from unknown[122.231.43.242]
Some chinese system is trying to authenticate against my Postfix time and again. And believe it or not – it was doing that for days. My log file grew up to 500 MB per day just from those dorks. Let’s put an end to that.
Installation
A standard piece of software I install on servers for that purpose is fail2ban. Go put it on your mail server:
apt install fail2ban
It knows about a lot of patterns in your log files that report something harmful. The software runs in the background and follows the various log files. Each new line in the log is compared against regular expressions. If a match is found it blocks the IP address that was mentioned in the log.
Adding a filter
As fail2ban does not know that “lost connection after AUTH” is a failure we need to add that pattern. Create a new file /etc/fail2ban/filter.d/postfix-ispmail.conf with this content:
[INCLUDES]
before = common.conf[Definition]
_daemon = postfix(-\w+)?/(?:submission/|smtps/)?smtp[ds]
failregex = ^%(__prefix_line)slost connection after AUTH from \S+\[<HOST>\]$
ignoreregex =[Init]
journalmatch = _SYSTEMD_UNIT=postfix.service
This defines a new filter called “postfix-ispmail”. The name is derived from the filename of the filter. The important line is the failregex. It described a regular expression – a machine-readable description of what are you searching for. %(…)s expands a variable. And __prefix_line is a pattern describing the left part of each log line: “Jan 25 00:32:01 jen postfix/smtpd[32600]: “. The “postfix/smtpd” part is defined by the _daemon variable. “\S+” means that at this place you expect one or more character that are not a space. The brackets need to be escaped by a backslash because they have a special meaning in regular expressions. And <HOST> is a built-in pattern of fail2ban that stands for an IP address. You should get yourself acquainted with regular expressions even if they may scare you at first. They are more powerful than scary actually. 🙂
Adding a jail
fail2ban can now understand the pattern we are looking for. But where do we look? And what do we do if we find a match? That’s our next task. Create another file /etc/fail2ban/jail.d/postfix-ispmail.conf with this content:
[postfix-ispmail] logpath = %(syslog_mail)s enabled = true port = smtp,submission
The first line refers to the name of the filter. We called the filter file “postfix-ispmail.conf” so this is the filter name (the “.conf” is ignored). The log file we want to follow is defined in the syslog_mail variable which gets defined elsewhere (in the paths-debian.conf file). It points to /var/log/mail.log. We want to have this jail enabled because the default in /etc/fail2ban/jail.conf is having it disabled. And finally we define which network ports we want to block in order to block an attacker.
Make it so
Time for action. Restart fail2ban to load your configuration changes:
service fail2ban restart
To see what fail2ban did take a look at /var/log/fail2ban.log. In my example it logged:
2018-01-24 23:59:34,443 fail2ban.filter [32003]: INFO [postfix-ispmail] Found 122.231.43.242
2018-01-24 23:59:35,430 fail2ban.filter [32003]: INFO [postfix-ispmail] Found 122.231.43.242
2018-01-24 23:59:36,434 fail2ban.filter [32003]: INFO [postfix-ispmail] Found 122.231.43.242
2018-01-24 23:59:37,427 fail2ban.filter [32003]: INFO [postfix-ispmail] Found 122.231.43.242
2018-01-24 23:59:38,413 fail2ban.filter [32003]: INFO [postfix-ispmail] Found 122.231.43.242
2018-01-24 23:59:38,781 fail2ban.actions [32003]: NOTICE [postfix-ispmail] Ban 122.231.43.242
It found the “lost connection after AUTH” line mentioning that chinese IP address five times and decided to ban it. The reason is part of the configuration in /etc/fail2ban/jail.conf:
# “bantime” is the number of seconds that a host is banned.
bantime = 600# A host is banned if it has generated “maxretry” during the last “findtime”
# seconds.
findtime = 600# “maxretry” is the number of failures before a host get banned.
maxretry = 5
So by default if such a rogue pattern occurs at least maxretry times within findtime seconds the IP will be banned for bantime seconds. You can customize these values but please do not edit the jail.conf. Instead create a jail.local file as described in “man 5 jail.conf”.
Banning magic
Do you wonder how that banning actually works? fail2ban adds new rules to your host’s firewall using iptables. The list of current iptables rules can be viewed using…
iptables -nvL
It will show your firewall rules. fail2ban creates a new chain for each jail you configure. I call my jail “ispmail-postfix” so my rules look like:
Chain f2b-postfix-ispmail (1 references)
pkts bytes target prot opt in out source destination
3 152 REJECT all -- * * 125.106.22.171 …
22 972 REJECT all -- * * 115.213.146.75 …
3 152 REJECT all -- * * 1.195.252.154 …
837 39050 RETURN all -- * * 0.0.0.0/0 0.0.0.0/0
But there is a nicer way to see the current status of a jail:
fail2ban-client status postfix-ispmail
It will show something like:
Status for the jail: postfix-ispmail |- Filter | |- Currently failed: 1 | |- Total failed: 168 | `- File list: /var/log/mail.log `- Actions |- Currently banned: 14 |- Total banned: 36 `- Banned IP list: 1.199.191.44 125.106.250.51 115.213.233.184 …
More jails
You may want to browse through the pre-defined filters in either /etc/fail2ban/jail.conf or by looking at the files in /etc/fail2ban/filter.d/. By default only the “sshd” filter is enabled by the /etc/fail2ban/jail.d/defaults-debian.conf file. But feel free to add further files enabling…
- dovecot
- postfix
- postfix-sasl
- roundcube-auth
That will put an end to brute force attacks once and for all. Fun fact… two days after writing this I had already banned 1645 IP addresses.
(If we come up with more patterns that fail2ban doesn’t know yet I will add them. Did you find any?)
IPv6 support
Many ISPs support IPv6 nowadays. Unfortunately the version of fail2ban in Debian Stretch does not. IPv6 support has been added in version 0.10 but you get version 0.9.6 by default. There is no official backport of the new version yet. But I can at least offer you a quick backport so that you can use version 0.10 on Debian Stretch.
Download fail2ban_0.10.2-1_all.deb
Install it using…
dpkg -i fail2ban_0.10.2-1_all.deb
…and make sure any missing dependencies are installed…
apt -f install
Now fail2ban will be able to block IPv6 addresses.
Hi,
Thanks for you great tutorial, it helped me a lot!
Maybe it would be an idea to add the detected spam to the postfix fail2ban jail?
I am not fit enough in fail2ban configuration, but I think with matching for the line
Jan 25 07:14:08 srv5 postfix/cleanup[4939]: 5877618800AB: milter-reject: END-OF-MESSAGE from q212.mkzz.xyz[103.203.148.212]: 4.7.1 Spam message rejected; from= to= proto=ESMTP helo=
This should work, shouldn’t it?
Tanks and best regards
Christof
Note: the filter is named roundcube-auth for me on Stretch instead of roundcube.
Thanks. Fixed.
Hi
Any ideas why my fail2ban is missing the “paths-debian.conf”?
I mean I fixed it, but why/how? What am I missing?
Maybe you have accidentally deleted it? According to https://packages.debian.org/stretch/all/fail2ban/filelist it’s part of the package. You are using Debian Stretch and the default APT repositories, right?
Okay… just to close this, it was my mistake. I screwed the config files up while migrating from an old system. Everything is like it should now….
Very good guide, could install my mailserver within half a day.
ipv6 is not blocked with version 0.9.6 of fail2ban which is currently delivered with debian stretch.
we have to wait until it is available via apt or install it manually.
i have a ipv6 enabled server and need failban for ipv6, too.
kind regards
t.
Thanks for the hint. I had that on my TODO list as my new server has an IPv6 address and of course I am using. As 0.10.x has not made it into Debian Stretch I did a quick backport of the package. See the last section on this page.
It may be helpful to point out that in order to activate existing jails, all you have to do is add a “/etc/fail2ban/jail.d/${service-name}.conf” file that contains
[${service-name}]
enabled = true
You don’t have to copy the complete configuration from “/etc/fail2ban/jail.conf”. May be obvious to some, but it wasn’t 100% clear to me right away.
It may also be better to move the custom filter instructions below the general description on how to set up fail2ban. It confused me for some time because I wasn’t sure how much of that I would have to replicate to enable f2b for those other services, i.e. “installation, explanation, more jails, custom filters”
I’ve also run into the minor issue that jail.conf defines “imap3” (version 0.9.6) which isn’t known to iptables. This results in errors in the log when postfix-sasl is enabled. There’s an existing issue on GitHub for that: https://github.com/fail2ban/fail2ban/issues/1942. As a workaround, you can simply replace “imap3” with “imap”.
I’ve added another filter to postfix-ispmail:
^%(__prefix_line)sdisconnect from \S+\[\].* auth=0/1.*$
Since Postfix 3.0.0:
“Per-session command profiles, logged at the end of each inbound SMTP session. For example, a password-guessing bot is logged as “disconnect from name[addr] ehlo=1 auth=0/1 commands=1/2″, meaning that the client sent one EHLO command that worked, one AUTH command that failed, and hung up without sending a QUIT command. This information is always logged, and can help to solve puzzles without verbose logging or network sniffers.”
So without this new filter, not all password-guessings were filtered. And I had LOTS of these entries in my logfile.
You didn’t mention the UFW Firewall anywhere or opening/closing ports on the machine or else where.
What are all the ports we have to open? And is it enough to open this for example with my VPS (AWS) or should i set them both in UFW and AWS?
Thank you for this awsome tutorial!
Hi, excellent guide!
I have a server running on debian wheezy from an earlier guide and still works like a charm.
I’ve been reading this new guide with the hope of replacing that old server, very few changes to be made but one thing I notice is the lack of user quotas. Is there a new way of doing that? or the old way having a column in the table virtual_users (quota_kb) is still the prefer way?
Thanks!
Hi, nice guide
Is it possible to do something about this
postfix/smtpd[28439]: connect from unknown[180.150.253.38]
postfix/smtpd[28439]: disconnect from unknown[180.150.253.38] helo=1 auth=0/1 quit=1 commands=2/3
Hello Christoph, thank you again for this beautiful tutorials!
I want to install a good iptables script but I don’t know from where to start.
Exist a good standard rules script to use?
I need also an help on how to implement it.
Thanks!
@Enrico: I suggest “shorewall”. Take a look at their examples and discover the beauty of simple firewall rules with few lines of readable text. 🙂
Thank you @Christoph for your quick answer! 🙂
Hello @Christoph, I have implemented “shorewall” as you suggested (very nice!).
I notice however that, when finished the configuration, the rules enabled by shorewall in your tutorial, does not appear anymore when I digit “iptables -nvL”.
However if I type “fail2ban-client status postfix-ispmail” the resulting message is like in your tutorial.
Is this correct? Are them always active in iptables?
Thank you again 🙂
Checking the fail2ban log, appear:
Couldn’t load target `f2b-postfix-ispmail’:No such file or directory\n\nTry `iptables -h’ or ‘iptables –help’ for more information.\niptables: No chain/target/match by that name.\niptables: No chain/target/match by that name.\n”
This message appears only if I start Shorewall
If I don’t start Shorewall the fail2ban rules will be loaded correctly.
Any suggestion?
Thanks
Missing link at the end of this section:
Next page:Troubleshooting your mail server
Thanks for the hint. Fixed.
I followed this tutorial to build out a new mail server here and it works wonderfully. Thanks for this. The last one I followed was the Squeeze tutorial many years ago, and it was still running fine up until recently when we decided to upgrade (we figured 12 years was long enough for a server in production!)
My issue is the following, we send out large batches of emails periodically (all subscription based, no UCE — on the order of 800k emails) and the first night we sent it, it went out blazingly fast (like 3k messages/sec) and nothing seemed to be amiss. The second batch we sent out seems to have been rate-limited to 20 messages/sec and it’s rather frustrating that I can’t figure out what is doing the rate limiting. I looked into the rspamd config and the postfix config, but nothing appears to be amiss. Anyone have any idea where I might look next to figure out what might be going on here? The emails are all coming from an internal IP address that has been added to an IP_Whitelist in rspamd (found some code online that seemed to work – https://gist.github.com/ThomasLeister/f41adad98bb46d0c8418de50b5efb4a0)
Thanks in advance!
Hi!
I recommend use fail2ban ban action with ipset. Default fail2ban add and remove iptable rules for each ban action and this is VERY slow. Use ipset, Luke!
Thanks for the hint. Yes, ipset is better for large amounts of blocked IPs. I’ll add that. However in my example I rarely have more than 20 or 30 blocked IPs at a given time.