Fighting brute force attacks

So many bad guys everywhere. Paranoia you say? Well, take a look at your mail log. While I was preparing this guide I could barely follow it. The mail server I used for testing was by far not ready and I already got visits by a bunch of chinese IP addresses trying to guess accounts of email users. Scoundrels constantly scan the internet for mail servers and do not tire to find a way to abuse them. Let’s do something about that before they actually guess email accounts successfully.

Attack patterns

There was one pattern that caught my eye:

Jan 24 23:59:34 jen postfix/smtpd[31879]: lost connection after AUTH from unknown[122.231.43.242]
Jan 24 23:59:35 jen postfix/smtpd[31879]: lost connection after AUTH from unknown[122.231.43.242]
Jan 24 23:59:36 jen postfix/smtpd[31879]: lost connection after AUTH from unknown[122.231.43.242]
Jan 24 23:59:37 jen postfix/smtpd[31879]: lost connection after AUTH from unknown[122.231.43.242]
Jan 24 23:59:38 jen postfix/smtpd[31879]: lost connection after AUTH from unknown[122.231.43.242]

Some chinese system is trying to authenticate against my Postfix time and again. And believe it or not – it was doing that for days. My log file grew up to 500 MB per day just from those dorks. Let’s put an end to that.

Installation

A standard piece of software I install on servers for that purpose is fail2ban. Go put it on your mail server:

apt install fail2ban

It knows about a lot of patterns in your log files that report something harmful. The software runs in the background and follows the various log files. Each new line in the log is compared against regular expressions. If a match is found it blocks the IP address that was mentioned in the log.

Adding a filter

As fail2ban does not know that “lost connection after AUTH” is a failure we need to add that pattern. Create a new file /etc/fail2ban/filter.d/postfix-ispmail.conf with this content:

[INCLUDES]
before = common.conf

[Definition]
_daemon = postfix(-\w+)?/(?:submission/|smtps/)?smtp[ds]
failregex = ^%(__prefix_line)slost connection after AUTH from \S+\[<HOST>\]$
ignoreregex =

[Init]
journalmatch = _SYSTEMD_UNIT=postfix.service

This defines a new filter called “postfix-ispmail”. The name is derived from the filename of the filter. The important line is the failregex. It described a regular expression – a machine-readable description of what are you searching for. %(…)s expands a variable. And __prefix_line is a pattern describing the left part of each log line: “Jan 25 00:32:01 jen postfix/smtpd[32600]: “. The “postfix/smtpd” part is defined by the _daemon variable. “\S+” means that at this place you expect one or more character that are not a space. The brackets need to be escaped by a backslash because they have a special meaning in regular expressions. And <HOST> is a built-in pattern of fail2ban that stands for an IP address. You should get yourself acquainted with regular expressions even if they may scare you at first. They are more powerful than scary actually. 🙂

Adding a jail

fail2ban can now understand the pattern we are looking for. But where do we look? And what do we do if we find a match? That’s our next task. Create another file /etc/fail2ban/jail.d/postfix-ispmail.conf with this content:

[postfix-ispmail]
logpath = %(syslog_mail)s
enabled = true
port = smtp,submission

The first line refers to the name of the filter. We called the filter file “postfix-ispmail.conf” so this is the filter name (the “.conf” is ignored). The log file we want to follow is defined in the syslog_mail variable which gets defined elsewhere (in the paths-debian.conf file). It points to /var/log/mail.log. We want to have this jail enabled because the default in /etc/fail2ban/jail.conf is having it disabled. And finally we define which network ports we want to block in order to block an attacker.

Make it so

Time for action. Restart fail2ban to load your configuration changes:

service fail2ban restart

To see what fail2ban did take a look at /var/log/fail2ban.log. In my example it logged:

2018-01-24 23:59:34,443 fail2ban.filter [32003]: INFO [postfix-ispmail] Found 122.231.43.242
2018-01-24 23:59:35,430 fail2ban.filter [32003]: INFO [postfix-ispmail] Found 122.231.43.242
2018-01-24 23:59:36,434 fail2ban.filter [32003]: INFO [postfix-ispmail] Found 122.231.43.242
2018-01-24 23:59:37,427 fail2ban.filter [32003]: INFO [postfix-ispmail] Found 122.231.43.242
2018-01-24 23:59:38,413 fail2ban.filter [32003]: INFO [postfix-ispmail] Found 122.231.43.242
2018-01-24 23:59:38,781 fail2ban.actions [32003]: NOTICE [postfix-ispmail] Ban 122.231.43.242

It found the “lost connection after AUTH” line mentioning that chinese IP address five times and decided to ban it. The reason is part of the configuration in /etc/fail2ban/jail.conf:

# “bantime” is the number of seconds that a host is banned.
bantime = 600

# A host is banned if it has generated “maxretry” during the last “findtime”
# seconds.
findtime = 600

# “maxretry” is the number of failures before a host get banned.
maxretry = 5

So by default if such a rogue pattern occurs at least maxretry times within findtime seconds the IP will be banned for bantime seconds. You can customize these values but please do not edit the jail.conf. Instead create a jail.local file as described in “man 5 jail.conf”.

Banning magic

Do you wonder how that banning actually works? fail2ban adds new rules to your host’s firewall using iptables. The list of current iptables rules can be viewed using…

iptables -nvL

It will show your firewall rules. fail2ban creates a new chain for each jail you configure. I call my jail “ispmail-postfix” so my rules look like:

Chain f2b-postfix-ispmail (1 references)
 pkts bytes target prot opt in out source destination 
    3 152   REJECT all  --  *  *   125.106.22.171 …
   22 972   REJECT all  --  *  *   115.213.146.75 …
    3 152   REJECT all  --  *  *   1.195.252.154 …
  837 39050 RETURN all  --  *  *   0.0.0.0/0 0.0.0.0/0

But there is a nicer way to see the current status of a jail:

fail2ban-client status postfix-ispmail

It will show something like:

Status for the jail: postfix-ispmail
|- Filter
| |- Currently failed: 1
| |- Total failed: 168
| `- File list: /var/log/mail.log
`- Actions
 |- Currently banned: 14
 |- Total banned: 36
 `- Banned IP list: 1.199.191.44 125.106.250.51 115.213.233.184 …

More jails

You may want to browse through the pre-defined filters in either /etc/fail2ban/jail.conf or by looking at the files in /etc/fail2ban/filter.d/. By default only the “sshd” filter is enabled by the /etc/fail2ban/jail.d/defaults-debian.conf file. But feel free to add further files enabling…

  • dovecot
  • postfix
  • postfix-sasl
  • roundcube-auth

That will put an end to brute force attacks once and for all. Fun fact… two days after writing this I had already banned 1645 IP addresses.

(If we come up with more patterns that fail2ban doesn’t know yet I will add them. Did you find any?)

IPv6 support

Many ISPs support IPv6 nowadays. Unfortunately the version of fail2ban in Debian Stretch does not. IPv6 support has been added in version 0.10 but you get version 0.9.6 by default. There is no official backport of the new version yet. But I can at least offer you a quick backport so that you can use version 0.10 on Debian Stretch.

Download fail2ban_0.10.2-1_all.deb

Install it using…

dpkg -i fail2ban_0.10.2-1_all.deb

…and make sure any missing dependencies are installed…

apt -f install

Now fail2ban will be able to block IPv6 addresses.

11 thoughts on “Fighting brute force attacks

  • 2018-01-25 at 13:56
    Permalink

    Hi,

    Thanks for you great tutorial, it helped me a lot!

    Maybe it would be an idea to add the detected spam to the postfix fail2ban jail?

    I am not fit enough in fail2ban configuration, but I think with matching for the line

    Jan 25 07:14:08 srv5 postfix/cleanup[4939]: 5877618800AB: milter-reject: END-OF-MESSAGE from q212.mkzz.xyz[103.203.148.212]: 4.7.1 Spam message rejected; from= to= proto=ESMTP helo=

    This should work, shouldn’t it?

    Tanks and best regards
    Christof

    Reply
  • 2018-01-26 at 13:48
    Permalink

    Note: the filter is named roundcube-auth for me on Stretch instead of roundcube.

    Reply
    • 2018-01-26 at 13:54
      Permalink

      Thanks. Fixed.

      Reply
  • 2018-02-04 at 12:27
    Permalink

    Hi

    Any ideas why my fail2ban is missing the “paths-debian.conf”?
    I mean I fixed it, but why/how? What am I missing?

    Reply
      • 2018-02-10 at 11:58
        Permalink

        Okay… just to close this, it was my mistake. I screwed the config files up while migrating from an old system. Everything is like it should now….

        Reply
  • 2018-02-19 at 19:42
    Permalink

    Very good guide, could install my mailserver within half a day.

    ipv6 is not blocked with version 0.9.6 of fail2ban which is currently delivered with debian stretch.
    we have to wait until it is available via apt or install it manually.

    i have a ipv6 enabled server and need failban for ipv6, too.

    kind regards

    t.

    Reply
    • 2018-03-09 at 17:34
      Permalink

      Thanks for the hint. I had that on my TODO list as my new server has an IPv6 address and of course I am using. As 0.10.x has not made it into Debian Stretch I did a quick backport of the package. See the last section on this page.

      Reply
  • 2018-03-11 at 09:56
    Permalink

    It may be helpful to point out that in order to activate existing jails, all you have to do is add a “/etc/fail2ban/jail.d/${service-name}.conf” file that contains

    [${service-name}]
    enabled = true

    You don’t have to copy the complete configuration from “/etc/fail2ban/jail.conf”. May be obvious to some, but it wasn’t 100% clear to me right away.

    It may also be better to move the custom filter instructions below the general description on how to set up fail2ban. It confused me for some time because I wasn’t sure how much of that I would have to replicate to enable f2b for those other services, i.e. “installation, explanation, more jails, custom filters”

    I’ve also run into the minor issue that jail.conf defines “imap3” (version 0.9.6) which isn’t known to iptables. This results in errors in the log when postfix-sasl is enabled. There’s an existing issue on GitHub for that: https://github.com/fail2ban/fail2ban/issues/1942. As a workaround, you can simply replace “imap3” with “imap”.

    Reply
  • 2018-04-22 at 18:53
    Permalink

    I’ve added another filter to postfix-ispmail:
    ^%(__prefix_line)sdisconnect from \S+\[\].* auth=0/1.*$

    Since Postfix 3.0.0:
    “Per-session command profiles, logged at the end of each inbound SMTP session. For example, a password-guessing bot is logged as “disconnect from name[addr] ehlo=1 auth=0/1 commands=1/2″, meaning that the client sent one EHLO command that worked, one AUTH command that failed, and hung up without sending a QUIT command. This information is always logged, and can help to solve puzzles without verbose logging or network sniffers.”

    So without this new filter, not all password-guessings were filtered. And I had LOTS of these entries in my logfile.

    Reply
  • 2018-05-07 at 11:24
    Permalink

    You didn’t mention the UFW Firewall anywhere or opening/closing ports on the machine or else where.
    What are all the ports we have to open? And is it enough to open this for example with my VPS (AWS) or should i set them both in UFW and AWS?

    Thank you for this awsome tutorial!

    Reply

Leave a Reply

Your email address will not be published. Required fields are marked *