Firewalling and brute force mitigation

So many bad guys everywhere. Paranoia you say? Well, take a look at your mail log. While I was preparing this guide I could barely follow it. The mail server I used for testing was by far not ready and I already got visits by a bunch of foreign IP addresses trying to guess accounts of email users. Scoundrels constantly scan the internet for mail servers and do not tire to find a way to abuse them. Let’s do something about that before they actually guess email accounts successfully.

First install the nftables package:

apt install nftables

Basic firewall rules with nftables

Let’s begin by blocking all incoming network ports except those that are needed to run your mail server. Previous versions of Debian used the iptables tool to help you add firewall rules. Debian Buster comes with a newer version of the Linux kernel that is moving from iptables to nftables. The good news is that you can still use iptables commands. The bad news is that most tools do not yet support nftables. However one distant future day the old syntax will not work any more so let’s dive straight into the new syntax and use nftables.

First let’s make the iptables command use nftables. Run:

update-alternatives --config iptables

You will be able to choose between iptables-nft (the new stuff) and iptables-legacy (the old stuff). Go with iptables-nft. Now the iptables command will manage nftables rules. However you can explicitly use the iptables-legacy command if you want to access the old-school iptables rules.

So where would you store the firewall rules? The recommended way is to create a file /etc/nftables.conf and add rules there. Although the new syntax takes some getting used to, it is at least pretty readable. This is my default file for new mail servers:

#!/usr/sbin/nft -f
flush ruleset
table inet filter {
  chain input {
    type filter hook input priority 0; policy drop;

    iifname lo accept
    ct state established,related accept
    tcp dport { ssh, http, https, imap2, imaps, pop3, pop3s, submission, smtp } ct state new accept

    # ICMP: errors, pings
    ip protocol icmp icmp type { echo-request, echo-reply, destination-unreachable, time-exceeded, parameter-problem, router-solicitation, router-advertisement } accept
    # ICMPv6: errors, pings, routing
    ip6 nexthdr icmpv6 counter accept comment "accept all ICMP types"

    # Reject other packets
    ip protocol tcp reject with tcp reset
  }
}

Bear with me that I will not explain this file in-depth. Feel free to read more about nftables in the Debian wiki and the wiki of the nftables project.

To active these firewall rules use systemctl:

systemctl enable nftables
systemctl start nftables

If all worked as expected you can see that systemd loaded your rules:

systemctl status nftables

You should see “Active: active (exited)” as the status. It is normal that its status is “exited” because there is no permanent process running. The rules are applied just once and that’s it.

Although nftables has been there for quite a while many projects still do not support it. Fortunately an important tool to block off attackers does: fail2ban. The “#include” line shown in the file above refers to that.

Brute-force attacks

Let’s install fail2ban:

apt install fail2ban

This is a pretty little tool. It knows about a lot of patterns in your log files that show something harmful. The software runs in the background and follows the various log files. Each new line in the log is compared against regular expressions. If a match is found it blocks the IP address that was mentioned in the log. However by default fail2ban uses the deprecated iptables tool. Let’s make it use nftables, too.

Once again edit your /etc/nftables.conf file and at the end add this line:

include "/etc/nftables/fail2ban.conf"

Create that /etc/nftables directory…

mkdir /etc/nftables

…and in there create the fail2ban.conf file containing these lines:

#!/usr/sbin/nft -f

table inet fail2ban {
  chain input {
    type filter hook input priority 100;
  }
}

Restart nftables…

systemctl restart nftables

…and you should see a new table called fail2ban if you run:

nft list tables

Very nice. fail2ban just needs a little fine-tuning to use that table. Create a new file /etc/fail2ban/action.d/nftables-common.local that overrides the settings in the nftables-common.conf file and write:

[Init]
nftables_table = fail2ban
blocktype = drop

What it means…

• nftables_table = fail2ban
  (That is the name of the nftables table you just created. There fail2ban will add IP addresses to be blocked.)
• blocktype = drop
  (This way we just ignore attackers. Packets from them are thrown away instead of answered with an ICMP response.)

Create another file /etc/fail2ban/jail.local with…

[DEFAULT]
banaction = nftables-multiport
chain     = input

If you are curious and would like to get informed about every attacker that was blocked you may add “action = %(action_mwl)s” that also gets WHOIS information about the attacker. However this gets boring quickly and often WHOIS information is censored anyway – especially from organisations that are home to spammers.

By default an IP address gets blocked for 10 minutes (bantime) if more than 5 (maxretry) failures happened within 10 minutes (findtime). Check the jail.conf file for more information and feel free to add your preferred values to your jail.local file.

Also by default only failed login attempts using SSH are detected. There are lots of filters for other events and you just need to enable them. This once again goes into your jail.local file. I suggest you add these lines:

[apache-auth]
enabled = true

[dovecot]
enabled = true
port    = pop3,pop3s,imap2,imaps,submission,465,sieve

[postfix]
enabled = true

After all these changes it is a good idea to check if your fail2ban configuration is still valid:

fail2ban-server -t

Is fail2ban happy? Great. Restart fail2ban after your last change:

systemctl restart fail2ban

To see what fail2ban did take a look at /var/log/fail2ban.log. This is what it looked like after a few minutes:

fail2ban.filter  : INFO    [sshd] Found 177.139.194.62 - 2019-12-27 20:50:43
 fail2ban.filter  : INFO    [sshd] Found 177.139.194.62 - 2019-12-27 20:50:43
 fail2ban.filter  : INFO    [sshd] Found 177.139.194.62 - 2019-12-27 20:50:44
 fail2ban.filter  : INFO    [sshd] Found 49.88.112.112 - 2019-12-27 20:50:46
 fail2ban.filter  : INFO    [sshd] Found 134.209.152.176 - 2019-12-27 20:50:46
 fail2ban.filter  : INFO    [sshd] Found 49.88.112.112 - 2019-12-27 20:50:48
 fail2ban.filter  : INFO    [sshd] Found 134.209.152.176 - 2019-12-27 20:50:49
 fail2ban.filter  : INFO    [sshd] Found 49.88.112.112 - 2019-12-27 20:50:50
 fail2ban.filter  : INFO    [sshd] Found 49.88.112.112 - 2019-12-27 20:50:54
 fail2ban.filter  : INFO    [dovecot] Found 85.25.72.76 - 2019-12-27 20:51:09
 fail2ban.filter  : INFO    [dovecot] Found 85.25.72.76 - 2019-12-27 20:51:29
 fail2ban.filter  : INFO    [sshd] Found 61.72.255.26 - 2019-12-27 20:51:33
 fail2ban.filter  : INFO    [sshd] Found 49.88.112.112 - 2019-12-27 20:51:35
 fail2ban.filter  : INFO    [sshd] Found 61.72.255.26 - 2019-12-27 20:51:35
 fail2ban.actions : NOTICE  [sshd] Ban 49.88.112.112
 fail2ban.filter  : INFO    [sshd] Found 49.88.112.112 - 2019-12-27 20:51:37
 fail2ban.filter  : INFO    [dovecot] Found 85.25.72.76 - 2019-12-27 20:51:42
 fail2ban.filter  : INFO    [dovecot] Found 85.25.72.76 - 2019-12-27 20:52:03
 fail2ban.filter  : INFO    [dovecot] Found 85.25.72.76 - 2019-12-27 20:52:18
 fail2ban.actions : NOTICE  [dovecot] Ban 85.25.72.76

Status please

fail2ban comes with a handy shell command called fail2ban-client. To see which jails (=where to look for attackers) are active:

$> fail2ban-client status
Status
|- Number of jail:    4
`- Jail list:    apache-auth, dovecot, postfix, sshd

So all four expected jails are active. Do you want to know which IP addresses are banned for failed SSH login attempts?

$> fail2ban-client status sshd
Status for the jail: sshd
|- Filter
|  |- Currently failed:    3
|  |- Total failed:    13
|  - File list:    /var/log/auth.log - Actions
   |- Currently banned:    4
   |- Total banned:    5
   `- Banned IP list:    222.186.175.148 222.186.190.2 49.88.112.116 2a03:4000:1c:3f0::1

Yes, it works for IPv6, too. 🙂

Banning magic

Do you wonder how that banning actually works? fail2ban just adds new rules to your host’s firewall. Take a look at the fail2ban table you created earlier:

$> nft list table inet fail2ban
table inet fail2ban {
  set f2b-sshd {
    type ipv4_addr
    elements = { 49.88.112.112, 49.88.112.116, 222.186.175.148 }
  }

  set f2b-dovecot {
    type ipv4_addr
  }

  set f2b-sshd6 {
    type ipv6_addr
    elements = { 2a03:4000:1c:3f0::1 }
  }

  chain input { … }
}

nftables is pretty efficient with handling multiple IP addresses. It uses sets which are basically lists of IP addresses. iptables used to have the ipset functionality for that purpose. You can see in the output that it uses sets it names “elements”.

Look at the current status of a jail:

fail2ban-client status postfix-ispmail

It will show something like:

Status for the jail: postfix-ispmail
|- Filter
| |- Currently failed: 1
| |- Total failed: 168
| `- File list: /var/log/mail.log
`- Actions
 |- Currently banned: 14
 |- Total banned: 36
 `- Banned IP list: 1.199.191.44 125.106.250.51 115.213.233.184 …

Grab a bag of popcorn and enjoy how your mail server grows its list of blocked idiots. 🙂