[2019-12-26]

Firewalling and brute force mitigation

So many bad guys everywhere. Paranoia you say? Well, take a look at your mail log. While I was preparing this guide I could barely follow it. The mail server I used for testing was by far not ready and I already got visits by a bunch of foreign IP addresses trying to guess accounts of email users. Scoundrels constantly scan the internet for mail servers and do not tire to find a way to abuse them. Let’s do something about that before they actually guess email accounts successfully.

First install the nftables package:

apt install nftables

Basic firewall rules with nftables

Let’s begin by blocking all incoming network ports except those that are needed to run your mail server. Previous versions of Debian used the iptables tool to help you add firewall rules. Debian Buster comes with a newer version of the Linux kernel that is moving from iptables to nftables. The good news is that you can still use iptables commands. The bad news is that most tools do not yet support nftables. However one distant future day the old syntax will not work any more so let’s dive straight into the new syntax and use nftables.

First let’s make the iptables command use nftables. Run:

update-alternatives --config iptables

You will be able to choose between iptables-nft (the new stuff) and iptables-legacy (the old stuff). Go with iptables-nft. Now the iptables command will manage nftables rules. However you can explicitly use the iptables-legacy command if you want to access the old-school iptables rules.

So where would you store the firewall rules? The recommended way is to create a file /etc/nftables.conf and add rules there. Although the new syntax takes some getting used to, it is at least pretty readable. This is my default file for new mail servers:

#!/usr/sbin/nft -f
flush ruleset
table inet filter {
  chain input {
    type filter hook input priority 0; policy drop;

    iifname lo accept
    ct state established,related accept
    tcp dport ssh ct state new accept
    tcp dport http ct state new accept
    tcp dport https ct state new accept
    tcp dport imap2 ct state new accept
    tcp dport imaps ct state new accept
    tcp dport pop3 ct state new accept
    tcp dport pop3s ct state new accept
    tcp dport submission ct state new accept
    tcp dport smtp ct state new accept

     # ICMP: errors, pings
     ip protocol icmp icmp type { echo-request, echo-reply, destination-unreachable, time-exceeded, parameter-problem, router-solicitation, router-advertisement } accept
     # ICMPv6: errors, pings, routing
     ip6 nexthdr icmpv6 counter accept comment "accept all ICMP types"

     # Reject other packets
     ip protocol tcp reject with tcp reset
  }
}

Bear with me that I will not explain this file in-depth. Feel free to read more about nftables in the Debian wiki and the wiki of the nftables project.

To active these firewall rules use systemctl:

systemctl enable nftables
systemctl start nftables

If all worked as expected you can see that systemd loaded your rules:

systemctl status nftables

You should see “Active: active (exited)” as the status. It is normal that its status is “exited” because there is no permanent process running. The rules are applied just once and that’s it.

Although nftables has been there for quite a while many projects still do not support it. Fortunately an important tool to block off attackers does: fail2ban. The “#include” line shown in the file above refers to that.

Brute-force attacks

Let’s install fail2ban:

apt install fail2ban

This is a pretty little tool. It knows about a lot of patterns in your log files that show something harmful. The software runs in the background and follows the various log files. Each new line in the log is compared against regular expressions. If a match is found it blocks the IP address that was mentioned in the log. However by default fail2ban uses the deprecated iptables tool. Let’s make it use nftables, too.

Once again edit your /etc/nftables.conf file and at the end add this line:

include "/etc/nftables/fail2ban.conf"

Create that /etc/nftables directory…

mkdir /etc/nftables

…and in there create the fail2ban.conf file containing these lines:

#!/usr/sbin/nft -f

table inet fail2ban {
  chain input {
    type filter hook input priority 100;
  }
}

Why not ‘table ip’?

As the firewall just needs to block IP addresses you might be tempted to use ‘table ip’ instead of ‘table inet’. However ‘ip’ only deals with IPv4 packets. You would have to manage ‘ip’ and ‘ipv6’ tables. So I am just going with ‘table inet’ here which is more versatile.

Restart nftables

systemctl restart nftables

…and you should see a new table called fail2ban if you run:

nft list tables

Very nice. fail2ban just needs a little fine-tuning to use that table. Create a new file /etc/fail2ban/action.d/nftables-common.local that overrides the settings in the nftables-common.conf file and write:

[Init]
nftables_table = fail2ban
blocktype = drop

What it means…

  • nftables_table = fail2ban
    (That is the name of the nftables table you just created. There fail2ban will add IP addresses to be blocked.)
  • blocktype = drop
    (This way we just ignore attackers. Packets from them are thrown away instead of answered with an ICMP response.)

Create another file /etc/fail2ban/jail.local with…

[DEFAULT]
banaction = nftables-multiport
chain     = input

If you are curious and would like to get informed about every attacker that was blocked you may add “action = %(action_mwl)s” that also gets WHOIS information about the attacker. However this gets boring quickly and often WHOIS information is censored anyway – especially from organisations that are home to spammers.

By default an IP address gets blocked for 10 minutes (bantime) if more than 5 (maxretry) failures happened within 10 minutes (findtime). Check the jail.conf file for more information and feel free to add your preferred values to your jail.local file.

Also by default only failed login attempts using SSH are detected. There are lots of filters for other events and you just need to enable them. This once again goes into your jail.local file. I suggest you add these lines:

[apache-auth]
enabled = true

[dovecot]
enabled = true
port    = pop3,pop3s,imap2,imaps,submission,465,sieve

[postfix]
enabled = true

Why the port definition for [dovecot]?

You may wonder why in the example above I suggest defining port in the dovecot section. The reason is Debian bug #867374. The default configuration calls the service “imap” instead of “imap2” and that leads to an error because there is no “imap” service defined in /etc/services.

After all these changes it is a good idea to check if your fail2ban configuration is still valid:

fail2ban-server -t

Is fail2ban happy? Great. Restart fail2ban after your last change:

systemctl restart fail2ban

To see what fail2ban did take a look at /var/log/fail2ban.log. This is what it looked like after a few minutes:

fail2ban.filter  : INFO    [sshd] Found 177.139.194.62 - 2019-12-27 20:50:43
 fail2ban.filter  : INFO    [sshd] Found 177.139.194.62 - 2019-12-27 20:50:43
 fail2ban.filter  : INFO    [sshd] Found 177.139.194.62 - 2019-12-27 20:50:44
 fail2ban.filter  : INFO    [sshd] Found 49.88.112.112 - 2019-12-27 20:50:46
 fail2ban.filter  : INFO    [sshd] Found 134.209.152.176 - 2019-12-27 20:50:46
 fail2ban.filter  : INFO    [sshd] Found 49.88.112.112 - 2019-12-27 20:50:48
 fail2ban.filter  : INFO    [sshd] Found 134.209.152.176 - 2019-12-27 20:50:49
 fail2ban.filter  : INFO    [sshd] Found 49.88.112.112 - 2019-12-27 20:50:50
 fail2ban.filter  : INFO    [sshd] Found 49.88.112.112 - 2019-12-27 20:50:54
 fail2ban.filter  : INFO    [dovecot] Found 85.25.72.76 - 2019-12-27 20:51:09
 fail2ban.filter  : INFO    [dovecot] Found 85.25.72.76 - 2019-12-27 20:51:29
 fail2ban.filter  : INFO    [sshd] Found 61.72.255.26 - 2019-12-27 20:51:33
 fail2ban.filter  : INFO    [sshd] Found 49.88.112.112 - 2019-12-27 20:51:35
 fail2ban.filter  : INFO    [sshd] Found 61.72.255.26 - 2019-12-27 20:51:35
 fail2ban.actions : NOTICE  [sshd] Ban 49.88.112.112
 fail2ban.filter  : INFO    [sshd] Found 49.88.112.112 - 2019-12-27 20:51:37
 fail2ban.filter  : INFO    [dovecot] Found 85.25.72.76 - 2019-12-27 20:51:42
 fail2ban.filter  : INFO    [dovecot] Found 85.25.72.76 - 2019-12-27 20:52:03
 fail2ban.filter  : INFO    [dovecot] Found 85.25.72.76 - 2019-12-27 20:52:18
 fail2ban.actions : NOTICE  [dovecot] Ban 85.25.72.76

Status please

fail2ban comes with a handy shell command called fail2ban-client. To see which jails (=where to look for attackers) are active:

$> fail2ban-client status
Status
|- Number of jail:    4
`- Jail list:    apache-auth, dovecot, postfix, sshd

So all four expected jails are active. Do you want to know which IP addresses are banned for failed SSH login attempts?

$> fail2ban-client status sshd
Status for the jail: sshd
|- Filter
|  |- Currently failed:    3
|  |- Total failed:    13
|  - File list:    /var/log/auth.log - Actions
   |- Currently banned:    4
   |- Total banned:    5
   `- Banned IP list:    222.186.175.148 222.186.190.2 49.88.112.116 2a03:4000:1c:3f0::1

Yes, it works for IPv6, too. 🙂

Banning magic

Do you wonder how that banning actually works? fail2ban just adds new rules to your host’s firewall. Take a look at the fail2ban table you created earlier:

$> nft list table inet fail2ban
table inet fail2ban {
  set f2b-sshd {
    type ipv4_addr
    elements = { 49.88.112.112, 49.88.112.116, 222.186.175.148 }
  }

  set f2b-dovecot {
    type ipv4_addr
  }

  set f2b-sshd6 {
    type ipv6_addr
    elements = { 2a03:4000:1c:3f0::1 }
  }

  chain input { … }
}

nftables is pretty efficient with handling multiple IP addresses. It uses sets which are basically lists of IP addresses. iptables used to have the ipset functionality for that purpose. You can see in the output that it uses sets it names “elements”.

Look at the current status of a jail:

fail2ban-client status postfix-ispmail

It will show something like:

Status for the jail: postfix-ispmail
|- Filter
| |- Currently failed: 1
| |- Total failed: 168
| `- File list: /var/log/mail.log
`- Actions
 |- Currently banned: 14
 |- Total banned: 36
 `- Banned IP list: 1.199.191.44 125.106.250.51 115.213.233.184 …

Grab a bag of popcorn and enjoy how your mail server grows its list of blocked idiots. 🙂

11 thoughts on “Firewalling and brute force mitigation

  • 2019-12-31 at 10:40
    Permalink

    nftables should be installed before systemctl enable nftables.service

    Reply
  • 2019-12-31 at 15:03
    Permalink

    It’s not clear what “and you should see a new table called fail2ban like this” means – how to see the table? 🙂

    For fail2ban, there is no “apache” ruleset. The configuration listed yields

    Found no accessible config files for ‘filter.d/apache’ under /etc/fail2ban
    Unable to read the filter ‘apache’
    Errors in jail ‘apache’.
    ERROR: test configuration failed

    Looking into /etc/fail2ban/jail.conf shows several Apache-related configurations, like apache-auth (which seems to be referred later on this page too).

    Reply
    • 2019-12-31 at 16:25
      Permalink

      Indeed. Fixed. And suprisingly on my test system I used “apache-auth”. Must have been a copy/paste error.

      Reply
  • 2019-12-31 at 16:59
    Permalink

    You have a typo on the shebang for /etc/nftables.conf, missing the “#” char

    Reply
    • 2020-01-01 at 11:25
      Permalink

      Dammit. WordPress keeps eating my # chars. 🙁

      Reply
  • 2020-01-09 at 00:23
    Permalink

    sorry, for me the nftables is completely new, someone can help me, after installing and configuring it closes my ssh port I can not enter, likewise close my bind9 dns ports since on that server I have the dns.

    Where do I add those rules? can you help me, please

    Reply
  • 2020-01-11 at 21:25
    Permalink

    Is it worth to set longer bantime and detection time ?

    Reply
    • 2020-01-14 at 11:56
      Permalink

      I have it set even stricter and am waiting until my users complain. 🙂

      Reply
  • 2020-04-19 at 05:27
    Permalink

    Hi Christoph,

    yeah, almost through your tutorial — thanks a lot at this point 🙂
    I noticed the listing of chain f2b-postfix-ispmail, which reminds me of iptables syntax. Since nftables are used in this tutorial, might these lines be a leftover of a previous version?

    Cheers
    Gabe

    Reply
    • 2020-04-19 at 14:29
      Permalink

      Hi Gabe. I hope you get through it this weekend. It feels good once everything is set up and running. 🙂
      Thanks for the hint of the iptables output. I have removed it.

      Reply

Leave a Reply to Azazel Mendoza Cancel reply

Your email address will not be published. Required fields are marked *