In another chapter we already dealt with restrictions during the SMTP dialog. You learned about real-time black lists (RBLs) which help you block most of the incoming spam. If you are still getting too much spam and want to do more then content-scanning can help you. A content scanner looks into the actual email instead of just doing simple checks during the SMTP session. Unfortunately content-scanning with Postfix is rather complicated. However the software of choice here is AMaViS - short for "a mail virus scanner". It's a software written in Perl that uses additional software like SpamAssassin (for spam detection) and ClamAV (virus scanning) as well as many other virus scanners.
- can detect spam and block the email or mark it as spam
- can detect viruses and block infested emails
- supports automatic DKIM signing
- uses a lot of CPU (lowers your delivery rate by factor 5 to 10)
- spam scanning only works globally (an individual user cannot train the spam filter to match their habits)
- using content filters with Postfix works but is error prone
To understand how Postfix and AMaViS work together please take a look at the big picture again. When Postfix passes an email to AMaViS it uses SMTP on a non-standard port - 10024 by default. AMaViS scans the email and if it's successful it will then pass it back to Postfix on another port - 10025 by default.
Install AMaViS and its suggested packages:
apt-get install amavisd-new spamassassin clamav clamav-daemon arj zoo nomarch cpio lzop cabextract apt-listchanges libnet-ldap-perl libauthen-sasl-perl libdbi-perl libmail-dkim-perl p7zip rpm unrar-free libsnmp-perl
This setup is complex and can go wrong. So if you are adding content filtering to your live server I strongly suggest you enable the soft bounce feature in Postfix. If Postfix intends to reject or bounce an email then this feature will rather keep the mail in the queue and try again later. Whenever you do major changes in your mail server setup I recommend you enable it:
postconf -e soft_bounce=yes
As explained above AMaViS will accept emails on TCP/10024 and re-inject them to Postfix on TCP/10025. By default Postfix doesn't listen on port 10025. Neither does it know of AMaViS. So first you need to edit /etc/postfix/master.cf and add these two services:
smtp-amavis unix - - n - 2 smtp
127.0.0.1:10025 inet n - n - - smtpd
(Note that all lines but the first lines of each service need to be indented by spaces.)
Restart Postfix to make it pick up these two new services:
And you need to tell Postfix to use the "smtp-amavis" service as a content filter. Making Postfix forward emails to AMaViS is done by setting the content_filter setting. Also set the "receive_override_options" setting that will be explained later by running these shell commands:
postconf -e content_filter=smtp-amavis:[127.0.0.1]:10024
postconf -e receive_override_options=no_address_mappings
The first line is probably obvious. All email is forwarded to the "smtp-amavis" service defined in the /etc/postfix/master.cf using TCP port 10024 on the IP address 127.0.0.1 (localhost).
The second line is less obvious. But it's important or you risk delivering email duplicates to your users. The reason is that Postfix processes an incoming email twice. First when it comes from the internet. And second when it comes back from AMaViS after being content-scanned. Without disabling address mappings here the aliases (forwardings) would be checked twice. Of course an alias is supposed to be checked just once to avoid duplicate delivery. That's why you use this setting here.
AMaViS is configured using different files in /etc/amavis/conf.d. You may want to check what is set there. Above all you need to change the 15-content_filter_mode file to enable virus and spam scanning. Confusingly the default settings are:
#@bypass_virus_checks_maps = (
# \%bypass_virus_checks, \@bypass_virus_checks_acl, \$bypass_virus_checks_re);
#@bypass_spam_checks_maps = (
# \%bypass_spam_checks, \@bypass_spam_checks_acl, \$bypass_spam_checks_re);
To enable scanning you have to remove the "#" signs in front of these four lines. It sounds like it would then bypass the scanning but instead it enables it.
The default scanner is ClamAV which is a free and decent anti-virus software. To use it you will have to get a group membership right:
adduser clamav amavis
The configuration files in /etc/amavis/conf.d are run in sorted order. A setting in "50-user" will override the same setting in "01-debian". So I recommend you only change the existing (but empty) "50-user" file there. A couple of settings that you should consider:
If AMaViS believes that an email is spam then it will change the subject to begin with this string. By default that string is "***SPAM***" which I found annoying. If you do not want to alter the subject you should set it to an empty string like this:
Users can still find out if AMaViS flagged the email as spam by checking the X-Spam-Status header.
Emails with a spam detection score greater or equal to this level will get spam headers added. Until you are happy with your AMaViS configuration I suggest you set it to a very low score ("undef" is the smallest possible value) so that headers are always added. Recommendation:
Emails with a spam score greater or equal to this level will be marked as spam. Usually the default (6.31) is reasonable. Lower it if too much spam gets through. Raise it if you get too many regular mails caught as spam.
If an email scores this level then AMaViS will act on the email according to the $final_spam_destiny setting. It should be set to the same value as $sa_tag2_level_deflt
I do not like the default D_BOUNCE setting here. The usual approach with spam is to flag an email as spam by adding email headers. Your users can then use these headers to decide if they want to sort an email into an extra folder or discard it. But if you bounced them you would hit some innocent person because spam mails never contain the correct sender address. Just let the user decide what to do with spam. So it is strongly recommended that you set this to D_PASS:
There are a lot of further settings that you can use to tweak AMaViS to your own preferences.
Make sure that your 50-user file ends with "1;" or else AMaViS will not start up properly.
Restart AMaViS if you have made changes to the config files:
Make sure that AMaViS is listening on TCP port 10024:
netstat -nap | grep 10024
You should get this output:
tcp 0 0 127.0.0.1:10024 0.0.0.0:* LISTEN 12345/amavisd
If you get such a line then AMaViS is running and waiting for incoming SMTP sessions. Otherwise check your /var/log/mail.log file - perhaps you have made a mistake in the configuration files.
This should get you started. For a more detailed documentation please see the README.Postfix that provided by AMaViS.
Why filter outbound email?
In former tutorials I used SQL queries to determine whether an email is incoming or outgoing. Many system administrators do not want to scan outgoing email for spam because:
- your own newsletter could be sent to your customers flagged as spam
- you are wasting CPU resources
But nowadays I would rather recommend to allow scanning any email because
- it helps determine a virus infestation of a local PC
- it helps AMaViS learn both ham (good email) and spam (unwanted email)
Ultimately you decide. There are different approaches described in the AMaViS documentation.
Now that everything is set up you will want to test your spam scanning process. Fortunately SpamAssassin comes with both a ham (good) and spam (bad) email for you to test with. You can send these emails into your mail system and watch the /var/log/mail.log file to see if AMaViS is doing the right thing.
First let's try the spam email:
sendmail email@example.com < /usr/share/doc/spamassassin/examples/sample-spam.txt
I addition to many lines in your /var/log/mail.log file dealing with "postfix" you should see one line dealing with "amavis". It will look like this:
May 7 12:42:33 debian amavis: (01834-01) Passed SPAM, <root@...> -> <firstname.lastname@example.org>, quarantine: T/spam-TrRe46tTm2+r.gz, Message-ID: <GTUBE1.email@example.com>, mail_id: TrRe46tTm2+r, Hits: 1002.448, size: 944, queued_as: 22DC9D40, 5022 ms
This shows that the email was accepted ("Passed") but flagged as spam ("SPAM"). The spam score is 1002.448 ("Hits") which is way above the limit I have set as $sa_tag2_level_deflt. Also the email was copied to /var/lib/amavis/virusmails/T/spam-TrRe46tTm2+r.gz in GZIPped format. To learn how to deal with quarantined emails please see the AMaViS documentation.
Did you enable soft bounce earlier? Everything works? If the
command shows that there are still emails in the queue that need to be delivered then re-queue them first:
postsuper -r ALL
Re-scheduling means that Postfix reconsiders what to do with every email in the queue. Transport (routing) and content filtering information stick to an email. So even if you reconfigure Postfix then emails in the queue would not pick that up until you re-queue them.
Now flush the queue
and the emails should get delivered. The /var/log/mail.log will give you information on what happens.
If everything works as you expect then switch off soft bounce mode again and you are done:
postconf -e soft_bounce=no
If you seriously want to run AMaViS on a production mail server then please spend time with its documentation. It's a complex piece of software that can integrate a lot of third-party software like virus scanners. It can be tweaked a lot. It can automatically add DKIM signatures on outgoing email. It has a quarantine system so you can moderate spam emails. It just offers many additional features outside of the scope of this tutorial. If you believe you have a valuable addition to this tutorial then please send a comment on this page. Thank you.
Just one word on spam scanning. SpamAssassin (as being used by AMaViS) uses different rules to look for spam but can also learn to distinguish ham from spam emails. To do that properly it needs to scan an equal amount of several hundred ham and spam emails before it can really do its job. And as far as I know there is no way to learn spam per-user. So whatever you teach SpamAssassin is good or bad will apply for all your users.