In another chapter we already dealt with restrictions during the SMTP dialog. You learned about real-time black lists (RBLs) which help you block most of the incoming spam. If you are still getting too much spam and want to do more then content-scanning can help you. A content scanner looks into the actual email instead of just doing simple checks during the SMTP session. Unfortunately content-scanning with Postfix is rather complicated. However the software of choice here is AMaViS - short for "a mail virus scanner". It's a software written in Perl that uses additional software like SpamAssassin (for spam detection) and ClamAV (virus scanning) as well as many other virus scanners.
Advantages:
- can detect spam and block the email or mark it as spam
- can detect viruses and block infested emails
- supports automatic DKIM signing
Disadvantages:
- uses a lot of CPU (lowers your delivery rate by factor 5 to 10)
- spam scanning only works globally (an individual user cannot train the spam filter to match their habits)
- using content filters with Postfix works but is error prone
To understand how Postfix and AMaViS work together please take a look at the big picture again. When Postfix passes an email to AMaViS it uses SMTP on a non-standard port - 10024 by default. AMaViS scans the email and if it's successful it will then pass it back to Postfix on another port - 10025 by default.
Installation
Install AMaViS and its suggested packages:
apt-get install amavisd-new spamassassin clamav clamav-daemon arj zoo nomarch cpio lzop cabextract apt-listchanges libnet-ldap-perl libauthen-sasl-perl libdbi-perl libmail-dkim-perl p7zip rpm unrar-free libsnmp-perl
This setup is complex and can go wrong. So if you are adding content filtering to your live server I strongly suggest you enable the soft bounce feature in Postfix. If Postfix intends to reject or bounce an email then this feature will rather keep the mail in the queue and try again later. Whenever you do major changes in your mail server setup I recommend you enable it:
postconf -e soft_bounce=yes
As explained above AMaViS will accept emails on TCP/10024 and re-inject them to Postfix on TCP/10025. By default Postfix doesn't listen on port 10025. Neither does it know of AMaViS. So first you need to edit /etc/postfix/master.cf and add these two services:
smtp-amavis unix - - n - 2 smtp
-o smtp_data_done_timeout=1200
-o smtp_send_xforward_command=yes
-o disable_dns_lookups=yes
-o max_use=20
127.0.0.1:10025 inet n - n - - smtpd
-o content_filter=
-o smtpd_delay_reject=no
-o smtpd_client_restrictions=permit_mynetworks,reject
-o smtpd_helo_restrictions=
-o smtpd_sender_restrictions=
-o smtpd_recipient_restrictions=permit_mynetworks,reject
-o smtpd_data_restrictions=reject_unauth_pipelining
-o smtpd_end_of_data_restrictions=
-o smtpd_restriction_classes=
-o mynetworks=127.0.0.0/8
-o smtpd_error_sleep_time=0
-o smtpd_soft_error_limit=1001
-o smtpd_hard_error_limit=1000
-o smtpd_client_connection_count_limit=0
-o smtpd_client_connection_rate_limit=0
-o receive_override_options=no_header_body_checks,no_unknown_recipient_checks,no_milters
-o local_header_rewrite_clients=
(Note that all lines but the first lines of each service need to be indented by spaces.)
Restart Postfix to make it pick up these two new services:
postfix reload
And you need to tell Postfix to use the "smtp-amavis" service as a content filter. Making Postfix forward emails to AMaViS is done by setting the content_filter setting. Also set the "receive_override_options" setting that will be explained later by running these shell commands:
postconf -e content_filter=smtp-amavis:[127.0.0.1]:10024
postconf -e receive_override_options=no_address_mappings
The first line is probably obvious. All email is forwarded to the "smtp-amavis" service defined in the /etc/postfix/master.cf using TCP port 10024 on the IP address 127.0.0.1 (localhost).
The second line is less obvious. But it's important or you risk delivering email duplicates to your users. The reason is that Postfix processes an incoming email twice. First when it comes from the internet. And second when it comes back from AMaViS after being content-scanned. Without disabling address mappings here the aliases (forwardings) would be checked twice. Of course an alias is supposed to be checked just once to avoid duplicate delivery. That's why you use this setting here.
Configuring AMaViS
AMaViS is configured using different files in /etc/amavis/conf.d. You may want to check what is set there. Above all you need to change the 15-content_filter_mode file to enable virus and spam scanning. Confusingly the default settings are:
#@bypass_virus_checks_maps = (
# \%bypass_virus_checks, \@bypass_virus_checks_acl, \$bypass_virus_checks_re);#@bypass_spam_checks_maps = (
# \%bypass_spam_checks, \@bypass_spam_checks_acl, \$bypass_spam_checks_re);
To enable scanning you have to remove the "#" signs in front of these four lines. It sounds like it would then bypass the scanning but instead it enables it.
The default scanner is ClamAV which is a free and decent anti-virus software. To use it you will have to get a group membership right:
adduser clamav amavis
/etc/init.d/clamav-daemon restart
The configuration files in /etc/amavis/conf.d are run in sorted order. A setting in "50-user" will override the same setting in "01-debian". So I recommend you only change the existing (but empty) "50-user" file there. A couple of settings that you should consider:
$sa_spam_subject_tag
If AMaViS believes that an email is spam then it will change the subject to begin with this string. By default that string is "***SPAM***" which I found annoying. If you do not want to alter the subject you should set it to an empty string like this:
$sa_spam_subject_tag=undef;
Users can still find out if AMaViS flagged the email as spam by checking the X-Spam-Status header.
$sa_tag_level_deflt
Emails with a spam detection score greater or equal to this level will get spam headers added. Until you are happy with your AMaViS configuration I suggest you set it to a very low score ("undef" is the smallest possible value) so that headers are always added. Recommendation:
$sa_tag_level_deflt=undef;
$sa_tag2_level_deflt
Emails with a spam score greater or equal to this level will be marked as spam. Usually the default (6.31) is reasonable. Lower it if too much spam gets through. Raise it if you get too many regular mails caught as spam.
$sa_kill_level_deflt
If an email scores this level then AMaViS will act on the email according to the $final_spam_destiny setting. It should be set to the same value as $sa_tag2_level_deflt
$final_spam_destiny
I do not like the default D_BOUNCE setting here. The usual approach with spam is to flag an email as spam by adding email headers. Your users can then use these headers to decide if they want to sort an email into an extra folder or discard it. But if you bounced them you would hit some innocent person because spam mails never contain the correct sender address. Just let the user decide what to do with spam. So it is strongly recommended that you set this to D_PASS:
$final_spam_destiny=D_PASS;
There are a lot of further settings that you can use to tweak AMaViS to your own preferences.
Make sure that your 50-user file ends with "1;" or else AMaViS will not start up properly.
Restart AMaViS if you have made changes to the config files:
/etc/init.d/amavis restart
Make sure that AMaViS is listening on TCP port 10024:
netstat -nap | grep 10024
You should get this output:
tcp 0 0 127.0.0.1:10024 0.0.0.0:* LISTEN 12345/amavisd
If you get such a line then AMaViS is running and waiting for incoming SMTP sessions. Otherwise check your /var/log/mail.log file - perhaps you have made a mistake in the configuration files.
This should get you started. For a more detailed documentation please see the README.Postfix that provided by AMaViS.
Why filter outbound email?
In former tutorials I used SQL queries to determine whether an email is incoming or outgoing. Many system administrators do not want to scan outgoing email for spam because:
- your own newsletter could be sent to your customers flagged as spam
- you are wasting CPU resources
But nowadays I would rather recommend to allow scanning any email because
- it helps determine a virus infestation of a local PC
- it helps AMaViS learn both ham (good email) and spam (unwanted email)
Ultimately you decide. There are different approaches described in the AMaViS documentation.
Testing
Now that everything is set up you will want to test your spam scanning process. Fortunately SpamAssassin comes with both a ham (good) and spam (bad) email for you to test with. You can send these emails into your mail system and watch the /var/log/mail.log file to see if AMaViS is doing the right thing.
First let's try the spam email:
sendmail john@example.com < /usr/share/doc/spamassassin/examples/sample-spam.txt
I addition to many lines in your /var/log/mail.log file dealing with "postfix" you should see one line dealing with "amavis". It will look like this:
May 7 12:42:33 debian amavis[1834]: (01834-01) Passed SPAM, <root@...> -> <john@example.org>, quarantine: T/spam-TrRe46tTm2+r.gz, Message-ID: <GTUBE1.1010101@example.net>, mail_id: TrRe46tTm2+r, Hits: 1002.448, size: 944, queued_as: 22DC9D40, 5022 ms
This shows that the email was accepted ("Passed") but flagged as spam ("SPAM"). The spam score is 1002.448 ("Hits") which is way above the limit I have set as $sa_tag2_level_deflt. Also the email was copied to /var/lib/amavis/virusmails/T/spam-TrRe46tTm2+r.gz in GZIPped format. To learn how to deal with quarantined emails please see the AMaViS documentation.
Nearly done
Did you enable soft bounce earlier? Everything works? If the
mailq
command shows that there are still emails in the queue that need to be delivered then re-queue them first:
postsuper -r ALL
Re-scheduling means that Postfix reconsiders what to do with every email in the queue. Transport (routing) and content filtering information stick to an email. So even if you reconfigure Postfix then emails in the queue would not pick that up until you re-queue them.
Now flush the queue
postfix flush
and the emails should get delivered. The /var/log/mail.log will give you information on what happens.
If everything works as you expect then switch off soft bounce mode again and you are done:
postconf -e soft_bounce=no
Want more?
If you seriously want to run AMaViS on a production mail server then please spend time with its documentation. It's a complex piece of software that can integrate a lot of third-party software like virus scanners. It can be tweaked a lot. It can automatically add DKIM signatures on outgoing email. It has a quarantine system so you can moderate spam emails. It just offers many additional features outside of the scope of this tutorial. If you believe you have a valuable addition to this tutorial then please send a comment on this page. Thank you.
Just one word on spam scanning. SpamAssassin (as being used by AMaViS) uses different rules to look for spam but can also learn to distinguish ham from spam emails. To do that properly it needs to scan an equal amount of several hundred ham and spam emails before it can really do its job. And as far as I know there is no way to learn spam per-user. So whatever you teach SpamAssassin is good or bad will apply for all your users.
31 Comments
XXX: Why are there no proper headers?
Submitted by Anonymous (not verified) on
Because spam and virus scanning is disabled by default in /etc/amavis/conf.d/15-content_filter_mode.
D'oh. Thanks for the hint.
Submitted by Christoph Haas on
D'oh. Thanks for the hint. Apparently I've been looking in the wrong place. :)
Why did you choose to use
Submitted by Anonymous (not verified) on
Why did you choose to use amavis instead of clamav-milter and spamass-milter?
Interesting question indeed.
Submitted by Christoph Haas on
Interesting question indeed. I was hesitating to propose AMaViS. But I received a couple of feedback emails telling me AMaViS is preferred due to it's additional features. I personally don't use AMaViS any more. And somehow I'm tempted to follow your suggestion...
Especially since power users
Submitted by Anonymous (not verified) on
Especially since power users who need additional features from amavis don't need a basic setup guide :)
can you tell what are you
Submitted by Anonymous (not verified) on
can you tell what are you using instead of amavis ?
Do you plan to add a section
Submitted by Anonymous (not verified) on
Do you plan to add a section for clamav-milter and spamass-milter? If yes I will be happy to wait for your tutorial, otherwise I will put it in my todo and google for an howto.
I will probably add that. But
Submitted by Christoph Haas on
I will probably add that. But due to other projects I got distracted by that may take up to a month. So if you ar ein a hurry you better start reading the documentation. The setup shouldn't be hard.
Fine, thank you
Submitted by Anonymous (not verified) on
No problem, I will wait for your tutorial because I have other projects too and I'm already quite satisfied even without spamassassin. Thank you for this wonderful howto, surely the best one I have ever read.
spamass-milter
Submitted by LightVision (not verified) on
Well, I've tried spamass-milter and it works, but not so reliable like amavisd-new. Plus it can't find the user (rcpt to) and give warnings that spamd is running as root then drop's to nobody.
The same for milter-manager.
Actually, it realy takes longer to process incomming mails with milters thant with amavis. This is my oppinion, no tests done.
So I'll stick with amavid and get read the documentation early provided.
Thanks Christoph for your great tutorial!
PS. I'll realy appreciate if you'll have the time to write how can be done with milters.
I just like to admit, that I
Submitted by Anonymous (not verified) on
I just like to admit, that I also would like to see a section for clamav-milter and spamass-milter. But there is no hurry ;-)
Realtime Whitelist?
Submitted by Anonymous (not verified) on
What's more urgent is finding a way to deal with RBL false positives coming from big providers!
Getting mails from gmail or hotmail trashed by spamcop is simply inconceivable! Is there no way to whitelist all the bigger providers like gmail, hotmail, aol, yahoo etc.?
I think about senderscore: have you to pay to get access to their whitelist? Also, how to implement a dinamic whitelist in postfix?
The answer is on the next page in the discussion
Submitted by Anonymous (not verified) on
smptd_proxy_filter
Submitted by Anonymous (not verified) on
Maybe think about integrating amavisd-new as smtpd_proxy_filter instead as content_filter!
Then Postfix is able to reject recognized spam within smtp connection and must not accept mail first and bounce it afterwards.
Greetz, Tobias
No headers added for spam
Submitted by Anonymous (not verified) on
I was unable to add headers to spam and found that /etc/mailname has to contain the hostname of the domain, modified it and worked.
Since this tutorial is about a multidomain mailserver (great one, thank you ;) ), shouldn't be added these lines of lenny tutorial on /etc/amavis/conf.d/50-user?
So edit the /etc/amavis/conf.d/50-user file and before the "1;" enter these lines:
@lookup_sql_dsn = ( ['DBI:mysql:database=mailserver;host=127.0.0.1;port=3306', 'mailuser', 'mailuser2009']); $sql_select_policy = 'SELECT name FROM virtual_domains WHERE CONCAT("@",name) IN (%k)';Or i've done something wrong?
Thank you so much for this
Submitted by Anonymous (not verified) on
Thank you so much for this pointer! I was already wondering why there were no headers in all GTUBE messages I spammed to my new server, but this fixed it :-)
Thanks
Submitted by Anonymous (not verified) on
This is a precious hint for the "virtual domain" & "Spam-Header" Configuration!
Thanks for paving this street. You made my day.
Also No Headers Added for Spam - Update?
Submitted by Anonymous (not verified) on
First, thank you for posting additional information to get spam headers properly added to filtered email.
In my case, the X-Virus-Scanned header was being properly added, and the logs showed the mail was being properly scanned and classified, however the actual spam headers were not being added.
Your first suggestion, /etc/mailname is already populated with my FQDN. I don't think that is/was the problem.
I added the @lookup_sql_dsn section you provided and it all began working properly and well.
1. What does the addition of the @lookup_sql_dsn section do?
2. Why is that section not included in and a part of the tutorial if that is required for spam filtering and added headers to function properly? I see the author responded to comments much later than yours... but did not respond/affirm your comments/additions.
Would the author be willing to comment and or update the documentation to include what is necessary for the filtering and headers to function properly?
Lastly, THANK YOU, and others, for such an excellent, wonderful resource and tutorial.
Note the system in discussion is Debian 6, and I have followed the tutorial to the letter, and all is working properly except for this spam filtering/header issue discussed here.
No headers added for spam
Submitted by LightVision (not verified) on
Paid attention!!!
If you want amavis to scan and filter only inbound mails then follow the comment above or Lenny tutorial and use <b>@lookup_sql_dsn = ... </b>alongside with <b>$sql_select_policy = ...</b>.
If you want amavis to scan and filter ALL mails, no matter inbound or outbound then add to <b>/etc/amavis/conf.d/50 - user</b> right before "... Do not modify anithig below this line" the folowing line:
<b>@local_domains_acl = qw(.);</b>
LightVision
What happen to sa-learn?
Submitted by Anonymous (not verified) on
Why was the sa-learn not included in this tutorial?
Several reasons. Too
Submitted by Christoph Haas on
Several reasons.
That's fine. I never could
Submitted by Anonymous (not verified) on
That's fine. I never could get it to learn anything on my Etch server anyway.
Are the other 2 filters better than AMaVIS? Are they esier to setup and maintain?
No mail after AmaVis configuration
Submitted by Anonymous (not verified) on
Hi:
After the Postfix flush mails weren't delivered.
Log showed:
Negative greeting: at (eval 101) line 596, <GEN33> line 659.
Any ideas what went wrong?
SPAM Folder
Submitted by Anonymous (not verified) on
Tanks for this Tutorial. Everything works fine. A X- header is added. But how to move messages marked as SPAM to the SPAM Folder? They are stored in Inbox...
There are basically two
Submitted by Christoph Haas on
There are basically two different ways.
Problems with amavis chroot
Submitted by Anonymous (not verified) on
Hi,
i dont really know why, but with your line for amavis in master.cf
(127.0.0.1:10025 inet n - n - - smtpd)
i got error messages in my logs.
Apr 1 18:42:59 srv postfix/qmgr[22928]: warning: connect to transport private/smtp-amavis: No such file or directory
Apr 1 18:42:59 srv postfix/error[27270]: 174B93264034: to=<abc@def.com>, relay=none, delay=0.12, delays=0.11/0/0/0, dsn=4.3.0, status=deferred (mail transport unavailable)
After changing the line without running in chroot environemnt it works.
127.0.0.1:10025 inet n - - - - smtpd
When I try to test my spam
Submitted by Anonymous (not verified) on
When I try to test my spam scanning with sendmail john@example.com < /usr/share/doc/spamassassin/examples/sample-spam.txt, my mail.log says:
Apr 16 20:41:17 server1 postfix/pickup[10148]: B270D1CB: uid=0 from=<root>
Apr 16 20:41:17 server1 postfix/cleanup[10498]: B270D1CB: message-id=<GTUBE1.1010101@example.net>
Apr 16 20:41:17 server1 postfix/qmgr[10149]: B270D1CB: from=<root@emailfriend.com>, size=941, nrcpt=1 (queue active)
Apr 16 20:41:17 server1 postfix/smtpd[10503]: connect from localhost[127.0.0.1]
Apr 16 20:41:17 server1 dovecot: auth(default): new auth connection: pid=10503
Apr 16 20:41:17 server1 postfix/smtpd[10503]: D46ED1CA: client=localhost[127.0.0.1]
Apr 16 20:41:17 server1 postfix/cleanup[10498]: D46ED1CA: message-id=<GTUBE1.1010101@example.net>
Apr 16 20:41:17 server1 postfix/qmgr[10149]: D46ED1CA: from=<root@emailfriend.com>, size=1399, nrcpt=1 (queue active)
Apr 16 20:41:17 server1 amavis[10099]: (10099-01) Passed SPAM, <root@emailfriend.com> -> <john@example.com>, quarantine: f/spam-fvyu2k+z0xzs.gz, Message-ID: <GTUBE1.1010101@example.net>, mail_id: fvyu2k+z0xzs, Hits: 1002.069, size: 941, queued_as: D46ED1CA, 144 ms
Apr 16 20:41:17 server1 postfix/smtp[10500]: B270D1CB: to=<john@example.com>, relay=127.0.0.1[127.0.0.1]:10024, delay=0.17, delays=0.01/0.01/0.01/0.14, dsn=2.0.0, status=sent (250 2.0.0 Ok, id=10099-01, from MTA([127.0.0.1]:10025): 250 2.0.0 Ok: queued as D46ED1CA)
Apr 16 20:41:17 server1 postfix/qmgr[10149]: B270D1CB: removed
Apr 16 20:41:17 server1 postfix/smtp[10493]: connect to example.com[192.0.43.10]:25: Connection refused
Apr 16 20:41:17 server1 postfix/smtp[10493]: D46ED1CA: to=<john@example.com>, relay=none, delay=0.09, delays=0.01/0/0.09/0, dsn=4.4.1, status=deferred (connect to example.com[192.0.43.10]:25: Connection refused)
Congratulations
Submitted by Christoph Haas on
Great to hear it works. AMaViS can be tricky at times.
example.org not example.com in the test
Submitted by Sven (not verified) on
The test email has a little error, it says example.com as destination domain instead of example.org
sendmail john@example.com < /usr/share/doc/spamassassin/examples/sample-spam.txt
should be
sendmail john@example.org < /usr/share/doc/spamassassin/examples/sample-spam.txt
Spamassassin
Submitted by Mark Taylor (not verified) on
Just a note to say I had to edit: /etc/default/spamassassin and set ENABLED=1 then do service spamd start before spam scanning worked :)
Thanks for the awesome guide.
No SpamAssassin headers
Submitted by Patschi (not verified) on
Hello,
first I thank you for the great tutorial. My problem is, that SpamAsassin is not working correctly on my system. It isn't cutting the content of the mail and not adding the headers, even if I added "@lookup_sql_dsn" in the "50-user" config file. The thing is: It is renaming the subject of spam mails and in the logs I can read, that SpamAssassin detected the mail sucessfully as spam. The test fail of SA went through the filter - even no subject rename, but also recognized as spam in the logfiles.
Thanks for any help! :)