Filtering out spam with rspamd

You have come a long way in this guide and your mail server is already fully functional. Now it’s time to deal with the dark side: spam. And there will be lots of it. So we need to detect spam emails and filter them out. I found that rspamd is a well-performing choice for that purpose both in speed and detection. rspamd keeps a permanent process running on your mail server that listens to connections from Postfix using the milter (=mail filter) protocol. Every time an email enters your system, Postfix will send it to rspamd to have its content checked. rspamd runs a lot of checks on the email and computes a total score. The higher the score – the more likely the email should be considered unsolicited.

How free is rspamd?

rspamd is pretty effective against spam thanks to its many kinds of checks that incoming emails run through. Several of these checks rely on services provided by the rspamd project. There are web sites like email.rspamd.com, uribl.rspamd.com and maps.rspamd.com that provide information used by the default rspamd installation. That data is not freely available and there are no public mirrors.

You are only allowed to use their data under certain conditions described in rspamd’s usage policy. Spoiler: heavy or commercial usage will get you blocked.

rspamd is being packaged and shipped by the Debian project. However the rspamd principal developer builds and endorses his own packages and has a history of getting angry and offensive against anyone using the packages shipped by Debian. I find that attitude quite suspicious and rather stay with the Debian packages that have been built by a process that I trust.

Make Postfix use rspamd

Let’s tell Postfix to send all incoming email through rspamd. Run these commands on the shell:

postconf smtpd_milters=inet:127.0.0.1:11332
postconf non_smtpd_milters=inet:127.0.0.1:11332
postconf milter_mail_macros="i {mail_addr} {client_addr} {client_name} {auth_authen}"

Postfix will connect to rspamd that is listening to TCP port 11332 on localhost (127.0.0.1) and send the email through that connection. The smtpd_milters setting defines that connection for emails that came into the system from the internet via the SMTP protocol. The non_smtpd_milters setting is optional – it makes Postfix have all emails checked that originate from the system itself. Finally the milter_mail_macros setting defines several variables that rspamd expects for better spam detection. rspamd then runs its checks and tells Postfix whether the email should pass or get rejected.

Testing spam detection

For testing we can use a sample spam email that comes with SpamAssassin. It is called GTUBE (Generic Test for Unsolicited Bulk Email). It contains a certain artificial pattern that is recognized as spam by SpamAssassin. Do you know EICAR.COM to test virus scanners? This is the same thing for spam.

I suggest that you download the file on the server:

wget http://spamassassin.apache.org/gtube/gtube.txt

…and send that email to our test user John…

sendmail john@example.org < gtube.txt

Check your /var/log/mail.log. You will find something like this:

Jan 13 09:21:01 mail postfix/cleanup[19945]: 95B7A42212: milter-reject: END-OF-MESSAGE from localhost[127.0.0.1]: 5.7.1 Gtube pattern; from=<root@mail.example.org> to=<john@example.org>
Jan 13 09:21:01 jen postfix/cleanup[19945]: 95B7A42212: to=<john@example.org>, relay=none, delay=0.18, delays=0.18/0/0/0, dsn=5.7.1, status=bounced (Gtube pattern)

The interesting parts are those printed in bold letters. “milter-reject” tells that the milter (rspamd) recommended to Postfix to reject the email. It gave the reason “5.7.1 Gtube pattern”. In mail communication you often find these three digit codes. They are defined in RFC 3463. The first digit is most important:

  • 2 = Success
  • 4 = Transient failure (temporary problem – come back later)
  • 5 = Permanent failure (do not try again in this way)

So 5.7.1 tells us that the result code is a permanent failure in delivery. If you looked up the RFC you would find that the 7 stands for an issue regarding the security policy. So it’s not a technical failure but instead a security-relevant component on the system has rejected the email. That’s what rspamd did. It even told us the reason: “Gtube pattern”. So you see that rspamd knows about the Gtube spam test pattern and reacts as expected.

Accordingly you won’t see this email in John’s inbox. This is a great advantage of using milters. Imagine Postfix receiving a spam email and confirm its reception. What should it do when it finds out that it’s unwanted email? According to the SMTP protocol it must not throw away any emails. Would you create a bounce message telling the sender that you did not accept the email? That would be a bad idea because the sender address in spam emails is very likely forged. You would send the bounce to an innocent person thus creating so called backscatter and make it even worse. So the better approach is to check the email while the sending server is still connected to your Postfix. This allows Postfix to reject the email with a 5.x.x error code and let the other side figure out what to do.

Score metrics

rspamd will however not reject all spam email. It computes a score that tells how likely a certain email is spam. You can tell it which scores you would accept, flag as spam or make the incoming email get rejected. Rspamd has a large ruleset that checks incoming emails in various ways and adds to the score. Take a look at the /etc/rspamd/actions.conf file:

actions {
  reject = 15;
  add_header = 6;
  greylist = 4;
}

These are the default actions. If rspamd computes a score of at least 15 then the email will get rejected at the doorstep just like the Gtube pattern in the previous test. Any other score above 6 will add a line “X-Spam: Yes” so that your mail software can detect them and maybe file the email to a different folder. And any other score above 4 will trigger greylisting which is a mechanism that temporarily rejects the email with a 4.x.x code and waits if the sending server will try again. After a waiting time of a few minutes greylisting will accept the email. The idea is to reject email from systems that do not have a sending queue. Malware like on infected Wind*ws computers used to try sending an email just once which triggered greylisting and successfully rejected the spammer. But even malware programmers have learned and may try again after a few minutes thus circumventing greylisting. Your mileage may vary. The problem with greylisting is that the recipient has to wait a couple of minutes for the email to be delivered which is often bothering the users.

If you like to change these defaults then create a new file in /etc/rspamd/local.d/actions.conf containing your desired limits:

reject = 150;
add_header = 6;
greylist = 4;

This would virtually never reject an email. The other two values are pretty sane defaults. I personally use this setting all the time so that users can find spam in their Junk folder but don’t have to ask me if the mail server rejected it.

Please take a moment to understand how to change rspamd defaults. You can either create files in /etc/rspamd/override.d/… which will replace entire sections; or create files in /etc/rspamd/local.d/… which will change only parts of the configuration. There is a helpful page in the rspamd documentation that contains examples. Whatever you do – never change the /etc/rspamd/* files directly because a software update will try to replace them.

Restart rspamd after any configuration changes:

systemctl restart rspamd

To check if rspamd has picked up your configuration use this command to see the current configuration:

rspamadm configdump

You may test your configuration using

rspamadm configtest

Alternatively you may check if all required rspamd processes are running…

pgrep -a rspam

141693 rspamd: main process
141694 rspamd: rspamd_proxy process (localhost:11332)
141695 rspamd: controller process (localhost:11334)
141696 rspamd: normal process (localhost:11333)
141697 rspamd: hs_helper process

Adding headers

As you may know an email consists of the headers and the body. Your users will usually only see common header information like the subject, the sender, the recipient and the date and time the email was sent. But there is way more information like the route the email traveled or extended headers added by the various mail server on the way to the destination. Such extended headers begin with an “X-“. rspamd can add such headers to help you filter out spam. For that purpose create a new configuration override file at /etc/rspamd/override.d/milter_headers.conf with this content:

extended_spam_headers = true;

Again restart rspamd:

systemctl restart rspamd

As documented it will add these headers:

X-Rspamd-Server: mail
Authentication-Results: dmarc=fail reason="No valid SPF, No valid DKIM" …
X-Rspamd-Queue-Id: C22E55A005B3
X-Spamd-Result: default: False [11.55 / 15.00]
 R_PARTS_DIFFER(0.27)[63.4%]
 FROM_NO_DN(0.00)[]
 RCVD_COUNT_ZERO(0.00)[0]
 R_DKIM_NA(0.00)[]
 FUZZY_DENIED(12.00)[1:19305c7fdd:1.00:bin,1:35699594fd:0.91:txt]
 RBL_SENDERSCORE(2.00)[55.181.23.94.bl.score.senderscore.com]
 ARC_NA(0.00)[]
 RCPT_COUNT_ONE(0.00)[1]
 RCVD_TLS_ALL(0.00)[]
 FROM_EQ_ENVFROM(0.00)[]
 R_SPF_SOFTFAIL(0.00)[~all]
 BAYES_HAM(-2.71)[98.75%]
 TO_MATCH_ENVRCPT_ALL(0.00)[]
 MIME_GOOD(-0.10)[multipart/related,multipart/alternative,text/plain]
 MID_RHS_MATCH_FROM(0.00)[]
 ASN(0.00)[asn:16276, ipnet:94.23.0.0/16, country:FR]
 TO_DN_NONE(0.00)[]
 DMARC_POLICY_SOFTFAIL(0.10)[Chronopost.fr : No valid SPF, No valid DKIM,none]
 SUBJECT_ENDS_EXCLAIM(0.00)[]
X-Spam: Yes

No headers?

Please note that rspamd only adds headers if an email is received from outside your server. So if you use swaks locally then you will not get anything.

Each of the uppercase symbols like FROM_HAS_DN means that a certain detection rule of rspamd was triggered. It does not necessarily mean something bad about the email. For example R_SPF_ALLOW has a negative score that lowers the total score because it is something good about the email. There are a several symbols with a 0.00 score. These do not change the score but show you what rspamd has found. But if you consider certain criteria good or bad then you can define your own scores for them.

The last line here is especially interesting because next on our list is…

Sending spam to the Junk folder

Your users will not realize that their spam emails have an added “X-Spam: Yes” header. It is not actively shown in their mail client. Nor does it move the email out of the inbox into their spam folder. Such emails just appear like normal in their inbox. So let’s aid them by moving spam to a separate Junk folder beneath their inbox automatically. Dovecot has support for Sieve filters which are scripts that run automatically whenever an email comes in.

John could create a new Sieve filter (e.g. using the Roundcube webmail interface) for himself that would save any emails to his “Junk” folder if the header line “X-Spam: Yes” was found. This rule would be useful for all your users though so let’s find a more general solution.

Dovecot supports global Sieve filters that apply to all users. Edit the file /etc/dovecot/conf.d/90-sieve.conf. Look for the “sieve_after” lines. They are commented out. Add a new line there:

sieve_after = /etc/dovecot/sieve-after

The “sieve after” filters are executed after the users’ filters. John can define his own filter rules. And after that Dovecot will run any filter rules it finds in files in /etc/dovecot/sieve-after. That is just an arbitrary directory that you create:

mkdir /etc/dovecot/sieve-after

And add a new file /etc/dovecot/sieve-after/spam-to-folder.sieve reading:

require ["fileinto"];

if header :contains "X-Spam" "Yes" {
 fileinto "Junk";
 stop;
}

The “require” lines include functionality to move emails into certain folders (fileinto) and to create folders if they don’t exist yet (mailbox). Then if rspamd marked an email as spam it gets moved into the INBOX.Junk folder which just appears as “Junk” to the user underneath their inbox.

Dovecot cannot deal with such human-readable files though. So we need to compile it:

sievec /etc/dovecot/sieve-after/spam-to-folder.sieve

That generated a machine-readable file /etc/dovecot/sieve-after/spam-to-folder.svbin.

Now all your users will automatically get spam emails moved to their Junk folder. Nice – isn’t it?

About Redis

Many features in Rspamd use Redis to persist their data. Let me give you a quick introduction.

Redis is a kind of database system. It is way more limited than a traditional SQL database because it just stores keys and values. There aren’t several fields/columns like in SQL. But it is lightning fast the way it works. On my aged server it handles around 50,000 requests per second. It gets it speed from its simplicity and from keeping the data in RAM. So it doesn’t access the disk to fetch information. (Yet it saves its data to disk frequently to prevent data loss.) People use Redis as a cache or for very fast lookups of simple data structures. And so does rspamd.

You installed the “redis-server” package earlier. And that’s all you needed to do. It started automatically and listens ton incoming connections on TCP port 6379 on localhost. In Rspamd the Redis backend is enabled by default. You just have to tell it the IP address of your Redis server. Add a file /etc/rspamd/override.d/redis.conf and insert:

servers = "127.0.0.1";

Restart rspamd and you are done.

systemctl restart rspamd

Feel free to use Redis for other lookups, too.

Training the spam detection

One of rspamd’s features is analyzing word patterns using probability theory. That functionality is contained in its “statistical module“. (Yes, the name is pretty misleading.) Essentially you show rspamd lots of ham (good) and spam (bad) emails and its detection gets better over time.

Redis replaces SQLite

Rspamd recommends (and defaults to) using Redis to store spam training data. That’s what we will use now.

(a) No Bayes training

You can start with an empty training database. This is not as bad as it sounds. rspamd has way more functionality to determine if an email is ham or spam. Autolearning takes email that are likely ham or spam and uses them to train the spam filter. The rspamd documentation has further examples how to fine-tune auto learning. After a few hundred emails the training will contribute towards a better detection rate.

If you want to use autolearning just create a new file /etc/rspamd/override.d/classifier-bayes.conf and make it contain:

autolearn = true;

(b) Migrating training data from previous mail server

Have you used the old SQLite-based training file on the old server? Look for files like /var/lib/rspamd/*.sqlite on the old server. In that case please follow these simple instructions from the rspamd documentation to convert them to data in Redis.

If instead you have used Redis already then you just need to copy over the Redis database from the old server. Stop Redis on the new server. Copy over the /var/lib/redis/dump.rdb from the old server to the new server. Start Redis again. And restart rspamd. So on the new server run:

systemctl stop redis
scp root@old-server:/var/lib/redis/dump.rdb /var/lib/redis
systemctl start redis
systemctl restart rspamd

To check if that worked you can ask Rspamd using “rspamc stat” and look for…

Statfile: BAYES_SPAM type: redis; length: 0; free blocks: 0; total blocks: 0; free: 0.00%; learned: 21164; users: 214; languages: 0
Statfile: BAYES_HAM type: redis; length: 0; free blocks: 0; total blocks: 0; free: 0.00%; learned: 1411; users: 62; languages: 0

(c) Training from your existing ham and spam emails

Have you been running a mail server with mailboxes in a Malidir structure before but without rspamd? Then you probably have a good amount of ham and spam emails. Let’s use those to train rspamd. It is important to train both ham and spam emails. The rspamc command will allow you to feed entire directories/folders of emails to the learning process. An example to train spam:

rspamc learn_spam /var/vmail/example.org/john/Maildir/.Junk/cur

And this would be an example to train ham from John’s inbox:

rspamc learn_ham /var/vmail/example.org/john/Maildir/cur

Of course the quality of spam detection will depend on how good the source data is. If users put emails in their Junk folder which are not typical spam they will soil the detection.

Check the number of emails you learned by running…

rspamc stat

Bayes spam checking will not work before it learned at least 200 spam and ham emails. Teaching rspamd fewer emails or just spam emails will not work. This is defined by the min_learns variable defined in /etc/rspamd/statistic.conf.

In the output you will find lines beginning with “Statfile” like these…

Statfile: BAYES_SPAM type: redis; length: 0; free blocks: 0; total blocks: 0; free: 0.00%; learned: 21164; users: 214; languages: 0
Statfile: BAYES_HAM type: redis; length: 0; free blocks: 0; total blocks: 0; free: 0.00%; learned: 1411; users: 62; languages: 0

This is what you usually start with. The more emails you feed into the training process the better the detection rate will be. Some emails however may not be long enough or too similar to previously trained emails. So don’t worry if you are training 1000 emails but just get a count of 500 emails here.

Per-user spam training

rspamd allows you to train the spam detection per user. It would not keep a global training database that applies to all users. Instead each user gets their own training.

Advantage: users work differently. Some have subscribed to a sales newsletter and now believe that marking it as spam gets them unsubscribed. Yes, that’s stupid but can thoroughly confuse the spam detection. Also you might be very interested in viagr* product information while others do not.

Disadvantage: training still requires many ham and spam mails before it has any effect. So unless a user gets 200 samples of good and evil emails the spam detection cannot work. Many users will not get that many emails so due to the lack of spam training the detection will not be improved.

If you decide you want to use per-user spam training then add/edit the file /etc/rspamd/local.d/classifier-bayes.conf and insert:

users_enabled = true;

Learning from user actions

Now we are getting to something really cool. Let’s tell Dovecot that moving emails into the Junk folder teaches rspamd instantly that the email is spam. And train an email as ham if it is moved out of the Junk folder. We will add triggers (actually “sieve scripts“) to the action of moving emails via IMAP.

The currently recommended way is to use the “IMAPSieve” plugin instead. There is nothing to install – it comes with the Dovecot packages. We just need to configure it.

First order of business is enabling the IMAPSieve plugin for the IMAP protocol/service in Dovecot. Edit the /etc/dovecot/conf.d/20-imap.conf file and look for the line reading “mail_plugins”. Turn it into:

mail_plugins = $mail_plugins quota imap_sieve

We also need to edit Dovecot’s Sieve configuration to enable two plugins that are required for our task. Sieve is a scripting language that automates things in conjunction with emails and folders. Edit the file /etc/dovecot/conf.d/90-sieve.conf and put these lines into the plugin {…} section:

# From elsewhere to Junk folder
imapsieve_mailbox1_name = Junk
imapsieve_mailbox1_causes = COPY
imapsieve_mailbox1_before = file:/etc/dovecot/sieve/learn-spam.sieve

# From Junk folder to elsewhere
imapsieve_mailbox2_name = *
imapsieve_mailbox2_from = Junk
imapsieve_mailbox2_causes = COPY
imapsieve_mailbox2_before = file:/etc/dovecot/sieve/learn-ham.sieve

sieve_pipe_bin_dir = /etc/dovecot/sieve
sieve_global_extensions = +vnd.dovecot.pipe
sieve_plugins = sieve_imapsieve sieve_extprograms

The first rule tells Dovecot to run the Sieve rules as defined in the /etc/dovecot/sieve/learn-spam.sieve file whenever an email is moved into a user’s “Junk” folder. We will create that Sieve script in a minute.

The second rule sets the other way. Whenever an email is moved from the “Junk” folder to any (*) folder then the /etc/dovecot/sieve/learn-ham.sieve Sieve script is called.

The “sieve_pipe_bin_dir” setting defines where executable scripts are allowed to reside. We will put our simple learning scripts there. And finally the “sieve_global_extensions” setting enables the pipe plugin that allows sending email to external commands.

Next up let’s create the Sieve scripts that we told Dovecot about. Create a new directory /etc/dovecot/sieve to put our new files in:

mkdir /etc/dovecot/sieve

Then create the file /etc/dovecot/sieve/learn-spam.sieve and let it contain:

require ["vnd.dovecot.pipe", "copy", "imapsieve"];
pipe :copy "rspamd-learn-spam.sh";

Let’s do the same for /etc/dovecot/sieve/learn-ham.sieve

require ["vnd.dovecot.pipe", "copy", "imapsieve", "variables"];
if string "${mailbox}" "Trash" {
  stop;
}
pipe :copy "rspamd-learn-ham.sh";

The above Sieve script avoids training an email as ham if the user moves it to the Trash folder. After all if you clear your Junk folder you do not want to train your spam as regular emails.

Restart Dovecot:

systemctl restart dovecot

These two scripts need to be compiled – that is turning them into machine-readable code:

sievec /etc/dovecot/sieve/learn-spam.sieve
sievec /etc/dovecot/sieve/learn-ham.sieve

This creates two new files “learn-ham.svbin” and “learn-spam.svbin” that look like gibberish inside but are now in a format that Dovecot’s Sieve plugin can understand.

Let’s fix the permissions of these files, too, while we are at it:

chmod u=rw,go= /etc/dovecot/sieve/learn-{spam,ham}.{sieve,svbin}
chown vmail.vmail /etc/dovecot/sieve/learn-{spam,ham}.{sieve,svbin}

And the last step is to create the simple shell scripts that do the actual spam/ham training. The first file is /etc/dovecot/sieve/rspamd-learn-spam.sh which contains:

#!/bin/sh
exec /usr/bin/rspamc learn_spam

That looks simple, doesn’t it? Nothing more is actually needed. The spam email is piped to this script and rspamd learns it as a spam email and adjusts its spam detection database accordingly.

The second script teaches ham and is called /etc/dovecot/sieve/rspamd-learn-ham.sh. Make it contain:

#!/bin/sh
exec /usr/bin/rspamc learn_ham

These two shell scripts must be made executable:

chmod u=rwx,go= /etc/dovecot/sieve/rspamd-learn-{spam,ham}.sh
chown vmail.vmail /etc/dovecot/sieve/rspamd-learn-{spam,ham}.sh

I hope you haven’t lost your mind yet. It’s really just a chain of things to happen. Let’s reiterate how this process works:

  1. a user moves a spam email into their “Junk” folder
  2. Dovecot realizes that this triggers the Sieve rule “imapsieve_mailbox1” so it calls the Sieve script /etc/dovecot/sieve/learn-spam.sieve (in fact the *.svbin version of the script)
  3. Sieve will take the email and send (“pipe”) it to the executable rspamd-learn-spam.sh shell script
  4. the script in turn runs the email through the “/usr/bin/rspamc learn_spam” command

This works equally for the other way or learning ham emails of course.

I am sure you are eager to try it out. Yes, it’s about time. However to see that it actually works I suggest you edit the /etc/dovecot/conf.d/10-logging.conf file and set “mail_debug=yes”. That will add a lot more detail to the /var/log/mail.log file but on a busy server may also lead to headaches. 🙂

Restart Dovecot…

systemctl restart dovecot

…and watch the /var/log/mail.log file…

tail -f /var/log/mail.log

Now open your IMAP client (Thunderbird, Evolution, Roundcube, mutt or whatever you prefer) and drag an email to your Junk folder. The mail log should read something along the lines of…

imap(john@example.org): Debug: imapsieve: mailbox INBOX.Junk: MOVE event
imap(john@example.org): Debug: imapsieve: Matched static mailbox rule [1]
imap(john@example.org): Debug: sieve: file storage: Using Sieve script path: /etc/dovecot/sieve/learn-spam.sieve
imap(john@example.org): Debug: sieve: file script: Opened script learn-spam' from/etc/dovecot/sieve/learn-spam.sieve'
imap(john@example.org): Debug: sieve: Opening script 1 of 1 from /etc/dovecot/sieve/learn-spam.sieve' imap(john@example.org): Debug: sieve: Loading script /etc/dovecot/sieve/learn-spam.sieve imap(john@example.org): Debug: sieve: Script binary /etc/dovecot/sieve/learn-spam.svbin successfully loaded imap(john@example.org): Debug: sieve: binary save: not saving binary /etc/dovecot/sieve/learn-spam.svbin, because it is already stored imap(john@example.org): Debug: sieve: Executing script from/etc/dovecot/sieve/learn-spam.svbin'
imap(john@example.org): Debug: sieve: action pipe: running program: rspamd-learn-spam.sh
imap(john@example.org): Debug: Mailbox INBOX.Junk: Opened mail UID=3978 because: mail stream
imap(john@example.org): Debug: waiting for program /etc/dovecot/sieve/rspamd-learn-spam.sh' to finish after 0 msecs imap(john@example.org): sieve: pipe action: piped message to programrspamd-learn-spam.sh'

Look for errors and warnings. If at the end you see that Dovecot called the “rspamd-learn-spam.sh” script then you should be fine.

And finally if you pull an email out of the “Junk” folder you should see mailbox rule [2] be called and the email being learned as ham:

imap(john@example.org): Debug: imapsieve: mailbox INBOX: MOVE event
imap(john@example.org): Debug: imapsieve: Matched static mailbox rule [2]
imap(john@example.org): Debug: sieve: file storage: Using Sieve script path: /etc/dovecot/sieve/learn-ham.sieve
imap(john@example.org): Debug: sieve: file script: Opened script learn-ham' from/etc/dovecot/sieve/learn-ham.sieve'
imap(john@example.org): Debug: sieve: Opening script 1 of 1 from /etc/dovecot/sieve/learn-ham.sieve' imap(john@example.org): Debug: sieve: Loading script /etc/dovecot/sieve/learn-ham.sieve imap(john@example.org): Debug: sieve: Script binary /etc/dovecot/sieve/learn-ham.svbin successfully loaded imap(john@example.org): Debug: sieve: binary save: not saving binary /etc/dovecot/sieve/learn-ham.svbin, because it is already stored imap(john@example.org): Debug: sieve: Executing script from /etc/dovecot/sieve/learn-ham.svbin'
imap(john@example.org): Debug: sieve: action pipe: running program: rspamd-learn-ham.sh
imap(john@example.org): Debug: Mailbox INBOX: Opened mail UID=28412 because: mail stream
imap(john@example.org): Debug: waiting for program /etc/dovecot/sieve/rspamd-learn-ham.sh' to finish after 0 msecs imap(john@example.org): sieve: pipe action: piped message to programrspamd-learn-ham.sh'

Should you sometimes get errors like “HTTP error: 500, Unknown statistics error, found when storing data on backend; classifier: (null)” then you are not alone. Several people have reported that issue but the author couldn’t yet provide a solution. I tend to just ignore that.

Autoexpunge

Andi Olsen pointed out that Dovecot has introduced a feature to automatically delete emails in a folder that reach a certain age. This is especially useful for the “Trash” and “Junk” folders. To enable this feature just edit the /etc/dovecot/conf.d/15-mailboxes.conf file and add the autoexpunge parameter where desired. Example:

mailbox Junk {
  special_use = \Junk
  auto = subscribe
  autoexpunge = 30d
}
mailbox Trash {
  special_use = \Trash
  auto = subscribe
  autoexpunge = 30d
}

Restart Dovecot after any configuration change.

Logging

rspamd keeps a verbose log of its actions in /var/log/rspamd/rspamd.log. If a user complains that a certain email got blocked or at least flagged as spam then take a look at this log. You can match the /var/log/mail.log with it by comparing the Postfix queue ID. Those are the 12-digit hexadecimal number like “95CE05A00547“. Those IDs can be found in the rspamd.log, too:

2018-01-14 06:39:45 #10424(normal) <40985d>; task; rspamd_task_write_log: id: <undef>, qid: <95CE05A00547>,  ip: 12.13.51.194, from: <…>, (default: F (no action):  [3.40/15.00]  [MISSING_MID(2.50){},MISSING_DATE(1.00){},MIME_GOOD(-0.10){text/plain;},ARC_NA(0.00){},ASN(0.00){asn:8220,  ipnet:212.123.192.0/18,  country:GB;},FROM_EQ_ENVFROM(0.00){},FROM_NO_DN(0.00){},RCPT_COUNT_ONE(0.00){1;},RCVD_COUNT_ZERO(0.00){0;},RCVD_TLS_ALL(0.00){},TO_DN_NONE(0.00){},TO_DOM_EQ_FROM_DOM(0.00){},TO_MATCH_ENVRCPT_ALL(0.00){}]),  len: 181, time: 16.000ms real, 6.385ms virtual, dns req: 0, digest:  <69b289a82827c11f759837c033cd800a>, rcpts: <…>, mime_rcpt:  <…>

Scan outgoing emails, too

In older guides I accidentally recommended to set smtpd_milters to an empty value to prevent scanning outgoing emails. But that was wrong. In fact you want to use rspamd to work on outgoing emails. rspamd determines automatically if an authenticated user wants to send an email out and behaves differently. For example it does not check for IP blacklists or DKIM signatures. Instead it adds a DKIM signature.

So please don’t use “-o smtpd_milters=” anywhere. Sorry for the confusion.

The web interface

rspamd’s web interface

rspamd comes with a neat bonus feature: a web interface. It allows you to check emails for spam, get statistics and fine-tune scores. It is already installed and enabled by default and expects HTTP requests on port 11334 on the localhost interface. I suggest you add a simple proxy configuration to your already working HTTPS-enabled web mail configuration to get access.

First you need to enable Apache’s modules for HTTP proxying and rewriting:

a2enmod proxy_http
a2enmod rewrite

You can either create a new virtual host configuration or just edit the /etc/apache2/sites-available/webmail.example.org-https.conf file. Anywhere within the VirtualHost tags add:

<Location /rspamd>
  Require all granted   
</Location>

RewriteEngine On
RewriteRule ^/rspamd$ /rspamd/ [R,L]
RewriteRule ^/rspamd/(.*) http://localhost:11334/$1 [P,L]

This piece of configuration will forward any requests to https://webmail.example.org/rspamd to localhost:11334 and thus give you access to the rspamd web interface.

The interface is password protected. Let’s generate a new access password:

pwgen 15 1

This gives you a password like “eiLi1lueTh9mia4”. You could put that password in an rspamd configuration file. But cleartext passwords in configuration files are not quite elegant. Let’s create a hash of the password:

rspamadm pw
Enter passphrase: …
$2$icoahes75e7g9wxapnrbmqnpuzjoq7z…

Feed it the password that pwgen created for you and you will get a long hashed password. This procedure by the way is also documented on the rspamd pages.

Create a new configuration file /etc/rspamd/local.d/worker-controller.inc and put your hashed password in there:

password = "$2$icoahes75e7g9wxapnrbmqnpuzjoq7z…"

That’s it for the configuration. Finally restart both rspamd and Apache to load your changed configuration:

systemctl restart rspamd
systemctl restart apache2

If everything went as expected you should now be able to access the rspamd web interface at https://webmail.example.org/rspamd

50 thoughts on “Filtering out spam with rspamd”

  1. It would be useful to see an example of how to incorporate one or more RBLs into rspamd if that isn’t more trouble than it is worth.

    1. Christoph Haas

      In fact rspamd by default already checks several RBLs. If the remote server is found in an RBL rspamd adds a value to the total score.

      Example:

      X-Spamd-Result: default: False [9.60 / 50.00];
      HAS_REPLYTO(0.00)[g…@gmail.com];
      R_SPF_ALLOW(0.00)[+mx];
      RBL_SPAMHAUS_CSS(2.00)[173.82.240.156:from];

  2. /etc/dovecot/sieve/learn-ham.sieve did not work as expected: When moving mail from Junk to Trash it called again the learn-ham script.

    The fix was to do adjust the learn-ham.sieve file in this way, after this fix it worked (do not forget to sievec + chown vmail:vmail):

    require [“vnd.dovecot.pipe”, “copy”, “imapsieve”, “environment”, “variables”];

    if environment :matches “imap.mailbox” “*” {
    set “mailbox” “${1}”;
    }

    if string “${mailbox}” “Trash” {
    stop;
    }

    pipe :copy “rspamd-learn-ham.sh”;

    1. Christoph Haas

      Out of curiosity: why do mails get moved from Junk to Trash? I personally consider Trash “ham mails that I don’t need any more”. But I purge spam mails directly from the Junk folder. But that’s just me and perhaps I’m making assumptions here for everyone else.

      1. George Kovachev

        Thunderbird has an option in “Junk Settings”
        Automatically delete junk mail older than [number of days] days

        If user activated it the filter will not work as expected as Andreas mentioned.

      2. I am using Evolution, the only thing I can do is “Empty Junk”, which moves everything to Trash, then “Empty Trash”, which removes the e-mails.

        I found a setting to purge junk mail in Evolution preferences, thank you for the hint on Thunderbird.

    2. Thank you for this comment. I have been using rspamd for months and was puzzled why bayes detection was not working properly. Luckily I found your comment and implemented your suggestion. I think this deserves mention in the tutorial since it is not uncommon to delete a single message in the spam folder and in Thunderbird, for example, this moves the message to trash, triggering the unwanted behavior.

  3. Hi Christophe
    Thank you for this beautiful and good reference work.

    I want to implement the autoexpunge feature on my mail server and noticed you didn’t use a backslach in the “mailbox” section with the “special_use” syntact.
    These were still there in your previous version.
    I myself use ubuntu 20.04 and got an error from doveconf when I use it to test the configuration of dovecot for syntacts errors
    fixed it to
    mailbox Junk {
    special_use =\ Junk
    auto = subscribe
    autoexpunge = 30d
    }
    mailbox Trash {
    special_use = \Trash
    auto = subscribe
    autoexpunge = 30d
    }

    removed the error message form doveconf

    I think this will also be the case with debain bullseye

    Regard Jan Kruis

    1. Christoph Haas

      Thanks for the hint. That was indeed a display problem here on the page. The backslash definitely must be there.

  4. Hi, What changes should I make to rspamd default configuration to make it free for commecial usage?

  5. Did I miss something, or did you drop virus/malware scanning from the bullseye guide? As far as I can tell, rspamd doesn’t scan for malware. I didn’t see any reference to clamav in the bullseye guide. It was in the buster guide.

    1. Christoph Haas

      I hid the answer to that in the FAQ. 🙂 No, you are right. Malware scanning is not dealt with in the Bullseye guide. I’m lazy. I don’t believe in traditional malware scanning. And thus I didn’t want anyone to rely on my half-donkeyed attempts to increase security.

  6. Hi,

    This is a fantastic guide, but in the Buster version you had:

    require [“fileinto”,”mailbox”];

    if header :contains “X-Spam” “Yes” {
    fileinto :create “Junk”;
    stop;
    }

    But in this version you only have:

    require [“fileinto”];

    if header :contains “X-Spam” “Yes” {
    fileinto :create “Junk”;
    stop;
    }

    Yet you still mention both fileinto AND mailbox. Is this a typo or is the require line meant to contain both fileinto and mailbox ?

    Regards

    1. The :create argument is provided by the mailbox extension, so if :create is used, “mailbox” is required.

      1. Martin Kuchar

        AFAIK :create and “mailbox” is mandatory, because Junk folder is not created before first users login with imap client and autosubscribe..

  7. Oscar Lemmer

    Error: “4.7.1 Try again later” in /var/log/mail.log with outbound mails from authenticated users.
    (on Ubuntu 20.04 with Postfix 3.4.13 and rsapmd v.3.1)

    Solution: remove the quotes [“] in main.cf
    from:
    milter_mail_macros = “i {mail_addr} {client_addr} {client_name} {auth_authen}”
    to:
    milter_mail_macros = i {mail_addr} {client_addr} {client_name} {auth_authen}

    Credits to:
    https://workaround.org/ispmail/stretch/filtering-out-spam-with-rspamd/#comment-41072

  8. Oscar Lemmer

    ARC-Signing can also be enabled very easy.
    It just needs a config file “arc.conf” with the same content as dkim_signing.conf.

    So basically just do (in addition to this tutorial):

    cp /etc/rspamd/local.d/dkim_signing.conf /etc/rspamd/local.d/arc.conf

    and the ARC module is working.

  9. Hello, thanks for tutorial, I use it for 5 years at all:).
    But now I have some problem with spam. Spam mail is always rejected and not copied to Junk folder (inbox too) – to any folder, is rejected and deleted (?)
    I do not know, what is wrong, I checked the config perhaps 100 times:)
    Have somebody the same problem please?

    1. Your question is answered in the “Score metrics” section of this page.

      Your reject setting is too low, and the message is simply rejected, and is not deleted, it was never saved on the system.

  10. Stuck at step to convert file to sieve readable with the command

    sievec /etc/dovecot/sieve-after/spam-to-folder.sieve

    Getting error

    sievec(root): Fatal: Couldn’t load required plugin /usr/lib/dovecot/modules/lib11_imap_quota_plugin.so: dlopen() failed: /usr/lib/dovecot/modules/lib11_imap_quota_plugin.so: undefined symbol: command_unregister

    Any thoughts on where to look?

    1. Apologies, I had googled around looking for help first and was getting nowhere. I posted the message and then went back through my config files and found an error. Corrected the error and now it works.

    2. Christoph Haas

      Check this file:
      conf.d/20-imap.conf: mail_plugins = $mail_plugins quota imap_sieve

  11. A note regarding the “Training from your existing ham and spam emails” chapter, explicitely to the rspamc command: I almost went crazy when I did not find a way to “unlearn” false-positive spam (or false-negative ham). No matter if I had per-user training enabled or not. Mails initially but wrongly sent to spam could not be learnt as ham, even if one learned many slightly different samples. The issue on my end was, that I did not know about the “–header ‘Learn-Type: bulk'” argument for rspamc. This can be used for “learn_ham” and “learn_spam” commands. Maybe that’s worth adding to the documentation? Background: https://github.com/rspamd/rspamd/issues/3600

  12. Florian F.

    Hey Christoph,

    i upgraded my mailserver from previous versions and for whatever reason the script when moving mails is not executed. There is no error in logfiles. No idea why. Did you have similar issue before?

    Thanks and best regards

    1. Florian F.

      in rspam log i get following error:

      2022-06-29 11:53:44 #2159(controller) ; csession; rspamd_stat_classifiers_process: skip statistics as SPAM class is missing
      2022-06-29 11:53:44 #2159(controller) ; csession; rspamd_task_process: skip learning: all learn conditions denied learning spam in default classifier
      2022-06-29 11:53:44 #2159(controller) ; csession; rspamd_controller_learn_fin_task: cannot learn : all learn conditions denied learning spam in default classifier

  13. Daniél Lecoq

    Hi and thanks for an awesome guide. I’ve gone through it a few times before.
    I seem to have a problem with rspamd. first when emailing the gtube.txt, I didn’t get the responses expected. After a few tries, I decided to continue with the guide anyways.
    So… the rspamd configdump
    root@vps-56e8986b:/home/debian# rspamd configdump
    2022-08-01 13:24:00 #0(main) ; main; detect_priv: cannot run rspamd workers as root user, please add -u and -g options to select a proper unprivilleged user or specify –insecure flag
    OK, so I exited the root user and became a normal user, then it worked.
    but…
    debian@vps-56e8986b:~$ rspamd configtest
    2022-08-01 13:17:15 #0(main) logger; rspamd_log_open_specific: cannot open specific logger: open_log: cannot open desired log file: /var/log/rspamd/rspamd.log, Permission denied\x0A
    debian@vps-56e8986b:~$ sudo rspamd configtest
    2022-08-01 13:17:24 #0(main) ; main; detect_priv: cannot run rspamd workers as root user, please add -u and -g options to select a proper unprivilleged user or specify –insecure flag

    Did I miss somewhere that we were supposed to create a rspamd user and group?

    1. Daniél Lecoq

      My error was about being tired and writing “rspamd” instead of “rspamadm”

  14. Hi Christoph, thanks again for the great work. I have one question concerning spam handling:

    I would like to add a blacklist of email-addresses and/or domains the server should reject. I would like using MySQL, but a textfile should be ok, too. What was the best way to do that?

    Thank you very much for your help.
    Kind regards
    Kostja

  15. Silver Dream !

    I am experiencing problems with sieve learning-{ham|spam}. The debug logs show (relevant lines):
    “`
    Debug: sieve: action pipe: running program: rspamd-learn-spam.sh
    Debug: program exec:/etc/dovecot/sieve/rspamd-learn-spam.sh: Created
    Debug: program exec:/etc/dovecot/sieve/rspamd-learn-spam.sh: Pass environment: USER=
    Debug: program exec:/etc/dovecot/sieve/rspamd-learn-spam.sh: Pass environment: HOME=
    Debug: program exec:/etc/dovecot/sieve/rspamd-learn-spam.sh: Pass environment: HOST=
    Debug: Mailbox Junk: UID 1027: Opened mail because: mail stream
    Debug: sieve: uid=1027: Finished executing pipe action (status=ok, keep=implicit)
    Debug: sieve: uid=1027: Finished executing actions (status=ok, keep=implicit)
    Debug: sieve: uid=1027: Finished executing result (no commit, status=ok, keep=yes)
    Debug: sieve: multi-script: Sequence active
    Debug: sieve: multi-script: Finishing sequence (status=ok)
    Debug: sieve: uid=1027: Executing result (status=ok, commit=yes)
    Debug: sieve: uid=1027: Starting execution of actions
    Debug: sieve: uid=1027: Executing actions
    Debug: sieve: uid=1027: Finished executing actions (status=ok, keep=implicit)
    Debug: sieve: uid=1027: Execute implicit keep (failure=no)
    Debug: sieve: uid=1027: Start storing into mailbox Junk
    Debug: sieve: uid=1027: Executing implicit keep action
    Debug: sieve: uid=1027: Execute storing into mailbox ‘Junk’
    Debug: sieve: uid=1027: Updated existing mail in mailbox ‘Junk’
    Debug: sieve: uid=1027: Finished executing implicit keep action (status=ok)
    Debug: sieve: uid=1027: Finalizing actions
    Debug: sieve: uid=1027: Finalize pipe action (status=ok, action_status=ok, commit_status=ok, pre-commit=yes)
    Debug: sieve: uid=1027: Commit pipe action
    Debug: program exec:/etc/dovecot/sieve/rspamd-learn-spam.sh: Establishing connection
    Debug: program exec:/etc/dovecot/sieve/rspamd-learn-spam.sh: Forked child process
    Debug: program exec:/etc/dovecot/sieve/rspamd-learn-spam.sh (168652): Connected to program
    Debug: program exec:/etc/dovecot/sieve/rspamd-learn-spam.sh (168652): Finished streaming payload to program
    Debug: program exec:/etc/dovecot/sieve/rspamd-learn-spam.sh (168652): Finished input to program
    Debug: program exec:/etc/dovecot/sieve/rspamd-learn-spam.sh (168652): Disconnected
    Debug: program exec:/etc/dovecot/sieve/rspamd-learn-spam.sh (168652): Waiting for program to finish after 0 msecs (timeout = 10000 msecs)
    Debug: program exec:/etc/dovecot/sieve/rspamd-learn-spam.sh (168652): Child process ended
    program exec:/etc/dovecot/sieve/rspamd-learn-spam.sh (168652): Terminated with non-zero exit code 1
    Debug: program exec:/etc/dovecot/sieve/rspamd-learn-spam.sh (168652): Destroy
    Error: sieve: failed to execute to program `rspamd-learn-spam.sh’: refer to server log for more information. [2023-02-24 17:03:47]
    “`
    To my less-trained eye everything looks correct until “Terminated with non-zero exit code”. Any clues as to what might be causing this? Thanks and Regards, P.

  16. After latest update users are reporting issues with receiving e-mails to specific mailboxes (not all of course). Logfile says:

    milter-reject: END-OF-MESSAGE from mail-xxxx.xxxxxxx.com[x.x.x.118]: 4.7.1 Ratelimit “to” exceeded

    I now get several miller-rejects for Ratelimit exceedings, not only “to”, but also “other_limit_alt” and some more. I have used the configuration described in this tutorial and it worked fine for more than a year without issues. I will get deeper into it, but if someone has experienced the same phenomena please share.

  17. Why do you use sieve-after for moving spam to Junk? I had a problem with user configured sieve filters executed on spam mail, so the spam actually wasn’t moved to Junk but “followed” the user defined filter rule to another folder. As this wasn’t the behavior we wanted I moved the rule to sieve-before and everything is working like expected now.

    1. Christoph Haas

      It probably depends on what the users expect. My personal use case is that I distribute emails to different folders for various mailing lists. That means that spam is also sent there. However the other way could mean that mails from lists would get junked accidentally.

  18. I have an address that started receiving spam and I would like any mail coming to that address to be silently dropped as I don’t use that address anymore. It was part of a leak a couple years ago. I’m not sure how I should achieve this. Could you give me some pointers?

    1. Christoph Haas

      There may be more sophisticated ways. This quick hack will do the job though:

      Add such an entry to /etc/aliases:

      devnull: /dev/null

      Run “newaliases” to load this new alias.

      Add an alias in virtual_aliases forwarding that email address to devnull@localhost. (This assumes that you have “localhost” defined as a local domain.)

      1. Thanks, I had come across that solution, but it wasn’t clear if this would discard the mail (and not send a response to the sending mailserver that our mailserver accepted the message, indicating that the address is “valid”).

        I have found another solution that is as followed:
        Create a file named /etc/postfix/blocked

        In main.cf append the virtual_alias_maps to include this file prepended with pcre: (eg virtual_alias_maps = mysql:/etc/postfix/mysql-virtual-alias-maps.cf,mysql:/etc/postfix/mysql-email2email.cf,pcre:/etc/postfix/blocked)

        In the file itself add the address you no longer wish to receive mails on, appended with discard: (eg compromised@yourdomain.com discard:)

        Restarting postfix should do the trick.

        I summarized it again here https://felixvandyck.be/kb/blog-page.php?page=postfix_reject_incoming_to_local_address.md

        I’m not sure why the stackoverflow post calls to use “discard:” (with colon appended), or which other options one could add. That would be a nice addition to my documentation.

  19. Do I still need autolearn = true; in /etc/rspamd/override.d/classifier-bayes.conf if I enabled per-user spam training (users_enabled = true; in /etc/rspamd/local.d/classifier-bayes.conf)?

  20. I am getting following in logs:

    Oct 08 07:05:55 imap(jack@mypostfix.com): Debug: sieve: include: sieve_global is not set; it is currently not possible to include `:global’ scripts

    And also:

    Oct 08 07:06:24 imap(jack@mypostfix.com): Debug: sieve: action pipe: socket path `/run/dovecot/sieve-pipe/rspamd-learn-spam.sh’ for program `rspamd-learn-spam.sh’ not found

    What am I missing???

    It couldn’t find socket

  21. Christoph Haas

    Hi Adam. Just guessing here, because I haven’t yet encountered that message. But have you set “sieve_pipe_bin_dir = /etc/dovecot/sieve”?

    1. Yes I have already set that:

      sieve_pipe_bin_dir = /etc/dovecot/sieve
      sieve_global_extensions = +vnd.dovecot.pipe
      sieve_plugins = sieve_imapsieve sieve_extprograms

  22. My ISPmail has been working well for a few years.
    For a few days, I have no longer been able to receive or send emails.
    I have this error :
    postfix/submission/smtpd[9078]: NOQUEUE: milter-reject: EHLO from localhost[127.0.0.1]: 451 4.7.1 Service unavailable
    postfix/smtpd[10315]: NOQUEUE: milter-reject: CONNECT from localhost[::1]: 451 4.7.1 Service unavailable – try again later; proto=SMTP
    postfix/smtpd[10315]: NOQUEUE: milter-reject: EHLO from localhost[::1]: 451 4.7.1 Service unavailable – try again later; proto=SMTP helo=

    My Rspamd daemon version is 2.7
    I restarted postfix, dovecot, rspamd but no change.

    When I disable on /etc/postfix/main.cf theses 2 lines, mails are ok
    #smtpd_milters = inet:127.0.0.1:11332
    #non_smtpd_milters = inet:127.0.0.1:11332
    milter_mail_macros = i {mail_addr} {client_addr} {client_name} {auth_authen}
    virtual_mailbox_maps = mysql:/etc/postfix/mysql-virtual-mailbox-maps.cf

    pgrep -a rspam
    6960 rspamd: main process
    6962 rspamd: rspamd_proxy process (localhost:11332)
    6963 rspamd: controller process (localhost:11334)
    6964 rspamd: normal process (localhost:11333)
    6965 rspamd: normal process (localhost:11333)

    What happens suddenly?

    1. II solved my problem by replacing 127.0.0.1 with localhost in these 2 lines :
      #smtpd_milters = inet:127.0.0.1:11332
      smtpd_milters = inet:localhost:11332
      #non_smtpd_milters = inet:127.0.0.1:11332
      non_smtpd_milters = inet:localhost:11332
      I don’t undestand why but all is well now !

  23. Hello CHRISTOPH!
    there are a lot of messages like this in the rspamd logs:
    2023-12-01 08:58:14 #2351130(normal) ; task; fuzzy_check_timer_callback: got IO timeout with server fuzzy1.rspamd.com:11335(135.181.136.158:11335), after 1 retransmits
    2023-12-01 09:00:32 #2351131(normal) ; task; fuzzy_check_timer_callback: got IO timeout with server fuzzy2.rspamd.com:11335(135.181.136.158:11335), after 1 retransmits

    Have you encountered this on your server? Any ideas on how to fix this?

Leave a Reply

Your email address will not be published. Required fields are marked *

Scroll to Top