[2019-12-02]

Filtering out spam with rspamd

You have come a long way in this guide and your mail server is already fully functional. Now it’s time to deal with the dark side: spam. The amount of spam nowadays has been insane for many years. So we need to detect spam emails and filter them out. I found that rspamd is a well-performing choice for that purpose. rspamd keeps a permanent process running on your mail server that listens to connections from Postfix using the milter (=mail filter) protocol. Every time an email enters your system, Postfix will send it to rspamd to have its content checked. rspamd runs a lot of checks on the email and computes a total score. The higher the score – the more likely the email should be considered unsolicited.

Make Postfix use rspamd

Let’s tell Postfix to send all incoming email through rspamd. Run these commands on the shell:

postconf smtpd_milters=inet:127.0.0.1:11332
postconf non_smtpd_milters=inet:127.0.0.1:11332
postconf milter_mail_macros="i {mail_addr} {client_addr} {client_name} {auth_authen}"

Postfix will connect to rspamd that is listening to TCP port 11332 on localhost (127.0.0.1) and pass the email over that connection. The smtpd_milters setting defines that connection for emails that came into the system from the internet via the SMTP protocol. The non_smtpd_milters setting is optional – it makes Postfix have all emails checked that originate from the system itself. Finally the milter_mail_macros setting defines several variables that rspamd expects for better spam detection. rspamd then runs its checks and tells Postfix whether the email should pass or get rejected.

Testing spam detection

For testing we can use a sample spam email that comes with SpamAssassin. It is called GTUBE (Generic Test for Unsolicited Bulk Email). It contains a certain articial pattern that is recognized as spam by SpamAssassin. Do you know EICAR.COM to test virus scanners? This is the same thing for spam.

I suggest that you download the file on the server:

wget http://spamassassin.apache.org/gtube/gtube.txt

…and send that email to our test user John…

sendmail john@example.org < gtube.txt

Check your /var/log/mail.log. You will find something like this:

Jan 13 09:21:01 mail postfix/cleanup[19945]: 95B7A42212: milter-reject: END-OF-MESSAGE from localhost[127.0.0.1]: 5.7.1 Gtube pattern; from=<root@mail.example.org> to=<john@example.org>
Jan 13 09:21:01 jen postfix/cleanup[19945]: 95B7A42212: to=<john@example.org>, relay=none, delay=0.18, delays=0.18/0/0/0, dsn=5.7.1, status=bounced (Gtube pattern)

The interesting parts are those printed in bold letters. “milter-reject” tells that the milter (rspamd) recommended to Postfix to reject the email. It gave the reason “5.7.1 Gtube pattern”. In mail communication you often find these three digit codes. They are defined in RFC 3463. The first digit is most important:

  • 2 = Success
  • 4 = Transient failure (temporary problem – come back later)
  • 5 = Permanent failure (do not try again in this way)

So 5.7.1 tells us that the result code is a permanent failure in delivery. If you looked up the RFC you would find that the 7 stands for an issue regarding the security policy. So it’s not a technical failure but instead a security-relevant component on the system has rejected the email. That’s what rspamd did. It even told us the reason: “Gtube pattern”. So you see that rspamd knows about the Gtube spam test pattern and reacts as expected.

Accordingly you won’t see this email in John’s inbox. This is a great advantage of using milters by the way. Imagine Postfix receiving a spam email and confirming its reception. What should it do when it finds out that it’s unwanted email? According to the SMTP protocol it must not throw away any emails. Would you create a bounce message telling the sender that you did not accept the email? That would be a bad idea because the sender address is very likely forged. You would send the bounce to an innocent person thus creating so called backscatter and make it even worse. So the right approach is to check the email while the sending server is still connected to your Postfix. This allows Postfix to reject the email with a 5.x.x error code and let the other side figure out what to do.

Score metrics (optional)

rspamd will however not reject all spam email. As it computes a score of the spam propability you can tell it which scores you would accept, flag as spam or make the incoming email get rejected. Take a look at the /etc/rspamd/actions.conf file. There are tons of scores defined for various conditions. At the beginning of the file you will find:

actions {
  reject = 15;
  add_header = 6;
  greylist = 4;
}

These are the default actions. If rspamd computes a score of at least 15 then the email will get rejected at the doorstep just like the Gtube pattern in the previous test. Any score above 6 will add header lines like “X-Spam: Yes” so that your mail software can detect them and maybe file the email to a different folder. Any score above 4 will trigger greylisting which is a mechanism that temporarily rejects the email with a 4.x.x code and waits if the sending server will try again. After a waiting time of a few minutes greylisting will accept the email. The idea is to reject email from systems that do not have a sending queue. Malware like on infected Wind*ws computers used to try sending an email just once which triggered greylisting and successfully rejected the spammer. But even malware programmers have learned and may try again after a few minutes thus circumventing greylisting. Your mileage may vary.

If you like to change these defaults then create a new file in /etc/rspamd/override.d/actions.conf containing your desired limits:

reject = 150;
add_header = 2;
greylist = 5;

This would virtually never reject an email. And it would flag any email with a score of at least 2 as spam. Greylisting would happen at a score above 5. These are not necessarily sane values – they are just meant as an example.

Please take a moment to understand how to change rspamd defaults. You can either create files in /etc/rspamd/local.d/… which will replace entire sections; or create files in /etc/rspamd/override.d/… which will change only small parts of the configuration. There is a helpful page in the rspamd documentation that contains examples. Whatever you do – never change the /etc/rspamd/* files directly because a software update will try to replace them.

Of course restart rspamd after any configuration changes:

systemctl restart rspamd

To check if rspamd has picked up your configuration use this command to see the current configuration:

rspamadm configdump

You may test your configuration using

rspamadm configtest

Alternatively you may check if all required rspamd processes are running…

pgrep -a rspam

11431 rspamd: main process                         
11555 rspamd: rspamd_proxy process (localhost:11332)
11557 rspamd: controller process (localhost:11334)
11559 rspamd: normal process (localhost:11333)

Adding headers

As you may know an email contains of the header and the body. Your users will only see header information like the subject, the sender, the recipient and the date and time the email was sent. But there is way more information like the router the email travelled or extended headers added by the various mail server on the way to the destination. Such extended headers begin with an “X-“. rspamd can add such headers to help you filter out spam. For that purpose create a new configuration override file at /etc/rspamd/override.d/milter_headers.conf with this content:

extended_spam_headers = true;

You would need to restart rspamd of course:

systemctl restart rspamd

As documented it will add these headers:

X-Rspamd-Server: mail
Authentication-Results: dmarc=fail reason="No valid SPF, No valid DKIM" …
X-Rspamd-Queue-Id: C22E55A005B3
X-Spamd-Result: default: False [11.55 / 15.00]
 R_PARTS_DIFFER(0.27)[63.4%]
 FROM_NO_DN(0.00)[]
 RCVD_COUNT_ZERO(0.00)[0]
 R_DKIM_NA(0.00)[]
 FUZZY_DENIED(12.00)[1:19305c7fdd:1.00:bin,1:35699594fd:0.91:txt]
 RBL_SENDERSCORE(2.00)[55.181.23.94.bl.score.senderscore.com]
 ARC_NA(0.00)[]
 RCPT_COUNT_ONE(0.00)[1]
 RCVD_TLS_ALL(0.00)[]
 FROM_EQ_ENVFROM(0.00)[]
 R_SPF_SOFTFAIL(0.00)[~all]
 BAYES_HAM(-2.71)[98.75%]
 TO_MATCH_ENVRCPT_ALL(0.00)[]
 MIME_GOOD(-0.10)[multipart/related,multipart/alternative,text/plain]
 MID_RHS_MATCH_FROM(0.00)[]
 ASN(0.00)[asn:16276, ipnet:94.23.0.0/16, country:FR]
 TO_DN_NONE(0.00)[]
 DMARC_POLICY_SOFTFAIL(0.10)[Chronopost.fr : No valid SPF, No valid DKIM,none]
 SUBJECT_ENDS_EXCLAIM(0.00)[]
X-Spam: Yes

No headers?

Please note that headers are only added by rspamd if you send the email from the outside of your server. If you use swaks locally then you will not get those headers.

Each of the uppercase symbols like FROM_HAS_DN means that a certain detection routing of rspamd was triggered. It does not necessarily mean something bad about the email. For example R_SPF_ALLOW has a negative score that lowers the total score because it is something good about the email. There are a several symbols with a 0.00 score. These do not change the score but show you what rspamd has found. But if you consider certain criteria good or bad then you can define your own scores for them.

The last line here is especially interesting because next on our list is…

Sending spam to the Junk folder

Your users will not realize that their spam emails have an added “X-Spam: Yes” header. It it not actively shown in their mail client. Nor does it move the email out of the inbox into their spam folder. Such emails just appear like normal in their inbox. So let’s aid them by moving spam to a separate Junk folder beneath their inbox automatically. Dovecot has support for Sieve filters which are scripts that run automatically whenever an email comes in.

John could create a new Sieve filter (e.g. using the Roundcube webmail interface) for himself that would save any emails to his “Junk” folder if the header line “X-Spam: Yes” was found. This rule would be useful for all your users though so let’s find a general solution.

Dovecot supports global Sieve filters that apply to all users. Edit the file /etc/dovecot/conf.d/90-sieve.conf. Look for the “sieve_after” lines. They are commented out. Add a new line there:

sieve_after = /etc/dovecot/sieve-after

The “sieve after” filters are executed after the users’ filters. John can define his own filter rules. And after that Dovecot will run any filter rules it finds in files in /etc/dovecot/sieve-after. That is just an arbitrary directory that you create:

mkdir /etc/dovecot/sieve-after

And add a new file /etc/dovecot/sieve-after/spam-to-folder.sieve reading:

require ["fileinto","mailbox"];

if header :contains "X-Spam" "Yes" {
 fileinto :create "Junk";
 stop;
}

Separatist warning

If your /var/vmail directory tree is (still) using the “.” separator then you need to call the destination folder “INBOX.Junk” instead of “Junk”.

The “require” lines include functionality to move emails into certain folders (fileinto) and to create folders if they don’t exist yet (mailbox). Then if rspamd marked an email as spam it gets moved into the INBOX.Junk folder which just appears as “Junk” to the user underneath their inbox.

Dovecot cannot deal with such human-readable files though. So we need to compile it:

sievec /etc/dovecot/sieve-after/spam-to-folder.sieve

That generated a machine-readable file /etc/dovecot/sieve-after/spam-to-folder.svbin.

Now all your users will automatically get spam emails moved to their Junk folder. Nice – isn’t it?

/etc/dovecot/sieve only

Please do not put global Sieve files anywhere else. Dovecot only obeys files there. The respective configuration parameter is called sieve_pipe_bin_dir that defines where such Sieve files are expected.

Training the spam detection

One of rspamd’s features is analyzing word patterns using probability theory. That functionality is contained in its “statistical module“. (Yes, the name is pretty misleading.)

(a) No Bayes training

You can start with an empty training database. This is not as bad as it sounds. rspamd has way more functionality to determine if an email is ham or spam. Autolearning takes email that are surely ham or spam and uses them to train the spam filter. The rspamd documentation has further examples how to fine-tune auto learning. After a few hundred emails the training will contribute towards a better detection rate.

If you want to use autolearning just create a new file /etc/rspamd/override.d/classifier-bayes.conf and make it contain:

autolearn = true;

(b) Migrating training data from previous mail server

If your previous mail server used rspamd on Debian Stretch then you can migrate over the training data. Stop the rspamd process on both the old and the new server so that the SQLite databases are closed properly…

systemctl stop rspamd

…and copy over all files like /var/lib/rspamd/*.sqlite to your new mail server.

Set the ownership of the database files on the new server correctly:

chown _rspamd._rspamd /var/lib/rspamd/*sqlite

Then restart rspamd on the both servers:

systemctl start rspamd

Verify that the database is now filled by running…

rspamc stat

At the end of the output you will see something like…

Statfile: BAYES_SPAM type: sqlite3; length: 36.59M; free blocks: 0; total blocks: 497.23k; free: 0.00%; learned: 3622; users: 233; languages: 14
Statfile: BAYES_HAM type: sqlite3; length: 28.00M; free blocks: 0; total blocks: 374.15k; free: 0.00%; learned: 31913; users: 97; languages: 6
Total learns: 35535

(c) Training from your existing ham and spam emails

Have you been running a mail server with mailboxes in a Malidir structure before but without rspamd? Then you probably have a good amount of ham and spam emails. Let’s use those to train rspamd. It is important to train both ham and spam emails. The rspamc command will allow you to feed entire directories/folders of emails to the learning process. An example to train spam:

rspamc learn_spam /var/vmail/example.org/john/Maildir/.INBOX.Junk/cur

And this would be an example to train ham from John’s inbox:

rspamc learn_ham /var/vmail/example.org/john/Maildir/cur

Of course the quality of spam detection will depend on how good the source data is. So do not train spam emails from users who randomly put mails into their Junk folder that is not actually spam.

Check the number of emails you learned by running…

rspamc stat

Bayes spam checking will not work before it learned at least 200 spam and ham emails. Teaching rspamd fewer emails or just spam emails will not work. This is defined by the min_learns variable defined in /etc/rspamd/statistic.conf.

In the output you will find lines beginning with “Statfile” like these…

Statfile: BAYES_SPAM type: sqlite3; length: 41.78M; free blocks: 0; total blocks: 0; free: 0.00%; learned: 0; users: 1; languages: 1
Statfile: BAYES_HAM type: sqlite3; length: 51.12M; free blocks: 0; total blocks: 0; free: 0.00%; learned: 0; users: 1; languages: 1
Total learns: 0

This is what you usually start with. The more emails you feed into the training process the better the detection rate will be. Some emails however may not be long enough or be too similar to previously trained emails. So don’t worry if you are training 1000 emails but just get a count of 500 emails here.

Per-user spam training

rspamd allows you to train the spam detection per user. It would not keep a global training database that applies to all users. Instead each user gets their own training.

Advantage: users work differently. Some have subscribed to a sales newsletter and now believe that marking it as spam gets them unsubscribed. Yes, that’s stupid but can thoroughly confuse the spam detection. Also you might be very interested in viagr* product informatoin while others do not.

Disadvantage: training still requires many ham and spam mails before it has any effect. So unless a user gets 200 samples of good and evil emails the spam detection cannot work.

If you decide you want to use per-user spam training then add/edit the file /etc/rspamd/override.d/statistic.conf /etc/rspamd/local.d/classifier-bayes.conf and insert:

users_enabled = true;

Better scaling with Redis

So far rspamd has stored its training (and other) data in SQLite databases. If you haven’t heard of it: it’s a database system much like MariaDB but it does not have a server process that you connect to. Instead you access the database files directly using either the “sqlite3” command or using libraries that integrate it SQLite into other programming languages. It is very simple to use. But compared to MariaDB or PostgreSQL its performance is limited. SQLite is completely sufficient for medium-size mail servers. But if you are going big you should instead use Redis as a data store.

Redis is also a kind of database system. It is way more limited than a traditional SQL database. But it is lightning fast the way it works. On my aged server it handles around 50,000 requests per second. It gets it speed from keeping the data in memory. So it doesn’t access the disk to fetch information. (Yet it saves its data to disk frequently to prevent data loss.) People use Redis as a cache or for very fast lookups of simple data structures. And so does rspamd.

Installing Redis is simple:

apt install redis

That’s it. No more configuration needed. Redis mainly consists of a daemon process called redis-server that listens for TCP connections on localhost on port 6379. Enabling Redis in rspamd is simple. Add a file /etc/rspamd/override.d/redis.conf and insert:

servers = "127.0.0.1";

Now you need to configure which rspamd modules should use Redis as a backend storage. Several modules support Redis. Just as an example let’s put the spam training data of the statistic module into Redis. Edit the file /etc/rspamd/override.d/statistic.conf and set the backend there. If you also use per-user spam training this file should now look like this:

classifier "bayes" {
   users_enabled = true;
   backend = "redis";
 }

Feel free to use Redis for other lookups, too.

Learning from user actions

Now we are getting to something really cool. Let’s tell Dovecot that moving emails into the Junk folder teaches rspamd instantly that the email is spam. And train an email as ham if it is moved out of the Junk folder. We will add triggers (actually “sieve scripts“) to the action of moving emails via IMAP.

The currently recommended way is to use the “IMAPSieve” plugin instead. There is nothing to install – it comes with the Dovecot packages. We just need to configure it.

First order of business is enabling the IMAPSieve plugin for the IMAP protocol/service in Dovecot. Edit the /etc/dovecot/conf.d/20-imap.conf file and look for the line reading “mail_plugins”. Turn it into:

mail_plugins = $mail_plugins imap_sieve

We also need to edit Dovecot’s Sieve configuration to enable two plugins that are required for our task. Sieve is a scripting language that automates things in conjunction with emails and folders. Edit the file /etc/dovecot/conf.d/90-sieve.conf and put these lines into the plugin {…} section:

# From elsewhere to Junk folder
imapsieve_mailbox1_name = Junk
imapsieve_mailbox1_causes = COPY
imapsieve_mailbox1_before = file:/etc/dovecot/sieve/learn-spam.sieve

# From Junk folder to elsewhere
imapsieve_mailbox2_name = *
imapsieve_mailbox2_from = Junk
imapsieve_mailbox2_causes = COPY
imapsieve_mailbox2_before = file:/etc/dovecot/sieve/learn-ham.sieve

sieve_pipe_bin_dir = /etc/dovecot/sieve
sieve_global_extensions = +vnd.dovecot.pipe
sieve_plugins = sieve_imapsieve sieve_extprograms

The first rule tells Dovecot to run the Sieve rules as defined in the /etc/dovecot/sieve/learn-spam.sieve file whenever an email is moved into a user’s “Junk” folder. We will create that Sieve script in a minute.

The second rule sets the other way. Whenever an email is moved from the “Junk” folder to any (*) folder then the /etc/dovecot/sieve/learn-ham.sieve Sieve script is called.

The “sieve_pipe_bin_dir” setting defines where executable scripts are allowed to reside. We will put our simple learning scripts there. And finally the “sieve_global_extensions” setting enables the pipe plugin that allows sending email to external commands.

Separators

If you come from previous versions of this guide, your mail directories may still use dots (“.”) instead of paths (“/”) to compose the folder structure. A clear sign is that you find directories like “INBOX.Junk” instead of just “Junk” in mail directories within /var/vmail. In that case replace the word “Junk” in the section above by “INBOX.Junk”.

Next up let’s create the Sieve scripts that we told Dovecot about. Create a new directory /etc/dovecot/sieve to put our new files in:

mkdir /etc/dovecot/sieve

Then create the file /etc/dovecot/sieve/learn-spam.sieve and let it contain:

require ["vnd.dovecot.pipe", "copy", "imapsieve"];
pipe :copy "rspamd-learn-spam.sh";

Let’s do the same for /etc/dovecot/sieve/learn-ham.sieve

require ["vnd.dovecot.pipe", "copy", "imapsieve", "variables"];
if string "${mailbox}" "Trash" {
  stop;
}
pipe :copy "rspamd-learn-ham.sh";

The above Sieve script avoids training an email as ham if the user moves it to the Trash folder. After all if you clear your Junk folder you do not want to train your spam as regular emails.

Restart Dovecot:

systemctl restart dovecot

These two scripts need to be compiled – that is turning them into machine-readable code:

sievec /etc/dovecot/sieve/learn-spam.sieve
sievec /etc/dovecot/sieve/learn-ham.sieve

This creates two new files “learn-ham.svbin” and “learn-spam.svbin” that look like gibberish inside but are now in a format that Dovecot’s Sieve plugin can understand.

Let’s fix the permissions of these files, too, while we are at it:

chmod u=rw,go= /etc/dovecot/sieve/learn-{spam,ham}.{sieve,svbin}
chown vmail.vmail /etc/dovecot/sieve/learn-{spam,ham}.{sieve,svbin}

And the last step is to create the simple shell scripts that do the actual spam/ham training. The first file is /etc/dovecot/sieve/rspamd-learn-spam.sh which contains:

#!/bin/sh
exec /usr/bin/rspamc learn_spam

That looks simple, doesn’t it? Nothing more is actually needed. The spam email is piped to this script and rspamd learns it as a spam email and adjusts its spam detection database accordingly.

The second script teaches ham and is called /etc/dovecot/sieve/rspamd-learn-ham.sh. Make it contain:

#!/bin/sh
exec /usr/bin/rspamc learn_ham

These two shell scripts must be executable so go fix that:

chmod u=rwx,go= /etc/dovecot/sieve/rspamd-learn-{spam,ham}.sh
chown vmail.vmail /etc/dovecot/sieve/rspamd-learn-{spam,ham}.sh

I hope you haven’t lost your mind yet. It’s really just a chain of things to happen. Let’s reiterate how this process works:

  1. a user moves a spam email into their “Junk” folder
  2. Dovecot realizes that this triggers the Sieve rule “imapsieve_mailbox1” so it calls the Sieve script /etc/dovecot/sieve/learn-spam.sieve (in fact the *.svbin version of the script)
  3. Sieve will take the email and send (“pipe”) it to the executable rspamd-learn-spam.sh shell script
  4. the script in turn runs the email through the “/usr/bin/rspamc learn_spam” command

This works equally for the other way or learning ham emails of course.

I am sure you are eager to try it out. Yes, it’s about time. However to see that it actually works I suggest you edit the /etc/dovecot/conf.d/10-logging.conf file and set “mail_debug=yes”. That will add a lot more detail to the /var/log/mail.log file but on a busy server may also lead to headaches. 🙂

Restart Dovecot…

systemctl restart dovecot

…and watch the /var/log/mail.log file…

tail -f /var/log/mail.log

Now open your IMAP client (Thunderbird, Evolution, Roundcube, mutt or whatever you prefer) and drag an email to your Junk folder. The mail log should read something along the lines of…

imap(john@example.org): Debug: imapsieve: mailbox INBOX.Junk: MOVE event
imap(john@example.org): Debug: imapsieve: Matched static mailbox rule [1]
imap(john@example.org): Debug: sieve: file storage: Using Sieve script path: /etc/dovecot/sieve/learn-spam.sieve
imap(john@example.org): Debug: sieve: file script: Opened script learn-spam' from/etc/dovecot/sieve/learn-spam.sieve'
imap(john@example.org): Debug: sieve: Opening script 1 of 1 from /etc/dovecot/sieve/learn-spam.sieve' imap(john@example.org): Debug: sieve: Loading script /etc/dovecot/sieve/learn-spam.sieve imap(john@example.org): Debug: sieve: Script binary /etc/dovecot/sieve/learn-spam.svbin successfully loaded imap(john@example.org): Debug: sieve: binary save: not saving binary /etc/dovecot/sieve/learn-spam.svbin, because it is already stored imap(john@example.org): Debug: sieve: Executing script from/etc/dovecot/sieve/learn-spam.svbin'
imap(john@example.org): Debug: sieve: action pipe: running program: rspamd-learn-spam.sh
imap(john@example.org): Debug: Mailbox INBOX.Junk: Opened mail UID=3978 because: mail stream
imap(john@example.org): Debug: waiting for program /etc/dovecot/sieve/rspamd-learn-spam.sh' to finish after 0 msecs imap(john@example.org): sieve: pipe action: piped message to programrspamd-learn-spam.sh'

Look for errors and warnings. If at the end you see that Dovecot called the “rspamd-learn-spam.sh” script then you should be fine.

And finally if you pull an email out of the “Junk” folder you should see mailbox rule [2] be called and the email being learned as ham:

imap(john@example.org): Debug: imapsieve: mailbox INBOX: MOVE event
imap(john@example.org): Debug: imapsieve: Matched static mailbox rule [2]
imap(john@example.org): Debug: sieve: file storage: Using Sieve script path: /etc/dovecot/sieve/learn-ham.sieve
imap(john@example.org): Debug: sieve: file script: Opened script learn-ham' from/etc/dovecot/sieve/learn-ham.sieve'
imap(john@example.org): Debug: sieve: Opening script 1 of 1 from /etc/dovecot/sieve/learn-ham.sieve' imap(john@example.org): Debug: sieve: Loading script /etc/dovecot/sieve/learn-ham.sieve imap(john@example.org): Debug: sieve: Script binary /etc/dovecot/sieve/learn-ham.svbin successfully loaded imap(john@example.org): Debug: sieve: binary save: not saving binary /etc/dovecot/sieve/learn-ham.svbin, because it is already stored imap(john@example.org): Debug: sieve: Executing script from/etc/dovecot/sieve/learn-ham.svbin'
imap(john@example.org): Debug: sieve: action pipe: running program: rspamd-learn-ham.sh
imap(john@example.org): Debug: Mailbox INBOX: Opened mail UID=28412 because: mail stream
imap(john@example.org): Debug: waiting for program /etc/dovecot/sieve/rspamd-learn-ham.sh' to finish after 0 msecs imap(john@example.org): sieve: pipe action: piped message to programrspamd-learn-ham.sh'

That’s it. Nifty, isn’t it?

Autoexpunge

Andi Olsen pointed out that Dovecot has introduced a feature to automatically delete emails in a folder that reach a certain age. This is especially useful for the “Trash” and “Junk” folders. To enable this feature just edit the /etc/dovecot/conf.d/15-mailboxes.conf file and add the autoexpunge parameter where desired. Example:

mailbox Junk {
  special_use = \Junk
  auto = subscribe
  autoexpunge = 30d
}
mailbox Trash {
  special_use = \Trash
  auto = subscribe
  autoexpunge = 30d
}

Restart Dovecot after any configuration change.

Logging

rspamd keeps a verbose log of its actions in /var/log/rspamd/rspamd.log. If a user complains that a certain email got blocked or at least flagged as spam then take a look at this log. You can match the /var/log/mail.log with it by comparing the Postfix queue ID. Those are the 12-digit hexadecimal number like “95CE05A00547“. Those IDs can be found in the rspamd.log, too:

2018-01-14 06:39:45 #10424(normal) <40985d>; task; rspamd_task_write_log: id: <undef>, qid: <95CE05A00547>,  ip: 12.13.51.194, from: <…>, (default: F (no action):  [3.40/15.00]  [MISSING_MID(2.50){},MISSING_DATE(1.00){},MIME_GOOD(-0.10){text/plain;},ARC_NA(0.00){},ASN(0.00){asn:8220,  ipnet:212.123.192.0/18,  country:GB;},FROM_EQ_ENVFROM(0.00){},FROM_NO_DN(0.00){},RCPT_COUNT_ONE(0.00){1;},RCVD_COUNT_ZERO(0.00){0;},RCVD_TLS_ALL(0.00){},TO_DN_NONE(0.00){},TO_DOM_EQ_FROM_DOM(0.00){},TO_MATCH_ENVRCPT_ALL(0.00){}]),  len: 181, time: 16.000ms real, 6.385ms virtual, dns req: 0, digest:  <69b289a82827c11f759837c033cd800a>, rcpts: <…>, mime_rcpt:  <…>

Scan outgoing emails, too

Previously I accidentlly recommended to set smtpd_milters to an empty value to prevent scanning outgoing emails. But that was wrong. In fact you want to use rspamd to work on outgoing emails. rspamd determines automatically if an authenticated user wants to send an email out and behaves differently. For example it does not check for IP blacklists or DKIM signatures. Instead it adds a DKIM signature.

So please don’t use “-o smtpd_milters=” anywhere. Sorry for the confusion.

The web interface

rspamd’s web interface

rspamd comes with a neat bonus feature: a web interface. It allows you to check emails for spam, get statistics and fine-tune scores. It is already installed and enabled by default and expects HTTP requests on port 11334 on the localhost interface. I suggest you add a simple proxy configuration to your already working HTTPS-enabled web mail configuration to get access.

First you need to enable Apache’s modules for HTTP proxying and rewriting:

a2enmod proxy_http
a2enmod rewrite

You can either create a new virtual host configuration or just edit the /etc/apache2/sites-available/webmail.example.org-https.conf file. Anywhere within the VirtualHost tags add:

<Location /rspamd>
  Require all granted   
</Location>

RewriteEngine On
RewriteRule ^/rspamd$ /rspamd/ [R,L]
RewriteRule ^/rspamd/(.*) http://localhost:11334/$1 [P,L]

This piece of configuration will forward any requests to https://webmail.example.org/rspamd to localhost:11334 and thus give you access to the rspamd web interface.

The interface is password protected. Let’s generate a new access password:

pwgen 15 1

This gives you a password like “eiLi1lueTh9mia4”. You could put that password in an rspamd configuration file. But cleartext passwords in configuration files are not quite elegant. Let’s create a hash of the password:

rspamadm pw
Enter passphrase: …
$2$icoahes75e7g9wxapnrbmqnpuzjoq7z…

Feed it the password that pwgen created for you and you will get a long hashed password. This procedure by the way is also documented on the rspamd pages.

Create a new configuration file /etc/rspamd/local.d/worker-controller.inc and put your hashed password in there:

password = "$2$icoahes75e7g9wxapnrbmqnpuzjoq7z…"

There is a nasty bug in the rspamd package in Debian Buster that breaks displaying of the charts. The Javascript D3 library that is used for that purpose was forgotten to get installed. Fortunately you just need to extract one Javascript file from the D3 release ZIP file to work around it:

wget https://github.com/d3/d3/releases/download/v5.15.0/d3.zip
unzip -d /usr/share/rspamd/www/js/lib d3.zip d3.min.js
rm d3.zip

That’s it for the configuration. Finally restart both rspamd and Apache to load your changed configuration:

systemctl restart rspamd
systemctl restart apache2

If everything went as expected you should now be able to access the rspamd web interface at https://webmail.example.org/rspamd

No graphs?

I could not get the (pie) charts working on the test server. Maybe that is a bug in rspamd 1.8.1. If anyone gets it working please leave a comment.

Anything else?

If you consider using rspamd for a larger environment then please take your time reading the good amount of documentation they provide. rspamd scales very well but requires additional setup like using Redis for faster spam analysis across multiple mail gateways.

Oh, and a…

Privacy Warning

rspamd “phones home” for its fuzzy checks. If that makes you feel bad you may want to disable that feature. Besides rspamd wants your money if you make money with your mail server and scan more than half a million emails per day.

92 thoughts on “Filtering out spam with rspamd

  • 2019-12-29 at 10:05
    Permalink

    When I try sievec /etc/dovecot/sieve/learn-spam.sieve

    I get the following error:
    sievec(root): Warning: sieve: ignored unknown extension ‘vnd.dovecot.pipe’ while configuring available extensions
    learn-spam: line 1: error: require command: unknown Sieve capability `vnd.dovecot.pipe’.
    learn-spam: line 1: error: require command: unknown Sieve capability `imapsieve’.
    learn-spam: line 2: error: unknown command ‘pipe’ (only reported once at first occurrence).
    learn-spam: error: validation failed.
    sievec(root): Fatal: failed to compile sieve script ‘/etc/dovecot/sieve/learn-spam.sieve’

    dovecot-sieve and dovecot-managedsieve are installed and I setup the 20-imap.conf with mail_plugins = $mail_plugins imap_sieve

    Reply
    • 2019-12-29 at 10:19
      Permalink

      In 90-sieve.conf for sieve_plugins the following must be set:
      sieve_plugins = sieve_extprograms sieve_imapsieve

      After this it works. 🙂

      Reply
      • 2019-12-29 at 11:47
        Permalink

        Indeed. And I put that into the configuration file on my test system, too. Wonder why that didn’t make it here. 🙂 Thanks.

        Reply
        • 2019-12-30 at 15:36
          Permalink

          root@debian-4gb-fsn1-1:~# sievec /etc/dovecot/sieve/learn-spam.sieve
          learn-spam: line 1: error: require command: unknown Sieve capability `vnd.dovecot.pipe’.
          learn-spam: line 1: error: require command: unknown Sieve capability `imapsieve’.
          learn-spam: line 2: error: unknown command ‘pipe’ (only reported once at first occurrence).
          learn-spam: error: validation failed.

          In my trial sieve_plugins = sieve_extprograms sieve_imapsieve was already set before getting to this error. Help ?

          Reply
          • 2019-12-30 at 17:22
            Permalink

            Got the same error, even tough I changed 90-sieve.conf.
            After a reload of dovecot it worked. @Christoph: you should a reload after touchind the dovecot files.

          • 2019-12-30 at 17:38
            Permalink

            Hiya Gompali,

            Should be a restart of dovecot before compiling

            systemctl restart dovecot

            Fixed it for me.

            LadyLinux

          • 2019-12-30 at 18:57
            Permalink

            Ok, thks 😉

          • 2019-12-31 at 11:11
            Permalink

            Thanks, I have edited the page to reload before using the pipe plugin.

  • 2019-12-31 at 18:18
    Permalink

    Can you advise on the best way to combine a catch-all address (without using aliases) with the sieve-after forward rule?

    I have one catch-all email domain that needs to be forwarded on to a gmail business account, but i want the forward to happen after rspamd. (I tried tricking postfix mailbox-maps, but dovecot complains the account doesn’t exist before the sieve-after runs). Any advice? Thanks so much

    Reply
    • 2020-01-01 at 11:47
      Permalink

      What about forwarding the catchall address to another user in your domain? And then forward that using the user’s sieve rule to Google?

      Okay, that was confusing. What I mean:

      1. Add a virtual_aliases entry for “@mydomain.com” to “all-the-stuff@mydomain.com”.
      2. Create a user “all-the-stuff@mydomain.com” in virtual_users.
      3. Login is through webmail as “all-the-stuff@mydomain.com” (Roundcube) and add a Sieve rule to forward all incoming email to “business-account@gmail.com”.

      That way you combined the advantage of having a catchall account with the ability to use Sieve.

      Reply
      • 2020-01-01 at 17:54
        Permalink

        Thanks Christoph. I was thinking that … although I wondered whether it’s still possible to maintain the reply-to address. I’ll do some further experimentation with this. Much obliged.

        Reply
        • 2020-01-01 at 19:35
          Permalink

          No problem. I would expect that sender and recipient are unchanged indeed.

          Reply
          • 2020-01-22 at 03:20
            Permalink

            Just reporting back for the benefit of others… this solution is working beautifully!

  • 2020-01-03 at 20:47
    Permalink

    I think, should be either “add_header = 2;” in the config snippet or “And it would flag any email with a score of at least 0 as spam.” in the text 🙂

    Reply
    • 2020-01-06 at 17:01
      Permalink

      You are right. Fixed. Thanks.

      Reply
    • 2020-01-06 at 17:03
      Permalink

      Good find. I already wondered why graphs suddenly stopped working on my test system but I suspected a typo on my own side. I’ll add the workaround for the bug right away.

      Reply
  • 2020-01-05 at 18:10
    Permalink

    Hey Christoph,

    I’ve found two problems that are not explained in this section.

    1. For the /etc/dovecot/sieve-after/spam-to-folder.sieve
    you used the old specs for “fileinto :create “INBOX.Junk”;”
    I think it’s a good idea to give a warning at this point for old/new version differences.

    The new variant should look like this:
    require [“fileinto”,”mailbox”];

    if header :contains “X-Spam” “Yes” {
    fileinto :create “Junk”;
    stop;
    }

    2. When the learn-{spam,ham}.svbin files get created the owner/group is root at default.
    So we need to make a chmod and chown for these files also. When the files are owned by
    root the sieve files couldn’t be used by rspamd.

    chmod u=rw,go= /etc/dovecot/sieve/learn-{spam,ham}.svbin
    chown vmail.vmail /etc/dovecot/sieve/learn-{spam,ham}.svbin

    Reply
    • 2020-01-06 at 17:21
      Permalink

      Thanks. I have added a huge warning for the folder naming depending on the separator. Once it again it shows that I my test system still runs on the old separator scheme. 🙂

      Reply
    • 2020-05-14 at 18:11
      Permalink

      Just for the record, the spam-to-folder.sieve has the same problem; needs the owner/permissions adjusted.

      Reply
  • 2020-01-09 at 13:18
    Permalink

    Hi,
    i like the new auto learning feature when moving messages to another folder. Great, BUT!
    If i empty my spam/junk folder the messages are moved to the Trash folder and then these messages are treated as “moved out from spam” and learned as ham. Currently i disabled the rule moving out from spam. Maybe you find a solution.

    Reply
    • 2020-01-09 at 17:09
      Permalink

      Thanks for spotting that. It probably depends on the email client or its settings on how to manage trashing spam. I set my Junk folder to “keep 30 days and then delete directly”. But to be honest I have not checked yet if my users do it similiarly. I’m open to suggestions. I took the inspiration from https://wiki2.dovecot.org/HowTo/AntispamWithSieve

      Reply
      • 2020-01-10 at 10:14
        Permalink

        There is a recommendation for that use case on the site you linked to. Simply search for “Trash”. There is a line added to “report-ham.sieve” that prevents further processing when an email is moved to the trash folder.

        if string “${mailbox}” “Trash” {
        stop;
        }

        Reply
        • 2020-01-14 at 11:48
          Permalink

          Works well. I have added that.

          Reply
        • 2020-01-22 at 03:21
          Permalink

          Thanks for this! I just noticed the same thing in my logs, and was going to share back. Love to see that the guide is already updated with a perfect solution. Thanks to you both!

          Reply
  • 2020-01-15 at 22:54
    Permalink

    Thanks for all !
    There is a problem with this added :
    sievec /etc/dovecot/sieve/learn-ham.sieve
    learn-ham: line 2: error: unknown test ‘string’ (only reported once at first occurrence).
    learn-ham: error: validation failed.
    sievec(root): Fatal: failed to compile sieve script ‘/etc/dovecot/sieve/learn-ham.sieve’

    Reply
    • 2020-01-17 at 14:32
      Permalink

      In /etc/dovecot/sieve/learn-ham.sieve, add “environment”, “variables” to the require line so it appears as require [“vnd.dovecot.pipe”, “copy”, “imapsieve”, “environment”, “variables”];

      Not sure if both are required but they are shown in the example code Christoph refers to in https://wiki2.dovecot.org/HowTo/AntispamWithSieve.

      HTH

      Reply
    • 2020-01-19 at 00:05
      Permalink

      I get the same error 🙁
      so frustrating :(((

      Reply
      • 2020-01-20 at 12:08
        Permalink

        Sorry, that was an error on my side. Just fixed the documentation, too.

        Reply
    • 2020-01-19 at 00:10
      Permalink

      just found the solution !

      replace require [“vnd.dovecot.pipe”, “copy”, “imapsieve”];
      with require [“vnd.dovecot.pipe”, “copy”, “imapsieve”, “variables”];

      Reply
      • 2020-01-19 at 10:07
        Permalink

        Thanks, there isn’t error now but same problem (deleted spam are added ham !):
        Soluce with this : /etc/dovecot/sieve/learn-ham.sieve (https://wiki2.dovecot.org/HowTo/AntispamWithSieve)
        require [“vnd.dovecot.pipe”, “copy”, “imapsieve”, “environment”, “variables”];
        if environment :matches “imap.mailbox” “*” {
        set “mailbox” “${1}”;
        }
        if string “${mailbox}” “Trash” {
        stop;
        }
        if environment :matches “imap.user” “*” {
        set “username” “${1}”;
        }
        pipe :copy “sa-learn-ham.sh” [ “${username}” ];

        Reply
      • 2020-01-19 at 17:48
        Permalink

        Ok, no error but the same issue when you delete a spam :
        sieve: pipe action: piped message to program `rspamd-learn-ham.sh’

        Why don’t you this :https://wiki2.dovecot.org/HowTo/AntispamWithSieve

        /etc/dovecot/sieve/learn-ham.sieve

        require [“vnd.dovecot.pipe”, “copy”, “imapsieve”, “environment”, “variables”];

        if environment :matches “imap.mailbox” “*” {
        set “mailbox” “${1}”;
        }

        if string “${mailbox}” “Trash” {
        stop;
        }

        if environment :matches “imap.user” “*” {
        set “username” “${1}”;
        }

        pipe :copy “rspamd-learn-ham.sh” [ “${username}” ];

        Reply
  • 2020-01-16 at 02:59
    Permalink

    I know we should only use the stable packages of debian buster but after configuring everything above for rspamd the charts were still not showing on the rspamd web interface.

    Online I found advise to upgrade to the latest version. What i instead did was adding below commands. Which didn’t upgrade to the latest version but installed missing files.

    After this the web charts are working.

    apt install -y lsb-release wget
    wget -O- https://rspamd.com/apt-stable/gpg.key | apt-key add –
    echo “deb http://rspamd.com/apt-stable/ $(lsb_release -c -s) main” > /etc/apt/sources.list.d/rspamd.list
    echo “deb-src http://rspamd.com/apt-stable/ $(lsb_release -c -s) main” >> /etc/apt/sources.list.d/rspamd.list

    apt update && apt upgrade -y

    Reply
    • 2020-01-20 at 09:52
      Permalink

      Thanks it solved the missing graph issue.

      Reply
    • 2020-01-20 at 12:22
      Permalink

      If you follow the instructions above you can fix the charts issue without breaking… erm… upgrading your system. 🙂

      Reply
      • 2020-01-30 at 18:58
        Permalink

        You are right, of course.
        I was happy the charts problem “was fixed”.

        Charts still didn’t work after:
        wget https://github.com/d3/d3/releases/download/v5.15.0/d3.zip
        unzip -d /usr/share/rspamd/www/js/lib d3.zip d3.min.js
        rm d3.zip

        Now i have a bigger problem:
        program `/etc/dovecot/sieve/rspamd-learn-spam.sh’ terminated with non-zero exit code 1
        Error: sieve: pipe action: failed to execute to program `rspamd-learn-spam.sh’: refer to server log for more information. [2020-01-30 14:14:23]
        sieve: left message in mailbox ‘Junk’
        Error: sieve: Execution of script /etc/dovecot/sieve/learn-spam.sieve failed
        imap-login: Login: user=, method=PLAIN, rip=192.168.168.4, lip=192.168.166.24, mpid=10366, TLS, session=
        program `/etc/dovecot/sieve/rspamd-learn-ham.sh’ terminated with non-zero exit code 1
        Error: sieve: pipe action: failed to execute to program `rspamd-learn-ham.sh’: refer to server log for more information. [2020-01-30 14:15:07]
        sieve: left message in mailbox ‘INBOX’
        Error: sieve: Execution of script /etc/dovecot/sieve/learn-ham.sieve failed

        Reply
        • 2020-03-14 at 20:18
          Permalink

          Hello,
          I got the exact same problem, like in you (since the beginning, just found out yet).
          Do you found out, why that happens? Just checked all logs and “rspamd.log” shows this:

          2020-03-14 20:51:28 #17033(controller) ; csession; rspamd_mime_part_detect_language: detected part language: en
          2020-03-14 20:51:28 #17033(controller) ; csession; rspamd_stat_classifiers_process: skip statistics as SPAM class is missing
          2020-03-14 20:51:28 #17033(controller) ; csession; rspamd_task_process: learn error: Unknown statistics error, found when storing data on backend; classifier: (null) 2020-03-14 20:51:28 #17033(controller) ; csession; rspamd_controller_learn_fin_task: cannot learn : Unknown statistics error, found when storing data on backend; classifier: (null)

          And “rspamadm configtest” shows this:

          2020-03-14 21:15:38 #0(main) ; symcache; rspamd_symcache_validate: symbol ‘BAYES_SPAM’ has its score defined but there is no corresponding rule registered
          2020-03-14 21:15:38 #0(main) ; symcache; rspamd_symcache_validate: symbol ‘BAYES_HAM’ has its score defined but there is no corresponding rule registered
          syntax OK

          Reply
  • 2020-01-23 at 21:21
    Permalink

    In Case someone else is struggling like I was, there are continuous typos on the /etc/dovecot/sieve/learn-ham.sieve example.

    There is a difference on ” and “. I got lots of error messages like learn-ham: line 1: error: unexpected character(s) starting with 0xe2. And then I noticed that it should be 0x22 (double quotation mark).

    Only the next line is wrong and should be taken under further inspection:
    require [“vnd.dovecot.pipe”, “copy”, “imapsieve”, “variables”];
    if string “${mailbox}” “Trash” {
    stop;
    }
    pipe :copy “rspamd-learn-ham.sh”;

    Reply
    • 2020-01-28 at 14:57
      Permalink

      Terribly sorry. Sometimes the HTML editor here tries to be smarter than me. That has given me quite a lot of headaches, too. Just fixed the quotes in the text.

      Reply
  • 2020-01-26 at 09:58
    Permalink

    I have all done “Training the spam detection” with redis but when I try
    rspamc learn_ham /var/vmail/mydomain.fr/frederic/Maildir/cur
    I have : “HTTP error: 500, Unknown statistics error”
    In rspamc stat, I haven’t the lines beginning with “Statfile”
    I am with rspamd 1.8.1 on a raspberry

    Reply
    • 2020-03-04 at 07:07
      Permalink

      I have similar symptoms with Training. And rspamd hasn’t learnt one eMail. I checked that the learning pipeline is working as seen on debug_log.

      root@smtp:~# rspamc stat
      Results for command: stat (0.036 seconds)
      Messages scanned: 5835
      Messages with action reject: 318, 5.45%
      Messages with action soft reject: 0, 0.00%
      Messages with action rewrite subject: 0, 0.00%
      Messages with action add header: 253, 4.34%
      Messages with action greylist: 973, 16.68%
      Messages with action no action: 4291, 73.54%
      Messages treated as spam: 571, 9.79%
      Messages treated as ham: 5264, 90.21%
      Messages learned: 0
      Connections count: 393
      Control connections count: 724
      Pools allocated: 2755
      Pools freed: 2726
      Bytes allocated: 18.87M
      Memory chunks allocated: 102
      Shared chunks allocated: 17
      Chunks freed: 0
      Oversized chunks: 128
      Fuzzy hashes in storage “rspamd.com”: 1057405579
      Fuzzy hashes stored: 1057405579
      Total learns: 0

      Reply
      • 2020-03-16 at 20:31
        Permalink

        Same issue here.
        When i use redis as backend i get the same error so i switched to sqlite and everything seems to work fine.
        When learning rspamc my spam mails (rspamc learning_spam ) i get

        > Results for file: success = true;

        And looking into rspamd.log it shows
        > #25867(controller) ; csession; rspamd_controller_learn_fin_task: learned message as spam:

        Running rspamc stat i’m also missing the stats file, not sure how to fix this 🙁

        Results for command: stat (0.012 seconds)
        Messages scanned: 5664
        Messages with action reject: 3503, 61.84%
        Messages with action soft reject: 0, 0.00%
        Messages with action rewrite subject: 0, 0.00%
        Messages with action add header: 878, 15.50%
        Messages with action greylist: 322, 5.68%
        Messages with action no action: 961, 16.96%
        Messages treated as spam: 4381, 77.34%
        Messages treated as ham: 1283, 22.65%
        Messages learned: 0
        Connections count: 3089
        Control connections count: 980
        Pools allocated: 71030
        Pools freed: 71006
        Bytes allocated: 29.54MiB
        Memory chunks allocated: 130
        Shared chunks allocated: 17
        Chunks freed: 0
        Oversized chunks: 1029
        Fuzzy hashes in storage “rspamd.com”: 1078929305
        Fuzzy hashes stored: 1078929305
        Total learns: 0

        Reply
        • 2020-05-17 at 12:26
          Permalink

          I have been following this tutorial for a few iterations now and it was always of great help. Thank you very much!

          With the current version and with switching from sqlite to Redis, I was experiencing the same error. After experimenting a lot with the configuration value (and by looking through all the helpful comment here, especially https://workaround.org/ispmail/buster/filtering-out-spam-with-rspamd-2/#comment-104387), I think I was able to solve the error:

          Instead of using /etc/rspamd/override.d/statistic.conf, I added the redis option in /etc/rspamd/override.d/classifier-bayes.conf. Now rspamd began to learn from messages again.

          Reply
    • 2020-05-16 at 10:16
      Permalink

      Same here “HTTP error: 500, Unknown statistics error”

      Reply
  • 2020-01-27 at 18:43
    Permalink

    How should a Maildirectory structure be like?

    INBOX
    ./Sent
    ./Trash
    -/Drafts

    How about Spam?

    a) under INBOX
    INBOX.Junk

    b) under “Maildirecory root”
    ./Junk

    I found myself in a situation that I was trying to figure out what is wrong since rspamc does not get spam/ham education when moving from/to Junk folder. But, when the first spam came in I noticed that a second Junk folder was created.

    I think that Junk folder under inbox is ok, but how do You see it?

    Reply
    • 2020-01-28 at 14:59
      Permalink

      The layout of your Maildirs depends on how you set the “separator” in the Dovecot configuration. (a) uses “.” as a seperator while (b) uses “/”. For historical reasons I still use “.” but “/” is today’s default.

      Reply
      • 2020-06-24 at 15:45
        Permalink

        Hi! First of all, thank you for this wonderful tutorial.

        It is pretty easy to follow and it is also very useful.

        I’m in doubt regarding separators in Dovecot. Although it seems new versions should be using “/” instead of “.”, I’m a brand new installation Dovecot still uses “.” (and it seems documentation states the same: https://wiki.dovecot.org/MailboxFormat/Maildir, under Directory Structure).

        What am I missing out?

        Thank you for your time and consideration.

        Reply
        • 2020-06-24 at 16:27
          Permalink

          Interesting. Thanks for pointing that out. I could have sworn that “/” was the newer default. But neither the distributed configuration files nor the documentation confirm that. So apparently I have made that up. 🙂

          Reply
          • 2020-06-24 at 17:23
            Permalink

            Thank you so much for your quick answer.

            For our documentation, I have been translating this tutorial to spanish. I would like to send you a copy, maybe others would benefit too.

  • 2020-02-10 at 00:16
    Permalink

    I have been following along so far, and everything seems to be working ok. However, I am seeing this message when I run “rspamadm configtest”:

    ==> rspamadm configtest
    symbol ‘BAYES_HAM’ has its score defined but there is no corresponding rule registered
    symbol ‘BAYES_SPAM’ has its score defined but there is no corresponding rule registered
    syntax OK

    I am not sure what this means; I need to go back and review what I did to make sure that I didn’t miss a step, but I wanted to ask about what this might be pointing to.

    Reply
  • 2020-03-15 at 02:37
    Permalink

    Hi Christoph,
    at “Per-user spam training” you obviously switched from statistics.conf to classifier-bayes.conf. But further down at “Better scaling with redis” you used the statistics.conf to change the backend. Is that meant to be that way altough?

    greets,
    Tobias

    (Btw: Great tutorial – I’m following this guide since sarge!)

    Reply
  • 2020-03-16 at 10:14
    Permalink

    Pie Chars are working for me. I did follow your guide and have not changed anything. When i first visited the webinterface, there was no pie chart at all. I guess it will take some time till rspam generates it.

    Thanks for another gread ISP mail tutorial. I am using this since years and could not imagine to use something else anymore 🙂

    Cheers Florian

    Reply
  • 2020-03-31 at 00:22
    Permalink

    Hi, thanks for the great tutorial! I am getting this kind of strange error message in my log:

    Mar 31 02:18:11 hostname.net dovecot[417]: lmtp(recipient@domain.tld): Error: open(/etc/dovecot/sieve-after/spam-to-folder.svbin.hostname.net.3586.40efc3f6ecdbce04) failed: Read-only file system

    Mar 31 02:18:11 hostname.net dovecot[417]: lmtp(recipient@domain.tld): Error: sieve: binary /etc/dovecot/sieve-after/spam-to-folder.svbin: save: failed to create temporary file: open(/etc/dovecot/sieve-after/spam-to-folder.svbin.) failed: Read-only file system

    Why is dovecot trying to create temporary files in its configuration directory? I chowned the sieve-after directory to vmail, but this does not seem to help…

    Reply
    • 2020-03-31 at 14:38
      Permalink

      I fixed it by putting

      [Service]
      ReadWritePaths=/etc/dovecot/sieve/before.d/

      in `systemctl edit dovecot` and then also setting

      chown vmail:vmail /etc/dovecot/sieve-after/

      Reply
  • 2020-04-06 at 18:12
    Permalink

    In my learn-ham.sieve, I have the following:
    require [“vnd.dovecot.pipe”, “copy”, “imapsieve”, “environment”, “variables”];
    if string “${mailbox}” “Trash” {
    stop;
    }
    pipe :copy “rspamd-learn-ham.sh”;

    But when moving an email from Junk to Trash, the learn-ham is still being triggered:
    Apr 6 20:10:04 letmein dovecot: imap(me@domain.tld): sieve: pipe action: piped message to program `rspamd-learn-ham.sh’
    Apr 6 20:10:04 letmein dovecot: imap(me@domain.tld): sieve: left message in mailbox ‘Trash’

    What could the issue be?

    Reply
    • 2020-04-07 at 14:39
      Permalink

      Found out what the issue was! 🙂

      The learn-ham.sieve file should look like this:
      require [“vnd.dovecot.pipe”, “copy”, “imapsieve”, “environment”, “variables”];

      if environment :matches “imap.mailbox” “*” {
      set “mailbox” “${1}”;
      }

      if string “${mailbox}” “Trash” {
      stop;
      }
      pipe :copy “rspamd-learn-ham.sh”;

      Reply
      • 2020-05-08 at 18:40
        Permalink

        Thanks for that – that fix worked for me.

        And many thanks to Christoph for an amazing HowTo!

        Reply
      • 2020-06-22 at 12:18
        Permalink

        Thank you that did do the trick for me too – it drove me crazy.

        Thanks for a great tutorial Christoph

        Reply
  • 2020-04-07 at 14:48
    Permalink

    I’m not sure that I understand the “Per-user spam training” section of this guide.
    Does it benefit at all from the manual learning in section “Learning from user actions”?

    The scripts (rspamd-learn-spam/ham.sh) does not include anu user details.
    So to me this looks like it will be added to the global database and not a user specific database or am I missing something?

    In reference to the guide here:
    https://wiki2.dovecot.org/HowTo/AntispamWithSieve
    We can see that Spamassassin and dspam both have a username value forwarded.
    spamassassin: exec /usr/bin/spamc -u ${1} -L spam -C report
    dspam: exec /usr/bin/dspam –client –user ${1} –class=spam –source=error

    And in the report-spam/ham we also see this (which is not included in your guide):
    if environment :matches “imap.user” “*” {
    set “username” “${1}”;
    }

    So do we benefit from “Per-user spam training” at all?

    Reply
    • 2020-04-12 at 12:31
      Permalink

      I got that information from https://rspamd.com/doc/configuration/statistic.html#statistics-configuration

      However I am struggling a lot with per-user training. The statistics/bayes module of rspamd is working unrealiably here. Also training spam per-user massively reduced the number of ham and spam samples. With many users all benefit from (proper) training. However if you have a user that only received 20 spam emails a month it will take a year before they have enough samples to be actually use bayes.

      In fact I’m experimenting with SpamAssassin again which shows a much better spam detection ratio despite all the good intentions of rspamd.

      Reply
  • 2020-04-21 at 20:01
    Permalink

    The first link on this page “rspamd” goes to rspam.com but it should be rspamd.com I think.

    Reply
    • 2020-04-21 at 20:30
      Permalink

      Ouch. Thanks. Fixed.

      Reply
  • 2020-05-03 at 12:24
    Permalink

    Hello,

    Just a few word about the line:
    postconf milter_mail_macros=”i {mail_addr} {client_addr} {client_name} {auth_authen}”

    As I was editing the main.cf file manually, I copy pasted the line with the ” (double quote) around the parameters. This is wrong!
    As a result I had this bug:
    https://github.com/rspamd/rspamd/issues/2622

    But if you use the postconf command like in your tutorial, then it works like a charm.

    So this is not an issue here, but I just want to let everyone figure that out :p

    Anyway, thanks a lot for this tuto, it helped a lot!

    Reply
  • 2020-05-06 at 15:48
    Permalink

    Hello Christoph,

    Love your guide, very useful. Many thanks.

    Read it earlier, now going through it again, to actually execute it, and I think I spotted something. I find the following statement a tad bit confusing:

    “The currently recommended way is to use the “IMAPSieve” plugin instead.”

    What does “instead” refer to here exactly?

    Regards,
    Dan

    Reply
  • 2020-05-06 at 17:07
    Permalink

    Couldn’t help but notice, that you seem to have a wee disagreement with the developer(s) of rspamd. 😉 I also noticed, that earlier tutorials were using SpamAssasin.

    I reckon, you had good reasons, to make the change. Can I be curious, and ask you about this? I am sure, others have been wondering too…

    Cheers,
    Dan

    Reply
  • 2020-05-16 at 09:40
    Permalink

    For nginx – maybe it helps:

    location ^~ /rspamd/ {
    proxy_pass http://localhost:11334/;
    proxy_set_header Host $host;
    proxy_set_header X-Forwarded-For $proxy_add_x_forwarded_for;
    }

    Import on my server config was “^~”

    Reply
  • 2020-05-16 at 11:02
    Permalink

    No graphs on Debian Buster?

    1. Go to Github: https://github.com/rspamd/rspamd
    2. Downlaod the zip-file
    3. Rename www to www_bak (mv /usr/share/rspamd/www /usr/share/rspamd/www_bak
    4. Upload the “interface” directory from zip-file to /usr/share/rspamd/ and rename to www (mv /usr/share/rspamd/interface /usr/share/rspamd/www)

    Now the graphs should work 😉

    Reply
    • 2020-05-18 at 02:00
      Permalink

      Yep that fixed it for me.
      Thank you Martin!

      Reply
  • 2020-05-20 at 16:33
    Permalink

    Hi,

    I am currently facing an issue with the dovecot sieve. Whenever the action is triggered I got this error in the log:

    Error: sieve: pipe action: failed to execute to program `rspamd-learn-spam.sh’: refer to server log for more information.

    I followed the guide thus far without problem but got nailed by this. I am not sure which server log the error refers to. Anyone encountered this problem before? Any advice?

    Thank you.

    Reply
    • 2020-05-30 at 23:39
      Permalink

      I had the same problem with the error “Error: sieve: pipe action: failed to execute to program `rspamd-learn-spam.sh’: refer to server log for more information.”

      as well as the error:
      symbol ‘BAYES_HAM’ has its score defined but there is no corresponding rule registered
      symbol ‘BAYES_SPAM’ has its score defined but there is no corresponding rule registered

      What fixed both issues for me was deleting /etc/rspamd/override.d/statistics.conf

      and making /etc/rspamd/override.d/classifier-bayes.conf look exactly like this:

      classifier “bayes” {
      users_enabled = true;
      backend = “redis”;
      autolearn = true;
      }

      Sources:

      /etc/rspamd/statistic.conf:

      #If you just need to change the default bayes classifier, you can also use
      # ‘local.d/classifier-bayes.conf’ or ‘override.d/classifier-bayes.conf’. But
      # never ever use both `statistics.conf` and `classifier-bayes.conf` locals files
      # together!

      AND

      https://github.com/rspamd/rspamd/issues/2678#issuecomment-629394419

      After making these changes, the previous problems are gone and it works perfectly now 🙂

      Reply
      • 2020-06-05 at 08:53
        Permalink

        I have the same problem. But your fixes didn’t help.

        My logs state this:
        Jun 05 10:40:23 mail dovecot[22535]: imap(user): Fatal: execvp(/etc/dovecot/sieve/global/rspamd-learn-spam.sh) failed: Permission denied
        Jun 05 10:40:23 mail dovecot[22535]: imap(user): Error: write(program stdin) failed: Broken pipe
        Jun 05 10:40:23 mail dovecot[22535]: imap(user): program `/etc/dovecot/sieve/global/rspamd-learn-spam.sh’ terminated with non-zero exit code 84
        Jun 05 10:40:23 mail dovecot[22535]: imap(user): Error: sieve: pipe action: failed to pipe message to program `rspamd-learn-spam.sh’: refer to server log for more information. [2020-06-05 10:40:23]
        Jun 05 10:40:23 mail dovecot[22535]: imap(user): sieve: left message in mailbox ‘Junk’
        Jun 05 10:40:23 mail dovecot[22535]: imap(user): Error: sieve: Execution of script /etc/dovecot/sieve/global/learn-spam.sieve failed

        to be honest I didn’t follow this sites guide to set up the mail server, it was set up by someone else many years ago. So there are some differences. But for all I know I should have compensated for them.
        First difference: our sieve scripts lie in sieve/global, not sieve/, but I changed that in the 90-plugin.conf so it contains:
        imapsieve_mailbox1_before = file:/etc/dovecot/sieve/global/learn-spam.sieve
        imapsieve_mailbox2_before = file:/etc/dovecot/sieve/global/learn-ham.sieve
        and
        sieve_pipe_bin_dir = /etc/dovecot/sieve/global

        Another difference is the mail directory. they are under /maildirs/user/Maildir. And we have neither user nor group “vmail”. But since the /maildirs folder is owned by root:root I set that owner instead of vmail:vmail so you get this view:
        10:49:07 [root @ mail] ~ # ls -la /etc/dovecot/sieve/global/
        insgesamt 44K
        drwxr-xr-x 2 dovecot root 4,0K Jun 5 10:26 .
        drwxr-xr-x 3 root root 4,0K Jul 29 2019 ..
        -rw-r–r– 1 root root 144 Jun 5 10:06 learn-ham.sieve
        -rw-r–r– 1 root root 306 Jun 5 10:07 learn-ham.svbin
        -rw-r–r– 1 root root 86 Jun 5 10:06 learn-spam.sieve
        -rw-r–r– 1 root root 250 Jun 5 10:06 learn-spam.svbin
        -rw-r–r– 1 root root 509 Mär 16 13:57 mailfilter.sieve
        -rw-r–r– 1 dovecot root 462 Jul 29 2019 mailfilter.sieve~
        -rw-r–r– 1 root root 398 Mai 6 18:02 mailfilter.svbin
        -rwx—— 1 root root 41 Jun 5 10:25 rspamd-learn-ham.sh
        -rwx—— 1 root root 42 Jun 5 10:25 rspamd-learn-spam.sh

        to be exact, the .sieve files are normally owned by dovecot:root. But it doesn’t matter how I change ownership, the errors are the same.
        Manually calling e.g. rspamc learn_spam /maildirs/user/Maildir/cur does work in rspamd 2.5. Do you have any idea what could be wrong?

        Reply
        • 2020-06-07 at 12:09
          Permalink

          The sieve files are likely okay. But take a look at the permissions of your *.sh files. Only root can execute them.

          On my test server they have these permissions:

          -rwxrw—- 1 vmail vmail 84 Mar 8 22:44 rspamd-learn-ham.sh
          -rwxrw—- 1 vmail vmail 86 Mar 8 22:44 rspamd-learn-spam.sh

          Reply
          • 2020-06-08 at 16:02
            Permalink

            thanks, I changed that. I’ll report back when I get another spam mail that wasn’t sorted properly

          • 2020-06-09 at 09:52
            Permalink

            ok, still the same error. I’m trying now with owner dovecot:root. Or might it help to even change permissions fro 760 to 766?

          • 2020-06-10 at 09:16
            Permalink

            nope, still same error. Strange.

          • 2020-06-16 at 12:27
            Permalink

            ok, it’s getting weirder all the time. Permissions for both shell scripts is at 764, both shell scripts and sieve files are owned by dovecot:root. Now the error message changed to

            Jun 16 14:16:49 mail dovecot[20820]: imap(user): Error: sieve: learn-spam: line 1: unexpected character(s) starting with 0xbe
            Jun 16 14:16:49 mail dovecot[20820]: imap(user): Error: sieve: learn-spam: line 1: unexpected unknown characters found at (the presumed) end of file
            Jun 16 14:16:49 mail dovecot[20820]: imap(user): Error: sieve: learn-spam: parse failed
            Jun 16 14:16:49 mail dovecot[20820]: imap(user): Error: sieve: Failed to compile script `/etc/dovecot/sieve/global/learn-spam.sieve’

            …what’s going on?

            PS: mails are stored as bzip2 compressed Maildirs. Might that be the problem of the first two error lines?
            If so, is it possible to pipe the mail through the zlib plugin of dovecot to rspamd so they are still saved compressed but rspamd gets the raw file for training?

          • 2020-06-16 at 18:05
            Permalink

            and how to fix that?

        • 2020-06-16 at 20:31
          Permalink

          Depends on the text editor you are using. Common editors like vi or nano do not mess around with it. Make sure that your text editor is not writing a byte-order-mark.

          Reply
          • 2020-06-17 at 09:33
            Permalink

            well, where should I use any text editor in the first place? The problem is piping the mail to rspamd when moving it to the spam folder in the mail application. There is no text editor involved at all.

          • 2020-06-17 at 12:25
            Permalink

            I meant the text editor you used to create the “sieve” files. That’s where your editor added a BOM in the file that is not accepted by Dovecot.

          • 2020-06-17 at 13:49
            Permalink

            ah, thanks. You’re right, the learn-spam.sieve was completely corrupted for some reason. I don’t know what did that. I only remember that at some point the file had the permission 740 even though I gave it 744 and did not change that. Very strange. Now it’s what it’s supposed to be. I’ll report back when I got to test it out

          • 2020-06-19 at 07:38
            Permalink

            and back to the start:
            Jun 19 09:36:39 mail dovecot[20820]: imap(user): Fatal: execvp(/etc/dovecot/sieve/global/rspamd-learn-spam.sh) failed: Permission denied
            Jun 19 09:36:39 mail dovecot[20820]: imap(user): program `/etc/dovecot/sieve/global/rspamd-learn-spam.sh’ terminated with non-zero exit code 84
            Jun 19 09:36:39 mail dovecot[20820]: imap(user): Error: sieve: pipe action: failed to execute to program `rspamd-learn-spam.sh’: refer to server log for more information. [2020-06-19 09:36:39]
            Jun 19 09:36:39 mail dovecot[20820]: imap(user): sieve: left message in mailbox ‘Junk’
            Jun 19 09:36:39 mail dovecot[20820]: imap(user): Error: sieve: Execution of script /etc/dovecot/sieve/global/learn-spam.sieve failed

      • 2020-06-08 at 08:07
        Permalink

        This fixed my problems, too. Had the same error messages. Thanks!

        Reply
  • 2020-06-09 at 12:27
    Permalink

    As per your last comment, I got the pie chart working on my install.

    I had to install the `unzip` package, and confirm erasure during the package extraction with `y`, but that’s it, otherwise it worked just following your instructions.

    You may want to add an `apt install unzip` and some forcing parameter to the `unzip` command to avoid prompting for confirmation.

    rspamd version 1.8.1

    Reply
    • 2020-06-16 at 19:50
      Permalink

      Isn’t it simpler, and just using the Buster (or in my case Devuan Beowulf) repositories, to install rspamd 2.5 from backports in place of rspamd 1.8.1 ? Worked for me.

      Reply
  • 2020-06-28 at 10:44
    Permalink

    Hi there,

    I would like to block some bad attachments like .exe or .exe inside an archive which means that local users will not be able to attach those files in their email. Is rspamd capable of doing that? Or do I need Amavis to do it?

    I had tried to add some config for the mime modules following the documentation of rspamd but it seems like it’s not working as expected. Anyone tried this before?

    Thanks.

    Reply

Leave a Reply to QP Cancel reply

Your email address will not be published. Required fields are marked *