You have come a long way in this guide and your mail server is already fully functional. Now it’s time to deal with the dark side: spam. The amount of spam nowadays has been insane for many years. So we need to detect spam emails and filter them out. I found that rspamd is a well-performing choice for that purpose. rspamd keeps a permanent process running on your mail server that listens to connections from Postfix using the milter (=mail filter) protocol. Every time an email enters your system, Postfix will send it to rspamd to have its content checked. rspamd runs a lot of checks on the email and computes a total score. The higher the score – the more likely the email should be considered unsolicited.
Make Postfix use rspamd
Let’s tell Postfix to send all incoming email through rspamd. Run these commands on the shell:
postconf smtpd_milters=inet:127.0.0.1:11332
postconf non_smtpd_milters=inet:127.0.0.1:11332
postconf milter_mail_macros="i {mail_addr} {client_addr} {client_name} {auth_authen}"
Postfix will connect to rspamd that is listening to TCP port 11332 on localhost (127.0.0.1) and pass the email over that connection. The smtpd_milters setting defines that connection for emails that came into the system from the internet via the SMTP protocol. The non_smtpd_milters setting is optional – it makes Postfix have all emails checked that originate from the system itself. Finally the milter_mail_macros setting defines several variables that rspamd expects for better spam detection. rspamd then runs its checks and tells Postfix whether the email should pass or get rejected.
Testing spam detection
For testing we can use a sample spam email that comes with SpamAssassin. It is called GTUBE (Generic Test for Unsolicited Bulk Email). It contains a certain articial pattern that is recognized as spam by SpamAssassin. Do you know EICAR.COM to test virus scanners? This is the same thing for spam.
I suggest that you download the file on the server:
wget http://spamassassin.apache.org/gtube/gtube.txt
…and send that email to our test user John…
sendmail john@example.org < gtube.txt
Check your /var/log/mail.log. You will find something like this:
Jan 13 09:21:01 mail postfix/cleanup[19945]: 95B7A42212: milter-reject: END-OF-MESSAGE from localhost[127.0.0.1]: 5.7.1 Gtube pattern; from=<root@mail.example.org> to=<john@example.org>
Jan 13 09:21:01 jen postfix/cleanup[19945]: 95B7A42212: to=<john@example.org>, relay=none, delay=0.18, delays=0.18/0/0/0, dsn=5.7.1, status=bounced (Gtube pattern)
The interesting parts are those printed in bold letters. “milter-reject” tells that the milter (rspamd) recommended to Postfix to reject the email. It gave the reason “5.7.1 Gtube pattern”. In mail communication you often find these three digit codes. They are defined in RFC 3463. The first digit is most important:
- 2 = Success
- 4 = Transient failure (temporary problem – come back later)
- 5 = Permanent failure (do not try again in this way)
So 5.7.1 tells us that the result code is a permanent failure in delivery. If you looked up the RFC you would find that the 7 stands for an issue regarding the security policy. So it’s not a technical failure but instead a security-relevant component on the system has rejected the email. That’s what rspamd did. It even told us the reason: “Gtube pattern”. So you see that rspamd knows about the Gtube spam test pattern and reacts as expected.
Accordingly you won’t see this email in John’s inbox. This is a great advantage of using milters by the way. Imagine Postfix receiving a spam email and confirming its reception. What should it do when it finds out that it’s unwanted email? According to the SMTP protocol it must not throw away any emails. Would you create a bounce message telling the sender that you did not accept the email? That would be a bad idea because the sender address is very likely forged. You would send the bounce to an innocent person thus creating so called backscatter and make it even worse. So the right approach is to check the email while the sending server is still connected to your Postfix. This allows Postfix to reject the email with a 5.x.x error code and let the other side figure out what to do.
Score metrics (optional)
rspamd will however not reject all spam email. As it computes a score of the spam propability you can tell it which scores you would accept, flag as spam or make the incoming email get rejected. Take a look at the /etc/rspamd/actions.conf file. There are tons of scores defined for various conditions. At the beginning of the file you will find:
actions {
reject = 15;
add_header = 6;
greylist = 4;
}
These are the default actions. If rspamd computes a score of at least 15 then the email will get rejected at the doorstep just like the Gtube pattern in the previous test. Any score above 6 will add header lines like “X-Spam: Yes” so that your mail software can detect them and maybe file the email to a different folder. Any score above 4 will trigger greylisting which is a mechanism that temporarily rejects the email with a 4.x.x code and waits if the sending server will try again. After a waiting time of a few minutes greylisting will accept the email. The idea is to reject email from systems that do not have a sending queue. Malware like on infected Wind*ws computers used to try sending an email just once which triggered greylisting and successfully rejected the spammer. But even malware programmers have learned and may try again after a few minutes thus circumventing greylisting. Your mileage may vary.
If you like to change these defaults then create a new file in /etc/rspamd/local.d/actions.conf containing your desired limits:
reject = 150; add_header = 2; greylist = 5;
This would virtually never reject an email. And it would flag any email with a score of at least 2 as spam. Greylisting would happen at a score above 5. These are not necessarily sane values – they are just meant as an example.
Please take a moment to understand how to change rspamd defaults. You can either create files in /etc/rspamd/override.d/… which will replace entire sections; or create files in /etc/rspamd/local.d/… which will change only small parts of the configuration. There is a helpful page in the rspamd documentation that contains examples. Whatever you do – never change the /etc/rspamd/* files directly because a software update will try to replace them.
Of course restart rspamd after any configuration changes:
systemctl restart rspamd
To check if rspamd has picked up your configuration use this command to see the current configuration:
rspamadm configdump
You may test your configuration using
rspamadm configtest
Alternatively you may check if all required rspamd processes are running…
pgrep -a rspam 11431 rspamd: main process 11555 rspamd: rspamd_proxy process (localhost:11332) 11557 rspamd: controller process (localhost:11334) 11559 rspamd: normal process (localhost:11333)
Adding headers
As you may know an email contains of the header and the body. Your users will only see header information like the subject, the sender, the recipient and the date and time the email was sent. But there is way more information like the router the email travelled or extended headers added by the various mail server on the way to the destination. Such extended headers begin with an “X-“. rspamd can add such headers to help you filter out spam. For that purpose create a new configuration override file at /etc/rspamd/override.d/milter_headers.conf with this content:
extended_spam_headers = true;
You would need to restart rspamd of course:
systemctl restart rspamd
As documented it will add these headers:
X-Rspamd-Server: mail Authentication-Results: dmarc=fail reason="No valid SPF, No valid DKIM" … X-Rspamd-Queue-Id: C22E55A005B3 X-Spamd-Result: default: False [11.55 / 15.00] R_PARTS_DIFFER(0.27)[63.4%] FROM_NO_DN(0.00)[] RCVD_COUNT_ZERO(0.00)[0] R_DKIM_NA(0.00)[] FUZZY_DENIED(12.00)[1:19305c7fdd:1.00:bin,1:35699594fd:0.91:txt] RBL_SENDERSCORE(2.00)[55.181.23.94.bl.score.senderscore.com] ARC_NA(0.00)[] RCPT_COUNT_ONE(0.00)[1] RCVD_TLS_ALL(0.00)[] FROM_EQ_ENVFROM(0.00)[] R_SPF_SOFTFAIL(0.00)[~all] BAYES_HAM(-2.71)[98.75%] TO_MATCH_ENVRCPT_ALL(0.00)[] MIME_GOOD(-0.10)[multipart/related,multipart/alternative,text/plain] MID_RHS_MATCH_FROM(0.00)[] ASN(0.00)[asn:16276, ipnet:94.23.0.0/16, country:FR] TO_DN_NONE(0.00)[] DMARC_POLICY_SOFTFAIL(0.10)[Chronopost.fr : No valid SPF, No valid DKIM,none] SUBJECT_ENDS_EXCLAIM(0.00)[] X-Spam: Yes
Each of the uppercase symbols like FROM_HAS_DN means that a certain detection routing of rspamd was triggered. It does not necessarily mean something bad about the email. For example R_SPF_ALLOW has a negative score that lowers the total score because it is something good about the email. There are a several symbols with a 0.00 score. These do not change the score but show you what rspamd has found. But if you consider certain criteria good or bad then you can define your own scores for them.
The last line here is especially interesting because next on our list is…
Sending spam to the Junk folder
Your users will not realize that their spam emails have an added “X-Spam: Yes” header. It it not actively shown in their mail client. Nor does it move the email out of the inbox into their spam folder. Such emails just appear like normal in their inbox. So let’s aid them by moving spam to a separate Junk folder beneath their inbox automatically. Dovecot has support for Sieve filters which are scripts that run automatically whenever an email comes in.
John could create a new Sieve filter (e.g. using the Roundcube webmail interface) for himself that would save any emails to his “Junk” folder if the header line “X-Spam: Yes” was found. This rule would be useful for all your users though so let’s find a general solution.
Dovecot supports global Sieve filters that apply to all users. Edit the file /etc/dovecot/conf.d/90-sieve.conf. Look for the “sieve_after” lines. They are commented out. Add a new line there:
sieve_after = /etc/dovecot/sieve-after
The “sieve after” filters are executed after the users’ filters. John can define his own filter rules. And after that Dovecot will run any filter rules it finds in files in /etc/dovecot/sieve-after. That is just an arbitrary directory that you create:
mkdir /etc/dovecot/sieve-after
And add a new file /etc/dovecot/sieve-after/spam-to-folder.sieve reading:
require ["fileinto","mailbox"];
if header :contains "X-Spam" "Yes" {
fileinto :create "Junk";
stop;
}
The “require” lines include functionality to move emails into certain folders (fileinto) and to create folders if they don’t exist yet (mailbox). Then if rspamd marked an email as spam it gets moved into the INBOX.Junk folder which just appears as “Junk” to the user underneath their inbox.
Dovecot cannot deal with such human-readable files though. So we need to compile it:
sievec /etc/dovecot/sieve-after/spam-to-folder.sieve
That generated a machine-readable file /etc/dovecot/sieve-after/spam-to-folder.svbin.
Now all your users will automatically get spam emails moved to their Junk folder. Nice – isn’t it?
Training the spam detection
One of rspamd’s features is analyzing word patterns using probability theory. That functionality is contained in its “statistical module“. (Yes, the name is pretty misleading.)
(a) No Bayes training
You can start with an empty training database. This is not as bad as it sounds. rspamd has way more functionality to determine if an email is ham or spam. Autolearning takes email that are surely ham or spam and uses them to train the spam filter. The rspamd documentation has further examples how to fine-tune auto learning. After a few hundred emails the training will contribute towards a better detection rate.
If you want to use autolearning just create a new file /etc/rspamd/override.d/classifier-bayes.conf and make it contain:
autolearn = true;
(b) Migrating training data from previous mail server
If your previous mail server used rspamd on Debian Stretch then you can migrate over the training data. Stop the rspamd process on both the old and the new server so that the SQLite databases are closed properly…
systemctl stop rspamd
…and copy over all files like /var/lib/rspamd/*.sqlite to your new mail server.
Set the ownership of the database files on the new server correctly:
chown _rspamd._rspamd /var/lib/rspamd/*sqlite
Then restart rspamd on the both servers:
systemctl start rspamd
Verify that the database is now filled by running…
rspamc stat
At the end of the output you will see something like…
Statfile: BAYES_SPAM type: sqlite3; length: 36.59M; free blocks: 0; total blocks: 497.23k; free: 0.00%; learned: 3622; users: 233; languages: 14 Statfile: BAYES_HAM type: sqlite3; length: 28.00M; free blocks: 0; total blocks: 374.15k; free: 0.00%; learned: 31913; users: 97; languages: 6 Total learns: 35535
(c) Training from your existing ham and spam emails
Have you been running a mail server with mailboxes in a Malidir structure before but without rspamd? Then you probably have a good amount of ham and spam emails. Let’s use those to train rspamd. It is important to train both ham and spam emails. The rspamc command will allow you to feed entire directories/folders of emails to the learning process. An example to train spam:
rspamc learn_spam /var/vmail/example.org/john/Maildir/.INBOX.Junk/cur
And this would be an example to train ham from John’s inbox:
rspamc learn_ham /var/vmail/example.org/john/Maildir/cur
Of course the quality of spam detection will depend on how good the source data is. So do not train spam emails from users who randomly put mails into their Junk folder that is not actually spam.
Check the number of emails you learned by running…
rspamc stat
Bayes spam checking will not work before it learned at least 200 spam and ham emails. Teaching rspamd fewer emails or just spam emails will not work. This is defined by the min_learns variable defined in /etc/rspamd/statistic.conf.
In the output you will find lines beginning with “Statfile” like these…
Statfile: BAYES_SPAM type: sqlite3; length: 41.78M; free blocks: 0; total blocks: 0; free: 0.00%; learned: 0; users: 1; languages: 1 Statfile: BAYES_HAM type: sqlite3; length: 51.12M; free blocks: 0; total blocks: 0; free: 0.00%; learned: 0; users: 1; languages: 1 Total learns: 0
This is what you usually start with. The more emails you feed into the training process the better the detection rate will be. Some emails however may not be long enough or be too similar to previously trained emails. So don’t worry if you are training 1000 emails but just get a count of 500 emails here.
Per-user spam training
rspamd allows you to train the spam detection per user. It would not keep a global training database that applies to all users. Instead each user gets their own training.
Advantage: users work differently. Some have subscribed to a sales newsletter and now believe that marking it as spam gets them unsubscribed. Yes, that’s stupid but can thoroughly confuse the spam detection. Also you might be very interested in viagr* product informatoin while others do not.
Disadvantage: training still requires many ham and spam mails before it has any effect. So unless a user gets 200 samples of good and evil emails the spam detection cannot work.
If you decide you want to use per-user spam training then add/edit the file /etc/rspamd/override.d/statistic.conf/etc/rspamd/local.d/classifier-bayes.conf and insert:
users_enabled = true;
Better scaling with Redis
So far rspamd has stored its training (and other) data in SQLite databases. If you haven’t heard of it: it’s a database system much like MariaDB but it does not have a server process that you connect to. Instead you access the database files directly using either the “sqlite3” command or using libraries that integrate it SQLite into other programming languages. It is very simple to use. But compared to MariaDB or PostgreSQL its performance is limited. SQLite is completely sufficient for medium-size mail servers. But if you are going big you should instead use Redis as a data store.
Redis is also a kind of database system. It is way more limited than a traditional SQL database. But it is lightning fast the way it works. On my aged server it handles around 50,000 requests per second. It gets it speed from keeping the data in memory. So it doesn’t access the disk to fetch information. (Yet it saves its data to disk frequently to prevent data loss.) People use Redis as a cache or for very fast lookups of simple data structures. And so does rspamd.
Installing Redis is simple:
apt install redis
That’s it. No more configuration needed. Redis mainly consists of a daemon process called redis-server that listens for TCP connections on localhost on port 6379. Enabling Redis in rspamd is simple. Add a file /etc/rspamd/override.d/redis.conf and insert:
servers = "127.0.0.1";
Now you need to configure which rspamd modules should use Redis as a backend storage. Several modules support Redis. Just as an example let’s put the spam training data of the statistic module into Redis. Edit the file /etc/rspamd/override.d/statistic.conf and set the backend there. If you also use per-user spam training this file should now look like this:
classifier "bayes" {
users_enabled = true;
backend = "redis";
}
Feel free to use Redis for other lookups, too.
Learning from user actions
Now we are getting to something really cool. Let’s tell Dovecot that moving emails into the Junk folder teaches rspamd instantly that the email is spam. And train an email as ham if it is moved out of the Junk folder. We will add triggers (actually “sieve scripts“) to the action of moving emails via IMAP.
The currently recommended way is to use the “IMAPSieve” plugin instead. There is nothing to install – it comes with the Dovecot packages. We just need to configure it.
First order of business is enabling the IMAPSieve plugin for the IMAP protocol/service in Dovecot. Edit the /etc/dovecot/conf.d/20-imap.conf file and look for the line reading “mail_plugins”. Turn it into:
mail_plugins = $mail_plugins imap_sieve
We also need to edit Dovecot’s Sieve configuration to enable two plugins that are required for our task. Sieve is a scripting language that automates things in conjunction with emails and folders. Edit the file /etc/dovecot/conf.d/90-sieve.conf and put these lines into the plugin {…} section:
# From elsewhere to Junk folder imapsieve_mailbox1_name = Junk imapsieve_mailbox1_causes = COPY imapsieve_mailbox1_before = file:/etc/dovecot/sieve/learn-spam.sieve # From Junk folder to elsewhere imapsieve_mailbox2_name = * imapsieve_mailbox2_from = Junk imapsieve_mailbox2_causes = COPY imapsieve_mailbox2_before = file:/etc/dovecot/sieve/learn-ham.sieve sieve_pipe_bin_dir = /etc/dovecot/sieve sieve_global_extensions = +vnd.dovecot.pipe sieve_plugins = sieve_imapsieve sieve_extprograms
The first rule tells Dovecot to run the Sieve rules as defined in the /etc/dovecot/sieve/learn-spam.sieve file whenever an email is moved into a user’s “Junk” folder. We will create that Sieve script in a minute.
The second rule sets the other way. Whenever an email is moved from the “Junk” folder to any (*) folder then the /etc/dovecot/sieve/learn-ham.sieve Sieve script is called.
The “sieve_pipe_bin_dir” setting defines where executable scripts are allowed to reside. We will put our simple learning scripts there. And finally the “sieve_global_extensions” setting enables the pipe plugin that allows sending email to external commands.
Next up let’s create the Sieve scripts that we told Dovecot about. Create a new directory /etc/dovecot/sieve to put our new files in:
mkdir /etc/dovecot/sieve
Then create the file /etc/dovecot/sieve/learn-spam.sieve and let it contain:
require ["vnd.dovecot.pipe", "copy", "imapsieve"]; pipe :copy "rspamd-learn-spam.sh";
Let’s do the same for /etc/dovecot/sieve/learn-ham.sieve
require ["vnd.dovecot.pipe", "copy", "imapsieve", "variables"];
if string "${mailbox}" "Trash" {
stop;
}
pipe :copy "rspamd-learn-ham.sh";
The above Sieve script avoids training an email as ham if the user moves it to the Trash folder. After all if you clear your Junk folder you do not want to train your spam as regular emails.
Restart Dovecot:
systemctl restart dovecot
These two scripts need to be compiled – that is turning them into machine-readable code:
sievec /etc/dovecot/sieve/learn-spam.sieve sievec /etc/dovecot/sieve/learn-ham.sieve
This creates two new files “learn-ham.svbin” and “learn-spam.svbin” that look like gibberish inside but are now in a format that Dovecot’s Sieve plugin can understand.
Let’s fix the permissions of these files, too, while we are at it:
chmod u=rw,go= /etc/dovecot/sieve/learn-{spam,ham}.{sieve,svbin}
chown vmail.vmail /etc/dovecot/sieve/learn-{spam,ham}.{sieve,svbin}
And the last step is to create the simple shell scripts that do the actual spam/ham training. The first file is /etc/dovecot/sieve/rspamd-learn-spam.sh which contains:
#!/bin/sh exec /usr/bin/rspamc learn_spam
That looks simple, doesn’t it? Nothing more is actually needed. The spam email is piped to this script and rspamd learns it as a spam email and adjusts its spam detection database accordingly.
The second script teaches ham and is called /etc/dovecot/sieve/rspamd-learn-ham.sh. Make it contain:
#!/bin/sh exec /usr/bin/rspamc learn_ham
These two shell scripts must be executable so go fix that:
chmod u=rwx,go= /etc/dovecot/sieve/rspamd-learn-{spam,ham}.sh
chown vmail.vmail /etc/dovecot/sieve/rspamd-learn-{spam,ham}.sh
I hope you haven’t lost your mind yet. It’s really just a chain of things to happen. Let’s reiterate how this process works:
- a user moves a spam email into their “Junk” folder
- Dovecot realizes that this triggers the Sieve rule “imapsieve_mailbox1” so it calls the Sieve script /etc/dovecot/sieve/learn-spam.sieve (in fact the *.svbin version of the script)
- Sieve will take the email and send (“pipe”) it to the executable rspamd-learn-spam.sh shell script
- the script in turn runs the email through the “/usr/bin/rspamc learn_spam” command
This works equally for the other way or learning ham emails of course.
I am sure you are eager to try it out. Yes, it’s about time. However to see that it actually works I suggest you edit the /etc/dovecot/conf.d/10-logging.conf file and set “mail_debug=yes”. That will add a lot more detail to the /var/log/mail.log file but on a busy server may also lead to headaches. 🙂
Restart Dovecot…
systemctl restart dovecot
…and watch the /var/log/mail.log file…
tail -f /var/log/mail.log
Now open your IMAP client (Thunderbird, Evolution, Roundcube, mutt or whatever you prefer) and drag an email to your Junk folder. The mail log should read something along the lines of…
imap(john@example.org): Debug: imapsieve: mailbox INBOX.Junk: MOVE event imap(john@example.org): Debug: imapsieve: Matched static mailbox rule [1] imap(john@example.org): Debug: sieve: file storage: Using Sieve script path: /etc/dovecot/sieve/learn-spam.sieve imap(john@example.org): Debug: sieve: file script: Opened scriptlearn-spam' from/etc/dovecot/sieve/learn-spam.sieve' imap(john@example.org): Debug: sieve: Opening script 1 of 1 from/etc/dovecot/sieve/learn-spam.sieve' imap(john@example.org): Debug: sieve: Loading script /etc/dovecot/sieve/learn-spam.sieve imap(john@example.org): Debug: sieve: Script binary /etc/dovecot/sieve/learn-spam.svbin successfully loaded imap(john@example.org): Debug: sieve: binary save: not saving binary /etc/dovecot/sieve/learn-spam.svbin, because it is already stored imap(john@example.org): Debug: sieve: Executing script from/etc/dovecot/sieve/learn-spam.svbin' imap(john@example.org): Debug: sieve: action pipe: running program: rspamd-learn-spam.sh imap(john@example.org): Debug: Mailbox INBOX.Junk: Opened mail UID=3978 because: mail stream imap(john@example.org): Debug: waiting for program/etc/dovecot/sieve/rspamd-learn-spam.sh' to finish after 0 msecs imap(john@example.org): sieve: pipe action: piped message to programrspamd-learn-spam.sh'
Look for errors and warnings. If at the end you see that Dovecot called the “rspamd-learn-spam.sh” script then you should be fine.
And finally if you pull an email out of the “Junk” folder you should see mailbox rule [2] be called and the email being learned as ham:
imap(john@example.org): Debug: imapsieve: mailbox INBOX: MOVE event imap(john@example.org): Debug: imapsieve: Matched static mailbox rule [2] imap(john@example.org): Debug: sieve: file storage: Using Sieve script path: /etc/dovecot/sieve/learn-ham.sieve imap(john@example.org): Debug: sieve: file script: Opened scriptlearn-ham' from/etc/dovecot/sieve/learn-ham.sieve' imap(john@example.org): Debug: sieve: Opening script 1 of 1 from/etc/dovecot/sieve/learn-ham.sieve' imap(john@example.org): Debug: sieve: Loading script /etc/dovecot/sieve/learn-ham.sieve imap(john@example.org): Debug: sieve: Script binary /etc/dovecot/sieve/learn-ham.svbin successfully loaded imap(john@example.org): Debug: sieve: binary save: not saving binary /etc/dovecot/sieve/learn-ham.svbin, because it is already stored imap(john@example.org): Debug: sieve: Executing script from/etc/dovecot/sieve/learn-ham.svbin' imap(john@example.org): Debug: sieve: action pipe: running program: rspamd-learn-ham.sh imap(john@example.org): Debug: Mailbox INBOX: Opened mail UID=28412 because: mail stream imap(john@example.org): Debug: waiting for program/etc/dovecot/sieve/rspamd-learn-ham.sh' to finish after 0 msecs imap(john@example.org): sieve: pipe action: piped message to programrspamd-learn-ham.sh'
That’s it. Nifty, isn’t it?
Autoexpunge
Andi Olsen pointed out that Dovecot has introduced a feature to automatically delete emails in a folder that reach a certain age. This is especially useful for the “Trash” and “Junk” folders. To enable this feature just edit the /etc/dovecot/conf.d/15-mailboxes.conf file and add the autoexpunge parameter where desired. Example:
mailbox Junk {
special_use = \Junk
auto = subscribe
autoexpunge = 30d
}
mailbox Trash {
special_use = \Trash
auto = subscribe
autoexpunge = 30d
}
Restart Dovecot after any configuration change.
Logging
rspamd keeps a verbose log of its actions in /var/log/rspamd/rspamd.log. If a user complains that a certain email got blocked or at least flagged as spam then take a look at this log. You can match the /var/log/mail.log with it by comparing the Postfix queue ID. Those are the 12-digit hexadecimal number like “95CE05A00547“. Those IDs can be found in the rspamd.log, too:
2018-01-14 06:39:45 #10424(normal) <40985d>; task; rspamd_task_write_log: id: <undef>, qid: <95CE05A00547>, ip: 12.13.51.194, from: <…>, (default: F (no action): [3.40/15.00] [MISSING_MID(2.50){},MISSING_DATE(1.00){},MIME_GOOD(-0.10){text/plain;},ARC_NA(0.00){},ASN(0.00){asn:8220, ipnet:212.123.192.0/18, country:GB;},FROM_EQ_ENVFROM(0.00){},FROM_NO_DN(0.00){},RCPT_COUNT_ONE(0.00){1;},RCVD_COUNT_ZERO(0.00){0;},RCVD_TLS_ALL(0.00){},TO_DN_NONE(0.00){},TO_DOM_EQ_FROM_DOM(0.00){},TO_MATCH_ENVRCPT_ALL(0.00){}]), len: 181, time: 16.000ms real, 6.385ms virtual, dns req: 0, digest: <69b289a82827c11f759837c033cd800a>, rcpts: <…>, mime_rcpt: <…>
Scan outgoing emails, too
Previously I accidentlly recommended to set smtpd_milters to an empty value to prevent scanning outgoing emails. But that was wrong. In fact you want to use rspamd to work on outgoing emails. rspamd determines automatically if an authenticated user wants to send an email out and behaves differently. For example it does not check for IP blacklists or DKIM signatures. Instead it adds a DKIM signature.
So please don’t use “-o smtpd_milters=” anywhere. Sorry for the confusion.
The web interface
rspamd comes with a neat bonus feature: a web interface. It allows you to check emails for spam, get statistics and fine-tune scores. It is already installed and enabled by default and expects HTTP requests on port 11334 on the localhost interface. I suggest you add a simple proxy configuration to your already working HTTPS-enabled web mail configuration to get access.
First you need to enable Apache’s modules for HTTP proxying and rewriting:
a2enmod proxy_http a2enmod rewrite
You can either create a new virtual host configuration or just edit the /etc/apache2/sites-available/webmail.example.org-https.conf file. Anywhere within the VirtualHost tags add:
<Location /rspamd> Require all granted </Location> RewriteEngine On RewriteRule ^/rspamd$ /rspamd/ [R,L] RewriteRule ^/rspamd/(.*) http://localhost:11334/$1 [P,L]
This piece of configuration will forward any requests to https://webmail.example.org/rspamd to localhost:11334 and thus give you access to the rspamd web interface.
The interface is password protected. Let’s generate a new access password:
pwgen 15 1
This gives you a password like “eiLi1lueTh9mia4”. You could put that password in an rspamd configuration file. But cleartext passwords in configuration files are not quite elegant. Let’s create a hash of the password:
rspamadm pw Enter passphrase: … $2$icoahes75e7g9wxapnrbmqnpuzjoq7z…
Feed it the password that pwgen created for you and you will get a long hashed password. This procedure by the way is also documented on the rspamd pages.
Create a new configuration file /etc/rspamd/local.d/worker-controller.inc and put your hashed password in there:
password = "$2$icoahes75e7g9wxapnrbmqnpuzjoq7z…"
There is a nasty bug in the rspamd package in Debian Buster that breaks displaying of the charts. The Javascript D3 library that is used for that purpose was forgotten to get installed. Fortunately you just need to extract one Javascript file from the D3 release ZIP file to work around it:
wget https://github.com/d3/d3/releases/download/v5.15.0/d3.zip unzip -d /usr/share/rspamd/www/js/lib d3.zip d3.min.js rm d3.zip
That’s it for the configuration. Finally restart both rspamd and Apache to load your changed configuration:
systemctl restart rspamd systemctl restart apache2
If everything went as expected you should now be able to access the rspamd web interface at https://webmail.example.org/rspamd
Anything else?
If you consider using rspamd for a larger environment then please take your time reading the good amount of documentation they provide. rspamd scales very well but requires additional setup like using Redis for faster spam analysis across multiple mail gateways.
Oh, and a…
When I try sievec /etc/dovecot/sieve/learn-spam.sieve
I get the following error:
sievec(root): Warning: sieve: ignored unknown extension ‘vnd.dovecot.pipe’ while configuring available extensions
learn-spam: line 1: error: require command: unknown Sieve capability `vnd.dovecot.pipe’.
learn-spam: line 1: error: require command: unknown Sieve capability `imapsieve’.
learn-spam: line 2: error: unknown command ‘pipe’ (only reported once at first occurrence).
learn-spam: error: validation failed.
sievec(root): Fatal: failed to compile sieve script ‘/etc/dovecot/sieve/learn-spam.sieve’
dovecot-sieve and dovecot-managedsieve are installed and I setup the 20-imap.conf with mail_plugins = $mail_plugins imap_sieve
In 90-sieve.conf for sieve_plugins the following must be set:
sieve_plugins = sieve_extprograms sieve_imapsieve
After this it works. 🙂
Indeed. And I put that into the configuration file on my test system, too. Wonder why that didn’t make it here. 🙂 Thanks.
root@debian-4gb-fsn1-1:~# sievec /etc/dovecot/sieve/learn-spam.sieve
learn-spam: line 1: error: require command: unknown Sieve capability `vnd.dovecot.pipe’.
learn-spam: line 1: error: require command: unknown Sieve capability `imapsieve’.
learn-spam: line 2: error: unknown command ‘pipe’ (only reported once at first occurrence).
learn-spam: error: validation failed.
In my trial sieve_plugins = sieve_extprograms sieve_imapsieve was already set before getting to this error. Help ?
Got the same error, even tough I changed 90-sieve.conf.
After a reload of dovecot it worked. @Christoph: you should a reload after touchind the dovecot files.
Hiya Gompali,
Should be a restart of dovecot before compiling
systemctl restart dovecot
Fixed it for me.
LadyLinux
Ok, thks 😉
Thanks, I have edited the page to reload before using the pipe plugin.
Can you advise on the best way to combine a catch-all address (without using aliases) with the sieve-after forward rule?
I have one catch-all email domain that needs to be forwarded on to a gmail business account, but i want the forward to happen after rspamd. (I tried tricking postfix mailbox-maps, but dovecot complains the account doesn’t exist before the sieve-after runs). Any advice? Thanks so much
What about forwarding the catchall address to another user in your domain? And then forward that using the user’s sieve rule to Google?
Okay, that was confusing. What I mean:
1. Add a virtual_aliases entry for “@mydomain.com” to “all-the-stuff@mydomain.com”.
2. Create a user “all-the-stuff@mydomain.com” in virtual_users.
3. Login is through webmail as “all-the-stuff@mydomain.com” (Roundcube) and add a Sieve rule to forward all incoming email to “business-account@gmail.com”.
That way you combined the advantage of having a catchall account with the ability to use Sieve.
Thanks Christoph. I was thinking that … although I wondered whether it’s still possible to maintain the reply-to address. I’ll do some further experimentation with this. Much obliged.
No problem. I would expect that sender and recipient are unchanged indeed.
Just reporting back for the benefit of others… this solution is working beautifully!
I think, should be either “add_header = 2;” in the config snippet or “And it would flag any email with a score of at least 0 as spam.” in the text 🙂
You are right. Fixed. Thanks.
A missing d3.min.js library is to blame for the charts not working. (found this http://debian.2.n7.nabble.com/Bug-918027-rspamd-Web-UI-d3-arc-is-not-a-function-td4444444.html)
wget https://github.com/d3/d3/releases/download/v5.15.0/d3.zip
unzip d3.zip
cp d3.min.js /usr/share/rspamd/www/js/lib/d3.min.js
Good find. I already wondered why graphs suddenly stopped working on my test system but I suspected a typo on my own side. I’ll add the workaround for the bug right away.
Did you forget to remove the blue box note saying “I could not get the (pie) charts working on the test server. Maybe that is a bug in rspamd 1.8.1. If anyone gets it working please leave a comment.”?
Hey Christoph,
I’ve found two problems that are not explained in this section.
1. For the /etc/dovecot/sieve-after/spam-to-folder.sieve
you used the old specs for “fileinto :create “INBOX.Junk”;”
I think it’s a good idea to give a warning at this point for old/new version differences.
The new variant should look like this:
require [“fileinto”,”mailbox”];
if header :contains “X-Spam” “Yes” {
fileinto :create “Junk”;
stop;
}
2. When the learn-{spam,ham}.svbin files get created the owner/group is root at default.
So we need to make a chmod and chown for these files also. When the files are owned by
root the sieve files couldn’t be used by rspamd.
chmod u=rw,go= /etc/dovecot/sieve/learn-{spam,ham}.svbin
chown vmail.vmail /etc/dovecot/sieve/learn-{spam,ham}.svbin
Thanks. I have added a huge warning for the folder naming depending on the separator. Once it again it shows that I my test system still runs on the old separator scheme. 🙂
Just for the record, the spam-to-folder.sieve has the same problem; needs the owner/permissions adjusted.
Hi,
i like the new auto learning feature when moving messages to another folder. Great, BUT!
If i empty my spam/junk folder the messages are moved to the Trash folder and then these messages are treated as “moved out from spam” and learned as ham. Currently i disabled the rule moving out from spam. Maybe you find a solution.
Thanks for spotting that. It probably depends on the email client or its settings on how to manage trashing spam. I set my Junk folder to “keep 30 days and then delete directly”. But to be honest I have not checked yet if my users do it similiarly. I’m open to suggestions. I took the inspiration from https://wiki2.dovecot.org/HowTo/AntispamWithSieve
There is a recommendation for that use case on the site you linked to. Simply search for “Trash”. There is a line added to “report-ham.sieve” that prevents further processing when an email is moved to the trash folder.
if string “${mailbox}” “Trash” {
stop;
}
Works well. I have added that.
Thanks for this! I just noticed the same thing in my logs, and was going to share back. Love to see that the guide is already updated with a perfect solution. Thanks to you both!
Hi Markus,
thank you very much for this great work! I’m using this tutorial since many years.
In the buster edition, I had the same problem that jens described. But even the “new” solution did not work – deleted mails were still learend as ham:
non-working content of /etc/dovecot/sieve/learn-ham.sieve:
require [“vnd.dovecot.pipe”, “copy”, “imapsieve”, “variables”];
if string “${mailbox}” “Trash” {
stop;
}
pipe :copy “rspamd-learn-ham.sh”;
I used vnd.dovecot.debug to debug my script and I found out that “${mailbox}” was always empty.
Instead I’m using the following solution which I like to share:
require [“vnd.dovecot.pipe”, “environment”, “copy”, “imapsieve”, “variables”];
if environment :matches “imap.mailbox” “*” {
set “mailbox” “${1}”;
}
if string :is “${mailbox}” “Trash” {
stop;
}
pipe :copy “rspamd-learn-ham.sh”;
I don’t really know if the first if-block is needed in all scenarios, however, it seems to be the only working solution on my installation. Fred did already post the same solution for a per-user setup.
Thank’s again for your work!
Jochen
I can confirm that Jochen is correct. With the previous fix, manually deleted Junk (at least on Apple Mail) is still learned as ham, but his modified sieve works as expected.
Thanks for all !
There is a problem with this added :
sievec /etc/dovecot/sieve/learn-ham.sieve
learn-ham: line 2: error: unknown test ‘string’ (only reported once at first occurrence).
learn-ham: error: validation failed.
sievec(root): Fatal: failed to compile sieve script ‘/etc/dovecot/sieve/learn-ham.sieve’
In /etc/dovecot/sieve/learn-ham.sieve, add “environment”, “variables” to the require line so it appears as require [“vnd.dovecot.pipe”, “copy”, “imapsieve”, “environment”, “variables”];
Not sure if both are required but they are shown in the example code Christoph refers to in https://wiki2.dovecot.org/HowTo/AntispamWithSieve.
HTH
Same issue here 🙁
I’ve got the same error..
I get the same error 🙁
so frustrating :(((
Sorry, that was an error on my side. Just fixed the documentation, too.
just found the solution !
replace require [“vnd.dovecot.pipe”, “copy”, “imapsieve”];
with require [“vnd.dovecot.pipe”, “copy”, “imapsieve”, “variables”];
Thanks! That worked.
Thanks, there isn’t error now but same problem (deleted spam are added ham !):
Soluce with this : /etc/dovecot/sieve/learn-ham.sieve (https://wiki2.dovecot.org/HowTo/AntispamWithSieve)
require [“vnd.dovecot.pipe”, “copy”, “imapsieve”, “environment”, “variables”];
if environment :matches “imap.mailbox” “*” {
set “mailbox” “${1}”;
}
if string “${mailbox}” “Trash” {
stop;
}
if environment :matches “imap.user” “*” {
set “username” “${1}”;
}
pipe :copy “sa-learn-ham.sh” [ “${username}” ];
Ok, no error but the same issue when you delete a spam :
sieve: pipe action: piped message to program `rspamd-learn-ham.sh’
Why don’t you this :https://wiki2.dovecot.org/HowTo/AntispamWithSieve
/etc/dovecot/sieve/learn-ham.sieve
require [“vnd.dovecot.pipe”, “copy”, “imapsieve”, “environment”, “variables”];
if environment :matches “imap.mailbox” “*” {
set “mailbox” “${1}”;
}
if string “${mailbox}” “Trash” {
stop;
}
if environment :matches “imap.user” “*” {
set “username” “${1}”;
}
pipe :copy “rspamd-learn-ham.sh” [ “${username}” ];
I know we should only use the stable packages of debian buster but after configuring everything above for rspamd the charts were still not showing on the rspamd web interface.
Online I found advise to upgrade to the latest version. What i instead did was adding below commands. Which didn’t upgrade to the latest version but installed missing files.
After this the web charts are working.
apt install -y lsb-release wget
wget -O- https://rspamd.com/apt-stable/gpg.key | apt-key add –
echo “deb http://rspamd.com/apt-stable/ $(lsb_release -c -s) main” > /etc/apt/sources.list.d/rspamd.list
echo “deb-src http://rspamd.com/apt-stable/ $(lsb_release -c -s) main” >> /etc/apt/sources.list.d/rspamd.list
apt update && apt upgrade -y
Thanks it solved the missing graph issue.
If you follow the instructions above you can fix the charts issue without breaking… erm… upgrading your system. 🙂
You are right, of course.
I was happy the charts problem “was fixed”.
Charts still didn’t work after:
wget https://github.com/d3/d3/releases/download/v5.15.0/d3.zip
unzip -d /usr/share/rspamd/www/js/lib d3.zip d3.min.js
rm d3.zip
Now i have a bigger problem:
program `/etc/dovecot/sieve/rspamd-learn-spam.sh’ terminated with non-zero exit code 1
Error: sieve: pipe action: failed to execute to program `rspamd-learn-spam.sh’: refer to server log for more information. [2020-01-30 14:14:23]
sieve: left message in mailbox ‘Junk’
Error: sieve: Execution of script /etc/dovecot/sieve/learn-spam.sieve failed
imap-login: Login: user=, method=PLAIN, rip=192.168.168.4, lip=192.168.166.24, mpid=10366, TLS, session=
program `/etc/dovecot/sieve/rspamd-learn-ham.sh’ terminated with non-zero exit code 1
Error: sieve: pipe action: failed to execute to program `rspamd-learn-ham.sh’: refer to server log for more information. [2020-01-30 14:15:07]
sieve: left message in mailbox ‘INBOX’
Error: sieve: Execution of script /etc/dovecot/sieve/learn-ham.sieve failed
Hello,
I got the exact same problem, like in you (since the beginning, just found out yet).
Do you found out, why that happens? Just checked all logs and “rspamd.log” shows this:
2020-03-14 20:51:28 #17033(controller) ; csession; rspamd_mime_part_detect_language: detected part language: en
2020-03-14 20:51:28 #17033(controller) ; csession; rspamd_stat_classifiers_process: skip statistics as SPAM class is missing
2020-03-14 20:51:28 #17033(controller) ; csession; rspamd_task_process: learn error: Unknown statistics error, found when storing data on backend; classifier: (null) 2020-03-14 20:51:28 #17033(controller) ; csession; rspamd_controller_learn_fin_task: cannot learn : Unknown statistics error, found when storing data on backend; classifier: (null)
And “rspamadm configtest” shows this:
2020-03-14 21:15:38 #0(main) ; symcache; rspamd_symcache_validate: symbol ‘BAYES_SPAM’ has its score defined but there is no corresponding rule registered
2020-03-14 21:15:38 #0(main) ; symcache; rspamd_symcache_validate: symbol ‘BAYES_HAM’ has its score defined but there is no corresponding rule registered
syntax OK
In Case someone else is struggling like I was, there are continuous typos on the /etc/dovecot/sieve/learn-ham.sieve example.
There is a difference on ” and “. I got lots of error messages like learn-ham: line 1: error: unexpected character(s) starting with 0xe2. And then I noticed that it should be 0x22 (double quotation mark).
Only the next line is wrong and should be taken under further inspection:
require [“vnd.dovecot.pipe”, “copy”, “imapsieve”, “variables”];
if string “${mailbox}” “Trash” {
stop;
}
pipe :copy “rspamd-learn-ham.sh”;
Terribly sorry. Sometimes the HTML editor here tries to be smarter than me. That has given me quite a lot of headaches, too. Just fixed the quotes in the text.
I have all done “Training the spam detection” with redis but when I try
rspamc learn_ham /var/vmail/mydomain.fr/frederic/Maildir/cur
I have : “HTTP error: 500, Unknown statistics error”
In rspamc stat, I haven’t the lines beginning with “Statfile”
I am with rspamd 1.8.1 on a raspberry
I have similar symptoms with Training. And rspamd hasn’t learnt one eMail. I checked that the learning pipeline is working as seen on debug_log.
root@smtp:~# rspamc stat
Results for command: stat (0.036 seconds)
Messages scanned: 5835
Messages with action reject: 318, 5.45%
Messages with action soft reject: 0, 0.00%
Messages with action rewrite subject: 0, 0.00%
Messages with action add header: 253, 4.34%
Messages with action greylist: 973, 16.68%
Messages with action no action: 4291, 73.54%
Messages treated as spam: 571, 9.79%
Messages treated as ham: 5264, 90.21%
Messages learned: 0
Connections count: 393
Control connections count: 724
Pools allocated: 2755
Pools freed: 2726
Bytes allocated: 18.87M
Memory chunks allocated: 102
Shared chunks allocated: 17
Chunks freed: 0
Oversized chunks: 128
Fuzzy hashes in storage “rspamd.com”: 1057405579
Fuzzy hashes stored: 1057405579
Total learns: 0
Same issue here.
When i use redis as backend i get the same error so i switched to sqlite and everything seems to work fine.
When learning rspamc my spam mails (rspamc learning_spam ) i get
> Results for file: success = true;
And looking into rspamd.log it shows
> #25867(controller) ; csession; rspamd_controller_learn_fin_task: learned message as spam:
Running rspamc stat i’m also missing the stats file, not sure how to fix this 🙁
Results for command: stat (0.012 seconds)
Messages scanned: 5664
Messages with action reject: 3503, 61.84%
Messages with action soft reject: 0, 0.00%
Messages with action rewrite subject: 0, 0.00%
Messages with action add header: 878, 15.50%
Messages with action greylist: 322, 5.68%
Messages with action no action: 961, 16.96%
Messages treated as spam: 4381, 77.34%
Messages treated as ham: 1283, 22.65%
Messages learned: 0
Connections count: 3089
Control connections count: 980
Pools allocated: 71030
Pools freed: 71006
Bytes allocated: 29.54MiB
Memory chunks allocated: 130
Shared chunks allocated: 17
Chunks freed: 0
Oversized chunks: 1029
Fuzzy hashes in storage “rspamd.com”: 1078929305
Fuzzy hashes stored: 1078929305
Total learns: 0
I have been following this tutorial for a few iterations now and it was always of great help. Thank you very much!
With the current version and with switching from sqlite to Redis, I was experiencing the same error. After experimenting a lot with the configuration value (and by looking through all the helpful comment here, especially https://workaround.org/ispmail/buster/filtering-out-spam-with-rspamd-2/#comment-104387), I think I was able to solve the error:
Instead of using /etc/rspamd/override.d/statistic.conf, I added the redis option in /etc/rspamd/override.d/classifier-bayes.conf. Now rspamd began to learn from messages again.
Same here “HTTP error: 500, Unknown statistics error”
Found solution here:
https://github.com/rspamd/rspamd/issues/2678#issuecomment-629394419
Leave blank: /etc/rspamd/override.d/statistic.conf
/etc/rspamd/override.d/statistic.conf:
autolearn = true;
servers = “127.0.0.1”;
backend = “redis”
users_enabled = true;
How should a Maildirectory structure be like?
INBOX
./Sent
./Trash
-/Drafts
How about Spam?
a) under INBOX
INBOX.Junk
b) under “Maildirecory root”
./Junk
I found myself in a situation that I was trying to figure out what is wrong since rspamc does not get spam/ham education when moving from/to Junk folder. But, when the first spam came in I noticed that a second Junk folder was created.
I think that Junk folder under inbox is ok, but how do You see it?
The layout of your Maildirs depends on how you set the “separator” in the Dovecot configuration. (a) uses “.” as a seperator while (b) uses “/”. For historical reasons I still use “.” but “/” is today’s default.
Hi! First of all, thank you for this wonderful tutorial.
It is pretty easy to follow and it is also very useful.
I’m in doubt regarding separators in Dovecot. Although it seems new versions should be using “/” instead of “.”, I’m a brand new installation Dovecot still uses “.” (and it seems documentation states the same: https://wiki.dovecot.org/MailboxFormat/Maildir, under Directory Structure).
What am I missing out?
Thank you for your time and consideration.
Interesting. Thanks for pointing that out. I could have sworn that “/” was the newer default. But neither the distributed configuration files nor the documentation confirm that. So apparently I have made that up. 🙂
Thank you so much for your quick answer.
For our documentation, I have been translating this tutorial to spanish. I would like to send you a copy, maybe others would benefit too.
I have been following along so far, and everything seems to be working ok. However, I am seeing this message when I run “rspamadm configtest”:
==> rspamadm configtest
symbol ‘BAYES_HAM’ has its score defined but there is no corresponding rule registered
symbol ‘BAYES_SPAM’ has its score defined but there is no corresponding rule registered
syntax OK
I am not sure what this means; I need to go back and review what I did to make sure that I didn’t miss a step, but I wanted to ask about what this might be pointing to.
Hi Christoph,
at “Per-user spam training” you obviously switched from statistics.conf to classifier-bayes.conf. But further down at “Better scaling with redis” you used the statistics.conf to change the backend. Is that meant to be that way altough?
greets,
Tobias
(Btw: Great tutorial – I’m following this guide since sarge!)
Pie Chars are working for me. I did follow your guide and have not changed anything. When i first visited the webinterface, there was no pie chart at all. I guess it will take some time till rspam generates it.
Thanks for another gread ISP mail tutorial. I am using this since years and could not imagine to use something else anymore 🙂
Cheers Florian
Hi, thanks for the great tutorial! I am getting this kind of strange error message in my log:
Mar 31 02:18:11 hostname.net dovecot[417]: lmtp(recipient@domain.tld): Error: open(/etc/dovecot/sieve-after/spam-to-folder.svbin.hostname.net.3586.40efc3f6ecdbce04) failed: Read-only file system
Mar 31 02:18:11 hostname.net dovecot[417]: lmtp(recipient@domain.tld): Error: sieve: binary /etc/dovecot/sieve-after/spam-to-folder.svbin: save: failed to create temporary file: open(/etc/dovecot/sieve-after/spam-to-folder.svbin.) failed: Read-only file system
Why is dovecot trying to create temporary files in its configuration directory? I chowned the sieve-after directory to vmail, but this does not seem to help…
I fixed it by putting
[Service]
ReadWritePaths=/etc/dovecot/sieve/before.d/
in `systemctl edit dovecot` and then also setting
chown vmail:vmail /etc/dovecot/sieve-after/
In my learn-ham.sieve, I have the following:
require [“vnd.dovecot.pipe”, “copy”, “imapsieve”, “environment”, “variables”];
if string “${mailbox}” “Trash” {
stop;
}
pipe :copy “rspamd-learn-ham.sh”;
But when moving an email from Junk to Trash, the learn-ham is still being triggered:
Apr 6 20:10:04 letmein dovecot: imap(me@domain.tld): sieve: pipe action: piped message to program `rspamd-learn-ham.sh’
Apr 6 20:10:04 letmein dovecot: imap(me@domain.tld): sieve: left message in mailbox ‘Trash’
What could the issue be?
Found out what the issue was! 🙂
The learn-ham.sieve file should look like this:
require [“vnd.dovecot.pipe”, “copy”, “imapsieve”, “environment”, “variables”];
if environment :matches “imap.mailbox” “*” {
set “mailbox” “${1}”;
}
if string “${mailbox}” “Trash” {
stop;
}
pipe :copy “rspamd-learn-ham.sh”;
Thanks for that – that fix worked for me.
And many thanks to Christoph for an amazing HowTo!
Thank you that did do the trick for me too – it drove me crazy.
Thanks for a great tutorial Christoph
I’m not sure that I understand the “Per-user spam training” section of this guide.
Does it benefit at all from the manual learning in section “Learning from user actions”?
The scripts (rspamd-learn-spam/ham.sh) does not include anu user details.
So to me this looks like it will be added to the global database and not a user specific database or am I missing something?
In reference to the guide here:
https://wiki2.dovecot.org/HowTo/AntispamWithSieve
We can see that Spamassassin and dspam both have a username value forwarded.
spamassassin: exec /usr/bin/spamc -u ${1} -L spam -C report
dspam: exec /usr/bin/dspam –client –user ${1} –class=spam –source=error
And in the report-spam/ham we also see this (which is not included in your guide):
if environment :matches “imap.user” “*” {
set “username” “${1}”;
}
So do we benefit from “Per-user spam training” at all?
I got that information from https://rspamd.com/doc/configuration/statistic.html#statistics-configuration
However I am struggling a lot with per-user training. The statistics/bayes module of rspamd is working unrealiably here. Also training spam per-user massively reduced the number of ham and spam samples. With many users all benefit from (proper) training. However if you have a user that only received 20 spam emails a month it will take a year before they have enough samples to be actually use bayes.
In fact I’m experimenting with SpamAssassin again which shows a much better spam detection ratio despite all the good intentions of rspamd.
The first link on this page “rspamd” goes to rspam.com but it should be rspamd.com I think.
Ouch. Thanks. Fixed.
Hello,
Just a few word about the line:
postconf milter_mail_macros=”i {mail_addr} {client_addr} {client_name} {auth_authen}”
As I was editing the main.cf file manually, I copy pasted the line with the ” (double quote) around the parameters. This is wrong!
As a result I had this bug:
https://github.com/rspamd/rspamd/issues/2622
But if you use the postconf command like in your tutorial, then it works like a charm.
So this is not an issue here, but I just want to let everyone figure that out :p
Anyway, thanks a lot for this tuto, it helped a lot!
Just wanted to say that I ran into this as well while building some Ansible roles to do all this for me.
Thanks for the awesome tutorial Christophe, I’ve been following and implementing it since Sarge.
Hello Christoph,
Love your guide, very useful. Many thanks.
Read it earlier, now going through it again, to actually execute it, and I think I spotted something. I find the following statement a tad bit confusing:
“The currently recommended way is to use the “IMAPSieve” plugin instead.”
What does “instead” refer to here exactly?
Regards,
Dan
Couldn’t help but notice, that you seem to have a wee disagreement with the developer(s) of rspamd. 😉 I also noticed, that earlier tutorials were using SpamAssasin.
I reckon, you had good reasons, to make the change. Can I be curious, and ask you about this? I am sure, others have been wondering too…
Cheers,
Dan
For nginx – maybe it helps:
location ^~ /rspamd/ {
proxy_pass http://localhost:11334/;
proxy_set_header Host $host;
proxy_set_header X-Forwarded-For $proxy_add_x_forwarded_for;
}
Import on my server config was “^~”
No graphs on Debian Buster?
1. Go to Github: https://github.com/rspamd/rspamd
2. Downlaod the zip-file
3. Rename www to www_bak (mv /usr/share/rspamd/www /usr/share/rspamd/www_bak
4. Upload the “interface” directory from zip-file to /usr/share/rspamd/ and rename to www (mv /usr/share/rspamd/interface /usr/share/rspamd/www)
Now the graphs should work 😉
Yep that fixed it for me.
Thank you Martin!
Hi,
I am currently facing an issue with the dovecot sieve. Whenever the action is triggered I got this error in the log:
Error: sieve: pipe action: failed to execute to program `rspamd-learn-spam.sh’: refer to server log for more information.
I followed the guide thus far without problem but got nailed by this. I am not sure which server log the error refers to. Anyone encountered this problem before? Any advice?
Thank you.
I had the same problem with the error “Error: sieve: pipe action: failed to execute to program `rspamd-learn-spam.sh’: refer to server log for more information.”
as well as the error:
symbol ‘BAYES_HAM’ has its score defined but there is no corresponding rule registered
symbol ‘BAYES_SPAM’ has its score defined but there is no corresponding rule registered
What fixed both issues for me was deleting /etc/rspamd/override.d/statistics.conf
and making /etc/rspamd/override.d/classifier-bayes.conf look exactly like this:
classifier “bayes” {
users_enabled = true;
backend = “redis”;
autolearn = true;
}
Sources:
/etc/rspamd/statistic.conf:
#If you just need to change the default bayes classifier, you can also use
# ‘local.d/classifier-bayes.conf’ or ‘override.d/classifier-bayes.conf’. But
# never ever use both `statistics.conf` and `classifier-bayes.conf` locals files
# together!
AND
https://github.com/rspamd/rspamd/issues/2678#issuecomment-629394419
After making these changes, the previous problems are gone and it works perfectly now 🙂
I have the same problem. But your fixes didn’t help.
My logs state this:
Jun 05 10:40:23 mail dovecot[22535]: imap(user): Fatal: execvp(/etc/dovecot/sieve/global/rspamd-learn-spam.sh) failed: Permission denied
Jun 05 10:40:23 mail dovecot[22535]: imap(user): Error: write(program stdin) failed: Broken pipe
Jun 05 10:40:23 mail dovecot[22535]: imap(user): program `/etc/dovecot/sieve/global/rspamd-learn-spam.sh’ terminated with non-zero exit code 84
Jun 05 10:40:23 mail dovecot[22535]: imap(user): Error: sieve: pipe action: failed to pipe message to program `rspamd-learn-spam.sh’: refer to server log for more information. [2020-06-05 10:40:23]
Jun 05 10:40:23 mail dovecot[22535]: imap(user): sieve: left message in mailbox ‘Junk’
Jun 05 10:40:23 mail dovecot[22535]: imap(user): Error: sieve: Execution of script /etc/dovecot/sieve/global/learn-spam.sieve failed
to be honest I didn’t follow this sites guide to set up the mail server, it was set up by someone else many years ago. So there are some differences. But for all I know I should have compensated for them.
First difference: our sieve scripts lie in sieve/global, not sieve/, but I changed that in the 90-plugin.conf so it contains:
imapsieve_mailbox1_before = file:/etc/dovecot/sieve/global/learn-spam.sieve
imapsieve_mailbox2_before = file:/etc/dovecot/sieve/global/learn-ham.sieve
and
sieve_pipe_bin_dir = /etc/dovecot/sieve/global
Another difference is the mail directory. they are under /maildirs/user/Maildir. And we have neither user nor group “vmail”. But since the /maildirs folder is owned by root:root I set that owner instead of vmail:vmail so you get this view:
10:49:07 [root @ mail] ~ # ls -la /etc/dovecot/sieve/global/
insgesamt 44K
drwxr-xr-x 2 dovecot root 4,0K Jun 5 10:26 .
drwxr-xr-x 3 root root 4,0K Jul 29 2019 ..
-rw-r–r– 1 root root 144 Jun 5 10:06 learn-ham.sieve
-rw-r–r– 1 root root 306 Jun 5 10:07 learn-ham.svbin
-rw-r–r– 1 root root 86 Jun 5 10:06 learn-spam.sieve
-rw-r–r– 1 root root 250 Jun 5 10:06 learn-spam.svbin
-rw-r–r– 1 root root 509 Mär 16 13:57 mailfilter.sieve
-rw-r–r– 1 dovecot root 462 Jul 29 2019 mailfilter.sieve~
-rw-r–r– 1 root root 398 Mai 6 18:02 mailfilter.svbin
-rwx—— 1 root root 41 Jun 5 10:25 rspamd-learn-ham.sh
-rwx—— 1 root root 42 Jun 5 10:25 rspamd-learn-spam.sh
to be exact, the .sieve files are normally owned by dovecot:root. But it doesn’t matter how I change ownership, the errors are the same.
Manually calling e.g. rspamc learn_spam /maildirs/user/Maildir/cur does work in rspamd 2.5. Do you have any idea what could be wrong?
The sieve files are likely okay. But take a look at the permissions of your *.sh files. Only root can execute them.
On my test server they have these permissions:
-rwxrw—- 1 vmail vmail 84 Mar 8 22:44 rspamd-learn-ham.sh
-rwxrw—- 1 vmail vmail 86 Mar 8 22:44 rspamd-learn-spam.sh
thanks, I changed that. I’ll report back when I get another spam mail that wasn’t sorted properly
ok, still the same error. I’m trying now with owner dovecot:root. Or might it help to even change permissions fro 760 to 766?
nope, still same error. Strange.
ok, it’s getting weirder all the time. Permissions for both shell scripts is at 764, both shell scripts and sieve files are owned by dovecot:root. Now the error message changed to
Jun 16 14:16:49 mail dovecot[20820]: imap(user): Error: sieve: learn-spam: line 1: unexpected character(s) starting with 0xbe
Jun 16 14:16:49 mail dovecot[20820]: imap(user): Error: sieve: learn-spam: line 1: unexpected unknown characters found at (the presumed) end of file
Jun 16 14:16:49 mail dovecot[20820]: imap(user): Error: sieve: learn-spam: parse failed
Jun 16 14:16:49 mail dovecot[20820]: imap(user): Error: sieve: Failed to compile script `/etc/dovecot/sieve/global/learn-spam.sieve’
…what’s going on?
PS: mails are stored as bzip2 compressed Maildirs. Might that be the problem of the first two error lines?
If so, is it possible to pipe the mail through the zlib plugin of dovecot to rspamd so they are still saved compressed but rspamd gets the raw file for training?
There’s your problem: https://de.wikipedia.org/wiki/Byte_Order_Mark
and how to fix that?
Depends on the text editor you are using. Common editors like vi or nano do not mess around with it. Make sure that your text editor is not writing a byte-order-mark.
well, where should I use any text editor in the first place? The problem is piping the mail to rspamd when moving it to the spam folder in the mail application. There is no text editor involved at all.
I meant the text editor you used to create the “sieve” files. That’s where your editor added a BOM in the file that is not accepted by Dovecot.
ah, thanks. You’re right, the learn-spam.sieve was completely corrupted for some reason. I don’t know what did that. I only remember that at some point the file had the permission 740 even though I gave it 744 and did not change that. Very strange. Now it’s what it’s supposed to be. I’ll report back when I got to test it out
and back to the start:
Jun 19 09:36:39 mail dovecot[20820]: imap(user): Fatal: execvp(/etc/dovecot/sieve/global/rspamd-learn-spam.sh) failed: Permission denied
Jun 19 09:36:39 mail dovecot[20820]: imap(user): program `/etc/dovecot/sieve/global/rspamd-learn-spam.sh’ terminated with non-zero exit code 84
Jun 19 09:36:39 mail dovecot[20820]: imap(user): Error: sieve: pipe action: failed to execute to program `rspamd-learn-spam.sh’: refer to server log for more information. [2020-06-19 09:36:39]
Jun 19 09:36:39 mail dovecot[20820]: imap(user): sieve: left message in mailbox ‘Junk’
Jun 19 09:36:39 mail dovecot[20820]: imap(user): Error: sieve: Execution of script /etc/dovecot/sieve/global/learn-spam.sieve failed
ok, I think I’ve just fixed my problem. it seems for some reason dovecot had no permission to execute the shell script. I fixed that with
sudo -u dovecot chmod +x *.sh
so correct file permissions in my case are
/etc/dovecot/sieve/global:
drwxr-xr-x 2 dovecot root 4,0K Jul 8 07:33 .
drwxr-xr-x 3 root root 4,0K Jul 29 2019 ..
-rw-r–r– 1 dovecot root 144 Jun 5 10:06 learn-ham.sieve
-rw-r–r– 1 root root 306 Jun 5 10:07 learn-ham.svbin
-rw-r–r– 1 dovecot root 86 Jun 17 15:45 learn-spam.sieve
-rw-r–r– 1 root root 250 Jun 17 15:45 learn-spam.svbin
-rw-r–r– 1 dovecot root 509 Mär 16 13:57 mailfilter.sieve
-rw-r–r– 1 dovecot root 462 Jul 29 2019 mailfilter.sieve~
-rw-r–r– 1 root root 398 Mai 6 18:02 mailfilter.svbin
-rwxrwxr-x 1 dovecot root 41 Jun 5 10:25 rspamd-learn-ham.sh
-rwxrwxr-x 1 dovecot root 42 Jul 8 07:33 rspamd-learn-spam.sh
This fixed my problems, too. Had the same error messages. Thanks!
As per your last comment, I got the pie chart working on my install.
I had to install the `unzip` package, and confirm erasure during the package extraction with `y`, but that’s it, otherwise it worked just following your instructions.
You may want to add an `apt install unzip` and some forcing parameter to the `unzip` command to avoid prompting for confirmation.
rspamd version 1.8.1
Isn’t it simpler, and just using the Buster (or in my case Devuan Beowulf) repositories, to install rspamd 2.5 from backports in place of rspamd 1.8.1 ? Worked for me.
Hi there,
I would like to block some bad attachments like .exe or .exe inside an archive which means that local users will not be able to attach those files in their email. Is rspamd capable of doing that? Or do I need Amavis to do it?
I had tried to add some config for the mime modules following the documentation of rspamd but it seems like it’s not working as expected. Anyone tried this before?
Thanks.
This might be what you are looking for: https://rspamd.com/doc/modules/mime_types.html
Do I need to enable it or is it enabled by default?
I managed to get it working but there is one problem so far, ie. if the .exe inside the archive were to be renamed, rspamd is unable to detect it and the mail will go through. Any advice or workaround for this?
Thanks.
Hello,
Am I the only one with problem with autolearning and Outlook?
When using Thunderbird, I can se the correct action in mail.log when I move emails to the Junk folder, but when I use Outlook, nothing happens.
Do we need to set anything to “INBOX.Junk” in the auto expunge section if we’re using the ‘.’ separator? I’m not seeing the INBOX.Junk folder being cleaned up after 30 days
Hello,
this is a helpful tutorial. Thank you.
One note to the very last warning box: “Besides rspamd wants your money if you make money with your mail server and scan more than half a million emails per day.”
This is not true at all. They want your money if you make money with _their_ servers (i.e. their “fuzzy feeds” and blacklists) which is IMO a very legitimate thing. On your own servers you can do anything you want for free, it’s all open source. Don’t blame them for not providing CPU power for free to commercial usage.
Hi Michael. You are right that it’s just fair to help the rspamd project cover their costs. After all it’s a nice piece of software. I admit that I have a prejudice against the main developer of rspamd due to the sociopathic behavior he shows on bug reports, mailing lists, Twitter etc. I’m tempted to recommend SpamAssassin instead of rspamd but I have to admit that it doesn’t scale as well. I’m reminded of what my former used to day: “No matter how brilliant you are. If you behave like an asshole then you don’t belong here.”
How does sieve_after interact with user sieves? I know that it runs after user sieves, but if a user has specified a rule “if it’s an email from *@paypal.com, move it into INBOX.paypal”, and rspamd missclassifies that email as a spam, will it move it out of INBOX.paypal into Junk? Because if it will, that’s not what the user wants. However, if it won’t – then that’s just perfect.
If it will, I guess I could have the user change their sieve rule to “if it’s an email from *@paypal.com, move it into INBOX.paypal and add X-MoveToJunk:no header” and edit my /etc/dovecot/sieve-after/spam-to-folder.sieve to stop if X-MoveToJunk is “no”.
Well, let’s not jump the shark and test how it behaves first. I likely won’t be able to test for a couple of days. Furthermore, I don’t know how rudimentary the sieve rule editing is in mail clients or the roundcube, perhaps it doesn’t even allow adding headers.
The “/etc/dovecot/sieve only” warning in the post seems rather random and confusing, especially after we just violated that warning by creating a custom /etc/dovecot/sieve-after/ directory. Was there some paragraph preceding it before that got removed?
This rspamd section needs to be expanded. rspamd documentation suggests using three Redis instances, one for Bayes, another for Fuzzy and yet another for everything else. Also, so that Redis doesn’t get OOM-killed, as it stored everything in RAM, with an on-disk backup in case you reboot, it suggests setting memory limits on those instances and setting eviction policies on the non-volatile data only: Bayes and Fuzzy (setting it on volatile data might break things, it says).
https://rspamd.com/doc/quickstart.html#caching-setup
https://rspamd.com/doc/tutorials/redis_replication.html
https://rspamd.com/doc/faq.html#which-backend-should-i-use-for-statistics
Debian’s Redis supports running multiple instances: systemd service, logrotate rules, etc. all account for this possibility. It’s described how to run multiple instances at the top of /lib/systemd/system/redis-server@.service.
Rspamd has an apt repository with recent packages, it can be found here: http://rspamd.com/apt-stable/
Hello for rspamd I’ve created another vhost (cleaner) and I’ve got IMG, CSS and JS with the configuration below.
ServerName rspamd.example.org
SSLEngine on
SSLCertificateFile /etc/letsencrypt/live/rspamd.example.org/fullchain.pem
SSLCertificateKeyFile /etc/letsencrypt/live/rspamd.example.org/privkey.pem
RewriteEngine On
RewriteRule ^/css/(.*) /usr/share/rspamd/www/css/$1 [L]
RewriteRule ^/font/(.*) /usr/share/rspamd/www/font/$1 [L]
RewriteRule ^/img/(.*) /usr/share/rspamd/www/img/$1 [L]
RewriteRule ^/js/(.*) /usr/share/rspamd/www/js/$1 [L]
RewriteRule ^/(.*) http://127.0.0.1:11334/$1 [P,L]
Options FollowSymLinks
Require all granted
For nginx with rspamd.example.org:
server {
listen 80;
listen 443 ssl;
if ($scheme != ‘https’) { rewrite ^ https://$host$request_uri? permanent; }
ssl_certificate /etc/letsencrypt/live/rspamd.example.org/fullchain.pem;
ssl_certificate_key /etc/letsencrypt/live/rspamd.example.org/privkey.pem;
server_name rspamd.example.org;
location /(css|font|img|js)/(.*)$ {
root /usr/share/rspamd/www/$1/$2;
}
location / {
proxy_pass http://127.0.0.1:11334/;
proxy_set_header Host $host;
proxy_set_header X-Forwarded-For $proxy_add_x_forwarded_for;
}
access_log /var/log/nginx/rspamd.example.org-access.log;
error_log /var/log/nginx/rspamd.example.org-error.log;
}
It should be mentioned that you must run a local DNS resolver for some spam lists to work.
Some spam lists use DNS to check IPs, e.g. if someone sends you an email from 157.52.193.92, rspamd will check DNS response of 91.193.52.157.list.dnswl.org, but dnswl.org allows only 100000 queries per day per DNS resolver, so if you use a public DNS resolver like Google’s, CloudFlare’s, or that of your ISP or hosting provider, it’s likely that their limit is exceeded and your query would be ignored.
To check if your DNS resolver has exceeded a limit, run:
host -tTXT 2.0.0.127.multi.uribl.com
Host is provided by dnsutils package. If you get “Query Refused” as a reply, then the limit is exceeded.
To fix this, run:
apt-get install unbound
the default config for unbound is fine, it listens only on localhost and only acts as a local resolver, and add:
dns {
nameserver = [“127.0.0.1”];
}
to /etc/rspamd/local.d/options.inc.
You don’t need to change resolver system-wide, e.g. in /etc/resolv.conf, using the local resolver only for rspamd is enough. However, you can use it system-wide if you want to. Note that if you do, postfix uses /var/spool/postfix/etc/resolv.conf.
To check if it worked, run:
host -tTXT 2.0.0.127.multi.uribl.com 127.0.0.1
You should no longer see “Query Refused”. In fact I get “permanent testpoint” TXT record.
Hi.
Thanks very much for this great tutorial.
I have a concern regarding email attachments. I would like to know what are the best practices in dealing with attachments for both sending and receiving. How do you guys go about it?
How do you control the attachment types as well as blocking user from attaching suspicious items? Do you do it using rspamd or clamav or both? Any thoughts around this?
Thanks.
Hi Cristoph,
First of all, great tutorial!
So you mentioned a way to migrate training data from another instance.
However, this only applies with SQLite. Is there a chance you can expand the Redis section with this?
Thank you
Thanks. Regarding Redis… that would be the best solution. Somehow my spam detection works really badly and I’m still looking for the cause on my test server. So I need to figure that out first.
If someone want whitelisting as I do, please have a look here:
https://gist.github.com/ThomasLeister/f41adad98bb46d0c8418de50b5efb4a0
At the end is a good complete config listing, IMO.
cu
hawe
Logrotate did not work proper. Files are rotated but log isn’t switched to new file. It seems the original prostrate “service rspamd reopenlog >/dev/null 2>&1 || true”did not work. My workaround is to reload config, but this restart the rspamc too.
Did someone know bette?
This is may logrotate file:
var/log/rspamd/rspamd.log{
daily
rotate 4
create 664 _rspamd _rspamd
delaycompress
compress
notifempty
missingok
postrotate
# service rspamd reopenlog >/dev/null 2>&1 || true /dev/null 2>&1 || true <- my workaround
endscript
}
cu
hawe
It seem something has eaten some parts of my log rotate script. Here is the second try:
/var/log/rspamd/rspamd.log{
daily
rotate 4
create 664 _rspamd _rspamd
delaycompress
compress
notifempty
missingok
postrotate
# service rspamd reopenlog >/dev/null 2>&1 || true /dev/null 2>&1 || true <- workaround
endscript
}
cu
hawe
happend again, sorry for that.
postrotate
# service rspamd reopenlog >/dev/null 2>&1 || true
service rspamd reload >/dev/null 2>&1 || true
endscript
Thanks for an excellent tutorial Christoph. Although I have worked with linux systems for years I had never before had any dealings with mail so your tutorial has been invaluable.
In order to get the sending of spam to the Junk folder to work, I found I had to edit 20-lmtd.conf and turn the mail_plugins line to
mail_plugins = $mail_plugins sieve
I also wondered how I could test it out and came across the rspamd testing patterns at https://rspamd.com/doc/gtube_patterns.html
Geoff
First of all thanks for the best mail server tutorial that is out there.
My Debian 10 does not have a sendmail (client) binary. So I cannot do “sendmail john@example.org < gtube.txt" (as you suggest to test spamassassin, eh rspamd) but rather would have to do something like "swaks -t john@example -s localhost -d gtube.txt".
Naively I did an "apt install -y sendmail" without much thinking and uninstalled postfix + postfix-mysql.
In /etc/dovecot/conf.d/90-sieve.conf with the parameter:
sieve = file:~/sieve;active=~/.dovecot.sieve
I got this error below:
Debug: sieve: file storage: Storage path `/var/vmail/example.org/postmaster//sieve’ not found
Jan 11 18:05:25 MailServer-10 dovecot: lmtp(postmaster@xxx): Debug: sieve: file storage: Storage path `/var/vmail/example.org/postmaster//.dovecot.sieve’ not found
Where can I fix this problem? Thanks.
Hi,
I’m not sure if this is a typo but you state that:
If you decide you want to use per-user spam training then add/edit the file
/etc/rspamd/override.d/statistic.conf**/etc/rspamd/local.d/classifier-bayes.conf** and insert:users_enabled = true;
but then you go on to state that:
Edit the file **/etc/rspamd/override.d/statistic.conf** and set the backend there. If you also use per-user spam training this file should now look like this:
classifier “bayes” {
users_enabled = true;
backend = “redis”;
}
This suggests that users_enabled = true; has to be set in both files. Is this true ?
Or is this last block of code actually supposed to be in classifier-bayes.conf ?
Wenn E-Mails im Junk-Ordner liegen und diese gelöscht werden (direkt im Junk-Ordner) dann wird das Ham Script ausgeführt. Gibt es eine Möglichkeit das zu verhindern? Nutzer neigen dazu, Junk-Mails im Junk-Ordner zu löschen.
I may be overheated but, haven’t you mixed up the use of local.d and override.d?
Rspamd’s documentation says [1];
> Settings in local.d will be merged with stock configuration (where possible: ie. the setting is a list [] or collection {}) where-as settings in override.d will always replace the stock configuration.
Where you say the opposite;
> You can either create files in /etc/rspamd/local.d/… which will replace entire sections; or create files in /etc/rspamd/override.d/… which will change only small parts of the configuration.
[1]: https://rspamd.com/doc/quickstart.html#manual-configuration
Thank you very much for your fantastic work with these guides over the years!
I think I’ve followed them along since etch or lenny 😀
Hi Thomas… thanks for the hint. I don’t know how I managed to get that mixed up. You are completely right. I have just fixed that section. And thanks for staying since Lenny… that’s quite some time. 🙂
The exclusion in the sieve filter which is intended to NOT do a ham learn when moving from the Spam folder to the deleted folder, wasn’t working for me. The ${mailbox} variable was always empty, so the deleted mail folder detection never did anything.
Here is my finished working version:
require [“vnd.dovecot.pipe”, “copy”, “imapsieve”, “environment”, “vnd.dovecot.imapsieve”];
if not environment :is “imap.mailbox” “Trash” {
pipe :copy “rspamd-learn-ham.sh”;
}
Here is another script that I used during the debug process which might be helpful for others (this version is specific to the local setup I’m working on, which uses INBOX. folder prefix, and some users use a “Deleted Items” mailbox for historical and localisation reasons).:
require [“vnd.dovecot.pipe”, “vnd.dovecot.debug”, “copy”, “imapsieve”, “variables”, “environment”, “fileinto”, “mailbox”, “vnd.dovecot.imapsieve”];
debug_log “debug TEST1: ${mailbox}”;
if environment :matches “imap.mailbox” “*” {
set “mailbox” “${1}”;
}
debug_log “debug TEST2: ${mailbox}”;
if environment :matches “vnd.dovecot.mailbox-from” “*” {
set “mailboxfrom” “${1}”;
}
if environment :matches “vnd.dovecot.mailbox-to” “*” {
set “mailboxto” “${1}”;
}
debug_log “debug TEST3: ${mailboxto} ${mailboxfrom}”;
if not anyof(
string :is “${mailbox}” “INBOX.Trash”,
string :matches “${mailbox}” “INBOX.Deleted*”
) {
debug_log “treating as ham – since it is NOT being moved from ${mailboxfrom} to mailbox for deleted items, but to ${mailboxto}”;
pipe :copy “rspamd-learn-ham.sh”;
} else {
debug_log “not treating as ham – since it is being moved from ${mailboxfrom} to the ${mailboxto} mailbox”;
}
n.b. to run the debug script, I set:
sieve_global_extensions = +vnd.dovecot.pipe +vnd.dovecot.debug +vnd.dovecot.environment +vnd.dovecot.imapsieve
in:
/etc/dovecot/conf.d/90-sieve.conf
:: No graphs? ::
My “solution” was installing the buster-backport version of rspamd.
This Webinterface is running fine!
1. rspamd comes configured with redis as the backend by default so unless it’s somehow changed to postgresql manually, redis will have to be installed and configured.
2. The instructions for configuring the bayes classifier are inconsistent. Sometimes they talk about classifier-bayes.conf and sometimes about statistics.conf. Better to have a consistent approach to reduce confusion.
When enabling `users_enabled`, should’t the `rspamc learn_spam` have the `–user` provided too?
“`
require [“vnd.dovecot.pipe”, “copy”, “imapsieve”, “environment”, “variables”];
if environment :matches “imap.user” “*” {
set “username” “${1}”;
}
pipe :copy “rspamd-learn-spam.sh” [ “${username}” ];
“`