Server-side filters: Sieve

What is server-based filtering?

You already have a fully functioning mail server that even tags spam. But so far you left the actual task of sorting out the tagged spam to the user. We can do better by setting up server-side filters. The task ahead is to move all spam emails to a user's "Spam" folder. We are using the Sieve feature of Dovecot which is a mail filter like procmail (which does not work for virtual mailboxes). Sieve has a simple scripting language that allows us to forward all emails that are tagged by SpamAssassin to a "Spam" folder automatically. At the end of the /etc/dovecot/dovecot.conf file you will find a "plugin" section. Within there you can define a "sieve_global_path" which points to a default filter file that takes effect if the recipient user hasn't defined any per-user filtering rules.

By the way the term "server-side filtering" means that the rules are executed on the server automatically. As opposed to "client-side filtering" which is configured in the mail program on the user's computer. Apparently server-side filtering is preferred as it happens even if the user is offline. And it gives you interesting options such as a vacation autoresponder or filtering based on header fields.

The global sieve filter script

A basic recipe will do the "Spam" sorting. Create a /var/vmail/globalsieverc file containing these lines:

require ["fileinto"];
# Move spam to spam folder
if header :contains "X-Spam-Flag" ["YES"] {
  fileinto "spam";
  stop;
}

Make sure that the globalsieverc file is readable by the vmail user:

$> chown vmail /var/vmail/globalsieverc

Define this file as a global filtering file within the "plugin { }" section of your /etc/dovecot/dovecot.conf file:

sieve_global_path = /var/vmail/globalsieverc

and restart Dovecot:

$> /etc/init.d/dovecot restart

Send John another spam email:
$> sendmail john@example.com < /usr/share/doc/spamassassin/examples/sample-spam.txt

Then watch the /var/vmail/dovecot-deliver.log file. The last line should look like:

deliver(john@example.com): 2009-07-01 01:00:22 Info: msgid=<GTUBE1.1010101@example.net>: saved mail to spam

This says that the email was saved into the "spam" folder. If you fetch email you should see the email show up in that folder automatically.

Note that different folders only work when IMAP is used to fetch emails. POP3 doesn't have a notion of folders and will only show emails in the inbox. And the user will have to subscribe to the "Spam" folder manually. Alternatively you can add "spam" to the "/var/vmail/example.com/john/Maildir/subscriptions" file which contains a list of folders that John wants to see in his mail program.

Managesieve server

We were talking about per-user filter scripts. How does a user manage such a script? This is done by a low-level interface called "managesieve". Previously additional software like "pysieved" was needed to offer "managesieve". In Lenny's Dovecot this is already supported out of the box.

To enable it you'll have to add "managesieve" to the "protocols" line in your /etc/dovecot/dovecot.conf file. In its "protocol managesieve" section the line "sieve=~/.dovecot.sieve" should already be set. This means that a user's sieve filtering script will be stored in his virtual "home directory". John's sieve script would end up in "/var/mail/example.com/john/.dovecot.sieve".

Managesieve client

Your users probably don't want to learn the sieve filtering language. So they need a comfortable way to administer their filtering rules. This can be accomplished by the "avelsieve" plugin to the "squirrelmail" web mail software. It's assumed that you have set up squirrelmail as decribed in its chapter. Install the "avelsieve" plugin:

$> aptitude install avelsieve

Fortunately that's all that needs to be done. If you login via squirrelmail you will see a new link at the top called "Filters". Click on it. You can now add rules of all kinds.

Note that avelsieve in Debian Lenny currently has a bug (Debian #516198 , Avelsieve #247) that can make existing rules disappear when using Dovecot and TLS. I have prepared a patched avelsieve Debian package that you can install with "dpkg -i" if you encounter the same problem.

31 Comments

Filter if not wrking

Dear,

Thanks for the nice tutorial. I have followed properly but my filtering not working. I can see filter link in squirrelmail but when click on that it say:

//##

Could not log on to timsieved daemon on your IMAP server localhost:2000.
Error Encountered: 111 Connection refused

##//

 Any advice...?

 

/Touhid

Horde/Ingo

Horde Ingo also offers managesieve-support (with tls), wich works like a charme, with this configuration in /pathtohorde/ingo/config/backends.php

/* Sieve Example */
$backends['sieve'] = array(
    'driver' => 'timsieved',
    'preferred' => 'imap.example.com',
    'hordeauth' => full,
    'params' => array(
        // Hostname of the timsieved server
        'hostspec' => 'imap.example.de',
        // Login type of the server
        'logintype' => 'PLAIN',
        // Enable/disable TLS encryption
        'usetls' => true,
        // Port number of the timsieved server
        'port' => 2000,
        // Name of the sieve script
        'scriptname' => 'ingo',
        // The following settings can be used to specify an administration
        // user to update all users' scripts. If you want to use an admin
        // user, you also need to disable 'hordeauth' above. You have to use
        // an admin user if you want to use shared rules.
        // 'username' => 'cyrus',
        // 'password' => '*****',
    ),
    'script' => 'sieve',
    'scriptparams' => array(),
    'shares' => false
);

hooks.php

u need also change config/hooks.php for standard debian packages

class Ingo_Hooks
{
public function transport_auth($driver)
{
switch ($driver) {
case 'timsieved':
$ob = $GLOBALS['registry']->call('mail/imapOb');
return array(
'password' => $ob->getParam('password'),
'username' => $ob->getParam('username')
);
}
return true;
}
}

thst's brecause default settings return only username, without domain, and u need whole user@example.com to authentication

Simple :)

That's simpler than you may think. If you have squirrelmail and avelsieve installed then just create a new rule and tick the "Vacation" box.

Sieve Filter not working

hello,

first of all, thanks for your great work. this  tutorial isactually one of the best ever read!!! thanks for that!

but still i have a problem with the sieve filtering: dovecot seems to totally ignore the rule. if i set a rule with -avelsieve- the rule is correctly saved into userdir/sieve/phpscript.sieve. the symlink is also set correctly userdir/.dovecot.sieve => userdir/sieve/phpscript.sieve.

if i logout of squirrelmail and then login again.... then avelsieve/squirrelmail does not find any filter rules (no filter defined). the filterfile itself still lies in the correct folder... as mentioned above...

when an email is sent to an adress with a filter defined... nothing happens... the mail is delivered and the filter is ignored (i did not use tls/ssl)

in the dovecot config i checked the protocols settings... but all seems ok to me:

protocols = imap imaps pop3 pop3s managesieve
 [ ... ]
protocol managesieve {
  sieve=~/.dovecot.sieve
  sieve_storage=~/sieve
}
 

i hope i provided all informations needed to detect the mistake i made ?!?!?

is there any other setting i have to take care of (in addition to the things i mentioned)??

thanks for any idea  // greetings from cologne

chris

Bug?

Did you read the last sentence on this page regarding my bugfixed version of avelsieve for Lenny? ;)

sieve is working now :) - avelsieve still blind...

 hello again,

i wrote you a message just a few minutes ago... telling you that my dovecot ignores all sieve filters and that squirrelmail tells me that there are no filters after a logout.

well at least the filter work generally. the mistake lay in the dovecot.conf

protocol lda {
  log_path = /var/vmail/dovecot-deliver.log
  auth_socket_path = /var/run/dovecot/auth-master
  postmaster_address = admin@cre8.info
  mail_plugins = cmusieve quota
  # mail_plugins = quota
}

originally i had the plugins defined in two separate lines
mail_plugins = cmusieve
mail_plugins = quota
and this makes dovecot ignore the first definition / replace it with -quota-

now i wonder why avelsieve still does not notice the filter placed in the correct folder??

any advice??

 

 

 

Fatal error: Allowed memory size of 268435456 bytes exhausted

Im spanish and my english is too poor :-( Sorry.

When I click on Filters I have this response:

"Fatal error: Allowed memory size of 268435456 bytes exhausted (tried to allocate 1025 bytes) in /usr/share/squirrelmail/plugins/avelsieve/include/managesieve.lib.php on line 873"

I have Lenny:

matrix:~# dpkg -l avelsieve
Desired=Unknown/Install/Remove/Purge/Hold
| Estado=No/Instalado/Config-files/Desempaquetado/Fallo-config/Medio-inst/espera-disparo/pendiente-disparo
|/ Err?=(ninguno)/Retenido/Requiere-reinst/X=ambos problemas (Estado,Err: mayúsc.=malo)
||/ Nombre         Versión       Descripción
+++-==============-==============-============================================
ii  avelsieve      1.9.7-6        Sieve mail filters plugin for SquirrelMail
matrix:~# dpkg -l squirrelmail
Desired=Unknown/Install/Remove/Purge/Hold
| Estado=No/Instalado/Config-files/Desempaquetado/Fallo-config/Medio-inst/espera-disparo/pendiente-disparo
|/ Err?=(ninguno)/Retenido/Requiere-reinst/X=ambos problemas (Estado,Err: mayúsc.=malo)
||/ Nombre         Versión       Descripción
+++-==============-==============-============================================
ii  squirrelmail   2:1.4.15-4+len Webmail for nuts
matrix:~#

What is it?

Thanks

Pepe

 

I answer to my same one (Resolved mistake) ;-)

Resolved mistake.

It seems to be that one had given an infinite curl, due to a wrong configuration of /etc/dovecot/dovecot.conf.

With the following configuration it works well:

##
## MANAGESIEVE specific settings
##

protocol managesieve {
  # Login executable location.
  #login_executable = /usr/lib/dovecot/managesieve-login

  # MANAGESIEVE executable location. See IMAP's mail_executable above for
  # examples how this could be changed.
  #mail_executable = /usr/lib/dovecot/managesieve

  # Maximum MANAGESIEVE command line length in bytes. This setting is
  # directly borrowed from IMAP. But, since long command lines are very
  # unlikely with MANAGESIEVE, changing this will not be very useful. 
  #managesieve_max_line_length = 65536

  # Specifies the location of the symlink pointing to the active script in
  # the sieve storage directory. This must match the SIEVE setting used by
  # deliver (refer to http://wiki.dovecot.org/LDA/Sieve#location for more
  # info). Variable substitution with % is recognized.
  #sieve=~/.dovecot.sieve

  # This specifies the path to the directory where the uploaded scripts must
  # be stored. In terms of '%' variable substitution it is identical to
  # dovecot's mail_location setting used by the mail protocol daemons.
  #sieve_storage=~/sieve

  # If, for some inobvious reason, the sieve_storage remains unset, the
  # managesieve daemon uses the specification of the mail_location to find out
  # where to store the sieve files (see explaination in README.managesieve).
  # The example below, when uncommented, overrides any global mail_location
  # specification and stores all the scripts in '~/mail/sieve' if sieve_storage
  # is unset. However, you should always use the sieve_storage setting.
  # mail_location = mbox:~/mail

  # To fool managesieve clients that are focused on timesieved you can
  # specify the IMPLEMENTATION capability that the dovecot reports to clients
  # (default: dovecot).
  #managesieve_implementation_string = Cyrus timsieved v2.2.13

}

##
## LDA specific settings
##

 protocol lda {
   log_path = /var/vmail/dovecot-deliver.log 
# Address to use when sending rejection mails.
   postmaster_address = postmaster@xxxxxxx.org

  # Hostname to use in various parts of sent mails, eg. in Message-Id.
  # Default is the system's real hostname.
  #hostname =

  # Support for dynamically loadable plugins. mail_plugins is a space separated
  # list of plugins to load.
  #mail_plugins =
  #mail_plugin_dir = /usr/lib/dovecot/modules/lda

  # Binary to use for sending mails.
  #sendmail_path = /usr/lib/sendmail

  # UNIX socket path to master authentication server to find users.
  auth_socket_path = /var/run/dovecot/auth-master

  # Enabling Sieve plugin for server-side mail filtering
   mail_plugins = sieve
  # global_script_path = /var/vmail/globalsieverc

plugin {
  # Path to a global sieve script file, which gets executed ONLY if
  # user's private Sieve script doesn't exist. (e.g. /etc/dovecot/default.sieve)
  # (v1.0.1 and older called this setting "global_script_path")

  sieve_global_path = /var/vmail/globalsieverc
 
  # The include extension fetches the :personal scripts from this
  # directory. When ManageSieve is used, this is also where scripts
  # are uploaded.

  sieve_dir = ~/sieve

  # The location of the user's active script:

  sieve = ~/.dovecot.sieve

  # Directory for :global include scripts (for v1.1 include extension)

  sieve_global_dir = /var/vmail/sieve
 
  sieve_maxscriptsize = 128
}

 }

It seems to be that I had put in two places simultaneously the same judgments; putting the symbol "#" in front of the duplicated ones, resolved problem, at the moment.

Regards
 

user sieve filtering overrides global filtering

Thank you for this excellent tutorial Christoph!

One small note:

I think you should mention somewhere, that if a user filter file is present (ie ~/.dovecot.sieve) the global filters won't be executed!

It took me quite a while to figure out, why my spam filtering suddenly stopped working ^^

Actually...

Actually it's there. Quoting from above: "Within there you can define a "sieve_global_path" which points to a default filter file that takes effect if the recipient user hasn't defined any per-user filtering rules." Maybe that can be rephrased to make it more clearly?

avelsieve Bug in Lenny

"Note that avelsieve in Debian Lenny currently has a bug (Debian #516198 , Avelsieve #247) that can make existing rules disappear when using Dovecot and TLS. I have prepared a patched avelsieve Debian package that you can install with "dpkg -i" if you encounter the same problem." I think this bug has been closed in the stable release (lenny) at 5th Sep. 2009. See also the following bugreport Debian Bug 542539

sieve location

The following options in /etc/dovecot/dovecot.conf would further streamline the resulting directory structure, since it avoids that managesieve creates directories like /var/vmail/example.com/jonas@example.com :
protocol managesieve {
	sieve=/var/vmail/%d/%n/.dovecot.sieve
	sieve_storage=/var/vmail/%d/%n/sieve
}

plugin {
...
	sieve=/var/vmail/%d/%n/.dovecot.sieve
	sieve_dir=/var/vmail/%d/%n/sieve
}

Sieve - Setup in general!?

Hi there, I have just recently setup my mailserver completely new after this tutorial... I am running a virtual "Debian GNU/Linux 4.0" here, with: * Dovecot 1.0.rc15 * Postfix 2.3.8 my managesieve plugin is telling me that it's not able to connect to sieveserver and on enabling the managesieve protocol in the dovecot onfig file on restarting dovecot it fails because the protocol managesieve is unknown. Will I have to compile dovecot from scratch with sieve included?

The driver said: (QUOTA) “Quota exceeded”

Please help:
My ingo (Horde frontend) by some users says when they trying to put some extra emails in whitelist or blacklist:
There was an error activating the script. The driver said: (QUOTA) “Quota exceeded”
The rest I got working. I see that the size of the .dovecot.sieve file is 99.865 bytes. I don’t use quota for Maildir. Please let me know how to increase that quota. I was searching everywhere – haven’t found.

# Error Driver (QUOTA) "Quota exceeded"

I have
$conf['storage']['params']['driverconfig'] = 'horde';
$conf['storage']['driver'] = 'sql';
in /var/www/html/horde/ingo/config/conf.php
I have 2,146 records for one of the users in ingo_lists table of horde DB.
The size of the script (.dovecot.sieve file) is 99.965 bytes.
When trying to activate script I get the following
# Changes saved.
# Error There was an error activating the script. The driver said: 
(QUOTA) "Quota exceeded"
Please let me know where I can change that size of QUOTA

global sieve filter ignored

First, this is a great tutorial! Lately though I considered filtering spam with a global sieve filter. I created the /var/vmail/globalsieverc file which contains: require ["fileinto"]; # Move spam to spam folder if header :contains "X-Spam-Flag" ["YES"] { fileinto "spam"; stop; } Made certain it was owned by vmail. Changed the last stanza of /etc/dovecot/dovecot.conf which in labeled plugin: sieve_global_path = /var/vmail/globalsieverc Restarted dovecot Sent the spam email (GTUBE) The result is here: 2010-07-19 03:35:19 deliver(larry@larkat.com): Info: msgid=: saved mail to INBOX Do I need to create the spam folder first?

global sieve filter ignored

How is your "spam" folder named ? Mine is labeled as "Junk" in both Squirrelmail and Roundcube. So i replaced "spam" with "Junk" and the the spam was filed into Junk folder. You might also need to check the message in the dovecot-deliver logs (if that is what you are using.

Sieve namespace

I had to change "fileinto 'spam'" to "fileinto 'INBOX.spam'", or it'd throw an error in dovecot-deliver.log about not finding the namespace for the mailbox.

> Error: sieve: msgid=unspecified: failed to find namespace for mailbox 'spam'
 

 

Move SPAM to spam

Actualy there are an error in the tutorial

If you follow the tutorial and install&configure AMAVIS the amamavis is marking the spam as 

X-Spam-Status: Yes

The other things you should take care: permission of the sieve rc and location

If you want that spam to be in Spam modify accordingly (spam -> Spam)

Learning Spam and Ham

Thanks for this great tutorial I'm following for some time now.
Most of our users use Apple's Mail application so I sort out Spam into a folder called Junk. This can be synchronised with the Mail-app. If the user gets a Spam delivered into his Inbox, he can kick it into the Junk-Folder just by klicking the "Werbung"-Button (german for Ad). If Mail is configured to sync this folder back to the server, there is a folder full of qualified Junk at the server. It would be consequent to use the Junk-Folder as a Spam-Learning source. This is no problem as this Spamassassin-Feature is well documented on the web. But how do I manage to use this on variable foldernames? Imagine, another user sortes his Spam into a folder called "Spam" (via managesieve), because his E-Mail Client does so…
Do you have an idea, how to get this working? Sebastian

Sieve reply from address

When using sieve successfully with vacation auto-reply I have a problem: the reply message has "from" address in the form: username@dovecot.server.hostname not users.mail.address@our.domain

The dovecot.conf lda-section postmaster address is configured as postmaster@our.domain so i thought it would send the reply message with that as from address - or at least with the same address it got the mail (users.mail.address@our.domain).

Port 1490 for avelsieve

I had to change the port in /etc/squirrelmail/avelsieve-config.php (auth failure) :

 $sieveport = 2000;

 

I do not know why Debian changed the default port?

Filter not working

I am using debian lenny box. All the configuration is ok, no error message but filter is not working. If i set any filter, nothing happen.

 

where to check for errors??

 

/Borua