ISPmail tutorial for Debian Lenny

This tutorial is for the former stable version "Debian Lenny". If you are using "Debian Squeeze" then please follow the new tutorial.

A spanish translation of this tutorial is also available - courtesy of José Ramón Magán Iglesias.

What this tutorial is about

You surely know the internet service providers that allow you to rent a domain and use it to receive emails. If you have a computer running Debian which is connected to the internet permanently you can do that yourself. You do not even need to have a fixed IP address thanks to dynamic DNS services like dyndns.org. All you need is this document, a cup of tea and a little time. When you are done your server will be able to...

  • receive and store emails for your users from other mail servers
  • let your users retrieve the email through IMAP and POP3 - even with SSL to encrypt to connection
  • receive and forward ("relay") email for your users if they are authenticated
  • offer a webmail interface to read emails in a web browser
  • detect most spam emails and filter them out or tag them

License/Copyright

This tutorial book is copyrighted 2009 Christoph Haas (email@christoph-haas.de). It can be used freely under the terms of the GNU General Public License. Don't forget to refer to this URL when using it. Thank you.

Changelog

  • 17.6.09: Lenny tutorial gets published.
  • 19.6.09: The page on SPF checks is temporarily offline. A new page on spam/virus/phishing fighting is currently prepared.

Things you will need

The server setup described here is totally standard work for a professional system administrator. Depending on your level of knowledge and experience you may walk through this document easily or curse the author and fail miserably. You will need to know or learn about different topics regarding basic system administration (e.g. how do I edit a text file, where are my log files), DNS, SMTP, MySQL and POP3/IMAP. You also need root access to an existing Debian Lenny server or be able to install one. Also you will likely have to change DNS records for your domain. If your server is protected by the firewall make sure you can change its rules. Going through this setup takes you between 2 hours (for a professional) and a week (if you are a beginner who will inevitably make mistakes). So make sure you are not in a hurry.

About this document

Many years ago I wanted to turn my Debian server into a mail server with virus scanning, spam detection, email forwarding ("aliasing"), POP3 and IMAP access and a webmail service. All the components were there but it took a while until they worked properly together. So I summed up my desk full of scrawly notes into a tutorial that has become pretty famous. According to my web server statistics it's the main reason why people visit workaround.org.

This document is not a simple copy-and-paste tutorial where you just copy the commands from the web site and run it on your server. Instead it will make you understand the different components that you are setting up. In the end you will be skilled enough to debug problems yourself. If you feel you need help with your setup then try the hints in the Troubleshooting section or ask on the mailing list. The setup in this tutorial has been tested very thoroughly by many readers. Unlike many other Postfix tutorials on the internet this is already the fifth edition. Writing this tutorial took a lot of work so these are not just quick draft notes thrown together but a consistent document guiding you.

The whole tutorial is split into several chapters. Please use the links on the right side or below to navigate through the tutorial. If you prefer all content on a single page (e.g. for printing the tutorial) then use the "printer-friendly" link below. You are also invited to comment on the pages - just click on the "Add new comment" link at the bottom.

If you like the tutorial then a tiny donation is appreciated to run the test server website hosts and to pay for the internet connection.

113 Comments

all on one page

Where is there a printer friendly version of the whole guide? the printer friendly links I've tried lead to a version for just that page.

One of the ways I've often used your other mail server guides is as a reference, I'll search the guide for any mention of a particular word so that I can quickly see everything you advise for that particular thing. This is _really_ useful for helping me through issues. The Lenny guide doesn't seem to enable this.

Pete Boyd

Hi, I tryed to upgrade from

Hi,

I tryed to upgrade from the etch Tutorial. Im using the etch Tutorial and GRsoft Mail-Manager from Peter Gutwein *thx*.

So I modified my mysql Tables in etch.

For all the Peopels with the same config, you get an error if you try to upgrade your sql tables:

This doesn`t work:

 

-- Remove the 'user' column
ALTER TABLE virtual_users DROP user;

I`ve done a new Installation with ne new Tutorial. Thanks for this realy great Tutorial.

Gruß Sven

This is the error

This is the error I get.

SQL-Befehl:

ALTER TABLE `virtual_users` DROP `user`

MySQL meldet: 

#1062 - Duplicate entry '1' for key 2

 

Basic Debian Installation - Choice of Filesystem

"XFS is way faster when dealing with lots of small files"

I am a little confused, because usually all the info on XFS always only tells something like "XFS is way faster when dealing with lots of LARGE files" (or as seen on Wikipedia: "XFS is particularly proficient at handling large files and at offering smooth data transfers"). Just did a read-up on XFS and typical filesystem comparations, and there are some downsides to XFS which make me wonder, if it really is suited best for a mailserver:

- Not crash-proof
- High CPU consumption
- Not shrinkable
- No undelete tool
- Creation and deletion of directory entries can be a much slower metadata operation than other file systems
- Failure-handling policies can be improved

There seem to be more reports on dataloss with XFS, so a power-outage without UPS practically ensures data-loss, if the machine was under load at that time (yes, every server should have a UPS, mine has).

I am usually just sticking with ext3. Your mentioned problem with lengthy fsck on boot can simply be avoided: "tune2fs -o journal_data /dev/sdX" & "tune2fs -i0 -c0 /dev/sdX", taken from the ext3 FS Hints & Tips in Gentoo Forums.

Performance-wise: If XFS really does better on lots of small files as well, I may take a closer look, but I am still not convinced.

XFS not a good choice for a mailserver

In the meantime, I did some more reading, and I came to the conclusion, that XFS is not a good choice for use on a mailserver, especially when using LVM and not having a Hardware-RAID-Controller with battery backed controller cache and cache in write back mode, see the full details here: http://forums.gentoo.org/viewtopic-p-5876285.html#5876285

Most people do not seem to be aware, that XFS

- was initially designed with large databases & really large files in mind (hence the default settings on filesystem creation are not optimized for a normal setup)
- does not perform well with the default settings, if used on a regular sized filesystem (it can even be worse than ext3, especially on file deletion)
- can cause serious troubles if not taking the barrier/nobarrier & write cache matter into account (that's the issue with dataloss and binary-zeroed-files on unclean shutdown / power-off)
- gets even more complicated when using in virtualized environments (again that matter with nobarrier, blockdevice virtualization and write caches)

Since I intend to stetup the mailserver in a VM with KQEMU virtualization on top of LVM with underlaying luks-encrypted Hardware-RAID1, that's pretty much a no-go (the same goes for my rootserver, which is running a similar setup, but with KVM virtualization and Software-RAID1).

Of course, filesystem choice can end in religious endless discussion, all I'm saying is, that maybe it's better to leave out the filesystem recommendation, or give a clear warning, that XFS is not to be used lightminded.

XFS vs. Ext3

Since I also did some research on that matter I would like to weigh in on that debate. But first of all I would also like to write a big thank you for the manual. Awesome stuff! To make things short: Ext3 for desktops, XFS for servers. If you are really good and exactly know what you are doing you can choose whatever you think is best. But as a rule of thumb I would stick with that. Ext3 is good when you have a desktop that crashes once in a while. The frequent (every 30 mounts or so) fscks will take care of issues. A server runs for a very long time without reboot. And when it crashes there is usually someone there to take care of issues should they arise. An XFS system will reboot without problems, but you should take the fs offline and check it (only takes a short time) after a bad crash. People tend to do that with servers. At least with important ones. But clueless people don't do that. And then Ext3 is a much better choice. So there is your reason for the default.

JFS experience

JFS fs size limit is 4PB. I use it everywhere since IBM published first linux kernel JFS patch. Without problem, and data loss. (At now: 5 servers (file store and mailer), ~8T disk backend.) The biggest proceeds: perfect and extreme fast fsck. Otherwise the real bottleneck is the network. Always. :D

nfs is not so bad!

hello,
I'm not so negative about ext3 (here I have more than 10000 servers and I see only a few problem with ext3, which is the most used, compared on problems with other fs.... everyone use upss for servers, but powerdown is not the only cause of crashes, and a fs shuould be alive after a crash!)
here an article about filesystems where we can understand why all the main distributions choose ext3 as default FS (of course it's only a default, if you know what you are doing you can make different decision for different particular use): http://www.linux-mag.com/id/7416/1/
yust my .02 cent
emilio

Great tutorial!

I wish i had known this a few years ago, now that i figured most of this the hard way.

While scanning through the tutorial, there's a few things i noticed:

- Why SPF? I feel that DKIM is a superior system, in the unlikely case that some kind of verification on the mta level will catch on.
- Is there still a benefit in using amavis? I'm a happy user of clamav-milter and spamass-milter, which ist just insanely simple to set up.
- Have a look at the (backported) clamav-unofficial-sigs package. This turns clamav into a *very* powerful spam filter without needing any aditional configuration to mess with. I had no issues with false positives up to now.
- Another powerful and zero-effort alternative to greylisting comes with the policyd-weight package, which does all kinds of checks as well as dnsrbl lookups. Again it didn't give me one false positive in years.
- Have a look at the RoundCube Webmailer. It might not be as proven as squirrelmail, but it's got the whole ajax / web 2.0 bling.

In the end this just proves that as usual there's more than just one way to do things, if you start to toy around, everyone will be able to find his/her perfect setup without too much hassle.

I suggest ASSP for anti-spam to every sysadmin in the world.

I totally agree with the drawbacks of Amavis*.

My favorite anti-spam approach is ASSP. It takes a while to learn and use all the features, and it needs some time tweaking it, but then it totally rocks!

The biggest advantage it has imho is that (most) spam will not enter the postfix queue! With most common setup with amavis (and spamassassin, dspam, bogofilter etc.) spam gets accepted and only then processed. With ASSP (and Hermes and afaik also with postfix-fwd and postfix-policy) spam does *not* get accepted, processed and then delivered (Except for the spam-collection postmaster if needed).

http://assp.sf.net/

Kind regards, Albinootje.

 

spamass-milter only on incoming mails

How did you manage, that spamass-milter is only used on incoming emails, and not on outgoing?

I spent all morning playing around with it, but there just seems to be no way. Defining the milter in main.cf  results in spam-check even on outgoing email, and that's what amavis is able to prevent. There seems to be no way to configure spamass-milter the same way as amavis, and you surely do not want to start with manually set trusted_networks or whitelist settings. The approach for applying content_filters only on incoming mails from external (as shown in the comments on the amavis page in this tutorial) does not seem to be working for milter-application.

I thought, using spamass-milter + clamav-milter is the superior way to amavis + spamass + clamav, but it seems not to be.

On the other side, it would make sense to virus-scan outgoing mail as well.

If there nevertheless should be a way, to restrict spamass-milter on incoming mails, please let me know.

Timestamp

Any chance you could put a timestamp on the tutorial so we can see when it was last updated?  I know you're still making changes/additions and it would be helpful in knowing if there was something new.  Unless it's there and I can't find it.

Just to chime in on the filesystem debate, I think another downside to ext3 vs. xfs is that ext3 allocates 5% of space as a reserved region (for the superuser in case you fill up the volume) so you end up with less usable space with ext3.  Supposedly you can override this but I've never been successful.  I use xfs for pretty much everything but I agree that when using lvm it's not ideal because it doesn't have the ability to shrink.

I've run a mailserver based on the etch tutorial for a few years and it's been flawless.  One thing I'd like to see is best practices for keeping spamassassin and clamav up to date.

I run my mail server in a VM (using kvm).  A neat idea might be to provide a virtual appliance of the full setup, although walking through the tutorial is extremely beneficial.

Thanks for all your work.

Amavis not scanning messages

After following this, I found that headers weren't being added to incoming messages and Amavis didn't appear to be doing anything with incoming mails.

I found that vim /etc/amavis/conf.d/15-content_filter_mode had the scanning of messages disabled, so just needed to uncomment a few lines to get things up and running!

Cheers,
Mark

Thanks, but a little question

Thank you very much for this excellent tutoria, it was very useful to help me setting up my new mail server at work, but i've got a little (security) question. If a user (or a spammer) telnet on my server on port 25 and send the commands:
 
mail from:user1@mydomain
rctp to: user2@mydomain
 
the system don't need authentication and anybody can send fake mail (in my domain) and stole user1's mail address. Is there a way to avoid this behaviour?

Dovecot password query

 Great tutorial, I'm using it since Debian Sarge.

I only found that there is something small missing in the Migrate from Edge section.

I had to update the password query to use the table 'virtual_users' instead of the 'view_users'. After I changed this, everything worked fine again :)

keep up the good work ;)

X-Original-To

Hi!

I followed the last 2 Howtos for ispmail. But with the new one I miss the "X-Original-To"-Header. In your example it is set, but not on my system. Do you know how i can get it back?

 

Greetings

X-Original-To

In the master.cf you need the "O" flag:
dovecot unix - n n - - pipe
flags=DORhu user=vmail:vmail argv=/usr/lib/dovecot/deliver -d ${recipient}
read man 8 pipe for more flags...
So I wonder why the "X-Originial-To"-Tag is in the Tutorial..


Greetings

Hallo Christoph! Meine

Hallo Christoph!
Meine Mailserver (3) laufen auf Basis Debian Etch, mit dem Wissen, welches ich mir aus deinem vorhergehenden Tutorial angeeignet habe. Dafür möchte ich mich recht herzlich bedanken. Danke auch für diese, lange ersehnte Ausgabe. Ich weis, wie viele Stunden an unbezahlter Arbeit darin im verborgenen liegen. Leider, so scheint es mir jedenfalls, verzögert sich der durchbrechende Erfolg des Tutorials diesmal auf Grund verschiedener Grundsatzfragen, wie - was wäre das am Besten geeignete FS oder die Performance diverser SQL Statements. Serveradministratoren sollten selbst in der Lage sein, über das für sie geeignete FS zu entscheiden - und über die Vor- und Nachteile von CONCATs und JOINS sollte folgender Gedankenanstoß anregen: bei 200.000 SMTP Connects pro Tag, von denen wahrscheinlich 80% durch eine effiziente Zusammenstellung der Postfix Policy Delegation abgelehnt werden, bevor es zu einem Zugriff auf die DB kommt, verbleiben gerade 0,47 Abfragen pro Sek. an die DB. 
Auch das beste Shell/Bash Script, zur Verwaltung der Mail-Benutzer, für einen Kunden mit 200 km entfernt liegendem Standort und mangelnder Berichtigung am Port 22, hilft nicht auf die Beine. Bitte nimm meine Worte nicht einfach als Kritik entgegen, denn das ist in keiner Art und Weise meine Absicht. Um Kritik zu üben, reicht mein bescheidenes Wissen lange nicht aus. Was fehlt ist einfach der Pep von  dovecot_common, libmail-dkim-perl oder postfix-policyd aus den Backports, Roundcubemail mit „manage sieve protocol“ Plugin (auch Vacation mittels sieve rules ohne Probleme möglich) und Postfixadmin, mit oder ohne hardcoded SQL-Statements (und ohne vacation.pl). Dass dies funktioniert, zeigen andere vor (iRedMail), zu erklären, wie es funktioniert, dazu benötige ich, oder vielleicht ist jemand meiner Meinung, dann wir - ein Tutorial.
Danke Alois

Manage your email accounts

Hallo, danke erstmal für das Klasse How-To, hat mir schon mehrfach geholfen.

Nun zu meiner Frage, wie steht's mit den neuen Tools um die Mailaccounts zu managen?

Ich weiß, bei OpenSource und erst recht bei Ein-Mann Projekten: It's done when it's done.

Vielleicht könnte ich ja auch als eine Art Betatester helfen.

LG

   Thomas

Very nice!

I like the ISPwebadmin frontend you've written very much.

Is there a way to get it to run on the https protocol instead of http? I've tried setting "use = egg:Paste#https" but that gives me errors when I try to start the service. I'd like sensitive information such as passwords and even email addresses to be encrypted.

Thanks

I got it working with SSL.

Now, I have another problem. I keep the passwords for my users in plaintext, but your ispwebadmin python script automatically uses md5 or some other hashing. Is there a way to force it to update people's passwords as plaintext?

ISPwebAdmin Vs Postfixadmin

Thanks for the "How to" it's really helped me out. Is there are any particular reason to use ISPwebAdmin rather than Postfixadmin? I've used Postfixadmin on previous Etch servers and it worked well - is there anything specific I'd need to do to run it rather than ISPwebAdmin? Thanks Wych

postfix main.cf virtual parameters sequence

when i config the postfix main.cf, if i put "virtual_mailbox_maps" before "virtual_transport = dovecot" the postfix send out error messages:"postfix/virtual[7950]: fatal: bad string length 0 < 1: virtual_mailbox_base =,postfix/master[4076]: warning: process /usr/lib/postfix/virtual pid 7950 exit status 1, postfix/master[4076]: warning: /usr/lib/postfix/virtual: bad command startup -- throttling". if i put "virtual_mailbox_maps" after "virtual_transport = dovecot" , all is ok! why?

Excellent Job!

This is an excellent article! And it works too!

I'm a former qmail / courier / vpopmail admin who has finally converted to postfix & dovecot. This only took a few hours to setup and saved me hours of upgrading, compiling & debugging my former setup.

Thanks so much for going into so much detail. I rarely get past scratching instructions on little bits of paper. This flows very well and is easy to understand. Nice writing.

Thanks again!

use the iredmail

you can use iredmail  (http://code.google.com/p/iredmail/wiki/Installation_on_Debian)

you can config the email system manual according to the document, but you also can use the iredmail script.

you would found use script, just need 5 minute. and also you can choose use ldap or mysql.

fetchmail

hi will there be a tutorial how to integrate fetchmail in this configuration in future?

would be nice 

Pages