When you are dealing with mail servers you probably know that IP blacklisting is the best way to defend incoming spam. Surely you will have a few false positives which means that an IP blacklist blocks incoming legit email. Blacklists need to be aggressive these days because spammers are, too. We may not care for blocked emails from tiny email servers in the darkest corner of earth. But when it comes to accidentally blacklisting large and well-known providers then you users may lose their sympathy. And by “well-known providers” I mean such organisations as Google, GMX or Deutsche Telekom. I don’t like any of them and it’s beyond my belief how anyone with an IQ above room temperature uses them for their (confidential) email communication. But they can surely be trusted to deal well with spam complaints and try hard to block outgoing spam. Considering the number of email users they have I would even say that they react fast and have a low amount of spam coming from them. Deutsche Telekom ist also an internet access provider and I don’t even want to start imagining how many malware-infested Windows workstations send their spam.
However recently I was positively surprised by an email from the Deutsche Telekom postmasters. They noticed that I blocked email from them because one of their outgoing IP addresses was listed on the SpamCop mailing list. And so I received this email:
Dear Sir or Madam, esteemed colleagues, we have received reports from our customers indicating that your mail system is blocking at least one of our mail server IP adresses from sending mail to your system. Example: | <firstname.lastname@example.org>: | host fry.workaround.org [220.127.116.11] said: | 554 5.7.1 Service unavailable; Client host [18.104.22.168] blocked using | bl.spamcop.net; Blocked - see http://www.spamcop.net/bl.shtml?22.214.171.124 | (in reply to RCPT TO command) The rejection occurs apparently due to a listing of our server on the so-called 'SpamCop Block List' [SCBL]. There is no option for a delisting ahead of time at SpamCop. In addition SpamCop refuses to increase the thresholds for our systems. To clarify the dimension we are talking about: We receive one to four complaints a day to average through SpamCop about our 14 servers for outgoing mails of ca. 20 million private and business customers. In case of a qualified complaint - most of the times it is qualified - we block the compromised email account from sending email at once, until the rightful bearer of it has changed his passwords. Outgoing emails are directed compulsory through a spam filter. But of course there is no spam filter which separates 100% spam. We go to any lengths to reduce the load of other systems to the least possible degree. But you cannot change the parameters in an arbitrary manner because an automatic spam filter would reject more and more emails as 'spam' which aren't spam. It is the responsibility of the operator of the receiving system if he rejects all emails from a system due to some unwanted emails. Unfortunately it swallowed up again and again that SpamCop itself clearly does NOT recommend the use of their SCBL in this way. SpamCop rather warns the administrators against the consequences which actually took place here: Implement the SCBL to Filter Spam The SCBL aims to stop most spam while not blocking wanted email. This is a difficult task. It is not possible for any blocking tool to avoid blocking wanted mail entirely. Given the power of the SCBL, SpamCop encourages use of the SCBL in concert with an actively maintained whitelist of wanted email senders. SpamCop encourages SCBL users to tag and divert email, rather than block it outright. Most SCBL users consider the amount of unwanted email successfully filtered to make the risks and additional efforts worthwhile. The SCBL is aggressive and often errs on the side of blocking mail. When implementing the SCBL, provide users with the information about how the SCBL and your mail system filter their email. Ideally, they should have a choice of filtering options. Many mailservers operate with blacklists in a "tag only" mode, which is preferable in many situations. Source: http://www.spamcop.net/bl.shtml It is actually an official internet standard of the 'Internet Engineering Taskforce' (IETF) that the operator of the receiving mail server (and not the operator of a block list) is solely responsible for the rejection of emails: It is the responsibility of the system administrators who adopt one or more DNSBLs to evaluate, understand, and make a determination of which DNSBLs are appropriate for the sites they administer. If you are going to allow a third party's information to guide your filtering decision-making process, you MUST understand the policies and practises of those third parties because responsibility for filter decisions remains ultimately with you, the postmaster. Source: http://tools.ietf.org/html/rfc6471#section-1.2 The mailservers ... 126.96.36.199 - 188.8.131.52 (private & small business users) 184.108.40.206 - 220.127.116.11 (private & small business users) 18.104.22.168 - 22.214.171.124 (business users) ... of Deutsche Telekom AG are used by approx. 20 million individual customers and some hundred thousands of domains of business customers. According to conservative estimates, about one percent of all end-user computers are infected with malicious software. Therefore we cannot avoid relaying a certain proportion of "spam". But please bear in mind that not only our millions of customers are the one to suffer for a too strict blocking policy but also all the customers of other providers (which are using your blacklist) when they lose the email contact to their friends, business partners, customers or employees. Deutsche Telekom ... • applies content filtering for outgoing and incoming emails • avoids backscattering by refusing acceptance of emails to unknown users and of emails categorized as "spam" by the content filtering system • put locks to spamming email accounts without prior warning to the responsible bearer of the account. Written as above, we do enforce our AUP, which includes disconnection. If there is an abuse issue that we need to deal with, please send full header information and message to email@example.com and our abuse team will investigate immediately. Apropos: The possibly complex maintenance of an own whitelist can be circumvented by the probably most relevant whitelist, the DNSWL www.dnswl.org. At http://dnswl.org/s?s=1972 you'll find our mailservers. Thanks in advance for a complaisant examination of our concern! Kind regards t-online.de Postmaster
I thought I’d quote the entire email here – bear with me. It taught me two things: first they really care for other organisations postmasters – that gained them extra karma points. And they pointed me to a whitelist that I wasn’t aware of: dnswl.org. That whitelist lists organisations that are high-volume providers that are at the same time known to deal timely with spam complaints. So using that whitelist is a pretty good idea to limit the number of false positives when using black lists. After all famous realtime blacklists like Sorbs or SpamCop have an unfortunte history of blocking large ISP due to automated processes and that annoyed my users big time.
Okay, let’s get a long story short. If you are using Postfix as your MTA then this is the simple setting to use the whitelist in addition to your blacklists (relevant change is printed in bold letters):
However the whitelist is even more fine-grained. There are four different levels of trusted ISPs. So I suggest you hop over to dnswl.org and check them out.
I hope you find this hint useful and your users will probably appreciate it, too.
Update: I had pretty mixed experiences with dnswl.org. For some reason their name server often responded incorrectly making incoming emails get stuck with a transient (4xx) status code. So I had to remove it from my production configuration again. Considering that blacklists and whitelist do not work very well as a single mean I nowadays recommend using SpamAssassin that takes blacklists into account but won’t reject an email just because of one of the lists. See the ISPmail Jessie tutorial for details on the setup.