I feel obliged to briefly discuss the use of Debian packages in contrast to just pulling software from their website and installing it. Why is this important? Well…
“The roundcube software in this guide is totally outdated. Just go to roundcube.net and download a new version.”
(Frequent advice in the user comments of these guides.)
…or an even less friendly quote…
“I wish I could convince Debian maintainers to stop providing Rspamd in Debian repos. Probably, I should change a license for that, but I’m really tired of that crap…”
Vsevolod Stakhov – principal developer of rspamd
So what’s the deal? Shouldn’t we just use Debian packages where we find it comfortable and install any other software in a newer version? Sure, nobody keeps you from doing that. Consider these aspects though…
Advantage: cool new software
That advantage is pretty obvious. Why use the old dusty Roundcube version that comes with Debian Buster if you can get the sexy shiny new version directly from roundcube.net? It’s just a tarball that can be extracted somewhere and works easily. Some software even comes in handy APT repositories so it’s simple.
Disadvantage: conflicts with the APT package system
Debian arguably has the best software package management of all operating systems. It keeps track of all files that are installed on your disk. And it knows how to upgrade to newer versions without damaging your configuration.
There are defined locations on your disk (according to the FHS) like /opt or /usr/local where additional software can live. That is certainly true but then you need to make sure that you do not have the same APT package installed, too. Otherwise you would have to specify exactly which copy of the software you want to use. This can lead to annoying mixups.
Disadvantage: conflicts with other packages
Debian package maintainers take great care that packages in a stable Debian distribution work well together. Dovecot in Debian Buster comes with helpful instructions on how to use the Postfix version in Debian Buster. If you take packages from other sources then you may find that they use unusual paths in your file system. Or that they somehow do not work with the “old” version of other packages that you installed from the Debian sources. This can quickly lead to an update frenzy where you need to update several unrelated packages that in turn break other packages.
Disadvantage: no automatic integration
Debian packages are not just a different format of the original (“upstream”) software tarball. Debian developers make sure that every part of the software gets into the right spot. For example configuration data always ends up somewhere in /etc. Also it integrates nicely with other parts. Debian’s Roundcube package offers working configuration snippets that plug into your Apache web server. If you install the software on your own you will lack that.
Disadvantage: no security support
Especially in public-facing services like mail servers this is my most important issue. Surely you can install the newest version of Roundcube. But as a responsible system administrator you will need to make sure that you check for updates every day and install them when they pop up. Debian packages get updates automatically. Your manual installation will not.
And that is no matter of paranoid phantasy. I have lost several systems in my life because the update mechanism failed and script kiddies found open security issues and did bad things on my servers.
And I have also broken my servers because I gave in to the temptation to use newer software. Some developers provide their own APT repositories and actively discourage you to use the packages provided by Debian. In fact I was literally shocked to read that the rspamd principal developer Vsevolod Stakhov not only discourages but even threatens Debian users if they dare using Debian packages. He even considers changing his license to exclude Debian from using his software. Of course bug reports of Debian packages should be sent to the Debian project first. However developers must not force users to ignore basic principles of Debian. In fact this is the most unprofessional behavior of an open source developer I have seen in the last years. Yikes.
That aside, developers hardly ever fix issues in old software. That’s just not fun. Developers like to go forward. While they implement new features they also fix issues on their way. I totally understand that motivation. But that way they force users to accept their changes and new features even if the users just want to fix the security issues.
So that way you are forced to upgrade. In Debian’s security updates you will get patches for the existing software version. If you installed version 1.3.8 then you will stay on version 1.3.8 – just with security fixes applied. You can rely on your system that your configuration will work throughout the entire lifetime of that Debian release. However if you installed version 1.4.x manually then you will have to use every new version that the author publishes. Security issues are then fixed by updating. And if that changes the configuration or the general UI then you and your users will have to deal with that. Or you just ignore further updates and are likely to find your server part of a botnet that keeps computing Bitcoins.
So be aware that when you manage software yourself you will have to check for updates. And expect broken servers.
The meaning of ‘stable’
However let me get one term straight: “stable”. Some consider software versions that are included in a stable distribution more reliable than those that get updated frequently. Unfortunately that is often just wrong. Think of stable as long-lasting. That version will stay on your system and even security updates will change as little as possible to prevent surprises. Please do not mistake it for a warrant to get you more secure software. Unfortunately there are software packages in Debian that have not been updated in years and have been suffering from security issues for a year. (Yes, I mean you, solr.) Even the rspamd package in Debian Buster is a bit buggy and need manual fixing (as explained in this guide). Still if you are a busy system administrator and have to puppeteer a hundred servers then you will want to stay with stable because it makes your life easier. And if you got something running it will keep running until the next stable release.
Conclusion
Throughout this guide I will always make you install Debian packages. Also please expect better support through the comments sections if you stay on the trail of what is good and righteous. 🙂
For more information see also the Debian FAQ on security.
Oh, and Roundcube was just a random example. There’s nothing wrong with Roundcube. 🙂
This is totally fair. I remember in the last guide I posted a similar comment about going rogue to install RainLoop or something like that. I get your position, and respect it.
Thanks for the comment. I don’t mind installing newer packages. I do that, too, on internal systems. Just wanted to make sure what consequences it has.
What you’re describing, especially about rspamd, is typical of a new mindset. The confusion between open-source and free software is probably not unrelated to this. A man like RMS is an exceptional programmer, but above all a man of exceptional moral rigor. The developer of rspamd is undoubtedly a good programmer, but he lacks this moral rigor.
For years I’ve been using Gentoo, which also has an excellent package versioning system, but it needs a lot of attention. A kind of linux from scratch that would handle dependencies.
But if you want to become the maintainer of all the packages you use, then 24 hours a day won’t be enough, and you won’t have a second to do anything else with your computer than keeping it up to date…
And I’ve been around Unix systems since 1980 or so, and I’m still the living proof that what’s old isn’t necessarily obsolete! 😉
I agree with all that you’re saying, Christoph.
What I do on my own servers when I really, absolutely need the latest and greatest package version (e.g. Nextcloud) is I spin up a Docker container.
Fortunately, most open-source server side projects do have an official Docker image these days, and you can set them up without the risk of screwing up your system’s package dependencies.
Containers such as Watchtower ensure that the latest version is always installed, but of course when doing that it’s important to specify a version tag when initially building the Docker container, so that there won’t be unexpected upgrades to new major versions which could contain breaking changes.
It does require some more attention to a project’s development roadmap, because you do have to keep track when a version of a project you’re using reaches end of life and upgrading to a new major version is advisable due to no patches being supplied for the older version.
This may be too much effort for admins running huge server clusters, but for me (with only a handful of critical services running on a handful servers) it’s definitely manageable.
BTW, I was *EXTREMELY* skeptical of the whole containerization movement when it first started, I used to think that it maybe could be useful for deploying services on a huge set of servers quickly for easy scalability, but my feeling was that the overhead of creating a whole container environment for just one service would not justify the benefits.
Having used Docker for over a year at this point though, I must say that it has made my life so much easier, especially when installing various services with conflicting dependencies on the same machine, or in case I want to try out a project with lots of dependencies (GitLab and Logstash are great examples here) while being able to easily get rid of it if I want to.
Ultimately it is up to each and every admin to decide what strategy is right for them, but in cases where installing the distro-supplied package is not feasible, I personally would always prefer setting up a container over installing the package directly onto the system by pulling from GIT or similar.
Also thanks for writing up these wonderful tutorials and keeping it up for so long, you’re a legend Christoph!
Thanks for your thoughts, Robin. Even though I don’t consider myself a “legend”. Lol. 🙂
In my professional life I work a lot with containers and Kubernetes. And I have already felt the urge to put up a mail server on Kubernetes despite all the hell that consists of YAML, Helm, CI etc. After all if done right you could just scale your mail server to the skies and back. But in reality that stuff keeps breaking for me.
Looking at the Dockerfile that create the images found e.g. on hub.docker.com I’m scared of their quality. Last week it took me two days to debug a stupid timezone issue in the “mariadb” image. The next two days went down the drain for a weird design decision in the “ruby” image. The Dockerfiles are often just very very weird and remind me of my first BASIC projects on my trusted old C=64.
And as you mentioned keeping the software up to date (e.g. security issues) is a problem, too.
Be aware of this warning from the rspamd authors of using older versions: https://twitter.com/rspamd/status/1229088580078837760
He’s such a moron. I wonder why he keeps broadcasting that. I feel bad for endorsing the work of such a person. If it’s not too childish maybe we should use some other software for the next ISPmail guide.
LOL, he blocked me on Twitter.
You called him a moron.
What are your thoughts on using the buster-backports packages from Debian? In your example of Roundcube: buster-backports offers Roundcube 1.4.6 (at the moment)
It tackles the problem of updates and dependencies. The only downside I see is that this might introduce dependencies to other packages in buster-backports, but my experience is that Debian tries to limit this effect by using sensible version dependencies (so packages in stable are compatible)
PS: Thanks for your excellent work!
I’m sorry. I see you already answered this in the comments on “Install the software packages “:
https://workaround.org/ispmail/buster/install-the-software-packages/#comment-97631
Hi Christoph,
calling a software developer a moron and using his software at the same time is a good example of an oxymoron and simply paradoxical as well as unbelievable. Maybe you just delete the insult from the thread…. doesn’t suit you and this great tutorial/workshop, to be honest.
Anyway, I definitely use the stable version from the rspamd site – an entry in the apt source list also ensures that I install relevant updates promptly.
Neither witchcraft nor does it destabilize the basic Buster installation.
jm2c
Best from Cologne
Wolfram
I admit that “moron” may not sound very scientific and overly simplified. Still even as a developer of a great piece of software like rspamd you are dealing with people who use your software. Insulting and even threatening those people is inadequate in my opinion. Such behavior has in the past lead to famous forks of large software projects. Other projects introduced Code-of-Conducts to get back to a healthy cooperation. And even large companies have triggered shitstorms in today’s internet and face significant losses due to their way they do PR and deal with their users/customers. I am aware that taking such a harsh position here makes me appear unprofessional. However I would give myself away if I focused completely on the software and not warn of the dangerous attitude of the people behind it. In today’s internet society I consider “trust” the most important factor in using any service, software or product. And I find it hard to use a piece of software in my infrastructure that comes from a person showing sociopatic behavior. Not just once because he had a bad day. But time and again.
Hi Cristoph!
Very nice setup here!
I have to say, I did NOT implement my setup based on your guide. I came here from 123qwe.com, because I am researching how can I turn my docker-compose based setup to a “native” one.
You are both great help, so thanks for your guide!
But let me kindly say something at least partly subjective.
I would adwise anybody to ditch Debian in favor of OpenSuse (or Ubuntu, if you must):
– you expect – and encourage – your readers to learn something new: as OpenSuse’s userbase is smaller, it is likely they will also learn a new distro. And most of them will seldomly look back, if they don’t have stockholm syndrome 😛
– But aside from the joke: “Debian arguably has the best software package management of all operating systems”. Now, it is easy to argue:
* so the packages are old, compared to other, but also stable distros. Ubuntu, OpenSuse (at least the Leap), etc. are far from bleeding edge. That already makes it impossible to be the best, and that is hard to argue. In fact, the old packages mean you will have to wait 1-2-3 years for a completely new software to reach the Debian repos, as new software projects will very well use the latest stable-ish frameworks (good decision IMO) and it makes them incompatible with Debian many times.
* So maybe the package variety is the best? Could be, but not significantly better in my experience. I like to try new things, and in the last 5 years I had maybe one or two occasions that I could not install something that I could on Debian. Mostly it was something ancient (by software standards) or proprietary. But I installed software at least a dozen times that I could not, was I to use Debian. Packages are plentiful not just in Debian. Even Microsoft provides rpm based repos for Teams and VS Code…
* Than the package management tools? Sorry, but I think it is subpar on multiple levels. For first argument, the dependency solving is decades behind the best. Fedora’s dnf and OpenSuse’s zypper use SAT solver for dependency solving, since about 10 years. Also hard to argue.
* package management tools II: The `apt-*` CLI is unwieldy. Small example: sequential `apt-get update && apt-get upgrade` call everywhere. Why? Because defaults are not thought out. On the other hand `zypper update` will refresh the repo data automatically (you can instruct it otherwise if you just did that a moment ago, likely in a script), AND than it will upDATE the packages. You can also call `zypper refresh` for only the former. `zypper up` and `zypper ref` are shorthands for them. And how is `apt-cache` even needed as a separate command? Yes, as it’s name suggests, it operates on the local repodata cache, but it is also true for `apt-get update`, so what IS the logic?
* package management III: So you should use aptitude or whatever for a better CLI? Then – recurring theme – use the better BY DEFAULT.
* the OpenSuse Build service is a treasure chest of user repos. I have a few of them for niche and/or bleeding-edge software, and it works well with the system repos!
– btrfs is a great filesystem if you own the hardware of your server, unless you plan to RAID 5/6. I use it on my PC since about 7 years, and no hiccups for me. It has a better rsync integrated. I think it has the best support on OpenSuse, and openSuse’s MicroOs transactional-update (which is built on top of btrfs snapshots) could be also a thing to potentially try on a self-owned hardware!
I managed to follow all Debian/Ubuntu server setup guides successfully in the past. Yes you will have to adjust some pathes, some users, and more package names if you use OpenSuse, but most of the time you’ll get fresher-but-not-less-stable version through the package manager, compared to Debian.
The worst are the numerous guides with docker(-compose), recommending/requireing exact Ubuntu LTS versions or Debian, than curling/wgeting the docker, because the “packages are old”. I thought docker should make deployment host distro agnostic.
Ansible (and also salt-stack I think) also have some wrapper command that is compatible with all relevant package managers (even though you’ll have to work around different package names), but other distros are sidestepped altogether in most of the tutorials.
But let me circle back to my opening words: I am grateful for you effort of putting this together!
Even though I consider the host distro techn(olog)ically subpar, the concentrated knowledge in this guide could be (in fact, it is) sold!
Krisztián, thanks for your elaborate comment. Much appreciated.
IMHO whether you use an RPM- or APT-based distro is a personal choice. Maybe you started with one of them and got used to it. Maybe you tried both and have developed a preference. I’m very biased towards APT because I understand it and Debian follows a very concise logic. My experience with RPM comes back from the dark ages of “rpmfind.net”. 8 years ago I was again forced to work with CentOS and having to use EPEL for the most basic of tools hasn’t improved my opinion.
“Debian packages are old” is a misconception by the way – albeit a common one. Debian packages are infact very up-to-date with only few exceptions. But to make use of them you need to change your configuration to “unstable” and you will get fresh software every day. The reason for “stable” distributions is to avoid breaking changes. You don’t want to wake up in the morning realizing that 500 server are down because a new version of some software was installed automatically that requires a configuration change. So “stable” means: the version does not change but it will get security updates. It doesn’t necessarily mean that your systems run more “stable” or “reliable”. In turn “unstable” does not mean “less reliable” – it means that software versions will change unexpectedly and require more attention.
Nobody keeps you from running “unstable” on your workstation. But you probably don’t want that adventure on your server farm.
Problems might arise when you want to mix “stable” with certain newer packages. In that case you can either use the backports repository or package the specific version yourself. There can’t be matching packages for all potential use cases and combinations of other packages. For my day job for example I run specific versions of MariaDB/MySQL on the production servers because our developers would kill me if a newer version would suddenly break any of their ten million SQL queries that rely on a certain MySQL version.
The worst part about Debian is that it’s unsexy. The bug tracker comes from the stone age. The web site makes me want to cry. Creating Debian packages has been improved a lot but I still don’t quite enjoy to build packages. But in terms of ease-of-use (of an installed system) and stability nothing beats it for me on a server. On my workstation I run either xubuntu or Mint which works perfectly.
So it’s good to hear that OpenSuse is worth a try. I always disliked RPM and the dependency hell – even with “yum”. Having EPEL seperately is weird. And SuSE was the worst of the worst for me. Then again most of my experience comes from using “yast” and that’s been twenty years ago. 🙂
In the end you will be able to run a mail server with any kind of Linux distribution. You are right… figure out the paths and you are good to go.
Not everyone wants to build a server from the ground up — especially if it is a hurry-up situation.
Since many of us are using Virtual Private Servers, a singular choice should be considered in that context. If you want to insure that your setup will be relatively portable between hosts, you have three universal choices (in alphabetical order): Centos, Debian and Ubuntu. There are certainly hosts that offer a wider array of choices but nobody wants to be locked in to a particular vendor because they made a confining choice early on.
Hi!
There are more fresh packages at https://repo.dovecot.org – you could maybe mention this as possible alternative to debian packages?
Thanks. I still believe that a stable version is to be preferred to avoid surprises.
Hi all,
regarding stable vs shiny, esp. for Roundcube I do the following:
I have a new Roundcube on a separate server that communicates with the Email server build upon your tutorial with little changes via IMAPS and SMTP (submission). You also can have the “change password” feature from Roundcube if you have an SSH tunnel to the email server to do the DB connection. One also may be able to have an encrypted sql connection directly, I didn’t check that.
This way I have a stable email server plus an up-to-date webmailer. Also has security and performance advantages.
best regards