Preparations for Debian Bookworm – a tale of getting nowhere slowly

Debian 12 – codename “bookworm” – was released in June 2023. And you surely expect a new version of the ISPmail guide. Do not despair – I am already working on it. Check out this section frequently for further updates. And if you want to know why I am so behind…

Starting early

In fact I started preparing the new guide in January 2023. But why is it not ready then? Well, the past months have surely been the worst period of procrastination I ever had. And I don’t mean that kind of procrastination where you just play video games, watch Youtube and work on some “maker” projects. Okay, okay, I also did that. But at the same time I thought I should make the new guide a bit more fun to write. Nobody likes to work on the same boring topic for twenty years without changing stuff. So what could I do to spice it up?

Let’s have some fun

Oh, I know. The feature we always lacked was a cool web interface to manage the mail server. No, wait, that’s not fair to say. Several people have published their work and I appreciate their work. But I’m also a control freak. I would have designed the web interface differently. And somehow we needed more features specific to the ISPmail guide. Like managing DKIM keys or parsing log files. Somehow I was dreaming of a cool web interface with a nice dashboard, a self-explaining management of domains, aliases and all that stuff. So I created a new Git repository and started collecting ideas. Where do I put the Git repository? GitHub? Nah, too mainstream. I usually host stuff on my old laptop-based “server” at home. Hmm, what happened to my old Git server I used to run there? It’s gone? Well, not too bad. What is a lightweight software to run a Git server. Searching… Searching… Oh, Gogs looks cool. Let’s try that.

Where is my Git server?

Wait, the installation is just one binary file? An entire web application that provides the basic features of GitHub in just one binary? What is this magic? Wouldn’t it be cool if my new web interface for ISPmail would also be just one binary? I usually develop web applications in Ruby on Rails. It’s a lot of fun but distributing a web application is either a complex process or I bundle it in a (quite large) Docker container. Gogs is open source software so I checked it out and found that it’s written in Go(lang). That’s bad. I don’t know anything about Go, although I always wanted to take a look at it. Maybe this is the time. You can in fact bundle up an entire directory and embed it into a binary. So I started learning Go.

How cool is Golang

A couple of tutorials later I felt ready to start developing. After all, there were several web applications written in Go and there was a lot of praise about the extensive standard library and the many useful libraries. An hour later a “hello world” showed in my web browser and I was happy. The syntax felt a little weird and I was pretty sure that I would not like to develop something complex like this. So I started searching for web frameworks. Reddit told me that I am stupid for looking for a web framework. As a Go developer you apparently don’t use web frameworks. Go offers so many features that you can just feel relaxed and hack along. I wrinkled my nose because as a Rails developer I know about all the complexity of templates, sessions, cookies, routing, controllers, debugging, caching, database migrations and testing. I was convinced that I needed a framework and started searching. There was Gin Gonic and Revel and a couple of others that have been discontinued. Documentation was lacking so I started to look at other projects that used these frameworks. I tried to replicate the features that I’m used to from Rails. There was nothing like that. Go’s standard library was incomplete compared to Python and Ruby. And then the syntax. Go’s syntax gave me sleepless nights. And “interfaces”. These damn interfaces. I’m so used to object-oriented programming. But Go’s inventors seem to hate OOP. Developing in Go felt like solving Sudoku puzzles blindfolded in a tornado. Even after spending two weeks with Go I failed to understand other people’s code. I really wanted to love Go but I couldn’t.

Or rather Javascript?

Okay, maybe trying Go was a stupid idea. I have lost so much time already. Let’s do the most mainstream thing possible to develop a web application. Let’s use JavaScript for the frontend. And maybe just a tiny REST API in the background with Flask or Sinatra. Or perhaps just NodeJS because all the cool guys use that. That will do it. The problem is: I hate JavaScript. But Reddit tells me that JavaScript has improved so much over the past years that it’s finally worth using it. And there is Typescript that makes it even better. So after 10 years of ignoring JavaScript I again learned about ECMAScript, JavaScript, the various ES versions… and I still did not like it. Sure, JavaScript has evolved over time. But if you were to re-invent JavaScript from scratch you would not end up with this mess. But wait. Developers nowadays don’t use bare JavaScript but Typescript.

Typescript it is

So let’s learn Typescript. One day later I figured out that Typescript just makes everything more verbose while in turn helping you to avoid common mistakes. But its main promise was to introduce static types. It somehow does that. During build time. But later on you just have your same mess of JavaScript that is not strictly typed. I was reminded of learning CoffeeScript back then to make JavaScript less painful. My frustration grew. I stared at my monitor. My IDE showed thousands of files in the node_modules directory for no apparent reason. I had set up a tangled mess of semi-automatic build processes to compile my basic Svelte frontend. It was neither simple nor fun.

Brain holidays

So I went back to Ruby on Rails. And after weeks of trying out Go and JavaScript and its toolchains, this felt like a much needed vacation for my brain. Someone once said that a good developer can program in any language. If that’s true then I suck as a developer. But my perfectionism and desire to learn the right tool for the job had taken me nowhere. So, Rails was the way to go. Be it so. I slapped together a process to turn my Rails project into a Docker container so that others can deploy it easily. The container measured 500 MB which is far away from the 50 MB of Go applications. But I finally needed some results. So let’s create something visible and usable and add perfectionism and features later on. After a few days of hacking at least I had something working…

I had also come up with a couple of features that really help you manage your server:

Those features are not implemented yet. But I found the idea cool to verify MX and SPF records, manage DKIM keys and watch your mail server logs.

Wasting time with WordPress malware

However the web interface was just a pet project. Debian “bookworm” had been released in the meantime and it was only a matter of days before I receive the first fan mail asking when the new ISPmail guide will be released. So lets put it aside and start working on the actual guide. This web site is running WordPress and so I started preparing a new category and setting up the chapters… when I realized that somehow the WordPress instance felt sluggish. My guts told me to check the installation. And despite my efforts to harden the server, run an application firewall in front, install updates daily and kick out abandoned WordPress plugins… I found several pieces of malware trying to hide in my DocumentRoot. That’s what you get for choosing WordPress, right? Everybody knows that it’s just a matter of time until you get “hacked”. Maybe WordPress was a bad choice after all. It’s not good at serving structured documentation like a multi-chapter guide/book. Several times I was asked why I can’t simply put the documentation in a Git repository and share it with others. And by today I did not figure out how to serve all the chapter as one complete file. So maybe it’s time to question WordPress and move on.

Static site generators

Alternatives? Typo 3… no thanks. Drupal… the same mess it has always been. Maybe something more documentation-centric like ReadTheDocs or Sphinx. Well, they use reStructuredText instead of Markdown. Who nowadays still uses reStructuredText? What about mkdocs? Let’s just ask our favorite search engine what people recommend in my case. Most of the results were: “How I migrated from WordPress to Hugo”. If there are many articles on the web about people migrating from one software to another, it’s usually a good sign. Or just a hype. With Hugo it was a hype. And I did what I always do: I fell for the hype. I had heard of static site generators before. They compile your documentation into plain HTML (and CSS) so you just upload your static files to a web server. And nobody will hack your CMS because there is no scripting involved – just static files. Hugo was apparently an attempt to rewrite Jekyll – a well known static site generator written in Ruby. And to make it fast they used Go(lang). And it was fast. It even had a huge library of themes to chose from. For documentation there were over twenty different themes. Most of them were so ugly that it hurt my eyes.

Docsy

But one of them stood out: Docsy. A beautiful theme written by Google. I could finally start writing the new ISPmail guide in Markdown. There was a feature to merge all chapters into one page – something that was asked for multiple times. It looked much nicer and more professional than my WordPress theme. It could be managed through Git because it was just text files. It just felt good writing documentation again. Hugo’s documentation was extensive and after a few days of struggling I understood its concepts.

GDPR

I just needed to avoid loading the font from Google Fonts and instead host them locally. No big deal thanks to the base template. And throw out all these other cloud-based features. Docsy uses Lunr.js – an offline search tool – that’s cool. And apparently it uses Bootstrap – so let’s download Bootstrap and also serve it locally. So far so good.

But how do I structure a large guide into chapters? Hugo seems to sort my chapters randomly. It is meant for documentation so it surely offers something. Yes, it does: weights. To create an order of your chapters you need to edit each Markdown file and add a numeric weight to it. This was awful because I had to open all my files and manually compare those weights to see the order of chapters. I had a flashback of using Commodore 64 BASIC having to renumber my lines time and again. However I really wanted to start writing and took a two-week vacation from work to get as far as possible.

A mess of the highest order

So, why don’t I just name my files 01-preface.md, 02-changes.md, 03-migrating.md and so on? That would apply an order. Actually, no it doesn’t. Hugo orders your pages by weight, then by date (reversed), then by title and last by the filename/path. And it used the actual file name in the URL which added those sorting numbers. And if I had to change the numbers I would screw up the SEO because the URL would change.

I did not give up though. Everything else was so smooth – I didn’t want to give up just because the sort order was wrong. So I took Docsy’s template that creates a table-of-contents of an ISPmail guide and wanted to pimp it. Hugo uses the Go(lang) template language which is quite eccentric –like many other parts of Go. Well, a good developer can develop in any language so let’s try to get used to the template language. Okay, now let’s figure out how Docsy creates the table-of-concents.

After following a dozen links within the theme’s files I finally found the TOC template. And without any exaggeration I can truly say that this is the worst piece of code I have ever seen in the last years. Just take one random line…

{{ $show := cond (or (lt $ulNr $ulShow) $activePath (and (not $shouldDelayActive) (eq $s.Parent $p.Parent)) (and (not $shouldDelayActive) (eq $s.Parent $p)) (not $p.Site.Params.ui.sidebar_menu_compact) (and (not $shouldDelayActive) ($p.IsDescendant $s.Parent))) true false -}}

It looks like LISP and Brainfuck were on a drunken one-night stand and had a segmentation fault. It took me two days to untangle and reverse engineer that uncommented and badly formatted mess of a Hugo template.

No comment

When considering to move from WordPress to Hugo there was another aspect I had completely forgotten: the comments. Having a good ISPmail guide is only part of the game. We need all the questions, corrections, discussions and ideas in the pages. They are way too valuable to lose. And we want to keep discussing the new guide as well. Of course any cloud-based discussion add-on like Disqus were completely out of question. But Commento seemed to be a nice open-source alternative. Unfortunately it took me a day until I realized that they have not updated their open-source code in quite a while. Maybe to encourage people to buy their cloud-based offering instead.

At least someone else came to the same conclusion and formed Commento into Comentario (which was surprisingly hard to find). That software worked very well. You can even import your existing WordPress comments into it and it will show the comments on the right pages. The only thing lacking is a GDPR-compliant spam detector. But otherwise it had actually worked. Kudos to Dmitry Kann for developing it further. It deserves much more fame than it has. So the comments feature had been solved but…

Screw that

I spent my entire vacation trying to improve the Docsy theme, add proper pagination, fix the templates and work around the limitations and opinions of Hugo to force it to work with ordered and structured documents. Many users had the same headaches that I had but the Hugo developers insisted that it works as designed. I was so deeply frustrated that after a sleepless night I decided to stop beating this dead horse. It took me an entire vacation to realize that Hugo is only good for one thing: simple blogs.

Conclusion

So here I am. My misguided curiosity and stubbornness made me waste half a year. Of course I have learned a lot along the way. But I missed my goal by a large margin. I am now back to WordPress for the guide and Rails for the web application. And as a fellow reader quoted: Perfect is the enemy of good. So if you don’t mind… I will have to create something good.

Have you had similar experiences like me? Please comment.

18 thoughts on “Preparations for Debian Bookworm – a tale of getting nowhere slowly”

  1. Hi,

    Well, I really like Hugo and use it on some small documentation projects.

    Have you tried to use page’s front-matter?
    (https://gohugo.io/content-management/urls/#front-matter)

    I use it like this:

    title: 01. Introduction
    slug: introduction

    Set the `slug` in front matter to override the last segment of the path.
    Set the `title`as the content’s title.

    This will sort pages by title, however you can customize the sort options (https://gohugo.io/templates/lists/#order-content)

    Thanks for you wonderful work! Looking forward for the Bookworm version!!!

  2. Hello, I appreciate your guide and your work. However, I would like to request that you keep the guide distinct from your interface and management tool, which I believe should remain optional—like a cherry on the cake. To highlight this point, I quote from your guide: “If the trend continues, only a few people will know how to run a mail server while everyone else is degrading into stupid consumers.”

    On a personal note, I feel that providing a ready-to-use Docker image seems to steer towards the same direction of degrading into “stupid consumers.” I earnestly hope that you’re not aligning your approach with systems like mailinabox or iRedMail. The little configuration required, in my opinion, should be handled with as minimal overhead as possible. It’s merely some database management, after all. A couple of PHP files would suffice.

    Lastly, allow me to extend my thanks, Christoph. I’ve been a follower of your guide for around 10 years, and if my critique sounds harsh, please understand it’s not a reflection of my lack of appreciation for your work. Rather, it’s an expression of my concerns regarding the integrity of your original intention, which has always resonated profoundly with me.

  3. Hi Christoph,

    Good to hear that you are working on bookworm version of the ISPMail. I have ISPMail running for many years and like the guides.
    Since openssl version 1 is expiring soon, do you have an upgrade path openssl version 3? is it working out of the box? How are you solving it for yourself? or are you already running bookworm on your production sites?

    looking forward for your remarks and or anserwers.

    Regards,
    Richard

  4. Dear Christoph,

    I am sure you already know that but your efforts on ISP Mail is very well appreciated. Thanks and regards.

    Nuri.

  5. Chris, I thank you very much for your past and ongoing work on these email server tutorials. I built and am currently using my first tutorial of yours (bullseye) using Devuan, and I must say that I love it. It has been solid with no problems for almost two years now.
    I would like to mention something that might be of interest to you, maybe a possibility of using Publii (https://getpublii.com/) and Comentario? I have not used Publii, yet, but a couple of friends have and it works better for them than does wordpress.
    I really don’t know enough to advise beyond this other than to say that Publii does use Markdown and you “should” be able to use Comentario within the Publii framework. This would enable static html websites with a comment system. I am just sharing information, hope it helps you out.
    I would also like to voice my support for not using only a docker image. I learned a lot with the detailed customization and configuration of the tutorial and I feel that would be lost in a docker image. I really don’t like docker, can you tell?
    Looking forward to your new tutorial, but until then, I have a very nice email system functioning just fine!

    -Craig

  6. Thank you, I’m looking forward to it. You must know that you are helping a lot of people

  7. I can do the needed administrative work without a control panel for my small installation, but what I would like to have (I added it in my installation) in a control panel is a comment field for aliases. I use aliases to track abuse by creating a new alias for almost every online service that wants an email-address. Having a comment field is therefore crucial for me to know were I used an alias.

  8. I am running an extremely old version of ispmail and would like to upgrade to bookworm once it’s available, so I’ve been checking in every couple of weeks. It would be nice if these news items had date stamps so we could have an idea how long it’s been since the post was made.

    1. Christoph Haas

      I am working on the new guide right now. I’ve taken a week of vacation from work to get some progress.

    2. Christoph Haas

      I just realized what you meant. WordPress really does not show the post date by default. Oh, my. Just fixed it. 🙂

  9. I once embarked on an assignment to convert a LAMP application to anything else. It was an unqualified train wreck.

    PHP isn’t glamorous but it works at the appropriate level without all of the gyrations and data interchange associated with web frameworks.

    One can easily spin up a LAMP server on a live machine, a VPS or a VM/container and be most of the way there in terms of the back end necessary for the project. Since Bookworm brings a much more modern version of PHP (8.2), maintenance should be less of an issue going forward.

    I can’t speak for others but adding e-mail addresses and domains isn’t something I do very often. Updating SQL tables certainly doesn’t require some complex RESTful API or a highly reactive multi-user interface.

    PHP certainly isn’t glamorous but neither is maintaining a mail server.

  10. I am running a small mail server on a virtual debian machine. We just updated the debian version to Bookworm, and the mail server seems to be running fine without modifications (knock on wood). Only rspamd was masked after upgrade and needed a re-install to function properly again.
    Christoph, thank you very much for your work, it really helped me to understand and install the necessary components for a modern mail server!

  11. Christoph Haas

    Thanks for the feedback, Boris. I’m on the last 4 pages now and nearly ready for the release of the Bookworm guide. Yes, you are right, most of the configuration can just stay as it is. I needed a couple of changes in Roundcube and the rspamd spam autolearning configuration. Keep us posted if you find further changes that you needed to do.

  12. Thank you for your effort. Especially for explaining how each component works. Regarding console cofiguration, I think it is a waste of time. An update will be required whenever any ISPmail component is changed.

Leave a Comment

Your email address will not be published. Required fields are marked *

Scroll to Top