Get a new server
I strongly recommend that you set up a new server and make sure that the new mail server is working correctly before you start migrating existing email users to it. You may argue that Debian can be upgraded easily using “apt-get dist-upgrade” but that is very dangerous on a live mail server. Automatic configuration changes will have evil side effects and you will likely lose email and cause chaos. And even if you made a backup of the old server it will not be easy to go back to the previous state without losing further email that poured in in the meantime.
So please get a new server and install Debian Jessie on it. Once the users are migrated properly you can tear down the old server so you will not waste that much resources.
Completely follow this guide first
Important: before attempting any migration please first work through this guide. Then at the end start migrating your data.
Read the following pages and set up your Jessie mail server from scratch. Play around with it until you are confident that everything works as expected. Inform your users about the migration and set a time when you intend to move email accounts to the new server. The change will require a DNS change which takes a while to be visible worldwide so your users will have a period where incoming email is delayed. If you proceed properly not a single email will be lost though.
Migrate the database
Now you need to copy the database that contains the control data about your email domains and accounts. On the (old) Wheezy server log in as root and backup the mailserver database. That is as easy as running
mysqldump mailserver > mailserver.sql
Copy that file to the new server (using scp) and import it there:
mysql mailserver < mailserver.sql
Fix the database schema
Do you remember that we are using more secure passwords from now on? So the database field that stores your passwords is not long enough for the new hashes. This command makes the field longer. Run it on your new server:
mysql mailserver -e 'alter table virtual_users modify password varchar(150) NOT NULL'
And as your database contains just MD5 hashes but the new default schema is SHA256 we need to prepend every password with “{PLAIN-MD5}” so that Dovecot knows which hashing algorithm we mean. Run this command on your new server:
mysql mailserver -e "update virtual_users set password=concat('{PLAIN-MD5}',password);"
Migrate the emails
Fortunately Dovecot uses the maildir format that stores emails as plain files on disk. Use rsync to copy over the mails from the old mail server:
rsync -va /var/vmail/ newserver:/var/vmail/
(Note the trailing slashes. Type them exactly as above or you will end up in wrong places.)
There is no need to shut down Dovecot on your live Wheezy server. Copying the files while Dovecot is running will not break anything. While the rsync runs only few files will change. So you keep the downtime low.
Once the rsync is done you need to shut down Dovecot. You told your users about the downtime, right? Okay. Let’s synchronize again. rsync will only copy those files that have changed which makes it much faster than the first sync:
rsync -va --delete /var/vmail/ newserver:/var/vmail/
(The “–delete” option makes sure that files that have been removed from the old server will also be deleted from the new server.)
Switch the DNS records
For all your domains you will have to change the DNS “MX” records to point to your new server. If possible you could lower the TTL (time-to-live) in advance to make the switch work faster.
Enable soft_bounce
Accidents happen. And you don’t want to lose emails. So run this command to enable your safety net on the new server:
postconf soft_bounce=yes
Now Postfix will keep emails in the queue that it would normally bounce back. Start Postfix and Dovecot on the new server again. And watch your /var/log/mail.log and run “mailq” from time to time to see what emails get stuck in the queue. If you are certain that emails can be removed then use “postsuper -d QUEUE-ID”.
Once you have a good feeling that emails are properly received and sent you can switch off the soft_bounce mode again:
postconf soft_bounce=no
Open the gates
Now start Postfix on the new server. It may take a while until everyone in the world respects the new MX record and sends email to the new server.
Shut down the old server
If possible do a final backup of the old server. Then when you have a good feeling about the new server and users seem to be happy you can dismiss the old system after a few days.
Hi!
Will this work from debian 5 to?
Hi Victor. Debian Lenny is no longer supported in terms of security updates. I suggest that you install a newer Debian version. But if you really like to use Lenny then feel free to follow the Lenny tutorial. You can find is as a PDF at https://workaround.org/ispmail
Hi!
The plan is to upgrade to debian 8, there for i wonder if this migration guide will work to migrate from a deb5 installation to deb8?
Yes, you can do that. I recommend a fresh installation of Debian 8 and then copy over the database and the emails as described in this Debian 8 (Jessie) guide. I’m not I understand why you want to install a mail server on a deprecated operating system though. 🙂
UPDATE: Sorry – I totally misread your comment. Yes, updating should not be a problem. The file system structure and database has not changed as far as I remember.
I think he means “will this guide work for upgrading *from* deb5?”, i.e. he currently has a working mailserver running on deb5 already.
Great tutorial, thanks for documenting all this!
Thanks for this.
Under rsync, I think you mean “Note” – not “Not” which suggests you don’t use them.
Best wishes, great job.
Paul, thanks for the hint. Fixed. I’m curious who (besides me) first set up their own mail server following the guide. 🙂
Me for sure. Without your considerable help, it would have been a non-starter. I cannot be more grateful. It’s nice to give a little back when I can.
In fact it was an acquaintance who first pointed me to your tutorials in the Lenny days and he certainly used them too.
If the truth be known, there’s probably hundreds, if not thousands that have good cause to be equally grateful too.
Thanks. I’m flattered. It’s always good to hear success stories. After all it’s a sophisticated hobby project. I’m not making money with it. (Except for a few companies who needed customisations and who I helped with afterwards.)
Seconder. Me too since Etch tutorial, about ten years ago.
Hello Christoph,
A bit of information for migration.
When attempting to expand the varchar field, I couldn’t get your command to work. The following however, worked for me using mysql Ver 14.14. Distrib 5.5.46 and changing to the database;
mysql> use mailserver;
Database changed
mysql> ALTER TABLE `virtual_users` CHANGE COLUMN `password` `password` varchar(150) NOT NULL;
The concat part worked perfectly.
That’s weird. What error message did you get? According to https://dev.mysql.com/doc/refman/5.5/en/alter-table.html that syntax is perfectly valid.
Looking at this again today, I think there’s a couple of issues with this and it’s a chicken and egg situation.
I’m used to running everything from within mysql, but I think you mean to run these commands straight from a shell prompt. BUT, I suspect they will only work IF you’ve invoked your auto mysql login fix described in the “Install the software packages” section. I looked back to the previous tutorial and as far as I’m aware, this is the first time you’ve used this.
So reading your tutorial and working my way through, the first thing I come across is the migration page, which I try to comply with. As a previous user of your tutorials and that point in time, I haven’t yet got to the part about the auto mysql login, as I’ve no reason to suspect I wouldn’t understand the rest of the tutorial and in any case, I’m migrating an existing server.
So (my fault) the first thing I do is log in to mysql and run the commands. They fail with:
ERROR 1064 (42000): You have an error in your SQL syntax; check the manual that corresponds to your MySQL server version for the right syntax to use near ‘mysql mailserver -e ‘alter table virtual_users modify password varchar(150) NOT NUL’ at line 1
So I looked for an alternative and came up with the command I posted yesterday.
Not surprisingly, now I’ve come to the conclusion they’re meant to be run from a shell prompt.
So by the time I’d had a think and got to line 2 – the concat, I decided starting off with the “update” part of the command (within mysql) might work and it did.
Setting up a test database today on a different machine, I tried to run the command straight from the shell prompt (as I believe you meant) and got this error:
mysql mailserver -e ‘alter table virtual_users modify password varchar(150) NOT NULL’
ERROR 1045 (28000): Access denied for user ‘root’@’localhost’ (using password: NO)
which suggests to me, the auto login MUST be invoked before this command can be run.
For those that don’t wish to invoke this, the command I posted (used from within mysql) works as expected.
So my conclusion is, it is a chicken and egg situation in as much as if you follow the order of the pages, a previous user would not yet be aware of the auto login.
I hope that helps.
Sorry to hear you had so much trouble. But the order of the tutorial really is important. So I wrote “Follow the tutorial” here and meant it. After that you have an empty database and a ~/.my.cnf file that allows the auto-login. The shell command to migrate the database will then run right through.
I know that there’s a lot of forks in path. But if you really follow the order of pages and instructions I’m sure it will work. 🙂
I’m sure you’re correct.
But the point I was making, as far as I was concerned, I WAS following the order of the tutorial. The Migration page (2nd page) comes after the “what’s new” so the way I read it, that was where one started if you’re migrating a server – sort out the database ready for the rest of the tutorial. The auto login to mysql doesn’t appear before the “installing software”, the 6th page.
To be truthful, it wasn’t much of a problem so please don’t apologize. It just needed a bit of thought to work through. It’s certainly not meant to be any criticism of you or your work. It’s just something that temporarily tripped me up and I thought you may wish to know about it.
What would you suggest to make it clearer that the mail server has to be set up completely before doing the migration? I could for example move the page to the end of the tutorial.
Yes, that’s one option, or just put a note on the page that the installation should be completed before migration is attempted. Whichever you prefer.
Dear Christoph,
I have to admit it, I didn’t follow your advice to get a new server but I have dist-upgraded mine.
However, everything ran fine, except for a mysterious issue.
I don’t know why – maybe at the time I was trying to enforce my mailserver – but I have enabled this in my former main.cf:
smtpd_relay_restrictions = permit_sasl_authenticated, reject_invalid_hostname, reject_non_fqdn_sender, reject_non_fqdn_recipient, reject_unknown_sender_domain, reject_unknown_recipient_domain, reject_unauth_destination, reject_unauth_pipelining, reject_unknown_reverse_client_hostname, reject_rbl_client bl.spamcop.net, reject_rbl_client zen.spamhaus.org, reject_rbl_client dnsbl-1.uceprotect.net.
which are a series of denying rules on smtpd_relay. I do not remember if you ever discussed smtpd_relay_restrictions, but I remember that this worked just fine before the upgrade,
However, in Jessie this line prevents me from sending e-mail outside (local delivering – of course – works). Amavis was complaining like that:
Blocked MTA-BLOCKED {RejectedOpenRelay}, [5.170.9.208]:22070 [5.170.9.208]
Do you have any idea on how to explain that? Something similar is described in
https://bananasfk.wordpress.com/2015/06/04/wheezy-to-jessie-debian-upgrades/
in the postfix paragraph, but that’s a little bit confused. It says that you have to add -o mynetworks=127.0.0.0/8 to the 10025 port in master.cf, but hey, that directive was already there.
Another minor issue is with opendkim, which I see you are about to cover. It works fine, but it complains that key data is not secure. I have checked permissions and they are all OK, so I have to add it this to /etc/opendkim.conf:
RequireSafeKeys False
and this seems to be a matter related to size key, rather than permissions:
https://bananasfk.wordpress.com/2015/01/16/opendkim-fail/
(again the same British guy)
However, it will be nice to fix this, too.
Thank you for your great guide, first time ever I am considering a donation 🙂
Luigi
For safety reasons I would recommend to change `rsync -va –delete /var/vmail/ newserver:/var/vmail/` to `rsync -va –delete /var/vmail// newserver:/var/vmail/`.
If you’re migrating mailboxes from two old servers to a single new server that hosts both domains this will prevent you from accidentally deleting the already migrated mailboxes from the other server. Or any other files you may have created in the new server’s vmail directory (.forward, .spamassasin, …).
I meant to write: “rsync -va –delete /var/vmail/{vdomain}/ newserver:/var/vmail/{vdomain}/”
(Hint: Comment preview would be nice. Being able to write < > would be nice 😉
Dear Christoph,
I followed your guide to upgrade to the new server. Everything is great, and the new system is working. However all the old emails are missing.
I have rsync’ed the files across in to my vmail directory, and the files are there, but they don’t show up on roundcube or direct IMAP connection. I have a bunch of empty accounts (although new emails are received ok).
Any idea what may have gone wrong?Or what I need to do to migrate the emails across properly?
Hi Christoph,
Thanks soooo much for this guide! Currently I am migrating a mail server debian 5 lenny which I built 5 years ago with this guide (yeah believe it) to a newly setup debian jessie. I have the new server all up and running, and now I want to transfer the emails. However I noticed that there are some dovecote files within the file structure of the emails which are different on the old server compared to the new file structure. Some binary log files and some other dovecot config files seem to be inside of each directory. Are these files needed on the new server? Can I omit them? Further the old server has a different fqdn as the new one, but all mails seem to be appended with the hostname. Will the transfer of the old mails to the new infrastructure have any problem with that?
Thanks in advance for your insights! You rock!
Thank you for this great tutorial. Much appreciated.