Bug#1037346: cyrus-imapd: all emails disappeared after upgrading from bullseye to bookworm (similar to #1007965)
Hi, I had the same problem (mails were not displayed in the mail client any more after upgrading from Bullseye to Bookworm). What worked in the end for me was this: - restored /var/lib/cyrus/ and /var/spool/cyrus/ from backup, to the old state from before the Debian upgrade. - built Cyrus 3.4.6, and installed and started it temporarily. The mails were now visible in the mail client again. - upgraded to the normal Cyrus version from Bookworm. The mails were still visible in the mail client. And since I had already kept the server running for some hours with the broken mail archive, I then also restored the getmail6 status files (/var/lib/getmail) to the state from before the Debian upgrade. This caused getmail to re-fetch the mails that were missing. I don't know how to detect whether the mail archive was really migrated successfully; but since Cyrus 3.6.1 is now running and Thunderbird can connect and sees all mails, I suppose the migration was successful? Regarding the build of Cyrus 3.4.6, in the end I built it using the existing Debian packaging data, and using the "debocker" tool to build inside a clean Docker container. Unfortunately debocker will always run the "lintian" tool to check the package, which failed for me with error "E: cyrus-common: depends-on-obsolete-package Depends: lsb-base". I didn't know how to fix this error or how to correctly disable the lintian step; so I edited the debocker files to disable this check. Very ugly, but it worked. So from my notes, these must have been the steps that I did for building the Cyrus 3.4.6 Debian package: - installed "debocker" and "devscripts" packages: `sudo apt install debocker devscripts` - downloaded the Debian cyrus-imapd packaging info: `debcheckout cyrus-imapd` - downloaded the original source package: `wget https://github.com/cyrusimap/cyrus-imapd/releases/download/cyrus-imapd-3.4.6/cyrus-imapd-3.4.6.tar.gz` - renamed the source package so it would be found in the next steps: `mv cyrus-imapd-3.4.6.tar.gz cyrus-imapd_3.4.6.orig.tar.gz` - `cd cyrus-imapd/` - started new Git branch at the state of Cyrus 3.4.3-4: `git checkout -b cyrus-3.4.6 debian/3.4.3-4` - added changelog entry: `dch -v '3.4.6-1.1' "use new upstream version 3.4.6"` - the warning about missing DEBEMAIL can be ignored - committed change: `git commit debian/changelog -m 'update changelog'` - modified the "debocker" template files to disable the lintian step: `sudo nano /usr/share/debocker/bundle-files/steps/05-build` and then commented out the line for "lintian --pedantic --display-info *.changes" - created build bundle: `debocker bundle --image debian:bookworm -f "-b -us -uc"` - did the actual build: `sudo debocker build-bundle ../cyrus-imapd_3.4.6-1.1_bundle.tar` - whether the "sudo" is necessary depends on your local Docker setup - the build took a while (half an hour or more?); and then there was a message like "LOG Build successful", and there were lots of Cyrus packages in the current directory. I then copied cyrus-clients_3.4.6-1.1_amd64.deb, cyrus-common_3.4.6-1.1_amd64.deb and cyrus-imapd_3.4.6-1.1_amd64.deb to the server and installed them. Remember to undo the change to the debocker file (/usr/share/debocker/bundle-files/steps/05-build) after the build. Kind regards, Oliver
Bug#1037346: cyrus-imapd: all emails disappeared after upgrading from bullseye to bookworm (similar to #1007965)
Op 06-02-2024 om 11:34 schreef Jens Georg: Package: cyrus-common Version: 3.2.6-2+deb11u2 Followup-For: Bug #1037346 Some additional information: - You can roll-back to package 3.2 after the upgrade to 3.6, the mailboxes will re-appear again - You might encounter some junk mailboxes from the failed conversion attempt, the remedy is to convert mailboxes.db to text, remove the offending lines, and convert it back to its original format. - The proper may to migrate to 3.6 without hassle is: - Stop cyrus-imapd service - Download and compile 3.4, run ./imap/ctl_cyrusdb -r once (make a copy of /var/lib/cyrus/mailboxes.db beforehand, of course) - Then proceed with the update It seems that the debian package, despite claiming that it has the necessary patches to make the upgrade possible, doesn't. I still have to do the upgrade on a few machines.. Did you do this on Debian 11 or Debian 12? What did you download exactly? Maybe this? https://github.com/cyrusimap/cyrus-imapd/tree/cyrus-imapd-3.4.6 After downloading and go to the right directory something like: systemctl stop cyrus-imap ./configure make sudo make install Did you use any Debian patches? sudo cp -a /var/lib/cyrus/mailboxes.db /path/to/backup/ ./imap/ctl_cyrusdb -r How did you remove this version? Then install the new Debian packages? Bye, Paul -- Paul van der Vlis Linux systeembeheer Groningen https://vandervlis.nl/
Bug#1037346: cyrus-imapd: all emails disappeared after upgrading from bullseye to bookworm (similar to #1007965)
Package: cyrus-common Version: 3.2.6-2+deb11u2 Followup-For: Bug #1037346 Some additional information: - You can roll-back to package 3.2 after the upgrade to 3.6, the mailboxes will re-appear again - You might encounter some junk mailboxes from the failed conversion attempt, the remedy is to convert mailboxes.db to text, remove the offending lines, and convert it back to its original format. - The proper may to migrate to 3.6 without hassle is: - Stop cyrus-imapd service - Download and compile 3.4, run ./imap/ctl_cyrusdb -r once (make a copy of /var/lib/cyrus/mailboxes.db beforehand, of course) - Then proceed with the update It seems that the debian package, despite claiming that it has the necessary patches to make the upgrade possible, doesn't. -- System Information: Debian Release: 12.4 APT prefers stable-updates APT policy: (500, 'stable-updates'), (500, 'stable-security'), (500, 'oldstable-updates'), (500, 'oldstable-security'), (500, 'stable'), (500, 'oldstable') Architecture: amd64 (x86_64) Kernel: Linux 6.1.0-16-amd64 (SMP w/2 CPU threads; PREEMPT) Kernel taint flags: TAINT_WARN Locale: LANG=C, LC_CTYPE=C.UTF-8 (charmap=UTF-8), LANGUAGE not set Shell: /bin/sh linked to /usr/bin/dash Init: systemd (via /run/systemd/system) LSM: AppArmor: enabled Versions of packages cyrus-common depends on: ii adduser 3.134 ii db-upgrade-util 5.3.1+nmu1 ii db-util 5.3.2 ii debconf [debconf-2.0] 1.5.82 ii e2fsprogs 1.47.0-2 ii gawk1:5.2.1-2 ii init-system-helpers 1.65.2 ii libc6 2.36-9+deb12u4 ii libclamav9 0.103.10+dfsg-0+deb11u1 ii libcom-err2 1.47.0-2 ii libgcc-s1 12.2.0-14 ii libical33.0.16-1+b1 ii libicu6767.1-7 ii libjansson4 2.14-2 ii libkrb5-3 1.20.1-2+deb12u1 ii libldap-2.4-2 2.4.57+dfsg-3+deb11u1 ii libpcre32:8.39-15 ii libpq5 15.5-0+deb12u1 ii libsasl2-2 2.1.28+dfsg-10 ii libsasl2-modules2.1.28+dfsg-10 ii libsnmp40 5.9.3+dfsg-2 ii libsqlite3-03.40.1-2 ii libssl1.1 1.1.1w-0+deb11u1 ii libstdc++6 12.2.0-14 ii libwrap07.6.q-32 ii libxapian30 1.4.22-1 ii libxml2 2.9.14+dfsg-1.3~deb12u1 ii libzephyr4 3.1.2-1.1+b1 ii lsb-base11.6 ii netbase 6.4 ii perl5.36.0-7+deb12u1 ii postfix [mail-transport-agent] 3.7.6-0+deb12u2 ii sysvinit-utils [lsb-base] 3.06-4 ii zlib1g 1:1.2.13.dfsg-1 Versions of packages cyrus-common recommends: hi cyrus-admin 3.2.6-2+deb11u2 hi cyrus-imapd 3.2.6-2+deb11u2 hi cyrus-pop3d 3.2.6-2+deb11u2 Versions of packages cyrus-common suggests: pn apt-listchanges hi cyrus-admin3.2.6-2+deb11u2 pn cyrus-caldav pn cyrus-clients hi cyrus-doc 3.2.6-2+deb11u2 hi cyrus-imapd3.2.6-2+deb11u2 pn cyrus-murder pn cyrus-nntpd hi cyrus-pop3d3.2.6-2+deb11u2 pn cyrus-replication ii sasl2-bin 2.1.28+dfsg-10 -- Configuration Files: /etc/cyrus.conf changed [not included] /etc/imapd.conf changed [not included] /etc/pam.d/sieve changed [not included] -- debconf information excluded
Bug#1037346: cyrus-imapd: all emails disappeared after upgrading from bullseye to bookworm (similar to #1007965)
Kai, Thanks so much for your work here, I've been panicking about my email. This script is a HUGE help. I noticed one thing though, in the linking, it doesn't really handle nested folders, so I get a ton of: ln: /var/spool/cyrus/mail/p/user/phil/lists/linuxdriver: hard link not allowed for directory And one small thing, it might be useful for others if the 'echo's before the first bash gave the command to run to feed the file to cyradm, so you don't have to go back to the script to find it. Thanks! -- Phil Dibowitz p...@ipom.com Open Source software and tech docsInsanity Palace of Metallica http://www.phildev.net/ http://www.ipom.com/ "Be who you are and say what you feel, because those who mind don't matter and those who matter don't mind." - Dr. Seuss
Bug#1037346: cyrus-imapd: all emails disappeared after upgrading from bullseye to bookworm (similar to #1007965)
Package: cyrus-imapd Version: 3.6.1-4 Severity: important Dear Maintainer, today I upgraded our famliy+friends email server system from bullseye to bookworm including the cyrus-imapd upgrade to v3.6.1-4. Unfortunately, all mails of all users disappeared and new inboxes were autocreated when users logged in: --syslog- Jun 11 12:27:15 xxx cyrus/imaps[1730301]: autocreateinbox: User x, INBOX was successfully created - The emails were still in the non-uuid directory of the cyrus spool dir. I did not figure out how this happened, but I solved it by 1. extracting a list of all sub-mailboxes of all users from the filesystem structure, 2. re-creating the same sub-mailboxes with the cm command of cyradm, 3. and hard-linking all directory content to the new uuid dirs (with the help of mbpath) The issue is now solved for me, but I can provide additional system information and logs if needed. (same report goes to https://github.com/cyrusimap/cyrus-imapd/issues/4035) -Kai -- System Information: Debian Release: 12.0 APT prefers stable APT policy: (500, 'stable') Architecture: amd64 (x86_64) Kernel: Linux 6.1.0-9-amd64 (SMP w/4 CPU threads; PREEMPT) Locale: LANG=en_US.UTF-8, LC_CTYPE=en_US.UTF-8 (charmap=UTF-8), LANGUAGE not set Shell: /bin/sh linked to /usr/bin/dash Init: systemd (via /run/systemd/system) LSM: AppArmor: enabled Versions of packages cyrus-imapd depends on: ii cyrus-common 3.6.1-4 ii libc6 2.36-9 ii libcom-err2 1.47.0-2 ii libsasl2-22.1.28+dfsg-10 ii libssl3 3.0.9-1 ii libwrap0 7.6.q-32 ii zlib1g1:1.2.13.dfsg-1 Versions of packages cyrus-imapd recommends: ii rsync 3.2.7-1 cyrus-imapd suggests no packages. -- no debconf information
