Re: Why do we need to run reconstruct after having moved mailboxes using cyradm's xfer command?
Dear ellie,
thanks again for answering my questions!
On 17.12.2018 23:38, ellie timoney wrote:
> Hi Binarus,
>
>> Could anybody please shortly explain why? What exactly are the
>> techniques and mechanisms cyradm's xfer uses to do its thing? I had been
>> quite sure that it just uses the IMAP protocol, but there seems to be
>> more to it ...
>
> You're right, but missing a detail. The XFER command is an IMAP extension,
> for use in cyrus clusters ("murders"), where mailboxes need to be identical
> between servers. I don't know much about the details, but it makes sense to
> me that it would copy the mailboxes across at a "deeper" level, to make sure
> internal identifiers and such match.
OK, got it. I am glad that I at least didn't misunderstand cyradm
completely. I originally thought that xfer was just a command which
cyradm internally would translate into standard IMAP commands, which is
wrong as I have understood now.
> Something like imapsync would probably be using the CREATE and
> APPEND commands instead, which let the destination server choose
> its own internal identifiers, and which will probably be different from the
> source ones. But they would be stored in the exact method the destination
> server prefers (such as using the newest version of the mailbox index
> format...)
Then it actually was the right idea to try cyradm xfer instead of
imapsync. After all, we want our mailboxes, folders and messages
transferred to the new server without anything being altered as far as
possible, don't we? Now that I can use scripts with cyradm (thanks to
your answer to one of my other posts), this will be a pleasure next time.
>> Now that I have used cyradm's xfer command to relocate the mailboxes /
>> messages / folders / subfolders, I surprisingly had to run the "heaviest"
>> form of reconstruct (-V max).
>
> That's not the "heaviest" form of reconstruct, it's just updating the mailbox
> indexes to the "max"imum (-V)ersion supported by the server. This ends up
> adding a bunch of extra information to the indexes that the newer server
> version can make use of. Generally if the index version is too old for what
> it's trying to do, it'll fall back to calculating missing values the long
> way, or provide less info, or just refuse the requested operation. The
> assertion failure you saw is clearly a bug, but this smells familiar, so
> maybe it's already been fixed. The latest version of the 2.5 release is
> 2.5.12, and is nearly 2 years newer than 2.5.10, for what it's worth.
OK, understood. In my case, it decided to refuse the operation ...
somehow. Obviously, it didn't tell the client that it refused, but
probably believed that the operation would be successful, sent an OK to
the client, began the operation (the message indeed disappeared from
Thunderbird's message list), noticed that it couldn't perform it and
then reverted it (the message re-appeared in Thunderbird's message list
about one second after it had disappeared).
This is no real issue to me even if the log entries are due to a bug, as
long as we can solve that problem by running the reconstruct. I just
wanted to know why we have to reconstruct; thanks again for the insight!
Regarding the version: In general, I don't have any problem with
compiling software myself. Unfortunately, I then am responsible for the
security updates as well, which I don't want, because I am not a full
time administrator (I have to care for apache2, ntp, cyrus, sendmail,
mysql and subversion servers, having only a few hours per week besides
my normal job for doing this). Therefore, I'd like the Linux
distribution / package manager to care for the security fixes, which
means that I have to use the distribution's stock packages.
However, this is an exceptional case because I will eventually have to
upgrade anyway due to the problem with the shared folder namespace
showing twice (described in one of my other posts). This is a thing
which our users (including myself) probably can't accept, so if
upgrading is the only remedy, I'll eventually do that. But I'll first
wait for some answers to that other post.
> Hope this helps!
It really did :-)
Thank you very much again!
Binarus
Cyrus Home Page: http://www.cyrusimap.org/
List Archives/Info: http://lists.andrew.cmu.edu/pipermail/info-cyrus/
To Unsubscribe:
https://lists.andrew.cmu.edu/mailman/listinfo/info-cyrus
Re: Why do we need to run reconstruct after having moved mailboxes using cyradm's xfer command?
Hi Binarus,
> Could anybody please shortly explain why? What exactly are the
> techniques and mechanisms cyradm's xfer uses to do its thing? I had been
> quite sure that it just uses the IMAP protocol, but there seems to be
> more to it ...
You're right, but missing a detail. The XFER command is an IMAP extension, for
use in cyrus clusters ("murders"), where mailboxes need to be identical between
servers. I don't know much about the details, but it makes sense to me that it
would copy the mailboxes across at a "deeper" level, to make sure internal
identifiers and such match.
cyradm uses the IMAP protocol – think of it as a command-driven IMAP client,
except that it knows about cyrus extensions (like XFER) that generic IMAP
clients (like imapsync) probably don't
Something like imapsync would probably be using the CREATE and APPEND
commands instead, which let the destination server choose its own
internal identifiers, and which will probably be different from the source
ones. But they would be stored in the exact method the destination server
prefers (such as using the newest version of the mailbox index format...)
> Now that I have used cyradm's xfer command to relocate the mailboxes /
> messages / folders / subfolders, I surprisingly had to run the "heaviest"
> form of reconstruct (-V max).
That's not the "heaviest" form of reconstruct, it's just updating the mailbox
indexes to the "max"imum (-V)ersion supported by the server. This ends up
adding a bunch of extra information to the indexes that the newer server
version can make use of. Generally if the index version is too old for what
it's trying to do, it'll fall back to calculating missing values the long way,
or provide less info, or just refuse the requested operation. The assertion
failure you saw is clearly a bug, but this smells familiar, so maybe it's
already been fixed. The latest version of the 2.5 release is 2.5.12, and is
nearly 2 years newer than 2.5.10, for what it's worth.
Hope this helps!
ellie
On Sun, Dec 16, 2018, at 9:06 PM, Binarus wrote:
> Dear all,
>
> yesterday, I have moved a bunch of user mailboxes and public folders
> from a server running 2.4.16 to another server running 2.5.10, using
> cyradm's xfer command (see my previous messages).
>
> After having finished the migration, I noticed a weird behavior in
> Thunderbird (which is our standard email client): When trying to move a
> message from the Inbox to the Junk or Trash folder, the message
> disappeared from the Inbox for a short time, then reappeared. The log
> files on the server were showing the dreaded entries:
>
> Dec 16 01:12:59 morn cyrus/imaps[14914]: Fatal error: Internal error:
> assertion failed: imap/mailbox.c: 2850: !message_guid_isnull(
> >guid)
>
> "Fortunately", a lot of other users are affected by such log entries and
> weird email client behavior as well, so finding the solution on the 'net
> was not too difficult: Running reconstruct did not lead to anywhere, but
> running reconstruct -V max was the solution.
>
> This lets me scratch my head:
>
> In the past, I have upgraded Cyrus imapd at least three times, each time
> using imapsync (instead of cyradm's xfer command) to move the mailboxes
> and the public namespace (including all messages and subfolders) from
> the old server to the new one. In none of these cases, it was necessary
> to reconstruct anything afterwards. This would have been illogical
> anyway: Each time, the new server had been setup from scratch, and no
> mailboxes / messages / folders / subfolders had been moved directly via
> file transfer from the old to the new server.
>
> Now that I have used cyradm's xfer command to relocate the mailboxes /
> messages / folders / subfolders, I surprisingly had to run the
> "heaviest" form of reconstruct (-V max).
>
> Could anybody please shortly explain why? What exactly are the
> techniques and mechanisms cyradm's xfer uses to do its thing? I had been
> quite sure that it just uses the IMAP protocol, but there seems to be
> more to it ...
>
> Thank you very much for any insight,
>
> Binarus
>
> Cyrus Home Page: http://www.cyrusimap.org/
> List Archives/Info: http://lists.andrew.cmu.edu/pipermail/info-cyrus/
> To Unsubscribe:
> https://lists.andrew.cmu.edu/mailman/listinfo/info-cyrus
Cyrus Home Page: http://www.cyrusimap.org/
List Archives/Info: http://lists.andrew.cmu.edu/pipermail/info-cyrus/
To Unsubscribe:
https://lists.andrew.cmu.edu/mailman/listinfo/info-cyrus
Why do we need to run reconstruct after having moved mailboxes using cyradm's xfer command?
Dear all, yesterday, I have moved a bunch of user mailboxes and public folders from a server running 2.4.16 to another server running 2.5.10, using cyradm's xfer command (see my previous messages). After having finished the migration, I noticed a weird behavior in Thunderbird (which is our standard email client): When trying to move a message from the Inbox to the Junk or Trash folder, the message disappeared from the Inbox for a short time, then reappeared. The log files on the server were showing the dreaded entries: Dec 16 01:12:59 morn cyrus/imaps[14914]: Fatal error: Internal error: assertion failed: imap/mailbox.c: 2850: !message_guid_isnull(>guid) "Fortunately", a lot of other users are affected by such log entries and weird email client behavior as well, so finding the solution on the 'net was not too difficult: Running reconstruct did not lead to anywhere, but running reconstruct -V max was the solution. This lets me scratch my head: In the past, I have upgraded Cyrus imapd at least three times, each time using imapsync (instead of cyradm's xfer command) to move the mailboxes and the public namespace (including all messages and subfolders) from the old server to the new one. In none of these cases, it was necessary to reconstruct anything afterwards. This would have been illogical anyway: Each time, the new server had been setup from scratch, and no mailboxes / messages / folders / subfolders had been moved directly via file transfer from the old to the new server. Now that I have used cyradm's xfer command to relocate the mailboxes / messages / folders / subfolders, I surprisingly had to run the "heaviest" form of reconstruct (-V max). Could anybody please shortly explain why? What exactly are the techniques and mechanisms cyradm's xfer uses to do its thing? I had been quite sure that it just uses the IMAP protocol, but there seems to be more to it ... Thank you very much for any insight, Binarus Cyrus Home Page: http://www.cyrusimap.org/ List Archives/Info: http://lists.andrew.cmu.edu/pipermail/info-cyrus/ To Unsubscribe: https://lists.andrew.cmu.edu/mailman/listinfo/info-cyrus
