Re: xfer problems between 2.3.15 and 2.4.17
Hi, did you use allowallsubscribe in your imapd.conf? copied from imapd.conf manpage allowallsubscribe: 0 Allow subscription to nonexistent mailboxes. This option is typically used on backend servers in a Murder so that users can subscribe to mailboxes that don't reside on their home server. This option can also be used as a workaround for IMAP clients which don't play well with nonexistent or unselectable mailboxes (e.g., Microsoft Outlook). Regards Michael Menge Quoting gavin.g...@ed.ac.uk: Actually we get very little logged on our mupdate master, just logins, checkpointing notices etc, nothing about the work done to maintain folder state. It's one of the reasons we feel so in the dark with this issue. On Sat, 7 Jun 2014, Wesley Craig wrote: On 07 Jun 2014, at 14:49, Gavin Gray gavin.g...@ed.ac.uk wrote: yes it looks like for some reason the mupdate procedure that happens within the murder upon folder creation is getting held up for some reason with regard to the 2.4 frontend. but we don't know why or how to correct it. You should have logs from the mupdate master indicating when the various folder state changes happen, when the various frontends receive that information, etc. :wes Gavin Gray Edinburgh University Information Services Rm 2013 JCMB Kings Buildings Edinburgh EH9 3JZ UK tel +44 (0)131 650 5987 email gavin.g...@ed.ac.uk -- The University of Edinburgh is a charitable body, registered in Scotland, with registration number SC005336. Cyrus Home Page: http://www.cyrusimap.org/ List Archives/Info: http://lists.andrew.cmu.edu/pipermail/info-cyrus/ To Unsubscribe: https://lists.andrew.cmu.edu/mailman/listinfo/info-cyrus M.MengeTel.: (49) 7071/29-70316 Universität Tübingen Fax.: (49) 7071/29-5912 Zentrum für Datenverarbeitung mail: michael.me...@zdv.uni-tuebingen.de Wächterstraße 76 72074 Tübingen smime.p7s Description: S/MIME Signatur Cyrus Home Page: http://www.cyrusimap.org/ List Archives/Info: http://lists.andrew.cmu.edu/pipermail/info-cyrus/ To Unsubscribe: https://lists.andrew.cmu.edu/mailman/listinfo/info-cyrus
Re: xfer problems between 2.3.15 and 2.4.17
yes we have allowallsubscribe: yes in our frontend imapd.conf thanks On Tue, 10 Jun 2014, Michael Menge wrote: Hi, did you use allowallsubscribe in your imapd.conf? copied from imapd.conf manpage allowallsubscribe: 0 Allow subscription to nonexistent mailboxes. This option is typically used on backend servers in a Murder so that users can subscribe to mailboxes that don't reside on their home server. This option can also be used as a workaround for IMAP clients which don't play well with nonexistent or unselectable mailboxes (e.g., Microsoft Outlook). Regards Michael Menge Quoting gavin.g...@ed.ac.uk: Actually we get very little logged on our mupdate master, just logins, checkpointing notices etc, nothing about the work done to maintain folder state. It's one of the reasons we feel so in the dark with this issue. On Sat, 7 Jun 2014, Wesley Craig wrote: On 07 Jun 2014, at 14:49, Gavin Gray gavin.g...@ed.ac.uk wrote: yes it looks like for some reason the mupdate procedure that happens within the murder upon folder creation is getting held up for some reason with regard to the 2.4 frontend. but we don't know why or how to correct it. You should have logs from the mupdate master indicating when the various folder state changes happen, when the various frontends receive that information, etc. : wes Gavin Gray Edinburgh University Information Services Rm 2013 JCMB Kings Buildings Edinburgh EH9 3JZ UK tel +44 (0)131 650 5987 email gavin.g...@ed.ac.uk -- The University of Edinburgh is a charitable body, registered in Scotland, with registration number SC005336. Cyrus Home Page: http://www.cyrusimap.org/ List Archives/Info: http://lists.andrew.cmu.edu/pipermail/info-cyrus/ To Unsubscribe: https://lists.andrew.cmu.edu/mailman/listinfo/info-cyrus M.MengeTel.: (49) 7071/29-70316 Universität Tübingen Fax.: (49) 7071/29-5912 Zentrum für Datenverarbeitung mail: michael.me...@zdv.uni-tuebingen.de Wächterstraße 76 72074 Tübingen Gavin Gray Edinburgh University Information Services Rm 2013 JCMB Kings Buildings Edinburgh EH9 3JZ UK tel +44 (0)131 650 5987 email gavin.g...@ed.ac.uk -- The University of Edinburgh is a charitable body, registered in Scotland, with registration number SC005336. Cyrus Home Page: http://www.cyrusimap.org/ List Archives/Info: http://lists.andrew.cmu.edu/pipermail/info-cyrus/ To Unsubscribe: https://lists.andrew.cmu.edu/mailman/listinfo/info-cyrus
Re: xfer problems between 2.3.15 and 2.4.17
Hi, Quoting gavin.g...@ed.ac.uk: yes we have allowallsubscribe: yes in our frontend imapd.conf the man page mentions backend servers thanks On Tue, 10 Jun 2014, Michael Menge wrote: Hi, did you use allowallsubscribe in your imapd.conf? copied from imapd.conf manpage allowallsubscribe: 0 Allow subscription to nonexistent mailboxes. This option is typically used on backend servers in a Murder so that users can subscribe to mailboxes that don't reside on their home server. This option can also be used as a workaround for IMAP clients which don't play well with nonexistent or unselectable mailboxes (e.g., Microsoft Outlook). Regards Michael Menge Quoting gavin.g...@ed.ac.uk: Actually we get very little logged on our mupdate master, just logins, checkpointing notices etc, nothing about the work done to maintain folder state. It's one of the reasons we feel so in the dark with this issue. On Sat, 7 Jun 2014, Wesley Craig wrote: On 07 Jun 2014, at 14:49, Gavin Gray gavin.g...@ed.ac.uk wrote: yes it looks like for some reason the mupdate procedure that happens within the murder upon folder creation is getting held up for some reason with regard to the 2.4 frontend. but we don't know why or how to correct it. You should have logs from the mupdate master indicating when the various folder state changes happen, when the various frontends receive that information, etc. : wes Gavin Gray Edinburgh University Information Services Rm 2013 JCMB Kings Buildings Edinburgh EH9 3JZ UK tel +44 (0)131 650 5987 email gavin.g...@ed.ac.uk -- The University of Edinburgh is a charitable body, registered in Scotland, with registration number SC005336. Cyrus Home Page: http://www.cyrusimap.org/ List Archives/Info: http://lists.andrew.cmu.edu/pipermail/info-cyrus/ To Unsubscribe: https://lists.andrew.cmu.edu/mailman/listinfo/info-cyrus M.MengeTel.: (49) 7071/29-70316 Universität Tübingen Fax.: (49) 7071/29-5912 Zentrum für Datenverarbeitung mail: michael.me...@zdv.uni-tuebingen.de Wächterstraße 76 72074 Tübingen Gavin Gray Edinburgh University Information Services Rm 2013 JCMB Kings Buildings Edinburgh EH9 3JZ UK tel +44 (0)131 650 5987 email gavin.g...@ed.ac.uk -- The University of Edinburgh is a charitable body, registered in Scotland, with registration number SC005336. M.MengeTel.: (49) 7071/29-70316 Universität Tübingen Fax.: (49) 7071/29-5912 Zentrum für Datenverarbeitung mail: michael.me...@zdv.uni-tuebingen.de Wächterstraße 76 72074 Tübingen smime.p7s Description: S/MIME Signatur Cyrus Home Page: http://www.cyrusimap.org/ List Archives/Info: http://lists.andrew.cmu.edu/pipermail/info-cyrus/ To Unsubscribe: https://lists.andrew.cmu.edu/mailman/listinfo/info-cyrus
Re: xfer problems between 2.3.15 and 2.4.17
sorry I should've been more explicit, we also have the same thing in our backend's imapd.conf On Tue, 10 Jun 2014, Michael Menge wrote: Hi, Quoting gavin.g...@ed.ac.uk: yes we have allowallsubscribe: yes in our frontend imapd.conf the man page mentions backend servers thanks On Tue, 10 Jun 2014, Michael Menge wrote: Hi, did you use allowallsubscribe in your imapd.conf? copied from imapd.conf manpage allowallsubscribe: 0 Allow subscription to nonexistent mailboxes. This option is typically used on backend servers in a Murder so that users can subscribe to mailboxes that don't reside on their home server. This option can also be used as a workaround for IMAP clients which don't play well with nonexistent or unselectable mailboxes (e.g., Microsoft Outlook). Regards Michael Menge Quoting gavin.g...@ed.ac.uk: Actually we get very little logged on our mupdate master, just logins, checkpointing notices etc, nothing about the work done to maintain folder state. It's one of the reasons we feel so in the dark with this issue. On Sat, 7 Jun 2014, Wesley Craig wrote: On 07 Jun 2014, at 14:49, Gavin Gray gavin.g...@ed.ac.uk wrote: yes it looks like for some reason the mupdate procedure that happens within the murder upon folder creation is getting held up for some reason with regard to the 2.4 frontend. but we don't know why or how to correct it. You should have logs from the mupdate master indicating when the various folder state changes happen, when the various frontends receive that information, etc. : wes Gavin Gray Edinburgh University Information Services Rm 2013 JCMB Kings Buildings Edinburgh EH9 3JZ UK tel +44 (0)131 650 5987 email gavin.g...@ed.ac.uk -- The University of Edinburgh is a charitable body, registered in Scotland, with registration number SC005336. Cyrus Home Page: http://www.cyrusimap.org/ List Archives/Info: http://lists.andrew.cmu.edu/pipermail/info-cyrus/ To Unsubscribe: https://lists.andrew.cmu.edu/mailman/listinfo/info-cyrus M.MengeTel.: (49) 7071/29-70316 Universität Tübingen Fax.: (49) 7071/29-5912 Zentrum für Datenverarbeitung mail: michael.me...@zdv.uni-tuebingen.de Wächterstraße 76 72074 Tübingen Gavin Gray Edinburgh University Information Services Rm 2013 JCMB Kings Buildings Edinburgh EH9 3JZ UK tel +44 (0)131 650 5987 email gavin.g...@ed.ac.uk -- The University of Edinburgh is a charitable body, registered in Scotland, with registration number SC005336. M.MengeTel.: (49) 7071/29-70316 Universität Tübingen Fax.: (49) 7071/29-5912 Zentrum für Datenverarbeitung mail: michael.me...@zdv.uni-tuebingen.de Wächterstraße 76 72074 Tübingen Gavin Gray Edinburgh University Information Services Rm 2013 JCMB Kings Buildings Edinburgh EH9 3JZ UK tel +44 (0)131 650 5987 email gavin.g...@ed.ac.uk -- The University of Edinburgh is a charitable body, registered in Scotland, with registration number SC005336. Cyrus Home Page: http://www.cyrusimap.org/ List Archives/Info: http://lists.andrew.cmu.edu/pipermail/info-cyrus/ To Unsubscribe: https://lists.andrew.cmu.edu/mailman/listinfo/info-cyrus
Re: xfer problems between 2.3.15 and 2.4.17
I've switched to debug logging now on our mupdate master, thanks On Mon, 9 Jun 2014, Wesley Craig wrote: On 09 Jun 2014, at 07:14, gavin.g...@ed.ac.uk wrote: Actually we get very little logged on our mupdate master, just logins, checkpointing notices etc, nothing about the work done to maintain folder state. It's one of the reasons we feel so in the dark with this issue. Set logging on the mupdate master to debug, it shares lots of details. In particular, you want to see cmd_set and cmd_find. The cmd_set is from the backend making changes to the mupdate master. The cmd_find is *either* a frontend looking for particular mailboxes *or* the normal streaming of state change events from the mupdate master to the frontends. Obviously, there can be timing issues, but the typical paradigm implemented on the frontends is that when an expected mailbox isn't there, the frontend will specifically ask the mupdate master about it using a kick_mupdate(). :wes Gavin Gray Edinburgh University Information Services Rm 2013 JCMB Kings Buildings Edinburgh EH9 3JZ UK tel +44 (0)131 650 5987 email gavin.g...@ed.ac.uk -- The University of Edinburgh is a charitable body, registered in Scotland, with registration number SC005336. Cyrus Home Page: http://www.cyrusimap.org/ List Archives/Info: http://lists.andrew.cmu.edu/pipermail/info-cyrus/ To Unsubscribe: https://lists.andrew.cmu.edu/mailman/listinfo/info-cyrus
Re: xfer problems between 2.3.15 and 2.4.17
Hi Wes, It looks like the whole mupdate thing is working perfectly well. If I create a folder while connected to my 2.4.17 frontend then the logs show the backend issuing the cmd_set and then a bunch of cmd_find going out including to the frontend in question. Furthermore the new folder really is there in the mailboxes.db on the frontend. So in a way that's reassuring, but then why is the frontend telling email clients that the folder doesn't exist when a request to subscribe to it comes in? We aren't seeing any kick_mupdate getting logged. The only thing is we see two cmd_sets is that normal? : Jun 10 11:07:29 mupd1 mupdate[31917]: cmd_set(fd:5, user.gaving.gavarella) Jun 10 11:07:30 mupd1 mupdate[31917]: cmd_set(fd:5, user.gaving.gavarella) Jun 10 11:07:30 mupd1 mupdate[31917]: cmd_find(fd:29, user.gaving.gavarella) Jun 10 11:07:30 mupd1 mupdate[31917]: cmd_find(fd:29, user.gaving.gavarella) Jun 10 11:07:30 mupd1 mupdate[31917]: cmd_find(fd:24, user.gaving.gavarella) Jun 10 11:07:30 mupd1 mupdate[31917]: cmd_find(fd:24, user.gaving.gavarella) Jun 10 11:07:30 mupd1 mupdate[31917]: cmd_find(fd:25, user.gaving.gavarella) Jun 10 11:07:30 mupd1 mupdate[31917]: cmd_find(fd:25, user.gaving.gavarella) Jun 10 11:07:30 mupd1 mupdate[31917]: cmd_find(fd:10, user.gaving.gavarella) Jun 10 11:07:30 mupd1 mupdate[31917]: cmd_find(fd:10, user.gaving.gavarella) Jun 10 11:07:31 mupd1 mupdate[31917]: cmd_find(fd:11, user.gaving.gavarella) Jun 10 11:07:31 mupd1 mupdate[31917]: cmd_find(fd:11, user.gaving.gavarella) Jun 10 11:07:31 mupd1 mupdate[31917]: cmd_find(fd:36, user.gaving.gavarella) Jun 10 11:07:31 mupd1 mupdate[31917]: cmd_find(fd:36, user.gaving.gavarella) Jun 10 11:07:31 mupd1 mupdate[31917]: cmd_find(fd:26, user.gaving.gavarella) Jun 10 11:07:31 mupd1 mupdate[31917]: cmd_find(fd:38, user.gaving.gavarella) Jun 10 11:07:31 mupd1 mupdate[31917]: cmd_find(fd:38, user.gaving.gavarella) Jun 10 11:07:31 mupd1 mupdate[31917]: cmd_find(fd:14, user.gaving.gavarella) Jun 10 11:07:31 mupd1 mupdate[31917]: cmd_find(fd:22, user.gaving.gavarella) Jun 10 11:07:31 mupd1 mupdate[31917]: cmd_find(fd:22, user.gaving.gavarella) Jun 10 11:07:31 mupd1 mupdate[31917]: cmd_find(fd:26, user.gaving.gavarella) ... fd 38, and 26 is the 2.4.17 frontend in question. Thanks for helping out with this. Gavin On Mon, 9 Jun 2014, Wesley Craig wrote: On 09 Jun 2014, at 07:14, gavin.g...@ed.ac.uk wrote: Actually we get very little logged on our mupdate master, just logins, checkpointing notices etc, nothing about the work done to maintain folder state. It's one of the reasons we feel so in the dark with this issue. Set logging on the mupdate master to debug, it shares lots of details. In particular, you want to see cmd_set and cmd_find. The cmd_set is from the backend making changes to the mupdate master. The cmd_find is *either* a frontend looking for particular mailboxes *or* the normal streaming of state change events from the mupdate master to the frontends. Obviously, there can be timing issues, but the typical paradigm implemented on the frontends is that when an expected mailbox isn't there, the frontend will specifically ask the mupdate master about it using a kick_mupdate(). :wes Gavin Gray Edinburgh University Information Services Rm 2013 JCMB Kings Buildings Edinburgh EH9 3JZ UK tel +44 (0)131 650 5987 email gavin.g...@ed.ac.uk -- The University of Edinburgh is a charitable body, registered in Scotland, with registration number SC005336. Cyrus Home Page: http://www.cyrusimap.org/ List Archives/Info: http://lists.andrew.cmu.edu/pipermail/info-cyrus/ To Unsubscribe: https://lists.andrew.cmu.edu/mailman/listinfo/info-cyrus
Re: xfer problems between 2.3.15 and 2.4.17
On Tue, 10 Jun 2014, gavin.g...@ed.ac.uk wrote: Hi Wes, It looks like the whole mupdate thing is working perfectly well. If I create a folder while connected to my 2.4.17 frontend then the logs show the backend issuing the cmd_set and then a bunch of cmd_find going out including to the frontend in question. Furthermore the new folder really is there in the mailboxes.db on the frontend. So in a way that's reassuring, but then why is the frontend telling email clients that the folder doesn't exist when a request to subscribe to it comes in? We aren't seeing any kick_mupdate getting logged. I'm pretty sure your problem is that you have the proxyservers variable set in imapd.conf on your frontend. See this message from the archives: http://asg.andrew.cmu.edu/archive/message.php?mailbox=archive.info-cyrussearchterm=Murder%20mailbox%20create%20race%20conditionmsg=54193 I ran into this same problem, which was introduced by changes in v2.4.13. The imapd.conf manpage says: proxyservers: none A list of users and groups that are allowed to proxy for other users, separated by spaces. Any user listed in this will be allowed to login for any other user: use with caution. In a standard murder this option should ONLY be set on backends. DO NOT SET on frontends or things won't work properly. Let us know if removing proxyservers from your frontends fixes the problem! Andy Cyrus Home Page: http://www.cyrusimap.org/ List Archives/Info: http://lists.andrew.cmu.edu/pipermail/info-cyrus/ To Unsubscribe: https://lists.andrew.cmu.edu/mailman/listinfo/info-cyrus
Re: xfer problems between 2.3.15 and 2.4.17
this fix, detailed below, has indeed solved our problem with folder creation on our new 2.4.17 frontends. many thanks On Tue, 10 Jun 2014, Andrew Morgan wrote: On Tue, 10 Jun 2014, gavin.g...@ed.ac.uk wrote: Hi Wes, It looks like the whole mupdate thing is working perfectly well. If I create a folder while connected to my 2.4.17 frontend then the logs show the backend issuing the cmd_set and then a bunch of cmd_find going out including to the frontend in question. Furthermore the new folder really is there in the mailboxes.db on the frontend. So in a way that's reassuring, but then why is the frontend telling email clients that the folder doesn't exist when a request to subscribe to it comes in? We aren't seeing any kick_mupdate getting logged. I'm pretty sure your problem is that you have the proxyservers variable set in imapd.conf on your frontend. See this message from the archives: http://asg.andrew.cmu.edu/archive/message.php?mailbox=archive.info-cyrussearchterm=Murder%20mailbox%20create%20race%20conditionmsg=54193 I ran into this same problem, which was introduced by changes in v2.4.13. The imapd.conf manpage says: proxyservers: none A list of users and groups that are allowed to proxy for other users, separated by spaces. Any user listed in this will be allowed to login for any other user: use with caution. In a standard murder this option should ONLY be set on backends. DO NOT SET on frontends or things won't work properly. Let us know if removing proxyservers from your frontends fixes the problem! Andy Gavin Gray Edinburgh University Information Services Rm 2013 JCMB Kings Buildings Edinburgh EH9 3JZ UK tel +44 (0)131 650 5987 email gavin.g...@ed.ac.uk -- The University of Edinburgh is a charitable body, registered in Scotland, with registration number SC005336. Cyrus Home Page: http://www.cyrusimap.org/ List Archives/Info: http://lists.andrew.cmu.edu/pipermail/info-cyrus/ To Unsubscribe: https://lists.andrew.cmu.edu/mailman/listinfo/info-cyrus
Re: xfer problems between 2.3.15 and 2.4.17
Actually we get very little logged on our mupdate master, just logins, checkpointing notices etc, nothing about the work done to maintain folder state. It's one of the reasons we feel so in the dark with this issue. On Sat, 7 Jun 2014, Wesley Craig wrote: On 07 Jun 2014, at 14:49, Gavin Gray gavin.g...@ed.ac.uk wrote: yes it looks like for some reason the mupdate procedure that happens within the murder upon folder creation is getting held up for some reason with regard to the 2.4 frontend. but we don't know why or how to correct it. You should have logs from the mupdate master indicating when the various folder state changes happen, when the various frontends receive that information, etc. :wes Gavin Gray Edinburgh University Information Services Rm 2013 JCMB Kings Buildings Edinburgh EH9 3JZ UK tel +44 (0)131 650 5987 email gavin.g...@ed.ac.uk -- The University of Edinburgh is a charitable body, registered in Scotland, with registration number SC005336. Cyrus Home Page: http://www.cyrusimap.org/ List Archives/Info: http://lists.andrew.cmu.edu/pipermail/info-cyrus/ To Unsubscribe: https://lists.andrew.cmu.edu/mailman/listinfo/info-cyrus
Re: xfer problems between 2.3.15 and 2.4.17
All we see is the request from the client to subscrobe to the newly created folder failing for the reason that, the mailbox does not exist. we get this from logging the client's imap session. thanks, Gavin Quoting Dave McMurtrie dav...@andrew.cmu.edu on Thu, 5 Jun 2014 16:18:16 +: On Thu, 2014-06-05 at 16:15 +0100, gavin.g...@ed.ac.uk wrote: As you may be aware we are attempting this and have run into various problems. Currently we have a mixed murder of 2.3.15 backends and 2.4.17 backends. We are now fairly confident that we can xfer accounts succesfully between these backends. The problems we had appear to have been with a very small number of accounts on our older backends that had corrupt cyrus.index files. However we are now having trouble configuring frontends that will work with this mixed murder environment while we xfer our users accross. If we use our existing 2.3.15 frontends then users have who have been migrated lose the ability to see other accounts in the Other Users name space. On the other hand if we introduce 2.4.17 frontends then we see strange behaviour around folder creation. Clients can create the folders but autosubscription fails with the client being told the new folder doesn't exist. If one waits a minute or two one can manually subscribe to the folder. So far we have not upgraded the mupdate master. Is this a mistake? In terms of the frontend config we have added suppress_capabilities: ESEARCH QRESYNC WITHIN XLIST LIST-EXTENDED to the 2.4.17 frontends, otherwise the config is identical to our 2.3.15 frontends. Is there any other config changes we should be aware of? any ideas greatly appreciated... What you're doing should work just fine. It's exactly what we did to migrate our murder environment from 2.3.x to 2.4.x. I think I posted to the info-cyrus list about how we upgraded, but maybe I didn't. I got all the 2.4 backend servers built and ready to go, then I upgraded all the frontends to 2.4, then I used XFER to move all the mail from 2.3 to 2.4 servers. I don't recall exactly when I upgraded our mupdate server, but I don't think it matters. I don't believe anything changed in mupdate protocol or in the mailbox.db format between 2.3 and 2.4. Have you tried grabbing telemetry on the 2.4 server when the subscription fails? Is there anything being logged? Thanks! Dave -- Gavin Gray Edinburgh University Information Services Rm 2013 JCMB Kings Buildings Edinburgh EH9 3JZ UK tel +44 (0)131 650 5987 email gavin.g...@ed.ac.uk -- The University of Edinburgh is a charitable body, registered in Scotland, with registration number SC005336. -- The University of Edinburgh is a charitable body, registered in Scotland, with registration number SC005336. Cyrus Home Page: http://www.cyrusimap.org/ List Archives/Info: http://lists.andrew.cmu.edu/pipermail/info-cyrus/ To Unsubscribe: https://lists.andrew.cmu.edu/mailman/listinfo/info-cyrus
Re: xfer problems between 2.3.15 and 2.4.17
yes it looks like for some reason the mupdate procedure that happens within the murder upon folder creation is getting held up for some reason with regard to the 2.4 frontend. but we don't know why or how to correct it. thanks for getting back to us, Quoting Andrew Morgan mor...@orst.edu on Thu, 5 Jun 2014 11:02:11 -0700 (PDT): On Thu, 5 Jun 2014, gavin.g...@ed.ac.uk wrote: As you may be aware we are attempting this and have run into various problems. Currently we have a mixed murder of 2.3.15 backends and 2.4.17 backends. We are now fairly confident that we can xfer accounts succesfully between these backends. The problems we had appear to have been with a very small number of accounts on our older backends that had corrupt cyrus.index files. However we are now having trouble configuring frontends that will work with this mixed murder environment while we xfer our users accross. If we use our existing 2.3.15 frontends then users have who have been migrated lose the ability to see other accounts in the Other Users name space. On the other hand if we introduce 2.4.17 frontends then we see strange behaviour around folder creation. Clients can create the folders but autosubscription fails with the client being told the new folder doesn't exist. If one waits a minute or two one can manually subscribe to the folder. This is tickling my memory, but I can't recall exactly what it was. I remember running into a problem like this as well. Something about the frontend's mailbox database not being updated in a timely fashion... So far we have not upgraded the mupdate master. Is this a mistake? In terms of the frontend config we have added suppress_capabilities: ESEARCH QRESYNC WITHIN XLIST LIST-EXTENDED to the 2.4.17 frontends, otherwise the config is identical to our 2.3.15 frontends. Is there any other config changes we should be aware of? I used the following when I upgraded from 2.3 to 2.4: suppress_capabilities: ESEARCH LIST-EXTENDED QRESYNC WITHIN XLIST ENABLE SORT=DISPLAY There was a thread I started back in October 2011 with the subject 2.3 to 2.4 Murder upgrade where I ran through the upgrade and the workarounds I had to make. Andy -- Gavin Gray Edinburgh University Information Services Rm 2013 JCMB Kings Buildings Edinburgh EH9 3JZ UK tel +44 (0)131 650 5987 email gavin.g...@ed.ac.uk -- The University of Edinburgh is a charitable body, registered in Scotland, with registration number SC005336. -- The University of Edinburgh is a charitable body, registered in Scotland, with registration number SC005336. Cyrus Home Page: http://www.cyrusimap.org/ List Archives/Info: http://lists.andrew.cmu.edu/pipermail/info-cyrus/ To Unsubscribe: https://lists.andrew.cmu.edu/mailman/listinfo/info-cyrus
Re: xfer problems between 2.3.15 and 2.4.17
On Thu, 2014-06-05 at 16:15 +0100, gavin.g...@ed.ac.uk wrote: As you may be aware we are attempting this and have run into various problems. Currently we have a mixed murder of 2.3.15 backends and 2.4.17 backends. We are now fairly confident that we can xfer accounts succesfully between these backends. The problems we had appear to have been with a very small number of accounts on our older backends that had corrupt cyrus.index files. However we are now having trouble configuring frontends that will work with this mixed murder environment while we xfer our users accross. If we use our existing 2.3.15 frontends then users have who have been migrated lose the ability to see other accounts in the Other Users name space. On the other hand if we introduce 2.4.17 frontends then we see strange behaviour around folder creation. Clients can create the folders but autosubscription fails with the client being told the new folder doesn't exist. If one waits a minute or two one can manually subscribe to the folder. So far we have not upgraded the mupdate master. Is this a mistake? In terms of the frontend config we have added suppress_capabilities: ESEARCH QRESYNC WITHIN XLIST LIST-EXTENDED to the 2.4.17 frontends, otherwise the config is identical to our 2.3.15 frontends. Is there any other config changes we should be aware of? any ideas greatly appreciated... What you're doing should work just fine. It's exactly what we did to migrate our murder environment from 2.3.x to 2.4.x. I think I posted to the info-cyrus list about how we upgraded, but maybe I didn't. I got all the 2.4 backend servers built and ready to go, then I upgraded all the frontends to 2.4, then I used XFER to move all the mail from 2.3 to 2.4 servers. I don't recall exactly when I upgraded our mupdate server, but I don't think it matters. I don't believe anything changed in mupdate protocol or in the mailbox.db format between 2.3 and 2.4. Have you tried grabbing telemetry on the 2.4 server when the subscription fails? Is there anything being logged? Thanks! Dave Cyrus Home Page: http://www.cyrusimap.org/ List Archives/Info: http://lists.andrew.cmu.edu/pipermail/info-cyrus/ To Unsubscribe: https://lists.andrew.cmu.edu/mailman/listinfo/info-cyrus
Re: xfer problems between 2.3.15 and 2.4.17
On Thu, 5 Jun 2014, gavin.g...@ed.ac.uk wrote: As you may be aware we are attempting this and have run into various problems. Currently we have a mixed murder of 2.3.15 backends and 2.4.17 backends. We are now fairly confident that we can xfer accounts succesfully between these backends. The problems we had appear to have been with a very small number of accounts on our older backends that had corrupt cyrus.index files. However we are now having trouble configuring frontends that will work with this mixed murder environment while we xfer our users accross. If we use our existing 2.3.15 frontends then users have who have been migrated lose the ability to see other accounts in the Other Users name space. On the other hand if we introduce 2.4.17 frontends then we see strange behaviour around folder creation. Clients can create the folders but autosubscription fails with the client being told the new folder doesn't exist. If one waits a minute or two one can manually subscribe to the folder. This is tickling my memory, but I can't recall exactly what it was. I remember running into a problem like this as well. Something about the frontend's mailbox database not being updated in a timely fashion... So far we have not upgraded the mupdate master. Is this a mistake? In terms of the frontend config we have added suppress_capabilities: ESEARCH QRESYNC WITHIN XLIST LIST-EXTENDED to the 2.4.17 frontends, otherwise the config is identical to our 2.3.15 frontends. Is there any other config changes we should be aware of? I used the following when I upgraded from 2.3 to 2.4: suppress_capabilities: ESEARCH LIST-EXTENDED QRESYNC WITHIN XLIST ENABLE SORT=DISPLAY There was a thread I started back in October 2011 with the subject 2.3 to 2.4 Murder upgrade where I ran through the upgrade and the workarounds I had to make. Andy Cyrus Home Page: http://www.cyrusimap.org/ List Archives/Info: http://lists.andrew.cmu.edu/pipermail/info-cyrus/ To Unsubscribe: https://lists.andrew.cmu.edu/mailman/listinfo/info-cyrus
Re: xfer problems between 2.3.15 and 2.4.17
Thanks Ken, We have tried the -G and -R options to no avail. I've begun to experiment with your seen database idea. It would be nice to discover the reason for the corruption. We still don't even know whether the problem was there before the xfer or if it's a result of the xfer process. Obviously there are some major changes from cyrus 2.3.15 to 2.4.17, but then mostly folders are migrated and 'upgraded' correctly. cheers, Gavin Gray On Thu, 15 May 2014, Ken Murchison wrote: Hi Gavin, Have you tried running reconstruct with the -G and/or -R options to see if they fix the corruption without having to remove cyrus.index? Another option is to place a copy of the user's user.seen file from the 2.3 machine on the 2.4 machine prior to removing cyrus.index and reconstructing. I *think* the server will then upgrade the seen state from user.seen to cyrus.index. On 05/15/2014 11:53 AM, gavin.g...@ed.ac.uk wrote: Any help with this would be much appreciated. We keep coming across folders that once they have been migrated seem to have corrupt cyrus.index files. The only way to fix them is to remove the files and do a reconstruct. This is not a workable solution from our users point of view as is sets all the messages back to flaggged as new etc. We have tried various tests, but we can't discover the cause of the corruption of the cyrus.index files. regards, Gavin Gray On Wed, 7 May 2014, gavin.g...@ed.ac.uk wrote: I have been testing xfer of accounts within a cyrus murder from 2.3.15 backends to new 2.4.17. backends. all the email and folders seem to migrate perfectly and the xfer'd accounts can send and receive email. However when reading email with an IMAP client I am having strange issues setting flags within folders on messages. In particular setting and unsetting deletion flags is very erratic. In some folders it doesn't work at all, on others I can set the deletion flag but can't unset it. All of the backends have delayed expunge switched on. debug output from the alpine IMAP client seems to suggest the server is doing what it's told: IMAP DEBUG 12:04:30 5/7: 0169 STORE 93 +Flags (\DELETED) IMAP DEBUG 12:04:30 5/7: 0169 OK Completed IMAP DEBUG 12:04:31 5/7: 016a STORE 94 +Flags (\DELETED) IMAP DEBUG 12:04:31 5/7: 016a OK Completed but then something seems to immediately remove the flag, because on issuing an expunge the client finds nothing to expunge. nothing of note seems to be logged on the backend, even logging in debug mode. the other baffling thing is that in some folders within the same users account, this whole process works perfectly. does anyone have any ideas what could be causing this and if there might be a solution? many thanks, Gavin Gray Edinburgh University Information Services Rm 2013 JCMB Kings Buildings Edinburgh EH9 3JZ UK tel +44 (0)131 650 5987 email gavin.g...@ed.ac.uk -- The University of Edinburgh is a charitable body, registered in Scotland, with registration number SC005336. Cyrus Home Page: http://www.cyrusimap.org/ List Archives/Info: http://lists.andrew.cmu.edu/pipermail/info-cyrus/ To Unsubscribe: https://lists.andrew.cmu.edu/mailman/listinfo/info-cyrus Gavin Gray Edinburgh University Information Services Rm 2013 JCMB Kings Buildings Edinburgh EH9 3JZ UK tel +44 (0)131 650 5987 email gavin.g...@ed.ac.uk -- The University of Edinburgh is a charitable body, registered in Scotland, with registration number SC005336. Cyrus Home Page: http://www.cyrusimap.org/ List Archives/Info: http://lists.andrew.cmu.edu/pipermail/info-cyrus/ To Unsubscribe: https://lists.andrew.cmu.edu/mailman/listinfo/info-cyrus -- Kenneth Murchison Principal Systems Software Engineer Carnegie Mellon University Gavin Gray Edinburgh University Information Services Rm 2013 JCMB Kings Buildings Edinburgh EH9 3JZ UK tel +44 (0)131 650 5987 email gavin.g...@ed.ac.uk -- The University of Edinburgh is a charitable body, registered in Scotland, with registration number SC005336. Cyrus Home Page: http://www.cyrusimap.org/ List Archives/Info: http://lists.andrew.cmu.edu/pipermail/info-cyrus/ To Unsubscribe: https://lists.andrew.cmu.edu/mailman/listinfo/info-cyrus
Re: xfer problems between 2.3.15 and 2.4.17
So I'm going to attach the cyrus metadata files for a folder that behaved the way I am describing having been xfer'd between 2.3.15 and 2.4.17. these are the files from the source server ie the 2.3.15 backend. If anyone with far deeper knowledge of cyrus than we do, could have a look to see if they can identify a problem and possibly find a solution then we would be very grateful. Gavin Gray On Thu, 15 May 2014, gavin.g...@ed.ac.uk wrote: Any help with this would be much appreciated. We keep coming across folders that once they have been migrated seem to have corrupt cyrus.index files. The only way to fix them is to remove the files and do a reconstruct. This is not a workable solution from our users point of view as is sets all the messages back to flaggged as new etc. We have tried various tests, but we can't discover the cause of the corruption of the cyrus.index files. regards, Gavin Gray On Wed, 7 May 2014, gavin.g...@ed.ac.uk wrote: I have been testing xfer of accounts within a cyrus murder from 2.3.15 backends to new 2.4.17. backends. all the email and folders seem to migrate perfectly and the xfer'd accounts can send and receive email. However when reading email with an IMAP client I am having strange issues setting flags within folders on messages. In particular setting and unsetting deletion flags is very erratic. In some folders it doesn't work at all, on others I can set the deletion flag but can't unset it. All of the backends have delayed expunge switched on. debug output from the alpine IMAP client seems to suggest the server is doing what it's told: IMAP DEBUG 12:04:30 5/7: 0169 STORE 93 +Flags (\DELETED) IMAP DEBUG 12:04:30 5/7: 0169 OK Completed IMAP DEBUG 12:04:31 5/7: 016a STORE 94 +Flags (\DELETED) IMAP DEBUG 12:04:31 5/7: 016a OK Completed but then something seems to immediately remove the flag, because on issuing an expunge the client finds nothing to expunge. nothing of note seems to be logged on the backend, even logging in debug mode. the other baffling thing is that in some folders within the same users account, this whole process works perfectly. does anyone have any ideas what could be causing this and if there might be a solution? many thanks, Gavin Gray Edinburgh University Information Services Rm 2013 JCMB Kings Buildings Edinburgh EH9 3JZ UK tel +44 (0)131 650 5987 email gavin.g...@ed.ac.uk -- The University of Edinburgh is a charitable body, registered in Scotland, with registration number SC005336. Cyrus Home Page: http://www.cyrusimap.org/ List Archives/Info: http://lists.andrew.cmu.edu/pipermail/info-cyrus/ To Unsubscribe: https://lists.andrew.cmu.edu/mailman/listinfo/info-cyrus Gavin Gray Edinburgh University Information Services Rm 2013 JCMB Kings Buildings Edinburgh EH9 3JZ UK tel +44 (0)131 650 5987 email gavin.g...@ed.ac.uk -- The University of Edinburgh is a charitable body, registered in Scotland, with registration number SC005336. Cyrus Home Page: http://www.cyrusimap.org/ List Archives/Info: http://lists.andrew.cmu.edu/pipermail/info-cyrus/ To Unsubscribe: https://lists.andrew.cmu.edu/mailman/listinfo/info-cyrus Gavin Gray Edinburgh University Information Services Rm 2013 JCMB Kings Buildings Edinburgh EH9 3JZ UK tel +44 (0)131 650 5987 email gavin.g...@ed.ac.uk -- The University of Edinburgh is a charitable body, registered in Scotland, with registration number SC005336. cyrus.index Description: Binary data �� Cyrus mailbox header The best thing about this system was that it had lots of goals. --Jim Morris on Andrew user.jaw76a0f44a45710262 $NotJunk JunkRecorded $Junk NonJunk jaw lrswipkxtecda cyrus.cache Description: Binary data cyrus.expunge Description: Binary data Cyrus Home Page: http://www.cyrusimap.org/ List Archives/Info: http://lists.andrew.cmu.edu/pipermail/info-cyrus/ To Unsubscribe: https://lists.andrew.cmu.edu/mailman/listinfo/info-cyrus
Re: xfer problems between 2.3.15 and 2.4.17
Any help with this would be much appreciated. We keep coming across folders that once they have been migrated seem to have corrupt cyrus.index files. The only way to fix them is to remove the files and do a reconstruct. This is not a workable solution from our users point of view as is sets all the messages back to flaggged as new etc. We have tried various tests, but we can't discover the cause of the corruption of the cyrus.index files. regards, Gavin Gray On Wed, 7 May 2014, gavin.g...@ed.ac.uk wrote: I have been testing xfer of accounts within a cyrus murder from 2.3.15 backends to new 2.4.17. backends. all the email and folders seem to migrate perfectly and the xfer'd accounts can send and receive email. However when reading email with an IMAP client I am having strange issues setting flags within folders on messages. In particular setting and unsetting deletion flags is very erratic. In some folders it doesn't work at all, on others I can set the deletion flag but can't unset it. All of the backends have delayed expunge switched on. debug output from the alpine IMAP client seems to suggest the server is doing what it's told: IMAP DEBUG 12:04:30 5/7: 0169 STORE 93 +Flags (\DELETED) IMAP DEBUG 12:04:30 5/7: 0169 OK Completed IMAP DEBUG 12:04:31 5/7: 016a STORE 94 +Flags (\DELETED) IMAP DEBUG 12:04:31 5/7: 016a OK Completed but then something seems to immediately remove the flag, because on issuing an expunge the client finds nothing to expunge. nothing of note seems to be logged on the backend, even logging in debug mode. the other baffling thing is that in some folders within the same users account, this whole process works perfectly. does anyone have any ideas what could be causing this and if there might be a solution? many thanks, Gavin Gray Edinburgh University Information Services Rm 2013 JCMB Kings Buildings Edinburgh EH9 3JZ UK tel +44 (0)131 650 5987 email gavin.g...@ed.ac.uk -- The University of Edinburgh is a charitable body, registered in Scotland, with registration number SC005336. Cyrus Home Page: http://www.cyrusimap.org/ List Archives/Info: http://lists.andrew.cmu.edu/pipermail/info-cyrus/ To Unsubscribe: https://lists.andrew.cmu.edu/mailman/listinfo/info-cyrus Gavin Gray Edinburgh University Information Services Rm 2013 JCMB Kings Buildings Edinburgh EH9 3JZ UK tel +44 (0)131 650 5987 email gavin.g...@ed.ac.uk -- The University of Edinburgh is a charitable body, registered in Scotland, with registration number SC005336. Cyrus Home Page: http://www.cyrusimap.org/ List Archives/Info: http://lists.andrew.cmu.edu/pipermail/info-cyrus/ To Unsubscribe: https://lists.andrew.cmu.edu/mailman/listinfo/info-cyrus
Re: xfer problems between 2.3.15 and 2.4.17
Hi Gavin, Have you tried running reconstruct with the -G and/or -R options to see if they fix the corruption without having to remove cyrus.index? Another option is to place a copy of the user's user.seen file from the 2.3 machine on the 2.4 machine prior to removing cyrus.index and reconstructing. I *think* the server will then upgrade the seen state from user.seen to cyrus.index. On 05/15/2014 11:53 AM, gavin.g...@ed.ac.uk wrote: Any help with this would be much appreciated. We keep coming across folders that once they have been migrated seem to have corrupt cyrus.index files. The only way to fix them is to remove the files and do a reconstruct. This is not a workable solution from our users point of view as is sets all the messages back to flaggged as new etc. We have tried various tests, but we can't discover the cause of the corruption of the cyrus.index files. regards, Gavin Gray On Wed, 7 May 2014, gavin.g...@ed.ac.uk wrote: I have been testing xfer of accounts within a cyrus murder from 2.3.15 backends to new 2.4.17. backends. all the email and folders seem to migrate perfectly and the xfer'd accounts can send and receive email. However when reading email with an IMAP client I am having strange issues setting flags within folders on messages. In particular setting and unsetting deletion flags is very erratic. In some folders it doesn't work at all, on others I can set the deletion flag but can't unset it. All of the backends have delayed expunge switched on. debug output from the alpine IMAP client seems to suggest the server is doing what it's told: IMAP DEBUG 12:04:30 5/7: 0169 STORE 93 +Flags (\DELETED) IMAP DEBUG 12:04:30 5/7: 0169 OK Completed IMAP DEBUG 12:04:31 5/7: 016a STORE 94 +Flags (\DELETED) IMAP DEBUG 12:04:31 5/7: 016a OK Completed but then something seems to immediately remove the flag, because on issuing an expunge the client finds nothing to expunge. nothing of note seems to be logged on the backend, even logging in debug mode. the other baffling thing is that in some folders within the same users account, this whole process works perfectly. does anyone have any ideas what could be causing this and if there might be a solution? many thanks, Gavin Gray Edinburgh University Information Services Rm 2013 JCMB Kings Buildings Edinburgh EH9 3JZ UK tel +44 (0)131 650 5987 email gavin.g...@ed.ac.uk -- The University of Edinburgh is a charitable body, registered in Scotland, with registration number SC005336. Cyrus Home Page: http://www.cyrusimap.org/ List Archives/Info: http://lists.andrew.cmu.edu/pipermail/info-cyrus/ To Unsubscribe: https://lists.andrew.cmu.edu/mailman/listinfo/info-cyrus Gavin Gray Edinburgh University Information Services Rm 2013 JCMB Kings Buildings Edinburgh EH9 3JZ UK tel +44 (0)131 650 5987 email gavin.g...@ed.ac.uk -- The University of Edinburgh is a charitable body, registered in Scotland, with registration number SC005336. Cyrus Home Page: http://www.cyrusimap.org/ List Archives/Info: http://lists.andrew.cmu.edu/pipermail/info-cyrus/ To Unsubscribe: https://lists.andrew.cmu.edu/mailman/listinfo/info-cyrus -- Kenneth Murchison Principal Systems Software Engineer Carnegie Mellon University Cyrus Home Page: http://www.cyrusimap.org/ List Archives/Info: http://lists.andrew.cmu.edu/pipermail/info-cyrus/ To Unsubscribe: https://lists.andrew.cmu.edu/mailman/listinfo/info-cyrus
