Re: time for cyrus-imap v3.2?

[Michael Menge]

Hi all,

there are some bugs in cyrus 3.0/3.1 that i would like to see fixed
and I want to make sure that these changes will be able to be
included after 3.2 is released or will be fixed before 3.2 is released:

#2659 allow rename back from deleted mailbox when conversations is enabled
#2599 bug renaming/deleting special use folders in murder setup
#2598 squat search_engine not used

Also fixing "#2774 Murder does not work with TLS" would be
appreciate, if not possible the murder documentation should
at least been updated

Quoting my mail  
https://lists.andrew.cmu.edu/pipermail/cyrus-devel/2018-July/004297.html



Quoting ellie timoney :


Anyway, it looks to me like the STARTTLS support in mupdate is just  
 fundamentally broken at the moment, and my recommendation is to  
not  use it.  If your IMAP servers need to connect to an mupdate  
server  that's not within their trusted network, I guess you'd need  
to set  up a VPN for it or something along those lines (but I'm no  
network  specialist).


could you add a warning in the relevant murder/installation guides  
and  manuals?


Quoting Bron Gondwana :


On Tue, Nov 5, 2019, at 12:04, Ricardo Signes wrote:
So, I think the plan was to cut a stable Cyrus 3.2 after we had  
stable JMAP. Is that time now? We talked about this on the Zoom  
call today.


I think we're pretty close to it. The big question is: do we fork  
what will eventually become 3.2 and keep stabilising on it while we  
ship UUID mailboxes on master, or do we finish 3.2 before we merge  
uuid mailboxes.


Cyrus master has pretty stable for JMAP core and mail. I think we  
need to do one more pass through to look for places where Cyrus  
extensions might leak through without the correct `using` options,  
but apart from that, I don't think we expect its mail API to change  
apart from bugfixes.


Yep, legit. The one big thing still missing there is  
PushSubscriptions. I'd be keen to finish writing that. I mean:


https://github.com/cyrusimap/cyrus-imapd/issues?q=is%3Aopen+is%3Aissue+label%3A3.2

We should probably do a push and resolve all of those, then boom let's go.



there are some bugs in cyrus 3.0/3.1 that i would like to see fixed
and I want to make sure that these changes will be able to be
included after 3.2 is released or will be fixed before 3.2 is released:

#2659 allow rename back from deleted mailbox when conversations is enabled
#2599 bug renaming/deleting special use folders in murder setup
#2598 squat search_engine not used

Also fixing "#2774 Murder does not work with TLS" would be
appreciate, if not possible the murder documentation should
at least been updated

Quoting my mail  
https://lists.andrew.cmu.edu/pipermail/cyrus-devel/2018-July/004297.html



Quoting ellie timoney :


Anyway, it looks to me like the STARTTLS support in mupdate is just  
 fundamentally broken at the moment, and my recommendation is to  
not  use it.  If your IMAP servers need to connect to an mupdate  
server  that's not within their trusted network, I guess you'd need  
to set  up a VPN for it or something along those lines (but I'm no  
network  specialist).


could you add a warning in the relevant murder/installation guides  
and  manuals?








M.MengeTel.: (49) 7071/29-70316
Universität Tübingen   Fax.: (49) 7071/29-5912
Zentrum für Datenverarbeitung  mail:  
michael.me...@zdv.uni-tuebingen.de

Wächterstraße 76
72074 Tübingen

Re: cyrus-imap 2.5.3 - `lost' messages

[Michael Menge]

Quoting Dirk-Willem van Gulik :

On 14 Aug 2018, at 09:38, Michael Menge  
 wrote:



Quoting Dirk-Willem van Gulik <mailto:di...@webweaving.org>>:


On a smal cyrus imap 2.5.3 setup (freebsd ports, default settings,  
5k users, 300 Gb mboxes) we are seeing very sporadic `loss' of  
messages (once every few months; 1 or 2 messages; out of a few  
100k to a million going through perfectly fine in the same period).


They disappear; but are still present as a /var/spool/imap/user/X/12345.
files.

But cannot be found; even when searching through IMAP on  
message-ID or otherwise unique elements. Searches in the same  
veign for the messages before/after the lost one work and return  
those.


A grep through the cyrus.index shows its unique message ID.

A full  reconstruct with /-r/-R/-G does not seem to bring it back  
as visible.


The mesaages lost are otherwise normal; fully valid mime; and when
re-injected - pass through and surface.

What is the right way to debug this ? We suspect it may have  
something to do with either (al)pine and thunderbird; as it seems  
to hit users that mix those two clients - during periods they use  
both. All else seems normal.


Do you use delayed expunge mode (man imapd.conf see expunge_mode)?
If yes, "unexpunge -l user/X" will show you the mails that have  
been deleted but are still on the disk.

You will also see the time the mail was deleted.


Yes we do. And did not know about that command.

Where is this recorded / what rebuilds this file (and no - in this  
case - it was not deleted, we do use delayed expunge) ?




The idea behind delayed expunge is twofold, on one hand it postpones
the I/O to an more convenient time, where less interactive I/O is done,
and on the other hand it is very useful for backup/restore as you can
keep the messages for a few days so that your normal backup has a chance
to see the deleted mails even if the user (op a POP3 client) did delete
before the backup is run. As a additional benefit you can restore these
mails very easy with unexpunge (see the manpage for details)

We use the unix hierarchy separator "/" (imapd.conf unixhierarchysep: 1)
which is not the default for Cyrus 2.5. If you do use "." als  
hierarchy separator

you have to use "unexpunge -l user.X"

For more information about this take a look at  
https://www.cyrusimap.org/2.5/imap/features/delayed-expunge.html
as well as the imapd.conf manpage ( deletedprefix, delete_mode,  
expunge_mode, expunge_days) and cyr_expire manpage.


Regards,


   Michael





M.MengeTel.: (49) 7071/29-70316
Universität Tübingen   Fax.: (49) 7071/29-5912
Zentrum für Datenverarbeitung  mail:  
michael.me...@zdv.uni-tuebingen.de

Wächterstraße 76
72074 Tübingen

Re: cyrus-imap 2.5.3 - `lost' messages

[Michael Menge]

Quoting Dirk-Willem van Gulik :

On a smal cyrus imap 2.5.3 setup (freebsd ports, default settings,  
5k users, 300 Gb mboxes) we are seeing very sporadic `loss' of  
messages (once every few months; 1 or 2 messages; out of a few 100k  
to a million going through perfectly fine in the same period).


They disappear; but are still present as a /var/spool/imap/user/X/12345.
 files.

But cannot be found; even when searching through IMAP on message-ID  
or otherwise unique elements. Searches in the same veign for the  
messages before/after the lost one work and return those.


A grep through the cyrus.index shows its unique message ID.

A full  reconstruct with /-r/-R/-G does not seem to bring it back as visible.

The mesaages lost are otherwise normal; fully valid mime; and when
re-injected - pass through and surface.

What is the right way to debug this ? We suspect it may have  
something to do with either (al)pine and thunderbird; as it seems to  
hit users that mix those two clients - during periods they use both.  
All else seems normal.


Do you use delayed expunge mode (man imapd.conf see expunge_mode)?
If yes, "unexpunge -l user/X" will show you the mails that have been  
deleted but are still on the disk.

You will also see the time the mail was deleted.

Regards,


   Michael Menge


M.MengeTel.: (49) 7071/29-70316
Universität Tübingen   Fax.: (49) 7071/29-5912
Zentrum für Datenverarbeitung  mail:  
michael.me...@zdv.uni-tuebingen.de

Wächterstraße 76
72074 Tübingen

Re: cvt_xlist_specialuse does not work with subfolders

[Michael Menge]

Quoting Dilyan Palauzov :


Hello,

you can use cyradm with setmetadata:  
https://www.cyrusimap.org/dev/imap/reference/manpages/systemcommands/cyradm.html#setmetadata .  It works the sameway on 3.0, despite the fact the 3.0's documentation is silent on  
this.


When making chnages on master, e.g. updating the documengation,  
please verify if they are relevant for 3.0 and in such case backport  
them.


There is no \spam special use, it is \junk .



Thanks for the hint to the documentation.
I did read a bit more about XLIST, SPECIAL-USE and Annotations and
I found some inconsistent behavior, between IMAP setmetadata command (imtest),
cvt_xlist_specialuse and cyradm setmetadata

Here the setup for the account user/zrstes1

mamx02.mail.localhost> cm user/zrstes3
mamx02.mail.localhost> cm user/zrstes3/Mail
mamx02.mail.localhost> cm user/zrstes3/Mail/v-spam
mamx02.mail.localhost> cm user/zrstes3/Mail/trash
mamx02.mail.localhost> cm user/zrstes3/Mail/sent
mamx02.mail.localhost> cm user/zrstes3/Mail/drafts
mamx02.mail.localhost> cm user/zrstes3/newdrafts
mamx02.mail.localhost> cm user/zrstes3/NewDrafts
mamx02.mail.localhost> getmetadata user/zrstes3*
mamx02.mail.localhost> lm user/zrstes3*
user/zrstes3 (\HasChildren)
user/zrstes3/Mail (\HasChildren)
user/zrstes3/Mail/drafts (\HasNoChildren)
user/zrstes3/Mail/sent (\HasNoChildren)
user/zrstes3/Mail/trash (\HasNoChildren)
user/zrstes3/Mail/v-spam (\HasNoChildren)
user/zrstes3/NewDrafts (\HasNoChildren)
user/zrstes3/newdrafts (\HasNoChildren)
mamx02.mail.localhost> setmetadata user/zrstes3/Mail/sent specialuse "\\Sent"
mamx02.mail.localhost> getmetadata user/zrstes3*
{user/zrstes3/Mail/sent}:
  private:
specialuse: \Sent
mamx02.mail.localhost>


Switching to imtest

a LIST "" "*"
* LIST (\HasNoChildren) "/" INBOX
* LIST (\HasChildren) "/" Mail
* LIST (\HasNoChildren) "/" Mail/drafts
* LIST (\HasNoChildren) "/" Mail/sent
* LIST (\HasNoChildren) "/" Mail/trash
* LIST (\HasNoChildren) "/" Mail/v-spam
* LIST (\HasNoChildren) "/" NewDrafts
* LIST (\HasNoChildren) "/" newdrafts
a OK Completed (0.000 secs 8 calls)
a XLIST "" "Mail/sent"
* XLIST (\HasNoChildren) "/" Mail/sent
a OK Completed (0.000 secs 1 calls)
b LIST (SPECIAL-USE) "" "*"
b OK Completed (0.000 secs 8 calls)
c LIST "" "Mail/sent" RETURN (SPECIAL-USE)
* LIST (\HasNoChildren) "/" Mail/sent
c OK Completed (0.000 secs 1 calls)


No Special use visible, lets set it via IMAP for an other folder.


d SETMETADATA "Mail/trash" (/private/specialuse "\\Trash")
d OK Completed
e XLIST "" "Mail/trash"
* XLIST (\HasNoChildren \Trash) "/" Mail/trash
e OK Completed (0.000 secs 1 calls)
f LIST (SPECIAL-USE) "" "*"
* LIST (\HasNoChildren \Trash) "/" Mail/trash
f OK Completed (0.000 secs 8 calls)

Lets check in cyradm again

mamx02.mail.localhost> getmetadata user/zrstes3*
{user/zrstes3/Mail/sent}:
  private:
specialuse: \Sent
mamx02.mail.localhost>

Still only the sent folder we configured via cyradm before.

So lets try cvt_xlist_specialuse running with "xlist-drafts: newdrafts"

cvt_xlist_specialuse -C /etc/imapd_be.conf -v user/zrstes3*
will set \drafts for folders named newdrafts
will set \sent for folders named Mail.sent
will set \trash for folders named Mail.trash
will set \junk for folders named Mail.s-spam
will set \spam for folders named Mail.v-spam
be/cvt_xlist_specialuse[3847]: couldn't authenticate to backend  
server: SASL library is not initialized

be/cvt_xlist_specialuse[3847]: mupdate_connect failed: no auth status
be/cvt_xlist_specialuse[3847]: cannot connect to mupdate server for  
update of 'user.zrstes3.newdrafts'
be/cvt_xlist_specialuse[3847]: auditlog: modseq  
sessionid=  
mailbox=  
uniqueid= highestmodseq=<2>

set specialuse \drafts for user.zrstes3.newdrafts

Checking in imtest.

g LIST (SPECIAL-USE) "" "*"
* LIST (\HasNoChildren \Trash) "/" Mail/trash
* LIST (\HasNoChildren \drafts) "/" newdrafts
g OK Completed (0.000 secs 8 calls)

and in cyradm

mamx02.mail.localhost> getmetadata user/zrstes3*
{user/zrstes3/Mail/sent}:
  private:
specialuse: \Sent
mamx02.mail.localhost>

I was wondering if the Annotation set by cvt_xlist_specialuse should  
be \\ instead of \drafts
so i tried again with "/" as delimiter for xlist-sent, xlist-trash,  
xlist-junk, xlist-spam and with

xlist-Drafts: NewDrafts



cvt_xlist_specialuse -C /etc/imapd_be.conf -v user/zrstes3*

will set \drafts for folders named NewDrafts
will set \sent for folders named Mail/sent
will set \trash for folders named Mail/trash
will set \junk for folders named Mail/s-spam
will set \spam for folders named Mail/v-spam
not s

cvt_xlist_specialuse does not work with subfolders

[Michael Menge]

Hi,

We are testing the migration from Cyrus 2.4.20 to 3.0.7
and have discovered a problem with cvt_xlist_specialuse
(migration of the xlist-* options to specialuse annotations).

Below are the important options from the imapd.conf of the current backends

altnamespace: 1
unixhierarchysep: 1
xlist-sent: Mail.sent
xlist-trash: Mail.trash
xlist-drafts: Mail.drafts
xlist-junk: Mail.v-spam
xlist-spam: Mail.v-spam
specialusealways: 1

--- default 
virtdomains: off


cvt_xlist_specialuse did only output

will set \drafts for folders named Mail.drafts
will set \sent for folders named Mail.sent
will set \trash for folders named Mail.trash
will set \junk for folders named Mail.s-spam
will set \spam for folders named Mail.v-spam

but was not able to find any mailboxes to update the annotation.

First we suspected that with the changed default setting for unixhierarchysep
the format of xlist- Options might have changed to unixhierarchysep as well.

But setting

xlist-sent: Mail/sent
xlist-trash: Mail/trash
xlist-drafts: Mail/drafts
xlist-junk: Mail/v-spam
xlist-spam: Mail/v-spam

did only change the output of cvt_xlist_specialuse to

will set \drafts for folders named Mail/drafts
will set \sent for folders named Mail/sent
will set \trash for folders named Mail/trash
will set \junk for folders named Mail/s-spam
will set \spam for folders named Mail/v-spam

but it still didn't find any mailboxes to update the annotations.

After changing the xlist- option to a direct folder of the INBOX e.g. Mail
cvt_xlist_specialuse was able to find the folder and set the annotation.

Is there a way to set the specialuse annotations with cyradm or perl script?

Regards

Michael Menge





M.MengeTel.: (49) 7071/29-70316
Universität Tübingen   Fax.: (49) 7071/29-5912
Zentrum für Datenverarbeitung  mail:  
michael.me...@zdv.uni-tuebingen.de

Wächterstraße 76
72074 Tübingen

Re: use valgrind / Re: SIGSEGV in cyrus-imapd 3.0.7 mupdate

[Michael Menge]

Hi Ellie

thanks for your replies,

Quoting ellie timoney :

From what I'm seeing here, it looks like mupdate calls  
tls_init_serverengine() for each new STARTTLS session, and then  
calls tls_shutdown_serverengine()  when that session ends.


The thing is though, the TLS state that these functions manage is  
something like a singleton, it should only exist once for each  
process.  And in fact, tls_init_serverengine() enforces this (which  
is why you don't see the dh_params being allocated for each  
connection: it only lets itself run once).  This works fine for all  
the single-threaded service programs, but mupdate is a threaded  
application and can have multiple connected client sessions in the  
same process.  The comments at the top of imap/tls.c are instructive:



* DESCRIPTION
*   This module is the interface between Cyrus Imapd and the  
OpenSSL library.

*   As of now only one filedescriptor can be handled, so only one
*   TLS channel can be open at a time.


I can't see anything in the code that makes me suspect this comment  
is out of date.  It doesn't look the slightest bit thread-safe.


At the moment, it looks to me like the STARTTLS support was added to  
mupdate with no consideration of its thread-safety, and I'm left to  
assume that people aren't really using STARTTLS for their mupdate  
connections, otherwise this probably would've been tripped over long  
ago.  (Or, maybe it was, and someone said "don't do that", and then  
it all got lost to history?  It would be interesting to hear from  
the front lines.)


I have been running mupdate with STARTTLS for a few years with 2.3.x  
and 2.4.x.

It didn't crash as predictable as 3.0, but we had some instances where the
mupdate master crashed and stopped working and we had to restart the service
(on an other host). At that time we where unable to debug it, as it would
run without crash for weeks or months. I suspect now that the  
non-thread-safe usage

of openssl could have been the cause for this.

Using STARTTLS for mupdate was the obvious solution after the default  
for "allowplaintext"

(Allow the use of cleartext passwords on the wire) was changed to 0.



Anyway, it looks to me like the STARTTLS support in mupdate is just  
fundamentally broken at the moment, and my recommendation is to not  
use it.  If your IMAP servers need to connect to an mupdate server  
that's not within their trusted network, I guess you'd need to set  
up a VPN for it or something along those lines (but I'm no network  
specialist).




could you add a warning in the relevant murder/installation guides and  
manuals?


I don't honestly see this being fixed any time soon -- it would  
require either:


 * a big rewrite of imap/tls.c to make it thread-safe
 * a big rewrite of mupdate to make it single-threaded like the rest of Cyrus


out of curiosity, why is mupdate multi-threaded in the first place?

 * a big rewrite of mupdate to make it do its own TLS handling  
(rather than using imap/tls.c)




This sounds like it is the same work as the first item, but it is
only used for mupdate.

All of these are serious tasks with serious testing requirements,  
especially considering the need to interact with OpenSSL correctly.   
Even if someone does produce patches for master, they won't make it  
back to the 3.0 series.




so is there a chance for a patch in 3.1?


Sorry to be the bearer of annoying news!


I will see how I can work around this.

The cyrus project has much improved since fastmail dedicated
some developers to it, but I think it is time that fastmail switches
to a murder setup, so that this part of the code gets the same love
and testing as the other parts of cyrus ;-)

Thanks to all helping debugging this problem. I will gladly help
with testing any fixes for this bug.

Kind regards,

   Michael


M.MengeTel.: (49) 7071/29-70316
Universität Tübingen   Fax.: (49) 7071/29-5912
Zentrum für Datenverarbeitung  mail:  
michael.me...@zdv.uni-tuebingen.de

Wächterstraße 76
72074 Tübingen

Re: use valgrind / Re: SIGSEGV in cyrus-imapd 3.0.7 mupdate

[Michael Menge]

Hi,

Дилян had suggested to add some debug outputs to imap/tls.c



diff --git a/imap/tls.c b/imap/tls.c
--- a/imap/tls.c
+++ b/imap/tls.c
@@ -893,7 +893,9 @@ EXPORTED int tls_init_serverengine(const char *ident,

 #if (OPENSSL_VERSION_NUMBER >= 0x0090800fL)
 /* Load DH params for DHE-* key exchanges */
+syslog(LOG_CRIT, "dh_params will be set, current value=%p", dh_params);
 dh_params = load_dh_param(server_key_file, server_cert_file);
+syslog(LOG_CRIT, "dh_params were set current_value=%p", dh_params);
 SSL_CTX_set_tmp_dh(s_ctx, dh_params);
 #endif

@@ -1308,7 +1310,11 @@ EXPORTED int tls_shutdown_serverengine(void)
 }

 #if (OPENSSL_VERSION_NUMBER >= 0x0090800fL)
-if (dh_params) DH_free(dh_params);
+if (dh_params) {
+   syslog(LOG_CRIT, "dh_params will be freed %p", dh_params);
+   DH_free(dh_params);
+   syslog(LOG_CRIT, "dh_params were freed %p", dh_params);
+   }
 #endif
 }
-

I did run mupdate with this debug output


Jul 12 10:17:25 mx02 mu/mupdate[6537]: dh_params will be set, current  
value=(nil)
Jul 12 10:17:25 mx02 mu/mupdate[6537]: inittls: Loading DH parameters  
from file
Jul 12 10:17:25 mx02 mu/mupdate[6537]: dh_params were set  
current_value=0x7fc7541b9600
Jul 12 10:17:25 mx02 mu/mupdate[6537]: starttls: TLSv1.2 with cipher  
ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits new) no authentication
Jul 12 10:17:25 mx02 mu/mupdate[6537]: login: msmx02.mail.localhost  
[10.23.21.78] cyrus PLAIN+TLS User logged in
Jul 12 10:18:37 mx02 mu/mupdate[6537]: starttls: TLSv1.2 with cipher  
ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits new) no authentication
Jul 12 10:18:37 mx02 mu/mupdate[6537]: login: msmx02.mail.localhost  
[10.23.21.78] cyrus PLAIN+TLS User logged in

Jul 12 10:18:37 mx02 mu/mupdate[6537]: dh_params will be freed 0x7fc7541b9600
Jul 12 10:18:37 mx02 mu/mupdate[6537]: dh_params were freed 0x7fc7541b9600
Jul 12 10:18:37 mx02 mu/mupdate[6537]: starttls: TLSv1.2 with cipher  
ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits new) no authentication
Jul 12 10:18:37 mx02 mu/mupdate[6537]: login: msmx02.mail.localhost  
[10.23.21.78] cyrus PLAIN+TLS User logged in

Jul 12 10:18:37 mx02 mu/mupdate[6537]: dh_params will be freed 0x7fc7541b9600
Jul 12 10:18:38 mx02 mu/master[6534]: process type:SERVICE  
name:mupdate path:/usr/local/libexec/mupdate age:89.131s pid:6537  
signaled to death by signal 11 (Segmentation fault, core dumped)
Jul 12 10:18:38 mx02 mu/master[6534]: service mupdate/ipv4 pid 6537 in  
READY state: terminated abnormally


so it seems to me that the dh_params were set once on startup but  
freed for each closed connection


Regards

Michael Menge


Quoting Michael Menge :


Hi Дилян,


Quoting Дилян Палаузов :


Hello Michael,

this is likely either a memory mishandling issue (use after free(),
double free(), invalid read()/write()...), which gets evident if cyrus
is run under valgrind --tool=memcheck.  I run it with

valgrind --num-callers=30 --leak-check=full --track-origins=yes --read-
var-info=yes --show-leak-kinds=all --trace-children=yes --track-fds=yes
/usr/local/cyrus/bin/master -D &> log-file-memcheck



thanks for the valgind options. Valgrind did find some "Invalid read  
of size 4"
and "size 8" in DH_free as well as "Invalid write of size 4" in  
CRYPTO_add_lock
and "size 8" in OPENSSL_cleanse. As far as i can tell the memory was  
always free'd

before by CRYPTO_free (mem.c:434).

I have attached the full log



Another reason can be multi-threaded inconsistencies: mutexes locked in
inconsistent order by differnt threads (while this is not a cause for
crash, it leads to deadlock), mutexes locked in one thread and unlocked
in another or alike.  This can be detected by valgrind/helgrind

valgrind --tool=helgrind --num-callers=30 --leak-check=full --track-
origins=yes --read-var-info=yes --trace-children=yes --track-fds=yes
/usr/local/cyrus/bin/master -D &> log-file-helgrind



I had to remove some options here, as my valgrind didn't know them with
--tool=helgrind

I used /usr/bin/valgrind --tool=helgrind --num-callers=30  
--read-var-info=yes --trace-children=yes --track-fds=yes  
/usr/local/libexec/master -C /etc/imapd_mu.conf -M  
/etc/cyrus_mu.conf -p /var/run/cyurs_mu.pid -D &>  
/tmp/cyrus-mupdate-log-file-helgrind


Valgrind did find some "Possible data race during read" and "This  
conflicts with a previous write"

I have attached the full log as well




Of course, it is useful to have debugging symbols.  Compiling with -O3
migh make things faster, while compiling with -O0 will make it run
considerably slower. If ld (=ld.bfd) is very, very recent (e.g. 2.31),
then link with  -Wl,-z,noseparate-code otherwise valgrind cannot read
the debug info (https://bugs.kde.org/show_bug.cgi?id=395682).

Then it shall be possible to see in o

Re: SIGSEGV in cyrus-imapd 3.0.7 mupdate

[Michael Menge]

Hi Ellie


Quoting ellie timoney :


Hi,

Are you confident that the backtrace in your earlier email is from  
the thread that crashed?




no, as I had no experience in debugging multithreaded programs before,
but looking at all threads of multiple core dumps DH_free is a
common pattern.


mupdate is multithreaded, so obtaining the correct backtrace from a  
crash is a bit of work (but maybe you already did this)


I used Redhats abrtd to collect the coredumps. This program also collects some
other information and produces an core_backtrace json output and  
groups similar

coredumps together. I can provide these if anyone is interested, but they
are to big to mail.



It's been a while since I've done any multithreaded debugging with  
gcc (since Cyrus is predominantly single-threaded), but my fingers  
remember using "thread apply all bt" and going from there, if that's  
any help :)




I have attached the output of "thread apply all bt" form three different
core dumps.



On Thu, Jul 5, 2018, at 7:40 PM, Michael Menge wrote:

Quoting Michael Menge :

> Hi,
>
> we are in the process of setting up our new production mailserver with
> cyrus-imapd 3.0.7 on RHEL 7.5 Servers.
>
> At the moment we encounter many crashes (SIGSEGV) of the mupdate  
process on

> the mupdate master instance. As soon as we issue a command that updates
> multiple mailboxes in short time we trigger a SIGSEGV.
>
>> sam user/zrstes* cyrus all
> Setting ACL on user/zrstes1...OK.
> Setting ACL on user/zrstes1/Mail...OK.
> Setting ACL on user/zrstes1/Mail/drafts...OK.
> Setting ACL on user/zrstes1/Mail/s-spam...OK.
> Setting ACL on user/zrstes1/Mail/sent...OK.
> Setting ACL on user/zrstes1/Mail/trash...OK.
> Setting ACL on user/zrstes1/Mail/v-spam...OK.
> Setting ACL on user/zrstes2...cyrus: lrswipkxtea: no connection to server
>
> I suspect that at this time the connection from the backend to the
> mupdate master
> is lost as the mupdate process received the SIGSEGV
>
> (gdb) bt
> #0  0x7fb49c98 in ?? ()
> #1  0x7fb4a9a59a85 in DH_free (r=0x7fb49c1b9600) at dh_lib.c:194
> #2  0x7fb4aa84dbef in tls_shutdown_serverengine () at imap/tls.c:1311
> #3  0x00404075 in conn_free (C=0x7fb4840009f0) at  
imap/mupdate.c:379

> #4  0x004062c0 in thread_main (rock=0x0) at imap/mupdate.c:1330
> #5  0x7fb4a771add5 in start_thread (arg=0x7fb4a22d9700) at
> pthread_create.c:308
> #6  0x7fb4a718fb3d in clone () at
> ../sysdeps/unix/sysv/linux/x86_64/clone.S:113
>
>
> Further information, as DH_free is in the bt, we have NOT configured
> dh_params in
> our ssl certificate/key or in the imapd.conf
>

As a follow up, we did generate dh_params, but the mupdate still crashes
with SIGSEGV. Any ideas how to debug/fix this? Is there any
information that is missing?

Regards

Michael Menge







M.MengeTel.: (49) 7071/29-70316
Universität Tübingen   Fax.: (49) 7071/29-5912
Zentrum für Datenverarbeitung  mail:  
michael.me...@zdv.uni-tuebingen.de

Wächterstraße 76
72074 Tübingen
GNU gdb (GDB) Red Hat Enterprise Linux 7.6.1-110.el7
Copyright (C) 2013 Free Software Foundation, Inc.
License GPLv3+: GNU GPL version 3 or later <http://gnu.org/licenses/gpl.html>
This is free software: you are free to change and redistribute it.
There is NO WARRANTY, to the extent permitted by law.  Type "show copying"
and "show warranty" for details.
This GDB was configured as "x86_64-redhat-linux-gnu".
For bug reporting instructions, please see:
<http://www.gnu.org/software/gdb/bugs/>...
Reading symbols from /usr/local/libexec/mupdate...done.
[New LWP 10478]
[New LWP 10482]
[New LWP 10481]
[New LWP 10479]
[New LWP 10480]
[New LWP 10476]
[New LWP 10477]
[Thread debugging using libthread_db enabled]
Using host libthread_db library "/lib64/libthread_db.so.1".
Core was generated by `mupdate -m -C /etc/imapd_mu.conf'.
Program terminated with signal 11, Segmentation fault.
#0  OPENSSL_cleanse () at x86_64cpuid.s:192
192 movq%rax,(%rdi)
(gdb) 
Thread 7 (Thread 0x7fc98341f700 (LWP 10477)):
#0  0x7fc986062a2d in accept () at ../sysdeps/unix/syscall-template.S:81
#1  0x004098fa in mupdate_placebo_kick_start (rock=0x0) at 
imap/mupdate-slave.c:351
#2  0x7fc98605bdd5 in start_thread (arg=0x7fc98341f700) at 
pthread_create.c:308
#3  0x7fc985ad0b3d in clone () at 
../sysdeps/unix/sysv/linux/x86_64/clone.S:113

Thread 6 (Thread 0x7fc98b43c8c0 (LWP 10476)):
#0  0x7fc986062a2d in accept () at ../sysdeps/unix/syscall-template.S:81
#1  0x0040a232 in main (argc=4, argv=0x7ffc0cfcfd28, 
envp=0x7ffc0cfcfd50) at master/service-thread.c:268

Thread 5 (Thread 0x7fc9

Re: SIGSEGV in cyrus-imapd 3.0.7 mupdate

[Michael Menge]

Quoting Michael Menge :


Hi,

we are in the process of setting up our new production mailserver with
cyrus-imapd 3.0.7 on RHEL 7.5 Servers.

At the moment we encounter many crashes (SIGSEGV) of the mupdate process on
the mupdate master instance. As soon as we issue a command that updates
multiple mailboxes in short time we trigger a SIGSEGV.


sam user/zrstes* cyrus all

Setting ACL on user/zrstes1...OK.
Setting ACL on user/zrstes1/Mail...OK.
Setting ACL on user/zrstes1/Mail/drafts...OK.
Setting ACL on user/zrstes1/Mail/s-spam...OK.
Setting ACL on user/zrstes1/Mail/sent...OK.
Setting ACL on user/zrstes1/Mail/trash...OK.
Setting ACL on user/zrstes1/Mail/v-spam...OK.
Setting ACL on user/zrstes2...cyrus: lrswipkxtea: no connection to server

I suspect that at this time the connection from the backend to the  
mupdate master

is lost as the mupdate process received the SIGSEGV

(gdb) bt
#0  0x7fb49c98 in ?? ()
#1  0x7fb4a9a59a85 in DH_free (r=0x7fb49c1b9600) at dh_lib.c:194
#2  0x7fb4aa84dbef in tls_shutdown_serverengine () at imap/tls.c:1311
#3  0x00404075 in conn_free (C=0x7fb4840009f0) at imap/mupdate.c:379
#4  0x004062c0 in thread_main (rock=0x0) at imap/mupdate.c:1330
#5  0x7fb4a771add5 in start_thread (arg=0x7fb4a22d9700) at  
pthread_create.c:308
#6  0x7fb4a718fb3d in clone () at  
../sysdeps/unix/sysv/linux/x86_64/clone.S:113



Further information, as DH_free is in the bt, we have NOT configured  
dh_params in

our ssl certificate/key or in the imapd.conf



As a follow up, we did generate dh_params, but the mupdate still crashes
with SIGSEGV. Any ideas how to debug/fix this? Is there any  
information that is missing?


Regards

   Michael Menge



M.MengeTel.: (49) 7071/29-70316
Universität Tübingen   Fax.: (49) 7071/29-5912
Zentrum für Datenverarbeitung  mail:  
michael.me...@zdv.uni-tuebingen.de

Wächterstraße 76
72074 Tübingen

SIGSEGV in cyrus-imapd 3.0.7 mupdate

[Michael Menge]

Hi,

we are in the process of setting up our new production mailserver with
cyrus-imapd 3.0.7 on RHEL 7.5 Servers.

At the moment we encounter many crashes (SIGSEGV) of the mupdate process on
the mupdate master instance. As soon as we issue a command that updates
multiple mailboxes in short time we trigger a SIGSEGV.


sam user/zrstes* cyrus all

Setting ACL on user/zrstes1...OK.
Setting ACL on user/zrstes1/Mail...OK.
Setting ACL on user/zrstes1/Mail/drafts...OK.
Setting ACL on user/zrstes1/Mail/s-spam...OK.
Setting ACL on user/zrstes1/Mail/sent...OK.
Setting ACL on user/zrstes1/Mail/trash...OK.
Setting ACL on user/zrstes1/Mail/v-spam...OK.
Setting ACL on user/zrstes2...cyrus: lrswipkxtea: no connection to server

I suspect that at this time the connection from the backend to the  
mupdate master

is lost as the mupdate process received the SIGSEGV

(gdb) bt
#0  0x7fb49c98 in ?? ()
#1  0x7fb4a9a59a85 in DH_free (r=0x7fb49c1b9600) at dh_lib.c:194
#2  0x7fb4aa84dbef in tls_shutdown_serverengine () at imap/tls.c:1311
#3  0x00404075 in conn_free (C=0x7fb4840009f0) at imap/mupdate.c:379
#4  0x004062c0 in thread_main (rock=0x0) at imap/mupdate.c:1330
#5  0x7fb4a771add5 in start_thread (arg=0x7fb4a22d9700) at  
pthread_create.c:308
#6  0x7fb4a718fb3d in clone () at  
../sysdeps/unix/sysv/linux/x86_64/clone.S:113



Further information, as DH_free is in the bt, we have NOT configured  
dh_params in

our ssl certificate/key or in the imapd.conf

Kind regards

   Michael Menge


M.MengeTel.: (49) 7071/29-70316
Universität Tübingen   Fax.: (49) 7071/29-5912
Zentrum für Datenverarbeitung  mail:  
michael.me...@zdv.uni-tuebingen.de

Wächterstraße 76
72074 Tübingen

Re: Cyrus meeting minutes: 13 Nov 2017

[Michael Menge]

Hi,

Quoting Nicola Nye :



Release plans:
   2.5 has some patches to go in, no ETA on next release
   3.0 has some patches to go in, no ETA on next release
   3.2 not until next year, with full JMAP and other juicy stuff



i have send an patch regarding mupdate rollback for 2.4.20 to the list  
on Sep 15,

which is still relevant for the 2.5 and 3.0 versions/branches.

As i didn't see any reactions from the developers till now I assume  
was missed.

I created a issues on github as this seems to be the place to report bugs now.

https://github.com/cyrusimap/cyrus-imapd/issues/2199

Regards

   Michael Menge


M.MengeTel.: (49) 7071/29-70316
Universität Tübingen   Fax.: (49) 7071/29-5912
Zentrum für Datenverarbeitung  mail:  
michael.me...@zdv.uni-tuebingen.de

Wächterstraße 76
72074 Tübingen

Re: moving mailboxes to other partitons in murder setup can lead missing entries in mailboxes.db

[Michael Menge]

Hi,


Quoting Stephan Lauffer :


Hello Michael,

Zitat von Michael Menge :


Hi,

i discovered that my patch for bug #3862 (rollback db changes on  
mupdate failure),
which was includes in cyrus-imapd 2.4.19, 2.5.8 and 3.0.0-beta2,  
has a bug if a

mailbox is moved to an other partition and a rollback is needed.

For the rollback my old patch did recreate the old entry and then  
delete the new entry.
But in case of moving the mailbox to a other partition the oldname  
and newname are

the same, so the mailbox will be deleted for the mailboxes.db
Recovery form this error is tricky. Reconstruct (reconstruct -p  
oldpartiton "mailboxname")

will fail because the mupdate master still has the entry the the mailbox.
So you have to manually add the mailbox to the mailboxes.db on the backend,
or delete the mailbox from the mailboxes.db on the mupdate master.


Attached is a patch for 2.4.20 which will reverse the order, so the  
"new" mailbox is

deleted first and then the old entry is recreated.

The same is needed for the other branches.


Btw: For me it looks like we have a wrong test for removing (or not), see:

https://build.opensuse.org/package/view_file/home:nixda:devel/cyrus-imapd-3.0/cyrus-imapd-3.0.2-mbx_rename.patch?expand=1

What do you think about this patch?



If i read this patch right it negates the requirement to keep the same  
mailboxname
on partition changes to require it to change the mailbox name. This  
would prevent
the bug, but it would be a burden to migrate data to an other  
partition, as it would

require an addition rename.

Also I don't know if there are other reasons to allow or disallow the  
change of

the mailboxname but requiring it seams wrong to me. This check predates my
original patch so there seems to have been a reason in the past to  
disallow it.


Where did this patch came from?

Regards

   Michael Menge



M.MengeTel.: (49) 7071/29-70316
Universität Tübingen   Fax.: (49) 7071/29-5912
Zentrum für Datenverarbeitung  mail:  
michael.me...@zdv.uni-tuebingen.de

Wächterstraße 76
72074 Tübingen

moving mailboxes to other partitons in murder setup can lead missing entries in mailboxes.db

[Michael Menge]

Hi,

i discovered that my patch for bug #3862 (rollback db changes on  
mupdate failure),
which was includes in cyrus-imapd 2.4.19, 2.5.8 and 3.0.0-beta2, has a  
bug if a

mailbox is moved to an other partition and a rollback is needed.

For the rollback my old patch did recreate the old entry and then  
delete the new entry.
But in case of moving the mailbox to a other partition the oldname and  
newname are

the same, so the mailbox will be deleted for the mailboxes.db
Recovery form this error is tricky. Reconstruct (reconstruct -p  
oldpartiton "mailboxname")

will fail because the mupdate master still has the entry the the mailbox.
So you have to manually add the mailbox to the mailboxes.db on the backend,
or delete the mailbox from the mailboxes.db on the mupdate master.


Attached is a patch for 2.4.20 which will reverse the order, so the  
"new" mailbox is

deleted first and then the old entry is recreated.

The same is needed for the other branches.

Regards

   Michael Menge



M.MengeTel.: (49) 7071/29-70316
Universität Tübingen   Fax.: (49) 7071/29-5912
Zentrum für Datenverarbeitung  mail:  
michael.me...@zdv.uni-tuebingen.de

Wächterstraße 76
72074 Tübingen
diff -u -r cyrus-imapd-2.4.20/imap/mboxlist.c cyrus-imapd-2.4.20.mupdate_commit_patch2/imap/mboxlist.c
--- cyrus-imapd-2.4.20/imap/mboxlist.c	2017-08-18 02:29:14.0 +0200
+++ cyrus-imapd-2.4.20.mupdate_commit_patch2/imap/mboxlist.c	2017-09-14 13:59:06.0 +0200
@@ -1186,14 +1186,14 @@
 if (mupdatecommiterror) {
 r = 0;
 
-/* recreate an old entry */
+/* delete the new entry */
 if (!isusermbox)
-r = DB->store(mbdb, oldname, strlen(oldname),
-  mboxent, strlen(mboxent), &tid);
+r = DB->delete(mbdb, newname, strlen(newname), &tid, 0);
 
-/* delete the new entry */
+/* recreate an old entry */
 if (!r)
-r = DB->delete(mbdb, newname, strlen(newname), &tid, 0);
+r = DB->store(mbdb, oldname, strlen(oldname),
+  mboxent, strlen(mboxent), &tid);
 
 /* Commit transaction */
 if (!r)

Re: Safe to delete apparently stale / aged 'spool/sync.' directory?

[Michael Menge via Cyrus-devel]

Hi,


Quoting Karl Pielorz via Cyrus-devel :


Hi,

We've upgraded a number of Cyrus servers through the years - we're  
currently running 2.5.x.


The other day I noticed on a couple of servers we have:

 /vol/imap/spool/sync.

This directory hasn't been touched in years - but has a number of  
sub-dirs (all numbers) - that all contain messages (again, created  
years ago).


Am I safe to remove this directory?

We also have:

 /vol/imap/sync

Which periodically has files created in it (the server is sync'ing  
to a another machine - so that's expected).


But I can't tell if 'vol/imap/sync' is a replacement [now, after  
upgrades] for 'vol/imap/spool/sync.' - i.e. if the 'sync.' directory  
can either be cleared out (and left), or just got rid of all together.





AFAIK the sync. folder is used by the sync_server (receiving side of  
the syncronisation)
as tmp storage for the incomming mails/data. The the number (subfolder  
name) is the pid

of sync_server process.

On clean shutdown these files/folders should be clenad up, but if the  
server crashes

there might be some old files leftover.

IHMO it should be safe to delete the folders on the active server  
(where the sync_client is running)
and old folders where the corresponding process is nor running any  
more on the receiving (sync_server)

side.





M.MengeTel.: (49) 7071/29-70316
Universität Tübingen   Fax.: (49) 7071/29-5912
Zentrum für Datenverarbeitung  mail:  
michael.me...@zdv.uni-tuebingen.de

Wächterstraße 76
72074 Tübingen

race conditon on 2 concurrent mailbox rename in murder setup

[Michael Menge via Cyrus-devel]

Hi,

I discovered a race condition in cyrus imapd 2.4.18 if 2 concurrent  
rename operation are running on the same folder



19:44:15 mailserv08 imap[1001]: Rename: user.userA.Folder ->  
user.userA.Mail.someotherFolder.Folder
19:45:27 mailserv08 imap[1002]: Rename: user.userA.Folder ->  
user.userA.jetanotherFolder.Folder

19:46:43 mailserv08 imap[1001]: Deleted mailbox user.userA.Folder
19:46:43 mailserv08 imap[1002]: MUPDATE: can't commit mailbox entry  
for 'user.userA.jetanotherFolder.Folder'
19:46:43 mailserv08 imap[1002]: Deleted mailbox  
user.userA.jetanotherFolder.Folder


The first rename succeeds, but the second rename recreates the  
mailboxdb entry for the original folder, which does not exist in the  
filesystem. Which will lead to IOErrors if the original folder is  
accessed (e.g. cyr_expung / squatter)


I am not sure if the race condition was introduced by my patch for Bug  
3862 (https://bugzilla.cyrusimap.org/show_bug.cgi?id=3862) or if it  
existed before.


At the moment i restore the original folder (mkdir Folder; touch  
cyrus.index cyrus.header; reconstruct)
and delete it with cyradm. Is there an better way to remove the non  
existing folder from the

mailboxdb / mupdate without restoring it?


Regards,


Michael Menge



M.MengeTel.: (49) 7071/29-70316
Universität Tübingen   Fax.: (49) 7071/29-5912
Zentrum für Datenverarbeitung  mail:  
michael.me...@zdv.uni-tuebingen.de

Wächterstraße 76
72074 Tübingen

Re: Request for docs help: lmtpproxyd, message_test, pop3proxyd, search_test, squat_dump, synctest

[Michael Menge via Cyrus-devel]

Hi,

Quoting Nicola Nye via Cyrus-devel :


G'day,

I'm trying to flesh out our man pages and corresponding html reference
files for all the programs and tooling that is shipped with Cyrus.

Is anyone able to shed some insight as to what the following tools do
and why you'd use them:

lmtpproxyd, message_test, pop3proxyd, search_test, squat_dump, synctest


lmtpproxyd and pop3proxyd are the same as lmtpd and pop3d.

AFAIK they are from the time where the murder frontend used its own
binaries with only the proxy functionality. I guess they are kept
for upgrade/compatibility reasons so the cyrus.conf could be kept
as is.

Maybe we should use links for lmtpproxyd and pop3proxyd to make
it more obvious.



Bonus marks if you know what all the options are.

Alternatively, if they're obsolete/nobody uses them anyway/confusing,
then that's good to know too, and we can look at no longer shipping them
with each release.

Cheers,

Nicola





M.MengeTel.: (49) 7071/29-70316
Universität Tübingen   Fax.: (49) 7071/29-5912
Zentrum für Datenverarbeitung  mail:  
michael.me...@zdv.uni-tuebingen.de

Wächterstraße 76
72074 Tübingen

Status of Bug 3862

[Michael Menge via Cyrus-devel]

Hi,

this night our mupdate Master had again a problem, which caused that  
no change could be

committed to the mupdate master mailbox.db.

This alone wouldn’t be much of a problem if there wasn't the bug 3862.
https://bugzilla.cyrusimap.org/show_bug.cgi?id=3862#c3

So now i have to cleanup all failed mailbox renames

We use Cyrus IMAP 2.4.17

can someone else reproduce this bug with cyrus 2.4.x?
  -- Setup non unified murder cluster,
  -- create some mailboxes
  -- block communication between backend and mupdate master
  -- try to rename mailbox

Is this bug fixed in Cyrus IMAP 2.5 or 3?

Regards

   Michael Menge


M.MengeTel.: (49) 7071/29-70316
Universität Tübingen   Fax.: (49) 7071/29-5912
Zentrum für Datenverarbeitung  mail:  
michael.me...@zdv.uni-tuebingen.de

Wächterstraße 76
72074 Tübingen

Re: Update to Murder docs (D69) and question on style.

[Michael Menge]

Hi,

Quoting Nic Bernstein :


On 08/21/2015 02:58 AM, Michael Menge wrote:

Hi,


Quoting Nic Bernstein :

I can write this up, I just wasn't sure if it was still needed.  I  
put a big ol' Note: in the replication page saying:


  Important

  Within a Cyrus /Murder/
<https://docs.cyrus.foundation/imap/developer/architecture.html#architecture-murder>
  environment, replicas must *not* be configured to invoke
  ctl_mboxlist(8)
<http://docs.cyrus.foundation/imap/admin/commands/ctl_mboxlist.html>
  on startup (pushing the local mailbox list to the *Mupdate Master*).
  This may only be done on the Master instance.

That's the only real gotcha I know of, but, having said that, I  
did write up a brief set of instructions about this very topic not  
that long ago (IIRC) for user mailing list.  I figured I could  
start with that.





For the initial configuration I second this. But there are IHMO things
to consider on failover.


1. ctl_mboxlist must be used with -m and -a Option on failover
2. on big installations updating all entries in mailbox.db on the
  mupdate server can take some time, on our setup we switch the IP address
  of master and replic on failover


And on 21 August at 06:12AM CDT, Ken Murchison wrote:
Right.  We don't even have our replicas as part of our Murder.   
They replicate their backend as if it were a standalone server.



So the replica has no clue about the Murder.  We switch DNS between  
the two hosts during failover, so no IP address change [the servers  
are in different data centers, so that wouldn't be practical in any  
event].  I doubt we actually need that last blob of stuff on the  
replica, but it doesn't seem to have hurt.


As for /etc/cyrus.conf, we do something similar, in regards to  
commenting/uncommenting START entries for ctl_mboxlist and  
sync_client, versus an SERVICES entry for sync_server.


It's not the cleanest process in failover, but a damn sight better  
than nothing.


I should have been more clearer in the fist place.

We change the IP and servername: in imapd.conf on the new master to  
the  value of the old master

so that there is no changes in the mailbox.db.

The advantage of changing the dns is you don't have to worry about  
duplicate IPs after a split brain,

but you have to make sure that the dns lookup is not cached.


@main devs

how do you plan to integrate mupdate and replication? I hope that it  
will not need to rewrite all

entries in mailbox.db on failover.

Regards

Michael






M.MengeTel.: (49) 7071/29-70316
Universität Tübingen   Fax.: (49) 7071/29-5912
Zentrum für Datenverarbeitung  mail:  
michael.me...@zdv.uni-tuebingen.de

Wächterstraße 76
72074 Tübingen

Re: Update to Murder docs (D69) and question on style.

[Michael Menge]

Hi,


Quoting Nic Bernstein :


On 08/20/2015 09:07 PM, Bron Gondwana wrote:

On Fri, Aug 21, 2015, at 11:42, Nicola Nye wrote:

On Fri, Aug 21, 2015, at 11:35 AM, Bron Gondwana wrote:
Murder plus replication is a giant ball of suck right now.  They  
don't know about each other, and they interact badly :(
So is that a case for documenting "Here be dragons, enter at your  
own risk" for the moment?
Does it mean that admins should look at other backup mechanisms to  
handle failover and redundancy in a murder environment, because  
Cyrus's replication doesn't play nicely with murder?

Or... ?
Or we need to fix murder and replication to work together nicely,  
but that's hard work[tm].




Ahem, some of us run Murderous Replicas all day long and it works  
well enough...  Doesn't it? ;-)


I can write this up, I just wasn't sure if it was still needed.  I  
put a big ol' Note: in the replication page saying:


   Important

   Within a Cyrus /Murder/



   environment, replicas must *not* be configured to invoke
   ctl_mboxlist(8)
   
   on startup (pushing the local mailbox list to the *Mupdate Master*).
   This may only be done on the Master instance.

That's the only real gotcha I know of, but, having said that, I did  
write up a brief set of instructions about this very topic not that  
long ago (IIRC) for user mailing list.  I figured I could start with  
that.





For the initial configuration I second this. But there are IHMO things
to consider on failover.


1. ctl_mboxlist must be used with -m and -a Option on failover
2. on big installations updating all entries in mailbox.db on the
   mupdate server can take some time, on our setup we switch the IP address
   of master and replic on failover

Regards

   Michael






M.MengeTel.: (49) 7071/29-70316
Universität Tübingen   Fax.: (49) 7071/29-5912
Zentrum für Datenverarbeitung  mail:  
michael.me...@zdv.uni-tuebingen.de

Wächterstraße 76
72074 Tübingen

Re: Today's pop quiz: replication

[Michael Menge]

Hi,


Quoting ellie timoney :


- a single cyrus instance may be the primary server for some users
  but a

replica server for other users

Are you sure about that?


I'm

 not sure about any of this.  But you make a good point: I thought I
 could see a way that this was possible, but now I don't think I can.

I think I can, again, though I guess not in a murder.

But in a non-murder setup, I believe the only difference between a
primary vs replica is "which one the user interacts with"?  Which, in a
non-murder setup, one is presumably managing in some way external to
cyrus (e.g. database, proxies, and some glue)...

So it seems like it should be plausible to have two (or more) cyrus
instances (hosted wherever), each running a sync_server plus a
channel/sync_client for each of the others, and whatever happens on any
replicates to the others.  You'd be trusting your outer, non-cyrus layer
to make sure not to proxy any individual user to the wrong instance (at
least, not while there's pending replication transactions for them).
And I guess shared folders would be thorny.  But I don't (yet) see a
reason from a replication standpoint why this wouldn't work.


One possible problem I am seeing is that after a split brain, or switch of
primary and replica for some users there might be some problems for renamed
folders, subsciptions and folder annotations. Cyrus is able to handle changes
on both sides in a mailbox  (new mail, deleded mail, changed mail) because
of modseq, but this is missing for mailbox metadata.




So:

- a single cyrus instance may* be the primary server for some users but
  a replica server for other users

[* with caveats and requiring custom tooling, and specifically not
in a murder]


On the other hand working with multiple instances on one host is working fine
(even in an murder setup).

See http://www.spinics.net/lists/info-cyrus/msg16035.html



ellie

On Fri, Jul 24, 2015, at 10:01 AM, ellie timoney wrote:



Okay, I'll bite.  Here's what a bit of a sync_log looks like:


Thanks!  So anything processing a sync_log (sync_client, squatter,
etc) needs to look at an actual copy of the user/mailbox in order to
determine its current state, and needs to look at both copies to work
out what to replicate between them.


- a single cyrus instance may be the primary server for some users
  but a

replica server for other users

Are you sure about that?


I'm not sure about any of this.  But you make a good point: I thought
I could see a way that this was possible, but now I don't think I can.

Cheers,

ellie

On Thu, Jul 23, 2015, at 11:42 PM, Nic Bernstein wrote:

On 07/23/2015 01:14 AM, ellie timoney wrote:

Is the file format of the sync log defined anywhere? I assume it
> correlates with a set of commands. (Not that this is important to
> a user: it may as well be opaque, but it made me wonder!)

I'm a bit confused about this myself.  Each time I go digging
into the

code my understanding flips back the opposite way.

I think, either:

* the sync log contains all the information needed to reproduce what's
  happened (e.g. if a message has arrived, the sync log will contain the
  message itself); OR
* the sync log contains just enough to identify things that have changed
  (e.g. if a message has arrived, the sync log contains a message id of
  some sort), and the sync_client processing the log just uses the log
  to discover which things to sync, but then uses the actual mailbox to
  construct the changes to send to the replica.

Either way I haven't seen any documentation on the sync log format.  I
suspect it's either the raw sync protocol or some subset thereof?


Okay, I'll bite.  Here's what a bit of a sync_log looks like:

MAILBOX user.newjersey

MAILBOX user.support USER onlight USER nic USER admin USER randy MAILBOX
user.randy.Trash USER lynn MAILBOX user.lynn.Trash

It's basically a list of either users or mailboxes which have been

altered.  When sync_log is enabled, all of the daemons which might
alter a mailbox or user will write a line to this log each time they
do so.  That means the obvious suspects -- imapd, pop3d, timsieved,
lmtpd, etc. -- but also cyr_expire and friends (as in the
USER...MAILBOX couplets at the end of the sample).



- a single cyrus instance may be the primary server for some users
  but a

replica server for other users

Are you sure about that?  How does one specify the users for which

such an instance would play each role?  A single HOST may run
separate instances, which may perform these different roles, but I
cannot fathom how to configure a single instance to do both at once
for different user cohorts.


This raises potential problems when one deploys replication within a

murder (Cyrus aggregation).  Only one server may claim ownership of
any given mailbox, via a mupdate call, so an instance which is a
replica MAY NOT push updates to mupdate master, or mayhem will
ensue.  Here's a commented section f

Re: Today's pop quiz: replication

[Michael Menge]

Hi,


Quoting ellie timoney :


Okay, I'll bite.  Here's what a bit of a sync_log looks like:


Thanks!  So anything processing a sync_log (sync_client, squatter, etc)
needs to look at an actual copy of the user/mailbox in order to
determine its current state, and needs to look at both copies to work
out what to replicate between them.



Only sync_client processes the sync_log. All other processes only log  
to sync_log

and therefore only need to know where they changed something.


- a single cyrus instance may be the primary server for some users
  but a

replica server for other users

Are you sure about that?


I'm not sure about any of this.  But you make a good point: I thought I
could see a way that this was possible, but now I don't think I can.

Cheers,

ellie





M.MengeTel.: (49) 7071/29-70316
Universität Tübingen   Fax.: (49) 7071/29-5912
Zentrum für Datenverarbeitung  mail:  
michael.me...@zdv.uni-tuebingen.de

Wächterstraße 76
72074 Tübingen

Re: Cyrus 2.5 status

[Michael Menge]

Quoting Bron Gondwana :



Here's a typed up version of the list::




* Mailbox on-disk paths == folder uniqueid
  * fast, atomic rename - including multiple folders
  * fix delayed_delete to just keep old uniqueid in mailboxes.db =>  
no DELETED. prefix

  * fast undelete of entire folders
  * store current mailbox name inside cyrus.header for reconstruct
  * only works now that we store uniqueid in mailboxes.db (DLIST format)


This will make finding mailboxes in filebased backups a lot harder,
as we would have to restore the mailboxes.db from backup first to get  
the uniqueid. Using softlinks (mailboxname == link name, link points  
to the uniqid)

might work depending how the backup program shows links.

Regards

   Michael


M.MengeTel.: (49) 7071/29-70316
Universität Tübingen   Fax.: (49) 7071/29-5912
Zentrum für Datenverarbeitung  mail:  
michael.me...@zdv.uni-tuebingen.de

Wächterstraße 76
72074 Tübingen

smime.p7s
Description: S/MIME Signatur

PATCH for lmtp was [frontend lmtp connections to mupdate master]

[Michael Menge]

Hi,


Quoting Michael Menge :


Quoting Michael Menge :


By thew way, the reason I was so surprised in the first place was, that
I have been fooled by the documentation:
Quoting https://cyrusimap.org/docs/cyrus-imapd/2.4.17/install-murder.php


Configuring the frontends
[...]
However, because the frontends only talk to the mupdate master via  
 a  slave running on the local machine, you will also need to set  
up  a  slave on the same machine, in the SERVICES section of   
cyrus.conf,  like so


And an other misleading DOC !
Quoting   
http://www.cyrusimap.org/mediawiki/index.php/Cyrus_Murder_Mail_Delivery

UMich patch

Patch Patches are against 2.2 but are being moved to 2.3 Lmtpproxyd  
 will now use the local mailboxes.db




Quoting Wesley Craig :

On 29 Sep 2014, at 10:08, Michael Menge  
 wrote:

thanks for your mail. By any chance do you still have the patch?


In fact, I don't.  Dusting off my notes, I recall now that instead  
of patching this (bogus) code, we instead decided to configure the  
frontends as "unified", while leaving the backends "standard".  The  
only downside of this approach was that a naive admin could  
potentially create a user's mailbox on a frontend.


Sorry I said that we ported that code forward: in fact we simply got  
the effect of consulting mailboxes.db without porting forward.


:wes


I have patched my lmtpd to use the local mailbox.db in all cases,
unified Murder, and standalone cyrus already did this.
I tested it with a few testmails on my test system,
but as i don't have a unified setup or a standalone test system
at the moment i didn't test these setups.

I would appreciate a review.


Regards,

   Michael Menge



M.MengeTel.: (49) 7071/29-70316
Universität Tübingen   Fax.: (49) 7071/29-5912
Zentrum für Datenverarbeitung  mail:  
michael.me...@zdv.uni-tuebingen.de

Wächterstraße 76
72074 Tübingen--- cyrus-imapd-2.4.17/imap/lmtpd.c	2012-12-01 20:57:54.0 +0100
+++ cyrus-imapd-2.4.17.lmtp_patch/imap/lmtpd.c	2014-09-29 15:27:55.0 +0200
@@ -186,56 +186,44 @@
 
 global_sasl_init(1, 1, mysasl_cb);
 
-if (config_mupdate_server &&
-	(config_mupdate_config == IMAP_ENUM_MUPDATE_CONFIG_STANDARD) &&
-	!config_getstring(IMAPOPT_PROXYSERVERS)) {
-	/* proxy only -- talk directly to mupdate master */
-	r = mupdate_connect(config_mupdate_server, NULL, &mhandle, NULL);
-	if (r) {
-	syslog(LOG_ERR, "couldn't connect to MUPDATE server %s: %s",
-		   config_mupdate_server, error_message(r));
-	fatal("error connecting with MUPDATE server", EC_TEMPFAIL);
-	}
-}
-else {
-	dupelim = config_getswitch(IMAPOPT_DUPLICATESUPPRESSION);
+dupelim = config_getswitch(IMAPOPT_DUPLICATESUPPRESSION);
 
 #ifdef USE_SIEVE
-	mylmtp.addheaders = xzmalloc(2 * sizeof(struct addheader));
-	mylmtp.addheaders[0].name = "X-Sieve";
-	mylmtp.addheaders[0].body = SIEVE_VERSION;
+mylmtp.addheaders = xzmalloc(2 * sizeof(struct addheader));
+mylmtp.addheaders[0].name = "X-Sieve";
+mylmtp.addheaders[0].body = SIEVE_VERSION;
 
-	/* setup sieve support */
-	sieve_interp = setup_sieve();
+/* setup sieve support */
+sieve_interp = setup_sieve();
 #else
-	if (dupelim)
+if (dupelim)
 #endif
-	{
-	/* initialize duplicate delivery database */
-	if (duplicate_init(NULL, 0) != 0) {
-		fatal("lmtpd: unable to init duplicate delivery database",
-		  EC_SOFTWARE);
-	}
+{
+/* initialize duplicate delivery database */
+	if (duplicate_init(NULL, 0) != 0) {
+	fatal("lmtpd: unable to init duplicate delivery database",
+	   EC_SOFTWARE);
 	}
+}
 
-	/* so we can do mboxlist operations */
-	mboxlist_init(0);
-	mboxlist_open(NULL);
-
-	/* so we can do quota operations */
-	quotadb_init(0);
-	quotadb_open(NULL);
-
-	/* Initialize the annotatemore db (for sieve on shared mailboxes) */
-	annotatemore_init(0, NULL, NULL);
-	annotatemore_open(NULL);
+/* so we can do mboxlist operations */
+mboxlist_init(0);
+mboxlist_open(NULL);
 
-	/* setup for statuscache invalidation */
-	statuscache_open(NULL);
+/* so we can do quota operations */
+quotadb_init(0);
+quotadb_open(NULL);
 
-	/* setup for sending IMAP IDLE notifications */
-	idle_enabled();
-}
+/* Initialize the annotatemore db (for sieve on shared mailboxes) */
+annotatemore_init(0, NULL, NULL);
+annotatemore_open(NULL);
+
+/* setup for statuscache invalidation */
+statuscache_open(NULL);
+
+/* setup for sending IMAP IDLE notifications */
+idle_enabled();
+
 
 /* Set namespace */
 if ((r = mboxname_init_namespace(&lmtpd_namespace, 0)) != 0) {
@@ -288,32 +276,8 @@
 snmp_increment(TOTAL_CONNECTIONS, 1);
 snmp_increment(ACTIVE_CONNE

assertion failed after master mupdate received SIGSEGV

[Michael Menge]

Hi,

on the mupdate master the mupdate process was again killed by SIGSEGV
and on the attempt to restart the process, assertion failed.

Can someone help me understand whats going on?
Cyrus-imapd 2.4.17, murder in stadard config.

mu/mupdate[147469]: Internal error: assertion failed:  
cyrusdb_skiplist.c: 497: db && db->map_len && db->fname &&  
db->map_base && db-

is_open && db->lock_status
mu/master[1486]: service mupdate pid 147469 in READY state: terminated  
abnormally





M.MengeTel.: (49) 7071/29-70316
Universität Tübingen   Fax.: (49) 7071/29-5912
Zentrum für Datenverarbeitung  mail:  
michael.me...@zdv.uni-tuebingen.de

Wächterstraße 76
72074 Tübingen

smime.p7s
Description: S/MIME Signatur

frontend lmtp connections to mupdate master

[Michael Menge]

Hi,

last night we had a problem with our mupdate master not
working any more, received SIGSEGV, and the next mupdate had an
internal error. See logs below.
I had been under the impression that only changes in the mailbox db
would fail if the mupdate master was not working.

But also mail delivery was failing. The frontend lmtpd tried
to connect to the mupdate master, and didn't use the frontend
mailboxdb.

I was wondering whats the reasoning?
I think the many connections to the mupdate master i have seen,
have not been backend impad processes creating/renaming/deleting
folders but frontend lmtpd processes asking for information which is
available local.

Regards

   Michael Menge



Sep 25 22:15:12 mailserv01 mu/master[8348]: process 8665 exited,  
signaled to death by 11
Sep 25 22:15:12 mailserv01 mu/master[8348]: service mupdate pid 8665  
in READY state: terminated abnormally
Sep 25 22:15:13 mailserv01 mu/mupdate[46721]: DBERROR: reading  
/srv/cyrus-mu/db/skipstamp, assuming the worst: No such file or  
directory
Sep 25 22:15:19 mailserv01 mu/mupdate[46721]: skiplist: checkpointed  
/srv/cyrus-mu/mailboxes.db (459230 records, 47495540 bytes) in 2 seconds
Sep 25 22:15:20 mailserv01 mu/mupdate[46721]: Internal error:  
assertion failed: cyrusdb_skiplist.c: 497: db && db->map_len &&  
db->fname && db->map_base && db->is_open && db->lock_status
Sep 25 22:15:20 mailserv01 mu/master[8348]: service mupdate pid 46721  
in READY state: terminated abnormally
Sep 25 22:15:20 mailserv01 mu/mupdate[46912]: DBERROR: reading  
/srv/cyrus-mu/db/skipstamp, assuming the worst: No such file or  
directory
Sep 25 22:15:23 mailserv01 mu/mupdate[46912]: skiplist: checkpointed  
/srv/cyrus-mu/mailboxes.db (459230 records, 47495540 bytes) in 2 seconds
Sep 25 22:15:23 mailserv01 mu/master[8348]: process 46912 exited,  
signaled to death by 6
Sep 25 22:15:23 mailserv01 mu/master[8348]: service mupdate pid 46912  
in READY state: terminated abnormally
Sep 25 22:15:23 mailserv01 mu/mupdate[46936]: DBERROR: reading  
/srv/cyrus-mu/db/skipstamp, assuming the worst: No such file or  
directory
Sep 25 22:15:26 mailserv01 mu/mupdate[46936]: skiplist: checkpointed  
/srv/cyrus-mu/mailboxes.db (459230 records, 47495540 bytes) in 2 seconds




M.MengeTel.: (49) 7071/29-70316
Universität Tübingen   Fax.: (49) 7071/29-5912
Zentrum für Datenverarbeitung  mail:  
michael.me...@zdv.uni-tuebingen.de

Wächterstraße 76
72074 Tübingen

smime.p7s
Description: S/MIME Signatur

recover form unclean failover

[Michael Menge]

Hi all,

We are testing cyrus 2.4.17 replication in a murder setup.

We try to resolve the scenario of an unclean failover.

In the beginning Server A is the backend and replicates to server B.
In case of failure the backend IP is switched to Server B and the
Server A will restart with the IP of the replica.

If the failover is unclean there will be a rolling replication log file
on Server A.

We have noticed that cyrus catches some problems with split brain.
On the other hand, mailboxes created on server A that are not synced jet
to server B may get lost if the sync_client is started on server B.
Sync_client will send "APPLY UNMAILBOX mailboxname" to the sync_server  
on Server A.


The Article "FastMail storage architecture" on
https://www.fastmail.fm/help/technical/architecture.html indicates that
cyrus is able to recover from split-brain without data loss.

Are there any tools/scripts/recommended steps to fill this gap ;-)
@Bron, how is this scenario handled at fastmail?

Regards

   Michael Menge




M.MengeTel.: (49) 7071/29-70316
Universität Tübingen   Fax.: (49) 7071/29-5912
Zentrum für Datenverarbeitung  mail:  
michael.me...@zdv.uni-tuebingen.de

Wächterstraße 76
72074 Tübingen

smime.p7s
Description: S/MIME Signatur

Wrong message size in cyrus.index if mime boundary limit is hit?

[Michael Menge]

Hi,

i had some problems with syncing a mailbox (see  
http://permalink.gmane.org/gmane.mail.imap.cyrus/37513 for details).  
Recreating the cyrus.index file

with reconstruct resulted in the same wrong size in cyrus.index.
The message hat much over boundary_limit mime parts.

I suspect that the size in cyrus.index is only size of the parsed parts,
which would explain why the recreated cyrus.index had the same wrong
message size.

Is my assumption correct, or did I miss something?

As this will break replication, should we store the filesize in
cyrus.index in case we hit the boundary limit.


Regards

 Michael Menge






M.MengeTel.: (49) 7071/29-70316
Universität Tübingen   Fax.: (49) 7071/29-5912
Zentrum für Datenverarbeitung  mail:  
michael.me...@zdv.uni-tuebingen.de

Wächterstraße 76
72074 Tübingen

smime.p7s
Description: S/MIME Signatur

Re: Sync after replica restarted...

[Michael Menge]

Quoting Karl Pielorz :



Ok, that's probably because the replica was restarting. However now  
it's restarted - the master doesn't seem to be pushing any more data  
to the replica :(


In the 'sync' directory I have an ever growing 'log' file - and  
'log-33213' (which I presume is what was in use by the process that  
failed - i.e. that matches the PID above).


How do I restart replication?

In cyrus.conf I have:

"
...
syncclientcmd="/usr/local/cyrus/bin/sync_client -r"
...
"

Can I just do:

"
su cyrus
/usr/local/cyrus/bin/sync_client -r
"

To restart that process? - It's not listed in 'ps ax | grep sync_client'.



Bevor you run "sync_client -r" you should run

  sync_client -r -f /PATH_TO_SYNC_DIR/log-33213

to sync what was left form the old sync process and confirm that the exit
code is 0 (echo $?) than you can delete log-33213

The Sync_client will catch up if you miss syncing log-33213, but it can
lead to some errors if sync_client tries to sync some mailboxes that have
not been created on the replic.

Regards

Michael








M.MengeTel.: (49) 7071/29-70316
Universität Tübingen   Fax.: (49) 7071/29-5912
Zentrum für Datenverarbeitung  mail:  
michael.me...@zdv.uni-tuebingen.de

Wächterstraße 76
72074 Tübingen

Re: Quota and Sievescripts

[Michael Menge]

Hi,

Quoting Дилян Палаузов :


Hello,

the motivation for the patch is to reduce the number of bounces,  
that would be avoided, if a spam mail is not bounced, but discarded.




The spam mails are one but not the only usecase.
E.g users can redirect/forward some of ther mails without
keeping a copy. Without this enhanchmant there would be an
overquota bounce even if no mail would be stored in the overquota
folder.


On 16.04.2012 11:42, Michael Menge wrote:

Hi all,

some time ago I added an enhencement request
to only return IMAP_QUOTA_EXCEEDED if needed
https://bugzilla.cyrusimap.org/show_bug.cgi?id=3057

The request was postponed past 2.4.
As the releas of 2.5 is in near future I wanted to ask if
this patch is considerd for 2.5? Is there anything I could do to
help to include it in 2.5?

I didn't have time to look into the source of 2.4 or 2.5 to
check how much work has to be done to change the patch that it will
apply to 2.5.

Regards

Michael Menge






M.MengeTel.: (49) 7071/29-70316
Universität Tübingen   Fax.: (49) 7071/29-5912
Zentrum für Datenverarbeitung  mail:  
michael.me...@zdv.uni-tuebingen.de

Wächterstraße 76
72074 Tübingen

smime.p7s
Description: S/MIME Signatur

Quota and Sievescripts

[Michael Menge]

Hi all,

some time ago I added an enhencement request
to only return IMAP_QUOTA_EXCEEDED if needed
https://bugzilla.cyrusimap.org/show_bug.cgi?id=3057

The request was postponed past 2.4.
As the releas of 2.5 is in near future I wanted to ask if
this patch is considerd for 2.5? Is there anything I could do to
help to include it in 2.5?

I didn't have time to look into the source of 2.4 or 2.5 to
check how much work has to be done to change the patch that it will
apply to 2.5.

Regards

Michael Menge



M.MengeTel.: (49) 7071/29-70316
Universität Tübingen   Fax.: (49) 7071/29-5912
Zentrum für Datenverarbeitung  mail:  
michael.me...@zdv.uni-tuebingen.de

Wächterstraße 76
72074 Tübingen

smime.p7s
Description: S/MIME Signatur

Re: MESSAGE quota resource implemention

[Michael Menge]

Quoting Bron Gondwana :


Actually, really I'd like to create a new UNIQUEID - and store
all the files in paths based on uniqueid rather than on folder
name.  This would not only mean renames could be fast and
atomic, but that delayed delete would be fast.  The downside is
a more opaque filesystem layout.  Oh, another upside - file path
limitations don't exist so much any more.



How do you restore a folder from a filebased backup?
Or exclude some folders form backup (e.g. trash folders)?




M.MengeTel.: (49) 7071/29-70316
Universität Tübingen   Fax.: (49) 7071/29-5912
Zentrum für Datenverarbeitung  mail:  
michael.me...@zdv.uni-tuebingen.de

Wächterstraße 76
72074 Tübingen

smime.p7s
Description: S/MIME Signatur

Re: LIST shows notexisting folder under Shared Folders

[Michael Menge]

Replaying to myself,

Quoting Michael Menge :


Hi,

I try to debug a problem with horde/imp 5.0 (git version)
and cyrus-imapd 2.4.7. Imp is unable to create folders.
And shows the error Folder already exists.

telemetry logging shows

a LIST "" (Test23)
* LIST (\NonExistent) "/" "Shared Folders/Test23"
a OK Completed (0.000 secs)

which would indicate that a folder Test23 is existing as a
shared folder. But this folder does not exist.

b LIST "" (* "Other Users/*" "Shared Folders/*")
* LIST (\Noinferiors \HasChildren) "/" INBOX
* LIST () "/" test
b OK Completed (0.000 secs 2 calls)

is this related to http://bugzilla.cyrusimap.org/show_bug.cgi?id=3404?



As Bron commited a fix for this bug, I applied the patch
4701b866c5eca0592f07f94a3b747d980291fa9e which fixed the problem

Thanks




M.MengeTel.: (49) 7071/29-70316
Universität Tübingen   Fax.: (49) 7071/29-5912
Zentrum für Datenverarbeitung  mail:  
michael.me...@zdv.uni-tuebingen.de

Wächterstraße 76
72074 Tübingen

smime.p7s
Description: S/MIME Signatur

LIST shows notexisting folder under Shared Folders

[Michael Menge]

Hi,

I try to debug a problem with horde/imp 5.0 (git version)
and cyrus-imapd 2.4.7. Imp is unable to create folders.
And shows the error Folder already exists.

telemetry logging shows

a LIST "" (Test23)
* LIST (\NonExistent) "/" "Shared Folders/Test23"
a OK Completed (0.000 secs)

which would indicate that a folder Test23 is existing as a
shared folder. But this folder does not exist.

b LIST "" (* "Other Users/*" "Shared Folders/*")
* LIST (\Noinferiors \HasChildren) "/" INBOX
* LIST () "/" test
b OK Completed (0.000 secs 2 calls)

is this related to http://bugzilla.cyrusimap.org/show_bug.cgi?id=3404?

regards


 Michael Menge



M.MengeTel.: (49) 7071/29-70316
Universität Tübingen   Fax.: (49) 7071/29-5912
Zentrum für Datenverarbeitung  mail:  
michael.me...@zdv.uni-tuebingen.de

Wächterstraße 76
72074 Tübingen

smime.p7s
Description: S/MIME Signatur

Re: Selection of most fitting partition/backend upon account creation

[Michael Menge]

Quoting Bron Gondwana :


On Wed, Dec 15, 2010 at 03:31:00PM +0100, Michael Menge wrote:

I would like to see this included in the official cyrus.

It would be nice if the space reserved by quota could be
taken into account.


Oooh... interesting.  Yes, that does make sense.  I don't
know about you, but we over-supply quota by heaps.


Yes, i think this is common, so the "not reserved" disk
space can get negative. And the code must be able to hanlde
this, and the case where the "not reserved" disk space is 0.


We
have a "balancing task" that watches for servers filling
up and moves users automagically behind the scenes.

We use sync_move, which I'm planning to write into Cyrus
for the future as well - where it uses the replication
engine rather than the evil XFER :)  Much saner, and it
means you can sync everything FIRST and then just
replicate the diff once you actually shut down the
mailbox, so the downtime is much shorter.



Very nice






M.MengeTel.: (49) 7071/29-70316
Universität Tübingen   Fax.: (49) 7071/29-5912
Zentrum für Datenverarbeitung  mail:  
michael.me...@zdv.uni-tuebingen.de

Wächterstraße 76
72074 Tübingen

smime.p7s
Description: S/MIME Signatur

Re: Selection of most fitting partition/backend upon account creation

[Michael Menge]

Quoting Julien Coloos :


Hi,

In latest cyrus versions (beginning with 2.3.x branch), cyrus is
capable of selecting the most fitting partition when creating a new
account. To do so, the default partition has to be left unspecified,
and in this case cyrus selects the partition with the most free space.
Similarly, when creating an account from a murder proxy, it is
possible to let cyrus select the backend with the most free space.

Due to some of our clients needs, we had to rework those features to
handle other modes of selection (see below). We currently have code
for 2.3.16 version.
If you are interested, we could create a patch out of it, adapted for
2.4.5 version, and add an entry for it on bugzilla.


Here are some details on the changes we made:

Partitions/backend are managed using a new structure which contains
all valuable data. Those data are retrieved the first time it is
needed, and then cached (configuration allows to refresh data).

New configuration options manage partition/backend selection:
   - partition_mode: how most fitting partition is selected; value is one of
  - random: (pseudo-)random selection
  - freespace-most: partition with the most free space (KiB)
 -> same as current cyrus behaviour
  - freespace-percent-most: partition with the most free space (%)
  - freespace-percent-weighted: each partition is weighted
according to its free space (%)
 -> the more free space the partition has, the more chances it
has to be selected
  - freespace-percent-weighted-delta: each partition is weighted
according to its difference of free space (%) compared to the most
used partition
 -> the more the partition is lagging behind the most used
partition, the more chances it has to be selected
 -> actually the code is made so that even the most used
partition has a few chances to be selected, and those chances increase
when other partitions get closer
   - partition_mode_exclude: list of partitions to exclude from  
selection mode

   - partition_mode_weighted_usage_limit: limit of partition usage (%)
  -> if a partition is over that limit, it is automatically
excluded from selection mode
  -> if all partitions are over that limit, this feature is not  
used anymore

   - partition_mode_usage_reinit: for a given session, number of
"operations" (e.g. partition selection) for which partitions usage
data are cached
  -> useful for clients that massively create new mailboxes using
the same session

   - serverlist_mode: same as partition_mode, but used on proxy for
selecting most fitting backend
  - random: (pseudo-)random
  - freespace-most: backend with the most (total) free space (KiB)
 -> same as current cyrus behaviour
  - freespace-percent-most: backend whose partition has the most
free space (%)
 -> and not the backend with the most (total) free space (%);
the goal is to compare the most fitting partition of each backend
  - freespace-percent-weighted: same as for partition selection,
comparing the free space (%) of the least used partition of each
backend
 -> again the goal is to compare the most fitting partition of
each backend
  - freespace-percent-weighted-delta: same as for partition
selection, comparing the free space (%) of the least used partition of
each backend
 -> again the goal is to compare the most fitting partition of
each backend
   - serverlist_mode_weighted_usage_limit: same as
partition_mode_weighted_usage_limit
   - serverlist_mode_usage_reinit: same as partition_mode_usage_reinit


I would like to see this included in the official cyrus.

It would be nice if the space reserved by quota could be
taken into account.



M.MengeTel.: (49) 7071/29-70316
Universität Tübingen   Fax.: (49) 7071/29-5912
Zentrum für Datenverarbeitung  mail:  
michael.me...@zdv.uni-tuebingen.de

Wächterstraße 76
72074 Tübingen

smime.p7s
Description: S/MIME Signatur

Re: Zero byte appends

[Michael Menge]

Quoting Bron Gondwana :


Can anyone think of any good reason we should allow zero byte
files to be appended via IMAP?  Just that I'm currently going
over the reconstruct code - and that's one of the few places
where it actually changes things.


Some applications (e.g. Kolab) store other objects than Mail (e.g.  
Addressbooks, Calendar, Tasks or Notes) in IMAP folders. They might

create empty objects for some reason.



Reconstruct will always unlink a zero byte file and remove the
record - but IMAP APPEND will actually create one.  We should
align the behaviour one way or the other!

Bron.







M.MengeTel.: (49) 7071/29-70316
Universität Tübingen   Fax.: (49) 7071/29-5912
Zentrum für Datenverarbeitung  mail:  
michael.me...@zdv.uni-tuebingen.de

Wächterstraße 76
72074 Tübingen

smime.p7s
Description: S/MIME Cryptographic Signature

Re: CONDSTORE by default?

[Michael Menge]

Hi,

doc/readme.html in 2.3.14 still says:

Upgrade Caveats

This section reserved for WARNING WARNING WARNING comments.
Note that the replication protocol currently does not have the facility
to support the IMAP CONDSTORE extension (modification sequences). It is
recommended that you do not try to use both CONDSTORE and replication
at this time. The deficiencies in the replication protocol will be
fixed in version 2.3.9.

Is this problem fixed or is it still valid?



Quoting Ken Murchison :

There is a higher bookkeeping cost when only \Seen state changes  
because rather than just updating seen.db, we also have to tweak  
cyrus.index. The cost may not be enough to worry about, but since  
\Seen state is most likely the most frequently used flag (and most  
likely is changed separately from other flags), I didn't want to  
force CONDSTORE on people until we had a good handle on the  
performance hit.



Bron Gondwana wrote:

Hi Ken, everyone,

Can anyone think of a good reason why CONDSTORE isn't enabled
by default for everyone all the time?  Back when it wasn't
stable, fair enough - but I think it's pretty close now, and
am willing to put a bit more effort in to finishing the job...

Thunderbird 3 will support CONDSTORE to reduce the FLAGS traffic
quite considerably, and hopefully other clients will follow suit
if there is enough server support to be worth the effort.

Looking at the code, the "bookkeeping" cost is quite low, and
it always touches records that are being updated anyway, so the
writes will be combined in the same block most of the time.

Besides, it will simplify the code considerably to remove all
the tests for "CONDSTORE ENABLED" everywhere.

Bron ( looking at turning it on for all new mailboxes soon,
  and then enabling it for all the old ones )



--
Kenneth Murchison
Systems Programmer
Carnegie Mellon University






M.MengeTel.: (49) 7071/29-70316
Universität Tübingen   Fax.: (49) 7071/29-5912
Zentrum für Datenverarbeitung  mail:  
michael.me...@zdv.uni-tuebingen.de

Wächterstraße 76
72074 Tübingen

smime.p7s
Description: S/MIME krytographische Unterschrift