Re: Cyrus 3.0.8 running :) and feedback of the upgrade
Hi mates! I don't really know where the difference is, but in the dev environment, searches are hugely faster. I know, in dev is not the same as production in terms of muas traffic and so... but... we are talking in differences like from 5 seconds to perhaps 40-50 seconds and obviously when saying this... production is a non loaded env... I mean we have there 4000 accounts, but io is fine, cpu and mem are nice... os limits too there's no any bottleneck... The real difference between my dev env and prod one... is basically the way they were built. FOR DEV I created a copy of one production machine. A copy... for being able to be the most close as we could, from the prod env in this testing env. That prod env, was at 2.3. I then, connected to it 2.4-Cyrus root fs (with Cyrus 2.4). Tested this version and converted to 12 version some databases (just the ones for testing, the same ones as where I have seen this speed effect). Later I disconnected the 2.4-Cyrus root disk and connected a 3.0-Cyrus root disk. Did a reconstruct -r -V max, ctl_conversationsdb -z on-testing-used-mailboxes and ctl_conversationsdb -b on-testing-used-mailboxes. FOR PROD we connected to 2.3 server the 2.4-Cyrus root disk. Later we setup an empty slave with 3.0 and started a user by user, user mode replication there. Later a rolling one (with the file generated between last "all users, user by user sync" and the moment we started the rolling one) when almost all in the slave was very near to the present state of the master in that moment. This replication was done with Csync not IMAP. These too are the basic differences... could this give you some kind of clue?... I'll try to debug code too in order to see something Cheers! --- EGOITZ AURREKOETXEA Departamento de sistemas 944 209 470 Parque Tecnológico. Edificio 103 48170 Zamudio (Bizkaia) ego...@sarenet.es www.sarenet.es [1] Antes de imprimir este correo electrónico piense si es necesario hacerlo. El 13-02-2019 18:19, Egoitz Aurrekoetxea escribió: > thanks a lot mate! > > I'm doing checks... for comparing the previous testing env and live > production :) > > Cheers! > > --- > > EGOITZ AURREKOETXEA > Departamento de sistemas > 944 209 470 > Parque Tecnológico. Edificio 103 > 48170 Zamudio (Bizkaia) > ego...@sarenet.es > www.sarenet.es [1] > Antes de imprimir este correo electrónico piense si es necesario hacerlo. > > El 13-02-2019 17:58, Robert Stepanek escribió: > On Wed, Feb 13, 2019, at 2:24 PM, Egoitz Aurrekoetxea wrote: > > Or.. if using in imapd.conf "search_fuzzy_always: 1" isn't it?. > > Yes, that will instruct imapd to always use FUZZY search for IMAP SEARCH > commands [1]. If you use JMAP, it always use fuzzy search (and hence the > Xapian backend). > > [1] https://github.com/cyrusimap/cyrus-imapd/blob/master/imap/imapd.c#L6059 Cyrus Home Page: http://www.cyrusimap.org/ List Archives/Info: http://lists.andrew.cmu.edu/pipermail/info-cyrus/ To Unsubscribe: https://lists.andrew.cmu.edu/mailman/listinfo/info-cyrus Links: -- [1] http://www.sarenet.es Cyrus Home Page: http://www.cyrusimap.org/ List Archives/Info: http://lists.andrew.cmu.edu/pipermail/info-cyrus/ To Unsubscribe: https://lists.andrew.cmu.edu/mailman/listinfo/info-cyrus
Re: Cyrus 3.0.8 running :) and feedback of the upgrade
thanks a lot mate! I'm doing checks... for comparing the previous testing env and live production :) Cheers! --- EGOITZ AURREKOETXEA Departamento de sistemas 944 209 470 Parque Tecnológico. Edificio 103 48170 Zamudio (Bizkaia) ego...@sarenet.es www.sarenet.es [1] Antes de imprimir este correo electrónico piense si es necesario hacerlo. El 13-02-2019 17:58, Robert Stepanek escribió: > On Wed, Feb 13, 2019, at 2:24 PM, Egoitz Aurrekoetxea wrote: > >> Or.. if using in imapd.conf "search_fuzzy_always: 1" isn't it?. > > Yes, that will instruct imapd to always use FUZZY search for IMAP SEARCH > commands [1]. If you use JMAP, it always use fuzzy search (and hence the > Xapian backend). > > [1] https://github.com/cyrusimap/cyrus-imapd/blob/master/imap/imapd.c#L6059 Links: -- [1] http://www.sarenet.es Cyrus Home Page: http://www.cyrusimap.org/ List Archives/Info: http://lists.andrew.cmu.edu/pipermail/info-cyrus/ To Unsubscribe: https://lists.andrew.cmu.edu/mailman/listinfo/info-cyrus
Re: Cyrus 3.0.8 running :) and feedback of the upgrade
On Wed, Feb 13, 2019, at 2:24 PM, Egoitz Aurrekoetxea wrote: > Or.. if using in imapd.conf "search_fuzzy_always: 1" isn't it?. Yes, that will instruct imapd to always use FUZZY search for IMAP SEARCH commands [1]. If you use JMAP, it always use fuzzy search (and hence the Xapian backend). [1]https://github.com/cyrusimap/cyrus-imapd/blob/master/imap/imapd.c#L6059 Cyrus Home Page: http://www.cyrusimap.org/ List Archives/Info: http://lists.andrew.cmu.edu/pipermail/info-cyrus/ To Unsubscribe: https://lists.andrew.cmu.edu/mailman/listinfo/info-cyrus
Re: Cyrus 3.0.8 running :) and feedback of the upgrade
Or.. if using in imapd.conf "search_fuzzy_always: 1" isn't it?. If a search returns results... I assume is going through it... isn't it?. else.. I assume nothing would be found... I really attached too the configs (in url format of pastebin), https://pastebin.com/CtCEedty just for... the cause you could see something missing there... which by the way... I don't think something could be missed but I'm new to Xapian :) :) although have been running Cyrus more than ten years ago :) :) ... Cheers! --- EGOITZ AURREKOETXEA Departamento de sistemas 944 209 470 Parque Tecnológico. Edificio 103 48170 Zamudio (Bizkaia) ego...@sarenet.es www.sarenet.es [1] Antes de imprimir este correo electrónico piense si es necesario hacerlo. El 13-02-2019 13:21, Robert Stepanek escribió: > Hi Egoitz, > > On Wed, Feb 13, 2019, at 1:10 PM, Egoitz Aurrekoetxea wrote: > >> Xapian version is xapian-core-1.4.9_1 (FreeBSD port). Could I see by some >> debbuging log Xapian is working in the searchs in order to know, that can't >> be faster and that all is ok?. At the moment I have tree tiers. But I'm just >> with the default one in almost all mailboxes > > If you use FUZZY search in your search expression then it must go through > Xapian. > > Cheers, > Robert > > > Cyrus Home Page: http://www.cyrusimap.org/ > List Archives/Info: http://lists.andrew.cmu.edu/pipermail/info-cyrus/ > To Unsubscribe: > https://lists.andrew.cmu.edu/mailman/listinfo/info-cyrus Links: -- [1] http://www.sarenet.es Cyrus Home Page: http://www.cyrusimap.org/ List Archives/Info: http://lists.andrew.cmu.edu/pipermail/info-cyrus/ To Unsubscribe: https://lists.andrew.cmu.edu/mailman/listinfo/info-cyrus
Re: Cyrus 3.0.8 running :) and feedback of the upgrade
Hi Egoitz, On Wed, Feb 13, 2019, at 1:10 PM, Egoitz Aurrekoetxea wrote: > Xapian version is xapian-core-1.4.9_1 (FreeBSD port). Could I see by > some debbuging log Xapian is working in the searchs in order to know, > that can't be faster and that all is ok?. At the moment I have tree > tiers. But I'm just with the default one in almost all mailboxes If you use FUZZY search in your search expression then it must go through Xapian. Cheers, Robert Cyrus Home Page: http://www.cyrusimap.org/ List Archives/Info: http://lists.andrew.cmu.edu/pipermail/info-cyrus/ To Unsubscribe: https://lists.andrew.cmu.edu/mailman/listinfo/info-cyrus
Re: disk space used by a mailbox without expunged
On 13/02/2019 13:08, Patrick Boutilier wrote: On 2/13/19 7:01 AM, Eric Luyten wrote: On 13/02/2019 09:17, Michael Menge wrote: Hi Marcus, Quoting Marcus Schopen : Hi, is there a way to count the disk space used by a mailbox without expunged messages? mbexamine user/LoginID | grep Size on older cyrus versions (2.3 and 2.4) mbexamine did examine all subfolders as well in 3.0 only the info for the given folder is shown O dear. We (2.3 server, upgrading this year) use the subfolder size information extensively in our management procedures. Use /* to get the subfolders. Patrick, So there is only a command syntax and no functionality change. We can handle that. Eric. Cyrus Home Page: http://www.cyrusimap.org/ List Archives/Info: http://lists.andrew.cmu.edu/pipermail/info-cyrus/ To Unsubscribe: https://lists.andrew.cmu.edu/mailman/listinfo/info-cyrus
Cyrus 3.0.8 running :) and feedback of the upgrade
Hi mates! This has been the great day. We have upgraded a server pair (master/slave). Almost all has gone fine, apart from the fact that we have two extrange issues : - Some users had found their folders unsubscribed and seems the same too, lost them Sieve filters. Perhaps due to autocreate or xlist?? which we weren't previously using¿? - I'm not really sure, if Xapian is "actively" working. We have upgraded from 2.4 Cyrus to 3.0.8 with the replication method. First has been finally fixed with a couple of scripts. The second one... I paste the master/slave config here : https://pastebin.com/CtCEedty I think it's all ok... and by the way log is saying : Feb 13 13:07:19 masterserver squatter[13446]: Xapian committed 1 updates in 0.146300 sec Feb 13 13:07:19 masterserver squatter[13446]: Xapian committed 3 updates in 0.075679 sec Feb 13 13:07:21 masterserver squatter[13446]: Xapian committed 36 updates in 1.219806 sec Feb 13 13:07:22 masterserver squatter[13446]: Xapian committed 1 updates in 0.256899 sec Feb 13 13:07:22 masterserver squatter[13446]: Xapian committed 5 updates in 0.165767 sec Feb 13 13:07:22 masterserver squatter[13446]: Xapian committed 1 updates in 0.252658 sec Feb 13 13:07:22 masterserver squatter[13446]: Xapian committed 4 updates in 0.083638 sec Feb 13 13:07:24 masterserver squatter[13446]: Xapian committed 4 updates in 0.952737 sec Xapian version is xapian-core-1.4.9_1 (FreeBSD port). Could I see by some debbuging log Xapian is working in the searchs in order to know, that can't be faster and that all is ok?. At the moment I have tree tiers. But I'm just with the default one in almost all mailboxes. Thank you so much mates :) :) (as always) Cheers! -- EGOITZ AURREKOETXEA Departamento de sistemas 944 209 470 Parque Tecnológico. Edificio 103 48170 Zamudio (Bizkaia) ego...@sarenet.es www.sarenet.es [1] Antes de imprimir este correo electrónico piense si es necesario hacerlo. Links: -- [1] http://www.sarenet.es Cyrus Home Page: http://www.cyrusimap.org/ List Archives/Info: http://lists.andrew.cmu.edu/pipermail/info-cyrus/ To Unsubscribe: https://lists.andrew.cmu.edu/mailman/listinfo/info-cyrus
Re: disk space used by a mailbox without expunged
On 2/13/19 7:01 AM, Eric Luyten wrote: On 13/02/2019 09:17, Michael Menge wrote: Hi Marcus, Quoting Marcus Schopen : Hi, is there a way to count the disk space used by a mailbox without expunged messages? mbexamine user/LoginID | grep Size on older cyrus versions (2.3 and 2.4) mbexamine did examine all subfolders as well in 3.0 only the info for the given folder is shown O dear. We (2.3 server, upgrading this year) use the subfolder size information extensively in our management procedures. Use /* to get the subfolders. Such as: /usr/local/cyrus/sbin/mbexamine user/testuser|grep 'Mailbox Size' Number of Messages: 21 Mailbox Size: 332856 bytes Annotations Size: 0 bytes /usr/local/cyrus/sbin/mbexamine user/testuser/*|grep 'Mailbox Size' Number of Messages: 61 Mailbox Size: 1578340 bytes Annotations Size: 0 bytes Number of Messages: 1 Mailbox Size: 22909 bytes Annotations Size: 0 bytes Number of Messages: 0 Mailbox Size: 0 bytes Annotations Size: 0 bytes Number of Messages: 1573 Mailbox Size: 50237802 bytes Annotations Size: 0 bytes Number of Messages: 41 Mailbox Size: 639825 bytes Annotations Size: 0 bytes Number of Messages: 0 Mailbox Size: 0 bytes Annotations Size: 0 bytes Eric Luyten. Cyrus Home Page: http://www.cyrusimap.org/ List Archives/Info: http://lists.andrew.cmu.edu/pipermail/info-cyrus/ To Unsubscribe: https://lists.andrew.cmu.edu/mailman/listinfo/info-cyrus <> Cyrus Home Page: http://www.cyrusimap.org/ List Archives/Info: http://lists.andrew.cmu.edu/pipermail/info-cyrus/ To Unsubscribe: https://lists.andrew.cmu.edu/mailman/listinfo/info-cyrus
Re: disk space used by a mailbox without expunged
On 13/02/2019 09:17, Michael Menge wrote: Hi Marcus, Quoting Marcus Schopen : Hi, is there a way to count the disk space used by a mailbox without expunged messages? mbexamine user/LoginID | grep Size on older cyrus versions (2.3 and 2.4) mbexamine did examine all subfolders as well in 3.0 only the info for the given folder is shown O dear. We (2.3 server, upgrading this year) use the subfolder size information extensively in our management procedures. Eric Luyten. Cyrus Home Page: http://www.cyrusimap.org/ List Archives/Info: http://lists.andrew.cmu.edu/pipermail/info-cyrus/ To Unsubscribe: https://lists.andrew.cmu.edu/mailman/listinfo/info-cyrus
Re: another problem with conversations db
On Wed, Feb 13, 2019, at 00:39, Michael Menge wrote:
> Hi Bron,
>
> sorry, i had to rearrange some quotes to put them my answers in a more
> meaningful order.
>
>
> Quoting Bron Gondwana :
>
> >> >> The file was already at
> >> /srv/cyrus-hdd-be/archive/ssd-part/L/user//2185.
>
> I was able to fix these problems with reconstruct, and the didn't
> reappear till now.
> Also there where other accounts which had IOERRORS regarding the
> conversation db,
> with no cyr_expire archive errors, so i believe that these problems
> are not related.
>
> I tried rebuilding the conversation db for the accounts with errors,
> but some other
> accounts will show up with errors some time later. I counldn't find a
> some thing in
> common jet.
OK. That's hard to disagnore from remotely :(
>
> >> >> > Anyway, I don't think that would break anything.
> >> >> >
> >> >> > metapartition-ssd: /srv/cyrus-ssd-be/meta/ssd-part
> >> >> > metapartition_files: header index cache expunge squat annotations
> >> >> > lock dav archivecache
> >> >> >
> >> >> > Ooh, I haven't tested having cache and archivecache on the same
> >> >> > location. That's really interesting. Again, I'd be in favour of
> >> >> > separation here, give them different paths. That might be tricky
> >> >> > with ssd though, the way this is laid out. I assume you have some
> >> >> > kind of symlink farm going on?
> >> >> >
> >> >>
> >> >> I didn't know that there could be a problem with cache and archivecache.
> >> >> At the time we decided on the configuration for cyrus 3.0 I looked at
> >> >> the
> >> >> imapd.conf man page and for metapartition_files decided that I want all
> >> >> meta files on the ssd storage. There was no indication in the man page
> >> >> that there could be a problem.
> >> >
> >> > Fair. I'd have to test that to see if it works correctly. I would
> >> > hope so, but I haven't tested that configuration. This is the
> >> > downside with having lots of different ways to do things!
> >> >
> >> >> How do I separate location of archivecache from the other
> >> >> metapartition path?
> >> >> And fix the cache and archivecache files?
> >> >
> >> > This I don't know a good answer for. I will test if having the same
> >> > path for cache and archivecache could fail. I THINK that I made the
> >> > code safe for it, but I'm not sure that it's been tested.
> >> >
> >> >> No there is no sysmlink farm. We have mounted different iSCSI volumes to
> >> >> /srv/cyrus-ssd-be, /srv/cyrus-hdd-be and /srv/cyrus-be
> >> >
> >> > Right. That makes sense.
>
> Did you have time to look into the cache/archivecache situation jet?
Yes, I've looked at the code!
in mailbox_archive():
/* got a new cache record to write */
if (differentcache)
{
dirtycache = 1;
copyrecord.cache_offset = 0;
if (mailbox_append_cache(mailbox, ))
continue;
}
And the code for differentcache is:
char *spoolcache = xstrdup(mailbox_meta_fname(mailbox, META_CACHE));
char *archivecache = xstrdup(mailbox_meta_fname(mailbox, META_ARCHIVECACHE));
int differentcache = strcmp(spoolcache, archivecache);
So it looks like the answer is cache/archivecache is fine. It is not your
problem.
>
> >> > Right! I do wonder if there are some bugs in 3.0.x which are fixed
> >> > on master around delivery to archive partition. We definitely had
> >> > bugs on master, but I thought they were newly introduced on master
> >> > as well, which is why the fixes weren't backported. But if you're
> >> > having files be in the wrong location, maybe there are bugs on 3.0.x
> >> > as well.
>
> Are all fixes from master backported to 3.0?
Unfortunately it's hard to tell, because many of the fixes on master are fixes
to bugs that were only introduced on master, and some bugs on 3.0 we just say
"the fix is so invasive that it's basically just backporting 90% of master,
which is pointless for a stable release".
> Is the new Commit "I will try your new commits regarding CID" related to the
> "IOERROR: conversations_audit on load:" and "IOERROR:
> conversations_audit on store"?
Shouldn't be. It just means we store the G keys regardless of whether the
record has a CID.
> I will try your new commits in the next days on my test servers to sea
> if the fix
> the endless loop in ctl_conversationsdb I have seen for some accounts.
I guess one more question - are you running the most recent index version?
(reconstruct -V max)
> Quoting myself (Re: prblems rebuilding conversations db) Jan 24, 2019
>
> > The program loops in build_cid_cb (imap/ctl_conversationsdb.c:189)
> >
> > For the problematic mailbox that I tested, for every message
> > record->cid was NULLCONVERSATION, so mailbox_cacherecord,
> > message_update_conversations and mailbox_rewrite_index_record
> > where called, each returned 0.
> >
> > After iterating trough all messages in the mailbox count was > 0, and r==0,
> > so the while condition (!r && count) was true for the next run.
> > At this point record->cid was still
Re: disk space used by a mailbox without expunged
Hi Marcus, Quoting Marcus Schopen : Hi, is there a way to count the disk space used by a mailbox without expunged messages? mbexamine user/LoginID | grep Size on older cyrus versions (2.3 and 2.4) mbexamine did examine all subfolders as well in 3.0 only the info for the given folder is shown M.MengeTel.: (49) 7071/29-70316 Universität Tübingen Fax.: (49) 7071/29-5912 Zentrum für Datenverarbeitung mail: michael.me...@zdv.uni-tuebingen.de Wächterstraße 76 72074 Tübingen Cyrus Home Page: http://www.cyrusimap.org/ List Archives/Info: http://lists.andrew.cmu.edu/pipermail/info-cyrus/ To Unsubscribe: https://lists.andrew.cmu.edu/mailman/listinfo/info-cyrus
disk space used by a mailbox without expunged
Hi, is there a way to count the disk space used by a mailbox without expunged messages? Ciao M. Cyrus Home Page: http://www.cyrusimap.org/ List Archives/Info: http://lists.andrew.cmu.edu/pipermail/info-cyrus/ To Unsubscribe: https://lists.andrew.cmu.edu/mailman/listinfo/info-cyrus
