Re: squatter segfaulting

[Paolo Cravero]

Hello.

> Vanilla Cyrus, no Xapian. I don't know how to run a debug-enable, I only know 
> the basics :/
> 
> ii  cyrus-imapd   2.5.10-3+deb9u1
> amd64Cyrus mail system - IMAP support
> 

There seems to be an open issue: 
https://bugs.debian.org/cgi-bin/bugreport.cgi?bug=871921

I've seen squatter segfaulting when the index file got too large (in an older 
Cyrus version). Can you check how large it is? How many messages are in the 
mailbox/folder causing the crash?

How about system's ulimit? Can the cyrus user open enough files?

Paolo

Cyrus Home Page: http://www.cyrusimap.org/
List Archives/Info: http://lists.andrew.cmu.edu/pipermail/info-cyrus/
To Unsubscribe:
https://lists.andrew.cmu.edu/mailman/listinfo/info-cyrus

Re: squatter segfaulting

[Heiler Bemerguy]

  
  
Hi,

Vanilla Cyrus, no Xapian. I don't know how to run a debug-enable,
  I only know the basics :/
ii  cyrus-imapd  
  2.5.10-3+deb9u1    amd64    Cyrus mail system -
  IMAP support


Best Regards,
  Heiler Bemerguy

Em 06/02/2020 09:54, Robert Stepanek escreveu:

  
  
  
  On Wed, Feb 5, 2020, at 9:36 PM, Heiler Bemerguy wrote:
  
  
Is this normal ?

  
  
  
  
Absolutely not. Which Cyrus version are you running? Are
  you using the Xapian search backend? Would you be able to run
  this with a debug-enabled build and look at the core dump?


  
  

Cheers,

Robert



  
  
  
  
  
  
Cyrus Home Page: http://www.cyrusimap.org/
List Archives/Info: http://lists.andrew.cmu.edu/pipermail/info-cyrus/
To Unsubscribe:
https://lists.andrew.cmu.edu/mailman/listinfo/info-cyrus

  


Cyrus Home Page: http://www.cyrusimap.org/
List Archives/Info: http://lists.andrew.cmu.edu/pipermail/info-cyrus/
To Unsubscribe:
https://lists.andrew.cmu.edu/mailman/listinfo/info-cyrus

Re: squatter segfaulting

[Robert Stepanek]

On Wed, Feb 5, 2020, at 9:36 PM, Heiler Bemerguy wrote:
> Is this normal ?

Absolutely not. Which Cyrus version are you running? Are you using the Xapian 
search backend? Would you be able to run this with a debug-enabled build and 
look at the core dump?

Cheers,
Robert


Cyrus Home Page: http://www.cyrusimap.org/
List Archives/Info: http://lists.andrew.cmu.edu/pipermail/info-cyrus/
To Unsubscribe:
https://lists.andrew.cmu.edu/mailman/listinfo/info-cyrus

squatter segfaulting

[Heiler Bemerguy]

Is this normal ?

[2191259.566831] squatter[28914]: segfault at 7fa584008008 ip 
563e2b9b04d9 sp 7ffe9fed4920 error 6 in squatter[563e2b9ab000+7000]
[2199792.860105] squatter[32985]: segfault at 7f3b83577008 ip 
55fa60ec64d9 sp 7ffe6c6c1140 error 6 in squatter[55fa60ec1000+7000]
[2207827.970904] squatter[38118]: segfault at 7f3d32a0c008 ip 
5612cbb614d9 sp 7ffe71a88f70 error 6 in squatter[5612cbb5c000+7000]
[2214224.052412] squatter[42190]: segfault at 7fd324b9d008 ip 
55fb547604d9 sp 7fff69321470 error 6 in squatter[55fb5475b000+7000]
[2215084.555691] squatter[41867]: segfault at 7f330be0e008 ip 
55a0e4f974d9 sp 7ffe4863a620 error 6 in squatter[55a0e4f92000+7000]
[2219506.071396] squatter[45617]: segfault at 7f1611096008 ip 
55d868dc74d9 sp 7ffcc77dcae0 error 6 in squatter[55d868dc2000+7000]
[2228006.707228] squatter[49796]: segfault at 7f94446d0008 ip 
55b1187cf4d9 sp 7ffc4bdd8b40 error 6 in squatter[55b1187ca000+7000]
[2233416.966628] squatter[69504]: segfault at 7f4d2233e008 ip 
55c3d36d24d9 sp 7ffd34afb080 error 6 in squatter[55c3d36cd000+7000]


--
Atenciosamente,

Heiler Bemerguy - CINBESA
Analista de Redes, Wi-Fi,
Virtualização e Serviços Internet
(55) 91 98151-4894


Cyrus Home Page: http://www.cyrusimap.org/
List Archives/Info: http://lists.andrew.cmu.edu/pipermail/info-cyrus/
To Unsubscribe:
https://lists.andrew.cmu.edu/mailman/listinfo/info-cyrus

Re: Question about squatter for Xapian

[Egoitz Aurrekoetxea via Info-cyrus]

Hi!!

Thank you so much Rob!!

I will launch it this weekend :) :)

Cheers!

> El 23 ene 2020, a las 23:04, Rob N ★  escribió:
> 
> On Fri, 24 Jan 2020, at 4:38 AM, ego...@sarenet.es <mailto:ego...@sarenet.es> 
> wrote:
>> - Does it regenerate all mailboxes indexes?. Just the non-indexed emails?. I 
>> assume it should be extremely slow… so could this be launched?. Could you 
>> advise me please, if another way is preferred? 
> 
> Normally, just the non-indexed emails.
> 
> squatter -i (incremental) should be all you need to fill the gaps in your 
> index.
> 
> Obviously how long it takes depends on how much mail has arrived and how good 
> your disks are, but for 12 hours worth I wouldn't expect more than a couple 
> of hours to fill the gaps.
> 
>> - I assume not, but as we move records between Xapian tiers nightly… if the 
>> Squatter launched by me, by hand (for those non indexed emails), runs at the 
>> same time as this between tiers movement of records or at the same time too 
>> as the rolling mode squatter (-R) could one squatter process interfere in 
>> the job of the other instance of squatter?.
> 
> It's ok to run them all at the same time. Cyrus has appropriate locks to make 
> sure that Xapian updates and repacks don't get in each others' way.
> 
> Rob N.
> 
> Cyrus Home Page: http://www.cyrusimap.org/ <http://www.cyrusimap.org/>
> List Archives/Info: http://lists.andrew.cmu.edu/pipermail/info-cyrus/ 
> <http://lists.andrew.cmu.edu/pipermail/info-cyrus/>
> To Unsubscribe:
> https://lists.andrew.cmu.edu/mailman/listinfo/info-cyrus 
> <https://lists.andrew.cmu.edu/mailman/listinfo/info-cyrus>

Cyrus Home Page: http://www.cyrusimap.org/
List Archives/Info: http://lists.andrew.cmu.edu/pipermail/info-cyrus/
To Unsubscribe:
https://lists.andrew.cmu.edu/mailman/listinfo/info-cyrus

Re: Question about squatter for Xapian

[Rob N ★]

On Fri, 24 Jan 2020, at 4:38 AM, ego...@sarenet.es wrote:
> - Does it regenerate all mailboxes indexes?. Just the non-indexed emails?. I 
> assume it should be extremely slow… so could this be launched?. Could you 
> advise me please, if another way is preferred? 

Normally, just the non-indexed emails.

squatter -i (incremental) should be all you need to fill the gaps in your index.

Obviously how long it takes depends on how much mail has arrived and how good 
your disks are, but for 12 hours worth I wouldn't expect more than a couple of 
hours to fill the gaps.

> - I assume not, but as we move records between Xapian tiers nightly… if the 
> Squatter launched by me, by hand (for those non indexed emails), runs at the 
> same time as this between tiers movement of records or at the same time too 
> as the rolling mode squatter (-R) could one squatter process interfere in the 
> job of the other instance of squatter?.

It's ok to run them all at the same time. Cyrus has appropriate locks to make 
sure that Xapian updates and repacks don't get in each others' way.

Rob N.
Cyrus Home Page: http://www.cyrusimap.org/
List Archives/Info: http://lists.andrew.cmu.edu/pipermail/info-cyrus/
To Unsubscribe:
https://lists.andrew.cmu.edu/mailman/listinfo/info-cyrus

Question about squatter for Xapian

[ego...@sarenet.es]

Good morning,

We have mailboxes indexed with Xapian. We run for that Squatter in rolling mode 
(-R) so that a log can create new index records while new mail enters.. is 
removed and so… Nightly too move records between tiers and so… 

In an emergency on one mailbox server, we moved traffic to what was the slave 
server, and from that moment obviously the master… but we did a mistake, we 
forgot setting in the new slave "sync_log: true" so no squatter rolling log was 
generated. We noted about it 12 hours later. As we have "search_fuzzy_always: 
1” set all searches go through Xapian, so no indexed mail won’t never appear in 
the searchs run by the users. I think we could run squatter for indexing non 
indexed emails, but I was wondering : 

- Does it regenerate all mailboxes indexes?. Just the non-indexed emails?. I 
assume it should be extremely slow… so could this be launched?. Could you 
advise me please, if another way is preferred? 
- I assume not, but as we move records between Xapian tiers nightly… if the 
Squatter launched by me, by hand (for those non indexed emails), runs at the 
same time as this between tiers movement of records or at the same time too as 
the rolling mode squatter (-R) could one squatter process interfere in the job 
of the other instance of squatter?.

Thank you so much,
Best regards,

Cyrus Home Page: http://www.cyrusimap.org/
List Archives/Info: http://lists.andrew.cmu.edu/pipermail/info-cyrus/
To Unsubscribe:
https://lists.andrew.cmu.edu/mailman/listinfo/info-cyrus

Re: Squatter segfault on big mailbox

[Gabriele Bulfon]

Tried looking into the source of squatter and verbosing on a specific folder 
that always shows the problem.
Here is how it ends:
 
Processing index character 128, 9 total words, temp file size is 41
Processing index character 130, 8 total words, temp file size is 52
Processing index character 147, 2 total words, temp file size is 12
Processing index character 152, 2 total words, temp file size is 12
Processing index character 153, 5 total words, temp file size is 26
Processing index character 172, 10 total words, temp file size is 66
Processing index character 176, 72 total words, temp file size is 498
Processing index character 191, 417128205 total words, temp file size is 
1251444163
Closing index: Not enough space
 
I will look and change the source to give more info on the backtrace of the 
error, but maybe someone here alreay has any idea?
 
Gabriele
 
 
Sonicle S.r.l. 
: 
http://www.sonicle.com
Music: 
http://www.gabrielebulfon.com
Quantum Mechanics : 
http://www.cdbaby.com/cd/gabrielebulfon
Da:
Gabriele Bulfon
A:
Vladislav Kurz
Mark
info-cyrus
Data:
17 dicembre 2019 7.26.35 CET
Oggetto:
Re: Squatter segfault on big mailbox
 
Hello, we're having the same problem with Cyrus 2.5.11 on XStreamOS/illumos
Any idea how to fix this?
Same problem both on 32 bit large files build, and 64 bit.
Looks like the Debian issue has no solution.
Can't believe this has never been addressed.
 
Gabriele
 
 
Sonicle S.r.l. 
: 
http://www.sonicle.com
Music: 
http://www.gabrielebulfon.com
Quantum Mechanics : 
http://www.cdbaby.com/cd/gabrielebulfon
--
Da: Vladislav Kurz
A: Mark
info-cyrus
Data: 11 ottobre 2017 9.37.21 CEST
Oggetto: Re: Squatter segfault on big mailbox
On 10/11/17 07:37, Mark wrote:
On 2017-10-10 16:05, Vladislav Kurz wrote:
Hello everyone,
we have recently migrated our mail server from cyrus 2.2 to 2.5 (both
debian packages).
We have a problem that squatter segfaults on some quite big mailboxes.
I had to add -i (--incremental) to at least index new mails. But users
cannot search in old mails (before migration). I used strace to find out
if it is some particular mail, but it parsed everything fine, and failed
when dealing with some of the many temporary files squat.
The problematic mailboxes have just inbox10 GB (and15 GB with all
subfolders) and2 emails in inbox.
Has anyone else hit similar problem and solved it somehow?
I'm not sure about fixes, but there are some reports of similar
problems.  Have a look at the thread with the subject "Cyrus 2.5.10 IMAP
search" from the beginning of June this year. 
Apparently the format of the squatter database changed between 2.4 and
2.5, and some people (at least) have seen a 10x increase in the size of
the squatter database for the same mailbox.  For systems that only allow
2GB mmap, this can be a problem.
I'm not sure if this increase in size is a bug or a feature.
Mark
Hello Mark,
Thanks for the hint about old discussion. Unfortunately, there was no
solution to this issue mentioned. I found out that this was already
reported to debian, I wonder whetrher the maintainer forwarded the bug
report upstream.
https://bugs.debian.org/cgi-bin/bugreport.cgi?bug=871921
This is on 64-bit Linux, so mmap should not be limited to 2 GB.
And I also have noticed the extreme increase in squat file sizes.
--
Best Regards
Vladislav Kurz

Cyrus Home Page: http://www.cyrusimap.org/
List Archives/Info: http://lists.andrew.cmu.edu/pipermail/info-cyrus/
To Unsubscribe:
https://lists.andrew.cmu.edu/mailman/listinfo/info-cyrus
Cyrus Home Page: http://www.cyrusimap.org/List Archives/Info: 
http://lists.andrew.cmu.edu/pipermail/info-cyrus/To 
Unsubscribe:https://lists.andrew.cmu.edu/mailman/listinfo/info-cyrus

Cyrus Home Page: http://www.cyrusimap.org/
List Archives/Info: http://lists.andrew.cmu.edu/pipermail/info-cyrus/
To Unsubscribe:
https://lists.andrew.cmu.edu/mailman/listinfo/info-cyrus

Re: Squatter segfault on big mailbox

[Gabriele Bulfon]

Hello, we're having the same problem with Cyrus 2.5.11 on XStreamOS/illumos
Any idea how to fix this?
Same problem both on 32 bit large files build, and 64 bit.
Looks like the Debian issue has no solution.
Can't believe this has never been addressed.
 
Gabriele
 
 
Sonicle S.r.l. 
: 
http://www.sonicle.com
Music: 
http://www.gabrielebulfon.com
Quantum Mechanics : 
http://www.cdbaby.com/cd/gabrielebulfon
--
Da: Vladislav Kurz
A: Mark
info-cyrus
Data: 11 ottobre 2017 9.37.21 CEST
Oggetto: Re: Squatter segfault on big mailbox
On 10/11/17 07:37, Mark wrote:
On 2017-10-10 16:05, Vladislav Kurz wrote:
Hello everyone,
we have recently migrated our mail server from cyrus 2.2 to 2.5 (both
debian packages).
We have a problem that squatter segfaults on some quite big mailboxes.
I had to add -i (--incremental) to at least index new mails. But users
cannot search in old mails (before migration). I used strace to find out
if it is some particular mail, but it parsed everything fine, and failed
when dealing with some of the many temporary files squat.
The problematic mailboxes have just inbox10 GB (and15 GB with all
subfolders) and2 emails in inbox.
Has anyone else hit similar problem and solved it somehow?
I'm not sure about fixes, but there are some reports of similar
problems.  Have a look at the thread with the subject "Cyrus 2.5.10 IMAP
search" from the beginning of June this year. 
Apparently the format of the squatter database changed between 2.4 and
2.5, and some people (at least) have seen a 10x increase in the size of
the squatter database for the same mailbox.  For systems that only allow
2GB mmap, this can be a problem.
I'm not sure if this increase in size is a bug or a feature.
Mark
Hello Mark,
Thanks for the hint about old discussion. Unfortunately, there was no
solution to this issue mentioned. I found out that this was already
reported to debian, I wonder whetrher the maintainer forwarded the bug
report upstream.
https://bugs.debian.org/cgi-bin/bugreport.cgi?bug=871921
This is on 64-bit Linux, so mmap should not be limited to 2 GB.
And I also have noticed the extreme increase in squat file sizes.
--
Best Regards
Vladislav Kurz

Cyrus Home Page: http://www.cyrusimap.org/
List Archives/Info: http://lists.andrew.cmu.edu/pipermail/info-cyrus/
To Unsubscribe:
https://lists.andrew.cmu.edu/mailman/listinfo/info-cyrus

Cyrus Home Page: http://www.cyrusimap.org/
List Archives/Info: http://lists.andrew.cmu.edu/pipermail/info-cyrus/
To Unsubscribe:
https://lists.andrew.cmu.edu/mailman/listinfo/info-cyrus

Re: squatter not used after upgrade to cyrus 3.0.8

[Robert Stepanek]

On Fri, May 10, 2019, at 3:58 PM, Michael Menge wrote:
> I can test again if search_fuzzy_always has an influence on
> the usage of the usage of the "search text".

Yes, please do.

Cheers,
Robert

Cyrus Home Page: http://www.cyrusimap.org/
List Archives/Info: http://lists.andrew.cmu.edu/pipermail/info-cyrus/
To Unsubscribe:
https://lists.andrew.cmu.edu/mailman/listinfo/info-cyrus

Re: squatter not used after upgrade to cyrus 3.0.8

[Michael Menge]

Quoting Robert Stepanek :


On Fri, May 10, 2019, at 2:18 PM, Michael Menge wrote:

no the problem is not completly resolved jet.
See for details.


I tested this right now on the 3.0 branch, and I can confirm there's  
an issue of matches not being found. I don't have an answer why this  
is so, yet. I'm working on it.



> I've put in my impad.conf:
> search_engine: squat
> search_fuzzy_always: 1

AFAIK the squat search engine does not support fuzzy search.
I am sure in my testing this setting didn't resolve the slow
search, but i don't remember if this setting did nothing,
or failed to find anything at all or what else happened
with this configuration.


If I recall correctly, Cyrus uses the configured search engine only  
for fuzzy search. Non-fuzzy search is matched using the  
Cyrus-builtin routines, which will be slow: for body search it has  
to examine every message for that mailbox.


If your client is sending something like:

 C: 6 search body "body"

then it won't use the squat/xapian index, unless you have  
search_fuzzy_always set.




Thanks Robert for looking into it.

I think there is some confusion here. I didn't test "search body",
not sure if this was supported by cyrus 2.4 and if yes if it made
use of the index.
What I did test was "search text" and "search header".
Both used the index in 2.4 and didn't use it in 3.0
in my initial configuration.

I was able to make cyrus use the squat index file again
for header search by enabling conversation db, though I had to
disable conversation db again on our production servers because
of other problems.

I can test again if search_fuzzy_always has an influence on
the usage of the usage of the "search text".



M.MengeTel.: (49) 7071/29-70316
Universität Tübingen   Fax.: (49) 7071/29-5912
Zentrum für Datenverarbeitung  mail:  
michael.me...@zdv.uni-tuebingen.de

Wächterstraße 76
72074 Tübingen


Cyrus Home Page: http://www.cyrusimap.org/
List Archives/Info: http://lists.andrew.cmu.edu/pipermail/info-cyrus/
To Unsubscribe:
https://lists.andrew.cmu.edu/mailman/listinfo/info-cyrus

Re: squatter not used after upgrade to cyrus 3.0.8

[Robert Stepanek]

On Fri, May 10, 2019, at 2:18 PM, Michael Menge wrote:
> no the problem is not completly resolved jet.
> See https://github.com/cyrusimap/cyrus-imapd/issues/2598 for details.

I tested this right now on the 3.0 branch, and I can confirm there's an issue 
of matches not being found. I don't have an answer why this is so, yet. I'm 
working on it.

> > I've put in my impad.conf:
> > search_engine: squat
> > search_fuzzy_always: 1
> 
> AFAIK the squat search engine does not support fuzzy search.
> I am sure in my testing this setting didn't resolve the slow
> search, but i don't remember if this setting did nothing,
> or failed to find anything at all or what else happened
> with this configuration.

If I recall correctly, Cyrus uses the configured search engine only for fuzzy 
search. Non-fuzzy search is matched using the Cyrus-builtin routines, which 
will be slow: for body search it has to examine every message for that mailbox.

If your client is sending something like:

 C: 6 search body "body"

then it won't use the squat/xapian index, unless you have search_fuzzy_always 
set.

Cheers,
Robert
Cyrus Home Page: http://www.cyrusimap.org/
List Archives/Info: http://lists.andrew.cmu.edu/pipermail/info-cyrus/
To Unsubscribe:
https://lists.andrew.cmu.edu/mailman/listinfo/info-cyrus

Re: squatter not used after upgrade to cyrus 3.0.8

[Michael Menge]

Hi Carlos,

Quoting Carlos Larrañaga :


Hi Michael,

Do you resolved this problem?
I'm having the same issue with cyrus 3.0.9 not accessing cyrus.squat files.



no the problem is not completly resolved jet.
See https://github.com/cyrusimap/cyrus-imapd/issues/2598 for details.


I've put in my impad.conf:
  search_engine: squat
  search_fuzzy_always: 1


AFAIK the squat search engine does not support fuzzy search.
I am sure in my testing this setting didn't resolve the slow
search, but i don't remember if this setting did nothing,
or  failed to find anything at all or what else happened
with this configuration.



Any recomendation will be appreciated.

Best regards,
Carlos.


El 25/10/2018 a las 15:09, Michael Menge escribió:

Hi,

Quoting Michael Menge :


Hi,

Quoting Albert Shih :


Le 17/09/2018 à 14:01:52+0200, Michael Menge a écrit

Hi,

we recently upgrade from Cyrus-Imapd 2.4.x to 3.0.8. After some initial
problems
which we could fix cyrus imapd 3.0.8 is running stable. The one remaining
problem
we receive reports about is, that the search is not working/too slow.

First we discovered that one of the options for Squatter are not
supported
anymore, "-s Skip mailboxes whose index file is older than their current
squat file (within a small time delta)." and that squatter does not like
"-r" in combination with the source "user"

  > squatter -C /etc/imapd_be.conf -r  user
  fatal error: Internal error: assertion failed: lib/cyrusdb_twoskip.c:
2339: keylen


But after reindexing all mailboxes the search is still slow. I tried to
debug this and
found with strace that cyrus didn't try to open the cyrus.squat
file for the
mailbox.

I suspect that I am missed some configuration change. So here is our
imapd.conf for our backends


If I'm correct you need :

 search_fuzzy_always: on

in your config.

If you can tell me if it's work...I would really appreciate. Because I
activated that but I'm not able to see if it's work really.


Thanks for your help.

I did tried it on my test server and it seems to be a bit faster,
but that could be due to caching. Strace still didn't show any access
to the cyrus.squat file.

For information: We only use squatter. No Xapia. And we had much faster
search with Cyrus-Imapd 2.3.x and 2.4.x. I don't have the timings form
the old system but our users are complaining and they receive timeouts
in our horde/imp webmailer, which they did't receive before.

Any other ideas are appreciated.


I still have the problem that search in cyrus imap 3.0.8 with search engine
squatter is slow compared to 2.4.20. I try to figure out if the squatter
search engine is working in cyurs imapd 3.0 and I messed up my
configuration,
or if my configuration should work but squatter is broken.

I did set up a test environment to compare the old and new versions.
To verify that the search is indeed slower with 3.0.8

I used two different searches 'B SEARCH TEXT "Test"' and 'B SEARCH
HEADER X-comment Unirundmail'

=== 2.4.20 === SEARCH TEXT

A SELECT INBOX
* 4321 EXISTS
* 4321 RECENT
* FLAGS (\Answered \Flagged \Draft \Deleted \Seen)
* OK [PERMANENTFLAGS (\Answered \Flagged \Draft \Deleted \Seen \*)] Ok
* OK [UNSEEN 1] Ok
* OK [UIDVALIDITY 1540372444] Ok
* OK [UIDNEXT 93369] Ok
* OK [HIGHESTMODSEQ 2] Ok
* OK [URLMECH INTERNAL] Ok
A OK [READ-WRITE] Completed
B SEARCH TEXT "Test"
* SEARCH 1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 19 20 21 22
23 24 25 26 27 28 29 30 31 33 34 35 36 37 38 39

4292 4294 4295 4296 4298 4299 4300 4301 4303 4306 4307 4309 4310
4315 4316 4317 4318 4321
B OK Completed (1996 msgs in 0.690 secs)


=== 3.0.8 === SEARCH TEXT

a SELECT INBOX
* 4321 EXISTS
* 0 RECENT
* FLAGS (\Answered \Flagged \Draft \Deleted \Seen NonJunk
$Forwarded $mdnsent $label1 $label2 $label3 hordetest testflag)
* OK [PERMANENTFLAGS (\Answered \Flagged \Draft \Deleted \Seen
NonJunk $Forwarded $mdnsent $label1 $label2 $label3 hordetest
testflag \*)] Ok
* OK [UNSEEN 3737] Ok
* OK [UIDVALIDITY 1238498084] Ok
* OK [UIDNEXT 93373] Ok
* OK [HIGHESTMODSEQ 98491] Ok
* OK [URLMECH INTERNAL] Ok
* OK [ANNOTATIONS 65536] Ok
a OK [READ-WRITE] Completed
B SEARCH TEXT "Test"
* SEARCH 1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 19 20 21 22
23 24 25 26 27 28 29 30 31 33 34 35 36 37 38 39

4274 4275 4276 4277 4278 4279 4285 4286 4287 4288 4292 4294 4295
4296 4298 4299 4300 4301 4303 4306 4307 4309 4310 4315 4316 4317
4318 4321
B OK Completed (1935 msgs in 2.580 secs)

 2.4.20 === SEARCH HEADER

a SELECT INBOX
* 4321 EXISTS
* 0 RECENT
* FLAGS (\Answered \Flagged \Draft \Deleted \Seen)
* OK [PERMANENTFLAGS (\Answered \Flagged \Draft \Deleted \Seen \*)] Ok
* OK [UNSEEN 1] Ok
* OK [UIDVALIDITY 1540372444] Ok
* OK [UIDNEXT 93369] Ok
* OK [HIGHESTMODSEQ 2] Ok
* OK [URLMECH INTERNAL] Ok
a OK [READ-WRITE] Completed
b SEARCH HEADER X-comment Unirundmail
* SEARCH 4283 4291 4313 431

Re: squatter not used after upgrade to cyrus 3.0.8

[Carlos Larrañaga]

Hi Michael,

Do you resolved this problem?
I'm having the same issue with cyrus 3.0.9 not accessing cyrus.squat files.

I've put in my impad.conf:
  search_engine: squat
  search_fuzzy_always: 1

Any recomendation will be appreciated.

Best regards,
Carlos.


El 25/10/2018 a las 15:09, Michael Menge escribió:

Hi,

Quoting Michael Menge :


Hi,

Quoting Albert Shih :


Le 17/09/2018 à 14:01:52+0200, Michael Menge a écrit

Hi,

we recently upgrade from Cyrus-Imapd 2.4.x to 3.0.8. After some initial
problems
which we could fix cyrus imapd 3.0.8 is running stable. The one remaining
problem
we receive reports about is, that the search is not working/too slow.

First we discovered that one of the options for Squatter are not supported
anymore, "-s Skip mailboxes whose index file is older than their current
squat file (within a small time delta)." and that squatter does not like
"-r" in combination with the source "user"

  > squatter -C /etc/imapd_be.conf -r  user
  fatal error: Internal error: assertion failed: lib/cyrusdb_twoskip.c:
2339: keylen


But after reindexing all mailboxes the search is still slow. I tried to
debug this and
found with strace that cyrus didn't try to open the cyrus.squat file for the
mailbox.

I suspect that I am missed some configuration change. So here is our
imapd.conf for our backends


If I'm correct you need :

 search_fuzzy_always: on

in your config.

If you can tell me if it's work...I would really appreciate. Because I
activated that but I'm not able to see if it's work really.


Thanks for your help.

I did tried it on my test server and it seems to be a bit faster,
but that could be due to caching. Strace still didn't show any access
to the cyrus.squat file.

For information: We only use squatter. No Xapia. And we had much faster
search with Cyrus-Imapd 2.3.x and 2.4.x. I don't have the timings form
the old system but our users are complaining and they receive timeouts
in our horde/imp webmailer, which they did't receive before.

Any other ideas are appreciated.


I still have the problem that search in cyrus imap 3.0.8 with search engine
squatter is slow compared to 2.4.20. I try to figure out if the squatter
search engine is working in cyurs imapd 3.0 and I messed up my configuration,
or if my configuration should work but squatter is broken.

I did set up a test environment to compare the old and new versions.
To verify that the search is indeed slower with 3.0.8

I used two different searches 'B SEARCH TEXT "Test"' and 'B SEARCH HEADER 
X-comment Unirundmail'


=== 2.4.20 === SEARCH TEXT

A SELECT INBOX
* 4321 EXISTS
* 4321 RECENT
* FLAGS (\Answered \Flagged \Draft \Deleted \Seen)
* OK [PERMANENTFLAGS (\Answered \Flagged \Draft \Deleted \Seen \*)] Ok
* OK [UNSEEN 1] Ok
* OK [UIDVALIDITY 1540372444] Ok
* OK [UIDNEXT 93369] Ok
* OK [HIGHESTMODSEQ 2] Ok
* OK [URLMECH INTERNAL] Ok
A OK [READ-WRITE] Completed
B SEARCH TEXT "Test"
* SEARCH 1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 19 20 21 22 23 24 25 26 
27 28 29 30 31 33 34 35 36 37 38 39


4292 4294 4295 4296 4298 4299 4300 4301 4303 4306 4307 4309 4310 4315 4316 
4317 4318 4321

B OK Completed (1996 msgs in 0.690 secs)


=== 3.0.8 === SEARCH TEXT

a SELECT INBOX
* 4321 EXISTS
* 0 RECENT
* FLAGS (\Answered \Flagged \Draft \Deleted \Seen NonJunk $Forwarded $mdnsent 
$label1 $label2 $label3 hordetest testflag)
* OK [PERMANENTFLAGS (\Answered \Flagged \Draft \Deleted \Seen NonJunk 
$Forwarded $mdnsent $label1 $label2 $label3 hordetest testflag \*)] Ok

* OK [UNSEEN 3737] Ok
* OK [UIDVALIDITY 1238498084] Ok
* OK [UIDNEXT 93373] Ok
* OK [HIGHESTMODSEQ 98491] Ok
* OK [URLMECH INTERNAL] Ok
* OK [ANNOTATIONS 65536] Ok
a OK [READ-WRITE] Completed
B SEARCH TEXT "Test"
* SEARCH 1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 19 20 21 22 23 24 25 26 
27 28 29 30 31 33 34 35 36 37 38 39


4274 4275 4276 4277 4278 4279 4285 4286 4287 4288 4292 4294 4295 4296 4298 
4299 4300 4301 4303 4306 4307 4309 4310 4315 4316 4317 4318 4321

B OK Completed (1935 msgs in 2.580 secs)

 2.4.20 === SEARCH HEADER

a SELECT INBOX
* 4321 EXISTS
* 0 RECENT
* FLAGS (\Answered \Flagged \Draft \Deleted \Seen)
* OK [PERMANENTFLAGS (\Answered \Flagged \Draft \Deleted \Seen \*)] Ok
* OK [UNSEEN 1] Ok
* OK [UIDVALIDITY 1540372444] Ok
* OK [UIDNEXT 93369] Ok
* OK [HIGHESTMODSEQ 2] Ok
* OK [URLMECH INTERNAL] Ok
a OK [READ-WRITE] Completed
b SEARCH HEADER X-comment Unirundmail
* SEARCH 4283 4291 4313 4319 4320
b OK Completed (5 msgs in 0.020 secs)

 3.0.8 === SEARCH HEADER

a SELECT INBOX
* 4321 EXISTS
* 0 RECENT
* FLAGS (\Answered \Flagged \Draft \Deleted \Seen NonJunk $Forwarded $mdnsent 
$label1 $label2 $label3 hordetest testflag)
* OK [PERMANENTFLAGS (\Answered \Flagged \Draft \Deleted \Seen NonJunk 
$Forwarded $mdnsent $label1 $label2 $label3 hordetest testflag \*)] Ok

* OK [UNSEEN 3737] Ok
* OK [UIDVALIDITY 1

squatter issue on Cyrus 2.4

[Marco]

Hello,

 I have Cyrus 2.4 with thousand mailboxes and separated partitions for 
data and metadata, with squat files in metamaildata partitions.


Every night at 2 I run "squatter -s -a -i" to update the indexes.

Two nights ago a metamaildata partition filled out. Starting from 2am 
many GiB busy in few hours. I can't understand why. No errors in Cyrus 
logs. I didn't find the files which caused the busy space.


Last night squatter didn't reproduce the error.

Do you have ever experienced problems like this?

Thank you very much for every hints.
Kind Regards
Marco

**
name   : Cyrus IMAPD
version: v2.4.17-Invoca-RPM-2.4.17-6.el6 d1df8aff 2012-12-01
vendor : Project Cyrus
support-url: http://www.cyrusimap.org
os : Linux
os-version : 2.6.32-279.el6.x86_64
environment: Built w/Cyrus SASL 2.1.23
 Running w/Cyrus SASL 2.1.23
 Built w/OpenSSL 1.0.0-fips 29 Mar 2010
 Running w/OpenSSL 1.0.0-fips 29 Mar 2010
 Built w/zlib 1.2.3
 Running w/zlib 1.2.3
 CMU Sieve 2.4
 TCP Wrappers
 NET-SNMP
 mmap = shared
 lock = fcntl
 nonblock = fcntl
 idle = idled

******
EVENTS {
  # Squatter
  squatter  cmd="squatter -s -a -i" at=0200
**
metapartition_files: header index cache expunge squat
**

Cyrus Home Page: http://www.cyrusimap.org/
List Archives/Info: http://lists.andrew.cmu.edu/pipermail/info-cyrus/
To Unsubscribe:
https://lists.andrew.cmu.edu/mailman/listinfo/info-cyrus

Re: squatter not used after upgrade to cyrus 3.0.8

[Michael Menge]

Hi,

Quoting Michael Menge :


Hi,

Quoting Albert Shih :


Le 17/09/2018 à 14:01:52+0200, Michael Menge a écrit

Hi,

we recently upgrade from Cyrus-Imapd 2.4.x to 3.0.8. After some initial
problems
which we could fix cyrus imapd 3.0.8 is running stable. The one remaining
problem
we receive reports about is, that the search is not working/too slow.

First we discovered that one of the options for Squatter are not supported
anymore, "-s Skip mailboxes whose index file is older than their current
squat file (within a small time delta)." and that squatter does not like
"-r" in combination with the source "user"

  > squatter -C /etc/imapd_be.conf -r  user
  fatal error: Internal error: assertion failed: lib/cyrusdb_twoskip.c:
2339: keylen


But after reindexing all mailboxes the search is still slow. I tried to
debug this and
found with strace that cyrus didn't try to open the cyrus.squat  
file for the

mailbox.

I suspect that I am missed some configuration change. So here is our
imapd.conf for our backends


If I'm correct you need :

 search_fuzzy_always: on

in your config.

If you can tell me if it's work...I would really appreciate. Because I
activated that but I'm not able to see if it's work really.


Thanks for your help.

I did tried it on my test server and it seems to be a bit faster,
but that could be due to caching. Strace still didn't show any access
to the cyrus.squat file.

For information: We only use squatter. No Xapia. And we had much faster
search with Cyrus-Imapd 2.3.x and 2.4.x. I don't have the timings form
the old system but our users are complaining and they receive timeouts
in our horde/imp webmailer, which they did't receive before.

Any other ideas are appreciated.


I still have the problem that search in cyrus imap 3.0.8 with search engine
squatter is slow compared to 2.4.20. I try to figure out if the squatter
search engine is working in cyurs imapd 3.0 and I messed up my configuration,
or if my configuration should work but squatter is broken.

I did set up a test environment to compare the old and new versions.
To verify that the search is indeed slower with 3.0.8

I used two different searches 'B SEARCH TEXT "Test"' and 'B SEARCH  
HEADER X-comment Unirundmail'


=== 2.4.20 === SEARCH TEXT

A SELECT INBOX
* 4321 EXISTS
* 4321 RECENT
* FLAGS (\Answered \Flagged \Draft \Deleted \Seen)
* OK [PERMANENTFLAGS (\Answered \Flagged \Draft \Deleted \Seen \*)] Ok
* OK [UNSEEN 1] Ok
* OK [UIDVALIDITY 1540372444] Ok
* OK [UIDNEXT 93369] Ok
* OK [HIGHESTMODSEQ 2] Ok
* OK [URLMECH INTERNAL] Ok
A OK [READ-WRITE] Completed
B SEARCH TEXT "Test"
* SEARCH 1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 19 20 21 22 23  
24 25 26 27 28 29 30 31 33 34 35 36 37 38 39


4292 4294 4295 4296 4298 4299 4300 4301 4303 4306 4307 4309 4310 4315  
4316 4317 4318 4321

B OK Completed (1996 msgs in 0.690 secs)


=== 3.0.8 === SEARCH TEXT

a SELECT INBOX
* 4321 EXISTS
* 0 RECENT
* FLAGS (\Answered \Flagged \Draft \Deleted \Seen NonJunk $Forwarded  
$mdnsent $label1 $label2 $label3 hordetest testflag)
* OK [PERMANENTFLAGS (\Answered \Flagged \Draft \Deleted \Seen NonJunk  
$Forwarded $mdnsent $label1 $label2 $label3 hordetest testflag \*)] Ok

* OK [UNSEEN 3737] Ok
* OK [UIDVALIDITY 1238498084] Ok
* OK [UIDNEXT 93373] Ok
* OK [HIGHESTMODSEQ 98491] Ok
* OK [URLMECH INTERNAL] Ok
* OK [ANNOTATIONS 65536] Ok
a OK [READ-WRITE] Completed
B SEARCH TEXT "Test"
* SEARCH 1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 19 20 21 22 23  
24 25 26 27 28 29 30 31 33 34 35 36 37 38 39


4274 4275 4276 4277 4278 4279 4285 4286 4287 4288 4292 4294 4295 4296  
4298 4299 4300 4301 4303 4306 4307 4309 4310 4315 4316 4317 4318 4321

B OK Completed (1935 msgs in 2.580 secs)

 2.4.20 === SEARCH HEADER

a SELECT INBOX
* 4321 EXISTS
* 0 RECENT
* FLAGS (\Answered \Flagged \Draft \Deleted \Seen)
* OK [PERMANENTFLAGS (\Answered \Flagged \Draft \Deleted \Seen \*)] Ok
* OK [UNSEEN 1] Ok
* OK [UIDVALIDITY 1540372444] Ok
* OK [UIDNEXT 93369] Ok
* OK [HIGHESTMODSEQ 2] Ok
* OK [URLMECH INTERNAL] Ok
a OK [READ-WRITE] Completed
b SEARCH HEADER X-comment Unirundmail
* SEARCH 4283 4291 4313 4319 4320
b OK Completed (5 msgs in 0.020 secs)

 3.0.8 === SEARCH HEADER

a SELECT INBOX
* 4321 EXISTS
* 0 RECENT
* FLAGS (\Answered \Flagged \Draft \Deleted \Seen NonJunk $Forwarded  
$mdnsent $label1 $label2 $label3 hordetest testflag)
* OK [PERMANENTFLAGS (\Answered \Flagged \Draft \Deleted \Seen NonJunk  
$Forwarded $mdnsent $label1 $label2 $label3 hordetest testflag \*)] Ok

* OK [UNSEEN 3737] Ok
* OK [UIDVALIDITY 1238498084] Ok
* OK [UIDNEXT 93373] Ok
* OK [HIGHESTMODSEQ 98491] Ok
* OK [URLMECH INTERNAL] Ok
* OK [ANNOTATIONS 65536] Ok
a OK [READ-WRITE] Completed
b SEARCH HEADER X-comment Unirundmail
* SEARCH 4283 4291 4313 4319 4320
b OK Completed (5 msgs in 0.370 secs)

===

There is also a big discrepancy between 

Re: squatter not used after upgrade to cyrus 3.0.8

[Michael Menge]

Hi,

Quoting Albert Shih :


Le 17/09/2018 à 14:01:52+0200, Michael Menge a écrit

Hi,

we recently upgrade from Cyrus-Imapd 2.4.x to 3.0.8. After some initial
problems
which we could fix cyrus imapd 3.0.8 is running stable. The one remaining
problem
we receive reports about is, that the search is not working/too slow.

First we discovered that one of the options for Squatter are not supported
anymore, "-s Skip mailboxes whose index file is older than their current
squat file (within a small time delta)." and that squatter does not like
"-r" in combination with the source "user"

   > squatter -C /etc/imapd_be.conf -r  user
   fatal error: Internal error: assertion failed: lib/cyrusdb_twoskip.c:
2339: keylen


But after reindexing all mailboxes the search is still slow. I tried to
debug this and
found with strace that cyrus didn't try to open the cyrus.squat file for the
mailbox.

I suspect that I am missed some configuration change. So here is our
imapd.conf for our backends


If I'm correct you need :

  search_fuzzy_always: on

in your config.

If you can tell me if it's work...I would really appreciate. Because I
activated that but I'm not able to see if it's work really.


Thanks for your help.

I did tried it on my test server and it seems to be a bit faster,
but that could be due to caching. Strace still didn't show any access
to the cyrus.squat file.

For information: We only use squatter. No Xapia. And we had much faster
search with Cyrus-Imapd 2.3.x and 2.4.x. I don't have the timings form
the old system but our users are complaining and they receive timeouts
in our horde/imp webmailer, which they did't receive before.

Any other ideas are appreciated.


Regards

   Michael Menge



M.MengeTel.: (49) 7071/29-70316
Universität Tübingen   Fax.: (49) 7071/29-5912
Zentrum für Datenverarbeitung  mail:  
michael.me...@zdv.uni-tuebingen.de

Wächterstraße 76
72074 Tübingen


Cyrus Home Page: http://www.cyrusimap.org/
List Archives/Info: http://lists.andrew.cmu.edu/pipermail/info-cyrus/
To Unsubscribe:
https://lists.andrew.cmu.edu/mailman/listinfo/info-cyrus

squatter not used after upgrade to cyrus 3.0.8

[Michael Menge]

Hi,

we recently upgrade from Cyrus-Imapd 2.4.x to 3.0.8. After some  
initial problems
which we could fix cyrus imapd 3.0.8 is running stable. The one  
remaining problem

we receive reports about is, that the search is not working/too slow.

First we discovered that one of the options for Squatter are not  
supported anymore, "-s Skip mailboxes whose index file is older than  
their current squat file (within a small time delta)." and that  
squatter does not like  "-r" in combination with the source "user"


   > squatter -C /etc/imapd_be.conf -r  user
   fatal error: Internal error: assertion failed:  
lib/cyrusdb_twoskip.c: 2339: keylen



But after reindexing all mailboxes the search is still slow. I tried  
to debug this and
found with strace that cyrus didn't try to open the cyrus.squat file  
for the mailbox.


I suspect that I am missed some configuration change. So here is our  
imapd.conf for our backends



=== Beginq Imapd.conf =
servername: ma0X.mail.localhost
configdirectory: /srv/cyrus-be
partition-default: /srv/cyrus-be
partition-ssd: /srv/cyrus-be/ssd-part
metapartition-ssd: /srv/cyrus-ssd-be/meta/ssd-part
metapartition_files: header index cache expunge squat annotations lock  
dav archivecache

archivepartition-ssd: /srv/cyrus-hdd-be/archive/ssd-part
archive_enabled: 1
## archive_days = 4*365
archive_days: 1460

proc_path: /srv/tmpfs/proc-be
mboxname_lockpath: /srv/tmpfs/lock-be
defaultpartition: ssd
admins: XXX

mupdate_server: mupdate.mail.localhost
mupdate_port: 3905
mupdate_authname: XXX
mupdate_password: XXX
proxy_authname: XXX
proxy_password: XXX
proxyservers: XXX

allowusermoves: 1
allowallsubscribe: 1
sync_host: sl0X.mail.localhost
sync_authname: XXX
sync_password: XXX
sync_port: 2005
guid_mode: sha1
sync_log: 1
sync_shutdown_file: /srv/cyrus-be/sync/shutdown

sievedir: /srv/cyrus-be/sieve
sieve_extensions: fileinto reject vacation imapflags notify include  
envelope body relational regex subaddress copy

sieve_maxscriptsize: 150

sasl_pwcheck_method: saslauthd
sasl_mech_list: plain login

allowanonymouslogin: no
reject8bit: no
quotawarn: 90
timeout: 45
poptimeout: 10
dracinterval: 0
drachost: localhost
lmtp_over_quota_perm_failure: 1
altnamespace: 1
#flushseenstate: 1
unixhierarchysep: 1
hashimapspool: 1
fulldirhash: 1
duplicatesuppression: 0
expunge_mode: delayed
delete_mode: delayed
deletedprefix: DELETED
anysievefolder: 1
quota_db: quotalegacy
subscriptions_db: flat
xlist-sent: Mail.sent
xlist-trash: Mail.trash
xlist-drafts: Mail.drafts
xlist-junk: Mail.v-spam
xlist-spam: Mail.v-spam
specialusealways: 1
syslog_prefix: be

tls_server_cert: XXX
tls_server_key: XXX
tls_client_ca_file: XXX
tls_ciphers: XXX
tls_prefer_server_ciphers: 1

auditlog: 1
reverseacls: 1
search_engine: squat
delete_unsubscribe: 1
statuscache: 1

=== End Imapd.conf =


M.MengeTel.: (49) 7071/29-70316
Universität Tübingen   Fax.: (49) 7071/29-5912
Zentrum für Datenverarbeitung  mail:  
michael.me...@zdv.uni-tuebingen.de

Wächterstraße 76
72074 Tübingen


Cyrus Home Page: http://www.cyrusimap.org/
List Archives/Info: http://lists.andrew.cmu.edu/pipermail/info-cyrus/
To Unsubscribe:
https://lists.andrew.cmu.edu/mailman/listinfo/info-cyrus

Re: Squatter segfault on big mailbox

[Vladislav Kurz]

On 10/11/17 07:37, Mark wrote:
> On 2017-10-10 16:05, Vladislav Kurz wrote:
>> Hello everyone,
>>
>> we have recently migrated our mail server from cyrus 2.2 to 2.5 (both
>> debian packages).
>>
>> We have a problem that squatter segfaults on some quite big mailboxes.
>> I had to add -i (--incremental) to at least index new mails. But users
>> cannot search in old mails (before migration). I used strace to find out
>> if it is some particular mail, but it parsed everything fine, and failed
>> when dealing with some of the many temporary files squat.
>>
>> The problematic mailboxes have just inbox > 10 GB (and >15 GB with all
>> subfolders) and >2 emails in inbox.
>>
>> Has anyone else hit similar problem and solved it somehow?
> 
> I'm not sure about fixes, but there are some reports of similar
> problems.  Have a look at the thread with the subject "Cyrus 2.5.10 IMAP
> search" from the beginning of June this year. 
> 
> Apparently the format of the squatter database changed between 2.4 and
> 2.5, and some people (at least) have seen a 10x increase in the size of
> the squatter database for the same mailbox.  For systems that only allow
> 2GB mmap, this can be a problem.
> 
> I'm not sure if this increase in size is a bug or a feature.
> 
> Mark
> 

Hello Mark,

Thanks for the hint about old discussion. Unfortunately, there was no
solution to this issue mentioned. I found out that this was already
reported to debian, I wonder whetrher the maintainer forwarded the bug
report upstream.
https://bugs.debian.org/cgi-bin/bugreport.cgi?bug=871921

This is on 64-bit Linux, so mmap should not be limited to 2 GB.

And I also have noticed the extreme increase in squat file sizes.


-- 
Best Regards
Vladislav Kurz


Cyrus Home Page: http://www.cyrusimap.org/
List Archives/Info: http://lists.andrew.cmu.edu/pipermail/info-cyrus/
To Unsubscribe:
https://lists.andrew.cmu.edu/mailman/listinfo/info-cyrus

Squatter segfault on big mailbox

[Vladislav Kurz]

Hello everyone,

we have recently migrated our mail server from cyrus 2.2 to 2.5 (both
debian packages).

We have a problem that squatter segfaults on some quite big mailboxes.
I had to add -i (--incremental) to at least index new mails. But users
cannot search in old mails (before migration). I used strace to find out
if it is some particular mail, but it parsed everything fine, and failed
when dealing with some of the many temporary files squat.

The problematic mailboxes have just inbox > 10 GB (and >15 GB with all
subfolders) and >2 emails in inbox.

Has anyone else hit similar problem and solved it somehow?

-- 
Best regards
Vladislav Kurz


Cyrus Home Page: http://www.cyrusimap.org/
List Archives/Info: http://lists.andrew.cmu.edu/pipermail/info-cyrus/
To Unsubscribe:
https://lists.andrew.cmu.edu/mailman/listinfo/info-cyrus

Re: squatter lower limits

[Robert Stepanek]

Hi,

I haven't worked with the squat backend, but I highly recommend to
switch to the latest stable 3.0 branch and the Xapian backend, if you
can.

The Xapian backend does not discriminate between short and long strings,
provides stemming and multi-tiered search databases, among other
improvements. Note that the squat backend and the squatter tool share
the same name, but really become two separate things, e.g. you'd still
use the squatter executable to index Xapian-backed search databases.

Bron wrote up a description of search setup in 2014:
https://blog.fastmail.com/2014/12/01/email-search-system/

Cheers,
Robert


On Fri, Sep 15, 2017, at 11:35, Paolo Cravero wrote:
> Hello.
> 
> While looking to do low-level disk usage optimization, some simple
> performance tests relied on full-text searches (2.4 branch). Metadata
> always resides on local disks, while messages are on slower hardware.
> 
> I noticed that full-text searches with short strings take much longer
> than longer text. For example, a FT search on 3 letters takes >60" while
> a 9-letter long string on the same corpus lasts ~20". These tests have
> been repeated over and over again to exclude disk caching being the
> culprit: reversing the search order - longer first - has no impact.
> 
> So I opened up the cyrus source code and looked for search-related code.
> As I understand it, squatter is not used if the search string is shorter
> than 4 symbols. From squat.h it's quite clear:
> 
> /*
> Don't change this unless you're SURE you know what you're doing.
> Its only effect on the API is that searches for strings that are
> shorter than SQUAT_WORD_SIZE are not allowed.
> In SQUAT, a 'word' simply refers to a string of SQUAT_WORD_SIZE
> arbitrary bytes.
> */
> 
> #define SQUAT_WORD_SIZE 4
> 
> So, question to who knows the squatter implementation in cyrus: is this
> lower limit applied to all searches? Body, subject, addresse(s)?
> 
> And, does this lower bound still apply to 3.0 branch and the new indexing
> engine Xapian?
> 
> Let alone low level disk compression or optimization, a client might not
> handle well long search times without receiving data on the IMAP channel
> and dismiss the connection (or a network device could do it). So, if
> searching for short strings means reading all raw message files, I should
> warn users through the client interface of possible failures since the
> mail corpus keeps growing and growing and growing. That's until we
> upgrade to 3.0, it that helps.
> 
> Thanks,
> 
> Paolo
> 
> Cyrus Home Page: http://www.cyrusimap.org/
> List Archives/Info: http://lists.andrew.cmu.edu/pipermail/info-cyrus/
> To Unsubscribe:
> https://lists.andrew.cmu.edu/mailman/listinfo/info-cyrus

Cyrus Home Page: http://www.cyrusimap.org/
List Archives/Info: http://lists.andrew.cmu.edu/pipermail/info-cyrus/
To Unsubscribe:
https://lists.andrew.cmu.edu/mailman/listinfo/info-cyrus

squatter lower limits

[Paolo Cravero]

Hello.

While looking to do low-level disk usage optimization, some simple performance 
tests relied on full-text searches (2.4 branch). Metadata always resides on 
local disks, while messages are on slower hardware.

I noticed that full-text searches with short strings take much longer than 
longer text. For example, a FT search on 3 letters takes >60" while a 9-letter 
long string on the same corpus lasts ~20". These tests have been repeated over 
and over again to exclude disk caching being the culprit: reversing the search 
order - longer first - has no impact.

So I opened up the cyrus source code and looked for search-related code. As I 
understand it, squatter is not used if the search string is shorter than 4 
symbols. From squat.h it's quite clear:

/*
Don't change this unless you're SURE you know what you're doing.
Its only effect on the API is that searches for strings that are
shorter than SQUAT_WORD_SIZE are not allowed.
In SQUAT, a 'word' simply refers to a string of SQUAT_WORD_SIZE
arbitrary bytes.
*/

#define SQUAT_WORD_SIZE 4

So, question to who knows the squatter implementation in cyrus: is this lower 
limit applied to all searches? Body, subject, addresse(s)?

And, does this lower bound still apply to 3.0 branch and the new indexing 
engine Xapian?

Let alone low level disk compression or optimization, a client might not handle 
well long search times without receiving data on the IMAP channel and dismiss 
the connection (or a network device could do it). So, if searching for short 
strings means reading all raw message files, I should warn users through the 
client interface of possible failures since the mail corpus keeps growing and 
growing and growing. That's until we upgrade to 3.0, it that helps.

Thanks,

Paolo

Cyrus Home Page: http://www.cyrusimap.org/
List Archives/Info: http://lists.andrew.cmu.edu/pipermail/info-cyrus/
To Unsubscribe:
https://lists.andrew.cmu.edu/mailman/listinfo/info-cyrus

Cyrus Squatter now running with Xapian on XStreamOS!

[Gabriele Bulfon]

I found the reason why squatter was not running : because I built with 
conversation support, squatter complained about these being a requirement but 
not enabled in imapd.conf.
Enabled on impad.conf, restarted cyrus, squatter started full indexing!
And I also found my xapian config produced a lot of stuff under 
".../cyrus/search..."!
Will I have full indexing of attachments (doc, pdf etc) by default too?
Or should I do something specific?
Thanks!
Gabriele
Sonicle S.r.l.
:
http://www.sonicle.com
Music:
http://www.gabrielebulfon.com
Quantum Mechanics :
http://www.cdbaby.com/cd/gabrielebulfon

Cyrus Home Page: http://www.cyrusimap.org/
List Archives/Info: http://lists.andrew.cmu.edu/pipermail/info-cyrus/
To Unsubscribe:
https://lists.andrew.cmu.edu/mailman/listinfo/info-cyrus

Re: Squatter after upgrade

[Anton via Info-cyrus]

Hi Simon.
At the beginning this year I migrated from ubuntu 12.10+cyrus2.4.16 to 
centos7+cyrus2.4.17. All it have worked fine, but month and half after 
migration I start to notice something was going wrong:  time to time I see lots 
of  imap-processes, users complain to access to their mboxes, mail delivery 
problems, squatter holds at bad mbox and etc.I managed to find who faced to the 
same problem https://github.com/cyrusimap/cyrus-imapd/issues/1599. 
In my case I caught the bug exactly as described in link above. I decided to 
look for more new cyrus version for centos7 and found in kolab repository 
http://obs.kolabsys.com/repositories/Kolab:/16/CentOS_7/x86_64

Also about how to control squatter task via system scheduler CRON or in-build 
one - Cyrus EVENS Section. I think it depends on habits and sysadmin style. For 
example, I like starting cyrus routine tasks via EVENTS Section in order to 
cyrus-master controls its tasks. 

> 1 мая 2017 г., в 10:39 ДП, Simon Wilson  написал(а):
> 
> - Message from Anton  -
>   Date: Fri, 28 Apr 2017 21:22:04 +0700
>   From: Anton 
> Subject: Re: Squatter after upgrade
> To: si...@simonandkate.net
> Cc: "info-cyrus lists.andrew.cmu.edu" 
> 
> 
>> Hi Simon!
>> Just put your squatter at the same place in EVENT section in new server. 
>> Actually I recommend you don't use stock cyrus version in centos 7 due to 
>> IDLE bugs. Jump to 2.5.10 from kolab repo.
>> 
>> 
> 
> Thanks. I've added squatter and it seems to be running OK.
> 
> I've not been able to find any information about IDLE bugs in the CentOS 7 
> packaged Cyrus. The whole intent of running CentOS is for package stability - 
> it would need to be a major issue to get me to break out of that and sideload 
> a newer Cyrus. Do you have more info on the IDLE bugs you mention?
> 
>>> 28 апр. 2017 г., в 8:13 ПП, Simon Wilson  
>>> написал(а):
>>> 
>>> Hi list,
>>> 
>>> I have just upgraded from Cyrus 2.3.7 on a CentOS 5 server to 2.4.17. on a 
>>> new CentOS 7 server.
>>> 
>>> On the old one events was:
>>> 
>>> EVENTS {
>>> # this is required
>>> checkpointcmd="ctl_cyrusdb -c" period=30
>>> 
>>> # this is only necessary if using duplicate delivery suppression,
>>> # Sieve or NNTP
>>> delprune  cmd="cyr_expire -E 3" at=0400
>>> 
>>> # this is only necessary if caching TLS sessions
>>> tlsprune  cmd="tls_prune" at=0400
>>> 
>>> # running squatter
>>> squatter cmd="/usr/lib/cyrus-imapd/squatter -r user" period=1440
>>> }
>>> 
>>> On the new one it's:
>>> 
>>> EVENTS {
>>> # this is required
>>> checkpointcmd="ctl_cyrusdb -c" period=30
>>> 
>>> # this is only necessary if using duplicate delivery suppression,
>>> # Sieve or NNTP
>>> delprune  cmd="cyr_expire -E 3" at=0400
>>> 
>>> # this is only necessary if caching TLS sessions
>>> tlsprune  cmd="tls_prune" at=0400
>>> }
>>> 
>>> 
>>> I notice there is no SQUATTER entry in the new one - is it required, or is 
>>> the functionality no longer needed? Should I put a SQUATTER entry in the 
>>> new server's cyrus.conf?
>>> 
>>> Thanks
>>> Simon.
>>> 
>>> --
>>> Simon Wilson
>>> M: 0400 12 11 16
>>> 
>>> 
>>> Cyrus Home Page: http://www.cyrusimap.org/
>>> List Archives/Info: http://lists.andrew.cmu.edu/pipermail/info-cyrus/
>>> To Unsubscribe:
>>> https://lists.andrew.cmu.edu/mailman/listinfo/info-cyrus
> 
> 
> - End message from Anton  -
> 
> 
> 
> -- 
> Simon Wilson
> M: 0400 12 11 16
> 
> 
> Cyrus Home Page: http://www.cyrusimap.org/
> List Archives/Info: http://lists.andrew.cmu.edu/pipermail/info-cyrus/
> To Unsubscribe:
> https://lists.andrew.cmu.edu/mailman/listinfo/info-cyrus


smime.p7s
Description: S/MIME cryptographic signature

Cyrus Home Page: http://www.cyrusimap.org/
List Archives/Info: http://lists.andrew.cmu.edu/pipermail/info-cyrus/
To Unsubscribe:
https://lists.andrew.cmu.edu/mailman/listinfo/info-cyrus

Re: Squatter after upgrade

[ellie timoney]

If you run it from the EVENTS section of cyrus.conf, it won't run when
Cyrus itself isn't running.  Depending on your expectation at the time,
that might be an advantage or disadvantage

On Mon, May 1, 2017, at 07:19 PM, Sebastian Hagedorn wrote:
> > What advantages / disadvantages to running squatter from cron vs. from
> > cyrus.conf?
> 
> Advantage: more control. Disadvantage: none that I know of.
> 
> > What does your cronjob look like?
> 
> There's a version in contrib. It's called squatrunner.pl.
> --
> Sebastian Hagedorn - Weyertal 121, Zimmer 2.02
> Regionales Rechenzentrum (RRZK)
> Universität zu Köln / Cologne University - Tel. +49-221-470-89578
> 
> Cyrus Home Page: http://www.cyrusimap.org/
> List Archives/Info: http://lists.andrew.cmu.edu/pipermail/info-cyrus/
> To Unsubscribe:
> https://lists.andrew.cmu.edu/mailman/listinfo/info-cyrus

Cyrus Home Page: http://www.cyrusimap.org/
List Archives/Info: http://lists.andrew.cmu.edu/pipermail/info-cyrus/
To Unsubscribe:
https://lists.andrew.cmu.edu/mailman/listinfo/info-cyrus

Re: Squatter after upgrade

[Sebastian Hagedorn]

What advantages / disadvantages to running squatter from cron vs. from
cyrus.conf?


Advantage: more control. Disadvantage: none that I know of.


What does your cronjob look like?


There's a version in contrib. It's called squatrunner.pl.
--
Sebastian Hagedorn - Weyertal 121, Zimmer 2.02
Regionales Rechenzentrum (RRZK)
Universität zu Köln / Cologne University - Tel. +49-221-470-89578

Cyrus Home Page: http://www.cyrusimap.org/
List Archives/Info: http://lists.andrew.cmu.edu/pipermail/info-cyrus/
To Unsubscribe:
https://lists.andrew.cmu.edu/mailman/listinfo/info-cyrus

Re: Squatter after upgrade

[Simon Wilson]

- Message from Sebastian Hagedorn  -
   Date: Fri, 28 Apr 2017 15:45:00 +0200
   From: Sebastian Hagedorn 
Subject: Re: Squatter after upgrade
 To: si...@simonandkate.net
 Cc: "info-cyrus lists.andrew.cmu.edu" 


--On 28. April 2017 um 23:13:46 +1000 Simon Wilson  
 wrote:



I notice there is no SQUATTER entry in the new one - is it required, or
is the functionality no longer needed? Should I put a SQUATTER entry in
the new server's cyrus.conf?


If you want to use the SQUAT index, you need to run squatter. That  
doesn't have to happen from cyrus.conf, though. We have a cronjob  
for that purpose that is a bit more involved.

--
   .:.Sebastian Hagedorn - Weyertal 121 (Gebäude 133), Zimmer 2.02.:.
.:.Regionales Rechenzentrum (RRZK).:.
  .:.Universität zu Köln / Cologne University - ✆ +49-221-470-89578.:.



- End message from Sebastian Hagedorn  -

Thanks Sebastian.

What advantages / disadvantages to running squatter from cron vs. from  
cyrus.conf? What does your cronjob look like?


Simon.

--
Simon Wilson
M: 0400 12 11 16


Cyrus Home Page: http://www.cyrusimap.org/
List Archives/Info: http://lists.andrew.cmu.edu/pipermail/info-cyrus/
To Unsubscribe:
https://lists.andrew.cmu.edu/mailman/listinfo/info-cyrus

Re: Squatter after upgrade

[Simon Wilson]

- Message from Anton  -
   Date: Fri, 28 Apr 2017 21:22:04 +0700
   From: Anton 
Subject: Re: Squatter after upgrade
 To: si...@simonandkate.net
 Cc: "info-cyrus lists.andrew.cmu.edu" 



Hi Simon!
Just put your squatter at the same place in EVENT section in new  
server. Actually I recommend you don't use stock cyrus version in  
centos 7 due to IDLE bugs. Jump to 2.5.10 from kolab repo.





Thanks. I've added squatter and it seems to be running OK.

I've not been able to find any information about IDLE bugs in the  
CentOS 7 packaged Cyrus. The whole intent of running CentOS is for  
package stability - it would need to be a major issue to get me to  
break out of that and sideload a newer Cyrus. Do you have more info on  
the IDLE bugs you mention?


28 апр. 2017 г., в 8:13 ПП, Simon Wilson   
написал(а):


Hi list,

I have just upgraded from Cyrus 2.3.7 on a CentOS 5 server to  
2.4.17. on a new CentOS 7 server.


On the old one events was:

EVENTS {
 # this is required
 checkpointcmd="ctl_cyrusdb -c" period=30

 # this is only necessary if using duplicate delivery suppression,
 # Sieve or NNTP
 delprune  cmd="cyr_expire -E 3" at=0400

 # this is only necessary if caching TLS sessions
 tlsprune  cmd="tls_prune" at=0400

 # running squatter
 squatter cmd="/usr/lib/cyrus-imapd/squatter -r user" period=1440
}

On the new one it's:

EVENTS {
 # this is required
 checkpointcmd="ctl_cyrusdb -c" period=30

 # this is only necessary if using duplicate delivery suppression,
 # Sieve or NNTP
 delprune  cmd="cyr_expire -E 3" at=0400

 # this is only necessary if caching TLS sessions
 tlsprune  cmd="tls_prune" at=0400
}


I notice there is no SQUATTER entry in the new one - is it  
required, or is the functionality no longer needed? Should I put a  
SQUATTER entry in the new server's cyrus.conf?


Thanks
Simon.

--
Simon Wilson
M: 0400 12 11 16


Cyrus Home Page: http://www.cyrusimap.org/
List Archives/Info: http://lists.andrew.cmu.edu/pipermail/info-cyrus/
To Unsubscribe:
https://lists.andrew.cmu.edu/mailman/listinfo/info-cyrus



- End message from Anton  -



--
Simon Wilson
M: 0400 12 11 16


Cyrus Home Page: http://www.cyrusimap.org/
List Archives/Info: http://lists.andrew.cmu.edu/pipermail/info-cyrus/
To Unsubscribe:
https://lists.andrew.cmu.edu/mailman/listinfo/info-cyrus

Re: Squatter after upgrade

[Anton via Info-cyrus]

Hi Simon!
Just put your squatter at the same place in EVENT section in new server. 
Actually I recommend you don't use stock cyrus version in centos 7 due to IDLE 
bugs. Jump to 2.5.10 from kolab repo. 


> 28 апр. 2017 г., в 8:13 ПП, Simon Wilson  написал(а):
> 
> Hi list,
> 
> I have just upgraded from Cyrus 2.3.7 on a CentOS 5 server to 2.4.17. on a 
> new CentOS 7 server.
> 
> On the old one events was:
> 
> EVENTS {
>  # this is required
>  checkpointcmd="ctl_cyrusdb -c" period=30
> 
>  # this is only necessary if using duplicate delivery suppression,
>  # Sieve or NNTP
>  delprune  cmd="cyr_expire -E 3" at=0400
> 
>  # this is only necessary if caching TLS sessions
>  tlsprune  cmd="tls_prune" at=0400
> 
>  # running squatter
>  squatter cmd="/usr/lib/cyrus-imapd/squatter -r user" period=1440
> }
> 
> On the new one it's:
> 
> EVENTS {
>  # this is required
>  checkpointcmd="ctl_cyrusdb -c" period=30
> 
>  # this is only necessary if using duplicate delivery suppression,
>  # Sieve or NNTP
>  delprune  cmd="cyr_expire -E 3" at=0400
> 
>  # this is only necessary if caching TLS sessions
>  tlsprune  cmd="tls_prune" at=0400
> }
> 
> 
> I notice there is no SQUATTER entry in the new one - is it required, or is 
> the functionality no longer needed? Should I put a SQUATTER entry in the new 
> server's cyrus.conf?
> 
> Thanks
> Simon.
> 
> -- 
> Simon Wilson
> M: 0400 12 11 16
> 
> 
> Cyrus Home Page: http://www.cyrusimap.org/
> List Archives/Info: http://lists.andrew.cmu.edu/pipermail/info-cyrus/
> To Unsubscribe:
> https://lists.andrew.cmu.edu/mailman/listinfo/info-cyrus


smime.p7s
Description: S/MIME cryptographic signature

Cyrus Home Page: http://www.cyrusimap.org/
List Archives/Info: http://lists.andrew.cmu.edu/pipermail/info-cyrus/
To Unsubscribe:
https://lists.andrew.cmu.edu/mailman/listinfo/info-cyrus

Re: Squatter after upgrade

[Sebastian Hagedorn]

--On 28. April 2017 um 23:13:46 +1000 Simon Wilson  
wrote:



I notice there is no SQUATTER entry in the new one - is it required, or
is the functionality no longer needed? Should I put a SQUATTER entry in
the new server's cyrus.conf?


If you want to use the SQUAT index, you need to run squatter. That doesn't 
have to happen from cyrus.conf, though. We have a cronjob for that purpose 
that is a bit more involved.

--
   .:.Sebastian Hagedorn - Weyertal 121 (Gebäude 133), Zimmer 2.02.:.
.:.Regionales Rechenzentrum (RRZK).:.
  .:.Universität zu Köln / Cologne University - ✆ +49-221-470-89578.:.

Cyrus Home Page: http://www.cyrusimap.org/
List Archives/Info: http://lists.andrew.cmu.edu/pipermail/info-cyrus/
To Unsubscribe:
https://lists.andrew.cmu.edu/mailman/listinfo/info-cyrus

Squatter after upgrade

[Simon Wilson]

Hi list,

I have just upgraded from Cyrus 2.3.7 on a CentOS 5 server to 2.4.17.  
on a new CentOS 7 server.


On the old one events was:

EVENTS {
  # this is required
  checkpointcmd="ctl_cyrusdb -c" period=30

  # this is only necessary if using duplicate delivery suppression,
  # Sieve or NNTP
  delprune  cmd="cyr_expire -E 3" at=0400

  # this is only necessary if caching TLS sessions
  tlsprune  cmd="tls_prune" at=0400

  # running squatter
  squatter cmd="/usr/lib/cyrus-imapd/squatter -r user" period=1440
}

On the new one it's:

EVENTS {
  # this is required
  checkpointcmd="ctl_cyrusdb -c" period=30

  # this is only necessary if using duplicate delivery suppression,
  # Sieve or NNTP
  delprune  cmd="cyr_expire -E 3" at=0400

  # this is only necessary if caching TLS sessions
  tlsprune  cmd="tls_prune" at=0400
}


I notice there is no SQUATTER entry in the new one - is it required,  
or is the functionality no longer needed? Should I put a SQUATTER  
entry in the new server's cyrus.conf?


Thanks
Simon.

--
Simon Wilson
M: 0400 12 11 16


Cyrus Home Page: http://www.cyrusimap.org/
List Archives/Info: http://lists.andrew.cmu.edu/pipermail/info-cyrus/
To Unsubscribe:
https://lists.andrew.cmu.edu/mailman/listinfo/info-cyrus

Volume of metadata - squatter - in 2.5

[Marc Patermann via Info-cyrus]

Hi,

we are in the process of migrating from old cyrus imapd 2.3.16 servers 
to new with cyrus 2.5.10.


We copy whith imapsync - i.e. to benefit from single instance strore.

Now the volume of the metadata is increasing to a factor of like 7-15!
It seems squatter is taking a lot more space now.

Is this right?

one server:
new: 2,3G/var/lib/imap/meta
old: 153M/var/lib/imap/meta

another server:
new: 3,6G/var/lib/imap/meta
old: 563M/var/lib/imap/meta


Marc

Cyrus Home Page: http://www.cyrusimap.org/
List Archives/Info: http://lists.andrew.cmu.edu/pipermail/info-cyrus/
To Unsubscribe:
https://lists.andrew.cmu.edu/mailman/listinfo/info-cyrus

Re: Cyrus Squatter: No left space on device.

[Jean-Christophe Delaye]

On 15/06/15 16:06, Tecnología UNNOBA wrote:
> Hello!
>
> i'm using cyrus IMAPD since 11 years, on Debian VM. [0]
>
> Now, I found this problem and I can't find docs or comments about it on web:
>
> # cyrus squatter -riv user/jgcha...@comunidad.unnoba.edu.ar
> Indexing mailbox user/jgcha...@comunidad.unnoba.edu.ar... Writing index
> update: No space left on device
>
>
> [0]
> localhost> ver
> name   : Cyrus IMAPD
> version: v2.4.17-caldav-beta10-Debian-2.4.17+caldav~beta10-18
> 97aa1c46 2014-07-22
> vendor : Project Cyrus
> support-url: http://www.cyrusimap.org
> os : Linux
> os-version : 3.14-1-amd64
> environment: Built w/Cyrus SASL 2.1.26
>   Running w/Cyrus SASL 2.1.26
>   Built w/Berkeley DB 5.3.28: (September  9, 2013)
>   Running w/Berkeley DB 5.3.28: (September  9, 2013)
>   Built w/OpenSSL 1.0.1k 8 Jan 2015
>   Running w/OpenSSL 1.0.2a 19 Mar 2015
>   Built w/zlib 1.2.8
>   Running w/zlib 1.2.8
>   CMU Sieve 2.4
>   TCP Wrappers
>   NET-SNMP
>   mmap = shared
>   lock = fcntl
>   nonblock = fcntl
>   idle = poll
>
>
> * Partition has 1.3 TB and only 400 GB used.

May be full in inode space???
Have you tried df -i
>
>
>
> * User mailbox has correct permissions.
>
> mda:/var/spool/cyrus/mail/domain/c/comunidad.unnoba.edu.ar/j/user/jgcharne#
> ls -all
> drwx-- 1 cyrus mail  150 jun 15 10:38 .
> drwx-- 1 cyrus mail   16 abr 13 11:17 ..
> -rw--- 1 cyrus mail 1105 jun 15 10:22 7.
> -rw--- 1 cyrus mail 7412 jun 15 10:22 cyrus.cache
> -rw--- 1 cyrus mail  227 jun 15 10:15 cyrus.header
> -rw--- 1 cyrus mail  800 jun 15 10:26 cyrus.index
> -rw--- 1 cyrus mail  112 jun 15 10:17 cyrus.squat
> drwx-- 1 cyrus mail   90 jun 10 05:17 Drafts
> drwx-- 1 cyrus mail  122 jun 15 10:00 Sent
> drwx-- 1 cyrus mail   90 jun 10 05:17 Spam
> drwx-- 1 cyrus mail   90 jun 10 05:17 Templates
> drwx-- 1 cyrus mail   90 jun 15 10:16 Trash
>
>
>
> Someone had this problem?
>
> Thanks in advance.
>
> Javier.-
>
>
>
> 
> Cyrus Home Page: http://www.cyrusimap.org/
> List Archives/Info: http://lists.andrew.cmu.edu/pipermail/info-cyrus/
> To Unsubscribe:
> https://lists.andrew.cmu.edu/mailman/listinfo/info-cyrus
>



Cyrus Home Page: http://www.cyrusimap.org/
List Archives/Info: http://lists.andrew.cmu.edu/pipermail/info-cyrus/
To Unsubscribe:
https://lists.andrew.cmu.edu/mailman/listinfo/info-cyrus

Cyrus Squatter: No left space on device.

[Tecnología UNNOBA]

Hello!

i'm using cyrus IMAPD since 11 years, on Debian VM. [0]

Now, I found this problem and I can't find docs or comments about it on web:

# cyrus squatter -riv user/jgcha...@comunidad.unnoba.edu.ar
Indexing mailbox user/jgcha...@comunidad.unnoba.edu.ar... Writing index
update: No space left on device


[0]
localhost> ver
name   : Cyrus IMAPD
version: v2.4.17-caldav-beta10-Debian-2.4.17+caldav~beta10-18
97aa1c46 2014-07-22
vendor : Project Cyrus
support-url: http://www.cyrusimap.org
os : Linux
os-version : 3.14-1-amd64
environment: Built w/Cyrus SASL 2.1.26
 Running w/Cyrus SASL 2.1.26
 Built w/Berkeley DB 5.3.28: (September  9, 2013)
 Running w/Berkeley DB 5.3.28: (September  9, 2013)
 Built w/OpenSSL 1.0.1k 8 Jan 2015
 Running w/OpenSSL 1.0.2a 19 Mar 2015
 Built w/zlib 1.2.8
 Running w/zlib 1.2.8
 CMU Sieve 2.4
 TCP Wrappers
 NET-SNMP
 mmap = shared
 lock = fcntl
 nonblock = fcntl
 idle = poll


* Partition has 1.3 TB and only 400 GB used.



* User mailbox has correct permissions.

mda:/var/spool/cyrus/mail/domain/c/comunidad.unnoba.edu.ar/j/user/jgcharne#
ls -all
drwx-- 1 cyrus mail  150 jun 15 10:38 .
drwx-- 1 cyrus mail   16 abr 13 11:17 ..
-rw--- 1 cyrus mail 1105 jun 15 10:22 7.
-rw--- 1 cyrus mail 7412 jun 15 10:22 cyrus.cache
-rw--- 1 cyrus mail  227 jun 15 10:15 cyrus.header
-rw--- 1 cyrus mail  800 jun 15 10:26 cyrus.index
-rw--- 1 cyrus mail  112 jun 15 10:17 cyrus.squat
drwx-- 1 cyrus mail   90 jun 10 05:17 Drafts
drwx-- 1 cyrus mail  122 jun 15 10:00 Sent
drwx-- 1 cyrus mail   90 jun 10 05:17 Spam
drwx-- 1 cyrus mail   90 jun 10 05:17 Templates
drwx-- 1 cyrus mail   90 jun 15 10:16 Trash



Someone had this problem?

Thanks in advance.

Javier.-




Cyrus Home Page: http://www.cyrusimap.org/
List Archives/Info: http://lists.andrew.cmu.edu/pipermail/info-cyrus/
To Unsubscribe:
https://lists.andrew.cmu.edu/mailman/listinfo/info-cyrus

Re: Cyrus squatter using only a single core

[Michael Menge]

Hi,

nice does only have an influence if there are 2 or more processes
waiting for CPU-time. If the other processes are not using the cpu i
would guess that they are waiting for I/O, or are ideling.

See http://en.wikipedia.org/wiki/Nice_%28Unix%29 for more details.

Most Cyrus Processes are I/O bound and not CPU bound

Regards

   Michael Menge

Quoting "Fabio S. Schmidt" :


Hi Jose ! Thanks for answering !

Your explanations are very appropriate and I couldn't agree more !

In fact, I'm just concerned if squatter causing a high load on a core even
when running with nice 19 is the normal behavior, but I realize that it is
more a question about Linux than Cyrus.

--

My best regards,
Fabio Soares Schmidt



On 15 July 2014 16:19, Jose Luis Rodriguez Garcia 
wrote:


I am not a cyrux expert, but why are you worried that a core is loaded?

It means that the core is working on the task. Does the performance of the
server suffer because of this (response time to users, etc)?
I think that monitor for detect a high load in a core isn't appropriate
measure of performance. It can be useful for detected a hanged process that
takes all your cpu, or historical usages of the cores, but not for other
purposes.

I think that it is more appropriate measure the total usage of cpu of the
server.


On Tue, Jul 15, 2014 at 6:58 PM, Fabio S. Schmidt 
wrote:


Thank you Bron ! Once I unfortunately do not have the knowledgement
necessary to create this patch, I will adjust my configurations to run
squatter at specific times and disable the CPU consumption trigger on my
monitoring solution.

Even if when running with nice 19 my squatter process causes a high load
on the core which it's running, it this behavior normal?

Thanks everyone for answering this thread, I know that my english is not
so good but I'm trying to improve it !

--

My best regards,
Fabio Soares Schmidt



On 15 July 2014 03:03, Bron Gondwana  wrote:


 Yes, all Cyrus processes are single threaded except mupdate.  I don't
think that running squatter on multiple cores would necessarily be an
improvement - and regardless, it's not a priority for me to implement.
We'd accept a patch if someone did it and it integrated with the other
stuff that's floating around on the fastmail branch.

Bron.


On Tue, Jul 15, 2014, at 04:13 AM, Fabio S. Schmidt wrote:

Hi !

I'm running Cyrus 2.4.14 on Debian 6 64 Bits and I've noticed that when
squatter is running it only uses a single core. Has this behavior been
improved on newer versions?

Here is my entry:

squatter_1cmd="/usr/bin/nice +n 19 /usr/lib/cyrus/bin/squatter -s
-i -r user" period=180


I do run it with "nice +19" but It causes a high load on the core it's
running and triggers an alert on my monitoring solution. I know that this
alert could be deactivated, but maybe I'm doing something wrong with
squatter.

--

My best regards,
Fabio Soares Schmidt


Linux Professional Institute - LPIC-3
Microsoft Certified Technology Specialist: Active Directory

 
Cyrus Home Page: http://www.cyrusimap.org/
List Archives/Info: http://lists.andrew.cmu.edu/pipermail/info-cyrus/
To Unsubscribe:
https://lists.andrew.cmu.edu/mailman/listinfo/info-cyrus


--
Bron Gondwana
br...@fastmail.fm




Cyrus Home Page: http://www.cyrusimap.org/
List Archives/Info: http://lists.andrew.cmu.edu/pipermail/info-cyrus/
To Unsubscribe:
https://lists.andrew.cmu.edu/mailman/listinfo/info-cyrus




Cyrus Home Page: http://www.cyrusimap.org/
List Archives/Info: http://lists.andrew.cmu.edu/pipermail/info-cyrus/
To Unsubscribe:
https://lists.andrew.cmu.edu/mailman/listinfo/info-cyrus











M.MengeTel.: (49) 7071/29-70316
Universität Tübingen   Fax.: (49) 7071/29-5912
Zentrum für Datenverarbeitung  mail:  
michael.me...@zdv.uni-tuebingen.de

Wächterstraße 76
72074 Tübingen

smime.p7s
Description: S/MIME Signatur

Cyrus Home Page: http://www.cyrusimap.org/
List Archives/Info: http://lists.andrew.cmu.edu/pipermail/info-cyrus/
To Unsubscribe:
https://lists.andrew.cmu.edu/mailman/listinfo/info-cyrus

Re: Cyrus squatter using only a single core

[Fabio S. Schmidt]

Hi Jose ! Thanks for answering !

Your explanations are very appropriate and I couldn't agree more !

In fact, I'm just concerned if squatter causing a high load on a core even
when running with nice 19 is the normal behavior, but I realize that it is
more a question about Linux than Cyrus.

-- 

My best regards,
Fabio Soares Schmidt



On 15 July 2014 16:19, Jose Luis Rodriguez Garcia 
wrote:

> I am not a cyrux expert, but why are you worried that a core is loaded?
>
> It means that the core is working on the task. Does the performance of the
> server suffer because of this (response time to users, etc)?
> I think that monitor for detect a high load in a core isn't appropriate
> measure of performance. It can be useful for detected a hanged process that
> takes all your cpu, or historical usages of the cores, but not for other
> purposes.
>
> I think that it is more appropriate measure the total usage of cpu of the
> server.
>
>
> On Tue, Jul 15, 2014 at 6:58 PM, Fabio S. Schmidt 
> wrote:
>
>> Thank you Bron ! Once I unfortunately do not have the knowledgement
>> necessary to create this patch, I will adjust my configurations to run
>> squatter at specific times and disable the CPU consumption trigger on my
>> monitoring solution.
>>
>> Even if when running with nice 19 my squatter process causes a high load
>> on the core which it's running, it this behavior normal?
>>
>> Thanks everyone for answering this thread, I know that my english is not
>> so good but I'm trying to improve it !
>>
>> --
>>
>> My best regards,
>> Fabio Soares Schmidt
>>
>>
>>
>> On 15 July 2014 03:03, Bron Gondwana  wrote:
>>
>>>  Yes, all Cyrus processes are single threaded except mupdate.  I don't
>>> think that running squatter on multiple cores would necessarily be an
>>> improvement - and regardless, it's not a priority for me to implement.
>>> We'd accept a patch if someone did it and it integrated with the other
>>> stuff that's floating around on the fastmail branch.
>>>
>>> Bron.
>>>
>>>
>>> On Tue, Jul 15, 2014, at 04:13 AM, Fabio S. Schmidt wrote:
>>>
>>> Hi !
>>>
>>> I'm running Cyrus 2.4.14 on Debian 6 64 Bits and I've noticed that when
>>> squatter is running it only uses a single core. Has this behavior been
>>> improved on newer versions?
>>>
>>> Here is my entry:
>>>
>>> squatter_1cmd="/usr/bin/nice +n 19 /usr/lib/cyrus/bin/squatter -s
>>> -i -r user" period=180
>>>
>>>
>>> I do run it with "nice +19" but It causes a high load on the core it's
>>> running and triggers an alert on my monitoring solution. I know that this
>>> alert could be deactivated, but maybe I'm doing something wrong with
>>> squatter.
>>>
>>> --
>>>
>>> My best regards,
>>> Fabio Soares Schmidt
>>>
>>>
>>> Linux Professional Institute - LPIC-3
>>> Microsoft Certified Technology Specialist: Active Directory
>>>
>>>  
>>> Cyrus Home Page: http://www.cyrusimap.org/
>>> List Archives/Info: http://lists.andrew.cmu.edu/pipermail/info-cyrus/
>>> To Unsubscribe:
>>> https://lists.andrew.cmu.edu/mailman/listinfo/info-cyrus
>>>
>>>
>>> --
>>> Bron Gondwana
>>> br...@fastmail.fm
>>>
>>>
>>>
>>> 
>>> Cyrus Home Page: http://www.cyrusimap.org/
>>> List Archives/Info: http://lists.andrew.cmu.edu/pipermail/info-cyrus/
>>> To Unsubscribe:
>>> https://lists.andrew.cmu.edu/mailman/listinfo/info-cyrus
>>
>>
>> 
>> Cyrus Home Page: http://www.cyrusimap.org/
>> List Archives/Info: http://lists.andrew.cmu.edu/pipermail/info-cyrus/
>> To Unsubscribe:
>> https://lists.andrew.cmu.edu/mailman/listinfo/info-cyrus
>
>

Cyrus Home Page: http://www.cyrusimap.org/
List Archives/Info: http://lists.andrew.cmu.edu/pipermail/info-cyrus/
To Unsubscribe:
https://lists.andrew.cmu.edu/mailman/listinfo/info-cyrus

Re: Cyrus squatter using only a single core

[Jose Luis Rodriguez Garcia]

I am not a cyrux expert, but why are you worried that a core is loaded?

It means that the core is working on the task. Does the performance of the
server suffer because of this (response time to users, etc)?
I think that monitor for detect a high load in a core isn't appropriate
measure of performance. It can be useful for detected a hanged process that
takes all your cpu, or historical usages of the cores, but not for other
purposes.

I think that it is more appropriate measure the total usage of cpu of the
server.


On Tue, Jul 15, 2014 at 6:58 PM, Fabio S. Schmidt 
wrote:

> Thank you Bron ! Once I unfortunately do not have the knowledgement
> necessary to create this patch, I will adjust my configurations to run
> squatter at specific times and disable the CPU consumption trigger on my
> monitoring solution.
>
> Even if when running with nice 19 my squatter process causes a high load
> on the core which it's running, it this behavior normal?
>
> Thanks everyone for answering this thread, I know that my english is not
> so good but I'm trying to improve it !
>
> --
>
> My best regards,
> Fabio Soares Schmidt
>
>
>
> On 15 July 2014 03:03, Bron Gondwana  wrote:
>
>>  Yes, all Cyrus processes are single threaded except mupdate.  I don't
>> think that running squatter on multiple cores would necessarily be an
>> improvement - and regardless, it's not a priority for me to implement.
>> We'd accept a patch if someone did it and it integrated with the other
>> stuff that's floating around on the fastmail branch.
>>
>> Bron.
>>
>>
>> On Tue, Jul 15, 2014, at 04:13 AM, Fabio S. Schmidt wrote:
>>
>> Hi !
>>
>> I'm running Cyrus 2.4.14 on Debian 6 64 Bits and I've noticed that when
>> squatter is running it only uses a single core. Has this behavior been
>> improved on newer versions?
>>
>> Here is my entry:
>>
>> squatter_1cmd="/usr/bin/nice +n 19 /usr/lib/cyrus/bin/squatter -s -i
>> -r user" period=180
>>
>>
>> I do run it with "nice +19" but It causes a high load on the core it's
>> running and triggers an alert on my monitoring solution. I know that this
>> alert could be deactivated, but maybe I'm doing something wrong with
>> squatter.
>>
>> --
>>
>> My best regards,
>> Fabio Soares Schmidt
>>
>>
>> Linux Professional Institute - LPIC-3
>> Microsoft Certified Technology Specialist: Active Directory
>>
>>  
>> Cyrus Home Page: http://www.cyrusimap.org/
>> List Archives/Info: http://lists.andrew.cmu.edu/pipermail/info-cyrus/
>> To Unsubscribe:
>> https://lists.andrew.cmu.edu/mailman/listinfo/info-cyrus
>>
>>
>> --
>> Bron Gondwana
>> br...@fastmail.fm
>>
>>
>>
>> 
>> Cyrus Home Page: http://www.cyrusimap.org/
>> List Archives/Info: http://lists.andrew.cmu.edu/pipermail/info-cyrus/
>> To Unsubscribe:
>> https://lists.andrew.cmu.edu/mailman/listinfo/info-cyrus
>
>
> 
> Cyrus Home Page: http://www.cyrusimap.org/
> List Archives/Info: http://lists.andrew.cmu.edu/pipermail/info-cyrus/
> To Unsubscribe:
> https://lists.andrew.cmu.edu/mailman/listinfo/info-cyrus
>

Cyrus Home Page: http://www.cyrusimap.org/
List Archives/Info: http://lists.andrew.cmu.edu/pipermail/info-cyrus/
To Unsubscribe:
https://lists.andrew.cmu.edu/mailman/listinfo/info-cyrus

Re: Cyrus squatter using only a single core

[Fabio S. Schmidt]

Thank you Bron ! Once I unfortunately do not have the knowledgement
necessary to create this patch, I will adjust my configurations to run
squatter at specific times and disable the CPU consumption trigger on my
monitoring solution.

Even if when running with nice 19 my squatter process causes a high load on
the core which it's running, it this behavior normal?

Thanks everyone for answering this thread, I know that my english is not so
good but I'm trying to improve it !

-- 

My best regards,
Fabio Soares Schmidt



On 15 July 2014 03:03, Bron Gondwana  wrote:

>  Yes, all Cyrus processes are single threaded except mupdate.  I don't
> think that running squatter on multiple cores would necessarily be an
> improvement - and regardless, it's not a priority for me to implement.
> We'd accept a patch if someone did it and it integrated with the other
> stuff that's floating around on the fastmail branch.
>
> Bron.
>
>
> On Tue, Jul 15, 2014, at 04:13 AM, Fabio S. Schmidt wrote:
>
> Hi !
>
> I'm running Cyrus 2.4.14 on Debian 6 64 Bits and I've noticed that when
> squatter is running it only uses a single core. Has this behavior been
> improved on newer versions?
>
> Here is my entry:
>
> squatter_1cmd="/usr/bin/nice +n 19 /usr/lib/cyrus/bin/squatter -s -i
> -r user" period=180
>
>
> I do run it with "nice +19" but It causes a high load on the core it's
> running and triggers an alert on my monitoring solution. I know that this
> alert could be deactivated, but maybe I'm doing something wrong with
> squatter.
>
> --
>
> My best regards,
> Fabio Soares Schmidt
>
>
> Linux Professional Institute - LPIC-3
> Microsoft Certified Technology Specialist: Active Directory
>
>  
> Cyrus Home Page: http://www.cyrusimap.org/
> List Archives/Info: http://lists.andrew.cmu.edu/pipermail/info-cyrus/
> To Unsubscribe:
> https://lists.andrew.cmu.edu/mailman/listinfo/info-cyrus
>
>
> --
> Bron Gondwana
> br...@fastmail.fm
>
>
>
> 
> Cyrus Home Page: http://www.cyrusimap.org/
> List Archives/Info: http://lists.andrew.cmu.edu/pipermail/info-cyrus/
> To Unsubscribe:
> https://lists.andrew.cmu.edu/mailman/listinfo/info-cyrus

Cyrus Home Page: http://www.cyrusimap.org/
List Archives/Info: http://lists.andrew.cmu.edu/pipermail/info-cyrus/
To Unsubscribe:
https://lists.andrew.cmu.edu/mailman/listinfo/info-cyrus

Re: Cyrus squatter using only a single core

[Bron Gondwana]

Yes, all Cyrus processes are single threaded except mupdate.  I don't think 
that running squatter on multiple cores would necessarily be an improvement - 
and regardless, it's not a priority for me to implement.  We'd accept a patch 
if someone did it and it integrated with the other stuff that's floating around 
on the fastmail branch.



Bron.





On Tue, Jul 15, 2014, at 04:13 AM, Fabio S. Schmidt wrote:

Hi !

I'm running Cyrus 2.4.14 on Debian 6 64 Bits and I've noticed that when 
squatter is running it only uses a single core. Has this behavior been improved 
on newer versions?

Here is my entry:

squatter_1cmd="/usr/bin/nice +n 19 /usr/lib/cyrus/bin/squatter -s -i -r 
user" period=180


I do run it with "nice +19" but It causes a high load on the core it's running 
and triggers an alert on my monitoring solution. I know that this alert could 
be deactivated, but maybe I'm doing something wrong with squatter.

--
My best regards,
Fabio Soares Schmidt


Linux Professional Institute - LPIC-3
Microsoft Certified Technology Specialist: Active Directory



Cyrus Home Page: [1]http://www.cyrusimap.org/

List Archives/Info: [2]http://lists.andrew.cmu.edu/pipermail/info-cyrus/

To Unsubscribe:

[3]https://lists.andrew.cmu.edu/mailman/listinfo/info-cyrus



--
Bron Gondwana
br...@fastmail.fm

References

1. http://www.cyrusimap.org/
2. http://lists.andrew.cmu.edu/pipermail/info-cyrus/
3. https://lists.andrew.cmu.edu/mailman/listinfo/info-cyrus

Cyrus Home Page: http://www.cyrusimap.org/
List Archives/Info: http://lists.andrew.cmu.edu/pipermail/info-cyrus/
To Unsubscribe:
https://lists.andrew.cmu.edu/mailman/listinfo/info-cyrus

Re: Cyrus squatter using only a single core

[Fabio S. Schmidt]

Hi Menge ! Thanks for the tips !

I realize that running squatter every 3h is too often, once my enviroment
is still growning, we had around a tousand users but the predictions are to
have 50.000 users in a month or two. We are planning to change it to run
once each night and maybe at specific times with less IO traffic. After
changing the squatter entry on cyrus.conf do I need to restart Cyrus or
just a reload is fine ?


On 14 July 2014 17:12, Michael Menge 
wrote:

> Hi,
>
>
> Quoting "Fabio S. Schmidt" :
>
>  Hi !
>>
>> I'm running Cyrus 2.4.14 on Debian 6 64 Bits and I've noticed that when
>> squatter is running it only uses a single core. Has this behavior been
>> improved on newer versions?
>>
>> Here is my entry:
>>
>> squatter_1cmd="/usr/bin/nice +n 19 /usr/lib/cyrus/bin/squatter -s -i
>> -r
>> user" period=180
>>
>>
> i think you are running the squatter to often.
>
> period=180 will run it every 3h. If there is more then one sqatter running,
> it second will catch up and the proceses will have to wait for the mailbox
> lock.
>
> We run our squatter, expire and backup once each night, when there is less
> IO traffic.
>
> you can use at= instead of period=. See cyrus.conf manpage for details
>
>
> Regards
>
>Michael Menge
>
>
>
>
>
>
>> I do run it with "nice +19" but It causes a high load on the core it's
>> running and triggers an alert on my monitoring solution. I know that this
>> alert could be deactivated, but maybe I'm doing something wrong with
>> squatter.
>>
>> --
>>
>> My best regards,
>> Fabio Soares Schmidt
>>
>>
>> Linux Professional Institute - LPIC-3
>> Microsoft Certified Technology Specialist: Active Directory
>>
>>
>
>
>
> 
> 
> M.MengeTel.: (49) 7071/29-70316
> Universität Tübingen   Fax.: (49) 7071/29-5912
> Zentrum für Datenverarbeitung  mail: michael.me...@zdv.uni-
> tuebingen.de
> Wächterstraße 76
> 72074 Tübingen
> 
> Cyrus Home Page: http://www.cyrusimap.org/
> List Archives/Info: http://lists.andrew.cmu.edu/pipermail/info-cyrus/
> To Unsubscribe:
> https://lists.andrew.cmu.edu/mailman/listinfo/info-cyrus
>



-- 

My best regards,
Fabio Soares Schmidt


Linux Professional Institute - LPIC-3
Microsoft Certified Technology Specialist: Active Directory

Cyrus Home Page: http://www.cyrusimap.org/
List Archives/Info: http://lists.andrew.cmu.edu/pipermail/info-cyrus/
To Unsubscribe:
https://lists.andrew.cmu.edu/mailman/listinfo/info-cyrus

Re: Cyrus squatter using only a single core

[Fabio S. Schmidt]

Hi Melander !

I think that "squatter -s -i -r user" launches a recursive process on all
users, am I wrong?


On 14 July 2014 16:41, Mogens Melander  wrote:

> Hi
>
> Did you launch a recursive process on all users, or did you
> launch a process per user ?
>
> On Mon, July 14, 2014 20:13, Fabio S. Schmidt wrote:
> > Hi !
> >
> > I'm running Cyrus 2.4.14 on Debian 6 64 Bits and I've noticed that when
> > squatter is running it only uses a single core. Has this behavior been
> > improved on newer versions?
> >
> > Here is my entry:
> >
> > squatter_1cmd="/usr/bin/nice +n 19 /usr/lib/cyrus/bin/squatter -s -i
> > -r
> > user" period=180
> >
> >
> > I do run it with "nice +19" but It causes a high load on the core it's
> > running and triggers an alert on my monitoring solution. I know that this
> > alert could be deactivated, but maybe I'm doing something wrong with
> > squatter.
> >
> > --
> >
> > My best regards,
> > Fabio Soares Schmidt
> >
> >
> > Linux Professional Institute - LPIC-3
> > Microsoft Certified Technology Specialist: Active Directory
>
> --
> Mogens Melander
> +66 8701 33224
>
>
> --
> This message has been scanned for viruses and
> dangerous content by MailScanner, and is
> believed to be clean.
>
> 
> Cyrus Home Page: http://www.cyrusimap.org/
> List Archives/Info: http://lists.andrew.cmu.edu/pipermail/info-cyrus/
> To Unsubscribe:
> https://lists.andrew.cmu.edu/mailman/listinfo/info-cyrus
>



-- 

My best regards,
Fabio Soares Schmidt


Linux Professional Institute - LPIC-3
Microsoft Certified Technology Specialist: Active Directory

Cyrus Home Page: http://www.cyrusimap.org/
List Archives/Info: http://lists.andrew.cmu.edu/pipermail/info-cyrus/
To Unsubscribe:
https://lists.andrew.cmu.edu/mailman/listinfo/info-cyrus

Re: Cyrus squatter using only a single core

[Michael Menge]

Hi,

Quoting "Fabio S. Schmidt" :


Hi !

I'm running Cyrus 2.4.14 on Debian 6 64 Bits and I've noticed that when
squatter is running it only uses a single core. Has this behavior been
improved on newer versions?

Here is my entry:

squatter_1cmd="/usr/bin/nice +n 19 /usr/lib/cyrus/bin/squatter -s -i -r
user" period=180



i think you are running the squatter to often.

period=180 will run it every 3h. If there is more then one sqatter running,
it second will catch up and the proceses will have to wait for the mailbox
lock.

We run our squatter, expire and backup once each night, when there is less
IO traffic.

you can use at= instead of period=. See cyrus.conf manpage for details


Regards

   Michael Menge






I do run it with "nice +19" but It causes a high load on the core it's
running and triggers an alert on my monitoring solution. I know that this
alert could be deactivated, but maybe I'm doing something wrong with
squatter.

--

My best regards,
Fabio Soares Schmidt


Linux Professional Institute - LPIC-3
Microsoft Certified Technology Specialist: Active Directory







M.MengeTel.: (49) 7071/29-70316
Universität Tübingen   Fax.: (49) 7071/29-5912
Zentrum für Datenverarbeitung  mail:  
michael.me...@zdv.uni-tuebingen.de

Wächterstraße 76
72074 Tübingen

smime.p7s
Description: S/MIME Signatur

Cyrus Home Page: http://www.cyrusimap.org/
List Archives/Info: http://lists.andrew.cmu.edu/pipermail/info-cyrus/
To Unsubscribe:
https://lists.andrew.cmu.edu/mailman/listinfo/info-cyrus

Re: Cyrus squatter using only a single core

[Mogens Melander]

Hi

Did you launch a recursive process on all users, or did you
launch a process per user ?

On Mon, July 14, 2014 20:13, Fabio S. Schmidt wrote:
> Hi !
>
> I'm running Cyrus 2.4.14 on Debian 6 64 Bits and I've noticed that when
> squatter is running it only uses a single core. Has this behavior been
> improved on newer versions?
>
> Here is my entry:
>
> squatter_1cmd="/usr/bin/nice +n 19 /usr/lib/cyrus/bin/squatter -s -i
> -r
> user" period=180
>
>
> I do run it with "nice +19" but It causes a high load on the core it's
> running and triggers an alert on my monitoring solution. I know that this
> alert could be deactivated, but maybe I'm doing something wrong with
> squatter.
>
> --
>
> My best regards,
> Fabio Soares Schmidt
>
>
> Linux Professional Institute - LPIC-3
> Microsoft Certified Technology Specialist: Active Directory

-- 
Mogens Melander
+66 8701 33224


-- 
This message has been scanned for viruses and
dangerous content by MailScanner, and is
believed to be clean.


Cyrus Home Page: http://www.cyrusimap.org/
List Archives/Info: http://lists.andrew.cmu.edu/pipermail/info-cyrus/
To Unsubscribe:
https://lists.andrew.cmu.edu/mailman/listinfo/info-cyrus

Cyrus squatter using only a single core

[Fabio S. Schmidt]

Hi !

I'm running Cyrus 2.4.14 on Debian 6 64 Bits and I've noticed that when
squatter is running it only uses a single core. Has this behavior been
improved on newer versions?

Here is my entry:

squatter_1cmd="/usr/bin/nice +n 19 /usr/lib/cyrus/bin/squatter -s -i -r
user" period=180


I do run it with "nice +19" but It causes a high load on the core it's
running and triggers an alert on my monitoring solution. I know that this
alert could be deactivated, but maybe I'm doing something wrong with
squatter.

-- 

My best regards,
Fabio Soares Schmidt


Linux Professional Institute - LPIC-3
Microsoft Certified Technology Specialist: Active Directory

Cyrus Home Page: http://www.cyrusimap.org/
List Archives/Info: http://lists.andrew.cmu.edu/pipermail/info-cyrus/
To Unsubscribe:
https://lists.andrew.cmu.edu/mailman/listinfo/info-cyrus

RE: Still getting SQUAT errors after adding squatter to events

[Joshua Battles]

On Feb 12, 2014 4:25 PM, "Dudi Goldenberg"  wrote:
>
> >I thought this was required to get it to index subfolders also? Am I
reading the man page wrong?
> >Thanks,
> >Josh
>
> From squatter manpage:
>
> "The mailbox argument can be omitted, which indexes all mailboxes.
Otherwise it needs to be specified in the same format as used in cyradm."
>
> So you can run it with -r with user specified to index all mailboxes
including subs.
>
> Regards,
>
> D.

In the previous message you told me to remove those options and now you're
telling me I need them. Which is it?

I'm really confused now.  I need to index all the mailboxes and their sub
folders.  What's the correct way to accomplish this?

Thanks,
Josh

Cyrus Home Page: http://www.cyrusimap.org/
List Archives/Info: http://lists.andrew.cmu.edu/pipermail/info-cyrus/
To Unsubscribe:
https://lists.andrew.cmu.edu/mailman/listinfo/info-cyrus

RE: Still getting SQUAT errors after adding squatter to events

[Dudi Goldenberg]

>So you can run it with -r with user specified to index all mailboxes including 
>subs.

Excuse the typo... should of course be:

"So you can run it with -r with no user" etc.

Regards,

D.


Cyrus Home Page: http://www.cyrusimap.org/
List Archives/Info: http://lists.andrew.cmu.edu/pipermail/info-cyrus/
To Unsubscribe:
https://lists.andrew.cmu.edu/mailman/listinfo/info-cyrus

RE: Still getting SQUAT errors after adding squatter to events

[Dudi Goldenberg]

>I thought this was required to get it to index subfolders also? Am I reading 
>the man page wrong?
>Thanks,
>Josh

>From squatter manpage:

"The mailbox argument can be omitted, which indexes all mailboxes. Otherwise it 
needs to be specified in the same format as used in cyradm."

So you can run it with -r with user specified to index all mailboxes including 
subs.

Regards,

D.

Cyrus Home Page: http://www.cyrusimap.org/
List Archives/Info: http://lists.andrew.cmu.edu/pipermail/info-cyrus/
To Unsubscribe:
https://lists.andrew.cmu.edu/mailman/listinfo/info-cyrus

RE: Still getting SQUAT errors after adding squatter to events

[Joshua Battles]

On Feb 12, 2014 2:58 PM, "Dan White"  wrote:
>
>> The entry has been in the config for 24 hours and I'm still seeing squat
>> errors. Here is the line I've added:
>> squatter   cmd="/use/sbin/squatter -r user" period=30
>
>
> You have a typo here, in the path.

Good catch on the path, however it was ok in the file.  Spell check must've
gotten it when I sent the email to the list.

On Feb 12, 2014 3:02 PM, "Dudi Goldenberg"  wrote:
>
> >I've added it as an event in Cyrus.conf but I am still getting the
errors. The entry has been in the config for 24 hours and I'm still seeing
squat errors. Here is the line I've added:
>
> >squatter   cmd="/use/sbin/squatter -r user" period=30
>
>
>
> Just omit "-r user" to index all mailboxes, like:
>
>
> squatter  cmd="/usr/sbin/squatter" at=0520
>
>
> Regards,
>
> D

I thought this was required to get it to index subfolders also? Am I
reading the man page wrong?

Thanks,
Josh

Cyrus Home Page: http://www.cyrusimap.org/
List Archives/Info: http://lists.andrew.cmu.edu/pipermail/info-cyrus/
To Unsubscribe:
https://lists.andrew.cmu.edu/mailman/listinfo/info-cyrus

RE: Still getting SQUAT errors after adding squatter to events

[Dudi Goldenberg]

>I've added it as an event in Cyrus.conf but I am still getting the errors. The 
>entry has been in the config for 24 hours and I'm still seeing squat errors. 
>Here is the line I've added:

>squatter   cmd="/use/sbin/squatter -r user" period=30



Just omit "-r user" to index all mailboxes, like:



squatter  cmd="/usr/sbin/squatter" at=0520





Regards,



D.


Cyrus Home Page: http://www.cyrusimap.org/
List Archives/Info: http://lists.andrew.cmu.edu/pipermail/info-cyrus/
To Unsubscribe:
https://lists.andrew.cmu.edu/mailman/listinfo/info-cyrus

Re: Still getting SQUAT errors after adding squatter to events

[Dan White]

On 02/12/14 13:36 -0600, Joshua Battles wrote:
>Hello,
>I am maintaining a mail server for a very active group of users with large
>mailboxes and am having trouble getting squatter to run.
>
>I've added it as an event in Cyrus.conf but I am still getting the errors.
>The entry has been in the config for 24 hours and I'm still seeing squat
>errors. Here is the line I've added:
>squatter   cmd="/use/sbin/squatter -r user" period=30

You have a typo here, in the path.

>Squatter resides in the sbin directory.
>
>What should I be checking to figure out why it isn't running? I didn't see
>anything in the logs.
>
>Does it matter that the cyrus user isn't "cyrus" ?
>
>I'm new to cyrus and was handed this server already in use so pardon my
>ignorance.
>
>Thanks,
>Josh

-- 
Dan White

Cyrus Home Page: http://www.cyrusimap.org/
List Archives/Info: http://lists.andrew.cmu.edu/pipermail/info-cyrus/
To Unsubscribe:
https://lists.andrew.cmu.edu/mailman/listinfo/info-cyrus

Still getting SQUAT errors after adding squatter to events

[Joshua Battles]

Hello,
I am maintaining a mail server for a very active group of users with large
mailboxes and am having trouble getting squatter to run.

I've added it as an event in Cyrus.conf but I am still getting the errors.
The entry has been in the config for 24 hours and I'm still seeing squat
errors. Here is the line I've added:
squatter   cmd="/use/sbin/squatter -r user" period=30

Squatter resides in the sbin directory.

What should I be checking to figure out why it isn't running? I didn't see
anything in the logs.

Does it matter that the cyrus user isn't "cyrus" ?

I'm new to cyrus and was handed this server already in use so pardon my
ignorance.

Thanks,
Josh

Cyrus Home Page: http://www.cyrusimap.org/
List Archives/Info: http://lists.andrew.cmu.edu/pipermail/info-cyrus/
To Unsubscribe:
https://lists.andrew.cmu.edu/mailman/listinfo/info-cyrus

Re: Squatter crash with statusdb

[Jeroen van Meeuwen (Kolab Systems)]

On 2013-06-24 13:24, Simon Matter wrote:
>> Hi Andy, could you file a bug for this?  Then it will not be 
>> forgotten...
> 
> Or, could you check this bug here
> http://bugzilla.cyrusimap.org/show_bug.cgi?id=3757
> 
> The patch below was the fix, could you verify if it also fixes your 
> issue?
> 
> Thanks,
> Simon
> 
>> From 1661683d453ea444aae5832b4a2cb7fd54489672 Mon Sep 17 00:00:00 2001
> From: Bron Gondwana 
> Date: Sun, 09 Dec 2012 19:42:17 +
> Subject: Bug #3757 - don't segfault on mailbox close with no user
> 

This seems to still apply indeed, barring a DB->foreach() => 
cyrusdb_foreach() merge conflict.

I have it in:

   [master 9766afa] Bug #3757 - don't segfault on mailbox close with no 
user

so it'll be in Cyrus IMAP 2.5 when I push (I have a couple of other 
things to push as well).

Kind regards,

Jeroen van Meeuwen

-- 
Systems Architect, Kolab Systems AG

e: vanmeeuwen at kolabsys.com
m: +44 74 2516 3817
w: http://www.kolabsys.com

pgp: 9342 BF08

Cyrus Home Page: http://www.cyrusimap.org/
List Archives/Info: http://lists.andrew.cmu.edu/pipermail/info-cyrus/
To Unsubscribe:
https://lists.andrew.cmu.edu/mailman/listinfo/info-cyrus

Re: Squatter crash with statusdb

[Andy Fiddaman]

On Mon, 24 Jun 2013, Simon Matter wrote:
;
; Or, could you check this bug here
; http://bugzilla.cyrusimap.org/show_bug.cgi?id=3757
;
; The patch below was the fix, could you verify if it also fixes your issue?

Yes that patch fixes my issue too. I did search bugzilla but didn't
see that one because it is marked as resolved. I didn't see it fixed
in git either (http://git.cyrusimap.org/cyrus-imapd/tree/imap/statuscache_db.c)
but I see now it's fixed in the 2.4 branch only.

Thanks,

Andy


Cyrus Home Page: http://www.cyrusimap.org/
List Archives/Info: http://lists.andrew.cmu.edu/pipermail/info-cyrus/
To Unsubscribe:
https://lists.andrew.cmu.edu/mailman/listinfo/info-cyrus

Re: Squatter crash with statusdb

[Simon Matter]

> Hi Andy, could you file a bug for this?  Then it will not be forgotten...

Or, could you check this bug here
http://bugzilla.cyrusimap.org/show_bug.cgi?id=3757

The patch below was the fix, could you verify if it also fixes your issue?

Thanks,
Simon

>From 1661683d453ea444aae5832b4a2cb7fd54489672 Mon Sep 17 00:00:00 2001
From: Bron Gondwana 
Date: Sun, 09 Dec 2012 19:42:17 +
Subject: Bug #3757 - don't segfault on mailbox close with no user

Broke squatter and possibly other users as well.

This is probably a 2.4 only fix - the code has changed
for statuscache a bit since.
---
diff --git a/imap/statuscache_db.c b/imap/statuscache_db.c
index fadc58d..41dfd2b 100644
--- a/imap/statuscache_db.c
+++ b/imap/statuscache_db.c
@@ -150,9 +150,11 @@ static char *statuscache_buildkey(const char
*mailboxname, const char *userid,

 /* Build statuscache key */
 len = strlcpy(key, mailboxname, sizeof(key));
+/* double % is a safe separator, it can't exist in a mailboxname */
 key[len++] = '%';
 key[len++] = '%';
-len += strlcpy(key + len, userid, sizeof(key) - len);
+if (userid)
+   len += strlcpy(key + len, userid, sizeof(key) - len);

 *keylen = len;

@@ -410,11 +412,9 @@ int statuscache_invalidate(const char *mboxname,
struct statusdata *sdata)
 drock.db = statuscachedb;
 drock.tid = NULL;

-key = statuscache_buildkey(mboxname, "", &keylen);
+key = statuscache_buildkey(mboxname, /*userid*/NULL, &keylen);

-/* strip off the second NULL that buildkey added, so we match
- * the entires for all users */
-r = DB->foreach(drock.db, key, keylen - 1, NULL, delete_cb,
+r = DB->foreach(drock.db, key, keylen, NULL, delete_cb,
&drock, &drock.tid);
 if (r != CYRUSDB_OK) {
syslog(LOG_ERR, "DBERROR: error invalidating: %s (%s)",


>
> Quoting Andy Fiddaman , Mon, 24 Jun 2013:
>
>> FWIW, this gets it working again:
>>
>> --- cyrus-imapd-2.4.17.dist/imap/statuscache_db.c   2013-06-24
>> 10:10:08.219203100 +
>> +++ cyrus-imapd-2.4.17/imap/statuscache_db.c2013-06-24
>> 10:10:20.537711377 +
>> @@ -152,7 +152,7 @@
>>  len = strlcpy(key, mailboxname, sizeof(key));
>>  key[len++] = '%';
>>  key[len++] = '%';
>> -len += strlcpy(key + len, userid, sizeof(key) - len);
>> +len += strlcpy(key + len, userid ? userid : "cyrus", sizeof(key) -
>> len);
>>
>>  *keylen = len;
>>
>>
>> On Mon, 24 Jun 2013, Andy Fiddaman wrote:
>>
>> ;
>> ; Hi,
>> ;
>> ; I've just upgraded my Cyrus installation to 2.4.17 and squatter is
>> ; crashing in statuscache_buildkey() because userid is NULL.
>> ;
>> ; I'm not sure what the best fix for this is. Should squatter even be
>> using
>> ; the statuscache or should it populate "cyrus" as the username when
>> ; initialising the index, or something else?
>> ;
>> ; Thanks,
>> ;
>> ; Andy
>> ;
>> ; Program received signal SIGSEGV, Segmentation fault.
>> ; [Switching to Thread 1 (LWP 1)]
>> ; 0xfd7ffe3dccb0 in .strlenalign16_loop () from /lib/64/libc.so.1
>> ; (gdb) where
>> ; #0  0xfd7ffe3dccb0 in .strlenalign16_loop () from
>> /lib/64/libc.so.1
>> ; #1  0xfd7ffe414149 in strlcpy () from /lib/64/libc.so.1
>> ; #2  0x004610ac in statuscache_buildkey (
>> ; mailboxname=0x5ab8b0 "example.net!user.silo", userid=0x0,
>> ; keylen=0xfd7fffdfe0cc) at statuscache_db.c:155
>> ; #3  0x0046169a in statuscache_update_txn (
>> ; mboxname=0x5ab8b0 "example.net!user.silo",
>> sdata=0xfd7fffdfe290,
>> ; tidptr=0xfd7fffdfe218) at statuscache_db.c:326
>> ; #4  0x004619ad in statuscache_invalidate (
>> ; mboxname=0x5ab8b0 "example.net!user.silo",
>> sdata=0xfd7fffdfe290)
>> ; at statuscache_db.c:425
>> ; #5  0x00434099 in mailbox_unlock_index (mailbox=0x5db998,
>> ; sdata=0xfd7fffdfe290) at mailbox.c:1637
>> ; #6  0x00422feb in index_unlock (state=0x5af6a0) at
>> index.c:1232
>> ; #7  0x00420c98 in index_open (name=0x5ac960
>> ; "example.net!user.silo",
>> ; init=0x0, stateptr=0xfd7fffdfec18) at index.c:246
>> ; #8  0x00420395 in index_me (name=0x5ac960
>> "example.net!user.silo",
>> ; matchlen=20, maycreate=0, rock=0xfd7fffdffcac) at
>> squatter.c:594
>> ; #9  0x004208f9 in main (argc=3, argv=0xfd7fffdffcf8)
>> ; at squatter.c:745
>> ;
>> ;
>> ; (gdb) frame 7
>> ;

Re: Squatter crash with statusdb

[Rudy Gevaert]

Hi Andy, could you file a bug for this?  Then it will not be forgotten...

Quoting Andy Fiddaman , Mon, 24 Jun 2013:

> FWIW, this gets it working again:
>
> --- cyrus-imapd-2.4.17.dist/imap/statuscache_db.c   2013-06-24
> 10:10:08.219203100 +
> +++ cyrus-imapd-2.4.17/imap/statuscache_db.c2013-06-24
> 10:10:20.537711377 +
> @@ -152,7 +152,7 @@
>  len = strlcpy(key, mailboxname, sizeof(key));
>  key[len++] = '%';
>  key[len++] = '%';
> -len += strlcpy(key + len, userid, sizeof(key) - len);
> +len += strlcpy(key + len, userid ? userid : "cyrus", sizeof(key) -
> len);
>
>  *keylen = len;
>
>
> On Mon, 24 Jun 2013, Andy Fiddaman wrote:
>
> ;
> ; Hi,
> ;
> ; I've just upgraded my Cyrus installation to 2.4.17 and squatter is
> ; crashing in statuscache_buildkey() because userid is NULL.
> ;
> ; I'm not sure what the best fix for this is. Should squatter even be using
> ; the statuscache or should it populate "cyrus" as the username when
> ; initialising the index, or something else?
> ;
> ; Thanks,
> ;
> ; Andy
> ;
> ; Program received signal SIGSEGV, Segmentation fault.
> ; [Switching to Thread 1 (LWP 1)]
> ; 0xfd7ffe3dccb0 in .strlenalign16_loop () from /lib/64/libc.so.1
> ; (gdb) where
> ; #0  0xfd7ffe3dccb0 in .strlenalign16_loop () from /lib/64/libc.so.1
> ; #1  0xfd7ffe414149 in strlcpy () from /lib/64/libc.so.1
> ; #2  0x004610ac in statuscache_buildkey (
> ; mailboxname=0x5ab8b0 "example.net!user.silo", userid=0x0,
> ; keylen=0xfd7fffdfe0cc) at statuscache_db.c:155
> ; #3  0x0046169a in statuscache_update_txn (
> ; mboxname=0x5ab8b0 "example.net!user.silo", sdata=0xfd7fffdfe290,
> ; tidptr=0xfd7fffdfe218) at statuscache_db.c:326
> ; #4  0x004619ad in statuscache_invalidate (
> ; mboxname=0x5ab8b0 "example.net!user.silo", sdata=0xfd7fffdfe290)
> ; at statuscache_db.c:425
> ; #5  0x00434099 in mailbox_unlock_index (mailbox=0x5db998,
> ; sdata=0xfd7fffdfe290) at mailbox.c:1637
> ; #6  0x00422feb in index_unlock (state=0x5af6a0) at index.c:1232
> ; #7  0x00420c98 in index_open (name=0x5ac960
> ; "example.net!user.silo",
> ; init=0x0, stateptr=0xfd7fffdfec18) at index.c:246
> ; #8  0x00420395 in index_me (name=0x5ac960 "example.net!user.silo",
> ; matchlen=20, maycreate=0, rock=0xfd7fffdffcac) at squatter.c:594
> ; #9  0x004208f9 in main (argc=3, argv=0xfd7fffdffcf8)
> ; at squatter.c:745
> ;
> ;
> ; (gdb) frame 7
> ; #7  0x00420c98 in index_open (name=0x5ac960
> ; "example.net!user.silo",
> ; init=0x0, stateptr=0xfd7fffdfec18) at index.c:246
> ; 246 in index.c
> ; (gdb) print *state
> ; $4 = {mailbox = 0x5db998, num_records = 6933, oldexists = 0, exists =
> ; 6933,
> ;   last_uid = 6934, highestmodseq = 1606, delayed_modseq = 0, map =
> ; 0x5dd740,
> ;   mapsize = 7168, internalseen = 0, skipped_expunge = 0, seen_dirty = 0,
> ;   keepingseen = 0, examining = 0, myrights = 0, numrecent = 0,
> ;   numunseen = 6933, firstnotseen = 1, flagname = {0x0  ; times>},
> ;   userid = 0x0, out = 0x0, qresync = 0, authstate = 0x0}
> ;
> ; 
> ; Cyrus Home Page: http://www.cyrusimap.org/
> ; List Archives/Info: http://lists.andrew.cmu.edu/pipermail/info-cyrus/
> ; To Unsubscribe:
> ; https://lists.andrew.cmu.edu/mailman/listinfo/info-cyrus
> ;
> 
> Cyrus Home Page: http://www.cyrusimap.org/
> List Archives/Info: http://lists.andrew.cmu.edu/pipermail/info-cyrus/
> To Unsubscribe:
> https://lists.andrew.cmu.edu/mailman/listinfo/info-cyrus

-- 
  -- -- -- -- -- -- -- -- -- -- -- -- -- -- -- -- -- -- -- -- -- -- -- --
  Rudy Gevaert e-mail: rudy.geva...@ugent.be
  Directie ICT, Afdeling Infrastructuur
  Groep Systemen  tel: +32 9 264 4750
  Universiteit Gent   fax: +32 9 264 4994
  Krijgslaan 281, gebouw S9, 9000 Gent, Belgie   www.UGent.be
  -- -- -- -- -- -- -- -- -- -- -- -- -- -- -- -- -- -- -- -- -- -- -- --



Cyrus Home Page: http://www.cyrusimap.org/
List Archives/Info: http://lists.andrew.cmu.edu/pipermail/info-cyrus/
To Unsubscribe:
https://lists.andrew.cmu.edu/mailman/listinfo/info-cyrus

Re: Squatter crash with statusdb

[Andy Fiddaman]

FWIW, this gets it working again:

--- cyrus-imapd-2.4.17.dist/imap/statuscache_db.c   2013-06-24
10:10:08.219203100 +
+++ cyrus-imapd-2.4.17/imap/statuscache_db.c2013-06-24
10:10:20.537711377 +
@@ -152,7 +152,7 @@
 len = strlcpy(key, mailboxname, sizeof(key));
 key[len++] = '%';
 key[len++] = '%';
-len += strlcpy(key + len, userid, sizeof(key) - len);
+len += strlcpy(key + len, userid ? userid : "cyrus", sizeof(key) -
len);

 *keylen = len;


On Mon, 24 Jun 2013, Andy Fiddaman wrote:

;
; Hi,
;
; I've just upgraded my Cyrus installation to 2.4.17 and squatter is
; crashing in statuscache_buildkey() because userid is NULL.
;
; I'm not sure what the best fix for this is. Should squatter even be using
; the statuscache or should it populate "cyrus" as the username when
; initialising the index, or something else?
;
; Thanks,
;
; Andy
;
; Program received signal SIGSEGV, Segmentation fault.
; [Switching to Thread 1 (LWP 1)]
; 0xfd7ffe3dccb0 in .strlenalign16_loop () from /lib/64/libc.so.1
; (gdb) where
; #0  0xfd7ffe3dccb0 in .strlenalign16_loop () from /lib/64/libc.so.1
; #1  0xfd7ffe414149 in strlcpy () from /lib/64/libc.so.1
; #2  0x004610ac in statuscache_buildkey (
; mailboxname=0x5ab8b0 "example.net!user.silo", userid=0x0,
; keylen=0xfd7fffdfe0cc) at statuscache_db.c:155
; #3  0x0046169a in statuscache_update_txn (
; mboxname=0x5ab8b0 "example.net!user.silo", sdata=0xfd7fffdfe290,
; tidptr=0xfd7fffdfe218) at statuscache_db.c:326
; #4  0x004619ad in statuscache_invalidate (
; mboxname=0x5ab8b0 "example.net!user.silo", sdata=0xfd7fffdfe290)
; at statuscache_db.c:425
; #5  0x00434099 in mailbox_unlock_index (mailbox=0x5db998,
; sdata=0xfd7fffdfe290) at mailbox.c:1637
; #6  0x00422feb in index_unlock (state=0x5af6a0) at index.c:1232
; #7  0x00420c98 in index_open (name=0x5ac960
; "example.net!user.silo",
; init=0x0, stateptr=0xfd7fffdfec18) at index.c:246
; #8  0x00420395 in index_me (name=0x5ac960 "example.net!user.silo",
; matchlen=20, maycreate=0, rock=0xfd7fffdffcac) at squatter.c:594
; #9  0x004208f9 in main (argc=3, argv=0xfd7fffdffcf8)
; at squatter.c:745
;
;
; (gdb) frame 7
; #7  0x00420c98 in index_open (name=0x5ac960
; "example.net!user.silo",
; init=0x0, stateptr=0xfd7fffdfec18) at index.c:246
; 246 in index.c
; (gdb) print *state
; $4 = {mailbox = 0x5db998, num_records = 6933, oldexists = 0, exists =
; 6933,
;   last_uid = 6934, highestmodseq = 1606, delayed_modseq = 0, map =
; 0x5dd740,
;   mapsize = 7168, internalseen = 0, skipped_expunge = 0, seen_dirty = 0,
;   keepingseen = 0, examining = 0, myrights = 0, numrecent = 0,
;   numunseen = 6933, firstnotseen = 1, flagname = {0x0 },
;   userid = 0x0, out = 0x0, qresync = 0, authstate = 0x0}
;
; 
; Cyrus Home Page: http://www.cyrusimap.org/
; List Archives/Info: http://lists.andrew.cmu.edu/pipermail/info-cyrus/
; To Unsubscribe:
; https://lists.andrew.cmu.edu/mailman/listinfo/info-cyrus
;

Cyrus Home Page: http://www.cyrusimap.org/
List Archives/Info: http://lists.andrew.cmu.edu/pipermail/info-cyrus/
To Unsubscribe:
https://lists.andrew.cmu.edu/mailman/listinfo/info-cyrus

Squatter crash with statusdb

[Andy Fiddaman]

Hi,

I've just upgraded my Cyrus installation to 2.4.17 and squatter is
crashing in statuscache_buildkey() because userid is NULL.

I'm not sure what the best fix for this is. Should squatter even be using
the statuscache or should it populate "cyrus" as the username when
initialising the index, or something else?

Thanks,

Andy

Program received signal SIGSEGV, Segmentation fault.
[Switching to Thread 1 (LWP 1)]
0xfd7ffe3dccb0 in .strlenalign16_loop () from /lib/64/libc.so.1
(gdb) where
#0  0xfd7ffe3dccb0 in .strlenalign16_loop () from /lib/64/libc.so.1
#1  0xfd7ffe414149 in strlcpy () from /lib/64/libc.so.1
#2  0x004610ac in statuscache_buildkey (
mailboxname=0x5ab8b0 "example.net!user.silo", userid=0x0,
keylen=0xfd7fffdfe0cc) at statuscache_db.c:155
#3  0x0046169a in statuscache_update_txn (
mboxname=0x5ab8b0 "example.net!user.silo", sdata=0xfd7fffdfe290,
tidptr=0xfd7fffdfe218) at statuscache_db.c:326
#4  0x004619ad in statuscache_invalidate (
mboxname=0x5ab8b0 "example.net!user.silo", sdata=0xfd7fffdfe290)
at statuscache_db.c:425
#5  0x00434099 in mailbox_unlock_index (mailbox=0x5db998,
sdata=0xfd7fffdfe290) at mailbox.c:1637
#6  0x00422feb in index_unlock (state=0x5af6a0) at index.c:1232
#7  0x00420c98 in index_open (name=0x5ac960
"example.net!user.silo",
init=0x0, stateptr=0xfd7fffdfec18) at index.c:246
#8  0x00420395 in index_me (name=0x5ac960 "example.net!user.silo",
matchlen=20, maycreate=0, rock=0xfd7fffdffcac) at squatter.c:594
#9  0x004208f9 in main (argc=3, argv=0xfd7fffdffcf8)
at squatter.c:745


(gdb) frame 7
#7  0x00420c98 in index_open (name=0x5ac960
"example.net!user.silo",
init=0x0, stateptr=0xfd7fffdfec18) at index.c:246
246 in index.c
(gdb) print *state
$4 = {mailbox = 0x5db998, num_records = 6933, oldexists = 0, exists =
6933,
  last_uid = 6934, highestmodseq = 1606, delayed_modseq = 0, map =
0x5dd740,
  mapsize = 7168, internalseen = 0, skipped_expunge = 0, seen_dirty = 0,
  keepingseen = 0, examining = 0, myrights = 0, numrecent = 0,
  numunseen = 6933, firstnotseen = 1, flagname = {0x0 },
  userid = 0x0, out = 0x0, qresync = 0, authstate = 0x0}


Cyrus Home Page: http://www.cyrusimap.org/
List Archives/Info: http://lists.andrew.cmu.edu/pipermail/info-cyrus/
To Unsubscribe:
https://lists.andrew.cmu.edu/mailman/listinfo/info-cyrus

squatter error "Closing index: Cannot allocate memory"

[mailing lists]

root@imap:~# su - cyrus -c '/usr/bin/nice -n 19 /usr/bin/ionice -c 3
/usr/sbin/squatter -vis user/archive'
Indexing mailbox user/archive... Closing index: Cannot allocate memory

The mailbox has ~1,3 million emails with a total size of ~124Gb

The 2.6.32-5-686 Debian squeeze kernel has
 total   used   free sharedbuffers cached
Mem:   20663081639956 426352  0 52 875672
-/+ buffers/cache: 7642321302076
Swap:   979832 118400 861432

Using cyrus-imapd-2.32.3.16-1

cyrus dot something files have the following sizes:
-rw--- 1 cyrus mail  58M Jun 20 09:28 cyrus.index
-rw--- 1 cyrus mail 1.3G Jun 20 09:28 cyrus.cache
-rw--- 1 cyrus mail 1.1G Jun 19 23:48 cyrus.squat.NEW
-rw--- 1 cyrus mail  197 Jan 13  2011 cyrus.header

Could this be a bug or should I make more memory room available for
squatter to do it's job?
Thanks.

Cyrus Home Page: http://www.cyrusimap.org/
List Archives/Info: http://lists.andrew.cmu.edu/pipermail/info-cyrus/
To Unsubscribe:
https://lists.andrew.cmu.edu/mailman/listinfo/info-cyrus

squatter processes locked

[Rudy Gevaert]

Hello

We have noticed that squatter sometimes hangs.  It's waiting for a lock

root@cyrprd6:/etc/cyrus-ugent/conf# strace -p 13858
Process 13858 attached - interrupt to quit
fcntl(13, F_SETLKW, {type=F_WRLCK, whence=SEEK_SET, start=0, len=0}


squatter 13858 cyrus   12uR  REG   0,170
86832239 /run/shm/mail26/lock/domain/u/ugent.be/s/user/a^user/Sent  
Messages.lock
squatter 13858 cyrus   13u   REG 253,34   181952  
3223281546 /mail/mail26/imap/domain/u/ugent.be/s/user/a^user/Sent  
Messages/cyrus.index

Are people doing any housekeeping on the squatter processes?  What  
kind?  Kill?

Or is it a bug?

Thanks

-- 
  -- -- -- -- -- -- -- -- -- -- -- -- -- -- -- -- -- -- -- -- -- -- -- --
  Rudy Gevaert e-mail: rudy.geva...@ugent.be
  Directie ICT, Afdeling Infrastructuur
  Groep Systemen  tel: +32 9 264 4750
  Universiteit Gent   fax: +32 9 264 4994
  Krijgslaan 281, gebouw S9, 9000 Gent, Belgie   www.UGent.be
  -- -- -- -- -- -- -- -- -- -- -- -- -- -- -- -- -- -- -- -- -- -- -- --



Cyrus Home Page: http://www.cyrusimap.org/
List Archives/Info: http://lists.andrew.cmu.edu/pipermail/info-cyrus/
To Unsubscribe:
https://lists.andrew.cmu.edu/mailman/listinfo/info-cyrus

Re: squatter segfault in 2.4.17

[Bron Gondwana]

On Sun, Dec 9, 2012, at 07:55 PM, Patrick Boutilier wrote:
> On 12/09/2012 11:26 AM, Bron Gondwana wrote:
> > Please file a bug in bugzilla. It looks not too hard to fix. Will check 
> > more when I'm not on a phone.

> https://bugzilla.cyrusimap.org/show_bug.cgi?id=3757

Yep - resolved in git now.  The bloody race condition in idle (#3744) is proving
less tractable to immediate fixes - though I have a potential fix there too now.

Bron.
-- 
  Bron Gondwana
  br...@fastmail.fm


Cyrus Home Page: http://www.cyrusimap.org/
List Archives/Info: http://lists.andrew.cmu.edu/pipermail/info-cyrus/
To Unsubscribe:
https://lists.andrew.cmu.edu/mailman/listinfo/info-cyrus

Re: squatter segfault in 2.4.17

[Patrick Boutilier]

On 12/09/2012 11:26 AM, Bron Gondwana wrote:

Please file a bug in bugzilla. It looks not too hard to fix. Will check more 
when I'm not on a phone.

Bron.

On Sun, Dec 9, 2012, at 03:00 PM, Khalid Mehmood wrote:

My squtter configuation was working fine in 2.4.16, but it has stopped working after I 
upgraded to 2.4.17 a couple of days ago. Running squatter manaually as cyrus user give 
the "Segmentation fault" error.
Has there been any changes regarding squatter since 2.4.16? I have read the 
changelog but can't find any squatter related changes.

The maillog gives the following error:

"squatter[39517]: error opening looking up *: Mailbox does not exist"


Thanks.


Cyrus Home Page: http://www.cyrusimap.org/
List Archives/Info: http://lists.andrew.cmu.edu/pipermail/info-cyrus/
To Unsubscribe:
https://lists.andrew.cmu.edu/mailman/listinfo/info-cyrus



https://bugzilla.cyrusimap.org/show_bug.cgi?id=3757
<>
Cyrus Home Page: http://www.cyrusimap.org/
List Archives/Info: http://lists.andrew.cmu.edu/pipermail/info-cyrus/
To Unsubscribe:
https://lists.andrew.cmu.edu/mailman/listinfo/info-cyrus

Re: squatter segfault in 2.4.17

[Bron Gondwana]

Please file a bug in bugzilla. It looks not too hard to fix. Will check more 
when I'm not on a phone.

Bron.

On Sun, Dec 9, 2012, at 03:00 PM, Khalid Mehmood wrote:
> My squtter configuation was working fine in 2.4.16, but it has stopped 
> working after I upgraded to 2.4.17 a couple of days ago. Running squatter 
> manaually as cyrus user give the "Segmentation fault" error.
> Has there been any changes regarding squatter since 2.4.16? I have read the 
> changelog but can't find any squatter related changes.
> 
> The maillog gives the following error:
> 
> "squatter[39517]: error opening looking up *: Mailbox does not exist"
> 
> 
> Thanks.
> 
> 
> Cyrus Home Page: http://www.cyrusimap.org/
> List Archives/Info: http://lists.andrew.cmu.edu/pipermail/info-cyrus/
> To Unsubscribe:
> https://lists.andrew.cmu.edu/mailman/listinfo/info-cyrus
-- 
  Bron Gondwana
  br...@fastmail.fm


Cyrus Home Page: http://www.cyrusimap.org/
List Archives/Info: http://lists.andrew.cmu.edu/pipermail/info-cyrus/
To Unsubscribe:
https://lists.andrew.cmu.edu/mailman/listinfo/info-cyrus

Re: squatter segfault in 2.4.17

[Patrick Boutilier]

On 12/09/2012 10:00 AM, Khalid Mehmood wrote:

My squtter configuation was working fine in 2.4.16, but it has stopped working after I 
upgraded to 2.4.17 a couple of days ago. Running squatter manaually as cyrus user give 
the "Segmentation fault" error.
Has there been any changes regarding squatter since 2.4.16? I have read the 
changelog but can't find any squatter related changes.

The maillog gives the following error:

"squatter[39517]: error opening looking up *: Mailbox does not exist"


Thanks.


Cyrus Home Page: http://www.cyrusimap.org/
List Archives/Info: http://lists.andrew.cmu.edu/pipermail/info-cyrus/
To Unsubscribe:
https://lists.andrew.cmu.edu/mailman/listinfo/info-cyrus



Seeing same thing. strace not very helpful.

...
open("/var/imap/statuscache.db", O_RDWR) = 9
fcntl(9, F_SETLKW, {type=F_RDLCK, whence=SEEK_SET, start=0, len=0}) = 0
fstat(9, {st_mode=S_IFREG|0600, st_size=11141528, ...}) = 0
stat("/var/imap/statuscache.db", {st_mode=S_IFREG|0600, 
st_size=11141528, ...}) = 0

mmap(NULL, 11157504, PROT_READ, MAP_SHARED, 9, 0) = 0x2b7584cc7000
fcntl(9, F_SETLKW, {type=F_UNLCK, whence=SEEK_SET, start=0, len=0}) = 0
fcntl(9, F_SETLKW, {type=F_WRLCK, whence=SEEK_SET, start=0, len=0}) = 0
fstat(9, {st_mode=S_IFREG|0600, st_size=11141528, ...}) = 0
stat("/var/imap/statuscache.db", {st_mode=S_IFREG|0600, 
st_size=11141528, ...}) = 0

lseek(9, 11141528, SEEK_SET)= 11141528
write(9, "\0\0\0\4\0\24\311\260", 8)= 8
lseek(9, 1362344, SEEK_SET) = 1362344
write(9, "\0\24\312\4", 4)  = 4
--- SIGSEGV (Segmentation fault) @ 0 (0) ---
+++ killed by SIGSEGV +++
<>
Cyrus Home Page: http://www.cyrusimap.org/
List Archives/Info: http://lists.andrew.cmu.edu/pipermail/info-cyrus/
To Unsubscribe:
https://lists.andrew.cmu.edu/mailman/listinfo/info-cyrus

Re: squatter stops when it encounters a locked mailbox?

[Michael D. Sofka]

Clement Hermann (nodens) wrote:
> Le 21/06/2011 18:28, Michael D. Sofka a écrit :
>> I run squatter in a perl program that forks three parallel squatter
>> processes on individual user's mailboxes. If a mailbox is locked the
>> particular squatter processing the mailbox quits, but the main program
>> continues to fork new processes for the remaining mailboxes. The program
>> checkpoints each user, shuts down at 6 A.M., and continues where it left
>> off the following day.
> Looks nice.
> 
> Maybe you could share this script ? I could use a squatter with some 
> parallelization and a way to stop it when the load is starting to grow 
> (like in the morning). I guess it could be modified to use other 
> parameters to know when to stop : number of logged-in users or imap/pop 
> processes, system load...
> 
> Cheers,
> 


I'm looking over the paperwork needed to share the program (welcome to
21st Century academia). If I get approval, I'll consider adding other
termination/pause options. Our 24 hour load-cycle is fairly predicable.
But I can see how logged-in users, and load average would be useful.

Mike
-- 
Michael D. Sofka   sof...@rpi.edu
C&MT Sr. Systems Programmer,   Email, HPC, TeX, Epistemology
Rensselaer Polytechnic Institute, Troy, NY.  http://www.rpi.edu/~sofkam/

Cyrus Home Page: http://www.cyrusimap.org/
List Archives/Info: http://lists.andrew.cmu.edu/pipermail/info-cyrus/

Re: squatter stops when it encounters a locked mailbox?

[Clement Hermann (nodens)]

Le 21/06/2011 18:28, Michael D. Sofka a écrit :
> I run squatter in a perl program that forks three parallel squatter
> processes on individual user's mailboxes. If a mailbox is locked the
> particular squatter processing the mailbox quits, but the main program
> continues to fork new processes for the remaining mailboxes. The program
> checkpoints each user, shuts down at 6 A.M., and continues where it left
> off the following day.
Looks nice.

Maybe you could share this script ? I could use a squatter with some 
parallelization and a way to stop it when the load is starting to grow 
(like in the morning). I guess it could be modified to use other 
parameters to know when to stop : number of logged-in users or imap/pop 
processes, system load...

Cheers,

-- 
Clement Hermann (nodens)
- "L'air pur ? c'est pas en RL, ça ? c'est pas hors charte ?"
Jean in L'Histoire des Pingouins, http://tnemeth.free.fr/fmbl/linuxsf/

Vous trouverez ma clef publique sur le serveur public pgp.mit.edu.
Please find my public key on the public keyserver pgp.mit.edu.


Cyrus Home Page: http://www.cyrusimap.org/
List Archives/Info: http://lists.andrew.cmu.edu/pipermail/info-cyrus/

Re: squatter stops when it encounters a locked mailbox?

[Michael D. Sofka]

Per olof Ljungmark wrote:
> All,
> 
> While migrating a few accounts from Exchange to Cyrus ( has nothing to 
> do with the below really) and staring at log files I noted the following
> 
> squatter[70553]: error locking index user/xyz: Mailbox is locked by POP 
> server
> 
> and squatter bails out without indexing the remaining mailboxes. Is this 
> expected behaviour in 2.3.16?

I run squatter in a perl program that forks three parallel squatter
processes on individual user's mailboxes. If a mailbox is locked the
particular squatter processing the mailbox quits, but the main program
continues to fork new processes for the remaining mailboxes. The program
checkpoints each user, shuts down at 6 A.M., and continues where it left
off the following day.

Mike

-- 
Michael D. Sofka   sof...@rpi.edu
C&MT Sr. Systems Programmer,   Email, HPC, TeX, Epistemology
Rensselaer Polytechnic Institute, Troy, NY.  http://www.rpi.edu/~sofkam/

Cyrus Home Page: http://www.cyrusimap.org/
List Archives/Info: http://lists.andrew.cmu.edu/pipermail/info-cyrus/

Re: squatter stops when it encounters a locked mailbox?

[Per olof Ljungmark]

On 06/21/11 08:23, Bron Gondwana wrote:
> On Tue, Jun 21, 2011 at 02:16:07AM +0200, Per olof Ljungmark wrote:
>> All,
>>
>> While migrating a few accounts from Exchange to Cyrus ( has nothing to
>> do with the below really) and staring at log files I noted the following
>>
>> squatter[70553]: error locking index user/xyz: Mailbox is locked by POP
>> server
>>
>> and squatter bails out without indexing the remaining mailboxes. Is this
>> expected behaviour in 2.3.16?
>
> Sorry, POP locking in cyrus before 2.4.x is a bit of a mess.  Yeah, it's
> "expected" for some value - the bailing out kind of sucks too, but I really
> don't want to try and fix 2.3.x any more.
>
> Bron ( particularly things like this that I changed the locking architecture
> specifically to avoid! )

OK, thanks, should have read more b4 posting I suppose. We have an 
upgrade plan in the works anyway and we would not want you to spend time 
on older stuff.

Cyrus Home Page: http://www.cyrusimap.org/
List Archives/Info: http://lists.andrew.cmu.edu/pipermail/info-cyrus/

Re: squatter stops when it encounters a locked mailbox?

[Bron Gondwana]

On Tue, Jun 21, 2011 at 02:16:07AM +0200, Per olof Ljungmark wrote:
> All,
> 
> While migrating a few accounts from Exchange to Cyrus ( has nothing to 
> do with the below really) and staring at log files I noted the following
> 
> squatter[70553]: error locking index user/xyz: Mailbox is locked by POP 
> server
> 
> and squatter bails out without indexing the remaining mailboxes. Is this 
> expected behaviour in 2.3.16?

Sorry, POP locking in cyrus before 2.4.x is a bit of a mess.  Yeah, it's
"expected" for some value - the bailing out kind of sucks too, but I really
don't want to try and fix 2.3.x any more.

Bron ( particularly things like this that I changed the locking architecture
   specifically to avoid! )

Cyrus Home Page: http://www.cyrusimap.org/
List Archives/Info: http://lists.andrew.cmu.edu/pipermail/info-cyrus/

squatter stops when it encounters a locked mailbox?

[Per olof Ljungmark]

All,

While migrating a few accounts from Exchange to Cyrus ( has nothing to 
do with the below really) and staring at log files I noted the following

squatter[70553]: error locking index user/xyz: Mailbox is locked by POP 
server

and squatter bails out without indexing the remaining mailboxes. Is this 
expected behaviour in 2.3.16?

Thanks,


Cyrus Home Page: http://www.cyrusimap.org/
List Archives/Info: http://lists.andrew.cmu.edu/pipermail/info-cyrus/

Re: incremental squatter

[Rob Mueller]

> Also, does anyone know what this means for searches on material that has
> changed since the last squatter run?  I have assumed, and hope, that the
> search procedure is something like this:
> search in the squatter index
> remove results referring to deleted items
> do an unindexed search on items added since last index.
>
> Is that right?  Or, for example, are new messages just ignored?

The squatter index isn't a perfect index. What it does is given a search 
term, it returns a list of messages that might contain the term, and 
excludes messages that definitely do not contain the search term. For each 
message that squatter says might contain the search term, cyrus then opens 
the message and does a complete search on it to see if it definitely 
contains the search term.

Because of that, if squatter sees a message id it hasn't indexed, it will 
always return that id, because that id might contain the term, it doesn't 
know.

The net result is that things work as expected. New messages that haven't 
been squatter indexed are always searched, you never miss messages.

Rob


Cyrus Home Page: http://cyrusimap.web.cmu.edu/
Cyrus Wiki/FAQ: http://cyrusimap.web.cmu.edu/twiki
List Archives/Info: http://asg.web.cmu.edu/cyrus/mailing-list.html

Re: incremental squatter

[Ross Boylan]

On Wed, 2009-09-02 at 08:44 -0400, Adam Tauno Williams wrote:
> On Wed, 2009-09-02 at 14:30 +0200, Paolo Cravero wrote:
> > Perhaps it is a bug in the documentation.
> > 'man squatter' in the DESCRIPTION says there's no incremental update:
> > "Squatter creates an index of ALL messages in the mailbox, not just those 
> > since the last time that it was run (i.e.,  it  does  NOT  do  incremental 
> > updates)."
> > but in the OPTIONS a "-i" switch mentions incremental.
> > Is it just a documentation error, so incremental indexing does work in 
> > 2.3.14?
> 
> It may be a matter of interpretation.
> 
> 
> Squatter creates an index of ALL messages in the mailbox, not just those
> since time that it was run (i.e., it does NOT do incremental updates).
> Any messages appended  to  the mailbox after squatter is run, will NOT
> be included in the index. To include new messages in the index, squatter
> must be run again. 
> 
> 
> I took "does NOT do incremental updates" to mean the index is not
> updated on-the-fly when new messages enter the mailbox, only when
> squatter is run.
I think that is a leftover from 2.2, and the sense of incremental there
was that when squatter runs reindexes the entire content, rather than
reusing what it can from the existing index.  So incremental referred to
a batch, rather than real-time, update.

Apparently 2.3 does have a batch incremental mode.  That sounds like a
huge win.

Also, does anyone know what this means for searches on material that has
changed since the last squatter run?  I have assumed, and hope, that the
search procedure is something like this:
search in the squatter index
remove results referring to deleted items
do an unindexed search on items added since last index.

Is that right?  Or, for example, are new messages just ignored?

Ross Boylan


Cyrus Home Page: http://cyrusimap.web.cmu.edu/
Cyrus Wiki/FAQ: http://cyrusimap.web.cmu.edu/twiki
List Archives/Info: http://asg.web.cmu.edu/cyrus/mailing-list.html

Re: incremental squatter

[Adam Tauno Williams]

On Wed, 2009-09-02 at 14:30 +0200, Paolo Cravero wrote:
> Perhaps it is a bug in the documentation.
> 'man squatter' in the DESCRIPTION says there's no incremental update:
> "Squatter creates an index of ALL messages in the mailbox, not just those 
> since the last time that it was run (i.e.,  it  does  NOT  do  incremental 
> updates)."
> but in the OPTIONS a "-i" switch mentions incremental.
> Is it just a documentation error, so incremental indexing does work in 2.3.14?

It may be a matter of interpretation.


Squatter creates an index of ALL messages in the mailbox, not just those
since time that it was run (i.e., it does NOT do incremental updates).
Any messages appended  to  the mailbox after squatter is run, will NOT
be included in the index. To include new messages in the index, squatter
must be run again. 


I took "does NOT do incremental updates" to mean the index is not
updated on-the-fly when new messages enter the mailbox, only when
squatter is run.

-- 
OpenGroupware developer: awill...@whitemice.org
<http://whitemiceconsulting.blogspot.com/>
OpenGroupare & Cyrus IMAPd documenation @
<http://docs.opengroupware.org/Members/whitemice/wmogag/file_view>


Cyrus Home Page: http://cyrusimap.web.cmu.edu/
Cyrus Wiki/FAQ: http://cyrusimap.web.cmu.edu/twiki
List Archives/Info: http://asg.web.cmu.edu/cyrus/mailing-list.html

Re: incremental squatter

[Sebastian Hagedorn]

--On 2. September 2009 14:30:18 +0200 Paolo Cravero  
wrote:



Is it just a documentation error, so incremental indexing does work in
2.3.14?


Yes, it does.
--
.:.Sebastian Hagedorn - RZKR-R1 (Gebäude 52), Zimmer 18.:.
.:.Regionales Rechenzentrum (RRZK).:.
.:.Universität zu Köln / Cologne University - ✆ +49-221-478-5587.:.

p7sckFeoLvaL7.p7s
Description: S/MIME cryptographic signature

Cyrus Home Page: http://cyrusimap.web.cmu.edu/
Cyrus Wiki/FAQ: http://cyrusimap.web.cmu.edu/twiki
List Archives/Info: http://asg.web.cmu.edu/cyrus/mailing-list.html

incremental squatter

[Paolo Cravero]

Hi.
Perhaps it is a bug in the documentation.

'man squatter' in the DESCRIPTION says there's no incremental update:

"Squatter creates an index of ALL messages in the mailbox, not just those 
since the last time that it was run (i.e.,  it  does  NOT  do  incremental 
updates)."

but in the OPTIONS a "-i" switch mentions incremental.

Is it just a documentation error, so incremental indexing does work in 2.3.14?

TIA,
Paolo



Cyrus Home Page: http://cyrusimap.web.cmu.edu/
Cyrus Wiki/FAQ: http://cyrusimap.web.cmu.edu/twiki
List Archives/Info: http://asg.web.cmu.edu/cyrus/mailing-list.html

squatter Writing index update: No such file or directory

[Ross Boylan]

# sudo -u cyrus squatter -v user.ross.R
Indexing mailbox user.ross.R... Writing index update: No such file or directory

Does anybody know what the error means, or what might be causing it?


I have squatter set to run nightly, but for about a month its only done
a few mailboxes and then stopped.  I believe it is failing on the one
shown above since that's consistent with the time stamps, the
alphabetical order of directories, and the error shown above.

That mailbox has the largest squat file by far:
171793 623216 -rw---   1 cyrusmail 637546664 Jul 14 05:58 
./mail/r/user/ross/R/cyrus.squat

However, that's only half of free space, if I've got my units right:
# df .
Filesystem   1K-blocks  Used Available Use% Mounted on
/dev/mapper/big_container-var
  14712376  13421392   1290984  92% /var

The biggest file it succeeds on is 39784890 bytes (40MB); /tmp has about
278440 1k blocks (278MB).

So if squatter is using /tmp, that might explain things.

Ross Boylan



Cyrus Home Page: http://cyrusimap.web.cmu.edu/
Cyrus Wiki/FAQ: http://cyrusimap.web.cmu.edu/twiki
List Archives/Info: http://asg.web.cmu.edu/cyrus/mailing-list.html

Re: squatter exits with "fatal error: Virtual memory exhausted" on huge mailbox

[Sebastian Hagedorn]

--On 13. Juli 2009 15:36:07 +0200 Gabor Gombas  wrote:


On Mon, Jul 13, 2009 at 02:09:40PM +0200, Sebastian Hagedorn wrote:


> 4 GB limit of 32 bit binaries?

Perhaps, although I haven't seen it.


That's only 3GB by default, 1GB of address space is reserved for the
kernel. Also, the stack, the executable, and all the shared libraries
the executable uses also occupy some address space.


Good to know.


Of course it's possible that it then tried to allocate one huge
chunk, but I can't see that. Are there better tools to monitor the
memory allocation of a process?


strace -e trace=brk,mmap,munmap (well, this actually traces glibc's
memory management, but it should show the failure).


You. I had to specify mmap2, but then I see this:

mmap2(NULL, 267374592, PROT_READ|PROT_WRITE, MAP_PRIVATE|MAP_ANONYMOUS, -1, 
0) = 0x2218b000

mmap2(NULL, 858675, PROT_READ, MAP_SHARED, 43, 0) = 0x220b9000
munmap(0x220b9000, 858675)  = 0
munmap(0x2218b000, 267374592)   = 0
mmap2(NULL, 682827776, PROT_READ|PROT_WRITE, MAP_PRIVATE|MAP_ANONYMOUS, -1, 
0) = -1 ENOMEM (Cannot allocate memory)

brk(0)  = 0x993d000
brk(0x32483000) = 0x993d000
mmap2(NULL, 682962944, PROT_READ|PROT_WRITE, MAP_PRIVATE|MAP_ANONYMOUS, -1, 
0) = -1 ENOMEM (Cannot allocate memory)
mmap2(NULL, 2097152, PROT_NONE, MAP_PRIVATE|MAP_ANONYMOUS|MAP_NORESERVE, 
-1, 0) = 0x337c1000

munmap(0x337c1000, 258048)  = 0
munmap(0x3390, 790528)  = 0
mmap2(NULL, 682827776, PROT_READ|PROT_WRITE, MAP_PRIVATE|MAP_ANONYMOUS, -1, 
0) = -1 ENOMEM (Cannot allocate memory)

fatal error: Virtual memory exhausted

So it really tries to allocate a rather large chunk! I guess that's one 
more reason to switch to 64-bit when we make he move to RHEL 5.


Thanks!
--
.:.Sebastian Hagedorn - RZKR-R1 (Gebäude 52), Zimmer 18.:.
.:.Regionales Rechenzentrum (RRZK).:.
.:.Universität zu Köln / Cologne University - ✆ +49-221-478-5587.:.

pgp0WmEua5sro.pgp
Description: PGP signature

Cyrus Home Page: http://cyrusimap.web.cmu.edu/
Cyrus Wiki/FAQ: http://cyrusimap.web.cmu.edu/twiki
List Archives/Info: http://asg.web.cmu.edu/cyrus/mailing-list.html

Re: squatter exits with "fatal error: Virtual memory exhausted" on huge mailbox

[Gabor Gombas]

On Mon, Jul 13, 2009 at 02:09:40PM +0200, Sebastian Hagedorn wrote:

> >4 GB limit of 32 bit binaries?
> 
> Perhaps, although I haven't seen it.

That's only 3GB by default, 1GB of address space is reserved for the
kernel. Also, the stack, the executable, and all the shared libraries
the executable uses also occupy some address space.

> Of course it's possible that it then tried to allocate one huge
> chunk, but I can't see that. Are there better tools to monitor the
> memory allocation of a process?

strace -e trace=brk,mmap,munmap (well, this actually traces glibc's
memory management, but it should show the failure).

Gabor

-- 
 -
 MTA SZTAKI Computer and Automation Research Institute
Hungarian Academy of Sciences
 -

Cyrus Home Page: http://cyrusimap.web.cmu.edu/
Cyrus Wiki/FAQ: http://cyrusimap.web.cmu.edu/twiki
List Archives/Info: http://asg.web.cmu.edu/cyrus/mailing-list.html

Re: squatter exits with "fatal error: Virtual memory exhausted" on huge mailbox

[Sebastian Hagedorn]

--On 13. Juli 2009 14:53:31 +0200 Pascal Gienger 
 wrote:



Sebastian Hagedorn schrieb:

fatal error: Virtual memory exhausted



Of course it's possible that it then tried to allocate one huge chunk,
but I can't see that. Are there better tools to monitor the memory
allocation of a process?


Swap file/partition full?


Unlikely.


Background:
I think the message "Virtual memory exhausted" is coming from your
operating system and not from the squatter process.


I disagree.


Squatter would have been said

   switch (err) {
   case SQUAT_ERR_OUT_OF_MEMORY:
 fprintf(stderr, "SQUAT: Out of memory (%s)\n", s);
 break;


But:

lib/xmalloc.c:fatal("Virtual memory exhausted", EC_TEMPFAIL);

This code actually can't be reached:

 b->buf = (char*)xrealloc(b->buf, len);
 if (b->buf == NULL) {
   squat_set_last_error(SQUAT_ERR_OUT_OF_MEMORY);
   return NULL;
 }

If the xrealloc fails, the "fatal" above is called within that routine.
--
.:.Sebastian Hagedorn - RZKR-R1 (Gebäude 52), Zimmer 18.:.
.:.Regionales Rechenzentrum (RRZK).:.
.:.Universität zu Köln / Cologne University - ✆ +49-221-478-5587.:.

pgpUnC6euB2si.pgp
Description: PGP signature

Cyrus Home Page: http://cyrusimap.web.cmu.edu/
Cyrus Wiki/FAQ: http://cyrusimap.web.cmu.edu/twiki
List Archives/Info: http://asg.web.cmu.edu/cyrus/mailing-list.html

Re: squatter exits with "fatal error: Virtual memory exhausted" on huge mailbox

[Pascal Gienger]

Sebastian Hagedorn schrieb:
>>> fatal error: Virtual memory exhausted

> Of course it's possible that it then tried to allocate one huge chunk, 
> but I can't see that. Are there better tools to monitor the memory 
> allocation of a process?

Swap file/partition full?
Background:
I think the message "Virtual memory exhausted" is coming from your 
operating system and not from the squatter process.


Squatter would have been said

   switch (err) {
   case SQUAT_ERR_OUT_OF_MEMORY:
 fprintf(stderr, "SQUAT: Out of memory (%s)\n", s);
 break;



So I think it is a Virtual Memory/Swap problem in your OS.

-- 
Pascal Gienger
University of Konstanz, IT Services Department ("Rechenzentrum")
Electronic Communications and Web Services
Building V, Room V404, Phone +49 7531 88 5048, Fax +49 7531 88 3739

Cyrus Home Page: http://cyrusimap.web.cmu.edu/
Cyrus Wiki/FAQ: http://cyrusimap.web.cmu.edu/twiki
List Archives/Info: http://asg.web.cmu.edu/cyrus/mailing-list.html

Re: squatter exits with "fatal error: Virtual memory exhausted" on huge mailbox

[Sebastian Hagedorn]

--On 13. Juli 2009 13:20:42 +0200 Pascal Gienger 
 wrote:



Sebastian Hagedorn schrieb:

Processing index character 101, 681642 total words, temp file size is
2107147
fatal error: Virtual memory exhausted


4 GB limit of 32 bit binaries?


Perhaps, although I haven't seen it.


How much RAM does squatter allocate before it dies?


I monitored the process like this:

# while true; do grep VmSize /proc/16815/status; sleep 1; done;

The last lines are:

VmSize:  2453772 kB
VmSize:  2454608 kB
VmSize:  2454608 kB
VmSize:  2454884 kB
VmSize:  2192664 kB
VmSize:  2192664 kB

Of course it's possible that it then tried to allocate one huge chunk, but 
I can't see that. Are there better tools to monitor the memory allocation 
of a process?


Thanks!
--
.:.Sebastian Hagedorn - RZKR-R1 (Gebäude 52), Zimmer 18.:.
.:.Regionales Rechenzentrum (RRZK).:.
.:.Universität zu Köln / Cologne University - ✆ +49-221-478-5587.:.

pgpGwJtb8GPrW.pgp
Description: PGP signature

Cyrus Home Page: http://cyrusimap.web.cmu.edu/
Cyrus Wiki/FAQ: http://cyrusimap.web.cmu.edu/twiki
List Archives/Info: http://asg.web.cmu.edu/cyrus/mailing-list.html

Re: squatter exits with "fatal error: Virtual memory exhausted" on huge mailbox

[Pascal Gienger]

Sebastian Hagedorn schrieb:
> Processing index character 101, 681642 total words, temp file size is 
> 2107147
> fatal error: Virtual memory exhausted

4 GB limit of 32 bit binaries?
How much RAM does squatter allocate before it dies?

-- 
Pascal Gienger
University of Konstanz, IT Services Department ("Rechenzentrum")
Electronic Communications and Web Services
Building V, Room V404, Phone +49 7531 88 5048, Fax +49 7531 88 3739

Cyrus Home Page: http://cyrusimap.web.cmu.edu/
Cyrus Wiki/FAQ: http://cyrusimap.web.cmu.edu/twiki
List Archives/Info: http://asg.web.cmu.edu/cyrus/mailing-list.html

squatter exits with "fatal error: Virtual memory exhausted" on huge mailbox

[Sebastian Hagedorn]

Hi,

we run 2.3.14 and we have one user with an enormous mailbox (not his 
INBOX!). It contains around 230,000 messages. We run squatter with a 
cronjob on all mailboxes. Recently this one mailbox has apparently become 
too big for squatter. When I try to squat it in verbose mode, it ends like 
this:


...
Skipping tiny document part 'b563140' (size 0)
Opening document part 's563140'
Processing index character 7, 5 total words, temp file size is 35
Processing index character 10, 152 total words, temp file size is 1064
Processing index character 13, 152 total words, temp file size is 1064
Processing index character 15, 3 total words, temp file size is 21
Processing index character 23, 5 total words, temp file size is 35
Processing index character 27, 5 total words, temp file size is 35
Processing index character 30, 3 total words, temp file size is 21
Processing index character 32, 12938 total words, temp file size is 66346
Processing index character 33, 2692 total words, temp file size is 11856
Processing index character 39, 12553 total words, temp file size is 57119
Processing index character 40, 60791 total words, temp file size is 203723
Processing index character 41, 57972 total words, temp file size is 195005
Processing index character 42, 20959 total words, temp file size is 68807
Processing index character 43, 51335 total words, temp file size is 172871
Processing index character 44, 78879 total words, temp file size is 270078
Processing index character 45, 183196 total words, temp file size is 588133
Processing index character 46, 233839 total words, temp file size is 756846
Processing index character 47, 56284 total words, temp file size is 195995
Processing index character 48, 226012 total words, temp file size is 701263
Processing index character 49, 168692 total words, temp file size is 530284
Processing index character 50, 153898 total words, temp file size is 487575
Processing index character 51, 122957 total words, temp file size is 391515
Processing index character 52, 100478 total words, temp file size is 322795
Processing index character 53, 81760 total words, temp file size is 265813
Processing index character 54, 78645 total words, temp file size is 258465
Processing index character 55, 83934 total words, temp file size is 272204
Processing index character 56, 82022 total words, temp file size is 266903
Processing index character 57, 76410 total words, temp file size is 250698
Processing index character 58, 135314 total words, temp file size is 437180
Processing index character 59, 44962 total words, temp file size is 151528
Processing index character 60, 49223 total words, temp file size is 197205
Processing index character 61, 39112 total words, temp file size is 133934
Processing index character 62, 110296 total words, temp file size is 363204
Processing index character 63, 3887 total words, temp file size is 18737
Processing index character 64, 60070 total words, temp file size is 233407
Processing index character 88, 238 total words, temp file size is 770
Processing index character 91, 22210 total words, temp file size is 7
Processing index character 92, 1490 total words, temp file size is 6242
Processing index character 93, 17768 total words, temp file size is 75520
Processing index character 94, 226 total words, temp file size is 962
Processing index character 95, 54059 total words, temp file size is 179583
Processing index character 96, 106 total words, temp file size is 506
Processing index character 97, 389316 total words, temp file size is 1227616
Processing index character 98, 140247 total words, temp file size is 457214
Processing index character 99, 243003 total words, temp file size is 782841
Processing index character 100, 262183 total words, temp file size is 833170
Processing index character 101, 681642 total words, temp file size is 
2107147

fatal error: Virtual memory exhausted

We will try to get this user to split this mailbox or to clean it up, but I 
wondered if there is something else we can do. The machine has ample 
amounts of memory, so I'm unsure why the call to malloc would fail ...

--
.:.Sebastian Hagedorn - RZKR-R1 (Gebäude 52), Zimmer 18.:.
.:.Regionales Rechenzentrum (RRZK).:.
.:.Universität zu Köln / Cologne University - ✆ +49-221-478-5587.:.

pgpP2d9eLBSG3.pgp
Description: PGP signature

Cyrus Home Page: http://cyrusimap.web.cmu.edu/
Cyrus Wiki/FAQ: http://cyrusimap.web.cmu.edu/twiki
List Archives/Info: http://asg.web.cmu.edu/cyrus/mailing-list.html

Re: Squatter causing load spike

[Rob Banz]

>>
> It seems like the default for cyr_expire runs at 4:00AM (delprune
> cmd="cyr_expire -E 3" at=0400) and I start squatter at 3:00AM. Do you
> think that this would cause the spike and server to lock up? We are
> running RHEL4U4

They both compete for a lot of resources... You probably shouldn't run  
them both at the same time.  My cyrii run on Solaris, so I'm not  
familiar with what platform specific issues you may be having.

You might try running just squatter one night without expire and see  
how it does.

Also, make sure you're running squatter with the option (think it's - 
s) that makes it only reindex mailboxes that have changed since the  
last squatter run.

-rob



Cyrus Home Page: http://cyrusimap.web.cmu.edu/
Cyrus Wiki/FAQ: http://cyrusimap.web.cmu.edu/twiki
List Archives/Info: http://asg.web.cmu.edu/cyrus/mailing-list.html

Re: Squatter causing load spike

[Gerard]

On Nov 13, 2007 5:07 PM, Rob Banz <[EMAIL PROTECTED]> wrote:
>
>
> On Nov 13, 2007, at 16:47, Gerard wrote:
>
> > I am running squatter at 3am as a cron job on all of my servers. Over
> > the passed week I have one server where squatter spikes the load and
> > ends up locking up the server at around 8am every morning. Yeah, it
> > seems to take that long to run which may be an issue in itself. Has
> > anyone come accross this or have a suggestion on how to get squatter
> > to perform better?
> > 
>
> What OS?
>
> Squatter shouldn't be causing "load spikes" unless there's some
> resource contention for lock files, etc.
>
> You should also make sure that if you're running cyr_expire, you're
> NOT running it concurrently with squatter.  Bad things happen.*  To
> get around this, I created a "nightly" shell script which runs
> cyr_expire, then squatter, and kick that off as a scheduled task in my
> cyrus.conf.
>
> * The bad things come into play if expire & squatter are processing
> the same mailbox at the same time.  Squatter will probably just
> "stop", and not tell you why.
>
> -rob
> 
> Cyrus Home Page: http://cyrusimap.web.cmu.edu/
> Cyrus Wiki/FAQ: http://cyrusimap.web.cmu.edu/twiki
> List Archives/Info: http://asg.web.cmu.edu/cyrus/mailing-list.html
>
It seems like the default for cyr_expire runs at 4:00AM (delprune
cmd="cyr_expire -E 3" at=0400) and I start squatter at 3:00AM. Do you
think that this would cause the spike and server to lock up? We are
running RHEL4U4

Cyrus Home Page: http://cyrusimap.web.cmu.edu/
Cyrus Wiki/FAQ: http://cyrusimap.web.cmu.edu/twiki
List Archives/Info: http://asg.web.cmu.edu/cyrus/mailing-list.html

Re: Squatter causing load spike

[Rob Banz]

On Nov 13, 2007, at 16:47, Gerard wrote:

> I am running squatter at 3am as a cron job on all of my servers. Over
> the passed week I have one server where squatter spikes the load and
> ends up locking up the server at around 8am every morning. Yeah, it
> seems to take that long to run which may be an issue in itself. Has
> anyone come accross this or have a suggestion on how to get squatter
> to perform better?
> ----

What OS?

Squatter shouldn't be causing "load spikes" unless there's some  
resource contention for lock files, etc.

You should also make sure that if you're running cyr_expire, you're  
NOT running it concurrently with squatter.  Bad things happen.*  To  
get around this, I created a "nightly" shell script which runs  
cyr_expire, then squatter, and kick that off as a scheduled task in my  
cyrus.conf.

* The bad things come into play if expire & squatter are processing  
the same mailbox at the same time.  Squatter will probably just  
"stop", and not tell you why.

-rob

Cyrus Home Page: http://cyrusimap.web.cmu.edu/
Cyrus Wiki/FAQ: http://cyrusimap.web.cmu.edu/twiki
List Archives/Info: http://asg.web.cmu.edu/cyrus/mailing-list.html

Squatter causing load spike

[Gerard]

I am running squatter at 3am as a cron job on all of my servers. Over
the passed week I have one server where squatter spikes the load and
ends up locking up the server at around 8am every morning. Yeah, it
seems to take that long to run which may be an issue in itself. Has
anyone come accross this or have a suggestion on how to get squatter
to perform better?

Cyrus Home Page: http://cyrusimap.web.cmu.edu/
Cyrus Wiki/FAQ: http://cyrusimap.web.cmu.edu/twiki
List Archives/Info: http://asg.web.cmu.edu/cyrus/mailing-list.html

Re: squatter running longer than 24 hours

[David Lang]

On Mon, 22 Oct 2007, Rob Mueller wrote:

>> squatter would really benefit from incremental updates. At the moment a
>> single new message in a mailbox containing 20k messages causes it to read
>> in all the existing messages in order to regenerate the index.
>
> We spoke to Ken about this ages back, and even offered to pay for the work
> to make it happen, but it was just around the time CMU hired him, so it
> never actually happened pity. It would be nice to be able to dedicate a
> couple of weeks to rummage around in there and actually make it happen...

postgres has full-text search capabilities at acceptable performance on very 
large databases, their code is BSD so anything relavent coudl be merged into 
cyrus. it may be worth someone looking into their logic.

David Lang

Cyrus Home Page: http://cyrusimap.web.cmu.edu/
Cyrus Wiki/FAQ: http://cyrusimap.web.cmu.edu/twiki
List Archives/Info: http://asg.web.cmu.edu/cyrus/mailing-list.html

Re: squatter running longer than 24 hours

[Rob Mueller]

> squatter would really benefit from incremental updates. At the moment a
> single new message in a mailbox containing 20k messages causes it to read
> in all the existing messages in order to regenerate the index.

We spoke to Ken about this ages back, and even offered to pay for the work 
to make it happen, but it was just around the time CMU hired him, so it 
never actually happened pity. It would be nice to be able to dedicate a 
couple of weeks to rummage around in there and actually make it happen...

Rob


Cyrus Home Page: http://cyrusimap.web.cmu.edu/
Cyrus Wiki/FAQ: http://cyrusimap.web.cmu.edu/twiki
List Archives/Info: http://asg.web.cmu.edu/cyrus/mailing-list.html

Re: squatter running longer than 24 hours

[David Carter]

On Sun, 21 Oct 2007, Vincent Fox wrote:

> I have seen squatter run more than 24 hours.
>
> This is on a large mail filesystem.  I've seen it start up a
> second one while the first is still running.  Should I:
>
> 1) Forget about squatter
> 2) Remove from cyrus.conf, run from cron every other day
> 3) Find some option to cyrus.conf for same effect as #2?

I squat a fraction of mailboxes each night using:

http://www-uxsup.csx.cam.ac.uk/~dpc22/cyrus/patches/2.3.8/squatter.patch

For example:

   squatter -s -m 0 -M 7

would update the squat indexes for 1 in 7 mailboxes, based on modulo 
arithmetic on the mailbox UniqueID.

squatter would really benefit from incremental updates. At the moment a 
single new message in a mailbox containing 20k messages causes it to read 
in all the existing messages in order to regenerate the index.

Unfortunately, the code is rather impenetrable. I infer that it is 
collecting information about adjacent characters in the message body. 
Presumably a 5 character search term provides 4 required pairing as a 
prefilter from the squat engine before message by message search kicks in.

-- 
David Carter Email: [EMAIL PROTECTED]
University Computing Service,Phone: (01223) 334502
New Museums Site, Pembroke Street,   Fax:   (01223) 334679
Cambridge UK. CB2 3QH.

Cyrus Home Page: http://cyrusimap.web.cmu.edu/
Cyrus Wiki/FAQ: http://cyrusimap.web.cmu.edu/twiki
List Archives/Info: http://asg.web.cmu.edu/cyrus/mailing-list.html

squatter running longer than 24 hours

[Vincent Fox]

I have seen squatter run more than 24 hours.

This is on a large mail filesystem.  I've seen it start up a
second one while the first is still running.  Should I:

1) Forget about squatter
2) Remove from cyrus.conf, run from cron every other day
3) Find some option to cyrus.conf for same effect as #2?


Cyrus Home Page: http://cyrusimap.web.cmu.edu/
Cyrus Wiki/FAQ: http://cyrusimap.web.cmu.edu/twiki
List Archives/Info: http://asg.web.cmu.edu/cyrus/mailing-list.html

Re: squatter problem

[Robert Banz]

>
> Are you running the command line squatter as root?  If so, then maybe
> there's a file early in the squatter run that's root owned and causing
> squatter to abort when run as the cyrus user - I'm afriad I don't know
> much about squatter, but that would be my first suspicion.

> Another theory - do you have a custom imapd.conf for the master
> process?  If so, the hand run squatter may be reading a different
> config file from the one run by master.
>
> Finally - have you tried changing the squatter command that master
> runs to do an strace dump to a file somewhere so you can see what
> it was trying?

I'm a good boy and, of course, don't run anything with cyrus under  
anyone other than cyrus ;)  And no funky multiple imapd.confs.

However, here's what I think may be going on...  I've got my  
cyr_expire process *and* squatter running at the same time.  I  
decided while I was sitting around this afternoon, to change sqatter  
start time to run, well, right then.  Squatter started running and  
seems to have not had a problem.

So, this begs my question:  Could there be an interaction between  
squatter and cyr_expire, where if they hit the same mailbox at the  
same time, could cause one (squatter) to decide that it doesn't have  
any more work to do?

-rob


Cyrus Home Page: http://cyrusimap.web.cmu.edu/
Cyrus Wiki/FAQ: http://cyrusimap.web.cmu.edu/twiki
List Archives/Info: http://asg.web.cmu.edu/cyrus/mailing-list.html

Re: squatter problem

[Bron Gondwana]

On Wed, Sep 05, 2007 at 03:15:33PM -0400, Robert Banz wrote:
> 
> On Sep 5, 2007, at 14:49, Michael D. Sofka wrote:
> 
> > On Wednesday 05 September 2007 09:44:16 am Robert Banz wrote:
> >
> >> Sep  5 02:00:00 ms1.mail.umbc.edu squatter[29759]: [ID 454541
> >> local6.debug] skipping mailbox user/a28/Spam
> >> Sep  5 02:00:00 ms1.mail.umbc.edu squatter[29759]: [ID 454541
> >> local6.debug] skipping mailbox user/a28/Trash
> >> Sep  5 02:00:00 ms1.mail.umbc.edu squatter[29759]: [ID 811920
> >> local6.notice] done indexing mailboxes
> >>
> >> Squatter seems to think its work is done rather quickly when running
> >> under the master's control, but if I run the same command line ( /
> >> local/cyrus/bin/squatter -s ) from the command line, it happily chugs
> >> along, indexing and indexing.
> >
> > I've been seeing the same behavior on both our back-end servers.
> > We're running Cyrus 2.2.12.
> 
> I'm up on 2.3.8 with a few of the fastmail.fm patches -- sorry,  
> forgot to include that in the mix.

Are you running the command line squatter as root?  If so, then maybe
there's a file early in the squatter run that's root owned and causing
squatter to abort when run as the cyrus user - I'm afriad I don't know
much about squatter, but that would be my first suspicion.

Another theory - do you have a custom imapd.conf for the master
process?  If so, the hand run squatter may be reading a different
config file from the one run by master.

Finally - have you tried changing the squatter command that master
runs to do an strace dump to a file somewhere so you can see what
it was trying?

Bron.

Cyrus Home Page: http://cyrusimap.web.cmu.edu/
Cyrus Wiki/FAQ: http://cyrusimap.web.cmu.edu/twiki
List Archives/Info: http://asg.web.cmu.edu/cyrus/mailing-list.html

Re: squatter problem

[Michael D. Sofka]

On Wednesday 05 September 2007 09:44:16 am Robert Banz wrote:

> Sep  5 02:00:00 ms1.mail.umbc.edu squatter[29759]: [ID 454541
> local6.debug] skipping mailbox user/a28/Spam
> Sep  5 02:00:00 ms1.mail.umbc.edu squatter[29759]: [ID 454541
> local6.debug] skipping mailbox user/a28/Trash
> Sep  5 02:00:00 ms1.mail.umbc.edu squatter[29759]: [ID 811920
> local6.notice] done indexing mailboxes
>
> Squatter seems to think its work is done rather quickly when running
> under the master's control, but if I run the same command line ( /
> local/cyrus/bin/squatter -s ) from the command line, it happily chugs
> along, indexing and indexing.

I've been seeing the same behavior on both our back-end servers.
We're running Cyrus 2.2.12.

Mike
-- 
Michael D. Sofka   [EMAIL PROTECTED]
C&MT Sr. Systems Programmer,   Email, TeX, Epistemology
Rensselaer Polytechnic Institute, Troy, NY.  http://www.rpi.edu/~sofkam/

Cyrus Home Page: http://cyrusimap.web.cmu.edu/
Cyrus Wiki/FAQ: http://cyrusimap.web.cmu.edu/twiki
List Archives/Info: http://asg.web.cmu.edu/cyrus/mailing-list.html

Re: squatter problem

[Robert Banz]

On Sep 5, 2007, at 14:49, Michael D. Sofka wrote:

> On Wednesday 05 September 2007 09:44:16 am Robert Banz wrote:
>
>> Sep  5 02:00:00 ms1.mail.umbc.edu squatter[29759]: [ID 454541
>> local6.debug] skipping mailbox user/a28/Spam
>> Sep  5 02:00:00 ms1.mail.umbc.edu squatter[29759]: [ID 454541
>> local6.debug] skipping mailbox user/a28/Trash
>> Sep  5 02:00:00 ms1.mail.umbc.edu squatter[29759]: [ID 811920
>> local6.notice] done indexing mailboxes
>>
>> Squatter seems to think its work is done rather quickly when running
>> under the master's control, but if I run the same command line ( /
>> local/cyrus/bin/squatter -s ) from the command line, it happily chugs
>> along, indexing and indexing.
>
> I've been seeing the same behavior on both our back-end servers.
> We're running Cyrus 2.2.12.

I'm up on 2.3.8 with a few of the fastmail.fm patches -- sorry,  
forgot to include that in the mix.

-rob


Cyrus Home Page: http://cyrusimap.web.cmu.edu/
Cyrus Wiki/FAQ: http://cyrusimap.web.cmu.edu/twiki
List Archives/Info: http://asg.web.cmu.edu/cyrus/mailing-list.html

squatter problem

[Robert Banz]

Hidey ho everybody,

So, one of my backend mailservers in my cyrus cluster is doing  
something silly:

Sep  5 02:00:00 ms1.mail.umbc.edu master[29759]: [ID 392559  
local6.debug] about to exec /local/cyrus/bin/squatter
Sep  5 02:00:00 ms1.mail.umbc.edu squatter[29759]: [ID 621814  
local6.notice] indexing mailboxes
Sep  5 02:00:00 ms1.mail.umbc.edu squatter[29759]: [ID 454541  
local6.debug] skipping mailbox shared/test
Sep  5 02:00:00 ms1.mail.umbc.edu squatter[29759]: [ID 454541  
local6.debug] skipping mailbox user/a11
...
Sep  5 02:00:00 ms1.mail.umbc.edu squatter[29759]: [ID 454541  
local6.debug] skipping mailbox user/a28/Spam
Sep  5 02:00:00 ms1.mail.umbc.edu squatter[29759]: [ID 454541  
local6.debug] skipping mailbox user/a28/Trash
Sep  5 02:00:00 ms1.mail.umbc.edu squatter[29759]: [ID 811920  
local6.notice] done indexing mailboxes

Squatter seems to think its work is done rather quickly when running  
under the master's control, but if I run the same command line ( / 
local/cyrus/bin/squatter -s ) from the command line, it happily chugs  
along, indexing and indexing.  I'm not seeing this behavior on any of  
my other backend servers...  Its obviously not dying of some horrible  
segfault or bus error, since it gets to log the 'done indexing  
mailboxes' message -- but what could possibly give it the idea that  
its done?

-rob


Cyrus Home Page: http://cyrusimap.web.cmu.edu/
Cyrus Wiki/FAQ: http://cyrusimap.web.cmu.edu/twiki
List Archives/Info: http://asg.web.cmu.edu/cyrus/mailing-list.html

Re: squatter segfaults on large mailboxes - FreeBSD

[Greg A. Woods]

At Sun, 22 Apr 2007 00:28:29 +0200, Per olof Ljungmark wrote:
Subject: Re: squatter segfaults on large mailboxes - FreeBSD
> 
> Sorry, no. Although the system itself is configured to dump cores there 
> are no traces left behind here. This is a test setup so it is likely 
> that I could recreate the scenario to get a dump if it is of any general 
> interest?

I should hope all Cyrus developers have at least some interest in making
the code more robust!  :-)

It scares the hell out of me whenever I find that some network daemon
code I run is making any kind of assumptions with the data it reads,
even from local disks, such that a violation of those assumptions can
lead to a core dump.

-- 
Greg A. Woods

H:+1 416 218-0098 W:+1 416 489-5852 x122 VE3TCP RoboHack <[EMAIL PROTECTED]>
Planix, Inc. <[EMAIL PROTECTED]>   Secrets of the Weird <[EMAIL PROTECTED]>

Cyrus Home Page: http://cyrusimap.web.cmu.edu/
Cyrus Wiki/FAQ: http://cyrusimap.web.cmu.edu/twiki
List Archives/Info: http://asg.web.cmu.edu/cyrus/mailing-list.html

Re: squatter segfaults on large mailboxes - FreeBSD

[Per olof Ljungmark]

Greg A. Woods wrote:

At Fri, 20 Apr 2007 21:45:05 +0200, Per olof Ljungmark wrote:
Subject: Re: squatter segfaults on large mailboxes - FreeBSD

The error is
Indexing mailbox user/spamdump/archive/2006/08... Segmentation fault
kernel: pid 6988 (squatter), uid 60: exited on signal 11

Never mind - corrupted mailbox - sorry for the noise.


It still shouldn't crash -- that's still a _major_ bug!

Maybe you've got a core dump that you can analyze?  At least to get a
stack backtrace and a dump of the local variables, etc.?


Sorry, no. Although the system itself is configured to dump cores there 
are no traces left behind here. This is a test setup so it is likely 
that I could recreate the scenario to get a dump if it is of any general 
interest?



Cyrus Home Page: http://cyrusimap.web.cmu.edu/
Cyrus Wiki/FAQ: http://cyrusimap.web.cmu.edu/twiki
List Archives/Info: http://asg.web.cmu.edu/cyrus/mailing-list.html

Re: squatter segfaults on large mailboxes - FreeBSD

[Greg A. Woods]

At Fri, 20 Apr 2007 21:45:05 +0200, Per olof Ljungmark wrote:
Subject: Re: squatter segfaults on large mailboxes - FreeBSD
> 
> > The error is
> > Indexing mailbox user/spamdump/archive/2006/08... Segmentation fault
> > kernel: pid 6988 (squatter), uid 60: exited on signal 11
> 
> Never mind - corrupted mailbox - sorry for the noise.

It still shouldn't crash -- that's still a _major_ bug!

Maybe you've got a core dump that you can analyze?  At least to get a
stack backtrace and a dump of the local variables, etc.?

-- 
Greg A. Woods

H:+1 416 218-0098 W:+1 416 489-5852 x122 VE3TCP RoboHack <[EMAIL PROTECTED]>
Planix, Inc. <[EMAIL PROTECTED]>   Secrets of the Weird <[EMAIL PROTECTED]>

Cyrus Home Page: http://cyrusimap.web.cmu.edu/
Cyrus Wiki/FAQ: http://cyrusimap.web.cmu.edu/twiki
List Archives/Info: http://asg.web.cmu.edu/cyrus/mailing-list.html

Re: squatter segfaults on large mailboxes - FreeBSD

[Per olof Ljungmark]

The error is
Indexing mailbox user/spamdump/archive/2006/08... Segmentation fault
kernel: pid 6988 (squatter), uid 60: exited on signal 11


Never mind - corrupted mailbox - sorry for the noise.

Cyrus Home Page: http://cyrusimap.web.cmu.edu/
Cyrus Wiki/FAQ: http://cyrusimap.web.cmu.edu/twiki
List Archives/Info: http://asg.web.cmu.edu/cyrus/mailing-list.html

squatter segfaults on large mailboxes - FreeBSD

[Per olof Ljungmark]

Cyrus-IMAP 2.3.8 with UOA autocreate patches
FreeBSD 6-STABLE

The command was:
su cyrus -c "/usr/local/cyrus/bin/squatter -r -s -v user"
The error is
Indexing mailbox user/spamdump/archive/2006/08... Segmentation fault
kernel: pid 6988 (squatter), uid 60: exited on signal 11

There is free memory available when this happens but still it could be a 
system limit somehow. Anyone else out there running FreeBSD that could 
give a hint?


Have two more boxes where this works fine on same mailbox but they do 
have a bit more RAM.


Thanks,

Cyrus Home Page: http://cyrusimap.web.cmu.edu/
Cyrus Wiki/FAQ: http://cyrusimap.web.cmu.edu/twiki
List Archives/Info: http://asg.web.cmu.edu/cyrus/mailing-list.html