Re: [Dovecot] Possible mdbox corruption
On 1/6/2012 12:32 PM, Daniel L. Miller wrote:
On 1/6/2012 9:36 AM, Timo Sirainen wrote:
On 6.1.2012, at 19.30, Daniel L. Miller wrote:
Jan 6 09:22:42 bubba dovecot: indexer-worker(us...@domain.com):
Error: fts_solr: Indexing failed: 400 Illegal character ((CTRL-CHAR,
code 18)) at [row,col {unknown-source}]: [482765,16]
Jan 6 09:22:42 bubba dovecot: indexer-worker: Error:
Google seems to indicate that Solr cannot handle invalid
characters - and that it is the responsibility of the calling
program to strip out such. A quick search shows me a both an
individual character comparison in Java and a regex used for the
purpose. Is there any illegal character protection in the Dovecot
Solr plugin?
Yes, there is. So I'm not really sure what it's complaining about.
Are you using the solr or solr_old backend?
Solr.
plugin {
fts = solr
fts_solr = url=http://localhost:8983/solr/
}
Now seeing:
Jan 8 15:40:09 bubba dovecot: imap(us...@domain.com): Error: fts_solr:
Lookup failed: 400 undefined field CC
Jan 8 15:40:09 bubba dovecot: imap: Error:
--
Daniel
Re: [Dovecot] Possible mdbox corruption
On 5.1.2012, at 2.24, Daniel L. Miller wrote: I thought I had cleared out the corruption I had before - perhaps I was mistaken. What steps should I take to help locate these issues? Currently using 2.1rc1. I see the following errors in my logs, including out of memory and message size issues (at 15:30): .. Jan 4 05:17:17 bubba dovecot: master: Error: service(indexer-worker): child 10896 returned error 83 (Out of memory (vsz_limit=256 MB, you may need to increase it)) Jan 4 06:17:17 bubba dovecot: indexer-worker(us...@domain.com): Fatal: pool_system_realloc(134217728): Out of memory The problem is clearly that index-worker's vsz_limit is too low. Increase it (or default_vsz_limit).
Re: [Dovecot] Possible mdbox corruption
On 6.1.2012, at 12.55, Timo Sirainen wrote: Jan 4 05:17:17 bubba dovecot: master: Error: service(indexer-worker): child 10896 returned error 83 (Out of memory (vsz_limit=256 MB, you may need to increase it)) Jan 4 06:17:17 bubba dovecot: indexer-worker(us...@domain.com): Fatal: pool_system_realloc(134217728): Out of memory The problem is clearly that index-worker's vsz_limit is too low. Increase it (or default_vsz_limit). Although the source of the out-of-memory /usr/local/lib/dovecot/libdovecot.so.0(buffer_write+0x7c) [0x7f0ec1a550ec] - /usr/local/lib/dovecot/lib21_fts_solr_plugin.so(+0x3292) [0x7f0ec024f292] - is something that shouldn't really be happening. I guess the Solr plugin wastes memory unnecessarily, I'll see what I can do about it. But for now just increase vsz limit.
Re: [Dovecot] Possible mdbox corruption
On Fri, 2012-01-06 at 12:57 +0200, Timo Sirainen wrote: On 6.1.2012, at 12.55, Timo Sirainen wrote: Jan 4 05:17:17 bubba dovecot: master: Error: service(indexer-worker): child 10896 returned error 83 (Out of memory (vsz_limit=256 MB, you may need to increase it)) Jan 4 06:17:17 bubba dovecot: indexer-worker(us...@domain.com): Fatal: pool_system_realloc(134217728): Out of memory The problem is clearly that index-worker's vsz_limit is too low. Increase it (or default_vsz_limit). Although the source of the out-of-memory /usr/local/lib/dovecot/libdovecot.so.0(buffer_write+0x7c) [0x7f0ec1a550ec] - /usr/local/lib/dovecot/lib21_fts_solr_plugin.so(+0x3292) [0x7f0ec024f292] - is something that shouldn't really be happening. I guess the Solr plugin wastes memory unnecessarily, I'll see what I can do about it. But for now just increase vsz limit. I don't see any obvious reason why it would be using a lot of memory, unless you have a message that has huge (MIME) headers. See if http://hg.dovecot.org/dovecot-2.1/rev/380b0667e0a5 helps / logs a warning about it.
Re: [Dovecot] Possible mdbox corruption
On 1/6/2012 2:57 AM, Timo Sirainen wrote:
On 6.1.2012, at 12.55, Timo Sirainen wrote:
Jan 4 05:17:17 bubba dovecot: master: Error: service(indexer-worker): child
10896 returned error 83 (Out of memory (vsz_limit=256 MB, you may need to
increase it))
Jan 4 06:17:17 bubba dovecot: indexer-worker(us...@domain.com): Fatal:
pool_system_realloc(134217728): Out of memory
The problem is clearly that index-worker's vsz_limit is too low. Increase it
(or default_vsz_limit).
Although the source of the out-of-memory
/usr/local/lib/dovecot/libdovecot.so.0(buffer_write+0x7c) [0x7f0ec1a550ec] -
/usr/local/lib/dovecot/lib21_fts_solr_plugin.so(+0x3292) [0x7f0ec024f292] -
is something that shouldn't really be happening. I guess the Solr plugin wastes
memory unnecessarily, I'll see what I can do about it. But for now just
increase vsz limit.
I set default_vsz_limit = 1024M. Those errors appear gone - but I do
have messages like:
Jan 6 09:22:42 bubba dovecot: indexer-worker(us...@domain.com): Error:
fts_solr: Indexing failed: 400 Illegal character ((CTRL-CHAR, code 18))
at [row,col {unknown-source}]: [482765,16]
Jan 6 09:22:42 bubba dovecot: indexer-worker: Error:
Google seems to indicate that Solr cannot handle invalid characters -
and that it is the responsibility of the calling program to strip out
such. A quick search shows me a both an individual character comparison
in Java and a regex used for the purpose. Is there any illegal
character protection in the Dovecot Solr plugin?
--
Daniel
Re: [Dovecot] Possible mdbox corruption
On 6.1.2012, at 19.30, Daniel L. Miller wrote:
Jan 6 09:22:42 bubba dovecot: indexer-worker(us...@domain.com): Error:
fts_solr: Indexing failed: 400 Illegal character ((CTRL-CHAR, code 18)) at
[row,col {unknown-source}]: [482765,16]
Jan 6 09:22:42 bubba dovecot: indexer-worker: Error:
Google seems to indicate that Solr cannot handle invalid characters - and
that it is the responsibility of the calling program to strip out such. A
quick search shows me a both an individual character comparison in Java and a
regex used for the purpose. Is there any illegal character protection in
the Dovecot Solr plugin?
Yes, there is. So I'm not really sure what it's complaining about. Are you
using the solr or solr_old backend?
Re: [Dovecot] Possible mdbox corruption
On 1/6/2012 9:36 AM, Timo Sirainen wrote:
On 6.1.2012, at 19.30, Daniel L. Miller wrote:
Jan 6 09:22:42 bubba dovecot: indexer-worker(us...@domain.com): Error:
fts_solr: Indexing failed: 400 Illegal character ((CTRL-CHAR, code 18)) at
[row,col {unknown-source}]: [482765,16]
Jan 6 09:22:42 bubba dovecot: indexer-worker: Error:
Google seems to indicate that Solr cannot handle invalid characters - and that it is
the responsibility of the calling program to strip out such. A quick search shows me a both an
individual character comparison in Java and a regex used for the purpose. Is there any
illegal character protection in the Dovecot Solr plugin?
Yes, there is. So I'm not really sure what it's complaining about. Are you using the
solr or solr_old backend?
Solr.
plugin {
fts = solr
fts_solr = url=http://localhost:8983/solr/
}
--
Daniel
