Re: v2.3.11.3 solr plugin search via MUA fails to match accented ascii characters; cmd line exec of `doveadm fts lookup` PANICs (assertion failed) [proposed patch]
On 11/2/20 9:03 AM, John Fawcett wrote: the panic on doveadm fts lookup is to be expected and solved by my previous patch. snip all noted. atm, I've managed to get solr-backend indexing, and cmd-line searchable with doveadm, correctly finding the unfolded, accented characters in my 'test'. so for my 'admin' needs, that's good enuf for now. waiting for your patch to get picked up into what seems like a new-release-coming-soon(ish) seems a good idea. i'll likely spin up a dovecot/master DIY-build to explore, anyway. and, it's all somewhat moot for end-users using TBird in that its backend server-side search is broken anyway. time to take a pause and leave it alone for a bit. thx!
Re: v2.3.11.3 solr plugin search via MUA fails to match accented ascii characters; cmd line exec of `doveadm fts lookup` PANICs (assertion failed) [proposed patch]
On 11/1/20 10:35 AM, John Fawcett wrote: Yes, getting more data about any potential problem would be useful. Just to clarify: I have a fully working search setup for some time now over various dovecot releases, so no patches needed to get it working. My setup does use fts plugin and fts-solr plugin, but it does not use lib-fts functionality (that has many features for example it was stopping you indexing excluded words like tambien). On my setup without lib-fts everything goes to solr which does the work of indexing without all the features of lib-fts. withOUT libfts - fts_solr = url=https://solr.presence-group.net:8984/solr/dovecot/ use_libfts soft_commit=yes batch_size=250 + fts_solr = url=https://solr.presence-group.net:8984/solr/dovecot/ soft_commit=yes batch_size=250 and unmodified dovecot-provided schema/config, /bin/cp -af /usr/share/doc/dovecot/solr-config-7.7.0.xml /path/to/solr/data/dovecot/conf/solrconfig.xml /bin/cp -af /usr/share/doc/dovecot/solr-schema-7.7.0.xml /path/to/solr/data/dovecot/conf/schema.xml i suspect my config's now more similar to yours. checking, doveadm fts rescan -u testu...@example.com doveadm index -u testu...@example.com -q '*' as before doveadm fts lookup -u testu...@example.com body "tésting" panics, doveadm(testu...@example.com): Panic: file mail-storage.c: line 2112 (mailbox_get_open_status): assertion failed: (box->opened) doveadm(testu...@example.com): Error: Raw backtrace: /usr/lib64/dovecot/libdovecot.so.0(backtrace_append+0x46) [0x7f7829b81cc6] -> /usr/lib64/dovecot/libdovecot.so.0(backtrace_get+0x22) [0x7f7829b81de2] -> /usr/lib64/dovecot/libdovecot.so.0(+0x10025b) [0x7f7829b8b25b] -> /usr/lib64/dovecot/libdovecot.so.0(+0x100297) [0x7f7829b8b297] -> /usr/lib64/dovecot/libdovecot.so.0(+0x59bc6) [0x7f7829ae4bc6] -> /usr/lib64/dovecot/libdovecot-storage.so.0(+0x4779e) [0x7f7829c9879e] -> /usr/lib64/dovecot/lib21_fts_solr_plugin.so(+0x5849) [0x7f78296ea849] -> /usr/lib64/dovecot/lib20_fts_plugin.so(fts_backend_lookup+0x51) [0x7f782930b7c1] -> /usr/lib64/dovecot/doveadm/lib20_doveadm_fts_plugin.so(+0x3280) [0x7f78270d0280] -> doveadm(+0x343cd) [0x55aa57edc3cd] -> doveadm(+0x34fe0) [0x55aa57edcfe0] -> doveadm(doveadm_cmd_ver2_to_mail_cmd_wrapper+0x22d) [0x55aa57edde2d] -> doveadm(doveadm_cmd_run_ver2+0x4e8) [0x55aa57eee8d8] -> doveadm(doveadm_cmd_try_run_ver2+0x3e) [0x55aa57eee92e] -> doveadm(main+0x1d4) [0x55aa57ecccf4] -> /lib64/libc.so.6(__libc_start_main+0xf2) [0x7f7829746042] -> doveadm(_start+0x2e) [0x55aa57ecd1ce] Aborted but search, even for accented characters, doveadm search -u testu...@example.com subject "tésting" 42d73837f133a05fad4df8839f03 1 813ef60e984f1b5f5fc25439fba4 293 doveadm search -u testu...@example.com body "tésting" ba899d0cfe33a05fbe4df8839f03 1 813ef60e984f1b5f5fc25439fba4 293 appears to work. next, to get tokenization -- at least email/url (UAX29URLEmailTokenizer) -- and lowercase & icu normalization working and verified.
Re: v2.3.11.3 solr plugin search via MUA fails to match accented ascii characters; cmd line exec of `doveadm fts lookup` PANICs (assertion failed) [proposed patch]
On 11/2/20 5:13 AM, Aki Tuomi wrote:
So what's the recommendation? use use_libfts, or not?
It's a choice. You can let solr perform the tokenization etc. or you can let
dovecot do it. There is no recommendation when using solr.
atm, my fts plugin conf is
plugin {
fts = solr
fts_solr = url=https://solr.example.com:8984/solr/dovecot/
use_libfts soft_commit=yes batch_size=250
fts_autoindex = yes
fts_autoindex_max_recent_msgs = 999
fts_autoindex_exclude = \Junk
fts_autoindex_exclude2 = \Trash
fts_enforced = yes
fts_filters = normalizer-icu snowball stopwords
fts_filters_en = lowercase snowball english-possessive stopwords
fts_languages = en es de fr it pt
fts_language_config = /usr/share/libexttextcat/fpdb.conf
fts_tokenizers = generic email-address
fts_tokenizer_generic = algorithm=simple
}
It seems that use_libfts is broken with solr due to reasons, so I guess the
only option for now is not to use it.
if I
- fts_solr = url=https://solr.example.com:8984/solr/dovecot/
use_libfts soft_commit=yes batch_size=250
+ fts_solr = url=https://solr.example.com:8984/solr/dovecot/
soft_commit=yes batch_size=250
how much of that^ config needs to be *removed* &/or simply stops functioning?
from its introduction
dovecot-2.2: fts-solr: fts_solr=use_libfts send data to Solr via...
https://dovecot.org/list/dovecot-cvs/2015-April/025715.html
fts-solr: fts_solr=use_libfts send data to Solr via
space-separated tokens.In this case Solr should be configured to not do any
kind of filtering anduse only WhitespaceTokenizerFactory
it's unclear to me what the effect of NOT using it is.
Reading
https://doc.dovecot.org/configuration_manual/fts/?highlight=fts%20solr%20plugin#dovecot-fts-architecture
https://doc.dovecot.org/configuration_manual/fts/tokenization/#fts-tokenization
refers to all of
fts_languages
fts_tokenizers
fts_tokenizer_generic
fts_filters
fts_filters_en
WithOUT 'use_libfts' which of those^ need modification/removal from dovecot
config?
Re: v2.3.11.3 solr plugin search via MUA fails to match accented ascii characters; cmd line exec of `doveadm fts lookup` PANICs (assertion failed) [proposed patch]
> On 02/11/2020 15:11 PGNet Dev wrote: > > > On 11/2/20 12:44 AM, Aki Tuomi wrote: > > you should try removing use_libfts from your config line and let solr do > > that part. > > sry, i'm a bit confused. > > you'd suggested I _add_ it, > > https://dovecot.org/pipermail/dovecot/2020-October/120258.html > > > I can reproduce your problem with the `fts lookup` command. Luckily it's > > equivalent to running `doveadm search`. I'll open a bug about this. > > Dovecot FTS tokenization is not done, unless you have `use_libfts` in > > fts_solr setting, in your case > > fts_solr = url=https://solr.example.com:8984/solr/dovecot/ use_libfts > > Without this, everything is sent to to solr as-is, which is then expected > > to do all the work. > > So what's the recommendation? use use_libfts, or not? It's a choice. You can let solr perform the tokenization etc. or you can let dovecot do it. There is no recommendation when using solr. It seems that use_libfts is broken with solr due to reasons, so I guess the only option for now is not to use it. Aki
Re: v2.3.11.3 solr plugin search via MUA fails to match accented ascii characters; cmd line exec of `doveadm fts lookup` PANICs (assertion failed) [proposed patch]
On 11/2/20 12:44 AM, Aki Tuomi wrote: you should try removing use_libfts from your config line and let solr do that part. sry, i'm a bit confused. you'd suggested I _add_ it, https://dovecot.org/pipermail/dovecot/2020-October/120258.html I can reproduce your problem with the `fts lookup` command. Luckily it's equivalent to running `doveadm search`. I'll open a bug about this. Dovecot FTS tokenization is not done, unless you have `use_libfts` in fts_solr setting, in your case fts_solr = url=https://solr.example.com:8984/solr/dovecot/ use_libfts Without this, everything is sent to to solr as-is, which is then expected to do all the work. So what's the recommendation? use use_libfts, or not?
Re: v2.3.11.3 solr plugin search via MUA fails to match accented ascii characters; cmd line exec of `doveadm fts lookup` PANICs (assertion failed) [proposed patch]
On 31/10/2020 17:55, John Fawcett wrote:
> On 31/10/2020 04:57, PGNet Dev wrote:
>> On 10/18/20 10:28 PM, Aki Tuomi wrote:
doveadm(myu...@example.com): Panic: file mail-storage.c: line
2112 (mailbox_get_open_status): assertion failed: (box->opened)
>> ...
>>
>>> I can reproduce your problem with the `fts lookup` command. Luckily
>>> it's equivalent to running `doveadm search`. I'll open a bug about this.
>> Can you provide any status on the bug/fix?
>>
>> Thanks.
> I can contribute a patch that solves the segfault. Unfortunately though
> fts search may be more broken than this. It does not give me search
> results, even though I see it querying solr and getting hits.
>
> diff -ur dovecot-2.3.11.3-orig/src/plugins/fts/doveadm-fts.c
> dovecot-2.3.11.3-patch/src/plugins/fts/doveadm-fts.c
> --- dovecot-2.3.11.3-orig/src/plugins/fts/doveadm-fts.c 2020-08-12
> 14:20:41.0 +0200
> +++ dovecot-2.3.11.3-patch/src/plugins/fts/doveadm-fts.c 2020-10-31
> 17:52:09.019388695 +0100
> @@ -47,6 +47,14 @@
> i_array_init(&result.scores, 16);
>
> box = mailbox_alloc(info->ns->list, info->vname, 0);
> + mailbox_set_reason(box,"fts search");
> + if (mailbox_open(box) < 0) {
> + i_error("Couldn't open mailbox: %s",
> + mailbox_get_last_internal_error(box, NULL));
> + doveadm_mail_failed_error(ctx, MAIL_ERROR_TEMP);
> + return -1;
> + }
> +
> if (fts_backend_lookup(backend, box, ctx->search_args->args,
> FTS_LOOKUP_FLAG_AND_ARGS, &result) < 0) {
> i_error("fts lookup failed");
>
> On a more minor issue, with this patch if you search for a non existent
> mailbox, it does give a segfault for a different assert, in mail-user.c
> (*user)->refcount == 1.
>
> doveadm(j...@voipsupport.it): Error: Couldn't open mailbox: Mailbox
> doesn't exist: inboxx
> doveadm(j...@voipsupport.it): Panic: file mail-user.c: line 229
> (mail_user_deinit): assertion failed: ((*user)->refcount == 1)
> doveadm(j...@voipsupport.it): Error: Raw backtrace:
> /usr/local/lib/dovecot/libdovecot.so.0(backtrace_append+0x42)
> [0x7f35c44c3ee2] ->
> /usr/local/lib/dovecot/libdovecot.so.0(backtrace_get+0x1e)
> [0x7f35c44c3fee] -> /usr/local/lib/dovecot/libdovecot.so.0(+0xec53e)
> [0x7f35c44ce53e] -> /usr/local/lib/dovecot/libdovecot.so.0(+0xec581)
> [0x7f35c44ce581] -> /usr/local/lib/dovecot/libdovecot.so.0(i_fatal+0)
> [0x7f35c44254ea] ->
> /usr/local/lib/dovecot/libdovecot-storage.so.0(+0x56d87)
> [0x7f35c47e4d87] -> doveadm(+0x2cb28) [0x55c0eaa57b28] ->
> doveadm(+0x2d77c) [0x55c0eaa5877c] ->
> doveadm(doveadm_cmd_ver2_to_mail_cmd_wrapper+0x21d) [0x55c0eaa5960d] ->
> doveadm(doveadm_cmd_run_ver2+0x472) [0x55c0eaa6a372] ->
> doveadm(doveadm_cmd_try_run_ver2+0x37) [0x55c0eaa6a497] ->
> doveadm(main+0x1d4) [0x55c0eaa47c54] ->
> /lib64/libc.so.6(__libc_start_main+0xf5) [0x7f35c3d34555] ->
> doveadm(+0x1d0ef) [0x55c0eaa480ef]
> Aborted
>
> John
>
After further testing, I believe the fix for doveadm fts lookup command
is ok (at least on a virtual users setup - I don't have system users to
test this with). What I stated above is wrong: it does produce search
results - I missed them in the debugging because the format is a bit
different to doveadm search, but they are there.
So unless someone with system users reports something not working, I'd
recommend the above patch to resolve the mailbox not open
assert/segfault issue.
John
Re: v2.3.11.3 solr plugin search via MUA fails to match accented ascii characters; cmd line exec of `doveadm fts lookup` PANICs (assertion failed) [proposed patch]
> On 01/11/2020 20:35 John Fawcett wrote:
>
>
> On 01/11/2020 15:20, PGNet Dev wrote:
> > On 11/1/20 1:56 AM, John Fawcett wrote:
> >> At the moment I don't see other corrections needed in dovecot apart from
> >> command line doveadm fts which is not a show stopper. Via doveadm search
> >> I confirm - on my simple config - that search for accented or non
> >> accented characters works correctly as it does via imap connection.
> >
> > thx. hopefully it'll get considered for a next release soon.
> >
> >> Only thing I cannot vouch for is bringing dovecot fts library and config
> >> into the equation because my setup delegates almost everything to solr.
> >
> > do i understand correctly that you're solr-indexing your dovecot mail
> > store withOUT using dovecot fts plugin, and that -- with your
> > aforementioned patch -- doveadm successfully uses the resulting indexes?
> >
> > i hadn't yet seriously considered _circumventing_ fts plugin; if this^
> > does get resolved soonish, then it's not a big deal. if not, an
> > fts-plugin-less setup would be interesting to know more abt!
> >
> >> Can you get evidence of things not working? For example tests run with
> >> soft_commit configured - that's important since without it the updates
> >> don't show up immediately in searches, that do show that the update is
> >> happening in solr via solr log, but then search is not working on
> >> accented characters, despite it working on other text in the same
> >> message? The solr logs also show whether the text was found or not via
> >> the "hits=" value in the logged searches, for example:
> >>
> >> 2020-11-01 08:32:42.231 INFO (qtp24119573-21) [ x:dovecot]
> >> o.a.s.c.S.Request [dovecot] webapp=/solr path=/select
> >> params={q={!lucene+q.op%3DAND}body:también&fl=uid,score&sort=uid+asc&fq=%2Bbox:b1626f0fe8d9145e5410c54a863a+%2Buser:j...@voipsupport.it&rows=3202&wt=xml}
> >>
> >> hits=3 status=0 QTime=3
> >>
> >> But if no hits are found, then dovecot cannot be expected to display
> >> results. It still may be an indexing problem though.
> >
> > my current config has soft_commit enabled,
> >
> > fts_solr = url=https://solr.example.com:8984/solr/dovecot/
> > use_libfts soft_commit=yes batch_size=250
> >
> > i'll see abt getting some clearer test results ...
>
> Yes, getting more data about any potential problem would be useful.
>
> Just to clarify: I have a fully working search setup for some time now
> over various dovecot releases, so no patches needed to get it working.
>
> My setup does use fts plugin and fts-solr plugin, but it does not use
> lib-fts functionality (that has many features for example it was
> stopping you indexing excluded words like tambien). On my setup without
> lib-fts everything goes to solr which does the work of indexing without
> all the features of lib-fts.
>
> My setup is like this not because of issues in lib-fts, but because I
> never had the need for it. There is no evidence at the moment however
> that even with lib-fts enabled there are issues with dovecot indexing or
> searching.
>
> What is currently nor working is "doveadm fts" command line utility. But
> this is mitigated by being able to use a similar command line utility
> "doveadm search". The issue on "dovecot fts" command line utility has
> (so far as the available evidence suggests) no effect on indexing or
> imap searches.
>
> fyi my working configuration includes fts and fts_solr plugins
>
> mail_plugins = quota notify replication fts fts_solr
>
> (and those are also recalled in the various specific plugin settings for
> imap, lmtp ecc), The specific config I am using for fts and fts_solr is:
>
> fts = solr
> fts_enforced = yes
> fts_solr = url=https://u...@server.example.com:443/solr/dovecot/
> batch_size=500 soft_commit=no
>
> BTW I use soft_commit=no because I have periodic soft commits setup on
> solr and I accept that newly indexed text won't become searchable for up
> to that interval, but for your testing purposes much more useful as you
> have it.
>
> John
PGNet,
you should try removing use_libfts from your config line and let solr do that
part.
Aki
Re: v2.3.11.3 solr plugin search via MUA fails to match accented ascii characters; cmd line exec of `doveadm fts lookup` PANICs (assertion failed) [proposed patch]
On 01/11/2020 15:20, PGNet Dev wrote:
> On 11/1/20 1:56 AM, John Fawcett wrote:
>> At the moment I don't see other corrections needed in dovecot apart from
>> command line doveadm fts which is not a show stopper. Via doveadm search
>> I confirm - on my simple config - that search for accented or non
>> accented characters works correctly as it does via imap connection.
>
> thx. hopefully it'll get considered for a next release soon.
>
>> Only thing I cannot vouch for is bringing dovecot fts library and config
>> into the equation because my setup delegates almost everything to solr.
>
> do i understand correctly that you're solr-indexing your dovecot mail
> store withOUT using dovecot fts plugin, and that -- with your
> aforementioned patch -- doveadm successfully uses the resulting indexes?
>
> i hadn't yet seriously considered _circumventing_ fts plugin; if this^
> does get resolved soonish, then it's not a big deal. if not, an
> fts-plugin-less setup would be interesting to know more abt!
>
>> Can you get evidence of things not working? For example tests run with
>> soft_commit configured - that's important since without it the updates
>> don't show up immediately in searches, that do show that the update is
>> happening in solr via solr log, but then search is not working on
>> accented characters, despite it working on other text in the same
>> message? The solr logs also show whether the text was found or not via
>> the "hits=" value in the logged searches, for example:
>>
>> 2020-11-01 08:32:42.231 INFO (qtp24119573-21) [ x:dovecot]
>> o.a.s.c.S.Request [dovecot] webapp=/solr path=/select
>> params={q={!lucene+q.op%3DAND}body:también&fl=uid,score&sort=uid+asc&fq=%2Bbox:b1626f0fe8d9145e5410c54a863a+%2Buser:j...@voipsupport.it&rows=3202&wt=xml}
>>
>> hits=3 status=0 QTime=3
>>
>> But if no hits are found, then dovecot cannot be expected to display
>> results. It still may be an indexing problem though.
>
> my current config has soft_commit enabled,
>
> fts_solr = url=https://solr.example.com:8984/solr/dovecot/
> use_libfts soft_commit=yes batch_size=250
>
> i'll see abt getting some clearer test results ...
Yes, getting more data about any potential problem would be useful.
Just to clarify: I have a fully working search setup for some time now
over various dovecot releases, so no patches needed to get it working.
My setup does use fts plugin and fts-solr plugin, but it does not use
lib-fts functionality (that has many features for example it was
stopping you indexing excluded words like tambien). On my setup without
lib-fts everything goes to solr which does the work of indexing without
all the features of lib-fts.
My setup is like this not because of issues in lib-fts, but because I
never had the need for it. There is no evidence at the moment however
that even with lib-fts enabled there are issues with dovecot indexing or
searching.
What is currently nor working is "doveadm fts" command line utility. But
this is mitigated by being able to use a similar command line utility
"doveadm search". The issue on "dovecot fts" command line utility has
(so far as the available evidence suggests) no effect on indexing or
imap searches.
fyi my working configuration includes fts and fts_solr plugins
mail_plugins = quota notify replication fts fts_solr
(and those are also recalled in the various specific plugin settings for
imap, lmtp ecc), The specific config I am using for fts and fts_solr is:
fts = solr
fts_enforced = yes
fts_solr = url=https://u...@server.example.com:443/solr/dovecot/
batch_size=500 soft_commit=no
BTW I use soft_commit=no because I have periodic soft commits setup on
solr and I accept that newly indexed text won't become searchable for up
to that interval, but for your testing purposes much more useful as you
have it.
John
Re: v2.3.11.3 solr plugin search via MUA fails to match accented ascii characters; cmd line exec of `doveadm fts lookup` PANICs (assertion failed) [proposed patch]
On 11/1/20 1:56 AM, John Fawcett wrote:
At the moment I don't see other corrections needed in dovecot apart from
command line doveadm fts which is not a show stopper. Via doveadm search
I confirm - on my simple config - that search for accented or non
accented characters works correctly as it does via imap connection.
thx. hopefully it'll get considered for a next release soon.
Only thing I cannot vouch for is bringing dovecot fts library and config
into the equation because my setup delegates almost everything to solr.
do i understand correctly that you're solr-indexing your dovecot mail store
withOUT using dovecot fts plugin, and that -- with your aforementioned patch --
doveadm successfully uses the resulting indexes?
i hadn't yet seriously considered _circumventing_ fts plugin; if this^ does get
resolved soonish, then it's not a big deal. if not, an fts-plugin-less setup
would be interesting to know more abt!
Can you get evidence of things not working? For example tests run with
soft_commit configured - that's important since without it the updates
don't show up immediately in searches, that do show that the update is
happening in solr via solr log, but then search is not working on
accented characters, despite it working on other text in the same
message? The solr logs also show whether the text was found or not via
the "hits=" value in the logged searches, for example:
2020-11-01 08:32:42.231 INFO (qtp24119573-21) [ x:dovecot]
o.a.s.c.S.Request [dovecot] webapp=/solr path=/select
params={q={!lucene+q.op%3DAND}body:también&fl=uid,score&sort=uid+asc&fq=%2Bbox:b1626f0fe8d9145e5410c54a863a+%2Buser:j...@voipsupport.it&rows=3202&wt=xml}
hits=3 status=0 QTime=3
But if no hits are found, then dovecot cannot be expected to display
results. It still may be an indexing problem though.
my current config has soft_commit enabled,
fts_solr = url=https://solr.example.com:8984/solr/dovecot/ use_libfts
soft_commit=yes batch_size=250
i'll see abt getting some clearer test results ...
Re: v2.3.11.3 solr plugin search via MUA fails to match accented ascii characters; cmd line exec of `doveadm fts lookup` PANICs (assertion failed) [proposed patch]
On 31/10/2020 22:01, PGNet Dev wrote: > On 10/31/20 9:55 AM, John Fawcett wrote: >> I can contribute a patch that solves the segfault. Unfortunately though >> fts search may be more broken than this. It does not give me search >> results, even though I see it querying solr and getting hits. > > Thx -- hopefully it moves this in the right direction. > > Also on the 'good news' page, it appears there's been some progress on > Thunderbird's use of backend/server search, > > TBird "search on server" doesn't -- NO comm with backend > IMAP/SOLR; appears to be local-only search > https://bugzilla.mozilla.org/show_bug.cgi?id=1673928 > > "A fix for this is upcoming." > > Remains to be seen if the doveadm search issues, and implications on > backend problems, have any effect on the Thunderbird searches. At the moment I don't see other corrections needed in dovecot apart from command line doveadm fts which is not a show stopper. Via doveadm search I confirm - on my simple config - that search for accented or non accented characters works correctly as it does via imap connection. For the imap test you can take Thunderbird out of the equation by running another imap client, for example this three line php script (that relies on having the php imap extension installed) can be run from the command line with php -f filename.php and for me produces the same results as doveadm search.
Re: v2.3.11.3 solr plugin search via MUA fails to match accented ascii characters; cmd line exec of `doveadm fts lookup` PANICs (assertion failed) [proposed patch]
On 10/31/20 9:55 AM, John Fawcett wrote: I can contribute a patch that solves the segfault. Unfortunately though fts search may be more broken than this. It does not give me search results, even though I see it querying solr and getting hits. Thx -- hopefully it moves this in the right direction. Also on the 'good news' page, it appears there's been some progress on Thunderbird's use of backend/server search, TBird "search on server" doesn't -- NO comm with backend IMAP/SOLR; appears to be local-only search https://bugzilla.mozilla.org/show_bug.cgi?id=1673928 "A fix for this is upcoming." Remains to be seen if the doveadm search issues, and implications on backend problems, have any effect on the Thunderbird searches.
Re: v2.3.11.3 solr plugin search via MUA fails to match accented ascii characters; cmd line exec of `doveadm fts lookup` PANICs (assertion failed) [proposed patch]
On 31/10/2020 04:57, PGNet Dev wrote:
> On 10/18/20 10:28 PM, Aki Tuomi wrote:
>>> doveadm(myu...@example.com): Panic: file mail-storage.c: line
>>> 2112 (mailbox_get_open_status): assertion failed: (box->opened)
> ...
>
>> I can reproduce your problem with the `fts lookup` command. Luckily
>> it's equivalent to running `doveadm search`. I'll open a bug about this.
>
> Can you provide any status on the bug/fix?
>
> Thanks.
I can contribute a patch that solves the segfault. Unfortunately though
fts search may be more broken than this. It does not give me search
results, even though I see it querying solr and getting hits.
diff -ur dovecot-2.3.11.3-orig/src/plugins/fts/doveadm-fts.c
dovecot-2.3.11.3-patch/src/plugins/fts/doveadm-fts.c
--- dovecot-2.3.11.3-orig/src/plugins/fts/doveadm-fts.c 2020-08-12
14:20:41.0 +0200
+++ dovecot-2.3.11.3-patch/src/plugins/fts/doveadm-fts.c 2020-10-31
17:52:09.019388695 +0100
@@ -47,6 +47,14 @@
i_array_init(&result.scores, 16);
box = mailbox_alloc(info->ns->list, info->vname, 0);
+ mailbox_set_reason(box,"fts search");
+ if (mailbox_open(box) < 0) {
+ i_error("Couldn't open mailbox: %s",
+ mailbox_get_last_internal_error(box, NULL));
+ doveadm_mail_failed_error(ctx, MAIL_ERROR_TEMP);
+ return -1;
+ }
+
if (fts_backend_lookup(backend, box, ctx->search_args->args,
FTS_LOOKUP_FLAG_AND_ARGS, &result) < 0) {
i_error("fts lookup failed");
On a more minor issue, with this patch if you search for a non existent
mailbox, it does give a segfault for a different assert, in mail-user.c
(*user)->refcount == 1.
doveadm(j...@voipsupport.it): Error: Couldn't open mailbox: Mailbox
doesn't exist: inboxx
doveadm(j...@voipsupport.it): Panic: file mail-user.c: line 229
(mail_user_deinit): assertion failed: ((*user)->refcount == 1)
doveadm(j...@voipsupport.it): Error: Raw backtrace:
/usr/local/lib/dovecot/libdovecot.so.0(backtrace_append+0x42)
[0x7f35c44c3ee2] ->
/usr/local/lib/dovecot/libdovecot.so.0(backtrace_get+0x1e)
[0x7f35c44c3fee] -> /usr/local/lib/dovecot/libdovecot.so.0(+0xec53e)
[0x7f35c44ce53e] -> /usr/local/lib/dovecot/libdovecot.so.0(+0xec581)
[0x7f35c44ce581] -> /usr/local/lib/dovecot/libdovecot.so.0(i_fatal+0)
[0x7f35c44254ea] ->
/usr/local/lib/dovecot/libdovecot-storage.so.0(+0x56d87)
[0x7f35c47e4d87] -> doveadm(+0x2cb28) [0x55c0eaa57b28] ->
doveadm(+0x2d77c) [0x55c0eaa5877c] ->
doveadm(doveadm_cmd_ver2_to_mail_cmd_wrapper+0x21d) [0x55c0eaa5960d] ->
doveadm(doveadm_cmd_run_ver2+0x472) [0x55c0eaa6a372] ->
doveadm(doveadm_cmd_try_run_ver2+0x37) [0x55c0eaa6a497] ->
doveadm(main+0x1d4) [0x55c0eaa47c54] ->
/lib64/libc.so.6(__libc_start_main+0xf5) [0x7f35c3d34555] ->
doveadm(+0x1d0ef) [0x55c0eaa480ef]
Aborted
John
