Re: Indexer error after upgrade to 2.3.11.3 [trial patch]

[Patrik Peng]

On 31.10.20 00:18, Scott Q. wrote:

I have implemented this change and the core dumps went away.

It appears so far that it's indeed the right fix.

Has this change been introduced in 2.3.11.3 ?

Thanks!

On Tuesday, 27/10/2020 at 08:20 John Fawcett wrote:

On 22/10/2020 10:23, John Fawcett wrote:

On 21/10/2020 19:00, John Fawcett wrote:

On 21/10/2020 16:44, Patrik Peng wrote:

On 16.10.20 18:34, Patrik Peng wrote:

On 16.10.20 18:00, Scott Q. wrote:

This reminds me, the way I was able to reproduce this
consistently was by having large headers ( 100+ lines ).


On Friday, 16/10/2020 at 11:49 Patrik Peng wrote:

On 19.08.20 17:37, Josef 'Jeff' Sipek wrote:


On Wed, Aug 19, 2020 at 17:03:57 +0200, Alessio Cecchi wrote:

Hi,
after the upgrade to Dovecot 2.3.11.3, from 2.3.10.1, I see frequently
these errors from different users:

It looks like this has been around for a while and you just got unlucky 
and
started seeing this now.  Here's a quick & dirty patch that should fix 
this.
If you can try it, let us know how it went.
Jeff.
diff --git a/src/plugins/fts-solr/solr-connection.c 
b/src/plugins/fts-solr/solr-connection.c
index 
ae720b5e2870a852c1b6c440939e3c7c0fa72b5c..9d364f93e2cd1b716b9ab61bd39656a6c5b1ea04
 100644
--- a/src/plugins/fts-solr/solr-connection.c
+++ b/src/plugins/fts-solr/solr-connection.c
@@ -103,7 +103,7 @@ int solr_connection_init(const struct 
fts_solr_settings *solr_set,
http_set.ssl = ssl_client_set;
http_set.debug = solr_set->debug;
http_set.rawlog_dir = solr_set->rawlog_dir;
-   solr_http_client = http_client_init(_set);
+   solr_http_client = http_client_init_private(_set);
}
*conn_r = conn;
diff --git a/src/plugins/fts/fts-parser-tika.c 
b/src/plugins/fts/fts-parser-tika.c
index 
a4b8b5c3034f57e22e77caa759c090da6b62f8ba..b8b57a350b9a710d101ac7ccbcc14560d415d905
 100644
--- a/src/plugins/fts/fts-parser-tika.c
+++ b/src/plugins/fts/fts-parser-tika.c
@@ -77,7 +77,7 @@ tika_get_http_client_url(struct mail_user *user, 
struct http_url **http_url_r)
http_set.request_timeout_msecs = 60*1000;
http_set.ssl = _set;
http_set.debug = user->mail_debug;
-   tika_http_client = http_client_init(_set);
+   tika_http_client = http_client_init_private(_set);
}
*http_url_r = tuser->http_url;
return 0;


Greetings

I'm also experiencing these issues while running Dovecot
2.3.11.3 with Solr 8.6.3 on FreeBSD 11.4. As mentioned in
a previous mail, the above patch is already applied to
Dovecot's FreeBSD Port, confirmed by the patches being
present in the portstree
(https://svnweb.freebsd.org/ports/branches/2020Q3/mail/dovecot/files/).

In a FreeBSD VM with the official image

(https://download.freebsd.org/ftp/releases/VM-IMAGES/12.1-RELEASE/amd64/Latest/)
I compiled dovecot from git and was able to reproduce the
error with the patch mentioned above applied and also
without any patches at all. From these results i
conclude, that neither the patches applied in FreeBSDs
portstree or the patch above have any influence.

I also managed to reproduce the same results on a Debian
10 machine (also with and without the patch):

doveadm(some.u...@example.com): Panic: file http-client-request.c: line 
1232 (http_client_request_send_more): assertion failed: (req->payload_input != 
NULL)
doveadm(some.u...@example.com): Error: Raw backtrace: 
/usr/local/lib/dovecot/libdovecot.so  
.0(backtrace_append+0x42) [0x7f093f7fc3c2]
-> /usr/local/lib/dovecot/libdovecot.so  
.0(backtrace_get+0x1e) [0x7f093f7fc4ce] -> 
/usr/local/lib/dovecot/libdovecot.so  .0(+0xea341) [0x7f093f807341]
-> /usr/local/lib/dovecot/libdovecot.so  .0(+0xea381) 
[0x7f093f807381] -> /usr/local/lib/dovecot/libdovecot.so  
.0(i_fatal+0) [0x7f093f75c074]
-> /usr/local/lib/dovecot/libdovecot.so  
.0(http_client_request_send_more+0x378) [0x7f093f7a47a8]
-> /usr/local/lib/dovecot/libdovecot.so  
.0(http_client_connection_output+0xe4) [0x7f093f7a90f4]
-> /usr/local/lib/dovecot/libssl_iostream_openssl.so  
(+0x8bff) [0x7f093ec71bff]
-> /usr/local/lib/dovecot/libdovecot.so  
.0(+0x1148b0) [0x7f093f8318b0]
-> /usr/local/lib/dovecot/libdovecot.so  
.0(io_loop_call_io+0x69) 

Re: Indexer error after upgrade to 2.3.11.3 [trial patch]

[John Fawcett]

On 31/10/2020 15:15, Scott Q. wrote:
> But do you know how this bug was introduced ?
>
> I checked the history of that file and don't see anything introduced
> recently that might cause
> this: 
> https://github.com/dovecot/core/commits/master/src/lib-http/http-client-request.c
> 
>
> The only one that might be somewhat related is this one from Apr 27 by
> Stephan Bosch
>
> https://github.com/dovecot/core/commit/799b52accf71e86756dde738d22c1c6a500a7e29#diff-1c02b3481573ffd33c9abccd3f5a6752a5cd81ca83389f4380657f7309c06366
> 
>
I don't know exactly. I'm guess it's not the above commit, since the
problem is happening only when payload_finished is TRUE on arriving at
the start of the http_client_request_send_more function and those lines
set it to FALSE. Though I can't exclude it that it might be an indirect
effect of this.

There are two places where http_client_request_send_more is called but
the problematic call is the one that occurs in http-client-connection.c
within the http_client_connection_continue_request function.

I wasn't able to identify a specific commit that introduced this
problem. The http-client-connection.c file and the whole library has
undergone numerous non trivial commits between 2.3.10 and 2.3.11.3 so
the answer is not easy to identify by code inspectiion. With the various
changes, the code now gets to http_client_request_send_more with
payload_finished true but with payload_input NULL whereas in 2.3.10
payload_input was not NULL. (It's a logical deduction from the fact we
didn't see the assert failures before and that part of the code has not
changed).

John

Re: Indexer error after upgrade to 2.3.11.3 [trial patch]

[Scott Q.]

But do you know how this bug was introduced ?

I checked the history of that file and don't see anything introduced
recently that might cause
this: 
https://github.com/dovecot/core/commits/master/src/lib-http/http-client-request.c


The only one that might be somewhat related is this one from Apr 27 by
Stephan Bosch


https://github.com/dovecot/core/commit/799b52accf71e86756dde738d22c1c6a500a7e29#diff-1c02b3481573ffd33c9abccd3f5a6752a5cd81ca83389f4380657f7309c06366




On Saturday, 31/10/2020 at 08:40 John Fawcett wrote:


 On 31/10/2020 00:18, Scott Q. wrote:



 I have implemented this change and the core dumps went away. 

It appears so far that it's indeed the right fix.


Has this change been introduced in 2.3.11.3 ? 


Thanks!


 

I've also been running it for 4 days without any further panics /
segfaults and no noticed adverse effects. I'd recommend the dovecot
team to consider including the patch in the next update.



John



diff -ur dovecot-2.3.11.3-orig/src/lib-http/http-client-request.c
dovecot-2.3.11.3/src/lib-http/http-client-request.c
--- dovecot-2.3.11.3-orig/src/lib-http/http-client-request.c   
2020-08-12 14:20:41.0 +0200
+++ dovecot-2.3.11.3/src/lib-http/http-client-request.c 2020-10-27
13:06:09.352973130 +0100
@@ -1229,12 +1229,12 @@
    const char *error;
    uoff_t offset;

-   i_assert(req->payload_input != NULL);
-   i_assert(req->payload_output != NULL);
-
    if (req->payload_finished)
    return
http_client_request_finish_payload_out(req);

+   i_assert(req->payload_input != NULL);
+   i_assert(req->payload_output != NULL);
+
    io_remove(>io_req_payload);

    /* chunked ostream needs to write to the parent
stream's buffer */

Re: Indexer error after upgrade to 2.3.11.3 [trial patch]

[John Fawcett]

On 31/10/2020 00:18, Scott Q. wrote:
> I have implemented this change and the core dumps went away.
>
> It appears so far that it's indeed the right fix.
>
> Has this change been introduced in 2.3.11.3 ? 
>
> Thanks!

I've also been running it for 4 days without any further panics /
segfaults and no noticed adverse effects. I'd recommend the dovecot team
to consider including the patch in the next update.

John

diff -ur dovecot-2.3.11.3-orig/src/lib-http/http-client-request.c
dovecot-2.3.11.3/src/lib-http/http-client-request.c
--- dovecot-2.3.11.3-orig/src/lib-http/http-client-request.c   
2020-08-12 14:20:41.0 +0200
+++ dovecot-2.3.11.3/src/lib-http/http-client-request.c 2020-10-27
13:06:09.352973130 +0100
@@ -1229,12 +1229,12 @@
    const char *error;
    uoff_t offset;

-   i_assert(req->payload_input != NULL);
-   i_assert(req->payload_output != NULL);
-
    if (req->payload_finished)
    return http_client_request_finish_payload_out(req);

+   i_assert(req->payload_input != NULL);
+   i_assert(req->payload_output != NULL);
+
    io_remove(>io_req_payload);

    /* chunked ostream needs to write to the parent stream's buffer */

Re: Indexer error after upgrade to 2.3.11.3 [trial patch]

[Scott Q.]

I have implemented this change and the core dumps went away.

It appears so far that it's indeed the right fix.


Has this change been introduced in 2.3.11.3 ? 


Thanks!

On Tuesday, 27/10/2020 at 08:20 John Fawcett wrote:


 On 22/10/2020 10:23, John Fawcett wrote:



 On 21/10/2020 19:00, John Fawcett wrote:



 On 21/10/2020 16:44, Patrik Peng wrote:



 On 16.10.20 18:34, Patrik Peng wrote:



 On 16.10.20 18:00, Scott Q. wrote:


 This reminds me, the way I was able to reproduce this consistently
was by having large headers ( 100+ lines ). 


On Friday, 16/10/2020 at 11:49 Patrik Peng wrote:


 

On 19.08.20 17:37, Josef 'Jeff' Sipek wrote:




 

On Wed, Aug 19, 2020 at 17:03:57 +0200, Alessio Cecchi wrote:  

 Hi, after the upgrade to Dovecot 2.3.11.3, from 2.3.10.1, I see
frequently these errors from different users:  

 It looks like this has been around for a while and you just got
unlucky and started seeing this now. Here's a quick & dirty patch that
should fix this. If you can try it, let us know how it went. Jeff.
diff --git a/src/plugins/fts-solr/solr-connection.c
b/src/plugins/fts-solr/solr-connection.c index
ae720b5e2870a852c1b6c440939e3c7c0fa72b5c..9d364f93e2cd1b716b9ab61bd39656a6c5b1ea04
100644 --- a/src/plugins/fts-solr/solr-connection.c +++
b/src/plugins/fts-solr/solr-connection.c @@ -103,7 +103,7 @@ int
solr_connection_init(const struct fts_solr_settings *solr_set,
http_set.ssl = ssl_client_set; http_set.debug = solr_set->debug;
http_set.rawlog_dir = solr_set->rawlog_dir; -solr_http_client =
http_client_init(_set); +solr_http_client =
http_client_init_private(_set); } *conn_r = conn; diff --git
a/src/plugins/fts/fts-parser-tika.c
b/src/plugins/fts/fts-parser-tika.c index
a4b8b5c3034f57e22e77caa759c090da6b62f8ba..b8b57a350b9a710d101ac7ccbcc14560d415d905
100644 --- a/src/plugins/fts/fts-parser-tika.c +++
b/src/plugins/fts/fts-parser-tika.c @@ -77,7 +77,7 @@
tika_get_http_client_url(struct mail_user *user, struct http_url
**http_url_r) http_set.request_timeout_msecs = 60*1000; http_set.ssl =
_set; http_set.debug = user->mail_debug; -tika_http_client =
http_client_init(_set); +tika_http_client =
http_client_init_private(_set); } *http_url_r = tuser->http_url;
return 0;  

  Greetings



I'm also experiencing these issues while running Dovecot 2.3.11.3 with
Solr 8.6.3 on FreeBSD 11.4. As mentioned in a previous mail, the above
patch is already applied to Dovecot's FreeBSD Port, confirmed by the
patches being present in the portstree
(https://svnweb.freebsd.org/ports/branches/2020Q3/mail/dovecot/files/).



In a FreeBSD VM with the official image
(https://download.freebsd.org/ftp/releases/VM-IMAGES/12.1-RELEASE/amd64/Latest/)
I compiled dovecot from git and was able to reproduce the error with
the patch mentioned above applied and also without any patches at all.
>From these results i conclude, that neither the patches applied in
FreeBSDs portstree or the patch above have any influence.



I also managed to reproduce the same results on a Debian 10 machine
(also with and without the patch):



doveadm(some.u...@example.com [1]): Panic: file http-client-request.c:
line 1232 (http_client_request_send_more): assertion failed:
(req->payload_input != NULL) doveadm(some.u...@example.com [1]):
Error: Raw backtrace: /usr/local/lib/dovecot/libdovecot.so
[2].0(backtrace_append+0x42) [0x7f093f7fc3c2] ->
/usr/local/lib/dovecot/libdovecot.so [2].0(backtrace_get+0x1e)
[0x7f093f7fc4ce] -> /usr/local/lib/dovecot/libdovecot.so
[2].0(+0xea341) [0x7f093f807341] ->
/usr/local/lib/dovecot/libdovecot.so [2].0(+0xea381) [0x7f093f807381]
-> /usr/local/lib/dovecot/libdovecot.so [2].0(i_fatal+0)
[0x7f093f75c074] -> /usr/local/lib/dovecot/libdovecot.so
[2].0(http_client_request_send_more+0x378) [0x7f093f7a47a8] ->
/usr/local/lib/dovecot/libdovecot.so
[2].0(http_client_connection_output+0xe4) [0x7f093f7a90f4] ->
/usr/local/lib/dovecot/libssl_iostream_openssl.so [3](+0x8bff)
[0x7f093ec71bff] -> /usr/local/lib/dovecot/libdovecot.so
[2].0(+0x1148b0) [0x7f093f8318b0] ->
/usr/local/lib/dovecot/libdovecot.so [2].0(io_loop_call_io+0x69)
[0x7f093f820259] -> /usr/local/lib/dovecot/libdovecot.so
[2].0(io_loop_handler_run_internal+0x11b) [0x7f093f821b6b] ->
/usr/local/lib/dovecot/libdovecot.so [2].0(io_loop_handler_run+0x59)
[0x7f093f820369] -> /usr/local/lib/dovecot/libdovecot.so
[2].0(io_loop_run+0x38) [0x7f093f820598] ->
/usr/local/lib/dovecot/libdovecot.so [2].0(+0x86d1e) [0x7f093f7a3d1e]
-> /usr/local/lib/dovecot/libdovecot.so
[2].0(http_client_request_finish_payload+0x2e) [0x7f093f7a407e] ->
/usr/local/lib/dovecot/lib21_fts_solr_plugin.so
[4](solr_connection_post_end+0x32) [0x7f093b8492c2] ->
/usr/local/lib/dovecot/lib21_fts_solr_plugin.so [4](+0x3a45)
[0x7f093b844a45] -> /usr/local/lib/dovecot/lib20_fts_plugin.so
[4](+0x94cc) [0x7f093e1104cc] ->
/usr/local/lib/dovecot/lib20_fts_plugin.so
[4](fts_backend_update_deinit+0x23) [0x7f093e110503] ->
/usr/local/lib/dovecot/lib20_fts_plugin.so [4](+0x10a9b)
[0x7f093e117a9b] -> 

Re: Indexer error after upgrade to 2.3.11.3 [trial patch]

[John Fawcett]

On 22/10/2020 10:23, John Fawcett wrote:
> On 21/10/2020 19:00, John Fawcett wrote:
>> On 21/10/2020 16:44, Patrik Peng wrote:
>>> On 16.10.20 18:34, Patrik Peng wrote:
 On 16.10.20 18:00, Scott Q. wrote:
> This reminds me, the way I was able to reproduce this consistently
> was by having large headers ( 100+ lines ).
>
>
> On Friday, 16/10/2020 at 11:49 Patrik Peng wrote:
>
> On 19.08.20 17:37, Josef 'Jeff' Sipek wrote:
>
>> On Wed, Aug 19, 2020 at 17:03:57 +0200, Alessio Cecchi wrote:
>>> Hi,
>>> after the upgrade to Dovecot 2.3.11.3, from 2.3.10.1, I see 
>>> frequently 
>>> these errors from different users:
>> It looks like this has been around for a while and you just got 
>> unlucky and
>> started seeing this now.  Here's a quick & dirty patch that should 
>> fix this.
>> If you can try it, let us know how it went.
>> Jeff.
>>
>> diff --git a/src/plugins/fts-solr/solr-connection.c 
>> b/src/plugins/fts-solr/solr-connection.c
>> index 
>> ae720b5e2870a852c1b6c440939e3c7c0fa72b5c..9d364f93e2cd1b716b9ab61bd39656a6c5b1ea04
>>  100644
>> --- a/src/plugins/fts-solr/solr-connection.c
>> +++ b/src/plugins/fts-solr/solr-connection.c
>> @@ -103,7 +103,7 @@ int solr_connection_init(const struct 
>> fts_solr_settings *solr_set,
>>  http_set.ssl = ssl_client_set;
>>  http_set.debug = solr_set->debug;
>>  http_set.rawlog_dir = solr_set->rawlog_dir;
>> -solr_http_client = http_client_init(_set);
>> +solr_http_client = http_client_init_private(_set);
>>  }
>>  *conn_r = conn;
>> diff --git a/src/plugins/fts/fts-parser-tika.c 
>> b/src/plugins/fts/fts-parser-tika.c
>> index 
>> a4b8b5c3034f57e22e77caa759c090da6b62f8ba..b8b57a350b9a710d101ac7ccbcc14560d415d905
>>  100644
>> --- a/src/plugins/fts/fts-parser-tika.c
>> +++ b/src/plugins/fts/fts-parser-tika.c
>> @@ -77,7 +77,7 @@ tika_get_http_client_url(struct mail_user *user, 
>> struct http_url **http_url_r)
>>  http_set.request_timeout_msecs = 60*1000;
>>  http_set.ssl = _set;
>>  http_set.debug = user->mail_debug;
>> -tika_http_client = http_client_init(_set);
>> +tika_http_client = http_client_init_private(_set);
>>  }
>>  *http_url_r = tuser->http_url;
>>  return 0;
>
> Greetings
>
> I'm also experiencing these issues while running Dovecot
> 2.3.11.3 with Solr 8.6.3 on FreeBSD 11.4. As mentioned in a
> previous mail, the above patch is already applied to Dovecot's
> FreeBSD Port, confirmed by the patches being present in the
> portstree
> 
> (https://svnweb.freebsd.org/ports/branches/2020Q3/mail/dovecot/files/).
>
> In a FreeBSD VM with the official image
> 
> (https://download.freebsd.org/ftp/releases/VM-IMAGES/12.1-RELEASE/amd64/Latest/)
> I compiled dovecot from git and was able to reproduce the
> error with the patch mentioned above applied and also without
> any patches at all. From these results i conclude, that
> neither the patches applied in FreeBSDs portstree or the patch
> above have any influence.
>
> I also managed to reproduce the same results on a Debian 10
> machine (also with and without the patch):
>
> doveadm(some.u...@example.com): Panic: file http-client-request.c: 
> line 1232 (http_client_request_send_more): assertion failed: 
> (req->payload_input != NULL)
> doveadm(some.u...@example.com): Error: Raw backtrace: 
> /usr/local/lib/dovecot/libdovecot.so 
> .0(backtrace_append+0x42) [0x7f093f7fc3c2]
> -> /usr/local/lib/dovecot/libdovecot.so 
> .0(backtrace_get+0x1e) [0x7f093f7fc4ce] -> 
> /usr/local/lib/dovecot/libdovecot.so .0(+0xea341) 
> [0x7f093f807341]
> -> /usr/local/lib/dovecot/libdovecot.so 
> .0(+0xea381) [0x7f093f807381] -> 
> /usr/local/lib/dovecot/libdovecot.so .0(i_fatal+0) 
> [0x7f093f75c074]
> -> /usr/local/lib/dovecot/libdovecot.so 
> .0(http_client_request_send_more+0x378) 
> [0x7f093f7a47a8]
> -> /usr/local/lib/dovecot/libdovecot.so 
> .0(http_client_connection_output+0xe4) 
> [0x7f093f7a90f4]
> -> /usr/local/lib/dovecot/libssl_iostream_openssl.so 
> (+0x8bff) [0x7f093ec71bff]
> -> /usr/local/lib/dovecot/libdovecot.so 
> .0(+0x1148b0) [0x7f093f8318b0]
> ->