Re: [PATCH] Re: Solr connection timeout hardwired to 60s

[Shawn Heisey via dovecot]

On 4/14/2019 7:59 AM, John Fawcett via dovecot wrote:

From dovecot point of view I can see the following as potentially useful
features:

1) a configurable batch size would enable to tune the number of emails
per request and help stay under the 60 seconds hard coded http request
timeout. A configurable http timeout would be less useful, since this
will potentially run into other timeouts on solr side.


Even if several thousand emails are sent per batch, unless they're 
incredibly large, I can't imagine indexing them taking more than a few 
seconds.  Does dovecot send attachments to Solr as well as the email 
itself?  Hopefully it doesn't.  If it does, then you would want a 
smaller batch size.


But if the heap size for Solr is not big enough, that can cause major 
delays no matter what requests are being sent, because Java will be 
spending most of its time doing garbage collection.


I'm also assuming that the Solr server is on the same LAN as dovecot and 
that transferring the update data does not take a long time.


Thanks,
Shawn

Re: Solr connection timeout hardwired to 60s

[John Fawcett via dovecot]

On 14/04/2019 17:55, John Fawcett via dovecot wrote:
> The solr server is a small test virtual machine with 0.2 (shared) vCPU
> and 0.6MB of memory and non SSD storage. It can index around 2000 emails
> per minute when there is no other activity. Average email size is about
> 45Kb. I'm not indexing attachments.
>
> John
>
I more than double the performance by using a 0.5 shared vCPU and SSD
storage. Approx 4500 emails per minute.

John

Re: Solr connection timeout hardwired to 60s

[John Fawcett via dovecot]

On 14/04/2019 17:16, Peter Mogensen via dovecot wrote:
> sorry... I got distracted half way and forgot to put a meaningfull
> subject so the archive could figure out the thread. - resending.
>
> On 4/14/19 4:04 PM, dovecot-requ...@dovecot.org wrote:
>
>>> Solr ships with autoCommit set to 15 seconds and openSearcher set to
>>> false on the autoCommit.? The autoSoftCommit setting is not enabled by
>>> default, but depending on how the index was created, Solr might try to
>>> set autoSoftCommit to 3 seconds ... which is WAY too short.
> I just run with the default. 15s autoCommit and no autoSoftCommit
>
>>> This thread says that dovecot is sending explicit commits.? 
> I see explicit /update req. with softCommit and waitSearcer=true in a
> tcpdump.
>
>>> One thing
>>> that might be happening to exceed 60 seconds is an extremely long
>>> commit, which is usually caused by excessive cache autowarming, but
>>> might be related to insufficient memory.? The max heap setting on an
>>> out-of-the-box Solr install (5.0 and later) is 512MB.? That's VERY
>>> small, and it doesn't take much index data before a much larger heap
>>> is required.
> I run with
>
> SOLR_JAVA_MEM="-Xmx8g -Xms2g"
>
>> I looked into the code (version 2.3.5.1):
> This is 2.2.35. I haven't checked the source difference to 2.3.x I must
> admit.
>
>> I immagine that one of the reasons dovecot sends softCommits is because
>> without autoindex active and even if mailboxes are periodically indexed
>> from cron, the last emails received with be indexed at the moment of the
>> search.? 
> I expect that dovecot has to because of it's default behavior by only
> bringing the index up-to-date just before search. So it has towait for
> the index result to be available if there's been any new mails indexed.
>
>> 1) a configurable batch size would enable to tune the number of emails
>> per request and help stay under the 60 seconds hard coded http request
>> timeout. A configurable http timeout would be less useful, since this
>> will potentially run into other timeouts on solr side.
> Being able to configure it is great.
> But I don't think it solves much. I recompiled with 100 as batch size
> and it still ended in timeouts.
> Then I recompiled with 10min timeout and now I see all the batches
> completing and their processesing time is mostly between 1 and 2 minutes
> (so all would have failed).
>
> To me it looks like Solr just takes too long time to index. This is no
> small machine. It's a 20 core Intel(R) Xeon(R) Silver 4114 CPU @ 2.20GHz
> and for this test it's not doing anything else, so I'm a bit surprised
> that even with only a few users this takes so long time.
>
> /Peter
>
>
Peter

I suppose you could go with a batch size of 50. If it's linear, you
could still keep under the default 60 seconds http request time :-)

I'm now testing with solr settings autoCommit 15 seconds, autoSoftCommit
60 seconds and sending no softCommits from dovecot and 500 batch size.

I've set up

    /usr/local/bin/doveadm index -A "*"

in crontab every 5 minutes so indexes will stay mostly up to date to
minimize amount of mail not already visible in the index when searches
are done.

The solr server is a small test virtual machine with 0.2 (shared) vCPU
and 0.6MB of memory and non SSD storage. It can index around 2000 emails
per minute when there is no other activity. Average email size is about
45Kb. I'm not indexing attachments.

John

Re: [PATCH] Re: Solr connection timeout hardwired to 60s

[John Fawcett via dovecot]

On 14/04/2019 16:04, Aki Tuomi via dovecot wrote:
>
>> On 14 April 2019 16:59 John Fawcett via dovecot < dovecot@dovecot.org
>> > wrote:
>>
>>
>> On 13/04/2019 17:16, Shawn Heisey via dovecot wrote:
>>> On 4/13/2019 4:29 AM, John Fawcett via dovecot wrote:
 If this value was made configurable people could set it to what they
 want. However the underlying problem is likely on solr configuration.
>>> The Jetty that is included in Solr has its idle timeout set to 50
>>> seconds.  But in practice, I have not seen this timeout trigger ...
>>> and if the OP is seeing a 60 second timeout, then the 50 second idle
>>> timeout in Jetty must not be occurring.
>>> There may be a socket timeout configured on inter-server requests --
>>> distributed queries or the load balancing that SolrCloud does.  I can
>>> never remember whether this is the case by default.  I think it is.
>> >> If there is an issue on initial indexing, where you are not really
>> >> concerned about qucik visibility but just getting things into the
>> index
>> >> efficiently, a better approach would be for dovecot plugin not to
>> send
>> >> any commit or softCommit (or waitSearcher either) and that should
>> speed
>> >> things up. You'd need to configure solr with a long autoSoftCommit
>> >> maxTime and a reasonable autoCommit maxTime, which you could then
>> >> reconfigure when the load was done.
>> >
>>> Solr ships with autoCommit set to 15 seconds and openSearcher set to
>>> false on the autoCommit.  The autoSoftCommit setting is not enabled by
>>> default, but depending on how the index was created, Solr might try to
>>> set autoSoftCommit to 3 seconds ... which is WAY too short.
>>> I will usually increase the autoCommit time to 60 seconds, just to
>>> reduce the amount of work that Solr is doing.  The autoSoftCommit
>>> time, if it is used, should be set to a reasonably long value ...
>>> values between two and five minutes would be good.  Attempting to use
>>> a very short autoSoftCommit time will usually lead to problems.
>>> This thread says that dovecot is sending explicit commits.  One thing
>>> that might be happening to exceed 60 seconds is an extremely long
>>> commit, which is usually caused by excessive cache autowarming, but
>>> might be related to insufficient memory.  The max heap setting on an
>>> out-of-the-box Solr install (5.0 and later) is 512MB.  That's VERY
>>> small, and it doesn't take much index data before a much larger heap
>>> is required.
>>> Thanks,
>>> Shawn
>> I looked into the code (version 2.3.5.1): the fts-solr plugin is not
>> sending softCommit every 1000 emails. Emails from a single folder are
>> batched in up to a maximum of 1000 emails per request, but the
>> softCommit gets sent once per mailbox folder at the end of all the
>> requests for that folder.
>>
>> I immagine that one of the reasons dovecot sends softCommits is because
>> without autoindex active and even if mailboxes are periodically indexed
>> from cron, the last emails received with be indexed at the moment of the
>> search.  So while sending softCommit has the advantage of including
>> recent mails in searches, it means that softCommits are being done upon
>> user request. Frequency depends on user activity.
>>
>> Going back to the original problem: seems the first advice to Peter is
>> to look into solr configuration as others have said.
>>
>> From dovecot point of view I can see the following as potentially useful
>> features:
>>
>> 1) a configurable batch size would enable to tune the number of emails
>> per request and help stay under the 60 seconds hard coded http request
>> timeout. A configurable http timeout would be less useful, since this
>> will potentially run into other timeouts on solr side.
>>
>> 2) abilty to turn off softCommits so as to have a more predictable
>> softCommit workload. In that case autoSoftCommit should be configured in
>> solr. In order to minimize risk of recent emails not appearing in search
>> results, periodic indexing could be set up by cron.
>>
>> I've attached a patch, any comments are welcome (especially about
>> getting settings from the backend context).
>>
>> Example config
>>
>> plugin {
>>   fts = solr
>>   fts_solr =
>> url= https://user:passw...@solr.example.com:443/solr/dovecot/
>> batch_size=500 no_soft_commit
>> }
>>
>> John
>
> Can you please open a pull request to https://github.com/dovecot/core ?
> ---
> Aki Tuomi

Done, thanks for considering it.

John

Re: Solr connection timeout hardwired to 60s

[Peter Mogensen via dovecot]

sorry... I got distracted half way and forgot to put a meaningfull
subject so the archive could figure out the thread. - resending.

On 4/14/19 4:04 PM, dovecot-requ...@dovecot.org wrote:

>> Solr ships with autoCommit set to 15 seconds and openSearcher set to
>> false on the autoCommit.? The autoSoftCommit setting is not enabled by
>> default, but depending on how the index was created, Solr might try to
>> set autoSoftCommit to 3 seconds ... which is WAY too short.

I just run with the default. 15s autoCommit and no autoSoftCommit

>> This thread says that dovecot is sending explicit commits.? 

I see explicit /update req. with softCommit and waitSearcer=true in a
tcpdump.

>> One thing
>> that might be happening to exceed 60 seconds is an extremely long
>> commit, which is usually caused by excessive cache autowarming, but
>> might be related to insufficient memory.? The max heap setting on an
>> out-of-the-box Solr install (5.0 and later) is 512MB.? That's VERY
>> small, and it doesn't take much index data before a much larger heap
>> is required.

I run with

SOLR_JAVA_MEM="-Xmx8g -Xms2g"

> I looked into the code (version 2.3.5.1):

This is 2.2.35. I haven't checked the source difference to 2.3.x I must
admit.

> I immagine that one of the reasons dovecot sends softCommits is because
> without autoindex active and even if mailboxes are periodically indexed
> from cron, the last emails received with be indexed at the moment of the
> search.? 

I expect that dovecot has to because of it's default behavior by only
bringing the index up-to-date just before search. So it has towait for
the index result to be available if there's been any new mails indexed.

> 1) a configurable batch size would enable to tune the number of emails
> per request and help stay under the 60 seconds hard coded http request
> timeout. A configurable http timeout would be less useful, since this
> will potentially run into other timeouts on solr side.

Being able to configure it is great.
But I don't think it solves much. I recompiled with 100 as batch size
and it still ended in timeouts.
Then I recompiled with 10min timeout and now I see all the batches
completing and their processesing time is mostly between 1 and 2 minutes
(so all would have failed).

To me it looks like Solr just takes too long time to index. This is no
small machine. It's a 20 core Intel(R) Xeon(R) Silver 4114 CPU @ 2.20GHz
and for this test it's not doing anything else, so I'm a bit surprised
that even with only a few users this takes so long time.

/Peter

Re: [PATCH] Re: Solr connection timeout hardwired to 60s

[Aki Tuomi via dovecot]

 
 
  
   
  
  
   
On 14 April 2019 16:59 John Fawcett via dovecot <
dovecot@dovecot.org> wrote:
   
   

   
   

   
   
On 13/04/2019 17:16, Shawn Heisey via dovecot wrote:
   
   

 On 4/13/2019 4:29 AM, John Fawcett via dovecot wrote:


 
  If this value was made configurable people could set it to what they
 
 
  want. However the underlying problem is likely on solr configuration.
 

   
   

 The Jetty that is included in Solr has its idle timeout set to 50


 seconds.  But in practice, I have not seen this timeout trigger ...


 and if the OP is seeing a 60 second timeout, then the 50 second idle


 timeout in Jetty must not be occurring.

   
   

 There may be a socket timeout configured on inter-server requests --


 distributed queries or the load balancing that SolrCloud does.  I can


 never remember whether this is the case by default.  I think it is.

   
   
>> If there is an issue on initial indexing, where you are not really
   
   
>> concerned about qucik visibility but just getting things into the index
   
   
>> efficiently, a better approach would be for dovecot plugin not to send
   
   
>> any commit or softCommit (or waitSearcher either) and that should speed
   
   
>> things up. You'd need to configure solr with a long autoSoftCommit
   
   
>> maxTime and a reasonable autoCommit maxTime, which you could then
   
   
>> reconfigure when the load was done.
   
   
>
   
   

 Solr ships with autoCommit set to 15 seconds and openSearcher set to


 false on the autoCommit.  The autoSoftCommit setting is not enabled by


 default, but depending on how the index was created, Solr might try to


 set autoSoftCommit to 3 seconds ... which is WAY too short.

   
   

 I will usually increase the autoCommit time to 60 seconds, just to


 reduce the amount of work that Solr is doing.  The autoSoftCommit


 time, if it is used, should be set to a reasonably long value ...


 values between two and five minutes would be good.  Attempting to use


 a very short autoSoftCommit time will usually lead to problems.

   
   

 This thread says that dovecot is sending explicit commits.  One thing


 that might be happening to exceed 60 seconds is an extremely long


 commit, which is usually caused by excessive cache autowarming, but


 might be related to insufficient memory.  The max heap setting on an


 out-of-the-box Solr install (5.0 and later) is 512MB.  That's VERY


 small, and it doesn't take much index data before a much larger heap


 is required.

   
   

 Thanks,


 Shawn

   
   
I looked into the code (version 2.3.5.1): the fts-solr plugin is not
   
   
sending softCommit every 1000 emails. Emails from a single folder are
   
   
batched in up to a maximum of 1000 emails per request, but the
   
   
softCommit gets sent once per mailbox folder at the end of all the
   
   
requests for that folder.
   
   

   
   
I immagine that one of the reasons dovecot sends softCommits is because
   
   
without autoindex active and even if mailboxes are periodically indexed
   
   
from cron, the last emails received with be indexed at the moment of the
   
   
search.  So while sending softCommit has the advantage of including
   
   
recent mails in searches, it means that softCommits are being done upon
   
   
user request. Frequency depends on user activity.
   
   

   
   
Going back to the original problem: seems the first advice to Peter is
   
   
to look into solr configuration as others have said.
   
   

   
   
From dovecot point of view I can see the following as potentially useful
   
   
features:
   
   

   
   
1) a configurable batch size would enable to tune the number of emails
   
   
per request and help stay under the 60 seconds hard coded http request
   
   
timeout. A configurable http timeout would be less useful, since this
   
   
will potentially run into other timeouts on solr side.
   
   

   
   
2) abilty to turn off softCommits so as to have a more predictable
   
   
softCommit workload. In that case autoSoftCommit should be configured in
   
   
solr. In order to minimize risk of recent emails not appearing in search
   
   
results, periodic indexing could be set up by cron.
   
   

   
   
I've attached a patch, any comments are welcome (especially about
   
   
getting settings from the backend context).
   
   

   
   
Example config
   
   

   
   
plugin {
   
   
  fts = solr
   
   
  fts_solr =
 

[PATCH] Re: Solr connection timeout hardwired to 60s

[John Fawcett via dovecot]

On 13/04/2019 17:16, Shawn Heisey via dovecot wrote:
> On 4/13/2019 4:29 AM, John Fawcett via dovecot wrote:
>> If this value was made configurable people could set it to what they
>> want. However the underlying problem is likely on solr configuration.
>
> The Jetty that is included in Solr has its idle timeout set to 50
> seconds.  But in practice, I have not seen this timeout trigger ...
> and if the OP is seeing a 60 second timeout, then the 50 second idle
> timeout in Jetty must not be occurring.
>
> There may be a socket timeout configured on inter-server requests --
> distributed queries or the load balancing that SolrCloud does.  I can
> never remember whether this is the case by default.  I think it is.
>
>> If there is an issue on initial indexing, where you are not really
>> concerned about qucik visibility but just getting things into the index
>> efficiently, a better approach would be for dovecot plugin not to send
>> any commit or softCommit (or waitSearcher either) and that should speed
>> things up. You'd need to configure solr with a long autoSoftCommit
>> maxTime and a reasonable autoCommit maxTime, which you could then
>> reconfigure when the load was done.
>
> Solr ships with autoCommit set to 15 seconds and openSearcher set to
> false on the autoCommit.  The autoSoftCommit setting is not enabled by
> default, but depending on how the index was created, Solr might try to
> set autoSoftCommit to 3 seconds ... which is WAY too short.
>
> I will usually increase the autoCommit time to 60 seconds, just to
> reduce the amount of work that Solr is doing.  The autoSoftCommit
> time, if it is used, should be set to a reasonably long value ...
> values between two and five minutes would be good.  Attempting to use
> a very short autoSoftCommit time will usually lead to problems.
>
> This thread says that dovecot is sending explicit commits.  One thing
> that might be happening to exceed 60 seconds is an extremely long
> commit, which is usually caused by excessive cache autowarming, but
> might be related to insufficient memory.  The max heap setting on an
> out-of-the-box Solr install (5.0 and later) is 512MB.  That's VERY
> small, and it doesn't take much index data before a much larger heap
> is required.
>
> Thanks,
> Shawn

I looked into the code (version 2.3.5.1): the fts-solr plugin is not
sending softCommit every 1000 emails. Emails from a single folder are
batched in up to a maximum of 1000 emails per request, but the
softCommit gets sent once per mailbox folder at the end of all the
requests for that folder.

I immagine that one of the reasons dovecot sends softCommits is because
without autoindex active and even if mailboxes are periodically indexed
from cron, the last emails received with be indexed at the moment of the
search.  So while sending softCommit has the advantage of including
recent mails in searches, it means that softCommits are being done upon
user request. Frequency depends on user activity.

Going back to the original problem: seems the first advice to Peter is
to look into solr configuration as others have said.

>From dovecot point of view I can see the following as potentially useful
features:

1) a configurable batch size would enable to tune the number of emails
per request and help stay under the 60 seconds hard coded http request
timeout. A configurable http timeout would be less useful, since this
will potentially run into other timeouts on solr side.

2) abilty to turn off softCommits so as to have a more predictable
softCommit workload. In that case autoSoftCommit should be configured in
solr. In order to minimize risk of recent emails not appearing in search
results, periodic indexing could be set up by cron.

I've attached a patch, any comments are welcome (especially about
getting settings from the backend context).

Example config

plugin {
  fts = solr
  fts_solr =
url=https://user:passw...@solr.example.com:443/solr/dovecot/
batch_size=500 no_soft_commit
}

John

--- src/plugins/fts-solr/fts-solr-plugin.h.orig 2019-04-14 15:12:07.694289402 
+0200
+++ src/plugins/fts-solr/fts-solr-plugin.h  2019-04-14 14:04:17.213939414 
+0200
@@ -12,8 +12,10 @@
 
 struct fts_solr_settings {
const char *url, *default_ns_prefix;
+   unsigned int batch_size;
bool use_libfts;
bool debug;
+   bool no_soft_commit;
 };
 
 struct fts_solr_user {
--- src/plugins/fts-solr/fts-solr-plugin.c.orig 2019-04-14 11:41:03.591782439 
+0200
+++ src/plugins/fts-solr/fts-solr-plugin.c  2019-04-14 14:37:46.059433864 
+0200
@@ -10,6 +10,8 @@
 #include "fts-solr-plugin.h"
 
 
+#define DEFAULT_SOLR_BATCH_SIZE 1000
+
 const char *fts_solr_plugin_version = DOVECOT_ABI_VERSION;
 struct http_client *solr_http_client = NULL;
 
@@ -37,6 +39,10 @@
} else if (str_begins(*tmp, "default_ns=")) {
set->default_ns_prefix =
p_strdup(user->pool, *tmp + 11);
+   } else if (str_begins(*tmp

Re: Solr connection timeout hardwired to 60s

[Shawn Heisey via dovecot]

On 4/13/2019 4:29 AM, John Fawcett via dovecot wrote:

If this value was made configurable people could set it to what they
want. However the underlying problem is likely on solr configuration.


The Jetty that is included in Solr has its idle timeout set to 50 
seconds.  But in practice, I have not seen this timeout trigger ... and 
if the OP is seeing a 60 second timeout, then the 50 second idle timeout 
in Jetty must not be occurring.


There may be a socket timeout configured on inter-server requests -- 
distributed queries or the load balancing that SolrCloud does.  I can 
never remember whether this is the case by default.  I think it is.



If there is an issue on initial indexing, where you are not really
concerned about qucik visibility but just getting things into the index
efficiently, a better approach would be for dovecot plugin not to send
any commit or softCommit (or waitSearcher either) and that should speed
things up. You'd need to configure solr with a long autoSoftCommit
maxTime and a reasonable autoCommit maxTime, which you could then
reconfigure when the load was done.


Solr ships with autoCommit set to 15 seconds and openSearcher set to 
false on the autoCommit.  The autoSoftCommit setting is not enabled by 
default, but depending on how the index was created, Solr might try to 
set autoSoftCommit to 3 seconds ... which is WAY too short.


I will usually increase the autoCommit time to 60 seconds, just to 
reduce the amount of work that Solr is doing.  The autoSoftCommit time, 
if it is used, should be set to a reasonably long value ... values 
between two and five minutes would be good.  Attempting to use a very 
short autoSoftCommit time will usually lead to problems.


This thread says that dovecot is sending explicit commits.  One thing 
that might be happening to exceed 60 seconds is an extremely long 
commit, which is usually caused by excessive cache autowarming, but 
might be related to insufficient memory.  The max heap setting on an 
out-of-the-box Solr install (5.0 and later) is 512MB.  That's VERY 
small, and it doesn't take much index data before a much larger heap is 
required.


Thanks,
Shawn

Re: Solr connection timeout hardwired to 60s

[John Fawcett via dovecot]

On 12/04/2019 12:09, Peter Mogensen via dovecot wrote:
> Looking further at tcpdumps of the Dovecot->Solr traffic and Solr
> metrics it doesn't seem like there's anything suspicious apart from the
> TCP windows running full and Dovecot backing of ... until it times out
> and close the connection.
>
> From my understanding of how Dovecot operates towards Solr it will flush
> ~1000 documents towards Solr in /update request until it has traversed
> the mailbox (let's say 20.000 mails), doing softCommits after each.
>
> But is it really reasonable for Dovecot to expect that no request will
> take more than 60s to process by Solr?
> It doesn't seem like my Solr can handle that, although it does process
> documents and it does reasonably fast clear pending documents after
> Dovecot closes the connection.
>
> On the surface it looks like Dovecot is too impatient.
>
> /Peter

If this value was made configurable people could set it to what they
want. However the underlying problem is likely on solr configuration.

Is this a problem only on initial indexing or an ongoing problem after
initial indexing?

The parameters that the solr plugin are using are designed to make
documents visible to searches quickly.

If there is an issue on initial indexing, where you are not really
concerned about qucik visibility but just getting things into the index
efficiently, a better approach would be for dovecot plugin not to send
any commit or softCommit (or waitSearcher either) and that should speed
things up. You'd need to configure solr with a long autoSoftCommit
maxTime and a reasonable autoCommit maxTime, which you could then
reconfigure when the load was done.

If you're using dovecot built from source code it should be possible to
test that by some minor modification of the code in  fts-backend-solr.c.

John

Re: Solr connection timeout hardwired to 60s

[Peter Mogensen via dovecot]

Looking further at tcpdumps of the Dovecot->Solr traffic and Solr
metrics it doesn't seem like there's anything suspicious apart from the
TCP windows running full and Dovecot backing of ... until it times out
and close the connection.

>From my understanding of how Dovecot operates towards Solr it will flush
~1000 documents towards Solr in /update request until it has traversed
the mailbox (let's say 20.000 mails), doing softCommits after each.

But is it really reasonable for Dovecot to expect that no request will
take more than 60s to process by Solr?
It doesn't seem like my Solr can handle that, although it does process
documents and it does reasonably fast clear pending documents after
Dovecot closes the connection.

On the surface it looks like Dovecot is too impatient.

/Peter

On 4/10/19 6:25 PM, Peter Mogensen wrote:
> 
> 
> On 4/4/19 6:57 PM, Peter Mogensen wrote:
>>
>>
>> On 4/4/19 6:47 PM, dovecot-requ...@dovecot.org wrote:
>>> For a typical Solr index, 60 seconds is an eternity.  Most people aim
>>> for query times of 100 milliseconds or less, and they often achieve
>>> that goal.
>>
>> I'm pretty sure I get these while indexing, not querying.
>>
>> Apr 04 16:44:50 host dovecot[114690]: indexer-worker(m...@example.com):
>> Error: fts_solr: Indexing failed: Request timed out (Request queued
>> 66.015 secs ago, 1 attempts in 66.005 secs, 63.146 in http ioloop, 0.000
>> in other ioloops, connected 94.903 secs ago)
> 
> Doing a TCP dump on indexing operations which consistently fail, I see
> that there's a lot of softCommits which never get an HTTP answer:
> 
> ==
> POST /solr/dovebody/update HTTP/1.1
> Host: localhost:8983
> Date: Wed, 10 Apr 2019 14:22:29 GMT
> Expect: 100-continue
> Content-Length: 47
> Connection: Keep-Alive
> Content-Type: text/xml
> 
> HTTP/1.1 100 Continue
> 
> 
> 

Re: Solr connection timeout hardwired to 60s

[Peter Mogensen via dovecot]

On 4/4/19 6:57 PM, Peter Mogensen wrote:
> 
> 
> On 4/4/19 6:47 PM, dovecot-requ...@dovecot.org wrote:
>> For a typical Solr index, 60 seconds is an eternity.  Most people aim
>> for query times of 100 milliseconds or less, and they often achieve
>> that goal.
> 
> I'm pretty sure I get these while indexing, not querying.
> 
> Apr 04 16:44:50 host dovecot[114690]: indexer-worker(m...@example.com):
> Error: fts_solr: Indexing failed: Request timed out (Request queued
> 66.015 secs ago, 1 attempts in 66.005 secs, 63.146 in http ioloop, 0.000
> in other ioloops, connected 94.903 secs ago)

Doing a TCP dump on indexing operations which consistently fail, I see
that there's a lot of softCommits which never get an HTTP answer:

==
POST /solr/dovebody/update HTTP/1.1
Host: localhost:8983
Date: Wed, 10 Apr 2019 14:22:29 GMT
Expect: 100-continue
Content-Length: 47
Connection: Keep-Alive
Content-Type: text/xml

HTTP/1.1 100 Continue





... in contrast to the first softCommit on the connection:


POST /solr/dovebody/update HTTP/1.1
Host: localhost:8983
Date: Wed, 10 Apr 2019 14:20:53 GMT
Expect: 100-continue
Content-Length: 47
Connection: Keep-Alive
Content-Type: text/xml

HTTP/1.1 100 Continue

HTTP/1.1 200 OK
Content-Type: application/xml; charset=UTF-8
Content-Length: 156





  0
  37


==

The missing softCommit responses seem to start right after the last
added document:
==

0

HTTP/1.1 200 OK
Content-Type: application/xml; charset=UTF-8
Content-Length: 156





  0
  12


POST /solr/dovebody/update HTTP/1.1
Host: localhost:8983
Date: Wed, 10 Apr 2019 14:22:29 GMT
Expect: 100-continue
Content-Length: 47
Connection: Keep-Alive
Content-Type: text/xml

HTTP/1.1 100 Continue


===

... and then the rest of the TCP dump doesn't get responses to
softCommit POSTs

/Peter

Re: Solr connection timeout hardwired to 60s

[Shawn Heisey via dovecot]

On 4/4/2019 6:42 PM, M. Balridge via dovecot wrote:

What is a general rule of thumb for RAM and SSD disk requirements as a
fraction of indexed document hive size to keep query performance at 200ms or
less? How do people deal with the JAVA GC world-stoppages, other than simply
doubling or tripling every instance?


There's no hard and fast rule for exactly how much memory you need for a 
search engine.  Some installs work well with half the index cached, 
others require more, some require less.


For ideal performance, you should have enough memory over and above your 
program requirements to cache the entire index.  That can be problematic 
with indexes that are hundreds of gigabytes, or even terabytes. 
Achieving the ideal is rarely necessary, though.


With a large enough heap, it is simply impossible to avoid long 
stop-the-world GC.  With proper tuning, those full garbage collections 
can happen far less frequently.  I've got another page about that.


https://wiki.apache.org/solr/ShawnHeisey#GC_Tuning_for_Solr

To handle extremely large indexes with good performance, I would 
recommend many servers running SolrCloud, and a sharded index.  That way 
each individual server will not be required to handle terabytes of data. 
 This can get very expensive very quickly.  You will also need a load 
balancer, to eliminate single points of failure.



I am wondering how well alternatives to Solr work in these situations
(ElasticSearch, Xapian, and any others I may have missed).


Assuming they are configured as similarly as possible, ElasticSearch and 
Solr will have nearly identical requirements, and perform similarly to 
each other.  They are both Lucene-based, and it is Lucene that primarily 
drives the requirements.  I know nothing about any other solutions.


With the extremely large index you have described, memory will be your 
achilles heel no matter what solution you find.


It is not Java that needs the extreme amounts of memory for very large 
indexes.  It is the operating system -- the disk cache.  You might also 
need a fairly large heap, but the on-disk size of the index will have 
less of an impact on heap requirements than the number of documents in 
the index.


Thanks,
Shawn

Re: Solr connection timeout hardwired to 60s

[M. Balridge via dovecot]

> I'm a denizen of the solr-u...@lucene.apache.org mailing list.
> [...]
> Here's a wiki page that I wrote about that topic.  This wiki is going
> away next month, but for now you can still access it:
> 
> https://wiki.apache.org/solr/SolrPerformanceProblems

That's a great resource, Shawn.

I am about to put together a test case to provide a comprehensive FTS setup
around Dovecot with a goal towards exposing proximity keyword searching, with
email silos containing tens of terabytes (most of the "bulk" is represented by
attachments, each of which get processed down to plaintext, if possible).
Figure thousands of users with decades of email (80,000 to 750,000) emails per
user).

My main background is in software engineering (C/C++/Python/Assembler), but I
have been forced into system admin tasks during many stretches of my work. I
do vividly remember the tedium of dealing with JAVA and GC, tuning it to avoid
stalls, and its ravenous appetite for RAM. 

It looks like those problems are still with us, many versions later.  For
corporations with infinite budgets, throwing lots of crazy money at the
problem is "fine" (>1TB RAM, all PCIe SSDs, etc), but I am worried that I will
be shoved forcefully into a wall of having to spend a fortune just to keep FTS
performing reasonably well before I even get to the 10,000 user mark.

I realise the only way to keep performance reasonable is to heavily shard the
index database, but I am concerned about how well the process works in
practice without needing a great deal of sysadmin hand-holding. I would
ideally prefer the decisions of how/where to shard be based on
volume/heuristics than something that is done manually. I realise that a human
will be necessary to add more hardware to the pools, but what are my options
for scaling the system by orders of magnitude?

What is a general rule of thumb for RAM and SSD disk requirements as a
fraction of indexed document hive size to keep query performance at 200ms or
less? How do people deal with the JAVA GC world-stoppages, other than simply
doubling or tripling every instance?

I am wondering how well alternatives to Solr work in these situations
(ElasticSearch, Xapian, and any others I may have missed).

Regards,

=M=

Re: Solr connection timeout hardwired to 60s

[Peter Mogensen via dovecot]

On 4/4/19 6:47 PM, dovecot-requ...@dovecot.org wrote:
> For a typical Solr index, 60 seconds is an eternity.  Most people aim
> for query times of 100 milliseconds or less, and they often achieve
> that goal.

I'm pretty sure I get these while indexing, not querying.

Apr 04 16:44:50 host dovecot[114690]: indexer-worker(m...@example.com):
Error: fts_solr: Indexing failed: Request timed out (Request queued
66.015 secs ago, 1 attempts in 66.005 secs, 63.146 in http ioloop, 0.000
in other ioloops, connected 94.903 secs ago)

/Peter

Re: Solr connection timeout hardwired to 60s

[Shawn Heisey via dovecot]

On 4/4/2019 2:21 AM, Peter Mogensen via dovecot wrote:

What's the recommended way to handling timeouts on large mailboxes given
the hardwired request timeout of 60s in solr-connection.c:

http_set.request_timeout_msecs = 60*1000;


I'm a denizen of the solr-u...@lucene.apache.org mailing list.

For a typical Solr index, 60 seconds is an eternity.  Most people aim 
for query times of 100 milliseconds or less, and they often achieve that 
goal.


If you have an index where queries really are taking longer than 60 
seconds, you're most likely going to need to get better hardware for 
Solr.  Memory is the resource that usually has the greatest impact on 
Solr performance.  Putting the index on SSD can help, but memory will 
help more.


Here's a wiki page that I wrote about that topic.  This wiki is going 
away next month, but for now you can still access it:


https://wiki.apache.org/solr/SolrPerformanceProblems

There's a section in that wiki page about asking for help on performance 
issues.  It describes how to create a particular process listing for a 
screenshot.  If you can get that screenshot and share it using a file 
sharing site (dropbox is usually a good choice), I may be able to offer 
some insight.


Thanks,
Shawn

Re: Solr connection timeout hardwired to 60s

[Daniel Lange via dovecot]

Hi Shawn

Am 04.04.19 um 16:12 schrieb Shawn Heisey via dovecot:
> On 4/4/2019 2:21 AM, Peter Mogensen via dovecot wrote:
> Here's a wiki page that I wrote about that topic.  This wiki is going 
> away next month, but for now you can still access it:
> 
> https://wiki.apache.org/solr/SolrPerformanceProblems

https://web.archive.org/web/20190404143817/https://wiki.apache.org/solr/SolrPerformanceProblems

That one will last longer :).

Best
Daniel

Re: Solr connection timeout hardwired to 60s

[Shawn Heisey via dovecot]

On 4/4/2019 2:21 AM, Peter Mogensen via dovecot wrote:

What's the recommended way to handling timeouts on large mailboxes given
the hardwired request timeout of 60s in solr-connection.c:

http_set.request_timeout_msecs = 60*1000;


I'm a denizen of the solr-u...@lucene.apache.org mailing list.

For a typical Solr index, 60 seconds is an eternity.  Most people aim 
for query times of 100 milliseconds or less, and they often achieve that 
goal.


If you have an index where queries really are taking longer than 60 
seconds, you're most likely going to need to get better hardware for 
Solr.  Memory is the resource that usually has the greatest impact on 
Solr performance.  Putting the index on SSD can help, but memory will 
help more.


Here's a wiki page that I wrote about that topic.  This wiki is going 
away next month, but for now you can still access it:


https://wiki.apache.org/solr/SolrPerformanceProblems

There's a section in that wiki page about asking for help on performance 
issues.  It describes how to create a particular process listing for a 
screenshot.  If you can get that screenshot and share it using a file 
sharing site (dropbox is usually a good choice), I may be able to offer 
some insight.


Thanks,
Shawn

Solr connection timeout hardwired to 60s

[Peter Mogensen via dovecot]

Hi,

What's the recommended way to handling timeouts on large mailboxes given
the hardwired request timeout of 60s in solr-connection.c:

   http_set.request_timeout_msecs = 60*1000;


/Peter