Re: [PATCH] Re: Solr connection timeout hardwired to 60s

[Aki Tuomi via dovecot]

On 14 April 2019 16:59 John Fawcett via dovecot < dovecot@dovecot.org> wrote:

On 13/04/2019 17:16, Shawn Heisey via dovecot wrote:

On 4/13/2019 4:29 AM, John Fawcett via dovecot wrote:

If this value was made configurable people could set it to what they

want. However the underlying problem is likely on solr configuration.

The Jetty that is included in Solr has its idle timeout set to 50

seconds.  But in practice, I have not seen this timeout trigger ...

and if the OP is seeing a 60 second timeout, then the 50 second idle

timeout in Jetty must not be occurring.

There may be a socket timeout configured on inter-server requests --

distributed queries or the load balancing that SolrCloud does.  I can

never remember whether this is the case by default.  I think it is.

>> If there is an issue on initial indexing, where you are not really

>> concerned about qucik visibility but just getting things into the index

>> efficiently, a better approach would be for dovecot plugin not to send

>> any commit or softCommit (or waitSearcher either) and that should speed

>> things up. You'd need to configure solr with a long autoSoftCommit

>> maxTime and a reasonable autoCommit maxTime, which you could then

>> reconfigure when the load was done.

>

Solr ships with autoCommit set to 15 seconds and openSearcher set to

false on the autoCommit.  The autoSoftCommit setting is not enabled by

default, but depending on how the index was created, Solr might try to

set autoSoftCommit to 3 seconds ... which is WAY too short.

I will usually increase the autoCommit time to 60 seconds, just to

reduce the amount of work that Solr is doing.  The autoSoftCommit

time, if it is used, should be set to a reasonably long value ...

values between two and five minutes would be good.  Attempting to use

a very short autoSoftCommit time will usually lead to problems.

This thread says that dovecot is sending explicit commits.  One thing

that might be happening to exceed 60 seconds is an extremely long

commit, which is usually caused by excessive cache autowarming, but

might be related to insufficient memory.  The max heap setting on an

out-of-the-box Solr install (5.0 and later) is 512MB.  That's VERY

small, and it doesn't take much index data before a much larger heap

is required.

Thanks,

Shawn

I looked into the code (version 2.3.5.1): the fts-solr plugin is not

sending softCommit every 1000 emails. Emails from a single folder are

batched in up to a maximum of 1000 emails per request, but the

softCommit gets sent once per mailbox folder at the end of all the

requests for that folder.

I immagine that one of the reasons dovecot sends softCommits is because

without autoindex active and even if mailboxes are periodically indexed

from cron, the last emails received with be indexed at the moment of the

search.  So while sending softCommit has the advantage of including

recent mails in searches, it means that softCommits are being done upon

user request. Frequency depends on user activity.

Going back to the original problem: seems the first advice to Peter is

to look into solr configuration as others have said.

From dovecot point of view I can see the following as potentially useful

features:

1) a configurable batch size would enable to tune the number of emails

per request and help stay under the 60 seconds hard coded http request

timeout. A configurable http timeout would be less useful, since this

will potentially run into other timeouts on solr side.

2) abilty to turn off softCommits so as to have a more predictable

softCommit workload. In that case autoSoftCommit should be configured in

solr. In order to minimize risk of recent emails not appearing in search

results, periodic indexing could be set up by cron.

I've attached a patch, any comments are welcome (especially about

getting settings from the backend context).

Example config

plugin {

  fts = solr

  fts_solr =

url= "" href="https://user:passw...@solr.example.com:443/solr/dovecot/" rel="noopener" target="_blank">https://user:passw...@solr.example.com:443/solr/dovecot/

batch_size=500 no_soft_commit

}

John

Can you please open a pull request to https://github.com/dovecot/core ?

---
Aki Tuomi