On 14/04/2019 16:04, Aki Tuomi via dovecot wrote: > >> On 14 April 2019 16:59 John Fawcett via dovecot < dovecot@dovecot.org >> <mailto:dovecot@dovecot.org>> wrote: >> >> >> On 13/04/2019 17:16, Shawn Heisey via dovecot wrote: >>> On 4/13/2019 4:29 AM, John Fawcett via dovecot wrote: >>>> If this value was made configurable people could set it to what they >>>> want. However the underlying problem is likely on solr configuration. >>> The Jetty that is included in Solr has its idle timeout set to 50 >>> seconds. But in practice, I have not seen this timeout trigger ... >>> and if the OP is seeing a 60 second timeout, then the 50 second idle >>> timeout in Jetty must not be occurring. >>> There may be a socket timeout configured on inter-server requests -- >>> distributed queries or the load balancing that SolrCloud does. I can >>> never remember whether this is the case by default. I think it is. >> >> If there is an issue on initial indexing, where you are not really >> >> concerned about qucik visibility but just getting things into the >> index >> >> efficiently, a better approach would be for dovecot plugin not to >> send >> >> any commit or softCommit (or waitSearcher either) and that should >> speed >> >> things up. You'd need to configure solr with a long autoSoftCommit >> >> maxTime and a reasonable autoCommit maxTime, which you could then >> >> reconfigure when the load was done. >> > >>> Solr ships with autoCommit set to 15 seconds and openSearcher set to >>> false on the autoCommit. The autoSoftCommit setting is not enabled by >>> default, but depending on how the index was created, Solr might try to >>> set autoSoftCommit to 3 seconds ... which is WAY too short. >>> I will usually increase the autoCommit time to 60 seconds, just to >>> reduce the amount of work that Solr is doing. The autoSoftCommit >>> time, if it is used, should be set to a reasonably long value ... >>> values between two and five minutes would be good. Attempting to use >>> a very short autoSoftCommit time will usually lead to problems. >>> This thread says that dovecot is sending explicit commits. One thing >>> that might be happening to exceed 60 seconds is an extremely long >>> commit, which is usually caused by excessive cache autowarming, but >>> might be related to insufficient memory. The max heap setting on an >>> out-of-the-box Solr install (5.0 and later) is 512MB. That's VERY >>> small, and it doesn't take much index data before a much larger heap >>> is required. >>> Thanks, >>> Shawn >> I looked into the code (version 2.3.5.1): the fts-solr plugin is not >> sending softCommit every 1000 emails. Emails from a single folder are >> batched in up to a maximum of 1000 emails per request, but the >> softCommit gets sent once per mailbox folder at the end of all the >> requests for that folder. >> >> I immagine that one of the reasons dovecot sends softCommits is because >> without autoindex active and even if mailboxes are periodically indexed >> from cron, the last emails received with be indexed at the moment of the >> search. So while sending softCommit has the advantage of including >> recent mails in searches, it means that softCommits are being done upon >> user request. Frequency depends on user activity. >> >> Going back to the original problem: seems the first advice to Peter is >> to look into solr configuration as others have said. >> >> From dovecot point of view I can see the following as potentially useful >> features: >> >> 1) a configurable batch size would enable to tune the number of emails >> per request and help stay under the 60 seconds hard coded http request >> timeout. A configurable http timeout would be less useful, since this >> will potentially run into other timeouts on solr side. >> >> 2) abilty to turn off softCommits so as to have a more predictable >> softCommit workload. In that case autoSoftCommit should be configured in >> solr. In order to minimize risk of recent emails not appearing in search >> results, periodic indexing could be set up by cron. >> >> I've attached a patch, any comments are welcome (especially about >> getting settings from the backend context). >> >> Example config >> >> plugin { >> fts = solr >> fts_solr = >> url= https://user:passw...@solr.example.com:443/solr/dovecot/ >> batch_size=500 no_soft_commit >> } >> >> John > > Can you please open a pull request to https://github.com/dovecot/core ? > --- > Aki Tuomi
Done, thanks for considering it. John