Re: [enhancement] fts-solr low performance
On 05.03.2018 11:07, azurIt wrote: >> Hi, >> >> we have activated fts-solr about a week ago and immediately started to >> experience really *low* performance with MOVE and EXPUNGE commands. >> After several days of googling, tcpdumping and straceing i was able to >> find and resolve the problem. >> >> We are using Dovecot 2.2.27 from Debian Jessie (jessie-backports), >> which is doing a soft commit in solr after every MOVE or EXPUNGE >> command - this behavior cannot be, currently, changed. The problem is >> that this was causing every MOVE/EXPUNGE to take about 6 seconds to >> complete. The problem appears to be in very old version of Solr - >> 3.6.2 (!!). This is the only version which is shipped with current >> (Jessie) and also next (Stretch) version of Debian, don't ask my why, >> i don't understand it either. Solr versions below 4.0 are NOT >> supporting soft commits, so all commits are hard and this was the >> problem. Finally, i decided to patch our Dovecot to not send a commit >> at all and everything started to be super fast. I'm doing hard commits >> every minute via cron so the only consequence of this is that you >> cannot search for messages delivered before less then a minute (which >> you, usually, don't need to do anyway). >> >> While googling i also find out that Solr supports autoCommit function >> (and from version 4.0 also autoSoftCommit), so there's no reason for >> Dovecot to handle this on it's own (and potentially doing hundreds or >> thousands of soft commits every second) - you can just set Solr to, >> for example, do autoSoftCommit every second and autoCommit every minute: >> https://cwiki.apache.org/confluence/display/solr/UpdateHandlers+in+SolrConfig#UpdateHandlersinSolrConfig-autoCommit >> >> Also this wiki page should be updated with warning about old versoins >> of Solr not supporting soft commits (you could also mention the >> auto[Soft]Commit function): >> http://wiki2.dovecot.org/Plugins/FTS/Solr >> >> I suggest to allow completely disable Solr commits in Dovecot by >> configuration, so people like me can handle this easily. What do you >> think? >> >> azur > > > > Hi, > > any news on this? Even Solr documentation suggests to NOT doing commits from > applications: > https://lucene.apache.org/solr/guide/6_6/shards-and-indexing-data-in-solrcloud.html#ShardsandIndexingDatainSolrCloud-IgnoringCommitsfromClientApplicationsinSolrCloud > > Thanks for not ignoring me. > > azur You are not being ignored. We'll attend to this eventually. Aki
Re: [enhancement] fts-solr low performance
>Hi, > >we have activated fts-solr about a week ago and immediately started to >experience really *low* performance with MOVE and EXPUNGE commands. >After several days of googling, tcpdumping and straceing i was able to >find and resolve the problem. > >We are using Dovecot 2.2.27 from Debian Jessie (jessie-backports), >which is doing a soft commit in solr after every MOVE or EXPUNGE >command - this behavior cannot be, currently, changed. The problem is >that this was causing every MOVE/EXPUNGE to take about 6 seconds to >complete. The problem appears to be in very old version of Solr - >3.6.2 (!!). This is the only version which is shipped with current >(Jessie) and also next (Stretch) version of Debian, don't ask my why, >i don't understand it either. Solr versions below 4.0 are NOT >supporting soft commits, so all commits are hard and this was the >problem. Finally, i decided to patch our Dovecot to not send a commit >at all and everything started to be super fast. I'm doing hard commits >every minute via cron so the only consequence of this is that you >cannot search for messages delivered before less then a minute (which >you, usually, don't need to do anyway). > >While googling i also find out that Solr supports autoCommit function >(and from version 4.0 also autoSoftCommit), so there's no reason for >Dovecot to handle this on it's own (and potentially doing hundreds or >thousands of soft commits every second) - you can just set Solr to, >for example, do autoSoftCommit every second and autoCommit every minute: >https://cwiki.apache.org/confluence/display/solr/UpdateHandlers+in+SolrConfig#UpdateHandlersinSolrConfig-autoCommit > >Also this wiki page should be updated with warning about old versoins >of Solr not supporting soft commits (you could also mention the >auto[Soft]Commit function): >http://wiki2.dovecot.org/Plugins/FTS/Solr > >I suggest to allow completely disable Solr commits in Dovecot by >configuration, so people like me can handle this easily. What do you >think? > >azur Hi, any news on this? Even Solr documentation suggests to NOT doing commits from applications: https://lucene.apache.org/solr/guide/6_6/shards-and-indexing-data-in-solrcloud.html#ShardsandIndexingDatainSolrCloud-IgnoringCommitsfromClientApplicationsinSolrCloud Thanks for not ignoring me. azur
[enhancement] fts-solr low performance
Hi, we have activated fts-solr about a week ago and immediately started to experience really *low* performance with MOVE and EXPUNGE commands. After several days of googling, tcpdumping and straceing i was able to find and resolve the problem. We are using Dovecot 2.2.27 from Debian Jessie (jessie-backports), which is doing a soft commit in solr after every MOVE or EXPUNGE command - this behavior cannot be, currently, changed. The problem is that this was causing every MOVE/EXPUNGE to take about 6 seconds to complete. The problem appears to be in very old version of Solr - 3.6.2 (!!). This is the only version which is shipped with current (Jessie) and also next (Stretch) version of Debian, don't ask my why, i don't understand it either. Solr versions below 4.0 are NOT supporting soft commits, so all commits are hard and this was the problem. Finally, i decided to patch our Dovecot to not send a commit at all and everything started to be super fast. I'm doing hard commits every minute via cron so the only consequence of this is that you cannot search for messages delivered before less then a minute (which you, usually, don't need to do anyway). While googling i also find out that Solr supports autoCommit function (and from version 4.0 also autoSoftCommit), so there's no reason for Dovecot to handle this on it's own (and potentially doing hundreds or thousands of soft commits every second) - you can just set Solr to, for example, do autoSoftCommit every second and autoCommit every minute: https://cwiki.apache.org/confluence/display/solr/UpdateHandlers+in+SolrConfig#UpdateHandlersinSolrConfig-autoCommit Also this wiki page should be updated with warning about old versoins of Solr not supporting soft commits (you could also mention the auto[Soft]Commit function): http://wiki2.dovecot.org/Plugins/FTS/Solr I suggest to allow completely disable Solr commits in Dovecot by configuration, so people like me can handle this easily. What do you think? azur
