Hi,
we have activated fts-solr about a week ago and immediately started to
experience really *low* performance with MOVE and EXPUNGE commands.
After several days of googling, tcpdumping and straceing i was able to
find and resolve the problem.
We are using Dovecot 2.2.27 from Debian Jessie (jessie-backports),
which is doing a soft commit in solr after every MOVE or EXPUNGE
command - this behavior cannot be, currently, changed. The problem is
that this was causing every MOVE/EXPUNGE to take about 6 seconds to
complete. The problem appears to be in very old version of Solr -
3.6.2 (!!). This is the only version which is shipped with current
(Jessie) and also next (Stretch) version of Debian, don't ask my why,
i don't understand it either. Solr versions below 4.0 are NOT
supporting soft commits, so all commits are hard and this was the
problem. Finally, i decided to patch our Dovecot to not send a commit
at all and everything started to be super fast. I'm doing hard commits
every minute via cron so the only consequence of this is that you
cannot search for messages delivered before less then a minute (which
you, usually, don't need to do anyway).
While googling i also find out that Solr supports autoCommit function
(and from version 4.0 also autoSoftCommit), so there's no reason for
Dovecot to handle this on it's own (and potentially doing hundreds or
thousands of soft commits every second) - you can just set Solr to,
for example, do autoSoftCommit every second and autoCommit every minute:
https://cwiki.apache.org/confluence/display/solr/UpdateHandlers+in+SolrConfig#UpdateHandlersinSolrConfig-autoCommit
Also this wiki page should be updated with warning about old versoins
of Solr not supporting soft commits (you could also mention the
auto[Soft]Commit function):
http://wiki2.dovecot.org/Plugins/FTS/Solr
I suggest to allow completely disable Solr commits in Dovecot by
configuration, so people like me can handle this easily. What do you
think?
azur
