Hi Shawn,
Thanks for your feedback.
In our scenario documents are added frequently (Approx 10 documents added in 1
minute) and we want to make it available for search near realtime (within 5
second). Even if we set autosoftcommit 5 second (so that document will be
available for search after 5 second), it flushes all documents from
documentCache. Just wanted to understand if we are doing something wrong or its
solr expected behavior.
<autoSoftCommit>
<maxTime>5000</maxTime>
</autoSoftCommit>
Regards,
Maulin
-----Original Message-----
From: Shawn Heisey [mailto:apa...@elyograg.org]
Sent: 17 August 2015 19:02
To: solr-user@lucene.apache.org
Subject: Re: Solr Caching (documentCache) not working
On 8/17/2015 7:04 AM, Maulin Rathod wrote:
> We have observed that Intermittently querying become slower when
> documentCache become empty. The documentCache is getting flushed whenever new
> document added to the collection.
>
> Is there any way by which we can ensure that newly added documents are
> visible without losing data in documentCache? We are trying to use soft
> commit but it also flushes all documents in documentCache.
<snip>
> <autoSoftCommit>
> <maxTime>50</maxTime>
> </autoSoftCommit>
You are doing a soft commit within 50 milliseconds of adding a new document.
Solr can have severe performance problems when autoSoftCommit is set to 1000 --
one second. 50 milliseconds is one twentieth of a very low value that is known
to cause problems. It can make the problem much more than 20 times worse.
Please read this article:
http://lucidworks.com/blog/understanding-transaction-logs-softcommit-and-commit-in-sorlcloud/
Note one particular section, which says the following: Don’t listen to your
product manager who says "we need no more than 1 second latency".
You need to set your commit interval as long as you possibly can. I personally
wouldn't go longer than 60 seconds, 30 seconds if the commits complete
particularly fast. It should be several minutes if that will meet your needs.
When your commit interval is very low, Solr's caches can become useless, as
you've noticed.
TL;DR info: Your autoCommit settings have openSearcher set to false, so they
do not matter for the problem you have described. I would probably increase
that to 5 minutes rather than 15 seconds, but that is not very important here,
and 15 seconds for hard commits that don't open a new searcher is known to have
a low impact on performance. "Low" impact isn't the same as NO impact, so I
keep this interval long as well.
Thanks,
Shawn
