On Mon, Aug 17, 2015 at 11:36 PM, Daniel Collins <danwcoll...@gmail.com> wrote:
> Just to open the can of worms, it *can* be possible to have very low commit > times, we have 250ms currently and are in production with that. But it > does come with pain (no such thing as a free lunch!), we had to turn off > ALL the Solr caches Gentlemen, Excuse me for hijacking, here are the small "segmentation" lunchbox: - segmented filters, which much cheaper for commit: http://blog.griddynamics.com/2014/01/segmented-filter-cache-in-solr.html - docvalues facets on steroids https://issues.apache.org/jira/browse/SOLR-7730 (since 5.3) But document's cache wasn't sliced yet. Thus, I prefer https://issues.apache.org/jira/browse/SOLR-7937 hangs for a while and to be pursued later. > (warming is useless at that kind of frequency, it will > take longer to warm the cache than the time before the next commit), and > throw a lot of RAM and expensive SSDs at the problem. > > That said, Shawn's advice is correct, anything less than 1s commit > shouldn't be needed for most users, and I would concur with staying away > from it unless you absolutely decide you have to have it. > > You only go that route if you are prepared to commit (no pun intended!) a > fair amount of time, money and resources to investigating and dealing with > issues. We will have a talk at Revolution this year about some of the > scale and latency issues we have to deal with (blatant plug for my team > lead who's giving the talk!) > > On 17 August 2015 at 14:31, Shawn Heisey <apa...@elyograg.org> wrote: > > > On 8/17/2015 7:04 AM, Maulin Rathod wrote: > > > We have observed that Intermittently querying become slower when > > documentCache become empty. The documentCache is getting flushed whenever > > new document added to the collection. > > > > > > Is there any way by which we can ensure that newly added documents are > > visible without losing data in documentCache? We are trying to use soft > > commit but it also flushes all documents in documentCache. > > > > <snip> > > > > > <autoSoftCommit> > > > <maxTime>50</maxTime> > > > </autoSoftCommit> > > > > You are doing a soft commit within 50 milliseconds of adding a new > > document. Solr can have severe performance problems when autoSoftCommit > > is set to 1000 -- one second. 50 milliseconds is one twentieth of a > > very low value that is known to cause problems. It can make the problem > > much more than 20 times worse. > > > > Please read this article: > > > > > > > http://lucidworks.com/blog/understanding-transaction-logs-softcommit-and-commit-in-sorlcloud/ > > > > Note one particular section, which says the following: Don’t listen to > > your product manager who says "we need no more than 1 second latency". > > > > You need to set your commit interval as long as you possibly can. I > > personally wouldn't go longer than 60 seconds, 30 seconds if the commits > > complete particularly fast. It should be several minutes if that will > > meet your needs. When your commit interval is very low, Solr's caches > > can become useless, as you've noticed. > > > > TL;DR info: Your autoCommit settings have openSearcher set to false, so > > they do not matter for the problem you have described. I would probably > > increase that to 5 minutes rather than 15 seconds, but that is not very > > important here, and 15 seconds for hard commits that don't open a new > > searcher is known to have a low impact on performance. "Low" impact > > isn't the same as NO impact, so I keep this interval long as well. > > > > Thanks, > > Shawn > > > > > -- Sincerely yours Mikhail Khludnev Principal Engineer, Grid Dynamics <http://www.griddynamics.com> <mkhlud...@griddynamics.com>
