Re: SolrCloud cluster does not accept new documents for indexing

Shawn, Mikhail, Chris, Thank you all for your feedback. Unfortunately I cannot try your recommendations right away - this week is busy. Will post my results here next week. Regards, Denis On Tue, Apr 24, 2018 at 11:33 AM Shawn Heisey wrote: > On 4/24/2018 6:30 AM, Chris

Re: SolrCloud cluster does not accept new documents for indexing

Hi Denis, Merge works on segments and depending on merge strategy it is triggered separately so there is no some queue between update executor and merge threads. Re SPM - I am using it on a daily bases for most of my consulting work and if you have SPM app you can invite me to it and I’ll take

Re: SolrCloud cluster does not accept new documents for indexing

On 4/24/2018 6:30 AM, Chris Ulicny wrote: I haven't worked with AWS, but recently we tried to move some of our solr instances to a cloud in Google's Cloud offering, and it did not go well. All of our problems ended up stemming from the fact that the I/O is throttled. Any complicated enough query

Re: SolrCloud cluster does not accept new documents for indexing

Denis, Can you enable infoSteam https://lucene.apache.org/solr/guide/6_6/indexconfig-in-solrconfig.html#IndexConfiginSolrConfig-OtherIndexingSettings and examine logs about throttling? And what if you try without auto-commit? On Tue, Apr 24, 2018 at 12:37 AM, Denis Demichev

Re: SolrCloud cluster does not accept new documents for indexing

I haven't worked with AWS, but recently we tried to move some of our solr instances to a cloud in Google's Cloud offering, and it did not go well. All of our problems ended up stemming from the fact that the I/O is throttled. Any complicated enough query would require too many disk reads to return

Re: SolrCloud cluster does not accept new documents for indexing

I conducted another experiment today with local SSD drives, but this did not seem to fix my problem. Don't see any extensive I/O in this case: Device:tpskB_read/skB_wrtn/skB_readkB_wrtn xvda 1.7688.83 5.521256191 77996 xvdb

Re: SolrCloud cluster does not accept new documents for indexing

When all indexing threads are occupied merging, incoming updates block until at least one thread frees up IIUC. The fact that you're not opening searchers doesn't matter as far as merging is concerned, that happens regardless on hard commits. Bumping your ram buffer up to 2G is usually

Re: SolrCloud cluster does not accept new documents for indexing

Mikhail, I see what you're saying. Thank you for the clarification. Yes, there's no single line in the client code that contains a commit statement. The only thing I do: solr.add(collectionName, dataToSend); where solr is a SolrClient. Autocommits are set up on the server side for 2 minutes and

Re: SolrCloud cluster does not accept new documents for indexing

Threads are hanging on merge io throthling at org.apache.lucene.index.MergePolicy$OneMergeProgress.pauseNanos(MergePolicy.java:150) at org.apache.lucene.index.MergeRateLimiter.maybePause(MergeRateLimiter.java:148) at

Re: SolrCloud cluster does not accept new documents for indexing

Erick, Thank you for your quick response. I/O bottleneck: Please see another screenshot attached, as you can see disk r/w operations are pretty low or not significant. iostat== Device: rrqm/s wrqm/s r/s w/srkB/swkB/s avgrq-sz avgqu-sz await r_await w_await

Re: SolrCloud cluster does not accept new documents for indexing

Have you changed any of the merge policy parameters? I doubt it but just asking. My guess: your I/O is your bottleneck. There are a limited number of threads (tunable) that are used for background merging. When they're all busy, incoming updates are queued up. This squares with your statement