Hi Shawn, Thanks for your time to check the logs. we are getting the " ErrorReporingConcurrentUpdateSolrClinet Error when calling solrcmdDistributor"
Please find the log details : https://drive.google.com/file/d/1lj6oR4pS9hfgySP-QBY5ipnAZXfAUWFT/view?usp=sharing And it comes frequently for the indexing that happens every 15 mins. This happens with solr 8.8.2. In production we have the same data-config.xml running Solr 4.10.4, we don't have any issues in production. In solr 4 we were using a solrserver connection passing the leader url for indexing. Now with solr8, as suggested we are using cloudsolrclient passing the ip's of the zookeepers. What we assume is when indexing is called it does it in one node, and when it tries to do a distributed update another node fails, not sure why this is happening. Any advice is highly appreciated. *Thanks,* *Reej* On Tue, Aug 17, 2021 at 9:52 PM Shawn Heisey <apa...@elyograg.org> wrote: > On 8/16/2021 5:17 AM, Reej Nayagam wrote: > > Hi Shawn , > > Please find the gc logs from our test server. Your help is much > appreciated. > > > > > https://drive.google.com/drive/folders/1VQqlRxiDnHWPUlmTUv83a8CqCtvBP4le?usp=sharing > > I analyzed the logs with the gceasy.io website. Garbage collection is > healthy. I did not expect to see Solr's default heap size of 512MB ... > most users must increase that for success. Are we dealing with a very > small index here? > > > Also our Solr version is 8.8.2, so if there is a fix after 8.4+ , the > 8.8.2 > > version should have this anyway right. > > > As for the "stall" message ... that's part of the code update for that > issue. The fact that you are seeing the message means that Solr is > breaking out of the stalled process -- there's nothing left to fix in > the code. > > > I've increased the autocommit maxtime to 60000, opensearcher is false > and > > autosoftcommit maxtime to 120000 to check just for 2 collections, and for > > those 2 I'm not getting the Overlapping on Decksearchers so frequently , > > but the below errors are happening frequently. > > > > *Request Handler Base : java.io.IOException:Task Queue processing has > > stalled for 20191 ms with 0 remaining elements to process* > > *Httpsolrcall : java.io.IOException:Task Queue processing has stalled for > > 20191 ms with 0 remaining elements to process* > > *Solrwriter : Exception while deleting by query (throws err in Post > Import > > Delete query)* > > > As already mentioned, the fact that you are seeing this message means > that you have the fix for that issue, which was not reported before. As > for what's causing it to stall, I have no idea. It's not garbage > collection pauses, which was the candidate I was expecting to be a problem. > > > Initially in solr.in.cmd the heap and GC settings were not enabled , now > > it's enabled. Added the details (solr.in.jpeg) in the above link as > well. > > Uploaded the error log screenshot (solr_logging.jpeg) > > You have explicitly enabled Solr's default heap size of 512MB. Solr has > GC tuning even if you don't specify it yourself. I believe with 8.8.2 > it will use G1GC. Your settings use CMS. I was actually the one that > came up with what's being used currently ... which I did long before the > bin\solr.cmd and bin/solr scripts were created. See my persoinal wiki > page on the Solr wiki: > > https://cwiki.apache.org/confluence/display/solr/shawnheisey > > The picture of the logging screen is not very helpful. Almost all of > the details are hidden. You can open each logging entry on the web page > by clicking the little "i" icon ... but the solr.log file would show all > the detail at once. > > Thanks, > Shawn > >
