Hi Shawn, Thank you so much for the assistance. Building is not a problem . Back in the days I have worked with linking, compiling and building C , C++ software . Java is a piece of cake. We have built the new war from the source version 4.10.3 and our preliminary tests have shown that our issue (replicas in recovery on high load)* is resolved *. We will continue to do more testing and confirm . Please note that the *patch is BUGGY*.
It removed the break statement within while loop because of which, whenever we send a list of docs it would hang (API CloudSolrServer.add) , but it would work if send one doc at a time. It took a while to figure out why that is happening. Once we put the break statement back it worked like a charm. Furthermore the patch has solr/solrj/src/java/org/apache/solr/client/solrj/impl/ConcurrentUpdateSolrClient.java which should be solr/solrj/src/java/org/apache/solr/client/solrj/impl/ConcurrentUpdateSolrServer.java Finally checking if(!offer) is sufficient than using if(offer == false) Last but not the least having a configurable queue size and timeouts (managed via solrconfig) would be quite helpful Thank you once again for your help. Vijay On Tue, Jan 27, 2015 at 6:20 PM, Shawn Heisey <apa...@elyograg.org> wrote: > On 1/27/2015 2:52 PM, Vijay Sekhri wrote: > > Hi Shawn, > > Here is some update. We found the main issue > > We have configured our cluster to run under jetty and when we tried full > > indexing, we did not see the original Invalid Chunk error. However the > > replicas still went into recovery > > All this time we been trying to look into replicas logs to diagnose the > > issue. The problem seem to be at the leader side. When we looked into > > leader logs, we found the following on all the leaders > > > > 3439873 [qtp1314570047-92] WARN > > org.apache.solr.update.processor.DistributedUpdateProcessor – Error > > sending update > > *java.lang.IllegalStateException: Queue full* > > <snip> > > > There is a similar bug reported around this > > https://issues.apache.org/jira/browse/SOLR-5850 > > > > and it seem to be in OPEN status. Is there a way we can configure the > queue > > size and increase it ? or is there a version of solr that has this issue > > resolved already? > > Can you suggest where we go from here to resolve this ? We can repatch > the > > war file if that is what you would recommend . > > In the end our initial speculation about solr unable to handle so many > > update is correct. We do not see this issue when the update load is less. > > Are you in a position where you can try the patch attached to > SOLR-5850? You would need to get the source code for the version you're > on (or perhaps a newer 4.x version), patch it, and build Solr yourself. > If you have no experience building java packages from source, this might > prove to be difficult. > > Thanks, > Shawn > > -- ********************************************* Vijay Sekhri *********************************************
