Vijay: Thanks for reporting this back! Could I ask you to post a new patch with your correction? Please use the same patch name (SOLR-5850.patch), and include a note about what you found (I've already added a comment).
Thanks! Erick On Wed, Jan 28, 2015 at 9:18 AM, Vijay Sekhri <sekhrivi...@gmail.com> wrote: > Hi Shawn, > Thank you so much for the assistance. Building is not a problem . Back in > the days I have worked with linking, compiling and building C , C++ > software . Java is a piece of cake. > We have built the new war from the source version 4.10.3 and our > preliminary tests have shown that our issue (replicas in recovery on high > load)* is resolved *. We will continue to do more testing and confirm . > Please note that the *patch is BUGGY*. > > It removed the break statement within while loop because of which, whenever > we send a list of docs it would hang (API CloudSolrServer.add) , but it > would work if send one doc at a time. > > It took a while to figure out why that is happening. Once we put the break > statement back it worked like a charm. > Furthermore the patch has > > solr/solrj/src/java/org/apache/solr/client/solrj/impl/ConcurrentUpdateSolrClient.java > which should be > > solr/solrj/src/java/org/apache/solr/client/solrj/impl/ConcurrentUpdateSolrServer.java > > Finally checking if(!offer) is sufficient than using if(offer == false) > Last but not the least having a configurable queue size and timeouts > (managed via solrconfig) would be quite helpful > Thank you once again for your help. > > Vijay > > On Tue, Jan 27, 2015 at 6:20 PM, Shawn Heisey <apa...@elyograg.org> wrote: > > > On 1/27/2015 2:52 PM, Vijay Sekhri wrote: > > > Hi Shawn, > > > Here is some update. We found the main issue > > > We have configured our cluster to run under jetty and when we tried > full > > > indexing, we did not see the original Invalid Chunk error. However the > > > replicas still went into recovery > > > All this time we been trying to look into replicas logs to diagnose the > > > issue. The problem seem to be at the leader side. When we looked into > > > leader logs, we found the following on all the leaders > > > > > > 3439873 [qtp1314570047-92] WARN > > > org.apache.solr.update.processor.DistributedUpdateProcessor – Error > > > sending update > > > *java.lang.IllegalStateException: Queue full* > > > > <snip> > > > > > There is a similar bug reported around this > > > https://issues.apache.org/jira/browse/SOLR-5850 > > > > > > and it seem to be in OPEN status. Is there a way we can configure the > > queue > > > size and increase it ? or is there a version of solr that has this > issue > > > resolved already? > > > Can you suggest where we go from here to resolve this ? We can repatch > > the > > > war file if that is what you would recommend . > > > In the end our initial speculation about solr unable to handle so many > > > update is correct. We do not see this issue when the update load is > less. > > > > Are you in a position where you can try the patch attached to > > SOLR-5850? You would need to get the source code for the version you're > > on (or perhaps a newer 4.x version), patch it, and build Solr yourself. > > If you have no experience building java packages from source, this might > > prove to be difficult. > > > > Thanks, > > Shawn > > > > > > > -- > ********************************************* > Vijay Sekhri > ********************************************* >
