[ https://issues.apache.org/jira/browse/SOLR-4816?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13765114#comment-13765114 ]
Shikhar Bhushan commented on SOLR-4816: --------------------------------------- We've run into some issues with CloudSolrServer leaking loads of LBHttpSolrServer's aliveCheckExecutor thread pools with {{parallelUpdates = true}}. The root cause here is that the RequestTask inner class is creating a new LBHttpSolrServer for each run() rather than utilizing CloudSolrServer.lbServer which is already available to it. Some detail: LBHttpSolrServer lazily initializes a single-threaded ScheduledExecutorService for the "aliveCheckExecutor" when e.g. there is some kind of error talking to a server. So this issue tends to come up when Solr nodes are unavailable and exceptions are thrown. There is also no call to shutdown() on that LBHttpSolrServer which gets created from RequestTask.run(). LBHttpSolrServer does have a finalizer that tries to shutdown the aliveCheckExecutor but there's no guarantee of finalizers executing (or maybe there is some other memory leak preventing that LBHttpSolrServer from being GC'ed at all). So the one-liner fix that should definitely go in is to simply have RequestTask use CloudSolrServer.lbServer. I have attached a patch that removes RequestTask altogether in favor of simply using Callable's and Future's which is much more idiomatic. (RequestTask-removal.patch) > Add document routing to CloudSolrServer > --------------------------------------- > > Key: SOLR-4816 > URL: https://issues.apache.org/jira/browse/SOLR-4816 > Project: Solr > Issue Type: Improvement > Components: SolrCloud > Affects Versions: 4.3 > Reporter: Joel Bernstein > Assignee: Mark Miller > Priority: Minor > Fix For: 4.5, 5.0 > > Attachments: RequestTask-removal.patch, SOLR-4816.patch, > SOLR-4816.patch, SOLR-4816.patch, SOLR-4816.patch, SOLR-4816.patch, > SOLR-4816.patch, SOLR-4816.patch, SOLR-4816.patch, SOLR-4816.patch, > SOLR-4816.patch, SOLR-4816.patch, SOLR-4816.patch, SOLR-4816.patch, > SOLR-4816.patch, SOLR-4816.patch, SOLR-4816.patch, SOLR-4816.patch, > SOLR-4816.patch, SOLR-4816.patch, SOLR-4816.patch, SOLR-4816.patch, > SOLR-4816.patch, SOLR-4816.patch, SOLR-4816.patch, SOLR-4816.patch, > SOLR-4816.patch, SOLR-4816.patch, SOLR-4816.patch, SOLR-4816-sriesenberg.patch > > > This issue adds the following enhancements to CloudSolrServer's update logic: > 1) Document routing: Updates are routed directly to the correct shard leader > eliminating document routing at the server. > 2) Optional parallel update execution: Updates for each shard are executed in > a separate thread so parallel indexing can occur across the cluster. > These enhancements should allow for near linear scalability on indexing > throughput. > Usage: > CloudSolrServer cloudClient = new CloudSolrServer(zkAddress); > cloudClient.setParallelUpdates(true); > SolrInputDocument doc1 = new SolrInputDocument(); > doc1.addField(id, "0"); > doc1.addField("a_t", "hello1"); > SolrInputDocument doc2 = new SolrInputDocument(); > doc2.addField(id, "2"); > doc2.addField("a_t", "hello2"); > UpdateRequest request = new UpdateRequest(); > request.add(doc1); > request.add(doc2); > request.setAction(AbstractUpdateRequest.ACTION.OPTIMIZE, false, false); > NamedList response = cloudClient.request(request); // Returns a backwards > compatible condensed response. > //To get more detailed response down cast to RouteResponse: > CloudSolrServer.RouteResponse rr = (CloudSolrServer.RouteResponse)response; -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators For more information on JIRA, see: http://www.atlassian.com/software/jira --------------------------------------------------------------------- To unsubscribe, e-mail: dev-unsubscr...@lucene.apache.org For additional commands, e-mail: dev-h...@lucene.apache.org