[
https://issues.apache.org/jira/browse/SOLR-4816?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13765114#comment-13765114
]
Shikhar Bhushan commented on SOLR-4816:
---------------------------------------
We've run into some issues with CloudSolrServer leaking loads of
LBHttpSolrServer's aliveCheckExecutor thread pools with {{parallelUpdates =
true}}.
The root cause here is that the RequestTask inner class is creating a new
LBHttpSolrServer for each run() rather than utilizing CloudSolrServer.lbServer
which is already available to it.
Some detail: LBHttpSolrServer lazily initializes a single-threaded
ScheduledExecutorService for the "aliveCheckExecutor" when e.g. there is some
kind of error talking to a server. So this issue tends to come up when Solr
nodes are unavailable and exceptions are thrown. There is also no call to
shutdown() on that LBHttpSolrServer which gets created from RequestTask.run().
LBHttpSolrServer does have a finalizer that tries to shutdown the
aliveCheckExecutor but there's no guarantee of finalizers executing (or maybe
there is some other memory leak preventing that LBHttpSolrServer from being
GC'ed at all).
So the one-liner fix that should definitely go in is to simply have RequestTask
use CloudSolrServer.lbServer.
I have attached a patch that removes RequestTask altogether in favor of simply
using Callable's and Future's which is much more idiomatic.
(RequestTask-removal.patch)
> Add document routing to CloudSolrServer
> ---------------------------------------
>
> Key: SOLR-4816
> URL: https://issues.apache.org/jira/browse/SOLR-4816
> Project: Solr
> Issue Type: Improvement
> Components: SolrCloud
> Affects Versions: 4.3
> Reporter: Joel Bernstein
> Assignee: Mark Miller
> Priority: Minor
> Fix For: 4.5, 5.0
>
> Attachments: RequestTask-removal.patch, SOLR-4816.patch,
> SOLR-4816.patch, SOLR-4816.patch, SOLR-4816.patch, SOLR-4816.patch,
> SOLR-4816.patch, SOLR-4816.patch, SOLR-4816.patch, SOLR-4816.patch,
> SOLR-4816.patch, SOLR-4816.patch, SOLR-4816.patch, SOLR-4816.patch,
> SOLR-4816.patch, SOLR-4816.patch, SOLR-4816.patch, SOLR-4816.patch,
> SOLR-4816.patch, SOLR-4816.patch, SOLR-4816.patch, SOLR-4816.patch,
> SOLR-4816.patch, SOLR-4816.patch, SOLR-4816.patch, SOLR-4816.patch,
> SOLR-4816.patch, SOLR-4816.patch, SOLR-4816.patch, SOLR-4816-sriesenberg.patch
>
>
> This issue adds the following enhancements to CloudSolrServer's update logic:
> 1) Document routing: Updates are routed directly to the correct shard leader
> eliminating document routing at the server.
> 2) Optional parallel update execution: Updates for each shard are executed in
> a separate thread so parallel indexing can occur across the cluster.
> These enhancements should allow for near linear scalability on indexing
> throughput.
> Usage:
> CloudSolrServer cloudClient = new CloudSolrServer(zkAddress);
> cloudClient.setParallelUpdates(true);
> SolrInputDocument doc1 = new SolrInputDocument();
> doc1.addField(id, "0");
> doc1.addField("a_t", "hello1");
> SolrInputDocument doc2 = new SolrInputDocument();
> doc2.addField(id, "2");
> doc2.addField("a_t", "hello2");
> UpdateRequest request = new UpdateRequest();
> request.add(doc1);
> request.add(doc2);
> request.setAction(AbstractUpdateRequest.ACTION.OPTIMIZE, false, false);
> NamedList response = cloudClient.request(request); // Returns a backwards
> compatible condensed response.
> //To get more detailed response down cast to RouteResponse:
> CloudSolrServer.RouteResponse rr = (CloudSolrServer.RouteResponse)response;
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators
For more information on JIRA, see: http://www.atlassian.com/software/jira
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
