[
https://issues.apache.org/jira/browse/SOLR-7294?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14377051#comment-14377051
]
Shalin Shekhar Mangar commented on SOLR-7294:
---------------------------------------------
The problem is in the OCP.migrateKey method:
{code}
log.info("Requesting merge of temp source collection replica to target leader");
params = new ModifiableSolrParams();
params.set(CoreAdminParams.ACTION, CoreAdminAction.MERGEINDEXES.toString());
params.set(CoreAdminParams.CORE, targetLeader.getStr("core"));
params.set(CoreAdminParams.SRC_CORE, tempCollectionReplica2);
setupAsyncRequest(asyncId, requestMap, params, sourceLeader.getNodeName());
sendShardRequest(targetLeader.getNodeName(), params, shardHandler);
collectShardResponses(results, true,
"MIGRATE failed to merge " + tempCollectionReplica2 +
" to " + targetLeader.getStr("core") + " on node: " +
targetLeader.getNodeName(),
shardHandler);
completeAsyncRequest(asyncId, requestMap, results);
{code}
Notice that the setupAsyncRequest is being called with
sourceLeader.getNodeName() but the actual request is being sent to the
targetLeader.getNodeName(). So fixing this part is easy enough.
I tried to see why our existing AsyncMigrateRouteKey test doesn't tickle this
problem and I was surprised that the test asks for the wrong node but always
gets the right status. Then I realized that it is because all the nodes in our
tests are loaded by the same classloader and since the core admin keeps the
requests in a static map, any node can give the status of an async core admin
API call. The request map in CoreAdminHandler doesn't need to be static. Once I
changed the request map to be an instance variable, this problem is reproduced
easily by the existing test.
We should refactor the code in OCP such that these situations become
impossible. I'll put up a patch.
I'll also create an issue to enforce a different class loader for each jetty.
> Migrate API fails with: Invalid status request: notfoundretried 6times
> ----------------------------------------------------------------------
>
> Key: SOLR-7294
> URL: https://issues.apache.org/jira/browse/SOLR-7294
> Project: Solr
> Issue Type: Bug
> Components: SolrCloud
> Affects Versions: 4.10.4, 5.0
> Reporter: Shalin Shekhar Mangar
> Assignee: Shalin Shekhar Mangar
> Fix For: Trunk, 5.1
>
> Attachments: source-leader.log, target.log
>
>
> Steps to reproduce:
> # Create a two node cluster
> # Create a collection called "source" with 1 shard, 1 replica
> # Add 1000 docs with prefix a!
> # Add 100 docs with prefix b! and c! each
> # Create a new target collection with 1 shard, 1 replica and ensure that it
> is created on a different node than "source"
> # Issue a migrate API call with an async parameter:
> {code}
> http://localhost:8983/solr/admin/collections?action=migrate&split.key=a!&collection=gettingstarted&target.collection=target&wt=json&async=acid
> {code}
> The above fails with:
> {code}
> ERROR - 2015-03-23 22:50:11.349; org.apache.solr.common.SolrException;
> Collection: gettingstarted operation: migrate
> failed:org.apache.solr.common.SolrException: Invalid status request:
> notfoundretried 6times
> at
> org.apache.solr.cloud.OverseerCollectionProcessor.waitForCoreAdminAsyncCallToComplete(OverseerCollectionProcessor.java:2807)
> at
> org.apache.solr.cloud.OverseerCollectionProcessor.waitForAsyncCallsToComplete(OverseerCollectionProcessor.java:2753)
> at
> org.apache.solr.cloud.OverseerCollectionProcessor.completeAsyncRequest(OverseerCollectionProcessor.java:2229)
> at
> org.apache.solr.cloud.OverseerCollectionProcessor.migrateKey(OverseerCollectionProcessor.java:2200)
> at
> org.apache.solr.cloud.OverseerCollectionProcessor.migrate(OverseerCollectionProcessor.java:1984)
> at
> org.apache.solr.cloud.OverseerCollectionProcessor.processMessage(OverseerCollectionProcessor.java:637)
> at
> org.apache.solr.cloud.OverseerCollectionProcessor$Runner.run(OverseerCollectionProcessor.java:2864)
> at
> java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142)
> at
> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)
> at java.lang.Thread.run(Thread.java:745)
> {code}
> Thanks to [~mewmewball] for finding this bug.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
