[
https://issues.apache.org/jira/browse/SPARK-13232?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15137222#comment-15137222
]
Atkins commented on SPARK-13232:
--------------------------------
If spark config "spark.yarn.executor.nodeLabelExpression" present,
*org.apache.spark.deploy.yarn.YarnAllocator#createContainerRequest* will create
a ContainerRequest instance with locality specification of nodes, racks, and
nodelabel which cause InvalidContainerRequestException be thrown.
This can reproduce by adding test suite in
*org.apache.spark.deploy.yarn.YarnAllocatorSuite*
{code}
test("request executors with locality") {
val handler = createAllocator(1)
handler.updateResourceRequests()
handler.getNumExecutorsRunning should be (0)
handler.getPendingAllocate.size should be (1)
handler.requestTotalExecutorsWithPreferredLocalities(3, 20, Map(("host1",
10), ("host2", 20)))
handler.updateResourceRequests()
handler.getPendingAllocate.size should be (3)
val container = createContainer("host1")
handler.handleAllocatedContainers(Array(container))
handler.getNumExecutorsRunning should be (1)
handler.allocatedContainerToHostMap.get(container.getId).get should be
("host1")
handler.allocatedHostToContainersMap.get("host1").get should contain
(container.getId)
}
{code}
> YARN executor node label expressions
> ------------------------------------
>
> Key: SPARK-13232
> URL: https://issues.apache.org/jira/browse/SPARK-13232
> Project: Spark
> Issue Type: Improvement
> Components: YARN
> Environment: Scala 2.11.7, Hadoop 2.7.2, Spark 1.6.0
> Reporter: Atkins
> Priority: Minor
>
> Using node label expression for executor failed to request container request
> and throws *InvalidContainerRequestException*.
> The code
> {code:title=AMRMClientImpl.java}
> /**
> * Valid if a node label expression specified on container request is valid
> or
> * not
> *
> * @param containerRequest
> */
> private void checkNodeLabelExpression(T containerRequest) {
> String exp = containerRequest.getNodeLabelExpression();
>
> if (null == exp || exp.isEmpty()) {
> return;
> }
> // Don't support specifying >= 2 node labels in a node label expression
> now
> if (exp.contains("&&") || exp.contains("||")) {
> throw new InvalidContainerRequestException(
> "Cannot specify more than two node labels"
> + " in a single node label expression");
> }
>
> // Don't allow specify node label against ANY request
> if ((containerRequest.getRacks() != null &&
> (!containerRequest.getRacks().isEmpty()))
> ||
> (containerRequest.getNodes() != null &&
> (!containerRequest.getNodes().isEmpty()))) {
> throw new InvalidContainerRequestException(
> "Cannot specify node label with rack and node");
> }
> }
> {code}
> doesn't allow node label with rack and node.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscr...@spark.apache.org
For additional commands, e-mail: issues-h...@spark.apache.org
