[
https://issues.apache.org/jira/browse/SPARK-41585?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Luca Canali updated SPARK-41585:
--------------------------------
Description:
The Spark exclude node functionality for YARN, introduced in SPARK-26688,
allows users to specify a list of node names that are excluded from resource
allocation. This is done using the configuration parameter:
{{spark.yarn.exclude.nodes}}
The feature currently works only for executors allocated via dynamic
allocation. To use the feature on Spark 3.3.1, for example, one may need also
to configure spark.dynamicAllocation.minExecutors=0 and
spark.executor.instances=0, therefore relying on executor resource allocation
only via dynamic allocation.
This proposes to extend the use of Spark exclude node functionality for YARN
beyond dynamic allocation, which I believe makes it more consistent also with
what the documentation reports for this feature/configuration parameter.
was:
The Spark exclude node functionality for YARN, introduced in SPARK-26688,
allows users to specify a list of node names that are excluded from resource
allocation. This is done using the configuration parameter:
{{spark.yarn.exclude.nodes}}
The feature currently works only for executors allocated via dynamic
allocation. To use the feature on Spark 3.3.1, for eaxmple, one needs to
configure spark.dynamicAllocation.minExecutors=0 and
spark.executor.instances=0, therefore relying on executor resource allocation
only via dynamic allocation.
This proposes to extend the use of Spark exclude node functionality for YARN
beyond dynamic allocation, which I believe makes it more consistent also with
what the documentation reports for this feature/configuration parameter.
> The Spark exclude node functionality for YARN should work independently of
> dynamic allocation
> ---------------------------------------------------------------------------------------------
>
> Key: SPARK-41585
> URL: https://issues.apache.org/jira/browse/SPARK-41585
> Project: Spark
> Issue Type: Bug
> Components: YARN
> Affects Versions: 3.3.1
> Reporter: Luca Canali
> Priority: Minor
>
> The Spark exclude node functionality for YARN, introduced in SPARK-26688,
> allows users to specify a list of node names that are excluded from resource
> allocation. This is done using the configuration parameter:
> {{spark.yarn.exclude.nodes}}
> The feature currently works only for executors allocated via dynamic
> allocation. To use the feature on Spark 3.3.1, for example, one may need also
> to configure spark.dynamicAllocation.minExecutors=0 and
> spark.executor.instances=0, therefore relying on executor resource allocation
> only via dynamic allocation.
> This proposes to extend the use of Spark exclude node functionality for YARN
> beyond dynamic allocation, which I believe makes it more consistent also with
> what the documentation reports for this feature/configuration parameter.
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscr...@spark.apache.org
For additional commands, e-mail: issues-h...@spark.apache.org
