Arnaud Nauwynck created SPARK-60794:
---------------------------------------
Summary: Allow YARN executor vCore request to differ from
spark.executor.cores (add spark.yarn.executor.request.cores as for k8s)
Key: SPARK-60794
URL: https://issues.apache.org/jira/browse/SPARK-60794
Project: Spark
Issue Type: Improvement
Components: YARN
Affects Versions: 3.5.9, 4.0.5, 4.1.3, 4.1.4, 4.2.1, 4.3.0, 5.0.0
Reporter: Arnaud Nauwynck
h3. Problem
On YARN, spark.executor.cores drives two things:
the vCores requested from YARN per executor container
(YarnAllocator.createYarnResourceForResourceProfile)
the number of task slots per executor (--cores passed by ExecutorRunnable,
which the driver registers as totalCores)
There is no way to decouple them. Workloads that are mostly I/O-bound (JDBC,
REST, object-store calls) want many concurrent tasks per executor while
reserving few YARN vCores. Today they must either over-reserve cluster CPU or
under-use executors.
Kubernetes already supports this through
spark.kubernetes.executor.request.cores. YARN has no equivalent.
h3. Proposal
Add a new optional config:
{code}
spark.yarn.executor.request.cores (int, optional, default =
spark.executor.cores)
{code}
Used only to build the YARN container Resource request.
spark.executor.cores keeps defining task slots, --cores, and dynamic allocation
sizing.
The value is capped at spark.executor.cores.
It applies to the default ResourceProfile only. Stage-level profiles keep their
explicit cores.
Example:
{code}
spark.executor.cores=20
spark.yarn.executor.request.cores=5
{code}
This gives 20 concurrent tasks per executor while reserving 5 vCores per
container.
h3. Implementation
deploy/yarn/config.scala: add the EXECUTOR_REQUEST_CORES ConfigEntry.
YarnAllocator.createYarnResourceForResourceProfile: use the request cores in
Resource.newInstance(totalMem, cores).
No change to runAllocatedContainers, ExecutorRunnable, ResourceProfile or the
driver-side scheduler.
Container matching (matchContainerToRequest) uses the stored
rpIdToYarnResource, so it stays consistent.
h3. minimal patch:
in config.scala, add a new parameter
{code:java}
private[spark] val YARN_EXECUTOR_REQUEST_CORES =
ConfigBuilder("spark.yarn.executor.request.cores")
.doc("vcores requested from YARN per executor; defaults to
spark.executor.cores")
.version("3.x-custom")
.intConf
.checkValue(_ > 0, "must be > 0")
.createOptional
{code}
in YarnAllocator.createYarnResourceForResourceProfile:
https://github.com/apache/spark/blob/master/resource-managers/yarn/src/main/scala/org/apache/spark/deploy/yarn/YarnAllocator.scala#L362
{code:java}
val sparkExecutorCores = resourcesWithDefaults.cores.get
val yarnRequestCores = sparkConf.get(YARN_EXECUTOR_REQUEST_CORES)
.map(c => math.min(c, sparkExecutorCores))
.getOrElse(sparkExecutorCores)
val resource = Resource.newInstance(totalMem, yarnRequestCores)
{code}
instead of
{code:java}
val resource = Resource.newInstance(
resourcesWithDefaults.totalMemMiB.toInt, resourcesWithDefaults.cores.get)
{code}
--
This message was sent by Atlassian Jira
(v10.3.26#10030026)
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
