sunchao commented on code in PR #57332:
URL: https://github.com/apache/spark/pull/57332#discussion_r3619159648
##########
core/src/main/scala/org/apache/spark/api/python/PythonRunner.scala:
##########
@@ -310,9 +310,11 @@ private[spark] abstract class BasePythonRunner[IN, OUT](
val memoryMb =
Option(context.getLocalProperty(PYSPARK_MEMORY_LOCAL_PROPERTY)).map(_.toLong)
val localdir = env.blockManager.diskBlockManager.localDirs.map(f =>
f.getPath()).mkString(",")
// If OMP_NUM_THREADS is not explicitly set, override it with the number
of task cpus.
- // See SPARK-42613 for details.
+ // See SPARK-42613 for details. `spark.task.cpus` may be fractional, so
round up to an integer
Review Comment:
[P2] This still uses the configured rather than actual executor capacity
The active task CPU amount is now correct, but `execCores` still falls back
to the config default of 1 when `spark.executor.cores` is unset. With
`local[4]`, `spark.task.cpus=0.5`, and 2 GiB configured, the helper computes
two slots and assigns 1 GiB to each worker, while `LocalSchedulerBackend` can
launch eight workers; their aggregate limits can therefore reach 8 GiB.
Standalone executors without an explicit core count have the same mismatch.
Please propagate the actual registered executor capacity; the new
single-partition integration test currently masks this by observing only one
worker.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: [email protected]
For queries about this service, please contact Infrastructure at:
[email protected]
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
