pan3793 commented on code in PR #57332:
URL: https://github.com/apache/spark/pull/57332#discussion_r3613588625
##########
core/src/main/scala/org/apache/spark/api/python/PythonRunner.scala:
##########
@@ -310,9 +310,11 @@ private[spark] abstract class BasePythonRunner[IN, OUT](
val memoryMb =
Option(context.getLocalProperty(PYSPARK_MEMORY_LOCAL_PROPERTY)).map(_.toLong)
val localdir = env.blockManager.diskBlockManager.localDirs.map(f =>
f.getPath()).mkString(",")
// If OMP_NUM_THREADS is not explicitly set, override it with the number
of task cpus.
- // See SPARK-42613 for details.
+ // See SPARK-42613 for details. `spark.task.cpus` may be fractional, so
round up to an integer
Review Comment:
good catch, updated to use the active resource profile instead of the global
`spark.task.cpus`.
BTW, this seems to be a pre-existing issue: with this change, when
`spark.task.cpus` > 1, each worker can receive more memory now.
##########
core/src/main/scala/org/apache/spark/api/python/PythonRunner.scala:
##########
@@ -310,9 +310,11 @@ private[spark] abstract class BasePythonRunner[IN, OUT](
val memoryMb =
Option(context.getLocalProperty(PYSPARK_MEMORY_LOCAL_PROPERTY)).map(_.toLong)
val localdir = env.blockManager.diskBlockManager.localDirs.map(f =>
f.getPath()).mkString(",")
// If OMP_NUM_THREADS is not explicitly set, override it with the number
of task cpus.
- // See SPARK-42613 for details.
+ // See SPARK-42613 for details. `spark.task.cpus` may be fractional, so
round up to an integer
+ // number of threads; it is validated to be > 0, so the ceiling is always
>= 1.
if (conf.getOption("spark.executorEnv.OMP_NUM_THREADS").isEmpty) {
- envVars.put("OMP_NUM_THREADS", conf.get("spark.task.cpus", "1"))
+ envVars.put("OMP_NUM_THREADS",
+ conf.get(CPUS_PER_TASK).setScale(0,
BigDecimal.RoundingMode.CEILING).intValue.toString)
Review Comment:
updated, basically the same as above
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: [email protected]
For queries about this service, please contact Infrastructure at:
[email protected]
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
