pan3793 commented on code in PR #57332:
URL: https://github.com/apache/spark/pull/57332#discussion_r3620345888
##########
core/src/main/scala/org/apache/spark/api/python/PythonRunner.scala:
##########
@@ -310,9 +310,11 @@ private[spark] abstract class BasePythonRunner[IN, OUT](
val memoryMb =
Option(context.getLocalProperty(PYSPARK_MEMORY_LOCAL_PROPERTY)).map(_.toLong)
val localdir = env.blockManager.diskBlockManager.localDirs.map(f =>
f.getPath()).mkString(",")
// If OMP_NUM_THREADS is not explicitly set, override it with the number
of task cpus.
- // See SPARK-42613 for details.
+ // See SPARK-42613 for details. `spark.task.cpus` may be fractional, so
round up to an integer
Review Comment:
this is a pre-existing issue reported in SPARK-30299, this change does not
change the situation, I would narrow the PR scope to not touch it
> Note that there are other places in the code that uses executor cores
which could also be wrong in standalone mode. for instance PythonRunner is
using it to split memory.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: [email protected]
For queries about this service, please contact Infrastructure at:
[email protected]
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
