pan3793 commented on code in PR #57332:
URL: https://github.com/apache/spark/pull/57332#discussion_r3627812140
##########
sql/core/src/main/scala/org/apache/spark/sql/execution/BaseScriptTransformationExec.scala:
##########
@@ -84,10 +100,19 @@ trait BaseScriptTransformationExec extends UnaryExecNode {
val path = System.getenv("PATH") + File.pathSeparator +
SparkFiles.getRootDirectory()
builder.environment().put("PATH", path)
- // if OMP_NUM_THREADS is not explicitly set, override it with the value of
"spark.task.cpus"
- if (System.getenv("OMP_NUM_THREADS") == null) {
- builder.environment().put("OMP_NUM_THREADS",
conf.getConfString("spark.task.cpus", "1"))
- }
+ // Derive the script process's OMP_NUM_THREADS from the task's own cpu
amount, which honors
+ // stage-level resource profiles (a script transform can be a child of
mapInPandas/mapInArrow
+ // carrying a different profile), rounded up to an integer >= 1. We check
the config rather
+ // than System.getenv for an explicit user override, because Spark seeds
the executor's
+ // OMP_NUM_THREADS with the global default, so a getenv check would always
suppress the
+ // per-stage value.
+ val ompUserOverride =
+
Option(SparkEnv.get).exists(_.conf.contains("spark.executorEnv.OMP_NUM_THREADS"))
Review Comment:
also cc @HyukjinKwon since you review the `OMP_NUM_THREADS` PRs, it seems
the stage-level profile and user-provided overrides case was totally missed.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: [email protected]
For queries about this service, please contact Infrastructure at:
[email protected]
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
