Sergiu Pocol created SPARK-55003:
------------------------------------
Summary: ResolveTranspose triggers Execution without SQLConf
Propagation
Key: SPARK-55003
URL: https://issues.apache.org/jira/browse/SPARK-55003
Project: Spark
Issue Type: Bug
Components: Spark Core
Affects Versions: 4.0.1, 4.0.0
Reporter: Sergiu Pocol
[Here|[https://github.com/apache/spark/blob/c7c1021693a01a476b234773dec2d30ca134d250/sql/core/src/main/scala/org/apache/spark/sql/catalyst/analysis/ResolveTranspose.scala#L169],]
this analyzer rule will trigger a SQL execution via the `executeCollect` API.
This API bypasses `withNewExecutionId` and in turn will miss out on session SQL
conf propagation to executors (see
[here|[https://github.com/apache/spark/blob/master/sql/core/src/main/scala/org/apache/spark/sql/execution/SQLExecution.scala#L147]).]
As a consequence, there can be a mismatch between SQL configs on the driver
and executors resulting in undefined/undesirable behaviour.
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
