HyukjinKwon opened a new pull request, #58808: URL: https://github.com/apache/spark/pull/58808
### What changes were proposed in this pull request? `DataFrame.rdd` in PySpark calls `Dataset.javaToPython` on the JVM side to build the Python RDD. Unlike the classic Scala `Dataset.rdd`, which was wrapped in `withNewRDDExecutionId` by SPARK-50994, the no-arg `javaToPython` executed the plan without an execution context. This PR wraps it in `withNewRDDExecutionId` so the RDD conversion runs under a tracked SQL execution, reusing the existing helper (generalized to accept the `QueryExecution` to track). ### Why are the changes needed? Without a tracked execution, the thread-local Spark properties carrying the session SQL configs are not propagated to the executors during the RDD conversion. Some operations depend on these being available in the `TaskContext` on the executor side, for example the non-vectorized Parquet reader reads `spark.sql.caseSensitive` from `SQLConf`. When the conf is not propagated, the read resolves columns incorrectly, producing wrong results or failing the read. This is the PySpark counterpart of SPARK-50994. ### Does this PR introduce _any_ user-facing change? Yes. Session SQL configs are now correctly propagated to the executors when a PySpark `DataFrame` is converted to an RDD, so operations that read configs on the executor side (e.g. the non-vectorized Parquet reader honoring `spark.sql.caseSensitive`) behave correctly. ### How was this patch tested? Added a test in `pyspark.sql.tests.test_dataframe` mirroring the SPARK-50994 scenario: with `spark.sql.caseSensitive=true` and the non-vectorized Parquet reader, a `DataFrame` with two same-named but differently-cased columns is written to Parquet, deduplicated, and converted to an RDD. Without the fix the executor-side read does not see the conf and fails; with the fix the RDD yields the correct deduplicated rows. ### Was this patch authored or co-authored using generative AI tooling? Generated-by: Isaac This pull request and its description were written by Isaac. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected] --------------------------------------------------------------------- To unsubscribe, e-mail: [email protected] For additional commands, e-mail: [email protected]
