alexandrefimov opened a new pull request, #56225: URL: https://github.com/apache/spark/pull/56225
### What changes were proposed in this pull request? This PR preserves the original SQL physical plan inside `SQLExecutionRDD` and uses it when building `SparkPlanInfo` for `RDDScanExec`. When a DataFrame is converted to an RDD and then back to a DataFrame, Spark currently shows only `Scan ExistingRDD` in the SQL UI. This change lets `SparkPlanInfo` discover the underlying SQL plan from the `SQLExecutionRDD` lineage and attach it under the `RDDScanExec` node. The RDD lineage traversal is defensive: - it deduplicates RDDs by id; - it stops at each discovered `SQLExecutionRDD`; - it tolerates dependency lookup failures by skipping that branch. ### Why are the changes needed? Without this, the SQL UI loses useful operator visibility and metrics after workflows such as: ```scala val source = spark.range(10).filter($"id" > 3) val recreated = spark.createDataFrame(source.rdd, source.schema) recreated.collect() ``` The UI only exposes `Scan ExistingRDD`, while the internal SQL operators, such as `Filter`, are hidden. This makes query diagnosis harder for DataFrame/RDD/DataFrame workflows. ### Does this PR introduce _any_ user-facing change? Yes. For SQL executions that scan an RDD backed by a previous SQL execution, the SQL UI can now show the internal SQL plan below `Scan ExistingRDD`, including metrics such as `number of output rows`. ### How was this patch tested? Added regression coverage in `SparkPlanInfoSuite` for: - exposing the internal SQL plan under `RDDScanExec`; - preserving SQL metrics in the SQL UI plan graph; - deduplicating shared RDD lineage; - nested `createDataFrame(df.rdd, df.schema)` cases. Ran: ```bash build/sbt "sql/testOnly *SparkPlanInfoSuite -- -z SPARK-47017" ``` Result: ```text Tests: succeeded 1, failed 0 All tests passed. ``` ### Was this patch authored or co-authored using generative AI tooling? Generated-by: OpenAI Codex (GPT-5) -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected] --------------------------------------------------------------------- To unsubscribe, e-mail: [email protected] For additional commands, e-mail: [email protected]
