Github user viirya commented on the issue: https://github.com/apache/spark/pull/14452 @davies Thanks for comment. This is proposed to reuse common subquery results in the query plan. For example, WITH cte as (SELECT * FROM src) SELECT * FROM cte a JOIN cte b In the above query, the subquery cte will be materialized twice. With this PR, we just execute cte once. In the benchmark of TPC-DS query64, we have cut half of the running time with this PR. Simply said, we find the common subqueries in the query plan. For each distinct subquery, we have only one SparkPlan for it and attach it to the subqueries which produce the same results. Once the first subquery among them is executed, the RDD is kept and reused by other subqueries. We would continue to benchmark TPC-DS queries with this PR. Once the results are done, I will post here. Please let me know if I need to explain more in details.
--- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. --- --------------------------------------------------------------------- To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org