pan3793 commented on PR #58229: URL: https://github.com/apache/spark/pull/58229#issuecomment-5615263383
> Add a switch to be able to materialize the View/CTE will give us real performance benefit I'm not sure about the View part, but the CTE part does match the work I'm currently doing. (I use the PostgreSQL 18 behavior as the reference, wasn't aware that DuckDB also supports it and has a different behavior.) > Spark doesn't have a cost model that we could use to estimate the benefits and costs of such materialization. Fully agree with this. In addition, data source statistics are also unreliable/missing in many cases. As a result, many of Spark's built-in optimization rules are very conservative. That is why I am trying to add some HINT-like functionality, allowing users to control the behavior manually. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected] --------------------------------------------------------------------- To unsubscribe, e-mail: [email protected] For additional commands, e-mail: [email protected]
