[ https://issues.apache.org/jira/browse/SPARK-27815?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
Hyukjin Kwon updated SPARK-27815: --------------------------------- Comment: was deleted (was: [~cloud_fan] Hi Wenchen, do you think it is viable solution mentioned below? Create a new V2WriteCommand case class and its Exec named maybe _OverwriteByQueryId_ to replace WriteToDataSourceV2, which accepts a QueryId so that tests can pass. Or should we keep WriteToDataSourceV2?) > do not leak SaveMode to file source v2 > -------------------------------------- > > Key: SPARK-27815 > URL: https://issues.apache.org/jira/browse/SPARK-27815 > Project: Spark > Issue Type: Improvement > Components: SQL > Affects Versions: 3.0.0 > Reporter: Yesheng Ma > Assignee: Yesheng Ma > Priority: Major > Fix For: 3.0.0 > > > The current catalyst optimizer's predicate pushdown is divided into two > separate rules: PushDownPredicate and PushThroughJoin. This is not efficient > for optimizing cascading joins such as TPC-DS q64, where a whole default > batch is re-executed just due to this. We need a more efficient approach to > pushdown predicate as much as possible in a single pass. -- This message was sent by Atlassian JIRA (v7.6.14#76016) --------------------------------------------------------------------- To unsubscribe, e-mail: issues-unsubscr...@spark.apache.org For additional commands, e-mail: issues-h...@spark.apache.org