[ https://issues.apache.org/jira/browse/SPARK-43866?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
Yuming Wang resolved SPARK-43866. --------------------------------- Resolution: Workaround > Partition filter condition should pushed down to metastore query if it is > equivalence Predicate > ----------------------------------------------------------------------------------------------- > > Key: SPARK-43866 > URL: https://issues.apache.org/jira/browse/SPARK-43866 > Project: Spark > Issue Type: Improvement > Components: SQL > Affects Versions: 3.4.0 > Reporter: zzzzming95 > Priority: Major > > Typically, hive partition fields are created as string types. > {code:java} > ``` > CREATE TABLE if not exists test_tb ( > id int > ) > PARTITIONED BY (dt string) > ```{code} > However, cast data conversions are often introduced inadvertently during use. > For example > {code:java} > ``` > select * from test_tb where dt=20230505; > ```{code} > it will prevent the condition `dt=20230505` from being pushed down into the > metastore , because `20230505` is an IntegralType. And resulting in a request > for all partitions. However, in the case of equivalent predicates, partition > field pushdown should be supported. > This can affect execution performance in cases where the data table has very > many partitions. -- This message was sent by Atlassian Jira (v8.20.10#820010) --------------------------------------------------------------------- To unsubscribe, e-mail: issues-unsubscr...@spark.apache.org For additional commands, e-mail: issues-h...@spark.apache.org