Re: please help why there is big difference in partitionFilter in spark2.4.2 and spark3.1.3.

2023-03-05 Thread Yuming Wang
Hi Liyun, This is because of this change: https://issues.apache.org/jira/browse/SPARK-27638. You can set spark.sql.legacy.typeCoercion.datetimeToString.enabled to true to restore the old behavior. On Mon, Mar 6, 2023 at 10:27 AM zhangliyun wrote: > Hi all > > > i have a spark sql , before

please help why there is big difference in partitionFilter in spark2.4.2 and spark3.1.3.

2023-03-05 Thread zhangliyun
Hi all i have a spark sql , before in spark 2.4.2 it runs correctly, when i upgrade to spark 3.1.3, it has some problem. the sql ``` select * from eds_rds.cdh_prpc63cgudba_pp_index_disputecasedetails_hourly where dt >= date_sub('${today}',30); ``` it will load the data of past