Hi, the query still gives the same error if we write "SELECT * FROM table_name WHERE data_partition > CURRENT_DATE() - INTERVAL 10 DAYS".
Also the queries work fine in SPARK 3.0.x, or in EMR 6.2.0. Thanks and Regards, Gourav Sengupta On Mon, Aug 23, 2021 at 1:16 PM Sean Owen <sro...@gmail.com> wrote: > Date handling was tightened up in Spark 3. I think you need to compare to > a date literal, not a string literal. > > On Mon, Aug 23, 2021 at 5:12 AM Gourav Sengupta < > gourav.sengupta.develo...@gmail.com> wrote: > >> Hi, >> >> while I am running in EMR 6.3.0 (SPARK 3.1.1) a simple query as "SELECT * >> FROM <table_name> WHERE <date parition field> > '2021-03-01'" the query is >> failing with error: >> >> --------------------------------------------------------------------------- >> pyspark.sql.utils.AnalysisException: >> org.apache.hadoop.hive.metastore.api.InvalidObjectException: Unsupported >> expression '2021 - 03 - 01' (Service: AWSGlue; Status Code: 400; Error >> Code: InvalidInputException; Request ID: >> dd3549c2-2eeb-4616-8dc5-5887ba43dd22; Proxy: null) >> >> --------------------------------------------------------------------------- >> >> The above query works fine in all previous versions of SPARK. >> >> Is this the expected behaviour in SPARK 3.1.1? If so can someone please >> let me know how to write this query. >> >> Also if this is the expected behaviour I think that a lot of users will >> have to make these changes in their existing code making transition to >> SPARK 3.1.1 expensive I think. >> >> Regards, >> Gourav Sengupta >> >
