as a workaround turn off pruning : spark.sql.hive.metastorePartitionPruning false spark.sql.hive.convertMetastoreParquet false
see https://github.com/awslabs/aws-glue-data-catalog-client-for-apache-hive-metastore/issues/45 On Tue Aug 24, 2021 at 9:18 AM CEST, Gourav Sengupta wrote: > Hi, > > I received a response from AWS, this is an issue with EMR, and they are > working on resolving the issue I believe. > > Thanks and Regards, > Gourav Sengupta > > On Mon, Aug 23, 2021 at 1:35 PM Gourav Sengupta < > gourav.sengupta.develo...@gmail.com> wrote: > > > Hi, > > > > the query still gives the same error if we write "SELECT * FROM table_name > > WHERE data_partition > CURRENT_DATE() - INTERVAL 10 DAYS". > > > > Also the queries work fine in SPARK 3.0.x, or in EMR 6.2.0. > > > > > > Thanks and Regards, > > Gourav Sengupta > > > > On Mon, Aug 23, 2021 at 1:16 PM Sean Owen <sro...@gmail.com> wrote: > > > >> Date handling was tightened up in Spark 3. I think you need to compare to > >> a date literal, not a string literal. > >> > >> On Mon, Aug 23, 2021 at 5:12 AM Gourav Sengupta < > >> gourav.sengupta.develo...@gmail.com> wrote: > >> > >>> Hi, > >>> > >>> while I am running in EMR 6.3.0 (SPARK 3.1.1) a simple query as "SELECT > >>> * FROM <table_name> WHERE <date parition field> > '2021-03-01'" the query > >>> is failing with error: > >>> > >>> --------------------------------------------------------------------------- > >>> pyspark.sql.utils.AnalysisException: > >>> org.apache.hadoop.hive.metastore.api.InvalidObjectException: Unsupported > >>> expression '2021 - 03 - 01' (Service: AWSGlue; Status Code: 400; Error > >>> Code: InvalidInputException; Request ID: > >>> dd3549c2-2eeb-4616-8dc5-5887ba43dd22; Proxy: null) > >>> > >>> --------------------------------------------------------------------------- > >>> > >>> The above query works fine in all previous versions of SPARK. > >>> > >>> Is this the expected behaviour in SPARK 3.1.1? If so can someone please > >>> let me know how to write this query. > >>> > >>> Also if this is the expected behaviour I think that a lot of users will > >>> have to make these changes in their existing code making transition to > >>> SPARK 3.1.1 expensive I think. > >>> > >>> Regards, > >>> Gourav Sengupta > >>> > >> --------------------------------------------------------------------- To unsubscribe e-mail: user-unsubscr...@spark.apache.org
