Hi, while I am running in EMR 6.3.0 (SPARK 3.1.1) a simple query as "SELECT * FROM <table_name> WHERE <date parition field> > '2021-03-01'" the query is failing with error: --------------------------------------------------------------------------- pyspark.sql.utils.AnalysisException: org.apache.hadoop.hive.metastore.api.InvalidObjectException: Unsupported expression '2021 - 03 - 01' (Service: AWSGlue; Status Code: 400; Error Code: InvalidInputException; Request ID: dd3549c2-2eeb-4616-8dc5-5887ba43dd22; Proxy: null) ---------------------------------------------------------------------------
The above query works fine in all previous versions of SPARK. Is this the expected behaviour in SPARK 3.1.1? If so can someone please let me know how to write this query. Also if this is the expected behaviour I think that a lot of users will have to make these changes in their existing code making transition to SPARK 3.1.1 expensive I think. Regards, Gourav Sengupta
