Hi Nicolas,
thanks a ton for your kind response, I will surely try this out.
Regards,
Gourav Sengupta
On Sun, Aug 29, 2021 at 11:01 PM Nicolas Paris
wrote:
> as a workaround turn off pruning :
>
> spark.sql.hive.metastorePartitionPruning false
> spark.sql.hive.convertMetastoreParquet false
>
as a workaround turn off pruning :
spark.sql.hive.metastorePartitionPruning false
spark.sql.hive.convertMetastoreParquet false
see
https://github.com/awslabs/aws-glue-data-catalog-client-for-apache-hive-metastore/issues/45
On Tue Aug 24, 2021 at 9:18 AM CEST, Gourav Sengupta wrote:
> Hi,
>
> I
Hi,
I received a response from AWS, this is an issue with EMR, and they are
working on resolving the issue I believe.
Thanks and Regards,
Gourav Sengupta
On Mon, Aug 23, 2021 at 1:35 PM Gourav Sengupta <
gourav.sengupta.develo...@gmail.com> wrote:
> Hi,
>
> the query still gives the same error
Hi,
the query still gives the same error if we write "SELECT * FROM table_name
WHERE data_partition > CURRENT_DATE() - INTERVAL 10 DAYS".
Also the queries work fine in SPARK 3.0.x, or in EMR 6.2.0.
Thanks and Regards,
Gourav Sengupta
On Mon, Aug 23, 2021 at 1:16 PM Sean Owen wrote:
> Date
Date handling was tightened up in Spark 3. I think you need to compare to a
date literal, not a string literal.
On Mon, Aug 23, 2021 at 5:12 AM Gourav Sengupta <
gourav.sengupta.develo...@gmail.com> wrote:
> Hi,
>
> while I am running in EMR 6.3.0 (SPARK 3.1.1) a simple query as "SELECT *
> FROM
Hi,
while I am running in EMR 6.3.0 (SPARK 3.1.1) a simple query as "SELECT *
FROM WHERE > '2021-03-01'" the query is
failing with error:
---
pyspark.sql.utils.AnalysisException:
