Understood, thanks Evyatar.
On Mon, Nov 7, 2022, 17:42 Evy M wrote:
> TBH I'm not sure why there is an issue casting the int to BigInt and I'm
> also not sure about the Jira ticket, I hope someone else can help here.
> Regarding the solution - IMO the more correct solution here would be to
>
Hi Evyatar,
Yes, directly reading the parquet data works. Since we use hive metastore
to obfuscate the underlying datastore details, we want to avoid directly
accessing the files.
I guess then the only option is to either change the data or change the
schema of the hive metastore as you suggested
TBH I'm not sure why there is an issue casting the int to BigInt and I'm
also not sure about the Jira ticket, I hope someone else can help here.
Regarding the solution - IMO the more correct solution here would be to
modify the Hive table to use INT since it seems that there is no need to
use
Hi Naresh,
Have you tried any of the following in order to resolve your issue:
1. Reading the Parquet files (directly, not via Hive [i.e,
spark.read.parquet()]), casting to LongType and creating the hive
table based on this dataframe? Hive's BigInt and Spark's Long should have
the
Hi all,
I am trying to read data (using spark sql) via a hive metastore which has a
column of type bigint. Underlying parquet data has int as the datatype for
the same column. I am getting the following error while trying to read the
data using spark sql -
java.lang.ClassCastException:
