Hi, I tried replacing just this JAR but getting errors.
From: Nagatomi Yasukazu
Sent: Friday, September 8, 2023 9:35 AM
To: Agrawal, Sanket
Cc: Chao Sun ; Yeachan Park ;
user@spark.apache.org
Subject: [EXT] Re: Spark 3.4.1 and Hive 3.1.3
Hi Sanket,
While migrating to Hive 3.1.3 may resolve
Hi Sanket,
While migrating to Hive 3.1.3 may resolve many issues, the link below
suggests that there might still be some vulnerabilities present.
Do you think the specific vulnerability you're concerned about can be
addressed with Hive 3.1.3?
I mean, have you checked if this is in your jar? Are you building an
assembly? Where do you expect elastic classes to be and are they there?
Need some basic debugging here
On Thu, Sep 7, 2023, 8:49 PM Dipayan Dev wrote:
> Hi Sean,
>
> Removed the provided thing, but still the same issue.
>
>
>
Hi Sean,
Removed the provided thing, but still the same issue.
org.elasticsearch
elasticsearch-spark-30_${scala.compat.version}
7.12.1
On Fri, Sep 8, 2023 at 4:41 AM Sean Owen wrote:
> By marking it provided, you are not including this dependency with your
> app. If it is also
Hi Chao,
The reason to migrate to Hive 3.1.3 is to remove a vulnerability from
hive-exec-2.3.9.jar.
Thanks
Sanket
From: Chao Sun
Sent: Thursday, September 7, 2023 10:23 PM
To: Agrawal, Sanket
Cc: Yeachan Park ; user@spark.apache.org
Subject: [EXT] Re: Spark 3.4.1 and Hive 3.1.3
Hi Sanket,
By marking it provided, you are not including this dependency with your
app. If it is also not somehow already provided by your spark cluster (this
is what it means), then yeah this is not anywhere on the class path at
runtime. Remove the provided scope.
On Thu, Sep 7, 2023, 4:09 PM Dipayan Dev
Thanks Mich figured that might be the case, regardless, appreciate the help
:)
On Thu, Sep 7, 2023 at 8:36 PM Mich Talebzadeh
wrote:
> Hi,
>
> As far as I am aware there is no Spark or JVM setting that can make Spark
> assume a different timezone during the initial load from Parquet as Parquet
Hi Sanket,
Spark 3.4.1 currently only works with Hive 2.3.9, and it would require a
lot of work to upgrade the Hive version to 3.x and up.
Normally though, you only need the Hive client in Spark to talk to
HiveMetastore (HMS) for things like table or partition metadata
information. In this case,
Hi,
Can you please elaborate your last response? I don’t have any external
dependencies added, and just updated the Spark version as mentioned below.
Can someone help me with this?
On Fri, 1 Sep 2023 at 5:58 PM, Koert Kuipers wrote:
> could the provided scope be the issue?
>
> On Sun, Aug 27,
++ Dev
On Thu, 7 Sep 2023 at 10:22 PM, Dipayan Dev wrote:
> Hi,
>
> Can you please elaborate your last response? I don’t have any external
> dependencies added, and just updated the Spark version as mentioned below.
>
> Can someone help me with this?
>
> On Fri, 1 Sep 2023 at 5:58 PM, Koert
Hi,
The maven option is good for testing but I wouldn't recommend it running in
production from a security perspective and also depending on your setup you
might be downloading jars at the start of every spark session.
By the way, Spark definitely not require all the jars from Hive, since from
Hi Varun,
With all that said, I forgot one worthy sentence.
"It doesn't really matter what background you come from or your wealth,
everything is possible. Use every negative source in your life as a
positive and you will never ever fail!"
Cheers
Mich Talebzadeh,
Distinguished Technologist,
Hi
I Tried using the maven option and it’s working. But we are not allowed to
download jars at runtime from maven because of some security restrictions.
So, I tried again with downloading hive 3.1.3 and giving the location of jars
and it worked this time. But now in our docker image we have 40
Hi,
As far as I am aware there is no Spark or JVM setting that can make Spark
assume a different timezone during the initial load from Parquet as Parquet
files store timestamps in UTC. The timezone conversion can be done (as I
described before) after the load.
HTH
Mich Talebzadeh,
Distinguished
14 matches
Mail list logo
