Re: Issue with joda-time library bundled in hive-exec:4.0.0

[Cheng Pan]

Simhadri, thank you for sharing the information.

Thanks,
Cheng Pan

Re: Issue with joda-time library bundled in hive-exec:4.0.0

[Simhadri G]

Hi Cheng Pan,

There is long running Hive mail thread discussing this here:
https://lists.apache.org/thread/sxcrcf4v9j630tl9domp0bn4m33bdq0s


On Thu, Apr 18, 2024 at 11:15 AM Cheng Pan  wrote:

> Hi Ayush,
>
> > Hive is already in discussion of marking Hive-2.x EOL, so at very best
> we would have one release and immediately after that we will announce it EOL
>
> Does the discussion happen in public? Is there an ETA for the final
> release of branch-2.3?
>
> Thanks,
> Cheng Pan
>
>
> > On Apr 17, 2024, at 18:03, Ayush Saxena  wrote:
> >
> > Thanx Cheng Pan for sharing the pointers, Do you have any list of issues
> or pointers on what are the challenges for Spark to move to a higher Hive
> version? I know upgrading libraries is quite challenging but it is
> inevitable.
> >
> > Hive is already in discussion of marking Hive-2.x EOL, so at very best
> we would have one release and immediately after that we will announce it
> EOL, maintaining a release line is quite an effort for us at Hive & doing
> it because other projects doesn't want to upgrade isn't a convincing reason
> for most of us. The best we can do is or are trying is to address issues
> for Spark whatever we can do as part of Hive code & would definitely need
> help/support from Spark side as well, since the move is from 2.x to 4.x, it
> would be a big change and would offer resistance on both sides.
> >
> > So, it would be great help if any pointers can be shared from Spark side
> for the move, if there is no help/interest from Spark then we can't do
> anything & there is no need for Hive-2.x either in that case :-)
> >
> > -Ayush
> >
> > On Wed, 17 Apr 2024 at 15:00, Cheng Pan  wrote:
> > > … we are exploring ways to get Spark move from 2.3.9 to 4.0, Our
> initial hunch is that it would be quite challenging without a hive-exec
> slim jar …
> >
> > It should be challenging to upgrade Spark’s built-in Hive version.
> Actually, we already did lots of work on branch-2.3 which focuses on CVE
> reduction, for example, allowing Spark to upgrade Guava to modern versions
> to get rid of Guava 14, it was tested with the latest Spark master
> branch[1], maybe we need a release for 2.3.10 now.
> >
> > [1] https://github.com/apache/spark/pull/45372
> >
> > Thanks,
> > Cheng Pan
> >
> >
>
>

Re: Issue with joda-time library bundled in hive-exec:4.0.0

[Cheng Pan]

Hi Ayush,

> Hive is already in discussion of marking Hive-2.x EOL, so at very best we 
> would have one release and immediately after that we will announce it EOL

Does the discussion happen in public? Is there an ETA for the final release of 
branch-2.3?

Thanks,
Cheng Pan


> On Apr 17, 2024, at 18:03, Ayush Saxena  wrote:
> 
> Thanx Cheng Pan for sharing the pointers, Do you have any list of issues or 
> pointers on what are the challenges for Spark to move to a higher Hive 
> version? I know upgrading libraries is quite challenging but it is inevitable.
> 
> Hive is already in discussion of marking Hive-2.x EOL, so at very best we 
> would have one release and immediately after that we will announce it EOL, 
> maintaining a release line is quite an effort for us at Hive & doing it 
> because other projects doesn't want to upgrade isn't a convincing reason for 
> most of us. The best we can do is or are trying is to address issues for 
> Spark whatever we can do as part of Hive code & would definitely need 
> help/support from Spark side as well, since the move is from 2.x to 4.x, it 
> would be a big change and would offer resistance on both sides.
> 
> So, it would be great help if any pointers can be shared from Spark side for 
> the move, if there is no help/interest from Spark then we can't do anything & 
> there is no need for Hive-2.x either in that case :-) 
> 
> -Ayush
> 
> On Wed, 17 Apr 2024 at 15:00, Cheng Pan  wrote:
> > … we are exploring ways to get Spark move from 2.3.9 to 4.0, Our initial 
> > hunch is that it would be quite challenging without a hive-exec slim jar …
> 
> It should be challenging to upgrade Spark’s built-in Hive version. Actually, 
> we already did lots of work on branch-2.3 which focuses on CVE reduction, for 
> example, allowing Spark to upgrade Guava to modern versions to get rid of 
> Guava 14, it was tested with the latest Spark master branch[1], maybe we need 
> a release for 2.3.10 now.
> 
> [1] https://github.com/apache/spark/pull/45372
> 
> Thanks,
> Cheng Pan
> 
> 

Re: Issue with joda-time library bundled in hive-exec:4.0.0

[Ayush Saxena]

Thanx Cheng Pan for sharing the pointers, Do you have any list of issues or
pointers on what are the challenges for Spark to move to a higher Hive
version? I know upgrading libraries is quite challenging but it
is inevitable.

Hive is already in discussion of marking Hive-2.x EOL, so at very best we
would have one release and immediately after that we will announce it EOL,
maintaining a release line is quite an effort for us at Hive & doing it
because other projects doesn't want to upgrade isn't a convincing reason
for most of us. The best we can do is or are trying is to address issues
for Spark whatever we can do as part of Hive code & would definitely need
help/support from Spark side as well, since the move is from 2.x to 4.x, it
would be a big change and would offer resistance on both sides.

So, it would be great help if any pointers can be shared from Spark side
for the move, if there is no help/interest from Spark then we can't do
anything & there is no need for Hive-2.x either in that case :-)

-Ayush

On Wed, 17 Apr 2024 at 15:00, Cheng Pan  wrote:

> > … we are exploring ways to get Spark move from 2.3.9 to 4.0, Our initial
> hunch is that it would be quite challenging without a hive-exec slim jar …
>
> It should be challenging to upgrade Spark’s built-in Hive version.
> Actually, we already did lots of work on branch-2.3 which focuses on CVE
> reduction, for example, allowing Spark to upgrade Guava to modern versions
> to get rid of Guava 14, it was tested with the latest Spark master
> branch[1], maybe we need a release for 2.3.10 now.
>
> [1] https://github.com/apache/spark/pull/45372
>
> Thanks,
> Cheng Pan
>
>
>

Re: Issue with joda-time library bundled in hive-exec:4.0.0

[Cheng Pan]

> … we are exploring ways to get Spark move from 2.3.9 to 4.0, Our initial 
> hunch is that it would be quite challenging without a hive-exec slim jar …

It should be challenging to upgrade Spark’s built-in Hive version. Actually, we 
already did lots of work on branch-2.3 which focuses on CVE reduction, for 
example, allowing Spark to upgrade Guava to modern versions to get rid of Guava 
14, it was tested with the latest Spark master branch[1], maybe we need a 
release for 2.3.10 now.

[1] https://github.com/apache/spark/pull/45372

Thanks,
Cheng Pan

Re: Issue with joda-time library bundled in hive-exec:4.0.0

[Cheng Pan]

There is a JIRA ticket[1] that tracks "upgrading built-in Hive to 3+”

BTW, regarding HMS API used by Spark, the Hive 2.3.9 client is compatible with 
HMS from 2.0 to 4.0, while the upcoming Hive 2.3.10 client should be compatible 
with HMS from 1.2 to 4.0, if we decide to upgrade the built-in Hive, it’s 
better to keep such compatibility.

[1] https://issues.apache.org/jira/browse/SPARK-44114

Thanks,
Cheng Pan


> On Apr 17, 2024, at 18:03, Ayush Saxena  wrote:
> 
> Thanx Cheng Pan for sharing the pointers, Do you have any list of issues or 
> pointers on what are the challenges for Spark to move to a higher Hive 
> version? I know upgrading libraries is quite challenging but it is inevitable.
> 
> Hive is already in discussion of marking Hive-2.x EOL, so at very best we 
> would have one release and immediately after that we will announce it EOL, 
> maintaining a release line is quite an effort for us at Hive & doing it 
> because other projects doesn't want to upgrade isn't a convincing reason for 
> most of us. The best we can do is or are trying is to address issues for 
> Spark whatever we can do as part of Hive code & would definitely need 
> help/support from Spark side as well, since the move is from 2.x to 4.x, it 
> would be a big change and would offer resistance on both sides.
> 
> So, it would be great help if any pointers can be shared from Spark side for 
> the move, if there is no help/interest from Spark then we can't do anything & 
> there is no need for Hive-2.x either in that case :-) 
> 
> -Ayush
> 
> On Wed, 17 Apr 2024 at 15:00, Cheng Pan  wrote:
> > … we are exploring ways to get Spark move from 2.3.9 to 4.0, Our initial 
> > hunch is that it would be quite challenging without a hive-exec slim jar …
> 
> It should be challenging to upgrade Spark’s built-in Hive version. Actually, 
> we already did lots of work on branch-2.3 which focuses on CVE reduction, for 
> example, allowing Spark to upgrade Guava to modern versions to get rid of 
> Guava 14, it was tested with the latest Spark master branch[1], maybe we need 
> a release for 2.3.10 now.
> 
> [1] https://github.com/apache/spark/pull/45372
> 
> Thanks,
> Cheng Pan
> 
> 

Re: Issue with joda-time library bundled in hive-exec:4.0.0

[Mergu Ravi]

I've observed the hive-exec:3.x version, where in the 3.1.0 version, I
found the core classifier library. When we can expect a similar core
classifier library in the 4.1.0 version.


On Wed, Apr 17, 2024 at 4:14 AM Ayush Saxena  wrote:

> Some can't be relocated, it is pretty challenging to relocate them, one I
> remember is calcite, I tried that here:
> https://issues.apache.org/jira/browse/HIVE-23772 & it didn't work, I am
> not sure if things have changed now.
>
> Relocated dependencies shouldn't be problematic I believe, having jars
> shaded but not relocated is problematic in some cases, you can't get rid of
> them via exclusion and then they tend to create classpath conflicts, we are
> exploring ways to get Spark move from 2.3.9 to 4.0, Our initial hunch is
>  that it would be quite challenging without a hive-exec slim jar which
> doesn't have these dependencies or has minimal shading.
>
> -Ayush
>
> On Tue, 16 Apr 2024 at 21:07, László Bodor 
> wrote:
>
>> isn't this as simple to solve as relocating joda (or what causes problems
>> in general)? I cannot recall when and why we removed the core jar, and I'm
>> not against it, but I feel that relocating something that is already shaded
>> in the uber jar is much easier
>>
>> looks like we shade a lot:
>> https://github.com/apache/hive/blob/f06cc2920424817da6405e0efe268ce6cd64a363/ql/pom.xml#L1032
>> but relocate only a few:
>> https://github.com/apache/hive/blob/f06cc2920424817da6405e0efe268ce6cd64a363/ql/pom.xml#L1110
>>
>> we already have a huge uber jar due to including everything, I feel
>> relocation doesn't make it any worse
>> relocation is only dangerous if the relocated classes are present in
>> hive-exec public method signatures (TODO check joda from this aspect)
>>
>> Mergu Ravi  ezt írta (időpont:
>> 2024. ápr. 16., K, 17:21):
>>
>>> When can we expect the core jar for 4.0.0 version.
>>>
>>> On Tue, Apr 16, 2024 at 4:57 PM Denys Kuzmenko 
>>> wrote:
>>>
 Hi Ravi,

 hive-exec is the uber jar, the same thing is done in Hive-3.x. We are
 considering restoring the hive-core jar that only has classes from ql

 On 2024/04/16 06:24:20 Mergu Ravi wrote:
 > Hi Hive Team,
 >
 > I'm currently using the Hive 4.0.0 libraries and facing an issue with
 the
 > hive-exec:4.0.0 library. It seems that the Joda-Time library is
 bundled
 > within it, preventing me from excluding it and using my existing
 Joda-Time
 > library. Could you please provide insight into why this bundling has
 been
 > implemented? Additionally, is there any plan to unbundle the Joda-Time
 > library in future releases?
 >
 > --
 >
 > Thanks & Regards,
 >
 >
 > 
 > Ravi Mergu
 > SMTS-2
 > o:  +91 40 64535355
 > m: +91 9959618687
 > w: www.gaiansolutions.com
 >
 > 
 > 
 > 
 >
 > *Please consider the environment before printing this email.* This
 message
 > contains confidential information and is intended only for the
 individual
 > named. If you are not the named addressee you should not disseminate,
 > distribute or copy this e-mail
 >

>>>
>>>
>>> --
>>>
>>> Thanks & Regards,
>>>
>>>
>>> 
>>> Ravi Mergu
>>> SMTS-2
>>> o:  +91 40 64535355
>>> m: +91 9959618687
>>> w: www.gaiansolutions.com
>>>
>>> 
>>> 
>>> 
>>>
>>> *Please consider the environment before printing this email.* This
>>> message contains confidential information and is intended only for the
>>> individual named. If you are not the named addressee you should not
>>> disseminate, distribute or copy this e-mail
>>>
>>>

-- 

Thanks & Regards,



Ravi Mergu
SMTS-2
o:  +91 40 64535355
m: +91 9959618687
w: www.gaiansolutions.com





*Please consider the environment before printing this email.* This message
contains confidential information and is intended only for the individual
named. If you are not the named addressee you should not disseminate,
distribute or copy this e-mail

Re: Issue with joda-time library bundled in hive-exec:4.0.0

[Ayush Saxena]

Some can't be relocated, it is pretty challenging to relocate them, one I
remember is calcite, I tried that here:
https://issues.apache.org/jira/browse/HIVE-23772 & it didn't work, I am not
sure if things have changed now.

Relocated dependencies shouldn't be problematic I believe, having jars
shaded but not relocated is problematic in some cases, you can't get rid of
them via exclusion and then they tend to create classpath conflicts, we are
exploring ways to get Spark move from 2.3.9 to 4.0, Our initial hunch is
 that it would be quite challenging without a hive-exec slim jar which
doesn't have these dependencies or has minimal shading.

-Ayush

On Tue, 16 Apr 2024 at 21:07, László Bodor 
wrote:

> isn't this as simple to solve as relocating joda (or what causes problems
> in general)? I cannot recall when and why we removed the core jar, and I'm
> not against it, but I feel that relocating something that is already shaded
> in the uber jar is much easier
>
> looks like we shade a lot:
> https://github.com/apache/hive/blob/f06cc2920424817da6405e0efe268ce6cd64a363/ql/pom.xml#L1032
> but relocate only a few:
> https://github.com/apache/hive/blob/f06cc2920424817da6405e0efe268ce6cd64a363/ql/pom.xml#L1110
>
> we already have a huge uber jar due to including everything, I feel
> relocation doesn't make it any worse
> relocation is only dangerous if the relocated classes are present in
> hive-exec public method signatures (TODO check joda from this aspect)
>
> Mergu Ravi  ezt írta (időpont:
> 2024. ápr. 16., K, 17:21):
>
>> When can we expect the core jar for 4.0.0 version.
>>
>> On Tue, Apr 16, 2024 at 4:57 PM Denys Kuzmenko 
>> wrote:
>>
>>> Hi Ravi,
>>>
>>> hive-exec is the uber jar, the same thing is done in Hive-3.x. We are
>>> considering restoring the hive-core jar that only has classes from ql
>>>
>>> On 2024/04/16 06:24:20 Mergu Ravi wrote:
>>> > Hi Hive Team,
>>> >
>>> > I'm currently using the Hive 4.0.0 libraries and facing an issue with
>>> the
>>> > hive-exec:4.0.0 library. It seems that the Joda-Time library is bundled
>>> > within it, preventing me from excluding it and using my existing
>>> Joda-Time
>>> > library. Could you please provide insight into why this bundling has
>>> been
>>> > implemented? Additionally, is there any plan to unbundle the Joda-Time
>>> > library in future releases?
>>> >
>>> > --
>>> >
>>> > Thanks & Regards,
>>> >
>>> >
>>> > 
>>> > Ravi Mergu
>>> > SMTS-2
>>> > o:  +91 40 64535355
>>> > m: +91 9959618687
>>> > w: www.gaiansolutions.com
>>> >
>>> > 
>>> > 
>>> > 
>>> >
>>> > *Please consider the environment before printing this email.* This
>>> message
>>> > contains confidential information and is intended only for the
>>> individual
>>> > named. If you are not the named addressee you should not disseminate,
>>> > distribute or copy this e-mail
>>> >
>>>
>>
>>
>> --
>>
>> Thanks & Regards,
>>
>>
>> 
>> Ravi Mergu
>> SMTS-2
>> o:  +91 40 64535355
>> m: +91 9959618687
>> w: www.gaiansolutions.com
>>
>> 
>> 
>> 
>>
>> *Please consider the environment before printing this email.* This
>> message contains confidential information and is intended only for the
>> individual named. If you are not the named addressee you should not
>> disseminate, distribute or copy this e-mail
>>
>>

Re: Issue with joda-time library bundled in hive-exec:4.0.0

[László Bodor]

isn't this as simple to solve as relocating joda (or what causes problems
in general)? I cannot recall when and why we removed the core jar, and I'm
not against it, but I feel that relocating something that is already shaded
in the uber jar is much easier

looks like we shade a lot:
https://github.com/apache/hive/blob/f06cc2920424817da6405e0efe268ce6cd64a363/ql/pom.xml#L1032
but relocate only a few:
https://github.com/apache/hive/blob/f06cc2920424817da6405e0efe268ce6cd64a363/ql/pom.xml#L1110

we already have a huge uber jar due to including everything, I feel
relocation doesn't make it any worse
relocation is only dangerous if the relocated classes are present in
hive-exec public method signatures (TODO check joda from this aspect)

Mergu Ravi  ezt írta (időpont:
2024. ápr. 16., K, 17:21):

> When can we expect the core jar for 4.0.0 version.
>
> On Tue, Apr 16, 2024 at 4:57 PM Denys Kuzmenko 
> wrote:
>
>> Hi Ravi,
>>
>> hive-exec is the uber jar, the same thing is done in Hive-3.x. We are
>> considering restoring the hive-core jar that only has classes from ql
>>
>> On 2024/04/16 06:24:20 Mergu Ravi wrote:
>> > Hi Hive Team,
>> >
>> > I'm currently using the Hive 4.0.0 libraries and facing an issue with
>> the
>> > hive-exec:4.0.0 library. It seems that the Joda-Time library is bundled
>> > within it, preventing me from excluding it and using my existing
>> Joda-Time
>> > library. Could you please provide insight into why this bundling has
>> been
>> > implemented? Additionally, is there any plan to unbundle the Joda-Time
>> > library in future releases?
>> >
>> > --
>> >
>> > Thanks & Regards,
>> >
>> >
>> > 
>> > Ravi Mergu
>> > SMTS-2
>> > o:  +91 40 64535355
>> > m: +91 9959618687
>> > w: www.gaiansolutions.com
>> >
>> > 
>> > 
>> > 
>> >
>> > *Please consider the environment before printing this email.* This
>> message
>> > contains confidential information and is intended only for the
>> individual
>> > named. If you are not the named addressee you should not disseminate,
>> > distribute or copy this e-mail
>> >
>>
>
>
> --
>
> Thanks & Regards,
>
>
> 
> Ravi Mergu
> SMTS-2
> o:  +91 40 64535355
> m: +91 9959618687
> w: www.gaiansolutions.com
>
> 
> 
> 
>
> *Please consider the environment before printing this email.* This
> message contains confidential information and is intended only for the
> individual named. If you are not the named addressee you should not
> disseminate, distribute or copy this e-mail
>
>

Re: Issue with joda-time library bundled in hive-exec:4.0.0

[Mergu Ravi]

When can we expect the core jar for 4.0.0 version.

On Tue, Apr 16, 2024 at 4:57 PM Denys Kuzmenko  wrote:

> Hi Ravi,
>
> hive-exec is the uber jar, the same thing is done in Hive-3.x. We are
> considering restoring the hive-core jar that only has classes from ql
>
> On 2024/04/16 06:24:20 Mergu Ravi wrote:
> > Hi Hive Team,
> >
> > I'm currently using the Hive 4.0.0 libraries and facing an issue with the
> > hive-exec:4.0.0 library. It seems that the Joda-Time library is bundled
> > within it, preventing me from excluding it and using my existing
> Joda-Time
> > library. Could you please provide insight into why this bundling has been
> > implemented? Additionally, is there any plan to unbundle the Joda-Time
> > library in future releases?
> >
> > --
> >
> > Thanks & Regards,
> >
> >
> > 
> > Ravi Mergu
> > SMTS-2
> > o:  +91 40 64535355
> > m: +91 9959618687
> > w: www.gaiansolutions.com
> >
> > 
> > 
> > 
> >
> > *Please consider the environment before printing this email.* This
> message
> > contains confidential information and is intended only for the individual
> > named. If you are not the named addressee you should not disseminate,
> > distribute or copy this e-mail
> >
>


-- 

Thanks & Regards,



Ravi Mergu
SMTS-2
o:  +91 40 64535355
m: +91 9959618687
w: www.gaiansolutions.com





*Please consider the environment before printing this email.* This message
contains confidential information and is intended only for the individual
named. If you are not the named addressee you should not disseminate,
distribute or copy this e-mail

Re: Issue with joda-time library bundled in hive-exec:4.0.0

[Denys Kuzmenko]

Hi Ravi,

hive-exec is the uber jar, the same thing is done in Hive-3.x. We are 
considering restoring the hive-core jar that only has classes from ql

On 2024/04/16 06:24:20 Mergu Ravi wrote:
> Hi Hive Team,
> 
> I'm currently using the Hive 4.0.0 libraries and facing an issue with the
> hive-exec:4.0.0 library. It seems that the Joda-Time library is bundled
> within it, preventing me from excluding it and using my existing Joda-Time
> library. Could you please provide insight into why this bundling has been
> implemented? Additionally, is there any plan to unbundle the Joda-Time
> library in future releases?
> 
> -- 
> 
> Thanks & Regards,
> 
> 
> 
> Ravi Mergu
> SMTS-2
> o:  +91 40 64535355
> m: +91 9959618687
> w: www.gaiansolutions.com
> 
> 
> 
> 
> 
> *Please consider the environment before printing this email.* This message
> contains confidential information and is intended only for the individual
> named. If you are not the named addressee you should not disseminate,
> distribute or copy this e-mail
>