Re: Calcite jars with shaded guava

[Gian Merlino]

Still working on this and gradually making progress. So far, I ran into these 
three things along the way:

https://issues.apache.org/jira/browse/CALCITE-5477
https://issues.apache.org/jira/browse/CALCITE-5478
https://issues.apache.org/jira/browse/CALCITE-5479

I've applied the linked PRs locally and continue to make progress integrating 
this new version into Druid.

On 2023/01/04 06:15:00 Gian Merlino wrote:
> Thank you! I've built a copy of main and have started working on integrating 
> it into Druid. I'll let the group know if anything interesting comes up. 
> There is also a good chunk of work to do here, as it's been a while since we 
> updated our version of Calcite.
> 
> On 2022/12/11 18:43:08 Julian Hyde wrote:
> > The bug is now fixed. (I started from your PR, but I had to do quite a lot 
> > of extra work. Despite what I said earlier, there were several places where 
> > Calcite was using APIs added between Guava 16 and 19.)
> > 
> > > On Dec 9, 2022, at 1:30 PM, Gian Merlino  wrote:
> > > 
> > > Hmm. In that case I'm not sure why we came to the belief that newer 
> > > Calcites could not be used with Guava 16.0.1. Maybe it was from reading 
> > > the Calcite release notes rather than actually testing it. It would be 
> > > great if this just works.
> > > 
> > > I raised this PR: https://github.com/apache/calcite/pull/2999 and this 
> > > JIRA: https://issues.apache.org/jira/browse/CALCITE-5428.
> > > 
> > >> On 2022/12/09 20:54:46 Julian Hyde wrote:
> > >> I ran with '-Dguava.version=16.0.1' and it seems to just work.
> > >> 
> > >> If you want this to be the new minimum, please create a Jira case and
> > >> PR. Changes will be needed to main.yml (GitHub CI), comments in
> > >> gradle.properties, and history.md (release notes for the upcoming
> > >> release).
> > >> 
> > >> Julian
> > >> 
> > >>> On Fri, Dec 9, 2022 at 12:30 PM Gian Merlino  wrote:
> > >>> 
> > >>> If Calcite ends up dropping the minimum to 16, that would certainly be 
> > >>> helpful, and appreciated, since it would mean Druid can upgrade Calcite 
> > >>> without needing to also tackle the Hadoop 2 problem. Please let us know 
> > >>> if it seems like that would be in the cards!
> > >>> 
> > >>> In case Calcite sticks on Guava 19+, I asked on the Druid issue 
> > >>> (https://github.com/apache/druid/issues/13532) to see if anyone's tried 
> > >>> Hadoop 2 with Guava 19. I'm not sure if we've tried that specific 
> > >>> combination; if not, maybe that would work for us. If that doesn't work 
> > >>> out, I think our best remaining idea would be to build our own version 
> > >>> of Calcite that uses a relocated Guava.
> > >>> 
> > >>> On 2022/12/08 20:49:17 Julian Hyde wrote:
> >  If it helps, it wouldn’t be impossible for Calcite to move its minimum 
> >  Guava version down to 16. (If we rely on APIs introduced after 16 we 
> >  can create a facade for those APIs.)
> >  
> >  I am frustrated that the Hadoop folks get locked into a particular 
> >  version of Guava (and JDK… don’t get me started on that one) and in 
> >  Calcite I would like to do the opposite.
> >  
> >  Julian
> >  
> >  
> > > On Dec 8, 2022, at 9:24 AM, Gian Merlino  wrote:
> > > 
> > > It pains me to admit that the reason Druid still uses Guava 16 is 
> > > that we still support Hadoop 2, which relies on APIs that have been 
> > > removed in newer versions of Guava. From time to time it is proposed 
> > > that the project drops support for Hadoop 2 in order to be free to 
> > > move up Guava versions, and also achieve various other benefits. 
> > > There are generally concerns from people that are stuck with Hadoop 2 
> > > for various reasons. The most recent thread is here: 
> > > https://lists.apache.org/thread/1j5w9dmt1gp8hx31tvrmyomcko4mlp03. I 
> > > hope at some point we decide to bite the bullet and do it, but the 
> > > community has not yet come to that decision.
> > > 
> > > In the meantime, we're stuck on an older Calcite version, and really 
> > > want to update it, so we're looking into options. There's a GitHub 
> > > issue here with our thoughts: 
> > > https://github.com/apache/druid/issues/13532
> > > 
> > > On 2022/03/30 17:55:44 Julian Hyde wrote:
> > >> Wow, I had no idea that anyone was on a version of Guava that old.
> > >> 
> > >> I checked our history of Guava versions:
> > >> * In https://issues.apache.org/jira/browse/CALCITE-1590 
> > >>  (Calcite 1.12) 
> > >> we changed the default from 19 to 20, but still supported Guava 
> > >> 14.0.1 and higher.
> > >> * In https://issues.apache.org/jira/browse/CALCITE-1715 
> > >>  (also Calcite 
> > >> 1.12) we changed the default version from 20 to 19
> > >> * In https://issues.apache.org/jira/browse/CALCITE-2191 
> > 

Re: Calcite jars with shaded guava

[Gian Merlino]

Thank you! I've built a copy of main and have started working on integrating it 
into Druid. I'll let the group know if anything interesting comes up. There is 
also a good chunk of work to do here, as it's been a while since we updated our 
version of Calcite.

On 2022/12/11 18:43:08 Julian Hyde wrote:
> The bug is now fixed. (I started from your PR, but I had to do quite a lot of 
> extra work. Despite what I said earlier, there were several places where 
> Calcite was using APIs added between Guava 16 and 19.)
> 
> > On Dec 9, 2022, at 1:30 PM, Gian Merlino  wrote:
> > 
> > Hmm. In that case I'm not sure why we came to the belief that newer 
> > Calcites could not be used with Guava 16.0.1. Maybe it was from reading the 
> > Calcite release notes rather than actually testing it. It would be great if 
> > this just works.
> > 
> > I raised this PR: https://github.com/apache/calcite/pull/2999 and this 
> > JIRA: https://issues.apache.org/jira/browse/CALCITE-5428.
> > 
> >> On 2022/12/09 20:54:46 Julian Hyde wrote:
> >> I ran with '-Dguava.version=16.0.1' and it seems to just work.
> >> 
> >> If you want this to be the new minimum, please create a Jira case and
> >> PR. Changes will be needed to main.yml (GitHub CI), comments in
> >> gradle.properties, and history.md (release notes for the upcoming
> >> release).
> >> 
> >> Julian
> >> 
> >>> On Fri, Dec 9, 2022 at 12:30 PM Gian Merlino  wrote:
> >>> 
> >>> If Calcite ends up dropping the minimum to 16, that would certainly be 
> >>> helpful, and appreciated, since it would mean Druid can upgrade Calcite 
> >>> without needing to also tackle the Hadoop 2 problem. Please let us know 
> >>> if it seems like that would be in the cards!
> >>> 
> >>> In case Calcite sticks on Guava 19+, I asked on the Druid issue 
> >>> (https://github.com/apache/druid/issues/13532) to see if anyone's tried 
> >>> Hadoop 2 with Guava 19. I'm not sure if we've tried that specific 
> >>> combination; if not, maybe that would work for us. If that doesn't work 
> >>> out, I think our best remaining idea would be to build our own version of 
> >>> Calcite that uses a relocated Guava.
> >>> 
> >>> On 2022/12/08 20:49:17 Julian Hyde wrote:
>  If it helps, it wouldn’t be impossible for Calcite to move its minimum 
>  Guava version down to 16. (If we rely on APIs introduced after 16 we can 
>  create a facade for those APIs.)
>  
>  I am frustrated that the Hadoop folks get locked into a particular 
>  version of Guava (and JDK… don’t get me started on that one) and in 
>  Calcite I would like to do the opposite.
>  
>  Julian
>  
>  
> > On Dec 8, 2022, at 9:24 AM, Gian Merlino  wrote:
> > 
> > It pains me to admit that the reason Druid still uses Guava 16 is that 
> > we still support Hadoop 2, which relies on APIs that have been removed 
> > in newer versions of Guava. From time to time it is proposed that the 
> > project drops support for Hadoop 2 in order to be free to move up Guava 
> > versions, and also achieve various other benefits. There are generally 
> > concerns from people that are stuck with Hadoop 2 for various reasons. 
> > The most recent thread is here: 
> > https://lists.apache.org/thread/1j5w9dmt1gp8hx31tvrmyomcko4mlp03. I 
> > hope at some point we decide to bite the bullet and do it, but the 
> > community has not yet come to that decision.
> > 
> > In the meantime, we're stuck on an older Calcite version, and really 
> > want to update it, so we're looking into options. There's a GitHub 
> > issue here with our thoughts: 
> > https://github.com/apache/druid/issues/13532
> > 
> > On 2022/03/30 17:55:44 Julian Hyde wrote:
> >> Wow, I had no idea that anyone was on a version of Guava that old.
> >> 
> >> I checked our history of Guava versions:
> >> * In https://issues.apache.org/jira/browse/CALCITE-1590 
> >>  (Calcite 1.12) we 
> >> changed the default from 19 to 20, but still supported Guava 14.0.1 
> >> and higher.
> >> * In https://issues.apache.org/jira/browse/CALCITE-1715 
> >>  (also Calcite 
> >> 1.12) we changed the default version from 20 to 19
> >> * In https://issues.apache.org/jira/browse/CALCITE-2191 
> >>  (Calcite 1.16) we 
> >> moved the minimum from 14.0.1 to 19 and cited a Druid email thread [1] 
> >> about upgrading Druid’s version of Guava
> >> 
> >> Foolish of me to assume that 4 years later Druid would have done 
> >> something.
> >> 
> >> Julian
> >> 
> >> [1] https://groups.google.com/g/druid-development/c/Dw2Qu1CWbuQ
> >> 
> >>> On Mar 30, 2022, at 10:22 AM, Abhishek Agarwal 
> >>>  wrote:
> >>> 
> >>> Thank you, Alessandro and Stamatis, for the info. Currently, 

Re: Calcite jars with shaded guava

[Julian Hyde]

The bug is now fixed. (I started from your PR, but I had to do quite a lot of 
extra work. Despite what I said earlier, there were several places where 
Calcite was using APIs added between Guava 16 and 19.)

> On Dec 9, 2022, at 1:30 PM, Gian Merlino  wrote:
> 
> Hmm. In that case I'm not sure why we came to the belief that newer Calcites 
> could not be used with Guava 16.0.1. Maybe it was from reading the Calcite 
> release notes rather than actually testing it. It would be great if this just 
> works.
> 
> I raised this PR: https://github.com/apache/calcite/pull/2999 and this JIRA: 
> https://issues.apache.org/jira/browse/CALCITE-5428.
> 
>> On 2022/12/09 20:54:46 Julian Hyde wrote:
>> I ran with '-Dguava.version=16.0.1' and it seems to just work.
>> 
>> If you want this to be the new minimum, please create a Jira case and
>> PR. Changes will be needed to main.yml (GitHub CI), comments in
>> gradle.properties, and history.md (release notes for the upcoming
>> release).
>> 
>> Julian
>> 
>>> On Fri, Dec 9, 2022 at 12:30 PM Gian Merlino  wrote:
>>> 
>>> If Calcite ends up dropping the minimum to 16, that would certainly be 
>>> helpful, and appreciated, since it would mean Druid can upgrade Calcite 
>>> without needing to also tackle the Hadoop 2 problem. Please let us know if 
>>> it seems like that would be in the cards!
>>> 
>>> In case Calcite sticks on Guava 19+, I asked on the Druid issue 
>>> (https://github.com/apache/druid/issues/13532) to see if anyone's tried 
>>> Hadoop 2 with Guava 19. I'm not sure if we've tried that specific 
>>> combination; if not, maybe that would work for us. If that doesn't work 
>>> out, I think our best remaining idea would be to build our own version of 
>>> Calcite that uses a relocated Guava.
>>> 
>>> On 2022/12/08 20:49:17 Julian Hyde wrote:
 If it helps, it wouldn’t be impossible for Calcite to move its minimum 
 Guava version down to 16. (If we rely on APIs introduced after 16 we can 
 create a facade for those APIs.)
 
 I am frustrated that the Hadoop folks get locked into a particular version 
 of Guava (and JDK… don’t get me started on that one) and in Calcite I 
 would like to do the opposite.
 
 Julian
 
 
> On Dec 8, 2022, at 9:24 AM, Gian Merlino  wrote:
> 
> It pains me to admit that the reason Druid still uses Guava 16 is that we 
> still support Hadoop 2, which relies on APIs that have been removed in 
> newer versions of Guava. From time to time it is proposed that the 
> project drops support for Hadoop 2 in order to be free to move up Guava 
> versions, and also achieve various other benefits. There are generally 
> concerns from people that are stuck with Hadoop 2 for various reasons. 
> The most recent thread is here: 
> https://lists.apache.org/thread/1j5w9dmt1gp8hx31tvrmyomcko4mlp03. I hope 
> at some point we decide to bite the bullet and do it, but the community 
> has not yet come to that decision.
> 
> In the meantime, we're stuck on an older Calcite version, and really want 
> to update it, so we're looking into options. There's a GitHub issue here 
> with our thoughts: https://github.com/apache/druid/issues/13532
> 
> On 2022/03/30 17:55:44 Julian Hyde wrote:
>> Wow, I had no idea that anyone was on a version of Guava that old.
>> 
>> I checked our history of Guava versions:
>> * In https://issues.apache.org/jira/browse/CALCITE-1590 
>>  (Calcite 1.12) we 
>> changed the default from 19 to 20, but still supported Guava 14.0.1 and 
>> higher.
>> * In https://issues.apache.org/jira/browse/CALCITE-1715 
>>  (also Calcite 1.12) 
>> we changed the default version from 20 to 19
>> * In https://issues.apache.org/jira/browse/CALCITE-2191 
>>  (Calcite 1.16) we 
>> moved the minimum from 14.0.1 to 19 and cited a Druid email thread [1] 
>> about upgrading Druid’s version of Guava
>> 
>> Foolish of me to assume that 4 years later Druid would have done 
>> something.
>> 
>> Julian
>> 
>> [1] https://groups.google.com/g/druid-development/c/Dw2Qu1CWbuQ
>> 
>>> On Mar 30, 2022, at 10:22 AM, Abhishek Agarwal 
>>>  wrote:
>>> 
>>> Thank you, Alessandro and Stamatis, for the info. Currently, the guava
>>> version being used in druid is 16. So simple exclusion is unlikely to 
>>> work.
>>> I will still give it a try. If it doesn't, we will shade the jars on our
>>> end.
>>> 
>>> On Tue, Mar 29, 2022 at 3:38 PM Alessandro Solimando <
>>> alessandro.solima...@gmail.com> wrote:
>>> 
 Hello Abhishek,
 maybe I am re-stating what Stamatis suggested without realising, but 
 since
 Calcite works well with a broad range of guava 

Re: Calcite jars with shaded guava

[Gian Merlino]

Hmm. In that case I'm not sure why we came to the belief that newer Calcites 
could not be used with Guava 16.0.1. Maybe it was from reading the Calcite 
release notes rather than actually testing it. It would be great if this just 
works.

I raised this PR: https://github.com/apache/calcite/pull/2999 and this JIRA: 
https://issues.apache.org/jira/browse/CALCITE-5428.

On 2022/12/09 20:54:46 Julian Hyde wrote:
> I ran with '-Dguava.version=16.0.1' and it seems to just work.
> 
> If you want this to be the new minimum, please create a Jira case and
> PR. Changes will be needed to main.yml (GitHub CI), comments in
> gradle.properties, and history.md (release notes for the upcoming
> release).
> 
> Julian
> 
> On Fri, Dec 9, 2022 at 12:30 PM Gian Merlino  wrote:
> >
> > If Calcite ends up dropping the minimum to 16, that would certainly be 
> > helpful, and appreciated, since it would mean Druid can upgrade Calcite 
> > without needing to also tackle the Hadoop 2 problem. Please let us know if 
> > it seems like that would be in the cards!
> >
> > In case Calcite sticks on Guava 19+, I asked on the Druid issue 
> > (https://github.com/apache/druid/issues/13532) to see if anyone's tried 
> > Hadoop 2 with Guava 19. I'm not sure if we've tried that specific 
> > combination; if not, maybe that would work for us. If that doesn't work 
> > out, I think our best remaining idea would be to build our own version of 
> > Calcite that uses a relocated Guava.
> >
> > On 2022/12/08 20:49:17 Julian Hyde wrote:
> > > If it helps, it wouldn’t be impossible for Calcite to move its minimum 
> > > Guava version down to 16. (If we rely on APIs introduced after 16 we can 
> > > create a facade for those APIs.)
> > >
> > > I am frustrated that the Hadoop folks get locked into a particular 
> > > version of Guava (and JDK… don’t get me started on that one) and in 
> > > Calcite I would like to do the opposite.
> > >
> > > Julian
> > >
> > >
> > > > On Dec 8, 2022, at 9:24 AM, Gian Merlino  wrote:
> > > >
> > > > It pains me to admit that the reason Druid still uses Guava 16 is that 
> > > > we still support Hadoop 2, which relies on APIs that have been removed 
> > > > in newer versions of Guava. From time to time it is proposed that the 
> > > > project drops support for Hadoop 2 in order to be free to move up Guava 
> > > > versions, and also achieve various other benefits. There are generally 
> > > > concerns from people that are stuck with Hadoop 2 for various reasons. 
> > > > The most recent thread is here: 
> > > > https://lists.apache.org/thread/1j5w9dmt1gp8hx31tvrmyomcko4mlp03. I 
> > > > hope at some point we decide to bite the bullet and do it, but the 
> > > > community has not yet come to that decision.
> > > >
> > > > In the meantime, we're stuck on an older Calcite version, and really 
> > > > want to update it, so we're looking into options. There's a GitHub 
> > > > issue here with our thoughts: 
> > > > https://github.com/apache/druid/issues/13532
> > > >
> > > > On 2022/03/30 17:55:44 Julian Hyde wrote:
> > > >> Wow, I had no idea that anyone was on a version of Guava that old.
> > > >>
> > > >> I checked our history of Guava versions:
> > > >> * In https://issues.apache.org/jira/browse/CALCITE-1590 
> > > >>  (Calcite 1.12) we 
> > > >> changed the default from 19 to 20, but still supported Guava 14.0.1 
> > > >> and higher.
> > > >> * In https://issues.apache.org/jira/browse/CALCITE-1715 
> > > >>  (also Calcite 
> > > >> 1.12) we changed the default version from 20 to 19
> > > >> * In https://issues.apache.org/jira/browse/CALCITE-2191 
> > > >>  (Calcite 1.16) we 
> > > >> moved the minimum from 14.0.1 to 19 and cited a Druid email thread [1] 
> > > >> about upgrading Druid’s version of Guava
> > > >>
> > > >> Foolish of me to assume that 4 years later Druid would have done 
> > > >> something.
> > > >>
> > > >> Julian
> > > >>
> > > >> [1] https://groups.google.com/g/druid-development/c/Dw2Qu1CWbuQ
> > > >>
> > > >>> On Mar 30, 2022, at 10:22 AM, Abhishek Agarwal 
> > > >>>  wrote:
> > > >>>
> > > >>> Thank you, Alessandro and Stamatis, for the info. Currently, the guava
> > > >>> version being used in druid is 16. So simple exclusion is unlikely to 
> > > >>> work.
> > > >>> I will still give it a try. If it doesn't, we will shade the jars on 
> > > >>> our
> > > >>> end.
> > > >>>
> > > >>> On Tue, Mar 29, 2022 at 3:38 PM Alessandro Solimando <
> > > >>> alessandro.solima...@gmail.com> wrote:
> > > >>>
> > >  Hello Abhishek,
> > >  maybe I am re-stating what Stamatis suggested without realising, but 
> > >  since
> > >  Calcite works well with a broad range of guava versions, can't you 
> > >  simply
> > >  exclude guava when you include it in druid?
> > > 
> > >  At that point if the guava version of druid is in the 

Re: Calcite jars with shaded guava

[Julian Hyde]

I ran with '-Dguava.version=16.0.1' and it seems to just work.

If you want this to be the new minimum, please create a Jira case and
PR. Changes will be needed to main.yml (GitHub CI), comments in
gradle.properties, and history.md (release notes for the upcoming
release).

Julian

On Fri, Dec 9, 2022 at 12:30 PM Gian Merlino  wrote:
>
> If Calcite ends up dropping the minimum to 16, that would certainly be 
> helpful, and appreciated, since it would mean Druid can upgrade Calcite 
> without needing to also tackle the Hadoop 2 problem. Please let us know if it 
> seems like that would be in the cards!
>
> In case Calcite sticks on Guava 19+, I asked on the Druid issue 
> (https://github.com/apache/druid/issues/13532) to see if anyone's tried 
> Hadoop 2 with Guava 19. I'm not sure if we've tried that specific 
> combination; if not, maybe that would work for us. If that doesn't work out, 
> I think our best remaining idea would be to build our own version of Calcite 
> that uses a relocated Guava.
>
> On 2022/12/08 20:49:17 Julian Hyde wrote:
> > If it helps, it wouldn’t be impossible for Calcite to move its minimum 
> > Guava version down to 16. (If we rely on APIs introduced after 16 we can 
> > create a facade for those APIs.)
> >
> > I am frustrated that the Hadoop folks get locked into a particular version 
> > of Guava (and JDK… don’t get me started on that one) and in Calcite I would 
> > like to do the opposite.
> >
> > Julian
> >
> >
> > > On Dec 8, 2022, at 9:24 AM, Gian Merlino  wrote:
> > >
> > > It pains me to admit that the reason Druid still uses Guava 16 is that we 
> > > still support Hadoop 2, which relies on APIs that have been removed in 
> > > newer versions of Guava. From time to time it is proposed that the 
> > > project drops support for Hadoop 2 in order to be free to move up Guava 
> > > versions, and also achieve various other benefits. There are generally 
> > > concerns from people that are stuck with Hadoop 2 for various reasons. 
> > > The most recent thread is here: 
> > > https://lists.apache.org/thread/1j5w9dmt1gp8hx31tvrmyomcko4mlp03. I hope 
> > > at some point we decide to bite the bullet and do it, but the community 
> > > has not yet come to that decision.
> > >
> > > In the meantime, we're stuck on an older Calcite version, and really want 
> > > to update it, so we're looking into options. There's a GitHub issue here 
> > > with our thoughts: https://github.com/apache/druid/issues/13532
> > >
> > > On 2022/03/30 17:55:44 Julian Hyde wrote:
> > >> Wow, I had no idea that anyone was on a version of Guava that old.
> > >>
> > >> I checked our history of Guava versions:
> > >> * In https://issues.apache.org/jira/browse/CALCITE-1590 
> > >>  (Calcite 1.12) we 
> > >> changed the default from 19 to 20, but still supported Guava 14.0.1 and 
> > >> higher.
> > >> * In https://issues.apache.org/jira/browse/CALCITE-1715 
> > >>  (also Calcite 1.12) 
> > >> we changed the default version from 20 to 19
> > >> * In https://issues.apache.org/jira/browse/CALCITE-2191 
> > >>  (Calcite 1.16) we 
> > >> moved the minimum from 14.0.1 to 19 and cited a Druid email thread [1] 
> > >> about upgrading Druid’s version of Guava
> > >>
> > >> Foolish of me to assume that 4 years later Druid would have done 
> > >> something.
> > >>
> > >> Julian
> > >>
> > >> [1] https://groups.google.com/g/druid-development/c/Dw2Qu1CWbuQ
> > >>
> > >>> On Mar 30, 2022, at 10:22 AM, Abhishek Agarwal 
> > >>>  wrote:
> > >>>
> > >>> Thank you, Alessandro and Stamatis, for the info. Currently, the guava
> > >>> version being used in druid is 16. So simple exclusion is unlikely to 
> > >>> work.
> > >>> I will still give it a try. If it doesn't, we will shade the jars on our
> > >>> end.
> > >>>
> > >>> On Tue, Mar 29, 2022 at 3:38 PM Alessandro Solimando <
> > >>> alessandro.solima...@gmail.com> wrote:
> > >>>
> >  Hello Abhishek,
> >  maybe I am re-stating what Stamatis suggested without realising, but 
> >  since
> >  Calcite works well with a broad range of guava versions, can't you 
> >  simply
> >  exclude guava when you include it in druid?
> > 
> >  At that point if the guava version of druid is in the list of those
> >  supported by Calcite (>= 19 IIRC) all should work fine.
> > 
> >  I feel like Stamatis concerning shading.
> > 
> >  Best regards,
> >  Alessandro
> > 
> >  On Tue, 29 Mar 2022 at 11:18, Stamatis Zampetakis 
> >  wrote:
> > 
> > > Hi Abhishek,
> > >
> > > Calcite supports multiple versions of Guava so in principle it should
> >  work
> > > without problems with older versions.
> > > Instead of shading you could possibly select explicitly the version
> > > that you want to use via maven/gradle.
> > >
> > > I share your pain 

Re: Calcite jars with shaded guava

[Gian Merlino]

If Calcite ends up dropping the minimum to 16, that would certainly be helpful, 
and appreciated, since it would mean Druid can upgrade Calcite without needing 
to also tackle the Hadoop 2 problem. Please let us know if it seems like that 
would be in the cards!

In case Calcite sticks on Guava 19+, I asked on the Druid issue 
(https://github.com/apache/druid/issues/13532) to see if anyone's tried Hadoop 
2 with Guava 19. I'm not sure if we've tried that specific combination; if not, 
maybe that would work for us. If that doesn't work out, I think our best 
remaining idea would be to build our own version of Calcite that uses a 
relocated Guava.

On 2022/12/08 20:49:17 Julian Hyde wrote:
> If it helps, it wouldn’t be impossible for Calcite to move its minimum Guava 
> version down to 16. (If we rely on APIs introduced after 16 we can create a 
> facade for those APIs.)
> 
> I am frustrated that the Hadoop folks get locked into a particular version of 
> Guava (and JDK… don’t get me started on that one) and in Calcite I would like 
> to do the opposite.
> 
> Julian
> 
> 
> > On Dec 8, 2022, at 9:24 AM, Gian Merlino  wrote:
> > 
> > It pains me to admit that the reason Druid still uses Guava 16 is that we 
> > still support Hadoop 2, which relies on APIs that have been removed in 
> > newer versions of Guava. From time to time it is proposed that the project 
> > drops support for Hadoop 2 in order to be free to move up Guava versions, 
> > and also achieve various other benefits. There are generally concerns from 
> > people that are stuck with Hadoop 2 for various reasons. The most recent 
> > thread is here: 
> > https://lists.apache.org/thread/1j5w9dmt1gp8hx31tvrmyomcko4mlp03. I hope at 
> > some point we decide to bite the bullet and do it, but the community has 
> > not yet come to that decision.
> > 
> > In the meantime, we're stuck on an older Calcite version, and really want 
> > to update it, so we're looking into options. There's a GitHub issue here 
> > with our thoughts: https://github.com/apache/druid/issues/13532
> > 
> > On 2022/03/30 17:55:44 Julian Hyde wrote:
> >> Wow, I had no idea that anyone was on a version of Guava that old.
> >> 
> >> I checked our history of Guava versions:
> >> * In https://issues.apache.org/jira/browse/CALCITE-1590 
> >>  (Calcite 1.12) we 
> >> changed the default from 19 to 20, but still supported Guava 14.0.1 and 
> >> higher.
> >> * In https://issues.apache.org/jira/browse/CALCITE-1715 
> >>  (also Calcite 1.12) 
> >> we changed the default version from 20 to 19
> >> * In https://issues.apache.org/jira/browse/CALCITE-2191 
> >>  (Calcite 1.16) we 
> >> moved the minimum from 14.0.1 to 19 and cited a Druid email thread [1] 
> >> about upgrading Druid’s version of Guava
> >> 
> >> Foolish of me to assume that 4 years later Druid would have done something.
> >> 
> >> Julian
> >> 
> >> [1] https://groups.google.com/g/druid-development/c/Dw2Qu1CWbuQ
> >> 
> >>> On Mar 30, 2022, at 10:22 AM, Abhishek Agarwal 
> >>>  wrote:
> >>> 
> >>> Thank you, Alessandro and Stamatis, for the info. Currently, the guava
> >>> version being used in druid is 16. So simple exclusion is unlikely to 
> >>> work.
> >>> I will still give it a try. If it doesn't, we will shade the jars on our
> >>> end.
> >>> 
> >>> On Tue, Mar 29, 2022 at 3:38 PM Alessandro Solimando <
> >>> alessandro.solima...@gmail.com> wrote:
> >>> 
>  Hello Abhishek,
>  maybe I am re-stating what Stamatis suggested without realising, but 
>  since
>  Calcite works well with a broad range of guava versions, can't you simply
>  exclude guava when you include it in druid?
>  
>  At that point if the guava version of druid is in the list of those
>  supported by Calcite (>= 19 IIRC) all should work fine.
>  
>  I feel like Stamatis concerning shading.
>  
>  Best regards,
>  Alessandro
>  
>  On Tue, 29 Mar 2022 at 11:18, Stamatis Zampetakis 
>  wrote:
>  
> > Hi Abhishek,
> > 
> > Calcite supports multiple versions of Guava so in principle it should
>  work
> > without problems with older versions.
> > Instead of shading you could possibly select explicitly the version
> > that you want to use via maven/gradle.
> > 
> > I share your pain but shading is quite complex to get right so I am not
>  in
> > favor of putting this burden on Calcite.
> > 
> > I've seen it used in many projects and it rarely works as expected. Most
>  of
> > the time it creates additional problems that need to be resolved.
> > 
> > Best,
> > Stamatis
> > 
>  
> >> 
> >> 
> 
> 

Re: Calcite jars with shaded guava

[Julian Hyde]

If it helps, it wouldn’t be impossible for Calcite to move its minimum Guava 
version down to 16. (If we rely on APIs introduced after 16 we can create a 
facade for those APIs.)

I am frustrated that the Hadoop folks get locked into a particular version of 
Guava (and JDK… don’t get me started on that one) and in Calcite I would like 
to do the opposite.

Julian


> On Dec 8, 2022, at 9:24 AM, Gian Merlino  wrote:
> 
> It pains me to admit that the reason Druid still uses Guava 16 is that we 
> still support Hadoop 2, which relies on APIs that have been removed in newer 
> versions of Guava. From time to time it is proposed that the project drops 
> support for Hadoop 2 in order to be free to move up Guava versions, and also 
> achieve various other benefits. There are generally concerns from people that 
> are stuck with Hadoop 2 for various reasons. The most recent thread is here: 
> https://lists.apache.org/thread/1j5w9dmt1gp8hx31tvrmyomcko4mlp03. I hope at 
> some point we decide to bite the bullet and do it, but the community has not 
> yet come to that decision.
> 
> In the meantime, we're stuck on an older Calcite version, and really want to 
> update it, so we're looking into options. There's a GitHub issue here with 
> our thoughts: https://github.com/apache/druid/issues/13532
> 
> On 2022/03/30 17:55:44 Julian Hyde wrote:
>> Wow, I had no idea that anyone was on a version of Guava that old.
>> 
>> I checked our history of Guava versions:
>> * In https://issues.apache.org/jira/browse/CALCITE-1590 
>>  (Calcite 1.12) we 
>> changed the default from 19 to 20, but still supported Guava 14.0.1 and 
>> higher.
>> * In https://issues.apache.org/jira/browse/CALCITE-1715 
>>  (also Calcite 1.12) we 
>> changed the default version from 20 to 19
>> * In https://issues.apache.org/jira/browse/CALCITE-2191 
>>  (Calcite 1.16) we moved 
>> the minimum from 14.0.1 to 19 and cited a Druid email thread [1] about 
>> upgrading Druid’s version of Guava
>> 
>> Foolish of me to assume that 4 years later Druid would have done something.
>> 
>> Julian
>> 
>> [1] https://groups.google.com/g/druid-development/c/Dw2Qu1CWbuQ
>> 
>>> On Mar 30, 2022, at 10:22 AM, Abhishek Agarwal 
>>>  wrote:
>>> 
>>> Thank you, Alessandro and Stamatis, for the info. Currently, the guava
>>> version being used in druid is 16. So simple exclusion is unlikely to work.
>>> I will still give it a try. If it doesn't, we will shade the jars on our
>>> end.
>>> 
>>> On Tue, Mar 29, 2022 at 3:38 PM Alessandro Solimando <
>>> alessandro.solima...@gmail.com> wrote:
>>> 
 Hello Abhishek,
 maybe I am re-stating what Stamatis suggested without realising, but since
 Calcite works well with a broad range of guava versions, can't you simply
 exclude guava when you include it in druid?
 
 At that point if the guava version of druid is in the list of those
 supported by Calcite (>= 19 IIRC) all should work fine.
 
 I feel like Stamatis concerning shading.
 
 Best regards,
 Alessandro
 
 On Tue, 29 Mar 2022 at 11:18, Stamatis Zampetakis 
 wrote:
 
> Hi Abhishek,
> 
> Calcite supports multiple versions of Guava so in principle it should
 work
> without problems with older versions.
> Instead of shading you could possibly select explicitly the version
> that you want to use via maven/gradle.
> 
> I share your pain but shading is quite complex to get right so I am not
 in
> favor of putting this burden on Calcite.
> 
> I've seen it used in many projects and it rarely works as expected. Most
 of
> the time it creates additional problems that need to be resolved.
> 
> Best,
> Stamatis
> 
 
>> 
>> 

Re: Calcite jars with shaded guava

[Gian Merlino]

It pains me to admit that the reason Druid still uses Guava 16 is that we still 
support Hadoop 2, which relies on APIs that have been removed in newer versions 
of Guava. From time to time it is proposed that the project drops support for 
Hadoop 2 in order to be free to move up Guava versions, and also achieve 
various other benefits. There are generally concerns from people that are stuck 
with Hadoop 2 for various reasons. The most recent thread is here: 
https://lists.apache.org/thread/1j5w9dmt1gp8hx31tvrmyomcko4mlp03. I hope at 
some point we decide to bite the bullet and do it, but the community has not 
yet come to that decision.

In the meantime, we're stuck on an older Calcite version, and really want to 
update it, so we're looking into options. There's a GitHub issue here with our 
thoughts: https://github.com/apache/druid/issues/13532

On 2022/03/30 17:55:44 Julian Hyde wrote:
> Wow, I had no idea that anyone was on a version of Guava that old.
> 
> I checked our history of Guava versions:
>  * In https://issues.apache.org/jira/browse/CALCITE-1590 
>  (Calcite 1.12) we 
> changed the default from 19 to 20, but still supported Guava 14.0.1 and 
> higher.
>  * In https://issues.apache.org/jira/browse/CALCITE-1715 
>  (also Calcite 1.12) we 
> changed the default version from 20 to 19
>  * In https://issues.apache.org/jira/browse/CALCITE-2191 
>  (Calcite 1.16) we moved 
> the minimum from 14.0.1 to 19 and cited a Druid email thread [1] about 
> upgrading Druid’s version of Guava
> 
> Foolish of me to assume that 4 years later Druid would have done something.
> 
> Julian
> 
> [1] https://groups.google.com/g/druid-development/c/Dw2Qu1CWbuQ
> 
> > On Mar 30, 2022, at 10:22 AM, Abhishek Agarwal 
> >  wrote:
> > 
> > Thank you, Alessandro and Stamatis, for the info. Currently, the guava
> > version being used in druid is 16. So simple exclusion is unlikely to work.
> > I will still give it a try. If it doesn't, we will shade the jars on our
> > end.
> > 
> > On Tue, Mar 29, 2022 at 3:38 PM Alessandro Solimando <
> > alessandro.solima...@gmail.com> wrote:
> > 
> >> Hello Abhishek,
> >> maybe I am re-stating what Stamatis suggested without realising, but since
> >> Calcite works well with a broad range of guava versions, can't you simply
> >> exclude guava when you include it in druid?
> >> 
> >> At that point if the guava version of druid is in the list of those
> >> supported by Calcite (>= 19 IIRC) all should work fine.
> >> 
> >> I feel like Stamatis concerning shading.
> >> 
> >> Best regards,
> >> Alessandro
> >> 
> >> On Tue, 29 Mar 2022 at 11:18, Stamatis Zampetakis 
> >> wrote:
> >> 
> >>> Hi Abhishek,
> >>> 
> >>> Calcite supports multiple versions of Guava so in principle it should
> >> work
> >>> without problems with older versions.
> >>> Instead of shading you could possibly select explicitly the version
> >>> that you want to use via maven/gradle.
> >>> 
> >>> I share your pain but shading is quite complex to get right so I am not
> >> in
> >>> favor of putting this burden on Calcite.
> >>> 
> >>> I've seen it used in many projects and it rarely works as expected. Most
> >> of
> >>> the time it creates additional problems that need to be resolved.
> >>> 
> >>> Best,
> >>> Stamatis
> >>> 
> >> 
> 
> 

Re: Calcite jars with shaded guava

[Julian Hyde]

Wow, I had no idea that anyone was on a version of Guava that old.

I checked our history of Guava versions:
 * In https://issues.apache.org/jira/browse/CALCITE-1590 
 (Calcite 1.12) we changed 
the default from 19 to 20, but still supported Guava 14.0.1 and higher.
 * In https://issues.apache.org/jira/browse/CALCITE-1715 
 (also Calcite 1.12) we 
changed the default version from 20 to 19
 * In https://issues.apache.org/jira/browse/CALCITE-2191 
 (Calcite 1.16) we moved 
the minimum from 14.0.1 to 19 and cited a Druid email thread [1] about 
upgrading Druid’s version of Guava

Foolish of me to assume that 4 years later Druid would have done something.

Julian

[1] https://groups.google.com/g/druid-development/c/Dw2Qu1CWbuQ

> On Mar 30, 2022, at 10:22 AM, Abhishek Agarwal 
>  wrote:
> 
> Thank you, Alessandro and Stamatis, for the info. Currently, the guava
> version being used in druid is 16. So simple exclusion is unlikely to work.
> I will still give it a try. If it doesn't, we will shade the jars on our
> end.
> 
> On Tue, Mar 29, 2022 at 3:38 PM Alessandro Solimando <
> alessandro.solima...@gmail.com> wrote:
> 
>> Hello Abhishek,
>> maybe I am re-stating what Stamatis suggested without realising, but since
>> Calcite works well with a broad range of guava versions, can't you simply
>> exclude guava when you include it in druid?
>> 
>> At that point if the guava version of druid is in the list of those
>> supported by Calcite (>= 19 IIRC) all should work fine.
>> 
>> I feel like Stamatis concerning shading.
>> 
>> Best regards,
>> Alessandro
>> 
>> On Tue, 29 Mar 2022 at 11:18, Stamatis Zampetakis 
>> wrote:
>> 
>>> Hi Abhishek,
>>> 
>>> Calcite supports multiple versions of Guava so in principle it should
>> work
>>> without problems with older versions.
>>> Instead of shading you could possibly select explicitly the version
>>> that you want to use via maven/gradle.
>>> 
>>> I share your pain but shading is quite complex to get right so I am not
>> in
>>> favor of putting this burden on Calcite.
>>> 
>>> I've seen it used in many projects and it rarely works as expected. Most
>> of
>>> the time it creates additional problems that need to be resolved.
>>> 
>>> Best,
>>> Stamatis
>>> 
>> 

Re: Calcite jars with shaded guava

[Abhishek Agarwal]

Thank you, Alessandro and Stamatis, for the info. Currently, the guava
version being used in druid is 16. So simple exclusion is unlikely to work.
I will still give it a try. If it doesn't, we will shade the jars on our
end.

On Tue, Mar 29, 2022 at 3:38 PM Alessandro Solimando <
alessandro.solima...@gmail.com> wrote:

> Hello Abhishek,
> maybe I am re-stating what Stamatis suggested without realising, but since
> Calcite works well with a broad range of guava versions, can't you simply
> exclude guava when you include it in druid?
>
> At that point if the guava version of druid is in the list of those
> supported by Calcite (>= 19 IIRC) all should work fine.
>
> I feel like Stamatis concerning shading.
>
> Best regards,
> Alessandro
>
> On Tue, 29 Mar 2022 at 11:18, Stamatis Zampetakis 
> wrote:
>
> > Hi Abhishek,
> >
> > Calcite supports multiple versions of Guava so in principle it should
> work
> > without problems with older versions.
> > Instead of shading you could possibly select explicitly the version
> > that you want to use via maven/gradle.
> >
> > I share your pain but shading is quite complex to get right so I am not
> in
> > favor of putting this burden on Calcite.
> >
> > I've seen it used in many projects and it rarely works as expected. Most
> of
> > the time it creates additional problems that need to be resolved.
> >
> > Best,
> > Stamatis
> >
>

Re: Calcite jars with shaded guava

[Alessandro Solimando]

Hello Abhishek,
maybe I am re-stating what Stamatis suggested without realising, but since
Calcite works well with a broad range of guava versions, can't you simply
exclude guava when you include it in druid?

At that point if the guava version of druid is in the list of those
supported by Calcite (>= 19 IIRC) all should work fine.

I feel like Stamatis concerning shading.

Best regards,
Alessandro

On Tue, 29 Mar 2022 at 11:18, Stamatis Zampetakis  wrote:

> Hi Abhishek,
>
> Calcite supports multiple versions of Guava so in principle it should work
> without problems with older versions.
> Instead of shading you could possibly select explicitly the version
> that you want to use via maven/gradle.
>
> I share your pain but shading is quite complex to get right so I am not in
> favor of putting this burden on Calcite.
>
> I've seen it used in many projects and it rarely works as expected. Most of
> the time it creates additional problems that need to be resolved.
>
> Best,
> Stamatis
>

Re: Calcite jars with shaded guava

[Stamatis Zampetakis]

Hi Abhishek,

Calcite supports multiple versions of Guava so in principle it should work
without problems with older versions.
Instead of shading you could possibly select explicitly the version
that you want to use via maven/gradle.

I share your pain but shading is quite complex to get right so I am not in
favor of putting this burden on Calcite.

I've seen it used in many projects and it rarely works as expected. Most of
the time it creates additional problems that need to be resolved.

Best,
Stamatis

Calcite jars with shaded guava

[Abhishek Agarwal]

Hi,
I am trying to upgrade calcite in the druid codebase. But because of the
guava version mismatch between druid and calcite, a straightforward upgrade
is not possible. We also can't upgrade guava in druid itself since Hadoop-2
still requires old guava jars. As a workaround, I created shaded artifacts
of calcite jars here - https://github.com/abhishekagarwal87/calcite-shaded.
I tested the calcite upgrade and didn't observe any other dependency
conflicts.

Instead of maintaining this repository, is it possible to generate and
upload shaded guava jars from the calcite main repo itself? If such an idea
is acceptable, I can create a PR in the calcite main repo.