The main reason why we don't shade protobuf is MR jobs for bulk load. We are using hbase-protocol in the classpath and clash between shaded and unshaded protobufs happen. Actually I'm not sure whether we still need this (I mean hbase-protocol in the classpath). Actually shading doesn't work well for all situations. Following cases are hard to shade : 1) uses auto generated classes (like Protobuf) 2) Uses services (like jackson, jaxb) 3) Popular libraries that used by others using Reflection API Don't remember why we unshade guava, but there was one of those cases just for sure. It would be interesting to try it with shaded HBase client, but it would not work for 0.98.
Thanks, Sergey On Sat, Sep 3, 2016 at 10:07 AM, James Taylor <jamestay...@apache.org> wrote: > Also agree - shading of guava & protobuf would be super valuable. Phoenix > ended up not supporting shading of protobuf because of difficulties getting > it to work (maybe because HBase dependency?). I think we support shading of > Guava, though. Is that correct, Sergey? > > On Sat, Sep 3, 2016 at 10:02 AM, Jacques Nadeau <jacq...@apache.org> > wrote: > >> +1 on shading guava/protobuf. >> >> On Sat, Sep 3, 2016 at 9:48 AM, Andrew Purtell <andrew.purt...@gmail.com> >> wrote: >> >> > Since Calcite should become a widely used library (smile) I think it >> would >> > be prudent to shade Guava and protobuf if Calcite depends on them. Then >> you >> > will play very nicely indeed on the classpath no matter what versions >> are >> > required by calling code. >> > >> > Jacques - Good lord. Let me see about shading HBase use of Guava, or >> > eliminating it. Unfortunately that will be no help in the short term. >> > Related, our Stack is wrestling with shading protobuf already, and is >> neck >> > deep in the Swamp of Classloading at the moment. >> > >> > > On Sep 3, 2016, at 9:06 AM, Jacques Nadeau <jacq...@apache.org> >> wrote: >> > > >> > > It isn't a real solution but in Drill we solved the HBase >> incompatibility >> > > issue on the server side (for tests only) by patching Guava 18 to >> allow >> > the >> > > HBase Guava calls that are missing. They are really quite trivial and >> > > support Andrew's arguments that Guava is the devil... >> > > >> > > https://github.com/apache/drill/blob/master/exec/java- >> > exec/src/main/java/org/apache/drill/exec/util/GuavaPatcher.java >> > > >> > > On Sat, Sep 3, 2016 at 8:16 AM, Andrew Purtell < >> andrew.purt...@gmail.com >> > > >> > > wrote: >> > > >> > >> While that seems very unfriendly of them, the main issue is Guava is >> the >> > >> devil (and protobuf is a minor demon). Would shading be an option? >> > >> >> > >>> On Sep 3, 2016, at 2:03 AM, CPC <acha...@gmail.com> wrote: >> > >>> >> > >>> Cassandra driver 3.x require min guava 16.0.1. If it detects an >> earlier >> > >>> version in classpath it stops working. >> > >>> >> > >>>> On Sep 3, 2016 04:26, "Julian Hyde" <jh...@apache.org> wrote: >> > >>>> >> > >>>> James & Andrew, I hear you. We’ll stay on Guava 12 if we have to. >> > >>>> >> > >>>> But can we try an experiment to see if it’s possible to get away >> with >> > >> 14? >> > >>>> >> > >>>> I propose that Maryann (who is developing the branch of Phoenix >> that >> > >> uses >> > >>>> Calcite) tries running with https://github.com/apache/ >> > calcite/pull/277 >> > >> < >> > >>>> https://github.com/apache/calcite/pull/277>. If we discover >> problems, >> > >> we >> > >>>> can try various solutions, like make the DateRangeRules disabled by >> > >> default >> > >>>> (these, and the Druid adapter, are the only parts of Calcite that >> need >> > >>>> Guava 14), or even copy the Guava classes that we need. If there >> > aren’t >> > >>>> problems, it means that we’ve slipped out of the shackles of >> inertia >> > >> that >> > >>>> are trying to drag us into an early grave. >> > >>>> >> > >>>> Julian >> > >>>> >> > >>>> >> > >>>>> On Sep 2, 2016, at 5:35 PM, James Taylor <jamestay...@apache.org> >> > >> wrote: >> > >>>>> >> > >>>>> On the server-side, HBase depends on Guava 12 (because Hadoop >> depends >> > >> on >> > >>>>> the same). For that reason, we've made sure Phoenix can work with >> > this >> > >>>>> version too. Phoenix may not need to depend on Calcite on the >> > >>>> server-side, >> > >>>>> and Phoenix and HBase both have shading, so there may be some >> avenues >> > >> of >> > >>>>> escape. >> > >>>>> >> > >>>>> Sorry for the muddled answer. >> > >>>>> >> > >>>>>> On Fri, Sep 2, 2016 at 5:21 PM, Andrew Purtell < >> apurt...@apache.org >> > > >> > >>>>> wrote: >> > >>>>> >> > >>>>>> Use of Guava 14 introduces at least a compile time problem with >> > HBase, >> > >>>> upon >> > >>>>>> which Phoenix depends, so I'm not sure Phoenix can move off of >> 13. >> > I'd >> > >>>> be >> > >>>>>> happy to be proven wrong. >> > >>>>>> >> > >>>>>>> On Fri, Sep 2, 2016 at 4:35 PM, Julian Hyde <jh...@apache.org> >> > >> wrote: >> > >>>>>>> >> > >>>>>>> Calcite currently supports a wide range of Guava versions, from >> > >> 12.0.1 >> > >>>> to >> > >>>>>>> 19.0*. For https://issues.apache.org/jira/browse/CALCITE-1334 < >> > >>>>>>> https://issues.apache.org/jira/browse/CALCITE-1334> I’d like to >> > use >> > >>>>>>> RangeSet, which was introduced in Guava 14. >> > >>>>>>> >> > >>>>>>> Would anyone have a problem if we made Calcite’s minimum Guava >> > >> version >> > >>>>>>> 14.0.1? >> > >>>>>>> >> > >>>>>>> I see that Hive uses 14.0.1, Phoenix uses 13, Drill uses 18. >> > >>>>>>> >> > >>>>>>> Julian >> > >>>>>>> >> > >>>>>>> * Except for the Druid adapter, which requires 14; see >> > >>>>>>> https://issues.apache.org/jira/browse/CALCITE-1325 < >> > >>>>>>> https://issues.apache.org/jira/browse/CALCITE-1325> >> > >>>>>> >> > >>>>>> >> > >>>>>> >> > >>>>>> >> > >>>>>> -- >> > >>>>>> Best regards, >> > >>>>>> >> > >>>>>> - Andy >> > >>>>>> >> > >>>>>> Problems worthy of attack prove their worth by hitting back. - >> Piet >> > >> Hein >> > >>>>>> (via Tom White) >> > >> >> > >> > >
