On Tue, Apr 11, 2017 at 10:23 AM, York, Zach <[email protected]> wrote:
> Should we allow dependent projects (such as Phoenix) to weigh in on this > issue since they are likely going to be the ones that benefit/are effected? > > I dumped a pointer to here into dev@phoenix. St.Ack > On 4/11/17, 10:17 AM, "York, Zach" <[email protected]> wrote: > > +1 (non-binding) > > This sounds like a good idea to me! > > Zach > > On 4/11/17, 9:48 AM, "[email protected] on behalf of Stack" < > [email protected] on behalf of [email protected]> wrote: > > Let me revive this thread. > > Recall, we are stuck on old or particular versions of critical > libs. We are > unable to update because our versions will clash w/ versions from > upstreamer hadoop2.7/2.8/3.0/spark, etc. We have a shaded client. > We need > to message downstreamers that they should use it going forward. > This will > help going forward but it will not inoculate our internals nor an > existing > context where we'd like to be a compatible drop-in. > > We could try hackery filtering transitive includes up in poms for > each > version of hadoop/spark that we support but in the end, its a > bunch of > effort, hard to test, and we are unable to dictate the CLASSPATH > order in > all situations. > > We could try some shading voodoo inline w/ build. Because shading > is a > post-package step and because we are modularized and shading > includes the > shaded classes in the artifact produced, we'd end up w/ multiple > copies of > guava/netty/etc. classes, an instance per module that makes a > reference. > > Lets do Sean's idea of a pre-build step where we package and > relocate > ('shade') critical dependencies (Going by the thread above, Ram, > Anoop, and > Andy seems good w/ general idea). > > In implementation, we (The HBase PMC) would ask for a new repo > [1]. In here > we'd create a new mvn project. This project would produce a single > artifact > (jar) called hbase-dependencies or hbase-3rdparty or > hbase-shaded-3rdparty > libs. In it would be relocated core libs such as guava and netty > (and maybe > protobuf). We'd publish this artifact and then have hbase depend > on it > changing all references to point at the relocation: e.g. rather > than import > com.google.common.collect.Maps, we'd import > org.apache.hadoop.hbase.com.google.common.collect.Maps. > > We (The HBase PMC) will have to make releases of this new artifact > and vote > on them. I think it will be a relatively rare event. > > I'd be up for doing the first cut if folks are game. > > St.Ack > > > 1. URL via Sean but for committers to view only: > https://reporeq.apache.org/ > > On Sun, Oct 2, 2016 at 10:29 PM, ramkrishna vasudevan < > [email protected]> wrote: > > > +1 for Sean's ideas. Bundling all the dependent libraries and > shading them > > into one jar and HBase referring to it makes sense and should > avoid some of > > the pain in terms of IDE usage. Stack's doc clearly talks about > the IDE > > issues that we may get after this protobuf shading goes in. It > may be > > difficult for new comers and those who don't know this > background of why it > > has to be like that. > > > > Regards > > Ram > > > > On Sun, Oct 2, 2016 at 10:51 AM, Stack <[email protected]> wrote: > > > > > On Sat, Oct 1, 2016 at 6:32 PM, Jerry He <[email protected]> > wrote: > > > > > > > How is the proposed going to impact the existing > shaded-client and > > > > shaded-server modules, making them unnecessary and go away? > > > > > > > > > > No. We still need the blanket shading of hbase client and > server. > > > > > > This effort is about our internals. We have a mess of other > components > > all > > > up inside us such as HDFS, etc., each with their own sets of > dependencies > > > many of which we have in common. This project t is about > making it so we > > > can upgrade at a rate independent of when our upstreamers > choose to > > change. > > > > > > > > > > It doesn't seem so. These modules are supposed to shade > HBase and > > > upstream > > > > from downstream users. > > > > > > > > > > Agree. > > > > > > Thanks for drawing out the difference between these two > shading efforts, > > > > > > St.Ack > > > > > > > > > > > > > Thanks. > > > > > > > > Jerry > > > > > > > > On Sat, Oct 1, 2016 at 2:33 PM, Andrew Purtell < > > [email protected] > > > > > > > > wrote: > > > > > > > > > > Sean has suggested a pre-build step where in another > repo we'd make > > > > hbase > > > > > > shaded versions of critical libs, 'release' them (votes, > etc.) and > > > then > > > > > > have core depend on these. It be a bunch of work but > would make the > > > > dev's > > > > > > life easier. > > > > > > > > > > So when we make changes that require updates to and > rebuild of the > > > > > supporting libraries, as a developer I would make local > changes, > > > install > > > > a > > > > > snapshot of that into the local maven cache, then point > the HBase > > build > > > > at > > > > > the snapshot, then do the other half of the work, then > push up to > > both? > > > > > > > > > > I think this could work. > > > > > > > > > > > > > >
