On Tue, Jul 4, 2017 at 6:16 AM, Chunhui Shen <zju...@163.com> wrote: > A small thing, we can't get the real version of the thirdparty library > after depend on the new generated hbase-shaded-*.jar. > With the above doubt, I found the file 'META-INF/DEPENDENCIES' which is > included in hbase-shaded-*.jar, it contains the information about the > shaded thirdparty library name and its version. > However, 'META-INF/DEPENDENCIES' of hbase-shaded-protobuf-1.0.0.jar > contains none... > > Thanks Chunhui. Makes sense that this should be populated by the hbase-thirdparty build. Let me fix in next release. I filed HBASE-18321 [hbase-thirdparty] Fix generation of META-INF/DEPENDENCIES to include dependency list and versions
Thanks, S > > Just realized such a different point, it's not a problem for most > developers I think, but better if we could get the real information about > thirdparty library in hbase project. > > At 2017-07-04 13:42:44, "Yu Li" <car...@gmail.com> wrote: > >bq. In branch-1, relocate all probobuf and guava references? > >Yes sir, IMHO this is nice to have (smile). Thanks. > > > >Best Regards, > >Yu > > > >On 4 July 2017 at 11:41, Stack <st...@duboce.net> wrote: > > > >> On Sun, Jul 2, 2017 at 11:44 PM, Yu Li <car...@gmail.com> wrote: > >> > > >> > bq. No plans to up guava or protobuf versions on branch-1. You think > we > >> > should? > >> > We met with guava conflict when using hbase in Flink, and I could see > >> > similar issue mentioned in spark. Since branch-1 is still the stable > >> > release line and will be in the near future, possible to do similar > >> shading > >> > w/o upgrading the guava/protobuf version? Thanks. > >> > > >> > In branch-1, relocate all probobuf and guava references? > >> > >> Thanks Yu Li, > >> S > >> > >> > >> > > >> > Best Regards, > >> > Yu > >> > > >> > On 3 July 2017 at 14:01, 张铎(Duo Zhang) <palomino...@gmail.com> wrote: > >> > > >> > > Then it will be a bit pain if developers want to see the source > code of > >> > the > >> > > relocated classes in IDE? > >> > > > >> > > 2017-07-03 12:31 GMT+08:00 Stack <st...@duboce.net>: > >> > > > >> > > > On Sat, Jul 1, 2017 at 8:16 PM, 张铎(Duo Zhang) < > palomino...@gmail.com > >> > > >> > > > wrote: > >> > > > > >> > > > > +1 (binding) > >> > > > > > >> > > > > Checked sums and signatures: OK > >> > > > > 'mvn clean install': OK, the size of the generated jar files are > >> > > > reasonable > >> > > > > Unzip the generated jar files: OK, all classes are relocated > under > >> > > > > org/apache/hadoop/hbase/shaded > >> > > > > > >> > > > > Only one minor question, seems we do not generate sources jars? > >> There > >> > > is > >> > > > > a hbase-shaded-protobuf-1.0.0-sources.jar after 'mvn install' > but > >> > the > >> > > > size > >> > > > > is only 22 bytes. > >> > > > > > >> > > > > > >> > > > Good question. This project is a little bit odd. It is just poms > >> (and a > >> > > few > >> > > > patches). The poms do a download, relocation, and then push to > deploy > >> > the > >> > > > jars to mvn repo. There is no real 'source' -- not anything that > >> makes > >> > > > sense to stuff in a src.jar -- other than the poms that are here > in > >> > the > >> > > > tgz (mvn install puts the jars w/ relocation into your local > repo.... > >> > > add a > >> > > > 'deploy' to push to mvn repository -- see README). > >> > > > > >> > > > Thanks Duo, > >> > > > S > >> > > > > >> > > > > >> > > > > >> > > > > Thanks. > >> > > > > > >> > > > > 2017-07-02 2:38 GMT+08:00 Stack <st...@duboce.net>: > >> > > > > > >> > > > > > +1 from me. > >> > > > > > St.Ack > >> > > > > > > >> > > > > > On Fri, Jun 30, 2017 at 3:07 PM, Stack <st...@duboce.net> > wrote: > >> > > > > > > >> > > > > > > This is an interesting vote. The vote is on an RC made out > of a > >> > new > >> > > > > hbase > >> > > > > > > project, hbase-thirdparty. > >> > > > > > > > >> > > > > > > First the usual stuff. > >> > > > > > > > >> > > > > > > The 1.0.0RC0 is available at: > >> > > > > > > > >> > > > > > > https://dist.apache.org/repos/dist/dev/hbase/hbase- > >> > > > > thirdparty/1.0.0RC0/ > >> > > > > > > > >> > > > > > > Maven artifacts are in the following staging repository: > >> > > > > > > > >> > > > > > > https://repository.apache.org/content/repositories/ > >> > > > > orgapachehbase-1171 > >> > > > > > > > >> > > > > > > Artifacts are signed with 8ACC93D2 which is at the tail of > our > >> > KEYS > >> > > > > file > >> > > > > > > http://www-us.apache.org/dist/hbase/KEYS. > >> > > > > > > > >> > > > > > > I tagged this RC as 1.0.0RC at > 8ffaf3dd561052bcc71772148ecd04 > >> > > > > cdf9e224f3 > >> > > > > > > in the new hbase-thirdparty repository at [1]. > >> > > > > > > > >> > > > > > > Now to the unusual. > >> > > > > > > > >> > > > > > > The artifact was made from a new repository, > >> hbase-thirdparty[1]. > >> > > > > > > hbase-thirdparty is a project overseen by the hbase PMC. It > >> came > >> > of > >> > > > > > > discussion on the dev list in the thread '[DISCUSS] More > >> Shading' > >> > > > [2]. > >> > > > > > > Its charge is the hosting of ornery thirdparty libs -- e.g. > >> > > protobuf, > >> > > > > > > guava, netty -- that need patching and/or shading so we are > no > >> > > longer > >> > > > > > bound > >> > > > > > > by whatever the version of a lib Hadoop happens to ship > (and we > >> > can > >> > > > > > change > >> > > > > > > versions w/o damaging out glorious downstreamers). In the > past, > >> > > when > >> > > > > > we've > >> > > > > > > been able, we've done this dirty patching and shading work > as > >> an > >> > > > > > > unacknowledged 'pre-build' hack step appended to our > protobuf > >> > file > >> > > > > > > generation. Moving this mess out of our mainline build to > live > >> > > > > elsewhere, > >> > > > > > > pulled in as a legit 'dependency', simplifies our build > >> > > particularly > >> > > > > > around > >> > > > > > > the shading of third-part libs; the narrower scope of a > >> dedicated > >> > > > > project > >> > > > > > > that produces transformed jars is easier to reason about, is > >> less > >> > > > > likely > >> > > > > > to > >> > > > > > > clash with existing dependencies whether transitive or > >> otherwise, > >> > > and > >> > > > > it > >> > > > > > > makes for less spaghetti poms. > >> > > > > > > > >> > > > > > > Other (minor) benefits include our being able to generate > >> > protobuf > >> > > > > files > >> > > > > > > inline now; no need of the pre-build anymore nor checking > in of > >> > > > > generated > >> > > > > > > protobuf files (25MB). There is a patch to remove them > >> > (predicated > >> > > > on > >> > > > > > this > >> > > > > > > hbase-thirdparty release): "HBASE-17056 Remove checked in PB > >> > > > generated > >> > > > > > > files". > >> > > > > > > > >> > > > > > > Downside is having to explicitly reference our version in > code > >> as > >> > > in > >> > > > > > > org.apache.hadoop.hbase.shaded.com.google.protobuf.Message > >> > rather > >> > > > than > >> > > > > > > just com.google.protobuf.Message and so on for guava > >> references, > >> > > etc. > >> > > > > > > (We've been doing this for a while now so hopefully it not > too > >> > > > strange > >> > > > > > > developing)*. > >> > > > > > > > >> > > > > > > There is little to this new project other than a few poms > to do > >> > > > > packaging > >> > > > > > > and a couple of patches we apply on build. It works by > pulling > >> > down > >> > > > > > > dependencies and doing whatever shading, renaming, or > patching > >> we > >> > > > need. > >> > > > > > > Currently hbase-thirdparty hosts protobuf (3.1.0 => 3.3.1), > >> guava > >> > > > (12.0 > >> > > > > > => > >> > > > > > > 22.0), gson, protobuf-util, and netty 4.1.12-FINAL. It > >> publishes > >> > > > three > >> > > > > > jars > >> > > > > > > for hbase to consume: hbase-shaded-protobuf, > >> hbase-shaded-netty, > >> > > and > >> > > > > then > >> > > > > > > hbase-shaded-miscellaneous for the remaining libs. > Philosophy > >> is > >> > if > >> > > > the > >> > > > > > lib > >> > > > > > > is troublesome, protobuf needs patching and netty includes > an > >> .so > >> > > > that > >> > > > > > > needs a rename, then give it its own module (See README in > the > >> > > > release > >> > > > > > for > >> > > > > > > more). I've arrived at this layout for hbase-thirdparty > after a > >> > > bunch > >> > > > > of > >> > > > > > > experimentation and messing with hbase dependencies (See > >> > sub-tasks > >> > > > > under > >> > > > > > > HBASE-17898 Update dependencies). That said, as we head to > >> > > > hbase-2.0.0, > >> > > > > > we > >> > > > > > > might have to make a an odd new release of hbase-thirdparty > but > >> > > > > > thereafter > >> > > > > > > I'd imagine it should settle down to be an infrequent > update. > >> > > > > > > > >> > > > > > > VOTE lasts 72 hours. Let +1 be an assent to the > >> hbase-thirdparty > >> > > > > project > >> > > > > > > and a vote for this RC. > >> > > > > > > > >> > > > > > > Questions on how it will go w/ this project welcome on this > >> > thread. > >> > > > If > >> > > > > > > heavy-detail on how the shading works, I suggest ask on the > >> > > > '[DISCUSS] > >> > > > > > More > >> > > > > > > Shading' thread. > >> > > > > > > > >> > > > > > > Thanks, > >> > > > > > > St.Ack > >> > > > > > > 1. https://git-wip-us.apache.org/repos/asf/hbase-thirdparty > >> > > > > > > 2. See the '[DISCUSS] More Shading' thread, > >> > > > > > http://apache-hbase.679495.n3. > >> > > > > > > nabble.com/DISCUSS-More-Shading-td4083025.html > >> > > > > > > > >> > > > > > > * Doc on how to make use of hbase-thirdparty is coming for > the > >> > > > > refguide. > >> > > > > > > > >> > > > > > > > >> > > > > > > > >> > > > > > > >> > > > > > >> > > > > >> > > > >> > > >> >
