A small thing, we can't get the real version of the thirdparty library after depend on the new generated hbase-shaded-*.jar. With the above doubt, I found the file 'META-INF/DEPENDENCIES' which is included in hbase-shaded-*.jar, it contains the information about the shaded thirdparty library name and its version. However, 'META-INF/DEPENDENCIES' of hbase-shaded-protobuf-1.0.0.jar contains none...
Just realized such a different point, it's not a problem for most developers I think, but better if we could get the real information about thirdparty library in hbase project. At 2017-07-04 13:42:44, "Yu Li" <car...@gmail.com> wrote: >bq. In branch-1, relocate all probobuf and guava references? >Yes sir, IMHO this is nice to have (smile). Thanks. > >Best Regards, >Yu > >On 4 July 2017 at 11:41, Stack <st...@duboce.net> wrote: > >> On Sun, Jul 2, 2017 at 11:44 PM, Yu Li <car...@gmail.com> wrote: >> > >> > bq. No plans to up guava or protobuf versions on branch-1. You think we >> > should? >> > We met with guava conflict when using hbase in Flink, and I could see >> > similar issue mentioned in spark. Since branch-1 is still the stable >> > release line and will be in the near future, possible to do similar >> shading >> > w/o upgrading the guava/protobuf version? Thanks. >> > >> > In branch-1, relocate all probobuf and guava references? >> >> Thanks Yu Li, >> S >> >> >> > >> > Best Regards, >> > Yu >> > >> > On 3 July 2017 at 14:01, 张铎(Duo Zhang) <palomino...@gmail.com> wrote: >> > >> > > Then it will be a bit pain if developers want to see the source code of >> > the >> > > relocated classes in IDE? >> > > >> > > 2017-07-03 12:31 GMT+08:00 Stack <st...@duboce.net>: >> > > >> > > > On Sat, Jul 1, 2017 at 8:16 PM, 张铎(Duo Zhang) <palomino...@gmail.com >> > >> > > > wrote: >> > > > >> > > > > +1 (binding) >> > > > > >> > > > > Checked sums and signatures: OK >> > > > > 'mvn clean install': OK, the size of the generated jar files are >> > > > reasonable >> > > > > Unzip the generated jar files: OK, all classes are relocated under >> > > > > org/apache/hadoop/hbase/shaded >> > > > > >> > > > > Only one minor question, seems we do not generate sources jars? >> There >> > > is >> > > > > a hbase-shaded-protobuf-1.0.0-sources.jar after 'mvn install' but >> > the >> > > > size >> > > > > is only 22 bytes. >> > > > > >> > > > > >> > > > Good question. This project is a little bit odd. It is just poms >> (and a >> > > few >> > > > patches). The poms do a download, relocation, and then push to deploy >> > the >> > > > jars to mvn repo. There is no real 'source' -- not anything that >> makes >> > > > sense to stuff in a src.jar -- other than the poms that are here in >> > the >> > > > tgz (mvn install puts the jars w/ relocation into your local repo.... >> > > add a >> > > > 'deploy' to push to mvn repository -- see README). >> > > > >> > > > Thanks Duo, >> > > > S >> > > > >> > > > >> > > > >> > > > > Thanks. >> > > > > >> > > > > 2017-07-02 2:38 GMT+08:00 Stack <st...@duboce.net>: >> > > > > >> > > > > > +1 from me. >> > > > > > St.Ack >> > > > > > >> > > > > > On Fri, Jun 30, 2017 at 3:07 PM, Stack <st...@duboce.net> wrote: >> > > > > > >> > > > > > > This is an interesting vote. The vote is on an RC made out of a >> > new >> > > > > hbase >> > > > > > > project, hbase-thirdparty. >> > > > > > > >> > > > > > > First the usual stuff. >> > > > > > > >> > > > > > > The 1.0.0RC0 is available at: >> > > > > > > >> > > > > > > https://dist.apache.org/repos/dist/dev/hbase/hbase- >> > > > > thirdparty/1.0.0RC0/ >> > > > > > > >> > > > > > > Maven artifacts are in the following staging repository: >> > > > > > > >> > > > > > > https://repository.apache.org/content/repositories/ >> > > > > orgapachehbase-1171 >> > > > > > > >> > > > > > > Artifacts are signed with 8ACC93D2 which is at the tail of our >> > KEYS >> > > > > file >> > > > > > > http://www-us.apache.org/dist/hbase/KEYS. >> > > > > > > >> > > > > > > I tagged this RC as 1.0.0RC at 8ffaf3dd561052bcc71772148ecd04 >> > > > > cdf9e224f3 >> > > > > > > in the new hbase-thirdparty repository at [1]. >> > > > > > > >> > > > > > > Now to the unusual. >> > > > > > > >> > > > > > > The artifact was made from a new repository, >> hbase-thirdparty[1]. >> > > > > > > hbase-thirdparty is a project overseen by the hbase PMC. It >> came >> > of >> > > > > > > discussion on the dev list in the thread '[DISCUSS] More >> Shading' >> > > > [2]. >> > > > > > > Its charge is the hosting of ornery thirdparty libs -- e.g. >> > > protobuf, >> > > > > > > guava, netty -- that need patching and/or shading so we are no >> > > longer >> > > > > > bound >> > > > > > > by whatever the version of a lib Hadoop happens to ship (and we >> > can >> > > > > > change >> > > > > > > versions w/o damaging out glorious downstreamers). In the past, >> > > when >> > > > > > we've >> > > > > > > been able, we've done this dirty patching and shading work as >> an >> > > > > > > unacknowledged 'pre-build' hack step appended to our protobuf >> > file >> > > > > > > generation. Moving this mess out of our mainline build to live >> > > > > elsewhere, >> > > > > > > pulled in as a legit 'dependency', simplifies our build >> > > particularly >> > > > > > around >> > > > > > > the shading of third-part libs; the narrower scope of a >> dedicated >> > > > > project >> > > > > > > that produces transformed jars is easier to reason about, is >> less >> > > > > likely >> > > > > > to >> > > > > > > clash with existing dependencies whether transitive or >> otherwise, >> > > and >> > > > > it >> > > > > > > makes for less spaghetti poms. >> > > > > > > >> > > > > > > Other (minor) benefits include our being able to generate >> > protobuf >> > > > > files >> > > > > > > inline now; no need of the pre-build anymore nor checking in of >> > > > > generated >> > > > > > > protobuf files (25MB). There is a patch to remove them >> > (predicated >> > > > on >> > > > > > this >> > > > > > > hbase-thirdparty release): "HBASE-17056 Remove checked in PB >> > > > generated >> > > > > > > files". >> > > > > > > >> > > > > > > Downside is having to explicitly reference our version in code >> as >> > > in >> > > > > > > org.apache.hadoop.hbase.shaded.com.google.protobuf.Message >> > rather >> > > > than >> > > > > > > just com.google.protobuf.Message and so on for guava >> references, >> > > etc. >> > > > > > > (We've been doing this for a while now so hopefully it not too >> > > > strange >> > > > > > > developing)*. >> > > > > > > >> > > > > > > There is little to this new project other than a few poms to do >> > > > > packaging >> > > > > > > and a couple of patches we apply on build. It works by pulling >> > down >> > > > > > > dependencies and doing whatever shading, renaming, or patching >> we >> > > > need. >> > > > > > > Currently hbase-thirdparty hosts protobuf (3.1.0 => 3.3.1), >> guava >> > > > (12.0 >> > > > > > => >> > > > > > > 22.0), gson, protobuf-util, and netty 4.1.12-FINAL. It >> publishes >> > > > three >> > > > > > jars >> > > > > > > for hbase to consume: hbase-shaded-protobuf, >> hbase-shaded-netty, >> > > and >> > > > > then >> > > > > > > hbase-shaded-miscellaneous for the remaining libs. Philosophy >> is >> > if >> > > > the >> > > > > > lib >> > > > > > > is troublesome, protobuf needs patching and netty includes an >> .so >> > > > that >> > > > > > > needs a rename, then give it its own module (See README in the >> > > > release >> > > > > > for >> > > > > > > more). I've arrived at this layout for hbase-thirdparty after a >> > > bunch >> > > > > of >> > > > > > > experimentation and messing with hbase dependencies (See >> > sub-tasks >> > > > > under >> > > > > > > HBASE-17898 Update dependencies). That said, as we head to >> > > > hbase-2.0.0, >> > > > > > we >> > > > > > > might have to make a an odd new release of hbase-thirdparty but >> > > > > > thereafter >> > > > > > > I'd imagine it should settle down to be an infrequent update. >> > > > > > > >> > > > > > > VOTE lasts 72 hours. Let +1 be an assent to the >> hbase-thirdparty >> > > > > project >> > > > > > > and a vote for this RC. >> > > > > > > >> > > > > > > Questions on how it will go w/ this project welcome on this >> > thread. >> > > > If >> > > > > > > heavy-detail on how the shading works, I suggest ask on the >> > > > '[DISCUSS] >> > > > > > More >> > > > > > > Shading' thread. >> > > > > > > >> > > > > > > Thanks, >> > > > > > > St.Ack >> > > > > > > 1. https://git-wip-us.apache.org/repos/asf/hbase-thirdparty >> > > > > > > 2. See the '[DISCUSS] More Shading' thread, >> > > > > > http://apache-hbase.679495.n3. >> > > > > > > nabble.com/DISCUSS-More-Shading-td4083025.html >> > > > > > > >> > > > > > > * Doc on how to make use of hbase-thirdparty is coming for the >> > > > > refguide. >> > > > > > > >> > > > > > > >> > > > > > > >> > > > > > >> > > > > >> > > > >> > > >> > >>
