On Sun, Jul 2, 2017 at 11:44 PM, Yu Li <car...@gmail.com> wrote:
>
> bq. No plans to up guava or protobuf versions on branch-1. You think we
> should?
> We met with guava conflict when using hbase in Flink, and I could see
> similar issue mentioned in spark. Since branch-1 is still the stable
> release line and will be in the near future, possible to do similar shading
> w/o upgrading the guava/protobuf version? Thanks.
>
> In branch-1, relocate all probobuf and guava references?

Thanks Yu Li,
S


>
> Best Regards,
> Yu
>
> On 3 July 2017 at 14:01, 张铎(Duo Zhang) <palomino...@gmail.com> wrote:
>
> > Then it will be a bit pain if developers want to see the source code of
> the
> > relocated classes in IDE?
> >
> > 2017-07-03 12:31 GMT+08:00 Stack <st...@duboce.net>:
> >
> > > On Sat, Jul 1, 2017 at 8:16 PM, 张铎(Duo Zhang) <palomino...@gmail.com>
> > > wrote:
> > >
> > > > +1 (binding)
> > > >
> > > > Checked sums and signatures: OK
> > > > 'mvn clean install': OK, the size of the generated jar files are
> > > reasonable
> > > > Unzip the generated jar files: OK, all classes are relocated under
> > > > org/apache/hadoop/hbase/shaded
> > > >
> > > > Only one minor question, seems we do not generate sources jars? There
> > is
> > > > a hbase-shaded-protobuf-1.0.0-sources.jar after 'mvn install' but
> the
> > > size
> > > > is only 22 bytes.
> > > >
> > > >
> > > Good question. This project is a little bit odd. It is just poms (and a
> > few
> > > patches). The poms do a download, relocation, and then push to deploy
> the
> > > jars to mvn repo. There is no real 'source' -- not anything that makes
> > > sense to stuff in a src.jar --  other than the poms that are here in
> the
> > > tgz (mvn install puts the jars w/ relocation into your local repo....
> > add a
> > > 'deploy' to push to mvn repository -- see README).
> > >
> > > Thanks Duo,
> > > S
> > >
> > >
> > >
> > > > Thanks.
> > > >
> > > > 2017-07-02 2:38 GMT+08:00 Stack <st...@duboce.net>:
> > > >
> > > > > +1 from me.
> > > > > St.Ack
> > > > >
> > > > > On Fri, Jun 30, 2017 at 3:07 PM, Stack <st...@duboce.net> wrote:
> > > > >
> > > > > > This is an interesting vote. The vote is on an RC made out of a
> new
> > > > hbase
> > > > > > project, hbase-thirdparty.
> > > > > >
> > > > > > First the usual stuff.
> > > > > >
> > > > > > The 1.0.0RC0 is available at:
> > > > > >
> > > > > >  https://dist.apache.org/repos/dist/dev/hbase/hbase-
> > > > thirdparty/1.0.0RC0/
> > > > > >
> > > > > > Maven artifacts are in the following staging repository:
> > > > > >
> > > > > >  https://repository.apache.org/content/repositories/
> > > > orgapachehbase-1171
> > > > > >
> > > > > > Artifacts are signed with 8ACC93D2 which is at the tail of our
> KEYS
> > > > file
> > > > > > http://www-us.apache.org/dist/hbase/KEYS.
> > > > > >
> > > > > > I tagged this RC as 1.0.0RC at 8ffaf3dd561052bcc71772148ecd04
> > > > cdf9e224f3
> > > > > > in the new hbase-thirdparty repository at [1].
> > > > > >
> > > > > > Now to the unusual.
> > > > > >
> > > > > > The artifact was made from a new repository, hbase-thirdparty[1].
> > > > > > hbase-thirdparty is a project overseen by the hbase PMC. It came
> of
> > > > > > discussion on the dev list in the thread '[DISCUSS] More Shading'
> > > [2].
> > > > > > Its charge is the hosting of ornery thirdparty libs -- e.g.
> > protobuf,
> > > > > > guava, netty -- that need patching and/or shading so we are no
> > longer
> > > > > bound
> > > > > > by whatever the version of a lib Hadoop happens to ship (and we
> can
> > > > > change
> > > > > > versions w/o damaging out glorious downstreamers). In the past,
> > when
> > > > > we've
> > > > > > been able, we've done this dirty patching and shading work as an
> > > > > > unacknowledged 'pre-build' hack step appended to our protobuf
> file
> > > > > > generation. Moving this mess out of our mainline build to live
> > > > elsewhere,
> > > > > > pulled in as a legit 'dependency', simplifies our build
> > particularly
> > > > > around
> > > > > > the shading of third-part libs; the narrower scope of a dedicated
> > > > project
> > > > > > that produces transformed jars is easier to reason about, is less
> > > > likely
> > > > > to
> > > > > > clash with existing dependencies whether transitive or otherwise,
> > and
> > > > it
> > > > > > makes for less spaghetti poms.
> > > > > >
> > > > > > Other (minor) benefits include our being able to generate
> protobuf
> > > > files
> > > > > > inline now; no need of the pre-build anymore nor checking in of
> > > > generated
> > > > > > protobuf files (25MB).  There is a patch to remove them
> (predicated
> > > on
> > > > > this
> > > > > > hbase-thirdparty release): "HBASE-17056 Remove checked in PB
> > > generated
> > > > > > files".
> > > > > >
> > > > > > Downside is having to explicitly reference our version in code as
> > in
> > > > > > org.apache.hadoop.hbase.shaded.com.google.protobuf.Message
> rather
> > > than
> > > > > > just com.google.protobuf.Message and so on for guava references,
> > etc.
> > > > > > (We've been doing this for a while now so hopefully it not too
> > > strange
> > > > > > developing)*.
> > > > > >
> > > > > > There is little to this new project other than a few poms to do
> > > > packaging
> > > > > > and a couple of patches we apply on build. It works by pulling
> down
> > > > > > dependencies and doing whatever shading, renaming, or patching we
> > > need.
> > > > > > Currently hbase-thirdparty hosts protobuf (3.1.0 => 3.3.1), guava
> > > (12.0
> > > > > =>
> > > > > > 22.0), gson, protobuf-util, and netty 4.1.12-FINAL. It publishes
> > > three
> > > > > jars
> > > > > > for hbase to consume: hbase-shaded-protobuf, hbase-shaded-netty,
> > and
> > > > then
> > > > > > hbase-shaded-miscellaneous for the remaining libs. Philosophy is
> if
> > > the
> > > > > lib
> > > > > > is troublesome, protobuf needs patching and netty includes an .so
> > > that
> > > > > > needs a rename, then give it its own module (See README in the
> > > release
> > > > > for
> > > > > > more). I've arrived at this layout for hbase-thirdparty after a
> > bunch
> > > > of
> > > > > > experimentation and messing with hbase dependencies (See
> sub-tasks
> > > > under
> > > > > > HBASE-17898 Update dependencies). That said, as we head to
> > > hbase-2.0.0,
> > > > > we
> > > > > > might have to make a an odd new release of hbase-thirdparty but
> > > > > thereafter
> > > > > > I'd imagine it should settle down to be an infrequent update.
> > > > > >
> > > > > > VOTE lasts 72 hours. Let +1 be an assent to the hbase-thirdparty
> > > > project
> > > > > > and a vote for this RC.
> > > > > >
> > > > > > Questions on how it will go w/ this project welcome on this
> thread.
> > > If
> > > > > > heavy-detail on how the shading works, I suggest ask on the
> > > '[DISCUSS]
> > > > > More
> > > > > > Shading' thread.
> > > > > >
> > > > > > Thanks,
> > > > > > St.Ack
> > > > > > 1. https://git-wip-us.apache.org/repos/asf/hbase-thirdparty
> > > > > > 2. See the '[DISCUSS] More Shading' thread,
> > > > > http://apache-hbase.679495.n3.
> > > > > > nabble.com/DISCUSS-More-Shading-td4083025.html
> > > > > >
> > > > > > * Doc on how to make use of hbase-thirdparty is coming for the
> > > > refguide.
> > > > > >
> > > > > >
> > > > > >
> > > > >
> > > >
> > >
> >
>

Reply via email to