I intend to do a mass commit late this weekend that moves us on to a shaded protobuf-3.1.0, either Sunday night or Monday morning.
If objection, please speak up or if need more time for consideration/review, just shout. I want to merge the branch HBASE-16264 into master (it is running here up on jenkins https://builds.apache.org/view/H-L/view/HBase/job/HBASE-16264/). The branch at HBASE-16264 has three significant bodies-of-work that unfortunately are tangled and can only go in of a piece. * HBASE-16264 <https://issues.apache.org/jira/browse/HBASE-16264> The shading of our protobuf usage so we can upgrade and/or run with a patched protobuf WITHOUT breaking REST, Spark, and in particular, Coprocessor Endpoints. * HBASE-16567 <https://issues.apache.org/jira/browse/HBASE-16567> A move up on to (shaded) protobuf-3.1.0 * HBASE-16741 <https://issues.apache.org/jira/browse/HBASE-16741> An amendment of our generate protobufs step to include shading and a bundling of protobuf src (with a means of calling a patch srcs hook) Together we're talking about 40MB of change mostly made of the movement of generated files or the application of a pattern that alters where we get imports from. When done, you should notice no difference and should be able to go about your business as per usual. Upside is that we will be able to avoid coming onheap doing protobuf marshalling/unmarshalling as protobuf 2.5.0 requires. Downside is that we repeat a good portion of our internal protos, once non-shaded so Coprocessor Endpoints can keep working and then again as shaded for internal use. I provide some more overview below on the changes. See the shading doc here: https://docs.google.com/document/d/1H4NgLXQ9Y9KejwobddCqaVMEDCGbyDcXtdF5iAfDIEk/edit# for more detail (Patches are up on review board -- except the latest HBASE-16264 which is too big for JIRA and RB). I am currently working on a devs chapter for the book on protobuf going forward that will go in as part of this patch. Thanks, St.Ack Items of note: * Two new modules; one named hbase-protocol-shaded that is used by hbase core. It has in it a shaded (and later patched) protobuf. The other new module is hbase-endpoint which goes after hbase-server and has those bundled endpoints that I was able to break out of core (there are a few that are hopelessly entangled that need to be undone as CPEPs but fortunately belong in core: Auth, Access, MultiRow). * I've tested running a branch-1 CPEP against a master with these patches in place and stuff like ACL (A CPEP) run from the branch-1 shell work against the branch-2 server. On Mon, Aug 22, 2016 at 5:20 PM, Stack <st...@duboce.net> wrote: > This project goes on. I updated HBASE-1563 "Shade protobuf" with some doc > on a final approach. We need to be able to refer to both shaded and > non-shaded protobuf so we can support sending HDFS old-school pb Messages > but also so Coprocessor Endpoints keep working though internally protobufs > have been relocated. Funny you should ask, but yes, there are some > downsides (as predicted by contributors on the JIRA). I'd be interested to > hear if they are too burdensome. In particular, your IDE experience gets a > little convoluted as you will need to add to your build path, a jar with > the relocated pbs. A pain. > > Thanks, > St.Ack > > > On Wed, Apr 13, 2016 at 6:09 AM, Stack <st...@duboce.net> wrote: > >> On Tue, Apr 12, 2016 at 9:26 PM, Sean Busbey <bus...@apache.org> wrote: >> >>> On Tue, Apr 12, 2016 at 6:17 PM, Stack <st...@duboce.net> wrote: >>> > >>> > >>> > On an initial pass, the only difficult part seems to be interaction >>> with >>> > HDFS in asyncwal (might just pull in the HDFS messages). >>> > >>> > >>> >>> I have some idea how we can make this work either by pushing asyncwal >>> upstream to HDFS or through some maven tricks, depending on how much >>> time we have. >>> >> >> Maven tricks? Tell us more. Here or drop a note up in the issue. >> Thanks Sean, >> St.Ack >> > >
