[
https://issues.apache.org/jira/browse/HBASE-15638?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
stack updated HBASE-15638:
--------------------------
Description:
We need to change our protobuf. Currently it is pb2.5.0. As is, protobufs
expect all buffers to be on-heap byte arrays. It does not have facility for
dealing in ByteBuffers and off-heap ByteBuffers in particular. This fact
frustrates the off-heaping-of-the-write-path project as
marshalling/unmarshalling of protobufs involves a copy on-heap first.
So, we need to patch our protobuf so it supports off-heap ByteBuffers. To
ensure we pick up the patched protobuf always, we need to relocate/shade our
protobuf and adjust all protobuf references accordingly.
Given as we have protobufs in our public facing API, Coprocessor Endpoints --
which use protobuf Service to describe new API -- a blind relocation/shading of
com.google.protobuf.* will break our API for CoProcessor EndPoints (CPEP) in
particular. For example, in the Table Interface, to invoke a method on a
registered CPEP, we have:
<T extends com.google.protobuf.Service,R> Map<byte[],R> coprocessorService(
Class<T> service, byte[] startKey, byte[] endKey,
org.apache.hadoop.hbase.client.coprocessor.Batch.Call<T,R>
callable)
throws com.google.protobuf.ServiceException, Throwable
This issue is how we intend to shade protobuf for hbase-2.0.0 while preserving
our API as is so CPEPs continue to work on the new hbase.
was:
Shade protobufs so we can move to a different version without breaking the
world. We want to get up on pb3 because it has unsafe methods that allow us
save on copies; it also has some means of dealing with BBs so we can pass it
offheap DBBs. We'll probably want to change PB3 to open it up some more too so
we can stay offheap as we traverse PB. This issue comes of [~anoop.hbase] and
[~ram_krish]'s offheaping of the readpath work.
This change is mostly straight-forward but there are some tricky bits:
# How to interface with HDFS? It wants its ByteStrings. Here in particular in
FanOutOneBlockAsyncDFSOutputSaslHelper:
{code}
if (payload != null) {
builder.setPayload(ByteString.copyFrom(payload));
}
{code}
# [~busbey] also points out that we need to take care of endpoints done as pb.
Test at least.
Let me raise this one on the dev list too.
> Shade protobuf
> --------------
>
> Key: HBASE-15638
> URL: https://issues.apache.org/jira/browse/HBASE-15638
> Project: HBase
> Issue Type: Bug
> Components: Protobufs
> Reporter: stack
> Assignee: stack
> Priority: Critical
> Attachments: 15638v2.patch, HBASE-15638.master.001.patch,
> HBASE-15638.master.002.patch, HBASE-15638.master.003 (1).patch,
> HBASE-15638.master.003 (1).patch, HBASE-15638.master.003 (1).patch,
> HBASE-15638.master.003.patch, HBASE-15638.master.003.patch,
> HBASE-15638.master.004.patch, HBASE-15638.master.005.patch,
> HBASE-15638.master.006.patch, HBASE-15638.master.007.patch,
> HBASE-15638.master.007.patch, HBASE-15638.master.008.patch,
> HBASE-15638.master.009.patch, as.far.as.server.patch
>
>
> We need to change our protobuf. Currently it is pb2.5.0. As is, protobufs
> expect all buffers to be on-heap byte arrays. It does not have facility for
> dealing in ByteBuffers and off-heap ByteBuffers in particular. This fact
> frustrates the off-heaping-of-the-write-path project as
> marshalling/unmarshalling of protobufs involves a copy on-heap first.
> So, we need to patch our protobuf so it supports off-heap ByteBuffers. To
> ensure we pick up the patched protobuf always, we need to relocate/shade our
> protobuf and adjust all protobuf references accordingly.
> Given as we have protobufs in our public facing API, Coprocessor Endpoints --
> which use protobuf Service to describe new API -- a blind relocation/shading
> of com.google.protobuf.* will break our API for CoProcessor EndPoints (CPEP)
> in particular. For example, in the Table Interface, to invoke a method on a
> registered CPEP, we have:
> <T extends com.google.protobuf.Service,R> Map<byte[],R> coprocessorService(
> Class<T> service, byte[] startKey, byte[] endKey,
> org.apache.hadoop.hbase.client.coprocessor.Batch.Call<T,R>
> callable)
> throws com.google.protobuf.ServiceException, Throwable
> This issue is how we intend to shade protobuf for hbase-2.0.0 while
> preserving our API as is so CPEPs continue to work on the new hbase.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
