BTW, we should also be looking at https://google.github.io/flatbuffers/ or https://capnproto.org/ for serialization as an option. The idea is to not allocate objects and prevent allocations altogether.
We are allocating PB objects for every Get / Put, then we allocate our Get / Put objects. At least we can save on 1. Although, just switching our serialization format again will be a huge undertaking with obvious wire-incompatibility issues. If PB3 or 2.x gives us what we want in terms of preventing big byte[] allocations, we would gain regardless. Enis On Tue, Feb 2, 2016 at 11:41 AM, Enis Söztutar <enis....@gmail.com> wrote: > Google guys over at > https://github.com/grpc/grpc-java/issues/1054#issuecomment-147295224 are > saying that CIS changes may be coming to 2.x from what I understand. If so, > our life would be easier. Even so, I'm 100% sure we have to do shading > since Hadoop will not change it's PB dependency anytime soon. > > We have to do this before doing shading: > https://issues.apache.org/jira/browse/HBASE-15174 > > Enis > > On Tue, Feb 2, 2016 at 8:15 AM, Stack <st...@duboce.net> wrote: > >> Thanks Duo. If proto3 had what we wanted, you are suggesting we might move >> to proto3 setting it to do proto2 support and shade it so we don't clash >> with other includes of pb? >> >> Regards Anoop comment, the note on the end of this issue looks promising >> but I don't know when it'd see the light of day: >> https://github.com/grpc/grpc-java/issues/1054#issuecomment-147295224 >> >> St.Ack >> >> >> On Mon, Feb 1, 2016 at 10:49 PM, Anoop John <anoop.hb...@gmail.com> >> wrote: >> >> > UnsafeByteStrings - This may help us to avoid copy even with out our >> > HBaseZeroCopyByteString stuff. But with a DirectByteBuffer, it has to >> copy >> > data to onheap byte[]. We even want a DBB backing ! >> > >> > -Anoop- >> > >> > On Tue, Feb 2, 2016 at 12:07 PM, 张铎 <palomino...@gmail.com> wrote: >> > >> > > https://groups.google.com/forum/#!topic/protobuf/wAqvtPLBsE8 >> > > >> > > PB2 and PB3 are wire compatible, and of course, protobuf-java is not >> > > compatible so dependency will be a problem... But I think the shaded >> > client >> > > and server can solve the problem? >> > > >> > > Thanks. >> > > >> > > 2016-02-02 14:27 GMT+08:00 Stack <st...@duboce.net>: >> > > >> > > > We are running into a few issues with protobufs. >> > > > >> > > > + PB always copies all data before making a Message. This generates >> > > garbage >> > > > unnecessarily. >> > > > + CodedInputStream does not support ByteBuffers in 2.5. In 2.6 it >> does >> > > but >> > > > again, copies the data out of the BB always; this is especially >> painful >> > > > when the BB is a DBB with its data offheap and intent is to keep >> data >> > > > offheap. >> > > > >> > > > There are other issues. CIS allocates 4k buffers regardless (See >> > > > HBASE-15177). >> > > > And then there was the HBaseZeroCopyByteString fun and games we had >> a >> > > while >> > > > back. >> > > > >> > > > 3.0 PB adds UnsafeByteStrings so can do zero copy. Thats good. But >> PB3 >> > is >> > > > incompatible with PB2 (or at least, it looks like PB2 clients can't >> > talk >> > > to >> > > > PB3 [1]). >> > > > >> > > > There is javanano protobufs. All is open access, but it too looks >> > > different >> > > > to PB2 (i've not tried it). >> > > > >> > > > Protostuff seems really quiet these times [2]. >> > > > >> > > > Fork (and shade)? >> > > > >> > > > Thoughts? >> > > > >> > > > St.Ack >> > > > >> > > > 1. https://github.com/google/protobuf/releases >> > > > 2. https://groups.google.com/forum/#!forum/protostuff >> > > > >> > > >> > >> > >
