It looks like flatbuffers breaks binary compat. If we are breaking binary compat then we can look at things other than protobuf descendants.
On Tue, Feb 2, 2016 at 1:14 PM, Stack <st...@duboce.net> wrote: > On Tue, Feb 2, 2016 at 12:16 PM, Enis Söztutar <enis....@gmail.com> wrote: > > > bq. java is second class in flatbuffers). > > This is an Android project, no? I thought Java is the primary target. > > > > I was looking at this Enis: > https://google.github.io/flatbuffers/flatbuffers_support.html > St.Ack > > > > > On Tue, Feb 2, 2016 at 12:14 PM, Stack <st...@duboce.net> wrote: > > > > > On Tue, Feb 2, 2016 at 11:47 AM, Enis Söztutar <enis....@gmail.com> > > wrote: > > > > > > > BTW, we should also be looking at > > https://google.github.io/flatbuffers/ > > > or > > > > https://capnproto.org/ for serialization as an option. The idea is > to > > > not > > > > allocate objects and prevent allocations altogether. > > > > > > > > > > > Agree (watching capnproto w/ a while, the java implementation has > > stalled. > > > java is second class in flatbuffers). > > > St.Ack > > > > > > > > > > > > > We are allocating PB objects for every Get / Put, then we allocate > our > > > Get > > > > / Put objects. At least we can save on 1. > > > > > > > > Although, just switching our serialization format again will be a > huge > > > > undertaking with obvious wire-incompatibility issues. If PB3 or 2.x > > gives > > > > us what we want in terms of preventing big byte[] allocations, we > would > > > > gain regardless. > > > > > > > > Enis > > > > > > > > On Tue, Feb 2, 2016 at 11:41 AM, Enis Söztutar <enis....@gmail.com> > > > wrote: > > > > > > > > > Google guys over at > > > > > > https://github.com/grpc/grpc-java/issues/1054#issuecomment-147295224 > > > are > > > > > saying that CIS changes may be coming to 2.x from what I > understand. > > If > > > > so, > > > > > our life would be easier. Even so, I'm 100% sure we have to do > > shading > > > > > since Hadoop will not change it's PB dependency anytime soon. > > > > > > > > > > We have to do this before doing shading: > > > > > https://issues.apache.org/jira/browse/HBASE-15174 > > > > > > > > > > Enis > > > > > > > > > > On Tue, Feb 2, 2016 at 8:15 AM, Stack <st...@duboce.net> wrote: > > > > > > > > > >> Thanks Duo. If proto3 had what we wanted, you are suggesting we > > might > > > > move > > > > >> to proto3 setting it to do proto2 support and shade it so we don't > > > clash > > > > >> with other includes of pb? > > > > >> > > > > >> Regards Anoop comment, the note on the end of this issue looks > > > promising > > > > >> but I don't know when it'd see the light of day: > > > > >> > > https://github.com/grpc/grpc-java/issues/1054#issuecomment-147295224 > > > > >> > > > > >> St.Ack > > > > >> > > > > >> > > > > >> On Mon, Feb 1, 2016 at 10:49 PM, Anoop John < > anoop.hb...@gmail.com> > > > > >> wrote: > > > > >> > > > > >> > UnsafeByteStrings - This may help us to avoid copy even with out > > our > > > > >> > HBaseZeroCopyByteString stuff. But with a DirectByteBuffer, it > > has > > > to > > > > >> copy > > > > >> > data to onheap byte[]. We even want a DBB backing ! > > > > >> > > > > > >> > -Anoop- > > > > >> > > > > > >> > On Tue, Feb 2, 2016 at 12:07 PM, 张铎 <palomino...@gmail.com> > > wrote: > > > > >> > > > > > >> > > https://groups.google.com/forum/#!topic/protobuf/wAqvtPLBsE8 > > > > >> > > > > > > >> > > PB2 and PB3 are wire compatible, and of course, protobuf-java > is > > > not > > > > >> > > compatible so dependency will be a problem... But I think the > > > shaded > > > > >> > client > > > > >> > > and server can solve the problem? > > > > >> > > > > > > >> > > Thanks. > > > > >> > > > > > > >> > > 2016-02-02 14:27 GMT+08:00 Stack <st...@duboce.net>: > > > > >> > > > > > > >> > > > We are running into a few issues with protobufs. > > > > >> > > > > > > > >> > > > + PB always copies all data before making a Message. This > > > > generates > > > > >> > > garbage > > > > >> > > > unnecessarily. > > > > >> > > > + CodedInputStream does not support ByteBuffers in 2.5. In > 2.6 > > > it > > > > >> does > > > > >> > > but > > > > >> > > > again, copies the data out of the BB always; this is > > especially > > > > >> painful > > > > >> > > > when the BB is a DBB with its data offheap and intent is to > > keep > > > > >> data > > > > >> > > > offheap. > > > > >> > > > > > > > >> > > > There are other issues. CIS allocates 4k buffers regardless > > (See > > > > >> > > > HBASE-15177). > > > > >> > > > And then there was the HBaseZeroCopyByteString fun and games > > we > > > > had > > > > >> a > > > > >> > > while > > > > >> > > > back. > > > > >> > > > > > > > >> > > > 3.0 PB adds UnsafeByteStrings so can do zero copy. Thats > good. > > > But > > > > >> PB3 > > > > >> > is > > > > >> > > > incompatible with PB2 (or at least, it looks like PB2 > clients > > > > can't > > > > >> > talk > > > > >> > > to > > > > >> > > > PB3 [1]). > > > > >> > > > > > > > >> > > > There is javanano protobufs. All is open access, but it too > > > looks > > > > >> > > different > > > > >> > > > to PB2 (i've not tried it). > > > > >> > > > > > > > >> > > > Protostuff seems really quiet these times [2]. > > > > >> > > > > > > > >> > > > Fork (and shade)? > > > > >> > > > > > > > >> > > > Thoughts? > > > > >> > > > > > > > >> > > > St.Ack > > > > >> > > > > > > > >> > > > 1. https://github.com/google/protobuf/releases > > > > >> > > > 2. https://groups.google.com/forum/#!forum/protostuff > > > > >> > > > > > > > >> > > > > > > >> > > > > > >> > > > > > > > > > > > > > > > > > > > >
