> On 3 Feb 2016, at 14:07, Adrian Nistor <anis...@redhat.com> wrote: > > Hi all, > > </snip> > > Re Protostream, it is just plain Protobuf folks! It's in no way a > customly-modified-encoding-loosely-based-on-protobuf. The only apparent > twist is our usage of Protobuf, which btw also follows Google's > recommendations (see [1]). All keys and values are wrapped into a struct > like this [2] (actually it is a union when looking closer). And for good > reason. Adding that metadata as a separate header at hot rod protocol > level would be another possibility, but really, having it stored in > cache with the key/value is much more appropriate. This type metadata > (the type name or type id) is needed for indexing too. And could also be > used by some smart cache stores to make some sense out of that blob and > map it to database columns or suchlike. And would also be needed by non > HR clients, like REST. I can go on .. :) The only downside I see is > space. Our caches are not homogeneous, unfortunately [let's not debate > now why], so the type info needs to be stored for each single key, value > instead of storing it once per cache.
^ I'm not sure Protostream is just plain Protobuf... If you look at [3], you see very specific encoding/decoding steps about the wrapped message. Could I just read Protostream stuff from a JS client with only a .proto file? Looking at [3], it seems like that's not the case and hence I think Protostream imposes certain type of encoding. It looks to me I'd have to encode/decode each basic primitive types manually in the JS client encoder to support Protostream, and only for custom types provided via a .proto file I can delegate to Protobuf itself. @Adrian, am I reading this properly? Cheers, [3] https://github.com/infinispan/protostream/blob/master/core/src/main/java/org/infinispan/protostream/WrappedMessage.java#L54 > > Adrian > > [1] > https://developers.google.com/protocol-buffers/docs/techniques#self-description > [2] > https://raw.githubusercontent.com/anistor/protostream/master/core/src/main/resources/org/infinispan/protostream/message-wrapping.proto > > On 02/03/2016 01:34 PM, Sanne Grinovero wrote: >> It sounds like a good idea, almost like a natural evolution, but to >> play devil's advocate I'll try to find some drawbacks for such a >> decision. >> >> One negative argument is overall complexity: there are many points in >> code in which one needs to consider that the encoding might be >> "something else". This isn't a new problem as we already support two >> modes, but we've soon how much more difficult it makes things and this >> is making things even a bit more complex. >> >> Another point which I don't like much is that people will have to >> reconfigure the server depending on specific needs of the clients; if >> we go this way there should be a way in the Hot Rod protocol to >> "negotiate encoding", so to avoid such configuration tasks for end >> users (at least when there's no need to actually deploy dependencies >> on the server to handle some custom encoder..). >> >> The other problem I see is that this severely complicates >> interoperability between different clients. Suppose two applications >> are being developed to use the Infinispan Server and decide to use two >> different encoders, I suspect that at some point they'll regret it >> when needing one to access some data from the other... not suggesting >> that this should be a good practice, but still it would be very >> inconvenient. >> >> Finally, tooling. We'll eventually need to work on better tooling and >> supporting multiple encoders would spread efforts thinly. >> >> That said, I'm not against the idea of toying with such options and >> make other encoders as an *experimental* feature: protobuf is the most >> suited choice but forever is a long time and we should not hinder >> research on better alternatives. >> >> Thanks, >> Sanne >> >> >> >> >> On 3 February 2016 at 10:38, Gustavo Fernandes <gust...@infinispan.org> >> wrote: >>> >>> On Mon, Jan 25, 2016 at 2:03 PM, Galder Zamarreño <gal...@redhat.com> wrote: >>>> Hi all, >>>> >>>> As I write the Javascript client for Hot Rod, and Vittorio writes the C++ >>>> client, the question the encoding of the byte arrays has popped up. >>>> >>>> The reason why encoding matters is mainly because of compatibility mode. >>>> How does a Hot Rod client know how it should transform something a REST >>>> client set? >>>> >>>> To be able to answer this question correctly, the Hot Rod client needs to >>>> know the type of the data. >>>> >>>> Although we could consider adding encoding information to the Hot Rod >>>> protocol long term, in the short term this question might already been >>>> answered by Protostream. >>> >>> >>> Compatibility between disparate clients should certainly be supported, but >>> what about a pluggable marshaller mechanism? Let's consider all usage >>> scenarios: >>> >>> * Only Java clients: in most cases, JBoss marshalling is adequate, no need >>> to involve protobuf or json and since JBoss Marshalling is default, no need >>> to configure anything. >>> >>> * Mix of Java and C++/C#/Python clients: the Protobuf encoding works >>> wonderfully for that. The client could be configured to use the protostream >>> marshaller, the same for the server. >>> >>> * Mix of Java, C++/C#/Python and REST clients: protobuf with json [1] >>> encoding is an interesting option. Same as above, a protostream marshaller >>> with json support could >>> be configured both on client and server. >>> >>> * Custom marshallers: consider FlatBuffers [2], for example, an interesting >>> new cross-platform serializer that allows to access serialized data without >>> de-serializing the whole payload, >>> and without generating any transient memory, and it optionally supports >>> JSON. It could be interesting to write a marshaller based on it and plug it >>> on Infinispan. >>> >>> Although we have a way of configuring the marshaller on client (via >>> RemoteCacheManager) and server (deployable jar), there'd be more work to do >>> in order make them really pluggable: >>> >>> - Some serialization formats like Protobuf and [2] require interface >>> description to be available on both client and server. This is not supported >>> for all marshallers, just for Protostream >>> - Remote query requires some metadata regarding which fields to index and >>> how, and this would need to be possible on every language regardless of the >>> marshalling. >>> Currently, this info can be encoded in the .proto file only, a custom >>> marshaller would not work. >>> - Currently remote query is settled on protobuf not only for the byte array >>> but for the the HotRod query request and response [3] >>> >>> Is having a pluggable serializer mechanism too far fetched, given the effort >>> already invested towards protobuf? >>> >>> Gustavo >>> >>> [1] https://developers.google.com/protocol-buffers/docs/proto3#json >>> [2] https://google.github.io/flatbuffers/ >>> [2] >>> https://code.facebook.com/posts/872547912839369/improving-facebook-s-performance-on-android-with-flatbuffers/ >>> [3] >>> https://github.com/infinispan/infinispan/blob/master/remote-query/remote-query-client/src/main/resources/org/infinispan/query/remote/client/query.proto >>> >>>> >>>> _______________________________________________ >>>> infinispan-dev mailing list >>>> infinispan-dev@lists.jboss.org >>>> https://lists.jboss.org/mailman/listinfo/infinispan-dev >>> >>> >>> _______________________________________________ >>> infinispan-dev mailing list >>> infinispan-dev@lists.jboss.org >>> https://lists.jboss.org/mailman/listinfo/infinispan-dev >> _______________________________________________ >> infinispan-dev mailing list >> infinispan-dev@lists.jboss.org >> https://lists.jboss.org/mailman/listinfo/infinispan-dev > > _______________________________________________ > infinispan-dev mailing list > infinispan-dev@lists.jboss.org > https://lists.jboss.org/mailman/listinfo/infinispan-dev _______________________________________________ infinispan-dev mailing list infinispan-dev@lists.jboss.org https://lists.jboss.org/mailman/listinfo/infinispan-dev