Ivan, Yes, this proposal is about the binary protocol - the way we serialize primitive values, the foundation for the thin client protocol.
Key-value API still has some uncertainty around it, we'll discuss it separately. On Wed, Jun 23, 2021 at 2:31 PM Ivan Daschinsky <ivanda...@gmail.com> wrote: > Pavel, let me clarify one thing. > > 1. If this proposal is about binary protocol, then there is no objection I > suppose. > 2. If this proposal about serialization of key-value, there are some > uncertainties, especially about complex objects. In this case this proposal > needs more work. > > ср, 23 июн. 2021 г. в 14:07, Pavel Tupitsyn <ptupit...@apache.org>: > > > Igniters, > > > > Looks like there are no objections and we can accept the proposal. > > I will close it tomorrow and move on to the thin client protocol itself. > > > > On Fri, Jun 18, 2021 at 12:10 PM Ivan Daschinsky <ivanda...@gmail.com> > > wrote: > > > > > >> To make it fair. Ignite uses thread-local reusable buffers, see [1]. > > > I know, but PooledMessageBufferOutput is not about thread-local, isn't > > it? > > > > > > I'm not against about MsgPack, I'm for fair and not biased comparison. > > > > > > I suppose that MsgPack is an ideal candidate for thin client binary > > > protocol, not only for serializing some user data. > > > I am analyzed some tarantool connectors [1] [2] [3] and found msgpack > > based > > > protocol is a really good idea. > > > Also there is realy super fast and just 1 header library with liberal > > BSD-2 > > > licence for C -- msgpuck [4]. It used in tarantool itself > > > and in [1] and is stable and bullet proof. > > > > > > [1] -- https://github.com/igorcoding/asynctnt > > > [2] -- https://github.com/tarantool/tarantool-python/ > > > [3] -- https://github.com/tarantool/go-tarantool > > > [4] -- https://github.com/rtsisyk/msgpuck > > > > > > пт, 18 июн. 2021 г. в 11:44, Pavel Tupitsyn <ptupit...@apache.org>: > > > > > > > Ivan, > > > > > > > > > why do you use PooledMessageBufferOutput in benchmarks? > > > > > > > > To make it fair. Ignite uses thread-local reusable buffers, see [1]. > > > > > > > > > > > > > why packer from msgpack-core show better performance than > > > > > BinaryWriter. And I suppose that benchmark is not quite fair. > > > > > > > > MsgPack writes and reads less bytes, so it should be faster. > > > > Benchmark is not 100% fair, there are some small extra things that > > > > BinaryWriter does. > > > > > > > > However: > > > > 1. I don't think we care about super-precise benchmarks here, just > the > > > > ballpark. > > > > 2. We are discussing the format, not the implementation. > > > > > > > > Important takeaway is: > > > > The format does not prevent someone from implementing it efficiently. > > > > > > > > > > > > > > > > [1] > > > > > > > > > > > > > > https://github.com/apache/ignite/blob/master/modules/core/src/main/java/org/apache/ignite/internal/binary/BinaryWriterExImpl.java#L101 > > > > > > > > On Thu, Jun 17, 2021 at 10:40 PM Ivan Daschinsky < > ivanda...@gmail.com> > > > > wrote: > > > > > > > > > Andrey, here we discuss serialization format, as far as I > understand. > > > > > Current implementation of ignite binary object serialization can be > > > > > rewritten. > > > > > If we do not care about fast (O(1)) field lookup, about schema > > > validation > > > > > and so on, msgpack is a really good option. It is also good for > > client > > > > > binary protocol, i.e. > > > > > tarantool uses it. > > > > > > > > > > >> Binarilizable interface forces user to write serialization code > > > > > I am talking about speed comparison. You can see from Pavel's data, > > > > > jackson-msgpack shows a pathetic performance comparing with a > > ignite's > > > > > default binary marshaller. If you want really fast serialization -- > > the > > > > > only option is to write code by yourself or use code generation. > > > Default > > > > > packer from msgpack-core java package is similar to BinaryWriter. > So > > I > > > am > > > > > wondering why packer from msgpack-core show better performance than > > > > > BinaryWriter. And I suppose that benchmark is not quite fair. > > > > > > > > > > > > > > > чт, 17 июн. 2021 г. в 22:19, Andrey Mashenkov < > > > > andrey.mashen...@gmail.com > > > > > >: > > > > > > > > > > > Ivan, thankd for clarification. > > > > > > > > > > > > Binarilizable interface forces user to write serialization code. > We > > > can > > > > > > support this or similar interface. > > > > > > But I'd like Ignite has some default serializer in addition. It > can > > > be > > > > > also > > > > > > useful e.g. in compute for param and result serialization. > > > > > > > > > > > > BinaryObjectBuider requires an Ignite node for object > construction, > > > but > > > > > we > > > > > > are looking for a detached builder and won't care about schemas. > > > > > > > > > > > > AFAIR, BinaryObject creates an objectReader on every single field > > > read > > > > > > operation. > > > > > > So, BO solution produces a lot of garbage and BO has noticable > > > overhead > > > > > > which affects the object footprint. > > > > > > > > > > > > чт, 17 июн. 2021 г., 21:41 Ivan Daschinsky <ivanda...@gmail.com > >: > > > > > > > > > > > > > >> Double checked -- there is not any links to PR either in IEP > > or > > > in > > > > > > jira > > > > > > > issue > > > > > > > Sorry, there is a link in IEP, but not in jira ticket. > > > > > > > > > > > > > > чт, 17 июн. 2021 г. в 21:39, Ivan Daschinsky < > > ivanda...@gmail.com > > > >: > > > > > > > > > > > > > > > Andrey, > > > > > > > > >> arbitrary object graph > > > > > > > > Also, that is not true, msgpack format doesn't handle > circular > > > > > graphs. > > > > > > > > Think about msgpack as binary json. You couldn't understand > > full > > > > > > > structure > > > > > > > > of message if you didn't deserialize it fully before, maps > and > > > > arrays > > > > > > are > > > > > > > > serialized just as contiguos chunks > > > > > > > > of values/kv-pairs. Msgpack is a really dumb and simple > > format. > > > > > > > > > > > > > > > > Also, as for me, I cannot understand why current ignite > > > > serialization > > > > > > > > (BinaryObjectBuilder or Binarilizable) is slower than raw > > message > > > > > pack > > > > > > > > serializer. > > > > > > > > I suppose that this is an issue and we should investigate it. > > > > > > > > > > > > > > > > Pavel, why do you use PooledMessageBufferOutput in > > benchmarks? > > > > I'm > > > > > > > > sorry, but is it fair to use it? > > > > > > > > > > > > > > > > >> The code is linked in the IEP [2] > > > > > > > > Double checked -- there is not any links to PR either in IEP > or > > > in > > > > > jira > > > > > > > > issue > > > > > > > > > > > > > > > > > > > > > > > > > > > > > -- > > > > > > > Sincerely yours, Ivan Daschinskiy > > > > > > > > > > > > > > > > > > > > > > > > > > > > -- > > > > > Sincerely yours, Ivan Daschinskiy > > > > > > > > > > > > > > > > > > -- > > > Sincerely yours, Ivan Daschinskiy > > > > > > > > -- > Sincerely yours, Ivan Daschinskiy >
