Thanks I'll take a look at both. Just to be sure we are talking about client version 0.82 right?
On Fri, Oct 24, 2014 at 8:39 AM, Guozhang Wang <wangg...@gmail.com> wrote: > Rajiv, > > The new producer does maintain a buffer per partition, but you need to > consider synchronizing the access to the buffer since it can take data from > multiple caller threads. I think Jay's suggestion 1) does the same thing > for your purpose if you already have the data buffer storing your data: by > creating a ProduceRecord it would not incur copying the data into a > temporary buffer, but instead mark the offset / length of in the byte > buffer that user specifies, the when producer.send() is called the > underlying buffer is copied to the producer buffer. If you do not have the > data buffer but just want to write directly to the producer buffer that is > one step further. > > As for the consumer, you can take a look at the MemoryRecord's iterator > implementation, which I think already implements what you want. > > Guozhang > > On Thu, Oct 23, 2014 at 6:13 PM, Rajiv Kurian <ra...@signalfuse.com> > wrote: > > > I want to avoid allocations since I am using Java in a C mode. Even > though > > creating objects is a mere thread local pointer bump in Java, freeing > them > > is not so cheap and causes uncontrollable jitter. The second motivation > is > > to avoid copying of data. Since I have objects which really look like C > > structs that can be sent over the wire it's most efficient for me to > write > > them out in the very exact buffer that will be sent over the wire. > > > > As for the bad API I completely agree - it is a very C style API and > > definitely not usable in a productive way by most developers. My point > was > > that this work is done by the protocol handling layer in any case, maybe > it > > can be extended to allow a user access to it's internals in a safe way > both > > during writing and reading. The present API then can be written as a > layer > > over this "ugly" non allocating API. > > > > Re (1) and (2). Instead of giving out keys, values as bytes which implies > > copies, I'd ideally like to scribble them straight into the buffer that > you > > are accumulating data onto before sending it. I am guessing you already > > need a single buffer per partition or you have a single buffer per > broker. > > All of this probably implies a single threaded producer where I can be in > > charge of the event loop. > > > > Right now my data is within ByteBuffer/Unsafe buffer based data > structures. > > They can be put on the wire without any serialization step if I was using > > Java NIO. Similarly they can be consumed on the other side without any > > deserialization step. But with the current kafka API I have to: > > i) Copy data from my ByteBuffers onto new byte arrays. > > ii) Wrap byte arrays from (i) in a new object. I can't even re-use this > > object since I don't know when kafka's send thread/serialization thread > is > > really done with it. > > iii) Write an encoder that just takes the byte array from this wrapper > > object and hands it to Kafka. > > > > Similarly on the consumer: > > i) Kafka will make copies of slices (representing user values) of the > > ByteBuffer that was transferred from a broker into byte arrays. > > ii) Allocate an object (using the decoder) that wraps these byte arrays > > and hand them to me. > > > > My imaginary (admittedly non-java-esque manual allocation style) API > would > > give me a pointer to Kafka's ByteBuffer that it has been accumulating > > protocol messages on for either writing (on producer) or reading (on > > consumer). I know it's a long shot but I still wanted to get the team's > > thoughts on it. I'd be happy to contribute if we can come to an agreement > > on the API design. My hypothesis is that if the internal protocol parsing > > and buffer creation logic is written like this, it wouldn't be too tough > to > > expose it's innards and have the current encoding/decoding APIs just use > > this low level API/ > > > > Thanks for listening to my rant. > > > > > > On Thu, Oct 23, 2014 at 5:19 PM, Jay Kreps <jay.kr...@gmail.com> wrote: > > > > > It sounds like you are primarily interested in optimizing the producer? > > > > > > There is no way to produce data without any allocation being done and I > > > think getting to that would be pretty hard and lead to bad apis, but > > > avoiding memory allocation entirely shouldn't be necessary. Small > > transient > > > objects in java are pretty cheap to allocate and deallocate. The new > > Kafka > > > producer API that is on trunk and will be in 0.8.2 is much more > > disciplined > > > in it's usage of memory though there is still some allocation. The goal > > is > > > to avoid copying the *data* multiple times, even if we do end up > creating > > > some small helper objects along the way (the idea is that the data may > be > > > rather large). > > > > > > If you wanted to further optimize the new producer there are two things > > > that could be done that would help: > > > 1. Avoid the copy when creating the ProducerRecord instance. This could > > be > > > done by accepting a length/offset along with the key and value and > making > > > use of this when writing to the records instance. As it is your key and > > > value need to be complete byte arrays. > > > 2. Avoid the copy during request serialization. This is a little > > trickier. > > > During request serialization we need to take the records for each > > partition > > > and create a request that contains all of them. It is possible to do > this > > > with no further recopying of data but somewhat tricky. > > > > > > My recommendation would be to try the new producer api and see how that > > > goes. If you need to optimize further we would definitely take patches > > for > > > (1) and (2). > > > > > > -Jay > > > > > > On Thu, Oct 23, 2014 at 4:03 PM, Rajiv Kurian <ra...@signalfuse.com> > > > wrote: > > > > > > > I have a flyweight style protocol that I use for my messages. Thus > they > > > > require no serialization/deserialization to be processed. The > messages > > > are > > > > just offset, length pairs within a ByteBuffer. > > > > > > > > Is there a producer and consumer API that forgoes allocation? I just > > want > > > > to give the kakfa producer offsets from a ByteBuffer. Similarly it > > would > > > be > > > > ideal if I could get a ByteBuffer and offsets into it from the > > consumer. > > > > Even if I could get byte arrays (implying a copy but no decoding > phase) > > > on > > > > the consumer that would be great. Right now it seems to me that the > > only > > > > way to get messages from Kafka is through a message object, which > > implies > > > > Kafka allocates these messages all the time. I am willing to use the > > > > upcoming 0.9 API too. > > > > > > > > Thanks. > > > > > > > > > > > > > -- > -- Guozhang >
