Hi all,
I have created KIP-286: producer.send() should not block on metadata
update. See
https://cwiki.apache.org/confluence/display/KAFKA/KIP-286%3A+producer.send%28%29+should+not+block+on+metadata+update
.
The KIP intends to improve user-experience of producer.send() when metadata
is not availa
Looks like per-topic queue is introduced.
In terms of memory consumption, how does the KIP allocate memory
between per-topic
queue and per-partition queue ?
Thanks
On Wed, Apr 11, 2018 at 8:50 PM, Dong Lin wrote:
> Hi all,
>
> I have created KIP-286: producer.send() should not block on metadat
Hey Ted,
Thanks for your comments. With the proposed solution in the KIP, the memory
is only allocated once for the given message, which is the same as the
existing implementation. The serialized message will be moved from
per-topic queue to per-partition queue without incurring additional memory
Thanks for the KIP, Dong.
In the current threading model, compression is done by the user threads,
therefore the producer sender thread can focus on IO. With the proposed
changes, does that mean the producer sender thread will have to do all the
compression as well? Would this become a performance
Dong:
Yes, that answers my question.
Thanks
On Thu, Apr 12, 2018 at 1:41 AM, Dong Lin wrote:
> Hey Ted,
>
> Thanks for your comments. With the proposed solution in the KIP, the memory
> is only allocated once for the given message, which is the same as the
> existing implementation. The seriali
Hey Becket,
Good point! Thanks for the comment.
I have updated the KIP to move the compression to user thread in the common
case. Basically user thread can be responsible for compressing and moving
messages from per-topic queue to per-partition queue once the metadata is
available. Only if IO thr
I am going to drop this KIP. Thinking about this more, the benefit of not
having to wait for metadata does not seem to worth the complexity added in
producer due to this KIP. Assuming that the Kafka cluster is available,
which should be the case, it should be fast to wait for the first metadata.
Af
