I think we should also consider tweaking the semantics of batch.size so that the sent batches can be larger if the batch is not ready to be sent (while still respecting max.request.size and perhaps a new max.batch.size).
Ismael On Sun, Oct 17, 2021, 12:08 PM Ismael Juma <ism...@juma.me.uk> wrote: > Hi Luke, > > Thanks for the KIP. Why do we have to reallocate the buffer? We can keep a > list of buffers instead and avoid reallocation. > > Ismael > > On Sun, Oct 17, 2021, 2:02 AM Luke Chen <show...@gmail.com> wrote: > >> Hi Kafka dev, >> I'd like to start the discussion for the proposal: KIP-782: Expandable >> batch size in producer. >> >> The main purpose for this KIP is to have better memory usage in producer, >> and also save users from the dilemma while setting the batch size >> configuration. After this KIP, users can set a higher batch.size without >> worries, and of course, with an appropriate "batch.initial.size" and >> "batch.reallocation.factor". >> >> Derailed description can be found here: >> >> https://cwiki.apache.org/confluence/display/KAFKA/KIP-782%3A+Expandable+batch+size+in+producer >> >> Any comments and feedback are welcome. >> >> Thank you. >> Luke >> >
