HI Steve, I am referring to version 0.8.0 and I would like to try in dev environment unbounded queue to try to figure upper bound of the application during the stress test and through put we want to achieve.
Also, additional features I would like to add At Run-time on Producer side : 1) Ability to change the tuning parameters at Run-time (eg Batch Size, Queue size, enqueue time etc) because our app run on various private cloud , physical box (Same app) so it would be great If Kafka allow tuning parameter to be reset at runtime. 2) During the batch Async send, occasionally we get MessageSizeTooLargeException, is it possible ensure from the producer side that batch can not exceed configure limit, and if it does exceed the limit, divide the batch into 1/2 and send it transparently unless single message exceed the limit. Our use case is "transferring log lines", we can configure batch size for optimal throughput but we do not want to loose data when batch size exceed configured limit (2MB). It there any way we can introduce this batch split strategy logic. 3) Purpose of injecting the Queue implementation, we want to preserve a latest message in queue and remove the oldest ( Priority Queue or drop message oldest message). Because by configuring the enqueue timeout, it drops latest message. eg. we have some monitoring message that take greater priority than regular "log Lines" etc... I hope this clarify the use cases. Thanks, Bhavesh On Wed, Jun 25, 2014 at 12:41 PM, Steve Morin <st...@stevemorin.com> wrote: > Bhavesh, > On item 1) you didn't say why you want to be able to change the queue > implementation? > -Steve > > > On Tue, Jun 24, 2014 at 4:37 PM, Bhavesh Mistry < > mistry.p.bhav...@gmail.com> > wrote: > > > Hi Kafka Dev Teeam, > > > > > > > > > > > > I would like to request following features in Kafka Async Producers: > > > > > > > > 1) Ability to Inject the implementation of the Blocking Queue similar > to > > serialize and partition class etc… > > > > > > > > I would like to inject following LIB for queue implementation. Of course > > wrapper upon Queue interface.. > > > > eg http://lmax-exchange.github.io/disruptor/ > > > > > > > > 2) Can we have configuration to send the data in Parallel to Brokers > using > > Pool Executor Service …etc or ability to inject Default Handlers in here > > which will run on back ground threads > > > > > > > > > > > https://github.com/apache/kafka/blob/0.8.1/core/src/main/scala/kafka/producer/async/ProducerSendThread.scala > > > > > > > > eg: Our event contains a timestamp so it is ok to have data out of order > > into partition due to parallel write to Brokers per partition... > > > > > > We are looking to dump the messages parallel not build up queue while it > > data is being send to brokers and the send thread will take time to > > complete entire batch if we have round robin on Partitions ( eg message > > count % number Of partitions). > > > > Please let me know if there is any alternative.. > > > > > > Thanks, > > Bhavesh > > >
