Hi Kefka Dev,
Please let me know if you guys agree with use case features presented on previous email. Thanks, Bhavesh On Fri, Jun 27, 2014 at 11:45 AM, Bhavesh Mistry <mistry.p.bhav...@gmail.com > wrote: > HI Steve, > > I am referring to version 0.8.0 and I would like to try in dev environment > unbounded queue to try to figure upper bound of the application during the > stress test and through put we want to achieve. > > Also, additional features I would like to add At Run-time on Producer side > : > > 1) Ability to change the tuning parameters at Run-time (eg Batch Size, > Queue size, enqueue time etc) because our app run on various private > cloud , physical box (Same app) so it would be great If Kafka allow tuning > parameter to be reset at runtime. > > 2) During the batch Async send, occasionally we get > MessageSizeTooLargeException, is it possible ensure from the producer side > that batch can not exceed configure limit, and if it does exceed the limit, > divide the batch into 1/2 and send it transparently unless single message > exceed the limit. > > Our use case is "transferring log lines", we can configure batch size for > optimal throughput but we do not want to loose data when batch size exceed > configured limit (2MB). It there any way we can introduce this batch > split strategy logic. > > 3) Purpose of injecting the Queue implementation, we want to preserve a > latest message in queue and remove the oldest ( Priority Queue or drop > message oldest message). Because by configuring the enqueue timeout, it > drops latest message. > > eg. we have some monitoring message that take greater priority than > regular "log Lines" etc... > > I hope this clarify the use cases. > > Thanks, > > Bhavesh > > > > On Wed, Jun 25, 2014 at 12:41 PM, Steve Morin <st...@stevemorin.com> > wrote: > >> Bhavesh, >> On item 1) you didn't say why you want to be able to change the queue >> implementation? >> -Steve >> >> >> On Tue, Jun 24, 2014 at 4:37 PM, Bhavesh Mistry < >> mistry.p.bhav...@gmail.com> >> wrote: >> >> > Hi Kafka Dev Teeam, >> > >> > >> > >> > >> > >> > I would like to request following features in Kafka Async Producers: >> > >> > >> > >> > 1) Ability to Inject the implementation of the Blocking Queue similar >> to >> > serialize and partition class etc… >> > >> > >> > >> > I would like to inject following LIB for queue implementation. Of >> course >> > wrapper upon Queue interface.. >> > >> > eg http://lmax-exchange.github.io/disruptor/ >> > >> > >> > >> > 2) Can we have configuration to send the data in Parallel to Brokers >> using >> > Pool Executor Service …etc or ability to inject Default Handlers in here >> > which will run on back ground threads >> > >> > >> > >> > >> > >> https://github.com/apache/kafka/blob/0.8.1/core/src/main/scala/kafka/producer/async/ProducerSendThread.scala >> > >> > >> > >> > eg: Our event contains a timestamp so it is ok to have data out of >> order >> > into partition due to parallel write to Brokers per partition... >> > >> > >> > We are looking to dump the messages parallel not build up queue while it >> > data is being send to brokers and the send thread will take time to >> > complete entire batch if we have round robin on Partitions ( eg message >> > count % number Of partitions). >> > >> > Please let me know if there is any alternative.. >> > >> > >> > Thanks, >> > Bhavesh >> > >> > >
