So to retrieve the message that was not transmitted due to error, I will have to store the message in each call back the message and store this in a file. Is there an alternative solution ?
What is best way to achieve QOS on producer side ? Also, will new Producer API will work with Broker version 0.8.0 ? Please let me know. Thanks, Bhavesh On Tue, May 6, 2014 at 8:46 AM, Jun Rao <jun...@gmail.com> wrote: > The callback will be invoked on completion. The exception tells you whether > the send completed or not. This api will be available in the next release. > > Thanks, > > Jun > > > On Mon, May 5, 2014 at 10:03 PM, Bhavesh Mistry > <mistry.p.bhav...@gmail.com>wrote: > > > Thanks for answers. > > > > Does the callback get call on failure only or for success as well ? > Also, > > how do I do this on Kafka 0.8.0 ? Is there any plan for adding > buffering > > on disk for next version ? Also, when application restart, kafka > producer > > will have to transmit the messages (buffered messages on disk) ? > > > > I can not upgrade to Kafka 0.8.1.1 right now. how can I achieve this on > > Kafka version 0.8.0. Also, when retransmitting the messages I wanted to > > transfer dedicated Kafka back-up or reserved partition for topic > (near-real > > time topic). > > > > How would you handle retransmission of messages ? > > > > Any idea about implementation will be very helpful. > > > > Thanks, > > > > Bhavesh > > > > On Mon, May 5, 2014 at 9:29 PM, Jun Rao <jun...@gmail.com> wrote: > > > > > At LinkedIn, we only buffer messages in memory in the producer. We try > as > > > much as we can to make the Kafka cluster always available (with > > > replication). The "on failure" feature you mentioned can be implemented > > by > > > providing a callback. You can see the example in the javadoc. > > > > > > *send > > > < > > > > > > http://empathybox.com/kafka-javadoc/kafka/clients/producer/KafkaProducer.html#send(kafka.clients.producer.ProducerRecord > > > , > > > kafka.clients.producer.Callback)>*(ProducerRecord< > > > > > > http://empathybox.com/kafka-javadoc/kafka/clients/producer/ProducerRecord.html > > > > > > > record, Callback< > > > > http://empathybox.com/kafka-javadoc/kafka/clients/producer/Callback.html > > > > > > callback) > > > > > > Thanks, > > > > > > Jun > > > > > > On Mon, May 5, 2014 at 6:30 PM, Bhavesh Mistry > > > <mistry.p.bhav...@gmail.com>wrote: > > > > > > > We are using Kafka for operation metrics and we do not want to loose > > any > > > > data at all if there is issue with Network or all brokers needs to be > > > > rebooted or operation reason down time while all producers are still > > > > producing data on front end side. We use async to publish messages > and > > > we > > > > are using Kafka version 0.8.0. > > > > > > > > > > > > > > > > Has any one implemented buffering on local disk (on producer side) > and > > > > transmit messages when network connection is restored? How do I get > > > > handle to list of messages async thread could not transfer after x > > reties > > > > ? I know new producer API has callback interface, but is per message > > not > > > > per producer instance ? > > > > > > > > > > > > > > > > Is this final new Producer API ? > > > > > > > > > > > > > > http://empathybox.com/kafka-javadoc/index.html?kafka/clients/producer/KafkaProducer.html > > > > > > > > > > > > is there a plan to add method like. onFailure(List<Messages> > messages, > > > > Exception exception ) ? > > > > > > > > > > > > > > > > > > > > Basically, I have to address the QOS on producer side, and be able to > > > > buffer on disk and retransmit all message to partitions that are > > reserved > > > > for messages that happened in past. > > > > > > > > > > > > > > > > How does Linked-in handle QOS on producer side ? > > > > > > > > > > > > Is there any plan to add this QOS feature on Producer Side with > > > Strategies > > > > to > > > > store and retransmit the message ? If we do get the list of messages > > is > > > > call back, will it be compressed data ? > > > > > > > > > > > > > > > > I would appreciate Kafka Developers and others feedback on how to > > > implement > > > > QOS. > > > > > > > > > > > > Thanks, > > > > > > > > > > > > Bhavesh > > > > > > > > > >
