Hello, I'm looking to insert log lines from log files into kafka, but I'm concerned with handling asynchronous send() failures. Specifically, if some of the log lines fail to send, I want to be notified of the failure so that I can attempt to resend them.
Based on previous threads on the mailing list (http://comments.gmane.org/gmane.comp.apache.kafka.user/1322), I know that the trunk version of kafka supports callbacks for dealing with failures. However, the callback function is not passed any metadata that can be used by the producer end to reference the original message. Including the key of the message in the RecordMetadata seems like it would be really useful for recovery purposes. Is anyone using the callback functionality to trigger resends of failed messages? If so, how are they tying the callbacks to messages? Is anyone using other methods for handling async errors/resending today? I can’t imagine that I am the only one trying to do this. I asked this question on the IRC channel today, and it sparked some discussion, but I wanted to hear from a wider audience. Thanks for the information, -Ryan
