Re: Handling send failures with async producer

2014-08-26 Thread Jay Kreps
Also, Jonathan, to answer your question, the new producer on trunk is running in prod for some use cases at LinkedIn and can be used with any 0.8.x. version. -Jay On Tue, Aug 26, 2014 at 12:38 PM, Jonathan Weeks wrote: > I am interested in this very topic as well. Also, can the trunk version of

Re: Handling send failures with async producer

2014-08-26 Thread Jun Rao
When you create the callback, you can pass in the original message. Thanks, Jun On Tue, Aug 26, 2014 at 12:35 PM, Ryan Persaud wrote: > Hello, > > I'm looking to insert log lines from log files into kafka, but I'm > concerned with handling asynchronous send() failures. Specifically, if > som

Re: Handling send failures with async producer

2014-08-26 Thread Christian Csar
TLDR: I use one Callback per job I send to Kafka and include that sort of information by reference in the Callback instance. Our system is currently moving data from beanstalkd to Kafka due to historical reasons so we use the callback to either delete or release the message depending on success. T

Re: Handling send failures with async producer

2014-08-26 Thread Jonathan Weeks
I am interested in this very topic as well. Also, can the trunk version of the producer be used with an existing 0.8.1.1 broker installation, or does one need to wait for 0.8.2 (at least)? Thanks, -Jonathan On Aug 26, 2014, at 12:35 PM, Ryan Persaud wrote: > Hello, > > I'm looking to insert

Handling send failures with async producer

2014-08-26 Thread Ryan Persaud
Hello, I'm looking to insert log lines from log files into kafka, but I'm concerned with handling asynchronous send() failures. Specifically, if some of the log lines fail to send, I want to be notified of the failure so that I can attempt to resend them. Based on previous threads on the mail