Re: Configuring client-side timeouts in the Java Producer so that send() doesn't block

2016-05-18 Thread Samuel Chase
Hello Ismael, On Wed, May 18, 2016 at 5:54 PM, Ismael Juma wrote: > Your second example should work as well. Can you please include the code > you are using to test the scenario and what Kafka version you are using > (0.9.0.0 or 0.9.0.1, I guess)? This is the code I'm using: ``` (with-open [p (

Configuring client-side timeouts in the Java Producer so that send() doesn't block

2016-05-18 Thread Samuel Chase
Hello Kafka Users, We've been using the old Scala Producer for quite some time now. In the Scala Producer, the send() call throws an exception when Kafka is down. We catch this exception, and perform error handling by spooling the message locally, which another thread consumes and retries with an

Re: What is the expected way to deal with out of space on brokers

2015-04-06 Thread Samuel Chase
@Pradeep, On Tue, Apr 7, 2015 at 9:30 AM, wrote: > I tried lot to unsubscribe from this list, even I send several e-mails. > Please unsubscribe from this list, since mailbox is getting filled. See here: http://kafka.apache.org/contact.html for instructions on how to unsubscribe from the ML.

Re: New Java Producer Client handling case where Kafka is unreachable

2015-03-21 Thread Samuel Chase
@Ewen On Fri, Mar 20, 2015 at 10:04 PM, Ewen Cheslack-Postava wrote: > Even if you have metadata cached, if the broker isn't available then > messages can get stuck in the producer indefinitely. Currently the new > producer doesn't have any client-side timeouts, which is a bug. See > https://issu

Re: New Java Producer Client handling case where Kafka is unreachable

2015-03-21 Thread Samuel Chase
@Mayuresh On Fri, Mar 20, 2015 at 10:30 PM, Mayuresh Gharat wrote: > But if the entire kafka cluster is down, it would try for some configured > number of retries and would return back an error in future. This is my > understanding. Please correct me if I am wrong. When I set the "retries" optio

New Java Producer Client handling unreachable Kafka

2015-03-20 Thread Samuel Chase
Hello Everyone, In the the new Java Producer API, the Callback code in KafkaProducer.send is run after there is a response from the Kafka server. This can be used if some error handling needs to be done based on the response. When using the new Java Kafka Producer, I've noticed that when the Kafk

Re: New Java Producer Client handling case where Kafka is unreachable

2015-03-20 Thread Samuel Chase
On Fri, Mar 20, 2015 at 3:43 PM, Samuel Chase wrote: > >> @Sunil >> >> On Fri, Mar 20, 2015 at 3:36 PM, sunil kalva wrote: >> > I think KafkaProducer.send method blocks until it fetches partition >> > metadata for configured time using "metadata.fetc

Re: New Java Producer Client handling case where Kafka is unreachable

2015-03-20 Thread Samuel Chase
@Sunil On Fri, Mar 20, 2015 at 3:36 PM, sunil kalva wrote: > I think KafkaProducer.send method blocks until it fetches partition > metadata for configured time using "metadata.fetch.timeout.ms", once time > out it throws TimeoutException. You might be experiencing TimeoutException ? My co-worker

Re: New Java Producer Client handling case where Kafka is unreachable

2015-03-20 Thread Samuel Chase
On Fri, Mar 20, 2015 at 3:24 PM, tao xiao wrote: > The underlining send runs in a different thread and doesn't block > producer.send(). One way I can think of to detect this is to set > block.on.buffer.full=false and catch BufferExhaustedException then check if > the broker is reachable. But this

Re: New Java Producer Client handling case where Kafka is unreachable

2015-03-20 Thread Samuel Chase
@Tao, On Fri, Mar 20, 2015 at 12:39 PM, tao xiao wrote: > You can set producer property retries not equal to 0. Details can be found > here > http://kafka.apache.org/documentation.html#newproducerconfigs I set "retries" to "1", but send is still blocking until the Kafka Server is reachable again

Re: New Java Producer Client handling case where Kafka is unreachable

2015-03-20 Thread Samuel Chase
Hello Tao, On Fri, Mar 20, 2015 at 12:39 PM, tao xiao wrote: > You can set producer property retries not equal to 0. Details can be found > here > http://kafka.apache.org/documentation.html#newproducerconfigs Thanks! I shall try that. Samuel

New Java Producer Client handling case where Kafka is unreachable

2015-03-20 Thread Samuel Chase
Hello Everyone, In the the new Java Producer API, the Callback code in KafkaProducer.send is run after there is a response from the Kafka server. This can be used if some error handling needs to be done based on the response. When using the new Java Kafka Producer, I've noticed that when the Kafk