Re: Getting timeouts with elastic load balancer in AWS

>From the log, it seems that while message 6 was being sent, it hit an exception (potentially due to broker down) and caused a resend. And it seems that message 6 reached the broker. When message 5 was sent, it didn't hit any exception. So there is no resend. The reason that message 5 didn't reach

Re: Getting timeouts with elastic load balancer in AWS

Just to remove all the variables regarding me restarting the broker, I did a test with Amazon ELB. (0.7.1 producer and 0.7.0 broker) Thus, no broker restarts. The connection was getting broken because Amazon ELB was closing all the connections. I found the exact same result. In spite of specifying

Re: Getting timeouts with elastic load balancer in AWS

Just to clarify: num.retries > 0 does not guarantee that all messages will be received at the broker. It guarantees retry on exceptions - so it cannot handle the corner case when the broker goes down after the message is written to the socket buffer but before the buffer is flushed (in which case n

Re: Getting timeouts with elastic load balancer in AWS

Jun, Here is the log with SynProducer and DefaultEventHandler trace enabled. http://pastebin.com/dTm5RSJ9 Here are my producer settings: properties.put("serializer.class", "kafka.serializer.StringEncoder") properties.put("broker.list", "0:localhost:9092") properties.put("producer.type", "async"

Re: Getting timeouts with elastic load balancer in AWS

Could you enable trace logging in DefaultEventHandler to see if the following message shows up after the warning? trace("kafka producer sent messages for topics %s to broker %s:%d (on attempt %d)" Thanks, Jun On Thu, Jun 28, 2012 at 10:44 AM, Vaibhav Puranik wrote: > Hi all, > > I don

Re: Getting timeouts with elastic load balancer in AWS

Hi all, I don't think the num.retries (0.7.1) is working. Here is how I tested it. I wrote a simple producer that sends messages with the following strings - "1_", "_2_". . As you can see all the messages are sequential. I tailed the topic log on broker. After sending every me

Re: Getting timeouts with elastic load balancer in AWS

0.7.1 has this: reconnect.time.interval.ms Thanks, Joel On Wed, Jun 27, 2012 at 3:31 PM, Vaibhav Puranik wrote: > That will be awesome. It will definitely address AWS ELB problem. > > +1 for "reconnect.interval". > > Regards, > Vaibhav > GumGum > > > On Wed, Jun 27, 2012 at 3:24 PM, Niek Sande

Re: Getting timeouts with elastic load balancer in AWS

That will be awesome. It will definitely address AWS ELB problem. +1 for "reconnect.interval". Regards, Vaibhav GumGum On Wed, Jun 27, 2012 at 3:24 PM, Niek Sanders wrote: > Do producers currently leave the sockets to the brokers open indefinitely? > > It might make sense to add a second produ

Re: Getting timeouts with elastic load balancer in AWS

Do producers currently leave the sockets to the brokers open indefinitely? It might make sense to add a second producer config param similar to "reconnect.interval" which limits on time instead of message count. (And then reconnect based on whichever criteria is hit first). For folks going throug

Re: Getting timeouts with elastic load balancer in AWS

Thanks Neha. I will try num.retries again with this version and post my feedback here. Regards, Vaibhav On Wed, Jun 27, 2012 at 3:13 PM, Neha Narkhede wrote: > You can download it from here - > > https://www.apache.org/dyn/closer.cgi/incubator/kafka/kafka-0.7.1-incubating/ > > Thanks, > Neha >

Re: Getting timeouts with elastic load balancer in AWS

You can download it from here - https://www.apache.org/dyn/closer.cgi/incubator/kafka/kafka-0.7.1-incubating/ Thanks, Neha On Wed, Jun 27, 2012 at 3:03 PM, Vaibhav Puranik wrote: > Thanks Jun. How do I download 0.7.1? > > I checked SVN tags but the last tag seems to be > kafka-0.7.1-incubating-c

Re: Getting timeouts with elastic load balancer in AWS

Thanks Jun. How do I download 0.7.1? I checked SVN tags but the last tag seems to be kafka-0.7.1-incubating-candidate-3/ Regards, Vaibhav On Wed, Jun 27, 2012 at 2:56 PM, Jun Rao wrote: > num.retries is a

Re: Getting timeouts with elastic load balancer in AWS

num.retries is added in 0.7.1, which is just out. Thanks, Jun On Wed, Jun 27, 2012 at 2:43 PM, Vaibhav Puranik wrote: > Jun, > > I wrote a test producer to test if num.retries working or not. But I found > that it's not working. No matter how many retries I set, whenever a > message send fail

Re: Getting timeouts with elastic load balancer in AWS

Vaibhav, >> No matter how many retries I set, whenever a message send fails, it always never gets to the broker. Please can you send across the error message that you see on the producer side ? Thanks, Neha On Wed, Jun 27, 2012 at 2:43 PM, Vaibhav Puranik wrote: > Jun, > > I wrote a test prod

Re: Getting timeouts with elastic load balancer in AWS

On 06/27/2012 05:43 PM, Vaibhav Puranik wrote: > Is this a known problem? Do I need to file a JIRA issue? Thanks! Please do.

Re: Getting timeouts with elastic load balancer in AWS

Jun, I wrote a test producer to test if num.retries working or not. But I found that it's not working. No matter how many retries I set, whenever a message send fails, it always never gets to the broker. I am using Kafka 0.7.0 Is this a known problem? Do I need to file a JIRA issue? Because we

Re: Getting timeouts with elastic load balancer in AWS

Set num.retries in producer config property file. It defaults to 0. Thanks, Jun On Tue, Jun 26, 2012 at 4:46 PM, Vaibhav Puranik wrote: > I reduced the batch size and reduced the pooled connections. Number of > errors have gone down significantly. But they are not eliminated yet. > > We defini

Re: Getting timeouts with elastic load balancer in AWS

I reduced the batch size and reduced the pooled connections. Number of errors have gone down significantly. But they are not eliminated yet. We definitely don't want to loose any events. Jun, how do I configure the client resend you mentioned below? I couldn't find any configuration. Regards, Va

Re: Getting timeouts with elastic load balancer in AWS

These are great pointers. I found some more discussion here: https://forums.aws.amazon.com/thread.jspa?threadID=33427 I can do the following to keep using the elastic load balancer: 1) Reduce the producer pool size to 1 or 2 because looks like connections are sitting idle. My volume does not desi

Re: Getting timeouts with elastic load balancer in AWS

ELBs will close connections that have no data going across them over a 60 sec period. A reference to this behavior can be found at the bottom of this page: http://aws.amazon.com/articles/1636185810492479 There is currently no way for customers to increase this timeout. If this timeout is in fac

Re: Getting timeouts with elastic load balancer in AWS

Vaibhav, Does elastic load balancer have any timeouts or quotas that kill existing socket connections? Does client resend succeed (you can configure resend in DefaultEventHandler)? Thanks, Jun On Mon, Jun 25, 2012 at 6:01 PM, Vaibhav Puranik wrote: > Hi all, > > We are sending our ad impressi