RE: Kafka consumers unable to process message

[Ghosh, Achintya (Contractor)]

Hi Jason,

Thanks for your response.

I know that is a known issue and I resolved it calling wakeup method by another 
thread. But here my problem is different, let me explain , it's very basic

I created one cluster with 6 nodes( 3 from one datacenter and 3 from 
another(remote) datacenter and kept replication factor 6 with 2 zookeeper 
servers one from each datacenter ). Now I brought down all 3 nodes of my local 
datacenter and produced few messages and I see producer is working fine even my 
local data center nodes are down. It successfully writes the messages to other 
data center nodes. But when I'm trying to consume the messages the 
consumer.poll method gets stuck as my local datacenter is down though other 
datacenter's nodes are up. 

My question is as the data has been written successfully to other datacenter 
why consumer part is not working?

Here is my Producer settings:

props.put("bootstrap.servers", 
"psaq1-wc.sys.comcast.net:61616,psaq2-wc.sys.comcast.net:61616,psaq3-wc.sys.comcast.net:61616,psaq1-ab.sys.comcast.net:61617,psaq2-ab.sys.comcast.net:61617,psaq3-ab.sys.comcast.net:61617");
props.put("acks", "1");
props.put("max.block.ms", 1000);
props.put("key.serializer", 
"org.apache.kafka.common.serialization.StringSerializer");
props.put("value.serializer", "com.comcast.ps.kafka.object.CustomMessageSer");

and here is Consumer settings:

props.put("group.id", "app-consumer");
                props.put("enable.auto.commit", "false");
                props.put("auto.offset.reset", "earliest");
                props.put("auto.commit.interval.ms", "500");
                props.put("session.timeout.ms", "120000");
                props.put("consumer.timeout.ms", "10000");
                props.put("zookeeper.session.timeout.ms", "120000");
                props.put("zookeeper.connection.timeout.ms", "60000");
                props.put("offsets.storage","kafka");
                props.put("request.timeout.ms", "150000");
                props.put("bootstrap.servers", 
"psaq1-wc.sys.comcast.net:61616,psaq2-wc.sys.comcast.net:61616,psaq3-wc.sys.comcast.net:61616,psaq1-ab.sys.comcast.net:61617,psaq2-ab.sys.comcast.net:61617,psaq3-ab.sys.comcast.net:61617");
            
                props.put("key.deserializer", 
"org.apache.kafka.common.serialization.StringDeserializer");
                props.put("value.deserializer", 
"com.comcast.ps.kafka.object.CustomMessageDeSer");

Is it because of consumer is not able to get the broker metadata if it is 
trying to connect other datacenter's zookeeper server? I tried with to increate 
the zookeeper session timeout and connection time out but no luck.

Please help on this.
Thanks
Achintya


-----Original Message-----
From: Jason Gustafson [mailto:ja...@confluent.io] 
Sent: Wednesday, August 31, 2016 4:05 PM
To: us...@kafka.apache.org
Cc: dev@kafka.apache.org
Subject: Re: Kafka consumers unable to process message

Hi Achintya,

We have a JIRA for this problem: https://issues.
apache.org/jira/browse/KAFKA-3834. Do you expect the client to raise an 
exception in this case or do you just want to keep it from blocking 
indefinitely? If the latter, you could escape the poll from another thread 
using wakeup().

Thanks,
Jason

On Wed, Aug 31, 2016 at 12:11 PM, Ghosh, Achintya (Contractor) < 
achintya_gh...@comcast.com> wrote:

> Hi there,
>
> Kafka consumer gets stuck at consumer.poll() method if my current 
> datacenter is down and replicated messages are in remote datacenter.
>
> How to solve that issue?
>
> Thanks
> Achintya
>