Re: Help with SocketTimeoutException while reading from Kafka cluster

2015-08-14 Thread Jaikiran Pai 

On Wednesday 12 August 2015 04:59 AM, venkatesh kavuluri wrote:

83799 [c3-onboard_-2-9571-1439334326956-cfa8b46a-leader-finder-thread]
INFO  kafka.consumer.SimpleConsumer  - Reconnect due to socket error:
java.net.SocketTimeoutException

163931 [c3-onboard_-2-9571-1439334326956-cfa8b46a-leader-finder-thread]
INFO  kafka.consumer.SimpleConsumer  - Reconnect due to socket error:
java.net.SocketTimeoutException


There's a patch in the JIRA here which logs the exact reason why the 
exception was thrown https://issues.apache.org/jira/browse/KAFKA-2221. 
It hasn't been merged since SimpleConsumer was considered deprecated. 
But you might want to apply that and see if that helps to narrow down 
the issue.


-Jaikiran

Help with SocketTimeoutException while reading from Kafka cluster

2015-08-11 Thread venkatesh kavuluri 
Hi All,


I am running a high level Kafka consumer and while it works fine against a
smaller staging cluster I am getting socket timeout exceptions while
running against larger LIVE cluster (against a topic with 100 partitions).
I am using "zookeeper.connect" to connect to the cluster.


*Stage cluster (Consumer gets messages)*:

1751 [ConsumerFetcherThread-local-enrich_-1439334511597-5eb31426-0-1]
INFO  kafka.consumer.ConsumerFetcherThread  -
[ConsumerFetcherThread-local-enrich_-1439334511597-5eb31426-0-1],
Starting

2344 [local-enrich_-1439334511597-5eb31426-leader-finder-thread] INFO
 kafka.consumer.ConsumerFetcherManager  -
[ConsumerFetcherManager-1439334511817] Added fetcher for partitions
ArrayBuffer([[fpti_onboarding_events,2], initOffset -1 to broker
id:1,host:XXX,port:9092] , [[fpti_onboarding_events,1], initOffset -1 to
broker id:1,host:,port:9092] , [[fpti_onboarding_events,0], initOffset
-1 to broker id:1,host:,port:9092] )

[ 2 ]** LISTENING TO EVENTS **

[ 0 ]** LISTENING TO EVENTS **

[ 1 ]** LISTENING TO EVENTS **


*Live cluster (Socket Timeouts )*:

3645
[ConsumerFetcherThread-c3-onboard_-2-9571-1439334326956-cfa8b46a-0-5]
INFO  kafka.consumer.ConsumerFetcherThread  -
[ConsumerFetcherThread-c3-onboard_-2-9571-1439334326956-cfa8b46a-0-5],
Starting

83799 [c3-onboard_-2-9571-1439334326956-cfa8b46a-leader-finder-thread]
INFO  kafka.consumer.SimpleConsumer  - Reconnect due to socket error:
java.net.SocketTimeoutException

163931 [c3-onboard_-2-9571-1439334326956-cfa8b46a-leader-finder-thread]
INFO  kafka.consumer.SimpleConsumer  - Reconnect due to socket error:
java.net.SocketTimeoutException

163969
[ConsumerFetcherThread-c3-onboard_-2-9571-1439334326956-cfa8b46a-0-1]
INFO  kafka.consumer.ConsumerFetcherThread  -
[ConsumerFetcherThread-c3-onboard_-2-9571-1439334326956-cfa8b46a-0-1],
Starting

Looks like leader-finder-thread is timing out in Live. Do you know why it
would happen.


I have the following timeouts defined in the configuration:

props.put("zookeeper.session.timeout.ms", "6400");

props.put("zookeeper.sync.time.ms", "600");

props.put("auto.offset.reset", "smallest");

props.put("socket.timeout.ms", "8");