Re: Consumer thread dies
Thanks Jun for your response. Here is my scenario: topicCountMap.put(topic, new Integer(2)); MapString, ListKafkaStreambyte[], byte[] consumerMap = consumer.createMessageStreams(topicCountMap); ListKafkaStreambyte[], byte[] streams = consumerMap.get(topic); So from above scenario (only 1 partition) , there will be 2 threads C1 and C2, and one of the thread would be holding one partition. In my scenario, C1 would be hung after consuming the message from stream and in hung state during processing of message. So in such scenario, looks like C2 will not get hold of partition stream unless C1 didn't die.. So considering such scenarios, I am planning to use SimpleConsumer where I will have more control on consuming Partition content. But I have to maintain offset state in zookeeper. Let me know if this approach is correct for such hung scenarios. Thanks and regards, Srini On Sun, Nov 9, 2014 at 9:12 PM, Jun Rao jun...@gmail.com wrote: If C1 dies, C2 will be owning that partition. However, C1 has to really die, which typically means that either you close the consumer connector or the jvm of C1 is gone. In your case, it seems that C1 didn't die, it just hung. Do you know why C1 hung? Thanks, Jun On Fri, Nov 7, 2014 at 3:34 PM, Srinivas Reddy Kancharla getre...@gmail.com wrote: Hi, I have a scenario where I have 1 partition and 1 consumer group having 2 consumer threads running say C1 and C2. Since there is only one partition for a given topic, say C1 is holding that partition. Now due to some reason if C1 dies, can C2 get hold of that partition? i.e. C1 was busy with KafkaStream instance, for any reason if C1 dies or in hung state, Can we make C2 talking to KafkaStream (for Partition 0). I am facing this issue where I have 10 messages in partition 0 and C1 was consuming it. At message 4, C1 went into hung state. Now I would like to make C2 to consumer other messages which are not consumed by C1. Thank and regards, Srini
Re: Consumer thread dies
Don't you have the same problem using SimpleConsumer? How does another process know a SimpleConsumer hangs? Thanks, Jun On Mon, Nov 10, 2014 at 9:47 AM, Srinivas Reddy Kancharla getre...@gmail.com wrote: Thanks Jun for your response. Here is my scenario: topicCountMap.put(topic, new Integer(2)); MapString, ListKafkaStreambyte[], byte[] consumerMap = consumer.createMessageStreams(topicCountMap); ListKafkaStreambyte[], byte[] streams = consumerMap.get(topic); So from above scenario (only 1 partition) , there will be 2 threads C1 and C2, and one of the thread would be holding one partition. In my scenario, C1 would be hung after consuming the message from stream and in hung state during processing of message. So in such scenario, looks like C2 will not get hold of partition stream unless C1 didn't die.. So considering such scenarios, I am planning to use SimpleConsumer where I will have more control on consuming Partition content. But I have to maintain offset state in zookeeper. Let me know if this approach is correct for such hung scenarios. Thanks and regards, Srini On Sun, Nov 9, 2014 at 9:12 PM, Jun Rao jun...@gmail.com wrote: If C1 dies, C2 will be owning that partition. However, C1 has to really die, which typically means that either you close the consumer connector or the jvm of C1 is gone. In your case, it seems that C1 didn't die, it just hung. Do you know why C1 hung? Thanks, Jun On Fri, Nov 7, 2014 at 3:34 PM, Srinivas Reddy Kancharla getre...@gmail.com wrote: Hi, I have a scenario where I have 1 partition and 1 consumer group having 2 consumer threads running say C1 and C2. Since there is only one partition for a given topic, say C1 is holding that partition. Now due to some reason if C1 dies, can C2 get hold of that partition? i.e. C1 was busy with KafkaStream instance, for any reason if C1 dies or in hung state, Can we make C2 talking to KafkaStream (for Partition 0). I am facing this issue where I have 10 messages in partition 0 and C1 was consuming it. At message 4, C1 went into hung state. Now I would like to make C2 to consumer other messages which are not consumed by C1. Thank and regards, Srini
Re: Consumer thread dies
Hi, Further I looked at this scenario, Is it correct that above scenario can be handled if I use SimpleConsumer approach instead of using ConsumerConnector.createMessageStreams() , this way I have better control on partition. This way my partition is not bound with any specific consumer thread. Please let me know if I am missing anything with ConsumerConnector.createMessageStreams(). Thanks and regards, Srini On Fri, Nov 7, 2014 at 3:34 PM, Srinivas Reddy Kancharla getre...@gmail.com wrote: Hi, I have a scenario where I have 1 partition and 1 consumer group having 2 consumer threads running say C1 and C2. Since there is only one partition for a given topic, say C1 is holding that partition. Now due to some reason if C1 dies, can C2 get hold of that partition? i.e. C1 was busy with KafkaStream instance, for any reason if C1 dies or in hung state, Can we make C2 talking to KafkaStream (for Partition 0). I am facing this issue where I have 10 messages in partition 0 and C1 was consuming it. At message 4, C1 went into hung state. Now I would like to make C2 to consumer other messages which are not consumed by C1. Thank and regards, Srini
Re: Consumer thread dies
If C1 dies, C2 will be owning that partition. However, C1 has to really die, which typically means that either you close the consumer connector or the jvm of C1 is gone. In your case, it seems that C1 didn't die, it just hung. Do you know why C1 hung? Thanks, Jun On Fri, Nov 7, 2014 at 3:34 PM, Srinivas Reddy Kancharla getre...@gmail.com wrote: Hi, I have a scenario where I have 1 partition and 1 consumer group having 2 consumer threads running say C1 and C2. Since there is only one partition for a given topic, say C1 is holding that partition. Now due to some reason if C1 dies, can C2 get hold of that partition? i.e. C1 was busy with KafkaStream instance, for any reason if C1 dies or in hung state, Can we make C2 talking to KafkaStream (for Partition 0). I am facing this issue where I have 10 messages in partition 0 and C1 was consuming it. At message 4, C1 went into hung state. Now I would like to make C2 to consumer other messages which are not consumed by C1. Thank and regards, Srini
