Re: Consumer thread dies
Thanks Jun for your response. Here is my scenario: topicCountMap.put(topic, new Integer(2)); Map>> consumerMap = consumer.createMessageStreams(topicCountMap); List> streams = consumerMap.get(topic); So from above scenario (only 1 partition) , there will be 2 threads C1 and C2, and one of the thread would be holding one partition. In my scenario, C1 would be hung after consuming the message from stream and in hung state during processing of message. So in such scenario, looks like C2 will not get hold of partition stream unless C1 didn't die.. So considering such scenarios, I am planning to use SimpleConsumer where I will have more control on consuming Partition content. But I have to maintain offset state in zookeeper. Let me know if this approach is correct for such hung scenarios. Thanks and regards, Srini On Sun, Nov 9, 2014 at 9:12 PM, Jun Rao wrote: > If C1 dies, C2 will be owning that partition. However, C1 has to really > die, which typically means that either you close the consumer connector or > the jvm of C1 is gone. > > In your case, it seems that C1 didn't die, it just hung. Do you know why C1 > hung? > > Thanks, > > Jun > > On Fri, Nov 7, 2014 at 3:34 PM, Srinivas Reddy Kancharla < > getre...@gmail.com > > wrote: > > > Hi, > > > > I have a scenario where I have 1 partition and 1 consumer group having 2 > > consumer threads running say C1 and C2. Since there is only one partition > > for a given topic, say C1 is holding that partition. Now due to some > reason > > if C1 dies, can C2 get hold of that partition? > > > > i.e. C1 was busy with KafkaStream instance, for any reason if C1 dies or > in > > hung state, Can we make C2 talking to KafkaStream (for Partition 0). > > I am facing this issue where I have 10 messages in partition 0 and C1 was > > consuming it. At message 4, C1 went into hung state. Now I would like to > > make C2 to consumer other messages which are not consumed by C1. > > > > Thank and regards, > > Srini > > >
Re: Consumer thread dies
Hi, Further I looked at this scenario, Is it correct that above scenario can be handled if I use SimpleConsumer approach instead of using "ConsumerConnector.createMessageStreams()" , this way I have better control on partition. This way my partition is not bound with any specific consumer thread. Please let me know if I am missing anything with "ConsumerConnector.createMessageStreams()". Thanks and regards, Srini On Fri, Nov 7, 2014 at 3:34 PM, Srinivas Reddy Kancharla wrote: > Hi, > > I have a scenario where I have 1 partition and 1 consumer group having 2 > consumer threads running say C1 and C2. Since there is only one partition > for a given topic, say C1 is holding that partition. Now due to some reason > if C1 dies, can C2 get hold of that partition? > > i.e. C1 was busy with KafkaStream instance, for any reason if C1 dies or > in hung state, Can we make C2 talking to KafkaStream (for Partition 0). > I am facing this issue where I have 10 messages in partition 0 and C1 was > consuming it. At message 4, C1 went into hung state. Now I would like to > make C2 to consumer other messages which are not consumed by C1. > > Thank and regards, > Srini >
Consumer thread dies
Hi, I have a scenario where I have 1 partition and 1 consumer group having 2 consumer threads running say C1 and C2. Since there is only one partition for a given topic, say C1 is holding that partition. Now due to some reason if C1 dies, can C2 get hold of that partition? i.e. C1 was busy with KafkaStream instance, for any reason if C1 dies or in hung state, Can we make C2 talking to KafkaStream (for Partition 0). I am facing this issue where I have 10 messages in partition 0 and C1 was consuming it. At message 4, C1 went into hung state. Now I would like to make C2 to consumer other messages which are not consumed by C1. Thank and regards, Srini
Re: Delete Specific Message from a Topic (partition)
Thanks for your response. I agree on sending messages to other new topic and thats what I am doing now. The reason of asking deleting of messages from log : Say I have a topic "InitialState_topic1" where one consumer group mygroupID1 is reading and is sending messages to new topic "InprogressState_topic1" -> which is read by another consumer group "mygroupID2". Now one of the consumer thread in mygroupID2 is stuck or hung in reading one of the partition, I was under impression that there should be expiry rule which I can set on each message so that I can delete such message from that partition and put it back to "initialState_topic1" topic so that other idle threads can pick it up again. So if a thread of a consumer group is stuck or hung (with one single message) and it holds a partition, how can I make other idle thread (from same consumer group) to take control of that partition so that it can read other pending messages? Thanks and regards, Srini On Sun, Oct 26, 2014 at 12:40 AM, Stevo Slavić wrote: > Have group 1 act like a filter, publish to a new topic all messages that > group 2 should process and then have group 2 actually consume only new > topic. > > Kind regards, > Stevo Slavic > On Oct 26, 2014 2:36 AM, "Srinivas Reddy Kancharla" > wrote: > > > Hi, > > > > I have a scenario where I produce messages for a given topic (say having > 10 > > partitions), and I have consumer group ( say mygroupID1) with 10 threads > > reading those 10 partitions. After consuming, I would like to delete > > specific messages from that topic (i.e. from a given partition). > > > > How should I restrict other consumer group say mygroupID2 to read > messages > > from the same topic? > > > > Thanks for the help, > > Srini > > >
Delete Specific Message from a Topic (partition)
Hi, I have a scenario where I produce messages for a given topic (say having 10 partitions), and I have consumer group ( say mygroupID1) with 10 threads reading those 10 partitions. After consuming, I would like to delete specific messages from that topic (i.e. from a given partition). How should I restrict other consumer group say mygroupID2 to read messages from the same topic? Thanks for the help, Srini
