[jira] [Commented] (KAFKA-2565) Offset Commit is not working if multiple consumers try to commit the offset
[ https://issues.apache.org/jira/browse/KAFKA-2565?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14952795#comment-14952795 ] Sreenivasulu Nallapati commented on KAFKA-2565: --- This happens basically in this scenario. Our batch works as below 1. Even though we are running our consumer in batch mode, we are using High Level Consumer for scalability. 2. Opens kafka consumer connector 3. Start of the consumer it will identify the latest offset(consumer stop offset -cso) for each partition 4. It will start reading messages till the cso for this batch. 5. Write the messages to a temp files on FTP server. Once we process all the messages, move the FTP server temp files to actual location on FTP server 6. If all data transfer is success, commit the offset to zookeeper. 7. If we run Single consumer to process all the partitions, there is no issue :) 8. If we start multiple consumers for a single topic ( say three consumers, three partitions. one consumer for one partition) the problem starts. What we observed here is: Out of three consumers if one consumer(c1-partition1) finishes its processing ahead of other two. The zookeeper sees a re balancing and start re balancing partition1 with one of other two running consumers(while zookeeper doing this task other consumers consumed all the messages and in the process of moving the temp files on FTP server). We are not closing the consumer connector till end of the batch. The re balancing is happening after we stopped consuming the message. Is there something we are missing here or doing wrong > Offset Commit is not working if multiple consumers try to commit the offset > --- > > Key: KAFKA-2565 > URL: https://issues.apache.org/jira/browse/KAFKA-2565 > Project: Kafka > Issue Type: Bug > Components: consumer >Affects Versions: 0.8.1, 0.8.2.1, 0.8.2.2 > Reporter: Sreenivasulu Nallapati >Assignee: Neha Narkhede > > We are seeing some strange behaviour with commitOffsets() method of > kafka.javaapi.consumer.ConsumerConnector. We committing the offsets to > zookeeper at the end of the consumer batch. We are running multiple consumers > for the same topic. > Test details: > 1.Created a topic with three partitions > 2.Started three consumers (cronjob) at the same time. The aim is that > each consumer to process one partition. > 3.Each consumer at the end of the batch, it will call the commitOffsets() > method on kafka.javaapi.consumer.ConsumerConnector > 4.The offsets are getting properly updated in zookeeper if we run the > consumers for small set (say 1000 messages) of messages. > 5.But for larger number of messages, commit offset is not working as > expected…sometimes only two offsets are properly committing and other one > remains as it was. > 6.Please see the below example > Partition: 0 Latest Offset: 1057585 > Partition: 1 Latest Offset: 1057715 > Partition: 2 Latest Offset: 1057590 > Earliest Offset after all consumers completed: {0=1057585, 1=724375, > 2=1057590} > Highlighted in red supposed to be committed as 1057715 but it did not. > Please check if it is bug with multiple consumers. When multiple consumers > are trying to update the same path in Zookeper, is there any synchronization > issue? > Kafka Cluster details > 1 zookeeper > 3 brokers -- This message was sent by Atlassian JIRA (v6.3.4#6332)
Offset Commit is not working if multiple consumers try to commit the offset
Hello, We are seeing some strange behaviour with commitOffsets() method of kafka.javaapi.consumer.ConsumerConnector. We committing the offsets to zookeeper at the end of the consumer batch. We are running multiple consumers for the same topic. Test details: 1. Created a topic with three partitions 2. Started three consumers (cronjob) at the same time. The aim is that each consumer to process one partition. 3. Each consumer at the end of the batch, it will call the commitOffsets() method on kafka.javaapi.consumer.ConsumerConnector 4. The offsets are getting properly updated in zookeeper if we run the consumers for small set (say 1000 messages) of messages. 5. But for larger number of messages, commit offset is not working as expected…sometimes only two offsets are properly committing and other one remains as it was. 6. Please see the below example Partition: 0 Latest Offset: 1057585 Partition: 1 Latest Offset: 1057715 Partition: 2 Latest Offset: 1057590 Earliest Offset after all consumers completed: {0=1057585, 1=724375, 2=1057590} Highlighted in red supposed to be committed as 1057715 but it did not. Please check if it is bug with multiple consumers. When multiple consumers are trying to update the same path in Zookeper, is there any synchronization issue? Kafka Cluster details 1 zookeeper 3 brokers Thanks Sreeni
[jira] [Commented] (KAFKA-2565) Offset Commit is not working if multiple consumers try to commit the offset
[ https://issues.apache.org/jira/browse/KAFKA-2565?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14902366#comment-14902366 ] Sreenivasulu Nallapati commented on KAFKA-2565: --- Earliest Offset after all consumers completed: {0=1057585, 1=724375, 2=1057590} 724375 supposed to be committed as 1057715 but it did not. > Offset Commit is not working if multiple consumers try to commit the offset > --- > > Key: KAFKA-2565 > URL: https://issues.apache.org/jira/browse/KAFKA-2565 > Project: Kafka > Issue Type: Bug > Components: consumer >Affects Versions: 0.8.1, 0.8.2.1, 0.8.2.2 > Reporter: Sreenivasulu Nallapati >Assignee: Neha Narkhede > > We are seeing some strange behaviour with commitOffsets() method of > kafka.javaapi.consumer.ConsumerConnector. We committing the offsets to > zookeeper at the end of the consumer batch. We are running multiple consumers > for the same topic. > Test details: > 1.Created a topic with three partitions > 2.Started three consumers (cronjob) at the same time. The aim is that > each consumer to process one partition. > 3.Each consumer at the end of the batch, it will call the commitOffsets() > method on kafka.javaapi.consumer.ConsumerConnector > 4.The offsets are getting properly updated in zookeeper if we run the > consumers for small set (say 1000 messages) of messages. > 5.But for larger number of messages, commit offset is not working as > expected…sometimes only two offsets are properly committing and other one > remains as it was. > 6.Please see the below example > Partition: 0 Latest Offset: 1057585 > Partition: 1 Latest Offset: 1057715 > Partition: 2 Latest Offset: 1057590 > Earliest Offset after all consumers completed: {0=1057585, 1=724375, > 2=1057590} > Highlighted in red supposed to be committed as 1057715 but it did not. > Please check if it is bug with multiple consumers. When multiple consumers > are trying to update the same path in Zookeper, is there any synchronization > issue? > Kafka Cluster details > 1 zookeeper > 3 brokers -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Created] (KAFKA-2565) Offset Commit is not working if multiple consumers try to commit the offset
Sreenivasulu Nallapati created KAFKA-2565: - Summary: Offset Commit is not working if multiple consumers try to commit the offset Key: KAFKA-2565 URL: https://issues.apache.org/jira/browse/KAFKA-2565 Project: Kafka Issue Type: Bug Components: consumer Affects Versions: 0.8.2.1, 0.8.1, 0.8.2.2 Reporter: Sreenivasulu Nallapati Assignee: Neha Narkhede We are seeing some strange behaviour with commitOffsets() method of kafka.javaapi.consumer.ConsumerConnector. We committing the offsets to zookeeper at the end of the consumer batch. We are running multiple consumers for the same topic. Test details: 1. Created a topic with three partitions 2. Started three consumers (cronjob) at the same time. The aim is that each consumer to process one partition. 3. Each consumer at the end of the batch, it will call the commitOffsets() method on kafka.javaapi.consumer.ConsumerConnector 4. The offsets are getting properly updated in zookeeper if we run the consumers for small set (say 1000 messages) of messages. 5. But for larger number of messages, commit offset is not working as expected…sometimes only two offsets are properly committing and other one remains as it was. 6. Please see the below example Partition: 0 Latest Offset: 1057585 Partition: 1 Latest Offset: 1057715 Partition: 2 Latest Offset: 1057590 Earliest Offset after all consumers completed: {0=1057585, 1=724375, 2=1057590} Highlighted in red supposed to be committed as 1057715 but it did not. Please check if it is bug with multiple consumers. When multiple consumers are trying to update the same path in Zookeper, is there any synchronization issue? Kafka Cluster details 1 zookeeper 3 brokers -- This message was sent by Atlassian JIRA (v6.3.4#6332)
Question regarding topic creation
Hello, I am new to Kafka and trying to evaluate for one of our use case. I have a basic question regarding topic. Assume we have a 10 broker cluster. If a create a Topic, will it be created in all 10 brokers? If not, how it manages internally if we try to send a message to that perticular topic? assume we created a producer with all brokers Thanks in advance. Thanks Sreeni
How producer gets the acknowledgement back
Hello, Can you please help me to get the acknowledgement in producer? After setting the property *request.required.acks to 1, *how producer gets the acknowledgement back? I am trying to get the acknowledgement in java producer. Thanks Sreeni