pooja sadashiv deokar created KAFKA-3553: --------------------------------------------
Summary: Issue with getting data of next offset after restarting the consumer and producer is always running Key: KAFKA-3553 URL: https://issues.apache.org/jira/browse/KAFKA-3553 Project: Kafka Issue Type: Bug Components: consumer, offset manager Affects Versions: 0.9.0.0 Environment: operating system : ubuntu 14.04 using kafka-python Reporter: pooja sadashiv deokar Assignee: Neha Narkhede I am pushing data to kafka topic after every 1 second in python. And written consumer to fetch data from topic with consumer timeout as 500 ms and enable_auto_commit as false. Following are the scripts: 1) prod.py from kafka import KafkaClient,SimpleConsumer from kafka.producer import SimpleProducer import random, time kafka = KafkaClient("localhost:9092") producer = SimpleProducer(kafka) i=0 while(True): l1 = ['Rahul','Narendra','NaMo','ManMohan','Sonia'] msg = str(i) + ',' + str(random.choice(l1)) print('putting data : '+msg) producer.send_messages("test", msg) i = i + 1 time.sleep(1) 2) con.py from kafka import KafkaConsumer consumer = KafkaConsumer('test',bootstrap_servers=['localhost:9092'],consumer_timeout_ms = 500, enable_auto_commit = False) for message in consumer: print ("%s:%d:%d: key=%s value=%s" % (message.topic, message.partition, message.offset, message.key, message.value)) consumer.commit() My producer is continuously running. And I am running consumer again when it stops. For first time consumer works well and gets data for me on time. But after first time its taking minimum 1 minute to maximum 5 minutes (or above 5 min also) to get next data. It should give me data after 1 second or maximum after 5 seconds. But its giving problem :( . Also 1 thing which I observe is If I wait for starting consumer about 1 min then data comes as expected. Please correct me if I am going wrong. -- This message was sent by Atlassian JIRA (v6.3.4#6332)