Hi, According to the wiki: https://cwiki.apache.org/confluence/display/KAFKA/Consumer+Client+Re-Design, allowing manual partition and topic access is a design goal.
Also the new API has functions to seek and subscribe at the partition level: http://people.apache.org/~nehanarkhede/kafka-0.9-consumer-javadoc/doc/ This for the KafkaConsumer from kafka version 0.9. Prajod -----Original Message----- From: Rajiv Kurian [mailto:[email protected]] Sent: 14 October 2015 01:44 To: [email protected] Subject: Questions on the new Kafka consumer I was reading the documentation for the new Kafka consumer API at https://github.com/apache/kafka/blob/trunk/clients/src/main/java/org/apache/kafka/clients/consumer/KafkaConsumer.java and came across this: "Each Kafka consumer must specify a consumer group that it belongs to." Currently we use Kafka but WITHOUT the consumer group functionality since I do the partition distribution based on my own app logic. I also persist my own offsets in an external store. I also sometimes need to consume from the beginning of a partition or just some arbitrary offset. Currently I use the SimpleConsumer to do this. I just give it the topics and partitions I want to consume from and I maintain the partition -> broker mapping myself. I also maintain the offset myself and when I need to consume from the beginning of a partition I ask the broker for the earliest offset and start consuming from there. I am glad that the new consumer does the broker -> partition mapping itself and I can get rid of a lot of code if I transitioned to it. But given that the documentation suggests that I have to use the Consumer group, it doesn't seem that it will support the lower level features that I need. I do see the commitSync method which suggests that I am in control of the offset management and for my use case I could just never call it. Is there also a way for me to specify which partition to consume from exactly (in addition to the topic) and also from which offset? Thanks, Rajiv The information contained in this electronic message and any attachments to this message are intended for the exclusive use of the addressee(s) and may contain proprietary, confidential or privileged information. If you are not the intended recipient, you should not disseminate, distribute or copy this e-mail. Please notify the sender immediately and destroy all copies of this message and any attachments. WARNING: Computer viruses can be transmitted via email. The recipient should check this email and any attachments for the presence of viruses. The company accepts no liability for any damage caused by any virus transmitted by this email. www.wipro.com
