Hi,
According to the wiki:
https://cwiki.apache.org/confluence/display/KAFKA/Consumer+Client+Re-Design,
allowing manual partition and topic access is a design goal.
Also the new API has functions to seek and subscribe at the partition level:
http://people.apache.org/~nehanarkhede/kafka-0.9-consumer-javadoc/doc/
This for the KafkaConsumer from kafka version 0.9.
Prajod
-Original Message-
From: Rajiv Kurian [mailto:ra...@signalfx.com]
Sent: 14 October 2015 01:44
To: users@kafka.apache.org
Subject: Questions on the new Kafka consumer
I was reading the documentation for the new Kafka consumer API at
https://github.com/apache/kafka/blob/trunk/clients/src/main/java/org/apache/kafka/clients/consumer/KafkaConsumer.java
and came across this:
"Each Kafka consumer must specify a consumer group that it belongs to."
Currently we use Kafka but WITHOUT the consumer group functionality since I do
the partition distribution based on my own app logic. I also persist my own
offsets in an external store.
I also sometimes need to consume from the beginning of a partition or just some
arbitrary offset. Currently I use the SimpleConsumer to do this. I just give it
the topics and partitions I want to consume from and I maintain the partition
-> broker mapping myself. I also maintain the offset myself and when I need to
consume from the beginning of a partition I ask the broker for the earliest
offset and start consuming from there.
I am glad that the new consumer does the broker -> partition mapping itself and
I can get rid of a lot of code if I transitioned to it. But given that the
documentation suggests that I have to use the Consumer group, it doesn't seem
that it will support the lower level features that I need. I do see the
commitSync method which suggests that I am in control of the offset management
and for my use case I could just never call it. Is there also a way for me to
specify which partition to consume from exactly (in addition to the topic) and
also from which offset?
Thanks,
Rajiv
The information contained in this electronic message and any attachments to
this message are intended for the exclusive use of the addressee(s) and may
contain proprietary, confidential or privileged information. If you are not the
intended recipient, you should not disseminate, distribute or copy this e-mail.
Please notify the sender immediately and destroy all copies of this message and
any attachments. WARNING: Computer viruses can be transmitted via email. The
recipient should check this email and any attachments for the presence of
viruses. The company accepts no liability for any damage caused by any virus
transmitted by this email. www.wipro.com