RE: Questions on the new Kafka consumer

2015-10-14 Thread prajod.vettiyattil
Hi,

According to the wiki: 
https://cwiki.apache.org/confluence/display/KAFKA/Consumer+Client+Re-Design, 
allowing manual partition and topic access is a design goal.

Also the new API has functions to seek and subscribe at the partition level: 
http://people.apache.org/~nehanarkhede/kafka-0.9-consumer-javadoc/doc/

This for the KafkaConsumer from kafka version 0.9.

Prajod

-Original Message-
From: Rajiv Kurian [mailto:ra...@signalfx.com]
Sent: 14 October 2015 01:44
To: users@kafka.apache.org
Subject: Questions on the new Kafka consumer

I was reading the documentation for the new Kafka consumer API at 
https://github.com/apache/kafka/blob/trunk/clients/src/main/java/org/apache/kafka/clients/consumer/KafkaConsumer.java
and came across this:

"Each Kafka consumer must specify a consumer group that it belongs to."

Currently we use Kafka but WITHOUT the consumer group functionality since I do 
the partition distribution based on my own app logic. I also persist my own 
offsets in an external store.
I also sometimes need to consume from the beginning of a partition or just some 
arbitrary offset. Currently I use the SimpleConsumer to do this. I just give it 
the topics and partitions I want to consume from and I maintain the partition 
-> broker mapping myself. I also maintain the offset myself and when I need to 
consume from the beginning of a partition I ask the broker for the earliest 
offset and start consuming from there.

I am glad that the new consumer does the broker -> partition mapping itself and 
I can get rid of a lot of code if I transitioned to it. But given that the 
documentation suggests that I have to use the Consumer group, it doesn't seem 
that it will support the lower level features that I need. I do see the 
commitSync method which suggests that I am in control of the offset management 
and for my use case I could just never call it. Is there also a way for me to 
specify which partition to consume from exactly (in addition to the topic) and 
also from which offset?

Thanks,
Rajiv
The information contained in this electronic message and any attachments to 
this message are intended for the exclusive use of the addressee(s) and may 
contain proprietary, confidential or privileged information. If you are not the 
intended recipient, you should not disseminate, distribute or copy this e-mail. 
Please notify the sender immediately and destroy all copies of this message and 
any attachments. WARNING: Computer viruses can be transmitted via email. The 
recipient should check this email and any attachments for the presence of 
viruses. The company accepts no liability for any damage caused by any virus 
transmitted by this email. www.wipro.com


Re: Questions on the new Kafka consumer

2015-10-13 Thread Rajiv Kurian
Aah I didn't read long enough. I just read the subscribing to specific
partitions, managing your own offsets and the controlling the consumer's
position sections.

So a follow up question - when I am subscribing to specific partitions and
managing my own offsets and controlling the consumer's position what does
the consumer group do? Do I still have to specify it all the time?

Thanks,
Rajiv

On Tue, Oct 13, 2015 at 1:14 PM, Rajiv Kurian  wrote:

> I was reading the documentation for the new Kafka consumer API at
> https://github.com/apache/kafka/blob/trunk/clients/src/main/java/org/apache/kafka/clients/consumer/KafkaConsumer.java
> and came across this:
>
> "Each Kafka consumer must specify a consumer group that it belongs to."
>
> Currently we use Kafka but WITHOUT the consumer group functionality since
> I do the partition distribution based on my own app logic. I also persist
> my own offsets in an external store.
> I also sometimes need to consume from the beginning of a partition or just
> some arbitrary offset. Currently I use the SimpleConsumer to do this. I
> just give it the topics and partitions I want to consume from and I
> maintain the partition -> broker mapping myself. I also maintain the offset
> myself and when I need to consume from the beginning of a partition I ask
> the broker for the earliest offset and start consuming from there.
>
> I am glad that the new consumer does the broker -> partition mapping
> itself and I can get rid of a lot of code if I transitioned to it. But
> given that the documentation suggests that I have to use the Consumer
> group, it doesn't seem that it will support the lower level features that I
> need. I do see the commitSync method which suggests that I am in control of
> the offset management and for my use case I could just never call it. Is
> there also a way for me to specify which partition to consume from exactly
> (in addition to the topic) and also from which offset?
>
> Thanks,
> Rajiv
>