Re: kafka partitions api

2015-02-26 Thread tao xiao
Alex, You can get partition from MessageAndMetadata as partition is exported via constructor parameter On Fri, Feb 27, 2015 at 2:12 PM, Alex Melville wrote: > Tao and Gaurav, > > > After looking through the source code in Kafka v8.2.0, I don't see any > partition() function on the MessageAndMet

Re: kafka partitions api

2015-02-26 Thread Alex Melville
Tao and Gaurav, After looking through the source code in Kafka v8.2.0, I don't see any partition() function on the MessageAndMetadata object. Here's the class's source: package kafka.message import kafka.serializer.Decoder import kafka.utils.Utils case class MessageAndMetadata[K, V](topic: St

Re: kafka partitions api

2015-02-26 Thread Gaurav Agarwal
that's fine to me , you can open a separate thread , But the original question when the consumerconnector got connected to a separate topic , Whether KafkaStream will have all the information of the partitions for that corresponding topic , Please confirm Thanks On Fri, Feb 27, 2015 at 11:20 AM,

Re: kafka partitions api

2015-02-26 Thread Alex Melville
I was going to make a separate email thread for this question but this thread's topic echoes what my own would have been. How can I query a broker or zookeeper for the number of partitions in a given topic? I'm trying to write a custom partitioner that sends a message to every partition within a

Re: kafka partitions api

2015-02-26 Thread tao xiao
Gaurav, You can get the partition number the message belongs to via MessageAndMetadata.partition() On Fri, Feb 27, 2015 at 5:16 AM, Jun Rao wrote: > The partition api is exposed to the consumer in 0.8.2. > > Thanks, > > Jun > > On Thu, Feb 26, 2015 at 10:53 AM, Gaurav Agarwal > wrote: > > > Af

Re: kafka partitions api

2015-02-26 Thread Jun Rao
The partition api is exposed to the consumer in 0.8.2. Thanks, Jun On Thu, Feb 26, 2015 at 10:53 AM, Gaurav Agarwal wrote: > After retrieving a kafka stream or kafka message how to get the > corresponding partition number to which it belongs ? I am using kafka > version 0.8.1. > More specifica

Fwd: kafka partitions api

2015-02-26 Thread Gaurav Agarwal
Hello After retrieving a kafka stream or kafka message how to get the corresponding partition number to which it belongs ? I am using kafka version 0.8.1. More specifically kafka.consumer.KafkaStream and kafka.message.MessageAndMetaData classes, does not provide API to retrieve partition number. Ar

kafka partitions api

2015-02-26 Thread Gaurav Agarwal
After retrieving a kafka stream or kafka message how to get the corresponding partition number to which it belongs ? I am using kafka version 0.8.1. More specifically kafka.consumer.KafkaStream and kafka.message.MessageAndMetaData classes, does not provide API to retrieve partition number. Are ther