What's the output of the ConsumerOffsetChecker tool?
Thanks,
Jun
On Tue, Oct 28, 2014 at 7:31 AM, Natarajan, Murugavel <
murugavel.natara...@softwareag.com> wrote:
> Hi,
>
> I have the following Kafka Setup
> Number of producer : 1
> Number of topics : 1
> Number of partitions : 2
> Number of c
Hi,
I have the following Kafka Setup
Number of producer : 1
Number of topics : 1
Number of partitions : 2
Number of consumers : 3 (with same group id)
Number of Kafka cluster : none(single Kafka server)
Zookeeper.session.timeout : 1000
Producer produces messages without any specific partitioning
With SimpleConsumer, you will have to handle leader discovery as well as
zookeeper based rebalancing. You can see an example here -
https://cwiki.apache.org/confluence/display/KAFKA/0.8.0+SimpleConsumer+Example
On Wed, Oct 8, 2014 at 11:45 AM, Sharninder wrote:
> Thanks Gwen. This really helped.
Thanks Gwen. This really helped.
Yes, Kafka is the best thing ever :)
Now how would this be done with the Simple consumer? I'm guessing I'll have
to maintain my own state in Zookeeper or something of that sort?
On Thu, Oct 9, 2014 at 12:01 AM, Gwen Shapira wrote:
> Here's an example (from Con
Here's an example (from ConsumerOffsetChecker tool) of 1 topic (t1)
and 1 consumer group (flume), each of the 3 topic partitions is being
read by a different machine running the flume consumer:
Group Topic Pid Offset
logSize Lag Owner
flume
yep. exactly.
On Wed, Oct 8, 2014 at 11:07 AM, Sharninder wrote:
> Thanks Gwen.
>
> When you're saying that I can add consumers to the same group, does that
> also hold true if those consumers are running on different machines? Or in
> different JVMs?
>
> --
> Sharninder
>
>
> On Wed, Oct 8, 2014
If you use the high level consumer implementation, and register all
consumers as part of the same group - they will load-balance
automatically.
When you add a consumer to the group, if there are enough partitions
in the topic, some of the partitions will be assigned to the new
consumer.
When a con
Thanks Gwen.
When you're saying that I can add consumers to the same group, does that
also hold true if those consumers are running on different machines? Or in
different JVMs?
--
Sharninder
On Wed, Oct 8, 2014 at 11:35 PM, Gwen Shapira wrote:
> If you use the high level consumer implementati
