[ 
https://issues.apache.org/jira/browse/KAFKA-6227?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Ramkumar updated KAFKA-6227:
----------------------------
    Description: 
In Kafka 0.8 High level consumers, the consumer.id under group.id 
differentiates the consumers connection and manage the rebalancing the 
partitions in zookeeper.  Our Service uses this logic and keeps the Kafka 
stream connection in a cache (Concurrent Hashmap). so that consecutive http 
client connection doesn’t have to make a stream connection, but takes from 
cache and read off the messages. This also helps multiple consumers under same 
group.id can simulatenously make connection to kafka and read off the message 
(load balancing).


In Kafka 0.11.0.1, the New consumer API the design has changed.  The 
consumer.id properties are no more supported and the connections with zookeeper 
are managed by Kafka itself. When 2 consumers instances under the same group 
attempts to make a connection simulatenously , one connection waits  on 
consumer.poll() method until the other one (which is already active) drops the 
connection. 
http://kafka.apache.org/0110/javadoc/index.html?org/apache/kafka/clients/consumer/KafkaConsumer.html
 That is at any point of only one active consumer instance is able to poll the 
messages from the topic. This slightly would change the behavior of our service 
that we have to restrict only one consumer connection for a group for a topic. 
That is we couldn’t hold the connection in cache if multiple consumer under 
same group needs to use the Kafka.
I couldn’t find any properties that aids to make multiple consumer connections 
on the same group

The manual partition assignment may be a work around but this is way complex to 
handle that in a service. This is complex because the service needs to track 
the consumer connections and assign the partitions of the topic and do the 
rebalancing (what Kafka 0.8 high level consumer does originally).   
Unfortunately we cannot use legacy api's n Kafka 0.11 since documentation 
refers about performance degradation in using old apis in new versions.

Was there a solution devised how the highlevel consumer of kafka 0.8 can be 
migrated with out any change to the behavior from the users perspective


  was:
In Kafka 0.8 High level consumers, the consumer.id under group.id 
differentiates the consumers connection and manage the rebalancing the 
partitions in zookeeper.  Our Service uses this logic and keeps the Kafka 
stream connection in a cache (Concurrent Hashmap). so that consecutive http 
client connection doesn’t have to make a stream connection, but takes from 
cache and read off the messages. This also helps multiple consumers under same 
group.id can simulatenously make connection to kafka and read off the message 
(load balancing).


In Kafka 0.11.0.1, the New consumer API the design has changed.  The 
consumer.id properties are no more supported and the connections with zookeeper 
are managed by Kafka itself. When 2 consumers instances under the same group 
attempts to make a connection simulatenously , one connection waits  on 
consumer.poll() method until the other one drops the connection. 
http://kafka.apache.org/0110/javadoc/index.html?org/apache/kafka/clients/consumer/KafkaConsumer.html
 That is at any point of only one active consumer instance is able to poll the 
messages from the topic. This slightly would change the behavior of our service 
that we have to restrict only one consumer connection for a group for a topic. 
That is we couldn’t hold the connection in cache if multiple consumer under 
same group needs to use the Kafka.
I couldn’t find any properties that aids to make multiple consumer connections 
on the same group

The manual partition assignment may be a work around but this is way complex to 
handle that in a service. This is complex because the service needs to track 
the consumer connections and assign the partitions of the topic and do the 
rebalancing (what Kafka 0.8 high level consumer does originally).   
Unfortunately we cannot use legacy api's n Kafka 0.11 since documentation 
refers about performance degradation in using old apis in new versions.

Was there a solution devised how the highlevel consumer of kafka 0.8 can be 
migrated with out any change to the behavior from the users perspective



> Kafka 0.11.01 New consumer - multiple consumers under same group not working 
> as expected
> ----------------------------------------------------------------------------------------
>
>                 Key: KAFKA-6227
>                 URL: https://issues.apache.org/jira/browse/KAFKA-6227
>             Project: Kafka
>          Issue Type: Bug
>          Components: consumer
>    Affects Versions: 0.11.0.1
>            Reporter: Ramkumar
>
> In Kafka 0.8 High level consumers, the consumer.id under group.id 
> differentiates the consumers connection and manage the rebalancing the 
> partitions in zookeeper.  Our Service uses this logic and keeps the Kafka 
> stream connection in a cache (Concurrent Hashmap). so that consecutive http 
> client connection doesn’t have to make a stream connection, but takes from 
> cache and read off the messages. This also helps multiple consumers under 
> same group.id can simulatenously make connection to kafka and read off the 
> message (load balancing).
> In Kafka 0.11.0.1, the New consumer API the design has changed.  The 
> consumer.id properties are no more supported and the connections with 
> zookeeper are managed by Kafka itself. When 2 consumers instances under the 
> same group attempts to make a connection simulatenously , one connection 
> waits  on consumer.poll() method until the other one (which is already 
> active) drops the connection. 
> http://kafka.apache.org/0110/javadoc/index.html?org/apache/kafka/clients/consumer/KafkaConsumer.html
>  That is at any point of only one active consumer instance is able to poll 
> the messages from the topic. This slightly would change the behavior of our 
> service that we have to restrict only one consumer connection for a group for 
> a topic. That is we couldn’t hold the connection in cache if multiple 
> consumer under same group needs to use the Kafka.
> I couldn’t find any properties that aids to make multiple consumer 
> connections on the same group
> The manual partition assignment may be a work around but this is way complex 
> to handle that in a service. This is complex because the service needs to 
> track the consumer connections and assign the partitions of the topic and do 
> the rebalancing (what Kafka 0.8 high level consumer does originally).   
> Unfortunately we cannot use legacy api's n Kafka 0.11 since documentation 
> refers about performance degradation in using old apis in new versions.
> Was there a solution devised how the highlevel consumer of kafka 0.8 can be 
> migrated with out any change to the behavior from the users perspective



--
This message was sent by Atlassian JIRA
(v6.4.14#64029)

Reply via email to