Re: New consumer - partitions auto assigned only on poll
Hello Jason, Thanks for feedback. I've created ticket for this https://issues.apache.org/jira/browse/KAFKA-2359 Kind regards, Stevo Slavic. On Wed, Jul 22, 2015 at 6:18 PM, Jason Gustafson wrote: > Hey Stevo, > > That's a good point. I think the javadoc is pretty clear that this could > return no partitions when the consumer has no active assignment, but it may > be a little unintuitive to have to call poll() after subscribing before you > can get the assigned partitions. I can't think of a strong reason not to go > ahead with the assignment in subscriptions() other than to keep it > non-blocking. Perhaps you can open a ticket and we can get feedback from > some other devs? > > Thanks, > Jason > > On Wed, Jul 22, 2015 at 2:09 AM, Stevo Slavić wrote: > > > Hello Apache Kafka community, > > > > In the new consumer I encountered unexpected behavior. After constructing > > KafakConsumer instance with configured consumer rebalance callback > handler, > > and subscribing to a topic with "consumer.subscribe(topic)", retrieving > > subscriptions would return empty set and callback handler would not get > > called (no partitions ever assigned or revoked), no matter how long > > instance was up. > > > > Then I found by inspecting KafkaConsumer code that partition assignment > > will only be triggered on first poll, pollOnce has: > > > > // ensure we have partitions assigned if we expect to > > if (subscriptions.partitionsAutoAssigned()) > > coordinator.ensurePartitionAssignment(); > > > > Would it make sense to include this fragment in > KafkaConsumer.subscriptions > > accessor as well? > > > > Kind regards, > > Stevo Slavic. > > >
Re: New consumer - partitions auto assigned only on poll
Hey Stevo, That's a good point. I think the javadoc is pretty clear that this could return no partitions when the consumer has no active assignment, but it may be a little unintuitive to have to call poll() after subscribing before you can get the assigned partitions. I can't think of a strong reason not to go ahead with the assignment in subscriptions() other than to keep it non-blocking. Perhaps you can open a ticket and we can get feedback from some other devs? Thanks, Jason On Wed, Jul 22, 2015 at 2:09 AM, Stevo Slavić wrote: > Hello Apache Kafka community, > > In the new consumer I encountered unexpected behavior. After constructing > KafakConsumer instance with configured consumer rebalance callback handler, > and subscribing to a topic with "consumer.subscribe(topic)", retrieving > subscriptions would return empty set and callback handler would not get > called (no partitions ever assigned or revoked), no matter how long > instance was up. > > Then I found by inspecting KafkaConsumer code that partition assignment > will only be triggered on first poll, pollOnce has: > > // ensure we have partitions assigned if we expect to > if (subscriptions.partitionsAutoAssigned()) > coordinator.ensurePartitionAssignment(); > > Would it make sense to include this fragment in KafkaConsumer.subscriptions > accessor as well? > > Kind regards, > Stevo Slavic. >
New consumer - partitions auto assigned only on poll
Hello Apache Kafka community, In the new consumer I encountered unexpected behavior. After constructing KafakConsumer instance with configured consumer rebalance callback handler, and subscribing to a topic with "consumer.subscribe(topic)", retrieving subscriptions would return empty set and callback handler would not get called (no partitions ever assigned or revoked), no matter how long instance was up. Then I found by inspecting KafkaConsumer code that partition assignment will only be triggered on first poll, pollOnce has: // ensure we have partitions assigned if we expect to if (subscriptions.partitionsAutoAssigned()) coordinator.ensurePartitionAssignment(); Would it make sense to include this fragment in KafkaConsumer.subscriptions accessor as well? Kind regards, Stevo Slavic.