[
https://issues.apache.org/jira/browse/KAFKA-2350?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14639657#comment-14639657
]
Guozhang Wang commented on KAFKA-2350:
--------------------------------------
Personally I would prefer not mixing the topic subscription (i.e. use Kafka for
assignment) with partition subscription (i.e. not using Kafka for assignment)
since I feel its flexibility benefits is overwhelmed by the complexity of API
semantics, mainly because (again, this is totally personally) I think there
will be very few use cases that need both during on life cycle of a consumer:
with the new consumer implementation, you can always have two consumers, one
for topic subscription and one for partition subscription since the
construction cost is quite low.
With that assumption that we are not going to support this mixing, I think what
I was mainly concerned (also probably as Jay intended to describe in case 3
above) is that the we will be effectively making the logic of whether throwing
exception or not with subscribe(topic) and subscribe(partition) at the same
time based on the call trace, for example:
{code}
subscribe(topicA);
unsubscribe(topicA-partition1);
subscribe(topicA-partition1); // this will be OK
subscribe(topicC-partition1); // this will be not OK, i.e. the logic depends
on what subscribe / unsubscribe() you have called before
{code}
Plus, what [~yasuhiro.matsuda] mentioned is also a valid point, though
rebalance will only happen during poll call. It is still wired to get
{code}
subscribe(topicA);
// then say assignment() return me topicA-partition1.
poll();
unsubscribe(topicA-partition1); // may throw an exception here, so you need to
get the assignment each time before you try to unsubscribe / subscribe, since
each poll() may change the assignment.
{code}
> Add KafkaConsumer pause capability
> ----------------------------------
>
> Key: KAFKA-2350
> URL: https://issues.apache.org/jira/browse/KAFKA-2350
> Project: Kafka
> Issue Type: Improvement
> Reporter: Jason Gustafson
> Assignee: Jason Gustafson
>
> There are some use cases in stream processing where it is helpful to be able
> to pause consumption of a topic. For example, when joining two topics, you
> may need to delay processing of one topic while you wait for the consumer of
> the other topic to catch up. The new consumer currently doesn't provide a
> nice way to do this. If you skip poll() or if you unsubscribe, then a
> rebalance will be triggered and your partitions will be reassigned.
> One way to achieve this would be to add two new methods to KafkaConsumer:
> {code}
> void pause(String... topics);
> void unpause(String... topics);
> {code}
> When a topic is paused, a call to KafkaConsumer.poll will not initiate any
> new fetches for that topic. After it is unpaused, fetches will begin again.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
