[
https://issues.apache.org/jira/browse/KAFKA-3727?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15346902#comment-15346902
]
Jason Gustafson commented on KAFKA-3727:
----------------------------------------
[~ecomar] Super late in chiming in (finally caught up to this JIRA), but here's
my basic thought and you can tell me if it makes sense or not. My main concern
at the moment is uncertainty over how KIP-4 will ultimately impact the client's
handling of unknown topics. One option that has been mentioned on the mail list
is to deprecate autocreation on the server and basically move the option over
to the clients (either to the producer only or to both the producer and
consumer).
Alternatively, if KIP-4 gives us an AdminClient, then I can imagine changing
the behavior to simply propagate an exception to the user when we encounter an
unknown topic. This lets them retry if that's the behavior they want or
alternatively create the topic using the admin client. They can even propagate
the exception further if they would rather fail. This is the option I'd
personally like to see long term, but we have to talk through the
compatibility.
The listener approach you're proposing is also viable and has the advantage of
preserving compatibility. I think you could either start a KIP for this change
now, or we can wait for the fallout of KIP-4 to become clearer. It would be
nice to hear what [~granthenke] thinks, since he's driving KIP-4 development.
> Consumer.poll() stuck in loop on non-existent topic manually assigned
> ---------------------------------------------------------------------
>
> Key: KAFKA-3727
> URL: https://issues.apache.org/jira/browse/KAFKA-3727
> Project: Kafka
> Issue Type: Bug
> Components: clients
> Reporter: Edoardo Comar
> Assignee: Edoardo Comar
> Priority: Critical
>
> The behavior of a consumer on poll() for a non-existing topic is surprisingly
> different/inconsistent
> between a consumer that subscribed to the topic and one that had the
> topic-partition manually assigned.
> The "subscribed" consumer will return an empty collection
> The "assigned" consumer will *loop forever* - this feels a bug to me.
> sample snippet to reproduce:
> {quote}
> KafkaConsumer<String, String> assignKc = new KafkaConsumer<>(props1);
> KafkaConsumer<String, String> subsKc = new KafkaConsumer<>(props2);
> List<TopicPartition> tps = new ArrayList<>();
> tps.add(new TopicPartition("topic-not-exists", 0));
> assignKc.assign(tps);
> subsKc.subscribe(Arrays.asList("topic-not-exists"));
> System.out.println("********* subscribe k consumer ");
> ConsumerRecords<String, String> crs2 = subsKc.poll(1000L);
> print("subscribeKc", crs2); // returns empty
> System.out.println("********* assign k consumer ");
> ConsumerRecords<String, String> crs1 = assignKc.poll(1000L);
> // will loop forever !
> print("assignKc", crs1);
> {quote}
> the logs for the "assigned" consumer show:
> [2016-05-18 17:33:09,907] DEBUG Updated cluster metadata version 8 to
> Cluster(nodes = [192.168.10.18:9093 (id: 0 rack: null)], partitions = [])
> (org.apache.kafka.clients.Metadata)
> [2016-05-18 17:33:09,908] DEBUG Partition topic-not-exists-0 is unknown for
> fetching offset, wait for metadata refresh
> (org.apache.kafka.clients.consumer.internals.Fetcher)
> [2016-05-18 17:33:10,010] DEBUG Sending metadata request
> {topics=[topic-not-exists]} to node 0 (org.apache.kafka.clients.NetworkClient)
> [2016-05-18 17:33:10,011] WARN Error while fetching metadata with correlation
> id 9 : {topic-not-exists=UNKNOWN_TOPIC_OR_PARTITION}
> (org.apache.kafka.clients.NetworkClient)
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
