[jira] [Updated] (KAFKA-3727) inconsistent behavior of Consumer.poll() on non-existent topic when assigned vs subscribed
[
https://issues.apache.org/jira/browse/KAFKA-3727?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Edoardo Comar updated KAFKA-3727:
-
Description:
Inconsistent behavior of Consumer.poll() on non-existent topic when assigned vs
subscribed
The behavior of a consumer on poll() for a non-existing topic is surprisingly
different between a consumer that subscribed to the topic and one that had the
topic-partition manually assigned.
the "subscribed" consumer will return an empty collection
the "assigned" consumer will *loop forever*.
the latter behavior feels a bug to me.
sample snippet to reproduce:
{quote}
KafkaConsumer assignKc = new KafkaConsumer<>(props1);
KafkaConsumer subsKc = new KafkaConsumer<>(props2);
List tps = new ArrayList<>();
tps.add(new TopicPartition("topic-not-exists", 0));
assignKc.assign(tps);
subsKc.subscribe(Arrays.asList("topic-not-exists"));
System.out.println("* subscribe k consumer ");
ConsumerRecords crs2 = subsKc.poll(1000L);
print("subscribeKc", crs2); // returns empty
System.out.println("* assign k consumer ");
ConsumerRecords crs1 = assignKc.poll(1000L);
// will loop forever !
print("assignKc", crs1);
{quote}
the logs for the "assigned" consumer show:
[2016-05-18 17:33:09,907] DEBUG Updated cluster metadata version 8 to
Cluster(nodes = [192.168.10.18:9093 (id: 0 rack: null)], partitions = [])
(org.apache.kafka.clients.Metadata)
[2016-05-18 17:33:09,908] DEBUG Partition topic-not-exists-0 is unknown for
fetching offset, wait for metadata refresh
(org.apache.kafka.clients.consumer.internals.Fetcher)
[2016-05-18 17:33:10,010] DEBUG Sending metadata request
{topics=[topic-not-exists]} to node 0 (org.apache.kafka.clients.NetworkClient)
[2016-05-18 17:33:10,011] WARN Error while fetching metadata with correlation
id 9 : {topic-not-exists=UNKNOWN_TOPIC_OR_PARTITION}
(org.apache.kafka.clients.NetworkClient)
was:
The behavior of a consumer on poll() for a non-existing topic is surprisingly
different between a consumer that subscribed to the topic and one that had the
topic-partition manually assigned.
the "subscribed" consumer will return an empty collection
the "assigned" consumer will *loop forever*.
the latter behavior feels a bug to me.
sample snippet to reproduce:
{quote}
KafkaConsumer assignKc = new KafkaConsumer<>(props1);
KafkaConsumer subsKc = new KafkaConsumer<>(props2);
List tps = new ArrayList<>();
tps.add(new TopicPartition("topic-not-exists", 0));
assignKc.assign(tps);
subsKc.subscribe(Arrays.asList("topic-not-exists"));
System.out.println("* subscribe k consumer ");
ConsumerRecords crs2 = subsKc.poll(1000L);
print("subscribeKc", crs2); // returns empty
System.out.println("* assign k consumer ");
ConsumerRecords crs1 = assignKc.poll(1000L);
// will loop forever !
print("assignKc", crs1);
{quote}
the logs for the "assigned" consumer show:
[2016-05-18 17:33:09,907] DEBUG Updated cluster metadata version 8 to
Cluster(nodes = [192.168.10.18:9093 (id: 0 rack: null)], partitions = [])
(org.apache.kafka.clients.Metadata)
[2016-05-18 17:33:09,908] DEBUG Partition topic-not-exists-0 is unknown for
fetching offset, wait for metadata refresh
(org.apache.kafka.clients.consumer.internals.Fetcher)
[2016-05-18 17:33:10,010] DEBUG Sending metadata request
{topics=[topic-not-exists]} to node 0 (org.apache.kafka.clients.NetworkClient)
[2016-05-18 17:33:10,011] WARN Error while fetching metadata with correlation
id 9 : {topic-not-exists=UNKNOWN_TOPIC_OR_PARTITION}
(org.apache.kafka.clients.NetworkClient)
> inconsistent behavior of Consumer.poll() on non-existent topic when assigned
> vs subscribed
> --
>
> Key: KAFKA-3727
> URL: https://issues.apache.org/jira/browse/KAFKA-3727
> Project: Kafka
> Issue Type: Bug
> Components: clients
>Reporter: Edoardo Comar
>
> Inconsistent behavior of Consumer.poll() on non-existent topic when assigned
> vs subscribed
> The behavior of a consumer on poll() for a non-existing topic is surprisingly
> different between a consumer that subscribed to the topic and one that had
> the topic-partition manually assigned.
> the "subscribed" consumer will return an empty collection
> the "assigned" consumer will *loop forever*.
> the latter behavior feels a bug to me.
> sample snippet to reproduce:
> {quote}
> KafkaConsumer assignKc = new KafkaConsumer<>(props1);
>
[jira] [Updated] (KAFKA-3727) inconsistent behavior of Consumer.poll() on non-existent topic when assigned vs subscribed
[
https://issues.apache.org/jira/browse/KAFKA-3727?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Edoardo Comar updated KAFKA-3727:
-
Description:
The behavior of a consumer on poll() for a non-existing topic is surprisingly
different between a consumer that subscribed to the topic and one that had the
topic-partition manually assigned.
the "subscribed" consumer will return an empty collection
the "assigned" consumer will *loop forever*.
the latter behavior feels a bug to me.
sample snippet to reproduce:
{quote}
KafkaConsumer assignKc = new KafkaConsumer<>(props1);
KafkaConsumer subsKc = new KafkaConsumer<>(props2);
List tps = new ArrayList<>();
tps.add(new TopicPartition("topic-not-exists", 0));
assignKc.assign(tps);
subsKc.subscribe(Arrays.asList("topic-not-exists"));
System.out.println("* subscribe k consumer ");
ConsumerRecords crs2 = subsKc.poll(1000L);
print("subscribeKc", crs2); // returns empty
System.out.println("* assign k consumer ");
ConsumerRecords crs1 = assignKc.poll(1000L);
// will loop forever !
print("assignKc", crs1);
{quote}
the logs for the "assigned" consumer show:
[2016-05-18 17:33:09,907] DEBUG Updated cluster metadata version 8 to
Cluster(nodes = [192.168.10.18:9093 (id: 0 rack: null)], partitions = [])
(org.apache.kafka.clients.Metadata)
[2016-05-18 17:33:09,908] DEBUG Partition topic-not-exists-0 is unknown for
fetching offset, wait for metadata refresh
(org.apache.kafka.clients.consumer.internals.Fetcher)
[2016-05-18 17:33:10,010] DEBUG Sending metadata request
{topics=[topic-not-exists]} to node 0 (org.apache.kafka.clients.NetworkClient)
[2016-05-18 17:33:10,011] WARN Error while fetching metadata with correlation
id 9 : {topic-not-exists=UNKNOWN_TOPIC_OR_PARTITION}
(org.apache.kafka.clients.NetworkClient)
was:
The behavior of a consumer on poll() for a non-existing topic is surprisingly
different between a consumer that subscribed to the topic and one that had the
topic-partition manually assigned.
the "subscribed" consumer will return an empty collection
the "assigned" consumer will *loop forever*.
the latter behavior feels a bug to me.
{quote}
KafkaConsumer assignKc = new KafkaConsumer<>(props1);
KafkaConsumer subsKc = new KafkaConsumer<>(props2);
List tps = new ArrayList<>();
tps.add(new TopicPartition("topic-not-exists", 0));
assignKc.assign(tps);
subsKc.subscribe(Arrays.asList("topic-not-exists"));
System.out.println("* subscribe k consumer ");
ConsumerRecords crs2 = subsKc.poll(1000L);
print("subscribeKc", crs2); // returns empty
System.out.println("* assign k consumer ");
ConsumerRecords crs1 = assignKc.poll(1000L);
// will loop forever !
print("assignKc", crs1);
{quote}
the logs for the "assigned" consumer show:
[2016-05-18 17:33:09,907] DEBUG Updated cluster metadata version 8 to
Cluster(nodes = [192.168.10.18:9093 (id: 0 rack: null)], partitions = [])
(org.apache.kafka.clients.Metadata)
[2016-05-18 17:33:09,908] DEBUG Partition topic-not-exists-0 is unknown for
fetching offset, wait for metadata refresh
(org.apache.kafka.clients.consumer.internals.Fetcher)
[2016-05-18 17:33:10,010] DEBUG Sending metadata request
{topics=[topic-not-exists]} to node 0 (org.apache.kafka.clients.NetworkClient)
[2016-05-18 17:33:10,011] WARN Error while fetching metadata with correlation
id 9 : {topic-not-exists=UNKNOWN_TOPIC_OR_PARTITION}
(org.apache.kafka.clients.NetworkClient)
> inconsistent behavior of Consumer.poll() on non-existent topic when assigned
> vs subscribed
> --
>
> Key: KAFKA-3727
> URL: https://issues.apache.org/jira/browse/KAFKA-3727
> Project: Kafka
> Issue Type: Bug
> Components: clients
>Reporter: Edoardo Comar
>
> The behavior of a consumer on poll() for a non-existing topic is surprisingly
> different between a consumer that subscribed to the topic and one that had
> the topic-partition manually assigned.
> the "subscribed" consumer will return an empty collection
> the "assigned" consumer will *loop forever*.
> the latter behavior feels a bug to me.
> sample snippet to reproduce:
> {quote}
> KafkaConsumer assignKc = new KafkaConsumer<>(props1);
> KafkaConsumer subsKc = new KafkaConsumer<>(props2);
> List tps = new ArrayList<>();
> tps.add(new TopicPartition("topic-not-exists", 0));
> assignKc.assign(tps);
>
[jira] [Updated] (KAFKA-3727) inconsistent behavior of Consumer.poll() on non-existent topic when assigned vs subscribed
[
https://issues.apache.org/jira/browse/KAFKA-3727?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Edoardo Comar updated KAFKA-3727:
-
Description:
The behavior of a consumer on poll() for a non-existing topic is surprisingly
different between a consumer that subscribed to the topic and one that had the
topic-partition manually assigned.
the "subscribed" consumer will return an empty collection
the "assigned" consumer will *loop forever*.
the latter behavior feels a bug to me.
{quote}
KafkaConsumer assignKc = new KafkaConsumer<>(props1);
KafkaConsumer subsKc = new KafkaConsumer<>(props2);
List tps = new ArrayList<>();
tps.add(new TopicPartition("topic-not-exists", 0));
assignKc.assign(tps);
subsKc.subscribe(Arrays.asList("topic-not-exists"));
System.out.println("* subscribe k consumer ");
ConsumerRecords crs2 = subsKc.poll(1000L);
print("subscribeKc", crs2); // returns empty
System.out.println("* assign k consumer ");
ConsumerRecords crs1 = assignKc.poll(1000L);
// will loop forever !
print("assignKc", crs1);
{quote}
the logs for the "assigned" consumer show:
[2016-05-18 17:33:09,907] DEBUG Updated cluster metadata version 8 to
Cluster(nodes = [192.168.10.18:9093 (id: 0 rack: null)], partitions = [])
(org.apache.kafka.clients.Metadata)
[2016-05-18 17:33:09,908] DEBUG Partition topic-not-exists-0 is unknown for
fetching offset, wait for metadata refresh
(org.apache.kafka.clients.consumer.internals.Fetcher)
[2016-05-18 17:33:10,010] DEBUG Sending metadata request
{topics=[topic-not-exists]} to node 0 (org.apache.kafka.clients.NetworkClient)
[2016-05-18 17:33:10,011] WARN Error while fetching metadata with correlation
id 9 : {topic-not-exists=UNKNOWN_TOPIC_OR_PARTITION}
(org.apache.kafka.clients.NetworkClient)
was:
The behavior of a consumer on poll() for a non-existing topic is surprisingly
different between a consumer that subscribed to the topic and one that had the
topic-partition manually assigned.
the "subscribed" consumer will return an empty collection
the "assigned" consumer will *loop forever*.
the latter behavior
{quote}
KafkaConsumer assignKc = new KafkaConsumer<>(props1);
KafkaConsumer subsKc = new KafkaConsumer<>(props2);
List tps = new ArrayList<>();
tps.add(new TopicPartition("topic-not-exists", 0));
assignKc.assign(tps);
subsKc.subscribe(Arrays.asList("topic-not-exists"));
System.out.println("* subscribe k consumer ");
ConsumerRecords crs2 = subsKc.poll(1000L);
print("subscribeKc", crs2); // returns empty
System.out.println("* assign k consumer ");
ConsumerRecords crs1 = assignKc.poll(1000L);
// will loop forever !
print("assignKc", crs1);
{quote}
the logs for the "assigned" consumer show:
[2016-05-18 17:33:09,907] DEBUG Updated cluster metadata version 8 to
Cluster(nodes = [192.168.10.18:9093 (id: 0 rack: null)], partitions = [])
(org.apache.kafka.clients.Metadata)
[2016-05-18 17:33:09,908] DEBUG Partition topic-not-exists-0 is unknown for
fetching offset, wait for metadata refresh
(org.apache.kafka.clients.consumer.internals.Fetcher)
[2016-05-18 17:33:10,010] DEBUG Sending metadata request
{topics=[topic-not-exists]} to node 0 (org.apache.kafka.clients.NetworkClient)
[2016-05-18 17:33:10,011] WARN Error while fetching metadata with correlation
id 9 : {topic-not-exists=UNKNOWN_TOPIC_OR_PARTITION}
(org.apache.kafka.clients.NetworkClient)
> inconsistent behavior of Consumer.poll() on non-existent topic when assigned
> vs subscribed
> --
>
> Key: KAFKA-3727
> URL: https://issues.apache.org/jira/browse/KAFKA-3727
> Project: Kafka
> Issue Type: Bug
> Components: clients
>Reporter: Edoardo Comar
>
> The behavior of a consumer on poll() for a non-existing topic is surprisingly
> different between a consumer that subscribed to the topic and one that had
> the topic-partition manually assigned.
> the "subscribed" consumer will return an empty collection
> the "assigned" consumer will *loop forever*.
> the latter behavior feels a bug to me.
> {quote}
> KafkaConsumer assignKc = new KafkaConsumer<>(props1);
> KafkaConsumer subsKc = new KafkaConsumer<>(props2);
> List tps = new ArrayList<>();
> tps.add(new TopicPartition("topic-not-exists", 0));
> assignKc.assign(tps);
> subsKc.subscribe(Arrays.asList("topic-not-exists"));
>
