[jira] [Commented] (KAFKA-5263) kakfa-clients consume 100% CPU with manual partition assignment when network connection is lost

[Konstantin Smirnov (JIRA)]

[ 
https://issues.apache.org/jira/browse/KAFKA-5263?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16024593#comment-16024593
 ] 

Konstantin Smirnov commented on KAFKA-5263:
---

Thanks a lot for your fix!

> kakfa-clients consume 100% CPU with manual partition assignment when network 
> connection is lost
> ---
>
> Key: KAFKA-5263
> URL: https://issues.apache.org/jira/browse/KAFKA-5263
> Project: Kafka
>  Issue Type: Bug
>  Components: clients
>Affects Versions: 0.10.1.0, 0.10.1.1, 0.10.2.0, 0.10.2.1
>Reporter: Konstantin Smirnov
>Assignee: Rajini Sivaram
> Fix For: 0.11.0.0, 0.11.1.0
>
> Attachments: cpu_consuming.log, cpu_consuming_profile.csv
>
>
> Noticed that lose of the connection to Kafka broker leads kafka-clients to 
> consume 100% CPU. The bug only appears when the manual partition assignmet is 
> used. It appears since the version 0.10.1.0. The bug is quite similar to 
> KAFKA-1642.



--
This message was sent by Atlassian JIRA
(v6.3.15#6346)

[jira] [Commented] (KAFKA-5263) kakfa-clients consume 100% CPU with manual partition assignment when network connection is lost

[ASF GitHub Bot (JIRA)]

[ 
https://issues.apache.org/jira/browse/KAFKA-5263?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16024492#comment-16024492
 ] 

ASF GitHub Bot commented on KAFKA-5263:
---

Github user asfgit closed the pull request at:

https://github.com/apache/kafka/pull/3124


> kakfa-clients consume 100% CPU with manual partition assignment when network 
> connection is lost
> ---
>
> Key: KAFKA-5263
> URL: https://issues.apache.org/jira/browse/KAFKA-5263
> Project: Kafka
>  Issue Type: Bug
>  Components: clients
>Affects Versions: 0.10.1.0, 0.10.1.1, 0.10.2.0, 0.10.2.1
>Reporter: Konstantin Smirnov
>Assignee: Rajini Sivaram
> Fix For: 0.11.0.0, 0.11.1.0
>
> Attachments: cpu_consuming.log, cpu_consuming_profile.csv
>
>
> Noticed that lose of the connection to Kafka broker leads kafka-clients to 
> consume 100% CPU. The bug only appears when the manual partition assignmet is 
> used. It appears since the version 0.10.1.0. The bug is quite similar to 
> KAFKA-1642.



--
This message was sent by Atlassian JIRA
(v6.3.15#6346)

[jira] [Commented] (KAFKA-5263) kakfa-clients consume 100% CPU with manual partition assignment when network connection is lost

[ASF GitHub Bot (JIRA)]

[ 
https://issues.apache.org/jira/browse/KAFKA-5263?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16021013#comment-16021013
 ] 

ASF GitHub Bot commented on KAFKA-5263:
---

GitHub user rajinisivaram opened a pull request:

https://github.com/apache/kafka/pull/3124

KAFKA-5263: Avoid tight polling loop in consumer with no ready nodes

If all brokers are unavailable, consumers using manual assignment poll in a 
tight loop with no channels to poll from. Producers and consumers using group 
management avoid this since they wait for metadata which attempt to create new 
connections with any necessary backoff. This PR does a similar wait for 
consumers using manual assignment when no nodes are ready.

You can merge this pull request into a Git repository by running:

$ git pull https://github.com/rajinisivaram/kafka KAFKA-5263

Alternatively you can review and apply these changes as the patch at:

https://github.com/apache/kafka/pull/3124.patch

To close this pull request, make a commit to your master/trunk branch
with (at least) the following in the commit message:

This closes #3124


commit e4cd05a61c3e5422b2a897004b41d523b5af8c16
Author: Rajini Sivaram 
Date:   2017-05-23T10:05:21Z

KAFKA-5263: Avoid tight polling loop in consumer with no ready nodes




> kakfa-clients consume 100% CPU with manual partition assignment when network 
> connection is lost
> ---
>
> Key: KAFKA-5263
> URL: https://issues.apache.org/jira/browse/KAFKA-5263
> Project: Kafka
>  Issue Type: Bug
>  Components: clients
>Affects Versions: 0.10.1.0, 0.10.1.1, 0.10.2.0, 0.10.2.1
>Reporter: Konstantin Smirnov
>Assignee: Rajini Sivaram
> Attachments: cpu_consuming.log, cpu_consuming_profile.csv
>
>
> Noticed that lose of the connection to Kafka broker leads kafka-clients to 
> consume 100% CPU. The bug only appears when the manual partition assignmet is 
> used. It appears since the version 0.10.1.0. The bug is quite similar to 
> KAFKA-1642.



--
This message was sent by Atlassian JIRA
(v6.3.15#6346)

[jira] [Commented] (KAFKA-5263) kakfa-clients consume 100% CPU with manual partition assignment

[Konstantin Smirnov (JIRA)]

[ 
https://issues.apache.org/jira/browse/KAFKA-5263?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16014226#comment-16014226
 ] 

Konstantin Smirnov commented on KAFKA-5263:
---

Sample code leading to the trouble:
{code}
public static void main(String[] args) {
System.out.println("Starting consumer");
Properties props = new Properties();
props.put("bootstrap.servers", "localhost:9092");
props.put("group.id", "test12345");
props.put("reconnect.backoff.ms", "1000");
props.put("retry.backoff.ms", "1000");
props.put("session.timeout.ms", "1");
props.put("key.deserializer", 
"org.apache.kafka.common.serialization.StringDeserializer");
props.put("value.deserializer", 
"org.apache.kafka.common.serialization.StringDeserializer");

try (KafkaConsumer consumer = new 
KafkaConsumer<>(props)) {
String topic = "test-topic"; 
//consumer.subscribe(Arrays.asList(topic));
List partitions = new ArrayList<>();
for (PartitionInfo partition : consumer.partitionsFor(topic)) {
partitions.add(new TopicPartition(topic, 
partition.partition()));
}
consumer.assign(partitions);
for (;;) {
ConsumerRecords records = consumer.poll(1000);
if (!records.isEmpty()) {
System.out.println("Records aren't empty!");
}
records.forEach(System.out::println);
}
}
}
{code}

> kakfa-clients consume 100% CPU with manual partition assignment
> ---
>
> Key: KAFKA-5263
> URL: https://issues.apache.org/jira/browse/KAFKA-5263
> Project: Kafka
>  Issue Type: Bug
>  Components: clients
>Affects Versions: 0.10.1.0, 0.10.1.1, 0.10.2.0, 0.10.2.1
>Reporter: Konstantin Smirnov
>
> Noticed that lose of the connection to Kafka broker leads kafka-clients to 
> consume 100% CPU. The bug only appears when the manual partition assignmet is 
> used. It appears since the version 0.10.1.0. The bug is quite similar to 
> KAFKA-1642.



--
This message was sent by Atlassian JIRA
(v6.3.15#6346)