[
https://issues.apache.org/jira/browse/KAFKA-1642?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14223161#comment-14223161
]
Bhavesh Mistry edited comment on KAFKA-1642 at 11/24/14 5:26 PM:
-----------------------------------------------------------------
[~ewencp],
The way to reproduce this is to simulate network instability by turning on and
off network service (or turn on/off physical cable). The connect and see if
recover and disconnect and connect again etc.. you will see the behavior again
and again.
The issue is also with connection state management :
{code}
private void initiateConnect(Node node, long now) {
try {
log.debug("Initiating connection to node {} at {}:{}.", node.id(),
node.host(), node.port());
// TODO FIX java.lang.IllegalStateException: No entry found for
node -3 (We need put before remove it..)..
this.connectionStates.connecting(node.id(), now); (This line has
problem because it will loose previous last attempt made get above exception
and it will try to connect to that node for ever and ever with exception )
selector.connect(node.id(), new InetSocketAddress(node.host(),
node.port()), this.socketSendBuffer, this.socketReceiveBuffer);
} catch (IOException e) {
/* attempt failed, we'll try again after the backoff */
connectionStates.disconnectedWhenConnectting(node.id());
/* maybe the problem is our metadata, update it */
metadata.requestUpdate();
log.debug("Error connecting to node {} at {}:{}:", node.id(),
node.host(), node.port(), e);
}
}
{code}
In my opinion, regardless of what node status is in run() method needs to be
safe-guarded from still CPU Cycle when there is no state for Node. (Hence I
have added exponential sleep as temp solution to not to still CPU cycle , I
think must protect it some how and check the execution time...)
Please let me know if you need more info and i will be more than happy to
reproduce bug and we can have conference call, and I can show you the problem.
Thanks,
Bhavesh
was (Author: bmis13):
[~ewencp],
The way to reproduce this is to simulate network instability by turning on and
off network service (or turn on/off physical cable). The connect and see if
recover and disconnect and connect again etc.. you will see the behavior again
and again.
The issue is also with connection state management :
{code}
private void initiateConnect(Node node, long now) {
try {
log.debug("Initiating connection to node {} at {}:{}.", node.id(),
node.host(), node.port());
// TODO FIX java.lang.IllegalStateException: No entry found for
node -3 (We need put before remove it..)..
this.connectionStates.connecting(node.id(), now); (This line has
problem because it will loose previous last attempt made get above exception
and it will try to connect to that node for ever and ever with exception )
selector.connect(node.id(), new InetSocketAddress(node.host(),
node.port()), this.socketSendBuffer, this.socketReceiveBuffer);
} catch (IOException e) {
/* attempt failed, we'll try again after the backoff */
connectionStates.disconnectedWhenConnectting(node.id());
/* maybe the problem is our metadata, update it */
metadata.requestUpdate();
log.debug("Error connecting to node {} at {}:{}:", node.id(),
node.host(), node.port(), e);
}
}
{code]
In my opinion, regardless of what node status is in run() method needs to be
safe-guarded from still CPU Cycle when there is no state for Node. (Hence I
have added exponential sleep as temp solution to not to still CPU cycle , I
think must protect it some how and check the execution time...)
Please let me know if you need more info and i will be more than happy to
reproduce bug and we can have conference call, and I can show you the problem.
Thanks,
Bhavesh
> [Java New Producer Kafka Trunk] CPU Usage Spike to 100% when network
> connection is lost
> ---------------------------------------------------------------------------------------
>
> Key: KAFKA-1642
> URL: https://issues.apache.org/jira/browse/KAFKA-1642
> Project: Kafka
> Issue Type: Bug
> Components: producer
> Affects Versions: 0.8.2
> Reporter: Bhavesh Mistry
> Assignee: Ewen Cheslack-Postava
> Fix For: 0.8.2
>
> Attachments:
> 0001-Initial-CPU-Hish-Usage-by-Kafka-FIX-and-Also-fix-CLO.patch,
> KAFKA-1642.patch, KAFKA-1642_2014-10-20_17:33:57.patch,
> KAFKA-1642_2014-10-23_16:19:41.patch
>
>
> I see my CPU spike to 100% when network connection is lost for while. It
> seems network IO thread are very busy logging following error message. Is
> this expected behavior ?
> 2014-09-17 14:06:16.830 [kafka-producer-network-thread] ERROR
> org.apache.kafka.clients.producer.internals.Sender - Uncaught error in kafka
> producer I/O thread:
> java.lang.IllegalStateException: No entry found for node -2
> at
> org.apache.kafka.clients.ClusterConnectionStates.nodeState(ClusterConnectionStates.java:110)
> at
> org.apache.kafka.clients.ClusterConnectionStates.disconnected(ClusterConnectionStates.java:99)
> at
> org.apache.kafka.clients.NetworkClient.initiateConnect(NetworkClient.java:394)
> at
> org.apache.kafka.clients.NetworkClient.maybeUpdateMetadata(NetworkClient.java:380)
> at org.apache.kafka.clients.NetworkClient.poll(NetworkClient.java:174)
> at org.apache.kafka.clients.producer.internals.Sender.run(Sender.java:175)
> at org.apache.kafka.clients.producer.internals.Sender.run(Sender.java:115)
> at java.lang.Thread.run(Thread.java:744)
> Thanks,
> Bhavesh
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
