[jira] [Commented] (CASSANDRA-5175) Unbounded (?) thread growth connecting to an removed node
[
https://issues.apache.org/jira/browse/CASSANDRA-5175?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13721497#comment-13721497
]
Vijay commented on CASSANDRA-5175:
--
Yes there was another commit on top the attached patch to fix the test cases,
yes the logic has changed since calling close() is the only time we need to
stop the thread.
Current code in the repo
{code}
if (m == CLOSE_SENTINEL)
{
disconnect();
if (isStopped)
break;
continue;
}
{code}
> Unbounded (?) thread growth connecting to an removed node
> -
>
> Key: CASSANDRA-5175
> URL: https://issues.apache.org/jira/browse/CASSANDRA-5175
> Project: Cassandra
> Issue Type: Bug
> Components: Core
>Affects Versions: 1.1.8
> Environment: EC2, JDK 7u9, Ubuntu 12.04.1 LTS
>Reporter: Janne Jalkanen
>Assignee: Vijay
>Priority: Minor
> Fix For: 1.1.10, 1.2.1
>
> Attachments: 0001-CASSANDRA-5175.patch
>
>
> The following lines started repeating every minute in the log file
> {noformat}
> INFO [GossipStage:1] 2013-01-19 19:35:43,929 Gossiper.java (line 831)
> InetAddress /10.238.x.y is now dead.
> INFO [GossipStage:1] 2013-01-19 19:35:43,930 StorageService.java (line 1291)
> Removing token 170141183460469231731687303715884105718 for /10.238.x.y
> {noformat}
> Also, I got about 3000 threads which all look like this:
> {noformat}
> Name: WRITE-/10.238.x.y
> State: WAITING on
> java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject@1bb65c0f
> Total blocked: 0 Total waited: 3
> Stack trace:
> sun.misc.Unsafe.park(Native Method)
> java.util.concurrent.locks.LockSupport.park(LockSupport.java:186)
> java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await(AbstractQueuedSynchronizer.java:2043)
> java.util.concurrent.LinkedBlockingQueue.take(LinkedBlockingQueue.java:442)
> org.apache.cassandra.net.OutboundTcpConnection.run(OutboundTcpConnection.java:104)
> {noformat}
> A new thread seems to be created every minute, and they never go away.
> The endpoint in question had been a part of the cluster weeks ago, and the
> node exhibiting the thread growth was added yesterday.
> Anyway, assassinating the endpoint in question stopped thread growth (but
> kept the existing threads running), so this isn't a huge issue. But I don't
> think the thread count is supposed to be increasing like this...
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (CASSANDRA-5175) Unbounded (?) thread growth connecting to an removed node
[
https://issues.apache.org/jira/browse/CASSANDRA-5175?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13721091#comment-13721091
]
Minh Do commented on CASSANDRA-5175:
Hi Vijay,
I am using your commit db8705294ba96fe2b746fea4f26a919538653ebd but I think the
logic in this commit is not the same as the attached patch. Please take a look.
if (m == CLOSE_SENTINEL)
{
disconnect();
+if (!isStopped)
+break;
continue;
}
I think it should be :
if (isStopped)
break;
Thanks.
> Unbounded (?) thread growth connecting to an removed node
> -
>
> Key: CASSANDRA-5175
> URL: https://issues.apache.org/jira/browse/CASSANDRA-5175
> Project: Cassandra
> Issue Type: Bug
> Components: Core
>Affects Versions: 1.1.8
> Environment: EC2, JDK 7u9, Ubuntu 12.04.1 LTS
>Reporter: Janne Jalkanen
>Assignee: Vijay
>Priority: Minor
> Fix For: 1.1.10, 1.2.1
>
> Attachments: 0001-CASSANDRA-5175.patch
>
>
> The following lines started repeating every minute in the log file
> {noformat}
> INFO [GossipStage:1] 2013-01-19 19:35:43,929 Gossiper.java (line 831)
> InetAddress /10.238.x.y is now dead.
> INFO [GossipStage:1] 2013-01-19 19:35:43,930 StorageService.java (line 1291)
> Removing token 170141183460469231731687303715884105718 for /10.238.x.y
> {noformat}
> Also, I got about 3000 threads which all look like this:
> {noformat}
> Name: WRITE-/10.238.x.y
> State: WAITING on
> java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject@1bb65c0f
> Total blocked: 0 Total waited: 3
> Stack trace:
> sun.misc.Unsafe.park(Native Method)
> java.util.concurrent.locks.LockSupport.park(LockSupport.java:186)
> java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await(AbstractQueuedSynchronizer.java:2043)
> java.util.concurrent.LinkedBlockingQueue.take(LinkedBlockingQueue.java:442)
> org.apache.cassandra.net.OutboundTcpConnection.run(OutboundTcpConnection.java:104)
> {noformat}
> A new thread seems to be created every minute, and they never go away.
> The endpoint in question had been a part of the cluster weeks ago, and the
> node exhibiting the thread growth was added yesterday.
> Anyway, assassinating the endpoint in question stopped thread growth (but
> kept the existing threads running), so this isn't a huge issue. But I don't
> think the thread count is supposed to be increasing like this...
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (CASSANDRA-5175) Unbounded (?) thread growth connecting to an removed node
[
https://issues.apache.org/jira/browse/CASSANDRA-5175?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13560455#comment-13560455
]
Vijay commented on CASSANDRA-5175:
--
Also added a commit db8705294ba96fe2b746fea4f26a919538653ebd for test failure
(dtest), basically we should not close the thread because we convicted the
node. Thanks!
> Unbounded (?) thread growth connecting to an removed node
> -
>
> Key: CASSANDRA-5175
> URL: https://issues.apache.org/jira/browse/CASSANDRA-5175
> Project: Cassandra
> Issue Type: Bug
> Components: Core
>Affects Versions: 1.1.8
> Environment: EC2, JDK 7u9, Ubuntu 12.04.1 LTS
>Reporter: Janne Jalkanen
>Assignee: Vijay
>Priority: Minor
> Fix For: 1.1.10
>
> Attachments: 0001-CASSANDRA-5175.patch
>
>
> The following lines started repeating every minute in the log file
> {noformat}
> INFO [GossipStage:1] 2013-01-19 19:35:43,929 Gossiper.java (line 831)
> InetAddress /10.238.x.y is now dead.
> INFO [GossipStage:1] 2013-01-19 19:35:43,930 StorageService.java (line 1291)
> Removing token 170141183460469231731687303715884105718 for /10.238.x.y
> {noformat}
> Also, I got about 3000 threads which all look like this:
> {noformat}
> Name: WRITE-/10.238.x.y
> State: WAITING on
> java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject@1bb65c0f
> Total blocked: 0 Total waited: 3
> Stack trace:
> sun.misc.Unsafe.park(Native Method)
> java.util.concurrent.locks.LockSupport.park(LockSupport.java:186)
> java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await(AbstractQueuedSynchronizer.java:2043)
> java.util.concurrent.LinkedBlockingQueue.take(LinkedBlockingQueue.java:442)
> org.apache.cassandra.net.OutboundTcpConnection.run(OutboundTcpConnection.java:104)
> {noformat}
> A new thread seems to be created every minute, and they never go away.
> The endpoint in question had been a part of the cluster weeks ago, and the
> node exhibiting the thread growth was added yesterday.
> Anyway, assassinating the endpoint in question stopped thread growth (but
> kept the existing threads running), so this isn't a huge issue. But I don't
> think the thread count is supposed to be increasing like this...
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (CASSANDRA-5175) Unbounded (?) thread growth connecting to an removed node
[
https://issues.apache.org/jira/browse/CASSANDRA-5175?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13559896#comment-13559896
]
Brandon Williams commented on CASSANDRA-5175:
-
Hard to know for certain if this is truly the cause, but it sounds plausible. +1
> Unbounded (?) thread growth connecting to an removed node
> -
>
> Key: CASSANDRA-5175
> URL: https://issues.apache.org/jira/browse/CASSANDRA-5175
> Project: Cassandra
> Issue Type: Bug
> Components: Core
>Affects Versions: 1.1.8
> Environment: EC2, JDK 7u9, Ubuntu 12.04.1 LTS
>Reporter: Janne Jalkanen
>Assignee: Vijay
>Priority: Minor
> Fix For: 1.1.10
>
> Attachments: 0001-CASSANDRA-5175.patch
>
>
> The following lines started repeating every minute in the log file
> {noformat}
> INFO [GossipStage:1] 2013-01-19 19:35:43,929 Gossiper.java (line 831)
> InetAddress /10.238.x.y is now dead.
> INFO [GossipStage:1] 2013-01-19 19:35:43,930 StorageService.java (line 1291)
> Removing token 170141183460469231731687303715884105718 for /10.238.x.y
> {noformat}
> Also, I got about 3000 threads which all look like this:
> {noformat}
> Name: WRITE-/10.238.x.y
> State: WAITING on
> java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject@1bb65c0f
> Total blocked: 0 Total waited: 3
> Stack trace:
> sun.misc.Unsafe.park(Native Method)
> java.util.concurrent.locks.LockSupport.park(LockSupport.java:186)
> java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await(AbstractQueuedSynchronizer.java:2043)
> java.util.concurrent.LinkedBlockingQueue.take(LinkedBlockingQueue.java:442)
> org.apache.cassandra.net.OutboundTcpConnection.run(OutboundTcpConnection.java:104)
> {noformat}
> A new thread seems to be created every minute, and they never go away.
> The endpoint in question had been a part of the cluster weeks ago, and the
> node exhibiting the thread growth was added yesterday.
> Anyway, assassinating the endpoint in question stopped thread growth (but
> kept the existing threads running), so this isn't a huge issue. But I don't
> think the thread count is supposed to be increasing like this...
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators
For more information on JIRA, see: http://www.atlassian.com/software/jira
