[jira] [Commented] (GEODE-8977) Thread monitoring service should also show locked monitors and synchronizers
[ https://issues.apache.org/jira/browse/GEODE-8977?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17550712#comment-17550712 ] ASF subversion and git services commented on GEODE-8977: Commit f05ef0d36322ac229bdfe790a89bc2e1b439645b in geode's branch refs/heads/support/1.15 from Darrel Schneider [ https://gitbox.apache.org/repos/asf?p=geode.git;h=f05ef0d363 ] GEODE-8977: change ThreadMonitor to reduce how long it does a "stop the world" ThreadDump vm op (#7751) Now uses a cheaper getThreadInfo that does not get lock info by default and calls getThreadInfo for each stuck thread. These are the defaults because they have the shortest time do the the VM ThreadDump operation. To get locks set the system property "gemfire.threadmonitor.showLocks" to "true". To get ThreadInfo on all stuck threads with a single call set the system property "gemfire.threadmonitor.batchCalls" to "true". (cherry picked from commit 3df1e76ddbf2ab8f95e4b337b99b65117054af76) > Thread monitoring service should also show locked monitors and synchronizers > > > Key: GEODE-8977 > URL: https://issues.apache.org/jira/browse/GEODE-8977 > Project: Geode > Issue Type: Improvement > Components: core >Affects Versions: 1.15.0 >Reporter: Darrel Schneider >Assignee: Darrel Schneider >Priority: Major > Labels: GeodeOperationAPI, blocks-1.15.0, pull-request-available > Fix For: 1.15.0, 1.16.0 > > > The thread monitoring service shows the call stack of a hung thread but it > does not show the synchronizations obtained by the frames in the call stack > like a normal stack dump does. > It looks like this is available from the ThreadInfo class that the service is > already using by calling getLockedMonitors and getLockedSynchronizers. The > getLockedMonitors returns a MonitorInfo which has information in it about > which frame of the stack obtained it. MonitorInfo subclasses LockInfo which > is what getLockedSynchronizers returns so it is possible that > getLockedSynchronizers does not provide any additional information to be > logged. -- This message was sent by Atlassian Jira (v8.20.7#820007)
[jira] [Commented] (GEODE-8977) Thread monitoring service should also show locked monitors and synchronizers
[ https://issues.apache.org/jira/browse/GEODE-8977?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17550710#comment-17550710 ] Darrel Schneider commented on GEODE-8977: - We figured out that getThreadInfo is an expensive VM ThreadDump operation that on most JVMs is done in a global safepoint. This means that all other threads are paused (once each of them reaches a safepoint in their execution) and then the ThreadDump is done. This can cause what appears to be a gc stop the world collection. To reduce the amount of time spend doing the ThreadDump operation, by default geode will not ask for lock/sync info but instead just get the stack trace. Also by default it will do the ThreadDump op on each stuck thread individually instead of with a single call that is passed all the thread ids. This reduces the amount of time spent in the op and on Java 15 and later it uses a thread-local safepoint instead of a global safepoint. To get locks set the system property "gemfire.threadmonitor.showLocks" to "true". To get ThreadInfo on all stuck threads with a single call set the system property "gemfire.threadmonitor.batchCalls" to "true". > Thread monitoring service should also show locked monitors and synchronizers > > > Key: GEODE-8977 > URL: https://issues.apache.org/jira/browse/GEODE-8977 > Project: Geode > Issue Type: Improvement > Components: core >Affects Versions: 1.15.0 >Reporter: Darrel Schneider >Assignee: Darrel Schneider >Priority: Major > Labels: GeodeOperationAPI, blocks-1.15.0, pull-request-available > Fix For: 1.15.0, 1.16.0 > > > The thread monitoring service shows the call stack of a hung thread but it > does not show the synchronizations obtained by the frames in the call stack > like a normal stack dump does. > It looks like this is available from the ThreadInfo class that the service is > already using by calling getLockedMonitors and getLockedSynchronizers. The > getLockedMonitors returns a MonitorInfo which has information in it about > which frame of the stack obtained it. MonitorInfo subclasses LockInfo which > is what getLockedSynchronizers returns so it is possible that > getLockedSynchronizers does not provide any additional information to be > logged. -- This message was sent by Atlassian Jira (v8.20.7#820007)
[jira] [Commented] (GEODE-8977) Thread monitoring service should also show locked monitors and synchronizers
[ https://issues.apache.org/jira/browse/GEODE-8977?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17550699#comment-17550699 ] ASF subversion and git services commented on GEODE-8977: Commit 3df1e76ddbf2ab8f95e4b337b99b65117054af76 in geode's branch refs/heads/develop from Darrel Schneider [ https://gitbox.apache.org/repos/asf?p=geode.git;h=3df1e76ddb ] GEODE-8977: change ThreadMonitor to reduce how long it does a "stop the world" ThreadDump vm op (#7751) Now uses a cheaper getThreadInfo that does not get lock info by default and calls getThreadInfo for each stuck thread. These are the defaults because they have the shortest time do the the VM ThreadDump operation. To get locks set the system property "gemfire.threadmonitor.showLocks" to "true". To get ThreadInfo on all stuck threads with a single call set the system property "gemfire.threadmonitor.batchCalls" to "true". > Thread monitoring service should also show locked monitors and synchronizers > > > Key: GEODE-8977 > URL: https://issues.apache.org/jira/browse/GEODE-8977 > Project: Geode > Issue Type: Improvement > Components: core >Affects Versions: 1.15.0 >Reporter: Darrel Schneider >Assignee: Darrel Schneider >Priority: Major > Labels: GeodeOperationAPI, blocks-1.15.0, pull-request-available > Fix For: 1.15.0 > > > The thread monitoring service shows the call stack of a hung thread but it > does not show the synchronizations obtained by the frames in the call stack > like a normal stack dump does. > It looks like this is available from the ThreadInfo class that the service is > already using by calling getLockedMonitors and getLockedSynchronizers. The > getLockedMonitors returns a MonitorInfo which has information in it about > which frame of the stack obtained it. MonitorInfo subclasses LockInfo which > is what getLockedSynchronizers returns so it is possible that > getLockedSynchronizers does not provide any additional information to be > logged. -- This message was sent by Atlassian Jira (v8.20.7#820007)
[jira] [Commented] (GEODE-8977) Thread monitoring service should also show locked monitors and synchronizers
[ https://issues.apache.org/jira/browse/GEODE-8977?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17335053#comment-17335053 ] ASF subversion and git services commented on GEODE-8977: Commit 8d99d08dade997b0e530d5d7be29ef0e02645e2a in geode's branch refs/heads/develop from Darrel Schneider [ https://gitbox.apache.org/repos/asf?p=geode.git;h=8d99d08 ] GEODE-8977: improve thread monitor to log lock information (#6377) * Revert "Revert "GEODE-8977: include syncs in thread monitor stack (#6248)" (#6368)" This reverts commit f366940bb4172084412ff6943c3793a6e6ed1b77. * getThreadInfo is now called at most once each time the thread monitor wakes up. Before it was called once for every thread that might be stuck. > Thread monitoring service should also show locked monitors and synchronizers > > > Key: GEODE-8977 > URL: https://issues.apache.org/jira/browse/GEODE-8977 > Project: Geode > Issue Type: Improvement > Components: core >Reporter: Darrel Schneider >Assignee: Darrel Schneider >Priority: Major > Labels: GeodeOperationAPI, pull-request-available > Fix For: 1.15.0 > > > The thread monitoring service shows the call stack of a hung thread but it > does not show the synchronizations obtained by the frames in the call stack > like a normal stack dump does. > It looks like this is available from the ThreadInfo class that the service is > already using by calling getLockedMonitors and getLockedSynchronizers. The > getLockedMonitors returns a MonitorInfo which has information in it about > which frame of the stack obtained it. MonitorInfo subclasses LockInfo which > is what getLockedSynchronizers returns so it is possible that > getLockedSynchronizers does not provide any additional information to be > logged. -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Commented] (GEODE-8977) Thread monitoring service should also show locked monitors and synchronizers
[ https://issues.apache.org/jira/browse/GEODE-8977?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17335054#comment-17335054 ] ASF subversion and git services commented on GEODE-8977: Commit 8d99d08dade997b0e530d5d7be29ef0e02645e2a in geode's branch refs/heads/develop from Darrel Schneider [ https://gitbox.apache.org/repos/asf?p=geode.git;h=8d99d08 ] GEODE-8977: improve thread monitor to log lock information (#6377) * Revert "Revert "GEODE-8977: include syncs in thread monitor stack (#6248)" (#6368)" This reverts commit f366940bb4172084412ff6943c3793a6e6ed1b77. * getThreadInfo is now called at most once each time the thread monitor wakes up. Before it was called once for every thread that might be stuck. > Thread monitoring service should also show locked monitors and synchronizers > > > Key: GEODE-8977 > URL: https://issues.apache.org/jira/browse/GEODE-8977 > Project: Geode > Issue Type: Improvement > Components: core >Reporter: Darrel Schneider >Assignee: Darrel Schneider >Priority: Major > Labels: GeodeOperationAPI, pull-request-available > Fix For: 1.15.0 > > > The thread monitoring service shows the call stack of a hung thread but it > does not show the synchronizations obtained by the frames in the call stack > like a normal stack dump does. > It looks like this is available from the ThreadInfo class that the service is > already using by calling getLockedMonitors and getLockedSynchronizers. The > getLockedMonitors returns a MonitorInfo which has information in it about > which frame of the stack obtained it. MonitorInfo subclasses LockInfo which > is what getLockedSynchronizers returns so it is possible that > getLockedSynchronizers does not provide any additional information to be > logged. -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Commented] (GEODE-8977) Thread monitoring service should also show locked monitors and synchronizers
[ https://issues.apache.org/jira/browse/GEODE-8977?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17329472#comment-17329472 ] ASF subversion and git services commented on GEODE-8977: Commit f366940bb4172084412ff6943c3793a6e6ed1b77 in geode's branch refs/heads/develop from Darrel Schneider [ https://gitbox.apache.org/repos/asf?p=geode.git;h=f366940 ] Revert "GEODE-8977: include syncs in thread monitor stack (#6248)" (#6368) This reverts commit 903afb5332b06b10e7f53a1abe9c53775a94f39e. > Thread monitoring service should also show locked monitors and synchronizers > > > Key: GEODE-8977 > URL: https://issues.apache.org/jira/browse/GEODE-8977 > Project: Geode > Issue Type: Improvement > Components: core >Reporter: Darrel Schneider >Assignee: Darrel Schneider >Priority: Major > Labels: GeodeOperationAPI, pull-request-available > Fix For: 1.15.0 > > > The thread monitoring service shows the call stack of a hung thread but it > does not show the synchronizations obtained by the frames in the call stack > like a normal stack dump does. > It looks like this is available from the ThreadInfo class that the service is > already using by calling getLockedMonitors and getLockedSynchronizers. The > getLockedMonitors returns a MonitorInfo which has information in it about > which frame of the stack obtained it. MonitorInfo subclasses LockInfo which > is what getLockedSynchronizers returns so it is possible that > getLockedSynchronizers does not provide any additional information to be > logged. -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Commented] (GEODE-8977) Thread monitoring service should also show locked monitors and synchronizers
[ https://issues.apache.org/jira/browse/GEODE-8977?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17314953#comment-17314953 ] ASF subversion and git services commented on GEODE-8977: Commit 903afb5332b06b10e7f53a1abe9c53775a94f39e in geode's branch refs/heads/develop from Darrel Schneider [ https://gitbox.apache.org/repos/asf?p=geode.git;h=903afb5 ] GEODE-8977: include syncs in thread monitor stack (#6248) Thread monitor stack now includes syncs and locked objects. It has also been reformated to look more like a normal stack dump. > Thread monitoring service should also show locked monitors and synchronizers > > > Key: GEODE-8977 > URL: https://issues.apache.org/jira/browse/GEODE-8977 > Project: Geode > Issue Type: Improvement > Components: core >Reporter: Darrel Schneider >Assignee: Darrel Schneider >Priority: Major > Labels: GeodeOperationAPI, pull-request-available > Fix For: 1.15.0 > > > The thread monitoring service shows the call stack of a hung thread but it > does not show the synchronizations obtained by the frames in the call stack > like a normal stack dump does. > It looks like this is available from the ThreadInfo class that the service is > already using by calling getLockedMonitors and getLockedSynchronizers. The > getLockedMonitors returns a MonitorInfo which has information in it about > which frame of the stack obtained it. MonitorInfo subclasses LockInfo which > is what getLockedSynchronizers returns so it is possible that > getLockedSynchronizers does not provide any additional information to be > logged. -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Commented] (GEODE-8977) Thread monitoring service should also show locked monitors and synchronizers
[
https://issues.apache.org/jira/browse/GEODE-8977?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17314952#comment-17314952
]
Darrel Schneider commented on GEODE-8977:
-
Before:
{noformat}
[warn 2021/04/02 10:01:17.496 PDT tid=0xf] Thread <58> (0x3a)
that was executed at <02 Apr 2021 10:01:10 PDT> has been stuck for <7.48
seconds> and number of thread monitor iteration <1>
Thread Name state
Executor Group
Monitored metric
Thread stack:
java.lang.Thread.sleep(Native Method)
TestFunction.sleep(TestFunction.java:86)
TestFunction.execute(TestFunction.java:35)
org.apache.geode.internal.cache.execute.AbstractExecution.executeFunctionLocally(AbstractExecution.java:328)
org.apache.geode.internal.cache.execute.AbstractExecution.lambda$executeFunctionOnLocalPRNode$0(AbstractExecution.java:273)
org.apache.geode.internal.cache.execute.AbstractExecution$$Lambda$317/1611044977.run(Unknown
Source)
java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142)
java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)
org.apache.geode.distributed.internal.ClusterOperationExecutors.runUntilShutdown(ClusterOperationExecutors.java:442)
org.apache.geode.distributed.internal.ClusterOperationExecutors.doFunctionExecutionThread(ClusterOperationExecutors.java:377)
org.apache.geode.distributed.internal.ClusterOperationExecutors$$Lambda$63/618199162.invoke(Unknown
Source)
org.apache.geode.logging.internal.executors.LoggingThreadFactory.lambda$newThread$0(LoggingThreadFactory.java:119)
org.apache.geode.logging.internal.executors.LoggingThreadFactory$$Lambda$53/873134840.run(Unknown
Source)
java.lang.Thread.run(Thread.java:745)
{noformat}
After:
{noformat}
[warn 2021/04/02 09:55:33.626 PDT tid=0xf] Thread <58> (0x3a)
that was executed at <02 Apr 2021 09:55:18 PDT> has been stuck for <14.758
seconds> and number of thread monitor iteration <2>
Thread Name state
Executor Group
Monitored metric
Thread stack for "Function Execution Processor2" (0x3a):
java.lang.ThreadState: TIMED_WAITING
at java.lang.Thread.sleep(Native Method)
at TestFunction.sleep(TestFunction.java:86)
at TestFunction.execute(TestFunction.java:35)
- locked java.lang.Object@587c50ca
at
org.apache.geode.internal.cache.execute.AbstractExecution.executeFunctionLocally(AbstractExecution.java:328)
at
org.apache.geode.internal.cache.execute.AbstractExecution.lambda$executeFunctionOnLocalPRNode$0(AbstractExecution.java:273)
at
org.apache.geode.internal.cache.execute.AbstractExecution$$Lambda$317/122802218.run(Unknown
Source)
at
java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142)
at
java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)
at
org.apache.geode.distributed.internal.ClusterOperationExecutors.runUntilShutdown(ClusterOperationExecutors.java:442)
at
org.apache.geode.distributed.internal.ClusterOperationExecutors.doFunctionExecutionThread(ClusterOperationExecutors.java:377)
at
org.apache.geode.distributed.internal.ClusterOperationExecutors$$Lambda$63/618199162.invoke(Unknown
Source)
at
org.apache.geode.logging.internal.executors.LoggingThreadFactory.lambda$newThread$0(LoggingThreadFactory.java:119)
at
org.apache.geode.logging.internal.executors.LoggingThreadFactory$$Lambda$53/873134840.run(Unknown
Source)
at java.lang.Thread.run(Thread.java:745)
Locked ownable synchronizers:
- java.util.concurrent.ThreadPoolExecutor$Worker@42d21af5
{noformat}
> Thread monitoring service should also show locked monitors and synchronizers
>
>
> Key: GEODE-8977
> URL: https://issues.apache.org/jira/browse/GEODE-8977
> Project: Geode
> Issue Type: Improvement
> Components: core
>Reporter: Darrel Schneider
>Assignee: Darrel Schneider
>Priority: Major
> Labels: GeodeOperationAPI, pull-request-available
>
> The thread monitoring service shows the call stack of a hung thread but it
> does not show the synchronizations obtained by the frames in the call stack
> like a normal stack dump does.
> It looks like this is available from the ThreadInfo class that the service is
> already using by calling getLockedMonitors and getLockedSynchronizers. The
> getLockedMonitors returns a MonitorInfo which has information in it about
> which frame of the stack obtained it. MonitorInfo subclasses LockInfo which
> is what getLockedSynchronizers returns so it is possible that
> getLockedSynchronizers does not provide any additional information to be
> logged.
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
