[jira] [Commented] (ARTEMIS-1527) [Artemis Testsuite] ActiveMQMessageHandlerTest#testServerShutdownAndReconnect fails
[ https://issues.apache.org/jira/browse/ARTEMIS-1527?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16724254#comment-16724254 ] ASF GitHub Bot commented on ARTEMIS-1527: - Github user JiriOndrusek commented on the issue: https://github.com/apache/activemq-artemis/pull/1761 I agree, closing as requested in previous comment. > [Artemis Testsuite] ActiveMQMessageHandlerTest#testServerShutdownAndReconnect > fails > --- > > Key: ARTEMIS-1527 > URL: https://issues.apache.org/jira/browse/ARTEMIS-1527 > Project: ActiveMQ Artemis > Issue Type: Bug >Affects Versions: 1.5.5 >Reporter: Jiri Ondrusek >Priority: Major > > {code} > java.lang.AssertionError: null > at org.junit.Assert.fail(Assert.java:86) > at org.junit.Assert.assertTrue(Assert.java:41) > at org.junit.Assert.assertTrue(Assert.java:52) > at > org.apache.activemq.artemis.tests.integration.ra.ActiveMQMessageHandlerTest.testServerShutdownAndReconnect(ActiveMQMessageHandlerTest.java:245) > {code} > Maximum size of clients global thread pool is computed as {{8 * > Runtime.getRuntime().availableProcessors()}}. In the case of CPU with one > core, the thread pool has maximum size 8. > The test fails because of following situation. When the server is stopped, > all ClientSessionFactoryImpl try to reconnect and they add executor to the > thread pool which tries to reconnect ad anfinitum, see \[1\]. These threads > employ the whole thread pool and the rest is not able to work in normal way. > For example, I can see warnings \[2\]. > This situation is known as Starvation or Livelock \[3\]. I think that the > system should work properly even if the maximum size of thread pool will be 2. > *Steps to reproduce:* > Modify the test in the following way: > {code} > @Test >public void testServerShutdownAndReconnect() throws Exception { > ActiveMQClient.clearThreadPools(); > ActiveMQClient.setGlobalThreadPoolProperties(2, 1); > ActiveMQResourceAdapter qResourceAdapter = newResourceAdapter(); > ... > {code} > \[1\] > {code} > Thread[Signal Dispatcher,5,main] > com.ibm.misc.SignalDispatcher.waitForSignal(Native Method) > com.ibm.misc.SignalDispatcher.run(SignalDispatcher.java:73) > Thread[Thread-4 > (ActiveMQ-client-global-threads--90143455),5,ActiveMQ-client-global-threads--90143455] > sun.misc.Unsafe.park(Native Method) > java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:226) > > java.util.concurrent.locks.AbstractQueuedSynchronizer.doAcquireSharedNanos(AbstractQueuedSynchronizer.java:1048) > > java.util.concurrent.locks.AbstractQueuedSynchronizer.tryAcquireSharedNanos(AbstractQueuedSynchronizer.java:1339) > java.util.concurrent.CountDownLatch.await(CountDownLatch.java:288) > > org.apache.activemq.artemis.core.protocol.core.impl.ActiveMQClientProtocolManager.waitOnLatch(ActiveMQClientProtocolManager.java:134) > > org.apache.activemq.artemis.core.client.impl.ClientSessionFactoryImpl.getConnectionWithRetry(ClientSessionFactoryImpl.java:829) > > org.apache.activemq.artemis.core.client.impl.ClientSessionFactoryImpl.reconnectSessions(ClientSessionFactoryImpl.java:753) > > org.apache.activemq.artemis.core.client.impl.ClientSessionFactoryImpl.failoverOrReconnect(ClientSessionFactoryImpl.java:615) > > org.apache.activemq.artemis.core.client.impl.ClientSessionFactoryImpl.handleConnectionFailure(ClientSessionFactoryImpl.java:513) > > org.apache.activemq.artemis.core.client.impl.ClientSessionFactoryImpl.access$400(ClientSessionFactoryImpl.java:70) > > org.apache.activemq.artemis.core.client.impl.ClientSessionFactoryImpl$DelegatingFailureListener.connectionFailed(ClientSessionFactoryImpl.java:1191) > > org.apache.activemq.artemis.spi.core.protocol.AbstractRemotingConnection.callFailureListeners(AbstractRemotingConnection.java:64) > > org.apache.activemq.artemis.core.protocol.core.impl.RemotingConnectionImpl.fail(RemotingConnectionImpl.java:210) > > org.apache.activemq.artemis.spi.core.protocol.AbstractRemotingConnection.fail(AbstractRemotingConnection.java:206) > > org.apache.activemq.artemis.core.client.impl.ClientSessionFactoryImpl$CloseRunnable.run(ClientSessionFactoryImpl.java:1014) > > org.apache.activemq.artemis.utils.OrderedExecutorFactory$OrderedExecutor$ExecutorTask.run(OrderedExecutorFactory.java:100) > > java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1153) > > java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:628) > java.lang.Thread.run(Thread.java:785) > Thread[Thread-5 > (ActiveMQ-client-global-threads--90143455),5,ActiveMQ-client-global-threads--90143455] > sun.misc.Unsafe.park(Native Method) > java.util.concurrent.locks.
[jira] [Commented] (ARTEMIS-1527) [Artemis Testsuite] ActiveMQMessageHandlerTest#testServerShutdownAndReconnect fails
[ https://issues.apache.org/jira/browse/ARTEMIS-1527?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16724255#comment-16724255 ] ASF GitHub Bot commented on ARTEMIS-1527: - Github user JiriOndrusek closed the pull request at: https://github.com/apache/activemq-artemis/pull/1761 > [Artemis Testsuite] ActiveMQMessageHandlerTest#testServerShutdownAndReconnect > fails > --- > > Key: ARTEMIS-1527 > URL: https://issues.apache.org/jira/browse/ARTEMIS-1527 > Project: ActiveMQ Artemis > Issue Type: Bug >Affects Versions: 1.5.5 >Reporter: Jiri Ondrusek >Priority: Major > > {code} > java.lang.AssertionError: null > at org.junit.Assert.fail(Assert.java:86) > at org.junit.Assert.assertTrue(Assert.java:41) > at org.junit.Assert.assertTrue(Assert.java:52) > at > org.apache.activemq.artemis.tests.integration.ra.ActiveMQMessageHandlerTest.testServerShutdownAndReconnect(ActiveMQMessageHandlerTest.java:245) > {code} > Maximum size of clients global thread pool is computed as {{8 * > Runtime.getRuntime().availableProcessors()}}. In the case of CPU with one > core, the thread pool has maximum size 8. > The test fails because of following situation. When the server is stopped, > all ClientSessionFactoryImpl try to reconnect and they add executor to the > thread pool which tries to reconnect ad anfinitum, see \[1\]. These threads > employ the whole thread pool and the rest is not able to work in normal way. > For example, I can see warnings \[2\]. > This situation is known as Starvation or Livelock \[3\]. I think that the > system should work properly even if the maximum size of thread pool will be 2. > *Steps to reproduce:* > Modify the test in the following way: > {code} > @Test >public void testServerShutdownAndReconnect() throws Exception { > ActiveMQClient.clearThreadPools(); > ActiveMQClient.setGlobalThreadPoolProperties(2, 1); > ActiveMQResourceAdapter qResourceAdapter = newResourceAdapter(); > ... > {code} > \[1\] > {code} > Thread[Signal Dispatcher,5,main] > com.ibm.misc.SignalDispatcher.waitForSignal(Native Method) > com.ibm.misc.SignalDispatcher.run(SignalDispatcher.java:73) > Thread[Thread-4 > (ActiveMQ-client-global-threads--90143455),5,ActiveMQ-client-global-threads--90143455] > sun.misc.Unsafe.park(Native Method) > java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:226) > > java.util.concurrent.locks.AbstractQueuedSynchronizer.doAcquireSharedNanos(AbstractQueuedSynchronizer.java:1048) > > java.util.concurrent.locks.AbstractQueuedSynchronizer.tryAcquireSharedNanos(AbstractQueuedSynchronizer.java:1339) > java.util.concurrent.CountDownLatch.await(CountDownLatch.java:288) > > org.apache.activemq.artemis.core.protocol.core.impl.ActiveMQClientProtocolManager.waitOnLatch(ActiveMQClientProtocolManager.java:134) > > org.apache.activemq.artemis.core.client.impl.ClientSessionFactoryImpl.getConnectionWithRetry(ClientSessionFactoryImpl.java:829) > > org.apache.activemq.artemis.core.client.impl.ClientSessionFactoryImpl.reconnectSessions(ClientSessionFactoryImpl.java:753) > > org.apache.activemq.artemis.core.client.impl.ClientSessionFactoryImpl.failoverOrReconnect(ClientSessionFactoryImpl.java:615) > > org.apache.activemq.artemis.core.client.impl.ClientSessionFactoryImpl.handleConnectionFailure(ClientSessionFactoryImpl.java:513) > > org.apache.activemq.artemis.core.client.impl.ClientSessionFactoryImpl.access$400(ClientSessionFactoryImpl.java:70) > > org.apache.activemq.artemis.core.client.impl.ClientSessionFactoryImpl$DelegatingFailureListener.connectionFailed(ClientSessionFactoryImpl.java:1191) > > org.apache.activemq.artemis.spi.core.protocol.AbstractRemotingConnection.callFailureListeners(AbstractRemotingConnection.java:64) > > org.apache.activemq.artemis.core.protocol.core.impl.RemotingConnectionImpl.fail(RemotingConnectionImpl.java:210) > > org.apache.activemq.artemis.spi.core.protocol.AbstractRemotingConnection.fail(AbstractRemotingConnection.java:206) > > org.apache.activemq.artemis.core.client.impl.ClientSessionFactoryImpl$CloseRunnable.run(ClientSessionFactoryImpl.java:1014) > > org.apache.activemq.artemis.utils.OrderedExecutorFactory$OrderedExecutor$ExecutorTask.run(OrderedExecutorFactory.java:100) > > java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1153) > > java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:628) > java.lang.Thread.run(Thread.java:785) > Thread[Thread-5 > (ActiveMQ-client-global-threads--90143455),5,ActiveMQ-client-global-threads--90143455] > sun.misc.Unsafe.park(Native Method) > java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:226) > > jav
[jira] [Commented] (ARTEMIS-1527) [Artemis Testsuite] ActiveMQMessageHandlerTest#testServerShutdownAndReconnect fails
[ https://issues.apache.org/jira/browse/ARTEMIS-1527?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16724220#comment-16724220 ] ASF GitHub Bot commented on ARTEMIS-1527: - Github user jbertram commented on the issue: https://github.com/apache/activemq-artemis/pull/1761 @JiriOndrusek, I don't think this is going to get merged. Can you go ahead and close this PR? > [Artemis Testsuite] ActiveMQMessageHandlerTest#testServerShutdownAndReconnect > fails > --- > > Key: ARTEMIS-1527 > URL: https://issues.apache.org/jira/browse/ARTEMIS-1527 > Project: ActiveMQ Artemis > Issue Type: Bug >Affects Versions: 1.5.5 >Reporter: Jiri Ondrusek >Priority: Major > > {code} > java.lang.AssertionError: null > at org.junit.Assert.fail(Assert.java:86) > at org.junit.Assert.assertTrue(Assert.java:41) > at org.junit.Assert.assertTrue(Assert.java:52) > at > org.apache.activemq.artemis.tests.integration.ra.ActiveMQMessageHandlerTest.testServerShutdownAndReconnect(ActiveMQMessageHandlerTest.java:245) > {code} > Maximum size of clients global thread pool is computed as {{8 * > Runtime.getRuntime().availableProcessors()}}. In the case of CPU with one > core, the thread pool has maximum size 8. > The test fails because of following situation. When the server is stopped, > all ClientSessionFactoryImpl try to reconnect and they add executor to the > thread pool which tries to reconnect ad anfinitum, see \[1\]. These threads > employ the whole thread pool and the rest is not able to work in normal way. > For example, I can see warnings \[2\]. > This situation is known as Starvation or Livelock \[3\]. I think that the > system should work properly even if the maximum size of thread pool will be 2. > *Steps to reproduce:* > Modify the test in the following way: > {code} > @Test >public void testServerShutdownAndReconnect() throws Exception { > ActiveMQClient.clearThreadPools(); > ActiveMQClient.setGlobalThreadPoolProperties(2, 1); > ActiveMQResourceAdapter qResourceAdapter = newResourceAdapter(); > ... > {code} > \[1\] > {code} > Thread[Signal Dispatcher,5,main] > com.ibm.misc.SignalDispatcher.waitForSignal(Native Method) > com.ibm.misc.SignalDispatcher.run(SignalDispatcher.java:73) > Thread[Thread-4 > (ActiveMQ-client-global-threads--90143455),5,ActiveMQ-client-global-threads--90143455] > sun.misc.Unsafe.park(Native Method) > java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:226) > > java.util.concurrent.locks.AbstractQueuedSynchronizer.doAcquireSharedNanos(AbstractQueuedSynchronizer.java:1048) > > java.util.concurrent.locks.AbstractQueuedSynchronizer.tryAcquireSharedNanos(AbstractQueuedSynchronizer.java:1339) > java.util.concurrent.CountDownLatch.await(CountDownLatch.java:288) > > org.apache.activemq.artemis.core.protocol.core.impl.ActiveMQClientProtocolManager.waitOnLatch(ActiveMQClientProtocolManager.java:134) > > org.apache.activemq.artemis.core.client.impl.ClientSessionFactoryImpl.getConnectionWithRetry(ClientSessionFactoryImpl.java:829) > > org.apache.activemq.artemis.core.client.impl.ClientSessionFactoryImpl.reconnectSessions(ClientSessionFactoryImpl.java:753) > > org.apache.activemq.artemis.core.client.impl.ClientSessionFactoryImpl.failoverOrReconnect(ClientSessionFactoryImpl.java:615) > > org.apache.activemq.artemis.core.client.impl.ClientSessionFactoryImpl.handleConnectionFailure(ClientSessionFactoryImpl.java:513) > > org.apache.activemq.artemis.core.client.impl.ClientSessionFactoryImpl.access$400(ClientSessionFactoryImpl.java:70) > > org.apache.activemq.artemis.core.client.impl.ClientSessionFactoryImpl$DelegatingFailureListener.connectionFailed(ClientSessionFactoryImpl.java:1191) > > org.apache.activemq.artemis.spi.core.protocol.AbstractRemotingConnection.callFailureListeners(AbstractRemotingConnection.java:64) > > org.apache.activemq.artemis.core.protocol.core.impl.RemotingConnectionImpl.fail(RemotingConnectionImpl.java:210) > > org.apache.activemq.artemis.spi.core.protocol.AbstractRemotingConnection.fail(AbstractRemotingConnection.java:206) > > org.apache.activemq.artemis.core.client.impl.ClientSessionFactoryImpl$CloseRunnable.run(ClientSessionFactoryImpl.java:1014) > > org.apache.activemq.artemis.utils.OrderedExecutorFactory$OrderedExecutor$ExecutorTask.run(OrderedExecutorFactory.java:100) > > java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1153) > > java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:628) > java.lang.Thread.run(Thread.java:785) > Thread[Thread-5 > (ActiveMQ-client-global-threads--90143455),5,ActiveMQ-client-global-threads--90143455] > sun.misc.Unsafe.park(Native
[jira] [Commented] (ARTEMIS-1527) [Artemis Testsuite] ActiveMQMessageHandlerTest#testServerShutdownAndReconnect fails
[ https://issues.apache.org/jira/browse/ARTEMIS-1527?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16320630#comment-16320630 ] ASF GitHub Bot commented on ARTEMIS-1527: - Github user clebertsuconic commented on the issue: https://github.com/apache/activemq-artemis/pull/1761 -1000 on this PR... it's a brittle change over a test failure. .you could find other way to fix the test? it's a huge change on the maintenance 1.x branch... it's a risky change... if this is a go... it needs to be on master. (I wouldn't ever cherry-pick something this size into 1.x anyways). > [Artemis Testsuite] ActiveMQMessageHandlerTest#testServerShutdownAndReconnect > fails > --- > > Key: ARTEMIS-1527 > URL: https://issues.apache.org/jira/browse/ARTEMIS-1527 > Project: ActiveMQ Artemis > Issue Type: Bug >Affects Versions: 1.5.5 >Reporter: Jiri Ondrusek > > {code} > java.lang.AssertionError: null > at org.junit.Assert.fail(Assert.java:86) > at org.junit.Assert.assertTrue(Assert.java:41) > at org.junit.Assert.assertTrue(Assert.java:52) > at > org.apache.activemq.artemis.tests.integration.ra.ActiveMQMessageHandlerTest.testServerShutdownAndReconnect(ActiveMQMessageHandlerTest.java:245) > {code} > Maximum size of clients global thread pool is computed as {{8 * > Runtime.getRuntime().availableProcessors()}}. In the case of CPU with one > core, the thread pool has maximum size 8. > The test fails because of following situation. When the server is stopped, > all ClientSessionFactoryImpl try to reconnect and they add executor to the > thread pool which tries to reconnect ad anfinitum, see \[1\]. These threads > employ the whole thread pool and the rest is not able to work in normal way. > For example, I can see warnings \[2\]. > This situation is known as Starvation or Livelock \[3\]. I think that the > system should work properly even if the maximum size of thread pool will be 2. > *Steps to reproduce:* > Modify the test in the following way: > {code} > @Test >public void testServerShutdownAndReconnect() throws Exception { > ActiveMQClient.clearThreadPools(); > ActiveMQClient.setGlobalThreadPoolProperties(2, 1); > ActiveMQResourceAdapter qResourceAdapter = newResourceAdapter(); > ... > {code} > \[1\] > {code} > Thread[Signal Dispatcher,5,main] > com.ibm.misc.SignalDispatcher.waitForSignal(Native Method) > com.ibm.misc.SignalDispatcher.run(SignalDispatcher.java:73) > Thread[Thread-4 > (ActiveMQ-client-global-threads--90143455),5,ActiveMQ-client-global-threads--90143455] > sun.misc.Unsafe.park(Native Method) > java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:226) > > java.util.concurrent.locks.AbstractQueuedSynchronizer.doAcquireSharedNanos(AbstractQueuedSynchronizer.java:1048) > > java.util.concurrent.locks.AbstractQueuedSynchronizer.tryAcquireSharedNanos(AbstractQueuedSynchronizer.java:1339) > java.util.concurrent.CountDownLatch.await(CountDownLatch.java:288) > > org.apache.activemq.artemis.core.protocol.core.impl.ActiveMQClientProtocolManager.waitOnLatch(ActiveMQClientProtocolManager.java:134) > > org.apache.activemq.artemis.core.client.impl.ClientSessionFactoryImpl.getConnectionWithRetry(ClientSessionFactoryImpl.java:829) > > org.apache.activemq.artemis.core.client.impl.ClientSessionFactoryImpl.reconnectSessions(ClientSessionFactoryImpl.java:753) > > org.apache.activemq.artemis.core.client.impl.ClientSessionFactoryImpl.failoverOrReconnect(ClientSessionFactoryImpl.java:615) > > org.apache.activemq.artemis.core.client.impl.ClientSessionFactoryImpl.handleConnectionFailure(ClientSessionFactoryImpl.java:513) > > org.apache.activemq.artemis.core.client.impl.ClientSessionFactoryImpl.access$400(ClientSessionFactoryImpl.java:70) > > org.apache.activemq.artemis.core.client.impl.ClientSessionFactoryImpl$DelegatingFailureListener.connectionFailed(ClientSessionFactoryImpl.java:1191) > > org.apache.activemq.artemis.spi.core.protocol.AbstractRemotingConnection.callFailureListeners(AbstractRemotingConnection.java:64) > > org.apache.activemq.artemis.core.protocol.core.impl.RemotingConnectionImpl.fail(RemotingConnectionImpl.java:210) > > org.apache.activemq.artemis.spi.core.protocol.AbstractRemotingConnection.fail(AbstractRemotingConnection.java:206) > > org.apache.activemq.artemis.core.client.impl.ClientSessionFactoryImpl$CloseRunnable.run(ClientSessionFactoryImpl.java:1014) > > org.apache.activemq.artemis.utils.OrderedExecutorFactory$OrderedExecutor$ExecutorTask.run(OrderedExecutorFactory.java:100) > > java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1153) > > java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoo
[jira] [Commented] (ARTEMIS-1527) [Artemis Testsuite] ActiveMQMessageHandlerTest#testServerShutdownAndReconnect fails
[ https://issues.apache.org/jira/browse/ARTEMIS-1527?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16318499#comment-16318499 ] ASF GitHub Bot commented on ARTEMIS-1527: - Github user JiriOndrusek commented on a diff in the pull request: https://github.com/apache/activemq-artemis/pull/1761#discussion_r160421530 --- Diff: artemis-core-client/src/main/java/org/apache/activemq/artemis/core/client/impl/ClientSessionFactoryImpl.java --- @@ -990,17 +,26 @@ public CloseRunnable(RemotingConnection conn, String scaleDownTargetNodeID) { // can cause reconnect loop @Override public void run() { - try { -CLOSE_RUNNABLES.add(this); -if (scaleDownTargetNodeID == null) { - conn.fail(ActiveMQClientMessageBundle.BUNDLE.disconnected()); -} else { - conn.fail(ActiveMQClientMessageBundle.BUNDLE.disconnected(), scaleDownTargetNodeID); -} - } finally { -CLOSE_RUNNABLES.remove(this); + CLOSE_RUNNABLES.add(this); + CountDownLatch latch; + if (scaleDownTargetNodeID == null) { +latch = conn.fail(ActiveMQClientMessageBundle.BUNDLE.disconnected()); + } else { +latch = conn.fail(ActiveMQClientMessageBundle.BUNDLE.disconnected(), scaleDownTargetNodeID); } + //TODO(jondruse) is there constant or property usable for this kind of wait interval? --- End diff -- I have too use "constant" for repetitive checking, whether action is already finished. May I use some generic timeout used in project (or define new attribute, ...) > [Artemis Testsuite] ActiveMQMessageHandlerTest#testServerShutdownAndReconnect > fails > --- > > Key: ARTEMIS-1527 > URL: https://issues.apache.org/jira/browse/ARTEMIS-1527 > Project: ActiveMQ Artemis > Issue Type: Bug >Affects Versions: 1.5.5 >Reporter: Jiri Ondrusek > > {code} > java.lang.AssertionError: null > at org.junit.Assert.fail(Assert.java:86) > at org.junit.Assert.assertTrue(Assert.java:41) > at org.junit.Assert.assertTrue(Assert.java:52) > at > org.apache.activemq.artemis.tests.integration.ra.ActiveMQMessageHandlerTest.testServerShutdownAndReconnect(ActiveMQMessageHandlerTest.java:245) > {code} > Maximum size of clients global thread pool is computed as {{8 * > Runtime.getRuntime().availableProcessors()}}. In the case of CPU with one > core, the thread pool has maximum size 8. > The test fails because of following situation. When the server is stopped, > all ClientSessionFactoryImpl try to reconnect and they add executor to the > thread pool which tries to reconnect ad anfinitum, see \[1\]. These threads > employ the whole thread pool and the rest is not able to work in normal way. > For example, I can see warnings \[2\]. > This situation is known as Starvation or Livelock \[3\]. I think that the > system should work properly even if the maximum size of thread pool will be 2. > *Steps to reproduce:* > Modify the test in the following way: > {code} > @Test >public void testServerShutdownAndReconnect() throws Exception { > ActiveMQClient.clearThreadPools(); > ActiveMQClient.setGlobalThreadPoolProperties(2, 1); > ActiveMQResourceAdapter qResourceAdapter = newResourceAdapter(); > ... > {code} > \[1\] > {code} > Thread[Signal Dispatcher,5,main] > com.ibm.misc.SignalDispatcher.waitForSignal(Native Method) > com.ibm.misc.SignalDispatcher.run(SignalDispatcher.java:73) > Thread[Thread-4 > (ActiveMQ-client-global-threads--90143455),5,ActiveMQ-client-global-threads--90143455] > sun.misc.Unsafe.park(Native Method) > java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:226) > > java.util.concurrent.locks.AbstractQueuedSynchronizer.doAcquireSharedNanos(AbstractQueuedSynchronizer.java:1048) > > java.util.concurrent.locks.AbstractQueuedSynchronizer.tryAcquireSharedNanos(AbstractQueuedSynchronizer.java:1339) > java.util.concurrent.CountDownLatch.await(CountDownLatch.java:288) > > org.apache.activemq.artemis.core.protocol.core.impl.ActiveMQClientProtocolManager.waitOnLatch(ActiveMQClientProtocolManager.java:134) > > org.apache.activemq.artemis.core.client.impl.ClientSessionFactoryImpl.getConnectionWithRetry(ClientSessionFactoryImpl.java:829) > > org.apache.activemq.artemis.core.client.impl.ClientSessionFactoryImpl.reconnectSessions(ClientSessionFactoryImpl.java:753) > > org.apache.activemq.artemis.core.client.impl.ClientSessionFactoryImpl.failoverOrReconnect(ClientSessionFactoryImpl.java:615) > > org.apache.activemq.artemis.core.client.impl.ClientSessionFactoryI
[jira] [Commented] (ARTEMIS-1527) [Artemis Testsuite] ActiveMQMessageHandlerTest#testServerShutdownAndReconnect fails
[ https://issues.apache.org/jira/browse/ARTEMIS-1527?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16318498#comment-16318498 ] ASF GitHub Bot commented on ARTEMIS-1527: - Github user JiriOndrusek commented on the issue: https://github.com/apache/activemq-artemis/pull/1761 In 2 places I have to use "constant" for repetitive checking, whether action is already finished. May I use some generic timeout used in project (or define new attribute, ...) (ClientSessionFactoryImpl line 1123 and RemotingConnectionImpl line 254) > [Artemis Testsuite] ActiveMQMessageHandlerTest#testServerShutdownAndReconnect > fails > --- > > Key: ARTEMIS-1527 > URL: https://issues.apache.org/jira/browse/ARTEMIS-1527 > Project: ActiveMQ Artemis > Issue Type: Bug >Affects Versions: 1.5.5 >Reporter: Jiri Ondrusek > > {code} > java.lang.AssertionError: null > at org.junit.Assert.fail(Assert.java:86) > at org.junit.Assert.assertTrue(Assert.java:41) > at org.junit.Assert.assertTrue(Assert.java:52) > at > org.apache.activemq.artemis.tests.integration.ra.ActiveMQMessageHandlerTest.testServerShutdownAndReconnect(ActiveMQMessageHandlerTest.java:245) > {code} > Maximum size of clients global thread pool is computed as {{8 * > Runtime.getRuntime().availableProcessors()}}. In the case of CPU with one > core, the thread pool has maximum size 8. > The test fails because of following situation. When the server is stopped, > all ClientSessionFactoryImpl try to reconnect and they add executor to the > thread pool which tries to reconnect ad anfinitum, see \[1\]. These threads > employ the whole thread pool and the rest is not able to work in normal way. > For example, I can see warnings \[2\]. > This situation is known as Starvation or Livelock \[3\]. I think that the > system should work properly even if the maximum size of thread pool will be 2. > *Steps to reproduce:* > Modify the test in the following way: > {code} > @Test >public void testServerShutdownAndReconnect() throws Exception { > ActiveMQClient.clearThreadPools(); > ActiveMQClient.setGlobalThreadPoolProperties(2, 1); > ActiveMQResourceAdapter qResourceAdapter = newResourceAdapter(); > ... > {code} > \[1\] > {code} > Thread[Signal Dispatcher,5,main] > com.ibm.misc.SignalDispatcher.waitForSignal(Native Method) > com.ibm.misc.SignalDispatcher.run(SignalDispatcher.java:73) > Thread[Thread-4 > (ActiveMQ-client-global-threads--90143455),5,ActiveMQ-client-global-threads--90143455] > sun.misc.Unsafe.park(Native Method) > java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:226) > > java.util.concurrent.locks.AbstractQueuedSynchronizer.doAcquireSharedNanos(AbstractQueuedSynchronizer.java:1048) > > java.util.concurrent.locks.AbstractQueuedSynchronizer.tryAcquireSharedNanos(AbstractQueuedSynchronizer.java:1339) > java.util.concurrent.CountDownLatch.await(CountDownLatch.java:288) > > org.apache.activemq.artemis.core.protocol.core.impl.ActiveMQClientProtocolManager.waitOnLatch(ActiveMQClientProtocolManager.java:134) > > org.apache.activemq.artemis.core.client.impl.ClientSessionFactoryImpl.getConnectionWithRetry(ClientSessionFactoryImpl.java:829) > > org.apache.activemq.artemis.core.client.impl.ClientSessionFactoryImpl.reconnectSessions(ClientSessionFactoryImpl.java:753) > > org.apache.activemq.artemis.core.client.impl.ClientSessionFactoryImpl.failoverOrReconnect(ClientSessionFactoryImpl.java:615) > > org.apache.activemq.artemis.core.client.impl.ClientSessionFactoryImpl.handleConnectionFailure(ClientSessionFactoryImpl.java:513) > > org.apache.activemq.artemis.core.client.impl.ClientSessionFactoryImpl.access$400(ClientSessionFactoryImpl.java:70) > > org.apache.activemq.artemis.core.client.impl.ClientSessionFactoryImpl$DelegatingFailureListener.connectionFailed(ClientSessionFactoryImpl.java:1191) > > org.apache.activemq.artemis.spi.core.protocol.AbstractRemotingConnection.callFailureListeners(AbstractRemotingConnection.java:64) > > org.apache.activemq.artemis.core.protocol.core.impl.RemotingConnectionImpl.fail(RemotingConnectionImpl.java:210) > > org.apache.activemq.artemis.spi.core.protocol.AbstractRemotingConnection.fail(AbstractRemotingConnection.java:206) > > org.apache.activemq.artemis.core.client.impl.ClientSessionFactoryImpl$CloseRunnable.run(ClientSessionFactoryImpl.java:1014) > > org.apache.activemq.artemis.utils.OrderedExecutorFactory$OrderedExecutor$ExecutorTask.run(OrderedExecutorFactory.java:100) > > java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1153) > > java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:628) > java.lang.Thread.run(Thread.java:785) > Th
[jira] [Commented] (ARTEMIS-1527) [Artemis Testsuite] ActiveMQMessageHandlerTest#testServerShutdownAndReconnect fails
[ https://issues.apache.org/jira/browse/ARTEMIS-1527?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16318493#comment-16318493 ] ASF GitHub Bot commented on ARTEMIS-1527: - Github user JiriOndrusek commented on a diff in the pull request: https://github.com/apache/activemq-artemis/pull/1761#discussion_r160420636 --- Diff: artemis-core-client/src/main/java/org/apache/activemq/artemis/core/protocol/core/impl/RemotingConnectionImpl.java --- @@ -205,14 +211,59 @@ public void fail(final ActiveMQException me, String scaleDownTargetNodeID) { } // Then call the listeners - callFailureListeners(me, scaleDownTargetNodeID); + List failureLatches = callFailureListeners(me, scaleDownTargetNodeID); callClosingListeners(); - internalClose(); + CountDownLatch latch = new CountDownLatch(1); - for (Channel channel : channels.values()) { - channel.returnBlocking(me); + new ScheduledInternalClose(failureLatches, me, latch).run(); + + return latch; + } + + // internalClose has to be called after failureListeners are finished. + private class ScheduledInternalClose implements Runnable { + + private final List failureLatches; + private final ActiveMQException me; + private final CountDownLatch latch; + + public ScheduledInternalClose(List failureLatches, final ActiveMQException me, CountDownLatch latch) { + this.failureLatches = failureLatches; + this.me = me; + this.latch = latch; + } + + @Override + public void run() { + List running = new LinkedList<>(); + failureLatches.forEach((l) -> { +try { + //interval is defined during scheduled execution + if (!l.await(1, TimeUnit.MILLISECONDS)) { + running.add(l); + } +} catch (InterruptedException e) { + ActiveMQClientLogger.LOGGER.warn(e.getMessage(), e); +} + }); + + if (!running.isEmpty()) { +//TODO(jondruse) is there constant or propertyF usable for this kind of wait interval? +scheduledExecutorService.schedule(new ScheduledInternalClose(running, me, latch), 500, TimeUnit.MILLISECONDS); --- End diff -- I have too use "constant" for repetitive checking, whether action is already finished. May I use some generic timeout used in project (or define new attribute, ...) > [Artemis Testsuite] ActiveMQMessageHandlerTest#testServerShutdownAndReconnect > fails > --- > > Key: ARTEMIS-1527 > URL: https://issues.apache.org/jira/browse/ARTEMIS-1527 > Project: ActiveMQ Artemis > Issue Type: Bug >Affects Versions: 1.5.5 >Reporter: Jiri Ondrusek > > {code} > java.lang.AssertionError: null > at org.junit.Assert.fail(Assert.java:86) > at org.junit.Assert.assertTrue(Assert.java:41) > at org.junit.Assert.assertTrue(Assert.java:52) > at > org.apache.activemq.artemis.tests.integration.ra.ActiveMQMessageHandlerTest.testServerShutdownAndReconnect(ActiveMQMessageHandlerTest.java:245) > {code} > Maximum size of clients global thread pool is computed as {{8 * > Runtime.getRuntime().availableProcessors()}}. In the case of CPU with one > core, the thread pool has maximum size 8. > The test fails because of following situation. When the server is stopped, > all ClientSessionFactoryImpl try to reconnect and they add executor to the > thread pool which tries to reconnect ad anfinitum, see \[1\]. These threads > employ the whole thread pool and the rest is not able to work in normal way. > For example, I can see warnings \[2\]. > This situation is known as Starvation or Livelock \[3\]. I think that the > system should work properly even if the maximum size of thread pool will be 2. > *Steps to reproduce:* > Modify the test in the following way: > {code} > @Test >public void testServerShutdownAndReconnect() throws Exception { > ActiveMQClient.clearThreadPools(); > ActiveMQClient.setGlobalThreadPoolProperties(2, 1); > ActiveMQResourceAdapter qResourceAdapter = newResourceAdapter(); > ... > {code} > \[1\] > {code} > Thread[Signal Dispatcher,5,main] > com.ibm.misc.SignalDispatcher.waitForSignal(Native Method) > com.ibm.misc.SignalDispatcher.run(SignalDispatcher.java:73) > Thread[Thread-4 > (ActiveMQ-client-global-threads--90143455),5,ActiveMQ-client-global-threads--90143455] > sun.misc.Unsafe.park(Native Method) > java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:226) > >
[jira] [Commented] (ARTEMIS-1527) [Artemis Testsuite] ActiveMQMessageHandlerTest#testServerShutdownAndReconnect fails
[ https://issues.apache.org/jira/browse/ARTEMIS-1527?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16318490#comment-16318490 ] ASF GitHub Bot commented on ARTEMIS-1527: - Github user JiriOndrusek commented on the issue: https://github.com/apache/activemq-artemis/pull/1676 Replaced with https://github.com/apache/activemq-artemis/pull/1761 > [Artemis Testsuite] ActiveMQMessageHandlerTest#testServerShutdownAndReconnect > fails > --- > > Key: ARTEMIS-1527 > URL: https://issues.apache.org/jira/browse/ARTEMIS-1527 > Project: ActiveMQ Artemis > Issue Type: Bug >Affects Versions: 1.5.5 >Reporter: Jiri Ondrusek > > {code} > java.lang.AssertionError: null > at org.junit.Assert.fail(Assert.java:86) > at org.junit.Assert.assertTrue(Assert.java:41) > at org.junit.Assert.assertTrue(Assert.java:52) > at > org.apache.activemq.artemis.tests.integration.ra.ActiveMQMessageHandlerTest.testServerShutdownAndReconnect(ActiveMQMessageHandlerTest.java:245) > {code} > Maximum size of clients global thread pool is computed as {{8 * > Runtime.getRuntime().availableProcessors()}}. In the case of CPU with one > core, the thread pool has maximum size 8. > The test fails because of following situation. When the server is stopped, > all ClientSessionFactoryImpl try to reconnect and they add executor to the > thread pool which tries to reconnect ad anfinitum, see \[1\]. These threads > employ the whole thread pool and the rest is not able to work in normal way. > For example, I can see warnings \[2\]. > This situation is known as Starvation or Livelock \[3\]. I think that the > system should work properly even if the maximum size of thread pool will be 2. > *Steps to reproduce:* > Modify the test in the following way: > {code} > @Test >public void testServerShutdownAndReconnect() throws Exception { > ActiveMQClient.clearThreadPools(); > ActiveMQClient.setGlobalThreadPoolProperties(2, 1); > ActiveMQResourceAdapter qResourceAdapter = newResourceAdapter(); > ... > {code} > \[1\] > {code} > Thread[Signal Dispatcher,5,main] > com.ibm.misc.SignalDispatcher.waitForSignal(Native Method) > com.ibm.misc.SignalDispatcher.run(SignalDispatcher.java:73) > Thread[Thread-4 > (ActiveMQ-client-global-threads--90143455),5,ActiveMQ-client-global-threads--90143455] > sun.misc.Unsafe.park(Native Method) > java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:226) > > java.util.concurrent.locks.AbstractQueuedSynchronizer.doAcquireSharedNanos(AbstractQueuedSynchronizer.java:1048) > > java.util.concurrent.locks.AbstractQueuedSynchronizer.tryAcquireSharedNanos(AbstractQueuedSynchronizer.java:1339) > java.util.concurrent.CountDownLatch.await(CountDownLatch.java:288) > > org.apache.activemq.artemis.core.protocol.core.impl.ActiveMQClientProtocolManager.waitOnLatch(ActiveMQClientProtocolManager.java:134) > > org.apache.activemq.artemis.core.client.impl.ClientSessionFactoryImpl.getConnectionWithRetry(ClientSessionFactoryImpl.java:829) > > org.apache.activemq.artemis.core.client.impl.ClientSessionFactoryImpl.reconnectSessions(ClientSessionFactoryImpl.java:753) > > org.apache.activemq.artemis.core.client.impl.ClientSessionFactoryImpl.failoverOrReconnect(ClientSessionFactoryImpl.java:615) > > org.apache.activemq.artemis.core.client.impl.ClientSessionFactoryImpl.handleConnectionFailure(ClientSessionFactoryImpl.java:513) > > org.apache.activemq.artemis.core.client.impl.ClientSessionFactoryImpl.access$400(ClientSessionFactoryImpl.java:70) > > org.apache.activemq.artemis.core.client.impl.ClientSessionFactoryImpl$DelegatingFailureListener.connectionFailed(ClientSessionFactoryImpl.java:1191) > > org.apache.activemq.artemis.spi.core.protocol.AbstractRemotingConnection.callFailureListeners(AbstractRemotingConnection.java:64) > > org.apache.activemq.artemis.core.protocol.core.impl.RemotingConnectionImpl.fail(RemotingConnectionImpl.java:210) > > org.apache.activemq.artemis.spi.core.protocol.AbstractRemotingConnection.fail(AbstractRemotingConnection.java:206) > > org.apache.activemq.artemis.core.client.impl.ClientSessionFactoryImpl$CloseRunnable.run(ClientSessionFactoryImpl.java:1014) > > org.apache.activemq.artemis.utils.OrderedExecutorFactory$OrderedExecutor$ExecutorTask.run(OrderedExecutorFactory.java:100) > > java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1153) > > java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:628) > java.lang.Thread.run(Thread.java:785) > Thread[Thread-5 > (ActiveMQ-client-global-threads--90143455),5,ActiveMQ-client-global-threads--90143455] > sun.misc.Unsafe.park(Native Method) > java.util.concurrent.locks.LockSupport.pa
[jira] [Commented] (ARTEMIS-1527) [Artemis Testsuite] ActiveMQMessageHandlerTest#testServerShutdownAndReconnect fails
[ https://issues.apache.org/jira/browse/ARTEMIS-1527?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16318488#comment-16318488 ] ASF GitHub Bot commented on ARTEMIS-1527: - GitHub user JiriOndrusek opened a pull request: https://github.com/apache/activemq-artemis/pull/1761 [ARTEMIS-1527] - [Artemis Testsuite] ActiveMQMessageHandlerTest#testS… …erverShutdownAndReconnect fails Issue: https://issues.apache.org/jira/browse/ARTEMIS-1527 You can merge this pull request into a Git repository by running: $ git pull https://github.com/JiriOndrusek/activemq-artemis ARTEMIS-1527-fix Alternatively you can review and apply these changes as the patch at: https://github.com/apache/activemq-artemis/pull/1761.patch To close this pull request, make a commit to your master/trunk branch with (at least) the following in the commit message: This closes #1761 commit 367e633846b020bb002e0cc0ff228df6bddefaa3 Author: JiriOndrusek Date: 2018-01-09T08:24:03Z [ARTEMIS-1527] - [Artemis Testsuite] ActiveMQMessageHandlerTest#testServerShutdownAndReconnect fails > [Artemis Testsuite] ActiveMQMessageHandlerTest#testServerShutdownAndReconnect > fails > --- > > Key: ARTEMIS-1527 > URL: https://issues.apache.org/jira/browse/ARTEMIS-1527 > Project: ActiveMQ Artemis > Issue Type: Bug >Affects Versions: 1.5.5 >Reporter: Jiri Ondrusek > > {code} > java.lang.AssertionError: null > at org.junit.Assert.fail(Assert.java:86) > at org.junit.Assert.assertTrue(Assert.java:41) > at org.junit.Assert.assertTrue(Assert.java:52) > at > org.apache.activemq.artemis.tests.integration.ra.ActiveMQMessageHandlerTest.testServerShutdownAndReconnect(ActiveMQMessageHandlerTest.java:245) > {code} > Maximum size of clients global thread pool is computed as {{8 * > Runtime.getRuntime().availableProcessors()}}. In the case of CPU with one > core, the thread pool has maximum size 8. > The test fails because of following situation. When the server is stopped, > all ClientSessionFactoryImpl try to reconnect and they add executor to the > thread pool which tries to reconnect ad anfinitum, see \[1\]. These threads > employ the whole thread pool and the rest is not able to work in normal way. > For example, I can see warnings \[2\]. > This situation is known as Starvation or Livelock \[3\]. I think that the > system should work properly even if the maximum size of thread pool will be 2. > *Steps to reproduce:* > Modify the test in the following way: > {code} > @Test >public void testServerShutdownAndReconnect() throws Exception { > ActiveMQClient.clearThreadPools(); > ActiveMQClient.setGlobalThreadPoolProperties(2, 1); > ActiveMQResourceAdapter qResourceAdapter = newResourceAdapter(); > ... > {code} > \[1\] > {code} > Thread[Signal Dispatcher,5,main] > com.ibm.misc.SignalDispatcher.waitForSignal(Native Method) > com.ibm.misc.SignalDispatcher.run(SignalDispatcher.java:73) > Thread[Thread-4 > (ActiveMQ-client-global-threads--90143455),5,ActiveMQ-client-global-threads--90143455] > sun.misc.Unsafe.park(Native Method) > java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:226) > > java.util.concurrent.locks.AbstractQueuedSynchronizer.doAcquireSharedNanos(AbstractQueuedSynchronizer.java:1048) > > java.util.concurrent.locks.AbstractQueuedSynchronizer.tryAcquireSharedNanos(AbstractQueuedSynchronizer.java:1339) > java.util.concurrent.CountDownLatch.await(CountDownLatch.java:288) > > org.apache.activemq.artemis.core.protocol.core.impl.ActiveMQClientProtocolManager.waitOnLatch(ActiveMQClientProtocolManager.java:134) > > org.apache.activemq.artemis.core.client.impl.ClientSessionFactoryImpl.getConnectionWithRetry(ClientSessionFactoryImpl.java:829) > > org.apache.activemq.artemis.core.client.impl.ClientSessionFactoryImpl.reconnectSessions(ClientSessionFactoryImpl.java:753) > > org.apache.activemq.artemis.core.client.impl.ClientSessionFactoryImpl.failoverOrReconnect(ClientSessionFactoryImpl.java:615) > > org.apache.activemq.artemis.core.client.impl.ClientSessionFactoryImpl.handleConnectionFailure(ClientSessionFactoryImpl.java:513) > > org.apache.activemq.artemis.core.client.impl.ClientSessionFactoryImpl.access$400(ClientSessionFactoryImpl.java:70) > > org.apache.activemq.artemis.core.client.impl.ClientSessionFactoryImpl$DelegatingFailureListener.connectionFailed(ClientSessionFactoryImpl.java:1191) > > org.apache.activemq.artemis.spi.core.protocol.AbstractRemotingConnection.callFailureListeners(AbstractRemotingConnection.java:64) > > org.apache.activemq.artemis.core.protocol.core.impl.RemotingConnectionImpl.fail(RemotingConnectionImpl.java:210) > > org.apache.activemq.artemi
[jira] [Commented] (ARTEMIS-1527) [Artemis Testsuite] ActiveMQMessageHandlerTest#testServerShutdownAndReconnect fails
[ https://issues.apache.org/jira/browse/ARTEMIS-1527?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16318491#comment-16318491 ] ASF GitHub Bot commented on ARTEMIS-1527: - Github user JiriOndrusek closed the pull request at: https://github.com/apache/activemq-artemis/pull/1676 > [Artemis Testsuite] ActiveMQMessageHandlerTest#testServerShutdownAndReconnect > fails > --- > > Key: ARTEMIS-1527 > URL: https://issues.apache.org/jira/browse/ARTEMIS-1527 > Project: ActiveMQ Artemis > Issue Type: Bug >Affects Versions: 1.5.5 >Reporter: Jiri Ondrusek > > {code} > java.lang.AssertionError: null > at org.junit.Assert.fail(Assert.java:86) > at org.junit.Assert.assertTrue(Assert.java:41) > at org.junit.Assert.assertTrue(Assert.java:52) > at > org.apache.activemq.artemis.tests.integration.ra.ActiveMQMessageHandlerTest.testServerShutdownAndReconnect(ActiveMQMessageHandlerTest.java:245) > {code} > Maximum size of clients global thread pool is computed as {{8 * > Runtime.getRuntime().availableProcessors()}}. In the case of CPU with one > core, the thread pool has maximum size 8. > The test fails because of following situation. When the server is stopped, > all ClientSessionFactoryImpl try to reconnect and they add executor to the > thread pool which tries to reconnect ad anfinitum, see \[1\]. These threads > employ the whole thread pool and the rest is not able to work in normal way. > For example, I can see warnings \[2\]. > This situation is known as Starvation or Livelock \[3\]. I think that the > system should work properly even if the maximum size of thread pool will be 2. > *Steps to reproduce:* > Modify the test in the following way: > {code} > @Test >public void testServerShutdownAndReconnect() throws Exception { > ActiveMQClient.clearThreadPools(); > ActiveMQClient.setGlobalThreadPoolProperties(2, 1); > ActiveMQResourceAdapter qResourceAdapter = newResourceAdapter(); > ... > {code} > \[1\] > {code} > Thread[Signal Dispatcher,5,main] > com.ibm.misc.SignalDispatcher.waitForSignal(Native Method) > com.ibm.misc.SignalDispatcher.run(SignalDispatcher.java:73) > Thread[Thread-4 > (ActiveMQ-client-global-threads--90143455),5,ActiveMQ-client-global-threads--90143455] > sun.misc.Unsafe.park(Native Method) > java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:226) > > java.util.concurrent.locks.AbstractQueuedSynchronizer.doAcquireSharedNanos(AbstractQueuedSynchronizer.java:1048) > > java.util.concurrent.locks.AbstractQueuedSynchronizer.tryAcquireSharedNanos(AbstractQueuedSynchronizer.java:1339) > java.util.concurrent.CountDownLatch.await(CountDownLatch.java:288) > > org.apache.activemq.artemis.core.protocol.core.impl.ActiveMQClientProtocolManager.waitOnLatch(ActiveMQClientProtocolManager.java:134) > > org.apache.activemq.artemis.core.client.impl.ClientSessionFactoryImpl.getConnectionWithRetry(ClientSessionFactoryImpl.java:829) > > org.apache.activemq.artemis.core.client.impl.ClientSessionFactoryImpl.reconnectSessions(ClientSessionFactoryImpl.java:753) > > org.apache.activemq.artemis.core.client.impl.ClientSessionFactoryImpl.failoverOrReconnect(ClientSessionFactoryImpl.java:615) > > org.apache.activemq.artemis.core.client.impl.ClientSessionFactoryImpl.handleConnectionFailure(ClientSessionFactoryImpl.java:513) > > org.apache.activemq.artemis.core.client.impl.ClientSessionFactoryImpl.access$400(ClientSessionFactoryImpl.java:70) > > org.apache.activemq.artemis.core.client.impl.ClientSessionFactoryImpl$DelegatingFailureListener.connectionFailed(ClientSessionFactoryImpl.java:1191) > > org.apache.activemq.artemis.spi.core.protocol.AbstractRemotingConnection.callFailureListeners(AbstractRemotingConnection.java:64) > > org.apache.activemq.artemis.core.protocol.core.impl.RemotingConnectionImpl.fail(RemotingConnectionImpl.java:210) > > org.apache.activemq.artemis.spi.core.protocol.AbstractRemotingConnection.fail(AbstractRemotingConnection.java:206) > > org.apache.activemq.artemis.core.client.impl.ClientSessionFactoryImpl$CloseRunnable.run(ClientSessionFactoryImpl.java:1014) > > org.apache.activemq.artemis.utils.OrderedExecutorFactory$OrderedExecutor$ExecutorTask.run(OrderedExecutorFactory.java:100) > > java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1153) > > java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:628) > java.lang.Thread.run(Thread.java:785) > Thread[Thread-5 > (ActiveMQ-client-global-threads--90143455),5,ActiveMQ-client-global-threads--90143455] > sun.misc.Unsafe.park(Native Method) > java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:226) > > java.util.concurrent.locks.Abstra
[jira] [Commented] (ARTEMIS-1527) [Artemis Testsuite] ActiveMQMessageHandlerTest#testServerShutdownAndReconnect fails
[ https://issues.apache.org/jira/browse/ARTEMIS-1527?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16270414#comment-16270414 ] ASF GitHub Bot commented on ARTEMIS-1527: - Github user JiriOndrusek commented on the issue: https://github.com/apache/activemq-artemis/pull/1676 Thank you for your responses. @clebertsuconic you are right, with this change FailoverTest is failing I'll try to find solution for FailoverTest to succeed. (it will take me some time to debug and understand it) @clebertsuconic @mtaylor I'm opened to any suggestions, as there are a lot of aspects, about which I don't know/understand. > [Artemis Testsuite] ActiveMQMessageHandlerTest#testServerShutdownAndReconnect > fails > --- > > Key: ARTEMIS-1527 > URL: https://issues.apache.org/jira/browse/ARTEMIS-1527 > Project: ActiveMQ Artemis > Issue Type: Bug >Affects Versions: 1.5.5 >Reporter: Jiri Ondrusek > > {code} > java.lang.AssertionError: null > at org.junit.Assert.fail(Assert.java:86) > at org.junit.Assert.assertTrue(Assert.java:41) > at org.junit.Assert.assertTrue(Assert.java:52) > at > org.apache.activemq.artemis.tests.integration.ra.ActiveMQMessageHandlerTest.testServerShutdownAndReconnect(ActiveMQMessageHandlerTest.java:245) > {code} > Maximum size of clients global thread pool is computed as {{8 * > Runtime.getRuntime().availableProcessors()}}. In the case of CPU with one > core, the thread pool has maximum size 8. > The test fails because of following situation. When the server is stopped, > all ClientSessionFactoryImpl try to reconnect and they add executor to the > thread pool which tries to reconnect ad anfinitum, see \[1\]. These threads > employ the whole thread pool and the rest is not able to work in normal way. > For example, I can see warnings \[2\]. > This situation is known as Starvation or Livelock \[3\]. I think that the > system should work properly even if the maximum size of thread pool will be 2. > *Steps to reproduce:* > Modify the test in the following way: > {code} > @Test >public void testServerShutdownAndReconnect() throws Exception { > ActiveMQClient.clearThreadPools(); > ActiveMQClient.setGlobalThreadPoolProperties(2, 1); > ActiveMQResourceAdapter qResourceAdapter = newResourceAdapter(); > ... > {code} > \[1\] > {code} > Thread[Signal Dispatcher,5,main] > com.ibm.misc.SignalDispatcher.waitForSignal(Native Method) > com.ibm.misc.SignalDispatcher.run(SignalDispatcher.java:73) > Thread[Thread-4 > (ActiveMQ-client-global-threads--90143455),5,ActiveMQ-client-global-threads--90143455] > sun.misc.Unsafe.park(Native Method) > java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:226) > > java.util.concurrent.locks.AbstractQueuedSynchronizer.doAcquireSharedNanos(AbstractQueuedSynchronizer.java:1048) > > java.util.concurrent.locks.AbstractQueuedSynchronizer.tryAcquireSharedNanos(AbstractQueuedSynchronizer.java:1339) > java.util.concurrent.CountDownLatch.await(CountDownLatch.java:288) > > org.apache.activemq.artemis.core.protocol.core.impl.ActiveMQClientProtocolManager.waitOnLatch(ActiveMQClientProtocolManager.java:134) > > org.apache.activemq.artemis.core.client.impl.ClientSessionFactoryImpl.getConnectionWithRetry(ClientSessionFactoryImpl.java:829) > > org.apache.activemq.artemis.core.client.impl.ClientSessionFactoryImpl.reconnectSessions(ClientSessionFactoryImpl.java:753) > > org.apache.activemq.artemis.core.client.impl.ClientSessionFactoryImpl.failoverOrReconnect(ClientSessionFactoryImpl.java:615) > > org.apache.activemq.artemis.core.client.impl.ClientSessionFactoryImpl.handleConnectionFailure(ClientSessionFactoryImpl.java:513) > > org.apache.activemq.artemis.core.client.impl.ClientSessionFactoryImpl.access$400(ClientSessionFactoryImpl.java:70) > > org.apache.activemq.artemis.core.client.impl.ClientSessionFactoryImpl$DelegatingFailureListener.connectionFailed(ClientSessionFactoryImpl.java:1191) > > org.apache.activemq.artemis.spi.core.protocol.AbstractRemotingConnection.callFailureListeners(AbstractRemotingConnection.java:64) > > org.apache.activemq.artemis.core.protocol.core.impl.RemotingConnectionImpl.fail(RemotingConnectionImpl.java:210) > > org.apache.activemq.artemis.spi.core.protocol.AbstractRemotingConnection.fail(AbstractRemotingConnection.java:206) > > org.apache.activemq.artemis.core.client.impl.ClientSessionFactoryImpl$CloseRunnable.run(ClientSessionFactoryImpl.java:1014) > > org.apache.activemq.artemis.utils.OrderedExecutorFactory$OrderedExecutor$ExecutorTask.run(OrderedExecutorFactory.java:100) > > java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1153) > > java.util.con
[jira] [Commented] (ARTEMIS-1527) [Artemis Testsuite] ActiveMQMessageHandlerTest#testServerShutdownAndReconnect fails
[ https://issues.apache.org/jira/browse/ARTEMIS-1527?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16270370#comment-16270370 ] ASF GitHub Bot commented on ARTEMIS-1527: - Github user mtaylor commented on the issue: https://github.com/apache/activemq-artemis/pull/1676 @clebertsuconic The problem here is that any thread that blocks indefinitely takes up resources in the Thread pool. If you have several threads trying reconnect, the ThreadPool may be exhausted and can not be used to perform any other tasks. My opinion here is that no task we add to the ThreadPool should block indefinitely, for retry here we should attempt to connect, if it fails schedule another attempt connect call, and give up the thread. Jiri's made a start on that, but we may need to flesh out some of the details. There's a similar problem with LargeMessage receive(). > [Artemis Testsuite] ActiveMQMessageHandlerTest#testServerShutdownAndReconnect > fails > --- > > Key: ARTEMIS-1527 > URL: https://issues.apache.org/jira/browse/ARTEMIS-1527 > Project: ActiveMQ Artemis > Issue Type: Bug >Affects Versions: 1.5.5 >Reporter: Jiri Ondrusek > > {code} > java.lang.AssertionError: null > at org.junit.Assert.fail(Assert.java:86) > at org.junit.Assert.assertTrue(Assert.java:41) > at org.junit.Assert.assertTrue(Assert.java:52) > at > org.apache.activemq.artemis.tests.integration.ra.ActiveMQMessageHandlerTest.testServerShutdownAndReconnect(ActiveMQMessageHandlerTest.java:245) > {code} > Maximum size of clients global thread pool is computed as {{8 * > Runtime.getRuntime().availableProcessors()}}. In the case of CPU with one > core, the thread pool has maximum size 8. > The test fails because of following situation. When the server is stopped, > all ClientSessionFactoryImpl try to reconnect and they add executor to the > thread pool which tries to reconnect ad anfinitum, see \[1\]. These threads > employ the whole thread pool and the rest is not able to work in normal way. > For example, I can see warnings \[2\]. > This situation is known as Starvation or Livelock \[3\]. I think that the > system should work properly even if the maximum size of thread pool will be 2. > *Steps to reproduce:* > Modify the test in the following way: > {code} > @Test >public void testServerShutdownAndReconnect() throws Exception { > ActiveMQClient.clearThreadPools(); > ActiveMQClient.setGlobalThreadPoolProperties(2, 1); > ActiveMQResourceAdapter qResourceAdapter = newResourceAdapter(); > ... > {code} > \[1\] > {code} > Thread[Signal Dispatcher,5,main] > com.ibm.misc.SignalDispatcher.waitForSignal(Native Method) > com.ibm.misc.SignalDispatcher.run(SignalDispatcher.java:73) > Thread[Thread-4 > (ActiveMQ-client-global-threads--90143455),5,ActiveMQ-client-global-threads--90143455] > sun.misc.Unsafe.park(Native Method) > java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:226) > > java.util.concurrent.locks.AbstractQueuedSynchronizer.doAcquireSharedNanos(AbstractQueuedSynchronizer.java:1048) > > java.util.concurrent.locks.AbstractQueuedSynchronizer.tryAcquireSharedNanos(AbstractQueuedSynchronizer.java:1339) > java.util.concurrent.CountDownLatch.await(CountDownLatch.java:288) > > org.apache.activemq.artemis.core.protocol.core.impl.ActiveMQClientProtocolManager.waitOnLatch(ActiveMQClientProtocolManager.java:134) > > org.apache.activemq.artemis.core.client.impl.ClientSessionFactoryImpl.getConnectionWithRetry(ClientSessionFactoryImpl.java:829) > > org.apache.activemq.artemis.core.client.impl.ClientSessionFactoryImpl.reconnectSessions(ClientSessionFactoryImpl.java:753) > > org.apache.activemq.artemis.core.client.impl.ClientSessionFactoryImpl.failoverOrReconnect(ClientSessionFactoryImpl.java:615) > > org.apache.activemq.artemis.core.client.impl.ClientSessionFactoryImpl.handleConnectionFailure(ClientSessionFactoryImpl.java:513) > > org.apache.activemq.artemis.core.client.impl.ClientSessionFactoryImpl.access$400(ClientSessionFactoryImpl.java:70) > > org.apache.activemq.artemis.core.client.impl.ClientSessionFactoryImpl$DelegatingFailureListener.connectionFailed(ClientSessionFactoryImpl.java:1191) > > org.apache.activemq.artemis.spi.core.protocol.AbstractRemotingConnection.callFailureListeners(AbstractRemotingConnection.java:64) > > org.apache.activemq.artemis.core.protocol.core.impl.RemotingConnectionImpl.fail(RemotingConnectionImpl.java:210) > > org.apache.activemq.artemis.spi.core.protocol.AbstractRemotingConnection.fail(AbstractRemotingConnection.java:206) > > org.apache.activemq.artemis.core.client.impl.ClientSessionFactoryImpl$CloseRunnable.run(ClientSessionFactoryImpl.java:1014) > > org.apache.ac
[jira] [Commented] (ARTEMIS-1527) [Artemis Testsuite] ActiveMQMessageHandlerTest#testServerShutdownAndReconnect fails
[ https://issues.apache.org/jira/browse/ARTEMIS-1527?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16270046#comment-16270046 ] ASF GitHub Bot commented on ARTEMIS-1527: - Github user clebertsuconic commented on the issue: https://github.com/apache/activemq-artemis/pull/1676 The failover is blocking because client applications would fail and would require to block.. it's customer's code. I reckon we could do asynchronously for MDBs or message listeners. so, how do you differentiate the two here? I suspect Failover (FailoverTest and others on the testsuite) wouldn't work. Also.. at least you should make this change on master.. it's a significant refactoring to make something like this to work properly. Failover itself was a pain.. making this level.. even more difficult. > [Artemis Testsuite] ActiveMQMessageHandlerTest#testServerShutdownAndReconnect > fails > --- > > Key: ARTEMIS-1527 > URL: https://issues.apache.org/jira/browse/ARTEMIS-1527 > Project: ActiveMQ Artemis > Issue Type: Bug >Affects Versions: 1.5.5 >Reporter: Jiri Ondrusek > > {code} > java.lang.AssertionError: null > at org.junit.Assert.fail(Assert.java:86) > at org.junit.Assert.assertTrue(Assert.java:41) > at org.junit.Assert.assertTrue(Assert.java:52) > at > org.apache.activemq.artemis.tests.integration.ra.ActiveMQMessageHandlerTest.testServerShutdownAndReconnect(ActiveMQMessageHandlerTest.java:245) > {code} > Maximum size of clients global thread pool is computed as {{8 * > Runtime.getRuntime().availableProcessors()}}. In the case of CPU with one > core, the thread pool has maximum size 8. > The test fails because of following situation. When the server is stopped, > all ClientSessionFactoryImpl try to reconnect and they add executor to the > thread pool which tries to reconnect ad anfinitum, see \[1\]. These threads > employ the whole thread pool and the rest is not able to work in normal way. > For example, I can see warnings \[2\]. > This situation is known as Starvation or Livelock \[3\]. I think that the > system should work properly even if the maximum size of thread pool will be 2. > *Steps to reproduce:* > Modify the test in the following way: > {code} > @Test >public void testServerShutdownAndReconnect() throws Exception { > ActiveMQClient.clearThreadPools(); > ActiveMQClient.setGlobalThreadPoolProperties(2, 1); > ActiveMQResourceAdapter qResourceAdapter = newResourceAdapter(); > ... > {code} > \[1\] > {code} > Thread[Signal Dispatcher,5,main] > com.ibm.misc.SignalDispatcher.waitForSignal(Native Method) > com.ibm.misc.SignalDispatcher.run(SignalDispatcher.java:73) > Thread[Thread-4 > (ActiveMQ-client-global-threads--90143455),5,ActiveMQ-client-global-threads--90143455] > sun.misc.Unsafe.park(Native Method) > java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:226) > > java.util.concurrent.locks.AbstractQueuedSynchronizer.doAcquireSharedNanos(AbstractQueuedSynchronizer.java:1048) > > java.util.concurrent.locks.AbstractQueuedSynchronizer.tryAcquireSharedNanos(AbstractQueuedSynchronizer.java:1339) > java.util.concurrent.CountDownLatch.await(CountDownLatch.java:288) > > org.apache.activemq.artemis.core.protocol.core.impl.ActiveMQClientProtocolManager.waitOnLatch(ActiveMQClientProtocolManager.java:134) > > org.apache.activemq.artemis.core.client.impl.ClientSessionFactoryImpl.getConnectionWithRetry(ClientSessionFactoryImpl.java:829) > > org.apache.activemq.artemis.core.client.impl.ClientSessionFactoryImpl.reconnectSessions(ClientSessionFactoryImpl.java:753) > > org.apache.activemq.artemis.core.client.impl.ClientSessionFactoryImpl.failoverOrReconnect(ClientSessionFactoryImpl.java:615) > > org.apache.activemq.artemis.core.client.impl.ClientSessionFactoryImpl.handleConnectionFailure(ClientSessionFactoryImpl.java:513) > > org.apache.activemq.artemis.core.client.impl.ClientSessionFactoryImpl.access$400(ClientSessionFactoryImpl.java:70) > > org.apache.activemq.artemis.core.client.impl.ClientSessionFactoryImpl$DelegatingFailureListener.connectionFailed(ClientSessionFactoryImpl.java:1191) > > org.apache.activemq.artemis.spi.core.protocol.AbstractRemotingConnection.callFailureListeners(AbstractRemotingConnection.java:64) > > org.apache.activemq.artemis.core.protocol.core.impl.RemotingConnectionImpl.fail(RemotingConnectionImpl.java:210) > > org.apache.activemq.artemis.spi.core.protocol.AbstractRemotingConnection.fail(AbstractRemotingConnection.java:206) > > org.apache.activemq.artemis.core.client.impl.ClientSessionFactoryImpl$CloseRunnable.run(ClientSessionFactoryImpl.java:1014) > > org.apache.activemq.artemis.utils.OrderedExecutorFa
[jira] [Commented] (ARTEMIS-1527) [Artemis Testsuite] ActiveMQMessageHandlerTest#testServerShutdownAndReconnect fails
[ https://issues.apache.org/jira/browse/ARTEMIS-1527?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16268624#comment-16268624 ] ASF GitHub Bot commented on ARTEMIS-1527: - GitHub user JiriOndrusek opened a pull request: https://github.com/apache/activemq-artemis/pull/1676 [ARTEMIS-1527] - [Artemis Testsuite] ActiveMQMessageHandlerTest#testS… …erverShutdownAndReconnect fails Issue: https://issues.apache.org/jira/browse/ARTEMIS-1527 You can merge this pull request into a Git repository by running: $ git pull https://github.com/JiriOndrusek/activemq-artemis ARTEMIS-1527 Alternatively you can review and apply these changes as the patch at: https://github.com/apache/activemq-artemis/pull/1676.patch To close this pull request, make a commit to your master/trunk branch with (at least) the following in the commit message: This closes #1676 commit 354409952ed7cd3c9d7883e21389e75ed9a022b5 Author: JiriOndrusek Date: 2017-11-22T14:38:03Z [ARTEMIS-1527] - [Artemis Testsuite] ActiveMQMessageHandlerTest#testServerShutdownAndReconnect fails > [Artemis Testsuite] ActiveMQMessageHandlerTest#testServerShutdownAndReconnect > fails > --- > > Key: ARTEMIS-1527 > URL: https://issues.apache.org/jira/browse/ARTEMIS-1527 > Project: ActiveMQ Artemis > Issue Type: Bug >Affects Versions: 1.5.5 >Reporter: Jiri Ondrusek > > {code} > java.lang.AssertionError: null > at org.junit.Assert.fail(Assert.java:86) > at org.junit.Assert.assertTrue(Assert.java:41) > at org.junit.Assert.assertTrue(Assert.java:52) > at > org.apache.activemq.artemis.tests.integration.ra.ActiveMQMessageHandlerTest.testServerShutdownAndReconnect(ActiveMQMessageHandlerTest.java:245) > {code} > Maximum size of clients global thread pool is computed as {{8 * > Runtime.getRuntime().availableProcessors()}}. In the case of CPU with one > core, the thread pool has maximum size 8. > The test fails because of following situation. When the server is stopped, > all ClientSessionFactoryImpl try to reconnect and they add executor to the > thread pool which tries to reconnect ad anfinitum, see \[1\]. These threads > employ the whole thread pool and the rest is not able to work in normal way. > For example, I can see warnings \[2\]. > This situation is known as Starvation or Livelock \[3\]. I think that the > system should work properly even if the maximum size of thread pool will be 2. > *Steps to reproduce:* > Modify the test in the following way: > {code} > @Test >public void testServerShutdownAndReconnect() throws Exception { > ActiveMQClient.clearThreadPools(); > ActiveMQClient.setGlobalThreadPoolProperties(2, 1); > ActiveMQResourceAdapter qResourceAdapter = newResourceAdapter(); > ... > {code} > \[1\] > {code} > Thread[Signal Dispatcher,5,main] > com.ibm.misc.SignalDispatcher.waitForSignal(Native Method) > com.ibm.misc.SignalDispatcher.run(SignalDispatcher.java:73) > Thread[Thread-4 > (ActiveMQ-client-global-threads--90143455),5,ActiveMQ-client-global-threads--90143455] > sun.misc.Unsafe.park(Native Method) > java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:226) > > java.util.concurrent.locks.AbstractQueuedSynchronizer.doAcquireSharedNanos(AbstractQueuedSynchronizer.java:1048) > > java.util.concurrent.locks.AbstractQueuedSynchronizer.tryAcquireSharedNanos(AbstractQueuedSynchronizer.java:1339) > java.util.concurrent.CountDownLatch.await(CountDownLatch.java:288) > > org.apache.activemq.artemis.core.protocol.core.impl.ActiveMQClientProtocolManager.waitOnLatch(ActiveMQClientProtocolManager.java:134) > > org.apache.activemq.artemis.core.client.impl.ClientSessionFactoryImpl.getConnectionWithRetry(ClientSessionFactoryImpl.java:829) > > org.apache.activemq.artemis.core.client.impl.ClientSessionFactoryImpl.reconnectSessions(ClientSessionFactoryImpl.java:753) > > org.apache.activemq.artemis.core.client.impl.ClientSessionFactoryImpl.failoverOrReconnect(ClientSessionFactoryImpl.java:615) > > org.apache.activemq.artemis.core.client.impl.ClientSessionFactoryImpl.handleConnectionFailure(ClientSessionFactoryImpl.java:513) > > org.apache.activemq.artemis.core.client.impl.ClientSessionFactoryImpl.access$400(ClientSessionFactoryImpl.java:70) > > org.apache.activemq.artemis.core.client.impl.ClientSessionFactoryImpl$DelegatingFailureListener.connectionFailed(ClientSessionFactoryImpl.java:1191) > > org.apache.activemq.artemis.spi.core.protocol.AbstractRemotingConnection.callFailureListeners(AbstractRemotingConnection.java:64) > > org.apache.activemq.artemis.core.protocol.core.impl.RemotingConnectionImpl.fail(RemotingConnectionImpl.java:210) > > org.apache.activemq.artemis.sp