[jira] [Work logged] (AMQ-9448) RedeliveryPolicy causing lock in dispatcher
[ https://issues.apache.org/jira/browse/AMQ-9448?focusedWorklogId=921617=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-921617 ] ASF GitHub Bot logged work on AMQ-9448: --- Author: ASF GitHub Bot Created on: 01/Jun/24 06:55 Start Date: 01/Jun/24 06:55 Worklog Time Spent: 10m Work Description: thezbyg commented on PR #1177: URL: https://github.com/apache/activemq/pull/1177#issuecomment-2143330637 Yes. This is the stack trace of "JobScheduler:JMS" thread blocked by itself due to acquiring read lock on store while iterating scheduled jobs in mainLoop() method and then attempting to acquire write lock to write a new scheduled job information: ` "JobScheduler:JMS" #28 daemon prio=5 os_prio=0 cpu=11.62ms elapsed=117.73s tid=0x7f43d1344b50 nid=0x185b waiting on condition [0x7f4356cfe000] java.lang.Thread.State: WAITING (parking) at jdk.internal.misc.Unsafe.park(java.base@17.0.8/Native Method) - parking to wait for <0x8bd25150> (a java.util.concurrent.locks.ReentrantReadWriteLock$NonfairSync) at java.util.concurrent.locks.LockSupport.park(java.base@17.0.8/LockSupport.java:211) at java.util.concurrent.locks.AbstractQueuedSynchronizer.acquire(java.base@17.0.8/AbstractQueuedSynchronizer.java:715) at java.util.concurrent.locks.AbstractQueuedSynchronizer.acquire(java.base@17.0.8/AbstractQueuedSynchronizer.java:938) at java.util.concurrent.locks.ReentrantReadWriteLock$WriteLock.lock(java.base@17.0.8/ReentrantReadWriteLock.java:959) at org.apache.activemq.store.kahadb.scheduler.JobSchedulerStoreImpl$8.visit(JobSchedulerStoreImpl.java:684) at org.apache.activemq.store.kahadb.data.KahaAddScheduledJobCommand.visit(KahaAddScheduledJobCommand.java:283) at org.apache.activemq.store.kahadb.scheduler.JobSchedulerStoreImpl.process(JobSchedulerStoreImpl.java:679) at org.apache.activemq.store.kahadb.AbstractKahaDBStore.store(AbstractKahaDBStore.java:495) at org.apache.activemq.store.kahadb.AbstractKahaDBStore.store(AbstractKahaDBStore.java:403) at org.apache.activemq.store.kahadb.scheduler.JobSchedulerImpl.doSchedule(JobSchedulerImpl.java:252) at org.apache.activemq.store.kahadb.scheduler.JobSchedulerImpl.schedule(JobSchedulerImpl.java:100) at org.apache.activemq.store.kahadb.scheduler.JobSchedulerImpl.mainLoop(JobSchedulerImpl.java:782) at org.apache.activemq.store.kahadb.scheduler.JobSchedulerImpl.run(JobSchedulerImpl.java:699) at java.lang.Thread.run(java.base@17.0.8/Thread.java:833) ` Full dump: [dump.txt](https://github.com/user-attachments/files/15520722/dump.txt) Issue Time Tracking --- Worklog Id: (was: 921617) Time Spent: 50m (was: 40m) > RedeliveryPolicy causing lock in dispatcher > --- > > Key: AMQ-9448 > URL: https://issues.apache.org/jira/browse/AMQ-9448 > Project: ActiveMQ Classic > Issue Type: Bug > Components: Broker >Affects Versions: 5.18.3 > Environment: ubuntu-22.04 > corretto-17 >Reporter: Oliver Holzmann >Priority: Critical > Attachments: doNotFireJobsWhileStoreLockIsHeld.patch, jstack1.out > > Time Spent: 50m > Remaining Estimate: 0h > > We are using ActiveMQ with redeliveryPolicy enabled: > {code:java} > > redeliveryDelay="1000" /> > {code} > When a message exceeds maximumRedeliveries the queue is "frozen". This means > the consumers can't read any messages and qrowing the queue in web console > causes error 504. > A thread dump shows a blocked JobScheduler:JMS thread: > {code:java} > priority : 5 > thread Id : x7fb985664960 > native Id : xae > state : BLOCKED > StackTrace: > "JobScheduler:JMS" #75 daemon prio=5 os_prio=0 cpu=478.63ms elapsed=757.29s > tid=0x7fb985664960 nid=0xae waiting for monitor entry [0x7fb80def1000] > java.lang.Thread.State: BLOCKED (on object monitor) > at > org.apache.activemq.broker.region.PrefetchSubscription.dispatchPending(PrefetchSubscription.java:647) > waiting to lock <0x00077ebc62a0> (a java.lang.Object) locked > <0x00077ebc6290> (a java.lang.Object) > at > org.apache.activemq.broker.region.PrefetchSubscription.add(PrefetchSubscription.java:159) > at org.apache.activemq.broker.region.Queue.doActualDispatch(Queue.java:2208) > at org.apache.activemq.broker.region.Queue.doDispatch(Queue.java:2156) > at org.apache.activemq.broker.region.Queue.pageInMessages(Queue.java:2307) > at org.apache.activemq.broker.region.Queue.iterate(Queue.java:1728) locked > <0x00077ebc5178> (a java.lang.Object) > at org.apache.activemq.broker.region.Queue.wakeup(Queue.java:1977) > at org.apache.activemq.broker.region.Queue.messageSent(Queue.java:1971) > at >
[jira] [Work logged] (AMQ-9448) RedeliveryPolicy causing lock in dispatcher
[ https://issues.apache.org/jira/browse/AMQ-9448?focusedWorklogId=921576=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-921576 ] ASF GitHub Bot logged work on AMQ-9448: --- Author: ASF GitHub Bot Created on: 31/May/24 19:37 Start Date: 31/May/24 19:37 Worklog Time Spent: 10m Work Description: mattrpav commented on PR #1177: URL: https://github.com/apache/activemq/pull/1177#issuecomment-2142874447 Do you have a thread dump of the deadlock occurring? Issue Time Tracking --- Worklog Id: (was: 921576) Time Spent: 40m (was: 0.5h) > RedeliveryPolicy causing lock in dispatcher > --- > > Key: AMQ-9448 > URL: https://issues.apache.org/jira/browse/AMQ-9448 > Project: ActiveMQ Classic > Issue Type: Bug > Components: Broker >Affects Versions: 5.18.3 > Environment: ubuntu-22.04 > corretto-17 >Reporter: Oliver Holzmann >Priority: Critical > Attachments: doNotFireJobsWhileStoreLockIsHeld.patch, jstack1.out > > Time Spent: 40m > Remaining Estimate: 0h > > We are using ActiveMQ with redeliveryPolicy enabled: > {code:java} > > redeliveryDelay="1000" /> > {code} > When a message exceeds maximumRedeliveries the queue is "frozen". This means > the consumers can't read any messages and qrowing the queue in web console > causes error 504. > A thread dump shows a blocked JobScheduler:JMS thread: > {code:java} > priority : 5 > thread Id : x7fb985664960 > native Id : xae > state : BLOCKED > StackTrace: > "JobScheduler:JMS" #75 daemon prio=5 os_prio=0 cpu=478.63ms elapsed=757.29s > tid=0x7fb985664960 nid=0xae waiting for monitor entry [0x7fb80def1000] > java.lang.Thread.State: BLOCKED (on object monitor) > at > org.apache.activemq.broker.region.PrefetchSubscription.dispatchPending(PrefetchSubscription.java:647) > waiting to lock <0x00077ebc62a0> (a java.lang.Object) locked > <0x00077ebc6290> (a java.lang.Object) > at > org.apache.activemq.broker.region.PrefetchSubscription.add(PrefetchSubscription.java:159) > at org.apache.activemq.broker.region.Queue.doActualDispatch(Queue.java:2208) > at org.apache.activemq.broker.region.Queue.doDispatch(Queue.java:2156) > at org.apache.activemq.broker.region.Queue.pageInMessages(Queue.java:2307) > at org.apache.activemq.broker.region.Queue.iterate(Queue.java:1728) locked > <0x00077ebc5178> (a java.lang.Object) > at org.apache.activemq.broker.region.Queue.wakeup(Queue.java:1977) > at org.apache.activemq.broker.region.Queue.messageSent(Queue.java:1971) > at > org.apache.activemq.broker.region.Queue.doPendingCursorAdditions(Queue.java:832) > at org.apache.activemq.broker.region.Queue.tryOrderedCursorAdd(Queue.java:927) > at org.apache.activemq.broker.region.Queue.doMessageSend(Queue.java:906) > at org.apache.activemq.broker.region.Queue.send(Queue.java:756) > at > org.apache.activemq.broker.region.DestinationFilter.send(DestinationFilter.java:138) > at > org.apache.activemq.broker.region.AbstractRegion.send(AbstractRegion.java:511) > at org.apache.activemq.broker.region.RegionBroker.send(RegionBroker.java:477) > at > org.apache.activemq.broker.jmx.ManagedRegionBroker.send(ManagedRegionBroker.java:343) > at org.apache.activemq.broker.BrokerFilter.send(BrokerFilter.java:154) > at > org.apache.activemq.broker.scheduler.SchedulerBroker.scheduledJob(SchedulerBroker.java:429) > at > org.apache.activemq.store.kahadb.scheduler.JobSchedulerImpl.fireJob(JobSchedulerImpl.java:823) > at > org.apache.activemq.store.kahadb.scheduler.JobSchedulerImpl.mainLoop(JobSchedulerImpl.java:753) > at > org.apache.activemq.store.kahadb.scheduler.JobSchedulerImpl.run(JobSchedulerImpl.java:699) > at java.lang.Thread.run(java.base@17.0.10/Thread.java:840) > Locked ownable synchronizers: <0x00077ebc3ed0> (a > java.util.concurrent.locks.ReentrantReadWriteLock$NonfairSync) > {code} > -- This message was sent by Atlassian Jira (v8.20.10#820010) - To unsubscribe, e-mail: issues-unsubscr...@activemq.apache.org For additional commands, e-mail: issues-h...@activemq.apache.org For further information, visit: https://activemq.apache.org/contact
[jira] [Work logged] (AMQ-9448) RedeliveryPolicy causing lock in dispatcher
[ https://issues.apache.org/jira/browse/AMQ-9448?focusedWorklogId=910667=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-910667 ] ASF GitHub Bot logged work on AMQ-9448: --- Author: ASF GitHub Bot Created on: 20/Mar/24 06:25 Start Date: 20/Mar/24 06:25 Worklog Time Spent: 10m Work Description: thezbyg commented on PR #1177: URL: https://github.com/apache/activemq/pull/1177#issuecomment-2008727676 I have now added a new unit test. Does existing JmsSchedulerTest#testCron unit test run successfully for you before this change? Issue Time Tracking --- Worklog Id: (was: 910667) Time Spent: 0.5h (was: 20m) > RedeliveryPolicy causing lock in dispatcher > --- > > Key: AMQ-9448 > URL: https://issues.apache.org/jira/browse/AMQ-9448 > Project: ActiveMQ Classic > Issue Type: Bug > Components: Broker >Affects Versions: 5.18.3 > Environment: ubuntu-22.04 > corretto-17 >Reporter: Oliver Holzmann >Priority: Critical > Attachments: doNotFireJobsWhileStoreLockIsHeld.patch, jstack1.out > > Time Spent: 0.5h > Remaining Estimate: 0h > > We are using ActiveMQ with redeliveryPolicy enabled: > {code:java} > > redeliveryDelay="1000" /> > {code} > When a message exceeds maximumRedeliveries the queue is "frozen". This means > the consumers can't read any messages and qrowing the queue in web console > causes error 504. > A thread dump shows a blocked JobScheduler:JMS thread: > {code:java} > priority : 5 > thread Id : x7fb985664960 > native Id : xae > state : BLOCKED > StackTrace: > "JobScheduler:JMS" #75 daemon prio=5 os_prio=0 cpu=478.63ms elapsed=757.29s > tid=0x7fb985664960 nid=0xae waiting for monitor entry [0x7fb80def1000] > java.lang.Thread.State: BLOCKED (on object monitor) > at > org.apache.activemq.broker.region.PrefetchSubscription.dispatchPending(PrefetchSubscription.java:647) > waiting to lock <0x00077ebc62a0> (a java.lang.Object) locked > <0x00077ebc6290> (a java.lang.Object) > at > org.apache.activemq.broker.region.PrefetchSubscription.add(PrefetchSubscription.java:159) > at org.apache.activemq.broker.region.Queue.doActualDispatch(Queue.java:2208) > at org.apache.activemq.broker.region.Queue.doDispatch(Queue.java:2156) > at org.apache.activemq.broker.region.Queue.pageInMessages(Queue.java:2307) > at org.apache.activemq.broker.region.Queue.iterate(Queue.java:1728) locked > <0x00077ebc5178> (a java.lang.Object) > at org.apache.activemq.broker.region.Queue.wakeup(Queue.java:1977) > at org.apache.activemq.broker.region.Queue.messageSent(Queue.java:1971) > at > org.apache.activemq.broker.region.Queue.doPendingCursorAdditions(Queue.java:832) > at org.apache.activemq.broker.region.Queue.tryOrderedCursorAdd(Queue.java:927) > at org.apache.activemq.broker.region.Queue.doMessageSend(Queue.java:906) > at org.apache.activemq.broker.region.Queue.send(Queue.java:756) > at > org.apache.activemq.broker.region.DestinationFilter.send(DestinationFilter.java:138) > at > org.apache.activemq.broker.region.AbstractRegion.send(AbstractRegion.java:511) > at org.apache.activemq.broker.region.RegionBroker.send(RegionBroker.java:477) > at > org.apache.activemq.broker.jmx.ManagedRegionBroker.send(ManagedRegionBroker.java:343) > at org.apache.activemq.broker.BrokerFilter.send(BrokerFilter.java:154) > at > org.apache.activemq.broker.scheduler.SchedulerBroker.scheduledJob(SchedulerBroker.java:429) > at > org.apache.activemq.store.kahadb.scheduler.JobSchedulerImpl.fireJob(JobSchedulerImpl.java:823) > at > org.apache.activemq.store.kahadb.scheduler.JobSchedulerImpl.mainLoop(JobSchedulerImpl.java:753) > at > org.apache.activemq.store.kahadb.scheduler.JobSchedulerImpl.run(JobSchedulerImpl.java:699) > at java.lang.Thread.run(java.base@17.0.10/Thread.java:840) > Locked ownable synchronizers: <0x00077ebc3ed0> (a > java.util.concurrent.locks.ReentrantReadWriteLock$NonfairSync) > {code} > -- This message was sent by Atlassian Jira (v8.20.10#820010)
[jira] [Work logged] (AMQ-9448) RedeliveryPolicy causing lock in dispatcher
[ https://issues.apache.org/jira/browse/AMQ-9448?focusedWorklogId=910217=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-910217 ] ASF GitHub Bot logged work on AMQ-9448: --- Author: ASF GitHub Bot Created on: 16/Mar/24 08:19 Start Date: 16/Mar/24 08:19 Worklog Time Spent: 10m Work Description: thezbyg commented on PR #1177: URL: https://github.com/apache/activemq/pull/1177#issuecomment-2001904263 Please use existing org.apache.activemq.broker.scheduler.JmsSchedulerTest#testCron test to confirm that persistent scheduler currently deadlocks on CRON jobs. This is not the same deadlock as reported in AMQ-9448, but it is caused by the same reason. Issue Time Tracking --- Worklog Id: (was: 910217) Time Spent: 20m (was: 10m) > RedeliveryPolicy causing lock in dispatcher > --- > > Key: AMQ-9448 > URL: https://issues.apache.org/jira/browse/AMQ-9448 > Project: ActiveMQ Classic > Issue Type: Bug > Components: Broker >Affects Versions: 5.18.3 > Environment: ubuntu-22.04 > corretto-17 >Reporter: Oliver Holzmann >Priority: Critical > Attachments: doNotFireJobsWhileStoreLockIsHeld.patch, jstack1.out > > Time Spent: 20m > Remaining Estimate: 0h > > We are using ActiveMQ with redeliveryPolicy enabled: > {code:java} > > redeliveryDelay="1000" /> > {code} > When a message exceeds maximumRedeliveries the queue is "frozen". This means > the consumers can't read any messages and qrowing the queue in web console > causes error 504. > A thread dump shows a blocked JobScheduler:JMS thread: > {code:java} > priority : 5 > thread Id : x7fb985664960 > native Id : xae > state : BLOCKED > StackTrace: > "JobScheduler:JMS" #75 daemon prio=5 os_prio=0 cpu=478.63ms elapsed=757.29s > tid=0x7fb985664960 nid=0xae waiting for monitor entry [0x7fb80def1000] > java.lang.Thread.State: BLOCKED (on object monitor) > at > org.apache.activemq.broker.region.PrefetchSubscription.dispatchPending(PrefetchSubscription.java:647) > waiting to lock <0x00077ebc62a0> (a java.lang.Object) locked > <0x00077ebc6290> (a java.lang.Object) > at > org.apache.activemq.broker.region.PrefetchSubscription.add(PrefetchSubscription.java:159) > at org.apache.activemq.broker.region.Queue.doActualDispatch(Queue.java:2208) > at org.apache.activemq.broker.region.Queue.doDispatch(Queue.java:2156) > at org.apache.activemq.broker.region.Queue.pageInMessages(Queue.java:2307) > at org.apache.activemq.broker.region.Queue.iterate(Queue.java:1728) locked > <0x00077ebc5178> (a java.lang.Object) > at org.apache.activemq.broker.region.Queue.wakeup(Queue.java:1977) > at org.apache.activemq.broker.region.Queue.messageSent(Queue.java:1971) > at > org.apache.activemq.broker.region.Queue.doPendingCursorAdditions(Queue.java:832) > at org.apache.activemq.broker.region.Queue.tryOrderedCursorAdd(Queue.java:927) > at org.apache.activemq.broker.region.Queue.doMessageSend(Queue.java:906) > at org.apache.activemq.broker.region.Queue.send(Queue.java:756) > at > org.apache.activemq.broker.region.DestinationFilter.send(DestinationFilter.java:138) > at > org.apache.activemq.broker.region.AbstractRegion.send(AbstractRegion.java:511) > at org.apache.activemq.broker.region.RegionBroker.send(RegionBroker.java:477) > at > org.apache.activemq.broker.jmx.ManagedRegionBroker.send(ManagedRegionBroker.java:343) > at org.apache.activemq.broker.BrokerFilter.send(BrokerFilter.java:154) > at > org.apache.activemq.broker.scheduler.SchedulerBroker.scheduledJob(SchedulerBroker.java:429) > at > org.apache.activemq.store.kahadb.scheduler.JobSchedulerImpl.fireJob(JobSchedulerImpl.java:823) > at > org.apache.activemq.store.kahadb.scheduler.JobSchedulerImpl.mainLoop(JobSchedulerImpl.java:753) > at > org.apache.activemq.store.kahadb.scheduler.JobSchedulerImpl.run(JobSchedulerImpl.java:699) > at java.lang.Thread.run(java.base@17.0.10/Thread.java:840) > Locked ownable synchronizers: <0x00077ebc3ed0> (a > java.util.concurrent.locks.ReentrantReadWriteLock$NonfairSync) > {code} > -- This message was sent by Atlassian Jira (v8.20.10#820010)
[jira] [Work logged] (AMQ-9448) RedeliveryPolicy causing lock in dispatcher
[ https://issues.apache.org/jira/browse/AMQ-9448?focusedWorklogId=910216=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-910216 ] ASF GitHub Bot logged work on AMQ-9448: --- Author: ASF GitHub Bot Created on: 16/Mar/24 08:14 Start Date: 16/Mar/24 08:14 Worklog Time Spent: 10m Work Description: thezbyg opened a new pull request, #1177: URL: https://github.com/apache/activemq/pull/1177 Do not fire or schedule jobs while holding read lock on store. Issue Time Tracking --- Worklog Id: (was: 910216) Remaining Estimate: 0h Time Spent: 10m > RedeliveryPolicy causing lock in dispatcher > --- > > Key: AMQ-9448 > URL: https://issues.apache.org/jira/browse/AMQ-9448 > Project: ActiveMQ Classic > Issue Type: Bug > Components: Broker >Affects Versions: 5.18.3 > Environment: ubuntu-22.04 > corretto-17 >Reporter: Oliver Holzmann >Priority: Critical > Attachments: doNotFireJobsWhileStoreLockIsHeld.patch, jstack1.out > > Time Spent: 10m > Remaining Estimate: 0h > > We are using ActiveMQ with redeliveryPolicy enabled: > {code:java} > > redeliveryDelay="1000" /> > {code} > When a message exceeds maximumRedeliveries the queue is "frozen". This means > the consumers can't read any messages and qrowing the queue in web console > causes error 504. > A thread dump shows a blocked JobScheduler:JMS thread: > {code:java} > priority : 5 > thread Id : x7fb985664960 > native Id : xae > state : BLOCKED > StackTrace: > "JobScheduler:JMS" #75 daemon prio=5 os_prio=0 cpu=478.63ms elapsed=757.29s > tid=0x7fb985664960 nid=0xae waiting for monitor entry [0x7fb80def1000] > java.lang.Thread.State: BLOCKED (on object monitor) > at > org.apache.activemq.broker.region.PrefetchSubscription.dispatchPending(PrefetchSubscription.java:647) > waiting to lock <0x00077ebc62a0> (a java.lang.Object) locked > <0x00077ebc6290> (a java.lang.Object) > at > org.apache.activemq.broker.region.PrefetchSubscription.add(PrefetchSubscription.java:159) > at org.apache.activemq.broker.region.Queue.doActualDispatch(Queue.java:2208) > at org.apache.activemq.broker.region.Queue.doDispatch(Queue.java:2156) > at org.apache.activemq.broker.region.Queue.pageInMessages(Queue.java:2307) > at org.apache.activemq.broker.region.Queue.iterate(Queue.java:1728) locked > <0x00077ebc5178> (a java.lang.Object) > at org.apache.activemq.broker.region.Queue.wakeup(Queue.java:1977) > at org.apache.activemq.broker.region.Queue.messageSent(Queue.java:1971) > at > org.apache.activemq.broker.region.Queue.doPendingCursorAdditions(Queue.java:832) > at org.apache.activemq.broker.region.Queue.tryOrderedCursorAdd(Queue.java:927) > at org.apache.activemq.broker.region.Queue.doMessageSend(Queue.java:906) > at org.apache.activemq.broker.region.Queue.send(Queue.java:756) > at > org.apache.activemq.broker.region.DestinationFilter.send(DestinationFilter.java:138) > at > org.apache.activemq.broker.region.AbstractRegion.send(AbstractRegion.java:511) > at org.apache.activemq.broker.region.RegionBroker.send(RegionBroker.java:477) > at > org.apache.activemq.broker.jmx.ManagedRegionBroker.send(ManagedRegionBroker.java:343) > at org.apache.activemq.broker.BrokerFilter.send(BrokerFilter.java:154) > at > org.apache.activemq.broker.scheduler.SchedulerBroker.scheduledJob(SchedulerBroker.java:429) > at > org.apache.activemq.store.kahadb.scheduler.JobSchedulerImpl.fireJob(JobSchedulerImpl.java:823) > at > org.apache.activemq.store.kahadb.scheduler.JobSchedulerImpl.mainLoop(JobSchedulerImpl.java:753) > at > org.apache.activemq.store.kahadb.scheduler.JobSchedulerImpl.run(JobSchedulerImpl.java:699) > at java.lang.Thread.run(java.base@17.0.10/Thread.java:840) > Locked ownable synchronizers: <0x00077ebc3ed0> (a > java.util.concurrent.locks.ReentrantReadWriteLock$NonfairSync) > {code} > -- This message was sent by Atlassian Jira (v8.20.10#820010)