[ https://issues.apache.org/jira/browse/AMQ-7221?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16860207#comment-16860207 ]
Alan Protasio commented on AMQ-7221: ------------------------------------ Hey [~gtully] [~cshannon] Can you guys take a look on this change when you have a some time? I think it is straight forward.. Thanks a lot! :D > Delete Scheduled messages causes ActiveMQ create/write a unnecessary huge > transaction file > ------------------------------------------------------------------------------------------ > > Key: AMQ-7221 > URL: https://issues.apache.org/jira/browse/AMQ-7221 > Project: ActiveMQ > Issue Type: Improvement > Components: KahaDB > Reporter: Alan Protasio > Priority: Major > Time Spent: 10m > Remaining Estimate: 0h > > Hi, > I tried to delete all my scheduled messages and I noticed that the broker > stop responding and a huge transaction file was created. > The Scheduled.db file had ~150mb and the transaction was already 140GB when I > give up and wiped the broker. > Looking at the code, I noticed that the cause of it is mostly this 2 lines: > [https://github.com/apache/activemq/blob/master/activemq-kahadb-store/src/main/java/org/apache/activemq/store/kahadb/scheduler/JobSchedulerImpl.java#L512] > [https://github.com/apache/activemq/blob/master/activemq-kahadb-store/src/main/java/org/apache/activemq/store/kahadb/scheduler/JobSchedulerStoreImpl.java#L514] > And > [https://github.com/apache/activemq/blob/master/activemq-kahadb-store/src/main/java/org/apache/activemq/store/kahadb/scheduler/JobSchedulerImpl.java#L501] > [https://github.com/apache/activemq/blob/master/activemq-kahadb-store/src/main/java/org/apache/activemq/store/kahadb/scheduler/JobSchedulerStoreImpl.java#L481] > On the first case, we are updating an List<Int> inside a transaction but in > order to do that, we get its value from the BTree and put back (with an added > element). When we do that, the whole list is again serialized and written > inside the transaction. > Similar thing is happening on the second case, We are decrementing one by a > number of references to a journal file but we could do it only one time in > the end saving lots of writes. > After the proposed patch, the transaction that was before more than 140GB > (and it didn't finish) was reduced to ~500 mb, and the broker could recover > in less than 1 minute (i gave up before with ~15 minutes). > On the modified test in the PR we can see that the transaction before the > change is ~50mb and after the change is ~500k > Before: > 2019-06-03 23:58:19,931 > [//localhost#1-1|https://issues.apache.org/localhost#1-1] - DEBUG Transaction > - Committing transaction 5177: Size 53320 kb > After: > 2019-06-03 23:59:13,578 > [//localhost#1-2|https://issues.apache.org/localhost#1-2] - DEBUG Transaction > - Committing transaction 5178: Size 496 kb > > PS: Keeping removedJobFileIds in memory will not increase the memory usage bc > we were loading it in memory anyway when reading from the index. -- This message was sent by Atlassian JIRA (v7.6.3#76005)