[
https://issues.apache.org/jira/browse/AMQ-7221?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16860207#comment-16860207
]
Alan Protasio commented on AMQ-7221:
------------------------------------
Hey [~gtully] [~cshannon]
Can you guys take a look on this change when you have a some time? I think it
is straight forward..
Thanks a lot! :D
> Delete Scheduled messages causes ActiveMQ create/write a unnecessary huge
> transaction file
> ------------------------------------------------------------------------------------------
>
> Key: AMQ-7221
> URL: https://issues.apache.org/jira/browse/AMQ-7221
> Project: ActiveMQ
> Issue Type: Improvement
> Components: KahaDB
> Reporter: Alan Protasio
> Priority: Major
> Time Spent: 10m
> Remaining Estimate: 0h
>
> Hi,
> I tried to delete all my scheduled messages and I noticed that the broker
> stop responding and a huge transaction file was created.
> The Scheduled.db file had ~150mb and the transaction was already 140GB when I
> give up and wiped the broker.
> Looking at the code, I noticed that the cause of it is mostly this 2 lines:
> [https://github.com/apache/activemq/blob/master/activemq-kahadb-store/src/main/java/org/apache/activemq/store/kahadb/scheduler/JobSchedulerImpl.java#L512]
> [https://github.com/apache/activemq/blob/master/activemq-kahadb-store/src/main/java/org/apache/activemq/store/kahadb/scheduler/JobSchedulerStoreImpl.java#L514]
> And
> [https://github.com/apache/activemq/blob/master/activemq-kahadb-store/src/main/java/org/apache/activemq/store/kahadb/scheduler/JobSchedulerImpl.java#L501]
> [https://github.com/apache/activemq/blob/master/activemq-kahadb-store/src/main/java/org/apache/activemq/store/kahadb/scheduler/JobSchedulerStoreImpl.java#L481]
> On the first case, we are updating an List<Int> inside a transaction but in
> order to do that, we get its value from the BTree and put back (with an added
> element). When we do that, the whole list is again serialized and written
> inside the transaction.
> Similar thing is happening on the second case, We are decrementing one by a
> number of references to a journal file but we could do it only one time in
> the end saving lots of writes.
> After the proposed patch, the transaction that was before more than 140GB
> (and it didn't finish) was reduced to ~500 mb, and the broker could recover
> in less than 1 minute (i gave up before with ~15 minutes).
> On the modified test in the PR we can see that the transaction before the
> change is ~50mb and after the change is ~500k
> Before:
> 2019-06-03 23:58:19,931
> [//localhost#1-1|https://issues.apache.org/localhost#1-1] - DEBUG Transaction
> - Committing transaction 5177: Size 53320 kb
> After:
> 2019-06-03 23:59:13,578
> [//localhost#1-2|https://issues.apache.org/localhost#1-2] - DEBUG Transaction
> - Committing transaction 5178: Size 496 kb
>
> PS: Keeping removedJobFileIds in memory will not increase the memory usage bc
> we were loading it in memory anyway when reading from the index.
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
