ok, maybe open a new issue, and attach your logs and configure a data small journal data file so that it can easily be attached once you find the block in gc.
If recovery of local xa multi store transactions is the problem, I would expect some log information from: org.apache.activemq.store.kahadb.MultiKahaDBTransactionStore use setJournalMaxFileLength=24k (it needs to be > 8k) on the persistence adapter. It may require killing the broker in the debugger to get to the bottom of this or using a system property that can kill the broker to force recovery in this case. An issue/jira ticket will put it on the radar in any event. I don't see that AMQ-3866 is using mkahadb. On 23 October 2012 14:47, Gilles Harloux <gilles.harl...@gmail.com> wrote: > I doubt I can reproduce it as a unit test: What I see happens > (sometimes, sometimes not) because I sigkill the process. If I replace > the kill with an orderly shutdown, everything works out ok. > > On Tue, Oct 23, 2012 at 3:38 PM, Gary Tully <gary.tu...@gmail.com> wrote: >> do you think you could come up with a unit test that shows growth. So >> something that uses mkahadb and produces or consumed in a transaction >> from destinations that span the stores. >> >> You could configure the size of journal data files to 16k or >> something, such that any issue shows up very quickly. >> >> A test case like that would help clarify the use case and if it can >> reproduce, would be a great start to getting a quick resolution. >> >> There is a test that does transacted send received across stores, but >> does not check for cleanup. It may be easy to extend it in that way. >> >> in activemq-core >> org.apache.activemq.store.StorePerDestinationTest#testTransactedSendReceiveAcrossStores >> >> >> On 23 October 2012 14:20, Gilles Harloux <gilles.harl...@gmail.com> wrote: >>> I found that link while you were answering, it seems <g>. I also found >>> https://issues.apache.org/jira/browse/AMQ-3866 and it looks a lot like >>> what I experience. >>> >>> I tried to delete the index, start a broker and debug as explained in >>> that ticket. Breakpointing at MessageDatabase#process, line 904 >>> (version 5.7.0) tells me it seems there is XA in the mix (data is a >>> KahaAddMessageCommand instance, with >>> data.f_transactionInfo.f_localTransactionId == null and >>> data.f_transactionInfo.f_xaTransactionId != null). I guess it's due to >>> the use of mKahaDB, where I am transacting across both stores. >>> >>> In that ticket, you asked for the relevant data file. I can share mine >>> if there's any interest -- I just have to find a convenient way to >>> provide you with a 32 Mb file. >>> >>> On Tue, Oct 23, 2012 at 12:24 PM, Gary Tully <gary.tu...@gmail.com> wrote: >>>> have a read of >>>> http://activemq.apache.org/why-do-kahadb-log-files-remain-after-cleanup.html >>>> >>>> The logging referenced there will show you what destinations are >>>> holding on to references to the journal data files. >>>> >>>> w.r.t the usage %, the journal size increases in chunks of data file >>>> size, so a new journal data file can push the usage over the limit. >>>> >>>> On 23 October 2012 09:15, Gilles Harloux <gilles.harl...@gmail.com> wrote: >>>>> Hi, >>>>> >>>>> I have a system embedding a broker with KahaDB as a store. I am trying >>>>> to get a feel about disaster recovery behavior. So what I am basically >>>>> doing is randomly kill & restart the process. I see a condition (after >>>>> killing & restarting multiple times) where the messages get consumed, >>>>> but kahaDB journal files (db-*.log) don't get reclaimed. As I set up a >>>>> storeUsage limit, it ends up blocking the system. >>>>> >>>>> I say the messages get consumed, because I can see my application >>>>> logging what messages it gets to handle; Also, JMX tells me that the >>>>> broker's TotalMesageCount is zero, while the StorePercentUsage is >>>>> above 100 (depending on parameters such as message size & rate, I saw >>>>> anything from 112 to 293 percent usage). >>>>> >>>>> So, two questions: >>>>> - How is it possible (for any reasons) to get a usage percentage >>>>> above 100? (In other situations, I saw it happen with memory too). >>>>> - Why is it I can't get kahaDB to reclaim seemingly unused journal files? >>>>> >>>>> TIA, >>>> >>>> >>>> >>>> -- >>>> http://redhat.com >>>> http://blog.garytully.com >> >> >> >> -- >> http://redhat.com >> http://blog.garytully.com -- http://redhat.com http://blog.garytully.com
