[
https://issues.apache.org/jira/browse/ARTEMIS-3992?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
SL reopened ARTEMIS-3992:
-------------------------
after a few days, an issue came back.
The full stacktrace on JournalImpl is gone, but we have a bunch of warnings like
{noformat}
2022-09-19 00:00:16,590 WARN [org.apache.activemq.artemis.core.server]
AMQ222029: Could not locate page transaction 190 588 717, ignoring message on
position PagePositionImpl [pageNr=1800, messageNr=3287, recordID=-1] ...
2022-09-19 00:00:16,590 WARN [org.apache.activemq.artemis.core.server]
AMQ222029: Could not locate page transaction 190 589 111, ignoring message on
position PagePositionImpl [pageNr=1800, messageNr=3288, recordID=-1] ...
2022-09-19 00:00:16,590 WARN [org.apache.activemq.artemis.core.server]
AMQ222029: Could not locate page transaction 190 588 472, ignoring message on
position PagePositionImpl [pageNr=1800, messageNr=3289, recordID=-1] ...
2022-09-19 00:00:16,590 WARN [org.apache.activemq.artemis.core.server]
AMQ222029: Could not locate page transaction 190 589 110, ignoring message on
position PagePositionImpl [pageNr=1800, messageNr=3292, recordID=-1] ...
2022-09-19 00:00:16,590 WARN [org.apache.activemq.artemis.core.server]
AMQ222029: Could not locate page transaction 190 588 727, ignoring message on
position PagePositionImpl [pageNr=1800, messageNr=3293, recordID=-1] ...
2022-09-19 00:00:17,741 WARN [org.apache.activemq.artemis.core.server]
AMQ222029: Could not locate page transaction 190 588 472, ignoring message on
position PagePositionImpl [pageNr=1800, messageNr=3303, recordID=-1] ...
2022-09-19 00:00:17,741 WARN [org.apache.activemq.artemis.core.server]
AMQ222029: Could not locate page transaction 190 589 111, ignoring message on
position PagePositionImpl [pageNr=1800, messageNr=3304, recordID=-1] ...
{noformat}
and
{noformat}
2022-09-19 01:18:07,715 WARN [org.apache.activemq.artemis.journal] AMQ142007:
Can not find record 191 873 186 during compact replay
2022-09-19 01:18:07,715 WARN [org.apache.activemq.artemis.journal] AMQ142007:
Can not find record 191 873 192 during compact replay
2022-09-19 01:18:07,715 WARN [org.apache.activemq.artemis.journal] AMQ142007:
Can not find record 191 873 207 during compact replay
2022-09-19 01:18:07,715 WARN [org.apache.activemq.artemis.journal] AMQ142007:
Can not find record 191 873 214 during compact replay
2022-09-19 01:18:07,715 WARN [org.apache.activemq.artemis.journal] AMQ142007:
Can not find record 191 873 219 during compact replay
2022-09-19 01:18:07,715 WARN [org.apache.activemq.artemis.journal] AMQ142007:
Can not find record 191 873 223 during compact replay
2022-09-19 01:18:07,715 WARN [org.apache.activemq.artemis.journal] AMQ142007:
Can not find record 191 873 199 during compact replay
2022-09-19 01:18:07,715 WARN [org.apache.activemq.artemis.journal] AMQ142007:
Can not find record 191 873 202 during compact replay
2022-09-19 01:18:07,715 WARN [org.apache.activemq.artemis.journal] AMQ142007:
Can not find record 191 873 210 during compact replay
2022-09-19 01:18:07,715 WARN [org.apache.activemq.artemis.journal] AMQ142007:
Can not find record 191 873 228 during compact replay
2022-09-19 01:18:07,715 WARN [org.apache.activemq.artemis.journal] AMQ142007:
Can not find record 191 873 375 during compact replay
2022-09-19 01:18:07,715 WARN [org.apache.activemq.artemis.journal] AMQ142007:
Can not find record 191 873 369 during compact replay
2022-09-19 01:18:07,715 WARN [org.apache.activemq.artemis.journal] AMQ142007:
Can not find record 191 873 372 during compact replay
2022-09-19 01:18:07,715 WARN [org.apache.activemq.artemis.journal] AMQ142007:
Can not find record 191 873 384 during compact replay
2022-09-19 01:18:07,715 WARN [org.apache.activemq.artemis.journal] AMQ142007:
Can not find record 191 873 378 during compact replay
2022-09-19 01:18:07,715 WARN [org.apache.activemq.artemis.journal] AMQ142007:
Can not find record 191 873 387 during compact replay
2022-09-19 01:18:07,715 WARN [org.apache.activemq.artemis.journal] AMQ142007:
Can not find record 191 873 396 during compact replay
2022-09-19 01:18:07,715 WARN [org.apache.activemq.artemis.journal] AMQ142007:
Can not find record 191 873 402 during compact replay
2022-09-19 01:18:07,715 WARN [org.apache.activemq.artemis.journal] AMQ142007:
Can not find record 191 873 408 during compact replay
2022-09-19 01:18:07,715 WARN [org.apache.activemq.artemis.journal] AMQ142007:
Can not find record 191 873 411 during compact replay
{noformat}
is this a sign of a serious issue with the persistant store ?
> Store corrumption and broker instabillty with rollback of XA transactions
> -------------------------------------------------------------------------
>
> Key: ARTEMIS-3992
> URL: https://issues.apache.org/jira/browse/ARTEMIS-3992
> Project: ActiveMQ Artemis
> Issue Type: Bug
> Components: Broker
> Affects Versions: 2.16.0
> Reporter: SL
> Priority: Major
>
> Edit : i had bad information about the time of the upgrade to 2.24.0, it was
> repeated just before upgrade, final status of the issue pending.
> We are experiancing a major stability issue with artemis which seems
> triggered by expired XA transactions.
> It starts with a bunch of timeouts like
> {noformat}
> 2022-09-13 00:00:02,970 WARN [org.apache.activemq.artemis.core.server]
> AMQ222103: transaction with xid XidImpl (2133539424 (...) timed out{noformat}
> Then a lot of recurring exceptions on the persistent store
> {noformat}
> MQ222055: Error on deleting duplicate cache: java.lang.IllegalStateException:
> Cannot find add info 228196096 on compactor or current records
> at
> org.apache.activemq.artemis.core.journal.impl.JournalImpl.checkKnownRecordID(JournalImpl.java:1152)
> [artemis-journal-2.16.0.jar:2.16.0]
> at
> org.apache.activemq.artemis.core.journal.impl.JournalImpl.appendDeleteRecord(JournalImpl.java:989)
> [artemis-journal-2.16.0.jar:2.16.0]
> at
> org.apache.activemq.artemis.core.persistence.impl.journal.AbstractJournalStorageManager.deleteDuplicateID(AbstractJournalStorageManager.java:482)
> [artemis-server-2.16.0.jar:2.16.0]
> at
> org.apache.activemq.artemis.core.postoffice.impl.DuplicateIDCacheImpl.addToCacheInMemory(DuplicateIDCacheImpl.java:265)
> [artemis-server-2.16.0.jar:2.16.0]
> at
> org.apache.activemq.artemis.core.postoffice.impl.DuplicateIDCacheImpl.access$000(DuplicateIDCacheImpl.java:41)
> [artemis-server-2.16.0.jar:2.16.0]
> at
> org.apache.activemq.artemis.core.postoffice.impl.DuplicateIDCacheImpl$AddDuplicateIDOperation.process(DuplicateIDCacheImpl.java:347)
> [artemis-server-2.16.0.jar:2.16.0]
> at
> org.apache.activemq.artemis.core.postoffice.impl.DuplicateIDCacheImpl$AddDuplicateIDOperation.beforeCommit(DuplicateIDCacheImpl.java:363)
> [artemis-server-2.16.0.jar:2.16.0]
> at
> org.apache.activemq.artemis.core.transaction.impl.TransactionImpl.beforeCommit(TransactionImpl.java:599)
> [artemis-server-2.16.0
> {noformat}
> From client side the consuming seems to slow down and at some point stops
> completely.
> The broker can partialy recover with a restart but its seems be still have
> issues if not given a new clean and empty persistant store.
> (Note : it might be similar to ARTEMIS-2373)
> Background :
> - It's a standalone artemis instance serving as front for other brokers
> (connected by bridges, working fine). It forwards messages submitted by
> clients to brokers connected to applications services and get back response
> messages which are consumed by the clients (basically a kind of reverse
> proxy).
> - It has been recently upgraded to 2.24.0 hoping that would fix the issue,
> but it remains identical.
> - It's a production system, the issue have not yet been reproduced on test
> environments (but it is repeated several times on this production environment)
> - We do not own the client trying to consume the messages and have little
> information on the specifics of its internals and XA usage.
> - Clients not using XA did not exhibit this kind of issue using the services
> for months, even years.
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
