[
https://issues.apache.org/jira/browse/HDDS-16399?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
ASF GitHub Bot updated HDDS-16399:
----------------------------------
Labels: pull-request-available (was: )
> Fix flaky testFinalizationEmptyClusterDataDistribution
> ------------------------------------------------------
>
> Key: HDDS-16399
> URL: https://issues.apache.org/jira/browse/HDDS-16399
> Project: Apache Ozone
> Issue Type: Bug
> Reporter: Stephen O'Donnell
> Assignee: Stephen O'Donnell
> Priority: Major
> Labels: pull-request-available
>
> [~erose]found that the test
> TestScmDataDistributionFinalization.testFinalizationEmptyClusterDataDistribution()
> fails about 1 time per 100 runs. This was also reported in HDDS-14050 as the
> test is marked as flaky in the code.
> The issue seems to be caused by race conditions in background services.
> Pasting directly from Claude:
> {code:java}
> 1. Line 209: addTransactions(txList) — incrDeletedBlocksSummary increments
> totalTxCount to 4 (in-memory, before the DB write), then writes txs + summary
> to the buffer.
> 2. Line 210: flushDBTransactionBuffer — the 4 new-format txs are now in
> RocksDB.
> 3. SCMBlockDeletingService runs concurrently (every 100ms): calls
> getTransactions, which calls getTransaction → addTxToTxSizeMap for each sent
> tx (line 330–335 in DeletedBlockLogImpl) — this populates txSizeMap for those
> tx IDs.
> 4. DNs confirm deletion fast in the mini-cluster → commitTransactions →
> removeTransactions → txSizeMap.remove(txID) returns the TxBlockInfo →
> descDeletedBlocksSummary × 4 → totalTxCount drops to 0.
> 5. But: the removal is only in the transaction buffer, not yet committed to
> RocksDB.
> 6. Line 212: getRowsInTable reads RocksDB directly → still sees 4 rows. ✓
> 7. Line 215: getTransactionSummary reads in-memory totalTxCount → 0.
> ✗txSizeMap is indeed never populated in addTransactions itself — it only gets
> populated in addTxToTxSizeMap when the block deleting service calls
> getTransactions and decides to send a tx to a DN. The service is the path
> that both populates txSizeMap and later drains it.The fix is the same pattern
> used in testFinalizationNonEmptyClusterDataDistribution: stop the
> SCMBlockDeletingService before the section that asserts the summary counts,
> and restart it before the waitFor that expects deletion to complete (line
> 221). {code}
> There was also a second type of failure also caused by race conditions with
> the background services:
> {code:java}
> [ERROR] Tests run: 2, Failures: 1, Errors: 0, Skipped: 0, Time elapsed:
> 98.10 s <<< FAILURE! -- in
> org.apache.hadoop.hdds.upgrade.TestScmDataDistributionFinalization
> [ERROR]
> org.apache.hadoop.hdds.upgrade.TestScmDataDistributionFinalization.testFinalizationEmptyClusterDataDistribution
> -- Time elapsed: 35.66 s <<< FAILURE!
> org.opentest4j.AssertionFailedError:
> expected: <totalTransactionCount: 0
> totalBlockCount: 0
> totalBlockSize: 0
> totalBlockReplicatedSize: 0
> > but was: <totalTransactionCount: 18446744073709551612
> totalBlockCount: 18446744073709551596
> totalBlockSize: 18446744073688580096
> totalBlockReplicatedSize: 18446744073646637056
> >
> {code}
> This appears to also be caused by the same background services and stopping
> it appears to fix it.
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
