sodonnel opened a new pull request, #11228: URL: https://github.com/apache/ozone/pull/11228
## What changes were proposed in this pull request? @errose28 found that the test TestScmDataDistributionFinalization.testFinalizationEmptyClusterDataDistribution() fails about 1 time per 100 runs. This was also reported in [HDDS-14050](https://issues.apache.org/jira/browse/HDDS-14050) as the test is marked as flaky in the code. The issue seems to be caused by race conditions in background services. Pasting directly from Claude: ``` 1. Line 209: addTransactions(txList) — incrDeletedBlocksSummary increments totalTxCount to 4 (in-memory, before the DB write), then writes txs + summary to the buffer. 2. Line 210: flushDBTransactionBuffer — the 4 new-format txs are now in RocksDB. 3. SCMBlockDeletingService runs concurrently (every 100ms): calls getTransactions, which calls getTransaction → addTxToTxSizeMap for each sent tx (line 330–335 in DeletedBlockLogImpl) — this populates txSizeMap for those tx IDs. 4. DNs confirm deletion fast in the mini-cluster → commitTransactions → removeTransactions → txSizeMap.remove(txID) returns the TxBlockInfo → descDeletedBlocksSummary × 4 → totalTxCount drops to 0. 5. But: the removal is only in the transaction buffer, not yet committed to RocksDB. 6. Line 212: getRowsInTable reads RocksDB directly → still sees 4 rows. ✓ 7. Line 215: getTransactionSummary reads in-memory totalTxCount → 0. ✗txSizeMap is indeed never populated in addTransactions itself — it only gets populated in addTxToTxSizeMap when the block deleting service calls getTransactions and decides to send a tx to a DN. The service is the path that both populates txSizeMap and later drains it.The fix is the same pattern used in testFinalizationNonEmptyClusterDataDistribution: stop the SCMBlockDeletingService before the section that asserts the summary counts, and restart it before the waitFor that expects deletion to complete (line 221). ``` There was also a second type of failure also caused by race conditions with the background services: ``` [ERROR] Tests run: 2, Failures: 1, Errors: 0, Skipped: 0, Time elapsed: 98.10 s <<< FAILURE! -- in org.apache.hadoop.hdds.upgrade.TestScmDataDistributionFinalization [ERROR] org.apache.hadoop.hdds.upgrade.TestScmDataDistributionFinalization.testFinalizationEmptyClusterDataDistribution -- Time elapsed: 35.66 s <<< FAILURE! org.opentest4j.AssertionFailedError: expected: <totalTransactionCount: 0 totalBlockCount: 0 totalBlockSize: 0 totalBlockReplicatedSize: 0 > but was: <totalTransactionCount: 18446744073709551612 totalBlockCount: 18446744073709551596 totalBlockSize: 18446744073688580096 totalBlockReplicatedSize: 18446744073646637056 > ``` This appears to also be caused by the same background services and stopping it appears to fix it. ## What is the link to the Apache JIRA https://issues.apache.org/jira/browse/HDDS-16399 ## How was this patch tested? Ran the changed test a few times and it passed. I will try to get a run of 100 iterations in CI to give better proof. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected] --------------------------------------------------------------------- To unsubscribe, e-mail: [email protected] For additional commands, e-mail: [email protected]
