Stephen O'Donnell created HDDS-16399:
----------------------------------------
Summary: Fix flaky testFinalizationEmptyClusterDataDistribution
Key: HDDS-16399
URL: https://issues.apache.org/jira/browse/HDDS-16399
Project: Apache Ozone
Issue Type: Bug
Reporter: Stephen O'Donnell
Assignee: Stephen O'Donnell
[~erose]found that the test
TestScmDataDistributionFinalization.testFinalizationEmptyClusterDataDistribution()
fails about 1 time per 100 runs. This was also reported in HDDS-14050 as the
test is marked as flaky in the code.
The issue seems to be caused by race conditions in background services. Pasting
directly from Claude:
{code:java}
1. Line 209: addTransactions(txList) — incrDeletedBlocksSummary increments
totalTxCount to 4 (in-memory, before the DB write), then writes txs + summary
to the buffer.
2. Line 210: flushDBTransactionBuffer — the 4 new-format txs are now in RocksDB.
3. SCMBlockDeletingService runs concurrently (every 100ms): calls
getTransactions, which calls getTransaction → addTxToTxSizeMap for each sent tx
(line 330–335 in DeletedBlockLogImpl) — this populates txSizeMap for those tx
IDs.
4. DNs confirm deletion fast in the mini-cluster → commitTransactions →
removeTransactions → txSizeMap.remove(txID) returns the TxBlockInfo →
descDeletedBlocksSummary × 4 → totalTxCount drops to 0.
5. But: the removal is only in the transaction buffer, not yet committed to
RocksDB.
6. Line 212: getRowsInTable reads RocksDB directly → still sees 4 rows. ✓
7. Line 215: getTransactionSummary reads in-memory totalTxCount → 0. ✗txSizeMap
is indeed never populated in addTransactions itself — it only gets populated in
addTxToTxSizeMap when the block deleting service calls getTransactions and
decides to send a tx to a DN. The service is the path that both populates
txSizeMap and later drains it.The fix is the same pattern used in
testFinalizationNonEmptyClusterDataDistribution: stop the
SCMBlockDeletingService before the section that asserts the summary counts, and
restart it before the waitFor that expects deletion to complete (line 221).
{code}
There was also a second type of failure also caused by race conditions with the
background services:
{code:java}
[ERROR] Tests run: 2, Failures: 1, Errors: 0, Skipped: 0, Time elapsed: 98.10
s <<< FAILURE! -- in
org.apache.hadoop.hdds.upgrade.TestScmDataDistributionFinalization
[ERROR]
org.apache.hadoop.hdds.upgrade.TestScmDataDistributionFinalization.testFinalizationEmptyClusterDataDistribution
-- Time elapsed: 35.66 s <<< FAILURE!
org.opentest4j.AssertionFailedError:
expected: <totalTransactionCount: 0
totalBlockCount: 0
totalBlockSize: 0
totalBlockReplicatedSize: 0
> but was: <totalTransactionCount: 18446744073709551612
totalBlockCount: 18446744073709551596
totalBlockSize: 18446744073688580096
totalBlockReplicatedSize: 18446744073646637056
>
{code}
This appears to also be caused by the same background services and stopping it
appears to fix it.
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
