kerneltime commented on PR #10953: URL: https://github.com/apache/ozone/pull/10953#issuecomment-5195291892
## Reproduction you can run yourself The analysis above is a code trace, so here is the race actually happening. Branch, on my fork, **not for merge**: **https://github.com/kerneltime/ozone/tree/HDDS-16092-repro** It is unmodified `master` plus one integration test. Nothing is staged, no delay is injected, no DB is edited by hand — a real 3-OM HA cluster, real client writes, the real Ratis snapshot trigger racing the real double-buffer flush. ``` mvn -pl hadoop-ozone/integration-test -am test \ -Dtest=TestHDDS16092TransactionInfoRegression ``` On that branch it **fails**: ``` HDDS-16092 HIT: omNode-3: persisted index went BACKWARDS 51 -> 50 AssertionFailedError: The persisted transaction index moved backwards, so the DB now contains transactions its own index disclaims ``` Cherry-pick this PR's commits on top and it passes — 0 regressions across 21,635,614 samples in a full 90-second run, versus a hit within ~30 seconds without them. ### Why the threshold is lowered The one setting changed from production is `ozone.om.ratis.snapshot.auto.trigger.threshold`, from its 400000 default down to 50. That does **not** create the race. It turns one draw per 400000 transactions into a draw every few transactions, so the existing window gets sampled thousands of times in half a minute instead of once. That ratio is the whole reason this has not been seen in the field. ### Why it is on a branch and not in this PR It depends on a non-production setting and on thread timing, so a green run proves nothing — it would be a CI gate that passes whether or not the bug is present. The committed tests in this PR are the deterministic equivalent, each verified by mutation to fail when the half of the fix it covers is removed. This branch is for seeing the real thing. ### The detector A watcher per OM polls `TRANSACTION_INFO_KEY` with `getSkipCache` (as `TransactionInfo.readTransactionInfo` does for this key) and records any move backwards. A persisted watermark decreasing is self-evidently wrong, so it needs no knowledge of the true applied index. All three OMs are watched, since each runs its own state machine updater and flush daemon and is an independent draw. Observed hits land on followers. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected] --------------------------------------------------------------------- To unsubscribe, e-mail: [email protected] For additional commands, e-mail: [email protected]
