smengcl commented on PR #11669: URL: https://github.com/apache/ozone/pull/11669#issuecomment-6488161845
> This is a stop gap before we solve the underlying issue. Yes. The real fix is HDDS-11452 (atomic purge). This patch only keeps a half-applied purge out of the DB. > snapshot chain corruption can happen due to race conditions Not quite a race. OM applies transactions one at a time. The purge only fails this way if the in-memory snapshot chain already lists a snapshot that has no row in `snapshotInfoTable`. An earlier bug has to cause that. > depending on whether the index is flushed or not, the transaction may be skipped Correct. If flushed, the purge is skipped on restart. The chain is rebuilt from the DB, which the purge never wrote to, and SnapshotDeletingService retries the purge later. > If not skipped, the race can still happen and may abort again. It may require restarting OM multiple times to get out of it. Replay is deterministic, not timing dependent. * If the entry that caused the mismatch is not replayed, the first restart fixes it. * If it is replayed, OM aborts again. The entries before the purge are usually flushed during that replay, so the next restart starts later in the log. The chain is rebuilt from `snapshotInfoTable` at startup, so once that entry is flushed, the purge goes through. To check progress, compare the transaction index in the abort message with the startup line `LastAppliedIndex is set from TransactionInfo from OM DB as (t:..., i:...)`. If the startup index stops going up, stop all OMs and run `ozone repair om skip-ratis-transaction --index <N>` on each one. Skipping a purge is safe: none of its changes reached the DB, and SnapshotDeletingService sends it again. Without this patch, OM keeps running in that state, and a later write can persist a broken previous snapshot ID. Then the chain fails to load at startup, snapshot GC stops, the next snapshot create terminates every OM, and recovery needs `ozone repair om snapshot chain` with OM offline. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected] --------------------------------------------------------------------- To unsubscribe, e-mail: [email protected] For additional commands, e-mail: [email protected]
