greenwich opened a new pull request, #11709:
URL: https://github.com/apache/ozone/pull/11709
The OM can terminate when Recon's delta sync races with RocksDB's WAL
archive purge.
Recon calls `OzoneManager.getDBUpdates` → `RDBStore.getUpdatesSince` →
`RocksDatabase.getUpdatesSince`. If RocksDB fails there with an `IOError`,
`RocksDatabase.closeOnError` closes the whole OM DB, because `shouldClose`
returns true for every `IOError` and `Corruption`. Within milliseconds
`OMDoubleBufferFlushThread` and the state machine apply thread hit the closed
DB and call `terminate(1)`. Nothing is corrupt. The OM restarts and replays the
Ratis log, but the Ratis group is without a leader for several seconds each
time.
The error is RocksDB reporting that a WAL file disappeared while
`GetUpdatesSince` was listing or opening WALs:
java.io.IOException: RocksDatabase[/data/om/db/om.db]: Failed to
getUpdatesSince 4890204708; status : IOError(Undefined);
message : while stat a file for size:
/data/om/db/om.db/archive/532231.log: No such file or directory
In `WalManager::GetUpdatesSince` (unchanged between RocksDB 7.7.3 and
10.10.1):
- **Listing:** `ReadFirstRecord` already tolerates a missing file, and the
size check tolerates a live WAL that has just moved to `archive/`. A file
listed in `archive/` and purged before the size check returns the error, as
does a live WAL whose archived copy is already gone.
- **Opening the iterator:** opening the first retained WAL has no fallback
for a purged archived file (`While opening a file for sequentially reading: …`).
Upstream RocksDB issue #11338 was closed as not planned, pointing to #5763.
Closing the DB is the wrong reaction. Archived WALs are obsolete for crash
recovery and exist only for `GetUpdatesSince`-style readers.
`RDBStore.getUpdatesSince` already tolerates errors while reading archived
batches; only this open step closed the DB.
This PR changes only `RocksDatabase.getUpdatesSince`:
- It recognises a vanished WAL file: an `IOError` whose message matches
`/<digits>.log: No such file or directory`. The message is matched because
rocksdbjni's `Status.SubCode` has no `PathNotFound` (the subcode reads
`Undefined`).
- On that error it retries once without closing the DB. The new listing no
longer contains the file.
- If the retry fails the same way, it throws `RocksDatabaseException`
without closing the DB. The OM stays up and Recon retries on its next cycle.
- Every other `RocksDBException` still goes through `closeOnError`, as
before. That includes `Corruption`, `IOError` with EIO or ENOSPC, and ENOENT on
SST, `MANIFEST` or `CURRENT` files.
If the vanished file held updates Recon still needed, the retry lists only
later WALs. The existing check in `RDBStore.getUpdatesSince` then throws
`SequenceNumberNotFoundException`, and Recon falls back to a full snapshot. The
race itself is deliberately not mapped to `SequenceNumberNotFoundException`,
because that would force a full OM DB snapshot even when the data is still
available.
Note: `No such file or directory` is the OS's ENOENT text from `strerror`,
which is stable under the C/English locale used in the containers.
## What is the link to the Apache JIRA
https://issues.apache.org/jira/browse/HDDS-8271
## How was this patch tested?
- Added `TestRocksDatabaseGetUpdatesSince`: 27 parameterized cases on a real
`RocksDatabase`, with the RocksDB call stubbed to throw.
- **Vanished WAL** (archived WAL during listing, live WAL during listing,
archived WAL on open): each is classified as a vanished WAL; on a single
failure the call is retried once and the caller gets the iterator; on a
repeated failure the caller gets `RocksDatabaseException`; in all cases the DB
stays open.
- **Other errors** (EIO, ENOSPC, missing `.sst`, missing `MANIFEST`,
permission denied, and `Corruption` with each vanished-WAL message): none are
classified as a vanished WAL; the DB is closed with no retry, as before.
- A successful call is not retried, and an exception with no status is not
classified.
- Existing `TestRDBStore` and `TestRDBStoreIteratorWithDBClose` pass.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: [email protected]
For queries about this service, please contact Infrastructure at:
[email protected]
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
