Devesh Kumar Singh created HDDS-15995:
-----------------------------------------
Summary: Recon SCM RocksDB closes with background IO error because
snapshot DB directory is renamed while the DB is still open
Key: HDDS-15995
URL: https://issues.apache.org/jira/browse/HDDS-15995
Project: Apache Ozone
Issue Type: Bug
Components: Ozone Recon
Affects Versions: 2.1.1, 2.2.0, 2.1.0, 2.0.0
Reporter: Devesh Kumar Singh
Assignee: Devesh Kumar Singh
*Summary*
During Recon's SCM DB snapshot swap, the downloaded snapshot directory
(scm.snapshot.db_<timestamp>) is renamed to the permanent name (recon-scm.db)
*while the RocksDB instance is still open on it*. RocksDB keeps its original
directory path internally, so a later attempt to create a new WAL/log file
fails with "No such file or directory", RocksDB sets a background IO error and
shuts itself down. All subsequent Recon SCM background work (PipelineSyncTask,
container tasks) then fails with "Rocks Database is closed".
*Observed RocksDB LOG (from Recon SCM DB folder)*
{code:java}
New memtable created with log file: #1517. Immutable memtables: 0.
[WARN] Background IO error IO error: No such file or directory: While open a
file
for appending: <reconScmDbDir>/scm.snapshot.db_1782737001797/001517.log: No
such file or directory
ErrorHandler: Set regular background error
Shutdown: canceling all background work
Shutdown complete
{code}
*How to reproduce*
# Enable snapshot sync and set ozone.recon.scm.container.threshold=0.
# Set a small memtable size (ozone.metastore.rocksdb.cf.write.buffer.size=64KB)
to make WAL rotation happen quickly.
# Create at least one container in SCM (write a key).
# Wipe Recon's SCM DB (recon-scm.db) and restart Recon so it downloads a fresh
snapshot -> triggers the rename-while-open.
# Generate continued write activity (create keys) and observe the Recon SCM
RocksDB LOG showing the background IO error on the scm.snapshot.db_<timestamp>
path, followed by "Rocks Database is closed" in Recon logs.
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
