Siyao Meng created HDDS-17078:
---------------------------------
Summary: OM startup takes hours with many snapshots because
snapshot local data YAMLs are parsed repeatedly on one thread
Key: HDDS-17078
URL: https://issues.apache.org/jira/browse/HDDS-17078
Project: Apache Ozone
Issue Type: Bug
Components: Ozone Manager, Snapshot
Reporter: Siyao Meng
Assignee: Siyao Meng
h3. Problem
On a cluster with about 19k snapshots, OM takes about 10 hours to open its RPC
port after a restart. During that time the OM main thread is fully busy (100%
of one core) in OmSnapshotLocalDataManager initialization:
{code}
HashMap.put <- SafeConstructor.processDuplicateKeys / flattenMapping
OmSnapshotLocalDataYaml$SnapshotLocalDataConstructor$ConstructSstFileInfo.construct
<- ConstructVersionMeta.construct <- ConstructSnapshotLocalData.construct
YamlSerializer.load
OmSnapshotLocalDataManager.tryLoadSnapshotLocalData
OmSnapshotLocalDataManager.addVersionNodeWithDependents
OmSnapshotLocalDataManager.init
OmSnapshotLocalDataManager.<init>
OmSnapshotManager.<init>
OzoneManager.instantiateServices
{code}
The whole delay falls between "Ozone filesystem snapshot feature is enabled"
and the next OM startup log line. The rest of startup takes seconds. This
affects 2.2.0 and current master.
h3. Analysis
OmSnapshotLocalDataManager#init lists all snapshot local data YAMLs, sorts them
by file name and loads each one. For each file, addVersionNodeWithDependents
walks the previous snapshot chain and loads every snapshot YAML not yet in the
version graph. A snapshot reached through that walk is loaded again when the
outer loop reaches its own file. Since the file names are UUIDs, the order is
effectively random with respect to the chain, so almost every YAML is parsed
twice, all on one thread.
Measurements on the affected cluster:
* 19,358 YAMLs, 164 GB in total (median 9 MB, max 17.7 MB). A median file holds
one version with about 36.7k SstFileInfo entries.
* Observed parse rate during startup: about 10 MB/s, 1.4 s per file.
* About 2 x 164 GB at 10 MB/s comes to about 9 hours, which matches the
observed startup time.
h3. Proposed fix
Parse every YAML exactly once in init, keeping only its version metadata
(SnapshotVersionsMeta), and let the dependency walk reuse that metadata instead
of reloading previous snapshots from disk. Run the parsing on a dedicated
thread pool of min(cores, 8) threads; 8 matches the YamlSerializer Yaml pool
size, so more threads would only wait. Handling of unreadable or mismatched
YAMLs is unchanged.
Expected improvement: about 2x from parsing once, times up to 8x from parallel
parsing.
h3. Testing
Extended TestOmSnapshotLocalDataManager#testInitWithExistingYamlFiles to count
YAML loads during init. It fails on the current code (3 loads for 2 files) and
passes with the fix (2 loads).
h3. Related
The YAMLs themselves are much larger than expected because each snapshot lists
tens of thousands of small SST files. That will be tracked separately.
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
