[
https://issues.apache.org/jira/browse/HDDS-17079?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Siyao Meng updated HDDS-17079:
------------------------------
Status: Patch Available (was: Open)
> Skip SST file lists when loading snapshot local data YAMLs for the snapshot
> version graph
> -----------------------------------------------------------------------------------------
>
> Key: HDDS-17079
> URL: https://issues.apache.org/jira/browse/HDDS-17079
> Project: Apache Ozone
> Issue Type: Sub-task
> Components: Ozone Manager, Snapshot
> Reporter: Siyao Meng
> Assignee: Siyao Meng
> Priority: Major
> Labels: pull-request-available
>
> h3. Problem
> To build its snapshot version graph, OmSnapshotLocalDataManager only needs
> each YAML's snapshotId, previousSnapshotId, version and the
> previousSnapshotVersion of each version. Every load still builds all
> SstFileInfo entries, which make up almost the whole file, and then throws
> them away. With many snapshots this is most of the CPU cost of OM startup,
> even after HDDS-17078 (parse each YAML once at startup).
> h3. Analysis
> Timing one 9 MB YAML from a cluster with about 19k snapshots (one version,
> about 36.7k SstFileInfo entries):
> * Full load: 0.153 s per file (59 MB/s)
> * Load without SST file lists: 0.088 s per file (103 MB/s)
> h3. Proposed fix
> Add a load mode to OmSnapshotLocalDataYaml (YamlFactory(true)) that drops
> each version's sstFiles node before construction, so none of its entries are
> built. All other fields load as before. OmSnapshotLocalDataManager uses this
> mode only in init and in the previous snapshot walk. All other reads and
> every write still use the full serializer.
> The serializer used in this mode refuses to save, so data loaded without SST
> file lists cannot overwrite a YAML.
> h3. Compatibility
> The on disk format is unchanged. New code reads YAMLs written by existing
> releases, and nothing new is written.
> One behavior change: startup still parses the whole document, so files that
> are not valid YAML are still skipped as before. But invalid values inside SST
> file entries are no longer detected at startup; they show up on the next full
> load of that snapshot.
> h3. Testing
> Added TestOmSnapshotLocalDataYaml#testLoadWithoutSstFiles. A YAML written by
> the regular serializer loads in the new mode with the same snapshot ids,
> versions, previousSnapshotVersion values, checksum and transaction info, and
> with empty SST file lists.
> Related to HDDS-17078; independent change, the speedups combine.
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
