Sammi Chen created HDDS-7321:
--------------------------------
Summary: Support CLI command to trigger rocksDB range compaction
Key: HDDS-7321
URL: https://issues.apache.org/jira/browse/HDDS-7321
Project: Apache Ozone
Issue Type: Sub-task
Reporter: Sammi Chen
Assignee: Sammi Chen
RocksDB has auto compaction itself, which is triggered by the total level file
size.
Once the total level file size exceeds the threshold, RocksDB will schedule the
compaction in backend.
When replicating containers between datanodes, current implementation leverages
RocksDB SstFileWriter to export container meta data to individual sst files,
and leverages RocksDB ingestExternalFile to import container meta data sst
files directly into target datanode RocksDB. If the imported container meta
data keys don't overlap with other sst files(Consider Merge RocksDB design,
container ID is used as prefix of each meta data key, this is true for most of
time), the imported sst file will be kept and remain the same without
compacting with other existing sst files.
The worst case, if thousands or dozens of thousands of containers are imported
on one datanode, there would be dozens of thousands of small sst files under
one RocksDB, or across all RocksDB instances of one datanode. By default,
RocksDB has no limit of open files. Dozens of thousands of small sst files
would exhaust the process open file quota, and service stability will be
impacted.
This task aims to provide an option for user to launch range compaction
manually to enforce compact all these small sst files into merged big ones. Of
course, the compaction will have impact on user data read/write performance on
this datanode, so the suggestion is only triggering the operation when the
datnode is not busy.
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
