Siyao Meng created HDDS-17011:
---------------------------------
Summary: Datanode automatic small SST file compaction leaks a
native CompactRangeOptions on every range compaction
Key: HDDS-17011
URL: https://issues.apache.org/jira/browse/HDDS-17011
Project: Apache Ozone
Issue Type: Bug
Reporter: Siyao Meng
Attachments:
CR-5-close-compact-range-options-in-auto-compaction.patch,
TestBugCR5CompactRangeOptionsLeak.java
h3. Mechanism
{{DatanodeStoreSchemaThreeImpl.compactionIfNeeded}} creates a
{{ManagedCompactRangeOptions}} for each column family level that has more small
SST files than the threshold and passes it to {{RocksDatabase.compactRange}}.
Neither closes it. RocksJava has no finalizer or cleaner for
{{CompactRangeOptions}} (checked in rocksdbjni 10.10.1.1), so its native memory
is never freed. The datanode leak detector reports it as
"ManagedCompactRangeOptions is not closed properly" once the Java object is
collected.
h3. Trigger
{{hdds.datanode.rocksdb.auto-compaction-small-sst-file}} is enabled (the
default) and a column family level of a per-disk DB has more than
{{hdds.datanode.rocksdb.auto-compaction-small-sst-file-num-threshold}} (default
512) SST files below the size threshold (default 1MB). Each scheduled
{{HddsVolume.compactDb}} run that compacts such a level leaks one object per
level.
h3. Impact
A small native leak for the life of the datanode process, plus leak detector
warnings. A {{CompactRangeOptions}} is a small fixed size native struct; at
most one leaks per column family level over the threshold per scheduled run,
and the run happens every 120 minutes per volume by default
({{hdds.datanode.rocksdb.auto-compaction-small-sst-file.interval.minutes}}), so
growth is slow. The byte count was not measured. No functional impact.
h3. Reproduction
PASS, unmodified source. [^TestBugCR5CompactRangeOptionsLeak.java]
({{hadoop-hdds/container-service}}) initializes a real schema V3 {{HddsVolume}}
with the file count threshold set to 1, writes and flushes blocks in three
rounds, calls {{HddsVolume.compactDb}} after each, and then runs GC until the
leak detector has reported one leaked {{ManagedCompactRangeOptions}} per
compaction (4 of 4). Each report's allocation stack points at
{{DatanodeStoreSchemaThreeImpl.compactionIfNeeded}} called from
{{HddsVolume.compactDb}}. A passing test means the defect is present.
h3. Patch
[^CR-5-close-compact-range-options-in-auto-compaction.patch], against
cce1ca28944886218132f01c0e271c1fd0700078. It also applies to master at
2536fe9cbc3.
The options are allocated in try with resources around the existing compaction
call, so they are closed after each compaction and also when it throws. Covered
by {{testAutoCompactionClosesCompactRangeOptions}} in the existing
{{TestKeyValueContainer}}, which does not depend on GC: it captures the options
passed to {{compactRange}} through spies on a real store and asserts that none
still owns its native handle after {{compactionIfNeeded}}. Without the change
it fails; with it {{TestKeyValueContainer}}, {{TestHddsVolume}} and
{{TestDatanodeStoreCache}} (233 tests) pass and checkstyle is clean. With the
patch the reproduction fails because no leak is reported.
The same unclosed allocation exists in {{RocksDBManualCompaction}} ({{ozone
repair ldb compact}}), which runs in a short lived CLI process; it could be
fixed in a follow up.
Found by code review of the datanode volume and per-disk RocksDB lifecycle
paths, as part of the TLA+ verification effort under HDDS-15926, on commit
cce1ca28944886218132f01c0e271c1fd0700078. Checked against HDDS issues and
apache/ozone pull requests for duplicates before filing. The attached patch is
a proposal for review. Generated with Specula (Claude Opus 5.5).
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
