[
https://issues.apache.org/jira/browse/HDDS-15860?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Siyao Meng updated HDDS-15860:
------------------------------
Description:
h2. Problem
Incremental snapshot defragmentation computes RocksDB SST files that differ
between two snapshot versions and applies the resulting changes to a temporary
checkpoint database.
When a table has exactly one candidate SST and the snapshot version is greater
than zero, {{SnapshotDefragService}} bypasses the normal spill-and-rewrite path
and passes the live SST directly to {{Table.loadFromFile}}.
The candidate is a live RocksDB SST selected at file granularity, not an
external SST containing an exact logical bucket delta. The observed behavior
differs by RocksDB version:
* With RocksDB 7.7.3 and 10.10.1.1, a DB-generated flush SST is rejected by
external-file ingestion with {{External file version not found}}. DB-generated
SSTs lack the {{rocksdb.external_sst_file.version}} and
{{rocksdb.external_sst_file.global_seqno}} properties, and their internal
sequence numbers are not valid external-file input.
* On current master with RocksDB 10.10.1.1, an SST previously ingested into a
live database can be accepted, but it installs the value stored in that live
SST instead of the logical value from the current snapshot. With RocksDB 7.7.3,
reusing such a file can instead fail global-sequence-number validation.
The direct path also bypasses bucket-prefix filtering, current-versus-previous
snapshot comparison, and tombstone generation.
h2. Trigger condition
* The snapshot version is greater than zero.
* Exactly one candidate SST is returned for an incremental table.
Once triggered, snapshot defragmentation cannot correctly advance the affected
snapshot to its next defragmented version.
h2. Proposed fix
Remove the single-file direct-ingestion shortcut. All candidate SST sets,
including singleton sets, should pass through {{spillTableDiffIntoSstFile}}.
This existing path:
* Filters keys by the target bucket prefix.
* Compares values in the current and previous snapshots.
* Emits only actual updates and insertions.
* Emits tombstones for deletions.
* Creates a fresh external SST suitable for RocksDB ingestion.
h2. Why existing tests did not catch this
The incremental-defragmentation control flow was covered, but the RocksDB
ingestion boundary was mocked:
* The original background-defrag change did not exercise real incremental SST
ingestion.
* {{testPerformIncrementalDefragmentation}} creates empty placeholder {{.sst}}
files and mocks {{SstFileSetReader}}, {{RDBSstFileWriter}}, and
{{Table.loadFromFile}}. For snapshot versions greater than zero, it explicitly
expected the single-file table to be passed directly to the mocked loader, so
the test codified the unsafe shortcut without invoking RocksDB JNI.
* Lower-level {{Table.loadFromFile}} tests first create a valid external SST
with {{dumpToFileWithPrefix}}. They verify external-SST ingestion, not
ingestion of a live DB-generated SST.
* The snapshot-defrag admin integration test verifies that the CLI can trigger
the service on OM nodes. It does not create snapshot data or verify incremental
SST ingestion and the resulting checkpoint contents.
As a result, no existing test combined a nonzero snapshot version, exactly one
live candidate SST, and a real {{Table.loadFromFile}} call.
h2. Test coverage
* Update the existing incremental-defragmentation unit test to expect rewriting
for every snapshot version and candidate-file count.
* Add a real RocksDB regression test covering both DB-generated live SSTs and
SSTs previously ingested into a live database.
was:
h2. Problem
Incremental snapshot defragmentation computes RocksDB SST files that differ
between two snapshot versions and applies the resulting changes to a temporary
checkpoint database.
When a table has exactly one candidate SST and the snapshot version is greater
than zero, {{SnapshotDefragService}} bypasses the normal spill-and-rewrite path
and passes the live SST directly to {{{}Table.loadFromFile{}}}.
The candidate is a live RocksDB SST selected at file granularity, not an
external SST containing an exact logical bucket delta. This causes two observed
failure modes on current master with RocksDB 10.10.1.1 (or with previous 7.7.3):
* A DB-generated flush SST is rejected by external-file ingestion with
{{External file version not found}} (because DB-generated SST lacks
rocksdb.external_sst_file.versionrocksdb.external_sst_file.version and
rocksdb.external_sst_file.global_seqno properties)
* An SST previously ingested into a live database can be accepted, but it
installs the value stored in that live SST instead of the value from the
current snapshot.
The direct path also bypasses bucket-prefix filtering, current-versus-previous
snapshot comparison, and tombstone generation.
h2. Trigger condition
* The snapshot version is greater than zero.
* Exactly one candidate SST is returned for an incremental table.
Once triggered, snapshot defragmentation cannot correctly advance the affected
snapshot to its next defragmented version.
h2. Proposed fix
Remove the single-file direct-ingestion shortcut. All candidate SST sets,
including singleton sets, should pass through
{{{}spillTableDiffIntoSstFile{}}}. This existing path:
* Filters keys by the target bucket prefix.
* Compares values in the current and previous snapshots.
* Emits only actual updates and insertions.
* Emits tombstones for deletions.
* Creates a fresh external SST suitable for RocksDB ingestion.
h2. Test coverage
* Update the existing incremental-defragmentation unit test to expect
rewriting for every snapshot version and candidate-file count.
* Add a real RocksDB regression test covering both DB-generated live SSTs and
SSTs previously ingested into a live database.
> Snapshot defrag fails when directly ingesting a single live RocksDB SST
> -----------------------------------------------------------------------
>
> Key: HDDS-15860
> URL: https://issues.apache.org/jira/browse/HDDS-15860
> Project: Apache Ozone
> Issue Type: Bug
> Reporter: Siyao Meng
> Assignee: Siyao Meng
> Priority: Major
>
> h2. Problem
> Incremental snapshot defragmentation computes RocksDB SST files that differ
> between two snapshot versions and applies the resulting changes to a
> temporary checkpoint database.
> When a table has exactly one candidate SST and the snapshot version is
> greater than zero, {{SnapshotDefragService}} bypasses the normal
> spill-and-rewrite path and passes the live SST directly to
> {{Table.loadFromFile}}.
> The candidate is a live RocksDB SST selected at file granularity, not an
> external SST containing an exact logical bucket delta. The observed behavior
> differs by RocksDB version:
> * With RocksDB 7.7.3 and 10.10.1.1, a DB-generated flush SST is rejected by
> external-file ingestion with {{External file version not found}}.
> DB-generated SSTs lack the {{rocksdb.external_sst_file.version}} and
> {{rocksdb.external_sst_file.global_seqno}} properties, and their internal
> sequence numbers are not valid external-file input.
> * On current master with RocksDB 10.10.1.1, an SST previously ingested into a
> live database can be accepted, but it installs the value stored in that live
> SST instead of the logical value from the current snapshot. With RocksDB
> 7.7.3, reusing such a file can instead fail global-sequence-number validation.
> The direct path also bypasses bucket-prefix filtering,
> current-versus-previous snapshot comparison, and tombstone generation.
> h2. Trigger condition
> * The snapshot version is greater than zero.
> * Exactly one candidate SST is returned for an incremental table.
> Once triggered, snapshot defragmentation cannot correctly advance the
> affected snapshot to its next defragmented version.
> h2. Proposed fix
> Remove the single-file direct-ingestion shortcut. All candidate SST sets,
> including singleton sets, should pass through {{spillTableDiffIntoSstFile}}.
> This existing path:
> * Filters keys by the target bucket prefix.
> * Compares values in the current and previous snapshots.
> * Emits only actual updates and insertions.
> * Emits tombstones for deletions.
> * Creates a fresh external SST suitable for RocksDB ingestion.
> h2. Why existing tests did not catch this
> The incremental-defragmentation control flow was covered, but the RocksDB
> ingestion boundary was mocked:
> * The original background-defrag change did not exercise real incremental SST
> ingestion.
> * {{testPerformIncrementalDefragmentation}} creates empty placeholder
> {{.sst}} files and mocks {{SstFileSetReader}}, {{RDBSstFileWriter}}, and
> {{Table.loadFromFile}}. For snapshot versions greater than zero, it
> explicitly expected the single-file table to be passed directly to the mocked
> loader, so the test codified the unsafe shortcut without invoking RocksDB JNI.
> * Lower-level {{Table.loadFromFile}} tests first create a valid external SST
> with {{dumpToFileWithPrefix}}. They verify external-SST ingestion, not
> ingestion of a live DB-generated SST.
> * The snapshot-defrag admin integration test verifies that the CLI can
> trigger the service on OM nodes. It does not create snapshot data or verify
> incremental SST ingestion and the resulting checkpoint contents.
> As a result, no existing test combined a nonzero snapshot version, exactly
> one live candidate SST, and a real {{Table.loadFromFile}} call.
> h2. Test coverage
> * Update the existing incremental-defragmentation unit test to expect
> rewriting for every snapshot version and candidate-file count.
> * Add a real RocksDB regression test covering both DB-generated live SSTs and
> SSTs previously ingested into a live database.
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
