Siyao Meng created HDDS-15860:
---------------------------------
Summary: Snapshot defrag fails when directly ingesting a single
live RocksDB SST
Key: HDDS-15860
URL: https://issues.apache.org/jira/browse/HDDS-15860
Project: Apache Ozone
Issue Type: Bug
Reporter: Siyao Meng
Assignee: Siyao Meng
h2. Problem
Incremental snapshot defragmentation computes RocksDB SST files that differ
between two snapshot versions and applies the resulting changes to a temporary
checkpoint database.
When a table has exactly one candidate SST and the snapshot version is greater
than zero, {{SnapshotDefragService}} bypasses the normal spill-and-rewrite path
and passes the live SST directly to {{Table.loadFromFile}}.
The candidate is a live RocksDB SST selected at file granularity, not an
external SST containing an exact logical bucket delta. This causes two observed
failure modes on current master with RocksDB 10.10.1.1:
* A DB-generated flush SST is rejected by external-file ingestion with
{{External file version not found}}.
* An SST previously ingested into a live database can be accepted, but it
installs the value stored in that live SST instead of the value from the
current snapshot.
The direct path also bypasses bucket-prefix filtering, current-versus-previous
snapshot comparison, and tombstone generation.
h2. Trigger condition
* The snapshot version is greater than zero.
* Exactly one candidate SST is returned for an incremental table.
Once triggered, snapshot defragmentation cannot correctly advance the affected
snapshot to its next defragmented version.
h2. Proposed fix
Remove the single-file direct-ingestion shortcut. All candidate SST sets,
including singleton sets, should pass through {{spillTableDiffIntoSstFile}}.
This existing path:
* Filters keys by the target bucket prefix.
* Compares values in the current and previous snapshots.
* Emits only actual updates and insertions.
* Emits tombstones for deletions.
* Creates a fresh external SST suitable for RocksDB ingestion.
h2. Test coverage
* Update the existing incremental-defragmentation unit test to expect rewriting
for every snapshot version and candidate-file count.
* Add a real RocksDB regression test covering both DB-generated live SSTs and
SSTs previously ingested into a live database.
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
