Siyao Meng created HDDS-17746:
---------------------------------
Summary: Optimized snapshot diff merge join compares objectIds as
signed longs while the lists are in unsigned byte order
Key: HDDS-17746
URL: https://issues.apache.org/jira/browse/HDDS-17746
Project: Apache Ozone
Issue Type: Bug
Reporter: Siyao Meng
Attachments: CR-2-compare-object-ids-unsigned-in-merge-join.patch,
TestBugCR2SignedObjectIdMergeJoin.java
h3. Mechanism
{{SnapDiffJobStore}} keys the new and old list column families by the objectId
as 8 big endian bytes, and the default bytewise comparator returns them in
unsigned order. {{MergeJoinSnapDiffWriter.classifyMergeJoin}} advances through
the two lists with signed {{<}} and {{>}}. The orders differ only when the
lists hold objectIds of both signs. Current OMs always use epoch 2, so new
objectIds are negative as signed longs. Older objectIds can be positive:
* Ozone 1.0.0 and earlier built objectIds as the transaction index shifted left
by 8.
* Ozone 1.1.0 to 1.4.1 used epoch 1 when {{ozone.om.ratis.enable}} was false
(the default was true).
ObjectIds are never rewritten, so a bucket holding keys written on 1.0.0
(LEGACY layout after the upgrade), or keys written by a non Ratis OM on 1.1.0
to 1.4.1, can hold both after an upgrade. The merge then loses step: matching
entries fall through to the CREATE and DELETE branches separately.
h3. Trigger
A bucket that holds a positive (legacy) objectId and current objectIds,
{{ozone.om.snapshot.diff.optimized.enabled=true}}. Take snapshot s1, delete the
legacy key (with several, the one with the highest objectId; a lower one is
matched correctly) and optionally modify a current key, take snapshot s2, and
request the diff s1 to s2, ordered or unordered, for an OBS, LEGACY or FSO
bucket. The merge join was added by HDDS-15391 and is not in any release yet
(the optimized diff is off by default).
h3. Impact
In a full diff, once the legacy key with the highest objectId is deleted, every
current key in the bucket is reported: unchanged ones as DELETE, modified ones
as DELETE plus CREATE. A consumer that replays the report deletes them all. In
a DAG diff the same happens to every changed current key. The report is a
persisted DONE report. Once the new list's head is a current (negative) id
while the old list's head is still the deleted positive id, every remaining new
entry is taken as a CREATE (present markers emit nothing) and every remaining
old entry as a DELETE. A unit run of the full diff shape (legacy1 kept, legacy2
deleted, four unchanged and one modified current key) gave {{[- ./legacy2, -
./cur1, - ./cur2, - ./cur3, - ./cur4, - ./cur5, + ./cur5]}} on the unfixed code
and {{[- ./legacy2, M ./cur5]}} with the patch; deleting legacy1 instead gave
{{[- ./legacy1, M ./cur5]}} on both.
h3. Reproduction
[^TestBugCR2SignedObjectIdMergeJoin.java] ({{hadoop-ozone/ozone-manager}})
opens its own RocksDB in a temporary directory, writes the list rows the
readers would write for one legacy positive id and two current ids, and runs
the real {{MergeJoinSnapDiffWriter.writeReport}}, for OBS and FSO. The mixed
sign case fails on the unfixed code with {{Expecting actual: [- ./older, -
./same, - ./modified, + ./modified] to contain exactly in any order: [-
./older, M ./modified]}}; a control with current ids only passes. With the
patch all four cases pass. The ids are written directly because a current OM
cannot create a positive objectId; the upgrade path itself was not run.
h3. Patch
[^CR-2-compare-object-ids-unsigned-in-merge-join.patch], against
d3b75c75bf9ad475551202477c3a21a3f4b968b7. It also applies to master at
bbc8505e00d.
The two comparisons in {{classifyMergeJoin}} use {{Long.compareUnsigned}},
matching the column family order. For ids of one sign, which is every bucket
created on 2.x, nothing changes.
Covered by {{testClassificationWithObjectIdsOfBothSigns}} in the existing
{{TestMergeJoinSnapDiffWriter}}, for OBS and FSO. Without the change both fail
as above. With it the snapshot diff unit suites and {{TestSnapshotDiffManager}}
(193 tests) pass, and checkstyle is clean.
Found by code review of the OM optimized snapshot diff paths, as part of the
TLA+ verification effort under HDDS-15926, on commit
d3b75c75bf9ad475551202477c3a21a3f4b968b7. Checked against HDDS issues and
apache/ozone pull requests for duplicates before filing. The attached patch is
a proposal for review. Generated with Specula (Claude Opus 5.5).
--
This message was sent by Atlassian Jira
(v10.3.26#10030026)
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]