KKcorps opened a new pull request, #19525:
URL: https://github.com/apache/pinot/pull/19525
Stacked on #19515. Only the last commit is new. The diff view includes
#19515 until that PR merges.
## TL;DR
Two replicas of an upsert partition can hold the same segments, the same
valid-doc counts, and still
disagree on which row is current or on its comparison value. This PR adds a
running XOR digest of the
live primary-key map to the snapshot report from #19515. Two replicas at the
same stream boundary hold
the same digest exactly when they hold the same set of `(primary key,
comparison value)` entries. It is
opt-in behind the existing `enableSnapshotMetadata` flag and costs one hash
and two XORs per map write.
## The problem
The saved-bitmap fingerprints in #19515 compare bytes on disk. They need
every segment to have a saved
file, they say nothing about comparison values, and a mismatch still needs a
logical bitmap compare to
confirm. The count-based consensus checks miss the same-count-different-rows
shape entirely.
```mermaid
flowchart LR
subgraph Before["❌ Today"]
A[replica A: key k at cv 120] --> C[equal counts, equal bitmaps]
B[replica B: key k at cv 100] --> C
C --> D[no signal]
end
subgraph After["✅ With this PR"]
E[replica A: XOR includes h k,120] --> G[digests differ at the same
boundary]
F[replica B: XOR includes h k,100] --> G
G --> H[bucket names 1/256th of the key space]
end
```
## The approach
1. Each partition keeps 256 `long` buckets. Bucket = top byte of
`hash64(storedKey)`.
2. Every live entry that is not a tombstone contributes `hash64(storedKey,
comparisonValue)` to its bucket.
An update XORs the old contribution out and the new one in. A removal
XORs out. A delete record XORs
the old contribution out and adds nothing.
3. Segment and docId are not hashed. A commit swap, reload or refresh that
lands the same rows leaves the
digest unchanged.
4. Segment-level operations on Helix threads (add, preload, replace, remove)
run inside a seqlock window.
The snapshot capture freezes the buckets at start and end and publishes
them only if both reads were
outside any window and nothing changed in between.
5. The report gains a `keyDigest` block: algorithm, total, base64 buckets,
entry count, stable flag.
## Key components
| Class / file | Role |
|---|---|
| `UpsertKeyDigest` | 256 XOR buckets, entry counter, seqlock, typed hashing
of keys and comparison values, freeze |
| `UpsertSnapshotMetadata.KeyDigest` | Report block, built from the start
and end marks of one capture |
| `UpsertSnapshotDiagnostics` | Freezes the digest at capture begin and
finish |
| `BasePartitionUpsertMetadataManager` | Owns the digest, opens the seqlock
window around segment-level operations |
| `ConcurrentMapPartitionUpsertMetadataManager` | Calls the digest at every
map write and removal; `RecordLocation` carries the tombstone flag |
| `docs/upsert-snapshot-metadata.md` | New "Key digest" section |
## Flow
### A consuming record
```mermaid
sequenceDiagram
participant C as Consumer thread
participant M as ConcurrentMapPartitionUpsertMetadataManager
participant K as ConcurrentHashMap
participant D as UpsertKeyDigest
C->>M: addRecord(segment, recordInfo)
M->>K: compute(storedKey)
alt new key
K-->>M: no entry
M->>D: add(storedKey, cv, isDelete)
else newer comparison value
K-->>M: current location
M->>D: update(storedKey, oldCv, oldIsDelete, cv, isDelete)
else out of order
K-->>M: current location
Note over M,D: no map change, no digest change
end
```
### A snapshot capture
```mermaid
sequenceDiagram
participant C as Consumer thread
participant B as BasePartitionUpsertMetadataManager
participant S as UpsertSnapshotDiagnostics
participant D as UpsertKeyDigest
participant H as Helix thread
C->>B: takeSnapshot(consumingSegment, startOffset)
B->>S: begin
S->>D: freeze (epoch, depth, copy, depth, epoch)
B->>B: write bitmap files
opt segment add / replace / remove
H->>D: beginUnstable
H->>D: add / update / remove per key
H->>D: endUnstable
end
B->>S: finish
S->>D: freeze
alt both marks stable and equal
S-->>B: keyDigest with total and buckets
else
S-->>B: keyDigest with stable=false, total and buckets null
end
```
## What the digest ignores on purpose
- **Tombstones.** A key whose latest record is a delete contributes nothing.
The deletedKeysTTL sweep only
removes tombstones, so it never touches the digest. A swallowed delete
still shows: one replica removed
the old contribution, the other still holds it.
- **Segment and docId.** Replicas assign docIds by build order and segment
ids by load order. Hashing
them would flag every reload.
- **The consuming segment's timing.** The digest includes consuming-segment
keys, so compare only reports
with the same `consumingSegmentName` and `startOffset`, taken at the
commit boundary.
## Sweeps
The on-heap manager applies the metadataTTL sweep to the digest. Its sweep
runs at the commit boundary
on every replica, and its map is rebuilt from segments on restart, so the
digest must describe the live
map. One capture after a restart on a metadataTTL table can differ until the
first sweep runs. A manager
whose sweep runs on a timer must keep the sweep off the digest and persist
the buckets instead. That is
the RocksDB companion, not this PR.
## Configuration
No new keys. The digest exists only when
`upsertConfig.metadataManagerConfigs.enableSnapshotMetadata`
is `true`, same as the rest of the report. When it is off, every hook is one
null check.
## Performance considerations
- Per map write: one XXH64 over the stored key bytes, one over the
comparison value, one SplitMix64
finalizer, one atomic XOR on a `long`, one `LongAdder` bump. For
`hashFunction=NONE` the key hash calls
`PrimaryKey.asBytes()`, which allocates once per write. Hashed key
functions use the stored 16 bytes.
- Per capture: two copies of 256 longs and one 2 KB base64 string in the
report and sidecar.
- State: 2 KB of buckets per partition. Nothing per key.
- `RecordLocation` does not grow. The tombstone flag rides in the sign bit
of the docId.
## Compatibility notes
- `UpsertSnapshotMetadata.FORMAT_VERSION` moves from 3 to 4. Version 3
sidecars read as unavailable.
Version 3 never shipped.
- `RecordLocation` gains a four-argument constructor and `isDeleteRecord()`.
The three-argument
constructor stays and means "not a delete".
- `UpsertSnapshotDiagnostics` takes the digest in its package-private
constructor.
- No change to `PartitionUpsertMetadataManager` or
`TableUpsertMetadataManager`.
## Validation
- `UpsertKeyDigestTest`: order independence, add/remove/update algebra,
tombstones contribute zero,
distinct hashes across comparison-value types, seqlock stability, report
gating.
- `ConcurrentMapPartitionUpsertMetadataManagerTest`: two replicas fed the
same records agree; a missed
update and a swallowed delete each break agreement until applied; an
out-of-order record and a segment
replace leave the digest unchanged; segment removal removes the owned key
on both; a fresh manager fed
only the surviving rows lands on the same digest; the metadataTTL sweep
removes swept keys from the
digest.
- `BasePartitionUpsertMetadataManagerTest`: the report carries a stable zero
digest, and a segment
operation in flight publishes `stable=false` with null total and buckets.
- `UpsertSnapshotMetadataStoreTest`: round trip, version 3 rejected.
- `TablesResourceTest` passes with the new report shape (19 tests).
- All 120 tests under `pinot-segment-local` upsert pass. Spotless and
Checkstyle pass.
## Release notes
Adds an opt-in per-partition XOR digest of upsert primary keys and
comparison values to the snapshot
metadata report.
Related: #19515, #19499. Documentation: `docs/upsert-snapshot-metadata.md`.
🤖 Generated with [Claude Code](https://claude.com/claude-code)
https://claude.ai/code/session_016sih7cgUT7mAFHtTYQq1mk
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: [email protected]
For queries about this service, please contact Infrastructure at:
[email protected]
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
