KKcorps opened a new pull request, #19515:
URL: https://github.com/apache/pinot/pull/19515
## Summary
Replica valid-doc counts currently lack the context of the snapshot that
produced them. This adds an
opt-in, server-local count summary and a read-only API so operators and
future comparison tasks can
inspect captured counts alongside the triggering consumer and its observed
start offset.
Every version-1 summary reports `boundaryStatus: UNVERIFIED`. Equal offsets
do not certify comparable
state while predecessor reconciliation or background mutations may differ.
Boundary verification is a
separate follow-up; this PR does not add a definitive divergence detector.
## Behavior
```mermaid
sequenceDiagram
participant C as Consumer startup
participant W as Bounded metadata writer
participant A as Server API
C->>C: Existing bitmap snapshot writes
C->>W: Captured counts and startup context
Note over C: Continue without waiting for metadata publication
W->>W: Atomically publish count summary
A->>A: Read last published summary
```
- Reuses counts produced by existing bitmap serialization. Skipped and
unchanged segments do not
inherit a newer capture context.
- Returns saved counts through
`GET /tables/{tableNameWithType}/upsertSnapshotMetadata/{partitionId}`.
Missing, invalid and unsupported
sidecars return unavailable evidence. The API never takes a snapshot or
reads primary keys/bitmaps.
- Keeps counts self-contained, so no bitmap hashing or historical bitmap
retention is needed.
- Preserves existing recovery writes and no-argument snapshot overrides.
## Performance and scope
`upsertConfig.metadataManagerConfigs.enableSnapshotMetadata` defaults to
`false`. There are no
per-record changes, additional bitmap serializations, or waits for
predecessor completion. Capture
allocates bounded count entries; JSON serialization and sidecar I/O run on
one background writer.
The queue holds eight summaries, each capped at 10,000 entries. Overload
drops diagnostic publication.
Allocation and background CPU/I/O still need workload measurements before
production enablement.
This first cut retains one summary per partition and defers lifecycle
instrumentation, history,
configuration/cleanup fingerprints, membership digests and controller
scheduling.
## Validation
- 49 selected tests passed: `BasePartitionUpsertMetadataManagerTest`,
`UpsertSnapshotMetadataStoreTest`,
`RealtimeSegmentDataManagerTest`, and
`TablesResourceTest#testUpsertSnapshotMetadataApi`.
- Spotless, Checkstyle, license formatting/checks, and `git diff --check`
passed.
- The extra reactor compiler-warning check was attempted twice but is
blocked by an unchanged
`ZstandardDecompressor` reference to missing
`org.jetbrains.annotations.NotNull`. The normal build
and tests above passed; no warnings on added lines appeared before that
failure.
## Release notes
Adds optional upsert snapshot count metadata and a server API for inspecting
its captured counts and
observed startup context. Version 1 exposes diagnostic evidence without
certifying replica divergence.
Related: #19499.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: [email protected]
For queries about this service, please contact Infrastructure at:
[email protected]
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
