deepthi912 opened a new pull request, #18853:
URL: https://github.com/apache/pinot/pull/18853
## Summary
- Adds an opt-in `includeBitmaps=true` query parameter to `POST
/tables/{tableNameWithType}/validDocIdsMetadata`.
- When set, each per-segment response entry now also carries the serialized
RoaringBitmap of the segment's validDocIds (new `bitmap` field on
`ValidDocIdsMetadataInfo`).
- Default behavior is unchanged: bitmaps are **omitted** unless the caller
asks for them, so existing consumers see no payload-size increase.
## Why
Today, callers that want cross-replica bitmap consensus (e.g. upsert
compaction task generators) have to do `N segments × M replicas` HTTP fetches
via the per-segment `validDocIdsBitmap` endpoint. With this opt-in flag, a
caller can fetch counts + bitmaps for a list of segments in a **single batched
request per server**, then do in-memory bitmap comparison.
Used together with the upsert compaction task changes in #17696, this lets
the task generator enforce bitmap-level `EQUAL` consensus before scheduling —
matching what the executor already enforces — without the per-segment HTTP cost.
## Wire compatibility
- `ValidDocIdsMetadataInfo` gains a nullable `bitmap` field.
`@JsonInclude(NON_NULL)` keeps it out of the serialized JSON when omitted.
`@JsonIgnoreProperties(ignoreUnknown = true)` on the class means older clients
ignore the field if a newer server emits it.
- The two-constructor pattern preserves existing callers (`new
ValidDocIdsMetadataInfo(...)` without bitmap still works; tests are unaffected).
- A newer client calling an older server simply gets `bitmap = null` (server
doesn't recognise the flag, doesn't emit the field) — caller must handle this
and fall back to the per-segment `validDocIdsBitmap` endpoint when needed.
## Test plan
- [x] `TablesResourceTest` (existing 16 tests still pass)
- [x] New `testValidDocIdsMetadataPostWithIncludeBitmaps` — verifies the
bitmap field is present, deserializes correctly, and matches `totalValidDocs`
cardinality.
- [x] Existing `testValidDocIdsMetadataPost` updated to assert the bitmap
field is **absent** by default.
- [x] `./mvnw -pl pinot-server,pinot-controller,pinot-common
checkstyle:check` clean.
🤖 Generated with [Claude Code](https://claude.com/claude-code)
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: [email protected]
For queries about this service, please contact Infrastructure at:
[email protected]
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
