raghavyadav01 opened a new pull request, #19901: URL: https://github.com/apache/pinot/pull/19901
A partition-hinted multi-stage query can silently drop rows of recently added segments. `SegmentPartitionMetadataManager` leaves a newly created segment out of its partition's info until the segment is online on every server that already holds the partition. That is meant for a segment whose replicas are still loading, so a half-online segment does not shrink the partition's fully replicated servers. The same check also excludes a segment that is already online on every server it is assigned to, just on different servers than the rest of its partition. That happens on any partitioned table without partition-aware instance assignment, where segments of one partition are spread across servers. Such a segment never covers the partition's servers, so it stays out for the whole new-segment window. A leaf assigned by partition (`tableOptions(partition_key=..., partition_size=...)`) then dispatches the partition without it, and its rows are skipped with no error. Once the segment stops being new, it is counted and the same query fails with `Failed to find enabled fully replicated server`. So the result depends on how long ago the segment was added. Measured on a 3-server cluster, two tables bucketed 4 ways on the join key, 2 segments per partition spread by the default assignment, queried shortly after the segments were added: | | colocated join (`is_colocated_by_join_keys` + `tableOptions`) | shuffled join | |---|---|---| | before | **23 rows**, no error | 60 rows | | after | `Failed to find enabled fully replicated server for table: ..., partition: 0` | 60 rows | With partition-aware instance assignment in place, the hinted join returns 60 rows both before and after this change. **Fix:** record how many replicas the ideal state assigns each segment (in ONLINE or CONSUMING state), and defer a new segment only while fewer of them are online. A new segment that is online on all of its assigned replicas is counted like any other segment, so a partition that no single server holds whole is reported immediately rather than hidden for the length of the new-segment window. A segment the ideal state does not list keeps the current behaviour. **Behaviour change:** on a partitioned table without partition-aware placement, a partition-hinted query now fails with the existing "no fully replicated server" error as soon as a new segment comes online elsewhere, instead of returning partial results for up to the new-segment expiration and failing afterwards. A segment that is still coming up is deferred exactly as before. **Testing:** `testFullyOnlineNewSegmentOnOtherServersIsNotDeferred` covers both sides (a segment still coming up stays deferred; a fully online one on other servers is counted and the partition reports no fully replicated server) and fails without the change. `pinot-broker` routing tests (269) and `WorkerManagerTest` / `ColocationGroupAnalyzerTest` (97) pass. 🤖 Generated with [Claude Code](https://claude.com/claude-code) -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected] --------------------------------------------------------------------- To unsubscribe, e-mail: [email protected] For additional commands, e-mail: [email protected]
