xiangfu0 opened a new pull request, #19287: URL: https://github.com/apache/pinot/pull/19287
## Summary Fix FULL-upsert vector queries so obsolete physical versions cannot consume the per-segment top-K candidate budget before Pinot applies the query-scoped upsert snapshot. - Treat the upsert snapshot as a mandatory candidate-generation scope while retaining the final outer bitmap AND as defense in depth. - Keep required upsert/publication scopes separate from optimizer-selected metadata filters. - Use filtered ANN when the reader supports it; otherwise scan exact distances only over allowed document IDs and fail clearly when no safe path exists. - Apply the same allowed-document contract to no-index exact search, threshold search, and vector radius predicates. - Make mutable HNSW filter-aware with explicit Pinot document IDs, same-generation NRT lookup/filtering, immutable async inputs, and a captured publication boundary. - Preserve adaptive metadata behavior for ordinary non-upsert queries and avoid vector bitmap allocation for non-vector plans. - Add explain attributes for candidate-filter cardinality, execution mode, skipped generation, and fallback reason. ## Root cause `FilterPlanNode` constructed and executed `VECTOR_SIMILARITY` before adding `SegmentContext.getDocIdsSnapshot()` as an outer AND. Obsolete versions could therefore occupy ANN top-K slots and be removed only afterward, producing fewer than K rows or omitting nearer current rows. ## Validation All commands ran with JDK 25 and `GITHUB_ACTIONS=true`. - Core vector/filter suite: 101 tests passed. - Mutable and immutable HNSW suite: 16 tests passed. - `VectorUpsertTableTest`: 2 query-engine invocations passed, covering both a single consuming segment and sealed-plus-consuming segments. - Fresh affected-reactor `clean test-compile` with `-Xlint:all` passed. - Spotless, Checkstyle, license format/check, and `git diff --check` passed. The integration test verifies six physical records/four current records, exact K and entity membership, scalar-distance equivalence, both query engines, two replicas, and a `skipUpsert=true` control proving obsolete rows are physically nearest. ## Performance and compatibility ANN approximation semantics are unchanged. Readers that cannot honor the mandatory bitmap use a correctness-first exact scan whose cost is proportional to the allowed-document count times vector dimension. Mutable HNSW currently opens an NRT reader per search; reusable reader management can be optimized separately. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected] --------------------------------------------------------------------- To unsubscribe, e-mail: [email protected] For additional commands, e-mail: [email protected]
