noob-se7en opened a new pull request, #19245: URL: https://github.com/apache/pinot/pull/19245
## Problem `ExpressionScanDocIdIterator` resolves match positions through a scratch docId buffer shared with the doc-id source operators it creates: every `nextBlock()` call on the source overwrites the buffer, and `processProjectionBlock` emitted matches via `matchingDocIds.add(_docIdBuffer[i])`. This is only correct while blocks are pulled and processed in lock-step — which the default `ProjectionOperator` happens to guarantee, but no contract documents. Since #11291 the projection operator is pluggable via `ProjectionOperatorUtils`; an implementation that pulls several blocks from the doc-id source ahead of consumption (e.g. to overlap remote storage reads with evaluation) leaves the buffer holding the last pulled batch while earlier batches are evaluated. Values and predicate verdicts stay correct (each block fetches values through its own docId copy), but matches get attributed to the wrong docIds — silently wrong results (phantom rows, undercounts from collisions) for any expression predicate AND-ed with an index-based filter. ## Fix - Resolve match positions through `projectionBlock.getDocIds()`, which is position-aligned with the fetched values by construction. With the default operator the block's docIds alias the scratch buffer, so behavior and performance there are unchanged. - Document the buffer-reuse contract on `DocIdSetBlock`, `BitmapDocIdSetOperator`, and `DocIdSetOperator`: block contents may be invalidated by the next `nextBlock()` call; consumers holding blocks across pulls must copy. - Regression test: a minimal pull-ahead `ProjectionOperator` registered via `ProjectionOperatorUtils`; fails before the fix with matches attributed to the last batch's docIds, passes after. Covers the plain, null-handling-enabled, and `PredicateEvaluationResult.NULL` emission paths, each under both the default and the pull-ahead operator. ## Impact - Default deployments: no behavior change — the emission reads the same array through a different reference; no added allocation or per-row cost. - Deployments plugging in look-ahead projection operators: expression-filter results under AND are now correct regardless of look-ahead depth. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected] --------------------------------------------------------------------- To unsubscribe, e-mail: [email protected] For additional commands, e-mail: [email protected]
