raghavyadav01 opened a new pull request, #20000: URL: https://github.com/apache/pinot/pull/20000
## Problem On a multi-value column, `arrayContainsString(col, 'v')` and `arrayContainsInt(col, 5)` are planned as a scalar-function expression filter on every segment. The equivalent `col = 'v'` (an MV `=` matches when any element matches) uses the inverted index and lets segments prune to empty through the dictionary, bloom filter or min/max value. On a table where the value is rare, the difference is large: the `=` form prunes almost every segment and scans nothing, while `arrayContains` evaluates the function on every document of every segment. ## Change New `ArrayContainsFilterOptimizer`, run right after `FlattenAndOrFilterOptimizer`, rewrites `arrayContainsInt/String(col, literal)` (bare or `= true`) into `col = literal` when: - the schema is available and `col` is a plain identifier of an MV column whose type matches the function (INT for `arrayContainsInt`, STRING for `arrayContainsString`); - the literal converts to that type with the same `PinotDataType` conversion the scalar path uses (`5.0` → `5`, `' 5'` → `5`, `5` → `'5'`), so the rewritten filter returns the same rows. If conversion fails (e.g. `arrayContainsInt(col, 'foo')`), the function is kept. It only walks AND/OR. It never rewrites under `NOT`, `= false` or `<> true`, because `!=` on an MV column means "no element equals", not the negation of "contains". The output is a normal EQUALS, so `MergeEqInFilterOptimizer` can merge an OR of them into IN (any element in the set, same semantics). The multi-stage leaf stage runs the same `QueryOptimizer`, so it gets the rewrite as well. Note: a table-level expression override keyed on the `arrayContains` expression would no longer match after the rewrite (documented in the class Javadoc). ## Tests - `QueryOptimizerTest.testArrayContainsFilter`: rewrites (bare, `= true`, snake_case, inside AND, merged into IN), literal conversion cases, and every shape that must stay unchanged (negations, column or function argument, missing column, SV column, type mismatch, unconvertible literal). - `ArrayContainsMultiValueQueriesTest`: on an MV INT column with an inverted index, the rewritten query returns the same count as the scalar evaluation and as `col = 225`, with `numEntriesScannedInFilter` = 0 (400,000 on master); also literal conversion, a missing value, `NOT arrayContains`, OR, same-column AND and nested AND/OR/NOT, each compared with the scalar result. `*Optimizer*`, `ArrayContainsMultiValueQueriesTest`, `InnerSegment*MultiValue*` and `InterSegment*MultiValue*` pass (117 tests). 🤖 Generated with [Claude Code](https://claude.com/claude-code) -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected] --------------------------------------------------------------------- To unsubscribe, e-mail: [email protected] For additional commands, e-mail: [email protected]
