raghavyadav01 opened a new pull request, #20001:
URL: https://github.com/apache/pinot/pull/20001
## Problem
On the multi-stage engine, `WHERE tags = 'urgent'` or `WHERE tags IN ('vip',
'normal')` on a multi-value column fails, while the single-stage engine answers
it with "any element matches" semantics.
MV columns are ARRAY-typed in the planner, and the lenient BABEL conformance
coerces the literal to an array: `=($3, CAST('urgent'):VARCHAR ARRAY)`. The
leaf then gets `equals(tags, <array literal>)`, which is not a recognised MV
predicate, so it runs as an expression filter that reads the MV column as
single-value:
```
java.lang.UnsupportedOperationException
at
org.apache.pinot.segment.spi.index.reader.ForwardIndexReader.readDictIds(ForwardIndexReader.java:106)
at
org.apache.pinot.core.common.DataFetcher$ColumnValueReader.readStringValues(DataFetcher.java:392)
...
```
Above the leaf the same comparison fails with `ClassCastException:
[Ljava.lang.String; cannot be cast to java.lang.Comparable`.
## Change
`PinotTypeCoercion`: for `=`, `<>`, `IN` and `NOT IN` between a plain MV
column reference and a non-array value, wrap the column in `ARRAY_TO_MV` before
the normal coercion. The query then plans exactly like the already supported
`ARRAY_TO_MV(tags) = 'urgent'`, which is pushed to the leaf and uses MV
semantics (`=`/`IN`: any element matches; `<>`/`NOT IN`: no element matches,
same as the single-stage engine).
Deliberately not rewritten (they fail validation as before):
- range comparisons (`>`, `<`, ...): Calcite simplifies `NOT (mv > x)` into
`mv <= x`, which has different MV semantics than the single-stage `NOT`;
- comparisons where coercion would cast the MV column itself (e.g. `tags =
5`); casting only the literal (`codes = '5'`) is fine;
- a NULL in the IN list, rejected with a clear error.
`ArrayToMvValidationVisitor`: when such a comparison ends up somewhere it
cannot run (projection/CASE, outer join condition, a filter that cannot be
pushed to the scan), the error now says that comparing a multi-value column to
a value is only supported in a pushable WHERE filter and suggests
`arrayContainsString` / `arrayContainsInt`.
None of the shapes that now get the new error returned rows before this
change (they failed at planning or at runtime), so no working query is affected.
## Tests
- `pinot-query-runtime` `MultiValueFilters.json` (new, 23 cases): `=`, value
on the left, `IN`, a 26-value `IN`, `= 'a' AND = 'b'`, INT MV, `<>`, `NOT (=)`,
`NOT IN`, mixed-literal `codes = '5'`, below a join, LEFT JOIN ON, `COUNT(*)
FILTER (WHERE ...)`, `arrayContainsString` above the leaf, plus error cases
(ranges, `tags = 5`, NULL in IN, non-pushable positions). Results checked
against single-stage semantics; the positive cases fail on master.
- Planner: `BasicQueryPlans.json` pins the `=(ARRAY_TO_MV($n), ...)` plan;
`ValidationErrorPlan.json` pins the projection error.
- `ResourceBasedQueriesTest`, `QueryRunnerTest`, `QueryCompilationTest`,
`ResourceBasedQueryPlansTest`, `SealedInListPlanningTest`,
`PinotTypeCoercionTest`, `PinotRuleUtilsTest` pass.
Not covered: MV TIMESTAMP columns (no test fixture supports them); the
TIMESTAMP-vs-BIGINT coercion rule is re-applied after wrapping.
🤖 Generated with [Claude Code](https://claude.com/claude-code)
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: [email protected]
For queries about this service, please contact Infrastructure at:
[email protected]
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
