xiangfu0 opened a new pull request, #20032: URL: https://github.com/apache/pinot/pull/20032
## What Brings VARIANT to the multi-stage engine: planner typing, plan serde, runtime evaluation of the Variant functions, and the same raw-value opacity guards as the single-stage engine. ## Changes - Planner: Variant operators in `PinotOperatorTable`. `PinotEvaluateLiteralRule` does not fold JSON-to-Variant calls. `RawVariantValidationVisitor` and `RawVariantJoinKeyValidator` reject raw VARIANT in join keys, set operations, sorts, window and group keys, and general aggregates at dispatch time. - Plan serde: `ColumnDataType.VARIANT` is wire value `24` in `expressions.proto`, after UUID/UUID_ARRAY (`22`/`23`). A compatibility test against the previous proto checks that older values don't move. **Using `24` permanently needs explicit reviewer sign-off before merge.** - Runtime: `VariantOperand` and `LiteralParseJsonOperand`, plus defensive guards in the aggregate, group-by, hash-join, sort, sorted-mailbox-receive, window and set operators. ## Compatibility An old server that receives a plan containing VARIANT fails deserialization with "Unsupported proto ColumnDataType: UNRECOGNIZED". 8/8 adds compatibility-verifier suites that assert this exact behavior during a rolling upgrade. ## Depends on 5/8 (`TransformFunctionType` registrations). ## About this PR This is **part 6 of 8** splitting #19101 (end-to-end Parquet VARIANT support) into layered PRs. #19101 stays open as the umbrella with the full feature description; [`docs/design/VARIANT.md`](https://github.com/apache/pinot/blob/xiangfu0/variant-split/01-spi-foundation/docs/design/VARIANT.md) (added in 1/8) is the design reference. The base branch is the previous PR in the stack, so the diff below is exactly this layer. Each layer compiles and passes its own tests on top of the one below it, and the top of the stack has the same tree as #19101. > VARIANT is not usable end to end until the whole stack has landed. Raw-value opacity guards arrive in 5/8 (single-stage) and 6/8 (multi-stage), so please don't create VARIANT columns on a build that has only part of the stack. The stack should land within one release. ### Full stack (merge bottom → top) _filled in once all PRs are open_ -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected] --------------------------------------------------------------------- To unsubscribe, e-mail: [email protected] For additional commands, e-mail: [email protected]
