xiangfu0 opened a new pull request, #20029: URL: https://github.com/apache/pinot/pull/20029
## What Stores VARIANT in segments as a raw, dictionary-free forward index, and rejects table and schema configurations that would need equality, ordering or a value index on raw Variant bytes. ## Changes - Segment creation and mutable segments: no dictionary, no sorted-column detection, and no min/max collection or regeneration for VARIANT, because envelope byte order has no logical meaning. The empty-bytes SQL-null sentinel is never decoded as an envelope (`ColumnarValueNormalizer`, `ColumnMetadataImpl`). - `SchemaUtils`: VARIANT must be a single-value dimension, with no custom default null value and no truncating or substituting max-length strategy. - `TableConfigUtils`: requires storage null handling; requires a RAW, enabled forward index; rejects every other index, dictionary, star-tree (default or explicit), sorted and partition columns, primary keys, upsert comparison columns, non-OVERWRITE partial-upsert strategies, and metrics-aggregation keys. - The segment-processing framework and the minion merge/rollup and realtime-to-offline generators reject non-CONCAT merge types for tables with VARIANT columns. - VARIANT maps to Avro `bytes`. The SQL DDL compiler and reverse emitter accept `VARIANT`. The controller DDL resource and data generator handle it explicitly. ## Depends on 2/8 (tests build envelopes with `VariantUtils`). ## About this PR This is **part 3 of 8** splitting #19101 (end-to-end Parquet VARIANT support) into layered PRs. #19101 stays open as the umbrella with the full feature description; [`docs/design/VARIANT.md`](https://github.com/apache/pinot/blob/xiangfu0/variant-split/01-spi-foundation/docs/design/VARIANT.md) (added in 1/8) is the design reference. The base branch is the previous PR in the stack, so the diff below is exactly this layer. Each layer compiles and passes its own tests on top of the one below it, and the top of the stack has the same tree as #19101. > VARIANT is not usable end to end until the whole stack has landed. Raw-value opacity guards arrive in 5/8 (single-stage) and 6/8 (multi-stage), so please don't create VARIANT columns on a build that has only part of the stack. The stack should land within one release. ### Full stack (merge bottom → top) _filled in once all PRs are open_ -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected] --------------------------------------------------------------------- To unsubscribe, e-mail: [email protected] For additional commands, e-mail: [email protected]
