xiangfu0 opened a new pull request, #18449: URL: https://github.com/apache/pinot/pull/18449
## Summary - Add `partitionExpression` as a self-contained table-config partitioning mode, mutually exclusive with existing `functionName`/`functionConfig` partition functions. - Reuse Pinot's scalar function framework for expression evaluation, with deterministic-function checks, single partition-column references, and `numPartitions` as a compile-time expression constant. - Preserve the existing `PartitionFunction` path in parallel while wiring expression-mode partitioning through segment generation, realtime writers, metadata serialization, routing, and segment pruning. - Persist partition expressions in segment/ZK metadata and add user-facing sample config/docs under `pinot-tools`. ## Notes - Expression mode intentionally has no `partitionIdNormalizer`; callers can express normalization directly, e.g. `positiveModulo(fnv1aHash32UTF8(md5(id)))`. - The scalar-backed evaluator supports a one-argument `positiveModulo(expr)` shorthand that uses the configured `numPartitions`. - Non-deterministic scalar functions and expressions that reference columns other than the partition column are rejected. ## Testing - `./mvnw -pl pinot-broker,pinot-core,pinot-controller,pinot-segment-local,pinot-tools -am -DskipTests compile` - `./mvnw -pl pinot-common -am -Dtest=InbuiltPartitionExpressionEvaluatorFactoryTest,ColumnPartitionMetadataTest -Dsurefire.failIfNoSpecifiedTests=false test` - `./mvnw -pl pinot-spi -Dtest=IndexingConfigTest test` - `./mvnw -pl pinot-segment-local,pinot-broker -am -Dtest=SegmentPartitionTest,SegmentPartitionMetadataManagerExpressionTest -Dsurefire.failIfNoSpecifiedTests=false test` - `./mvnw -pl pinot-core -am -Dtest=ColumnValueSegmentPrunerTest -Dsurefire.failIfNoSpecifiedTests=false test` - `./mvnw spotless:apply -pl pinot-spi,pinot-segment-spi,pinot-common,pinot-segment-local,pinot-broker,pinot-core,pinot-controller,pinot-tools` - `./mvnw checkstyle:check -pl pinot-spi,pinot-segment-spi,pinot-common,pinot-segment-local,pinot-broker,pinot-core,pinot-controller,pinot-tools` - `./mvnw license:format -pl pinot-spi,pinot-segment-spi,pinot-common,pinot-segment-local,pinot-broker,pinot-core,pinot-controller,pinot-tools` - `./mvnw license:check -pl pinot-spi,pinot-segment-spi,pinot-common,pinot-segment-local,pinot-broker,pinot-core,pinot-controller,pinot-tools` -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected] --------------------------------------------------------------------- To unsubscribe, e-mail: [email protected] For additional commands, e-mail: [email protected]
