This is an automated email from the ASF dual-hosted git repository.
xiangfu0 pushed a change to branch
data-2616-disallow-nondeterministic-transforms
in repository https://gitbox.apache.org/repos/asf/pinot.git
discard 1d1249622bc Use independent ingestion replay determinism flag
discard 2dc8cc6395d Separate ingestion replay determinism metadata
discard b2d1de621ee Disallow non-deterministic ingestion transform configs
add dce035f9ce7 Add jsonPath*Fast / jsonPath*FirstMatch extractor
functions (#18972)
add fae8080bc78 [core] Optimize TDigest percentile aggregation (#18996)
add 28922369c88 Upgrade Pinot to JDK 25 (#19014)
add 26110ff9c76 Bump software.amazon.awssdk:bom from 2.48.2 to 2.48.3
(#19018)
add e7c9043b17f Bump axios from 1.16.0 to 1.18.0 in
/pinot-controller/src/main/resources (#19022)
add b9e538a117e [multistage] Fix flaky RAW_MESSAGES assertion in
GrpcSenderBackpressureTest (#19020)
add a6d16ef3772 Fix ClassCastException in skip-leaf direct aggregation
with deduplicated agg calls (#19023)
add 7d62928d7e2 OPEN_STRUCT storage layer — columnar two-tier dense/sparse
index (PR 2b/4) (#18643)
add ed2240ee74b Render non-String scalars in jsonPathString without JSON
quotes (#19029)
add 29cabf90e33 fix(minion): fail single-segment task when segment upload
fails (#18813)
add ffb1ffb1fab Bump com.microsoft.azure:msal4j from 1.25.0 to 1.25.1
(#19030)
add 7a6c7d954ae Bump software.amazon.awssdk:bom from 2.48.3 to 2.48.4
(#19031)
add 385e38a9c0f Fix checkstyle failure in
BaseSingleSegmentConversionExecutorTest (#19037)
add bde7223cf29 Implement canProduceBitmaps and getBitmaps for AND and OR
filter operators (#19038)
add c3cd3c70ad2 Fix TDigest accumulator serialization through generic
serdes for capacity-preserving state (#19017)
add de8cbd33808 [MSE] Remove the experimental enriched join feature
(#19033)
add 51b9d74dd7b Widen INT arithmetic to LONG in plus/minus/mult scalar
functions (#19036)
add 45869821669 Bump com.squareup.okio:okio-bom from 3.17.0 to 3.18.0
(#19052)
add fff8c9cf5a9 Bump software.amazon.awssdk:bom from 2.48.4 to 2.49.0
(#19050)
add d512b22ecbd Track segments with snapshot to improve upsert snapshot
taking flow (#19028)
add 86535a9b6b3 Fix explicit tableOptions hint handling in
PinotImplicitTableHintRule (#19003)
add 2298c6d8c62 Skip selection LIMIT pruning for segments with
externally-deleted docs (#19039)
add e3f2bf5fc36 Bump software.amazon.awssdk:bom from 2.49.0 to 2.49.1
(#19059)
add 682b94b24bb Bump org.webjars:swagger-ui from 5.32.8 to 5.32.11 (#19060)
add 33433f056d2 Bump org.mongodb:bson from 5.9.0 to 5.9.1 (#19061)
add 280f6b655a8 Bump io.grpc:grpc-bom from 1.82.2 to 1.83.0 (#19062)
add 3b2eeca5e7b Bump it.unimi.dsi:fastutil from 8.5.18 to 8.5.19 (#19063)
add afbf3049019 Cache segment to tier map at rebalance start (#19055)
add 867b53b7489 Skip full target recompute on IdealState version change
for strict realtime rebalance that only moves tier segments (#19054)
add 706b772334e Simplify getMovingConsumingSegments and drop per-segment
stream allocation (#19068)
add dded09d82a8 [UUID 2/8] UUID ingest and segment storage (#18870)
add a3fedd7bb8a Skip active-tasks validation in table config validate/tune
preflight (#19079)
add 1cde957d524 Bump docker/login-action from 4.4.0 to 4.5.0 (#19074)
add 3a5ae85e1f0 Bump org.codehaus.jettison:jettison from 1.5.6 to 1.5.7
(#19075)
add 729f22c9ff4 Bump software.amazon.awssdk:bom from 2.49.1 to 2.49.2
(#19076)
add 9d6a650df8c Bump org.roaringbitmap:RoaringBitmap from 1.6.15 to 1.6.16
(#19077)
add c6226556ff8 [core] Upgrade t-digest to 3.3 with mixed-version
compatibility (#19015)
add 2bcbdfed0af Bump postcss in /pinot-controller/src/main/resources
(#19080)
add ab3fd2b214b Support backfilling null value vector from the default
null value (#19072)
add e40d93100bc Add fast jsonExtractScalar transform variants (#19012)
add 2050fce9bd0 [UUID 3/8] UUID result rendering (DataSchema, Arrow/JSON
encoders) (#18871)
add 81b82b26850 Support GROUP BY GROUPING SETS / ROLLUP / CUBE in the
multi-stage query engine (#18817)
add 71475e49ef3 add broker config options for sql log redaction (#18604)
add c43f05a355f Block UpsertCompactionTask and UpsertCompactMergeTask when
metadataTTL is enabled (#19057)
add 826a4d9455e Bump docker/login-action from 4.5.0 to 4.5.1 (#19095)
add 9b3f3eee517 Bump org.roaringbitmap:RoaringBitmap from 1.6.16 to 1.6.18
(#19096)
add 97691708c79 Bump software.amazon.awssdk:bom from 2.49.2 to 2.49.3
(#19097)
add 72f0d4928d3 Bump joda-time:joda-time from 2.14.2 to 2.14.3 (#19098)
add e68502b972e Bump com.github.luben:zstd-jni from 1.5.7-11 to 1.5.7-12
(#19099)
add 52051c180bb Bump io.opentelemetry.contrib:opentelemetry-gcp-resources
(#19100)
add c26d315512d Merge redundant quickstarts into the batch and stream
quickstarts (#19090)
add a8b207e7815 Generate Avro schemas from Pinot logical data types
(#19071) (#19073)
add dc2d0772613 Bump org.eclipse.sisu:sisu-maven-plugin from 1.0.1 to
1.1.0 (#19104)
add a0e9a9f8fdf Bump actions/stale from 10 to 11 (#19105)
add cea04a3a2fb Bump docker/login-action from 4.5.1 to 4.5.2 (#19106)
add 8f2eea37e39 Add unit tests for IFSTIndexHandler index lifecycle
(#18963)
add fc125943a55 Fix /tables status diff viewer crash (React error #130)
(#19109)
add e3bda5d53d9 Add IGNORE NULLS support for LEAD and LAG window functions
(#18202)
add 5606479534c Bump software.amazon.awssdk:bom from 2.49.3 to 2.49.4
(#19107)
add ce202033a62 Bump com.mycila:license-maven-plugin from 5.0.0 to 5.1.1
(#19051)
add 250654502f7 Add unit tests for JsonIndexHandler index lifecycle
(#19045)
add 6d5cdfb93ab Verify the Java 11 pinned client artifacts actually run on
a Java 11 JVM in CI (#19102)
add 7ecf65743e6 Harden schema case-collision validation with Locale.ROOT
(#19089)
add 6fd0774ac87 Use PostgreSQL function volatility for ingestion transforms
This update added new revisions after undoing existing revisions.
That is to say, some revisions that were in the old version of the
branch are not in the new version. This situation occurs
when a user --force pushes a change and generates a repository
containing something like this:
* -- * -- B -- O -- O -- O (1d1249622bc)
\
N -- N -- N
refs/heads/data-2616-disallow-nondeterministic-transforms (6fd0774ac87)
You should already have received notification emails for all of the O
revisions, and so the following emails describe only the N revisions
from the common base, B.
Any revisions marked "omit" are not gone; other references still
refer to them. Any revisions marked "discard" are gone forever.
No new revisions were added by this update.
Summary of changes:
.github/workflows/build-all-pinot-docker-image.yml | 12 +-
.../workflows/build-pinot-base-docker-image.yml | 4 +-
.github/workflows/build-pinot-docker-image.yml | 4 +-
.github/workflows/build-superset-docker-image.yml | 2 +-
.github/workflows/close_stale_issues.yml | 2 +-
.../nightly-build-all-pinot-docker-image.yml | 12 +-
...htly-build-multi-arch-superset-docker-image.yml | 4 +-
.github/workflows/pinot_compatibility_checks.yml | 12 +-
.github/workflows/pinot_compatibility_tests.yml | 4 +-
.github/workflows/pinot_integration_tests.yml | 2 +-
.../pinot_java11_client_compatibility.yml | 114 ++
...ulti_stage_query_engine_compatibility_tests.yml | 4 +-
.github/workflows/pinot_quickstart_tests.yml | 2 +-
.github/workflows/pinot_unit_tests.yml | 6 +-
.../scripts/.pinot_java11_client_compat.sh | 97 ++
.github/workflows/scripts/.pinot_quickstart.sh | 144 +-
.../docker/.pinot_docker_image_build_and_push.sh | 2 +-
...not_multi_arch_docker_image_manifest_package.sh | 2 +-
.../docker/.pinot_package_single_platform.sh | 6 +-
CLAUDE.md | 4 +-
LICENSE-binary | 3 +-
README.md | 2 +-
compatibility-verifier/checkoutAndBuild.sh | 5 +-
docker/images/pinot-base/README.md | 10 +-
docker/images/pinot/Dockerfile | 4 +-
docker/images/pinot/Dockerfile.build | 4 +-
docker/images/pinot/Dockerfile.package | 2 +-
helm/pinot/values.yaml | 8 +-
pinot-bom/pom.xml | 2 +-
.../apache/pinot/broker/querylog/QueryLogger.java | 65 +-
.../BaseSingleStageBrokerRequestHandler.java | 93 +-
.../MultiStageBrokerRequestHandler.java | 44 +-
.../pinot/broker/querylog/QueryLoggerTest.java | 167 ++-
.../common/function/FastJsonPathExtractor.java | 337 +++++
.../apache/pinot/common/function/FunctionInfo.java | 39 +-
.../pinot/common/function/PinotScalarFunction.java | 5 +-
.../pinot/common/function/SimpleJsonPath.java | 217 +++
.../common/function/TransformFunctionType.java | 56 +-
.../function/scalar/ArithmeticFunctions.java | 3 +-
.../common/function/scalar/DateTimeFunctions.java | 9 +-
.../common/function/scalar/InternalFunctions.java | 12 +-
.../common/function/scalar/JsonFunctions.java | 155 +-
.../scalar/arithmetic/MinusScalarFunction.java | 10 +-
.../scalar/arithmetic/MultScalarFunction.java | 10 +-
.../scalar/arithmetic/PlusScalarFunction.java | 10 +-
.../apache/pinot/common/metrics/ServerMeter.java | 2 +
.../apache/pinot/common/request/PinotQuery.java | 317 ++--
.../pinot/common/request/context/GroupingSets.java | 109 +-
.../response/encoder/ArrowResponseEncoder.java | 6 +
.../response/encoder/JsonResponseEncoder.java | 2 +
.../org/apache/pinot/common/utils/DataSchema.java | 172 ++-
.../apache/pinot/sql/parsers/CalciteSqlParser.java | 83 +-
.../org/apache/pinot/sql/parsers/ParserUtils.java | 19 +-
...nAggregationGroupByToDistinctQueryRewriter.java | 2 +-
pinot-common/src/main/proto/plan.proto | 23 +-
.../common/function/FastJsonPathExtractorTest.java | 737 +++++++++
.../pinot/common/function/FunctionUtilsTest.java | 99 +-
.../pinot/common/function/JsonFunctionsTest.java | 40 +
.../arithmetic/ArithmeticScalarFunctionTest.java | 43 +
.../common/request/context/GroupingSetsTest.java | 66 +
.../response/encoder/ArrowResponseEncoderTest.java | 79 +-
.../response/encoder/JsonResponseEncoderTest.java | 52 +-
.../apache/pinot/common/utils/DataSchemaTest.java | 88 +-
.../pinot/sql/parsers/GroupingSetsParserTest.java | 59 +-
pinot-common/src/thrift/query.thrift | 16 +-
.../connector/flink/sink/FlinkSegmentWriter.java | 3 +-
.../api/resources/PinotTableRestletResource.java | 2 +-
.../api/resources/TableConfigsRestletResource.java | 15 +-
.../assignment/segment/BaseSegmentAssignment.java | 25 +-
.../assignment/segment/SegmentAssignmentUtils.java | 82 +-
.../helix/core/rebalance/TableRebalancer.java | 66 +-
.../src/main/resources/package-lock.json | 48 +-
pinot-controller/src/main/resources/package.json | 2 +-
pinot-controller/src/main/resources/vite.config.ts | 10 +-
.../api/PinotSchemaRestletResourceTest.java | 8 +-
.../api/PinotTableRestletResourceTest.java | 8 +-
.../api/TableConfigsRestletResourceTest.java | 16 +-
...NonReplicaGroupTieredSegmentAssignmentTest.java | 64 +
...NonReplicaGroupTieredSegmentAssignmentTest.java | 8 +
.../StrictRealtimeSegmentAssignmentTest.java | 14 +-
.../TableRebalancerClusterStatelessTest.java | 10 +-
.../apache/pinot/core/common/ObjectSerDeUtils.java | 97 +-
.../core/common/datablock/DataBlockBuilder.java | 40 +-
.../apache/pinot/core/data/table/TableResizer.java | 30 +-
.../core/function/scalar/SketchFunctions.java | 85 +-
.../blocks/results/GroupByResultsBlock.java | 4 +-
.../core/operator/filter/AndFilterOperator.java | 25 +-
.../core/operator/filter/OrFilterOperator.java | 25 +-
.../JsonExtractScalarTransformFunction.java | 173 ++-
.../function/TransformFunctionFactory.java | 4 +
...ValueIntegerTupleSketchAggregationFunction.java | 4 +-
.../DistinctCountCPCSketchAggregationFunction.java | 4 +-
...inctCountRawThetaSketchAggregationFunction.java | 6 +-
...istinctCountThetaSketchAggregationFunction.java | 184 +--
.../FrequentLongsSketchAggregationFunction.java | 59 +-
.../FrequentStringsSketchAggregationFunction.java | 57 +-
.../IntegerTupleSketchAggregationFunction.java | 22 +-
.../function/PercentileKLLAggregationFunction.java | 4 +-
.../PercentileSmartTDigestAggregationFunction.java | 34 +-
.../function/PercentileTDigestAccumulator.java | 1585 ++++++++++++++++++++
.../PercentileTDigestAggregationFunction.java | 137 +-
...aluesIntegerTupleSketchAggregationFunction.java | 4 +-
.../FunnelCountAggregationFunctionFactory.java | 14 +-
.../funnel/ThetaSketchAggregationStrategy.java | 25 +-
.../function/funnel/ThetaSketchMergeStrategy.java | 22 +-
.../ThetaSketchResultExtractionStrategy.java | 13 +-
.../groupby/GroupingSetsGroupKeyGenerator.java | 36 +-
.../query/pruner/SelectionQuerySegmentPruner.java | 8 +-
.../core/query/reduce/GroupByDataTableReducer.java | 4 +-
.../core/query/reduce/PostAggregationHandler.java | 36 +-
.../context/utils/QueryContextConverterUtils.java | 63 +-
.../DistinctCountThetaSketchAggregator.java | 16 +-
.../aggregator/IntegerTupleSketchAggregator.java | 16 +-
.../aggregator/PercentileTDigestAggregator.java | 9 +-
.../pinot/core/util/SegmentProcessorAvroUtils.java | 258 +++-
.../core/common/DataSketchesSerialCompatTest.java | 540 +++++++
.../pinot/core/common/ObjectSerDeUtilsTest.java | 86 +-
.../common/datablock/DataBlockBuilderTest.java | 23 +
.../core/common/datablock/DataBlockTestUtils.java | 13 +
.../core/common/datatable/DataTableSerDeTest.java | 26 +
.../pinot/core/data/table/IndexedTableTest.java | 103 ++
.../pinot/core/function/FunctionRegistryTest.java | 3 +-
.../core/function/scalar/SketchFunctionsTest.java | 9 +-
.../operator/filter/AndFilterOperatorTest.java | 155 +-
.../core/operator/filter/OrFilterOperatorTest.java | 136 +-
.../JsonExtractScalarTransformFunctionTest.java | 260 ++--
...tinctCountCPCSketchAggregationFunctionTest.java | 4 +-
...ercentileRawTDigestAggregationFunctionTest.java | 177 +++
...centileSmartTDigestAggregationFunctionTest.java | 227 ++-
.../PercentileTDigestAggregationFunctionTest.java | 1490 +++++++++++++++++-
...PercentileTDigestMVAggregationFunctionTest.java | 10 +-
.../PostAggregationFunctionTest.java | 17 +
.../BrokerRequestToQueryContextConverterTest.java | 44 +-
.../selection/SelectionOperatorUtilsTest.java | 23 +
.../DistinctCountThetaSketchAggregatorTest.java | 26 +-
.../IntegerTupleSketchAggregatorTest.java | 22 +-
.../PercentileTDigestAggregatorTest.java | 61 +-
...ctCountIntegerSumTupleSketchStarTreeV2Test.java | 18 +-
.../v2/DistinctCountThetaSketchStarTreeV2Test.java | 14 +-
.../apache/pinot/core/util/SchemaUtilsTest.java | 35 +-
.../core/util/SegmentProcessorAvroUtilsTest.java | 405 +++++
.../DistinctCountThetaSketchQueriesTest.java | 15 +-
.../queries/FrequentItemsSketchQueriesTest.java | 46 +-
.../queries/FunnelCountQueriesThetaSketchTest.java | 4 +-
.../queries/PercentileTDigestMVQueriesTest.java | 14 +-
.../queries/PercentileTDigestQueriesTest.java | 66 +-
.../tests/NullHandlingIntegrationTest.java | 47 +-
.../tests/custom/GroupingSetsQueriesTest.java | 444 ++++--
.../integration/tests/custom/JsonPathTest.java | 68 +-
.../OpenStructIngestionCommitOfflineTest.java | 24 +-
.../OpenStructIngestionCommitRealtimeTest.java | 90 ++
.../custom/OpenStructIngestionCommitTestBase.java | 287 ++++
.../integration/tests/custom/ThetaSketchTest.java | 8 +-
.../integration/tests/custom/TupleSketchTest.java | 20 +-
.../resources/udf-test-results/all-functions.yaml | 32 +
pinot-java11-client-verifier/pom.xml | 140 ++
.../pinot/java11/CannedResponseTransport.java | 73 +
.../pinot/java11/ClasspathClosureScanner.java | 334 +++++
.../pinot/java11/Java11CompatibilityVerifier.java | 803 ++++++++++
.../src/main/resources/sample-broker-response.json | 53 +
.../pinot/java11/ClasspathClosureScannerTest.java | 367 +++++
pinot-perf/README.md | 6 +
.../percentile-tdigest-aggregation-100m-results.md | 268 ++++
.../pinot/perf/BenchmarkJsonPathExtraction.java | 172 +++
.../apache/pinot/perf/BenchmarkSketchSerDe.java | 151 ++
.../BenchmarkPercentileTDigestAggregation.java | 114 ++
.../BenchmarkPercentileTDigestCombine.java | 977 ++++++++++++
...chmarkPercentileTDigestStarTreeAggregation.java | 222 +++
.../BenchmarkPercentileTDigestValueAggregator.java | 141 ++
.../perf/aggregation/TDigestBenchmarkUtils.java | 88 ++
.../avro/AvroIngestionSchemaValidator.java | 2 +-
.../plugin/inputformat/avro/AvroSchemaUtil.java | 179 ++-
.../pinot/plugin/inputformat/avro/AvroUtils.java | 83 +-
.../inputformat/avro/AvroSchemaUtilTest.java | 106 +-
.../plugin/inputformat/avro/AvroUtilsTest.java | 139 ++
.../tasks/BaseSingleSegmentConversionExecutor.java | 33 +-
.../UpsertCompactionTaskGenerator.java | 10 +-
.../UpsertCompactMergeTaskGenerator.java | 6 +
.../BaseSingleSegmentConversionExecutorTest.java | 243 +++
.../UpsertCompactionTaskGeneratorTest.java | 11 +
.../UpsertCompactMergeTaskGeneratorTest.java | 12 +
.../filebased/FileBasedSegmentWriter.java | 3 +-
.../filebased/FileBasedSegmentWriterTest.java | 46 +
.../calcite/rel/logical/PinotLogicalAggregate.java | 18 +
.../rel/logical/PinotLogicalEnrichedJoin.java | 247 ---
.../calcite/rel/rules/GroupingSetsPlanUtils.java | 198 +++
.../PinotAggregateExchangeNodeInsertRule.java | 180 ++-
.../calcite/rel/rules/PinotEnrichedJoinRule.java | 396 -----
.../rel/rules/PinotImplicitTableHintRule.java | 79 +-
.../pinot/calcite/rel/rules/TableOptions.java | 7 +
.../pinot/calcite/sql/fun/PinotOperatorTable.java | 37 +-
.../org/apache/pinot/query/QueryEnvironment.java | 8 +-
.../planner/explain/ExplainNodeSimplifier.java | 1 +
.../explain/PhysicalExplainPlanVisitor.java | 1 +
.../query/planner/explain/PlanNodeMerger.java | 1 +
.../query/planner/explain/PlanNodeSorter.java | 1 +
.../planner/logical/EquivalentStagesFinder.java | 1 +
.../query/planner/logical/PlanFragmenter.java | 1 +
.../planner/logical/PlanNodeToRelConverter.java | 1 +
.../planner/logical/RelToPlanNodeConverter.java | 93 +-
.../query/planner/logical/SubPlanFragmenter.java | 1 +
.../planner/physical/DispatchablePlanVisitor.java | 1 +
.../physical/v2/PRelToPlanNodeConverter.java | 6 +-
.../planner/physical/v2/RelToPRelConverter.java | 4 -
.../physical/v2/nodes/PhysicalAggregate.java | 19 +
.../planner/physical/v2/nodes/PhysicalJoin.java | 22 +-
.../v2/opt/rules/AggregatePushdownRule.java | 144 +-
.../query/planner/plannode/AggregateNode.java | 35 +-
.../plannode/DefaultPostOrderTraversalVisitor.java | 1 +
.../query/planner/plannode/EnrichedJoinNode.java | 9 +
.../query/planner/plannode/PlanNodeVisitor.java | 5 +
.../query/planner/serde/PlanNodeDeserializer.java | 10 +-
.../query/planner/serde/PlanNodeSerializer.java | 11 +-
.../validation/ArrayToMvValidationVisitor.java | 1 +
.../validate/RowExpressionValidationVisitor.java | 7 +-
.../PinotAggregateExchangeNodeInsertRuleTest.java | 121 ++
.../rel/rules/PinotEnrichedJoinRuleTest.java | 638 --------
.../rel/rules/PinotImplicitTableHintRuleTest.java | 125 ++
.../apache/pinot/query/QueryCompilationTest.java | 154 ++
.../pinot/query/QueryEnvironmentTestBase.java | 7 +
.../pinot/query/QueryPlannerRuleOptionsTest.java | 7 +-
.../logical/RelToPlanNodeConverterTest.java | 56 -
.../query/planner/serde/PlanNodeSerDeTest.java | 53 +
.../src/test/resources/queries/JoinPlans.json | 123 +-
.../test/resources/queries/PinotHintablePlans.json | 22 +
.../query/runtime/InStageStatsTreeBuilder.java | 1 +
.../query/runtime/operator/AggregateOperator.java | 4 +-
.../runtime/operator/EnrichedHashJoinOperator.java | 417 -----
.../query/runtime/operator/MultiStageOperator.java | 10 +-
.../query/runtime/operator/RepeatOperator.java | 196 +++
.../operator/factory/AggregateOperatorFactory.java | 11 +
.../factory/DefaultJoinOperatorFactory.java | 24 +-
.../operator/factory/JoinOperatorFactory.java | 4 +
.../runtime/operator/window/WindowFunction.java | 2 +
.../window/value/LagValueWindowFunction.java | 45 +-
.../window/value/LeadValueWindowFunction.java | 45 +-
.../window/value/OffsetValueWindowFunction.java | 121 ++
.../query/runtime/plan/PlanNodeToOpChain.java | 62 +
.../plan/server/ServerPlanRequestVisitor.java | 8 +
.../query/mailbox/GrpcSenderBackpressureTest.java | 19 +-
.../operator/EnrichedHashJoinOperatorTest.java | 650 --------
.../query/runtime/operator/RepeatOperatorTest.java | 272 ++++
.../operator/WindowAggregateOperatorTest.java | 448 ++++++
.../factory/DefaultJoinOperatorFactoryTest.java | 23 +-
.../query/runtime/queries/QueryRunnerTest.java | 17 +
.../src/test/resources/queries/EnrichedJoin.json | 93 --
.../src/test/resources/queries/UDFAggregates.json | 6 +-
.../test/resources/queries/WindowFunctions.json | 110 ++
pinot-segment-local/pom.xml | 5 +
.../DistinctCountThetaSketchValueAggregator.java | 56 +-
.../IntegerTupleSketchValueAggregator.java | 44 +-
.../PercentileTDigestValueAggregator.java | 581 ++++++-
.../SerializedFrequentLongsSketch.java | 12 +-
.../SerializedFrequentStringsSketch.java | 12 +-
.../local/customobject/ThetaSketchAccumulator.java | 26 +-
.../customobject/TupleIntSketchAccumulator.java | 20 +-
.../immutable/ImmutableSegmentImpl.java | 55 +-
.../immutable/ImmutableSegmentLoader.java | 12 +
.../indexsegment/mutable/MutableSegmentImpl.java | 119 +-
.../stats/MutableNoDictColumnStatistics.java | 4 +-
.../stats/RealtimeSegmentStatsContainer.java | 8 +
.../dictionary/BytesOffHeapMutableDictionary.java | 9 +-
.../dictionary/BytesOnHeapMutableDictionary.java | 9 +-
.../forward/FixedByteMVMutableForwardIndex.java | 60 +
.../recordtransformer/DataTypeTransformer.java | 78 +-
.../segment/creator/impl/BaseSegmentCreator.java | 34 +
.../creator/impl/SegmentColumnarIndexCreator.java | 48 +-
.../impl/nullvalue/NullValueVectorCreator.java | 37 +-
.../impl/openstruct/OpenStructColumnSplitter.java | 550 +++++++
.../stats/BytesColumnPreIndexStatsCollector.java | 7 +-
.../stats/NoDictColumnStatisticsCollector.java | 6 +-
.../DefaultNullValueVirtualColumnProvider.java | 1 +
.../segment/index/forward/ForwardIndexType.java | 9 +-
.../segment/index/loader/IndexLoadingConfig.java | 43 +
.../segment/index/loader/SegmentPreProcessor.java | 5 +
.../ColumnMinMaxValueGenerator.java | 2 +
.../index/nullvalue/NullValueIndexType.java | 93 +-
.../index/nullvalue/NullValueVectorHandler.java | 372 +++++
.../openstruct/ImmutableOpenStructDataSource.java | 183 +++
.../segment/index/openstruct/MutableKeyColumn.java | 142 ++
.../openstruct/MutableOpenStructDataSource.java | 180 +++
.../index/openstruct/MutableOpenStructIndex.java | 267 ++++
.../index/openstruct/OpenStructIndexType.java | 15 +-
.../segment/readers/PinotSegmentColumnReader.java | 2 +
.../segment/readers/PinotSegmentRecordReader.java | 45 +-
.../upsert/BasePartitionUpsertMetadataManager.java | 206 ++-
.../segment/local/utils/CustomSerDeUtils.java | 44 +-
.../pinot/segment/local/utils/HashUtils.java | 23 +-
.../local/utils/NullValueTransformerUtils.java | 33 +-
.../pinot/segment/local/utils/SchemaUtils.java | 22 +-
.../pinot/segment/local/utils/TDigestUtils.java | 642 ++++++++
.../segment/local/utils/TableConfigUtils.java | 64 +-
...istinctCountThetaSketchValueAggregatorTest.java | 89 +-
.../IntegerTupleSketchValueAggregatorTest.java | 30 +-
.../PercentileTDigestValueAggregatorTest.java | 336 +++++
.../customobject/ThetaSketchAccumulatorTest.java | 31 +-
.../TupleIntSketchAccumulatorTest.java | 24 +-
.../impl/dictionary/MultiValueDictionaryTest.java | 53 +
.../recordtransformer/DataTypeTransformerTest.java | 112 ++
.../openstruct/OpenStructColumnSplitterTest.java | 414 +++++
.../openstruct/OpenStructSegmentCreationTest.java | 157 ++
.../DefaultNullValueVirtualColumnProviderTest.java | 10 +
.../index/creator/BloomFilterCreatorTest.java | 28 +
.../ColumnMinMaxValueGeneratorTest.java | 43 +-
...xHandlerTest.java => IFSTIndexHandlerTest.java} | 54 +-
.../index/nullvalue/NullValueIndexTypeTest.java | 105 +-
.../nullvalue/NullValueVectorHandlerTest.java | 511 +++++++
.../ImmutableOpenStructDataSourceTest.java | 200 +++
.../MutableOpenStructDataSourceTest.java | 93 ++
.../openstruct/MutableOpenStructIndexTest.java | 160 ++
.../segment/local/segment/readers/LazyRowTest.java | 12 +
.../BasePartitionUpsertMetadataManagerTest.java | 73 +-
...ertMetadataManagerForConsistentDeletesTest.java | 14 +
...rrentMapPartitionUpsertMetadataManagerTest.java | 17 +
.../pinot/segment/local/utils/HashUtilsTest.java | 42 +
.../segment/local/utils/TDigestUtilsTest.java | 423 ++++++
.../segment/local/utils/TableConfigUtilsTest.java | 89 +-
.../apache/pinot/segment/spi/ColumnMetadata.java | 9 +
.../org/apache/pinot/segment/spi/IndexSegment.java | 7 +
.../org/apache/pinot/segment/spi/V1Constants.java | 6 +
.../pinot/segment/spi/index/StandardIndexes.java | 5 +-
.../spi/index/creator/BloomFilterCreator.java | 16 +
.../spi/index/metadata/ColumnMetadataImpl.java | 31 +-
.../PredownloadPredownloadStatusRecorderTest.java | 4 +-
.../pinot/spi/annotations/FunctionVolatility.java | 30 +-
.../pinot/spi/annotations/ScalarFunction.java | 19 +-
.../spi/config/table/NullValueVectorConfig.java | 66 +
.../apache/pinot/spi/utils/CommonConstants.java | 7 +
pinot-tools/pom.xml | 61 -
.../pinot/tools/ColocatedJoinQuickStart.java | 2 +-
.../org/apache/pinot/tools/JoinQuickStart.java | 73 -
.../apache/pinot/tools/JsonIndexQuickStart.java | 55 -
.../pinot/tools/LookupJoinEngineQuickStart.java | 2 +-
.../pinot/tools/MultistageEngineQuickStart.java | 313 ----
.../OfflineComplexTypeHandlingQuickStart.java | 53 -
.../pinot/tools/PartialUpsertQuickStart.java | 76 -
.../org/apache/pinot/tools/QuickStartBase.java | 36 +
.../java/org/apache/pinot/tools/Quickstart.java | 187 ++-
.../apache/pinot/tools/QuickstartTableRequest.java | 5 +
.../RealtimeComplexTypeHandlingQuickStart.java | 55 -
.../pinot/tools/RealtimeJsonIndexQuickStart.java | 54 -
.../org/apache/pinot/tools/RealtimeQuickStart.java | 190 ++-
.../org/apache/pinot/tools/TPCHQuickStart.java | 2 +-
.../pinot/tools/TimestampIndexQuickstart.java | 66 -
.../apache/pinot/tools/UpsertJsonQuickStart.java | 55 -
.../org/apache/pinot/tools/UpsertQuickStart.java | 53 -
.../tools/admin/command/QuickStartCommand.java | 17 +-
.../tools/admin/command/QuickstartRunner.java | 12 +
.../converter/PinotSegmentToAvroConverter.java | 3 +-
.../converter/PinotSegmentToParquetConverter.java | 1 +
...IngestionJobSpec.yaml => ingestionJobSpec.yaml} | 0
...IngestionJobSpec.yaml => ingestionJobSpec.yaml} | 0
...IngestionJobSpec.yaml => ingestionJobSpec.yaml} | 0
...IngestionJobSpec.yaml => ingestionJobSpec.yaml} | 0
...IngestionJobSpec.yaml => ingestionJobSpec.yaml} | 0
.../org/apache/pinot/tools/BaseQuickStartTest.java | 93 ++
.../pinot/tools/admin/command/QuickStartTest.java | 119 ++
.../tools/admin/command/TestQuickStartCommand.java | 125 --
.../converter/PinotSegmentConverterTest.java | 132 ++
pom.xml | 41 +-
360 files changed, 25572 insertions(+), 6337 deletions(-)
create mode 100644 .github/workflows/pinot_java11_client_compatibility.yml
create mode 100755 .github/workflows/scripts/.pinot_java11_client_compat.sh
create mode 100644
pinot-common/src/main/java/org/apache/pinot/common/function/FastJsonPathExtractor.java
create mode 100644
pinot-common/src/main/java/org/apache/pinot/common/function/SimpleJsonPath.java
create mode 100644
pinot-common/src/test/java/org/apache/pinot/common/function/FastJsonPathExtractorTest.java
create mode 100644
pinot-common/src/test/java/org/apache/pinot/common/request/context/GroupingSetsTest.java
create mode 100644
pinot-core/src/main/java/org/apache/pinot/core/query/aggregation/function/PercentileTDigestAccumulator.java
create mode 100644
pinot-core/src/test/java/org/apache/pinot/core/common/DataSketchesSerialCompatTest.java
create mode 100644
pinot-core/src/test/java/org/apache/pinot/core/query/aggregation/function/PercentileRawTDigestAggregationFunctionTest.java
create mode 100644
pinot-core/src/test/java/org/apache/pinot/core/util/SegmentProcessorAvroUtilsTest.java
copy
pinot-segment-local/src/test/java/org/apache/pinot/segment/local/utils/HyperLogLogUtilsTest.java
=>
pinot-integration-tests/src/test/java/org/apache/pinot/integration/tests/custom/OpenStructIngestionCommitOfflineTest.java
(57%)
create mode 100644
pinot-integration-tests/src/test/java/org/apache/pinot/integration/tests/custom/OpenStructIngestionCommitRealtimeTest.java
create mode 100644
pinot-integration-tests/src/test/java/org/apache/pinot/integration/tests/custom/OpenStructIngestionCommitTestBase.java
create mode 100644 pinot-java11-client-verifier/pom.xml
create mode 100644
pinot-java11-client-verifier/src/main/java/org/apache/pinot/java11/CannedResponseTransport.java
create mode 100644
pinot-java11-client-verifier/src/main/java/org/apache/pinot/java11/ClasspathClosureScanner.java
create mode 100644
pinot-java11-client-verifier/src/main/java/org/apache/pinot/java11/Java11CompatibilityVerifier.java
create mode 100644
pinot-java11-client-verifier/src/main/resources/sample-broker-response.json
create mode 100644
pinot-java11-client-verifier/src/test/java/org/apache/pinot/java11/ClasspathClosureScannerTest.java
create mode 100644
pinot-perf/benchmark-results/percentile-tdigest-aggregation-100m-results.md
create mode 100644
pinot-perf/src/main/java/org/apache/pinot/perf/BenchmarkJsonPathExtraction.java
create mode 100644
pinot-perf/src/main/java/org/apache/pinot/perf/BenchmarkSketchSerDe.java
create mode 100644
pinot-perf/src/main/java/org/apache/pinot/perf/aggregation/BenchmarkPercentileTDigestAggregation.java
create mode 100644
pinot-perf/src/main/java/org/apache/pinot/perf/aggregation/BenchmarkPercentileTDigestCombine.java
create mode 100644
pinot-perf/src/main/java/org/apache/pinot/perf/aggregation/BenchmarkPercentileTDigestStarTreeAggregation.java
create mode 100644
pinot-perf/src/main/java/org/apache/pinot/perf/aggregation/BenchmarkPercentileTDigestValueAggregator.java
create mode 100644
pinot-perf/src/main/java/org/apache/pinot/perf/aggregation/TDigestBenchmarkUtils.java
create mode 100644
pinot-plugins/pinot-minion-tasks/pinot-minion-builtin-tasks/src/test/java/org/apache/pinot/plugin/minion/tasks/BaseSingleSegmentConversionExecutorTest.java
delete mode 100644
pinot-query-planner/src/main/java/org/apache/pinot/calcite/rel/logical/PinotLogicalEnrichedJoin.java
create mode 100644
pinot-query-planner/src/main/java/org/apache/pinot/calcite/rel/rules/GroupingSetsPlanUtils.java
delete mode 100644
pinot-query-planner/src/main/java/org/apache/pinot/calcite/rel/rules/PinotEnrichedJoinRule.java
create mode 100644
pinot-query-planner/src/test/java/org/apache/pinot/calcite/rel/rules/PinotAggregateExchangeNodeInsertRuleTest.java
delete mode 100644
pinot-query-planner/src/test/java/org/apache/pinot/calcite/rel/rules/PinotEnrichedJoinRuleTest.java
create mode 100644
pinot-query-planner/src/test/java/org/apache/pinot/calcite/rel/rules/PinotImplicitTableHintRuleTest.java
delete mode 100644
pinot-query-runtime/src/main/java/org/apache/pinot/query/runtime/operator/EnrichedHashJoinOperator.java
create mode 100644
pinot-query-runtime/src/main/java/org/apache/pinot/query/runtime/operator/RepeatOperator.java
create mode 100644
pinot-query-runtime/src/main/java/org/apache/pinot/query/runtime/operator/window/value/OffsetValueWindowFunction.java
delete mode 100644
pinot-query-runtime/src/test/java/org/apache/pinot/query/runtime/operator/EnrichedHashJoinOperatorTest.java
create mode 100644
pinot-query-runtime/src/test/java/org/apache/pinot/query/runtime/operator/RepeatOperatorTest.java
copy
pinot-common/src/test/java/org/apache/pinot/common/function/scalar/arithmetic/ArithmeticScalarFunctionTest.java
=>
pinot-query-runtime/src/test/java/org/apache/pinot/query/runtime/operator/factory/DefaultJoinOperatorFactoryTest.java
(57%)
delete mode 100644
pinot-query-runtime/src/test/resources/queries/EnrichedJoin.json
create mode 100644
pinot-segment-local/src/main/java/org/apache/pinot/segment/local/segment/creator/impl/openstruct/OpenStructColumnSplitter.java
create mode 100644
pinot-segment-local/src/main/java/org/apache/pinot/segment/local/segment/index/nullvalue/NullValueVectorHandler.java
create mode 100644
pinot-segment-local/src/main/java/org/apache/pinot/segment/local/segment/index/openstruct/ImmutableOpenStructDataSource.java
create mode 100644
pinot-segment-local/src/main/java/org/apache/pinot/segment/local/segment/index/openstruct/MutableKeyColumn.java
create mode 100644
pinot-segment-local/src/main/java/org/apache/pinot/segment/local/segment/index/openstruct/MutableOpenStructDataSource.java
create mode 100644
pinot-segment-local/src/main/java/org/apache/pinot/segment/local/segment/index/openstruct/MutableOpenStructIndex.java
create mode 100644
pinot-segment-local/src/main/java/org/apache/pinot/segment/local/utils/TDigestUtils.java
create mode 100644
pinot-segment-local/src/test/java/org/apache/pinot/segment/local/aggregator/PercentileTDigestValueAggregatorTest.java
create mode 100644
pinot-segment-local/src/test/java/org/apache/pinot/segment/local/segment/creator/impl/openstruct/OpenStructColumnSplitterTest.java
create mode 100644
pinot-segment-local/src/test/java/org/apache/pinot/segment/local/segment/creator/impl/openstruct/OpenStructSegmentCreationTest.java
copy
pinot-segment-local/src/test/java/org/apache/pinot/segment/local/segment/index/loader/invertedindex/{JsonIndexHandlerTest.java
=> IFSTIndexHandlerTest.java} (75%)
create mode 100644
pinot-segment-local/src/test/java/org/apache/pinot/segment/local/segment/index/nullvalue/NullValueVectorHandlerTest.java
create mode 100644
pinot-segment-local/src/test/java/org/apache/pinot/segment/local/segment/index/openstruct/ImmutableOpenStructDataSourceTest.java
create mode 100644
pinot-segment-local/src/test/java/org/apache/pinot/segment/local/segment/index/openstruct/MutableOpenStructDataSourceTest.java
create mode 100644
pinot-segment-local/src/test/java/org/apache/pinot/segment/local/segment/index/openstruct/MutableOpenStructIndexTest.java
create mode 100644
pinot-segment-local/src/test/java/org/apache/pinot/segment/local/utils/TDigestUtilsTest.java
copy
pinot-common/src/main/java/org/apache/pinot/common/restlet/resources/TablesList.java
=>
pinot-spi/src/main/java/org/apache/pinot/spi/annotations/FunctionVolatility.java
(50%)
create mode 100644
pinot-spi/src/main/java/org/apache/pinot/spi/config/table/NullValueVectorConfig.java
delete mode 100644
pinot-tools/src/main/java/org/apache/pinot/tools/JoinQuickStart.java
delete mode 100644
pinot-tools/src/main/java/org/apache/pinot/tools/JsonIndexQuickStart.java
delete mode 100644
pinot-tools/src/main/java/org/apache/pinot/tools/MultistageEngineQuickStart.java
delete mode 100644
pinot-tools/src/main/java/org/apache/pinot/tools/OfflineComplexTypeHandlingQuickStart.java
delete mode 100644
pinot-tools/src/main/java/org/apache/pinot/tools/PartialUpsertQuickStart.java
delete mode 100644
pinot-tools/src/main/java/org/apache/pinot/tools/RealtimeComplexTypeHandlingQuickStart.java
delete mode 100644
pinot-tools/src/main/java/org/apache/pinot/tools/RealtimeJsonIndexQuickStart.java
delete mode 100644
pinot-tools/src/main/java/org/apache/pinot/tools/TimestampIndexQuickstart.java
delete mode 100644
pinot-tools/src/main/java/org/apache/pinot/tools/UpsertJsonQuickStart.java
delete mode 100644
pinot-tools/src/main/java/org/apache/pinot/tools/UpsertQuickStart.java
rename
pinot-tools/src/main/resources/examples/batch/ssb/customer/{IngestionJobSpec.yaml
=> ingestionJobSpec.yaml} (100%)
rename
pinot-tools/src/main/resources/examples/batch/ssb/dates/{IngestionJobSpec.yaml
=> ingestionJobSpec.yaml} (100%)
rename
pinot-tools/src/main/resources/examples/batch/ssb/lineorder/{IngestionJobSpec.yaml
=> ingestionJobSpec.yaml} (100%)
rename
pinot-tools/src/main/resources/examples/batch/ssb/part/{IngestionJobSpec.yaml
=> ingestionJobSpec.yaml} (100%)
rename
pinot-tools/src/main/resources/examples/batch/ssb/supplier/{IngestionJobSpec.yaml
=> ingestionJobSpec.yaml} (100%)
create mode 100644
pinot-tools/src/test/java/org/apache/pinot/tools/BaseQuickStartTest.java
create mode 100644
pinot-tools/src/test/java/org/apache/pinot/tools/admin/command/QuickStartTest.java
delete mode 100644
pinot-tools/src/test/java/org/apache/pinot/tools/admin/command/TestQuickStartCommand.java
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
