This is an automated email from the ASF dual-hosted git repository.
xiangfu0 pushed a change to branch xiangfu0/codex2/ratis-minion-execution
in repository https://gitbox.apache.org/repos/asf/pinot.git
omit 4d1ceb46bee Add fenced native minion scheduling and offline execution
omit ee0084a4450 Add durable Ratis task ledger and minion metadata adapter
omit 28d71a769bb Add authenticated native table-size observations for Ratis
omit 701da8dda10 Expose Ratis controller metadata REST APIs and UI
omit c86ba95b6f4 Wire Ratis into normal starters with query listeners
disabled
omit 859ff5939d0 Integrate native serving permits and immutable query
deadlines
omit 14e51bcd4f2 Add independent native offline participant runtime and
query identity fences
omit 4ca81de19aa Activate Ratis protocol generation on every build platform
omit 8587dd338ac Add independent Ratis metadata and membership foundation
add ec462d40c27 Mark deprecated table config properties with
@Deprecated(since, forRemoval = true) (#18411)
add 16ef89030be Remove unused Prometheus test resource helper (#19932)
add 2dc0baa9a8b Defer a new segment only while its replicas are still
coming up (#19901)
add 34fee4dcf52 Remove unused InMemoryColumnReader test fixture (#19929)
add ad850480ef2 Revert "Map DECIMAL(p, 0) to a type that holds all its
values in the multi-stage engine (#19790)" (#19949)
add 1ea404bda57 Remove unused upsert compact merge executor test
scaffolding (#19931)
add 14124f31626 Use fast LZ4 for .tar.lz4 archives (#19946)
add 607576c7a8b Fix endless range index rebuild for MV columns with
rangeIndexVersion 2 (#19936)
add 88c3262d10a Support chained, forward-index-disabled, and
disabled-source derived columns (#19581)
add 0d02adb733b Exercise JDBC invalid decimal parsing on a valid row
(#19935)
add 89b84bad2e2 Resume a pauseless commit start after its IdealState
update fails (#19740)
add 8170af55d32 Bump follow-redirects in
/pinot-controller/src/main/resources (#19979)
add 1f977d977f8 Bump org.ow2.asm:asm from 9.11 to 9.11.1 (#19971)
add b1d6f042e3f Add onlyIfNeeded option in segment reload (#19861)
add 3007f88d181 Bump vite from 8.0.16 to 8.3.4 in
/pinot-controller/src/main/resources (#19989)
add 71299210708 Remove redundant snake_case scalar function aliases
(#19984)
add e8647425026 Remove the warning on multiple PinotMetricsFactory classes
(#19985)
add 2630b03651d Add null-ignoring variants of array union, concat and push
scalar functions (#19981)
add 881a1f74dc6 Honor nullableParameters for scalar functions in the
single-stage engine (#19983)
add edc48dff392 Let a retried validation-manager IdealState update add the
segments it created (#19739)
add be94fc67e47 Replace approximateFunctionApplied with
approximatedResult, set when a SMART function turns approximate (#19972)
add beb891d2282 Keep partitioned workers for window and WITHIN GROUP sorts
over hash exchanges (#19954)
add e574c52eff6 Push ARRAY_TO_MV filters below joins again in MSE (#19973)
add 30cc4973142 Bump com.github.luben:zstd-jni from 1.5.7-23 to 1.5.7-24
(#19994)
add d6146aad3cd Bump com.fasterxml.jackson:jackson-bom from 2.22.3 to
2.22.4 (#19993)
add 0b053a1f3c2 Fix stale realtime Lucene searchable document counts
(#19945)
add 2877723da38 Repair ERROR replicas on pauseless dedup and
partial-upsert tables (#19909)
add 9fbd72ae42a Run the MSE approximate aggregate rewrite after
CASE-to-FILTER conversion (#19974)
add 310d7604fcd Materialize non-order-by columns late on the sorted
order-by path (#19879)
add 24c520be6e0 Make FixedBitMVForwardIndexReader.getNumValuesMV advance
the reader context (#19992)
add 736f27c2fab Add independent Ratis metadata and membership foundation
add 4357538d18e Activate Ratis protocol generation on every build platform
add 8b8525f550d Add independent native offline participant runtime and
query identity fences
add ada424e4a7b Preserve terminal native participant fencing diagnostics
add 4ffa65d0dce Integrate native serving permits and immutable query
deadlines
add 1f3653bdf87 Wire Ratis into normal starters with query listeners
disabled
add 3bc62648f8d Expose Ratis controller metadata REST APIs and UI
add e095ccdc7c4 Add authenticated native table-size observations for Ratis
add 32cf7d22578 Add durable Ratis task ledger and minion metadata adapter
add 841edf3f374 Add fenced native minion scheduling and offline execution
This update added new revisions after undoing existing revisions.
That is to say, some revisions that were in the old version of the
branch are not in the new version. This situation occurs
when a user --force pushes a change and generates a repository
containing something like this:
* -- * -- B -- O -- O -- O (4d1ceb46bee)
\
N -- N -- N refs/heads/xiangfu0/codex2/ratis-minion-execution
(841edf3f374)
You should already have received notification emails for all of the O
revisions, and so the following emails describe only the N revisions
from the common base, B.
Any revisions marked "omit" are not gone; other references still
refer to them. Any revisions marked "discard" are gone forever.
No new revisions were added by this update.
Summary of changes:
.../apache/pinot/broker/querylog/QueryLogger.java | 4 +-
.../BaseSingleStageBrokerRequestHandler.java | 14 +-
.../MultiStageBrokerRequestHandler.java | 12 +-
.../SegmentPartitionMetadataManager.java | 65 +-
.../pinot/broker/querylog/QueryLoggerTest.java | 2 +-
.../SegmentPartitionMetadataManagerTest.java | 75 ++
.../apache/pinot/client/PinotResultSetTest.java | 13 +-
.../compression/Lz4CompressorStreamProvider.java | 138 +++
.../apache/pinot/common/datatable/DataTable.java | 7 +-
.../function/scalar/ArithmeticFunctions.java | 2 +-
.../common/function/scalar/ArrayFunctions.java | 188 +++++
.../common/function/scalar/IpAddressFunctions.java | 10 +-
.../common/function/scalar/StringFunctions.java | 30 +-
.../common/messages/SegmentReloadMessage.java | 22 +
.../apache/pinot/common/metrics/BrokerMeter.java | 9 +-
.../pinot/common/response/BrokerResponse.java | 17 +-
.../response/broker/BrokerResponseNative.java | 17 +-
.../response/broker/BrokerResponseNativeV2.java | 29 +-
.../response/broker/CursorResponseNative.java | 3 +-
.../pinot/common/utils/TarCompressionUtils.java | 14 +-
.../common/utils/config/QueryOptionsUtils.java | 8 +
...s.compress.compressors.CompressorStreamProvider | 12 +-
.../function/scalar/StringFunctionsTest.java | 6 +
.../common/messages/SegmentReloadMessageTest.java | 13 +
.../prometheus/PinotPrometheusMetricsTest.java | 12 -
.../response/broker/BrokerResponseNativeTest.java | 13 +
.../common/utils/TarCompressionUtilsTest.java | 72 ++
.../api/resources/PinotTableReloadResource.java | 11 +-
.../helix/core/PinotHelixResourceManager.java | 147 +++-
.../realtime/BlockingSegmentCompletionFSM.java | 5 +-
.../realtime/PinotLLCRealtimeSegmentManager.java | 423 ++++++++--
.../services/PinotTableReloadService.java | 24 +-
.../RealtimeSegmentValidationManager.java | 3 +-
.../src/main/resources/package-lock.json | 400 ++++-----
pinot-controller/src/main/resources/package.json | 2 +-
.../controller/helix/core/SegmentResetTest.java | 201 +++++
.../PinotLLCRealtimeSegmentManagerTest.java | 924 ++++++++++++++++++++-
.../services/PinotTableReloadServiceTest.java | 94 ++-
.../core/data/manager/BaseTableDataManager.java | 169 +++-
.../core/data/manager/InstanceDataManager.java | 29 +-
.../blocks/results/AggregationResultsBlock.java | 44 +-
.../blocks/results/GroupByResultsBlock.java | 6 +
.../query/LinearSelectionOrderByOperator.java | 216 ++++-
.../SelectionPartiallyOrderedByDescOperation.java | 11 +-
.../SelectionPartiallyOrderedByLinearOperator.java | 11 +-
.../function/ScalarTransformFunctionWrapper.java | 250 +++++-
.../apache/pinot/core/plan/SelectionPlanNode.java | 27 +-
.../aggregation/function/AggregationFunction.java | 12 +
.../function/AggregationFunctionUtils.java | 14 +
...istinctCountSmartSketchAggregationFunction.java | 15 +
.../DistinctCountSmartHLLAggregationFunction.java | 1 +
...stinctCountSmartHLLPlusAggregationFunction.java | 1 +
.../DistinctCountSmartULLAggregationFunction.java | 1 +
.../PercentileSmartTDigestAggregationFunction.java | 11 +
.../core/query/reduce/BrokerReduceService.java | 6 +
.../query/reduce/ExecutionStatsAggregator.java | 6 +
.../core/data/function/ArrayFunctionsTest.java | 92 ++
.../data/manager/BaseTableDataManagerTest.java | 198 +++++
.../results/AggregationResultsBlockTest.java | 85 ++
.../ScalarTransformFunctionWrapperTest.java | 65 ++
.../SmartFunctionApproximatedResultTest.java | 83 ++
.../core/query/reduce/BrokerReduceServiceTest.java | 52 ++
.../query/reduce/ExecutionStatsAggregatorTest.java | 16 +
...nnerSegmentSelectionSingleValueQueriesTest.java | 94 ++-
.../queries/NullHandlingEnabledQueriesTest.java | 58 ++
...ApproximateFunctionOverrideIntegrationTest.java | 147 +++-
.../PauselessDedupErrorReplicaRepairTest.java | 281 +++++++
.../realtime/utils/PauselessRealtimeTestUtils.java | 28 +-
.../resources/udf-test-results/all-functions.yaml | 68 ++
.../UpsertCompactMergeTaskExecutorTest.java | 47 +-
.../PinotApproximateAggregateRewriteRule.java | 15 +-
.../rel/rules/PinotEvaluateLiteralRule.java | 7 +-
.../calcite/rel/rules/PinotQueryRuleSets.java | 11 +-
.../pinot/calcite/rel/rules/PinotRuleUtils.java | 18 +-
.../pinot/calcite/sql/fun/PinotOperatorTable.java | 32 +-
.../apache/pinot/query/context/PlannerContext.java | 14 -
.../planner/logical/RelToPlanNodeConverter.java | 40 +-
.../query/planner/logical/RexExpressionUtils.java | 44 +-
.../planner/physical/DispatchablePlanVisitor.java | 21 +-
.../physical/v2/PRelToPlanNodeConverter.java | 8 +-
.../validation/ArrayToMvValidationVisitor.java | 3 +-
.../calcite/rel/rules/PinotRuleUtilsTest.java | 19 +
.../query/ApproximateAggregateRewriteTest.java | 44 +
.../apache/pinot/query/QueryCompilationTest.java | 91 ++
.../logical/RelToPlanNodeConverterTest.java | 53 +-
.../planner/logical/RexExpressionUtilsTest.java | 36 +-
.../physical/DispatchablePlanVisitorTest.java | 93 +++
.../physical/v2/PRelToPlanNodeConverterTest.java | 28 -
.../src/test/resources/queries/AggregatePlans.json | 26 -
.../resources/queries/ExplainPhysicalPlans.json | 18 +
.../src/test/resources/queries/JoinPlans.json | 17 +
.../resources/queries/ValidationErrorPlan.json | 5 +
.../query/runtime/operator/AggregateOperator.java | 19 +-
.../pinot/query/runtime/operator/LeafOperator.java | 8 +-
.../query/runtime/operator/MultiStageOperator.java | 1 +
.../runtime/operator/operands/FilterOperand.java | 40 +-
.../runtime/operator/AggregateOperatorTest.java | 36 +
.../query/runtime/operator/FilterOperatorTest.java | 36 -
.../query/runtime/operator/LeafOperatorTest.java | 24 +
.../testutils/MockInstanceDataManagerFactory.java | 7 +-
.../MockInstanceDataManagerFactoryTest.java | 76 ++
.../src/test/resources/queries/Joins.json | 56 ++
.../src/test/resources/queries/NumericTypes.json | 127 ---
.../local/data/manager/TableDataManager.java | 32 +-
.../invertedindex/RealtimeLuceneTextIndex.java | 22 +-
...IndexAndDictionaryBasedForwardIndexCreator.java | 17 +-
.../segment/index/loader/SegmentPreProcessor.java | 2 +-
.../defaultcolumn/BaseDefaultColumnHandler.java | 586 +++++++++++--
.../defaultcolumn/DefaultColumnHandlerFactory.java | 15 +-
.../defaultcolumn/V1DefaultColumnHandler.java | 7 +
.../defaultcolumn/V3DefaultColumnHandler.java | 26 +-
.../loader/invertedindex/RangeIndexHandler.java | 22 +-
.../forward/FixedBitMVForwardIndexReader.java | 5 +
.../pinot/segment/local/utils/ReloadJobStatus.java | 20 +
.../local/utils/ServerReloadJobStatusCache.java | 12 +
.../invertedindex/LuceneMutableTextIndexTest.java | 40 +-
.../index/forward/FixedBitMVForwardIndexTest.java | 69 ++
.../index/loader/SegmentPreProcessorTest.java | 509 +++++++++++-
.../RangeIndexHandlerMultiValueTest.java | 142 ++++
.../segment/readers/InMemoryColumnReader.java | 176 ----
.../utils/ServerReloadJobStatusCacheTest.java | 20 +
.../pinot/segment/spi/AggregationFunctionType.java | 111 +--
.../api/resources/ControllerJobStatusResource.java | 19 +-
.../server/starter/NativeInstanceDataManager.java | 8 +-
.../server/starter/NativeServerParticipant.java | 11 +-
.../starter/helix/HelixInstanceDataManager.java | 14 +-
.../helix/SegmentMessageHandlerFactory.java | 10 +-
.../resources/ControllerJobStatusResourceTest.java | 97 +++
.../starter/NativeServerParticipantTest.java | 3 +
.../apache/pinot/spi/config/table/DedupConfig.java | 6 +-
.../apache/pinot/spi/config/table/FieldConfig.java | 3 +-
.../pinot/spi/config/table/IndexingConfig.java | 6 +-
.../pinot/spi/config/table/RoutingConfig.java | 2 +
.../SegmentsValidationAndRetentionConfig.java | 10 +-
.../pinot/spi/config/table/UpsertConfig.java | 9 +-
.../InstanceReplicaGroupPartitionConfig.java | 2 +
.../apache/pinot/spi/metrics/PinotMetricUtils.java | 7 +-
.../apache/pinot/spi/utils/CommonConstants.java | 10 +
pom.xml | 6 +-
139 files changed, 6772 insertions(+), 1542 deletions(-)
create mode 100644
pinot-common/src/main/java/org/apache/pinot/common/compression/Lz4CompressorStreamProvider.java
copy pinot-clients/pinot-jdbc-client/src/main/resources/version.properties =>
pinot-common/src/main/resources/META-INF/services/org.apache.commons.compress.compressors.CompressorStreamProvider
(63%)
create mode 100644
pinot-core/src/test/java/org/apache/pinot/core/operator/blocks/results/AggregationResultsBlockTest.java
create mode 100644
pinot-core/src/test/java/org/apache/pinot/core/query/aggregation/function/SmartFunctionApproximatedResultTest.java
create mode 100644
pinot-integration-tests/src/test/java/org/apache/pinot/integration/tests/PauselessDedupErrorReplicaRepairTest.java
create mode 100644
pinot-query-planner/src/test/java/org/apache/pinot/query/planner/physical/DispatchablePlanVisitorTest.java
create mode 100644
pinot-query-runtime/src/test/java/org/apache/pinot/query/testutils/MockInstanceDataManagerFactoryTest.java
create mode 100644
pinot-segment-local/src/test/java/org/apache/pinot/segment/local/segment/index/loader/invertedindex/RangeIndexHandlerMultiValueTest.java
delete mode 100644
pinot-segment-local/src/test/java/org/apache/pinot/segment/local/segment/readers/InMemoryColumnReader.java
create mode 100644
pinot-server/src/test/java/org/apache/pinot/server/api/resources/ControllerJobStatusResourceTest.java
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
