This is an automated email from the ASF dual-hosted git repository.
xiangfu0 pushed a change to branch xiangfu0/data-3221-12-parallel-value-pruning
in repository https://gitbox.apache.org/repos/asf/pinot.git
discard 785926d7367 DATA-3221 (14): prune values across the query executor
omit 9a696892abe DATA-3221 (13): prune segments from column metadata
instead of materializing them
omit 1bad6fdfc20 DATA-3221 (12): make the new column-name accessor sorted
and null-safe
omit 187be1c87b4 DATA-3221 (11): list physical columns without building the
segment schema
omit f0a20da161a DATA-3221 (10): publish the segment column arrays as one
immutable pair
omit 624b8118c09 DATA-3221 (10): hold segment column metadata in sorted
arrays, derive the map on demand
omit 42fc38a698b DATA-3221 (8+9 follow-up): drop the shared shape, pack
bitsPerElement instead
omit a43da48fc71 DATA-3221 (8+9): shared column shape and primitive min/max
in ColumnMetadataImpl
omit 289f83918e5 Slim ColumnMetadataImpl to 72 bytes and derive the
per-segment Schema lazily
omit 19dbb53b3bd Materialize immutable-segment columns lazily behind an
opt-in instance config (default off)
omit fdde54d032b Share segment-derived FieldSpec instances across segments
through a weak interner
omit 0bbde43bad4 Fold PhysicalColumnIndexContainer's IndexTypeMap into a
presence mask plus dense readers
omit 2b73f0db6cf Delegate immutable DataSourceMetadata to ColumnMetadata
instead of snapshotting it
omit b61c9233453 Canonicalize the default null value and intern per-column
strings at metadata parse time
omit fb925d71340 Allocate ColumnMetadataImpl index sizes lazily and skip
the index_map lookup without an index dir
add 568df83b4f4 Add broker startup warmup for the query serve path (SSE)
(#19500)
add d666cb5e1c0 Bump actions/upload-artifact from 4.6.2 to 7.0.1 (#19516)
add 74659c76ff6 Bump actions/checkout from 4.4.0 to 7.0.1 (#19517)
add ceb1fabf4ef Bump actions/setup-python from 5.6.0 to 7.0.0 (#19518)
add f727b44f6ef Bump js-yaml from 4.3.1 to 4.3.2 in
/pinot-controller/src/main/resources (#19522)
add 940284ded36 Add an opt-in controller-copy METADATA push for same-name
segment refreshes (#19488)
add 63b3b1f1b3c Bump org.jetbrains.kotlin:kotlin-bom from 2.4.10 to 2.4.20
(#19524)
add 26f47121863 Broker startup pre-connect: remove the straggler-grace
window (#19519)
add f2800272e44 docs: add RELEASE_POLICY.md for versioning and upgrades
#11642 (#19451)
add c375b713cb7 Parallelize value-based segment pruning
This update added new revisions after undoing existing revisions.
That is to say, some revisions that were in the old version of the
branch are not in the new version. This situation occurs
when a user --force pushes a change and generates a repository
containing something like this:
* -- * -- B -- O -- O -- O (785926d7367)
\
N -- N -- N
refs/heads/xiangfu0/data-3221-12-parallel-value-pruning (c375b713cb7)
You should already have received notification emails for all of the O
revisions, and so the following emails describe only the N revisions
from the common base, B.
Any revisions marked "omit" are not gone; other references still
refer to them. Any revisions marked "discard" are gone forever.
No new revisions were added by this update.
Summary of changes:
.github/workflows/build-all-pinot-docker-image.yml | 8 +-
.../workflows/build-pinot-base-docker-image.yml | 4 +-
.github/workflows/build-pinot-docker-image.yml | 2 +-
.github/workflows/build-superset-docker-image.yml | 2 +-
.../nightly-build-all-pinot-docker-image.yml | 8 +-
...htly-build-multi-arch-superset-docker-image.yml | 2 +-
.github/workflows/pinot_compatibility_checks.yml | 8 +-
.github/workflows/pinot_compatibility_tests.yml | 4 +-
.github/workflows/pinot_integration_tests.yml | 4 +-
.../pinot_java11_client_compatibility.yml | 2 +-
...ulti_stage_query_engine_compatibility_tests.yml | 4 +-
.github/workflows/pinot_quickstart_tests.yml | 2 +-
.github/workflows/pinot_unit_tests.yml | 6 +-
.github/workflows/pinot_vuln_check.yml | 2 +-
.github/workflows/pr-flow-checks.yml | 4 +-
.github/workflows/pr-flow.yml | 10 +-
README.md | 1 +
RELEASE_POLICY.md | 126 +++
.../broker/broker/helix/BaseBrokerStarter.java | 148 +++-
.../requesthandler/BrokerRequestHandler.java | 27 +
.../BrokerRequestHandlerDelegate.java | 8 +
.../broker/requesthandler/BrokerWarmupConfig.java | 60 ++
.../broker/requesthandler/ServerPreConnector.java | 72 +-
.../SingleConnectionBrokerRequestHandler.java | 536 ++++++++++++
.../broker/broker/helix/BrokerWarmupGateTest.java | 50 ++
.../broker/requesthandler/BrokerWarmupTest.java | 454 ++++++++++
.../requesthandler/ServerPreConnectorTest.java | 106 ++-
.../metadata/segment/SegmentZKMetadataUtils.java | 10 +-
.../apache/pinot/common/metrics/BrokerGauge.java | 11 +
.../apache/pinot/common/metrics/BrokerMetrics.java | 7 +
.../apache/pinot/common/metrics/BrokerTimer.java | 3 +
.../src/main/resources/package-lock.json | 6 +-
.../controller/utils/SegmentMetadataMockUtils.java | 27 +-
.../query/pruner/ColumnValueSegmentPruner.java | 29 +-
.../core/query/pruner/ValueBasedSegmentPruner.java | 19 +-
.../query/pruner/ColumnValueSegmentPrunerTest.java | 111 ++-
.../tests/BrokerStartupWarmupIntegrationTest.java | 142 ++++
.../tasks/BaseSingleSegmentConversionExecutor.java | 104 ++-
.../plugin/minion/tasks/BaseTaskExecutor.java | 45 +-
.../minion/tasks/SegmentConversionUtils.java | 29 +-
.../BaseSingleSegmentConversionExecutorTest.java | 281 ++++++-
.../dedup/BasePartitionDedupMetadataManager.java | 2 +-
.../indexsegment/immutable/ColumnMaterializer.java | 125 ---
.../indexsegment/immutable/EmptyIndexSegment.java | 6 +-
.../immutable/ImmutableSegmentImpl.java | 240 +-----
.../immutable/ImmutableSegmentLoader.java | 98 +--
.../immutable/PhysicalColumnNames.java | 103 ---
.../index/column/PhysicalColumnIndexContainer.java | 157 ++--
.../index/datasource/ImmutableDataSource.java | 61 +-
.../segment/index/loader/ForwardIndexHandler.java | 2 +-
.../segment/index/loader/IndexLoadingConfig.java | 20 +-
.../ColumnMinMaxValueGenerator.java | 70 +-
.../defaultcolumn/BaseDefaultColumnHandler.java | 2 +-
.../LegacyRawValueInvertedIndexCleanup.java | 2 +-
.../segment/index/map/ImmutableMapDataSource.java | 47 +-
.../readers/PinotSegmentColumnReaderImpl.java | 5 +-
.../PartitionIdVirtualColumnProvider.java | 13 +-
.../VirtualColumnProviderFactory.java | 34 +-
.../startree/v2/store/StarTreeIndexContainer.java | 11 +-
.../startree/v2/store/StarTreeLoaderUtils.java | 11 +-
.../upsert/BasePartitionUpsertMetadataManager.java | 2 +-
...apPartitionDedupMetadataManagerWithTTLTest.java | 5 +-
.../local/dedup/DedupWithTimestampColumnTest.java | 7 +-
.../immutable/ColumnMaterializerTest.java | 124 ---
.../immutable/EmptyIndexSegmentTest.java | 36 -
.../immutable/ImmutableSegmentImplTest.java | 411 +--------
.../immutable/MockSegmentMetadata.java | 55 --
.../segment/index/SegmentMetadataImplTest.java | 536 +-----------
.../column/PhysicalColumnIndexContainerTest.java | 412 +--------
.../index/datasource/ImmutableDataSourceTest.java | 179 ----
.../index/loader/IndexLoadingConfigTest.java | 14 -
.../local/segment/index/loader/LoaderTest.java | 84 --
.../index/map/ImmutableMapDataSourceTest.java | 94 ---
...ertMetadataManagerForConsistentDeletesTest.java | 5 +-
...rrentMapPartitionUpsertMetadataManagerTest.java | 10 +-
.../apache/pinot/segment/spi/ColumnMetadata.java | 10 -
.../org/apache/pinot/segment/spi/IndexSegment.java | 13 -
.../apache/pinot/segment/spi/SegmentMetadata.java | 80 +-
.../pinot/segment/spi/index/FieldIndexConfigs.java | 18 -
.../spi/index/metadata/ColumnMetadataImpl.java | 532 +++---------
.../spi/index/metadata/SegmentMetadataImpl.java | 334 +-------
.../spi/index/metadata/SortedStringArraySet.java | 313 -------
.../segment/spi/SegmentMetadataDefaultsTest.java | 87 --
.../spi/index/metadata/ColumnMetadataImplTest.java | 929 +--------------------
.../index/metadata/SortedStringArraySetTest.java | 153 ----
.../resources/SegmentCompressionStatsReader.java | 2 +-
.../pinot/server/api/resources/TablesResource.java | 10 +-
.../helix/HelixInstanceDataManagerConfig.java | 10 -
.../SegmentCompressionStatsReaderTest.java | 8 +-
.../helix/HelixInstanceDataManagerConfigTest.java | 12 -
.../config/instance/InstanceDataManagerConfig.java | 7 -
.../apache/pinot/spi/utils/CommonConstants.java | 26 +
pom.xml | 2 +-
93 files changed, 2774 insertions(+), 5239 deletions(-)
create mode 100644 RELEASE_POLICY.md
create mode 100644
pinot-broker/src/main/java/org/apache/pinot/broker/requesthandler/BrokerWarmupConfig.java
create mode 100644
pinot-broker/src/test/java/org/apache/pinot/broker/broker/helix/BrokerWarmupGateTest.java
create mode 100644
pinot-broker/src/test/java/org/apache/pinot/broker/requesthandler/BrokerWarmupTest.java
create mode 100644
pinot-integration-tests/src/test/java/org/apache/pinot/integration/tests/BrokerStartupWarmupIntegrationTest.java
delete mode 100644
pinot-segment-local/src/main/java/org/apache/pinot/segment/local/indexsegment/immutable/ColumnMaterializer.java
delete mode 100644
pinot-segment-local/src/main/java/org/apache/pinot/segment/local/indexsegment/immutable/PhysicalColumnNames.java
delete mode 100644
pinot-segment-local/src/test/java/org/apache/pinot/segment/local/indexsegment/immutable/ColumnMaterializerTest.java
delete mode 100644
pinot-segment-local/src/test/java/org/apache/pinot/segment/local/indexsegment/immutable/MockSegmentMetadata.java
delete mode 100644
pinot-segment-local/src/test/java/org/apache/pinot/segment/local/segment/index/datasource/ImmutableDataSourceTest.java
delete mode 100644
pinot-segment-local/src/test/java/org/apache/pinot/segment/local/segment/index/map/ImmutableMapDataSourceTest.java
delete mode 100644
pinot-segment-spi/src/main/java/org/apache/pinot/segment/spi/index/metadata/SortedStringArraySet.java
delete mode 100644
pinot-segment-spi/src/test/java/org/apache/pinot/segment/spi/SegmentMetadataDefaultsTest.java
delete mode 100644
pinot-segment-spi/src/test/java/org/apache/pinot/segment/spi/index/metadata/SortedStringArraySetTest.java
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
