This is an automated email from the ASF dual-hosted git repository.
xiangfu0 pushed a change to branch xiangfu0/data-3221-11-metadata-only-pruning
in repository https://gitbox.apache.org/repos/asf/pinot.git
discard 68c65e2ea9e DATA-3221 (13): prune segments from column metadata
instead of materializing them
discard 2080eae7e05 DATA-3221 (12): make the new column-name accessor sorted
and null-safe
discard 794c4282ea1 DATA-3221 (11): list physical columns without building the
segment schema
discard 8de113ef3d9 DATA-3221 (10): publish the segment column arrays as one
immutable pair
discard 2d33e7e23d6 DATA-3221 (10): hold segment column metadata in sorted
arrays, derive the map on demand
discard 54d95310c94 DATA-3221 (8+9 follow-up): drop the shared shape, pack
bitsPerElement instead
discard f61001298f4 DATA-3221 (8+9): shared column shape and primitive min/max
in ColumnMetadataImpl
discard e85ba680462 Slim ColumnMetadataImpl to 72 bytes and derive the
per-segment Schema lazily
discard 4e4c5972832 Materialize immutable-segment columns lazily behind an
opt-in instance config (default off)
discard 1d3e9d20e95 Share segment-derived FieldSpec instances across segments
through a weak interner
discard 19ae462a005 Fold PhysicalColumnIndexContainer's IndexTypeMap into a
presence mask plus dense readers
discard 3eb0c08b517 Delegate immutable DataSourceMetadata to ColumnMetadata
instead of snapshotting it
discard 987446c09f6 Canonicalize the default null value and intern per-column
strings at metadata parse time
omit 22186c0c6a1 Grow index-size storage lazily with LongArrayList
omit fb925d71340 Allocate ColumnMetadataImpl index sizes lazily and skip
the index_map lookup without an index dir
add 568df83b4f4 Add broker startup warmup for the query serve path (SSE)
(#19500)
add d666cb5e1c0 Bump actions/upload-artifact from 4.6.2 to 7.0.1 (#19516)
add 74659c76ff6 Bump actions/checkout from 4.4.0 to 7.0.1 (#19517)
add ceb1fabf4ef Bump actions/setup-python from 5.6.0 to 7.0.0 (#19518)
add f727b44f6ef Bump js-yaml from 4.3.1 to 4.3.2 in
/pinot-controller/src/main/resources (#19522)
add 940284ded36 Add an opt-in controller-copy METADATA push for same-name
segment refreshes (#19488)
add 63b3b1f1b3c Bump org.jetbrains.kotlin:kotlin-bom from 2.4.10 to 2.4.20
(#19524)
add 26f47121863 Broker startup pre-connect: remove the straggler-grace
window (#19519)
add f2800272e44 docs: add RELEASE_POLICY.md for versioning and upgrades
#11642 (#19451)
add 86107957b03 Star-tree offheap build: read dims from sort buffer
instead of re-fetching from segment (#19430)
add 30f5d208a1e Prune values across the query executor instead of on the
calling thread (#19513)
add 718f4100551 Bump org.scala-lang.modules:scala-xml_2.13 from 2.4.0 to
2.5.0 (#19527)
add 372c535f053 Bump org.jline:jline from 4.4.2 to 4.4.3 (#19528)
add b65be1d599c Allocate ColumnMetadataImpl index sizes lazily and skip
the index_map lookup without an index dir (#19480)
add e08955f13fd Canonicalize the default null value and intern per-column
strings at metadata parse time
add 87b1c6a5cb9 Delegate immutable DataSourceMetadata to ColumnMetadata
instead of snapshotting it
add 5d1ebde82c1 Fold PhysicalColumnIndexContainer's IndexTypeMap into a
presence mask plus dense readers
add 865cdc1180d Share segment-derived FieldSpec instances across segments
through a weak interner
add c85c5cf1a46 Materialize immutable-segment columns lazily behind an
opt-in instance config (default off)
add 125c9acfb3f Slim ColumnMetadataImpl to 72 bytes and derive the
per-segment Schema lazily
add 32698b27c6b DATA-3221 (8+9): shared column shape and primitive min/max
in ColumnMetadataImpl
add f12f5cc0ac2 DATA-3221 (8+9 follow-up): drop the shared shape, pack
bitsPerElement instead
add 3023a0eb141 DATA-3221 (10): hold segment column metadata in sorted
arrays, derive the map on demand
add 5eb5311d229 DATA-3221 (10): publish the segment column arrays as one
immutable pair
add 6a35d50822e DATA-3221 (11): list physical columns without building the
segment schema
add 9a5974472d9 DATA-3221 (12): make the new column-name accessor sorted
and null-safe
add 490463e7db9 DATA-3221 (13): prune segments from column metadata
instead of materializing them
This update added new revisions after undoing existing revisions.
That is to say, some revisions that were in the old version of the
branch are not in the new version. This situation occurs
when a user --force pushes a change and generates a repository
containing something like this:
* -- * -- B -- O -- O -- O (68c65e2ea9e)
\
N -- N -- N
refs/heads/xiangfu0/data-3221-11-metadata-only-pruning (490463e7db9)
You should already have received notification emails for all of the O
revisions, and so the following emails describe only the N revisions
from the common base, B.
Any revisions marked "omit" are not gone; other references still
refer to them. Any revisions marked "discard" are gone forever.
No new revisions were added by this update.
Summary of changes:
.github/workflows/build-all-pinot-docker-image.yml | 8 +-
.../workflows/build-pinot-base-docker-image.yml | 4 +-
.github/workflows/build-pinot-docker-image.yml | 2 +-
.github/workflows/build-superset-docker-image.yml | 2 +-
.../nightly-build-all-pinot-docker-image.yml | 8 +-
...htly-build-multi-arch-superset-docker-image.yml | 2 +-
.github/workflows/pinot_compatibility_checks.yml | 8 +-
.github/workflows/pinot_compatibility_tests.yml | 4 +-
.github/workflows/pinot_integration_tests.yml | 4 +-
.../pinot_java11_client_compatibility.yml | 2 +-
...ulti_stage_query_engine_compatibility_tests.yml | 4 +-
.github/workflows/pinot_quickstart_tests.yml | 2 +-
.github/workflows/pinot_unit_tests.yml | 6 +-
.github/workflows/pinot_vuln_check.yml | 2 +-
.github/workflows/pr-flow-checks.yml | 4 +-
.github/workflows/pr-flow.yml | 10 +-
README.md | 1 +
RELEASE_POLICY.md | 126 +++++
.../broker/broker/helix/BaseBrokerStarter.java | 148 +++++-
.../requesthandler/BrokerRequestHandler.java | 27 ++
.../BrokerRequestHandlerDelegate.java | 8 +
.../broker/requesthandler/BrokerWarmupConfig.java | 60 +++
.../broker/requesthandler/ServerPreConnector.java | 72 ++-
.../SingleConnectionBrokerRequestHandler.java | 536 +++++++++++++++++++++
.../broker/broker/helix/BrokerWarmupGateTest.java | 50 ++
.../broker/requesthandler/BrokerWarmupTest.java | 454 +++++++++++++++++
.../requesthandler/ServerPreConnectorTest.java | 106 ++--
.../apache/pinot/common/metrics/BrokerGauge.java | 11 +
.../apache/pinot/common/metrics/BrokerMetrics.java | 7 +
.../apache/pinot/common/metrics/BrokerTimer.java | 3 +
.../src/main/resources/package-lock.json | 6 +-
.../query/pruner/BloomFilterSegmentPruner.java | 99 +---
.../core/query/pruner/ValueBasedSegmentPruner.java | 90 +++-
.../query/pruner/BloomFilterSegmentPrunerTest.java | 136 ++++++
.../query/pruner/ColumnValueSegmentPrunerTest.java | 189 ++++++++
.../tests/BrokerStartupWarmupIntegrationTest.java | 142 ++++++
.../tasks/BaseSingleSegmentConversionExecutor.java | 104 +++-
.../plugin/minion/tasks/BaseTaskExecutor.java | 45 +-
.../minion/tasks/SegmentConversionUtils.java | 29 +-
.../BaseSingleSegmentConversionExecutorTest.java | 281 +++++++++--
.../startree/v2/builder/BaseSingleTreeBuilder.java | 20 +
.../v2/builder/OffHeapSingleTreeBuilder.java | 108 +++--
.../v2/builder/OffHeapSingleTreeBuilderTest.java | 254 ++++++++++
.../apache/pinot/spi/utils/CommonConstants.java | 26 +
pom.xml | 6 +-
45 files changed, 2914 insertions(+), 302 deletions(-)
create mode 100644 RELEASE_POLICY.md
create mode 100644
pinot-broker/src/main/java/org/apache/pinot/broker/requesthandler/BrokerWarmupConfig.java
create mode 100644
pinot-broker/src/test/java/org/apache/pinot/broker/broker/helix/BrokerWarmupGateTest.java
create mode 100644
pinot-broker/src/test/java/org/apache/pinot/broker/requesthandler/BrokerWarmupTest.java
create mode 100644
pinot-integration-tests/src/test/java/org/apache/pinot/integration/tests/BrokerStartupWarmupIntegrationTest.java
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
