xiangfu0 opened a new pull request, #19539:
URL: https://github.com/apache/pinot/pull/19539
Broker partition pruning currently revisits the filter tree and rehashes the
same literals for every candidate segment. This change prepares visited
predicates and reuses literal partition IDs within each pruning call, while
checking every segment's current partition set.
Reuse is keyed by partition-function implementation, name, partition count,
normalizer and original constructor configuration. Retaining the original
configuration also covers plugins that inherit the default null
`getFunctionConfig()` implementation. AND/OR/IN keep their evaluation order and
short-circuit behavior. Empty/singleton inputs use uncached evaluation; mixed
configurations reuse a lookup probe without allocating one key per segment.
## Benchmark
Independently packaged baseline `5771d6acea60cd72738116835111cf7c49965373`
and candidate `8b15fbc6561e39dca7c17d14aa172657a63913db`, with identical
benchmark source and bytecode. `BenchmarkPartitionPruning` measures the
**complete prune call**, including the returned set's allocation. It cycles
through 32 prebuilt requests; parsing, real ZNRecord metadata loading, and
independent result-set checks run outside measurement. Homogeneous metadata
uses Murmur/128; interleaved metadata alternates Murmur/128 and Murmur/256.
Queries combine the measured predicate with two unrelated time filters.
Apple M4 Pro, 24 GiB RAM, Homebrew OpenJDK 25.0.4, JMH 1.37, one benchmark
thread, fixed 1 GiB heap, `-XX:ActiveProcessorCount=4`, GC profiler. Serial
baseline/PR/PR/baseline order: two independent forks per case per revision,
each with five 1-second warmups and five 1-second measurements. For
4,096-segment EQ and IN128, a supplemental ABBA comparison adds two forks per
revision with five 2-second warmups and five 1-second measurements, for four
forks per revision on those two cases. Every measured iteration is retained;
preliminary pilots are excluded.
The following are means across the retained iterations. This
shared-workstation benchmark demonstrates local pruning time and allocation;
production broker CPU, QPS, and end-to-end latency require workload replay.
| Metadata | Segments | Predicate | Baseline µs/call | PR µs/call | Time
reduction | Allocated B/call, baseline → PR |
|---|---:|---|---:|---:|---:|---:|
| homogeneous | 1 | EQ | 0.031 | 0.030 | 2.6% | 128 → 128 |
| homogeneous | 1 | IN_128 | 0.902 | 0.884 | 2.0% | 2503 → 2503 |
| homogeneous | 1 | UNRELATED | 0.033 | 0.031 | 4.2% | 208 → 208 |
| homogeneous | 256 | EQ | 6.988 | 2.541 | 63.6% | 8432 → 680 |
| homogeneous | 256 | IN_128 | 297.725 | 40.575 | 86.4% | 671551 → 12356 |
| homogeneous | 256 | IN_16 | 62.454 | 8.585 | 86.3% | 125316 → 2528 |
| homogeneous | 256 | UNRELATED | 7.295 | 5.111 | 29.9% | 12416 → 12800 |
| homogeneous | 4096 | EQ | 117.634 | 59.751 | 49.2% | 132689 → 2088 |
| homogeneous | 4096 | IN_128 | 5597.190 | 599.838 | 89.3% | 10743264 →
120997 |
| homogeneous | 4096 | IN_16 | 933.724 | 178.656 | 80.9% | 2003928 → 24705 |
| homogeneous | 4096 | UNRELATED | 130.144 | 86.770 | 33.3% | 196801 →
197185 |
| interleaved | 256 | EQ | 6.430 | 7.225 | -12.4% | 8416 → 1000 |
| interleaved | 256 | IN_128 | 363.839 | 57.059 | 84.3% | 751245 → 17301 |
| interleaved | 256 | UNRELATED | 7.467 | 6.799 | 9.0% | 12416 → 13072 |
**Tradeoffs and measurement limits:** Interleaved EQ is 12.4% slower (6.430
→ 7.225 µs) while allocating 88.1% fewer bytes. Queries without a partition
predicate allocate an extra 384 B/call for homogeneous metadata, or 656 B/call
in the interleaved case. Singleton controls are approximately unchanged; their
small timing differences should not be read as wins. The baseline's
large-segment EQ/IN128 cases vary between JVM forks despite stable iterations
within each fork; the per-fork means below expose that variation rather than
implying precise confidence in the pooled percentages. The first run began with
a host load average of 19.25 after packaging. This task ran no concurrent
builds or benchmark JVMs.
<details>
<summary>Per-fork means (µs per complete pruning call)</summary>
| Metadata | Segments | Predicate | Baseline forks A / B / C / D | PR forks
A / B / C / D |
|---|---:|---|---|---|
| homogeneous | 1 | EQ | 0.030 / 0.032 | 0.030 / 0.030 |
| homogeneous | 1 | IN_128 | 0.894 / 0.910 | 0.881 / 0.887 |
| homogeneous | 1 | UNRELATED | 0.033 / 0.032 | 0.032 / 0.031 |
| homogeneous | 256 | EQ | 7.267 / 6.709 | 2.523 / 2.558 |
| homogeneous | 256 | IN_128 | 312.676 / 282.774 | 40.851 / 40.299 |
| homogeneous | 256 | IN_16 | 62.223 / 62.686 | 8.478 / 8.692 |
| homogeneous | 256 | UNRELATED | 7.226 / 7.364 | 4.822 / 5.400 |
| homogeneous | 4096 | EQ | 95.980 / 124.076 / 136.227 / 114.252 | 60.270 /
59.315 / 59.386 / 60.034 |
| homogeneous | 4096 | IN_128 | 3737.818 / 5998.606 / 6369.155 / 6283.183 |
597.865 / 604.576 / 599.421 / 597.491 |
| homogeneous | 4096 | IN_16 | 934.902 / 932.546 | 174.327 / 182.985 |
| homogeneous | 4096 | UNRELATED | 129.133 / 131.155 | 83.924 / 89.615 |
| interleaved | 256 | EQ | 6.601 / 6.259 | 7.224 / 7.225 |
| interleaved | 256 | IN_128 | 357.585 / 370.093 | 56.524 / 57.594 |
| interleaved | 256 | UNRELATED | 7.575 / 7.360 | 6.803 / 6.796 |
</details>
## Reproduce
Copy `BenchmarkPartitionPruning.java` into a separate worktree at the
baseline commit. Package each worktree independently with JDK 25:
```sh
./mvnw -pl pinot-perf -am package -Ppinot-fastdev -DskipTests \
-Dskip.npm -Dskip.installnodenpm -Denforcer.skip \
-Dassembly.skipAssembly=true -Dmaven.javadoc.skip=true
```
Run each group in each worktree in baseline/PR/PR/baseline order, with
distinct output filenames. Verify the packaged broker/perf jars match their
module target jars before running.
```sh
"$JAVA_HOME/bin/java" -Xmx1g \
-cp 'pinot-perf/target/pinot-perf-pkg/lib/*' org.openjdk.jmh.Main \
BenchmarkPartitionPruning -wi 5 -i 5 -w 1s -r 1s -f 1 -t 1 \
-p _metadata=HOMOGENEOUS -p _numSegments=256,4096 \
-p _predicate=EQ,IN_16,IN_128,UNRELATED \
-prof gc -foe true -jvmArgs '-Xms1g -Xmx1g -XX:ActiveProcessorCount=4' \
-rf json -rff results.json
```
For singleton controls, use `_numSegments=1` and
`_predicate=EQ,IN_128,UNRELATED`. For interleaved controls, use
`_metadata=INTERLEAVED`, `_numSegments=256`, and
`_predicate=EQ,IN_128,UNRELATED`. Repeat the additional ABBA comparison for
`_metadata=HOMOGENEOUS`, `_numSegments=4096`, `_predicate=EQ,IN_128`, and `-w
2s`.
## Validation
- 22 tests passed, zero failures/errors/skips:
`SinglePartitionColumnSegmentPrunerTest` (11 cases), `SegmentPrunerTest` (8),
`SegmentPartitionMetadataManagerTest` (3).
- Hash-count assertions cover reuse across distinct function objects and
interleaved configurations. Tests also cover hidden plugin config, normalizers,
raw-byte hashing, lazy errors in both cached/uncached paths, missing metadata,
refresh, and concurrent-query isolation.
- Broker/perf `spotless:apply`, `license:format`, `checkstyle:check`, and
`license:check` passed.
- Warning/deprecation-enabled reactor `test-compile` passed, with no
warnings in changed sources.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: [email protected]
For queries about this service, please contact Infrastructure at:
[email protected]
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
