viirya opened a new pull request, #7693:
URL: https://github.com/apache/datafusion-comet/pull/7693
## Which issue does this PR close?
Part of #6499. Follow-up to #6883, found while benchmarking #7692.
## Rationale for this change
A local query runs at most as many partitions at once as Spark runs tasks.
Under the current scheduler, the next partition starts only when the first
undelivered one is delivered. So a partition that finishes before its turn
keeps its slot until it is delivered, and one slow partition leaves the slots
behind it idle. Spark instead starts the next task whenever any task ends.
Range partitions are uniform, so this rarely shows with the ranges #6883
admits. Scan partitions are not: their splits differ in size and some read a
file footer. With #7692, collecting a Parquet scan of 320 small partitions on
`local[8]` took about 45% longer locally than with ordinary Comet.
Per-partition timing of that case showed:
- Planning and starting a partition took about 50 µs.
- Running a partition took a median of 340 µs and up to 2.9 ms.
- Finished partitions then waited a mean of 1.4 ms for delivery while
keeping their slots.
- On average only 2.9 of the 8 slots were busy.
## What changes are included in this PR?
- When a partition's task finishes, it drops the partition's stream, which
releases its operators and their memory consumers, and wakes the reader so that
another partition starts. The bound on running partitions now counts only
partitions that have not finished. It still bounds how many partitions share
the memory pool.
- A finished partition waits for its turn holding the batches it has not
delivered. Once as many partitions as may run at once are started and not
delivered, another starts only while the undelivered batches total less than
the read-ahead limit (`spark.driver.maxResultSize`). Without this bound,
finished partitions could pile up their batches behind a slow one; in the
experiment without it, up to 57 were waiting at once.
- `local-execution.md` describes the new rule.
## How are these changes tested?
- New tests in `native/local`:
- `a_partition_that_finishes_before_its_turn_gives_up_its_slot`: while the
first partition runs, each later partition that finishes drops its stream and
lets the next one start.
- `finished_partitions_wait_for_their_turn_within_the_read_ahead_limit`:
with no read-ahead, a finished partition does not let a partition start beyond
those that may run at once.
- Two tests were rewritten because they asserted the old rule, that a
partition starts only after the first is delivered:
`partition_order_runs_at_most_the_given_number_of_partitions` and
`partition_order_follows_a_changing_running_limit`. They now use partitions
that run until the test lets them finish, so the bound is checked against
running partitions. The first also replaces
`partitions_from_a_start_function_are_planned_as_they_start`, since it plans
its partitions with a start function.
- Without the fix,
`a_partition_that_finishes_before_its_turn_gives_up_its_slot` and the rewritten
`partition_order_runs_at_most_the_given_number_of_partitions` fail. The
read-ahead test passes either way, since it guards the bound this change adds.
Benchmark (`collect`, `local[8]`, Apple M4 Max, release build, 8M rows in 64
Parquet files, 10 iterations, average ms). The scan cases run with this change
applied on top of #7692:
Both runs are on the same machine, but the run with this change was noisier:
Spark itself was slower in every case. So compare local execution with ordinary
Comet within each run.
| Case | Partitions | Before: Comet | Before: local | After: Comet | After:
local |
|---|---|---|---|---|---|
| Selective filter (8K rows returned) | 320 | 74 | 107 | 106 | 50 |
| Selective filter (8K rows returned) | 8 | 34 | 33 | 42 | 40 |
| All rows | 320 | 1279 | 1235 | 1109 | 1196 |
| All rows | 8 | 1178 | 1254 | 1247 | 1169 |
| All rows, half in one file | 8 | 1574 | 1124 | 1874 | 1539 |
| Range, all rows | 8 | 541 | 439 | 673 | 569 |
| Range, selective filter | 8 | 17 | 16 | 15 | 13 |
With many small partitions, local execution goes from about 45% slower than
ordinary Comet to about 2x faster. The other cases stay within noise.
`CometLocalExecutionSuite` also passes (35 tests) with this change applied on
top of #7692.
This pull request and its description were written by Isaac.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: [email protected]
For queries about this service, please contact Infrastructure at:
[email protected]
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]