viirya opened a new pull request, #7693:
URL: https://github.com/apache/datafusion-comet/pull/7693

   ## Which issue does this PR close?
   
   Part of #6499. Follow-up to #6883, found while benchmarking #7692.
   
   ## Rationale for this change
   
   A local query runs at most as many partitions at once as Spark runs tasks. 
Under the current scheduler, the next partition starts only when the first 
undelivered one is delivered. So a partition that finishes before its turn 
keeps its slot until it is delivered, and one slow partition leaves the slots 
behind it idle. Spark instead starts the next task whenever any task ends.
   
   Range partitions are uniform, so this rarely shows with the ranges #6883 
admits. Scan partitions are not: their splits differ in size and some read a 
file footer. With #7692, collecting a Parquet scan of 320 small partitions on 
`local[8]` took about 45% longer locally than with ordinary Comet. 
Per-partition timing of that case showed:
   
   - Planning and starting a partition took about 50 µs.
   - Running a partition took a median of 340 µs and up to 2.9 ms.
   - Finished partitions then waited a mean of 1.4 ms for delivery while 
keeping their slots.
   - On average only 2.9 of the 8 slots were busy.
   
   ## What changes are included in this PR?
   
   - When a partition's task finishes, it drops the partition's stream, which 
releases its operators and their memory consumers, and wakes the reader so that 
another partition starts. The bound on running partitions now counts only 
partitions that have not finished. It still bounds how many partitions share 
the memory pool.
   - A finished partition waits for its turn holding the batches it has not 
delivered. Once as many partitions as may run at once are started and not 
delivered, another starts only while the undelivered batches total less than 
the read-ahead limit (`spark.driver.maxResultSize`). Without this bound, 
finished partitions could pile up their batches behind a slow one; in the 
experiment without it, up to 57 were waiting at once.
   - `local-execution.md` describes the new rule.
   
   ## How are these changes tested?
   
   - New tests in `native/local`:
     - `a_partition_that_finishes_before_its_turn_gives_up_its_slot`: while the 
first partition runs, each later partition that finishes drops its stream and 
lets the next one start.
     - `finished_partitions_wait_for_their_turn_within_the_read_ahead_limit`: 
with no read-ahead, a finished partition does not let a partition start beyond 
those that may run at once.
   - Two tests were rewritten because they asserted the old rule, that a 
partition starts only after the first is delivered: 
`partition_order_runs_at_most_the_given_number_of_partitions` and 
`partition_order_follows_a_changing_running_limit`. They now use partitions 
that run until the test lets them finish, so the bound is checked against 
running partitions. The first also replaces 
`partitions_from_a_start_function_are_planned_as_they_start`, since it plans 
its partitions with a start function.
   - Without the fix, 
`a_partition_that_finishes_before_its_turn_gives_up_its_slot` and the rewritten 
`partition_order_runs_at_most_the_given_number_of_partitions` fail. The 
read-ahead test passes either way, since it guards the bound this change adds.
   
   Benchmark (`collect`, `local[8]`, Apple M4 Max, release build, 8M rows in 64 
Parquet files, 10 iterations, average ms). The scan cases run with this change 
applied on top of #7692:
   
   Both runs are on the same machine, but the run with this change was noisier: 
Spark itself was slower in every case. So compare local execution with ordinary 
Comet within each run.
   
   | Case | Partitions | Before: Comet | Before: local | After: Comet | After: 
local |
   |---|---|---|---|---|---|
   | Selective filter (8K rows returned) | 320 | 74 | 107 | 106 | 50 |
   | Selective filter (8K rows returned) | 8 | 34 | 33 | 42 | 40 |
   | All rows | 320 | 1279 | 1235 | 1109 | 1196 |
   | All rows | 8 | 1178 | 1254 | 1247 | 1169 |
   | All rows, half in one file | 8 | 1574 | 1124 | 1874 | 1539 |
   | Range, all rows | 8 | 541 | 439 | 673 | 569 |
   | Range, selective filter | 8 | 17 | 16 | 15 | 13 |
   
   With many small partitions, local execution goes from about 45% slower than 
ordinary Comet to about 2x faster. The other cases stay within noise. 
`CometLocalExecutionSuite` also passes (35 tests) with this change applied on 
top of #7692.
   
   This pull request and its description were written by Isaac.
   


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: [email protected]

For queries about this service, please contact Infrastructure at:
[email protected]


---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]

Reply via email to