avantgardnerio commented on PR #2419:
URL:
https://github.com/apache/datafusion-ballista/pull/2419#issuecomment-5533003888
AI slop good enough worth reposting verbatim:
```
q08 s1 and q09 s2, same run, same cluster, same SortShuffleWriterExec, both
into 256 partitions:
┌────────┬─────────────────────────────────────────────────┬────────────────────┬────────┬────────┐
│ │ plan │ rows/task
(median) │ median │ µs/row │
├────────┼─────────────────────────────────────────────────┼────────────────────┼────────┼────────┤
│ q08 s1 │ SortShuffleWriter ← DataSourceExec │
187.0M │ 62.45s │ 0.334 │
├────────┼─────────────────────────────────────────────────┼────────────────────┼────────┼────────┤
│ q09 s1 │ SortShuffleWriter ← DataSourceExec │
142.1M │ 19.18s │ 0.135 │
├────────┼─────────────────────────────────────────────────┼────────────────────┼────────┼────────┤
│ q09 s2 │ SortShuffleWriter ← HashJoin ← 2× ShuffleReader │
197.7M │ 6.88s │ 0.035 │
└────────┴─────────────────────────────────────────────────┴────────────────────┴────────┴────────┘
q09 s2 pushes more rows through the same writer and runs a partitioned
hash join on top — and it's 9.6× faster per row than q08 s1. If
buffering/spilling/consolidating ~190M rows
into 256 partitions were inherently a 60s operation, q09 s2 could not be
6.9s.
And q08 s1 vs q09 s1 are the identical operator pair —
SortShuffleWriterExec: partitioning=Hash([l_partkey@1], 256) ← DataSourceExec,
same 5,999,989,709 input rows, same
11,999,979,418 output — differing 3.3× on the median.
The variable that tracks it is the source: the two slow stages read
Parquet, the fast one reads ShuffleReaderExec. Task wall includes waiting on
S3, and elapsed_compute is task
wall.</pre>
```
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: [email protected]
For queries about this service, please contact Infrastructure at:
[email protected]
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
