wirybeaver opened a new pull request, #2701:
URL: https://github.com/apache/datafusion-ballista/pull/2701
## Motivation
Distributed runtime filtering needs a bounded summary that executors can
build, transport, and apply consistently. This is the foundation of the
three-part runtime-filter series; the following AQE layer handles filter
placement and coordination.
Implements task 1 of #2700.
## Modification
Add fixed-seed batch hashing, an 8 KiB Bloom summary, a bounded IN hash set,
and optional MinMax bounds. Build taps publish only after all local input
partitions reach EOF. Apply operators retain candidate rows and expose pruning
metrics; a missing payload passes the input through.
```text
build input -> build tap -> shuffle writer
|
successful task report
probe task plan + payload -> apply operator -> candidate rows
```
Wire the operators through the physical-plan codec and collect completed
summaries in both executor scheduling paths. Successful-task protobuf fixtures
are updated together with the transport contract. Automatic planning and
scheduler coordination are supplied by the next PR.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: [email protected]
For queries about this service, please contact Infrastructure at:
[email protected]
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
