liuzqt opened a new pull request, #40629: URL: https://github.com/apache/spark/pull/40629
### What changes were proposed in this pull request? The current TorrentBroadcast implementation is originally designed for large data transmission where driver might become the bottleneck and memory might also be the concern. Therefore it's kind of heavy and comes with some fixed overhead: - torrent protocol: more round traffic - disk level persistency, BlockManager, extra overhead which makes it inefficient for some small data transmission. We can have a lightweight broadcast implementation, e.g, SmallBroadcast, which implement star-topology broadcast(which avoids the unnecessary round traffic), and maybe in-memory only. Note: The current factory style API seem to encourage only one broadcast style on runtime, however this small broadcast implementation is a supplement to `TorrentBroadcast`. So I introduce new small broadcast APIs along the code path. ### Why are the changes needed? Provide a lightweight implementation of `Broadcast`, which is suitable for situation where data size is small and latency is critical. ### Does this PR introduce _any_ user-facing change? NO ### How was this patch tested? Unit test. More testcases are from `BroadcastSuite` -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org --------------------------------------------------------------------- To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
