andygrove opened a new pull request, #218: URL: https://github.com/apache/datafusion-site/pull/218
Draft announcement for Ballista 56.0.0, written ahead of the release from what has merged to `main` since 55.0.0. It focuses on performance, memory usage, adaptive query execution and robustness. Contents: - **Performance**: TPC-H SF1000 results on the same cluster as the 55.0.0 post. Ballista is now 2.2x faster than Spark 4.1.3 (up from 1.79x) and 1.12x faster than Spark + Comet 1.1.0 (from 1.10x slower), and 3.4x faster than Spark with the new experimental pipelined mode. Includes the changes behind the gain and the same caveats as before. - **Memory usage**: the smallest executor memory at which TPC-H q18 finishes, for Ballista (8 GiB), Spark (12 GiB) and Spark + Comet 1.2.0 (20 GiB), and how each engine's time changes as memory shrinks. From apache/datafusion-ballista#2855. - **AQE**: HyperLogLog distinct counts on shuffles, measuring joins before broadcasting them, exchange reuse, physical plans through AQE, partition coalescing and skew-join splitting (opt-in), and the groundwork for runtime filters. - **Robustness**: graceful shutdown, the standalone result service, plans that used to return wrong results silently, and the merge-queue test suites. - **Python**, other changes, and upgrading notes. Charts are copied from the Ballista docs (`tpch_sf1000_total.svg` from `main`, the two memory charts from apache/datafusion-ballista#2855). To do before publishing (also listed in a comment at the top of the post): - [ ] Set the date to the release date (front matter and file name). - [ ] Update the commit and contributor counts and the thank-you list from the 56.0.0 changelog (currently 189 commits from 20 authors on `main`). - [ ] Confirm the DataFusion version 56.0.0 ships with. - [ ] Merge apache/datafusion-ballista#2855 so the memory usage page link resolves. 🤖 Generated with [Claude Code](https://claude.com/claude-code) -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected] --------------------------------------------------------------------- To unsubscribe, e-mail: [email protected] For additional commands, e-mail: [email protected]
