andygrove opened a new issue, #2085: URL: https://github.com/apache/datafusion-ballista/issues/2085
**Describe the bug** The config key `ballista.optimizer.broadcast_join_threshold_bytes` (default 10 MiB) controls the byte-size threshold below which a hash join's smaller side is promoted to a `CollectLeft` broadcast join. It is only consumed by the **static** distributed planner, in `DefaultDistributedPlanner::maybe_promote_to_broadcast` (`ballista/scheduler/src/planner.rs:139`, `:274-304`). When adaptive query planning is enabled (`ballista.planner.adaptive.enabled=true`), planning goes through `AdaptivePlanner` / the `state/aqe/` module instead, which never calls `maybe_promote_to_broadcast` and never reads `broadcast_join_threshold_bytes`. This is even noted in the AQE module header comment (`ballista/scheduler/src/state/aqe/mod.rs:47-53`). A `grep` confirms the key has no reference anywhere under `state/aqe/`. Under AQE, broadcast (BHJ / `CollectLeft`) selection is instead driven by DataFusion's `optimizer.hash_join_single_partition_threshold` (default **1 MiB**) and `hash_join_single_partition_threshold_rows` (default 128K), read in `DynamicJoinSelectionExec::to_actual_join` (`ballista/scheduler/src/state/aqe/execution_plan/dynamic_join.rs:217-220`). The result: a user who sets `broadcast_join_threshold_bytes` to tune broadcast behavior gets **no effect** under AQE, and the effective threshold is silently a different value (1 MiB) driven by a different, DataFusion-namespaced key. This is a confusing config foot-gun, made worse because the two thresholds use different byte semantics. **To Reproduce** 1. Enable AQE: `ballista.planner.adaptive.enabled=true`. 2. Set `ballista.optimizer.broadcast_join_threshold_bytes` to a large value (e.g. 100 MiB). 3. Run a join whose build side is, say, 20 MiB. It is not promoted to a broadcast join, despite being well under the configured threshold. 4. Conversely, the actual broadcast cutoff is DataFusion's `hash_join_single_partition_threshold` (1 MiB default), which the Ballista key does not influence. **Expected behavior** Either the AQE join-selection path honors `ballista.optimizer.broadcast_join_threshold_bytes` (routing it into `DynamicJoinSelectionExec::to_actual_join`), or the key is documented as static-planner-only and AQE exposes an equivalent, clearly-named knob. A single broadcast-threshold config should behave consistently regardless of which planner is active. **Additional context** - Static path consumes the key: `ballista/scheduler/src/planner.rs:274-304`. - AQE path uses DataFusion thresholds instead: `ballista/scheduler/src/state/aqe/execution_plan/dynamic_join.rs:217-230`. - AQE header comment acknowledging `maybe_promote_to_broadcast` does not fire: `ballista/scheduler/src/state/aqe/mod.rs:47-53`. - Related: the two thresholds also differ in what "bytes" means (static estimate vs. on-disk Arrow IPC shuffle size), which is a separate concern worth tracking. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected] --------------------------------------------------------------------- To unsubscribe, e-mail: [email protected] For additional commands, e-mail: [email protected]
