Riza Suminto has uploaded this change for review. ( http://gerrit.cloudera.org:8080/23258
Change subject: IMPALA-14263: Add broadcast_cost_scale to tweak broadcast join cost ...................................................................... IMPALA-14263: Add broadcast_cost_scale to tweak broadcast join cost This commit enhances the distributed planner's costing model for broadcast joins by introducing the `broadcast_cost_scale` query option. This option enables users to fine-tune the planner's decision between broadcast and partitioned joins. Key changes: - The total broadcast cost is scaled by the new `broadcast_cost_scale` query option, allowing users to favor or penalize broadcast joins as needed. - Updated the planner logic and test cases to reflect the new costing model and options. This addresses scenarios where the default costing could lead to suboptimal join distribution choices, particularly in a large-scale cluster where the number of executors can increase broadcast cost, while choosing a partitioned strategy can lead to data skew. Admin can set `broadcast_cost_scale` less than 1.0 to make DistributedPlanner favor broadcast more than partitioned join (with possible downside of higher memory usage per query and higher network transmission). Existing query hints still take precedence over this option. Testing: Added FE tests. Change-Id: I475f8a26b2171e87952b69f66a5c18f77c2b3133 --- M be/src/service/query-options.cc M be/src/service/query-options.h M common/thrift/ImpalaService.thrift M common/thrift/Query.thrift M fe/src/main/java/org/apache/impala/planner/DistributedPlanner.java M testdata/workloads/functional-planner/queries/PlannerTest/tpcds-dist-method.test 6 files changed, 561 insertions(+), 3 deletions(-) git pull ssh://gerrit.cloudera.org:29418/Impala-ASF refs/changes/58/23258/1 -- To view, visit http://gerrit.cloudera.org:8080/23258 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-MessageType: newchange Gerrit-Change-Id: I475f8a26b2171e87952b69f66a5c18f77c2b3133 Gerrit-Change-Number: 23258 Gerrit-PatchSet: 1 Gerrit-Owner: Riza Suminto <[email protected]>
