Riza Suminto has posted comments on this change. ( http://gerrit.cloudera.org:8080/20612 )
Change subject: IMPALA-3825: Delegate runtime filter aggregation to some executors ...................................................................... Patch Set 8: (3 comments) http://gerrit.cloudera.org:8080/#/c/20612/7//COMMIT_MSG Commit Message: http://gerrit.cloudera.org:8080/#/c/20612/7//COMMIT_MSG@20 PS7, Line 20: Query option MAX_NUM_FILTERS_AGGREGATED_PER_HOST is added : to control this feature. Given N as the number of backend executors : excluding the coordinator, the selected number of intermediate : aggregators M = ceil(N / MAX_NUM_FILTERS_AGGREGATED_PER_HOST). Setting : MAX_NUM_FILTERS_AGGREGATED_PER_HOST <= 1 will disable > MAX_NUM_FILTER_AGGREGATOR seems ok for me in the short term to test this fe Done. Changed MAX_NUM_FILTER_AGGREGATOR to MAX_NUM_FILTERS_AGGREGATED_PER_HOST. http://gerrit.cloudera.org:8080/#/c/20612/7/be/src/scheduling/scheduler.cc File be/src/scheduling/scheduler.cc: http://gerrit.cloudera.org:8080/#/c/20612/7/be/src/scheduling/scheduler.cc@282 PS7, Line 282: typedef vector<pair<int, int>> InstanceToAggPairs; : : v > Can you move this to network-util.h? Done http://gerrit.cloudera.org:8080/#/c/20612/7/be/src/scheduling/scheduler.cc@317 PS7, Line 317: // Put coordinator grou > Does num_agg = 1 make sense? I think it still make sense. Say a query has 50 bloom runtime filters and running in 100 executors cluster. num_agg=1 means coordinator handle only 50 finalized filter instead of 5000 filter updates. In this case, coordinator will only responsible to publish the finalized filter to all executors, while the aggregation happen in other nodes. Note that only coordinator has full KRPC addresses of cluster members via Coordinator::BackendState. -- To view, visit http://gerrit.cloudera.org:8080/20612 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-MessageType: comment Gerrit-Change-Id: I11d38ed0f223d6e5b32a19ebe725af7738ee4ab0 Gerrit-Change-Number: 20612 Gerrit-PatchSet: 8 Gerrit-Owner: Riza Suminto <riza.sumi...@cloudera.com> Gerrit-Reviewer: Abhishek Rawat <ara...@cloudera.com> Gerrit-Reviewer: Csaba Ringhofer <csringho...@cloudera.com> Gerrit-Reviewer: Impala Public Jenkins <impala-public-jenk...@cloudera.com> Gerrit-Reviewer: Kurt Deschler <kdesc...@cloudera.com> Gerrit-Reviewer: Michael Smith <michael.sm...@cloudera.com> Gerrit-Reviewer: Riza Suminto <riza.sumi...@cloudera.com> Gerrit-Comment-Date: Wed, 01 Nov 2023 20:19:56 +0000 Gerrit-HasComments: Yes
