Riza Suminto has posted comments on this change. ( http://gerrit.cloudera.org:8080/19033 )
Change subject: IMPALA-11604 Planner changes for CPU usage ...................................................................... Patch Set 41: (7 comments) Found some issue and potential improvement. http://gerrit.cloudera.org:8080/#/c/19033/41/common/thrift/Frontend.thrift File common/thrift/Frontend.thrift: http://gerrit.cloudera.org:8080/#/c/19033/41/common/thrift/Frontend.thrift@769 PS41, Line 769: // The following is a list of limits to help selecting a suitable executor group to : // run for a query. The group is identified by the name prefix 'exec_group_name_prefix' : // from the pool service. For each query, the frontend computes a value for each of : // the limit dimension and compares it with the limit. An executor group is chosen : // if all such values are equal to or less than their corresponding limits defined : // for the group. : : // The memory limit provides the per host estimated-memory limit. Restore the comment. http://gerrit.cloudera.org:8080/#/c/19033/41/common/thrift/Query.thrift File common/thrift/Query.thrift: http://gerrit.cloudera.org:8080/#/c/19033/41/common/thrift/Query.thrift@912 PS41, Line 912: optional i32 cores_required; This need comment. http://gerrit.cloudera.org:8080/#/c/19033/41/common/thrift/Query.thrift@914 PS41, Line 914: optional string debug_core_sizing_trace; Consider dropping this and put it to LOG instead. http://gerrit.cloudera.org:8080/#/c/19033/41/fe/src/main/java/org/apache/impala/planner/PlanFragment.java File fe/src/main/java/org/apache/impala/planner/PlanFragment.java: http://gerrit.cloudera.org:8080/#/c/19033/41/fe/src/main/java/org/apache/impala/planner/PlanFragment.java@150 PS41, Line 150: adjustedInstanceCount_ This can be set multiple time during effective parallelism computation. I think we should have separate boolean flag to mark if certain fragment is set to a fixed count by some plan node specification (leaf node such as scan, empty source node, shared join build, etc). http://gerrit.cloudera.org:8080/#/c/19033/41/fe/src/main/java/org/apache/impala/planner/PlanFragment.java@953 PS41, Line 953: protected ProcessingCost traverseBlockingAwareCost(int maxInstancesPerNode, : List<ProcessingCost> blockingTreeCosts, int parentInstanceCount, : boolean allowIncrement) { I believe effective parallelism adjustment and processing cost finding can be separated into their own function. I'll look into this. http://gerrit.cloudera.org:8080/#/c/19033/41/fe/src/main/java/org/apache/impala/planner/Planner.java File fe/src/main/java/org/apache/impala/planner/Planner.java: http://gerrit.cloudera.org:8080/#/c/19033/41/fe/src/main/java/org/apache/impala/planner/Planner.java@317 PS41, Line 317: str.append("Efficient parallelism: "); : str.append(request.getCores_required()); : str.append("\n"); This might still be useful to show in EXTENDED explain level. http://gerrit.cloudera.org:8080/#/c/19033/41/fe/src/main/java/org/apache/impala/planner/Planner.java@320 PS41, Line 320: str.append("Efficient parallelism trace: "); : str.append(request.getDebug_core_sizing_trace()); : str.append("\n"); move to LOG instead. -- To view, visit http://gerrit.cloudera.org:8080/19033 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-MessageType: comment Gerrit-Change-Id: If32dc770dfffcdd0be2b5555a789a7720952c68a Gerrit-Change-Number: 19033 Gerrit-PatchSet: 41 Gerrit-Owner: Qifan Chen <qfc...@hotmail.com> Gerrit-Reviewer: Impala Public Jenkins <impala-public-jenk...@cloudera.com> Gerrit-Reviewer: Kurt Deschler <kdesc...@cloudera.com> Gerrit-Reviewer: Qifan Chen <qfc...@hotmail.com> Gerrit-Reviewer: Riza Suminto <riza.sumi...@cloudera.com> Gerrit-Reviewer: Wenzhe Zhou <wz...@cloudera.com> Gerrit-Comment-Date: Fri, 03 Feb 2023 17:08:16 +0000 Gerrit-HasComments: Yes
