Riza Suminto has posted comments on this change. ( http://gerrit.cloudera.org:8080/19033 )
Change subject: IMPALA-11604 Planner changes for CPU usage ...................................................................... Patch Set 46: (4 comments) http://gerrit.cloudera.org:8080/#/c/19033/46//COMMIT_MSG Commit Message: http://gerrit.cloudera.org:8080/#/c/19033/46//COMMIT_MSG@138 PS46, Line 138: a list of ProcessingCost > I think a single processingCost (instead of a list of processingCost) may b The MergeJoin scenario you mention sounds possible. I believe such case can be resolved in step IV with careful inspection of fragment members, without the need to merge step II, III, and IV into a single pass. I also think the separation now is better to keep since step II and III focus on individual fragment while step IV reconcile result from step III by looking at connected fragments and reason about their combined parallelism. That being said, I talked with Aman and found out that currently Impala planner does not support SortMerge Join. Impala planner never create a fragment plan with a chain of SCAN -> SORT -> JOIN since the JOIN is either a Hash based or just a plain Nested Loop join. In both cases, sorted input is not needed. I do see EXCHANGE -> AGG -> JOIN within a fragment in some TPC-DS queries (ie., Q47). That is similar with your example, in that AGG is also blocking operator like SORT. But with that case, the current algorithm in step IV is helped by the fact that such fragment always has child fragment feeding into it that is also considered a blocking fragment (has preAGG), leading to correct comparison between root subtree vs child subtrees. For further reassurance, I have tested the current algorithm against TPC-DS scale 3TB. I don't see any major regression due to skewed parallelism from bad decision on step IV. There are 2 queries that regress by more than 10% compared to regular MT_DOP plan, but that is due to wrong cardinality estimates caused by runtime filters and the fact that this patch does not change scan fragment parallelism yet. I plan to improve that in follow up work. http://gerrit.cloudera.org:8080/#/c/19033/46//COMMIT_MSG@247 PS46, Line 247: A blocking fragment is a fragment that has a blocking plan node. The : costing algorithm split the query plan tree into several blocking : subtrees divided by blocking fragments. Each blocking subtree has a : blocking fragment as a root and non-blocking fragments as leaves. All : other fragments in the subtree are non-blocking. > This part of the para should be merged into the para at line 142. I can move the Blocking-fragment definition to paragraph at line 142. But I think Blocking-subtree should stay here since discussion in step II is only focus within a single fragment. http://gerrit.cloudera.org:8080/#/c/19033/46//COMMIT_MSG@253 PS46, Line 253: F03, F00 The order is flipped here. http://gerrit.cloudera.org:8080/#/c/19033/46//COMMIT_MSG@258 PS46, Line 258: This means that all fragments within a : blocking subtree can run in parallel and should be assigned 1 core per : fragment instance > This may overestimate when HJ is involved, since the build side and the pro The sequential nature of build-probe is resolved through the CoreRequirement formula. At F00, the CoreRequirement is resolved to: max(F00, sum((F05+F02), (F06+F01)) = max(12, sum(4, 4)) = 12 The join node subtree (probe) is more dominant than the sum of the child subtrees (both builds). -- To view, visit http://gerrit.cloudera.org:8080/19033 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-MessageType: comment Gerrit-Change-Id: If32dc770dfffcdd0be2b5555a789a7720952c68a Gerrit-Change-Number: 19033 Gerrit-PatchSet: 46 Gerrit-Owner: Qifan Chen <qfc...@hotmail.com> Gerrit-Reviewer: Impala Public Jenkins <impala-public-jenk...@cloudera.com> Gerrit-Reviewer: Kurt Deschler <kdesc...@cloudera.com> Gerrit-Reviewer: Qifan Chen <qfc...@hotmail.com> Gerrit-Reviewer: Riza Suminto <riza.sumi...@cloudera.com> Gerrit-Reviewer: Wenzhe Zhou <wz...@cloudera.com> Gerrit-Comment-Date: Mon, 13 Feb 2023 20:36:09 +0000 Gerrit-HasComments: Yes
