Impala Public Jenkins has posted comments on this change. ( http://gerrit.cloudera.org:8080/16822 )
Change subject: IMPALA-10377 Improve the accuracy of resource estimation PlanNode does not consider some factors when estimating memory, this will cause a large error rate ...................................................................... Patch Set 1: (39 comments) http://gerrit.cloudera.org:8080/#/c/16822/1/fe/src/main/java/org/apache/impala/planner/AggregationNode.java File fe/src/main/java/org/apache/impala/planner/AggregationNode.java: http://gerrit.cloudera.org:8080/#/c/16822/1/fe/src/main/java/org/apache/impala/planner/AggregationNode.java@64 PS1, Line 64: // If the group clause is empty ( aggInfo.getGroupingExprs() is empty ), line has trailing whitespace http://gerrit.cloudera.org:8080/#/c/16822/1/fe/src/main/java/org/apache/impala/planner/AggregationNode.java@538 PS1, Line 538: // A skew factor of 1.5 was added to account for data skew among multiple fragment instances. line has trailing whitespace http://gerrit.cloudera.org:8080/#/c/16822/1/fe/src/main/java/org/apache/impala/planner/AggregationNode.java@538 PS1, Line 538: // A skew factor of 1.5 was added to account for data skew among multiple fragment instances. line too long (106 > 90) http://gerrit.cloudera.org:8080/#/c/16822/1/fe/src/main/java/org/apache/impala/planner/AggregationNode.java@539 PS1, Line 539: // This number was derived using empirical analysis of real-world and benchmark (tpch, tpcds) queries. line too long (114 > 90) http://gerrit.cloudera.org:8080/#/c/16822/1/fe/src/main/java/org/apache/impala/planner/AggregationNode.java@544 PS1, Line 544: perInstanceInputCardinality = (long) Math.ceil(inputCardinality / numInstances); line too long (92 > 90) http://gerrit.cloudera.org:8080/#/c/16822/1/fe/src/main/java/org/apache/impala/planner/AggregationNode.java@551 PS1, Line 551: line has trailing whitespace http://gerrit.cloudera.org:8080/#/c/16822/1/fe/src/main/java/org/apache/impala/planner/AggregationNode.java@553 PS1, Line 553: // A reduction factor of 2 (input rows divided by output rows) was added to grow hash tables. line too long (103 > 90) http://gerrit.cloudera.org:8080/#/c/16822/1/fe/src/main/java/org/apache/impala/planner/AggregationNode.java@554 PS1, Line 554: // If the reduction factor is lower than 2, only part of the data will be inserted into the hash table. line too long (113 > 90) http://gerrit.cloudera.org:8080/#/c/16822/1/fe/src/main/java/org/apache/impala/planner/AggregationNode.java@562 PS1, Line 562: // The memory of the data stored in hash table and the memory of the hash table‘s structure line too long (99 > 90) http://gerrit.cloudera.org:8080/#/c/16822/1/fe/src/main/java/org/apache/impala/planner/AggregationNode.java@563 PS1, Line 563: perInstanceDataBytes = (long)Math.ceil(perInstanceCardinality * line has trailing whitespace http://gerrit.cloudera.org:8080/#/c/16822/1/fe/src/main/java/org/apache/impala/planner/AggregationNode.java@564 PS1, Line 564: (avgRowSize_ + PlannerContext.SIZE_OF_BUCKET)); line too long (94 > 90) http://gerrit.cloudera.org:8080/#/c/16822/1/fe/src/main/java/org/apache/impala/planner/HashJoinNode.java File fe/src/main/java/org/apache/impala/planner/HashJoinNode.java: http://gerrit.cloudera.org:8080/#/c/16822/1/fe/src/main/java/org/apache/impala/planner/HashJoinNode.java@239 PS1, Line 239: // The memory of the data stored in hash table and line has trailing whitespace http://gerrit.cloudera.org:8080/#/c/16822/1/fe/src/main/java/org/apache/impala/planner/HashJoinNode.java@242 PS1, Line 242: BitUtil.roundUpToPowerOf2((long) Math.ceil(3 * rhsCard / 2)) * line has trailing whitespace http://gerrit.cloudera.org:8080/#/c/16822/1/fe/src/main/java/org/apache/impala/planner/HashJoinNode.java@245 PS1, Line 245: perBuildInstanceDataBytes += (rhsCard - rhsNdv) * line has trailing whitespace http://gerrit.cloudera.org:8080/#/c/16822/1/fe/src/main/java/org/apache/impala/planner/PlannerContext.java File fe/src/main/java/org/apache/impala/planner/PlannerContext.java: http://gerrit.cloudera.org:8080/#/c/16822/1/fe/src/main/java/org/apache/impala/planner/PlannerContext.java@45 PS1, Line 45: line has trailing whitespace http://gerrit.cloudera.org:8080/#/c/16822/1/fe/src/main/java/org/apache/impala/planner/SortNode.java File fe/src/main/java/org/apache/impala/planner/SortNode.java: http://gerrit.cloudera.org:8080/#/c/16822/1/fe/src/main/java/org/apache/impala/planner/SortNode.java@338 PS1, Line 338: perInstanceMemEstimate = fullInputSize < 0 ? line has trailing whitespace http://gerrit.cloudera.org:8080/#/c/16822/1/fe/src/test/java/org/apache/impala/planner/CardinalityTest.java File fe/src/test/java/org/apache/impala/planner/CardinalityTest.java: http://gerrit.cloudera.org:8080/#/c/16822/1/fe/src/test/java/org/apache/impala/planner/CardinalityTest.java@847 PS1, Line 847: verifyApproxMemoryEstimate("SELECT COUNT(*),MAX(B.BIGINT_COL) FROM FUNCTIONAL.ALLTYPES A ," + line has trailing whitespace http://gerrit.cloudera.org:8080/#/c/16822/1/fe/src/test/java/org/apache/impala/planner/CardinalityTest.java@847 PS1, Line 847: verifyApproxMemoryEstimate("SELECT COUNT(*),MAX(B.BIGINT_COL) FROM FUNCTIONAL.ALLTYPES A ," + line too long (98 > 90) http://gerrit.cloudera.org:8080/#/c/16822/1/fe/src/test/java/org/apache/impala/planner/CardinalityTest.java@848 PS1, Line 848: "FUNCTIONAL.ALLTYPESSMALL B, FUNCTIONAL.NULLROWS C GROUP BY A.ID," + line has trailing whitespace http://gerrit.cloudera.org:8080/#/c/16822/1/fe/src/test/java/org/apache/impala/planner/CardinalityTest.java@849 PS1, Line 849: "B.TIMESTAMP_COL,C.BOOL_NULLS", line has trailing whitespace http://gerrit.cloudera.org:8080/#/c/16822/1/fe/src/test/java/org/apache/impala/planner/CardinalityTest.java@852 PS1, Line 852: verifyApproxMemoryEstimate("SELECT COUNT(*),MAX(B.BIGINT_COL) FROM FUNCTIONAL.ALLTYPES A ," + line has trailing whitespace http://gerrit.cloudera.org:8080/#/c/16822/1/fe/src/test/java/org/apache/impala/planner/CardinalityTest.java@852 PS1, Line 852: verifyApproxMemoryEstimate("SELECT COUNT(*),MAX(B.BIGINT_COL) FROM FUNCTIONAL.ALLTYPES A ," + line too long (98 > 90) http://gerrit.cloudera.org:8080/#/c/16822/1/fe/src/test/java/org/apache/impala/planner/CardinalityTest.java@853 PS1, Line 853: "FUNCTIONAL.ALLTYPESSMALL B, FUNCTIONAL.NULLROWS C GROUP BY A.ID," + line has trailing whitespace http://gerrit.cloudera.org:8080/#/c/16822/1/fe/src/test/java/org/apache/impala/planner/CardinalityTest.java@854 PS1, Line 854: "B.TIMESTAMP_COL,C.BOOL_NULLS", line has trailing whitespace http://gerrit.cloudera.org:8080/#/c/16822/1/fe/src/test/java/org/apache/impala/planner/CardinalityTest.java@861 PS1, Line 861: verifyApproxMemoryEstimate("SELECT COUNT(*) FROM FUNCTIONAL.ALLTYPES A ", line has trailing whitespace http://gerrit.cloudera.org:8080/#/c/16822/1/fe/src/test/java/org/apache/impala/planner/CardinalityTest.java@864 PS1, Line 864: verifyApproxMemoryEstimate("SELECT COUNT(*) FROM FUNCTIONAL.ALLTYPES A ", line has trailing whitespace http://gerrit.cloudera.org:8080/#/c/16822/1/fe/src/test/java/org/apache/impala/planner/CardinalityTest.java@875 PS1, Line 875: verifyApproxMemoryEstimate("SELECT A.TIMESTAMP_COL,B.TIMESTAMP_COL FROM " + line has trailing whitespace http://gerrit.cloudera.org:8080/#/c/16822/1/fe/src/test/java/org/apache/impala/planner/CardinalityTest.java@876 PS1, Line 876: "FUNCTIONAL.ALLTYPES A , FUNCTIONAL.ALLTYPESSMALL B " + line has trailing whitespace http://gerrit.cloudera.org:8080/#/c/16822/1/fe/src/test/java/org/apache/impala/planner/CardinalityTest.java@877 PS1, Line 877: "ORDER BY A.TIMESTAMP_COL,B.TIMESTAMP_COL", line has trailing whitespace http://gerrit.cloudera.org:8080/#/c/16822/1/fe/src/test/java/org/apache/impala/planner/CardinalityTest.java@885 PS1, Line 885: line has trailing whitespace http://gerrit.cloudera.org:8080/#/c/16822/1/fe/src/test/java/org/apache/impala/planner/CardinalityTest.java@894 PS1, Line 894: "FUNCTIONAL.ALLTYPES A1,FUNCTIONAL.ALLTYPES B1 GROUP BY A1.ID,B1.DATE_STRING_COL) " + line too long (93 > 90) http://gerrit.cloudera.org:8080/#/c/16822/1/fe/src/test/java/org/apache/impala/planner/CardinalityTest.java@895 PS1, Line 895: "B ON A.ID = B.ID AND A.IDB=B.IDB", line has trailing whitespace http://gerrit.cloudera.org:8080/#/c/16822/1/fe/src/test/java/org/apache/impala/planner/CardinalityTest.java@898 PS1, Line 898: line has trailing whitespace http://gerrit.cloudera.org:8080/#/c/16822/1/fe/src/test/java/org/apache/impala/planner/CardinalityTest.java@908 PS1, Line 908: "FUNCTIONAL.ALLTYPES B1) B ON A.ID = B.ID AND A.IDB=B.IDB", line has trailing whitespace http://gerrit.cloudera.org:8080/#/c/16822/1/fe/src/test/java/org/apache/impala/planner/CardinalityTest.java@909 PS1, Line 909: 53290000L*(4L+8L+4L+8L)+134217728L*16L+(53290000L-5372800L)*24L, true, true, line has trailing whitespace http://gerrit.cloudera.org:8080/#/c/16822/1/fe/src/test/java/org/apache/impala/planner/CardinalityTest.java@917 PS1, Line 917: verifyApproxMemoryEstimate("SELECT ID FROM FUNCTIONAL_KUDU.ALLTYPESSMALL", line has trailing whitespace http://gerrit.cloudera.org:8080/#/c/16822/1/fe/src/test/java/org/apache/impala/planner/CardinalityTest.java@918 PS1, Line 918: 393216*1*2, true, false, line has trailing whitespace http://gerrit.cloudera.org:8080/#/c/16822/1/fe/src/test/java/org/apache/impala/planner/CardinalityTest.java@1113 PS1, Line 1113: protected PlanNode getPlanNode(String query, boolean isDistributedPlan, line has trailing whitespace http://gerrit.cloudera.org:8080/#/c/16822/1/fe/src/test/java/org/apache/impala/planner/CardinalityTest.java@1142 PS1, Line 1142: protected void verifyApproxMemoryEstimate(String query, long expected, line has trailing whitespace -- To view, visit http://gerrit.cloudera.org:8080/16822 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-MessageType: comment Gerrit-Change-Id: I46c656bc88b969f4de99e187df16be3887592f3d Gerrit-Change-Number: 16822 Gerrit-PatchSet: 1 Gerrit-Owner: Anonymous Coward <54liu...@163.com> Gerrit-Reviewer: Impala Public Jenkins <impala-public-jenk...@cloudera.com> Gerrit-Comment-Date: Sat, 05 Dec 2020 03:17:53 +0000 Gerrit-HasComments: Yes
