Abhishek Rawat has posted comments on this change. ( http://gerrit.cloudera.org:8080/21762 )
Change subject: IMPALA-13333: Limit memory estimation if PlanNode can spill ...................................................................... Patch Set 33: (3 comments) http://gerrit.cloudera.org:8080/#/c/21762/33/common/thrift/ImpalaService.thrift File common/thrift/ImpalaService.thrift: http://gerrit.cloudera.org:8080/#/c/21762/33/common/thrift/ImpalaService.thrift@1018 PS33, Line 1018: Setting to 0.0 or unsetting scratch_dirs flag will revert Planner to old behavior : // where it assumes that all rows must be held in memory I think this contradicts the statement above that, "Setting value closer to 1.0 will make Planner bias towards keeping as much rows as possible in memory, while setting value closer to 0.0 will make Planner bias towards spilling rows to disk under memory pressure." If the intent is to disable the scaling of memory estimate for Spillable operators i.e., the old behavior then we could maybe use -1 for that? The other option would be to use 0.0 for planner biasing towards no spilling and 1.0 for spilling bias. http://gerrit.cloudera.org:8080/#/c/21762/31/fe/src/main/java/org/apache/impala/analysis/Analyzer.java File fe/src/main/java/org/apache/impala/analysis/Analyzer.java: http://gerrit.cloudera.org:8080/#/c/21762/31/fe/src/main/java/org/apache/impala/analysis/Analyzer.java@765 PS31, Line 765: if (opts.isSetMem_limit() && opts.getMem_limit() > 0) { mem_limit will be clamped if clamp_mem_limit_query_option is set in pool config. http://gerrit.cloudera.org:8080/#/c/21762/33/fe/src/main/java/org/apache/impala/analysis/Analyzer.java File fe/src/main/java/org/apache/impala/analysis/Analyzer.java: http://gerrit.cloudera.org:8080/#/c/21762/33/fe/src/main/java/org/apache/impala/analysis/Analyzer.java@765 PS33, Line 765: if (opts.isSetMem_limit() && opts.getMem_limit() > 0) { mem_limit is also clamped when clamp_mem_limit_query_option is set in pool config. -- To view, visit http://gerrit.cloudera.org:8080/21762 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-MessageType: comment Gerrit-Change-Id: I290c4e889d4ab9e921e356f0f55a9c8b11d0854e Gerrit-Change-Number: 21762 Gerrit-PatchSet: 33 Gerrit-Owner: Riza Suminto <[email protected]> Gerrit-Reviewer: Abhishek Rawat <[email protected]> Gerrit-Reviewer: Csaba Ringhofer <[email protected]> Gerrit-Reviewer: Daniel Becker <[email protected]> Gerrit-Reviewer: Impala Public Jenkins <[email protected]> Gerrit-Reviewer: Jason Fehr <[email protected]> Gerrit-Reviewer: Riza Suminto <[email protected]> Gerrit-Reviewer: Wenzhe Zhou <[email protected]> Gerrit-Reviewer: Yida Wu <[email protected]> Gerrit-Comment-Date: Fri, 07 Mar 2025 00:57:17 +0000 Gerrit-HasComments: Yes
