Bikramjeet Vig has posted comments on this change. ( http://gerrit.cloudera.org:8080/13740 )
Change subject: IMPALA-7486: Add specialized estimation scheme for dedicated coordinators ...................................................................... Patch Set 18: Notes on my latest patch: so I was looking at my patch and found a bug where i was always calculating cluster mem to admit as (per_backend_mem_to_admit() * (per_backend_exec_params_.size() - 1) + coord_backend_mem_to_admit()) even for the cases where there is no fragment running on the coord host (for insert or CTAS queries). Digging in further I realized that for these cases the coord_backend_mem_to_admit was equal to the insert fragment mem estimate + 100 MB buffer which is due to the fact that in my calculations during planning i was assuming that the root fragment will always be the coordinator fragment (which is obviously not true in this case). Now tying this back to IMPALA-8791 (https://issues.apache.org/jira/browse/IMPALA-8791) we can run into a case where the mem requirement for the insert fragment is huge which results in a huge mem limit being set on the coord's query state (when auto setting of mem limit is enabled through pool configs), thereby blocking that amount of mem on the coordinator for admission purposes (See IMPALA-8791). Another peculiar case can be "insert into values" case where there is no coord fragment but the insert fragment gets scheduled on the coordinator host. This results in a situation where the mem admitted is per_backend_mem_limit_ but the mem limit applied to the coord query state is coord_backend_mem_limit_ and those 2 values can differ by a lot. Considering all that I think we should fix IMPALA-8791 right after this patch otherwise this can cause a bunch of confusing inconsistency problems. -- To view, visit http://gerrit.cloudera.org:8080/13740 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-MessageType: comment Gerrit-Change-Id: I2b94e7293b91dec8a18491079c34923eadd94b21 Gerrit-Change-Number: 13740 Gerrit-PatchSet: 18 Gerrit-Owner: Bikramjeet Vig <bikramjeet....@cloudera.com> Gerrit-Reviewer: Andrew Sherman <asher...@cloudera.com> Gerrit-Reviewer: Bikramjeet Vig <bikramjeet....@cloudera.com> Gerrit-Reviewer: Impala Public Jenkins <impala-public-jenk...@cloudera.com> Gerrit-Reviewer: Lars Volker <l...@cloudera.com> Gerrit-Reviewer: Tim Armstrong <tarmstr...@cloudera.com> Gerrit-Comment-Date: Wed, 31 Jul 2019 16:54:36 +0000 Gerrit-HasComments: No
