David Rorke has posted comments on this change. ( http://gerrit.cloudera.org:8080/21277 )
Change subject: IMPALA-12988: Calculate an unbounded version of CpuAsk ...................................................................... Patch Set 7: (1 comment) http://gerrit.cloudera.org:8080/#/c/21277/7//COMMIT_MSG Commit Message: http://gerrit.cloudera.org:8080/#/c/21277/7//COMMIT_MSG@22 PS7, Line 22: During executor group set : selection, Frontend should use the maximum between bounded CpuAsk and : unbounded CpuAsk numbers to avoid assigning a query to a small executor : group set too soon. The last executor group set stays as the "catch-all" : executor group set. > This can potentially cause the issue where most queries would get assigned Testing so far does confirm that unbounding will cause queries to flow to larger (sometimes much larger EGs). I mostly see this as a problem for cases where we overestimate cost, e.g. because of bad cardinality overestimates but unfortunately this is common. The problem with the current bounded model though is that it bounds in a way thats kind of arbitrary and hard to predict or reason about. If we want to have some (optional) bounding so queries don't flow so quickly to very large group sets I'd prefer some (optional) sublinear function that drives the growth of the CPU requirement. For example the growth in bounded CPU could be calculated as the sqrt (or some nth root) of the originally calculated growth in unbounded CPU. -- To view, visit http://gerrit.cloudera.org:8080/21277 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-MessageType: comment Gerrit-Change-Id: I5441e31088f90761062af35862be4ce09d116923 Gerrit-Change-Number: 21277 Gerrit-PatchSet: 7 Gerrit-Owner: Riza Suminto <riza.sumi...@cloudera.com> Gerrit-Reviewer: Abhishek Rawat <ara...@cloudera.com> Gerrit-Reviewer: David Rorke <dro...@cloudera.com> Gerrit-Reviewer: Impala Public Jenkins <impala-public-jenk...@cloudera.com> Gerrit-Reviewer: Kurt Deschler <kdesc...@cloudera.com> Gerrit-Reviewer: Riza Suminto <riza.sumi...@cloudera.com> Gerrit-Comment-Date: Tue, 16 Apr 2024 15:06:26 +0000 Gerrit-HasComments: Yes
