[
https://issues.apache.org/jira/browse/IMPALA-7486?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Lars Volker updated IMPALA-7486:
--------------------------------
Labels: scalability (was: )
> Admit less memory on dedicated coordinator for admission control purposes
> -------------------------------------------------------------------------
>
> Key: IMPALA-7486
> URL: https://issues.apache.org/jira/browse/IMPALA-7486
> Project: IMPALA
> Issue Type: Improvement
> Components: Backend
> Reporter: Tim Armstrong
> Assignee: Bikramjeet Vig
> Priority: Major
> Labels: scalability
>
> Following on from IMPALA-7349, we should consider handling dedicated
> coordinators specially rather than admitting a uniform amount of memory on
> all backends.
> The specific scenario I'm interested in targeting is the case where we a
> coordinator that is executing many "lightweight" coordinator fragments, e.g.
> just an ExchangeNode and PlanRootSink, plus maybe other lightweight operators
> like UnionNode that don't use much memory or CPU. With the current behaviour
> it's possible for a coordinator to reach capacity from the point-of-view of
> admission control when at runtime it is actually very lightly loaded.
> This is particularly true if coordinators and executors have different
> process mem limits. This will be somewhat common since they're often deployed
> on different hardware or the coordinator will have more memory dedicated to
> its embedded JVM for the catalog cache.
> More generally we could admit different amounts per backend depending on how
> many fragments are running, but I think this incremental step would address
> the most important cases and be a little easier to understand.
> We may want to defer this work until we've implemented distributed runtime
> filter aggregation, which will significantly reduce coordinator memory
> pressure, and until we've improved distributed overadmission (since the
> coordinator behaviour may help throttle overadmission ).
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-all-unsubscr...@impala.apache.org
For additional commands, e-mail: issues-all-h...@impala.apache.org
