Tim Armstrong created IMPALA-6847:
-------------------------------------
Summary: Consider adding workaround for high memory estimates in
admission control
Key: IMPALA-6847
URL: https://issues.apache.org/jira/browse/IMPALA-6847
Project: IMPALA
Issue Type: Improvement
Components: Frontend
Affects Versions: Product Backlog
Reporter: Tim Armstrong
Assignee: Tim Armstrong
A scenario we've seen play out a couple of times is this.
1. An Impala admin sets up memory-based admission control but sets the default
query mem_limit to 0. This means that admission control will use memory
estimates instead of mem_limit. Typically admins want some protection from
large queries consuming excessive memory but can't or don't want to set a
single mem_limit because the workload is unknown or unpredictable. This
configuration has caveats and we recommend against it, but this happens and
often works well enough as long as the workload is comprised of relatively
simple queries. The caveats include:
* There is no enforcement that a query stays within the memory estimate. This
means that a query can fail or force other queries to fail.
* Memory estimates are often inaccurate (this is unavoidable since they depend
on cardinality estimates, which are commonly off by 10x even with
state-of-the-art query planners). This means that runnable queries may be
impossible to admit without setting a mem_limit.
The last problem is the problem this JIRA is intending to work around. The
preferred solutions, which work most of the time, are:
1. Configure a default query memory limit.
2. Set a mem_limit for the problematic query
It would however, be useful to have a third fallback in cases where the first
two options are difficult or impossible to apply. The basic requirement is to
allow an administrator to configure a resource pool such that queries with very
high estimates can run.
The long-term solution to this is IMPALA-6460. This is a short-term workaround
that can tide users over until we have a comprehensive solution.
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
