Hello Impala Public Jenkins,
I'd like you to reexamine a change. Please visit
http://gerrit.cloudera.org:8080/21616
to look at the new patch set (#3).
Change subject: IMPALA-12345: Add user quotas to Admission Control
......................................................................
IMPALA-12345: Add user quotas to Admission Control
Allow administrators to configure per user limits on queries that can
run in the Impala system.
In order to do this, there are two parts. Firstly we must track the
total counts of queries in the system on a per-user basis. Secondly
there must be a user model that allows rules that control per-user
limits on the number of queries that can be run.
TPoolStats (the data that is shared between Admission Control instances)
is extended to include a map from user name to a count of queries
running. This (along with some derived data structures) is updated when
queries are queued and when they are released from Admission Control.
This lifecycle is slightly different from other TPoolStats data which
usually tracks data about queries that are running. Queries can be
rejected because of user quotas at submission time. This is done for
two reasons: (1) as queries can only be admitted from the front of the
queue and we do not want to block other queries due to quotas, and
(2) it is easy for users to understand what is going on when queries
are rejected at submission time.
Note that when running in configurations without an Admission Daemon
then Admission Control does not have perfect information about the
system and over-admission is possible for User-Level Admission Quotas
in the same way that it is for other Admission Control controls.
The User Model is implemented by extending the format of the
fair-scheduler.xml file.
Two new elements ‘userQueryLimit’ and ‘groupQueryLimit’ can be added to
the fair-scheduler.xml file. These elements can be placed on the root
configuration, which applies to all pools, or the pool configuration.
Each element contains a comma separated list of names, and an integer
value. In ‘userQueryLimit’ the names are the short names of users, or a
wildcard name “*” which matches all users. In ‘groupQueryLimit’ the
names are group names. Each name/value pair constitutes a rule limiting
the number of concurrent queries that a user can run. The root level
rules and pool level rules must both be passed for a new query to be
queued. The rules dictate a maximum number of queries that can run.
When evaluating rules at either the root level, or at the pool level,
when a rule matches a user then there is no more evaluation done.
To support reading the ‘userQueryLimit’ and ‘groupQueryLimit’ fields the
RequestPoolService is enhanced.
If user quotas are enabled for a pool then a list of the users with
running or queued queries in that pool is visible on the coordinator
webui admission control page.
More comprehensive documentation of the user model will be provided in
IMPALA-12943
TESTING
New end-to-end tests are added to test_admission_controller.py, and
admission-controller-test is extended to provide unit tests for the
user model.
Change-Id: I4c33f3f2427db57fb9b6c593a4b22d5029549b41
---
M be/src/common/global-flags.cc
M be/src/scheduling/admission-controller-test.cc
M be/src/scheduling/admission-controller.cc
M be/src/scheduling/admission-controller.h
M be/src/scheduling/cluster-membership-mgr.h
M be/src/util/backend-gflag-util.cc
M be/src/util/collection-metrics.h
M be/src/util/runtime-profile.h
M common/thrift/BackendGflags.thrift
M common/thrift/ImpalaInternalService.thrift
M common/thrift/StatestoreService.thrift
M common/thrift/metrics.json
M fe/src/main/java/org/apache/impala/common/JniUtil.java
M fe/src/main/java/org/apache/impala/service/BackendConfig.java
M fe/src/main/java/org/apache/impala/service/JniFrontend.java
M fe/src/main/java/org/apache/impala/util/RequestPoolService.java
M fe/src/test/java/org/apache/impala/util/JniUtilTest.java
M fe/src/test/java/org/apache/impala/util/TestRequestPoolService.java
M fe/src/test/resources/fair-scheduler-test.xml
M fe/src/test/resources/fair-scheduler-test2.xml
A fe/src/test/resources/fair-scheduler-test3.xml
M
java/yarn-extras/src/main/java/org/apache/impala/yarn/server/resourcemanager/scheduler/fair/AllocationConfiguration.java
M
java/yarn-extras/src/main/java/org/apache/impala/yarn/server/resourcemanager/scheduler/fair/AllocationFileLoaderService.java
M tests/custom_cluster/test_admission_controller.py
M www/admission_controller.tmpl
25 files changed, 1,887 insertions(+), 435 deletions(-)
git pull ssh://gerrit.cloudera.org:29418/Impala-ASF refs/changes/16/21616/3
--
To view, visit http://gerrit.cloudera.org:8080/21616
To unsubscribe, visit http://gerrit.cloudera.org:8080/settings
Gerrit-Project: Impala-ASF
Gerrit-Branch: master
Gerrit-MessageType: newpatchset
Gerrit-Change-Id: I4c33f3f2427db57fb9b6c593a4b22d5029549b41
Gerrit-Change-Number: 21616
Gerrit-PatchSet: 3
Gerrit-Owner: Andrew Sherman <asher...@cloudera.com>
Gerrit-Reviewer: Impala Public Jenkins <impala-public-jenk...@cloudera.com>
