[
https://issues.apache.org/jira/browse/HDFS-17341?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Lei Yang updated HDFS-17341:
----------------------------
Description:
Some service users today in namenode like ETL, metrics collection, ad-hoc users
that are critical to run business critical job accounts for many traffic in
namenode and shouldn't be throttled the same way as other individual users in
FCQ.
There is [feature|https://issues.apache.org/jira/browse/HADOOP-17165] in
namenode to always prioritize some service users to not subject to FCQ
scheduling. (Those users are always p0) but it is not perfect and it doesn't
account for traffic surge from those users.
The idea is to allocate dedicated rpc queues for those service users with
bounded queue capacity and allocate processing weight for those users. If queue
is full, those users are expected to backoff and retry.
New configs:
{code:java}
"faircallqueue.reserved.users"; // list of service users that are assigned to
dedicated queue
"faircallqueue.reserved.users.max"; // max number of service users allowed
"faircallqueue.reserved.users.capacities"; // custom queue capacities for each
service user
"faircallqueue.multiplexer.reserved.weights"; // processing weights for each
dedicated queue{code}
For instance, for a FCQ with 4 priority levels, 2 reserved users(a, b)
FCQ would look like:
{code:java}
P0: shared queue
P1: shared queue
P2: shared queue
P3: shared queue
P4: dedicated for user a
P5: dedicated for user b{code}
{color:#172b4d}The Multiplexer would have following weights{color}
{color:#172b4d}shared queue default weights: [8, 4, 2, 1]{color}
{color:#172b4d}reserved queue weights=[3, 2]{color}
{color:#172b4d}So user a gets 15% of total cycles, user b gets 10% of total
cycles.{color}
was:
Some service users today in namenode like ETL, metrics collection, ad-hoc users
that are critical to run business critical job accounts for many traffic in
namenode and shouldn't be throttled the same way as other individual users in
FCQ.
There is feature in namenode to always prioritize some service users to not
subject to FCQ scheduling. (Those users are always p0) but it is not perfect
and it doesn't account for traffic surge from those users.
The idea is to allocate dedicated rpc queues for those service users with
bounded queue capacity and allocate processing weight for those users. If queue
is full, those users are expected to backoff and retry.
New configs:
{code:java}
"faircallqueue.reserved.users"; // list of service users that are assigned to
dedicated queue
"faircallqueue.reserved.users.max"; // max number of service users allowed
"faircallqueue.reserved.users.capacities"; // custom queue capacities for each
service user
"faircallqueue.multiplexer.reserved.weights"; // processing weights for each
dedicated queue{code}
For instance, for a FCQ with 4 priority levels, 2 reserved users(a, b)
FCQ would look like:
{code:java}
P0: shared queue
P1: shared queue
P2: shared queue
P3: shared queue
P4: dedicated for user a
P5: dedicated for user b{code}
{color:#172b4d}The Multiplexer would have following weights{color}
{color:#172b4d}shared queue default weights: [8, 4, 2, 1]{color}
{color:#172b4d}reserved queue weights=[3, 2]{color}
{color:#172b4d}So user a gets 15% of total cycles, user b gets 10% of total
cycles.{color}
> Support dedicated user queues in Namenode FairCallQueue
> -------------------------------------------------------
>
> Key: HDFS-17341
> URL: https://issues.apache.org/jira/browse/HDFS-17341
> Project: Hadoop HDFS
> Issue Type: Improvement
> Components: namenode
> Affects Versions: 2.10.0, 3.4.0
> Reporter: Lei Yang
> Priority: Major
> Labels: pull-request-available
>
> Some service users today in namenode like ETL, metrics collection, ad-hoc
> users that are critical to run business critical job accounts for many
> traffic in namenode and shouldn't be throttled the same way as other
> individual users in FCQ.
> There is [feature|https://issues.apache.org/jira/browse/HADOOP-17165] in
> namenode to always prioritize some service users to not subject to FCQ
> scheduling. (Those users are always p0) but it is not perfect and it doesn't
> account for traffic surge from those users.
> The idea is to allocate dedicated rpc queues for those service users with
> bounded queue capacity and allocate processing weight for those users. If
> queue is full, those users are expected to backoff and retry.
>
> New configs:
> {code:java}
> "faircallqueue.reserved.users"; // list of service users that are assigned to
> dedicated queue
> "faircallqueue.reserved.users.max"; // max number of service users allowed
> "faircallqueue.reserved.users.capacities"; // custom queue capacities for
> each service user
> "faircallqueue.multiplexer.reserved.weights"; // processing weights for each
> dedicated queue{code}
> For instance, for a FCQ with 4 priority levels, 2 reserved users(a, b)
> FCQ would look like:
>
> {code:java}
> P0: shared queue
> P1: shared queue
> P2: shared queue
> P3: shared queue
> P4: dedicated for user a
> P5: dedicated for user b{code}
> {color:#172b4d}The Multiplexer would have following weights{color}
> {color:#172b4d}shared queue default weights: [8, 4, 2, 1]{color}
> {color:#172b4d}reserved queue weights=[3, 2]{color}
> {color:#172b4d}So user a gets 15% of total cycles, user b gets 10% of total
> cycles.{color}
>
>
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
---------------------------------------------------------------------
To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
