[
https://issues.apache.org/jira/browse/HDFS-17302?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Jian Zhang updated HDFS-17302:
------------------------------
Summary: RBF: ProportionRouterRpcFairnessPolicyController-Sharing and
Isolating. (was: RBF: ProportionRouterRpcFairnessPolicyController-support
proportional allocation of semaphores)
> RBF: ProportionRouterRpcFairnessPolicyController-Sharing and Isolating.
> -----------------------------------------------------------------------
>
> Key: HDFS-17302
> URL: https://issues.apache.org/jira/browse/HDFS-17302
> Project: Hadoop HDFS
> Issue Type: New Feature
> Components: rbf
> Reporter: Jian Zhang
> Assignee: Jian Zhang
> Priority: Major
> Labels: pull-request-available
> Attachments: HDFS-17302.001.patch, HDFS-17302.002.patch
>
>
> h2. Current shortcomings
> [HDFS-14090|https://issues.apache.org/jira/browse/HDFS-14090] provides a
> StaticRouterRpcFairnessPolicyController to support configuring different
> handlers for different ns. Using the StaticRouterRpcFairnessPolicyController
> allows the router to isolate different ns, and the ns with a higher load will
> not affect the router's access to the ns with a normal load. But the
> StaticRouterRpcFairnessPolicyController still falls short in many ways, such
> as:
> 1. *Configuration is inconvenient and error-prone*: When I use
> StaticRouterRpcFairnessPolicyController, I first need to know how many
> handlers the router has in total, then I have to know how many nameservices
> the router currently has, and then carefully calculate how many handlers to
> allocate to each ns so that the sum of handlers for all ns will not exceed
> the total handlers of the router, and I also need to consider how many
> handlers to allocate to each ns to achieve better performance. Therefore, I
> need to be very careful when configuring. Even if I configure only one more
> handler for a certain ns, the total number is more than the number of
> handlers owned by the router, which will also cause the router to fail to
> start. At this time, I had to investigate the reason why the router failed to
> start. After finding the reason, I had to reconsider the number of handlers
> for each ns.
> 2. *Extension ns is not supported*: During the running of the router, if a
> new ns is added to the cluster and a mount is added for the ns, but because
> no handler is allocated for the ns, the ns cannot be accessed through the
> router. We must reconfigure the number of handlers and then refresh the
> configuration. At this time, the router can access the ns normally. When we
> reconfigure the number of handlers, we have to face disadvantage 1:
> Configuration is inconvenient and error-prone.
> 3. *Waste handlers*: The main purpose of proposing
> RouterRpcFairnessPolicyController is to enable the router to access ns with
> normal load and not be affected by ns with higher load. First of all, not all
> ns have high loads; secondly, ns with high loads do not have high loads 24
> hours a day. It may be that only certain time periods, such as 0 to 8
> o'clock, have high loads, and other time periods have normal loads. Assume
> there are 2 ns, and each ns is allocated half of the number of handlers.
> Assume that ns1 has many requests from 0 to 14 o'clock, and almost no
> requests from 14 to 24 o'clock, ns2 has many requests from 12 to 24 o'clock,
> and almost no requests from 0 to 14 o'clock; when it is between 0 o'clock and
> 12 o'clock and between 14 o'clock and 24 o'clock, only one ns has more
> requests and the other ns has almost no requests, so we have wasted half of
> the number of handlers.
> 4. *Only isolation, no sharing*: The staticRouterRpcFairnessPolicyController
> does not support sharing, only isolation. I think isolation is just a means
> to improve the performance of router access to normal ns, not the purpose. It
> is impossible for all ns in the cluster to have high loads. On the contrary,
> in most scenarios, only a few ns in the cluster have high loads, and the
> loads of most other ns are normal. For ns with higher load and ns with normal
> load, we need to isolate their handlers so that the ns with higher load will
> not affect the performance of ns with lower load. However, for nameservices
> that are also under normal load, or are under higher load, we do not need to
> isolate them, these ns of the same nature can share the handlers of the
> router; The performance is better than assigning a fixed number of handlers
> to each ns, because each ns can use all the handlers of the router.
> h2. New features
> Based on the above staticRouterRpcFairnessPolicyController, there are
> deficiencies in usage and performance. I provide a new
> RouterRpcFairnessPolicyController:
> ProportionRouterRpcFairnessPolicyController (maybe with a better name) to
> solve the above major shortcomings.
> 1. *More user-friendly configuration* : Supports allocating handlers
> proportionally to each ns. For example, we can give ns1 a handler ratio of
> 0.2, then ns1 will use 0.2 of the total number of handlers on the router.
> Using this method, we do not need to confirm in advance how many handlers the
> router has.
> 2. *Sharing* :
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
---------------------------------------------------------------------
To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
