[jira] [Updated] (HDFS-17302) RBF: ProportionRouterRpcFairnessPolicyController-support proportional allocation of semaphores
[ https://issues.apache.org/jira/browse/HDFS-17302?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Jian Zhang updated HDFS-17302: -- Description: h2. Current shortcomings [HDFS-14090|https://issues.apache.org/jira/browse/HDFS-14090] provides a StaticRouterRpcFairnessPolicyController to support configuring different handlers for different ns. Using the StaticRouterRpcFairnessPolicyController allows the router to isolate different ns, and the ns with a higher load will not affect the router's access to the ns with a normal load. But the StaticRouterRpcFairnessPolicyController still falls short in many ways, such as: 1. *Configuration is inconvenient and error-prone*: When I use StaticRouterRpcFairnessPolicyController, I first need to know how many handlers the router has in total, then I have to know how many nameservices the router currently has, and then carefully calculate how many handlers to allocate to each ns so that the sum of handlers for all ns will not exceed the total handlers of the router, and I also need to consider how many handlers to allocate to each ns to achieve better performance. Therefore, I need to be very careful when configuring. Even if I configure only one more handler for a certain ns, the total number is more than the number of handlers owned by the router, which will also cause the router to fail to start. At this time, I had to investigate the reason why the router failed to start. After finding the reason, I had to reconsider the number of handlers for each ns. 2. *Extension ns is not supported*: During the running of the router, if a new ns is added to the cluster and a mount is added for the ns, but because no handler is allocated for the ns, the ns cannot be accessed through the router. We must reconfigure the number of handlers and then refresh the configuration. At this time, the router can access the ns normally. When we reconfigure the number of handlers, we have to face disadvantage 1: Configuration is inconvenient and error-prone. 3. *Waste handlers*: The main purpose of proposing RouterRpcFairnessPolicyController is to enable the router to access ns with normal load and not be affected by ns with higher load. First of all, not all ns have high loads; secondly, ns with high loads do not have high loads 24 hours a day. It may be that only certain time periods, such as 0 to 8 o'clock, have high loads, and other time periods have normal loads. Assume there are 2 ns, and each ns is allocated half of the number of handlers. Assume that ns1 has many requests from 0 to 14 o'clock, and almost no requests from 14 to 24 o'clock, ns2 has many requests from 12 to 24 o'clock, and almost no requests from 0 to 14 o'clock; when it is between 0 o'clock and 12 o'clock and between 14 o'clock and 24 o'clock, only one ns has more requests and the other ns has almost no requests, so we have wasted half of the number of handlers. 4. *Only isolation, no sharing*: The staticRouterRpcFairnessPolicyController does not support sharing, only isolation. I think isolation is just a means to improve the performance of router access to normal ns, not the purpose. It is impossible for all ns in the cluster to have high loads. On the contrary, in most scenarios, only a few ns in the cluster have high loads, and the loads of most other ns are normal. For ns with higher load and ns with normal load, we need to isolate their handlers so that the ns with higher load will not affect the performance of ns with lower load. However, for nameservices that are also under normal load, or are under higher load, we do not need to isolate them, these ns of the same nature can share the handlers of the router; The performance is better than assigning a fixed number of handlers to each ns, because each ns can use all the handlers of the router. h2. New features Based on the above staticRouterRpcFairnessPolicyController, there are deficiencies in usage and performance. I provide a new RouterRpcFairnessPolicyController: ProportionRouterRpcFairnessPolicyController (maybe with a better name) to solve the above major shortcomings. 1. *More user-friendly configuration* : Supports allocating handlers proportionally to each ns. For example, we can give ns1 a handler ratio of 0.2, then ns1 will use 0.2 of the total number of handlers on the router. Using this method, we do not need to confirm in advance how many handlers the router has. 2. *Sharing* : was: [HDFS-14090|https://issues.apache.org/jira/browse/HDFS-14090] provides a StaticRouterRpcFairnessPolicyController to support configuring different handlers for different ns. Using the StaticRouterRpcFairnessPolicyController allows the router to isolate different ns, and the ns with a higher load will not affect the router's access to the ns with a normal load. But the StaticRouterRpcFairnessPolicyController still falls short in many ways,
[jira] [Updated] (HDFS-17302) RBF: ProportionRouterRpcFairnessPolicyController-support proportional allocation of semaphores
[ https://issues.apache.org/jira/browse/HDFS-17302?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Jian Zhang updated HDFS-17302: -- Description: [HDFS-14090|https://issues.apache.org/jira/browse/HDFS-14090] provides a StaticRouterRpcFairnessPolicyController to support configuring different handlers for different ns. Using the StaticRouterRpcFairnessPolicyController allows the router to isolate different ns, and the ns with a higher load will not affect the router's access to the ns with a normal load. But the StaticRouterRpcFairnessPolicyController still falls short in many ways, such as: 1. *Configuration is inconvenient and error-prone*: When I use StaticRouterRpcFairnessPolicyController, I first need to know how many handlers the router has in total, then I have to know how many nameservices the router currently has, and then carefully calculate how many handlers to allocate to each ns so that the sum of handlers for all ns will not exceed the total handlers of the router, and I also need to consider how many handlers to allocate to each ns to achieve better performance. Therefore, I need to be very careful when configuring. Even if I configure only one more handler for a certain ns, the total number is more than the number of handlers owned by the router, which will also cause the router to fail to start. At this time, I had to investigate the reason why the router failed to start. After finding the reason, I had to reconsider the number of handlers for each ns. 2. *Extension ns is not supported*: During the running of the router, if a new ns is added to the cluster and a mount is added for the ns, but because no handler is allocated for the ns, the ns cannot be accessed through the router. We must reconfigure the number of handlers and then refresh the configuration. At this time, the router can access the ns normally. When we reconfigure the number of handlers, we have to face disadvantage 1: Configuration is inconvenient and error-prone. 3. *Waste handlers*: The main purpose of proposing RouterRpcFairnessPolicyController is to enable the router to access ns with normal load and not be affected by ns with higher load. First of all, not all ns have high loads; secondly, ns with high loads do not have high loads 24 hours a day. It may be that only certain time periods, such as 0 to 8 o'clock, have high loads, and other time periods have normal loads. Assume there are 2 ns, and each ns is allocated half of the number of handlers. Assume that ns1 has many requests from 0 to 14 o'clock, and almost no requests from 14 to 24 o'clock, ns2 has many requests from 12 to 24 o'clock, and almost no requests from 0 to 14 o'clock; when it is between 0 o'clock and 12 o'clock and between 14 o'clock and 24 o'clock, only one ns has more requests and the other ns has almost no requests, so we have wasted half of the number of handlers. 4. *Only isolation, no sharing*: The staticRouterRpcFairnessPolicyController does not support sharing, only isolation. I think isolation is just a means to improve the performance of router access to normal ns, not the purpose. It is impossible for all ns in the cluster to have high loads. On the contrary, in most scenarios, only a few ns in the cluster have high loads, and the loads of most other ns are normal. For ns with higher load and ns with normal load, we need to isolate their handlers so that the ns with higher load will not affect the performance of ns with lower load. However, for nameservices that are also under normal load, or are under higher load, we do not need to isolate them, these ns of the same nature can share the handlers of the router; The performance is better than assigning a fixed number of handlers to each ns, because each ns can use all the handlers of the router. Based on the above staticRouterRpcFairnessPolicyController, there are deficiencies in usage and performance. I provide a new RouterRpcFairnessPolicyController: ProportionRouterRpcFairnessPolicyController (maybe with a better name) to solve the above major shortcomings. 1.* More user-friendly configuration*: Supports allocating handlers proportionally to each ns. was: [HDFS-14090|https://issues.apache.org/jira/browse/HDFS-14090] provides a StaticRouterRpcFairnessPolicyController to support configuring different handlers for different ns. Using the StaticRouterRpcFairnessPolicyController allows the router to isolate different ns, and the ns with a higher load will not affect the router's access to the ns with a normal load. But the StaticRouterRpcFairnessPolicyController still falls short in many ways, such as: 1. *Configuration is inconvenient and error-prone*: When I use StaticRouterRpcFairnessPolicyController, I first need to know how many handlers the router has in total, then I have to know how many nameservices the router currently has, and then carefully
[jira] [Updated] (HDFS-17302) RBF: ProportionRouterRpcFairnessPolicyController-support proportional allocation of semaphores
[ https://issues.apache.org/jira/browse/HDFS-17302?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Jian Zhang updated HDFS-17302: -- Description: [HDFS-14090|https://issues.apache.org/jira/browse/HDFS-14090] provides a StaticRouterRpcFairnessPolicyController to support configuring different handlers for different ns. Using the StaticRouterRpcFairnessPolicyController allows the router to isolate different ns, and the ns with a higher load will not affect the router's access to the ns with a normal load. But the StaticRouterRpcFairnessPolicyController still falls short in many ways, such as: 1. *Configuration is inconvenient and error-prone*: When I use StaticRouterRpcFairnessPolicyController, I first need to know how many handlers the router has in total, then I have to know how many nameservices the router currently has, and then carefully calculate how many handlers to allocate to each ns so that the sum of handlers for all ns will not exceed the total handlers of the router, and I also need to consider how many handlers to allocate to each ns to achieve better performance. Therefore, I need to be very careful when configuring. Even if I configure only one more handler for a certain ns, the total number is more than the number of handlers owned by the router, which will also cause the router to fail to start. At this time, I had to investigate the reason why the router failed to start. After finding the reason, I had to reconsider the number of handlers for each ns. 2. *Extension ns is not supported*: During the running of the router, if a new ns is added to the cluster and a mount is added for the ns, but because no handler is allocated for the ns, the ns cannot be accessed through the router. We must reconfigure the number of handlers and then refresh the configuration. At this time, the router can access the ns normally. When we reconfigure the number of handlers, we have to face disadvantage 1: Configuration is inconvenient and error-prone. 3. *Waste handlers*: The main purpose of proposing RouterRpcFairnessPolicyController is to enable the router to access ns with normal load and not be affected by ns with higher load. First of all, not all ns have high loads; secondly, ns with high loads do not have high loads 24 hours a day. It may be that only certain time periods, such as 0 to 8 o'clock, have high loads, and other time periods have normal loads. Assume there are 2 ns, and each ns is allocated half of the number of handlers. Assume that ns1 has many requests from 0 to 14 o'clock, and almost no requests from 14 to 24 o'clock, ns2 has many requests from 12 to 24 o'clock, and almost no requests from 0 to 14 o'clock; when it is between 0 o'clock and 12 o'clock and between 14 o'clock and 24 o'clock, only one ns has more requests and the other ns has almost no requests, so we have wasted half of the number of handlers. 4. *Only isolation, no sharing*: The staticRouterRpcFairnessPolicyController does not support sharing, only isolation. I think isolation is just a means to improve the performance of router access to normal ns, not the purpose. It is impossible for all ns in the cluster to have high loads. On the contrary, in most scenarios, only a few ns in the cluster have high loads, and the loads of most other ns are normal. For ns with higher load and ns with normal load, we need to isolate their handlers so that the ns with higher load will not affect the performance of ns with lower load. However, for nameservices that are also under normal load, or are under higher load, we do not need to isolate them, these ns of the same nature can share the handlers of the router; The performance is better than assigning a fixed number of handlers to each ns, because each ns can use all the handlers of the router. Based on the above staticRouterRpcFairnessPolicyController, there are deficiencies in usage and performance. I provide a new RouterRpcFairnessPolicyController: ProportionRouterRpcFairnessPolicyController (maybe with a better name) to solve the above major shortcomings. was: [HDFS-14090|https://issues.apache.org/jira/browse/HDFS-14090] provides a StaticRouterRpcFairnessPolicyController to support configuring different handlers for different ns. Using the StaticRouterRpcFairnessPolicyController allows the router to isolate different ns, and the ns with a higher load will not affect the router's access to the ns with a normal load. But the StaticRouterRpcFairnessPolicyController still falls short in many ways, such as: 1. *Configuration is inconvenient and error-prone*: When I use StaticRouterRpcFairnessPolicyController, I first need to know how many handlers the router has in total, then I have to know how many nameservices the router currently has, and then carefully calculate how many handlers to allocate to each ns so that the sum of handlers for all ns will not
[jira] [Updated] (HDFS-17302) RBF: ProportionRouterRpcFairnessPolicyController-support proportional allocation of semaphores
[ https://issues.apache.org/jira/browse/HDFS-17302?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Jian Zhang updated HDFS-17302: -- Description: [HDFS-14090|https://issues.apache.org/jira/browse/HDFS-14090] provides a StaticRouterRpcFairnessPolicyController to support configuring different handlers for different ns. Using the StaticRouterRpcFairnessPolicyController allows the router to isolate different ns, and the ns with a higher load will not affect the router's access to the ns with a normal load. But the StaticRouterRpcFairnessPolicyController still falls short in many ways, such as: 1. *Configuration is inconvenient and error-prone*: When I use StaticRouterRpcFairnessPolicyController, I first need to know how many handlers the router has in total, then I have to know how many nameservices the router currently has, and then carefully calculate how many handlers to allocate to each ns so that the sum of handlers for all ns will not exceed the total handlers of the router, and I also need to consider how many handlers to allocate to each ns to achieve better performance. Therefore, I need to be very careful when configuring. Even if I configure only one more handler for a certain ns, the total number is more than the number of handlers owned by the router, which will also cause the router to fail to start. At this time, I had to investigate the reason why the router failed to start. After finding the reason, I had to reconsider the number of handlers for each ns. 2. *Extension ns is not supported*: During the running of the router, if a new ns is added to the cluster and a mount is added for the ns, but because no handler is allocated for the ns, the ns cannot be accessed through the router. We must reconfigure the number of handlers and then refresh the configuration. At this time, the router can access the ns normally. When we reconfigure the number of handlers, we have to face disadvantage 1: Configuration is inconvenient and error-prone. 3. *Waste handlers*: The main purpose of proposing RouterRpcFairnessPolicyController is to enable the router to access ns with normal load and not be affected by ns with higher load. First of all, not all ns have high loads; secondly, ns with high loads do not have high loads 24 hours a day. It may be that only certain time periods, such as 0 to 8 o'clock, have high loads, and other time periods have normal loads. Assume there are 2 ns, and each ns is allocated half of the number of handlers. Assume that ns1 has many requests from 0 to 14 o'clock, and almost no requests from 14 to 24 o'clock, ns2 has many requests from 12 to 24 o'clock, and almost no requests from 0 to 14 o'clock; when it is between 0 o'clock and 12 o'clock and between 14 o'clock and 24 o'clock, only one ns has more requests and the other ns has almost no requests, so we have wasted half of the number of handlers. 4. *Only isolation, no sharing*: The staticRouterRpcFairnessPolicyController does not support sharing, only isolation. I think isolation is just a means to improve the performance of router access to normal ns, not the purpose. It is impossible for all ns in the cluster to have high loads. On the contrary, in most scenarios, only a few ns in the cluster have high loads, and the loads of most other ns are normal. For ns with higher load and ns with normal load, we need to isolate their handlers so that the ns with higher load will not affect the performance of ns with lower load. However, for nameservices that are also under normal load, or are under higher load, we do not need to isolate them, these ns of the same nature can share the handlers of the router; The performance is better than assigning a fixed number of handlers to each ns, because each ns can use all the handlers of the router. Based on the above staticRouterRpcFairnessPolicyController, there are deficiencies in usage and performance. was: [HDFS-14090|https://issues.apache.org/jira/browse/HDFS-14090] provides a StaticRouterRpcFairnessPolicyController to support configuring different handlers for different ns. Using the StaticRouterRpcFairnessPolicyController allows the router to isolate different ns, and the ns with a higher load will not affect the router's access to the ns with a normal load. But the StaticRouterRpcFairnessPolicyController still falls short in many ways, such as: 1. *Configuration is inconvenient and error-prone*: When I use StaticRouterRpcFairnessPolicyController, I first need to know how many handlers the router has in total, then I have to know how many nameservices the router currently has, and then carefully calculate how many handlers to allocate to each ns so that the sum of handlers for all ns will not exceed the total handlers of the router, and I also need to consider how many handlers to allocate to each ns to achieve better performance. Therefore, I need to be
[jira] [Updated] (HDFS-17302) RBF: ProportionRouterRpcFairnessPolicyController-support proportional allocation of semaphores
[ https://issues.apache.org/jira/browse/HDFS-17302?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Jian Zhang updated HDFS-17302: -- Description: [HDFS-14090|https://issues.apache.org/jira/browse/HDFS-14090] provides a StaticRouterRpcFairnessPolicyController to support configuring different handlers for different ns. Using the StaticRouterRpcFairnessPolicyController allows the router to isolate different ns, and the ns with a higher load will not affect the router's access to the ns with a normal load. But the StaticRouterRpcFairnessPolicyController still falls short in many ways, such as: 1. *Configuration is inconvenient and error-prone*: When I use StaticRouterRpcFairnessPolicyController, I first need to know how many handlers the router has in total, then I have to know how many nameservices the router currently has, and then carefully calculate how many handlers to allocate to each ns so that the sum of handlers for all ns will not exceed the total handlers of the router, and I also need to consider how many handlers to allocate to each ns to achieve better performance. Therefore, I need to be very careful when configuring. Even if I configure only one more handler for a certain ns, the total number is more than the number of handlers owned by the router, which will also cause the router to fail to start. At this time, I had to investigate the reason why the router failed to start. After finding the reason, I had to reconsider the number of handlers for each ns. 2. *Extension ns is not supported*: During the running of the router, if a new ns is added to the cluster and a mount is added for the ns, but because no handler is allocated for the ns, the ns cannot be accessed through the router. We must reconfigure the number of handlers and then refresh the configuration. At this time, the router can access the ns normally. When we reconfigure the number of handlers, we have to face disadvantage 1: Configuration is inconvenient and error-prone. 3. *Waste handlers*: The main purpose of proposing RouterRpcFairnessPolicyController is to enable the router to access ns with normal load and not be affected by ns with higher load. First of all, not all ns have high loads; secondly, ns with high loads do not have high loads 24 hours a day. It may be that only certain time periods, such as 0 to 8 o'clock, have high loads, and other time periods have normal loads. Assume there are 2 ns, and each ns is allocated half of the number of handlers. Assume that ns1 has many requests from 0 to 14 o'clock, and almost no requests from 14 to 24 o'clock, ns2 has many requests from 12 to 24 o'clock, and almost no requests from 0 to 14 o'clock; when it is between 0 o'clock and 12 o'clock and between 14 o'clock and 24 o'clock, only one ns has more requests and the other ns has almost no requests, so we have wasted half of the number of handlers. 4.*Only isolation, no sharing*: The staticRouterRpcFairnessPolicyController does not support sharing, only isolation. I think isolation is just a means to improve the performance of router access to normal ns, not the purpose. It is impossible for all ns in the cluster to have high loads. On the contrary, in most scenarios, only a few ns in the cluster have high loads, and the loads of most other ns are normal. For ns with higher load and ns with normal load, we need to isolate their handlers so that the ns with higher load will not affect the performance of ns with lower load. However, for nameservices that are also under normal load, or are under higher load, we do not need to isolate them, these ns of the same nature can share the handlers of the router; The performance is better than assigning a fixed number of handlers to each ns, because each ns can use all the handlers of the router. was: [HDFS-14090|https://issues.apache.org/jira/browse/HDFS-14090] provides a StaticRouterRpcFairnessPolicyController to support configuring different handlers for different ns. Using the StaticRouterRpcFairnessPolicyController allows the router to isolate different ns, and the ns with a higher load will not affect the router's access to the ns with a normal load. But the StaticRouterRpcFairnessPolicyController still falls short in many ways, such as: 1. *Configuration is inconvenient and error-prone*: When I use StaticRouterRpcFairnessPolicyController, I first need to know how many handlers the router has in total, then I have to know how many nameservices the router currently has, and then carefully calculate how many handlers to allocate to each ns so that the sum of handlers for all ns will not exceed the total handlers of the router, and I also need to consider how many handlers to allocate to each ns to achieve better performance. Therefore, I need to be very careful when configuring. Even if I configure only one more handler for a certain ns, the total number is
[jira] [Updated] (HDFS-17302) RBF: ProportionRouterRpcFairnessPolicyController-support proportional allocation of semaphores
[ https://issues.apache.org/jira/browse/HDFS-17302?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Jian Zhang updated HDFS-17302: -- Description: [HDFS-14090|https://issues.apache.org/jira/browse/HDFS-14090] provides a StaticRouterRpcFairnessPolicyController to support configuring different handlers for different ns. Using the StaticRouterRpcFairnessPolicyController allows the router to isolate different ns, and the ns with a higher load will not affect the router's access to the ns with a normal load. But the StaticRouterRpcFairnessPolicyController still falls short in many ways, such as: 1. *Configuration is inconvenient and error-prone*: When I use StaticRouterRpcFairnessPolicyController, I first need to know how many handlers the router has in total, then I have to know how many nameservices the router currently has, and then carefully calculate how many handlers to allocate to each ns so that the sum of handlers for all ns will not exceed the total handlers of the router, and I also need to consider how many handlers to allocate to each ns to achieve better performance. Therefore, I need to be very careful when configuring. Even if I configure only one more handler for a certain ns, the total number is more than the number of handlers owned by the router, which will also cause the router to fail to start. At this time, I had to investigate the reason why the router failed to start. After finding the reason, I had to reconsider the number of handlers for each ns. 2. *Extension ns is not supported*: During the running of the router, if a new ns is added to the cluster and a mount is added for the ns, but because no handler is allocated for the ns, the ns cannot be accessed through the router. We must reconfigure the number of handlers and then refresh the configuration. At this time, the router can access the ns normally. When we reconfigure the number of handlers, we have to face disadvantage 1: Configuration is inconvenient and error-prone. 3. *Waste handlers*: The main purpose of proposing RouterRpcFairnessPolicyController is to enable the router to access ns with normal load and not be affected by ns with higher load. First of all, not all ns have high loads; secondly, ns with high loads do not have high loads 24 hours a day. It may be that only certain time periods, such as 0 to 8 o'clock, have high loads, and other time periods have normal loads. Assume there are 2 ns, and each ns is allocated half of the number of handlers. Assume that ns1 has many requests from 0 o'clock to 14 o'clock, and almost no requests from 14 o'clock to 24 o'clock. ns2 has many requests from 12 o'clock to 24 o'clock, and 0 o'clock - There are almost no requests at 12 o'clock; when it is between 0 o'clock and 12 o'clock and between 14 o'clock and 24 o'clock, only one ns has more requests and the other ns has almost no requests, so we have wasted half of the number of handlers. was: [HDFS-14090|https://issues.apache.org/jira/browse/HDFS-14090] provides a StaticRouterRpcFairnessPolicyController to support configuring different handlers for different ns. Using the StaticRouterRpcFairnessPolicyController allows the router to isolate different ns, and the ns with a higher load will not affect the router's access to the ns with a normal load. But the StaticRouterRpcFairnessPolicyController still falls short in many ways, such as: 1. *Configuration is inconvenient and error-prone*: When I use StaticRouterRpcFairnessPolicyController, I first need to know how many handlers the router has in total, then I have to know how many nameservices the router currently has, and then carefully calculate how many handlers to allocate to each ns so that the sum of handlers for all ns will not exceed the total handlers of the router, and I also need to consider how many handlers to allocate to each ns to achieve better performance. Therefore, I need to be very careful when configuring. Even if I configure only one more handler for a certain ns, the total number is more than the number of handlers owned by the router, which will also cause the router to fail to start. At this time, I had to investigate the reason why the router failed to start. After finding the reason, I had to reconsider the number of handlers for each ns. 2. *Extension ns is not supported*: During the running of the router, if a new ns is added to the cluster and a mount is added for the ns, but because no handler is allocated for the ns, the ns cannot be accessed through the router. We must reconfigure the number of handlers and then refresh the configuration. At this time, the router can access the ns normally. When we reconfigure the number of handlers, we have to face disadvantage 1: Configuration is inconvenient and error-prone. 3. *Cannot share handlers*: The main purpose of proposing RouterRpcFairnessPolicyController is to enable the
[jira] [Updated] (HDFS-17302) RBF: ProportionRouterRpcFairnessPolicyController-support proportional allocation of semaphores
[ https://issues.apache.org/jira/browse/HDFS-17302?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Jian Zhang updated HDFS-17302: -- Description: [HDFS-14090|https://issues.apache.org/jira/browse/HDFS-14090] provides a StaticRouterRpcFairnessPolicyController to support configuring different handlers for different ns. Using the StaticRouterRpcFairnessPolicyController allows the router to isolate different ns, and the ns with a higher load will not affect the router's access to the ns with a normal load. But the StaticRouterRpcFairnessPolicyController still falls short in many ways, such as: 1. *Configuration is inconvenient and error-prone*: When I use StaticRouterRpcFairnessPolicyController, I first need to know how many handlers the router has in total, then I have to know how many nameservices the router currently has, and then carefully calculate how many handlers to allocate to each ns so that the sum of handlers for all ns will not exceed the total handlers of the router, and I also need to consider how many handlers to allocate to each ns to achieve better performance. Therefore, I need to be very careful when configuring. Even if I configure only one more handler for a certain ns, the total number is more than the number of handlers owned by the router, which will also cause the router to fail to start. At this time, I had to investigate the reason why the router failed to start. After finding the reason, I had to reconsider the number of handlers for each ns. 2. *Extension ns is not supported*: During the running of the router, if a new ns is added to the cluster and a mount is added for the ns, but because no handler is allocated for the ns, the ns cannot be accessed through the router. We must reconfigure the number of handlers and then refresh the configuration. At this time, the router can access the ns normally. When we reconfigure the number of handlers, we have to face disadvantage 1: Configuration is inconvenient and error-prone. 3. *Cannot share handlers*: The main purpose of proposing RouterRpcFairnessPolicyController is to enable the router to access ns with normal load and not be affected by ns with higher load. First of all, not all ns have high loads; secondly, ns with high loads do not have high loads 24 hours a day. It may be that only certain time periods, such as 0 to 8 o'clock, have high loads, and other time periods have normal loads. Assume there are 2 ns, and each ns is allocated half of the number of handlers. Assume that ns1 has many requests from 0 o'clock to 14 o'clock, and almost no requests from 14 o'clock to 24 o'clock. ns2 has many requests from 12 o'clock to 24 o'clock, and 0 o'clock - There are almost no requests at 12 o'clock; when it is between 0 o'clock and 12 o'clock and between 14 o'clock and 24 o'clock, only one ns has more requests and the other ns has almost no requests, so we have wasted half of the number of handlers. was: [HDFS-14090|https://issues.apache.org/jira/browse/HDFS-14090] provides a StaticRouterRpcFairnessPolicyController to support configuring different handlers for different ns. Using the StaticRouterRpcFairnessPolicyController allows the router to isolate different ns, and the ns with a higher load will not affect the router's access to the ns with a normal load. But the StaticRouterRpcFairnessPolicyController still falls short in many ways, such as: 1. *Configuration is inconvenient and error-prone*: When I use StaticRouterRpcFairnessPolicyController, I first need to know how many handlers the router has in total, then I have to know how many nameservices the router currently has, and then carefully calculate how many handlers to allocate to each ns so that the sum of handlers for all ns will not exceed the total handlers of the router, and I also need to consider how many handlers to allocate to each ns to achieve better performance. Therefore, I need to be very careful when configuring. Even if I configure only one more handler for a certain ns, the total number is more than the number of hadnlers owned by the router, which will also cause the router to fail to start. At this time, I had to investigate the reason why the router failed to start. After finding the reason, I had to reconsider the number of handlers for each ns. 2. *Extension ns is not supported*: During the running of the router, if a new ns is added to the cluster and a mount is added for the ns, but because no handler is allocated for the ns, the ns cannot be accessed through the router. We must reconfigure the number of handlers and then refresh the configuration. At this time, the router can access the ns normally. When we reconfigure the number of handlers, we have to face disadvantage 1: Configuration is inconvenient and error-prone. 3. *Cannot share handlers*: > RBF: ProportionRouterRpcFairnessPolicyController-support proportional >
[jira] [Updated] (HDFS-17302) RBF: ProportionRouterRpcFairnessPolicyController-support proportional allocation of semaphores
[ https://issues.apache.org/jira/browse/HDFS-17302?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Jian Zhang updated HDFS-17302: -- Description: [HDFS-14090|https://issues.apache.org/jira/browse/HDFS-14090] provides a StaticRouterRpcFairnessPolicyController to support configuring different handlers for different ns. Using the StaticRouterRpcFairnessPolicyController allows the router to isolate different ns, and the ns with a higher load will not affect the router's access to the ns with a normal load. But the StaticRouterRpcFairnessPolicyController still falls short in many ways, such as: 1. *Configuration is inconvenient and error-prone*: When I use StaticRouterRpcFairnessPolicyController, I first need to know how many handlers the router has in total, then I have to know how many nameservices the router currently has, and then carefully calculate how many handlers to allocate to each ns so that the sum of handlers for all ns will not exceed the total handlers of the router, and I also need to consider how many handlers to allocate to each ns to achieve better performance. Therefore, I need to be very careful when configuring. Even if I configure only one more handler for a certain ns, the total number is more than the number of hadnlers owned by the router, which will also cause the router to fail to start. At this time, I had to investigate the reason why the router failed to start. After finding the reason, I had to reconsider the number of handlers for each ns. 2. *Extension ns is not supported*: During the running of the router, if a new ns is added to the cluster and a mount is added for the ns, but because no handler is allocated for the ns, the ns cannot be accessed through the router. We must reconfigure the number of handlers and then refresh the configuration. At this time, the router can access the ns normally. When we reconfigure the number of handlers, we have to face disadvantage 1: Configuration is inconvenient and error-prone. 3. *Cannot share handlers*: was: [HDFS-14090|https://issues.apache.org/jira/browse/HDFS-14090] provides a StaticRouterRpcFairnessPolicyController to support configuring different handlers for different ns. Using the StaticRouterRpcFairnessPolicyController allows the router to isolate different ns, and the ns with a higher load will not affect the router's access to the ns with a normal load. But the StaticRouterRpcFairnessPolicyController still falls short in many ways, such as: 1. Configuration is inconvenient and error-prone: When I use StaticRouterRpcFairnessPolicyController, I first need to know how many handlers the router has in total, then I have to know how many nameservices the router currently has, and then carefully calculate how many handlers to allocate to each ns so that the sum of handlers for all ns will not exceed the total handlers of the router, and I also need to consider how many handlers to allocate to each ns to achieve better performance. Therefore, I need to be very careful when configuring. Even if I configure only one more handler for a certain ns, the total number is more than the number of hadnlers owned by the router, which will also cause the router to fail to start. At this time, I had to investigate the reason why the router failed to start. After finding the reason, I had to reconsider the number of handlers for each ns. 2. Extension ns is not supported: During the running of the router, if a new ns is added to the cluster and a mount is added for the ns, but because no handler is allocated for the ns, the ns cannot be accessed through the router. We must reconfigure the number of handlers and then refresh the configuration. At this time, the router can access the ns normally. When we reconfigure the number of handlers, we have to face disadvantage 1: Configuration is inconvenient and error-prone. 3. Cannot share handlers: > RBF: ProportionRouterRpcFairnessPolicyController-support proportional > allocation of semaphores > -- > > Key: HDFS-17302 > URL: https://issues.apache.org/jira/browse/HDFS-17302 > Project: Hadoop HDFS > Issue Type: New Feature > Components: rbf >Reporter: Jian Zhang >Assignee: Jian Zhang >Priority: Major > Labels: pull-request-available > Attachments: HDFS-17302.001.patch, HDFS-17302.002.patch > > > [HDFS-14090|https://issues.apache.org/jira/browse/HDFS-14090] provides a > StaticRouterRpcFairnessPolicyController to support configuring different > handlers for different ns. Using the StaticRouterRpcFairnessPolicyController > allows the router to isolate different ns, and the ns with a higher load will > not affect the router's access to the ns with a normal load. But the >
[jira] [Updated] (HDFS-17302) RBF: ProportionRouterRpcFairnessPolicyController-support proportional allocation of semaphores
[ https://issues.apache.org/jira/browse/HDFS-17302?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Jian Zhang updated HDFS-17302: -- Description: [HDFS-14090|https://issues.apache.org/jira/browse/HDFS-14090] provides a StaticRouterRpcFairnessPolicyController to support configuring different handlers for different ns. Using the StaticRouterRpcFairnessPolicyController allows the router to isolate different ns, and the ns with a higher load will not affect the router's access to the ns with a normal load. But the StaticRouterRpcFairnessPolicyController still falls short in many ways, such as: 1. Configuration is inconvenient and error-prone: When I use StaticRouterRpcFairnessPolicyController, I first need to know how many handlers the router has in total, then I have to know how many nameservices the router currently has, and then carefully calculate how many handlers to allocate to each ns so that the sum of handlers for all ns will not exceed the total handlers of the router, and I also need to consider how many handlers to allocate to each ns to achieve better performance. Therefore, I need to be very careful when configuring. Even if I configure only one more handler for a certain ns, the total number is more than the number of hadnlers owned by the router, which will also cause the router to fail to start. At this time, I had to investigate the reason why the router failed to start. After finding the reason, I had to reconsider the number of handlers for each ns. 2. Extension ns is not supported: During the running of the router, if a new ns is added to the cluster and a mount is added for the ns, but because no handler is allocated for the ns, the ns cannot be accessed through the router. We must reconfigure the number of handlers and then refresh the configuration. At this time, the router can access the ns normally. When we reconfigure the number of handlers, we have to face disadvantage 1: Configuration is inconvenient and error-prone. 3. Cannot share handlers: was: [HDFS-14090|https://issues.apache.org/jira/browse/HDFS-14090] provides a StaticRouterRpcFairnessPolicyController to support configuring different handlers for different ns. Using the StaticRouterRpcFairnessPolicyController allows the router to isolate different ns, and the ns with a higher load will not affect the router's access to the ns with a normal load. But the StaticRouterRpcFairnessPolicyController still falls short in many ways, such as: 1. Configuration is inconvenient and error-prone: When I use StaticRouterRpcFairnessPolicyController, I first need to know how many handlers the router has in total, then I have to know how many nameservices the router currently has, and then carefully calculate how many handlers to allocate to each ns so that the sum of handlers for all ns will not exceed the total handlers of the router, and I also need to consider how many handlers to allocate to each ns to achieve better performance. Therefore, I need to be very careful when configuring. Even if I configure only one more handler for a certain ns, the total number is more than the number of hadnlers owned by the router, which will also cause the router to fail to start. At this time, I had to investigate the reason why the router failed to start. After finding the reason, I had to reconsider the number of handlers for each ns. 2. Extension ns is not supported: During the running of the router, if a new ns is added to the cluster and a mount is added for the ns, but because no handler is allocated for the ns, the ns cannot be accessed through the router. We must reconfigure the number of handlers and then refresh the configuration. At this time, the router can access the ns normally. When we reconfigure the number of handlers, we have to face disadvantage 1: Configuration is inconvenient and error-prone. 3. Cannot share handlers: > RBF: ProportionRouterRpcFairnessPolicyController-support proportional > allocation of semaphores > -- > > Key: HDFS-17302 > URL: https://issues.apache.org/jira/browse/HDFS-17302 > Project: Hadoop HDFS > Issue Type: New Feature > Components: rbf >Reporter: Jian Zhang >Assignee: Jian Zhang >Priority: Major > Labels: pull-request-available > Attachments: HDFS-17302.001.patch, HDFS-17302.002.patch > > > [HDFS-14090|https://issues.apache.org/jira/browse/HDFS-14090] provides a > StaticRouterRpcFairnessPolicyController to support configuring different > handlers for different ns. Using the StaticRouterRpcFairnessPolicyController > allows the router to isolate different ns, and the ns with a higher load will > not affect the router's access to the ns with a normal load. But the >
[jira] [Updated] (HDFS-17302) RBF: ProportionRouterRpcFairnessPolicyController-support proportional allocation of semaphores
[ https://issues.apache.org/jira/browse/HDFS-17302?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Jian Zhang updated HDFS-17302: -- Description: [HDFS-14090|https://issues.apache.org/jira/browse/HDFS-14090] provides a StaticRouterRpcFairnessPolicyController to support configuring different handlers for different ns. Using the StaticRouterRpcFairnessPolicyController allows the router to isolate different ns, and the ns with a higher load will not affect the router's access to the ns with a normal load. But the StaticRouterRpcFairnessPolicyController still falls short in many ways, such as: 1. Configuration is inconvenient and error-prone: When I use StaticRouterRpcFairnessPolicyController, I first need to know how many handlers the router has in total, then I have to know how many nameservices the router currently has, and then carefully calculate how many handlers to allocate to each ns so that the sum of handlers for all ns will not exceed the total handlers of the router, and I also need to consider how many handlers to allocate to each ns to achieve better performance. Therefore, I need to be very careful when configuring. Even if I configure only one more handler for a certain ns, the total number is more than the number of hadnlers owned by the router, which will also cause the router to fail to start. At this time, I had to investigate the reason why the router failed to start. After finding the reason, I had to reconsider the number of handlers for each ns. 2. Extension ns is not supported: During the running of the router, if a new ns is added to the cluster and a mount is added for the ns, but because no handler is allocated for the ns, the ns cannot be accessed through the router. We must reconfigure the number of handlers and then refresh the configuration. At this time, the router can access the ns normally. When we reconfigure the number of handlers, we have to face disadvantage 1: Configuration is inconvenient and error-prone. 3. Cannot share handlers: was:[HDFS-14090|https://issues.apache.org/jira/browse/HDFS-14090] which provides a StaticRouterRpcFairnessPolicyController to support configuring different handlers for different ns. Using the StaticRouterRpcFairnessPolicyController allows the router to isolate different ns, and the ns with a higher load will not affect the router's access to the ns with a normal load. > RBF: ProportionRouterRpcFairnessPolicyController-support proportional > allocation of semaphores > -- > > Key: HDFS-17302 > URL: https://issues.apache.org/jira/browse/HDFS-17302 > Project: Hadoop HDFS > Issue Type: New Feature > Components: rbf >Reporter: Jian Zhang >Assignee: Jian Zhang >Priority: Major > Labels: pull-request-available > Attachments: HDFS-17302.001.patch, HDFS-17302.002.patch > > > [HDFS-14090|https://issues.apache.org/jira/browse/HDFS-14090] provides a > StaticRouterRpcFairnessPolicyController to support configuring different > handlers for different ns. Using the StaticRouterRpcFairnessPolicyController > allows the router to isolate different ns, and the ns with a higher load will > not affect the router's access to the ns with a normal load. But the > StaticRouterRpcFairnessPolicyController still falls short in many ways, such > as: > 1. Configuration is inconvenient and error-prone: When I use > StaticRouterRpcFairnessPolicyController, I first need to know how many > handlers the router has in total, then I have to know how many nameservices > the router currently has, and then carefully calculate how many handlers to > allocate to each ns so that the sum of handlers for all ns will not exceed > the total handlers of the router, and I also need to consider how many > handlers to allocate to each ns to achieve better performance. Therefore, I > need to be very careful when configuring. Even if I configure only one more > handler for a certain ns, the total number is more than the number of > hadnlers owned by the router, which will also cause the router to fail to > start. At this time, I had to investigate the reason why the router failed to > start. After finding the reason, I had to reconsider the number of handlers > for each ns. > 2. Extension ns is not supported: During the running of the router, if a new > ns is added to the cluster and a mount is added for the ns, but because no > handler is allocated for the ns, the ns cannot be accessed through the > router. We must reconfigure the number of handlers and then refresh the > configuration. At this time, the router can access the ns normally. When we > reconfigure the number of handlers, we have to face disadvantage 1: > Configuration is inconvenient and error-prone. >
[jira] [Updated] (HDFS-17302) RBF: ProportionRouterRpcFairnessPolicyController-support proportional allocation of semaphores
[ https://issues.apache.org/jira/browse/HDFS-17302?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Jian Zhang updated HDFS-17302: -- Description: [HDFS-14090|https://issues.apache.org/jira/browse/HDFS-14090] : Improved isolation for downstream name nodes, which provides a StaticRouterRpcFairnessPolicyController to support configuring different handlers for different ns. Using the StaticRouterRpcFairnessPolicyController allows the router to isolate different ns, and the ns with a higher load will not affect the router's access to the ns with a normal load. > RBF: ProportionRouterRpcFairnessPolicyController-support proportional > allocation of semaphores > -- > > Key: HDFS-17302 > URL: https://issues.apache.org/jira/browse/HDFS-17302 > Project: Hadoop HDFS > Issue Type: New Feature > Components: rbf >Reporter: Jian Zhang >Assignee: Jian Zhang >Priority: Major > Labels: pull-request-available > Attachments: HDFS-17302.001.patch, HDFS-17302.002.patch > > > [HDFS-14090|https://issues.apache.org/jira/browse/HDFS-14090] : Improved > isolation for downstream name nodes, which provides a > StaticRouterRpcFairnessPolicyController to support configuring different > handlers for different ns. Using the StaticRouterRpcFairnessPolicyController > allows the router to isolate different ns, and the ns with a higher load will > not affect the router's access to the ns with a normal load. -- This message was sent by Atlassian Jira (v8.20.10#820010) - To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Updated] (HDFS-17302) RBF: ProportionRouterRpcFairnessPolicyController-support proportional allocation of semaphores
[ https://issues.apache.org/jira/browse/HDFS-17302?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Jian Zhang updated HDFS-17302: -- Description: [HDFS-14090|https://issues.apache.org/jira/browse/HDFS-14090] which provides a StaticRouterRpcFairnessPolicyController to support configuring different handlers for different ns. Using the StaticRouterRpcFairnessPolicyController allows the router to isolate different ns, and the ns with a higher load will not affect the router's access to the ns with a normal load. (was: [HDFS-14090|https://issues.apache.org/jira/browse/HDFS-14090] : Improved isolation for downstream name nodes, which provides a StaticRouterRpcFairnessPolicyController to support configuring different handlers for different ns. Using the StaticRouterRpcFairnessPolicyController allows the router to isolate different ns, and the ns with a higher load will not affect the router's access to the ns with a normal load.) > RBF: ProportionRouterRpcFairnessPolicyController-support proportional > allocation of semaphores > -- > > Key: HDFS-17302 > URL: https://issues.apache.org/jira/browse/HDFS-17302 > Project: Hadoop HDFS > Issue Type: New Feature > Components: rbf >Reporter: Jian Zhang >Assignee: Jian Zhang >Priority: Major > Labels: pull-request-available > Attachments: HDFS-17302.001.patch, HDFS-17302.002.patch > > > [HDFS-14090|https://issues.apache.org/jira/browse/HDFS-14090] which provides > a StaticRouterRpcFairnessPolicyController to support configuring different > handlers for different ns. Using the StaticRouterRpcFairnessPolicyController > allows the router to isolate different ns, and the ns with a higher load will > not affect the router's access to the ns with a normal load. -- This message was sent by Atlassian Jira (v8.20.10#820010) - To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Updated] (HDFS-17302) RBF: ProportionRouterRpcFairnessPolicyController-support proportional allocation of semaphores
[ https://issues.apache.org/jira/browse/HDFS-17302?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Jian Zhang updated HDFS-17302: -- Attachment: HDFS-17302.002.patch > RBF: ProportionRouterRpcFairnessPolicyController-support proportional > allocation of semaphores > -- > > Key: HDFS-17302 > URL: https://issues.apache.org/jira/browse/HDFS-17302 > Project: Hadoop HDFS > Issue Type: New Feature > Components: rbf >Reporter: Jian Zhang >Assignee: Jian Zhang >Priority: Major > Labels: pull-request-available > Attachments: HDFS-17302.001.patch, HDFS-17302.002.patch > > -- This message was sent by Atlassian Jira (v8.20.10#820010) - To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Updated] (HDFS-17302) RBF: ProportionRouterRpcFairnessPolicyController-support proportional allocation of semaphores
[ https://issues.apache.org/jira/browse/HDFS-17302?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Jian Zhang updated HDFS-17302: -- Attachment: HDFS-17302.001.patch Status: Patch Available (was: Open) > RBF: ProportionRouterRpcFairnessPolicyController-support proportional > allocation of semaphores > -- > > Key: HDFS-17302 > URL: https://issues.apache.org/jira/browse/HDFS-17302 > Project: Hadoop HDFS > Issue Type: New Feature > Components: rbf >Reporter: Jian Zhang >Assignee: Jian Zhang >Priority: Major > Labels: pull-request-available > Attachments: HDFS-17302.001.patch > > -- This message was sent by Atlassian Jira (v8.20.10#820010) - To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Updated] (HDFS-17302) RBF: ProportionRouterRpcFairnessPolicyController-support proportional allocation of semaphores
[ https://issues.apache.org/jira/browse/HDFS-17302?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] ASF GitHub Bot updated HDFS-17302: -- Labels: pull-request-available (was: ) > RBF: ProportionRouterRpcFairnessPolicyController-support proportional > allocation of semaphores > -- > > Key: HDFS-17302 > URL: https://issues.apache.org/jira/browse/HDFS-17302 > Project: Hadoop HDFS > Issue Type: New Feature > Components: rbf >Reporter: Jian Zhang >Assignee: Jian Zhang >Priority: Major > Labels: pull-request-available > -- This message was sent by Atlassian Jira (v8.20.10#820010) - To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
