[jira] [Updated] (HADOOP-17268) Add RPC Quota to NameNode.
[ https://issues.apache.org/jira/browse/HADOOP-17268?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Jinglun updated HADOOP-17268: - Description: My users recently complained 'The NameNode is much slower than before' to me. The reason is the cluster and jobs are getting bigger and bigger. So is the pressure of NameNode. I explained the pressure was heavy so the rpc requests must wait, but they were not satisfied. Because they thought the original quality of the service should be guaranteed. They were never told the NameNode would be so slow and all their services were built based on the assumption that the NameNode would always respond as fast as before. From the user's standpoint they are right. So my question is how to give the user a guarantee about RPC requests. The natural idea is RPC Quota, just like name quota and space quota. The quota can help users to understand the rpc requests are also a limit resource. And when they apply quota to the administrator, the admin would have the chance to distribute the resource and make a plan for the cluster. e.g. We have 200 quota for addBlock and they are all allocated. Even the peak doesn't reach 200, I should reject other users from applying to reserve the resource. The new user should be mounted to other namespaces. It's still an initial idea now. I'll think again carefully and make a detailed proposal. All advice are welcome! was: My users recently complained 'The NameNode is much slower than before' to me. The reason is the cluster and jobs are getting bigger and bigger. So is the pressure of NameNode. I explained the pressure was heavy so the rpc requests must wait, but they were not satisfied. Because they thought the original quality of the service should be guaranteed. They were never told the NameNode would be so slow and all their services were built based on the assumption that the NameNode would always respond as fast as before. >From the user's standpoint they are right. So my question is how to give the >user a guarantee about RPC requests. The natural idea is RPC Quota, just like >name quota and space quota. The quota can help users to understand the rpc >requests are also a limit resource. And when they apply quota to the >administrator, the admin would have the chance to distribute the resource and >make a plan for the cluster. e.g. We have 200 quota for addBlock and they are >all allocated. Even the peak doesn't reach 200, I should reject other users >from applying to reserve the resource. The new user should be mounted to an other namespace. It's still an initial idea now. I'll think again carefully and make a detailed proposal. > Add RPC Quota to NameNode. > -- > > Key: HADOOP-17268 > URL: https://issues.apache.org/jira/browse/HADOOP-17268 > Project: Hadoop Common > Issue Type: Improvement >Reporter: Jinglun >Assignee: Jinglun >Priority: Major > Attachments: HADOOP-17268.001.patch > > > My users recently complained 'The NameNode is much slower than before' to me. > The reason is the cluster and jobs are getting bigger and bigger. So is the > pressure of NameNode. I explained the pressure was heavy so the rpc requests > must wait, but they were not satisfied. Because they thought the original > quality of the service should be guaranteed. They were never told the > NameNode would be so slow and all their services were built based on the > assumption that the NameNode would always respond as fast as before. > From the user's standpoint they are right. So my question is how to give the > user a guarantee about RPC requests. The natural idea is RPC Quota, just like > name quota and space quota. The quota can help users to understand the rpc > requests are also a limit resource. And when they apply quota to the > administrator, the admin would have the chance to distribute the resource and > make a plan for the cluster. e.g. We have 200 quota for addBlock and they are > all allocated. Even the peak doesn't reach 200, I should reject other users > from applying to reserve the resource. The new user should be mounted to > other namespaces. > It's still an initial idea now. I'll think again carefully and make a > detailed proposal. All advice are welcome! -- This message was sent by Atlassian Jira (v8.3.4#803005) - To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: common-issues-h...@hadoop.apache.org
[jira] [Updated] (HADOOP-17268) Add RPC Quota to NameNode.
[ https://issues.apache.org/jira/browse/HADOOP-17268?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Jinglun updated HADOOP-17268: - Description: My users recently complained 'The NameNode is much slower than before' to me. The reason is the cluster and jobs are getting bigger and bigger. So is the pressure of NameNode. I explained the pressure was heavy so the rpc requests must wait, but they were not satisfied. Because they thought the original quality of the service should be guaranteed. They were never told the NameNode would be so slow and all their services were built based on the assumption that the NameNode would always respond as fast as before. >From the user's standpoint they are right. So my question is how to give the >user a guarantee about RPC requests. The natural idea is RPC Quota, just like >name quota and space quota. The quota can help users to understand the rpc >requests are also a limit resource. And when they apply quota to the >administrator, the admin would have the chance to distribute the resource and >make a plan for the cluster. e.g. We have 200 quota for addBlock and they are >all allocated. Even the peak doesn't reach 200, I should reject other users >from applying to reserve the resource. The new user should be mounted to an other namespace. It's still an initial idea now. I'll think again carefully and make a detailed proposal. was:Add the ability of rpc request quota to NameNode. All the requests exceeding quota would end with a 'Server too busy' exception. This can prevent users from overusing. > Add RPC Quota to NameNode. > -- > > Key: HADOOP-17268 > URL: https://issues.apache.org/jira/browse/HADOOP-17268 > Project: Hadoop Common > Issue Type: Improvement >Reporter: Jinglun >Assignee: Jinglun >Priority: Major > Attachments: HADOOP-17268.001.patch > > > My users recently complained 'The NameNode is much slower than before' to me. > The reason is the cluster and jobs are getting bigger and bigger. So is the > pressure of NameNode. I explained the pressure was heavy so the rpc requests > must wait, but they were not satisfied. Because they thought the original > quality of the service should be guaranteed. They were never told the > NameNode would be so slow and all their services were built based on the > assumption that the NameNode would always respond as fast as before. > From the user's standpoint they are right. So my question is how to give the > user a guarantee about RPC requests. The natural idea is RPC Quota, just like > name quota and space quota. The quota can help users to understand the rpc > requests are also a limit resource. And when they apply quota to the > administrator, the admin would have the chance to distribute the resource and > make a plan for the cluster. e.g. We have 200 quota for addBlock and they are > all allocated. Even the peak doesn't reach 200, I should reject other users > from applying to reserve the resource. The new user should be mounted to an > other namespace. > It's still an initial idea now. I'll think again carefully and make a > detailed proposal. -- This message was sent by Atlassian Jira (v8.3.4#803005) - To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: common-issues-h...@hadoop.apache.org
[jira] [Updated] (HADOOP-17268) Add RPC Quota to NameNode.
[ https://issues.apache.org/jira/browse/HADOOP-17268?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Jinglun updated HADOOP-17268: - Attachment: (was: HADOOP-17268.001.patch) > Add RPC Quota to NameNode. > -- > > Key: HADOOP-17268 > URL: https://issues.apache.org/jira/browse/HADOOP-17268 > Project: Hadoop Common > Issue Type: Improvement >Reporter: Jinglun >Assignee: Jinglun >Priority: Major > Attachments: HADOOP-17268.001.patch > > > Add the ability of rpc request quota to NameNode. All the requests exceeding > quota would end with a 'Server too busy' exception. This can prevent users > from overusing. -- This message was sent by Atlassian Jira (v8.3.4#803005) - To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: common-issues-h...@hadoop.apache.org
[jira] [Updated] (HADOOP-17268) Add RPC Quota to NameNode.
[ https://issues.apache.org/jira/browse/HADOOP-17268?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Jinglun updated HADOOP-17268: - Status: Open (was: Patch Available) > Add RPC Quota to NameNode. > -- > > Key: HADOOP-17268 > URL: https://issues.apache.org/jira/browse/HADOOP-17268 > Project: Hadoop Common > Issue Type: Improvement >Reporter: Jinglun >Assignee: Jinglun >Priority: Major > Attachments: HADOOP-17268.001.patch > > > Add the ability of rpc request quota to NameNode. All the requests exceeding > quota would end with a 'Server too busy' exception. This can prevent users > from overusing. -- This message was sent by Atlassian Jira (v8.3.4#803005) - To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: common-issues-h...@hadoop.apache.org
[jira] [Updated] (HADOOP-17268) Add RPC Quota to NameNode.
[ https://issues.apache.org/jira/browse/HADOOP-17268?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Jinglun updated HADOOP-17268: - Attachment: HADOOP-17268.001.patch Status: Patch Available (was: Open) Re-upload patch to trigger jenkins. > Add RPC Quota to NameNode. > -- > > Key: HADOOP-17268 > URL: https://issues.apache.org/jira/browse/HADOOP-17268 > Project: Hadoop Common > Issue Type: Improvement >Reporter: Jinglun >Assignee: Jinglun >Priority: Major > Attachments: HADOOP-17268.001.patch > > > Add the ability of rpc request quota to NameNode. All the requests exceeding > quota would end with a 'Server too busy' exception. This can prevent users > from overusing. -- This message was sent by Atlassian Jira (v8.3.4#803005) - To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: common-issues-h...@hadoop.apache.org
[jira] [Updated] (HADOOP-17268) Add RPC Quota to NameNode.
[ https://issues.apache.org/jira/browse/HADOOP-17268?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Jinglun updated HADOOP-17268: - Attachment: HADOOP-17268.001.patch Status: Patch Available (was: Open) > Add RPC Quota to NameNode. > -- > > Key: HADOOP-17268 > URL: https://issues.apache.org/jira/browse/HADOOP-17268 > Project: Hadoop Common > Issue Type: Improvement >Reporter: Jinglun >Assignee: Jinglun >Priority: Major > Attachments: HADOOP-17268.001.patch > > > Add the ability of rpc request quota to NameNode. All the requests exceeding > quota would end with a 'Server too busy' exception. This can prevent users > from overusing. -- This message was sent by Atlassian Jira (v8.3.4#803005) - To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: common-issues-h...@hadoop.apache.org
