[jira] [Updated] (HADOOP-13128) Manage Hadoop RPC resource usage via resource coupon
[ https://issues.apache.org/jira/browse/HADOOP-13128?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Xiaoyu Yao updated HADOOP-13128: Attachment: HADOOP-13128-Proposal-20160511.pdf Attach a draft proposal for discussion. > Manage Hadoop RPC resource usage via resource coupon > > > Key: HADOOP-13128 > URL: https://issues.apache.org/jira/browse/HADOOP-13128 > Project: Hadoop Common > Issue Type: Improvement >Reporter: Xiaoyu Yao >Assignee: Xiaoyu Yao > Attachments: HADOOP-13128-Proposal-20160511.pdf > > > HADOOP-9640 added RPC Fair Call Queue and HADOOP-10597 added RPC backoff to > ensure the fairness usage of the HDFS namenode resources. YARN, the Hadoop > cluster resource manager currently manages the CPU and Memory resources for > jobs/tasks but not the storage resources such as HDFS namenode and datanode > usage directly. As a result of that, a high priority Yarn Job may send too > many RPC requests to HDFS namenode and get demoted into low priority call > queues due to lack of reservation/coordination. > To better support multi-tenancy use cases like above, we propose to manage > RPC server resource usage via coupon mechanism integrated with YARN. The idea > is to allow YARN request HDFS storage resource coupon (e.g., namenode RPC > calls, datanode I/O bandwidth) from namenode on behalf of the job upon > submission time. Once granted, the tasks will include the coupon identifier > in RPC header for the subsequent calls. HDFS namenode RPC scheduler maintains > the state of the coupon usage based on the scheduler policy (fairness or > priority) to match the RPC priority with the YARN scheduling priority. > I will post a proposal with more detail shortly. -- This message was sent by Atlassian JIRA (v6.3.4#6332) - To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: common-issues-h...@hadoop.apache.org
[jira] [Updated] (HADOOP-13128) Manage Hadoop RPC resource usage via resource coupon
[ https://issues.apache.org/jira/browse/HADOOP-13128?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Xiaoyu Yao updated HADOOP-13128: Description: HADOOP-9640 added RPC Fair Call Queue and HADOOP-10597 added RPC backoff to ensure the fairness usage of the HDFS namenode resources. YARN, the Hadoop cluster resource manager currently manages the CPU and Memory resources for jobs/tasks but not the storage resources such as HDFS namenode and datanode usage directly. As a result of that, a high priority Yarn Job may send too many RPC requests to HDFS namenode and get demoted into low priority call queues due to lack of reservation/coordination. To better support multi-tenancy use cases like above, we propose to manage RPC server resource usage via coupon mechanism integrated with YARN. The idea is to allow YARN request HDFS storage resource coupon (e.g., namenode RPC calls, datanode I/O bandwidth) from namenode on behalf of the job upon submission time. Once granted, the tasks will include the coupon identifier in RPC header for the subsequent calls. HDFS namenode RPC scheduler maintains the state of the coupon usage based on the scheduler policy (fairness or priority) to match the RPC priority with the YARN scheduling priority. I will post a proposal with more detail shortly. was: HADOOP-9640 added RPC Fair Call Queue and HADOOP-10597 added RPC backoff to ensure the fairness usage of the HDFS namenode resources. YARN, the Hadoop cluster resource manager currently manages the CPU and Memory resources for jobs/tasks but not the storage resources such as HDFS namenode and datanode usage directly. As a result of that, a high priority Yarn Job may send too many RPC requests to HDFS namenode and get demoted into low priority call queues due to lack of reservation/coordination. To better support multi-tenancy use cases like above, we propose to manage RPC server resource usage via coupon mechanism integrated with YARN. The idea is to allow YARN request HDFS storage resource coupon (e.g., namenode RPC calls, datanode I/O bandwidth) from namenode on behalf of the job upon submission time. Once granted, the tasks will include the coupon identifier in RPC header for the subsequent calls. HDFS namenode RPC scheduler maintains the state of the coupon usage based on the scheduler policy (fairness or priority) to match the RPC priority with the YARN scheduling priority. > Manage Hadoop RPC resource usage via resource coupon > > > Key: HADOOP-13128 > URL: https://issues.apache.org/jira/browse/HADOOP-13128 > Project: Hadoop Common > Issue Type: Improvement >Reporter: Xiaoyu Yao >Assignee: Xiaoyu Yao > > HADOOP-9640 added RPC Fair Call Queue and HADOOP-10597 added RPC backoff to > ensure the fairness usage of the HDFS namenode resources. YARN, the Hadoop > cluster resource manager currently manages the CPU and Memory resources for > jobs/tasks but not the storage resources such as HDFS namenode and datanode > usage directly. As a result of that, a high priority Yarn Job may send too > many RPC requests to HDFS namenode and get demoted into low priority call > queues due to lack of reservation/coordination. > To better support multi-tenancy use cases like above, we propose to manage > RPC server resource usage via coupon mechanism integrated with YARN. The idea > is to allow YARN request HDFS storage resource coupon (e.g., namenode RPC > calls, datanode I/O bandwidth) from namenode on behalf of the job upon > submission time. Once granted, the tasks will include the coupon identifier > in RPC header for the subsequent calls. HDFS namenode RPC scheduler maintains > the state of the coupon usage based on the scheduler policy (fairness or > priority) to match the RPC priority with the YARN scheduling priority. > I will post a proposal with more detail shortly. -- This message was sent by Atlassian JIRA (v6.3.4#6332) - To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: common-issues-h...@hadoop.apache.org
[jira] [Updated] (HADOOP-13128) Manage Hadoop RPC resource usage via resource coupon
[ https://issues.apache.org/jira/browse/HADOOP-13128?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Xiaoyu Yao updated HADOOP-13128: Description: HADOOP-9640 added RPC Fair Call Queue and HADOOP-10597 added RPC backoff to ensure the fairness usage of the HDFS namenode resources. YARN, the Hadoop cluster resource manager currently manages the CPU and Memory resources for jobs/tasks but not the storage resources such as HDFS namenode and datanode usage directly. As a result of that, a high priority Yarn Job may send too many RPC requests to HDFS namenode and get demoted into low priority call queues due to lack of reservation/coordination. To better support multi-tenancy use cases like above, we propose to manage RPC server resource usage via coupon mechanism integrated with YARN. The idea is to allow YARN request HDFS storage resource coupon (e.g., namenode RPC calls, datanode I/O bandwidth) from namenode on behalf of the job upon submission time. Once granted, the tasks will include the coupon identifier in RPC header for the subsequent calls. HDFS namenode RPC scheduler maintains the state of the coupon usage based on the scheduler policy (fairness or priority) to match the RPC priority with the YARN scheduling priority. was: HADOOP-9640 added RPC Fair Call Queue and HADOOP-10597 added RPC backoff to ensure the fairness usage of the HDFS namenode resources. YARN, the Hadoop cluster resource manager currently manages the CPU and Memory resources for jobs/tasks but not the storage resources such as HDFS namenode and datanode usage directly. As a result of that, a high priority Yarn Job may send too many RPC requests to HDFS namenode call queue and get demoted into low priority namenode call queue due to lack of coordination. To better support multi-tenancy use cases like above, we propose to manage RPC server resource usage via coupon mechanism integrated with YARN. The idea is to allow YARN request HDFS storage resource coupon (e.g., namenode RPC calls, datanode I/O bandwidth) from namenode on behalf of the job upon submission time. Once granted, the tasks will include the coupon identifier in RPC header for the subsequent calls. HDFS namenode RPC scheduler maintains the state of the coupon usage based on the scheduler policy (fairness or priority) to match the RPC priority with the YARN scheduling priority. > Manage Hadoop RPC resource usage via resource coupon > > > Key: HADOOP-13128 > URL: https://issues.apache.org/jira/browse/HADOOP-13128 > Project: Hadoop Common > Issue Type: Improvement >Reporter: Xiaoyu Yao >Assignee: Xiaoyu Yao > > HADOOP-9640 added RPC Fair Call Queue and HADOOP-10597 added RPC backoff to > ensure the fairness usage of the HDFS namenode resources. YARN, the Hadoop > cluster resource manager currently manages the CPU and Memory resources for > jobs/tasks but not the storage resources such as HDFS namenode and datanode > usage directly. As a result of that, a high priority Yarn Job may send too > many RPC requests to HDFS namenode and get demoted into low priority call > queues due to lack of reservation/coordination. > To better support multi-tenancy use cases like above, we propose to manage > RPC server resource usage via coupon mechanism integrated with YARN. The idea > is to allow YARN request HDFS storage resource coupon (e.g., namenode RPC > calls, datanode I/O bandwidth) from namenode on behalf of the job upon > submission time. Once granted, the tasks will include the coupon identifier > in RPC header for the subsequent calls. HDFS namenode RPC scheduler maintains > the state of the coupon usage based on the scheduler policy (fairness or > priority) to match the RPC priority with the YARN scheduling priority. -- This message was sent by Atlassian JIRA (v6.3.4#6332) - To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: common-issues-h...@hadoop.apache.org
