[ https://issues.apache.org/jira/browse/YARN-2681?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
cntic updated YARN-2681: ------------------------ Attachment: HADOOP-2681.patch - fix findbugs warnings - testing purpose: +TC class rate can be given by read HDFS file defined in YARN configuration + TC class burst can be defined in configration. Otherwise default value will be set when TC class is added > Support bandwidth enforcement for containers while reading from HDFS > -------------------------------------------------------------------- > > Key: YARN-2681 > URL: https://issues.apache.org/jira/browse/YARN-2681 > Project: Hadoop YARN > Issue Type: New Feature > Components: capacityscheduler, nodemanager, resourcemanager > Affects Versions: 2.5.1 > Environment: Linux > Reporter: cntic > Attachments: HADOOP-2681.patch, HADOOP-2681.patch, Traffic Control > Design.png > > > To read/write data from HDFS on data node, applications establise TCP/IP > connections with the datanode. The HDFS read can be controled by setting > Linux Traffic Control (TC) subsystem on the data node to make filters on > appropriate connections. > The current cgroups net_cls concept can not be applied on the node where the > container is launched, netheir on data node since: > - TC hanldes outgoing bandwidth only, so it can be set on container node > (HDFS read = incoming data for the container) > - Since HDFS data node is handled by only one process, it is not possible > to use net_cls to separate connections from different containers to the > datanode. > Tasks: > 1) Extend Resource model to define bandwidth enforcement rate > 2) Monitor TCP/IP connection estabilised by container handling process and > its child processes > 3) Set Linux Traffic Control rules on data node base on address:port pairs in > order to enforce bandwidth of outgoing data > Concept: > http://www.hit.bme.hu/~do/papers/EnforcementDesign.pdf -- This message was sent by Atlassian JIRA (v6.3.4#6332)