[
https://issues.apache.org/jira/browse/YARN-2681?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
cntic updated YARN-2681:
------------------------
Description:
To read/write data from HDFS on data node, applications establise TCP/IP
connections with the datanode. The HDFS read can be controled by setting Linux
Traffic Control (TC) subsystem on the data node to make filters on appropriate
connections.
The current cgroups net_cls concept can not be applied on the node where the
container is launched, netheir on data node since:
- TC hanldes outgoing bandwidth only, so it can be set on container node
(HDFS read = incoming data for the container)
- Since HDFS data node is handled by only one process, it is not possible to
use net_cls to separate connections from different containers to the datanode.
Tasks:
1) Extend Resource model to define bandwidth enforcement rate
2) Monitor TCP/IP connection estabilised by container handling process and its
child processes
3) Set Linux Traffic Control rules on data node base on address:port pairs in
order to enforce bandwidth of outgoing data
Concept: http://www.hit.bme.hu/~do/papers/EnforcementDesign.pdf
Implementation: http://www.hit.bme.hu/~dohoai/documents/HdfsTrafficControl.pdf
was:
To read/write data from HDFS on data node, applications establise TCP/IP
connections with the datanode. The HDFS read can be controled by setting Linux
Traffic Control (TC) subsystem on the data node to make filters on appropriate
connections.
The current cgroups net_cls concept can not be applied on the node where the
container is launched, netheir on data node since:
- TC hanldes outgoing bandwidth only, so it can be set on container node
(HDFS read = incoming data for the container)
- Since HDFS data node is handled by only one process, it is not possible to
use net_cls to separate connections from different containers to the datanode.
Tasks:
1) Extend Resource model to define bandwidth enforcement rate
2) Monitor TCP/IP connection estabilised by container handling process and its
child processes
3) Set Linux Traffic Control rules on data node base on address:port pairs in
order to enforce bandwidth of outgoing data
Concept:
http://www.hit.bme.hu/~do/papers/EnforcementDesign.pdf
> Support bandwidth enforcement for containers while reading from HDFS
> --------------------------------------------------------------------
>
> Key: YARN-2681
> URL: https://issues.apache.org/jira/browse/YARN-2681
> Project: Hadoop YARN
> Issue Type: New Feature
> Components: nodemanager
> Affects Versions: 2.5.1
> Environment: Linux
> Reporter: cntic
> Labels: BB2015-05-TBR
> Fix For: 2.7.0
>
> Attachments: HADOOP-2681.patch, HADOOP-2681.patch,
> HDFS-2681.02.patch, HdfsTrafficControl_UML.png, Traffic Control Design.png
>
>
> To read/write data from HDFS on data node, applications establise TCP/IP
> connections with the datanode. The HDFS read can be controled by setting
> Linux Traffic Control (TC) subsystem on the data node to make filters on
> appropriate connections.
> The current cgroups net_cls concept can not be applied on the node where the
> container is launched, netheir on data node since:
> - TC hanldes outgoing bandwidth only, so it can be set on container node
> (HDFS read = incoming data for the container)
> - Since HDFS data node is handled by only one process, it is not possible
> to use net_cls to separate connections from different containers to the
> datanode.
> Tasks:
> 1) Extend Resource model to define bandwidth enforcement rate
> 2) Monitor TCP/IP connection estabilised by container handling process and
> its child processes
> 3) Set Linux Traffic Control rules on data node base on address:port pairs in
> order to enforce bandwidth of outgoing data
> Concept: http://www.hit.bme.hu/~do/papers/EnforcementDesign.pdf
> Implementation: http://www.hit.bme.hu/~dohoai/documents/HdfsTrafficControl.pdf
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
