[
https://issues.apache.org/jira/browse/HDFS-5175?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14164219#comment-14164219
]
Ming Ma commented on HDFS-5175:
-------------------------------
[~cnauroth], thanks for the suggestion.
We want the ability to config different instances of DFSClient with different
DSCP values. We ended up using the approach that Lohit mentioned e.g., from
application point of view, they will set the configuration property
dfs.client.dscp.bit.intvalue and DSClient will call setTrafficClass before
establishing the connection. Your suggestion of custom SocketFactory approach
will also work. Each instance of DFSClient will create its own instance of
DSCPSocketFactory; DSCPSocketFactory also needs to implement Configurable so
that it can access the configured DSCP value. If we want to go with custom
SocketFactory approach, we can add it to hadoop-common so that applications and
other hadoop projects can use it directly.
At the higher level, there are different scenarios. The protocol could be based
on DFSClient/DataTransferProtocol over TCP, or Hftp/WebHDFS over HTTP. The
data transfer should be read or write.
The primary use case for us is cross DC remote read scenario with Hftp. In
order to config Socket for Hftp scenario, we ended up using
httpclient.Protocol.registerProtocol with custom class of
ProtocolSocketFactory, given HttpURLConnection doesn't provide a mechanism for
custom SocketFactory.
As we try to open source our work to support WebHDFS, it brings up the question
of "how to config the socket in WebHDFS given httpclient isn't used by HDFS".
We did some investigation and found some possible approaches.
1. HttpsURLConnection has setSSLSocketFactory. Need to test out if it works for
regular http.
2. Call Socket.setSocketFactroy() for the entire JVM. This requires custom
SocketImplFactory to create custom socket based on webHDFS address.
We would like to get more inputs from others.
> Provide clients a way to set IP header bits on connections
> ----------------------------------------------------------
>
> Key: HDFS-5175
> URL: https://issues.apache.org/jira/browse/HDFS-5175
> Project: Hadoop HDFS
> Issue Type: Bug
> Affects Versions: 2.0.5-alpha
> Reporter: Lohit Vijayarenu
>
> It would be very helpful if we had ability for clients to set IP headers when
> they make socket connections for data transfers. We were looking into setting
> up QoS using DSCP bit and saw that there is no easy way to let clients pass
> down a specific value when clients make connection to DataNode.
> As a quick fix we did something similar to io.file.buffer.size where client
> could pass down DSCP integer value and when DFSClient opens a stream, it
> could set the value on socket using setTrafficClass
> Opening this JIRA to get more inputs from others who have had experience and
> might have already thought about this.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
