[
https://issues.apache.org/jira/browse/HDFS-13248?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16884283#comment-16884283
]
Ayush Saxena edited comment on HDFS-13248 at 7/13/19 5:32 AM:
--------------------------------------------------------------
Hey Brahma!!!
Literally, I don't have those stats with an MR Job. Actually I guess you are
pointing towards as if client resides on the same site as that of Router.
In that case, Yes, Correct. technically this won't be a problem as the router
and the client node share the same addr.
Here is to tackle when this isn't true and the *client resides on a node having
a DN but not a Router* In that scenario the BPP shall be satisfied wrt to
Router not the client, and hence these locality problems shall occur and the
DN's shall be sorted wrt to Router not client. And by far I guess the number of
clients sites shall be more than Routers. Let me know if you require any stats
for analysis, we shall try grab them up.
Well we had couple solutions, All Stuck as of now :
* Add proxy address in IPC connection (HADOOP-16254) --> This had some
security concerns.
* The RouterRPCServer should transfer CallerContext and client ip to
NamenodeRpcServer (HDFS-13293) --> This tend to little opaque and couple of
more problems stated above.
* Favored Nodes --> I guess the last patch here. Pass the local node as favored
node. But this isn't a complete solution. This doesn't take into account the
fallback in case of non availability of local nodes and couple of more.
Do give a check, if you can help, or give some pointers to any of the
solutions, Or a new solutions. this had been still since quite a long.
was (Author: ayushtkn):
Hey Brahma!!!
Literally, I don't have those stats with an MR Job. Actually I guess you are
pointing towards as if client resides on the same site as that of Router.
In that case, Yes, Correct. technically this won't be a problem as the router
and the client node share the same addr.
Here is to tackle when this isn't true and the *client resides on a node having
a DN but not a Router* In that scenario the BPP shall be satisfied wrt to
Router not the client, and hence these locality problems shall occur and the
DN's shall be sorted wrt to Router not client. And by far I guess the number of
clients sides shall be more than Routers. Let me know if you require any stats
for analysis, we shall try grab them up.
Well he had couple solutions, All Stuck as of now :
* Add proxy address in IPC connection (HADOOP-16254) --> This had some
security concerns.
* The RouterRPCServer should transfer CallerContext and client ip to
NamenodeRpcServer (HDFS-13293) --> This tend to little opaque and couple of
more problems stated above.
* Favored Nodes --> I guess the last patch here. Pass the local node as favored
node. But this isn't a complete solution. This doesn't take into account the
fallback in case of non availability of local nodes and couple of more.
Do give a check, if you can help, or give some pointers to any of the
solutions, Or a new solutions. this had been still since quite a long.
> RBF: Namenode need to choose block location for the client
> ----------------------------------------------------------
>
> Key: HDFS-13248
> URL: https://issues.apache.org/jira/browse/HDFS-13248
> Project: Hadoop HDFS
> Issue Type: Sub-task
> Reporter: Wu Weiwei
> Assignee: Íñigo Goiri
> Priority: Major
> Attachments: HDFS-13248.000.patch, HDFS-13248.001.patch,
> HDFS-13248.002.patch, HDFS-13248.003.patch, HDFS-13248.004.patch,
> HDFS-13248.005.patch, HDFS-Router-Data-Locality.odt, RBF Data Locality
> Design.pdf, clientMachine-call-path.jpeg, debug-info-1.jpeg, debug-info-2.jpeg
>
>
> When execute a put operation via router, the NameNode will choose block
> location for the router, not for the real client. This will affect the file's
> locality.
> I think on both NameNode and Router, we should add a new addBlock method, or
> add a parameter for the current addBlock method, to pass the real client
> information.
--
This message was sent by Atlassian JIRA
(v7.6.14#76016)
---------------------------------------------------------------------
To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
