[jira] [Commented] (HDFS-2576) Namenode should have a favored nodes hint to enable clients to have control over block placement.
[ https://issues.apache.org/jira/browse/HDFS-2576?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=14909735#comment-14909735 ] 郭同 commented on HDFS-2576: -- http://stackoverflow.com/questions/32779439/how-to-let-the-hdfss-replica-blocks-position-be-set-by-myself/32808390#32808390 > Namenode should have a favored nodes hint to enable clients to have control > over block placement. > - > > Key: HDFS-2576 > URL: https://issues.apache.org/jira/browse/HDFS-2576 > Project: Hadoop HDFS > Issue Type: New Feature > Components: hdfs-client, namenode >Reporter: Pritam Damania >Assignee: Devaraj Das > Fix For: 2.1.0-beta > > Attachments: hdfs-2576-1.txt, hdfs-2576-trunk-1.patch, > hdfs-2576-trunk-2.patch, hdfs-2576-trunk-7.1.patch, hdfs-2576-trunk-7.patch, > hdfs-2576-trunk-8.1.patch, hdfs-2576-trunk-8.2.patch, > hdfs-2576-trunk-8.3.patch, hdfs-2576-trunk-8.patch > > > Sometimes Clients like HBase are required to dynamically compute the > datanodes it wishes to place the blocks for a file for higher level of > locality. For this purpose there is a need of a way to give the Namenode a > hint in terms of a favoredNodes parameter about the locations where the > client wants to put each block. The proposed solution is a favored nodes > parameter in the addBlock() method and in the create() file method to enable > the clients to give the hints to the NameNode about the locations of each > replica of the block. Note that this would be just a hint and finally the > NameNode would look at disk usage, datanode load etc. and decide whether it > can respect the hints or not. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (HDFS-2576) Namenode should have a favored nodes hint to enable clients to have control over block placement.
[ https://issues.apache.org/jira/browse/HDFS-2576?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=14909739#comment-14909739 ] 郭同 commented on HDFS-2576: -- http://stackoverflow.com/questions/32779439/how-to-let-the-hdfss-replica-blocks-position-be-set-by-myself/32808390#32808390 > Namenode should have a favored nodes hint to enable clients to have control > over block placement. > - > > Key: HDFS-2576 > URL: https://issues.apache.org/jira/browse/HDFS-2576 > Project: Hadoop HDFS > Issue Type: New Feature > Components: hdfs-client, namenode >Reporter: Pritam Damania >Assignee: Devaraj Das > Fix For: 2.1.0-beta > > Attachments: hdfs-2576-1.txt, hdfs-2576-trunk-1.patch, > hdfs-2576-trunk-2.patch, hdfs-2576-trunk-7.1.patch, hdfs-2576-trunk-7.patch, > hdfs-2576-trunk-8.1.patch, hdfs-2576-trunk-8.2.patch, > hdfs-2576-trunk-8.3.patch, hdfs-2576-trunk-8.patch > > > Sometimes Clients like HBase are required to dynamically compute the > datanodes it wishes to place the blocks for a file for higher level of > locality. For this purpose there is a need of a way to give the Namenode a > hint in terms of a favoredNodes parameter about the locations where the > client wants to put each block. The proposed solution is a favored nodes > parameter in the addBlock() method and in the create() file method to enable > the clients to give the hints to the NameNode about the locations of each > replica of the block. Note that this would be just a hint and finally the > NameNode would look at disk usage, datanode load etc. and decide whether it > can respect the hints or not. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (HDFS-2576) Namenode should have a favored nodes hint to enable clients to have control over block placement.
[ https://issues.apache.org/jira/browse/HDFS-2576?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=14909047#comment-14909047 ] 郭同 commented on HDFS-2576: -- I have the same problem,Thank you very much ... > Namenode should have a favored nodes hint to enable clients to have control > over block placement. > - > > Key: HDFS-2576 > URL: https://issues.apache.org/jira/browse/HDFS-2576 > Project: Hadoop HDFS > Issue Type: New Feature > Components: hdfs-client, namenode >Reporter: Pritam Damania >Assignee: Devaraj Das > Fix For: 2.1.0-beta > > Attachments: hdfs-2576-1.txt, hdfs-2576-trunk-1.patch, > hdfs-2576-trunk-2.patch, hdfs-2576-trunk-7.1.patch, hdfs-2576-trunk-7.patch, > hdfs-2576-trunk-8.1.patch, hdfs-2576-trunk-8.2.patch, > hdfs-2576-trunk-8.3.patch, hdfs-2576-trunk-8.patch > > > Sometimes Clients like HBase are required to dynamically compute the > datanodes it wishes to place the blocks for a file for higher level of > locality. For this purpose there is a need of a way to give the Namenode a > hint in terms of a favoredNodes parameter about the locations where the > client wants to put each block. The proposed solution is a favored nodes > parameter in the addBlock() method and in the create() file method to enable > the clients to give the hints to the NameNode about the locations of each > replica of the block. Note that this would be just a hint and finally the > NameNode would look at disk usage, datanode load etc. and decide whether it > can respect the hints or not. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (HDFS-2576) Namenode should have a favored nodes hint to enable clients to have control over block placement.
[ https://issues.apache.org/jira/browse/HDFS-2576?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=14909049#comment-14909049 ] 郭同 commented on HDFS-2576: -- I have the same problem,Thank you very much ... > Namenode should have a favored nodes hint to enable clients to have control > over block placement. > - > > Key: HDFS-2576 > URL: https://issues.apache.org/jira/browse/HDFS-2576 > Project: Hadoop HDFS > Issue Type: New Feature > Components: hdfs-client, namenode >Reporter: Pritam Damania >Assignee: Devaraj Das > Fix For: 2.1.0-beta > > Attachments: hdfs-2576-1.txt, hdfs-2576-trunk-1.patch, > hdfs-2576-trunk-2.patch, hdfs-2576-trunk-7.1.patch, hdfs-2576-trunk-7.patch, > hdfs-2576-trunk-8.1.patch, hdfs-2576-trunk-8.2.patch, > hdfs-2576-trunk-8.3.patch, hdfs-2576-trunk-8.patch > > > Sometimes Clients like HBase are required to dynamically compute the > datanodes it wishes to place the blocks for a file for higher level of > locality. For this purpose there is a need of a way to give the Namenode a > hint in terms of a favoredNodes parameter about the locations where the > client wants to put each block. The proposed solution is a favored nodes > parameter in the addBlock() method and in the create() file method to enable > the clients to give the hints to the NameNode about the locations of each > replica of the block. Note that this would be just a hint and finally the > NameNode would look at disk usage, datanode load etc. and decide whether it > can respect the hints or not. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (HDFS-2576) Namenode should have a favored nodes hint to enable clients to have control over block placement.
[ https://issues.apache.org/jira/browse/HDFS-2576?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=14909051#comment-14909051 ] 郭同 commented on HDFS-2576: -- I have the same problem,Thank you very much ... > Namenode should have a favored nodes hint to enable clients to have control > over block placement. > - > > Key: HDFS-2576 > URL: https://issues.apache.org/jira/browse/HDFS-2576 > Project: Hadoop HDFS > Issue Type: New Feature > Components: hdfs-client, namenode >Reporter: Pritam Damania >Assignee: Devaraj Das > Fix For: 2.1.0-beta > > Attachments: hdfs-2576-1.txt, hdfs-2576-trunk-1.patch, > hdfs-2576-trunk-2.patch, hdfs-2576-trunk-7.1.patch, hdfs-2576-trunk-7.patch, > hdfs-2576-trunk-8.1.patch, hdfs-2576-trunk-8.2.patch, > hdfs-2576-trunk-8.3.patch, hdfs-2576-trunk-8.patch > > > Sometimes Clients like HBase are required to dynamically compute the > datanodes it wishes to place the blocks for a file for higher level of > locality. For this purpose there is a need of a way to give the Namenode a > hint in terms of a favoredNodes parameter about the locations where the > client wants to put each block. The proposed solution is a favored nodes > parameter in the addBlock() method and in the create() file method to enable > the clients to give the hints to the NameNode about the locations of each > replica of the block. Note that this would be just a hint and finally the > NameNode would look at disk usage, datanode load etc. and decide whether it > can respect the hints or not. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (HDFS-2576) Namenode should have a favored nodes hint to enable clients to have control over block placement.
[ https://issues.apache.org/jira/browse/HDFS-2576?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=14909052#comment-14909052 ] 郭同 commented on HDFS-2576: -- I have the same problem,Thank you very much ... > Namenode should have a favored nodes hint to enable clients to have control > over block placement. > - > > Key: HDFS-2576 > URL: https://issues.apache.org/jira/browse/HDFS-2576 > Project: Hadoop HDFS > Issue Type: New Feature > Components: hdfs-client, namenode >Reporter: Pritam Damania >Assignee: Devaraj Das > Fix For: 2.1.0-beta > > Attachments: hdfs-2576-1.txt, hdfs-2576-trunk-1.patch, > hdfs-2576-trunk-2.patch, hdfs-2576-trunk-7.1.patch, hdfs-2576-trunk-7.patch, > hdfs-2576-trunk-8.1.patch, hdfs-2576-trunk-8.2.patch, > hdfs-2576-trunk-8.3.patch, hdfs-2576-trunk-8.patch > > > Sometimes Clients like HBase are required to dynamically compute the > datanodes it wishes to place the blocks for a file for higher level of > locality. For this purpose there is a need of a way to give the Namenode a > hint in terms of a favoredNodes parameter about the locations where the > client wants to put each block. The proposed solution is a favored nodes > parameter in the addBlock() method and in the create() file method to enable > the clients to give the hints to the NameNode about the locations of each > replica of the block. Note that this would be just a hint and finally the > NameNode would look at disk usage, datanode load etc. and decide whether it > can respect the hints or not. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (HDFS-2576) Namenode should have a favored nodes hint to enable clients to have control over block placement.
[ https://issues.apache.org/jira/browse/HDFS-2576?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=14909058#comment-14909058 ] 郭同 commented on HDFS-2576: -- I have the same problem,Thank you very much ... > Namenode should have a favored nodes hint to enable clients to have control > over block placement. > - > > Key: HDFS-2576 > URL: https://issues.apache.org/jira/browse/HDFS-2576 > Project: Hadoop HDFS > Issue Type: New Feature > Components: hdfs-client, namenode >Reporter: Pritam Damania >Assignee: Devaraj Das > Fix For: 2.1.0-beta > > Attachments: hdfs-2576-1.txt, hdfs-2576-trunk-1.patch, > hdfs-2576-trunk-2.patch, hdfs-2576-trunk-7.1.patch, hdfs-2576-trunk-7.patch, > hdfs-2576-trunk-8.1.patch, hdfs-2576-trunk-8.2.patch, > hdfs-2576-trunk-8.3.patch, hdfs-2576-trunk-8.patch > > > Sometimes Clients like HBase are required to dynamically compute the > datanodes it wishes to place the blocks for a file for higher level of > locality. For this purpose there is a need of a way to give the Namenode a > hint in terms of a favoredNodes parameter about the locations where the > client wants to put each block. The proposed solution is a favored nodes > parameter in the addBlock() method and in the create() file method to enable > the clients to give the hints to the NameNode about the locations of each > replica of the block. Note that this would be just a hint and finally the > NameNode would look at disk usage, datanode load etc. and decide whether it > can respect the hints or not. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (HDFS-2576) Namenode should have a favored nodes hint to enable clients to have control over block placement.
[ https://issues.apache.org/jira/browse/HDFS-2576?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=14909057#comment-14909057 ] 郭同 commented on HDFS-2576: -- I have the same problem,Thank you very much ... > Namenode should have a favored nodes hint to enable clients to have control > over block placement. > - > > Key: HDFS-2576 > URL: https://issues.apache.org/jira/browse/HDFS-2576 > Project: Hadoop HDFS > Issue Type: New Feature > Components: hdfs-client, namenode >Reporter: Pritam Damania >Assignee: Devaraj Das > Fix For: 2.1.0-beta > > Attachments: hdfs-2576-1.txt, hdfs-2576-trunk-1.patch, > hdfs-2576-trunk-2.patch, hdfs-2576-trunk-7.1.patch, hdfs-2576-trunk-7.patch, > hdfs-2576-trunk-8.1.patch, hdfs-2576-trunk-8.2.patch, > hdfs-2576-trunk-8.3.patch, hdfs-2576-trunk-8.patch > > > Sometimes Clients like HBase are required to dynamically compute the > datanodes it wishes to place the blocks for a file for higher level of > locality. For this purpose there is a need of a way to give the Namenode a > hint in terms of a favoredNodes parameter about the locations where the > client wants to put each block. The proposed solution is a favored nodes > parameter in the addBlock() method and in the create() file method to enable > the clients to give the hints to the NameNode about the locations of each > replica of the block. Note that this would be just a hint and finally the > NameNode would look at disk usage, datanode load etc. and decide whether it > can respect the hints or not. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (HDFS-2576) Namenode should have a favored nodes hint to enable clients to have control over block placement.
[ https://issues.apache.org/jira/browse/HDFS-2576?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=14909053#comment-14909053 ] 郭同 commented on HDFS-2576: -- I have the same problem,Thank you very much ... > Namenode should have a favored nodes hint to enable clients to have control > over block placement. > - > > Key: HDFS-2576 > URL: https://issues.apache.org/jira/browse/HDFS-2576 > Project: Hadoop HDFS > Issue Type: New Feature > Components: hdfs-client, namenode >Reporter: Pritam Damania >Assignee: Devaraj Das > Fix For: 2.1.0-beta > > Attachments: hdfs-2576-1.txt, hdfs-2576-trunk-1.patch, > hdfs-2576-trunk-2.patch, hdfs-2576-trunk-7.1.patch, hdfs-2576-trunk-7.patch, > hdfs-2576-trunk-8.1.patch, hdfs-2576-trunk-8.2.patch, > hdfs-2576-trunk-8.3.patch, hdfs-2576-trunk-8.patch > > > Sometimes Clients like HBase are required to dynamically compute the > datanodes it wishes to place the blocks for a file for higher level of > locality. For this purpose there is a need of a way to give the Namenode a > hint in terms of a favoredNodes parameter about the locations where the > client wants to put each block. The proposed solution is a favored nodes > parameter in the addBlock() method and in the create() file method to enable > the clients to give the hints to the NameNode about the locations of each > replica of the block. Note that this would be just a hint and finally the > NameNode would look at disk usage, datanode load etc. and decide whether it > can respect the hints or not. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (HDFS-2576) Namenode should have a favored nodes hint to enable clients to have control over block placement.
[ https://issues.apache.org/jira/browse/HDFS-2576?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=14909055#comment-14909055 ] 郭同 commented on HDFS-2576: -- I have the same problem,Thank you very much ... > Namenode should have a favored nodes hint to enable clients to have control > over block placement. > - > > Key: HDFS-2576 > URL: https://issues.apache.org/jira/browse/HDFS-2576 > Project: Hadoop HDFS > Issue Type: New Feature > Components: hdfs-client, namenode >Reporter: Pritam Damania >Assignee: Devaraj Das > Fix For: 2.1.0-beta > > Attachments: hdfs-2576-1.txt, hdfs-2576-trunk-1.patch, > hdfs-2576-trunk-2.patch, hdfs-2576-trunk-7.1.patch, hdfs-2576-trunk-7.patch, > hdfs-2576-trunk-8.1.patch, hdfs-2576-trunk-8.2.patch, > hdfs-2576-trunk-8.3.patch, hdfs-2576-trunk-8.patch > > > Sometimes Clients like HBase are required to dynamically compute the > datanodes it wishes to place the blocks for a file for higher level of > locality. For this purpose there is a need of a way to give the Namenode a > hint in terms of a favoredNodes parameter about the locations where the > client wants to put each block. The proposed solution is a favored nodes > parameter in the addBlock() method and in the create() file method to enable > the clients to give the hints to the NameNode about the locations of each > replica of the block. Note that this would be just a hint and finally the > NameNode would look at disk usage, datanode load etc. and decide whether it > can respect the hints or not. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (HDFS-2576) Namenode should have a favored nodes hint to enable clients to have control over block placement.
[ https://issues.apache.org/jira/browse/HDFS-2576?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=14909050#comment-14909050 ] 郭同 commented on HDFS-2576: -- I have the same problem,Thank you very much ... > Namenode should have a favored nodes hint to enable clients to have control > over block placement. > - > > Key: HDFS-2576 > URL: https://issues.apache.org/jira/browse/HDFS-2576 > Project: Hadoop HDFS > Issue Type: New Feature > Components: hdfs-client, namenode >Reporter: Pritam Damania >Assignee: Devaraj Das > Fix For: 2.1.0-beta > > Attachments: hdfs-2576-1.txt, hdfs-2576-trunk-1.patch, > hdfs-2576-trunk-2.patch, hdfs-2576-trunk-7.1.patch, hdfs-2576-trunk-7.patch, > hdfs-2576-trunk-8.1.patch, hdfs-2576-trunk-8.2.patch, > hdfs-2576-trunk-8.3.patch, hdfs-2576-trunk-8.patch > > > Sometimes Clients like HBase are required to dynamically compute the > datanodes it wishes to place the blocks for a file for higher level of > locality. For this purpose there is a need of a way to give the Namenode a > hint in terms of a favoredNodes parameter about the locations where the > client wants to put each block. The proposed solution is a favored nodes > parameter in the addBlock() method and in the create() file method to enable > the clients to give the hints to the NameNode about the locations of each > replica of the block. Note that this would be just a hint and finally the > NameNode would look at disk usage, datanode load etc. and decide whether it > can respect the hints or not. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (HDFS-2576) Namenode should have a favored nodes hint to enable clients to have control over block placement.
[ https://issues.apache.org/jira/browse/HDFS-2576?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=14909059#comment-14909059 ] 郭同 commented on HDFS-2576: -- I have the same problem,Thank you very much ... > Namenode should have a favored nodes hint to enable clients to have control > over block placement. > - > > Key: HDFS-2576 > URL: https://issues.apache.org/jira/browse/HDFS-2576 > Project: Hadoop HDFS > Issue Type: New Feature > Components: hdfs-client, namenode >Reporter: Pritam Damania >Assignee: Devaraj Das > Fix For: 2.1.0-beta > > Attachments: hdfs-2576-1.txt, hdfs-2576-trunk-1.patch, > hdfs-2576-trunk-2.patch, hdfs-2576-trunk-7.1.patch, hdfs-2576-trunk-7.patch, > hdfs-2576-trunk-8.1.patch, hdfs-2576-trunk-8.2.patch, > hdfs-2576-trunk-8.3.patch, hdfs-2576-trunk-8.patch > > > Sometimes Clients like HBase are required to dynamically compute the > datanodes it wishes to place the blocks for a file for higher level of > locality. For this purpose there is a need of a way to give the Namenode a > hint in terms of a favoredNodes parameter about the locations where the > client wants to put each block. The proposed solution is a favored nodes > parameter in the addBlock() method and in the create() file method to enable > the clients to give the hints to the NameNode about the locations of each > replica of the block. Note that this would be just a hint and finally the > NameNode would look at disk usage, datanode load etc. and decide whether it > can respect the hints or not. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (HDFS-2576) Namenode should have a favored nodes hint to enable clients to have control over block placement.
[ https://issues.apache.org/jira/browse/HDFS-2576?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=14909056#comment-14909056 ] 郭同 commented on HDFS-2576: -- I have the same problem,Thank you very much ... > Namenode should have a favored nodes hint to enable clients to have control > over block placement. > - > > Key: HDFS-2576 > URL: https://issues.apache.org/jira/browse/HDFS-2576 > Project: Hadoop HDFS > Issue Type: New Feature > Components: hdfs-client, namenode >Reporter: Pritam Damania >Assignee: Devaraj Das > Fix For: 2.1.0-beta > > Attachments: hdfs-2576-1.txt, hdfs-2576-trunk-1.patch, > hdfs-2576-trunk-2.patch, hdfs-2576-trunk-7.1.patch, hdfs-2576-trunk-7.patch, > hdfs-2576-trunk-8.1.patch, hdfs-2576-trunk-8.2.patch, > hdfs-2576-trunk-8.3.patch, hdfs-2576-trunk-8.patch > > > Sometimes Clients like HBase are required to dynamically compute the > datanodes it wishes to place the blocks for a file for higher level of > locality. For this purpose there is a need of a way to give the Namenode a > hint in terms of a favoredNodes parameter about the locations where the > client wants to put each block. The proposed solution is a favored nodes > parameter in the addBlock() method and in the create() file method to enable > the clients to give the hints to the NameNode about the locations of each > replica of the block. Note that this would be just a hint and finally the > NameNode would look at disk usage, datanode load etc. and decide whether it > can respect the hints or not. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (HDFS-2576) Namenode should have a favored nodes hint to enable clients to have control over block placement.
[ https://issues.apache.org/jira/browse/HDFS-2576?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=14909048#comment-14909048 ] 郭同 commented on HDFS-2576: -- I have the same problem,Thank you very much ... > Namenode should have a favored nodes hint to enable clients to have control > over block placement. > - > > Key: HDFS-2576 > URL: https://issues.apache.org/jira/browse/HDFS-2576 > Project: Hadoop HDFS > Issue Type: New Feature > Components: hdfs-client, namenode >Reporter: Pritam Damania >Assignee: Devaraj Das > Fix For: 2.1.0-beta > > Attachments: hdfs-2576-1.txt, hdfs-2576-trunk-1.patch, > hdfs-2576-trunk-2.patch, hdfs-2576-trunk-7.1.patch, hdfs-2576-trunk-7.patch, > hdfs-2576-trunk-8.1.patch, hdfs-2576-trunk-8.2.patch, > hdfs-2576-trunk-8.3.patch, hdfs-2576-trunk-8.patch > > > Sometimes Clients like HBase are required to dynamically compute the > datanodes it wishes to place the blocks for a file for higher level of > locality. For this purpose there is a need of a way to give the Namenode a > hint in terms of a favoredNodes parameter about the locations where the > client wants to put each block. The proposed solution is a favored nodes > parameter in the addBlock() method and in the create() file method to enable > the clients to give the hints to the NameNode about the locations of each > replica of the block. Note that this would be just a hint and finally the > NameNode would look at disk usage, datanode load etc. and decide whether it > can respect the hints or not. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (HDFS-2576) Namenode should have a favored nodes hint to enable clients to have control over block placement.
[ https://issues.apache.org/jira/browse/HDFS-2576?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=14909046#comment-14909046 ] 郭同 commented on HDFS-2576: -- I have the same problem,Thank you very much ... > Namenode should have a favored nodes hint to enable clients to have control > over block placement. > - > > Key: HDFS-2576 > URL: https://issues.apache.org/jira/browse/HDFS-2576 > Project: Hadoop HDFS > Issue Type: New Feature > Components: hdfs-client, namenode >Reporter: Pritam Damania >Assignee: Devaraj Das > Fix For: 2.1.0-beta > > Attachments: hdfs-2576-1.txt, hdfs-2576-trunk-1.patch, > hdfs-2576-trunk-2.patch, hdfs-2576-trunk-7.1.patch, hdfs-2576-trunk-7.patch, > hdfs-2576-trunk-8.1.patch, hdfs-2576-trunk-8.2.patch, > hdfs-2576-trunk-8.3.patch, hdfs-2576-trunk-8.patch > > > Sometimes Clients like HBase are required to dynamically compute the > datanodes it wishes to place the blocks for a file for higher level of > locality. For this purpose there is a need of a way to give the Namenode a > hint in terms of a favoredNodes parameter about the locations where the > client wants to put each block. The proposed solution is a favored nodes > parameter in the addBlock() method and in the create() file method to enable > the clients to give the hints to the NameNode about the locations of each > replica of the block. Note that this would be just a hint and finally the > NameNode would look at disk usage, datanode load etc. and decide whether it > can respect the hints or not. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (HDFS-2576) Namenode should have a favored nodes hint to enable clients to have control over block placement.
[ https://issues.apache.org/jira/browse/HDFS-2576?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13996841#comment-13996841 ] samira elhami commented on HDFS-2576: - This patch is really helpful. I know it is added to hadoop-2.1.0-Beta. I've install this version of hadoop but I don't know how to set the block placement. I've searched everywhere. please help me. Thanks Namenode should have a favored nodes hint to enable clients to have control over block placement. - Key: HDFS-2576 URL: https://issues.apache.org/jira/browse/HDFS-2576 Project: Hadoop HDFS Issue Type: New Feature Components: hdfs-client, namenode Reporter: Pritam Damania Assignee: Devaraj Das Fix For: 2.1.0-beta Attachments: hdfs-2576-1.txt, hdfs-2576-trunk-1.patch, hdfs-2576-trunk-2.patch, hdfs-2576-trunk-7.1.patch, hdfs-2576-trunk-7.patch, hdfs-2576-trunk-8.1.patch, hdfs-2576-trunk-8.2.patch, hdfs-2576-trunk-8.3.patch, hdfs-2576-trunk-8.patch Sometimes Clients like HBase are required to dynamically compute the datanodes it wishes to place the blocks for a file for higher level of locality. For this purpose there is a need of a way to give the Namenode a hint in terms of a favoredNodes parameter about the locations where the client wants to put each block. The proposed solution is a favored nodes parameter in the addBlock() method and in the create() file method to enable the clients to give the hints to the NameNode about the locations of each replica of the block. Note that this would be just a hint and finally the NameNode would look at disk usage, datanode load etc. and decide whether it can respect the hints or not. -- This message was sent by Atlassian JIRA (v6.2#6252)
[jira] [Commented] (HDFS-2576) Namenode should have a favored nodes hint to enable clients to have control over block placement.
[ https://issues.apache.org/jira/browse/HDFS-2576?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13651560#comment-13651560 ] Jian He commented on HDFS-2576: --- I think this patch breaks branch-2, can you please take a look ? Namenode should have a favored nodes hint to enable clients to have control over block placement. - Key: HDFS-2576 URL: https://issues.apache.org/jira/browse/HDFS-2576 Project: Hadoop HDFS Issue Type: New Feature Components: hdfs-client, namenode Reporter: Pritam Damania Assignee: Devaraj Das Fix For: 2.0.5-beta Attachments: hdfs-2576-1.txt, hdfs-2576-trunk-1.patch, hdfs-2576-trunk-2.patch, hdfs-2576-trunk-7.1.patch, hdfs-2576-trunk-7.patch, hdfs-2576-trunk-8.1.patch, hdfs-2576-trunk-8.2.patch, hdfs-2576-trunk-8.3.patch, hdfs-2576-trunk-8.patch Sometimes Clients like HBase are required to dynamically compute the datanodes it wishes to place the blocks for a file for higher level of locality. For this purpose there is a need of a way to give the Namenode a hint in terms of a favoredNodes parameter about the locations where the client wants to put each block. The proposed solution is a favored nodes parameter in the addBlock() method and in the create() file method to enable the clients to give the hints to the NameNode about the locations of each replica of the block. Note that this would be just a hint and finally the NameNode would look at disk usage, datanode load etc. and decide whether it can respect the hints or not. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HDFS-2576) Namenode should have a favored nodes hint to enable clients to have control over block placement.
[ https://issues.apache.org/jira/browse/HDFS-2576?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13651580#comment-13651580 ] Devaraj Das commented on HDFS-2576: --- I am looking. Sorry for the inconvenience. Namenode should have a favored nodes hint to enable clients to have control over block placement. - Key: HDFS-2576 URL: https://issues.apache.org/jira/browse/HDFS-2576 Project: Hadoop HDFS Issue Type: New Feature Components: hdfs-client, namenode Reporter: Pritam Damania Assignee: Devaraj Das Fix For: 2.0.5-beta Attachments: hdfs-2576-1.txt, hdfs-2576-trunk-1.patch, hdfs-2576-trunk-2.patch, hdfs-2576-trunk-7.1.patch, hdfs-2576-trunk-7.patch, hdfs-2576-trunk-8.1.patch, hdfs-2576-trunk-8.2.patch, hdfs-2576-trunk-8.3.patch, hdfs-2576-trunk-8.patch Sometimes Clients like HBase are required to dynamically compute the datanodes it wishes to place the blocks for a file for higher level of locality. For this purpose there is a need of a way to give the Namenode a hint in terms of a favoredNodes parameter about the locations where the client wants to put each block. The proposed solution is a favored nodes parameter in the addBlock() method and in the create() file method to enable the clients to give the hints to the NameNode about the locations of each replica of the block. Note that this would be just a hint and finally the NameNode would look at disk usage, datanode load etc. and decide whether it can respect the hints or not. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HDFS-2576) Namenode should have a favored nodes hint to enable clients to have control over block placement.
[ https://issues.apache.org/jira/browse/HDFS-2576?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13651584#comment-13651584 ] Devaraj Das commented on HDFS-2576: --- Committed a patch that fixes the compilation issues on branch-2. Namenode should have a favored nodes hint to enable clients to have control over block placement. - Key: HDFS-2576 URL: https://issues.apache.org/jira/browse/HDFS-2576 Project: Hadoop HDFS Issue Type: New Feature Components: hdfs-client, namenode Reporter: Pritam Damania Assignee: Devaraj Das Fix For: 2.0.5-beta Attachments: hdfs-2576-1.txt, hdfs-2576-trunk-1.patch, hdfs-2576-trunk-2.patch, hdfs-2576-trunk-7.1.patch, hdfs-2576-trunk-7.patch, hdfs-2576-trunk-8.1.patch, hdfs-2576-trunk-8.2.patch, hdfs-2576-trunk-8.3.patch, hdfs-2576-trunk-8.patch Sometimes Clients like HBase are required to dynamically compute the datanodes it wishes to place the blocks for a file for higher level of locality. For this purpose there is a need of a way to give the Namenode a hint in terms of a favoredNodes parameter about the locations where the client wants to put each block. The proposed solution is a favored nodes parameter in the addBlock() method and in the create() file method to enable the clients to give the hints to the NameNode about the locations of each replica of the block. Note that this would be just a hint and finally the NameNode would look at disk usage, datanode load etc. and decide whether it can respect the hints or not. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HDFS-2576) Namenode should have a favored nodes hint to enable clients to have control over block placement.
[ https://issues.apache.org/jira/browse/HDFS-2576?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13646503#comment-13646503 ] Hudson commented on HDFS-2576: -- Integrated in Hadoop-Yarn-trunk #200 (See [https://builds.apache.org/job/Hadoop-Yarn-trunk/200/]) HDFS-2576. Moved the commit message lines for HDFS-4778 and HDFS-2576 in CHANGES.txt to the right places. (Revision 1477855) HDFS-4778. Fixes some issues that the first patch on HDFS-2576 missed. Contributed by Devaraj Das. (Revision 1477849) Result = SUCCESS ddas : http://svn.apache.org/viewcvs.cgi/?root=Apache-SVNview=revrev=1477855 Files : * /hadoop/common/trunk/hadoop-hdfs-project/hadoop-hdfs/CHANGES.txt ddas : http://svn.apache.org/viewcvs.cgi/?root=Apache-SVNview=revrev=1477849 Files : * /hadoop/common/trunk/hadoop-hdfs-project/hadoop-hdfs/CHANGES.txt * /hadoop/common/trunk/hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/blockmanagement/BlockPlacementPolicyDefault.java * /hadoop/common/trunk/hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/blockmanagement/DatanodeManager.java * /hadoop/common/trunk/hadoop-hdfs-project/hadoop-hdfs/src/test/java/org/apache/hadoop/hdfs/server/namenode/TestFavoredNodesEndToEnd.java Namenode should have a favored nodes hint to enable clients to have control over block placement. - Key: HDFS-2576 URL: https://issues.apache.org/jira/browse/HDFS-2576 Project: Hadoop HDFS Issue Type: New Feature Components: hdfs-client, namenode Reporter: Pritam Damania Assignee: Devaraj Das Fix For: 2.0.5-beta Attachments: hdfs-2576-1.txt, hdfs-2576-trunk-1.patch, hdfs-2576-trunk-2.patch, hdfs-2576-trunk-7.1.patch, hdfs-2576-trunk-7.patch, hdfs-2576-trunk-8.1.patch, hdfs-2576-trunk-8.2.patch, hdfs-2576-trunk-8.3.patch, hdfs-2576-trunk-8.patch Sometimes Clients like HBase are required to dynamically compute the datanodes it wishes to place the blocks for a file for higher level of locality. For this purpose there is a need of a way to give the Namenode a hint in terms of a favoredNodes parameter about the locations where the client wants to put each block. The proposed solution is a favored nodes parameter in the addBlock() method and in the create() file method to enable the clients to give the hints to the NameNode about the locations of each replica of the block. Note that this would be just a hint and finally the NameNode would look at disk usage, datanode load etc. and decide whether it can respect the hints or not. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HDFS-2576) Namenode should have a favored nodes hint to enable clients to have control over block placement.
[ https://issues.apache.org/jira/browse/HDFS-2576?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13646549#comment-13646549 ] Hudson commented on HDFS-2576: -- Integrated in Hadoop-Hdfs-trunk #1389 (See [https://builds.apache.org/job/Hadoop-Hdfs-trunk/1389/]) HDFS-2576. Moved the commit message lines for HDFS-4778 and HDFS-2576 in CHANGES.txt to the right places. (Revision 1477855) HDFS-4778. Fixes some issues that the first patch on HDFS-2576 missed. Contributed by Devaraj Das. (Revision 1477849) Result = FAILURE ddas : http://svn.apache.org/viewcvs.cgi/?root=Apache-SVNview=revrev=1477855 Files : * /hadoop/common/trunk/hadoop-hdfs-project/hadoop-hdfs/CHANGES.txt ddas : http://svn.apache.org/viewcvs.cgi/?root=Apache-SVNview=revrev=1477849 Files : * /hadoop/common/trunk/hadoop-hdfs-project/hadoop-hdfs/CHANGES.txt * /hadoop/common/trunk/hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/blockmanagement/BlockPlacementPolicyDefault.java * /hadoop/common/trunk/hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/blockmanagement/DatanodeManager.java * /hadoop/common/trunk/hadoop-hdfs-project/hadoop-hdfs/src/test/java/org/apache/hadoop/hdfs/server/namenode/TestFavoredNodesEndToEnd.java Namenode should have a favored nodes hint to enable clients to have control over block placement. - Key: HDFS-2576 URL: https://issues.apache.org/jira/browse/HDFS-2576 Project: Hadoop HDFS Issue Type: New Feature Components: hdfs-client, namenode Reporter: Pritam Damania Assignee: Devaraj Das Fix For: 2.0.5-beta Attachments: hdfs-2576-1.txt, hdfs-2576-trunk-1.patch, hdfs-2576-trunk-2.patch, hdfs-2576-trunk-7.1.patch, hdfs-2576-trunk-7.patch, hdfs-2576-trunk-8.1.patch, hdfs-2576-trunk-8.2.patch, hdfs-2576-trunk-8.3.patch, hdfs-2576-trunk-8.patch Sometimes Clients like HBase are required to dynamically compute the datanodes it wishes to place the blocks for a file for higher level of locality. For this purpose there is a need of a way to give the Namenode a hint in terms of a favoredNodes parameter about the locations where the client wants to put each block. The proposed solution is a favored nodes parameter in the addBlock() method and in the create() file method to enable the clients to give the hints to the NameNode about the locations of each replica of the block. Note that this would be just a hint and finally the NameNode would look at disk usage, datanode load etc. and decide whether it can respect the hints or not. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HDFS-2576) Namenode should have a favored nodes hint to enable clients to have control over block placement.
[ https://issues.apache.org/jira/browse/HDFS-2576?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13646602#comment-13646602 ] Hudson commented on HDFS-2576: -- Integrated in Hadoop-Mapreduce-trunk #1416 (See [https://builds.apache.org/job/Hadoop-Mapreduce-trunk/1416/]) HDFS-2576. Moved the commit message lines for HDFS-4778 and HDFS-2576 in CHANGES.txt to the right places. (Revision 1477855) HDFS-4778. Fixes some issues that the first patch on HDFS-2576 missed. Contributed by Devaraj Das. (Revision 1477849) Result = FAILURE ddas : http://svn.apache.org/viewcvs.cgi/?root=Apache-SVNview=revrev=1477855 Files : * /hadoop/common/trunk/hadoop-hdfs-project/hadoop-hdfs/CHANGES.txt ddas : http://svn.apache.org/viewcvs.cgi/?root=Apache-SVNview=revrev=1477849 Files : * /hadoop/common/trunk/hadoop-hdfs-project/hadoop-hdfs/CHANGES.txt * /hadoop/common/trunk/hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/blockmanagement/BlockPlacementPolicyDefault.java * /hadoop/common/trunk/hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/blockmanagement/DatanodeManager.java * /hadoop/common/trunk/hadoop-hdfs-project/hadoop-hdfs/src/test/java/org/apache/hadoop/hdfs/server/namenode/TestFavoredNodesEndToEnd.java Namenode should have a favored nodes hint to enable clients to have control over block placement. - Key: HDFS-2576 URL: https://issues.apache.org/jira/browse/HDFS-2576 Project: Hadoop HDFS Issue Type: New Feature Components: hdfs-client, namenode Reporter: Pritam Damania Assignee: Devaraj Das Fix For: 2.0.5-beta Attachments: hdfs-2576-1.txt, hdfs-2576-trunk-1.patch, hdfs-2576-trunk-2.patch, hdfs-2576-trunk-7.1.patch, hdfs-2576-trunk-7.patch, hdfs-2576-trunk-8.1.patch, hdfs-2576-trunk-8.2.patch, hdfs-2576-trunk-8.3.patch, hdfs-2576-trunk-8.patch Sometimes Clients like HBase are required to dynamically compute the datanodes it wishes to place the blocks for a file for higher level of locality. For this purpose there is a need of a way to give the Namenode a hint in terms of a favoredNodes parameter about the locations where the client wants to put each block. The proposed solution is a favored nodes parameter in the addBlock() method and in the create() file method to enable the clients to give the hints to the NameNode about the locations of each replica of the block. Note that this would be just a hint and finally the NameNode would look at disk usage, datanode load etc. and decide whether it can respect the hints or not. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HDFS-2576) Namenode should have a favored nodes hint to enable clients to have control over block placement.
[ https://issues.apache.org/jira/browse/HDFS-2576?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13646038#comment-13646038 ] Hudson commented on HDFS-2576: -- Integrated in Hadoop-trunk-Commit #3701 (See [https://builds.apache.org/job/Hadoop-trunk-Commit/3701/]) HDFS-4778. Fixes some issues that the first patch on HDFS-2576 missed. Contributed by Devaraj Das. (Revision 1477849) Result = SUCCESS ddas : http://svn.apache.org/viewcvs.cgi/?root=Apache-SVNview=revrev=1477849 Files : * /hadoop/common/trunk/hadoop-hdfs-project/hadoop-hdfs/CHANGES.txt * /hadoop/common/trunk/hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/blockmanagement/BlockPlacementPolicyDefault.java * /hadoop/common/trunk/hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/blockmanagement/DatanodeManager.java * /hadoop/common/trunk/hadoop-hdfs-project/hadoop-hdfs/src/test/java/org/apache/hadoop/hdfs/server/namenode/TestFavoredNodesEndToEnd.java Namenode should have a favored nodes hint to enable clients to have control over block placement. - Key: HDFS-2576 URL: https://issues.apache.org/jira/browse/HDFS-2576 Project: Hadoop HDFS Issue Type: New Feature Components: hdfs-client, namenode Reporter: Pritam Damania Assignee: Devaraj Das Fix For: 2.0.5-beta Attachments: hdfs-2576-1.txt, hdfs-2576-trunk-1.patch, hdfs-2576-trunk-2.patch, hdfs-2576-trunk-7.1.patch, hdfs-2576-trunk-7.patch, hdfs-2576-trunk-8.1.patch, hdfs-2576-trunk-8.2.patch, hdfs-2576-trunk-8.3.patch, hdfs-2576-trunk-8.patch Sometimes Clients like HBase are required to dynamically compute the datanodes it wishes to place the blocks for a file for higher level of locality. For this purpose there is a need of a way to give the Namenode a hint in terms of a favoredNodes parameter about the locations where the client wants to put each block. The proposed solution is a favored nodes parameter in the addBlock() method and in the create() file method to enable the clients to give the hints to the NameNode about the locations of each replica of the block. Note that this would be just a hint and finally the NameNode would look at disk usage, datanode load etc. and decide whether it can respect the hints or not. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HDFS-2576) Namenode should have a favored nodes hint to enable clients to have control over block placement.
[ https://issues.apache.org/jira/browse/HDFS-2576?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13646084#comment-13646084 ] Hudson commented on HDFS-2576: -- Integrated in Hadoop-trunk-Commit #3702 (See [https://builds.apache.org/job/Hadoop-trunk-Commit/3702/]) HDFS-2576. Moved the commit message lines for HDFS-4778 and HDFS-2576 in CHANGES.txt to the right places. (Revision 1477855) Result = SUCCESS ddas : http://svn.apache.org/viewcvs.cgi/?root=Apache-SVNview=revrev=1477855 Files : * /hadoop/common/trunk/hadoop-hdfs-project/hadoop-hdfs/CHANGES.txt Namenode should have a favored nodes hint to enable clients to have control over block placement. - Key: HDFS-2576 URL: https://issues.apache.org/jira/browse/HDFS-2576 Project: Hadoop HDFS Issue Type: New Feature Components: hdfs-client, namenode Reporter: Pritam Damania Assignee: Devaraj Das Fix For: 2.0.5-beta Attachments: hdfs-2576-1.txt, hdfs-2576-trunk-1.patch, hdfs-2576-trunk-2.patch, hdfs-2576-trunk-7.1.patch, hdfs-2576-trunk-7.patch, hdfs-2576-trunk-8.1.patch, hdfs-2576-trunk-8.2.patch, hdfs-2576-trunk-8.3.patch, hdfs-2576-trunk-8.patch Sometimes Clients like HBase are required to dynamically compute the datanodes it wishes to place the blocks for a file for higher level of locality. For this purpose there is a need of a way to give the Namenode a hint in terms of a favoredNodes parameter about the locations where the client wants to put each block. The proposed solution is a favored nodes parameter in the addBlock() method and in the create() file method to enable the clients to give the hints to the NameNode about the locations of each replica of the block. Note that this would be just a hint and finally the NameNode would look at disk usage, datanode load etc. and decide whether it can respect the hints or not. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HDFS-2576) Namenode should have a favored nodes hint to enable clients to have control over block placement.
[ https://issues.apache.org/jira/browse/HDFS-2576?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13643623#comment-13643623 ] Hudson commented on HDFS-2576: -- Integrated in Hadoop-Yarn-trunk #196 (See [https://builds.apache.org/job/Hadoop-Yarn-trunk/196/]) HDFS-2576. Enhances the DistributedFileSystem's create API so that clients can specify favored datanodes for a file's blocks. Contributed by Devaraj Das and Pritam Damania. (Revision 1476395) Result = SUCCESS ddas : http://svn.apache.org/viewcvs.cgi/?root=Apache-SVNview=revrev=1476395 Files : * /hadoop/common/trunk/hadoop-common-project/hadoop-common/src/main/java/org/apache/hadoop/net/NetworkTopology.java * /hadoop/common/trunk/hadoop-hdfs-project/hadoop-hdfs/CHANGES.txt * /hadoop/common/trunk/hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/DFSClient.java * /hadoop/common/trunk/hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/DFSOutputStream.java * /hadoop/common/trunk/hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/DistributedFileSystem.java * /hadoop/common/trunk/hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/protocol/ClientProtocol.java * /hadoop/common/trunk/hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/protocolPB/ClientNamenodeProtocolServerSideTranslatorPB.java * /hadoop/common/trunk/hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/protocolPB/ClientNamenodeProtocolTranslatorPB.java * /hadoop/common/trunk/hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/blockmanagement/BlockManager.java * /hadoop/common/trunk/hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/blockmanagement/BlockPlacementPolicy.java * /hadoop/common/trunk/hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/blockmanagement/BlockPlacementPolicyDefault.java * /hadoop/common/trunk/hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/blockmanagement/DatanodeManager.java * /hadoop/common/trunk/hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/namenode/FSNamesystem.java * /hadoop/common/trunk/hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/namenode/NameNodeRpcServer.java * /hadoop/common/trunk/hadoop-hdfs-project/hadoop-hdfs/src/main/proto/ClientNamenodeProtocol.proto * /hadoop/common/trunk/hadoop-hdfs-project/hadoop-hdfs/src/test/java/org/apache/hadoop/hdfs/TestDFSClientRetries.java * /hadoop/common/trunk/hadoop-hdfs-project/hadoop-hdfs/src/test/java/org/apache/hadoop/hdfs/TestFileCreation.java * /hadoop/common/trunk/hadoop-hdfs-project/hadoop-hdfs/src/test/java/org/apache/hadoop/hdfs/server/namenode/NNThroughputBenchmark.java * /hadoop/common/trunk/hadoop-hdfs-project/hadoop-hdfs/src/test/java/org/apache/hadoop/hdfs/server/namenode/TestAddBlockRetry.java * /hadoop/common/trunk/hadoop-hdfs-project/hadoop-hdfs/src/test/java/org/apache/hadoop/hdfs/server/namenode/TestFavoredNodesEndToEnd.java Namenode should have a favored nodes hint to enable clients to have control over block placement. - Key: HDFS-2576 URL: https://issues.apache.org/jira/browse/HDFS-2576 Project: Hadoop HDFS Issue Type: New Feature Components: hdfs-client, namenode Reporter: Pritam Damania Assignee: Devaraj Das Fix For: 2.0.5-beta Attachments: hdfs-2576-1.txt, hdfs-2576-trunk-1.patch, hdfs-2576-trunk-2.patch, hdfs-2576-trunk-7.1.patch, hdfs-2576-trunk-7.patch, hdfs-2576-trunk-8.1.patch, hdfs-2576-trunk-8.2.patch, hdfs-2576-trunk-8.3.patch, hdfs-2576-trunk-8.patch Sometimes Clients like HBase are required to dynamically compute the datanodes it wishes to place the blocks for a file for higher level of locality. For this purpose there is a need of a way to give the Namenode a hint in terms of a favoredNodes parameter about the locations where the client wants to put each block. The proposed solution is a favored nodes parameter in the addBlock() method and in the create() file method to enable the clients to give the hints to the NameNode about the locations of each replica of the block. Note that this would be just a hint and finally the NameNode would look at disk usage, datanode load etc. and decide whether it can respect the hints or not. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HDFS-2576) Namenode should have a favored nodes hint to enable clients to have control over block placement.
[ https://issues.apache.org/jira/browse/HDFS-2576?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13643654#comment-13643654 ] Hudson commented on HDFS-2576: -- Integrated in Hadoop-Hdfs-trunk #1385 (See [https://builds.apache.org/job/Hadoop-Hdfs-trunk/1385/]) HDFS-2576. Enhances the DistributedFileSystem's create API so that clients can specify favored datanodes for a file's blocks. Contributed by Devaraj Das and Pritam Damania. (Revision 1476395) Result = FAILURE ddas : http://svn.apache.org/viewcvs.cgi/?root=Apache-SVNview=revrev=1476395 Files : * /hadoop/common/trunk/hadoop-common-project/hadoop-common/src/main/java/org/apache/hadoop/net/NetworkTopology.java * /hadoop/common/trunk/hadoop-hdfs-project/hadoop-hdfs/CHANGES.txt * /hadoop/common/trunk/hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/DFSClient.java * /hadoop/common/trunk/hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/DFSOutputStream.java * /hadoop/common/trunk/hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/DistributedFileSystem.java * /hadoop/common/trunk/hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/protocol/ClientProtocol.java * /hadoop/common/trunk/hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/protocolPB/ClientNamenodeProtocolServerSideTranslatorPB.java * /hadoop/common/trunk/hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/protocolPB/ClientNamenodeProtocolTranslatorPB.java * /hadoop/common/trunk/hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/blockmanagement/BlockManager.java * /hadoop/common/trunk/hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/blockmanagement/BlockPlacementPolicy.java * /hadoop/common/trunk/hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/blockmanagement/BlockPlacementPolicyDefault.java * /hadoop/common/trunk/hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/blockmanagement/DatanodeManager.java * /hadoop/common/trunk/hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/namenode/FSNamesystem.java * /hadoop/common/trunk/hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/namenode/NameNodeRpcServer.java * /hadoop/common/trunk/hadoop-hdfs-project/hadoop-hdfs/src/main/proto/ClientNamenodeProtocol.proto * /hadoop/common/trunk/hadoop-hdfs-project/hadoop-hdfs/src/test/java/org/apache/hadoop/hdfs/TestDFSClientRetries.java * /hadoop/common/trunk/hadoop-hdfs-project/hadoop-hdfs/src/test/java/org/apache/hadoop/hdfs/TestFileCreation.java * /hadoop/common/trunk/hadoop-hdfs-project/hadoop-hdfs/src/test/java/org/apache/hadoop/hdfs/server/namenode/NNThroughputBenchmark.java * /hadoop/common/trunk/hadoop-hdfs-project/hadoop-hdfs/src/test/java/org/apache/hadoop/hdfs/server/namenode/TestAddBlockRetry.java * /hadoop/common/trunk/hadoop-hdfs-project/hadoop-hdfs/src/test/java/org/apache/hadoop/hdfs/server/namenode/TestFavoredNodesEndToEnd.java Namenode should have a favored nodes hint to enable clients to have control over block placement. - Key: HDFS-2576 URL: https://issues.apache.org/jira/browse/HDFS-2576 Project: Hadoop HDFS Issue Type: New Feature Components: hdfs-client, namenode Reporter: Pritam Damania Assignee: Devaraj Das Fix For: 2.0.5-beta Attachments: hdfs-2576-1.txt, hdfs-2576-trunk-1.patch, hdfs-2576-trunk-2.patch, hdfs-2576-trunk-7.1.patch, hdfs-2576-trunk-7.patch, hdfs-2576-trunk-8.1.patch, hdfs-2576-trunk-8.2.patch, hdfs-2576-trunk-8.3.patch, hdfs-2576-trunk-8.patch Sometimes Clients like HBase are required to dynamically compute the datanodes it wishes to place the blocks for a file for higher level of locality. For this purpose there is a need of a way to give the Namenode a hint in terms of a favoredNodes parameter about the locations where the client wants to put each block. The proposed solution is a favored nodes parameter in the addBlock() method and in the create() file method to enable the clients to give the hints to the NameNode about the locations of each replica of the block. Note that this would be just a hint and finally the NameNode would look at disk usage, datanode load etc. and decide whether it can respect the hints or not. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HDFS-2576) Namenode should have a favored nodes hint to enable clients to have control over block placement.
[ https://issues.apache.org/jira/browse/HDFS-2576?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13643671#comment-13643671 ] Hudson commented on HDFS-2576: -- Integrated in Hadoop-Mapreduce-trunk #1412 (See [https://builds.apache.org/job/Hadoop-Mapreduce-trunk/1412/]) HDFS-2576. Enhances the DistributedFileSystem's create API so that clients can specify favored datanodes for a file's blocks. Contributed by Devaraj Das and Pritam Damania. (Revision 1476395) Result = SUCCESS ddas : http://svn.apache.org/viewcvs.cgi/?root=Apache-SVNview=revrev=1476395 Files : * /hadoop/common/trunk/hadoop-common-project/hadoop-common/src/main/java/org/apache/hadoop/net/NetworkTopology.java * /hadoop/common/trunk/hadoop-hdfs-project/hadoop-hdfs/CHANGES.txt * /hadoop/common/trunk/hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/DFSClient.java * /hadoop/common/trunk/hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/DFSOutputStream.java * /hadoop/common/trunk/hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/DistributedFileSystem.java * /hadoop/common/trunk/hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/protocol/ClientProtocol.java * /hadoop/common/trunk/hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/protocolPB/ClientNamenodeProtocolServerSideTranslatorPB.java * /hadoop/common/trunk/hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/protocolPB/ClientNamenodeProtocolTranslatorPB.java * /hadoop/common/trunk/hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/blockmanagement/BlockManager.java * /hadoop/common/trunk/hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/blockmanagement/BlockPlacementPolicy.java * /hadoop/common/trunk/hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/blockmanagement/BlockPlacementPolicyDefault.java * /hadoop/common/trunk/hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/blockmanagement/DatanodeManager.java * /hadoop/common/trunk/hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/namenode/FSNamesystem.java * /hadoop/common/trunk/hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/namenode/NameNodeRpcServer.java * /hadoop/common/trunk/hadoop-hdfs-project/hadoop-hdfs/src/main/proto/ClientNamenodeProtocol.proto * /hadoop/common/trunk/hadoop-hdfs-project/hadoop-hdfs/src/test/java/org/apache/hadoop/hdfs/TestDFSClientRetries.java * /hadoop/common/trunk/hadoop-hdfs-project/hadoop-hdfs/src/test/java/org/apache/hadoop/hdfs/TestFileCreation.java * /hadoop/common/trunk/hadoop-hdfs-project/hadoop-hdfs/src/test/java/org/apache/hadoop/hdfs/server/namenode/NNThroughputBenchmark.java * /hadoop/common/trunk/hadoop-hdfs-project/hadoop-hdfs/src/test/java/org/apache/hadoop/hdfs/server/namenode/TestAddBlockRetry.java * /hadoop/common/trunk/hadoop-hdfs-project/hadoop-hdfs/src/test/java/org/apache/hadoop/hdfs/server/namenode/TestFavoredNodesEndToEnd.java Namenode should have a favored nodes hint to enable clients to have control over block placement. - Key: HDFS-2576 URL: https://issues.apache.org/jira/browse/HDFS-2576 Project: Hadoop HDFS Issue Type: New Feature Components: hdfs-client, namenode Reporter: Pritam Damania Assignee: Devaraj Das Fix For: 2.0.5-beta Attachments: hdfs-2576-1.txt, hdfs-2576-trunk-1.patch, hdfs-2576-trunk-2.patch, hdfs-2576-trunk-7.1.patch, hdfs-2576-trunk-7.patch, hdfs-2576-trunk-8.1.patch, hdfs-2576-trunk-8.2.patch, hdfs-2576-trunk-8.3.patch, hdfs-2576-trunk-8.patch Sometimes Clients like HBase are required to dynamically compute the datanodes it wishes to place the blocks for a file for higher level of locality. For this purpose there is a need of a way to give the Namenode a hint in terms of a favoredNodes parameter about the locations where the client wants to put each block. The proposed solution is a favored nodes parameter in the addBlock() method and in the create() file method to enable the clients to give the hints to the NameNode about the locations of each replica of the block. Note that this would be just a hint and finally the NameNode would look at disk usage, datanode load etc. and decide whether it can respect the hints or not. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HDFS-2576) Namenode should have a favored nodes hint to enable clients to have control over block placement.
[
https://issues.apache.org/jira/browse/HDFS-2576?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13642706#comment-13642706
]
Hadoop QA commented on HDFS-2576:
-
{color:green}+1 overall{color}. Here are the results of testing the latest
attachment
http://issues.apache.org/jira/secure/attachment/12580672/hdfs-2576-trunk-8.3.patch
against trunk revision .
{color:green}+1 @author{color}. The patch does not contain any @author
tags.
{color:green}+1 tests included{color}. The patch appears to include 5 new
or modified test files.
{color:green}+1 javac{color}. The applied patch does not increase the
total number of javac compiler warnings.
{color:green}+1 javadoc{color}. The javadoc tool did not generate any
warning messages.
{color:green}+1 eclipse:eclipse{color}. The patch built with
eclipse:eclipse.
{color:green}+1 findbugs{color}. The patch does not introduce any new
Findbugs (version 1.3.9) warnings.
{color:green}+1 release audit{color}. The applied patch does not increase
the total number of release audit warnings.
{color:green}+1 core tests{color}. The patch passed unit tests in
hadoop-common-project/hadoop-common hadoop-hdfs-project/hadoop-hdfs.
{color:green}+1 contrib tests{color}. The patch passed contrib unit tests.
Test results:
https://builds.apache.org/job/PreCommit-HDFS-Build/4323//testReport/
Console output: https://builds.apache.org/job/PreCommit-HDFS-Build/4323//console
This message is automatically generated.
Namenode should have a favored nodes hint to enable clients to have control
over block placement.
-
Key: HDFS-2576
URL: https://issues.apache.org/jira/browse/HDFS-2576
Project: Hadoop HDFS
Issue Type: New Feature
Reporter: Pritam Damania
Assignee: Devaraj Das
Fix For: 2.0.5-beta
Attachments: hdfs-2576-1.txt, hdfs-2576-trunk-1.patch,
hdfs-2576-trunk-2.patch, hdfs-2576-trunk-7.1.patch, hdfs-2576-trunk-7.patch,
hdfs-2576-trunk-8.1.patch, hdfs-2576-trunk-8.2.patch,
hdfs-2576-trunk-8.3.patch, hdfs-2576-trunk-8.patch
Sometimes Clients like HBase are required to dynamically compute the
datanodes it wishes to place the blocks for a file for higher level of
locality. For this purpose there is a need of a way to give the Namenode a
hint in terms of a favoredNodes parameter about the locations where the
client wants to put each block. The proposed solution is a favored nodes
parameter in the addBlock() method and in the create() file method to enable
the clients to give the hints to the NameNode about the locations of each
replica of the block. Note that this would be just a hint and finally the
NameNode would look at disk usage, datanode load etc. and decide whether it
can respect the hints or not.
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HDFS-2576) Namenode should have a favored nodes hint to enable clients to have control over block placement.
[ https://issues.apache.org/jira/browse/HDFS-2576?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13643230#comment-13643230 ] Hudson commented on HDFS-2576: -- Integrated in Hadoop-trunk-Commit #3672 (See [https://builds.apache.org/job/Hadoop-trunk-Commit/3672/]) HDFS-2576. Enhances the DistributedFileSystem's create API so that clients can specify favored datanodes for a file's blocks. Contributed by Devaraj Das and Pritam Damania. (Revision 1476395) Result = SUCCESS ddas : http://svn.apache.org/viewcvs.cgi/?root=Apache-SVNview=revrev=1476395 Files : * /hadoop/common/trunk/hadoop-common-project/hadoop-common/src/main/java/org/apache/hadoop/net/NetworkTopology.java * /hadoop/common/trunk/hadoop-hdfs-project/hadoop-hdfs/CHANGES.txt * /hadoop/common/trunk/hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/DFSClient.java * /hadoop/common/trunk/hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/DFSOutputStream.java * /hadoop/common/trunk/hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/DistributedFileSystem.java * /hadoop/common/trunk/hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/protocol/ClientProtocol.java * /hadoop/common/trunk/hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/protocolPB/ClientNamenodeProtocolServerSideTranslatorPB.java * /hadoop/common/trunk/hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/protocolPB/ClientNamenodeProtocolTranslatorPB.java * /hadoop/common/trunk/hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/blockmanagement/BlockManager.java * /hadoop/common/trunk/hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/blockmanagement/BlockPlacementPolicy.java * /hadoop/common/trunk/hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/blockmanagement/BlockPlacementPolicyDefault.java * /hadoop/common/trunk/hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/blockmanagement/DatanodeManager.java * /hadoop/common/trunk/hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/namenode/FSNamesystem.java * /hadoop/common/trunk/hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/namenode/NameNodeRpcServer.java * /hadoop/common/trunk/hadoop-hdfs-project/hadoop-hdfs/src/main/proto/ClientNamenodeProtocol.proto * /hadoop/common/trunk/hadoop-hdfs-project/hadoop-hdfs/src/test/java/org/apache/hadoop/hdfs/TestDFSClientRetries.java * /hadoop/common/trunk/hadoop-hdfs-project/hadoop-hdfs/src/test/java/org/apache/hadoop/hdfs/TestFileCreation.java * /hadoop/common/trunk/hadoop-hdfs-project/hadoop-hdfs/src/test/java/org/apache/hadoop/hdfs/server/namenode/NNThroughputBenchmark.java * /hadoop/common/trunk/hadoop-hdfs-project/hadoop-hdfs/src/test/java/org/apache/hadoop/hdfs/server/namenode/TestAddBlockRetry.java * /hadoop/common/trunk/hadoop-hdfs-project/hadoop-hdfs/src/test/java/org/apache/hadoop/hdfs/server/namenode/TestFavoredNodesEndToEnd.java Namenode should have a favored nodes hint to enable clients to have control over block placement. - Key: HDFS-2576 URL: https://issues.apache.org/jira/browse/HDFS-2576 Project: Hadoop HDFS Issue Type: New Feature Components: hdfs-client, namenode Reporter: Pritam Damania Assignee: Devaraj Das Fix For: 2.0.5-beta Attachments: hdfs-2576-1.txt, hdfs-2576-trunk-1.patch, hdfs-2576-trunk-2.patch, hdfs-2576-trunk-7.1.patch, hdfs-2576-trunk-7.patch, hdfs-2576-trunk-8.1.patch, hdfs-2576-trunk-8.2.patch, hdfs-2576-trunk-8.3.patch, hdfs-2576-trunk-8.patch Sometimes Clients like HBase are required to dynamically compute the datanodes it wishes to place the blocks for a file for higher level of locality. For this purpose there is a need of a way to give the Namenode a hint in terms of a favoredNodes parameter about the locations where the client wants to put each block. The proposed solution is a favored nodes parameter in the addBlock() method and in the create() file method to enable the clients to give the hints to the NameNode about the locations of each replica of the block. Note that this would be just a hint and finally the NameNode would look at disk usage, datanode load etc. and decide whether it can respect the hints or not. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HDFS-2576) Namenode should have a favored nodes hint to enable clients to have control over block placement.
[ https://issues.apache.org/jira/browse/HDFS-2576?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13643235#comment-13643235 ] Devaraj Das commented on HDFS-2576: --- Thanks Pritam for the work on the fb branch from which the trunk patch was derived. Thanks, everyone for the review of the patches. I'll submit a patch for branch-1 shortly. Namenode should have a favored nodes hint to enable clients to have control over block placement. - Key: HDFS-2576 URL: https://issues.apache.org/jira/browse/HDFS-2576 Project: Hadoop HDFS Issue Type: New Feature Components: hdfs-client, namenode Reporter: Pritam Damania Assignee: Devaraj Das Fix For: 2.0.5-beta Attachments: hdfs-2576-1.txt, hdfs-2576-trunk-1.patch, hdfs-2576-trunk-2.patch, hdfs-2576-trunk-7.1.patch, hdfs-2576-trunk-7.patch, hdfs-2576-trunk-8.1.patch, hdfs-2576-trunk-8.2.patch, hdfs-2576-trunk-8.3.patch, hdfs-2576-trunk-8.patch Sometimes Clients like HBase are required to dynamically compute the datanodes it wishes to place the blocks for a file for higher level of locality. For this purpose there is a need of a way to give the Namenode a hint in terms of a favoredNodes parameter about the locations where the client wants to put each block. The proposed solution is a favored nodes parameter in the addBlock() method and in the create() file method to enable the clients to give the hints to the NameNode about the locations of each replica of the block. Note that this would be just a hint and finally the NameNode would look at disk usage, datanode load etc. and decide whether it can respect the hints or not. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HDFS-2576) Namenode should have a favored nodes hint to enable clients to have control over block placement.
[ https://issues.apache.org/jira/browse/HDFS-2576?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13643236#comment-13643236 ] Devaraj Das commented on HDFS-2576: --- Forgot to mention that I committed the patch in trunk. Namenode should have a favored nodes hint to enable clients to have control over block placement. - Key: HDFS-2576 URL: https://issues.apache.org/jira/browse/HDFS-2576 Project: Hadoop HDFS Issue Type: New Feature Components: hdfs-client, namenode Reporter: Pritam Damania Assignee: Devaraj Das Fix For: 2.0.5-beta Attachments: hdfs-2576-1.txt, hdfs-2576-trunk-1.patch, hdfs-2576-trunk-2.patch, hdfs-2576-trunk-7.1.patch, hdfs-2576-trunk-7.patch, hdfs-2576-trunk-8.1.patch, hdfs-2576-trunk-8.2.patch, hdfs-2576-trunk-8.3.patch, hdfs-2576-trunk-8.patch Sometimes Clients like HBase are required to dynamically compute the datanodes it wishes to place the blocks for a file for higher level of locality. For this purpose there is a need of a way to give the Namenode a hint in terms of a favoredNodes parameter about the locations where the client wants to put each block. The proposed solution is a favored nodes parameter in the addBlock() method and in the create() file method to enable the clients to give the hints to the NameNode about the locations of each replica of the block. Note that this would be just a hint and finally the NameNode would look at disk usage, datanode load etc. and decide whether it can respect the hints or not. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HDFS-2576) Namenode should have a favored nodes hint to enable clients to have control over block placement.
[ https://issues.apache.org/jira/browse/HDFS-2576?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13642189#comment-13642189 ] Tsz Wo (Nicholas), SZE commented on HDFS-2576: -- - In BlockPlacementPolicyDefault, it should use the oldExcludedNodes when choosing the remaining nodes. Also, what should happen if a node is in both favoredNodes and excludedNodes? - In BlockPlacementPolicy and BlockPlacementPolicyDefault, use Collections.DatanodeDescriptoremptyList() instead of new ArrayListDatanodeDescriptor(). - Change void resolveNetworkLocation(DatanodeDescriptor) to String resolveNetworkLocation(DatanodeID). Then, we only need to create DatanodeID but not DatanodeDescriptor in the new resolveNetworkLocation(String) method. - Move getDatanodeDescriptor(String address) from BlockManager to DatanodeManager. Namenode should have a favored nodes hint to enable clients to have control over block placement. - Key: HDFS-2576 URL: https://issues.apache.org/jira/browse/HDFS-2576 Project: Hadoop HDFS Issue Type: New Feature Reporter: Pritam Damania Assignee: Devaraj Das Fix For: 2.0.5-beta Attachments: hdfs-2576-1.txt, hdfs-2576-trunk-1.patch, hdfs-2576-trunk-2.patch, hdfs-2576-trunk-7.1.patch, hdfs-2576-trunk-7.patch, hdfs-2576-trunk-8.1.patch, hdfs-2576-trunk-8.patch Sometimes Clients like HBase are required to dynamically compute the datanodes it wishes to place the blocks for a file for higher level of locality. For this purpose there is a need of a way to give the Namenode a hint in terms of a favoredNodes parameter about the locations where the client wants to put each block. The proposed solution is a favored nodes parameter in the addBlock() method and in the create() file method to enable the clients to give the hints to the NameNode about the locations of each replica of the block. Note that this would be just a hint and finally the NameNode would look at disk usage, datanode load etc. and decide whether it can respect the hints or not. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HDFS-2576) Namenode should have a favored nodes hint to enable clients to have control over block placement.
[
https://issues.apache.org/jira/browse/HDFS-2576?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13642566#comment-13642566
]
Hadoop QA commented on HDFS-2576:
-
{color:red}-1 overall{color}. Here are the results of testing the latest
attachment
http://issues.apache.org/jira/secure/attachment/12580648/hdfs-2576-trunk-8.2.patch
against trunk revision .
{color:green}+1 @author{color}. The patch does not contain any @author
tags.
{color:green}+1 tests included{color}. The patch appears to include 5 new
or modified test files.
{color:green}+1 javac{color}. The applied patch does not increase the
total number of javac compiler warnings.
{color:green}+1 javadoc{color}. The javadoc tool did not generate any
warning messages.
{color:green}+1 eclipse:eclipse{color}. The patch built with
eclipse:eclipse.
{color:green}+1 findbugs{color}. The patch does not introduce any new
Findbugs (version 1.3.9) warnings.
{color:green}+1 release audit{color}. The applied patch does not increase
the total number of release audit warnings.
{color:red}-1 core tests{color}. The patch failed these unit tests in
hadoop-common-project/hadoop-common hadoop-hdfs-project/hadoop-hdfs:
org.apache.hadoop.hdfs.server.blockmanagement.TestReplicationPolicy
org.apache.hadoop.hdfs.server.datanode.TestBlockReplacement
org.apache.hadoop.net.TestNetworkTopology
org.apache.hadoop.cli.TestHDFSCLI
org.apache.hadoop.hdfs.server.balancer.TestBalancerWithNodeGroup
org.apache.hadoop.hdfs.server.namenode.TestHostsFiles
org.apache.hadoop.hdfs.server.blockmanagement.TestBlocksWithNotEnoughRacks
org.apache.hadoop.hdfs.TestReplication
{color:green}+1 contrib tests{color}. The patch passed contrib unit tests.
Test results:
https://builds.apache.org/job/PreCommit-HDFS-Build/4321//testReport/
Console output: https://builds.apache.org/job/PreCommit-HDFS-Build/4321//console
This message is automatically generated.
Namenode should have a favored nodes hint to enable clients to have control
over block placement.
-
Key: HDFS-2576
URL: https://issues.apache.org/jira/browse/HDFS-2576
Project: Hadoop HDFS
Issue Type: New Feature
Reporter: Pritam Damania
Assignee: Devaraj Das
Fix For: 2.0.5-beta
Attachments: hdfs-2576-1.txt, hdfs-2576-trunk-1.patch,
hdfs-2576-trunk-2.patch, hdfs-2576-trunk-7.1.patch, hdfs-2576-trunk-7.patch,
hdfs-2576-trunk-8.1.patch, hdfs-2576-trunk-8.2.patch, hdfs-2576-trunk-8.patch
Sometimes Clients like HBase are required to dynamically compute the
datanodes it wishes to place the blocks for a file for higher level of
locality. For this purpose there is a need of a way to give the Namenode a
hint in terms of a favoredNodes parameter about the locations where the
client wants to put each block. The proposed solution is a favored nodes
parameter in the addBlock() method and in the create() file method to enable
the clients to give the hints to the NameNode about the locations of each
replica of the block. Note that this would be just a hint and finally the
NameNode would look at disk usage, datanode load etc. and decide whether it
can respect the hints or not.
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HDFS-2576) Namenode should have a favored nodes hint to enable clients to have control over block placement.
[
https://issues.apache.org/jira/browse/HDFS-2576?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13639928#comment-13639928
]
Hadoop QA commented on HDFS-2576:
-
{color:red}-1 overall{color}. Here are the results of testing the latest
attachment
http://issues.apache.org/jira/secure/attachment/12580179/hdfs-2576-trunk-8.patch
against trunk revision .
{color:green}+1 @author{color}. The patch does not contain any @author
tags.
{color:green}+1 tests included{color}. The patch appears to include 5 new
or modified test files.
{color:green}+1 javac{color}. The applied patch does not increase the
total number of javac compiler warnings.
{color:green}+1 javadoc{color}. The javadoc tool did not generate any
warning messages.
{color:green}+1 eclipse:eclipse{color}. The patch built with
eclipse:eclipse.
{color:red}-1 findbugs{color}. The patch appears to introduce 1 new
Findbugs (version 1.3.9) warnings.
{color:green}+1 release audit{color}. The applied patch does not increase
the total number of release audit warnings.
{color:green}+1 core tests{color}. The patch passed unit tests in
hadoop-common-project/hadoop-common hadoop-hdfs-project/hadoop-hdfs.
{color:green}+1 contrib tests{color}. The patch passed contrib unit tests.
Test results:
https://builds.apache.org/job/PreCommit-HDFS-Build/4298//testReport/
Findbugs warnings:
https://builds.apache.org/job/PreCommit-HDFS-Build/4298//artifact/trunk/patchprocess/newPatchFindbugsWarningshadoop-hdfs.html
Console output: https://builds.apache.org/job/PreCommit-HDFS-Build/4298//console
This message is automatically generated.
Namenode should have a favored nodes hint to enable clients to have control
over block placement.
-
Key: HDFS-2576
URL: https://issues.apache.org/jira/browse/HDFS-2576
Project: Hadoop HDFS
Issue Type: New Feature
Reporter: Pritam Damania
Assignee: Devaraj Das
Fix For: 2.0.5-beta
Attachments: hdfs-2576-1.txt, hdfs-2576-trunk-1.patch,
hdfs-2576-trunk-2.patch, hdfs-2576-trunk-7.1.patch, hdfs-2576-trunk-7.patch,
hdfs-2576-trunk-8.patch
Sometimes Clients like HBase are required to dynamically compute the
datanodes it wishes to place the blocks for a file for higher level of
locality. For this purpose there is a need of a way to give the Namenode a
hint in terms of a favoredNodes parameter about the locations where the
client wants to put each block. The proposed solution is a favored nodes
parameter in the addBlock() method and in the create() file method to enable
the clients to give the hints to the NameNode about the locations of each
replica of the block. Note that this would be just a hint and finally the
NameNode would look at disk usage, datanode load etc. and decide whether it
can respect the hints or not.
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HDFS-2576) Namenode should have a favored nodes hint to enable clients to have control over block placement.
[
https://issues.apache.org/jira/browse/HDFS-2576?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13640050#comment-13640050
]
Hadoop QA commented on HDFS-2576:
-
{color:red}-1 overall{color}. Here are the results of testing the latest
attachment
http://issues.apache.org/jira/secure/attachment/12580212/hdfs-2576-trunk-8.1.patch
against trunk revision .
{color:green}+1 @author{color}. The patch does not contain any @author
tags.
{color:green}+1 tests included{color}. The patch appears to include 5 new
or modified test files.
{color:green}+1 javac{color}. The applied patch does not increase the
total number of javac compiler warnings.
{color:green}+1 javadoc{color}. The javadoc tool did not generate any
warning messages.
{color:green}+1 eclipse:eclipse{color}. The patch built with
eclipse:eclipse.
{color:green}+1 findbugs{color}. The patch does not introduce any new
Findbugs (version 1.3.9) warnings.
{color:green}+1 release audit{color}. The applied patch does not increase
the total number of release audit warnings.
{color:red}-1 core tests{color}. The patch failed these unit tests in
hadoop-common-project/hadoop-common hadoop-hdfs-project/hadoop-hdfs:
org.apache.hadoop.hdfs.server.blockmanagement.TestBlocksWithNotEnoughRacks
{color:green}+1 contrib tests{color}. The patch passed contrib unit tests.
Test results:
https://builds.apache.org/job/PreCommit-HDFS-Build/4302//testReport/
Console output: https://builds.apache.org/job/PreCommit-HDFS-Build/4302//console
This message is automatically generated.
Namenode should have a favored nodes hint to enable clients to have control
over block placement.
-
Key: HDFS-2576
URL: https://issues.apache.org/jira/browse/HDFS-2576
Project: Hadoop HDFS
Issue Type: New Feature
Reporter: Pritam Damania
Assignee: Devaraj Das
Fix For: 2.0.5-beta
Attachments: hdfs-2576-1.txt, hdfs-2576-trunk-1.patch,
hdfs-2576-trunk-2.patch, hdfs-2576-trunk-7.1.patch, hdfs-2576-trunk-7.patch,
hdfs-2576-trunk-8.1.patch, hdfs-2576-trunk-8.patch
Sometimes Clients like HBase are required to dynamically compute the
datanodes it wishes to place the blocks for a file for higher level of
locality. For this purpose there is a need of a way to give the Namenode a
hint in terms of a favoredNodes parameter about the locations where the
client wants to put each block. The proposed solution is a favored nodes
parameter in the addBlock() method and in the create() file method to enable
the clients to give the hints to the NameNode about the locations of each
replica of the block. Note that this would be just a hint and finally the
NameNode would look at disk usage, datanode load etc. and decide whether it
can respect the hints or not.
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HDFS-2576) Namenode should have a favored nodes hint to enable clients to have control over block placement.
[ https://issues.apache.org/jira/browse/HDFS-2576?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13640120#comment-13640120 ] Devaraj Das commented on HDFS-2576: --- Have seen TestBlocksWithNotEnoughRacks failing before. Example - https://builds.apache.org/job/PreCommit-HDFS-Build/4302/. Namenode should have a favored nodes hint to enable clients to have control over block placement. - Key: HDFS-2576 URL: https://issues.apache.org/jira/browse/HDFS-2576 Project: Hadoop HDFS Issue Type: New Feature Reporter: Pritam Damania Assignee: Devaraj Das Fix For: 2.0.5-beta Attachments: hdfs-2576-1.txt, hdfs-2576-trunk-1.patch, hdfs-2576-trunk-2.patch, hdfs-2576-trunk-7.1.patch, hdfs-2576-trunk-7.patch, hdfs-2576-trunk-8.1.patch, hdfs-2576-trunk-8.patch Sometimes Clients like HBase are required to dynamically compute the datanodes it wishes to place the blocks for a file for higher level of locality. For this purpose there is a need of a way to give the Namenode a hint in terms of a favoredNodes parameter about the locations where the client wants to put each block. The proposed solution is a favored nodes parameter in the addBlock() method and in the create() file method to enable the clients to give the hints to the NameNode about the locations of each replica of the block. Note that this would be just a hint and finally the NameNode would look at disk usage, datanode load etc. and decide whether it can respect the hints or not. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HDFS-2576) Namenode should have a favored nodes hint to enable clients to have control over block placement.
[
https://issues.apache.org/jira/browse/HDFS-2576?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13627567#comment-13627567
]
Hadoop QA commented on HDFS-2576:
-
{color:green}+1 overall{color}. Here are the results of testing the latest
attachment
http://issues.apache.org/jira/secure/attachment/12577959/hdfs-2576-trunk-7.1.patch
against trunk revision .
{color:green}+1 @author{color}. The patch does not contain any @author
tags.
{color:green}+1 tests included{color}. The patch appears to include 4 new
or modified test files.
{color:green}+1 javac{color}. The applied patch does not increase the
total number of javac compiler warnings.
{color:green}+1 javadoc{color}. The javadoc tool did not generate any
warning messages.
{color:green}+1 eclipse:eclipse{color}. The patch built with
eclipse:eclipse.
{color:green}+1 findbugs{color}. The patch does not introduce any new
Findbugs (version 1.3.9) warnings.
{color:green}+1 release audit{color}. The applied patch does not increase
the total number of release audit warnings.
{color:green}+1 core tests{color}. The patch passed unit tests in
hadoop-common-project/hadoop-common hadoop-hdfs-project/hadoop-hdfs.
{color:green}+1 contrib tests{color}. The patch passed contrib unit tests.
Test results:
https://builds.apache.org/job/PreCommit-HDFS-Build/4215//testReport/
Console output: https://builds.apache.org/job/PreCommit-HDFS-Build/4215//console
This message is automatically generated.
Namenode should have a favored nodes hint to enable clients to have control
over block placement.
-
Key: HDFS-2576
URL: https://issues.apache.org/jira/browse/HDFS-2576
Project: Hadoop HDFS
Issue Type: New Feature
Reporter: Pritam Damania
Assignee: Devaraj Das
Fix For: 2.0.5-beta
Attachments: hdfs-2576-1.txt, hdfs-2576-trunk-1.patch,
hdfs-2576-trunk-2.patch, hdfs-2576-trunk-7.1.patch, hdfs-2576-trunk-7.patch
Sometimes Clients like HBase are required to dynamically compute the
datanodes it wishes to place the blocks for a file for higher level of
locality. For this purpose there is a need of a way to give the Namenode a
hint in terms of a favoredNodes parameter about the locations where the
client wants to put each block. The proposed solution is a favored nodes
parameter in the addBlock() method and in the create() file method to enable
the clients to give the hints to the NameNode about the locations of each
replica of the block. Note that this would be just a hint and finally the
NameNode would look at disk usage, datanode load etc. and decide whether it
can respect the hints or not.
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HDFS-2576) Namenode should have a favored nodes hint to enable clients to have control over block placement.
[
https://issues.apache.org/jira/browse/HDFS-2576?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13627430#comment-13627430
]
Hadoop QA commented on HDFS-2576:
-
{color:red}-1 overall{color}. Here are the results of testing the latest
attachment
http://issues.apache.org/jira/secure/attachment/12577936/hdfs-2576-trunk-7.patch
against trunk revision .
{color:green}+1 @author{color}. The patch does not contain any @author
tags.
{color:green}+1 tests included{color}. The patch appears to include 4 new
or modified test files.
{color:green}+1 javac{color}. The applied patch does not increase the
total number of javac compiler warnings.
{color:green}+1 javadoc{color}. The javadoc tool did not generate any
warning messages.
{color:green}+1 eclipse:eclipse{color}. The patch built with
eclipse:eclipse.
{color:red}-1 findbugs{color}. The patch appears to introduce 1 new
Findbugs (version 1.3.9) warnings.
{color:green}+1 release audit{color}. The applied patch does not increase
the total number of release audit warnings.
{color:green}+1 core tests{color}. The patch passed unit tests in
hadoop-common-project/hadoop-common hadoop-hdfs-project/hadoop-hdfs.
{color:green}+1 contrib tests{color}. The patch passed contrib unit tests.
Test results:
https://builds.apache.org/job/PreCommit-HDFS-Build/4214//testReport/
Findbugs warnings:
https://builds.apache.org/job/PreCommit-HDFS-Build/4214//artifact/trunk/patchprocess/newPatchFindbugsWarningshadoop-hdfs.html
Console output: https://builds.apache.org/job/PreCommit-HDFS-Build/4214//console
This message is automatically generated.
Namenode should have a favored nodes hint to enable clients to have control
over block placement.
-
Key: HDFS-2576
URL: https://issues.apache.org/jira/browse/HDFS-2576
Project: Hadoop HDFS
Issue Type: New Feature
Reporter: Pritam Damania
Assignee: Devaraj Das
Fix For: 2.0.5-beta
Attachments: hdfs-2576-1.txt, hdfs-2576-trunk-1.patch,
hdfs-2576-trunk-2.patch, hdfs-2576-trunk-7.patch
Sometimes Clients like HBase are required to dynamically compute the
datanodes it wishes to place the blocks for a file for higher level of
locality. For this purpose there is a need of a way to give the Namenode a
hint in terms of a favoredNodes parameter about the locations where the
client wants to put each block. The proposed solution is a favored nodes
parameter in the addBlock() method and in the create() file method to enable
the clients to give the hints to the NameNode about the locations of each
replica of the block. Note that this would be just a hint and finally the
NameNode would look at disk usage, datanode load etc. and decide whether it
can respect the hints or not.
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HDFS-2576) Namenode should have a favored nodes hint to enable clients to have control over block placement.
[ https://issues.apache.org/jira/browse/HDFS-2576?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13597468#comment-13597468 ] Suresh Srinivas commented on HDFS-2576: --- A lot of discussion to catchup with. I will post a comment later when I get some time. Devaraj, should DistributedFileSystem include a new version of the create method, for HBase to use? Namenode should have a favored nodes hint to enable clients to have control over block placement. - Key: HDFS-2576 URL: https://issues.apache.org/jira/browse/HDFS-2576 Project: Hadoop HDFS Issue Type: New Feature Reporter: Pritam Damania Attachments: hdfs-2576-1.txt, hdfs-2576-trunk-1.patch Sometimes Clients like HBase are required to dynamically compute the datanodes it wishes to place the blocks for a file for higher level of locality. For this purpose there is a need of a way to give the Namenode a hint in terms of a favoredNodes parameter about the locations where the client wants to put each block. The proposed solution is a favored nodes parameter in the addBlock() method and in the create() file method to enable the clients to give the hints to the NameNode about the locations of each replica of the block. Note that this would be just a hint and finally the NameNode would look at disk usage, datanode load etc. and decide whether it can respect the hints or not. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HDFS-2576) Namenode should have a favored nodes hint to enable clients to have control over block placement.
[ https://issues.apache.org/jira/browse/HDFS-2576?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13597548#comment-13597548 ] Hari Mankude commented on HDFS-2576: Is data skew going to be an issue where some DNs are overloaded vs other DNs? Would this an issue when there is other data stored in hdfs along with hbase? Namenode should have a favored nodes hint to enable clients to have control over block placement. - Key: HDFS-2576 URL: https://issues.apache.org/jira/browse/HDFS-2576 Project: Hadoop HDFS Issue Type: New Feature Reporter: Pritam Damania Attachments: hdfs-2576-1.txt, hdfs-2576-trunk-1.patch Sometimes Clients like HBase are required to dynamically compute the datanodes it wishes to place the blocks for a file for higher level of locality. For this purpose there is a need of a way to give the Namenode a hint in terms of a favoredNodes parameter about the locations where the client wants to put each block. The proposed solution is a favored nodes parameter in the addBlock() method and in the create() file method to enable the clients to give the hints to the NameNode about the locations of each replica of the block. Note that this would be just a hint and finally the NameNode would look at disk usage, datanode load etc. and decide whether it can respect the hints or not. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HDFS-2576) Namenode should have a favored nodes hint to enable clients to have control over block placement.
[ https://issues.apache.org/jira/browse/HDFS-2576?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13591017#comment-13591017 ] Devaraj Das commented on HDFS-2576: --- Thanks to everyone who chimed in on this issue. I take it that there is an agreement that the proposed approach is fine (a good incremental step). I'll work on a trunk patch for this. Namenode should have a favored nodes hint to enable clients to have control over block placement. - Key: HDFS-2576 URL: https://issues.apache.org/jira/browse/HDFS-2576 Project: Hadoop HDFS Issue Type: New Feature Reporter: Pritam Damania Attachments: hdfs-2576-1.txt Sometimes Clients like HBase are required to dynamically compute the datanodes it wishes to place the blocks for a file for higher level of locality. For this purpose there is a need of a way to give the Namenode a hint in terms of a favoredNodes parameter about the locations where the client wants to put each block. The proposed solution is a favored nodes parameter in the addBlock() method and in the create() file method to enable the clients to give the hints to the NameNode about the locations of each replica of the block. Note that this would be just a hint and finally the NameNode would look at disk usage, datanode load etc. and decide whether it can respect the hints or not. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HDFS-2576) Namenode should have a favored nodes hint to enable clients to have control over block placement.
[ https://issues.apache.org/jira/browse/HDFS-2576?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13591028#comment-13591028 ] Jonathan Hsieh commented on HDFS-2576: -- yup, I agree. Namenode should have a favored nodes hint to enable clients to have control over block placement. - Key: HDFS-2576 URL: https://issues.apache.org/jira/browse/HDFS-2576 Project: Hadoop HDFS Issue Type: New Feature Reporter: Pritam Damania Attachments: hdfs-2576-1.txt Sometimes Clients like HBase are required to dynamically compute the datanodes it wishes to place the blocks for a file for higher level of locality. For this purpose there is a need of a way to give the Namenode a hint in terms of a favoredNodes parameter about the locations where the client wants to put each block. The proposed solution is a favored nodes parameter in the addBlock() method and in the create() file method to enable the clients to give the hints to the NameNode about the locations of each replica of the block. Note that this would be just a hint and finally the NameNode would look at disk usage, datanode load etc. and decide whether it can respect the hints or not. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HDFS-2576) Namenode should have a favored nodes hint to enable clients to have control over block placement.
[ https://issues.apache.org/jira/browse/HDFS-2576?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13588600#comment-13588600 ] Devaraj Das commented on HDFS-2576: --- Thanks [~stack] for the review, and sorry for the delay in getting back. I was busy with getting the hbase side in shape to use this feature.. On your comments: bq. Here we are doing resolve from the client's POV. It might come up w/ a different hostname than that which the NN has for the favored node (vagaries of the local resolve setup). Anything we can do to make it so we for sure have same name for the favored node as NN has? I can't see how we can easily handle this problem. But the good news is that the resolution problem shouldn't bite the HBase use case. In the HBase use case, the HBase Master (most likely co-located in the same cluster as the Namenode) does the favored nodes assignment for regions. The Master and the Namenode should see the same address/hostname... bq. I am unclear on the comment below. Does it mean local to the supplied favoredNode? Yes. Look at the implementation of chooseTarget that is invoked from there. Finally, a call to chooseLocalNode is made. This is done for all the favored nodes (for loop over the favored nodes). Namenode should have a favored nodes hint to enable clients to have control over block placement. - Key: HDFS-2576 URL: https://issues.apache.org/jira/browse/HDFS-2576 Project: Hadoop HDFS Issue Type: New Feature Reporter: Pritam Damania Attachments: hdfs-2576-1.txt Sometimes Clients like HBase are required to dynamically compute the datanodes it wishes to place the blocks for a file for higher level of locality. For this purpose there is a need of a way to give the Namenode a hint in terms of a favoredNodes parameter about the locations where the client wants to put each block. The proposed solution is a favored nodes parameter in the addBlock() method and in the create() file method to enable the clients to give the hints to the NameNode about the locations of each replica of the block. Note that this would be just a hint and finally the NameNode would look at disk usage, datanode load etc. and decide whether it can respect the hints or not. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HDFS-2576) Namenode should have a favored nodes hint to enable clients to have control over block placement.
[
https://issues.apache.org/jira/browse/HDFS-2576?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13579533#comment-13579533
]
stack commented on HDFS-2576:
-
Do you think this could cause us issue Devaraj:
{code}
+favoredNodeInfos[i] = new DatanodeInfo(new DatanodeID(
+favoredNodes[i].getAddress().getHostAddress() + : +
+favoredNodes[i].getPort()));
{code}
Here we are doing resolve from the client's POV. It might come up w/ a
different hostname than that which the NN has for the favored node (vagaries of
the local resolve setup). Anything we can do to make it so we for sure have
same name for the favored node as NN has?
nit: presize w/ replication count?
+ ListDatanodeDescriptor results = new ArrayListDatanodeDescriptor();
I am unclear on the comment below:
{code}
+// Choose a single node which is local to favoredNode.
+DatanodeDescriptor[] locations = replicator.chooseTarget(1,
+favoredNode, new ArrayListDatanodeDescriptor(),
+excludedNodes, blockSize);
{code}
Does it mean local to the supplied favoredNode? If so, why not Choose the
favored node. Or is it saying choose as one of the replicas, the node that is
making the request; i.e. enforcing a replica local to the location the
dfsclient is making the request from? If the latter, I have further comment
but will hold for clarification (and looking at code, it is the former it
seems).
Patch looks great DD.
Namenode should have a favored nodes hint to enable clients to have control
over block placement.
-
Key: HDFS-2576
URL: https://issues.apache.org/jira/browse/HDFS-2576
Project: Hadoop HDFS
Issue Type: New Feature
Reporter: Pritam Damania
Attachments: hdfs-2576-1.txt
Sometimes Clients like HBase are required to dynamically compute the
datanodes it wishes to place the blocks for a file for higher level of
locality. For this purpose there is a need of a way to give the Namenode a
hint in terms of a favoredNodes parameter about the locations where the
client wants to put each block. The proposed solution is a favored nodes
parameter in the addBlock() method and in the create() file method to enable
the clients to give the hints to the NameNode about the locations of each
replica of the block. Note that this would be just a hint and finally the
NameNode would look at disk usage, datanode load etc. and decide whether it
can respect the hints or not.
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators
For more information on JIRA, see: http://www.atlassian.com/software/jira
