[jira] [Commented] (HADOOP-6616) Improve documentation for rack awareness
[ https://issues.apache.org/jira/browse/HADOOP-6616?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14239681#comment-14239681 ] Allen Wittenauer commented on HADOOP-6616: -- It's more than unfortunate: it's downright hostile. > Improve documentation for rack awareness > > > Key: HADOOP-6616 > URL: https://issues.apache.org/jira/browse/HADOOP-6616 > Project: Hadoop Common > Issue Type: Improvement > Components: documentation >Reporter: Jeff Hammerbacher >Assignee: Adam Faris > Labels: newbie > Fix For: 3.0.0 > > Attachments: hadoop-6616.patch, hadoop-6616.patch.2, > hadoop-6616.patch.3, hadoop-6616.patch.4 > > > The current documentation for rack awareness > (http://hadoop.apache.org/common/docs/r0.20.0/cluster_setup.html#Hadoop+Rack+Awareness) > should be augmented to include a sample script. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (HADOOP-6616) Improve documentation for rack awareness
[ https://issues.apache.org/jira/browse/HADOOP-6616?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14239067#comment-14239067 ] Adam Faris commented on HADOOP-6616: It looks like all the topology information regarding rack awareness was removed as 'cruft' in HADOOP-8427, when an effort to convert forrest docs to APT. See patch numbered 5 and the diff shows everything related to rack awareness has been removed. Removal is unfortunate as the documentation is still relevant for current versions of Hadoop. > Improve documentation for rack awareness > > > Key: HADOOP-6616 > URL: https://issues.apache.org/jira/browse/HADOOP-6616 > Project: Hadoop Common > Issue Type: Improvement > Components: documentation >Reporter: Jeff Hammerbacher >Assignee: Adam Faris > Labels: newbie > Fix For: 3.0.0 > > Attachments: hadoop-6616.patch, hadoop-6616.patch.2, > hadoop-6616.patch.3, hadoop-6616.patch.4 > > > The current documentation for rack awareness > (http://hadoop.apache.org/common/docs/r0.20.0/cluster_setup.html#Hadoop+Rack+Awareness) > should be augmented to include a sample script. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (HADOOP-6616) Improve documentation for rack awareness
[ https://issues.apache.org/jira/browse/HADOOP-6616?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14238661#comment-14238661 ] Allen Wittenauer commented on HADOOP-6616: -- Somewhere along the way, this change got dropped. At least, I can't find a record of it in branch-2 or trunk. > Improve documentation for rack awareness > > > Key: HADOOP-6616 > URL: https://issues.apache.org/jira/browse/HADOOP-6616 > Project: Hadoop Common > Issue Type: Improvement > Components: documentation >Reporter: Jeff Hammerbacher >Assignee: Adam Faris > Labels: newbie > Fix For: 3.0.0 > > Attachments: hadoop-6616.patch, hadoop-6616.patch.2, > hadoop-6616.patch.3, hadoop-6616.patch.4 > > > The current documentation for rack awareness > (http://hadoop.apache.org/common/docs/r0.20.0/cluster_setup.html#Hadoop+Rack+Awareness) > should be augmented to include a sample script. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (HADOOP-6616) Improve documentation for rack awareness
[ https://issues.apache.org/jira/browse/HADOOP-6616?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13501177#comment-13501177 ] Hudson commented on HADOOP-6616: Integrated in Hadoop-Mapreduce-trunk #1263 (See [https://builds.apache.org/job/Hadoop-Mapreduce-trunk/1263/]) HADOOP-6616. Improve documentation for rack awareness. Contributed by Adam Faris. (Revision 1411359) Result = FAILURE jghoman : http://svn.apache.org/viewcvs.cgi/?root=Apache-SVN&view=rev&rev=1411359 Files : * /hadoop/common/trunk/hadoop-common-project/hadoop-common/CHANGES.txt * /hadoop/common/trunk/hadoop-common-project/hadoop-common/src/main/docs/src/documentation/content/xdocs/cluster_setup.xml > Improve documentation for rack awareness > > > Key: HADOOP-6616 > URL: https://issues.apache.org/jira/browse/HADOOP-6616 > Project: Hadoop Common > Issue Type: Improvement > Components: documentation >Reporter: Jeff Hammerbacher >Assignee: Adam Faris > Labels: newbie > Fix For: 3.0.0 > > Attachments: hadoop-6616.patch, hadoop-6616.patch.2, > hadoop-6616.patch.3, hadoop-6616.patch.4 > > > The current documentation for rack awareness > (http://hadoop.apache.org/common/docs/r0.20.0/cluster_setup.html#Hadoop+Rack+Awareness) > should be augmented to include a sample script. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HADOOP-6616) Improve documentation for rack awareness
[ https://issues.apache.org/jira/browse/HADOOP-6616?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13501158#comment-13501158 ] Hudson commented on HADOOP-6616: Integrated in Hadoop-Hdfs-trunk #1232 (See [https://builds.apache.org/job/Hadoop-Hdfs-trunk/1232/]) HADOOP-6616. Improve documentation for rack awareness. Contributed by Adam Faris. (Revision 1411359) Result = SUCCESS jghoman : http://svn.apache.org/viewcvs.cgi/?root=Apache-SVN&view=rev&rev=1411359 Files : * /hadoop/common/trunk/hadoop-common-project/hadoop-common/CHANGES.txt * /hadoop/common/trunk/hadoop-common-project/hadoop-common/src/main/docs/src/documentation/content/xdocs/cluster_setup.xml > Improve documentation for rack awareness > > > Key: HADOOP-6616 > URL: https://issues.apache.org/jira/browse/HADOOP-6616 > Project: Hadoop Common > Issue Type: Improvement > Components: documentation >Reporter: Jeff Hammerbacher >Assignee: Adam Faris > Labels: newbie > Fix For: 3.0.0 > > Attachments: hadoop-6616.patch, hadoop-6616.patch.2, > hadoop-6616.patch.3, hadoop-6616.patch.4 > > > The current documentation for rack awareness > (http://hadoop.apache.org/common/docs/r0.20.0/cluster_setup.html#Hadoop+Rack+Awareness) > should be augmented to include a sample script. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HADOOP-6616) Improve documentation for rack awareness
[ https://issues.apache.org/jira/browse/HADOOP-6616?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13500989#comment-13500989 ] Hudson commented on HADOOP-6616: Integrated in Hadoop-Yarn-trunk #42 (See [https://builds.apache.org/job/Hadoop-Yarn-trunk/42/]) HADOOP-6616. Improve documentation for rack awareness. Contributed by Adam Faris. (Revision 1411359) Result = SUCCESS jghoman : http://svn.apache.org/viewcvs.cgi/?root=Apache-SVN&view=rev&rev=1411359 Files : * /hadoop/common/trunk/hadoop-common-project/hadoop-common/CHANGES.txt * /hadoop/common/trunk/hadoop-common-project/hadoop-common/src/main/docs/src/documentation/content/xdocs/cluster_setup.xml > Improve documentation for rack awareness > > > Key: HADOOP-6616 > URL: https://issues.apache.org/jira/browse/HADOOP-6616 > Project: Hadoop Common > Issue Type: Improvement > Components: documentation >Reporter: Jeff Hammerbacher >Assignee: Adam Faris > Labels: newbie > Fix For: 3.0.0 > > Attachments: hadoop-6616.patch, hadoop-6616.patch.2, > hadoop-6616.patch.3, hadoop-6616.patch.4 > > > The current documentation for rack awareness > (http://hadoop.apache.org/common/docs/r0.20.0/cluster_setup.html#Hadoop+Rack+Awareness) > should be augmented to include a sample script. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HADOOP-6616) Improve documentation for rack awareness
[ https://issues.apache.org/jira/browse/HADOOP-6616?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13500546#comment-13500546 ] Hudson commented on HADOOP-6616: Integrated in Hadoop-trunk-Commit #3047 (See [https://builds.apache.org/job/Hadoop-trunk-Commit/3047/]) HADOOP-6616. Improve documentation for rack awareness. Contributed by Adam Faris. (Revision 1411359) Result = SUCCESS jghoman : http://svn.apache.org/viewcvs.cgi/?root=Apache-SVN&view=rev&rev=1411359 Files : * /hadoop/common/trunk/hadoop-common-project/hadoop-common/CHANGES.txt * /hadoop/common/trunk/hadoop-common-project/hadoop-common/src/main/docs/src/documentation/content/xdocs/cluster_setup.xml > Improve documentation for rack awareness > > > Key: HADOOP-6616 > URL: https://issues.apache.org/jira/browse/HADOOP-6616 > Project: Hadoop Common > Issue Type: Improvement > Components: documentation >Reporter: Jeff Hammerbacher >Assignee: Adam Faris > Labels: newbie > Fix For: 3.0.0 > > Attachments: hadoop-6616.patch, hadoop-6616.patch.2, > hadoop-6616.patch.3, hadoop-6616.patch.4 > > > The current documentation for rack awareness > (http://hadoop.apache.org/common/docs/r0.20.0/cluster_setup.html#Hadoop+Rack+Awareness) > should be augmented to include a sample script. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HADOOP-6616) Improve documentation for rack awareness
[ https://issues.apache.org/jira/browse/HADOOP-6616?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13500522#comment-13500522 ] Jakob Homan commented on HADOOP-6616: - +1. Looks good. Verified generated html (working around HADOOP-8810). Committing. > Improve documentation for rack awareness > > > Key: HADOOP-6616 > URL: https://issues.apache.org/jira/browse/HADOOP-6616 > Project: Hadoop Common > Issue Type: Improvement > Components: documentation >Reporter: Jeff Hammerbacher >Assignee: Adam Faris > Labels: newbie > Attachments: hadoop-6616.patch, hadoop-6616.patch.2, > hadoop-6616.patch.3, hadoop-6616.patch.4 > > > The current documentation for rack awareness > (http://hadoop.apache.org/common/docs/r0.20.0/cluster_setup.html#Hadoop+Rack+Awareness) > should be augmented to include a sample script. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HADOOP-6616) Improve documentation for rack awareness
[
https://issues.apache.org/jira/browse/HADOOP-6616?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13476547#comment-13476547
]
Hadoop QA commented on HADOOP-6616:
---
{color:green}+1 overall{color}. Here are the results of testing the latest
attachment
http://issues.apache.org/jira/secure/attachment/12549208/hadoop-6616.patch.4
against trunk revision .
{color:green}+1 @author{color}. The patch does not contain any @author
tags.
{color:green}+0 tests included{color}. The patch appears to be a
documentation patch that doesn't require tests.
{color:green}+1 javac{color}. The applied patch does not increase the
total number of javac compiler warnings.
{color:green}+1 javadoc{color}. The javadoc tool did not generate any
warning messages.
{color:green}+1 eclipse:eclipse{color}. The patch built with
eclipse:eclipse.
{color:green}+1 findbugs{color}. The patch does not introduce any new
Findbugs (version 1.3.9) warnings.
{color:green}+1 release audit{color}. The applied patch does not increase
the total number of release audit warnings.
{color:green}+1 core tests{color}. The patch passed unit tests in
hadoop-common-project/hadoop-common.
{color:green}+1 contrib tests{color}. The patch passed contrib unit tests.
Test results:
https://builds.apache.org/job/PreCommit-HADOOP-Build/1629//testReport/
Console output:
https://builds.apache.org/job/PreCommit-HADOOP-Build/1629//console
This message is automatically generated.
> Improve documentation for rack awareness
>
>
> Key: HADOOP-6616
> URL: https://issues.apache.org/jira/browse/HADOOP-6616
> Project: Hadoop Common
> Issue Type: Improvement
> Components: documentation
>Reporter: Jeff Hammerbacher
>Assignee: Adam Faris
> Labels: newbie
> Attachments: hadoop-6616.patch, hadoop-6616.patch.2,
> hadoop-6616.patch.3, hadoop-6616.patch.4
>
>
> The current documentation for rack awareness
> (http://hadoop.apache.org/common/docs/r0.20.0/cluster_setup.html#Hadoop+Rack+Awareness)
> should be augmented to include a sample script.
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HADOOP-6616) Improve documentation for rack awareness
[ https://issues.apache.org/jira/browse/HADOOP-6616?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13476259#comment-13476259 ] Joep Rottinghuis commented on HADOOP-6616: -- Adam, when you'er happy with the patch as it is (or with Jakob's last suggestion for the "it's - > its" type, you can attach the last patch, edit the jira and set the status to "patch available" to mark it to be ready to be committed. You probably also want to indicate which branch you think this applies to (you'll probably have noticed that the project structure has changed significantely between the 0.20/1.0 ; 0.21/0.22 ; and 0.23/0.24/2.0 families of branches). > Improve documentation for rack awareness > > > Key: HADOOP-6616 > URL: https://issues.apache.org/jira/browse/HADOOP-6616 > Project: Hadoop Common > Issue Type: Improvement > Components: documentation >Reporter: Jeff Hammerbacher >Assignee: Adam Faris > Labels: newbie > Attachments: hadoop-6616.patch, hadoop-6616.patch.2, > hadoop-6616.patch.3 > > > The current documentation for rack awareness > (http://hadoop.apache.org/common/docs/r0.20.0/cluster_setup.html#Hadoop+Rack+Awareness) > should be augmented to include a sample script. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HADOOP-6616) Improve documentation for rack awareness
[ https://issues.apache.org/jira/browse/HADOOP-6616?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13475502#comment-13475502 ] Joep Rottinghuis commented on HADOOP-6616: -- LGTM > Improve documentation for rack awareness > > > Key: HADOOP-6616 > URL: https://issues.apache.org/jira/browse/HADOOP-6616 > Project: Hadoop Common > Issue Type: Improvement > Components: documentation >Reporter: Jeff Hammerbacher >Assignee: Adam Faris > Labels: newbie > Attachments: hadoop-6616.patch, hadoop-6616.patch.2, > hadoop-6616.patch.3 > > > The current documentation for rack awareness > (http://hadoop.apache.org/common/docs/r0.20.0/cluster_setup.html#Hadoop+Rack+Awareness) > should be augmented to include a sample script. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HADOOP-6616) Improve documentation for rack awareness
[ https://issues.apache.org/jira/browse/HADOOP-6616?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13475467#comment-13475467 ] Jakob Homan commented on HADOOP-6616: - bq. # 1) each rack is it's own layer 3 network with a /24 subnet, which could be typical where each rack has it's own bq. # can create it's 'off-rack' block copy. s/it's/its/g Otherwise looks good and ready for commit. > Improve documentation for rack awareness > > > Key: HADOOP-6616 > URL: https://issues.apache.org/jira/browse/HADOOP-6616 > Project: Hadoop Common > Issue Type: Improvement > Components: documentation >Reporter: Jeff Hammerbacher >Assignee: Adam Faris > Labels: newbie > Attachments: hadoop-6616.patch, hadoop-6616.patch.2, > hadoop-6616.patch.3 > > > The current documentation for rack awareness > (http://hadoop.apache.org/common/docs/r0.20.0/cluster_setup.html#Hadoop+Rack+Awareness) > should be augmented to include a sample script. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HADOOP-6616) Improve documentation for rack awareness
[
https://issues.apache.org/jira/browse/HADOOP-6616?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13471805#comment-13471805
]
Joep Rottinghuis commented on HADOOP-6616:
--
Update looks good. I ran the perl and python example scripts. The first two ran
fine. See comment below on the BASH script.
Nit: (line 18 of the patch)
The NameNode and the JobTracker obtains
NN and JT obtain (singular).
I think this same error already exists in
http://hadoop.apache.org/docs/r1.0.3/cluster_setup.html#Hadoop+Rack+Awareness
One refinement to
{noformat}
The jobtracker uses rack awareness to reduce network transfers of HDFS data
blocks, as it will schedule tasks on nodes located within the same rack
containing the needed HDFS data blocks.
{noformat}
If the tasks cannot be scheduled on the DNs containing the needed HDFS blocks,
then the tasks will be scheduled on the same rack to reduce network transfers
if possible.
Line 41 again the NN and JT obtain instead of obtains (plural).
Line 65:
{noformat}
Hadoop will send multiple IP addresses on STDIN when forking the topology
script.
{noformat}
I think IP addresses are passed as arguments, not on STDIN. The first Perl
script reads this correctly from ARGV but the comment in the script reads that
it gets it from STDIN.
The BASH example for flat network always returns /rack-unkown
I think that is due to
{noformat}
if [ -n $# ];
{noformat}
Something like
{noformat}
if [ "$1" == "" ]; then
{noformat}
Once that is fixed, the script errors out missing closing brace in the for
statement. This should be:
{noformat}
for host in ${BASH_ARGV[*]}; do
{noformat}
I had some trouble with mis-matching single quote (which is strange as they
occurred only in the comments).
Same STDIN comment in other scripts.
I could not get the last python script (the one that makes assumptions about
the physical environment) to work because I do not have hosts that are called
"dn" something.
> Improve documentation for rack awareness
>
>
> Key: HADOOP-6616
> URL: https://issues.apache.org/jira/browse/HADOOP-6616
> Project: Hadoop Common
> Issue Type: Improvement
> Components: documentation
>Reporter: Jeff Hammerbacher
> Labels: newbie
> Attachments: hadoop-6616.patch, hadoop-6616.patch.2
>
>
> The current documentation for rack awareness
> (http://hadoop.apache.org/common/docs/r0.20.0/cluster_setup.html#Hadoop+Rack+Awareness)
> should be augmented to include a sample script.
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HADOOP-6616) Improve documentation for rack awareness
[ https://issues.apache.org/jira/browse/HADOOP-6616?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13470873#comment-13470873 ] Joep Rottinghuis commented on HADOOP-6616: -- Perhaps you can add a few words about topology.script.number.args which IIRC defaults to 100 and drives the number of host IPs passed to the script in one go, to allow the script to do some internal caching. When set to 1 then a process will be spawned to invoke the script for each host. You describe how the rack awareness works. Would it be useful to add a sentence or two about why this is used (ie. block placement uses rack awareness for fault tolerance)? > Improve documentation for rack awareness > > > Key: HADOOP-6616 > URL: https://issues.apache.org/jira/browse/HADOOP-6616 > Project: Hadoop Common > Issue Type: Improvement > Components: documentation >Reporter: Jeff Hammerbacher > Labels: newbie > Attachments: hadoop-6616.patch > > > The current documentation for rack awareness > (http://hadoop.apache.org/common/docs/r0.20.0/cluster_setup.html#Hadoop+Rack+Awareness) > should be augmented to include a sample script. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HADOOP-6616) Improve documentation for rack awareness
[
https://issues.apache.org/jira/browse/HADOOP-6616?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13470862#comment-13470862
]
Joep Rottinghuis commented on HADOOP-6616:
--
Nice Adam. Nit int the second-last paragraph:
{noformat}
If neither topology.script.file.name or
topology.script.file.name is
not set, the rack id '/default-rack' is returned for any passed IP address.
{noformat}
Neither not is a double negative.
You mention the property topology.script.file.name twice. Did you mean the
following?
{noformat}
If neither topology.script.file.name nor
topology.node.switch.mapping.impl is set, the rack id
'/default-rack' is returned for any passed IP address.
{noformat}
> Improve documentation for rack awareness
>
>
> Key: HADOOP-6616
> URL: https://issues.apache.org/jira/browse/HADOOP-6616
> Project: Hadoop Common
> Issue Type: Improvement
> Components: documentation
>Reporter: Jeff Hammerbacher
> Labels: newbie
> Attachments: hadoop-6616.patch
>
>
> The current documentation for rack awareness
> (http://hadoop.apache.org/common/docs/r0.20.0/cluster_setup.html#Hadoop+Rack+Awareness)
> should be augmented to include a sample script.
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators
For more information on JIRA, see: http://www.atlassian.com/software/jira
