[jira] [Commented] (MAPREDUCE-3664) HDFS Federation Documentation has incorrect configuration example
[ https://issues.apache.org/jira/browse/MAPREDUCE-3664?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13185155#comment-13185155 ] Jitendra Nath Pandey commented on MAPREDUCE-3664: - Committed. Thanks to Brandon Li! HDFS Federation Documentation has incorrect configuration example - Key: MAPREDUCE-3664 URL: https://issues.apache.org/jira/browse/MAPREDUCE-3664 Project: Hadoop Map/Reduce Issue Type: Bug Components: documentation Affects Versions: 0.23.0, 0.24.0 Reporter: praveen sripati Priority: Minor Attachments: HDFS-2778.txt, HDFS-2778.txt HDFS Federation documentation example (1) has the following property namedfs.namenode.rpc-address.ns1/name valuehdfs://nn-host1:rpc-port/value /property dfs.namenode.rpc-address.* should be set to hostname:port, hdfs:// should not be there. (1) - http://hadoop.apache.org/common/docs/r0.23.0/hadoop-yarn/hadoop-yarn-site/Federation.html -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (MAPREDUCE-3236) Distcp with hdfs:// passed with error in JT log while copying from .20.204 to .20.205 ( with useIp=false)
[ https://issues.apache.org/jira/browse/MAPREDUCE-3236?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13131973#comment-13131973 ] Jitendra Nath Pandey commented on MAPREDUCE-3236: - It is expected that JT can't renew tokens from a different cluster over hdfs:// (i.e. using rpc) if rpc ports are closed for outside cluster access. It is usually expected that http ports are open so it should work over hftp. It seems to me its not a bug. Daryn, Lets address the issue you pointed out in a separate jira. Distcp with hdfs:// passed with error in JT log while copying from .20.204 to .20.205 ( with useIp=false) -- Key: MAPREDUCE-3236 URL: https://issues.apache.org/jira/browse/MAPREDUCE-3236 Project: Hadoop Map/Reduce Issue Type: Bug Components: jobtracker, security Affects Versions: 0.20.205.0 Reporter: Rajit Saha Assignee: Daryn Sharp Attachments: HDFS-2447-1.patch, HDFS-2447.patch I tried to copy file from .20.204 to .20.205 by distcp over hdfs:// while using hadoop.security.token.service.use_ip=false in core-site.xml. The copy was successful but found error org.apache.hadoop.mapreduce.security.token.DelegationTokenRenewal: exception in .20.205 JT. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (MAPREDUCE-2764) Fix renewal of dfs delegation tokens
[ https://issues.apache.org/jira/browse/MAPREDUCE-2764?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13127798#comment-13127798 ] Jitendra Nath Pandey commented on MAPREDUCE-2764: - Created jira MAPREDUCE-3192 to fix the javadoc warning. Fix renewal of dfs delegation tokens Key: MAPREDUCE-2764 URL: https://issues.apache.org/jira/browse/MAPREDUCE-2764 Project: Hadoop Map/Reduce Issue Type: Bug Reporter: Daryn Sharp Assignee: Owen O'Malley Fix For: 0.20.205.0, 0.23.0, 0.24.0 Attachments: MAPREDUCE-2764-2.patch, MAPREDUCE-2764-3.patch, MAPREDUCE-2764-4.patch, MAPREDUCE-2764-trunk.patch, MAPREDUCE-2764-trunk.patch, MAPREDUCE-2764-trunk.patch, MAPREDUCE-2764.patch, delegation.patch, token-renew-trunk.patch, token-renew.patch, token-renew.patch The JT may have issues renewing hftp tokens which disrupt long distcp jobs. The problem is the JT's delegation token renewal code is built on brittle assumptions. The token's service field contains only the ip:port pair. The renewal process assumes that the scheme must be hdfs. If that fails due to a {{VersionMismatchException}}, it tries https based on another assumption that it must be hftp if it's not hdfs. A number of other exceptions, most commonly {{IOExceptions}}, can be generated which fouls up the renewal since it won't fallback to https. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (MAPREDUCE-2764) Fix renewal of dfs delegation tokens
[ https://issues.apache.org/jira/browse/MAPREDUCE-2764?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13128018#comment-13128018 ] Jitendra Nath Pandey commented on MAPREDUCE-2764: - Merged to 23 as well. Fix renewal of dfs delegation tokens Key: MAPREDUCE-2764 URL: https://issues.apache.org/jira/browse/MAPREDUCE-2764 Project: Hadoop Map/Reduce Issue Type: Bug Reporter: Daryn Sharp Assignee: Owen O'Malley Fix For: 0.20.205.0, 0.23.0, 0.24.0 Attachments: MAPREDUCE-2764-2.patch, MAPREDUCE-2764-3.patch, MAPREDUCE-2764-4.patch, MAPREDUCE-2764-trunk.patch, MAPREDUCE-2764-trunk.patch, MAPREDUCE-2764-trunk.patch, MAPREDUCE-2764.patch, delegation.patch, token-renew-trunk.patch, token-renew.patch, token-renew.patch The JT may have issues renewing hftp tokens which disrupt long distcp jobs. The problem is the JT's delegation token renewal code is built on brittle assumptions. The token's service field contains only the ip:port pair. The renewal process assumes that the scheme must be hdfs. If that fails due to a {{VersionMismatchException}}, it tries https based on another assumption that it must be hftp if it's not hdfs. A number of other exceptions, most commonly {{IOExceptions}}, can be generated which fouls up the renewal since it won't fallback to https. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (MAPREDUCE-2764) Fix renewal of dfs delegation tokens
[ https://issues.apache.org/jira/browse/MAPREDUCE-2764?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13126421#comment-13126421 ] Jitendra Nath Pandey commented on MAPREDUCE-2764: - test patch results: -1 overall. +1 @author. The patch does not contain any @author tags. +1 tests included. The patch appears to include 19 new or modified tests. +1 javadoc. The javadoc tool did not generate any warning messages. +1 javac. The applied patch does not increase the total number of javac compiler warnings. +1 findbugs. The patch does not introduce any new Findbugs (version ) warnings. -1 release audit. The applied patch generated 1 release audit warnings (more than the trunk's current 0 warnings). The release audit warning is related to Apache License header for hadoop-assemblies/src/main/resources/assemblies/hadoop-mapreduce-dist.xml. It is unrelated to this patch. Fix renewal of dfs delegation tokens Key: MAPREDUCE-2764 URL: https://issues.apache.org/jira/browse/MAPREDUCE-2764 Project: Hadoop Map/Reduce Issue Type: Bug Reporter: Daryn Sharp Assignee: Owen O'Malley Fix For: 0.20.205.0, 0.23.0 Attachments: MAPREDUCE-2764-2.patch, MAPREDUCE-2764-3.patch, MAPREDUCE-2764-4.patch, MAPREDUCE-2764-trunk.patch, MAPREDUCE-2764.patch, delegation.patch, token-renew-trunk.patch, token-renew.patch, token-renew.patch The JT may have issues renewing hftp tokens which disrupt long distcp jobs. The problem is the JT's delegation token renewal code is built on brittle assumptions. The token's service field contains only the ip:port pair. The renewal process assumes that the scheme must be hdfs. If that fails due to a {{VersionMismatchException}}, it tries https based on another assumption that it must be hftp if it's not hdfs. A number of other exceptions, most commonly {{IOExceptions}}, can be generated which fouls up the renewal since it won't fallback to https. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (MAPREDUCE-2764) Fix renewal of dfs delegation tokens
[ https://issues.apache.org/jira/browse/MAPREDUCE-2764?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13127121#comment-13127121 ] Jitendra Nath Pandey commented on MAPREDUCE-2764: - I manually ran tests for common, hdfs, yarn. Following tests failed TestFailoverProxy, TestJMXJsonServlet TestDistributedFileSystem TestRestartDFS TestDfsOverAvroRpc TestDeleteBlockPool TestNMWebServer They fail on trunk without this patch as well. (TestDistributedFileSystem fails with java.io.IOException: Unknown protocol: org.apache.hadoop.hdfs.protocol.ClientProtocol) Fix renewal of dfs delegation tokens Key: MAPREDUCE-2764 URL: https://issues.apache.org/jira/browse/MAPREDUCE-2764 Project: Hadoop Map/Reduce Issue Type: Bug Reporter: Daryn Sharp Assignee: Owen O'Malley Fix For: 0.20.205.0, 0.23.0 Attachments: MAPREDUCE-2764-2.patch, MAPREDUCE-2764-3.patch, MAPREDUCE-2764-4.patch, MAPREDUCE-2764-trunk.patch, MAPREDUCE-2764-trunk.patch, MAPREDUCE-2764.patch, delegation.patch, token-renew-trunk.patch, token-renew.patch, token-renew.patch The JT may have issues renewing hftp tokens which disrupt long distcp jobs. The problem is the JT's delegation token renewal code is built on brittle assumptions. The token's service field contains only the ip:port pair. The renewal process assumes that the scheme must be hdfs. If that fails due to a {{VersionMismatchException}}, it tries https based on another assumption that it must be hftp if it's not hdfs. A number of other exceptions, most commonly {{IOExceptions}}, can be generated which fouls up the renewal since it won't fallback to https. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (MAPREDUCE-3149) add a test to verify that buildDTAuthority works for cases with no authority.
[ https://issues.apache.org/jira/browse/MAPREDUCE-3149?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13125995#comment-13125995 ] Jitendra Nath Pandey commented on MAPREDUCE-3149: - +1 for the trunk patch. add a test to verify that buildDTAuthority works for cases with no authority. - Key: MAPREDUCE-3149 URL: https://issues.apache.org/jira/browse/MAPREDUCE-3149 Project: Hadoop Map/Reduce Issue Type: Bug Affects Versions: 0.24.0 Reporter: John George Assignee: John George Attachments: HADOOP-7602.patch Add a test to verify that buildDTAuthority works for cases with no Authority. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (MAPREDUCE-2764) Fix renewal of dfs delegation tokens
[ https://issues.apache.org/jira/browse/MAPREDUCE-2764?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13126263#comment-13126263 ] Jitendra Nath Pandey commented on MAPREDUCE-2764: - I agree with Daryn regarding the use_ip change. Apart from that, the trunk patch looks good to me. +1 Fix renewal of dfs delegation tokens Key: MAPREDUCE-2764 URL: https://issues.apache.org/jira/browse/MAPREDUCE-2764 Project: Hadoop Map/Reduce Issue Type: Bug Reporter: Daryn Sharp Assignee: Owen O'Malley Fix For: 0.20.205.0, 0.23.0 Attachments: MAPREDUCE-2764-2.patch, MAPREDUCE-2764-3.patch, MAPREDUCE-2764-4.patch, MAPREDUCE-2764.patch, delegation.patch, token-renew-trunk.patch, token-renew.patch, token-renew.patch The JT may have issues renewing hftp tokens which disrupt long distcp jobs. The problem is the JT's delegation token renewal code is built on brittle assumptions. The token's service field contains only the ip:port pair. The renewal process assumes that the scheme must be hdfs. If that fails due to a {{VersionMismatchException}}, it tries https based on another assumption that it must be hftp if it's not hdfs. A number of other exceptions, most commonly {{IOExceptions}}, can be generated which fouls up the renewal since it won't fallback to https. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa For more information on JIRA, see: http://www.atlassian.com/software/jira