[jira] [Commented] (MAPREDUCE-3011) TT should remove bad local dirs from conf to prevent constant disk checking
[ https://issues.apache.org/jira/browse/MAPREDUCE-3011?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13126387#comment-13126387 ] Ravi Gummadi commented on MAPREDUCE-3011: - LocalDirAllocator#AllocatorPerContext#confChanged() is actually updating the savedLocalDirs everytime conf is changed. So every call to confChanged() is not resulting into disk checks (except once per configuration change i.e. once per every bad disk). Right ? TT should remove bad local dirs from conf to prevent constant disk checking --- Key: MAPREDUCE-3011 URL: https://issues.apache.org/jira/browse/MAPREDUCE-3011 Project: Hadoop Map/Reduce Issue Type: Sub-task Components: tasktracker Affects Versions: 0.20.204.0 Reporter: Eli Collins Fix For: 0.20.205.0 Per HADOOP-7551 the TT does not remove bad mapred.local.dirs from the conf so after a single disk failure *every* call to get a local path for reading or writing results in a disk check of *all* configured local dirs. After detecting that a local dir is bad we should remove it from the conf so that we don't repeatedly perform this expensive operation. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (MAPREDUCE-2858) MRv2 WebApp Security
[ https://issues.apache.org/jira/browse/MAPREDUCE-2858?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13126395#comment-13126395 ] Allen Wittenauer commented on MAPREDUCE-2858: - What prevents me from setting up a web server on my desktop or (in Y! parlance) a gateway and hijacking the same creds? MRv2 WebApp Security Key: MAPREDUCE-2858 URL: https://issues.apache.org/jira/browse/MAPREDUCE-2858 Project: Hadoop Map/Reduce Issue Type: Sub-task Components: applicationmaster, mrv2, security Affects Versions: 0.23.0 Reporter: Luke Lu Assignee: Luke Lu Priority: Blocker Fix For: 0.23.0 In MRv2, while the system servers (ResourceManager (RM), NodeManager (NM) and NameNode (NN)) run as trusted system users, the application masters (AM) run as users who submit the application. While this offers great flexibility to run multiple version of mapreduce frameworks (including their UI) on the same Hadoop cluster, it has significant implication for the security of webapps (Please do not discuss company specific vulnerabilities here). Requirements: # Secure authentication for AM (for app/job level ACLs). # Webapp security should be optional via site configuration. # Support existing pluggable single sign on mechanisms. # Should not require per app/user configuration for deployment. # Should not require special site-wide DNS configuration for deployment. This the top jira for webapp security. A design doc/notes of threat-modeling and counter measures will be posted on the wiki. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Updated] (MAPREDUCE-3133) Running a set of methods in a Single Test Class
[ https://issues.apache.org/jira/browse/MAPREDUCE-3133?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Mahadev konar updated MAPREDUCE-3133: - Resolution: Fixed Hadoop Flags: Reviewed Status: Resolved (was: Patch Available) Just committed this. Thanks John. Running a set of methods in a Single Test Class --- Key: MAPREDUCE-3133 URL: https://issues.apache.org/jira/browse/MAPREDUCE-3133 Project: Hadoop Map/Reduce Issue Type: Improvement Components: build Affects Versions: 0.23.0 Reporter: Jonathan Eagles Assignee: Jonathan Eagles Attachments: MAPREDUCE-3132.patch, MAPREDUCE-3132.patch, MAPREDUCE-3132.patch Instead of running every test method in a class, limit to specific testing methods as describe in the link below. http://maven.apache.org/plugins/maven-surefire-plugin/examples/single-test.html Upgrade to the latest version of maven-surefire-plugin that has this feature. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Updated] (MAPREDUCE-3133) Running a set of methods in a Single Test Class
[ https://issues.apache.org/jira/browse/MAPREDUCE-3133?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Mahadev konar updated MAPREDUCE-3133: - Fix Version/s: 0.23.0 Running a set of methods in a Single Test Class --- Key: MAPREDUCE-3133 URL: https://issues.apache.org/jira/browse/MAPREDUCE-3133 Project: Hadoop Map/Reduce Issue Type: Improvement Components: build Affects Versions: 0.23.0 Reporter: Jonathan Eagles Assignee: Jonathan Eagles Fix For: 0.23.0 Attachments: MAPREDUCE-3132.patch, MAPREDUCE-3132.patch, MAPREDUCE-3132.patch Instead of running every test method in a class, limit to specific testing methods as describe in the link below. http://maven.apache.org/plugins/maven-surefire-plugin/examples/single-test.html Upgrade to the latest version of maven-surefire-plugin that has this feature. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (MAPREDUCE-3133) Running a set of methods in a Single Test Class
[ https://issues.apache.org/jira/browse/MAPREDUCE-3133?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13126399#comment-13126399 ] Hudson commented on MAPREDUCE-3133: --- Integrated in Hadoop-Common-trunk-Commit #1070 (See [https://builds.apache.org/job/Hadoop-Common-trunk-Commit/1070/]) MAPREDUCE-3133. Running a set of methods in a Single Test Class. (Jonathan Eagles via mahadev) mahadev : http://svn.apache.org/viewcvs.cgi/?root=Apache-SVNview=revrev=1182688 Files : * /hadoop/common/trunk/hadoop-mapreduce-project/CHANGES.txt * /hadoop/common/trunk/hadoop-mapreduce-project/hadoop-yarn/pom.xml * /hadoop/common/trunk/hadoop-mapreduce-project/pom.xml Running a set of methods in a Single Test Class --- Key: MAPREDUCE-3133 URL: https://issues.apache.org/jira/browse/MAPREDUCE-3133 Project: Hadoop Map/Reduce Issue Type: Improvement Components: build Affects Versions: 0.23.0 Reporter: Jonathan Eagles Assignee: Jonathan Eagles Fix For: 0.23.0 Attachments: MAPREDUCE-3132.patch, MAPREDUCE-3132.patch, MAPREDUCE-3132.patch Instead of running every test method in a class, limit to specific testing methods as describe in the link below. http://maven.apache.org/plugins/maven-surefire-plugin/examples/single-test.html Upgrade to the latest version of maven-surefire-plugin that has this feature. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (MAPREDUCE-3133) Running a set of methods in a Single Test Class
[ https://issues.apache.org/jira/browse/MAPREDUCE-3133?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13126400#comment-13126400 ] Hudson commented on MAPREDUCE-3133: --- Integrated in Hadoop-Hdfs-trunk-Commit #1148 (See [https://builds.apache.org/job/Hadoop-Hdfs-trunk-Commit/1148/]) MAPREDUCE-3133. Running a set of methods in a Single Test Class. (Jonathan Eagles via mahadev) mahadev : http://svn.apache.org/viewcvs.cgi/?root=Apache-SVNview=revrev=1182688 Files : * /hadoop/common/trunk/hadoop-mapreduce-project/CHANGES.txt * /hadoop/common/trunk/hadoop-mapreduce-project/hadoop-yarn/pom.xml * /hadoop/common/trunk/hadoop-mapreduce-project/pom.xml Running a set of methods in a Single Test Class --- Key: MAPREDUCE-3133 URL: https://issues.apache.org/jira/browse/MAPREDUCE-3133 Project: Hadoop Map/Reduce Issue Type: Improvement Components: build Affects Versions: 0.23.0 Reporter: Jonathan Eagles Assignee: Jonathan Eagles Fix For: 0.23.0 Attachments: MAPREDUCE-3132.patch, MAPREDUCE-3132.patch, MAPREDUCE-3132.patch Instead of running every test method in a class, limit to specific testing methods as describe in the link below. http://maven.apache.org/plugins/maven-surefire-plugin/examples/single-test.html Upgrade to the latest version of maven-surefire-plugin that has this feature. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (MAPREDUCE-2858) MRv2 WebApp Security
[ https://issues.apache.org/jira/browse/MAPREDUCE-2858?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13126406#comment-13126406 ] Luke Lu commented on MAPREDUCE-2858: bq. What prevents me from setting up a web server on my desktop or (in Y! parlance) a gateway and hijacking the same creds? The weirdness of asking people to visit your cool site? As I mentioned numerous time here, the main thrust for the proxy is the dramatically increased threat surface area (any host:port combination is potentially a valid webapp) due to the fact that untrusted webapps are linked from the RM app list, where admins/users browse applications. OTOH, there is actually another (deployment specific) technical reason that your desktop web server is less of an issue, which I can't talk about here :) MRv2 WebApp Security Key: MAPREDUCE-2858 URL: https://issues.apache.org/jira/browse/MAPREDUCE-2858 Project: Hadoop Map/Reduce Issue Type: Sub-task Components: applicationmaster, mrv2, security Affects Versions: 0.23.0 Reporter: Luke Lu Assignee: Luke Lu Priority: Blocker Fix For: 0.23.0 In MRv2, while the system servers (ResourceManager (RM), NodeManager (NM) and NameNode (NN)) run as trusted system users, the application masters (AM) run as users who submit the application. While this offers great flexibility to run multiple version of mapreduce frameworks (including their UI) on the same Hadoop cluster, it has significant implication for the security of webapps (Please do not discuss company specific vulnerabilities here). Requirements: # Secure authentication for AM (for app/job level ACLs). # Webapp security should be optional via site configuration. # Support existing pluggable single sign on mechanisms. # Should not require per app/user configuration for deployment. # Should not require special site-wide DNS configuration for deployment. This the top jira for webapp security. A design doc/notes of threat-modeling and counter measures will be posted on the wiki. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (MAPREDUCE-3133) Running a set of methods in a Single Test Class
[ https://issues.apache.org/jira/browse/MAPREDUCE-3133?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13126410#comment-13126410 ] Hudson commented on MAPREDUCE-3133: --- Integrated in Hadoop-Mapreduce-trunk-Commit #1090 (See [https://builds.apache.org/job/Hadoop-Mapreduce-trunk-Commit/1090/]) MAPREDUCE-3133. Running a set of methods in a Single Test Class. (Jonathan Eagles via mahadev) mahadev : http://svn.apache.org/viewcvs.cgi/?root=Apache-SVNview=revrev=1182688 Files : * /hadoop/common/trunk/hadoop-mapreduce-project/CHANGES.txt * /hadoop/common/trunk/hadoop-mapreduce-project/hadoop-yarn/pom.xml * /hadoop/common/trunk/hadoop-mapreduce-project/pom.xml Running a set of methods in a Single Test Class --- Key: MAPREDUCE-3133 URL: https://issues.apache.org/jira/browse/MAPREDUCE-3133 Project: Hadoop Map/Reduce Issue Type: Improvement Components: build Affects Versions: 0.23.0 Reporter: Jonathan Eagles Assignee: Jonathan Eagles Fix For: 0.23.0 Attachments: MAPREDUCE-3132.patch, MAPREDUCE-3132.patch, MAPREDUCE-3132.patch Instead of running every test method in a class, limit to specific testing methods as describe in the link below. http://maven.apache.org/plugins/maven-surefire-plugin/examples/single-test.html Upgrade to the latest version of maven-surefire-plugin that has this feature. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Updated] (MAPREDUCE-2764) Fix renewal of dfs delegation tokens
[
https://issues.apache.org/jira/browse/MAPREDUCE-2764?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Jitendra Nath Pandey updated MAPREDUCE-2764:
Attachment: MAPREDUCE-2764-trunk.patch
The attached patch addresses the us_ip concern and fixes a few javac warnings.
Fix renewal of dfs delegation tokens
Key: MAPREDUCE-2764
URL: https://issues.apache.org/jira/browse/MAPREDUCE-2764
Project: Hadoop Map/Reduce
Issue Type: Bug
Reporter: Daryn Sharp
Assignee: Owen O'Malley
Fix For: 0.20.205.0, 0.23.0
Attachments: MAPREDUCE-2764-2.patch, MAPREDUCE-2764-3.patch,
MAPREDUCE-2764-4.patch, MAPREDUCE-2764-trunk.patch, MAPREDUCE-2764.patch,
delegation.patch, token-renew-trunk.patch, token-renew.patch,
token-renew.patch
The JT may have issues renewing hftp tokens which disrupt long distcp jobs.
The problem is the JT's delegation token renewal code is built on brittle
assumptions. The token's service field contains only the ip:port pair.
The renewal process assumes that the scheme must be hdfs. If that fails due
to a {{VersionMismatchException}}, it tries https based on another assumption
that it must be hftp if it's not hdfs. A number of other exceptions, most
commonly {{IOExceptions}}, can be generated which fouls up the renewal since
it won't fallback to https.
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators:
https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (MAPREDUCE-2764) Fix renewal of dfs delegation tokens
[
https://issues.apache.org/jira/browse/MAPREDUCE-2764?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13126419#comment-13126419
]
Hadoop QA commented on MAPREDUCE-2764:
--
-1 overall. Here are the results of testing the latest attachment
http://issues.apache.org/jira/secure/attachment/12498858/MAPREDUCE-2764-trunk.patch
against trunk revision .
+1 @author. The patch does not contain any @author tags.
+1 tests included. The patch appears to include 19 new or modified tests.
-1 patch. The patch command could not apply the patch.
Console output:
https://builds.apache.org/job/PreCommit-MAPREDUCE-Build/1002//console
This message is automatically generated.
Fix renewal of dfs delegation tokens
Key: MAPREDUCE-2764
URL: https://issues.apache.org/jira/browse/MAPREDUCE-2764
Project: Hadoop Map/Reduce
Issue Type: Bug
Reporter: Daryn Sharp
Assignee: Owen O'Malley
Fix For: 0.20.205.0, 0.23.0
Attachments: MAPREDUCE-2764-2.patch, MAPREDUCE-2764-3.patch,
MAPREDUCE-2764-4.patch, MAPREDUCE-2764-trunk.patch, MAPREDUCE-2764.patch,
delegation.patch, token-renew-trunk.patch, token-renew.patch,
token-renew.patch
The JT may have issues renewing hftp tokens which disrupt long distcp jobs.
The problem is the JT's delegation token renewal code is built on brittle
assumptions. The token's service field contains only the ip:port pair.
The renewal process assumes that the scheme must be hdfs. If that fails due
to a {{VersionMismatchException}}, it tries https based on another assumption
that it must be hftp if it's not hdfs. A number of other exceptions, most
commonly {{IOExceptions}}, can be generated which fouls up the renewal since
it won't fallback to https.
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators:
https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (MAPREDUCE-2764) Fix renewal of dfs delegation tokens
[
https://issues.apache.org/jira/browse/MAPREDUCE-2764?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13126421#comment-13126421
]
Jitendra Nath Pandey commented on MAPREDUCE-2764:
-
test patch results:
-1 overall.
+1 @author. The patch does not contain any @author tags.
+1 tests included. The patch appears to include 19 new or modified tests.
+1 javadoc. The javadoc tool did not generate any warning messages.
+1 javac. The applied patch does not increase the total number of javac
compiler warnings.
+1 findbugs. The patch does not introduce any new Findbugs (version )
warnings.
-1 release audit. The applied patch generated 1 release audit warnings
(more than the trunk's current 0 warnings).
The release audit warning is related to Apache License header for
hadoop-assemblies/src/main/resources/assemblies/hadoop-mapreduce-dist.xml. It
is unrelated to this patch.
Fix renewal of dfs delegation tokens
Key: MAPREDUCE-2764
URL: https://issues.apache.org/jira/browse/MAPREDUCE-2764
Project: Hadoop Map/Reduce
Issue Type: Bug
Reporter: Daryn Sharp
Assignee: Owen O'Malley
Fix For: 0.20.205.0, 0.23.0
Attachments: MAPREDUCE-2764-2.patch, MAPREDUCE-2764-3.patch,
MAPREDUCE-2764-4.patch, MAPREDUCE-2764-trunk.patch, MAPREDUCE-2764.patch,
delegation.patch, token-renew-trunk.patch, token-renew.patch,
token-renew.patch
The JT may have issues renewing hftp tokens which disrupt long distcp jobs.
The problem is the JT's delegation token renewal code is built on brittle
assumptions. The token's service field contains only the ip:port pair.
The renewal process assumes that the scheme must be hdfs. If that fails due
to a {{VersionMismatchException}}, it tries https based on another assumption
that it must be hftp if it's not hdfs. A number of other exceptions, most
commonly {{IOExceptions}}, can be generated which fouls up the renewal since
it won't fallback to https.
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators:
https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Updated] (MAPREDUCE-2733) Gridmix v3 cpu emulation system tests.
[ https://issues.apache.org/jira/browse/MAPREDUCE-2733?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Vinay Kumar Thota updated MAPREDUCE-2733: - Attachment: MAPREDUCE-2733.v2.patch Uploaded new patch by addressing review comments from Amar. Gridmix v3 cpu emulation system tests. -- Key: MAPREDUCE-2733 URL: https://issues.apache.org/jira/browse/MAPREDUCE-2733 Project: Hadoop Map/Reduce Issue Type: Task Reporter: Vinay Kumar Thota Assignee: Vinay Kumar Thota Attachments: MAPREDUCE-2733.patch, MAPREDUCE-2733.v2.patch 1. Enable CPU emulation with default resource usage interval and run Gridmix v3 with a trace file that contains the CPU resource usage details. 2. Enable CPU emulation with custom resource usage interval and run Gridmix v3 with a trace file that contains the CPU resource usage details. 3. Disable CPU emulation and run Gridmix v3 with a trace file that contains the CPU resource usage details. 4. Enable CPU emulation with default resource usage interval and run Gridmix v3 with a trace file that doesn't contains the CPU resource usage details. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Updated] (MAPREDUCE-3009) RM UI - Applications - Application(Job History) - Map Tasks - Task ID - Node link is not working
[ https://issues.apache.org/jira/browse/MAPREDUCE-3009?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] chackaravarthy updated MAPREDUCE-3009: -- Attachment: MAPREDUCE-3009-1.patch RM UI - Applications - Application(Job History) - Map Tasks - Task ID - Node link is not working - Key: MAPREDUCE-3009 URL: https://issues.apache.org/jira/browse/MAPREDUCE-3009 Project: Hadoop Map/Reduce Issue Type: Bug Components: jobhistoryserver, mrv2 Affects Versions: 0.23.0, 0.24.0 Reporter: chackaravarthy Assignee: chackaravarthy Fix For: 0.24.0 Attachments: IE_node_link_not_working.png, MAPREDUCE-3009-1.patch, MAPREDUCE-3009.patch, chrome_node_link_working.png, invalid_url.png, invalid_url_latest_code.png RM UI - Applications - Application(Job History) - Map Tasks - Task ID - Node link is not working. The URL contains extra '/' which is causing the problem. Please find in the attached screen shots. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Updated] (MAPREDUCE-3009) RM UI - Applications - Application(Job History) - Map Tasks - Task ID - Node link is not working
[ https://issues.apache.org/jira/browse/MAPREDUCE-3009?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] chackaravarthy updated MAPREDUCE-3009: -- Status: Patch Available (was: Open) Attached the patch. pls review. Thanks. RM UI - Applications - Application(Job History) - Map Tasks - Task ID - Node link is not working - Key: MAPREDUCE-3009 URL: https://issues.apache.org/jira/browse/MAPREDUCE-3009 Project: Hadoop Map/Reduce Issue Type: Bug Components: jobhistoryserver, mrv2 Affects Versions: 0.23.0, 0.24.0 Reporter: chackaravarthy Assignee: chackaravarthy Fix For: 0.24.0 Attachments: IE_node_link_not_working.png, MAPREDUCE-3009-1.patch, MAPREDUCE-3009.patch, chrome_node_link_working.png, invalid_url.png, invalid_url_latest_code.png RM UI - Applications - Application(Job History) - Map Tasks - Task ID - Node link is not working. The URL contains extra '/' which is causing the problem. Please find in the attached screen shots. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (MAPREDUCE-3009) RM UI - Applications - Application(Job History) - Map Tasks - Task ID - Node link is not working
[ https://issues.apache.org/jira/browse/MAPREDUCE-3009?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13126454#comment-13126454 ] chackaravarthy commented on MAPREDUCE-3009: --- Hi Vinod, In Firefox and Chrome, even if we give URL as http:///www.google.com; , it will be taken as http://www.google.com; But in Internet Explorer, its been taken as same what the URL we are giving. ie. as http:///www.google.com; only. Hence the problem. Even though the URL being constructed in HsTaskPage as http:///hostname;, its not getting reflected in the browser. With the attached patch (MAPREDUCE-3009-1), its working fine in all the browsers. Thanks RM UI - Applications - Application(Job History) - Map Tasks - Task ID - Node link is not working - Key: MAPREDUCE-3009 URL: https://issues.apache.org/jira/browse/MAPREDUCE-3009 Project: Hadoop Map/Reduce Issue Type: Bug Components: jobhistoryserver, mrv2 Affects Versions: 0.23.0, 0.24.0 Reporter: chackaravarthy Assignee: chackaravarthy Fix For: 0.24.0 Attachments: IE_node_link_not_working.png, MAPREDUCE-3009-1.patch, MAPREDUCE-3009.patch, chrome_node_link_working.png, invalid_url.png, invalid_url_latest_code.png RM UI - Applications - Application(Job History) - Map Tasks - Task ID - Node link is not working. The URL contains extra '/' which is causing the problem. Please find in the attached screen shots. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Updated] (MAPREDUCE-2733) Gridmix v3 cpu emulation system tests.
[ https://issues.apache.org/jira/browse/MAPREDUCE-2733?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Vinay Kumar Thota updated MAPREDUCE-2733: - Attachment: MAPREDUCE-2733.v3.patch Addressed some more comments from Amar. Gridmix v3 cpu emulation system tests. -- Key: MAPREDUCE-2733 URL: https://issues.apache.org/jira/browse/MAPREDUCE-2733 Project: Hadoop Map/Reduce Issue Type: Task Reporter: Vinay Kumar Thota Assignee: Vinay Kumar Thota Attachments: MAPREDUCE-2733.patch, MAPREDUCE-2733.v2.patch, MAPREDUCE-2733.v3.patch 1. Enable CPU emulation with default resource usage interval and run Gridmix v3 with a trace file that contains the CPU resource usage details. 2. Enable CPU emulation with custom resource usage interval and run Gridmix v3 with a trace file that contains the CPU resource usage details. 3. Disable CPU emulation and run Gridmix v3 with a trace file that contains the CPU resource usage details. 4. Enable CPU emulation with default resource usage interval and run Gridmix v3 with a trace file that doesn't contains the CPU resource usage details. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (MAPREDUCE-3009) RM UI - Applications - Application(Job History) - Map Tasks - Task ID - Node link is not working
[ https://issues.apache.org/jira/browse/MAPREDUCE-3009?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13126460#comment-13126460 ] Hadoop QA commented on MAPREDUCE-3009: -- -1 overall. Here are the results of testing the latest attachment http://issues.apache.org/jira/secure/attachment/12498863/MAPREDUCE-3009-1.patch against trunk revision . +1 @author. The patch does not contain any @author tags. -1 tests included. The patch doesn't appear to include any new or modified tests. Please justify why no new tests are needed for this patch. Also please list what manual steps were performed to verify this patch. +1 javadoc. The javadoc tool did not generate any warning messages. +1 javac. The applied patch does not increase the total number of javac compiler warnings. +1 findbugs. The patch does not introduce any new Findbugs (version 1.3.9) warnings. +1 release audit. The applied patch does not increase the total number of release audit warnings. +1 core tests. The patch passed unit tests in . +1 contrib tests. The patch passed contrib unit tests. Test results: https://builds.apache.org/job/PreCommit-MAPREDUCE-Build/1003//testReport/ Console output: https://builds.apache.org/job/PreCommit-MAPREDUCE-Build/1003//console This message is automatically generated. RM UI - Applications - Application(Job History) - Map Tasks - Task ID - Node link is not working - Key: MAPREDUCE-3009 URL: https://issues.apache.org/jira/browse/MAPREDUCE-3009 Project: Hadoop Map/Reduce Issue Type: Bug Components: jobhistoryserver, mrv2 Affects Versions: 0.23.0, 0.24.0 Reporter: chackaravarthy Assignee: chackaravarthy Fix For: 0.24.0 Attachments: IE_node_link_not_working.png, MAPREDUCE-3009-1.patch, MAPREDUCE-3009.patch, chrome_node_link_working.png, invalid_url.png, invalid_url_latest_code.png RM UI - Applications - Application(Job History) - Map Tasks - Task ID - Node link is not working. The URL contains extra '/' which is causing the problem. Please find in the attached screen shots. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (MAPREDUCE-2733) Gridmix v3 cpu emulation system tests.
[ https://issues.apache.org/jira/browse/MAPREDUCE-2733?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13126462#comment-13126462 ] Amar Kamat commented on MAPREDUCE-2733: --- +1. Kindly share the output of test-patch, ant-tests and system tests. Gridmix v3 cpu emulation system tests. -- Key: MAPREDUCE-2733 URL: https://issues.apache.org/jira/browse/MAPREDUCE-2733 Project: Hadoop Map/Reduce Issue Type: Task Reporter: Vinay Kumar Thota Assignee: Vinay Kumar Thota Attachments: MAPREDUCE-2733.patch, MAPREDUCE-2733.v2.patch, MAPREDUCE-2733.v3.patch 1. Enable CPU emulation with default resource usage interval and run Gridmix v3 with a trace file that contains the CPU resource usage details. 2. Enable CPU emulation with custom resource usage interval and run Gridmix v3 with a trace file that contains the CPU resource usage details. 3. Disable CPU emulation and run Gridmix v3 with a trace file that contains the CPU resource usage details. 4. Enable CPU emulation with default resource usage interval and run Gridmix v3 with a trace file that doesn't contains the CPU resource usage details. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Updated] (MAPREDUCE-3175) Yarn httpservers not created with access Control lists
[ https://issues.apache.org/jira/browse/MAPREDUCE-3175?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Vinod Kumar Vavilapalli updated MAPREDUCE-3175: --- Issue Type: Sub-task (was: Bug) Parent: MAPREDUCE-3101 Yarn httpservers not created with access Control lists -- Key: MAPREDUCE-3175 URL: https://issues.apache.org/jira/browse/MAPREDUCE-3175 Project: Hadoop Map/Reduce Issue Type: Sub-task Components: mrv2 Affects Versions: 0.23.0 Reporter: Thomas Graves Priority: Blocker RM, NM, job history, and application master httpservers are not created with access Control lists. I believe this means that anyone can access any of the standard servlets that check to see if the user has administrator access - like /jmx, /stacks, etc and ops has no way to restrict access to these things. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (MAPREDUCE-3035) MR V2 jobhistory does not contain rack information
[
https://issues.apache.org/jira/browse/MAPREDUCE-3035?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13126471#comment-13126471
]
Amar Kamat commented on MAPREDUCE-3035:
---
W.r.t Rumen, {{LoggedTaskAttempt}} should now support {{setNodeName(RackName,
HostName)}} API. Also in {{JobBuilder}} (e.g. see line-532), calls to
{{LoggedTaskAttempt.setHostName()}} should be modified to
{{LoggedTaskAttempt.setNodeName(RackName, HostName)}}.
Hopefully this will help you guys get started.
MR V2 jobhistory does not contain rack information
--
Key: MAPREDUCE-3035
URL: https://issues.apache.org/jira/browse/MAPREDUCE-3035
Project: Hadoop Map/Reduce
Issue Type: Bug
Components: mrv2
Affects Versions: 0.23.0
Reporter: Karam Singh
Assignee: chackaravarthy
Priority: Blocker
Fix For: 0.23.0
Attachments: MAPREDUCE-3035.patch
When topology.node.switch.mapping.impl is set to enable rack-locality
resolution via the topology script, from the RM web-UI, we can see the rack
information for each node. Running a job also reveals the information about
rack-local map tasks launched at end of job completion on the client side.
But the hostname field for attempts in the JobHistory does not contain this
rack information.
In case of hadoop-0.20 securiy or MRV1, hostname field of job history does
contain rackid/hostname whereas in MRV2, hostname field only contains the
hostIP. Thus this is a regression.
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators:
https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Updated] (MAPREDUCE-3174) app master UI goes away when app finishes - not very user friendly
[ https://issues.apache.org/jira/browse/MAPREDUCE-3174?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Vinod Kumar Vavilapalli updated MAPREDUCE-3174: --- Priority: Blocker (was: Minor) Thomas, thanks for reporting. This is more than a minor annoyance, we need to fix this before the release. Bumping its status to a blocker. Regarding possible solutions, we can have a Http-proxy at RM like Todd mentioned or we could have a simple IFrame/JS in the AM web-UI which periodically checks the liveness of AM UI and redirects to JobHistory when AM exits. app master UI goes away when app finishes - not very user friendly -- Key: MAPREDUCE-3174 URL: https://issues.apache.org/jira/browse/MAPREDUCE-3174 Project: Hadoop Map/Reduce Issue Type: Improvement Components: mrv2 Affects Versions: 0.23.0 Reporter: Thomas Graves Priority: Blocker A user can go to the application master UI to see the stats on the app, but as soon as the app finishes that UI goes away and user is left with nothing. A redirect to history server or similar would be much better. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (MAPREDUCE-3035) MR V2 jobhistory does not contain rack information
[ https://issues.apache.org/jira/browse/MAPREDUCE-3035?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13126478#comment-13126478 ] chackaravarthy commented on MAPREDUCE-3035: --- yeah thanks for your inputs Amar :-) MR V2 jobhistory does not contain rack information -- Key: MAPREDUCE-3035 URL: https://issues.apache.org/jira/browse/MAPREDUCE-3035 Project: Hadoop Map/Reduce Issue Type: Bug Components: mrv2 Affects Versions: 0.23.0 Reporter: Karam Singh Assignee: chackaravarthy Priority: Blocker Fix For: 0.23.0 Attachments: MAPREDUCE-3035.patch When topology.node.switch.mapping.impl is set to enable rack-locality resolution via the topology script, from the RM web-UI, we can see the rack information for each node. Running a job also reveals the information about rack-local map tasks launched at end of job completion on the client side. But the hostname field for attempts in the JobHistory does not contain this rack information. In case of hadoop-0.20 securiy or MRV1, hostname field of job history does contain rackid/hostname whereas in MRV2, hostname field only contains the hostIP. Thus this is a regression. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (MAPREDUCE-3009) RM UI - Applications - Application(Job History) - Map Tasks - Task ID - Node link is not working
[ https://issues.apache.org/jira/browse/MAPREDUCE-3009?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13126479#comment-13126479 ] chackaravarthy commented on MAPREDUCE-3009: --- Test cases are not required because it is a trivial change RM UI - Applications - Application(Job History) - Map Tasks - Task ID - Node link is not working - Key: MAPREDUCE-3009 URL: https://issues.apache.org/jira/browse/MAPREDUCE-3009 Project: Hadoop Map/Reduce Issue Type: Bug Components: jobhistoryserver, mrv2 Affects Versions: 0.23.0, 0.24.0 Reporter: chackaravarthy Assignee: chackaravarthy Fix For: 0.24.0 Attachments: IE_node_link_not_working.png, MAPREDUCE-3009-1.patch, MAPREDUCE-3009.patch, chrome_node_link_working.png, invalid_url.png, invalid_url_latest_code.png RM UI - Applications - Application(Job History) - Map Tasks - Task ID - Node link is not working. The URL contains extra '/' which is causing the problem. Please find in the attached screen shots. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (MAPREDUCE-2789) [MR:279] Update the scheduling info on CLI
[ https://issues.apache.org/jira/browse/MAPREDUCE-2789?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13126531#comment-13126531 ] Hudson commented on MAPREDUCE-2789: --- Integrated in Hadoop-Hdfs-0.23-Build #38 (See [https://builds.apache.org/job/Hadoop-Hdfs-0.23-Build/38/]) Merge -c 1182616 from trunk to branch-0.23 to fix MAPREDUCE-2789. acmurthy : http://svn.apache.org/viewcvs.cgi/?root=Apache-SVNview=revrev=1182617 Files : * /hadoop/common/branches/branch-0.23/hadoop-mapreduce-project/CHANGES.txt * /hadoop/common/branches/branch-0.23/hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-common/src/main/java/org/apache/hadoop/mapreduce/TypeConverter.java * /hadoop/common/branches/branch-0.23/hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-common/src/test/java/org/apache/hadoop/mapreduce/TestTypeConverter.java * /hadoop/common/branches/branch-0.23/hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-core/src/main/java/org/apache/hadoop/mapreduce/JobStatus.java * /hadoop/common/branches/branch-0.23/hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-core/src/main/java/org/apache/hadoop/mapreduce/tools/CLI.java * /hadoop/common/branches/branch-0.23/hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-jobclient/src/main/java/org/apache/hadoop/mapred/NotRunningJob.java * /hadoop/common/branches/branch-0.23/hadoop-mapreduce-project/hadoop-yarn/hadoop-yarn-api/src/main/java/org/apache/hadoop/yarn/api/records/ApplicationReport.java * /hadoop/common/branches/branch-0.23/hadoop-mapreduce-project/hadoop-yarn/hadoop-yarn-api/src/main/java/org/apache/hadoop/yarn/api/records/ApplicationResourceUsageReport.java * /hadoop/common/branches/branch-0.23/hadoop-mapreduce-project/hadoop-yarn/hadoop-yarn-api/src/main/java/org/apache/hadoop/yarn/api/records/impl/pb/ApplicationReportPBImpl.java * /hadoop/common/branches/branch-0.23/hadoop-mapreduce-project/hadoop-yarn/hadoop-yarn-api/src/main/java/org/apache/hadoop/yarn/api/records/impl/pb/ApplicationResourceUsageReportPBImpl.java * /hadoop/common/branches/branch-0.23/hadoop-mapreduce-project/hadoop-yarn/hadoop-yarn-api/src/main/java/org/apache/hadoop/yarn/util/ProtoUtils.java * /hadoop/common/branches/branch-0.23/hadoop-mapreduce-project/hadoop-yarn/hadoop-yarn-api/src/main/proto/yarn_protos.proto * /hadoop/common/branches/branch-0.23/hadoop-mapreduce-project/hadoop-yarn/hadoop-yarn-common/src/main/java/org/apache/hadoop/yarn/util/BuilderUtils.java * /hadoop/common/branches/branch-0.23/hadoop-mapreduce-project/hadoop-yarn/hadoop-yarn-common/src/test/java/org/apache/hadoop/yarn/MockApps.java * /hadoop/common/branches/branch-0.23/hadoop-mapreduce-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager/src/main/java/org/apache/hadoop/yarn/server/resourcemanager/rmapp/RMAppImpl.java * /hadoop/common/branches/branch-0.23/hadoop-mapreduce-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager/src/main/java/org/apache/hadoop/yarn/server/resourcemanager/rmapp/attempt/RMAppAttempt.java * /hadoop/common/branches/branch-0.23/hadoop-mapreduce-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager/src/main/java/org/apache/hadoop/yarn/server/resourcemanager/rmapp/attempt/RMAppAttemptImpl.java [MR:279] Update the scheduling info on CLI -- Key: MAPREDUCE-2789 URL: https://issues.apache.org/jira/browse/MAPREDUCE-2789 Project: Hadoop Map/Reduce Issue Type: Bug Components: mrv2 Affects Versions: 0.23.0 Reporter: Ramya Sunil Assignee: Eric Payne Fix For: 0.23.0 Attachments: MAPREDUCE-2789.patch, MAPREDUCE-2789.v1.txt, MAPREDUCE-2789.v2.txt, MAPREDUCE-2789.v3.txt, MAPREDUCE-2789.v4.txt The scheduling information such as number of containers running, memory usage and reservations per job is not available on bin/mapred job -list CLI. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (MAPREDUCE-2666) MR-279: Need to retrieve shuffle port number on ApplicationMaster restart
[ https://issues.apache.org/jira/browse/MAPREDUCE-2666?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13126530#comment-13126530 ] Hudson commented on MAPREDUCE-2666: --- Integrated in Hadoop-Hdfs-0.23-Build #38 (See [https://builds.apache.org/job/Hadoop-Hdfs-0.23-Build/38/]) Merge -c 1182613 from trunk to branch-0.23 to fix MAPREDUCE-2666. acmurthy : http://svn.apache.org/viewcvs.cgi/?root=Apache-SVNview=revrev=1182614 Files : * /hadoop/common/branches/branch-0.23/hadoop-mapreduce-project/CHANGES.txt * /hadoop/common/branches/branch-0.23/hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-app/src/main/java/org/apache/hadoop/mapreduce/v2/app/job/impl/TaskAttemptImpl.java * /hadoop/common/branches/branch-0.23/hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-core/src/main/avro/Events.avpr * /hadoop/common/branches/branch-0.23/hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-core/src/main/java/org/apache/hadoop/mapreduce/jobhistory/JobHistoryParser.java * /hadoop/common/branches/branch-0.23/hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-core/src/main/java/org/apache/hadoop/mapreduce/jobhistory/TaskAttemptStartedEvent.java * /hadoop/common/branches/branch-0.23/hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-hs/src/test/java/org/apache/hadoop/mapreduce/v2/hs/TestJobHistoryParsing.java * /hadoop/common/branches/branch-0.23/hadoop-mapreduce-project/src/java/org/apache/hadoop/mapred/JobInProgress.java * /hadoop/common/branches/branch-0.23/hadoop-mapreduce-project/src/test/mapred/org/apache/hadoop/mapreduce/jobhistory/TestJobHistoryEvents.java * /hadoop/common/branches/branch-0.23/hadoop-mapreduce-project/src/tools/org/apache/hadoop/tools/rumen/TaskAttempt20LineEventEmitter.java MR-279: Need to retrieve shuffle port number on ApplicationMaster restart - Key: MAPREDUCE-2666 URL: https://issues.apache.org/jira/browse/MAPREDUCE-2666 Project: Hadoop Map/Reduce Issue Type: Sub-task Components: mrv2 Affects Versions: 0.23.0 Reporter: Robert Joseph Evans Assignee: Jonathan Eagles Priority: Blocker Fix For: 0.23.0 Attachments: MAPREDUCE-2666.patch, MAPREDUCE-2666.patch MAPREDUCE-2652 allows ShuffleHandler to return the port it is operating on. In the case of an ApplicationMaster crash where it needs to be restarted that information is lost. We either need to re-query it from each of the NodeManagers or to persist it to the JobHistory logs and retrieve it again. The job history logs is probably the simpler solution. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (MAPREDUCE-3133) Running a set of methods in a Single Test Class
[ https://issues.apache.org/jira/browse/MAPREDUCE-3133?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13126533#comment-13126533 ] Hudson commented on MAPREDUCE-3133: --- Integrated in Hadoop-Hdfs-0.23-Build #38 (See [https://builds.apache.org/job/Hadoop-Hdfs-0.23-Build/38/]) MAPREDUCE-3133. Running a set of methods in a Single Test Class. (Jonathan Eagles via mahadev) - Merging r1182688 from trunk mahadev : http://svn.apache.org/viewcvs.cgi/?root=Apache-SVNview=revrev=1182691 Files : * /hadoop/common/branches/branch-0.23/hadoop-mapreduce-project/CHANGES.txt * /hadoop/common/branches/branch-0.23/hadoop-mapreduce-project/hadoop-yarn/pom.xml * /hadoop/common/branches/branch-0.23/hadoop-mapreduce-project/pom.xml Running a set of methods in a Single Test Class --- Key: MAPREDUCE-3133 URL: https://issues.apache.org/jira/browse/MAPREDUCE-3133 Project: Hadoop Map/Reduce Issue Type: Improvement Components: build Affects Versions: 0.23.0 Reporter: Jonathan Eagles Assignee: Jonathan Eagles Fix For: 0.23.0 Attachments: MAPREDUCE-3132.patch, MAPREDUCE-3132.patch, MAPREDUCE-3132.patch Instead of running every test method in a class, limit to specific testing methods as describe in the link below. http://maven.apache.org/plugins/maven-surefire-plugin/examples/single-test.html Upgrade to the latest version of maven-surefire-plugin that has this feature. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (MAPREDUCE-2666) MR-279: Need to retrieve shuffle port number on ApplicationMaster restart
[ https://issues.apache.org/jira/browse/MAPREDUCE-2666?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13126547#comment-13126547 ] Hudson commented on MAPREDUCE-2666: --- Integrated in Hadoop-Mapreduce-0.23-Build #47 (See [https://builds.apache.org/job/Hadoop-Mapreduce-0.23-Build/47/]) Merge -c 1182613 from trunk to branch-0.23 to fix MAPREDUCE-2666. acmurthy : http://svn.apache.org/viewcvs.cgi/?root=Apache-SVNview=revrev=1182614 Files : * /hadoop/common/branches/branch-0.23/hadoop-mapreduce-project/CHANGES.txt * /hadoop/common/branches/branch-0.23/hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-app/src/main/java/org/apache/hadoop/mapreduce/v2/app/job/impl/TaskAttemptImpl.java * /hadoop/common/branches/branch-0.23/hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-core/src/main/avro/Events.avpr * /hadoop/common/branches/branch-0.23/hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-core/src/main/java/org/apache/hadoop/mapreduce/jobhistory/JobHistoryParser.java * /hadoop/common/branches/branch-0.23/hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-core/src/main/java/org/apache/hadoop/mapreduce/jobhistory/TaskAttemptStartedEvent.java * /hadoop/common/branches/branch-0.23/hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-hs/src/test/java/org/apache/hadoop/mapreduce/v2/hs/TestJobHistoryParsing.java * /hadoop/common/branches/branch-0.23/hadoop-mapreduce-project/src/java/org/apache/hadoop/mapred/JobInProgress.java * /hadoop/common/branches/branch-0.23/hadoop-mapreduce-project/src/test/mapred/org/apache/hadoop/mapreduce/jobhistory/TestJobHistoryEvents.java * /hadoop/common/branches/branch-0.23/hadoop-mapreduce-project/src/tools/org/apache/hadoop/tools/rumen/TaskAttempt20LineEventEmitter.java MR-279: Need to retrieve shuffle port number on ApplicationMaster restart - Key: MAPREDUCE-2666 URL: https://issues.apache.org/jira/browse/MAPREDUCE-2666 Project: Hadoop Map/Reduce Issue Type: Sub-task Components: mrv2 Affects Versions: 0.23.0 Reporter: Robert Joseph Evans Assignee: Jonathan Eagles Priority: Blocker Fix For: 0.23.0 Attachments: MAPREDUCE-2666.patch, MAPREDUCE-2666.patch MAPREDUCE-2652 allows ShuffleHandler to return the port it is operating on. In the case of an ApplicationMaster crash where it needs to be restarted that information is lost. We either need to re-query it from each of the NodeManagers or to persist it to the JobHistory logs and retrieve it again. The job history logs is probably the simpler solution. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (MAPREDUCE-3133) Running a set of methods in a Single Test Class
[ https://issues.apache.org/jira/browse/MAPREDUCE-3133?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13126550#comment-13126550 ] Hudson commented on MAPREDUCE-3133: --- Integrated in Hadoop-Mapreduce-0.23-Build #47 (See [https://builds.apache.org/job/Hadoop-Mapreduce-0.23-Build/47/]) MAPREDUCE-3133. Running a set of methods in a Single Test Class. (Jonathan Eagles via mahadev) - Merging r1182688 from trunk mahadev : http://svn.apache.org/viewcvs.cgi/?root=Apache-SVNview=revrev=1182691 Files : * /hadoop/common/branches/branch-0.23/hadoop-mapreduce-project/CHANGES.txt * /hadoop/common/branches/branch-0.23/hadoop-mapreduce-project/hadoop-yarn/pom.xml * /hadoop/common/branches/branch-0.23/hadoop-mapreduce-project/pom.xml Running a set of methods in a Single Test Class --- Key: MAPREDUCE-3133 URL: https://issues.apache.org/jira/browse/MAPREDUCE-3133 Project: Hadoop Map/Reduce Issue Type: Improvement Components: build Affects Versions: 0.23.0 Reporter: Jonathan Eagles Assignee: Jonathan Eagles Fix For: 0.23.0 Attachments: MAPREDUCE-3132.patch, MAPREDUCE-3132.patch, MAPREDUCE-3132.patch Instead of running every test method in a class, limit to specific testing methods as describe in the link below. http://maven.apache.org/plugins/maven-surefire-plugin/examples/single-test.html Upgrade to the latest version of maven-surefire-plugin that has this feature. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (MAPREDUCE-2789) [MR:279] Update the scheduling info on CLI
[ https://issues.apache.org/jira/browse/MAPREDUCE-2789?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13126548#comment-13126548 ] Hudson commented on MAPREDUCE-2789: --- Integrated in Hadoop-Mapreduce-0.23-Build #47 (See [https://builds.apache.org/job/Hadoop-Mapreduce-0.23-Build/47/]) Merge -c 1182616 from trunk to branch-0.23 to fix MAPREDUCE-2789. acmurthy : http://svn.apache.org/viewcvs.cgi/?root=Apache-SVNview=revrev=1182617 Files : * /hadoop/common/branches/branch-0.23/hadoop-mapreduce-project/CHANGES.txt * /hadoop/common/branches/branch-0.23/hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-common/src/main/java/org/apache/hadoop/mapreduce/TypeConverter.java * /hadoop/common/branches/branch-0.23/hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-common/src/test/java/org/apache/hadoop/mapreduce/TestTypeConverter.java * /hadoop/common/branches/branch-0.23/hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-core/src/main/java/org/apache/hadoop/mapreduce/JobStatus.java * /hadoop/common/branches/branch-0.23/hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-core/src/main/java/org/apache/hadoop/mapreduce/tools/CLI.java * /hadoop/common/branches/branch-0.23/hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-jobclient/src/main/java/org/apache/hadoop/mapred/NotRunningJob.java * /hadoop/common/branches/branch-0.23/hadoop-mapreduce-project/hadoop-yarn/hadoop-yarn-api/src/main/java/org/apache/hadoop/yarn/api/records/ApplicationReport.java * /hadoop/common/branches/branch-0.23/hadoop-mapreduce-project/hadoop-yarn/hadoop-yarn-api/src/main/java/org/apache/hadoop/yarn/api/records/ApplicationResourceUsageReport.java * /hadoop/common/branches/branch-0.23/hadoop-mapreduce-project/hadoop-yarn/hadoop-yarn-api/src/main/java/org/apache/hadoop/yarn/api/records/impl/pb/ApplicationReportPBImpl.java * /hadoop/common/branches/branch-0.23/hadoop-mapreduce-project/hadoop-yarn/hadoop-yarn-api/src/main/java/org/apache/hadoop/yarn/api/records/impl/pb/ApplicationResourceUsageReportPBImpl.java * /hadoop/common/branches/branch-0.23/hadoop-mapreduce-project/hadoop-yarn/hadoop-yarn-api/src/main/java/org/apache/hadoop/yarn/util/ProtoUtils.java * /hadoop/common/branches/branch-0.23/hadoop-mapreduce-project/hadoop-yarn/hadoop-yarn-api/src/main/proto/yarn_protos.proto * /hadoop/common/branches/branch-0.23/hadoop-mapreduce-project/hadoop-yarn/hadoop-yarn-common/src/main/java/org/apache/hadoop/yarn/util/BuilderUtils.java * /hadoop/common/branches/branch-0.23/hadoop-mapreduce-project/hadoop-yarn/hadoop-yarn-common/src/test/java/org/apache/hadoop/yarn/MockApps.java * /hadoop/common/branches/branch-0.23/hadoop-mapreduce-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager/src/main/java/org/apache/hadoop/yarn/server/resourcemanager/rmapp/RMAppImpl.java * /hadoop/common/branches/branch-0.23/hadoop-mapreduce-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager/src/main/java/org/apache/hadoop/yarn/server/resourcemanager/rmapp/attempt/RMAppAttempt.java * /hadoop/common/branches/branch-0.23/hadoop-mapreduce-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager/src/main/java/org/apache/hadoop/yarn/server/resourcemanager/rmapp/attempt/RMAppAttemptImpl.java [MR:279] Update the scheduling info on CLI -- Key: MAPREDUCE-2789 URL: https://issues.apache.org/jira/browse/MAPREDUCE-2789 Project: Hadoop Map/Reduce Issue Type: Bug Components: mrv2 Affects Versions: 0.23.0 Reporter: Ramya Sunil Assignee: Eric Payne Fix For: 0.23.0 Attachments: MAPREDUCE-2789.patch, MAPREDUCE-2789.v1.txt, MAPREDUCE-2789.v2.txt, MAPREDUCE-2789.v3.txt, MAPREDUCE-2789.v4.txt The scheduling information such as number of containers running, memory usage and reservations per job is not available on bin/mapred job -list CLI. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (MAPREDUCE-2666) MR-279: Need to retrieve shuffle port number on ApplicationMaster restart
[ https://issues.apache.org/jira/browse/MAPREDUCE-2666?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13126556#comment-13126556 ] Hudson commented on MAPREDUCE-2666: --- Integrated in Hadoop-Mapreduce-trunk #859 (See [https://builds.apache.org/job/Hadoop-Mapreduce-trunk/859/]) MAPREDUCE-2666. Retrieve shuffle port number from JobHistory on MR AM restart. Contributed by Jonathan Eagles. acmurthy : http://svn.apache.org/viewcvs.cgi/?root=Apache-SVNview=revrev=1182613 Files : * /hadoop/common/trunk/hadoop-mapreduce-project/CHANGES.txt * /hadoop/common/trunk/hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-app/src/main/java/org/apache/hadoop/mapreduce/v2/app/job/impl/TaskAttemptImpl.java * /hadoop/common/trunk/hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-core/src/main/avro/Events.avpr * /hadoop/common/trunk/hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-core/src/main/java/org/apache/hadoop/mapreduce/jobhistory/JobHistoryParser.java * /hadoop/common/trunk/hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-core/src/main/java/org/apache/hadoop/mapreduce/jobhistory/TaskAttemptStartedEvent.java * /hadoop/common/trunk/hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-hs/src/test/java/org/apache/hadoop/mapreduce/v2/hs/TestJobHistoryParsing.java * /hadoop/common/trunk/hadoop-mapreduce-project/src/java/org/apache/hadoop/mapred/JobInProgress.java * /hadoop/common/trunk/hadoop-mapreduce-project/src/test/mapred/org/apache/hadoop/mapreduce/jobhistory/TestJobHistoryEvents.java * /hadoop/common/trunk/hadoop-mapreduce-project/src/tools/org/apache/hadoop/tools/rumen/TaskAttempt20LineEventEmitter.java MR-279: Need to retrieve shuffle port number on ApplicationMaster restart - Key: MAPREDUCE-2666 URL: https://issues.apache.org/jira/browse/MAPREDUCE-2666 Project: Hadoop Map/Reduce Issue Type: Sub-task Components: mrv2 Affects Versions: 0.23.0 Reporter: Robert Joseph Evans Assignee: Jonathan Eagles Priority: Blocker Fix For: 0.23.0 Attachments: MAPREDUCE-2666.patch, MAPREDUCE-2666.patch MAPREDUCE-2652 allows ShuffleHandler to return the port it is operating on. In the case of an ApplicationMaster crash where it needs to be restarted that information is lost. We either need to re-query it from each of the NodeManagers or to persist it to the JobHistory logs and retrieve it again. The job history logs is probably the simpler solution. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (MAPREDUCE-2789) [MR:279] Update the scheduling info on CLI
[ https://issues.apache.org/jira/browse/MAPREDUCE-2789?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13126557#comment-13126557 ] Hudson commented on MAPREDUCE-2789: --- Integrated in Hadoop-Mapreduce-trunk #859 (See [https://builds.apache.org/job/Hadoop-Mapreduce-trunk/859/]) MAPREDUCE-2789. Complete schedulingInfo on CLI. Contributed by Eric Payne. acmurthy : http://svn.apache.org/viewcvs.cgi/?root=Apache-SVNview=revrev=1182616 Files : * /hadoop/common/trunk/hadoop-mapreduce-project/CHANGES.txt * /hadoop/common/trunk/hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-common/src/main/java/org/apache/hadoop/mapreduce/TypeConverter.java * /hadoop/common/trunk/hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-common/src/test/java/org/apache/hadoop/mapreduce/TestTypeConverter.java * /hadoop/common/trunk/hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-core/src/main/java/org/apache/hadoop/mapreduce/JobStatus.java * /hadoop/common/trunk/hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-core/src/main/java/org/apache/hadoop/mapreduce/tools/CLI.java * /hadoop/common/trunk/hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-jobclient/src/main/java/org/apache/hadoop/mapred/NotRunningJob.java * /hadoop/common/trunk/hadoop-mapreduce-project/hadoop-yarn/hadoop-yarn-api/src/main/java/org/apache/hadoop/yarn/api/records/ApplicationReport.java * /hadoop/common/trunk/hadoop-mapreduce-project/hadoop-yarn/hadoop-yarn-api/src/main/java/org/apache/hadoop/yarn/api/records/ApplicationResourceUsageReport.java * /hadoop/common/trunk/hadoop-mapreduce-project/hadoop-yarn/hadoop-yarn-api/src/main/java/org/apache/hadoop/yarn/api/records/impl/pb/ApplicationReportPBImpl.java * /hadoop/common/trunk/hadoop-mapreduce-project/hadoop-yarn/hadoop-yarn-api/src/main/java/org/apache/hadoop/yarn/api/records/impl/pb/ApplicationResourceUsageReportPBImpl.java * /hadoop/common/trunk/hadoop-mapreduce-project/hadoop-yarn/hadoop-yarn-api/src/main/java/org/apache/hadoop/yarn/util/ProtoUtils.java * /hadoop/common/trunk/hadoop-mapreduce-project/hadoop-yarn/hadoop-yarn-api/src/main/proto/yarn_protos.proto * /hadoop/common/trunk/hadoop-mapreduce-project/hadoop-yarn/hadoop-yarn-common/src/main/java/org/apache/hadoop/yarn/util/BuilderUtils.java * /hadoop/common/trunk/hadoop-mapreduce-project/hadoop-yarn/hadoop-yarn-common/src/test/java/org/apache/hadoop/yarn/MockApps.java * /hadoop/common/trunk/hadoop-mapreduce-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager/src/main/java/org/apache/hadoop/yarn/server/resourcemanager/rmapp/RMAppImpl.java * /hadoop/common/trunk/hadoop-mapreduce-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager/src/main/java/org/apache/hadoop/yarn/server/resourcemanager/rmapp/attempt/RMAppAttempt.java * /hadoop/common/trunk/hadoop-mapreduce-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager/src/main/java/org/apache/hadoop/yarn/server/resourcemanager/rmapp/attempt/RMAppAttemptImpl.java [MR:279] Update the scheduling info on CLI -- Key: MAPREDUCE-2789 URL: https://issues.apache.org/jira/browse/MAPREDUCE-2789 Project: Hadoop Map/Reduce Issue Type: Bug Components: mrv2 Affects Versions: 0.23.0 Reporter: Ramya Sunil Assignee: Eric Payne Fix For: 0.23.0 Attachments: MAPREDUCE-2789.patch, MAPREDUCE-2789.v1.txt, MAPREDUCE-2789.v2.txt, MAPREDUCE-2789.v3.txt, MAPREDUCE-2789.v4.txt The scheduling information such as number of containers running, memory usage and reservations per job is not available on bin/mapred job -list CLI. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (MAPREDUCE-3171) normalize nodemanager native code compilation with common/hdfs native
[ https://issues.apache.org/jira/browse/MAPREDUCE-3171?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13126558#comment-13126558 ] Hudson commented on MAPREDUCE-3171: --- Integrated in Hadoop-Mapreduce-trunk #859 (See [https://builds.apache.org/job/Hadoop-Mapreduce-trunk/859/]) MAPREDUCE-3171. normalize nodemanager native code compilation with common/hdfs native. (tucu) tucu : http://svn.apache.org/viewcvs.cgi/?root=Apache-SVNview=revrev=1182575 Files : * /hadoop/common/trunk/hadoop-assemblies/src/main/resources/assemblies/hadoop-mapreduce-dist.xml * /hadoop/common/trunk/hadoop-mapreduce-project/CHANGES.txt * /hadoop/common/trunk/hadoop-mapreduce-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-nodemanager/pom.xml * /hadoop/common/trunk/hadoop-mapreduce-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-nodemanager/src/main/c * /hadoop/common/trunk/hadoop-mapreduce-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-nodemanager/src/main/native * /hadoop/common/trunk/hadoop-mapreduce-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-nodemanager/src/main/native/container-executor/.gitignore normalize nodemanager native code compilation with common/hdfs native - Key: MAPREDUCE-3171 URL: https://issues.apache.org/jira/browse/MAPREDUCE-3171 Project: Hadoop Map/Reduce Issue Type: Improvement Components: build Affects Versions: 0.23.0, 0.24.0 Reporter: Alejandro Abdelnur Assignee: Alejandro Abdelnur Fix For: 0.23.0, 0.24.0 Attachments: MAPREDUCE-3171.patch, MAPREDUCE-3171.sh Use same build pattern as used by common/hdfs native: * rename src/c to src/native * run autoreconf, configure and make under target not to pollute the src tree * use maven-make-plugin in an identical way as in common/hdfs native -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (MAPREDUCE-3133) Running a set of methods in a Single Test Class
[ https://issues.apache.org/jira/browse/MAPREDUCE-3133?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13126560#comment-13126560 ] Hudson commented on MAPREDUCE-3133: --- Integrated in Hadoop-Mapreduce-trunk #859 (See [https://builds.apache.org/job/Hadoop-Mapreduce-trunk/859/]) MAPREDUCE-3133. Running a set of methods in a Single Test Class. (Jonathan Eagles via mahadev) mahadev : http://svn.apache.org/viewcvs.cgi/?root=Apache-SVNview=revrev=1182688 Files : * /hadoop/common/trunk/hadoop-mapreduce-project/CHANGES.txt * /hadoop/common/trunk/hadoop-mapreduce-project/hadoop-yarn/pom.xml * /hadoop/common/trunk/hadoop-mapreduce-project/pom.xml Running a set of methods in a Single Test Class --- Key: MAPREDUCE-3133 URL: https://issues.apache.org/jira/browse/MAPREDUCE-3133 Project: Hadoop Map/Reduce Issue Type: Improvement Components: build Affects Versions: 0.23.0 Reporter: Jonathan Eagles Assignee: Jonathan Eagles Fix For: 0.23.0 Attachments: MAPREDUCE-3132.patch, MAPREDUCE-3132.patch, MAPREDUCE-3132.patch Instead of running every test method in a class, limit to specific testing methods as describe in the link below. http://maven.apache.org/plugins/maven-surefire-plugin/examples/single-test.html Upgrade to the latest version of maven-surefire-plugin that has this feature. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (MAPREDUCE-2789) [MR:279] Update the scheduling info on CLI
[ https://issues.apache.org/jira/browse/MAPREDUCE-2789?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13126568#comment-13126568 ] Hudson commented on MAPREDUCE-2789: --- Integrated in Hadoop-Hdfs-trunk #829 (See [https://builds.apache.org/job/Hadoop-Hdfs-trunk/829/]) MAPREDUCE-2789. Complete schedulingInfo on CLI. Contributed by Eric Payne. acmurthy : http://svn.apache.org/viewcvs.cgi/?root=Apache-SVNview=revrev=1182616 Files : * /hadoop/common/trunk/hadoop-mapreduce-project/CHANGES.txt * /hadoop/common/trunk/hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-common/src/main/java/org/apache/hadoop/mapreduce/TypeConverter.java * /hadoop/common/trunk/hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-common/src/test/java/org/apache/hadoop/mapreduce/TestTypeConverter.java * /hadoop/common/trunk/hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-core/src/main/java/org/apache/hadoop/mapreduce/JobStatus.java * /hadoop/common/trunk/hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-core/src/main/java/org/apache/hadoop/mapreduce/tools/CLI.java * /hadoop/common/trunk/hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-jobclient/src/main/java/org/apache/hadoop/mapred/NotRunningJob.java * /hadoop/common/trunk/hadoop-mapreduce-project/hadoop-yarn/hadoop-yarn-api/src/main/java/org/apache/hadoop/yarn/api/records/ApplicationReport.java * /hadoop/common/trunk/hadoop-mapreduce-project/hadoop-yarn/hadoop-yarn-api/src/main/java/org/apache/hadoop/yarn/api/records/ApplicationResourceUsageReport.java * /hadoop/common/trunk/hadoop-mapreduce-project/hadoop-yarn/hadoop-yarn-api/src/main/java/org/apache/hadoop/yarn/api/records/impl/pb/ApplicationReportPBImpl.java * /hadoop/common/trunk/hadoop-mapreduce-project/hadoop-yarn/hadoop-yarn-api/src/main/java/org/apache/hadoop/yarn/api/records/impl/pb/ApplicationResourceUsageReportPBImpl.java * /hadoop/common/trunk/hadoop-mapreduce-project/hadoop-yarn/hadoop-yarn-api/src/main/java/org/apache/hadoop/yarn/util/ProtoUtils.java * /hadoop/common/trunk/hadoop-mapreduce-project/hadoop-yarn/hadoop-yarn-api/src/main/proto/yarn_protos.proto * /hadoop/common/trunk/hadoop-mapreduce-project/hadoop-yarn/hadoop-yarn-common/src/main/java/org/apache/hadoop/yarn/util/BuilderUtils.java * /hadoop/common/trunk/hadoop-mapreduce-project/hadoop-yarn/hadoop-yarn-common/src/test/java/org/apache/hadoop/yarn/MockApps.java * /hadoop/common/trunk/hadoop-mapreduce-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager/src/main/java/org/apache/hadoop/yarn/server/resourcemanager/rmapp/RMAppImpl.java * /hadoop/common/trunk/hadoop-mapreduce-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager/src/main/java/org/apache/hadoop/yarn/server/resourcemanager/rmapp/attempt/RMAppAttempt.java * /hadoop/common/trunk/hadoop-mapreduce-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager/src/main/java/org/apache/hadoop/yarn/server/resourcemanager/rmapp/attempt/RMAppAttemptImpl.java [MR:279] Update the scheduling info on CLI -- Key: MAPREDUCE-2789 URL: https://issues.apache.org/jira/browse/MAPREDUCE-2789 Project: Hadoop Map/Reduce Issue Type: Bug Components: mrv2 Affects Versions: 0.23.0 Reporter: Ramya Sunil Assignee: Eric Payne Fix For: 0.23.0 Attachments: MAPREDUCE-2789.patch, MAPREDUCE-2789.v1.txt, MAPREDUCE-2789.v2.txt, MAPREDUCE-2789.v3.txt, MAPREDUCE-2789.v4.txt The scheduling information such as number of containers running, memory usage and reservations per job is not available on bin/mapred job -list CLI. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (MAPREDUCE-2666) MR-279: Need to retrieve shuffle port number on ApplicationMaster restart
[ https://issues.apache.org/jira/browse/MAPREDUCE-2666?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13126567#comment-13126567 ] Hudson commented on MAPREDUCE-2666: --- Integrated in Hadoop-Hdfs-trunk #829 (See [https://builds.apache.org/job/Hadoop-Hdfs-trunk/829/]) MAPREDUCE-2666. Retrieve shuffle port number from JobHistory on MR AM restart. Contributed by Jonathan Eagles. acmurthy : http://svn.apache.org/viewcvs.cgi/?root=Apache-SVNview=revrev=1182613 Files : * /hadoop/common/trunk/hadoop-mapreduce-project/CHANGES.txt * /hadoop/common/trunk/hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-app/src/main/java/org/apache/hadoop/mapreduce/v2/app/job/impl/TaskAttemptImpl.java * /hadoop/common/trunk/hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-core/src/main/avro/Events.avpr * /hadoop/common/trunk/hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-core/src/main/java/org/apache/hadoop/mapreduce/jobhistory/JobHistoryParser.java * /hadoop/common/trunk/hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-core/src/main/java/org/apache/hadoop/mapreduce/jobhistory/TaskAttemptStartedEvent.java * /hadoop/common/trunk/hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-hs/src/test/java/org/apache/hadoop/mapreduce/v2/hs/TestJobHistoryParsing.java * /hadoop/common/trunk/hadoop-mapreduce-project/src/java/org/apache/hadoop/mapred/JobInProgress.java * /hadoop/common/trunk/hadoop-mapreduce-project/src/test/mapred/org/apache/hadoop/mapreduce/jobhistory/TestJobHistoryEvents.java * /hadoop/common/trunk/hadoop-mapreduce-project/src/tools/org/apache/hadoop/tools/rumen/TaskAttempt20LineEventEmitter.java MR-279: Need to retrieve shuffle port number on ApplicationMaster restart - Key: MAPREDUCE-2666 URL: https://issues.apache.org/jira/browse/MAPREDUCE-2666 Project: Hadoop Map/Reduce Issue Type: Sub-task Components: mrv2 Affects Versions: 0.23.0 Reporter: Robert Joseph Evans Assignee: Jonathan Eagles Priority: Blocker Fix For: 0.23.0 Attachments: MAPREDUCE-2666.patch, MAPREDUCE-2666.patch MAPREDUCE-2652 allows ShuffleHandler to return the port it is operating on. In the case of an ApplicationMaster crash where it needs to be restarted that information is lost. We either need to re-query it from each of the NodeManagers or to persist it to the JobHistory logs and retrieve it again. The job history logs is probably the simpler solution. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (MAPREDUCE-3166) Make Rumen use job history api instead of relying on current history file name format
[ https://issues.apache.org/jira/browse/MAPREDUCE-3166?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13126565#comment-13126565 ] Hudson commented on MAPREDUCE-3166: --- Integrated in Hadoop-Hdfs-trunk #829 (See [https://builds.apache.org/job/Hadoop-Hdfs-trunk/829/]) MAPREDUCE-3166. [Rumen] Make Rumen use job history api instead of relying on current history file name format. ravigummadi : http://svn.apache.org/viewcvs.cgi/?root=Apache-SVNview=revrev=1182330 Files : * /hadoop/common/trunk/hadoop-mapreduce-project/CHANGES.txt * /hadoop/common/trunk/hadoop-mapreduce-project/ivy.xml * /hadoop/common/trunk/hadoop-mapreduce-project/src/test/mapred/org/apache/hadoop/tools/rumen/TestRumenJobTraces.java * /hadoop/common/trunk/hadoop-mapreduce-project/src/tools/org/apache/hadoop/tools/rumen/JobHistoryUtils.java Make Rumen use job history api instead of relying on current history file name format - Key: MAPREDUCE-3166 URL: https://issues.apache.org/jira/browse/MAPREDUCE-3166 Project: Hadoop Map/Reduce Issue Type: Bug Components: tools/rumen Affects Versions: 0.23.0 Reporter: Ravi Gummadi Assignee: Ravi Gummadi Fix For: 0.23.0 Attachments: MR3166.patch Rumen should not depend on the regular expression of job history file name format and should use the newly added api like isValidJobHistoryFileName(), getJobIDFromHistoryFilePath(). -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (MAPREDUCE-3171) normalize nodemanager native code compilation with common/hdfs native
[ https://issues.apache.org/jira/browse/MAPREDUCE-3171?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13126569#comment-13126569 ] Hudson commented on MAPREDUCE-3171: --- Integrated in Hadoop-Hdfs-trunk #829 (See [https://builds.apache.org/job/Hadoop-Hdfs-trunk/829/]) MAPREDUCE-3171. normalize nodemanager native code compilation with common/hdfs native. (tucu) tucu : http://svn.apache.org/viewcvs.cgi/?root=Apache-SVNview=revrev=1182575 Files : * /hadoop/common/trunk/hadoop-assemblies/src/main/resources/assemblies/hadoop-mapreduce-dist.xml * /hadoop/common/trunk/hadoop-mapreduce-project/CHANGES.txt * /hadoop/common/trunk/hadoop-mapreduce-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-nodemanager/pom.xml * /hadoop/common/trunk/hadoop-mapreduce-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-nodemanager/src/main/c * /hadoop/common/trunk/hadoop-mapreduce-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-nodemanager/src/main/native * /hadoop/common/trunk/hadoop-mapreduce-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-nodemanager/src/main/native/container-executor/.gitignore normalize nodemanager native code compilation with common/hdfs native - Key: MAPREDUCE-3171 URL: https://issues.apache.org/jira/browse/MAPREDUCE-3171 Project: Hadoop Map/Reduce Issue Type: Improvement Components: build Affects Versions: 0.23.0, 0.24.0 Reporter: Alejandro Abdelnur Assignee: Alejandro Abdelnur Fix For: 0.23.0, 0.24.0 Attachments: MAPREDUCE-3171.patch, MAPREDUCE-3171.sh Use same build pattern as used by common/hdfs native: * rename src/c to src/native * run autoreconf, configure and make under target not to pollute the src tree * use maven-make-plugin in an identical way as in common/hdfs native -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (MAPREDUCE-3133) Running a set of methods in a Single Test Class
[ https://issues.apache.org/jira/browse/MAPREDUCE-3133?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13126571#comment-13126571 ] Hudson commented on MAPREDUCE-3133: --- Integrated in Hadoop-Hdfs-trunk #829 (See [https://builds.apache.org/job/Hadoop-Hdfs-trunk/829/]) MAPREDUCE-3133. Running a set of methods in a Single Test Class. (Jonathan Eagles via mahadev) mahadev : http://svn.apache.org/viewcvs.cgi/?root=Apache-SVNview=revrev=1182688 Files : * /hadoop/common/trunk/hadoop-mapreduce-project/CHANGES.txt * /hadoop/common/trunk/hadoop-mapreduce-project/hadoop-yarn/pom.xml * /hadoop/common/trunk/hadoop-mapreduce-project/pom.xml Running a set of methods in a Single Test Class --- Key: MAPREDUCE-3133 URL: https://issues.apache.org/jira/browse/MAPREDUCE-3133 Project: Hadoop Map/Reduce Issue Type: Improvement Components: build Affects Versions: 0.23.0 Reporter: Jonathan Eagles Assignee: Jonathan Eagles Fix For: 0.23.0 Attachments: MAPREDUCE-3132.patch, MAPREDUCE-3132.patch, MAPREDUCE-3132.patch Instead of running every test method in a class, limit to specific testing methods as describe in the link below. http://maven.apache.org/plugins/maven-surefire-plugin/examples/single-test.html Upgrade to the latest version of maven-surefire-plugin that has this feature. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (MAPREDUCE-2858) MRv2 WebApp Security
[ https://issues.apache.org/jira/browse/MAPREDUCE-2858?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13126588#comment-13126588 ] Allen Wittenauer commented on MAPREDUCE-2858: - OK, I guess I need to put my whole thought process here. The web app linked to from the RM could trivially redirect the user to an unknown/untrusted source. (There are countless ways to do that.) That location could then grab whatever creds are trying to be stolen. Worse yet, IIRC, the Hadoop token system uses addressless/forwardable tokens... MRv2 WebApp Security Key: MAPREDUCE-2858 URL: https://issues.apache.org/jira/browse/MAPREDUCE-2858 Project: Hadoop Map/Reduce Issue Type: Sub-task Components: applicationmaster, mrv2, security Affects Versions: 0.23.0 Reporter: Luke Lu Assignee: Luke Lu Priority: Blocker Fix For: 0.23.0 In MRv2, while the system servers (ResourceManager (RM), NodeManager (NM) and NameNode (NN)) run as trusted system users, the application masters (AM) run as users who submit the application. While this offers great flexibility to run multiple version of mapreduce frameworks (including their UI) on the same Hadoop cluster, it has significant implication for the security of webapps (Please do not discuss company specific vulnerabilities here). Requirements: # Secure authentication for AM (for app/job level ACLs). # Webapp security should be optional via site configuration. # Support existing pluggable single sign on mechanisms. # Should not require per app/user configuration for deployment. # Should not require special site-wide DNS configuration for deployment. This the top jira for webapp security. A design doc/notes of threat-modeling and counter measures will be posted on the wiki. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Updated] (MAPREDUCE-2775) [MR-279] Decommissioned node does not shutdown
[ https://issues.apache.org/jira/browse/MAPREDUCE-2775?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Devaraj K updated MAPREDUCE-2775: - Attachment: MAPREDUCE-2775-1.patch [MR-279] Decommissioned node does not shutdown -- Key: MAPREDUCE-2775 URL: https://issues.apache.org/jira/browse/MAPREDUCE-2775 Project: Hadoop Map/Reduce Issue Type: Bug Components: mrv2 Affects Versions: 0.23.0 Reporter: Ramya Sunil Assignee: Devaraj K Priority: Blocker Fix For: 0.23.0 Attachments: MAPREDUCE-2775-1.patch, MAPREDUCE-2775.patch A Nodemanager which is decommissioned by an admin via refreshnodes does not automatically shutdown. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Updated] (MAPREDUCE-2775) [MR-279] Decommissioned node does not shutdown
[ https://issues.apache.org/jira/browse/MAPREDUCE-2775?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Devaraj K updated MAPREDUCE-2775: - Status: Patch Available (was: Open) Thanks Vinod for looking into the patch. I have updated the patch by incorporating the changes for the above comments. [MR-279] Decommissioned node does not shutdown -- Key: MAPREDUCE-2775 URL: https://issues.apache.org/jira/browse/MAPREDUCE-2775 Project: Hadoop Map/Reduce Issue Type: Bug Components: mrv2 Affects Versions: 0.23.0 Reporter: Ramya Sunil Assignee: Devaraj K Priority: Blocker Fix For: 0.23.0 Attachments: MAPREDUCE-2775-1.patch, MAPREDUCE-2775.patch A Nodemanager which is decommissioned by an admin via refreshnodes does not automatically shutdown. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (MAPREDUCE-2747) [MR-279] [Security] Cleanup LinuxContainerExecutor binary sources
[ https://issues.apache.org/jira/browse/MAPREDUCE-2747?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13126601#comment-13126601 ] Vinod Kumar Vavilapalli commented on MAPREDUCE-2747: I wanted to look at this carefully, but couldn't find any time today. I am also out tomorrow, so will review this the day after tomorrow. Thanks. [MR-279] [Security] Cleanup LinuxContainerExecutor binary sources - Key: MAPREDUCE-2747 URL: https://issues.apache.org/jira/browse/MAPREDUCE-2747 Project: Hadoop Map/Reduce Issue Type: Sub-task Components: mrv2, nodemanager, security Affects Versions: 0.23.0, 0.24.0 Reporter: Vinod Kumar Vavilapalli Assignee: Robert Joseph Evans Priority: Blocker Fix For: 0.23.0, 0.24.0 Attachments: MAPREDUCE-2747.patch, MR-2747.txt There are a lot of references to the old task-controller nomenclature still, job/task refs instead of app/container. Also the configuration file is named as taskcontroller.cfg and the configured variables are also from the mapred world (mrv1). These SHOULD be fixed before we make a release. Marking this as blocker. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Updated] (MAPREDUCE-3009) RM UI - Applications - Application(Job History) - Map Tasks - Task ID - Node link is not working
[
https://issues.apache.org/jira/browse/MAPREDUCE-3009?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Vinod Kumar Vavilapalli updated MAPREDUCE-3009:
---
Status: Open (was: Patch Available)
I wanted to say I am amused that someone is using IE :P But I kind of started
liking your screen-shots, so holding that back :)
Anyways, thanks for the analysis. But can we fix the url method itself so that
{{url(http://;, address)}} returns a correct URL? Thanks.
RM UI - Applications - Application(Job History) - Map Tasks - Task ID -
Node link is not working
-
Key: MAPREDUCE-3009
URL: https://issues.apache.org/jira/browse/MAPREDUCE-3009
Project: Hadoop Map/Reduce
Issue Type: Bug
Components: jobhistoryserver, mrv2
Affects Versions: 0.23.0, 0.24.0
Reporter: chackaravarthy
Assignee: chackaravarthy
Fix For: 0.24.0
Attachments: IE_node_link_not_working.png, MAPREDUCE-3009-1.patch,
MAPREDUCE-3009.patch, chrome_node_link_working.png, invalid_url.png,
invalid_url_latest_code.png
RM UI - Applications - Application(Job History) - Map Tasks - Task ID -
Node link is not working. The URL contains extra '/' which is causing the
problem. Please find in the attached screen shots.
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators:
https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (MAPREDUCE-2775) [MR-279] Decommissioned node does not shutdown
[ https://issues.apache.org/jira/browse/MAPREDUCE-2775?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13126612#comment-13126612 ] Vinod Kumar Vavilapalli commented on MAPREDUCE-2775: Thanks. Am in a hurry now, will take a look in a day. I also want to make sure the test-suite is complete and 'd like to test the decommissioning feature on my single node setup. 'Coz we've never tested this ever, end to end in YARN - checking that containers are killed, NM is down properly, RM UI is updated with the decommissioned node count etc. In the mean while, please see if you can add tests for one/more of the above. Thanks again! [MR-279] Decommissioned node does not shutdown -- Key: MAPREDUCE-2775 URL: https://issues.apache.org/jira/browse/MAPREDUCE-2775 Project: Hadoop Map/Reduce Issue Type: Bug Components: mrv2 Affects Versions: 0.23.0 Reporter: Ramya Sunil Assignee: Devaraj K Priority: Blocker Fix For: 0.23.0 Attachments: MAPREDUCE-2775-1.patch, MAPREDUCE-2775.patch A Nodemanager which is decommissioned by an admin via refreshnodes does not automatically shutdown. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (MAPREDUCE-2775) [MR-279] Decommissioned node does not shutdown
[ https://issues.apache.org/jira/browse/MAPREDUCE-2775?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13126611#comment-13126611 ] Hadoop QA commented on MAPREDUCE-2775: -- +1 overall. Here are the results of testing the latest attachment http://issues.apache.org/jira/secure/attachment/12498882/MAPREDUCE-2775-1.patch against trunk revision . +1 @author. The patch does not contain any @author tags. +1 tests included. The patch appears to include 12 new or modified tests. +1 javadoc. The javadoc tool did not generate any warning messages. +1 javac. The applied patch does not increase the total number of javac compiler warnings. +1 findbugs. The patch does not introduce any new Findbugs (version 1.3.9) warnings. +1 release audit. The applied patch does not increase the total number of release audit warnings. +1 core tests. The patch passed unit tests in . +1 contrib tests. The patch passed contrib unit tests. Test results: https://builds.apache.org/job/PreCommit-MAPREDUCE-Build/1004//testReport/ Console output: https://builds.apache.org/job/PreCommit-MAPREDUCE-Build/1004//console This message is automatically generated. [MR-279] Decommissioned node does not shutdown -- Key: MAPREDUCE-2775 URL: https://issues.apache.org/jira/browse/MAPREDUCE-2775 Project: Hadoop Map/Reduce Issue Type: Bug Components: mrv2 Affects Versions: 0.23.0 Reporter: Ramya Sunil Assignee: Devaraj K Priority: Blocker Fix For: 0.23.0 Attachments: MAPREDUCE-2775-1.patch, MAPREDUCE-2775.patch A Nodemanager which is decommissioned by an admin via refreshnodes does not automatically shutdown. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Created] (MAPREDUCE-3178) Capacity Schedular shows incorrect cluster information in the RM logs
Capacity Schedular shows incorrect cluster information in the RM logs - Key: MAPREDUCE-3178 URL: https://issues.apache.org/jira/browse/MAPREDUCE-3178 Project: Hadoop Map/Reduce Issue Type: Bug Components: contrib/capacity-sched Affects Versions: 0.24.0 Reporter: Bhallamudi Venkata Siva Kamesh When we start the NM, after stopping it (in a quick session) CS shows incorrect information about clusterResource in the logs. I have encountered this issue in a pseudo cluster mode and steps to reproduce are 1) start the YARN cluster 2) stop a NM and start the NM again (in a quick session) There should be a NM running in the cluster however as I observed RM detects NM as dead, after default time since its actual unavailability(In this case NM has been stopped). If you start your NM before this time (default time), ResourceTracker throws IOEx, however, CS adds the NM's capacity to the clusterResource. After elapsed time (default time) when RM detects NM as dead, RM removes the NM and hence capacity of the cluster will be subtracted by the amount NM capacity. Eventually there is no NM running in the cluster, but capacity of the cluster is NM's capacity (by default) -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (MAPREDUCE-3178) Capacity Schedular shows incorrect cluster information in the RM logs
[
https://issues.apache.org/jira/browse/MAPREDUCE-3178?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13126615#comment-13126615
]
Bhallamudi Venkata Siva Kamesh commented on MAPREDUCE-3178:
---
When we start the NM again, as part of the registration, the following object
will be created.
{code}
RMNode rmNode = new RMNodeImpl(nodeId, rmContext, host, cmPort,
httpPort, resolve(host), capability);
{code}
The above code internally calls {code}
context.getDispatcher().getEventHandler().handle(new
NodeAddedSchedulerEvent(this)); {code}, which calls the CS#addNode() method.
Here again this node's capability will be added to the clusterResource. In the
mean time, the following part of the code throws exception
{code}
if (this.rmContext.getRMNodes().putIfAbsent(nodeId, rmNode) != null) {
throw new IOException(Duplicate registration from the node!);
}
{code}
Capacity Schedular shows incorrect cluster information in the RM logs
-
Key: MAPREDUCE-3178
URL: https://issues.apache.org/jira/browse/MAPREDUCE-3178
Project: Hadoop Map/Reduce
Issue Type: Bug
Components: contrib/capacity-sched
Affects Versions: 0.24.0
Reporter: Bhallamudi Venkata Siva Kamesh
When we start the NM, after stopping it (in a quick session) CS shows
incorrect information about clusterResource in the logs.
I have encountered this issue in a pseudo cluster mode and steps to reproduce
are
1) start the YARN cluster
2) stop a NM and start the NM again (in a quick session)
There should be a NM running in the cluster however as I observed RM detects
NM as dead, after default time since its actual unavailability(In this case
NM has been stopped).
If you start your NM before this time (default time), ResourceTracker throws
IOEx, however, CS adds the NM's capacity to the clusterResource.
After elapsed time (default time) when RM detects NM as dead, RM removes the
NM and hence capacity of the cluster will be subtracted by the amount NM
capacity.
Eventually there is no NM running in the cluster, but capacity of the cluster
is NM's capacity (by default)
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators:
https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (MAPREDUCE-2747) [MR-279] [Security] Cleanup LinuxContainerExecutor binary sources
[ https://issues.apache.org/jira/browse/MAPREDUCE-2747?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13126617#comment-13126617 ] Robert Joseph Evans commented on MAPREDUCE-2747: That is fine. I don't want to break anything related to security so take your time to review it. [MR-279] [Security] Cleanup LinuxContainerExecutor binary sources - Key: MAPREDUCE-2747 URL: https://issues.apache.org/jira/browse/MAPREDUCE-2747 Project: Hadoop Map/Reduce Issue Type: Sub-task Components: mrv2, nodemanager, security Affects Versions: 0.23.0, 0.24.0 Reporter: Vinod Kumar Vavilapalli Assignee: Robert Joseph Evans Priority: Blocker Fix For: 0.23.0, 0.24.0 Attachments: MAPREDUCE-2747.patch, MR-2747.txt There are a lot of references to the old task-controller nomenclature still, job/task refs instead of app/container. Also the configuration file is named as taskcontroller.cfg and the configured variables are also from the mapred world (mrv1). These SHOULD be fixed before we make a release. Marking this as blocker. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (MAPREDUCE-3009) RM UI - Applications - Application(Job History) - Map Tasks - Task ID - Node link is not working
[
https://issues.apache.org/jira/browse/MAPREDUCE-3009?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13126620#comment-13126620
]
chackaravarthy commented on MAPREDUCE-3009:
---
:-) here url method internally using ujoin method which is being used in many
places.
And actually here now i feel like no need to use url method at all.
simply ,
{code} td().a(.nodelink, http://+ nodeHttpAddr, nodeHttpAddr)._(); {code}
this may be enough i think. Because I see the similar usage for getting node
link in NodesPage.java
{code} td().a(http://; + ni.getHttpAddress(), ni.getHttpAddress())._(). {code}
shall i proceed with simple concatenation or fixing the ujoin method?
Thanks
RM UI - Applications - Application(Job History) - Map Tasks - Task ID -
Node link is not working
-
Key: MAPREDUCE-3009
URL: https://issues.apache.org/jira/browse/MAPREDUCE-3009
Project: Hadoop Map/Reduce
Issue Type: Bug
Components: jobhistoryserver, mrv2
Affects Versions: 0.23.0, 0.24.0
Reporter: chackaravarthy
Assignee: chackaravarthy
Fix For: 0.24.0
Attachments: IE_node_link_not_working.png, MAPREDUCE-3009-1.patch,
MAPREDUCE-3009.patch, chrome_node_link_working.png, invalid_url.png,
invalid_url_latest_code.png
RM UI - Applications - Application(Job History) - Map Tasks - Task ID -
Node link is not working. The URL contains extra '/' which is causing the
problem. Please find in the attached screen shots.
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators:
https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (MAPREDUCE-3133) Running a set of methods in a Single Test Class
[ https://issues.apache.org/jira/browse/MAPREDUCE-3133?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13126619#comment-13126619 ] Jonathan Eagles commented on MAPREDUCE-3133: Thanks, Mahadev! Running a set of methods in a Single Test Class --- Key: MAPREDUCE-3133 URL: https://issues.apache.org/jira/browse/MAPREDUCE-3133 Project: Hadoop Map/Reduce Issue Type: Improvement Components: build Affects Versions: 0.23.0 Reporter: Jonathan Eagles Assignee: Jonathan Eagles Fix For: 0.23.0 Attachments: MAPREDUCE-3132.patch, MAPREDUCE-3132.patch, MAPREDUCE-3132.patch Instead of running every test method in a class, limit to specific testing methods as describe in the link below. http://maven.apache.org/plugins/maven-surefire-plugin/examples/single-test.html Upgrade to the latest version of maven-surefire-plugin that has this feature. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Updated] (MAPREDUCE-3124) mapper failed with failed to load native libs
[
https://issues.apache.org/jira/browse/MAPREDUCE-3124?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Vinod Kumar Vavilapalli updated MAPREDUCE-3124:
---
Status: Open (was: Patch Available)
Hard coding Linux-i386-32 won't fly. I guess you can use the {{PlatformName}}
class in Hadoop common utils package, but please test and verify. Thanks.
mapper failed with failed to load native libs
-
Key: MAPREDUCE-3124
URL: https://issues.apache.org/jira/browse/MAPREDUCE-3124
Project: Hadoop Map/Reduce
Issue Type: Bug
Components: mrv2
Affects Versions: 0.23.0, 0.24.0
Reporter: Thomas Graves
Assignee: John George
Priority: Blocker
Fix For: 0.23.0
Attachments: MAPREDUCE-3124.patch, MAPREDUCE-3124.patch
hadoop jar hadoop-mapreduce-examples-*.jar sort -Dmapreduce.job.acl-view
-job=* -Dmapreduce.map.output.compress=true
-Dmapreduce.map.output.compress.codec=org.apache.hadoop.io.compress.GzipCodec
-Dmapreduce.output.fileoutputformat.compress=true
-Dmapreduce.output.fileoutputformat.compression.type=NONE -Dmap
reduce.output.fileoutputformat.compression.codec=org.apache.hadoop.io.compress.GzipCodec
-outKey
org.apache.hadoop.io.Text -outValue org.apache.hadoop.io.Text
Compression/textinput Compression/textoutput-1317315994
This will fail with native libs not found error unless
-Dmapred.child.java.opts='-Djava.library.path=${HADOOP_COMMON_HOME}/lib/native/Linux-i386-32'
is added.
The error in container log:
2011-09-29 17:06:56,787 DEBUG org.apache.hadoop.util.NativeCodeLoader: Trying
to load the custom-built native-hadoop
library...2011-09-29 17:06:56,787 DEBUG
org.apache.hadoop.util.NativeCodeLoader: Failed to load native-hadoop with
error: java.lang.UnsatisfiedLinkError: no hadoop in
java.library.path2011-09-29 17:06:56,787 DEBUG
org.apache.hadoop.util.NativeCodeLoader:
java.library.path=/share/gridjdk-1.6.0_21/jre/lib/i386/server:/share/gridjdk-1.6.0_21/jre/lib/i386:/share/gridjdk-1.6.0_21/jre/../lib/i386:/tmp/mapred-local/usercache/hadoopqa/appcache/application_1317314754104_0012/container_1317314754104_0012_01_02:/current/lib:/usr/java/packages/lib/i386:/lib:/usr/lib2011-09-29
17:06:56,787 WARN org.apache.hadoop.util.NativeCodeLoader: Unable to load
native-hadoop library for your platform...
using builtin-java classes where applicable
Also note that the error that shows up at the application master for this is
terrible:
Container killed by the ApplicationMaster. Container killed on request. Exit
code is 137 Too Many fetch failures.Failing the attempt
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators:
https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (MAPREDUCE-2858) MRv2 WebApp Security
[
https://issues.apache.org/jira/browse/MAPREDUCE-2858?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13126627#comment-13126627
]
Robert Joseph Evans commented on MAPREDUCE-2858:
{quote}
bq. Is the proxy going to try and rewrite URLs so that they always pass through
the proxy or is it simply going to rely on the application master to only
output relative URLs?
Proxy should only allow a (configurable) whitelist of hosts to be used in
absolute URLs.
{quote}
Luke I have a basic proxy working and all of the URLs that start with '/' are
failing because they are absolute with respect to the machine that they are on,
not absolute to all machines. This is the URL rewriting I was curious about.
How did you solve this issue?
MRv2 WebApp Security
Key: MAPREDUCE-2858
URL: https://issues.apache.org/jira/browse/MAPREDUCE-2858
Project: Hadoop Map/Reduce
Issue Type: Sub-task
Components: applicationmaster, mrv2, security
Affects Versions: 0.23.0
Reporter: Luke Lu
Assignee: Luke Lu
Priority: Blocker
Fix For: 0.23.0
In MRv2, while the system servers (ResourceManager (RM), NodeManager (NM) and
NameNode (NN)) run as trusted
system users, the application masters (AM) run as users who submit the
application. While this offers great flexibility
to run multiple version of mapreduce frameworks (including their UI) on the
same Hadoop cluster, it has significant
implication for the security of webapps (Please do not discuss company
specific vulnerabilities here).
Requirements:
# Secure authentication for AM (for app/job level ACLs).
# Webapp security should be optional via site configuration.
# Support existing pluggable single sign on mechanisms.
# Should not require per app/user configuration for deployment.
# Should not require special site-wide DNS configuration for deployment.
This the top jira for webapp security. A design doc/notes of threat-modeling
and counter measures will be posted on the wiki.
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators:
https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Updated] (MAPREDUCE-3124) mapper failed with failed to load native libs
[
https://issues.apache.org/jira/browse/MAPREDUCE-3124?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
John George updated MAPREDUCE-3124:
---
Status: Patch Available (was: Open)
mapper failed with failed to load native libs
-
Key: MAPREDUCE-3124
URL: https://issues.apache.org/jira/browse/MAPREDUCE-3124
Project: Hadoop Map/Reduce
Issue Type: Bug
Components: mrv2
Affects Versions: 0.23.0, 0.24.0
Reporter: Thomas Graves
Assignee: John George
Priority: Blocker
Fix For: 0.23.0
Attachments: MAPREDUCE-3124.patch, MAPREDUCE-3124.patch,
MAPREDUCE-3124.patch
hadoop jar hadoop-mapreduce-examples-*.jar sort -Dmapreduce.job.acl-view
-job=* -Dmapreduce.map.output.compress=true
-Dmapreduce.map.output.compress.codec=org.apache.hadoop.io.compress.GzipCodec
-Dmapreduce.output.fileoutputformat.compress=true
-Dmapreduce.output.fileoutputformat.compression.type=NONE -Dmap
reduce.output.fileoutputformat.compression.codec=org.apache.hadoop.io.compress.GzipCodec
-outKey
org.apache.hadoop.io.Text -outValue org.apache.hadoop.io.Text
Compression/textinput Compression/textoutput-1317315994
This will fail with native libs not found error unless
-Dmapred.child.java.opts='-Djava.library.path=${HADOOP_COMMON_HOME}/lib/native/Linux-i386-32'
is added.
The error in container log:
2011-09-29 17:06:56,787 DEBUG org.apache.hadoop.util.NativeCodeLoader: Trying
to load the custom-built native-hadoop
library...2011-09-29 17:06:56,787 DEBUG
org.apache.hadoop.util.NativeCodeLoader: Failed to load native-hadoop with
error: java.lang.UnsatisfiedLinkError: no hadoop in
java.library.path2011-09-29 17:06:56,787 DEBUG
org.apache.hadoop.util.NativeCodeLoader:
java.library.path=/share/gridjdk-1.6.0_21/jre/lib/i386/server:/share/gridjdk-1.6.0_21/jre/lib/i386:/share/gridjdk-1.6.0_21/jre/../lib/i386:/tmp/mapred-local/usercache/hadoopqa/appcache/application_1317314754104_0012/container_1317314754104_0012_01_02:/current/lib:/usr/java/packages/lib/i386:/lib:/usr/lib2011-09-29
17:06:56,787 WARN org.apache.hadoop.util.NativeCodeLoader: Unable to load
native-hadoop library for your platform...
using builtin-java classes where applicable
Also note that the error that shows up at the application master for this is
terrible:
Container killed by the ApplicationMaster. Container killed on request. Exit
code is 137 Too Many fetch failures.Failing the attempt
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators:
https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Updated] (MAPREDUCE-3124) mapper failed with failed to load native libs
[
https://issues.apache.org/jira/browse/MAPREDUCE-3124?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
John George updated MAPREDUCE-3124:
---
Attachment: MAPREDUCE-3124.patch
Thanks Vinod. Attaching another patch using getPlatform().
mapper failed with failed to load native libs
-
Key: MAPREDUCE-3124
URL: https://issues.apache.org/jira/browse/MAPREDUCE-3124
Project: Hadoop Map/Reduce
Issue Type: Bug
Components: mrv2
Affects Versions: 0.23.0, 0.24.0
Reporter: Thomas Graves
Assignee: John George
Priority: Blocker
Fix For: 0.23.0
Attachments: MAPREDUCE-3124.patch, MAPREDUCE-3124.patch,
MAPREDUCE-3124.patch
hadoop jar hadoop-mapreduce-examples-*.jar sort -Dmapreduce.job.acl-view
-job=* -Dmapreduce.map.output.compress=true
-Dmapreduce.map.output.compress.codec=org.apache.hadoop.io.compress.GzipCodec
-Dmapreduce.output.fileoutputformat.compress=true
-Dmapreduce.output.fileoutputformat.compression.type=NONE -Dmap
reduce.output.fileoutputformat.compression.codec=org.apache.hadoop.io.compress.GzipCodec
-outKey
org.apache.hadoop.io.Text -outValue org.apache.hadoop.io.Text
Compression/textinput Compression/textoutput-1317315994
This will fail with native libs not found error unless
-Dmapred.child.java.opts='-Djava.library.path=${HADOOP_COMMON_HOME}/lib/native/Linux-i386-32'
is added.
The error in container log:
2011-09-29 17:06:56,787 DEBUG org.apache.hadoop.util.NativeCodeLoader: Trying
to load the custom-built native-hadoop
library...2011-09-29 17:06:56,787 DEBUG
org.apache.hadoop.util.NativeCodeLoader: Failed to load native-hadoop with
error: java.lang.UnsatisfiedLinkError: no hadoop in
java.library.path2011-09-29 17:06:56,787 DEBUG
org.apache.hadoop.util.NativeCodeLoader:
java.library.path=/share/gridjdk-1.6.0_21/jre/lib/i386/server:/share/gridjdk-1.6.0_21/jre/lib/i386:/share/gridjdk-1.6.0_21/jre/../lib/i386:/tmp/mapred-local/usercache/hadoopqa/appcache/application_1317314754104_0012/container_1317314754104_0012_01_02:/current/lib:/usr/java/packages/lib/i386:/lib:/usr/lib2011-09-29
17:06:56,787 WARN org.apache.hadoop.util.NativeCodeLoader: Unable to load
native-hadoop library for your platform...
using builtin-java classes where applicable
Also note that the error that shows up at the application master for this is
terrible:
Container killed by the ApplicationMaster. Container killed on request. Exit
code is 137 Too Many fetch failures.Failing the attempt
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators:
https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (MAPREDUCE-2863) Support web-services for RM NM
[
https://issues.apache.org/jira/browse/MAPREDUCE-2863?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13126665#comment-13126665
]
Thomas Graves commented on MAPREDUCE-2863:
--
Ok so I compared hamlet to jersey. I wrote a small app to list the nodes from
RM in json (equivalent of webapp page your_RM:8088/cluster/nodes). here is
what I came up with. If anyone see better ways to do these or sees something I
did wrong please point it out. Any comments/opinions are welcome!
Note since jersey is integrated into httpserver its a simple call to that to
add the jersey resource and you get all the filtering/security already in
httpserver if you register it properly. It would take a few functions to add
that into existing WebApp yarn framework but not much at all. I verified that
both go through any filters you have defined.
Could be as simple as (In RMWebApp which extends WebApp):
setJerseyPackageName(TomJerseyResource.class.getPackage().getName());
setJerseyPath(/jersey/*);
setAttribute(rm, this.rm); // inject RM instance to jersey
HttpServer server =
new HttpServer(name, bindAddress, port, findPort, conf,
webapp.getServePathSpecs() and webapp.getJerseyPath );
Assuming we have a data structure or produce data structure with info we want
to dump in json since that is needed for regular webapp html and its the same
in either hamlet or jersey here is the code I came up with:
Jersey:
@Path()
public class TestJerseyResource {
@GET
@Produces({MediaType.APPLICATION_JSON})
public Response get() throws IOException {
final ResourceManager rm = (ResourceManager)context.getAttribute(rm);
final RMContext rmContext = rm.getRMContext();
MapString, Object allNodes = getNodesInfoMap(rm, rmContext);
final String js = JSON.toString(allNodes);
return Response.ok(js).type(MediaType.APPLICATION_JSON).build();
}
}
Hamlet: in RMController.java
public void nodesjson() {
RMContext rmContext = getInstance(RMContext.class);
ResourceManager rm = getInstance(ResourceManager.class);
MapString, Object allNodes = getNodesInfoMap(rm, rmContext);
renderJSON(allNodes);
}
A few side notes. It took me longer to figure out how to do the hamlet stuff
then jersey. Probably because jersey is well documented and I was able to copy
example from existing hdfs code. I also still need to look further on how to
make it integrate nicely with the html stuff as some of the functions like
renderJSON were only available at the Controller level. I also need to see how
hamlet handles requesting different content type (I'm assuming it will) so that
you could go to same uri and just ask for the content type differently.
So in conclusion the code sizes seems pretty comparable. If we want to keep
with one framework I could do it in hamlet. I found it a little harder to use
but that is probably just a learning curve.
Ideally whichever we use the html for web ui and json web services use the same
underlying functions to get the data although looking at a bit of the html
render stuff that looks harder then I expected but perhaps there is some fancy
hamlet stuff I haven't found yet.
Support web-services for RM NM
Key: MAPREDUCE-2863
URL: https://issues.apache.org/jira/browse/MAPREDUCE-2863
Project: Hadoop Map/Reduce
Issue Type: Improvement
Components: mrv2, nodemanager, resourcemanager
Reporter: Arun C Murthy
Assignee: Thomas Graves
It will be very useful for RM and NM to support web-services to export
json/xml.
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators:
https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (MAPREDUCE-3124) mapper failed with failed to load native libs
[
https://issues.apache.org/jira/browse/MAPREDUCE-3124?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13126669#comment-13126669
]
Hadoop QA commented on MAPREDUCE-3124:
--
-1 overall. Here are the results of testing the latest attachment
http://issues.apache.org/jira/secure/attachment/12498887/MAPREDUCE-3124.patch
against trunk revision .
+1 @author. The patch does not contain any @author tags.
-1 tests included. The patch doesn't appear to include any new or modified
tests.
Please justify why no new tests are needed for this
patch.
Also please list what manual steps were performed to
verify this patch.
+1 javadoc. The javadoc tool did not generate any warning messages.
+1 javac. The applied patch does not increase the total number of javac
compiler warnings.
+1 findbugs. The patch does not introduce any new Findbugs (version 1.3.9)
warnings.
+1 release audit. The applied patch does not increase the total number of
release audit warnings.
+1 core tests. The patch passed unit tests in .
+1 contrib tests. The patch passed contrib unit tests.
Test results:
https://builds.apache.org/job/PreCommit-MAPREDUCE-Build/1005//testReport/
Console output:
https://builds.apache.org/job/PreCommit-MAPREDUCE-Build/1005//console
This message is automatically generated.
mapper failed with failed to load native libs
-
Key: MAPREDUCE-3124
URL: https://issues.apache.org/jira/browse/MAPREDUCE-3124
Project: Hadoop Map/Reduce
Issue Type: Bug
Components: mrv2
Affects Versions: 0.23.0, 0.24.0
Reporter: Thomas Graves
Assignee: John George
Priority: Blocker
Fix For: 0.23.0
Attachments: MAPREDUCE-3124.patch, MAPREDUCE-3124.patch,
MAPREDUCE-3124.patch
hadoop jar hadoop-mapreduce-examples-*.jar sort -Dmapreduce.job.acl-view
-job=* -Dmapreduce.map.output.compress=true
-Dmapreduce.map.output.compress.codec=org.apache.hadoop.io.compress.GzipCodec
-Dmapreduce.output.fileoutputformat.compress=true
-Dmapreduce.output.fileoutputformat.compression.type=NONE -Dmap
reduce.output.fileoutputformat.compression.codec=org.apache.hadoop.io.compress.GzipCodec
-outKey
org.apache.hadoop.io.Text -outValue org.apache.hadoop.io.Text
Compression/textinput Compression/textoutput-1317315994
This will fail with native libs not found error unless
-Dmapred.child.java.opts='-Djava.library.path=${HADOOP_COMMON_HOME}/lib/native/Linux-i386-32'
is added.
The error in container log:
2011-09-29 17:06:56,787 DEBUG org.apache.hadoop.util.NativeCodeLoader: Trying
to load the custom-built native-hadoop
library...2011-09-29 17:06:56,787 DEBUG
org.apache.hadoop.util.NativeCodeLoader: Failed to load native-hadoop with
error: java.lang.UnsatisfiedLinkError: no hadoop in
java.library.path2011-09-29 17:06:56,787 DEBUG
org.apache.hadoop.util.NativeCodeLoader:
java.library.path=/share/gridjdk-1.6.0_21/jre/lib/i386/server:/share/gridjdk-1.6.0_21/jre/lib/i386:/share/gridjdk-1.6.0_21/jre/../lib/i386:/tmp/mapred-local/usercache/hadoopqa/appcache/application_1317314754104_0012/container_1317314754104_0012_01_02:/current/lib:/usr/java/packages/lib/i386:/lib:/usr/lib2011-09-29
17:06:56,787 WARN org.apache.hadoop.util.NativeCodeLoader: Unable to load
native-hadoop library for your platform...
using builtin-java classes where applicable
Also note that the error that shows up at the application master for this is
terrible:
Container killed by the ApplicationMaster. Container killed on request. Exit
code is 137 Too Many fetch failures.Failing the attempt
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators:
https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (MAPREDUCE-2863) Support web-services for RM NM
[ https://issues.apache.org/jira/browse/MAPREDUCE-2863?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13126680#comment-13126680 ] Robert Joseph Evans commented on MAPREDUCE-2863: One thing to point out about JERSEY and Jackson is that if we use JAXB annotations in the classes that we read/write we cat get JSON/XML support seamlessly for free. It may not be that critical if we only want to support JSON, but it is very convenient if we want to support multiple types. Support web-services for RM NM Key: MAPREDUCE-2863 URL: https://issues.apache.org/jira/browse/MAPREDUCE-2863 Project: Hadoop Map/Reduce Issue Type: Improvement Components: mrv2, nodemanager, resourcemanager Reporter: Arun C Murthy Assignee: Thomas Graves It will be very useful for RM and NM to support web-services to export json/xml. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (MAPREDUCE-3177) mapreduce tar layout does not conform new layout
[ https://issues.apache.org/jira/browse/MAPREDUCE-3177?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13126691#comment-13126691 ] Alejandro Abdelnur commented on MAPREDUCE-3177: --- I'll open a JIRA to keep track of the Maven related JIRAs to backport to 0.23. Doing subtasks is not feasible because subtasks cannot span across projects. mapreduce tar layout does not conform new layout Key: MAPREDUCE-3177 URL: https://issues.apache.org/jira/browse/MAPREDUCE-3177 Project: Hadoop Map/Reduce Issue Type: Bug Components: build Affects Versions: 0.23.0, 0.24.0 Reporter: Alejandro Abdelnur Priority: Critical Fix For: 0.23.0, 0.24.0 The tar generated by MR does not follow the layout of common hdfs, instead, it uses a arbitrary layout which is also different from the old legacy layout (there is a modules/ directory with all the MR jars) -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Created] (MAPREDUCE-3179) Incorrect exit code for hadoop-mapreduce-test tests when exception thrown
Incorrect exit code for hadoop-mapreduce-test tests when exception thrown - Key: MAPREDUCE-3179 URL: https://issues.apache.org/jira/browse/MAPREDUCE-3179 Project: Hadoop Map/Reduce Issue Type: Bug Components: mrv2, test Affects Versions: 0.23.0, 0.24.0 Reporter: Jonathan Eagles Assignee: Jonathan Eagles Exit code for test jar is 0 despite exception thrown hadoop jar hadoop-mapreduce-test-0.23.0-SNAPSHOT.jar loadgen -Dmapreduce.job.acl-view -m 18 -r 0 -outKey org.apache.hadoop.io.Text -outValue org.apache.hadoop.io.Text -indir nonexistentdir Loadgen output snippet org.apache.hadoop.mapred.InvalidInputException: Input path does not exist: hdfs://machine.name.example.com:9000/user/exampleuser/nonexistentdir at org.apache.hadoop.mapred.FileInputFormat.listStatus(FileInputFormat.java:234) at org.apache.hadoop.mapred.FileInputFormat.getSplits(FileInputFormat.java:254) at org.apache.hadoop.mapreduce.JobSubmitter.writeOldSplits(JobSubmitter.java:470) at org.apache.hadoop.mapreduce.JobSubmitter.writeSplits(JobSubmitter.java:462) at org.apache.hadoop.mapreduce.JobSubmitter.submitJobInternal(JobSubmitter.java:358) at org.apache.hadoop.mapreduce.Job$2.run(Job.java:1159) at org.apache.hadoop.mapreduce.Job$2.run(Job.java:1156) at java.security.AccessController.doPrivileged(Native Method) at javax.security.auth.Subject.doAs(Subject.java:396) at org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1135) at org.apache.hadoop.mapreduce.Job.submit(Job.java:1156) at org.apache.hadoop.mapred.JobClient.submitJob(JobClient.java:539) at org.apache.hadoop.mapred.JobClient.runJob(JobClient.java:778) at org.apache.hadoop.mapred.GenericMRLoadGenerator.run(GenericMRLoadGenerator.java:200) at org.apache.hadoop.util.ToolRunner.run(ToolRunner.java:69) at org.apache.hadoop.mapred.GenericMRLoadGenerator.main(GenericMRLoadGenerator.java:214) at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method) at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:39) at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:25) at java.lang.reflect.Method.invoke(Method.java:597) at org.apache.hadoop.util.ProgramDriver$ProgramDescription.invoke(ProgramDriver.java:72) at org.apache.hadoop.util.ProgramDriver.driver(ProgramDriver.java:144) at org.apache.hadoop.test.MapredTestDriver.run(MapredTestDriver.java:111) at org.apache.hadoop.test.MapredTestDriver.main(MapredTestDriver.java:118) at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method) at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:39) at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:25) at java.lang.reflect.Method.invoke(Method.java:597) at org.apache.hadoop.util.RunJar.main(RunJar.java:189) -bash-3.2$ echo $? 0 This differs from example jar which correctly returns the correct exit code hadoop jar hadoop-mapreduce-examples-0.23.0-SNAPSHOT.jar wordcount nonexistentdir /outputdir wordcount output snippet org.apache.hadoop.mapreduce.lib.input.InvalidInputException: Input path does not exist: hdfs://machine.name.example.com:9000/user/exampleuser/nonexistentdir at org.apache.hadoop.mapreduce.lib.input.FileInputFormat.listStatus(FileInputFormat.java:243) at org.apache.hadoop.mapreduce.lib.input.FileInputFormat.getSplits(FileInputFormat.java:269) at org.apache.hadoop.mapreduce.JobSubmitter.writeNewSplits(JobSubmitter.java:443) at org.apache.hadoop.mapreduce.JobSubmitter.writeSplits(JobSubmitter.java:460) at org.apache.hadoop.mapreduce.JobSubmitter.submitJobInternal(JobSubmitter.java:358) at org.apache.hadoop.mapreduce.Job$2.run(Job.java:1159) at org.apache.hadoop.mapreduce.Job$2.run(Job.java:1156) at java.security.AccessController.doPrivileged(Native Method) at javax.security.auth.Subject.doAs(Subject.java:396) at org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1135) at org.apache.hadoop.mapreduce.Job.submit(Job.java:1156) at org.apache.hadoop.mapreduce.Job.waitForCompletion(Job.java:1176) at org.apache.hadoop.examples.WordCount.main(WordCount.java:84) at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method) at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:39) at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:25) at java.lang.reflect.Method.invoke(Method.java:597) at
[jira] [Updated] (MAPREDUCE-3179) Incorrect exit code for hadoop-mapreduce-test tests when exception thrown
[ https://issues.apache.org/jira/browse/MAPREDUCE-3179?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Jonathan Eagles updated MAPREDUCE-3179: --- Attachment: MAPREDUCE-3179.patch Incorrect exit code for hadoop-mapreduce-test tests when exception thrown - Key: MAPREDUCE-3179 URL: https://issues.apache.org/jira/browse/MAPREDUCE-3179 Project: Hadoop Map/Reduce Issue Type: Bug Components: mrv2, test Affects Versions: 0.23.0, 0.24.0 Reporter: Jonathan Eagles Assignee: Jonathan Eagles Attachments: MAPREDUCE-3179.patch Exit code for test jar is 0 despite exception thrown hadoop jar hadoop-mapreduce-test-0.23.0-SNAPSHOT.jar loadgen -Dmapreduce.job.acl-view -m 18 -r 0 -outKey org.apache.hadoop.io.Text -outValue org.apache.hadoop.io.Text -indir nonexistentdir Loadgen output snippet org.apache.hadoop.mapred.InvalidInputException: Input path does not exist: hdfs://machine.name.example.com:9000/user/exampleuser/nonexistentdir at org.apache.hadoop.mapred.FileInputFormat.listStatus(FileInputFormat.java:234) at org.apache.hadoop.mapred.FileInputFormat.getSplits(FileInputFormat.java:254) at org.apache.hadoop.mapreduce.JobSubmitter.writeOldSplits(JobSubmitter.java:470) at org.apache.hadoop.mapreduce.JobSubmitter.writeSplits(JobSubmitter.java:462) at org.apache.hadoop.mapreduce.JobSubmitter.submitJobInternal(JobSubmitter.java:358) at org.apache.hadoop.mapreduce.Job$2.run(Job.java:1159) at org.apache.hadoop.mapreduce.Job$2.run(Job.java:1156) at java.security.AccessController.doPrivileged(Native Method) at javax.security.auth.Subject.doAs(Subject.java:396) at org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1135) at org.apache.hadoop.mapreduce.Job.submit(Job.java:1156) at org.apache.hadoop.mapred.JobClient.submitJob(JobClient.java:539) at org.apache.hadoop.mapred.JobClient.runJob(JobClient.java:778) at org.apache.hadoop.mapred.GenericMRLoadGenerator.run(GenericMRLoadGenerator.java:200) at org.apache.hadoop.util.ToolRunner.run(ToolRunner.java:69) at org.apache.hadoop.mapred.GenericMRLoadGenerator.main(GenericMRLoadGenerator.java:214) at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method) at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:39) at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:25) at java.lang.reflect.Method.invoke(Method.java:597) at org.apache.hadoop.util.ProgramDriver$ProgramDescription.invoke(ProgramDriver.java:72) at org.apache.hadoop.util.ProgramDriver.driver(ProgramDriver.java:144) at org.apache.hadoop.test.MapredTestDriver.run(MapredTestDriver.java:111) at org.apache.hadoop.test.MapredTestDriver.main(MapredTestDriver.java:118) at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method) at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:39) at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:25) at java.lang.reflect.Method.invoke(Method.java:597) at org.apache.hadoop.util.RunJar.main(RunJar.java:189) -bash-3.2$ echo $? 0 This differs from example jar which correctly returns the correct exit code hadoop jar hadoop-mapreduce-examples-0.23.0-SNAPSHOT.jar wordcount nonexistentdir /outputdir wordcount output snippet org.apache.hadoop.mapreduce.lib.input.InvalidInputException: Input path does not exist: hdfs://machine.name.example.com:9000/user/exampleuser/nonexistentdir at org.apache.hadoop.mapreduce.lib.input.FileInputFormat.listStatus(FileInputFormat.java:243) at org.apache.hadoop.mapreduce.lib.input.FileInputFormat.getSplits(FileInputFormat.java:269) at org.apache.hadoop.mapreduce.JobSubmitter.writeNewSplits(JobSubmitter.java:443) at org.apache.hadoop.mapreduce.JobSubmitter.writeSplits(JobSubmitter.java:460) at org.apache.hadoop.mapreduce.JobSubmitter.submitJobInternal(JobSubmitter.java:358) at org.apache.hadoop.mapreduce.Job$2.run(Job.java:1159) at org.apache.hadoop.mapreduce.Job$2.run(Job.java:1156) at java.security.AccessController.doPrivileged(Native Method) at javax.security.auth.Subject.doAs(Subject.java:396) at org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1135) at org.apache.hadoop.mapreduce.Job.submit(Job.java:1156) at org.apache.hadoop.mapreduce.Job.waitForCompletion(Job.java:1176) at
[jira] [Commented] (MAPREDUCE-2858) MRv2 WebApp Security
[ https://issues.apache.org/jira/browse/MAPREDUCE-2858?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13126710#comment-13126710 ] Alejandro Abdelnur commented on MAPREDUCE-2858: --- My be I'm not getting enough hours of sleep lately but I'm not able to get a full understanding of how the proxy works. Would be possible to have a sequence diagram showing the steps and data that flow for a request? Asking because the proxy seems to be talking with the RM, seems to be rewriting data (URLs), etc, etc. That is not simple stuff. On a complete different twist if the concern is about an AM seeing the 'company-wide' cookies from a user because of the single sign on; wouldn't be simpler that the AM container provides an API to register filtersservlets and filters out all cookies before giving control to the AM filter/servlet? And, to avoid the AM code to open an arbitrary port to listen to non-curated HTTP requests, the AM container would run with a security manager that prevents opening new sockets. Recapping: * The AM container initializes an AM HTTP server. * The AM HTTP server is preinitialized with any 'company' specific authentication filter. * The AM runs in a SecurityManager that forbids AM code to open ports. * The AM code can register servlets and filters to this AM HTTP server. * The AM code starts the AM HTTP server server once all servlets filters are register. * The AM HTTP server filters out all cookies, thus the AM code does not see them. IMO this addresses the original issues without having to do introduce Application Proxies with complex logic. Thoughts? MRv2 WebApp Security Key: MAPREDUCE-2858 URL: https://issues.apache.org/jira/browse/MAPREDUCE-2858 Project: Hadoop Map/Reduce Issue Type: Sub-task Components: applicationmaster, mrv2, security Affects Versions: 0.23.0 Reporter: Luke Lu Assignee: Luke Lu Priority: Blocker Fix For: 0.23.0 In MRv2, while the system servers (ResourceManager (RM), NodeManager (NM) and NameNode (NN)) run as trusted system users, the application masters (AM) run as users who submit the application. While this offers great flexibility to run multiple version of mapreduce frameworks (including their UI) on the same Hadoop cluster, it has significant implication for the security of webapps (Please do not discuss company specific vulnerabilities here). Requirements: # Secure authentication for AM (for app/job level ACLs). # Webapp security should be optional via site configuration. # Support existing pluggable single sign on mechanisms. # Should not require per app/user configuration for deployment. # Should not require special site-wide DNS configuration for deployment. This the top jira for webapp security. A design doc/notes of threat-modeling and counter measures will be posted on the wiki. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (MAPREDUCE-2863) Support web-services for RM NM
[ https://issues.apache.org/jira/browse/MAPREDUCE-2863?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13126709#comment-13126709 ] Thomas Graves commented on MAPREDUCE-2863: -- Good point. Other things I'm looking at is how to do the query parameters in hamlet and requesting the different types. I know this is really easy in Jersey with the annotations. I'm also not sure if guice and jersey integrate nicely to allow you to use the same uri and just specify the type of output you want. Otherwise we might need to use different uris for web services then for the web UI which would be ugly. Support web-services for RM NM Key: MAPREDUCE-2863 URL: https://issues.apache.org/jira/browse/MAPREDUCE-2863 Project: Hadoop Map/Reduce Issue Type: Improvement Components: mrv2, nodemanager, resourcemanager Reporter: Arun C Murthy Assignee: Thomas Graves It will be very useful for RM and NM to support web-services to export json/xml. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Updated] (MAPREDUCE-3179) Incorrect exit code for hadoop-mapreduce-test tests when exception thrown
[ https://issues.apache.org/jira/browse/MAPREDUCE-3179?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Jonathan Eagles updated MAPREDUCE-3179: --- Target Version/s: 0.23.0, 0.24.0 Status: Patch Available (was: Open) Incorrect exit code for hadoop-mapreduce-test tests when exception thrown - Key: MAPREDUCE-3179 URL: https://issues.apache.org/jira/browse/MAPREDUCE-3179 Project: Hadoop Map/Reduce Issue Type: Bug Components: mrv2, test Affects Versions: 0.23.0, 0.24.0 Reporter: Jonathan Eagles Assignee: Jonathan Eagles Attachments: MAPREDUCE-3179.patch Exit code for test jar is 0 despite exception thrown hadoop jar hadoop-mapreduce-test-0.23.0-SNAPSHOT.jar loadgen -Dmapreduce.job.acl-view -m 18 -r 0 -outKey org.apache.hadoop.io.Text -outValue org.apache.hadoop.io.Text -indir nonexistentdir Loadgen output snippet org.apache.hadoop.mapred.InvalidInputException: Input path does not exist: hdfs://machine.name.example.com:9000/user/exampleuser/nonexistentdir at org.apache.hadoop.mapred.FileInputFormat.listStatus(FileInputFormat.java:234) at org.apache.hadoop.mapred.FileInputFormat.getSplits(FileInputFormat.java:254) at org.apache.hadoop.mapreduce.JobSubmitter.writeOldSplits(JobSubmitter.java:470) at org.apache.hadoop.mapreduce.JobSubmitter.writeSplits(JobSubmitter.java:462) at org.apache.hadoop.mapreduce.JobSubmitter.submitJobInternal(JobSubmitter.java:358) at org.apache.hadoop.mapreduce.Job$2.run(Job.java:1159) at org.apache.hadoop.mapreduce.Job$2.run(Job.java:1156) at java.security.AccessController.doPrivileged(Native Method) at javax.security.auth.Subject.doAs(Subject.java:396) at org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1135) at org.apache.hadoop.mapreduce.Job.submit(Job.java:1156) at org.apache.hadoop.mapred.JobClient.submitJob(JobClient.java:539) at org.apache.hadoop.mapred.JobClient.runJob(JobClient.java:778) at org.apache.hadoop.mapred.GenericMRLoadGenerator.run(GenericMRLoadGenerator.java:200) at org.apache.hadoop.util.ToolRunner.run(ToolRunner.java:69) at org.apache.hadoop.mapred.GenericMRLoadGenerator.main(GenericMRLoadGenerator.java:214) at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method) at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:39) at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:25) at java.lang.reflect.Method.invoke(Method.java:597) at org.apache.hadoop.util.ProgramDriver$ProgramDescription.invoke(ProgramDriver.java:72) at org.apache.hadoop.util.ProgramDriver.driver(ProgramDriver.java:144) at org.apache.hadoop.test.MapredTestDriver.run(MapredTestDriver.java:111) at org.apache.hadoop.test.MapredTestDriver.main(MapredTestDriver.java:118) at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method) at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:39) at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:25) at java.lang.reflect.Method.invoke(Method.java:597) at org.apache.hadoop.util.RunJar.main(RunJar.java:189) -bash-3.2$ echo $? 0 This differs from example jar which correctly returns the correct exit code hadoop jar hadoop-mapreduce-examples-0.23.0-SNAPSHOT.jar wordcount nonexistentdir /outputdir wordcount output snippet org.apache.hadoop.mapreduce.lib.input.InvalidInputException: Input path does not exist: hdfs://machine.name.example.com:9000/user/exampleuser/nonexistentdir at org.apache.hadoop.mapreduce.lib.input.FileInputFormat.listStatus(FileInputFormat.java:243) at org.apache.hadoop.mapreduce.lib.input.FileInputFormat.getSplits(FileInputFormat.java:269) at org.apache.hadoop.mapreduce.JobSubmitter.writeNewSplits(JobSubmitter.java:443) at org.apache.hadoop.mapreduce.JobSubmitter.writeSplits(JobSubmitter.java:460) at org.apache.hadoop.mapreduce.JobSubmitter.submitJobInternal(JobSubmitter.java:358) at org.apache.hadoop.mapreduce.Job$2.run(Job.java:1159) at org.apache.hadoop.mapreduce.Job$2.run(Job.java:1156) at java.security.AccessController.doPrivileged(Native Method) at javax.security.auth.Subject.doAs(Subject.java:396) at org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1135) at org.apache.hadoop.mapreduce.Job.submit(Job.java:1156) at org.apache.hadoop.mapreduce.Job.waitForCompletion(Job.java:1176)
[jira] [Commented] (MAPREDUCE-3179) Incorrect exit code for hadoop-mapreduce-test tests when exception thrown
[ https://issues.apache.org/jira/browse/MAPREDUCE-3179?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13126728#comment-13126728 ] Hadoop QA commented on MAPREDUCE-3179: -- +1 overall. Here are the results of testing the latest attachment http://issues.apache.org/jira/secure/attachment/12498899/MAPREDUCE-3179.patch against trunk revision . +1 @author. The patch does not contain any @author tags. +1 tests included. The patch appears to include 3 new or modified tests. +1 javadoc. The javadoc tool did not generate any warning messages. +1 javac. The applied patch does not increase the total number of javac compiler warnings. +1 findbugs. The patch does not introduce any new Findbugs (version 1.3.9) warnings. +1 release audit. The applied patch does not increase the total number of release audit warnings. +1 core tests. The patch passed unit tests in . +1 contrib tests. The patch passed contrib unit tests. Test results: https://builds.apache.org/job/PreCommit-MAPREDUCE-Build/1006//testReport/ Console output: https://builds.apache.org/job/PreCommit-MAPREDUCE-Build/1006//console This message is automatically generated. Incorrect exit code for hadoop-mapreduce-test tests when exception thrown - Key: MAPREDUCE-3179 URL: https://issues.apache.org/jira/browse/MAPREDUCE-3179 Project: Hadoop Map/Reduce Issue Type: Bug Components: mrv2, test Affects Versions: 0.23.0, 0.24.0 Reporter: Jonathan Eagles Assignee: Jonathan Eagles Attachments: MAPREDUCE-3179.patch Exit code for test jar is 0 despite exception thrown hadoop jar hadoop-mapreduce-test-0.23.0-SNAPSHOT.jar loadgen -Dmapreduce.job.acl-view -m 18 -r 0 -outKey org.apache.hadoop.io.Text -outValue org.apache.hadoop.io.Text -indir nonexistentdir Loadgen output snippet org.apache.hadoop.mapred.InvalidInputException: Input path does not exist: hdfs://machine.name.example.com:9000/user/exampleuser/nonexistentdir at org.apache.hadoop.mapred.FileInputFormat.listStatus(FileInputFormat.java:234) at org.apache.hadoop.mapred.FileInputFormat.getSplits(FileInputFormat.java:254) at org.apache.hadoop.mapreduce.JobSubmitter.writeOldSplits(JobSubmitter.java:470) at org.apache.hadoop.mapreduce.JobSubmitter.writeSplits(JobSubmitter.java:462) at org.apache.hadoop.mapreduce.JobSubmitter.submitJobInternal(JobSubmitter.java:358) at org.apache.hadoop.mapreduce.Job$2.run(Job.java:1159) at org.apache.hadoop.mapreduce.Job$2.run(Job.java:1156) at java.security.AccessController.doPrivileged(Native Method) at javax.security.auth.Subject.doAs(Subject.java:396) at org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1135) at org.apache.hadoop.mapreduce.Job.submit(Job.java:1156) at org.apache.hadoop.mapred.JobClient.submitJob(JobClient.java:539) at org.apache.hadoop.mapred.JobClient.runJob(JobClient.java:778) at org.apache.hadoop.mapred.GenericMRLoadGenerator.run(GenericMRLoadGenerator.java:200) at org.apache.hadoop.util.ToolRunner.run(ToolRunner.java:69) at org.apache.hadoop.mapred.GenericMRLoadGenerator.main(GenericMRLoadGenerator.java:214) at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method) at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:39) at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:25) at java.lang.reflect.Method.invoke(Method.java:597) at org.apache.hadoop.util.ProgramDriver$ProgramDescription.invoke(ProgramDriver.java:72) at org.apache.hadoop.util.ProgramDriver.driver(ProgramDriver.java:144) at org.apache.hadoop.test.MapredTestDriver.run(MapredTestDriver.java:111) at org.apache.hadoop.test.MapredTestDriver.main(MapredTestDriver.java:118) at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method) at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:39) at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:25) at java.lang.reflect.Method.invoke(Method.java:597) at org.apache.hadoop.util.RunJar.main(RunJar.java:189) -bash-3.2$ echo $? 0 This differs from example jar which correctly returns the correct exit code hadoop jar hadoop-mapreduce-examples-0.23.0-SNAPSHOT.jar wordcount nonexistentdir /outputdir wordcount output snippet org.apache.hadoop.mapreduce.lib.input.InvalidInputException: Input path does not exist: hdfs://machine.name.example.com:9000/user/exampleuser/nonexistentdir
[jira] [Commented] (MAPREDUCE-2461) Hudson jobs failing because mapred staging directory is full
[ https://issues.apache.org/jira/browse/MAPREDUCE-2461?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13126732#comment-13126732 ] Thomas Weise commented on MAPREDUCE-2461: - We see the same issue with 0.20.204. Hudson jobs failing because mapred staging directory is full Key: MAPREDUCE-2461 URL: https://issues.apache.org/jira/browse/MAPREDUCE-2461 Project: Hadoop Map/Reduce Issue Type: Bug Affects Versions: 0.23.0 Reporter: Todd Lipcon Fix For: 0.23.0 All of the tests that submit MR jobs are failing on the h7 build machine. This is because the staging directory is entirely full: hudson@h7:/tmp/mr/mr$ ls -l /tmp/hadoop-hudson/mapred/staging/ | wc -l 31999 This makes me think that there's some bug where we're leaking things in the staging directory. I will manually clean this for now, but we should investigate. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (MAPREDUCE-2858) MRv2 WebApp Security
[ https://issues.apache.org/jira/browse/MAPREDUCE-2858?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13126734#comment-13126734 ] Robert Joseph Evans commented on MAPREDUCE-2858: I could try to draw up a MSC or something for you, and yes, it is looking ever more complex, and also potentially very brittle. The issue with just configuring a filter is that there is no guarantee that the AM will honor that, or if the AM is not written in Java there is no way that it could support that (Yes I know because of how the RPC currently works it has to be in Java, but that might change in the future). There is also the possibility that the AM could return JavaScript that will pull out the cookies itself and send them off somewhere. So the proxy has to verify that what is being returned by the server is acceptable. I personally think that we need to move all of the web interfaces on to separate stateless servers that can then communicate with the AM/RM/NM through the already existing RPC. We would have to buff up the RPC to have it return the full set of data that the UI or web servers need. It would also be cool if we could add in some sort of caching to the RPC, so that we don't have to hit the RM every time someone loads up the Applications page. The Pros for this: # Less security issues. We are running trusted code on a trusted server as a trusted user. # All data on the UI is accessible programatically. (No need to ever scrape a web page) # MAPREDUCE-3174 goes away and we can get a more unified user experience (Without proxies of any sort). The Cons for this: # Users cannot change the web interface themselves (But since most M/R jobs only run for a couple of mins I don't think it is that critical) # App Masters that are not trusted do not get a web interface. # Possibly slower because we need an extra hop to get to the data (but if we are going through a proxy it is not really that different) MRv2 WebApp Security Key: MAPREDUCE-2858 URL: https://issues.apache.org/jira/browse/MAPREDUCE-2858 Project: Hadoop Map/Reduce Issue Type: Sub-task Components: applicationmaster, mrv2, security Affects Versions: 0.23.0 Reporter: Luke Lu Assignee: Luke Lu Priority: Blocker Fix For: 0.23.0 In MRv2, while the system servers (ResourceManager (RM), NodeManager (NM) and NameNode (NN)) run as trusted system users, the application masters (AM) run as users who submit the application. While this offers great flexibility to run multiple version of mapreduce frameworks (including their UI) on the same Hadoop cluster, it has significant implication for the security of webapps (Please do not discuss company specific vulnerabilities here). Requirements: # Secure authentication for AM (for app/job level ACLs). # Webapp security should be optional via site configuration. # Support existing pluggable single sign on mechanisms. # Should not require per app/user configuration for deployment. # Should not require special site-wide DNS configuration for deployment. This the top jira for webapp security. A design doc/notes of threat-modeling and counter measures will be posted on the wiki. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (MAPREDUCE-2858) MRv2 WebApp Security
[ https://issues.apache.org/jira/browse/MAPREDUCE-2858?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13126736#comment-13126736 ] Alejandro Abdelnur commented on MAPREDUCE-2858: --- Thanks Rob. Regarding the AM not honoring, that would not be an option as the AM would be started from a wrapper class that setups the SecurityManager and creates the Http Server for the AM to use. Yes, you are right, this would work only for Java. Wouldn't that be enough for a first cut? MRv2 WebApp Security Key: MAPREDUCE-2858 URL: https://issues.apache.org/jira/browse/MAPREDUCE-2858 Project: Hadoop Map/Reduce Issue Type: Sub-task Components: applicationmaster, mrv2, security Affects Versions: 0.23.0 Reporter: Luke Lu Assignee: Luke Lu Priority: Blocker Fix For: 0.23.0 In MRv2, while the system servers (ResourceManager (RM), NodeManager (NM) and NameNode (NN)) run as trusted system users, the application masters (AM) run as users who submit the application. While this offers great flexibility to run multiple version of mapreduce frameworks (including their UI) on the same Hadoop cluster, it has significant implication for the security of webapps (Please do not discuss company specific vulnerabilities here). Requirements: # Secure authentication for AM (for app/job level ACLs). # Webapp security should be optional via site configuration. # Support existing pluggable single sign on mechanisms. # Should not require per app/user configuration for deployment. # Should not require special site-wide DNS configuration for deployment. This the top jira for webapp security. A design doc/notes of threat-modeling and counter measures will be posted on the wiki. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (MAPREDUCE-3177) mapreduce tar layout does not conform new layout
[ https://issues.apache.org/jira/browse/MAPREDUCE-3177?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13126741#comment-13126741 ] Mahadev konar commented on MAPREDUCE-3177: -- Sounds good! mapreduce tar layout does not conform new layout Key: MAPREDUCE-3177 URL: https://issues.apache.org/jira/browse/MAPREDUCE-3177 Project: Hadoop Map/Reduce Issue Type: Bug Components: build Affects Versions: 0.23.0, 0.24.0 Reporter: Alejandro Abdelnur Priority: Critical Fix For: 0.23.0, 0.24.0 The tar generated by MR does not follow the layout of common hdfs, instead, it uses a arbitrary layout which is also different from the old legacy layout (there is a modules/ directory with all the MR jars) -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (MAPREDUCE-2863) Support web-services for RM NM
[ https://issues.apache.org/jira/browse/MAPREDUCE-2863?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13126744#comment-13126744 ] Luke Lu commented on MAPREDUCE-2863: bq. So in conclusion the code sizes seems pretty comparable. You call jersey's having to modify two files adding 100%+ more lines of code besides 3 more annotations (you probably need 4 for the @Context as the jersey snippet you post won't even compile) and an *unsafe* *cast* vs yarn webapp (BTW, it's not Hamlet, which is a strictly a view technology)'s *type* *safe* and concise solution comparable? Since rm and rmcontext is also used in two other methods, you can essentially reduce the particular solution to one line. Wait until you try to write a unit test for the method! The longer time you spent figuring out yarn webapp details is probably because it's a new (and IMO better) way of doing things via Guice, which has extensive documentation. I'll add better docs for yarn webapp framework itself (which is pretty small) RSN :) IMO, yarn webapp is far superior, which is already equivalent or better than the draft JAX-RS 2.0 (still a *special* purpose DI solution), which obsoletes JAX-RS 1.1 that jersey implements, because yarn webapp is backed by a superior *general* purpose DI library: Guice, which is a reference implementation of a (IMO) more useful Java standard (JSR-330). Support web-services for RM NM Key: MAPREDUCE-2863 URL: https://issues.apache.org/jira/browse/MAPREDUCE-2863 Project: Hadoop Map/Reduce Issue Type: Improvement Components: mrv2, nodemanager, resourcemanager Reporter: Arun C Murthy Assignee: Thomas Graves It will be very useful for RM and NM to support web-services to export json/xml. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (MAPREDUCE-2858) MRv2 WebApp Security
[ https://issues.apache.org/jira/browse/MAPREDUCE-2858?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13126746#comment-13126746 ] Allen Wittenauer commented on MAPREDUCE-2858: - bq. Please do not discuss specific vulnerabilities here Too bad. Let's play break the system. User connects to RM which is redirect to the AM through the proxy. The AM has an embedded object such as a flash animation or utilizes a trojan image. (After all, it passes through the white list since it is loaded from my AM). We write the creds we just gained through our hack to some place in HDFS... let's say /tmp. Through an Oozie workflow that has a file watch, I push the cookie/whatever i just gained back to my super secret lair hosting provider. (So even if you block data in and out of the grid, the workflow manager requires access out...) Now I can impersonate all of the Yahoo! employees I want through their cookie auth. I'm fairly convinced that signing is the only way to go without turning the knobs so far up that the AM is pretty useless (no images! no embedded objects! no links off the system! no js at all! ... ) . The cert could actually be verified by the RM to mark AM as trusted or not trusted. MRv2 WebApp Security Key: MAPREDUCE-2858 URL: https://issues.apache.org/jira/browse/MAPREDUCE-2858 Project: Hadoop Map/Reduce Issue Type: Sub-task Components: applicationmaster, mrv2, security Affects Versions: 0.23.0 Reporter: Luke Lu Assignee: Luke Lu Priority: Blocker Fix For: 0.23.0 In MRv2, while the system servers (ResourceManager (RM), NodeManager (NM) and NameNode (NN)) run as trusted system users, the application masters (AM) run as users who submit the application. While this offers great flexibility to run multiple version of mapreduce frameworks (including their UI) on the same Hadoop cluster, it has significant implication for the security of webapps (Please do not discuss company specific vulnerabilities here). Requirements: # Secure authentication for AM (for app/job level ACLs). # Webapp security should be optional via site configuration. # Support existing pluggable single sign on mechanisms. # Should not require per app/user configuration for deployment. # Should not require special site-wide DNS configuration for deployment. This the top jira for webapp security. A design doc/notes of threat-modeling and counter measures will be posted on the wiki. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Updated] (MAPREDUCE-3178) Capacity Schedular shows incorrect cluster information in the RM logs
[ https://issues.apache.org/jira/browse/MAPREDUCE-3178?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Mahadev konar updated MAPREDUCE-3178: - Component/s: (was: contrib/capacity-sched) mrv2 Affects Version/s: (was: 0.24.0) 0.23.0 Capacity Schedular shows incorrect cluster information in the RM logs - Key: MAPREDUCE-3178 URL: https://issues.apache.org/jira/browse/MAPREDUCE-3178 Project: Hadoop Map/Reduce Issue Type: Bug Components: mrv2 Affects Versions: 0.23.0 Reporter: Bhallamudi Venkata Siva Kamesh When we start the NM, after stopping it (in a quick session) CS shows incorrect information about clusterResource in the logs. I have encountered this issue in a pseudo cluster mode and steps to reproduce are 1) start the YARN cluster 2) stop a NM and start the NM again (in a quick session) There should be a NM running in the cluster however as I observed RM detects NM as dead, after default time since its actual unavailability(In this case NM has been stopped). If you start your NM before this time (default time), ResourceTracker throws IOEx, however, CS adds the NM's capacity to the clusterResource. After elapsed time (default time) when RM detects NM as dead, RM removes the NM and hence capacity of the cluster will be subtracted by the amount NM capacity. Eventually there is no NM running in the cluster, but capacity of the cluster is NM's capacity (by default) -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (MAPREDUCE-2858) MRv2 WebApp Security
[ https://issues.apache.org/jira/browse/MAPREDUCE-2858?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13126752#comment-13126752 ] Luke Lu commented on MAPREDUCE-2858: @Allen: since everything go through the proxy the proxy will check any non-relative URLs against a whitelist, so redirect via http or html meta or whatever method is not an issue. I don't think you actually understand how this proxy would work. It authenticates the user and if the user is not the owner of AM, the filtering will happen. MRv2 WebApp Security Key: MAPREDUCE-2858 URL: https://issues.apache.org/jira/browse/MAPREDUCE-2858 Project: Hadoop Map/Reduce Issue Type: Sub-task Components: applicationmaster, mrv2, security Affects Versions: 0.23.0 Reporter: Luke Lu Assignee: Luke Lu Priority: Blocker Fix For: 0.23.0 In MRv2, while the system servers (ResourceManager (RM), NodeManager (NM) and NameNode (NN)) run as trusted system users, the application masters (AM) run as users who submit the application. While this offers great flexibility to run multiple version of mapreduce frameworks (including their UI) on the same Hadoop cluster, it has significant implication for the security of webapps (Please do not discuss company specific vulnerabilities here). Requirements: # Secure authentication for AM (for app/job level ACLs). # Webapp security should be optional via site configuration. # Support existing pluggable single sign on mechanisms. # Should not require per app/user configuration for deployment. # Should not require special site-wide DNS configuration for deployment. This the top jira for webapp security. A design doc/notes of threat-modeling and counter measures will be posted on the wiki. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (MAPREDUCE-2858) MRv2 WebApp Security
[ https://issues.apache.org/jira/browse/MAPREDUCE-2858?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13126757#comment-13126757 ] Luke Lu commented on MAPREDUCE-2858: bq. This is the URL rewriting I was curious about. How did you solve this issue? I just prepended base href=/am:port/ to the output, it's not pretty (not really valid html after proxy) but it works (a la google cache links) :) A better fix would be inserting base href to the right places in the scanner. MRv2 WebApp Security Key: MAPREDUCE-2858 URL: https://issues.apache.org/jira/browse/MAPREDUCE-2858 Project: Hadoop Map/Reduce Issue Type: Sub-task Components: applicationmaster, mrv2, security Affects Versions: 0.23.0 Reporter: Luke Lu Assignee: Luke Lu Priority: Blocker Fix For: 0.23.0 In MRv2, while the system servers (ResourceManager (RM), NodeManager (NM) and NameNode (NN)) run as trusted system users, the application masters (AM) run as users who submit the application. While this offers great flexibility to run multiple version of mapreduce frameworks (including their UI) on the same Hadoop cluster, it has significant implication for the security of webapps (Please do not discuss company specific vulnerabilities here). Requirements: # Secure authentication for AM (for app/job level ACLs). # Webapp security should be optional via site configuration. # Support existing pluggable single sign on mechanisms. # Should not require per app/user configuration for deployment. # Should not require special site-wide DNS configuration for deployment. This the top jira for webapp security. A design doc/notes of threat-modeling and counter measures will be posted on the wiki. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (MAPREDUCE-2858) MRv2 WebApp Security
[ https://issues.apache.org/jira/browse/MAPREDUCE-2858?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13126775#comment-13126775 ] Luke Lu commented on MAPREDUCE-2858: bq. I'm fairly convinced that signing is the only way to go. Signing is pretty much useless in practice due to key management issues besides the hairy implementation issues (need to sign am jar along with all its dependencies and any stuff that gets added to distributed cache): if only certain authorized people can sign, you can guarantee it degenerating into rubber stamping. If any user can sign, it's no different from authentication. The proxy enables testing/explore/experiment by users and is orthogonal to signing if a strict signing process is followed (proxy can bypass signed AM). The proxy will also leave a timestamped auditable trail in the form of logs for each access. MRv2 WebApp Security Key: MAPREDUCE-2858 URL: https://issues.apache.org/jira/browse/MAPREDUCE-2858 Project: Hadoop Map/Reduce Issue Type: Sub-task Components: applicationmaster, mrv2, security Affects Versions: 0.23.0 Reporter: Luke Lu Assignee: Luke Lu Priority: Blocker Fix For: 0.23.0 In MRv2, while the system servers (ResourceManager (RM), NodeManager (NM) and NameNode (NN)) run as trusted system users, the application masters (AM) run as users who submit the application. While this offers great flexibility to run multiple version of mapreduce frameworks (including their UI) on the same Hadoop cluster, it has significant implication for the security of webapps (Please do not discuss company specific vulnerabilities here). Requirements: # Secure authentication for AM (for app/job level ACLs). # Webapp security should be optional via site configuration. # Support existing pluggable single sign on mechanisms. # Should not require per app/user configuration for deployment. # Should not require special site-wide DNS configuration for deployment. This the top jira for webapp security. A design doc/notes of threat-modeling and counter measures will be posted on the wiki. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (MAPREDUCE-2858) MRv2 WebApp Security
[ https://issues.apache.org/jira/browse/MAPREDUCE-2858?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13126793#comment-13126793 ] Allen Wittenauer commented on MAPREDUCE-2858: - bq. It authenticates the user and if the user is not the owner of AM, the filtering will happen. Correct, but with two very big points: a) The only allowed access is via the proxy (no HDFS, etc) b) The filter drops all embedded content. Bye-bye MR SVG graphs. bq. Signing is pretty much useless in practice due to key management issues besides the hairy implementation issues (need to sign am jar along with all its dependencies and any stuff that gets added to distributed cache) Actually, you'd be amazed at the enterprises that actually have effective signing infrastructures in place. But more importantly, you missed a key thing here: you can still allow untrusted code to run. You just mark it as such so that whoever can make the informed decision. This is much better than a black box that gives a false sense of security. MRv2 WebApp Security Key: MAPREDUCE-2858 URL: https://issues.apache.org/jira/browse/MAPREDUCE-2858 Project: Hadoop Map/Reduce Issue Type: Sub-task Components: applicationmaster, mrv2, security Affects Versions: 0.23.0 Reporter: Luke Lu Assignee: Luke Lu Priority: Blocker Fix For: 0.23.0 In MRv2, while the system servers (ResourceManager (RM), NodeManager (NM) and NameNode (NN)) run as trusted system users, the application masters (AM) run as users who submit the application. While this offers great flexibility to run multiple version of mapreduce frameworks (including their UI) on the same Hadoop cluster, it has significant implication for the security of webapps (Please do not discuss company specific vulnerabilities here). Requirements: # Secure authentication for AM (for app/job level ACLs). # Webapp security should be optional via site configuration. # Support existing pluggable single sign on mechanisms. # Should not require per app/user configuration for deployment. # Should not require special site-wide DNS configuration for deployment. This the top jira for webapp security. A design doc/notes of threat-modeling and counter measures will be posted on the wiki. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (MAPREDUCE-2858) MRv2 WebApp Security
[ https://issues.apache.org/jira/browse/MAPREDUCE-2858?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13126804#comment-13126804 ] Luke Lu commented on MAPREDUCE-2858: bq. The only allowed access is via the proxy (no HDFS, etc) This is only for webapp. The usual Hdfs security over kerberos works fine. bq. The filter drops all embedded content. Bye-bye MR SVG graphs. Again, you're commenting without reading the design notes here. The user can use whatever he wants in the webapp. Embedded content is fine for other users as well as long as src/href is relative *and/or* whitelisted. bq. This is much better than a black box that gives a false sense of security. Actually, it's the code signing that is a black box that gives a false sense of security. The proxy don't trust user submitted code per request with mandatory logging. MRv2 WebApp Security Key: MAPREDUCE-2858 URL: https://issues.apache.org/jira/browse/MAPREDUCE-2858 Project: Hadoop Map/Reduce Issue Type: Sub-task Components: applicationmaster, mrv2, security Affects Versions: 0.23.0 Reporter: Luke Lu Assignee: Luke Lu Priority: Blocker Fix For: 0.23.0 In MRv2, while the system servers (ResourceManager (RM), NodeManager (NM) and NameNode (NN)) run as trusted system users, the application masters (AM) run as users who submit the application. While this offers great flexibility to run multiple version of mapreduce frameworks (including their UI) on the same Hadoop cluster, it has significant implication for the security of webapps (Please do not discuss company specific vulnerabilities here). Requirements: # Secure authentication for AM (for app/job level ACLs). # Webapp security should be optional via site configuration. # Support existing pluggable single sign on mechanisms. # Should not require per app/user configuration for deployment. # Should not require special site-wide DNS configuration for deployment. This the top jira for webapp security. A design doc/notes of threat-modeling and counter measures will be posted on the wiki. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Updated] (MAPREDUCE-3179) Incorrect exit code for hadoop-mapreduce-test tests when exception thrown
[ https://issues.apache.org/jira/browse/MAPREDUCE-3179?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Jonathan Eagles updated MAPREDUCE-3179: --- Target Version/s: 0.23.0, 0.24.0 (was: 0.24.0, 0.23.0) Affects Version/s: 0.20.205.0 Incorrect exit code for hadoop-mapreduce-test tests when exception thrown - Key: MAPREDUCE-3179 URL: https://issues.apache.org/jira/browse/MAPREDUCE-3179 Project: Hadoop Map/Reduce Issue Type: Bug Components: mrv2, test Affects Versions: 0.20.205.0, 0.23.0, 0.24.0 Reporter: Jonathan Eagles Assignee: Jonathan Eagles Attachments: MAPREDUCE-3179.patch Exit code for test jar is 0 despite exception thrown hadoop jar hadoop-mapreduce-test-0.23.0-SNAPSHOT.jar loadgen -Dmapreduce.job.acl-view -m 18 -r 0 -outKey org.apache.hadoop.io.Text -outValue org.apache.hadoop.io.Text -indir nonexistentdir Loadgen output snippet org.apache.hadoop.mapred.InvalidInputException: Input path does not exist: hdfs://machine.name.example.com:9000/user/exampleuser/nonexistentdir at org.apache.hadoop.mapred.FileInputFormat.listStatus(FileInputFormat.java:234) at org.apache.hadoop.mapred.FileInputFormat.getSplits(FileInputFormat.java:254) at org.apache.hadoop.mapreduce.JobSubmitter.writeOldSplits(JobSubmitter.java:470) at org.apache.hadoop.mapreduce.JobSubmitter.writeSplits(JobSubmitter.java:462) at org.apache.hadoop.mapreduce.JobSubmitter.submitJobInternal(JobSubmitter.java:358) at org.apache.hadoop.mapreduce.Job$2.run(Job.java:1159) at org.apache.hadoop.mapreduce.Job$2.run(Job.java:1156) at java.security.AccessController.doPrivileged(Native Method) at javax.security.auth.Subject.doAs(Subject.java:396) at org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1135) at org.apache.hadoop.mapreduce.Job.submit(Job.java:1156) at org.apache.hadoop.mapred.JobClient.submitJob(JobClient.java:539) at org.apache.hadoop.mapred.JobClient.runJob(JobClient.java:778) at org.apache.hadoop.mapred.GenericMRLoadGenerator.run(GenericMRLoadGenerator.java:200) at org.apache.hadoop.util.ToolRunner.run(ToolRunner.java:69) at org.apache.hadoop.mapred.GenericMRLoadGenerator.main(GenericMRLoadGenerator.java:214) at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method) at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:39) at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:25) at java.lang.reflect.Method.invoke(Method.java:597) at org.apache.hadoop.util.ProgramDriver$ProgramDescription.invoke(ProgramDriver.java:72) at org.apache.hadoop.util.ProgramDriver.driver(ProgramDriver.java:144) at org.apache.hadoop.test.MapredTestDriver.run(MapredTestDriver.java:111) at org.apache.hadoop.test.MapredTestDriver.main(MapredTestDriver.java:118) at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method) at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:39) at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:25) at java.lang.reflect.Method.invoke(Method.java:597) at org.apache.hadoop.util.RunJar.main(RunJar.java:189) -bash-3.2$ echo $? 0 This differs from example jar which correctly returns the correct exit code hadoop jar hadoop-mapreduce-examples-0.23.0-SNAPSHOT.jar wordcount nonexistentdir /outputdir wordcount output snippet org.apache.hadoop.mapreduce.lib.input.InvalidInputException: Input path does not exist: hdfs://machine.name.example.com:9000/user/exampleuser/nonexistentdir at org.apache.hadoop.mapreduce.lib.input.FileInputFormat.listStatus(FileInputFormat.java:243) at org.apache.hadoop.mapreduce.lib.input.FileInputFormat.getSplits(FileInputFormat.java:269) at org.apache.hadoop.mapreduce.JobSubmitter.writeNewSplits(JobSubmitter.java:443) at org.apache.hadoop.mapreduce.JobSubmitter.writeSplits(JobSubmitter.java:460) at org.apache.hadoop.mapreduce.JobSubmitter.submitJobInternal(JobSubmitter.java:358) at org.apache.hadoop.mapreduce.Job$2.run(Job.java:1159) at org.apache.hadoop.mapreduce.Job$2.run(Job.java:1156) at java.security.AccessController.doPrivileged(Native Method) at javax.security.auth.Subject.doAs(Subject.java:396) at org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1135) at org.apache.hadoop.mapreduce.Job.submit(Job.java:1156) at
[jira] [Updated] (MAPREDUCE-3179) Incorrect exit code for hadoop-mapreduce-test tests when exception thrown
[ https://issues.apache.org/jira/browse/MAPREDUCE-3179?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Jonathan Eagles updated MAPREDUCE-3179: --- Attachment: MAPREDUCE-3179-branch-0.20-security.patch Incorrect exit code for hadoop-mapreduce-test tests when exception thrown - Key: MAPREDUCE-3179 URL: https://issues.apache.org/jira/browse/MAPREDUCE-3179 Project: Hadoop Map/Reduce Issue Type: Bug Components: mrv2, test Affects Versions: 0.20.205.0, 0.23.0, 0.24.0 Reporter: Jonathan Eagles Assignee: Jonathan Eagles Attachments: MAPREDUCE-3179-branch-0.20-security.patch, MAPREDUCE-3179.patch Exit code for test jar is 0 despite exception thrown hadoop jar hadoop-mapreduce-test-0.23.0-SNAPSHOT.jar loadgen -Dmapreduce.job.acl-view -m 18 -r 0 -outKey org.apache.hadoop.io.Text -outValue org.apache.hadoop.io.Text -indir nonexistentdir Loadgen output snippet org.apache.hadoop.mapred.InvalidInputException: Input path does not exist: hdfs://machine.name.example.com:9000/user/exampleuser/nonexistentdir at org.apache.hadoop.mapred.FileInputFormat.listStatus(FileInputFormat.java:234) at org.apache.hadoop.mapred.FileInputFormat.getSplits(FileInputFormat.java:254) at org.apache.hadoop.mapreduce.JobSubmitter.writeOldSplits(JobSubmitter.java:470) at org.apache.hadoop.mapreduce.JobSubmitter.writeSplits(JobSubmitter.java:462) at org.apache.hadoop.mapreduce.JobSubmitter.submitJobInternal(JobSubmitter.java:358) at org.apache.hadoop.mapreduce.Job$2.run(Job.java:1159) at org.apache.hadoop.mapreduce.Job$2.run(Job.java:1156) at java.security.AccessController.doPrivileged(Native Method) at javax.security.auth.Subject.doAs(Subject.java:396) at org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1135) at org.apache.hadoop.mapreduce.Job.submit(Job.java:1156) at org.apache.hadoop.mapred.JobClient.submitJob(JobClient.java:539) at org.apache.hadoop.mapred.JobClient.runJob(JobClient.java:778) at org.apache.hadoop.mapred.GenericMRLoadGenerator.run(GenericMRLoadGenerator.java:200) at org.apache.hadoop.util.ToolRunner.run(ToolRunner.java:69) at org.apache.hadoop.mapred.GenericMRLoadGenerator.main(GenericMRLoadGenerator.java:214) at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method) at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:39) at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:25) at java.lang.reflect.Method.invoke(Method.java:597) at org.apache.hadoop.util.ProgramDriver$ProgramDescription.invoke(ProgramDriver.java:72) at org.apache.hadoop.util.ProgramDriver.driver(ProgramDriver.java:144) at org.apache.hadoop.test.MapredTestDriver.run(MapredTestDriver.java:111) at org.apache.hadoop.test.MapredTestDriver.main(MapredTestDriver.java:118) at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method) at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:39) at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:25) at java.lang.reflect.Method.invoke(Method.java:597) at org.apache.hadoop.util.RunJar.main(RunJar.java:189) -bash-3.2$ echo $? 0 This differs from example jar which correctly returns the correct exit code hadoop jar hadoop-mapreduce-examples-0.23.0-SNAPSHOT.jar wordcount nonexistentdir /outputdir wordcount output snippet org.apache.hadoop.mapreduce.lib.input.InvalidInputException: Input path does not exist: hdfs://machine.name.example.com:9000/user/exampleuser/nonexistentdir at org.apache.hadoop.mapreduce.lib.input.FileInputFormat.listStatus(FileInputFormat.java:243) at org.apache.hadoop.mapreduce.lib.input.FileInputFormat.getSplits(FileInputFormat.java:269) at org.apache.hadoop.mapreduce.JobSubmitter.writeNewSplits(JobSubmitter.java:443) at org.apache.hadoop.mapreduce.JobSubmitter.writeSplits(JobSubmitter.java:460) at org.apache.hadoop.mapreduce.JobSubmitter.submitJobInternal(JobSubmitter.java:358) at org.apache.hadoop.mapreduce.Job$2.run(Job.java:1159) at org.apache.hadoop.mapreduce.Job$2.run(Job.java:1156) at java.security.AccessController.doPrivileged(Native Method) at javax.security.auth.Subject.doAs(Subject.java:396) at org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1135) at org.apache.hadoop.mapreduce.Job.submit(Job.java:1156) at
[jira] [Commented] (MAPREDUCE-3179) Incorrect exit code for hadoop-mapreduce-test tests when exception thrown
[ https://issues.apache.org/jira/browse/MAPREDUCE-3179?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13126837#comment-13126837 ] Jonathan Eagles commented on MAPREDUCE-3179: I have uploaded a patch for branch 20 security in case that is needed as well. Currently I have only targeted 23 and 24. Incorrect exit code for hadoop-mapreduce-test tests when exception thrown - Key: MAPREDUCE-3179 URL: https://issues.apache.org/jira/browse/MAPREDUCE-3179 Project: Hadoop Map/Reduce Issue Type: Bug Components: mrv2, test Affects Versions: 0.20.205.0, 0.23.0, 0.24.0 Reporter: Jonathan Eagles Assignee: Jonathan Eagles Attachments: MAPREDUCE-3179-branch-0.20-security.patch, MAPREDUCE-3179.patch Exit code for test jar is 0 despite exception thrown hadoop jar hadoop-mapreduce-test-0.23.0-SNAPSHOT.jar loadgen -Dmapreduce.job.acl-view -m 18 -r 0 -outKey org.apache.hadoop.io.Text -outValue org.apache.hadoop.io.Text -indir nonexistentdir Loadgen output snippet org.apache.hadoop.mapred.InvalidInputException: Input path does not exist: hdfs://machine.name.example.com:9000/user/exampleuser/nonexistentdir at org.apache.hadoop.mapred.FileInputFormat.listStatus(FileInputFormat.java:234) at org.apache.hadoop.mapred.FileInputFormat.getSplits(FileInputFormat.java:254) at org.apache.hadoop.mapreduce.JobSubmitter.writeOldSplits(JobSubmitter.java:470) at org.apache.hadoop.mapreduce.JobSubmitter.writeSplits(JobSubmitter.java:462) at org.apache.hadoop.mapreduce.JobSubmitter.submitJobInternal(JobSubmitter.java:358) at org.apache.hadoop.mapreduce.Job$2.run(Job.java:1159) at org.apache.hadoop.mapreduce.Job$2.run(Job.java:1156) at java.security.AccessController.doPrivileged(Native Method) at javax.security.auth.Subject.doAs(Subject.java:396) at org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1135) at org.apache.hadoop.mapreduce.Job.submit(Job.java:1156) at org.apache.hadoop.mapred.JobClient.submitJob(JobClient.java:539) at org.apache.hadoop.mapred.JobClient.runJob(JobClient.java:778) at org.apache.hadoop.mapred.GenericMRLoadGenerator.run(GenericMRLoadGenerator.java:200) at org.apache.hadoop.util.ToolRunner.run(ToolRunner.java:69) at org.apache.hadoop.mapred.GenericMRLoadGenerator.main(GenericMRLoadGenerator.java:214) at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method) at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:39) at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:25) at java.lang.reflect.Method.invoke(Method.java:597) at org.apache.hadoop.util.ProgramDriver$ProgramDescription.invoke(ProgramDriver.java:72) at org.apache.hadoop.util.ProgramDriver.driver(ProgramDriver.java:144) at org.apache.hadoop.test.MapredTestDriver.run(MapredTestDriver.java:111) at org.apache.hadoop.test.MapredTestDriver.main(MapredTestDriver.java:118) at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method) at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:39) at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:25) at java.lang.reflect.Method.invoke(Method.java:597) at org.apache.hadoop.util.RunJar.main(RunJar.java:189) -bash-3.2$ echo $? 0 This differs from example jar which correctly returns the correct exit code hadoop jar hadoop-mapreduce-examples-0.23.0-SNAPSHOT.jar wordcount nonexistentdir /outputdir wordcount output snippet org.apache.hadoop.mapreduce.lib.input.InvalidInputException: Input path does not exist: hdfs://machine.name.example.com:9000/user/exampleuser/nonexistentdir at org.apache.hadoop.mapreduce.lib.input.FileInputFormat.listStatus(FileInputFormat.java:243) at org.apache.hadoop.mapreduce.lib.input.FileInputFormat.getSplits(FileInputFormat.java:269) at org.apache.hadoop.mapreduce.JobSubmitter.writeNewSplits(JobSubmitter.java:443) at org.apache.hadoop.mapreduce.JobSubmitter.writeSplits(JobSubmitter.java:460) at org.apache.hadoop.mapreduce.JobSubmitter.submitJobInternal(JobSubmitter.java:358) at org.apache.hadoop.mapreduce.Job$2.run(Job.java:1159) at org.apache.hadoop.mapreduce.Job$2.run(Job.java:1156) at java.security.AccessController.doPrivileged(Native Method) at javax.security.auth.Subject.doAs(Subject.java:396) at
[jira] [Commented] (MAPREDUCE-3179) Incorrect exit code for hadoop-mapreduce-test tests when exception thrown
[ https://issues.apache.org/jira/browse/MAPREDUCE-3179?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13126845#comment-13126845 ] Hadoop QA commented on MAPREDUCE-3179: -- -1 overall. Here are the results of testing the latest attachment http://issues.apache.org/jira/secure/attachment/12498912/MAPREDUCE-3179-branch-0.20-security.patch against trunk revision . +1 @author. The patch does not contain any @author tags. +1 tests included. The patch appears to include 3 new or modified tests. -1 patch. The patch command could not apply the patch. Console output: https://builds.apache.org/job/PreCommit-MAPREDUCE-Build/1007//console This message is automatically generated. Incorrect exit code for hadoop-mapreduce-test tests when exception thrown - Key: MAPREDUCE-3179 URL: https://issues.apache.org/jira/browse/MAPREDUCE-3179 Project: Hadoop Map/Reduce Issue Type: Bug Components: mrv2, test Affects Versions: 0.20.205.0, 0.23.0, 0.24.0 Reporter: Jonathan Eagles Assignee: Jonathan Eagles Attachments: MAPREDUCE-3179-branch-0.20-security.patch, MAPREDUCE-3179.patch Exit code for test jar is 0 despite exception thrown hadoop jar hadoop-mapreduce-test-0.23.0-SNAPSHOT.jar loadgen -Dmapreduce.job.acl-view -m 18 -r 0 -outKey org.apache.hadoop.io.Text -outValue org.apache.hadoop.io.Text -indir nonexistentdir Loadgen output snippet org.apache.hadoop.mapred.InvalidInputException: Input path does not exist: hdfs://machine.name.example.com:9000/user/exampleuser/nonexistentdir at org.apache.hadoop.mapred.FileInputFormat.listStatus(FileInputFormat.java:234) at org.apache.hadoop.mapred.FileInputFormat.getSplits(FileInputFormat.java:254) at org.apache.hadoop.mapreduce.JobSubmitter.writeOldSplits(JobSubmitter.java:470) at org.apache.hadoop.mapreduce.JobSubmitter.writeSplits(JobSubmitter.java:462) at org.apache.hadoop.mapreduce.JobSubmitter.submitJobInternal(JobSubmitter.java:358) at org.apache.hadoop.mapreduce.Job$2.run(Job.java:1159) at org.apache.hadoop.mapreduce.Job$2.run(Job.java:1156) at java.security.AccessController.doPrivileged(Native Method) at javax.security.auth.Subject.doAs(Subject.java:396) at org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1135) at org.apache.hadoop.mapreduce.Job.submit(Job.java:1156) at org.apache.hadoop.mapred.JobClient.submitJob(JobClient.java:539) at org.apache.hadoop.mapred.JobClient.runJob(JobClient.java:778) at org.apache.hadoop.mapred.GenericMRLoadGenerator.run(GenericMRLoadGenerator.java:200) at org.apache.hadoop.util.ToolRunner.run(ToolRunner.java:69) at org.apache.hadoop.mapred.GenericMRLoadGenerator.main(GenericMRLoadGenerator.java:214) at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method) at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:39) at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:25) at java.lang.reflect.Method.invoke(Method.java:597) at org.apache.hadoop.util.ProgramDriver$ProgramDescription.invoke(ProgramDriver.java:72) at org.apache.hadoop.util.ProgramDriver.driver(ProgramDriver.java:144) at org.apache.hadoop.test.MapredTestDriver.run(MapredTestDriver.java:111) at org.apache.hadoop.test.MapredTestDriver.main(MapredTestDriver.java:118) at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method) at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:39) at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:25) at java.lang.reflect.Method.invoke(Method.java:597) at org.apache.hadoop.util.RunJar.main(RunJar.java:189) -bash-3.2$ echo $? 0 This differs from example jar which correctly returns the correct exit code hadoop jar hadoop-mapreduce-examples-0.23.0-SNAPSHOT.jar wordcount nonexistentdir /outputdir wordcount output snippet org.apache.hadoop.mapreduce.lib.input.InvalidInputException: Input path does not exist: hdfs://machine.name.example.com:9000/user/exampleuser/nonexistentdir at org.apache.hadoop.mapreduce.lib.input.FileInputFormat.listStatus(FileInputFormat.java:243) at org.apache.hadoop.mapreduce.lib.input.FileInputFormat.getSplits(FileInputFormat.java:269) at org.apache.hadoop.mapreduce.JobSubmitter.writeNewSplits(JobSubmitter.java:443) at org.apache.hadoop.mapreduce.JobSubmitter.writeSplits(JobSubmitter.java:460) at
[jira] [Commented] (MAPREDUCE-3179) Incorrect exit code for hadoop-mapreduce-test tests when exception thrown
[
https://issues.apache.org/jira/browse/MAPREDUCE-3179?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13126859#comment-13126859
]
Jonathan Eagles commented on MAPREDUCE-3179:
Investigation of ProgramDriver usage reveals two more instances of this bug. I
will file corresponding JIRAs for common and hdfs to address those issues.
-bash-3.2$ ack --java ProgramDriver
hadoop-common-project/hadoop-common/src/main/java/org/apache/hadoop/util/ProgramDriver.java
33:public class ProgramDriver {
42: public ProgramDriver(){
hadoop-common-project/hadoop-common/src/test/java/org/apache/hadoop/test/CoreTestDriver.java
25:import org.apache.hadoop.util.ProgramDriver;
32: private ProgramDriver pgd;
35:this(new ProgramDriver());
38: public CoreTestDriver(ProgramDriver pgd) {
hadoop-hdfs-project/hadoop-hdfs/src/test/java/org/apache/hadoop/test/HdfsTestDriver.java
22:import org.apache.hadoop.util.ProgramDriver;
29: private ProgramDriver pgd;
32:this(new ProgramDriver());
35: public HdfsTestDriver(ProgramDriver pgd) {
hadoop-mapreduce-project/src/test/mapred/org/apache/hadoop/test/MapredTestDriver.java
33:import org.apache.hadoop.util.ProgramDriver;
50: private ProgramDriver pgd;
53:this(new ProgramDriver());
56: public MapredTestDriver(ProgramDriver pgd) {
hadoop-mapreduce-project/src/examples/org/apache/hadoop/examples/ExampleDriver.java
27:import org.apache.hadoop.util.ProgramDriver;
37:ProgramDriver pgd = new ProgramDriver();
Incorrect exit code for hadoop-mapreduce-test tests when exception thrown
-
Key: MAPREDUCE-3179
URL: https://issues.apache.org/jira/browse/MAPREDUCE-3179
Project: Hadoop Map/Reduce
Issue Type: Bug
Components: mrv2, test
Affects Versions: 0.20.205.0, 0.23.0, 0.24.0
Reporter: Jonathan Eagles
Assignee: Jonathan Eagles
Attachments: MAPREDUCE-3179-branch-0.20-security.patch,
MAPREDUCE-3179.patch
Exit code for test jar is 0 despite exception thrown
hadoop jar hadoop-mapreduce-test-0.23.0-SNAPSHOT.jar loadgen
-Dmapreduce.job.acl-view -m 18 -r 0 -outKey org.apache.hadoop.io.Text
-outValue org.apache.hadoop.io.Text -indir nonexistentdir
Loadgen output snippet
org.apache.hadoop.mapred.InvalidInputException: Input path does not exist:
hdfs://machine.name.example.com:9000/user/exampleuser/nonexistentdir
at
org.apache.hadoop.mapred.FileInputFormat.listStatus(FileInputFormat.java:234)
at
org.apache.hadoop.mapred.FileInputFormat.getSplits(FileInputFormat.java:254)
at
org.apache.hadoop.mapreduce.JobSubmitter.writeOldSplits(JobSubmitter.java:470)
at
org.apache.hadoop.mapreduce.JobSubmitter.writeSplits(JobSubmitter.java:462)
at
org.apache.hadoop.mapreduce.JobSubmitter.submitJobInternal(JobSubmitter.java:358)
at org.apache.hadoop.mapreduce.Job$2.run(Job.java:1159)
at org.apache.hadoop.mapreduce.Job$2.run(Job.java:1156)
at java.security.AccessController.doPrivileged(Native Method)
at javax.security.auth.Subject.doAs(Subject.java:396)
at
org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1135)
at org.apache.hadoop.mapreduce.Job.submit(Job.java:1156)
at org.apache.hadoop.mapred.JobClient.submitJob(JobClient.java:539)
at org.apache.hadoop.mapred.JobClient.runJob(JobClient.java:778)
at
org.apache.hadoop.mapred.GenericMRLoadGenerator.run(GenericMRLoadGenerator.java:200)
at org.apache.hadoop.util.ToolRunner.run(ToolRunner.java:69)
at
org.apache.hadoop.mapred.GenericMRLoadGenerator.main(GenericMRLoadGenerator.java:214)
at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
at
sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:39)
at
sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:25)
at java.lang.reflect.Method.invoke(Method.java:597)
at
org.apache.hadoop.util.ProgramDriver$ProgramDescription.invoke(ProgramDriver.java:72)
at org.apache.hadoop.util.ProgramDriver.driver(ProgramDriver.java:144)
at
org.apache.hadoop.test.MapredTestDriver.run(MapredTestDriver.java:111)
at
org.apache.hadoop.test.MapredTestDriver.main(MapredTestDriver.java:118)
at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
at
sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:39)
at
sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:25)
at java.lang.reflect.Method.invoke(Method.java:597)
at org.apache.hadoop.util.RunJar.main(RunJar.java:189)
-bash-3.2$ echo $?
0
This differs from example jar
[jira] [Updated] (MAPREDUCE-2905) CapBasedLoadManager incorrectly allows assignment when assignMultiple is true (was: assignmultiple per job)
[ https://issues.apache.org/jira/browse/MAPREDUCE-2905?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Jeff Bean updated MAPREDUCE-2905: - Attachment: MR-2905.10-13-2011 Unit test included. Unit test found typo, which was fixed (assign map assign reduce, whatever) CapBasedLoadManager incorrectly allows assignment when assignMultiple is true (was: assignmultiple per job) --- Key: MAPREDUCE-2905 URL: https://issues.apache.org/jira/browse/MAPREDUCE-2905 Project: Hadoop Map/Reduce Issue Type: Bug Components: contrib/fair-share Affects Versions: 0.20.2 Reporter: Jeff Bean Attachments: MR-2905.10-13-2011, MR-2905.patch, MR-2905.patch.2 We encountered a situation where in the same cluster, large jobs benefit from mapred.fairscheduler.assignmultiple, but small jobs with small numbers of mappers do not: the mappers all clump to fully occupy just a few nodes, which causes those nodes to saturate and bottleneck. The desired behavior is to spread the job across more nodes so that a relatively small job doesn't saturate any node in the cluster. Testing has shown that setting mapred.fairscheduler.assignmultiple to false gives the desired behavior for small jobs, but is unnecessary for large jobs. However, since this is a cluster-wide setting, we can't properly tune. It'd be nice if jobs can set a param similar to mapred.fairscheduler.assignmultiple on submission to better control the task distribution of a particular job. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Updated] (MAPREDUCE-2905) CapBasedLoadManager incorrectly allows assignment when assignMultiple is true (was: assignmultiple per job)
[ https://issues.apache.org/jira/browse/MAPREDUCE-2905?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Jeff Bean updated MAPREDUCE-2905: - Attachment: screenshot-1.jpg Unit test failure exposes the issue. When assignmultiple is true, a load manager might be asked to assign 3 maps in a loop, and it allows all of them. CapBasedLoadManager incorrectly allows assignment when assignMultiple is true (was: assignmultiple per job) --- Key: MAPREDUCE-2905 URL: https://issues.apache.org/jira/browse/MAPREDUCE-2905 Project: Hadoop Map/Reduce Issue Type: Bug Components: contrib/fair-share Affects Versions: 0.20.2 Reporter: Jeff Bean Attachments: MR-2905.10-13-2011, MR-2905.patch, MR-2905.patch.2, screenshot-1.jpg We encountered a situation where in the same cluster, large jobs benefit from mapred.fairscheduler.assignmultiple, but small jobs with small numbers of mappers do not: the mappers all clump to fully occupy just a few nodes, which causes those nodes to saturate and bottleneck. The desired behavior is to spread the job across more nodes so that a relatively small job doesn't saturate any node in the cluster. Testing has shown that setting mapred.fairscheduler.assignmultiple to false gives the desired behavior for small jobs, but is unnecessary for large jobs. However, since this is a cluster-wide setting, we can't properly tune. It'd be nice if jobs can set a param similar to mapred.fairscheduler.assignmultiple on submission to better control the task distribution of a particular job. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Updated] (MAPREDUCE-3179) Incorrect exit code for hadoop-mapreduce-test tests when exception thrown
[ https://issues.apache.org/jira/browse/MAPREDUCE-3179?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Jonathan Eagles updated MAPREDUCE-3179: --- Attachment: (was: MAPREDUCE-3179-branch-0.20-security.patch) Incorrect exit code for hadoop-mapreduce-test tests when exception thrown - Key: MAPREDUCE-3179 URL: https://issues.apache.org/jira/browse/MAPREDUCE-3179 Project: Hadoop Map/Reduce Issue Type: Bug Components: mrv2, test Affects Versions: 0.20.205.0, 0.23.0, 0.24.0 Reporter: Jonathan Eagles Assignee: Jonathan Eagles Attachments: MAPREDUCE-3179.patch Exit code for test jar is 0 despite exception thrown hadoop jar hadoop-mapreduce-test-0.23.0-SNAPSHOT.jar loadgen -Dmapreduce.job.acl-view -m 18 -r 0 -outKey org.apache.hadoop.io.Text -outValue org.apache.hadoop.io.Text -indir nonexistentdir Loadgen output snippet org.apache.hadoop.mapred.InvalidInputException: Input path does not exist: hdfs://machine.name.example.com:9000/user/exampleuser/nonexistentdir at org.apache.hadoop.mapred.FileInputFormat.listStatus(FileInputFormat.java:234) at org.apache.hadoop.mapred.FileInputFormat.getSplits(FileInputFormat.java:254) at org.apache.hadoop.mapreduce.JobSubmitter.writeOldSplits(JobSubmitter.java:470) at org.apache.hadoop.mapreduce.JobSubmitter.writeSplits(JobSubmitter.java:462) at org.apache.hadoop.mapreduce.JobSubmitter.submitJobInternal(JobSubmitter.java:358) at org.apache.hadoop.mapreduce.Job$2.run(Job.java:1159) at org.apache.hadoop.mapreduce.Job$2.run(Job.java:1156) at java.security.AccessController.doPrivileged(Native Method) at javax.security.auth.Subject.doAs(Subject.java:396) at org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1135) at org.apache.hadoop.mapreduce.Job.submit(Job.java:1156) at org.apache.hadoop.mapred.JobClient.submitJob(JobClient.java:539) at org.apache.hadoop.mapred.JobClient.runJob(JobClient.java:778) at org.apache.hadoop.mapred.GenericMRLoadGenerator.run(GenericMRLoadGenerator.java:200) at org.apache.hadoop.util.ToolRunner.run(ToolRunner.java:69) at org.apache.hadoop.mapred.GenericMRLoadGenerator.main(GenericMRLoadGenerator.java:214) at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method) at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:39) at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:25) at java.lang.reflect.Method.invoke(Method.java:597) at org.apache.hadoop.util.ProgramDriver$ProgramDescription.invoke(ProgramDriver.java:72) at org.apache.hadoop.util.ProgramDriver.driver(ProgramDriver.java:144) at org.apache.hadoop.test.MapredTestDriver.run(MapredTestDriver.java:111) at org.apache.hadoop.test.MapredTestDriver.main(MapredTestDriver.java:118) at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method) at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:39) at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:25) at java.lang.reflect.Method.invoke(Method.java:597) at org.apache.hadoop.util.RunJar.main(RunJar.java:189) -bash-3.2$ echo $? 0 This differs from example jar which correctly returns the correct exit code hadoop jar hadoop-mapreduce-examples-0.23.0-SNAPSHOT.jar wordcount nonexistentdir /outputdir wordcount output snippet org.apache.hadoop.mapreduce.lib.input.InvalidInputException: Input path does not exist: hdfs://machine.name.example.com:9000/user/exampleuser/nonexistentdir at org.apache.hadoop.mapreduce.lib.input.FileInputFormat.listStatus(FileInputFormat.java:243) at org.apache.hadoop.mapreduce.lib.input.FileInputFormat.getSplits(FileInputFormat.java:269) at org.apache.hadoop.mapreduce.JobSubmitter.writeNewSplits(JobSubmitter.java:443) at org.apache.hadoop.mapreduce.JobSubmitter.writeSplits(JobSubmitter.java:460) at org.apache.hadoop.mapreduce.JobSubmitter.submitJobInternal(JobSubmitter.java:358) at org.apache.hadoop.mapreduce.Job$2.run(Job.java:1159) at org.apache.hadoop.mapreduce.Job$2.run(Job.java:1156) at java.security.AccessController.doPrivileged(Native Method) at javax.security.auth.Subject.doAs(Subject.java:396) at org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1135) at org.apache.hadoop.mapreduce.Job.submit(Job.java:1156) at org.apache.hadoop.mapreduce.Job.waitForCompletion(Job.java:1176)
[jira] [Updated] (MAPREDUCE-3163) JobClient spews errors when killing MR2 job
[ https://issues.apache.org/jira/browse/MAPREDUCE-3163?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Mahadev konar updated MAPREDUCE-3163: - Priority: Blocker (was: Critical) Seeing this too often. Will upload a fix soon. JobClient spews errors when killing MR2 job --- Key: MAPREDUCE-3163 URL: https://issues.apache.org/jira/browse/MAPREDUCE-3163 Project: Hadoop Map/Reduce Issue Type: Bug Components: job submission, mrv2 Affects Versions: 0.23.0 Reporter: Todd Lipcon Assignee: Mahadev konar Priority: Blocker Attachments: log.txt When I used the hadoop job command line to kill a running MR2 job, I got a bunch of error spew on the console, despite the kill actually taking effect. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Updated] (MAPREDUCE-3144) Augment JobHistory to include information needed for serving aggregated logs.
[ https://issues.apache.org/jira/browse/MAPREDUCE-3144?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Siddharth Seth updated MAPREDUCE-3144: -- Status: Patch Available (was: Open) Augment JobHistory to include information needed for serving aggregated logs. - Key: MAPREDUCE-3144 URL: https://issues.apache.org/jira/browse/MAPREDUCE-3144 Project: Hadoop Map/Reduce Issue Type: Sub-task Components: mrv2 Affects Versions: 0.23.0 Reporter: Vinod Kumar Vavilapalli Assignee: Siddharth Seth Fix For: 0.23.0 Attachments: MR3144.patch, MR3144_v2.txt -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Updated] (MAPREDUCE-3144) Augment JobHistory to include information needed for serving aggregated logs.
[ https://issues.apache.org/jira/browse/MAPREDUCE-3144?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Siddharth Seth updated MAPREDUCE-3144: -- Attachment: MR3144_v2.txt Updated patch. Adds - AM info being logged to history. - getAMInfos API in Job - to be used later for the UI. Augment JobHistory to include information needed for serving aggregated logs. - Key: MAPREDUCE-3144 URL: https://issues.apache.org/jira/browse/MAPREDUCE-3144 Project: Hadoop Map/Reduce Issue Type: Sub-task Components: mrv2 Affects Versions: 0.23.0 Reporter: Vinod Kumar Vavilapalli Assignee: Siddharth Seth Fix For: 0.23.0 Attachments: MR3144.patch, MR3144_v2.txt -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Created] (MAPREDUCE-3180) TaskTracker.java.orig accidentally checked in to 0.20-security-205
TaskTracker.java.orig accidentally checked in to 0.20-security-205 -- Key: MAPREDUCE-3180 URL: https://issues.apache.org/jira/browse/MAPREDUCE-3180 Project: Hadoop Map/Reduce Issue Type: Bug Affects Versions: 0.20.205.0 Reporter: Matt Foley Priority: Trivial The file src/mapred/org/apache/hadoop/mapred/TaskTracker.java.orig was accidentally checked in as part of r1179465. It is only in 0.20-security-205, not 0.20-security. If there is a 0.20.205.1, remove it then. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Assigned] (MAPREDUCE-3180) TaskTracker.java.orig accidentally checked in to 0.20-security-205
[ https://issues.apache.org/jira/browse/MAPREDUCE-3180?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Matt Foley reassigned MAPREDUCE-3180: - Assignee: Matt Foley TaskTracker.java.orig accidentally checked in to 0.20-security-205 -- Key: MAPREDUCE-3180 URL: https://issues.apache.org/jira/browse/MAPREDUCE-3180 Project: Hadoop Map/Reduce Issue Type: Bug Affects Versions: 0.20.205.0 Reporter: Matt Foley Assignee: Matt Foley Priority: Trivial The file src/mapred/org/apache/hadoop/mapred/TaskTracker.java.orig was accidentally checked in as part of r1179465. It is only in 0.20-security-205, not 0.20-security. If there is a 0.20.205.1, remove it then. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Created] (MAPREDUCE-3181) Terasort fails with Kerberos exception on secure cluster
Terasort fails with Kerberos exception on secure cluster Key: MAPREDUCE-3181 URL: https://issues.apache.org/jira/browse/MAPREDUCE-3181 Project: Hadoop Map/Reduce Issue Type: Bug Components: mrv2 Affects Versions: 0.23.0 Reporter: Anupam Seth Assignee: Anupam Seth We are seeing the following Kerberos exception upon trying to run terasort on secure single and multi-node clusters using the latest build from branch 0.23. java.io.IOException: Can't get JobTracker Kerberos principal for use as renewer at org.apache.hadoop.mapreduce.security.TokenCache.obtainTokensForNamenodesInternal(TokenCache.java:106) at org.apache.hadoop.mapreduce.security.TokenCache.obtainTokensForNamenodesInternal(TokenCache.java:90) at org.apache.hadoop.mapreduce.security.TokenCache.obtainTokensForNamenodes(TokenCache.java:83) at org.apache.hadoop.mapreduce.lib.input.FileInputFormat.listStatus(FileInputFormat.java:205) at org.apache.hadoop.mapreduce.lib.input.FileInputFormat.getSplits(FileInputFormat.java:269) at org.apache.hadoop.examples.terasort.TeraInputFormat.getSplits(TeraInputFormat.java:318) at org.apache.hadoop.examples.terasort.TeraInputFormat.writePartitionFile(TeraInputFormat.java:169) at org.apache.hadoop.examples.terasort.TeraSort.run(TeraSort.java:306) at org.apache.hadoop.util.ToolRunner.run(ToolRunner.java:69) at org.apache.hadoop.examples.terasort.TeraSort.main(TeraSort.java:325) at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method) at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:39) at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:25) at java.lang.reflect.Method.invoke(Method.java:597) at org.apache.hadoop.util.ProgramDriver$ProgramDescription.invoke(ProgramDriver.java:72) at org.apache.hadoop.util.ProgramDriver.driver(ProgramDriver.java:144) at org.apache.hadoop.examples.ExampleDriver.main(ExampleDriver.java:68) at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method) at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:39) at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:25) at java.lang.reflect.Method.invoke(Method.java:597) at org.apache.hadoop.util.RunJar.main(RunJar.java:189) Adding debug output shows that the job configuration is not loading up yarn-site.xml causing the above failure to happen. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (MAPREDUCE-2858) MRv2 WebApp Security
[
https://issues.apache.org/jira/browse/MAPREDUCE-2858?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13126941#comment-13126941
]
Robert Joseph Evans commented on MAPREDUCE-2858:
@Luke
I must be doing something wrong with the base tag, because it is not working
for me, and I am not really sure how it would work. The issue I am having is
the following.
The page has a reference to the following image
{code}
img src=/static/hadoop-st.png
{code}
I am not sure what I would insert into a base tag that would cause it to load
from
{code}http://myproxy.com:port/proxy/orighost:origport/static/hadoop-st.png{code}
it always wants to load from
{code}http://myproxy.com:port/static/hadoop-st.png{code} no matter what I put
in for base. This is on both Chrome and Firefox.
From what I have read BASE is used to take a moved page and point the
references back to the original location. At least that is how the google
cache uses it. The href is supposed to point to a fully qualified URI that is
used in resolving partial URIs as if that were the page that we were currently
on.
Could you give me a more concrete example of what the base tag should look like
to support this? The only thing that I can come up with to do this is
{code}base href=http://orighost:origport/path/to/page/{code}
but that would bypass the proxy for all links, which seems broken to me.
MRv2 WebApp Security
Key: MAPREDUCE-2858
URL: https://issues.apache.org/jira/browse/MAPREDUCE-2858
Project: Hadoop Map/Reduce
Issue Type: Sub-task
Components: applicationmaster, mrv2, security
Affects Versions: 0.23.0
Reporter: Luke Lu
Assignee: Luke Lu
Priority: Blocker
Fix For: 0.23.0
In MRv2, while the system servers (ResourceManager (RM), NodeManager (NM) and
NameNode (NN)) run as trusted
system users, the application masters (AM) run as users who submit the
application. While this offers great flexibility
to run multiple version of mapreduce frameworks (including their UI) on the
same Hadoop cluster, it has significant
implication for the security of webapps (Please do not discuss company
specific vulnerabilities here).
Requirements:
# Secure authentication for AM (for app/job level ACLs).
# Webapp security should be optional via site configuration.
# Support existing pluggable single sign on mechanisms.
# Should not require per app/user configuration for deployment.
# Should not require special site-wide DNS configuration for deployment.
This the top jira for webapp security. A design doc/notes of threat-modeling
and counter measures will be posted on the wiki.
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators:
https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Updated] (MAPREDUCE-3057) Job History Server goes of OutOfMemory with 1200 Jobs and Heap Size set to 10 GB
[ https://issues.apache.org/jira/browse/MAPREDUCE-3057?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Eric Payne updated MAPREDUCE-3057: -- Attachment: MAPREDUCE-3057.v1.txt I'm having trouble reproducing this on my 1-node cluster. I'm continuing to try to test this, but since it's just a change to the default config value, I think it's safe to just post the patch and make sure this is what was agreed to. Job History Server goes of OutOfMemory with 1200 Jobs and Heap Size set to 10 GB Key: MAPREDUCE-3057 URL: https://issues.apache.org/jira/browse/MAPREDUCE-3057 Project: Hadoop Map/Reduce Issue Type: Bug Components: jobhistoryserver, mrv2 Affects Versions: 0.23.0 Reporter: Karam Singh Assignee: Eric Payne Priority: Blocker Fix For: 0.23.0 Attachments: MAPREDUCE-3057.v1.txt History server was started with -Xmx1m Ran GridMix V3 with 1200 Jobs trace in STRESS mode on 350 nodes with each node 4 NMS. All jobs finished as reported by RM Web UI and HADOOP_MAPRED_HOME/bin/mapred job -list all But found that GridMix job client was stuck while trying connect to HistoryServer Then tried to do HADOOP_MAPRED_HOME/bin/mapred job -status jobid JobClient also got stuck while looking for token to connect to History server Then looked at History Server logs and found History is trowing java.lang.OutOfMemoryError: GC overhead limit exceeded error. With 10GB of Heap space and 1200 Jobs, History Server should not go out of memory . No matter what are the type of jobs. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Updated] (MAPREDUCE-3057) Job History Server goes of OutOfMemory with 1200 Jobs and Heap Size set to 10 GB
[ https://issues.apache.org/jira/browse/MAPREDUCE-3057?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Eric Payne updated MAPREDUCE-3057: -- Status: Patch Available (was: Open) Job History Server goes of OutOfMemory with 1200 Jobs and Heap Size set to 10 GB Key: MAPREDUCE-3057 URL: https://issues.apache.org/jira/browse/MAPREDUCE-3057 Project: Hadoop Map/Reduce Issue Type: Bug Components: jobhistoryserver, mrv2 Affects Versions: 0.23.0 Reporter: Karam Singh Assignee: Eric Payne Priority: Blocker Fix For: 0.23.0 Attachments: MAPREDUCE-3057.v1.txt History server was started with -Xmx1m Ran GridMix V3 with 1200 Jobs trace in STRESS mode on 350 nodes with each node 4 NMS. All jobs finished as reported by RM Web UI and HADOOP_MAPRED_HOME/bin/mapred job -list all But found that GridMix job client was stuck while trying connect to HistoryServer Then tried to do HADOOP_MAPRED_HOME/bin/mapred job -status jobid JobClient also got stuck while looking for token to connect to History server Then looked at History Server logs and found History is trowing java.lang.OutOfMemoryError: GC overhead limit exceeded error. With 10GB of Heap space and 1200 Jobs, History Server should not go out of memory . No matter what are the type of jobs. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (MAPREDUCE-3144) Augment JobHistory to include information needed for serving aggregated logs.
[ https://issues.apache.org/jira/browse/MAPREDUCE-3144?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13126951#comment-13126951 ] Hadoop QA commented on MAPREDUCE-3144: -- -1 overall. Here are the results of testing the latest attachment http://issues.apache.org/jira/secure/attachment/12498923/MR3144_v2.txt against trunk revision . +1 @author. The patch does not contain any @author tags. +1 tests included. The patch appears to include 24 new or modified tests. -1 javadoc. The javadoc tool appears to have generated 1 warning messages. +1 javac. The applied patch does not increase the total number of javac compiler warnings. -1 findbugs. The patch appears to introduce 1 new Findbugs (version 1.3.9) warnings. +1 release audit. The applied patch does not increase the total number of release audit warnings. +1 core tests. The patch passed unit tests in . +1 contrib tests. The patch passed contrib unit tests. Test results: https://builds.apache.org/job/PreCommit-MAPREDUCE-Build/1008//testReport/ Findbugs warnings: https://builds.apache.org/job/PreCommit-MAPREDUCE-Build/1008//artifact/trunk/hadoop-mapreduce-project/patchprocess/newPatchFindbugsWarningshadoop-mapreduce-client-core.html Console output: https://builds.apache.org/job/PreCommit-MAPREDUCE-Build/1008//console This message is automatically generated. Augment JobHistory to include information needed for serving aggregated logs. - Key: MAPREDUCE-3144 URL: https://issues.apache.org/jira/browse/MAPREDUCE-3144 Project: Hadoop Map/Reduce Issue Type: Sub-task Components: mrv2 Affects Versions: 0.23.0 Reporter: Vinod Kumar Vavilapalli Assignee: Siddharth Seth Fix For: 0.23.0 Attachments: MR3144.patch, MR3144_v2.txt -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (MAPREDUCE-1932) record skipping doesn't work with the new map/reduce api
[ https://issues.apache.org/jira/browse/MAPREDUCE-1932?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13126971#comment-13126971 ] Tom White commented on MAPREDUCE-1932: -- I wonder whether we want to add this to the new API, when we could instead suggest that people launch their own subprocess (as Owen suggests here: http://mail-archives.apache.org/mod_mbox/hadoop-common-user/201108.mbox/%3ccafqou9ekv+sbvav-bsf5dorjo68vsj6ztqxywwut+qhs3v3...@mail.gmail.com%3e). As I understand it, the record skipping feature finds bad records by doing a binary search on the record range covered by a given task, so it has to re-run the task many times until the size of the window is below a given threshold. Also, I'm not sure how it copes with the case of multiple corrupted records in a single split. record skipping doesn't work with the new map/reduce api Key: MAPREDUCE-1932 URL: https://issues.apache.org/jira/browse/MAPREDUCE-1932 Project: Hadoop Map/Reduce Issue Type: Bug Components: task Affects Versions: 0.20.1 Reporter: Owen O'Malley Assignee: Harsh J Attachments: mapreduce.1932.skippingreader.r1.diff The new HADOOP-1230 map/reduce api doesn't support the record skipping features. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (MAPREDUCE-3057) Job History Server goes of OutOfMemory with 1200 Jobs and Heap Size set to 10 GB
[ https://issues.apache.org/jira/browse/MAPREDUCE-3057?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13126975#comment-13126975 ] Hadoop QA commented on MAPREDUCE-3057: -- -1 overall. Here are the results of testing the latest attachment http://issues.apache.org/jira/secure/attachment/12498928/MAPREDUCE-3057.v1.txt against trunk revision . +1 @author. The patch does not contain any @author tags. -1 tests included. The patch doesn't appear to include any new or modified tests. Please justify why no new tests are needed for this patch. Also please list what manual steps were performed to verify this patch. +1 javadoc. The javadoc tool did not generate any warning messages. +1 javac. The applied patch does not increase the total number of javac compiler warnings. +1 findbugs. The patch does not introduce any new Findbugs (version 1.3.9) warnings. +1 release audit. The applied patch does not increase the total number of release audit warnings. +1 core tests. The patch passed unit tests in . +1 contrib tests. The patch passed contrib unit tests. Test results: https://builds.apache.org/job/PreCommit-MAPREDUCE-Build/1009//testReport/ Console output: https://builds.apache.org/job/PreCommit-MAPREDUCE-Build/1009//console This message is automatically generated. Job History Server goes of OutOfMemory with 1200 Jobs and Heap Size set to 10 GB Key: MAPREDUCE-3057 URL: https://issues.apache.org/jira/browse/MAPREDUCE-3057 Project: Hadoop Map/Reduce Issue Type: Bug Components: jobhistoryserver, mrv2 Affects Versions: 0.23.0 Reporter: Karam Singh Assignee: Eric Payne Priority: Blocker Fix For: 0.23.0 Attachments: MAPREDUCE-3057.v1.txt History server was started with -Xmx1m Ran GridMix V3 with 1200 Jobs trace in STRESS mode on 350 nodes with each node 4 NMS. All jobs finished as reported by RM Web UI and HADOOP_MAPRED_HOME/bin/mapred job -list all But found that GridMix job client was stuck while trying connect to HistoryServer Then tried to do HADOOP_MAPRED_HOME/bin/mapred job -status jobid JobClient also got stuck while looking for token to connect to History server Then looked at History Server logs and found History is trowing java.lang.OutOfMemoryError: GC overhead limit exceeded error. With 10GB of Heap space and 1200 Jobs, History Server should not go out of memory . No matter what are the type of jobs. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Updated] (MAPREDUCE-3181) Terasort fails with Kerberos exception on secure cluster
[ https://issues.apache.org/jira/browse/MAPREDUCE-3181?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Mahadev konar updated MAPREDUCE-3181: - Priority: Blocker (was: Major) Terasort fails with Kerberos exception on secure cluster Key: MAPREDUCE-3181 URL: https://issues.apache.org/jira/browse/MAPREDUCE-3181 Project: Hadoop Map/Reduce Issue Type: Bug Components: mrv2 Affects Versions: 0.23.0 Reporter: Anupam Seth Assignee: Anupam Seth Priority: Blocker We are seeing the following Kerberos exception upon trying to run terasort on secure single and multi-node clusters using the latest build from branch 0.23. java.io.IOException: Can't get JobTracker Kerberos principal for use as renewer at org.apache.hadoop.mapreduce.security.TokenCache.obtainTokensForNamenodesInternal(TokenCache.java:106) at org.apache.hadoop.mapreduce.security.TokenCache.obtainTokensForNamenodesInternal(TokenCache.java:90) at org.apache.hadoop.mapreduce.security.TokenCache.obtainTokensForNamenodes(TokenCache.java:83) at org.apache.hadoop.mapreduce.lib.input.FileInputFormat.listStatus(FileInputFormat.java:205) at org.apache.hadoop.mapreduce.lib.input.FileInputFormat.getSplits(FileInputFormat.java:269) at org.apache.hadoop.examples.terasort.TeraInputFormat.getSplits(TeraInputFormat.java:318) at org.apache.hadoop.examples.terasort.TeraInputFormat.writePartitionFile(TeraInputFormat.java:169) at org.apache.hadoop.examples.terasort.TeraSort.run(TeraSort.java:306) at org.apache.hadoop.util.ToolRunner.run(ToolRunner.java:69) at org.apache.hadoop.examples.terasort.TeraSort.main(TeraSort.java:325) at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method) at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:39) at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:25) at java.lang.reflect.Method.invoke(Method.java:597) at org.apache.hadoop.util.ProgramDriver$ProgramDescription.invoke(ProgramDriver.java:72) at org.apache.hadoop.util.ProgramDriver.driver(ProgramDriver.java:144) at org.apache.hadoop.examples.ExampleDriver.main(ExampleDriver.java:68) at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method) at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:39) at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:25) at java.lang.reflect.Method.invoke(Method.java:597) at org.apache.hadoop.util.RunJar.main(RunJar.java:189) Adding debug output shows that the job configuration is not loading up yarn-site.xml causing the above failure to happen. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Created] (MAPREDUCE-3182) loadgen ignore -m command line when using writing random data
loadgen ignore -m command line when using writing random data - Key: MAPREDUCE-3182 URL: https://issues.apache.org/jira/browse/MAPREDUCE-3182 Project: Hadoop Map/Reduce Issue Type: Bug Components: mrv2, test Affects Versions: 0.23.0, 0.24.0 Reporter: Jonathan Eagles Assignee: Jonathan Eagles If no input directories are specified, loadgen goes into a special mode where random data is generated and written. In that mode, setting the number of mappers (-m command line option) is overridden by a calculation. Instead, it should take into consideration the user specified number of mappers and fall back to the calculation. In addition, update the documentation as well to match the new behavior in the code. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Updated] (MAPREDUCE-3182) loadgen ignores -m command line when writing random data
[ https://issues.apache.org/jira/browse/MAPREDUCE-3182?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Jonathan Eagles updated MAPREDUCE-3182: --- Target Version/s: 0.23.0, 0.24.0 (was: 0.24.0, 0.23.0) Summary: loadgen ignores -m command line when writing random data (was: loadgen ignore -m command line when using writing random data) loadgen ignores -m command line when writing random data Key: MAPREDUCE-3182 URL: https://issues.apache.org/jira/browse/MAPREDUCE-3182 Project: Hadoop Map/Reduce Issue Type: Bug Components: mrv2, test Affects Versions: 0.23.0, 0.24.0 Reporter: Jonathan Eagles Assignee: Jonathan Eagles If no input directories are specified, loadgen goes into a special mode where random data is generated and written. In that mode, setting the number of mappers (-m command line option) is overridden by a calculation. Instead, it should take into consideration the user specified number of mappers and fall back to the calculation. In addition, update the documentation as well to match the new behavior in the code. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Updated] (MAPREDUCE-3170) Trunk nightly commit builds are failing.
[ https://issues.apache.org/jira/browse/MAPREDUCE-3170?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Hitesh Shah updated MAPREDUCE-3170: --- Attachment: MR-3170.1.patch Fixes NPE in ant commit tests as well as a bug in fileoutputcommitter causing one of the failures. Trunk nightly commit builds are failing. Key: MAPREDUCE-3170 URL: https://issues.apache.org/jira/browse/MAPREDUCE-3170 Project: Hadoop Map/Reduce Issue Type: Bug Affects Versions: 0.23.0 Reporter: Mahadev konar Assignee: Hitesh Shah Priority: Critical Fix For: 0.23.0 Attachments: MR-3170.1.patch Looks like the trunk commit builds are failing after MAPREDUCE-3148 and MAPREDUCE-3126 were committed. I suspect its MAPREDUCE-3148. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa For more information on JIRA, see: http://www.atlassian.com/software/jira
