[jira] [Commented] (MAPREDUCE-2708) [MR-279] Design and implement MR Application Master recovery
[ https://issues.apache.org/jira/browse/MAPREDUCE-2708?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13128689#comment-13128689 ] Sharad Agarwal commented on MAPREDUCE-2708: --- bq. All hadoop-mapreduce-client passing. correction: All hadoop-mapreduce-client tests passing. [MR-279] Design and implement MR Application Master recovery Key: MAPREDUCE-2708 URL: https://issues.apache.org/jira/browse/MAPREDUCE-2708 Project: Hadoop Map/Reduce Issue Type: Sub-task Components: applicationmaster, mrv2 Affects Versions: 0.23.0 Reporter: Sharad Agarwal Assignee: Sharad Agarwal Priority: Blocker Fix For: 0.23.0 Attachments: mr2708_v1.patch, mr2708_v2.patch Design recovery of MR AM from crashes/node failures. The running job should recover from the state it left off. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Updated] (MAPREDUCE-2708) [MR-279] Design and implement MR Application Master recovery
[ https://issues.apache.org/jira/browse/MAPREDUCE-2708?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Sharad Agarwal updated MAPREDUCE-2708: -- Attachment: mr2708_v2.patch Rebased to latest 23 branch. All hadoop-mapreduce-client passing. [MR-279] Design and implement MR Application Master recovery Key: MAPREDUCE-2708 URL: https://issues.apache.org/jira/browse/MAPREDUCE-2708 Project: Hadoop Map/Reduce Issue Type: Sub-task Components: applicationmaster, mrv2 Affects Versions: 0.23.0 Reporter: Sharad Agarwal Assignee: Sharad Agarwal Priority: Blocker Fix For: 0.23.0 Attachments: mr2708_v1.patch, mr2708_v2.patch Design recovery of MR AM from crashes/node failures. The running job should recover from the state it left off. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Updated] (MAPREDUCE-2762) [MR-279] - Cleanup staging dir after job completion
[
https://issues.apache.org/jira/browse/MAPREDUCE-2762?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Mahadev konar updated MAPREDUCE-2762:
-
Attachment: MAPREDUCE-2762.patch
Changed the patch to do cleanup after calling stop on all services in
MRAppMaster.
- Added a job cleanup event which is synchronously called from the MRAppMaster
Jobfinished handler.
We could also add a job api for cleanupo but either of those are not very clean
implementation. The more I think, the earlier suggestion on sending events for
de registering and then cleanup is probably much cleaner.
[MR-279] - Cleanup staging dir after job completion
---
Key: MAPREDUCE-2762
URL: https://issues.apache.org/jira/browse/MAPREDUCE-2762
Project: Hadoop Map/Reduce
Issue Type: Bug
Components: mrv2
Affects Versions: 0.23.0
Reporter: Ramya Sunil
Assignee: Mahadev konar
Priority: Blocker
Fix For: 0.23.0
Attachments: MAPREDUCE-2762.patch, MAPREDUCE-2762.patch,
MAPREDUCE-2762.patch, MAPREDUCE-2762.patch
The files created under the staging dir have to be deleted after job
completion. Currently, all job.* files remain forever in the
${yarn.apps.stagingDir}
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators:
https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (MAPREDUCE-2858) MRv2 WebApp Security
[ https://issues.apache.org/jira/browse/MAPREDUCE-2858?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13128712#comment-13128712 ] Luke Lu commented on MAPREDUCE-2858: bq. The number of ways to get JavaScript running, or to embed external content inside HTML content is amazing. Properly sandboxing untrusted content inside a page without resorting to IFRAMEs is practically research territory. Writing our own implementation of this is totally crazy. If we go that route, we should use something like Google's Caja framework. This comment is a misunderstanding of the current proposal. We're not trying to reinvent Caja at all. In fact, I personally don't trust the approach of Caja. It's too complex/slow and an overkill for our purpose. The only reason to rewrite the urls in html is for non-transparent proxy. If we deploy transparent proxy (which can be implemented straightforwardly via the full requestURI from the GET header) via layer 4+ switches, no rewrite is needed. This is orthogonal to the js whitelisting feature of the scanner. To give you an extreme example, it's completely acceptable for the scanner to always return true for the maybeUnsafe method. The whitelisting feature is a simple/safe/fast way to improve user experience for admins and other users who are not the owners of the target webapps, which is ultimately a nice thing to have but not absolutely required for the proxy to achieve its purpose. MRv2 WebApp Security Key: MAPREDUCE-2858 URL: https://issues.apache.org/jira/browse/MAPREDUCE-2858 Project: Hadoop Map/Reduce Issue Type: Sub-task Components: applicationmaster, mrv2, security Affects Versions: 0.23.0 Reporter: Luke Lu Assignee: Luke Lu Priority: Blocker Fix For: 0.23.0 In MRv2, while the system servers (ResourceManager (RM), NodeManager (NM) and NameNode (NN)) run as trusted system users, the application masters (AM) run as users who submit the application. While this offers great flexibility to run multiple version of mapreduce frameworks (including their UI) on the same Hadoop cluster, it has significant implication for the security of webapps (Please do not discuss company specific vulnerabilities here). Requirements: # Secure authentication for AM (for app/job level ACLs). # Webapp security should be optional via site configuration. # Support existing pluggable single sign on mechanisms. # Should not require per app/user configuration for deployment. # Should not require special site-wide DNS configuration for deployment. This the top jira for webapp security. A design doc/notes of threat-modeling and counter measures will be posted on the wiki. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Created] (MAPREDUCE-3193) NextGen Mapreduce framework is not able to read the job input recursively.Input is read only for one folder level deep
NextGen Mapreduce framework is not able to read the job input recursively.Input is read only for one folder level deep -- Key: MAPREDUCE-3193 URL: https://issues.apache.org/jira/browse/MAPREDUCE-3193 Project: Hadoop Map/Reduce Issue Type: Bug Components: mrv2 Affects Versions: 0.23.0 Reporter: Ramgopal N java.io.FileNotFoundException is thrown,if input file is more than one folder level deep and the job is getting failed. Example:Input file is /r1/r2/input.txt -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (MAPREDUCE-3136) Add docs for setting up real-world MRv2 clusters
[ https://issues.apache.org/jira/browse/MAPREDUCE-3136?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13128827#comment-13128827 ] Hudson commented on MAPREDUCE-3136: --- Integrated in Hadoop-Hdfs-0.23-Build #42 (See [https://builds.apache.org/job/Hadoop-Hdfs-0.23-Build/42/]) Merge -c 1184981 from trunk to branch-0.23 to complete fix for MAPREDUCE-3136. Merge -c 1184977 from trunk to branch-0.23 to fix MAPREDUCE-3136. acmurthy : http://svn.apache.org/viewcvs.cgi/?root=Apache-SVNview=revrev=1184982 Files : * /hadoop/common/branches/branch-0.23/hadoop-mapreduce-project/hadoop-yarn/hadoop-yarn-site/src/site/apt/index.apt.vm acmurthy : http://svn.apache.org/viewcvs.cgi/?root=Apache-SVNview=revrev=1184978 Files : * /hadoop/common/branches/branch-0.23/hadoop-mapreduce-project/CHANGES.txt * /hadoop/common/branches/branch-0.23/hadoop-mapreduce-project/hadoop-yarn/hadoop-yarn-site/src/site/apt/CapacityScheduler.apt.vm * /hadoop/common/branches/branch-0.23/hadoop-mapreduce-project/hadoop-yarn/hadoop-yarn-site/src/site/apt/ClusterSetup.apt.vm Add docs for setting up real-world MRv2 clusters Key: MAPREDUCE-3136 URL: https://issues.apache.org/jira/browse/MAPREDUCE-3136 Project: Hadoop Map/Reduce Issue Type: Sub-task Components: documentation, mrv2 Reporter: Arun C Murthy Assignee: Arun C Murthy Priority: Blocker Fix For: 0.23.0 Attachments: MAPREDUCE-3136.patch, MAPREDUCE-3136.patch, MAPREDUCE-3136.patch, MAPREDUCE-3136.patch, MAPREDUCE-3136.patch_1 Add docs for setting up real-world MRv2 clusters - MR portion of http://hadoop.apache.org/common/docs/stable/cluster_setup.html -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (MAPREDUCE-3189) Add link decoration back to MR2's CSS
[ https://issues.apache.org/jira/browse/MAPREDUCE-3189?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13128828#comment-13128828 ] Hudson commented on MAPREDUCE-3189: --- Integrated in Hadoop-Hdfs-0.23-Build #42 (See [https://builds.apache.org/job/Hadoop-Hdfs-0.23-Build/42/]) MAPREDUCE-3189. Add link decoration back to MR2's CSS. (Todd Lipcon via mahadev) - Merging r1184979 from trunk mahadev : http://svn.apache.org/viewcvs.cgi/?root=Apache-SVNview=revrev=1184980 Files : * /hadoop/common/branches/branch-0.23/hadoop-mapreduce-project/CHANGES.txt * /hadoop/common/branches/branch-0.23/hadoop-mapreduce-project/hadoop-yarn/hadoop-yarn-common/src/main/resources/webapps/static/yarn.css Add link decoration back to MR2's CSS - Key: MAPREDUCE-3189 URL: https://issues.apache.org/jira/browse/MAPREDUCE-3189 Project: Hadoop Map/Reduce Issue Type: Improvement Components: mrv2 Affects Versions: 0.23.0 Reporter: Todd Lipcon Assignee: Todd Lipcon Fix For: 0.23.0 Attachments: mr-3189.txt I found the MRv2 web UI very difficult to use because it's not clear which items are links and which aren't. I'd like to change the CSS so that links are underlined, making it easier to see them (since they're also not in any different color) -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (MAPREDUCE-2840) mr279 TestUberAM.testSleepJob test fails
[ https://issues.apache.org/jira/browse/MAPREDUCE-2840?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13128830#comment-13128830 ] Hudson commented on MAPREDUCE-2840: --- Integrated in Hadoop-Hdfs-0.23-Build #42 (See [https://builds.apache.org/job/Hadoop-Hdfs-0.23-Build/42/]) MAPREDUCE-2840. mr279 TestUberAM.testSleepJob test fails. (jonathan eagles via mahadev) - Merging r1184971 from trunk mahadev : http://svn.apache.org/viewcvs.cgi/?root=Apache-SVNview=revrev=1184972 Files : * /hadoop/common/branches/branch-0.23/hadoop-mapreduce-project/CHANGES.txt * /hadoop/common/branches/branch-0.23/hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-jobclient/src/test/java/org/apache/hadoop/mapreduce/v2/TestMRJobs.java * /hadoop/common/branches/branch-0.23/hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-jobclient/src/test/java/org/apache/hadoop/mapreduce/v2/TestUberAM.java mr279 TestUberAM.testSleepJob test fails Key: MAPREDUCE-2840 URL: https://issues.apache.org/jira/browse/MAPREDUCE-2840 Project: Hadoop Map/Reduce Issue Type: Bug Components: mrv2 Affects Versions: 0.23.0 Reporter: Thomas Graves Assignee: Jonathan Eagles Priority: Minor Fix For: 0.23.0 Attachments: MAPREDUCE-2840.patch Currently the TestUberAM.testSleepJob is failing on the mr279 branch. snippet of failure: junit.framework.AssertionFailedError: null at junit.framework.Assert.fail(Assert.java:47) at junit.framework.Assert.assertTrue(Assert.java:20) at junit.framework.Assert.assertTrue(Assert.java:27) at org.apache.hadoop.mapreduce.v2.TestMRJobs.testSleepJob(TestMRJobs.java:150) at org.apache.hadoop.mapreduce.v2.TestUberAM.testSleepJob(TestUberAM.java:58) at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method) at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:39) at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:25) at java.lang.reflect.Method.invoke(Method.java:597) -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (MAPREDUCE-3187) Add names for various unnamed threads in MR2
[ https://issues.apache.org/jira/browse/MAPREDUCE-3187?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13128829#comment-13128829 ] Hudson commented on MAPREDUCE-3187: --- Integrated in Hadoop-Hdfs-0.23-Build #42 (See [https://builds.apache.org/job/Hadoop-Hdfs-0.23-Build/42/]) MAPREDUCE-3187. Add names for various unnamed threads in MR2. (Todd Lipcon and Siddharth Seth via mahadev) - Merging r1184904 from trunk mahadev : http://svn.apache.org/viewcvs.cgi/?root=Apache-SVNview=revrev=1184905 Files : * /hadoop/common/branches/branch-0.23/hadoop-mapreduce-project/CHANGES.txt * /hadoop/common/branches/branch-0.23/hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-app/src/main/java/org/apache/hadoop/mapreduce/v2/app/TaskHeartbeatHandler.java * /hadoop/common/branches/branch-0.23/hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-app/src/main/java/org/apache/hadoop/mapreduce/v2/app/launcher/ContainerLauncherImpl.java * /hadoop/common/branches/branch-0.23/hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-app/src/main/java/org/apache/hadoop/mapreduce/v2/app/rm/RMCommunicator.java * /hadoop/common/branches/branch-0.23/hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-app/src/main/java/org/apache/hadoop/mapreduce/v2/app/taskclean/TaskCleanerImpl.java * /hadoop/common/branches/branch-0.23/hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-hs/src/main/java/org/apache/hadoop/mapreduce/v2/hs/JobHistory.java * /hadoop/common/branches/branch-0.23/hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-shuffle/src/main/java/org/apache/hadoop/mapred/ShuffleHandler.java * /hadoop/common/branches/branch-0.23/hadoop-mapreduce-project/hadoop-yarn/hadoop-yarn-common/src/main/java/org/apache/hadoop/yarn/event/AsyncDispatcher.java * /hadoop/common/branches/branch-0.23/hadoop-mapreduce-project/hadoop-yarn/hadoop-yarn-common/src/main/java/org/apache/hadoop/yarn/service/CompositeService.java * /hadoop/common/branches/branch-0.23/hadoop-mapreduce-project/hadoop-yarn/hadoop-yarn-common/src/main/java/org/apache/hadoop/yarn/util/AbstractLivelinessMonitor.java * /hadoop/common/branches/branch-0.23/hadoop-mapreduce-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-nodemanager/src/main/java/org/apache/hadoop/yarn/server/nodemanager/DeletionService.java * /hadoop/common/branches/branch-0.23/hadoop-mapreduce-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-nodemanager/src/main/java/org/apache/hadoop/yarn/server/nodemanager/NodeStatusUpdaterImpl.java * /hadoop/common/branches/branch-0.23/hadoop-mapreduce-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-nodemanager/src/main/java/org/apache/hadoop/yarn/server/nodemanager/containermanager/launcher/ContainersLauncher.java * /hadoop/common/branches/branch-0.23/hadoop-mapreduce-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-nodemanager/src/main/java/org/apache/hadoop/yarn/server/nodemanager/containermanager/localizer/ContainerLocalizer.java * /hadoop/common/branches/branch-0.23/hadoop-mapreduce-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-nodemanager/src/main/java/org/apache/hadoop/yarn/server/nodemanager/containermanager/localizer/ResourceLocalizationService.java * /hadoop/common/branches/branch-0.23/hadoop-mapreduce-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-nodemanager/src/main/java/org/apache/hadoop/yarn/server/nodemanager/containermanager/logaggregation/LogAggregationService.java * /hadoop/common/branches/branch-0.23/hadoop-mapreduce-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-nodemanager/src/main/java/org/apache/hadoop/yarn/server/nodemanager/containermanager/monitor/ContainersMonitorImpl.java * /hadoop/common/branches/branch-0.23/hadoop-mapreduce-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-nodemanager/src/main/resources/container-log4j.properties * /hadoop/common/branches/branch-0.23/hadoop-mapreduce-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager/src/main/java/org/apache/hadoop/yarn/server/resourcemanager/ResourceManager.java * /hadoop/common/branches/branch-0.23/hadoop-mapreduce-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager/src/main/java/org/apache/hadoop/yarn/server/resourcemanager/amlauncher/ApplicationMasterLauncher.java * /hadoop/common/branches/branch-0.23/hadoop-mapreduce-project/src/test/log4j.properties Add names for various unnamed threads in MR2 Key: MAPREDUCE-3187 URL: https://issues.apache.org/jira/browse/MAPREDUCE-3187 Project: Hadoop Map/Reduce Issue Type: Improvement Components: mrv2 Affects Versions: 0.23.0 Reporter: Todd Lipcon Assignee:
[jira] [Commented] (MAPREDUCE-3190) bin/yarn should barf early if HADOOP_COMMON_HOME or HADOOP_HDFS_HOME are not set
[
https://issues.apache.org/jira/browse/MAPREDUCE-3190?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13128826#comment-13128826
]
Hudson commented on MAPREDUCE-3190:
---
Integrated in Hadoop-Hdfs-0.23-Build #42 (See
[https://builds.apache.org/job/Hadoop-Hdfs-0.23-Build/42/])
Merge -c 1184975 from trunk to branch-0.23 to fix MAPREDUCE-3190.
acmurthy :
http://svn.apache.org/viewcvs.cgi/?root=Apache-SVNview=revrev=1184976
Files :
* /hadoop/common/branches/branch-0.23/hadoop-mapreduce-project/CHANGES.txt
*
/hadoop/common/branches/branch-0.23/hadoop-mapreduce-project/hadoop-yarn/bin/yarn
bin/yarn should barf early if HADOOP_COMMON_HOME or HADOOP_HDFS_HOME are not
set
Key: MAPREDUCE-3190
URL: https://issues.apache.org/jira/browse/MAPREDUCE-3190
Project: Hadoop Map/Reduce
Issue Type: Improvement
Components: mrv2
Affects Versions: 0.23.0
Reporter: Todd Lipcon
Assignee: Todd Lipcon
Fix For: 0.23.0
Attachments: mr-3190.txt, mr-3190.txt
Currently, if these env vars are not set when you run bin/yarn, it will crash
with various ClassNotFoundExceptions, having added {{/share/hadoop/hdfs}} to
the classpath. Rather, we should check for these env vars in the wrapper
script and display a reasonable error message.
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators:
https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (MAPREDUCE-3127) Unable to restrict users based on resourcemanager.admin.acls value set
[ https://issues.apache.org/jira/browse/MAPREDUCE-3127?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13128832#comment-13128832 ] Hudson commented on MAPREDUCE-3127: --- Integrated in Hadoop-Hdfs-0.23-Build #42 (See [https://builds.apache.org/job/Hadoop-Hdfs-0.23-Build/42/]) MAPREDUCE-3127. Fixed CHANGES.txt to reflect correct jira. acmurthy : http://svn.apache.org/viewcvs.cgi/?root=Apache-SVNview=revrev=1184993 Files : * /hadoop/common/branches/branch-0.23/hadoop-mapreduce-project/CHANGES.txt Unable to restrict users based on resourcemanager.admin.acls value set -- Key: MAPREDUCE-3127 URL: https://issues.apache.org/jira/browse/MAPREDUCE-3127 Project: Hadoop Map/Reduce Issue Type: Sub-task Components: mrv2, resourcemanager Affects Versions: 0.23.0 Reporter: Amol Kekre Assignee: Arun C Murthy Priority: Blocker Fix For: 0.23.0 Attachments: MAPREDUCE-3101.patch, MAPREDUCE-3101.patch, MAPREDUCE-3101.patch Setting the following property in yarn-site.xml with user ids to restrict ability to run 'rmadmin -refreshQueues is not honoured property nameyarn.server.resourcemanager.admin.acls/name valuehadoop1/value description/description final/final /property Should it be the same for rmadmin -refreshNodes? -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (MAPREDUCE-3101) Security issues in YARN
[ https://issues.apache.org/jira/browse/MAPREDUCE-3101?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13128831#comment-13128831 ] Hudson commented on MAPREDUCE-3101: --- Integrated in Hadoop-Hdfs-0.23-Build #42 (See [https://builds.apache.org/job/Hadoop-Hdfs-0.23-Build/42/]) Merge -c 1184989 from trunk to branch-0.23 to complete fix for MAPREDUCE-3101. acmurthy : http://svn.apache.org/viewcvs.cgi/?root=Apache-SVNview=revrev=1184990 Files : * /hadoop/common/branches/branch-0.23/hadoop-mapreduce-project/CHANGES.txt * /hadoop/common/branches/branch-0.23/hadoop-mapreduce-project/hadoop-yarn/hadoop-yarn-common/src/main/java/org/apache/hadoop/yarn/conf/YarnConfiguration.java * /hadoop/common/branches/branch-0.23/hadoop-mapreduce-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-common/src/main/resources/yarn-default.xml * /hadoop/common/branches/branch-0.23/hadoop-mapreduce-project/hadoop-yarn/hadoop-yarn-site/src/site/apt/ClusterSetup.apt.vm Security issues in YARN --- Key: MAPREDUCE-3101 URL: https://issues.apache.org/jira/browse/MAPREDUCE-3101 Project: Hadoop Map/Reduce Issue Type: Bug Components: security Affects Versions: 0.23.0 Reporter: Vinod Kumar Vavilapalli Assignee: Vinod Kumar Vavilapalli Fix For: 0.23.0 Most of the chassis for security in YARN is set up and is working. There are known bugs and security holes though. This JIRA is an umbrella ticket for tracking those. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (MAPREDUCE-3190) bin/yarn should barf early if HADOOP_COMMON_HOME or HADOOP_HDFS_HOME are not set
[
https://issues.apache.org/jira/browse/MAPREDUCE-3190?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13128850#comment-13128850
]
Hudson commented on MAPREDUCE-3190:
---
Integrated in Hadoop-Mapreduce-trunk #863 (See
[https://builds.apache.org/job/Hadoop-Mapreduce-trunk/863/])
MAPREDUCE-3190. Ensure bin/yarn fails early with a clear error message when
HADOOP_COMMON_HOME or HADOOP_HDFS_HOME are not set. Contributed by todd
acmurthy.
acmurthy :
http://svn.apache.org/viewcvs.cgi/?root=Apache-SVNview=revrev=1184975
Files :
* /hadoop/common/trunk/hadoop-mapreduce-project/CHANGES.txt
* /hadoop/common/trunk/hadoop-mapreduce-project/hadoop-yarn/bin/yarn
bin/yarn should barf early if HADOOP_COMMON_HOME or HADOOP_HDFS_HOME are not
set
Key: MAPREDUCE-3190
URL: https://issues.apache.org/jira/browse/MAPREDUCE-3190
Project: Hadoop Map/Reduce
Issue Type: Improvement
Components: mrv2
Affects Versions: 0.23.0
Reporter: Todd Lipcon
Assignee: Todd Lipcon
Fix For: 0.23.0
Attachments: mr-3190.txt, mr-3190.txt
Currently, if these env vars are not set when you run bin/yarn, it will crash
with various ClassNotFoundExceptions, having added {{/share/hadoop/hdfs}} to
the classpath. Rather, we should check for these env vars in the wrapper
script and display a reasonable error message.
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators:
https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (MAPREDUCE-3136) Add docs for setting up real-world MRv2 clusters
[ https://issues.apache.org/jira/browse/MAPREDUCE-3136?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13128851#comment-13128851 ] Hudson commented on MAPREDUCE-3136: --- Integrated in Hadoop-Mapreduce-trunk #863 (See [https://builds.apache.org/job/Hadoop-Mapreduce-trunk/863/]) MAPREDUCE-3136. Added the missing link to index.html. MAPREDUCE-3136. Added documentation for setting up Hadoop clusters in both non-secure and secure mode for both HDFS YARN. acmurthy : http://svn.apache.org/viewcvs.cgi/?root=Apache-SVNview=revrev=1184981 Files : * /hadoop/common/trunk/hadoop-mapreduce-project/hadoop-yarn/hadoop-yarn-site/src/site/apt/index.apt.vm acmurthy : http://svn.apache.org/viewcvs.cgi/?root=Apache-SVNview=revrev=1184977 Files : * /hadoop/common/trunk/hadoop-mapreduce-project/CHANGES.txt * /hadoop/common/trunk/hadoop-mapreduce-project/hadoop-yarn/hadoop-yarn-site/src/site/apt/CapacityScheduler.apt.vm * /hadoop/common/trunk/hadoop-mapreduce-project/hadoop-yarn/hadoop-yarn-site/src/site/apt/ClusterSetup.apt.vm Add docs for setting up real-world MRv2 clusters Key: MAPREDUCE-3136 URL: https://issues.apache.org/jira/browse/MAPREDUCE-3136 Project: Hadoop Map/Reduce Issue Type: Sub-task Components: documentation, mrv2 Reporter: Arun C Murthy Assignee: Arun C Murthy Priority: Blocker Fix For: 0.23.0 Attachments: MAPREDUCE-3136.patch, MAPREDUCE-3136.patch, MAPREDUCE-3136.patch, MAPREDUCE-3136.patch, MAPREDUCE-3136.patch_1 Add docs for setting up real-world MRv2 clusters - MR portion of http://hadoop.apache.org/common/docs/stable/cluster_setup.html -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (MAPREDUCE-3189) Add link decoration back to MR2's CSS
[ https://issues.apache.org/jira/browse/MAPREDUCE-3189?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13128852#comment-13128852 ] Hudson commented on MAPREDUCE-3189: --- Integrated in Hadoop-Mapreduce-trunk #863 (See [https://builds.apache.org/job/Hadoop-Mapreduce-trunk/863/]) MAPREDUCE-3189. Add link decoration back to MR2's CSS. (Todd Lipcon via mahadev) mahadev : http://svn.apache.org/viewcvs.cgi/?root=Apache-SVNview=revrev=1184979 Files : * /hadoop/common/trunk/hadoop-mapreduce-project/CHANGES.txt * /hadoop/common/trunk/hadoop-mapreduce-project/hadoop-yarn/hadoop-yarn-common/src/main/resources/webapps/static/yarn.css Add link decoration back to MR2's CSS - Key: MAPREDUCE-3189 URL: https://issues.apache.org/jira/browse/MAPREDUCE-3189 Project: Hadoop Map/Reduce Issue Type: Improvement Components: mrv2 Affects Versions: 0.23.0 Reporter: Todd Lipcon Assignee: Todd Lipcon Fix For: 0.23.0 Attachments: mr-3189.txt I found the MRv2 web UI very difficult to use because it's not clear which items are links and which aren't. I'd like to change the CSS so that links are underlined, making it easier to see them (since they're also not in any different color) -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (MAPREDUCE-3101) Security issues in YARN
[ https://issues.apache.org/jira/browse/MAPREDUCE-3101?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13128855#comment-13128855 ] Hudson commented on MAPREDUCE-3101: --- Integrated in Hadoop-Mapreduce-trunk #863 (See [https://builds.apache.org/job/Hadoop-Mapreduce-trunk/863/]) MAPREDUCE-3101. Changed default value of yarn.resourcemanager.acl.enable to true and added some more documentation. acmurthy : http://svn.apache.org/viewcvs.cgi/?root=Apache-SVNview=revrev=1184989 Files : * /hadoop/common/trunk/hadoop-mapreduce-project/CHANGES.txt * /hadoop/common/trunk/hadoop-mapreduce-project/hadoop-yarn/hadoop-yarn-common/src/main/java/org/apache/hadoop/yarn/conf/YarnConfiguration.java * /hadoop/common/trunk/hadoop-mapreduce-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-common/src/main/resources/yarn-default.xml * /hadoop/common/trunk/hadoop-mapreduce-project/hadoop-yarn/hadoop-yarn-site/src/site/apt/ClusterSetup.apt.vm Security issues in YARN --- Key: MAPREDUCE-3101 URL: https://issues.apache.org/jira/browse/MAPREDUCE-3101 Project: Hadoop Map/Reduce Issue Type: Bug Components: security Affects Versions: 0.23.0 Reporter: Vinod Kumar Vavilapalli Assignee: Vinod Kumar Vavilapalli Fix For: 0.23.0 Most of the chassis for security in YARN is set up and is working. There are known bugs and security holes though. This JIRA is an umbrella ticket for tracking those. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (MAPREDUCE-2840) mr279 TestUberAM.testSleepJob test fails
[ https://issues.apache.org/jira/browse/MAPREDUCE-2840?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13128854#comment-13128854 ] Hudson commented on MAPREDUCE-2840: --- Integrated in Hadoop-Mapreduce-trunk #863 (See [https://builds.apache.org/job/Hadoop-Mapreduce-trunk/863/]) MAPREDUCE-2840. mr279 TestUberAM.testSleepJob test fails. (jonathan eagles via mahadev) mahadev : http://svn.apache.org/viewcvs.cgi/?root=Apache-SVNview=revrev=1184971 Files : * /hadoop/common/trunk/hadoop-mapreduce-project/CHANGES.txt * /hadoop/common/trunk/hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-jobclient/src/test/java/org/apache/hadoop/mapreduce/v2/TestMRJobs.java * /hadoop/common/trunk/hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-jobclient/src/test/java/org/apache/hadoop/mapreduce/v2/TestUberAM.java mr279 TestUberAM.testSleepJob test fails Key: MAPREDUCE-2840 URL: https://issues.apache.org/jira/browse/MAPREDUCE-2840 Project: Hadoop Map/Reduce Issue Type: Bug Components: mrv2 Affects Versions: 0.23.0 Reporter: Thomas Graves Assignee: Jonathan Eagles Priority: Minor Fix For: 0.23.0 Attachments: MAPREDUCE-2840.patch Currently the TestUberAM.testSleepJob is failing on the mr279 branch. snippet of failure: junit.framework.AssertionFailedError: null at junit.framework.Assert.fail(Assert.java:47) at junit.framework.Assert.assertTrue(Assert.java:20) at junit.framework.Assert.assertTrue(Assert.java:27) at org.apache.hadoop.mapreduce.v2.TestMRJobs.testSleepJob(TestMRJobs.java:150) at org.apache.hadoop.mapreduce.v2.TestUberAM.testSleepJob(TestUberAM.java:58) at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method) at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:39) at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:25) at java.lang.reflect.Method.invoke(Method.java:597) -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (MAPREDUCE-3127) Unable to restrict users based on resourcemanager.admin.acls value set
[ https://issues.apache.org/jira/browse/MAPREDUCE-3127?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13128856#comment-13128856 ] Hudson commented on MAPREDUCE-3127: --- Integrated in Hadoop-Mapreduce-trunk #863 (See [https://builds.apache.org/job/Hadoop-Mapreduce-trunk/863/]) MAPREDUCE-3127. Fixed CHANGES.txt to reflect correct jira. acmurthy : http://svn.apache.org/viewcvs.cgi/?root=Apache-SVNview=revrev=1184992 Files : * /hadoop/common/trunk/hadoop-mapreduce-project/CHANGES.txt Unable to restrict users based on resourcemanager.admin.acls value set -- Key: MAPREDUCE-3127 URL: https://issues.apache.org/jira/browse/MAPREDUCE-3127 Project: Hadoop Map/Reduce Issue Type: Sub-task Components: mrv2, resourcemanager Affects Versions: 0.23.0 Reporter: Amol Kekre Assignee: Arun C Murthy Priority: Blocker Fix For: 0.23.0 Attachments: MAPREDUCE-3101.patch, MAPREDUCE-3101.patch, MAPREDUCE-3101.patch Setting the following property in yarn-site.xml with user ids to restrict ability to run 'rmadmin -refreshQueues is not honoured property nameyarn.server.resourcemanager.admin.acls/name valuehadoop1/value description/description final/final /property Should it be the same for rmadmin -refreshNodes? -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (MAPREDUCE-3187) Add names for various unnamed threads in MR2
[ https://issues.apache.org/jira/browse/MAPREDUCE-3187?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13128853#comment-13128853 ] Hudson commented on MAPREDUCE-3187: --- Integrated in Hadoop-Mapreduce-trunk #863 (See [https://builds.apache.org/job/Hadoop-Mapreduce-trunk/863/]) MAPREDUCE-3187. Add names for various unnamed threads in MR2. (Todd Lipcon and Siddharth Seth via mahadev) mahadev : http://svn.apache.org/viewcvs.cgi/?root=Apache-SVNview=revrev=1184904 Files : * /hadoop/common/trunk/hadoop-mapreduce-project/CHANGES.txt * /hadoop/common/trunk/hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-app/src/main/java/org/apache/hadoop/mapreduce/v2/app/TaskHeartbeatHandler.java * /hadoop/common/trunk/hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-app/src/main/java/org/apache/hadoop/mapreduce/v2/app/launcher/ContainerLauncherImpl.java * /hadoop/common/trunk/hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-app/src/main/java/org/apache/hadoop/mapreduce/v2/app/rm/RMCommunicator.java * /hadoop/common/trunk/hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-app/src/main/java/org/apache/hadoop/mapreduce/v2/app/taskclean/TaskCleanerImpl.java * /hadoop/common/trunk/hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-hs/src/main/java/org/apache/hadoop/mapreduce/v2/hs/JobHistory.java * /hadoop/common/trunk/hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-shuffle/src/main/java/org/apache/hadoop/mapred/ShuffleHandler.java * /hadoop/common/trunk/hadoop-mapreduce-project/hadoop-yarn/hadoop-yarn-common/src/main/java/org/apache/hadoop/yarn/event/AsyncDispatcher.java * /hadoop/common/trunk/hadoop-mapreduce-project/hadoop-yarn/hadoop-yarn-common/src/main/java/org/apache/hadoop/yarn/service/CompositeService.java * /hadoop/common/trunk/hadoop-mapreduce-project/hadoop-yarn/hadoop-yarn-common/src/main/java/org/apache/hadoop/yarn/util/AbstractLivelinessMonitor.java * /hadoop/common/trunk/hadoop-mapreduce-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-nodemanager/src/main/java/org/apache/hadoop/yarn/server/nodemanager/DeletionService.java * /hadoop/common/trunk/hadoop-mapreduce-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-nodemanager/src/main/java/org/apache/hadoop/yarn/server/nodemanager/NodeStatusUpdaterImpl.java * /hadoop/common/trunk/hadoop-mapreduce-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-nodemanager/src/main/java/org/apache/hadoop/yarn/server/nodemanager/containermanager/launcher/ContainersLauncher.java * /hadoop/common/trunk/hadoop-mapreduce-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-nodemanager/src/main/java/org/apache/hadoop/yarn/server/nodemanager/containermanager/localizer/ContainerLocalizer.java * /hadoop/common/trunk/hadoop-mapreduce-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-nodemanager/src/main/java/org/apache/hadoop/yarn/server/nodemanager/containermanager/localizer/ResourceLocalizationService.java * /hadoop/common/trunk/hadoop-mapreduce-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-nodemanager/src/main/java/org/apache/hadoop/yarn/server/nodemanager/containermanager/logaggregation/LogAggregationService.java * /hadoop/common/trunk/hadoop-mapreduce-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-nodemanager/src/main/java/org/apache/hadoop/yarn/server/nodemanager/containermanager/monitor/ContainersMonitorImpl.java * /hadoop/common/trunk/hadoop-mapreduce-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-nodemanager/src/main/resources/container-log4j.properties * /hadoop/common/trunk/hadoop-mapreduce-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager/src/main/java/org/apache/hadoop/yarn/server/resourcemanager/ResourceManager.java * /hadoop/common/trunk/hadoop-mapreduce-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager/src/main/java/org/apache/hadoop/yarn/server/resourcemanager/amlauncher/ApplicationMasterLauncher.java * /hadoop/common/trunk/hadoop-mapreduce-project/src/test/log4j.properties Add names for various unnamed threads in MR2 Key: MAPREDUCE-3187 URL: https://issues.apache.org/jira/browse/MAPREDUCE-3187 Project: Hadoop Map/Reduce Issue Type: Improvement Components: mrv2 Affects Versions: 0.23.0 Reporter: Todd Lipcon Assignee: Todd Lipcon Priority: Minor Fix For: 0.23.0 Attachments: mr-3187.txt, mr-3187.txt, mr3187_increment.txt Simple patch to add thread names for all the places we use Executors, etc. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA
[jira] [Commented] (MAPREDUCE-3190) bin/yarn should barf early if HADOOP_COMMON_HOME or HADOOP_HDFS_HOME are not set
[
https://issues.apache.org/jira/browse/MAPREDUCE-3190?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13128858#comment-13128858
]
Hudson commented on MAPREDUCE-3190:
---
Integrated in Hadoop-Mapreduce-0.23-Build #54 (See
[https://builds.apache.org/job/Hadoop-Mapreduce-0.23-Build/54/])
Merge -c 1184975 from trunk to branch-0.23 to fix MAPREDUCE-3190.
acmurthy :
http://svn.apache.org/viewcvs.cgi/?root=Apache-SVNview=revrev=1184976
Files :
* /hadoop/common/branches/branch-0.23/hadoop-mapreduce-project/CHANGES.txt
*
/hadoop/common/branches/branch-0.23/hadoop-mapreduce-project/hadoop-yarn/bin/yarn
bin/yarn should barf early if HADOOP_COMMON_HOME or HADOOP_HDFS_HOME are not
set
Key: MAPREDUCE-3190
URL: https://issues.apache.org/jira/browse/MAPREDUCE-3190
Project: Hadoop Map/Reduce
Issue Type: Improvement
Components: mrv2
Affects Versions: 0.23.0
Reporter: Todd Lipcon
Assignee: Todd Lipcon
Fix For: 0.23.0
Attachments: mr-3190.txt, mr-3190.txt
Currently, if these env vars are not set when you run bin/yarn, it will crash
with various ClassNotFoundExceptions, having added {{/share/hadoop/hdfs}} to
the classpath. Rather, we should check for these env vars in the wrapper
script and display a reasonable error message.
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators:
https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (MAPREDUCE-3187) Add names for various unnamed threads in MR2
[ https://issues.apache.org/jira/browse/MAPREDUCE-3187?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13128861#comment-13128861 ] Hudson commented on MAPREDUCE-3187: --- Integrated in Hadoop-Mapreduce-0.23-Build #54 (See [https://builds.apache.org/job/Hadoop-Mapreduce-0.23-Build/54/]) MAPREDUCE-3187. Add names for various unnamed threads in MR2. (Todd Lipcon and Siddharth Seth via mahadev) - Merging r1184904 from trunk mahadev : http://svn.apache.org/viewcvs.cgi/?root=Apache-SVNview=revrev=1184905 Files : * /hadoop/common/branches/branch-0.23/hadoop-mapreduce-project/CHANGES.txt * /hadoop/common/branches/branch-0.23/hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-app/src/main/java/org/apache/hadoop/mapreduce/v2/app/TaskHeartbeatHandler.java * /hadoop/common/branches/branch-0.23/hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-app/src/main/java/org/apache/hadoop/mapreduce/v2/app/launcher/ContainerLauncherImpl.java * /hadoop/common/branches/branch-0.23/hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-app/src/main/java/org/apache/hadoop/mapreduce/v2/app/rm/RMCommunicator.java * /hadoop/common/branches/branch-0.23/hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-app/src/main/java/org/apache/hadoop/mapreduce/v2/app/taskclean/TaskCleanerImpl.java * /hadoop/common/branches/branch-0.23/hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-hs/src/main/java/org/apache/hadoop/mapreduce/v2/hs/JobHistory.java * /hadoop/common/branches/branch-0.23/hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-shuffle/src/main/java/org/apache/hadoop/mapred/ShuffleHandler.java * /hadoop/common/branches/branch-0.23/hadoop-mapreduce-project/hadoop-yarn/hadoop-yarn-common/src/main/java/org/apache/hadoop/yarn/event/AsyncDispatcher.java * /hadoop/common/branches/branch-0.23/hadoop-mapreduce-project/hadoop-yarn/hadoop-yarn-common/src/main/java/org/apache/hadoop/yarn/service/CompositeService.java * /hadoop/common/branches/branch-0.23/hadoop-mapreduce-project/hadoop-yarn/hadoop-yarn-common/src/main/java/org/apache/hadoop/yarn/util/AbstractLivelinessMonitor.java * /hadoop/common/branches/branch-0.23/hadoop-mapreduce-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-nodemanager/src/main/java/org/apache/hadoop/yarn/server/nodemanager/DeletionService.java * /hadoop/common/branches/branch-0.23/hadoop-mapreduce-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-nodemanager/src/main/java/org/apache/hadoop/yarn/server/nodemanager/NodeStatusUpdaterImpl.java * /hadoop/common/branches/branch-0.23/hadoop-mapreduce-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-nodemanager/src/main/java/org/apache/hadoop/yarn/server/nodemanager/containermanager/launcher/ContainersLauncher.java * /hadoop/common/branches/branch-0.23/hadoop-mapreduce-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-nodemanager/src/main/java/org/apache/hadoop/yarn/server/nodemanager/containermanager/localizer/ContainerLocalizer.java * /hadoop/common/branches/branch-0.23/hadoop-mapreduce-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-nodemanager/src/main/java/org/apache/hadoop/yarn/server/nodemanager/containermanager/localizer/ResourceLocalizationService.java * /hadoop/common/branches/branch-0.23/hadoop-mapreduce-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-nodemanager/src/main/java/org/apache/hadoop/yarn/server/nodemanager/containermanager/logaggregation/LogAggregationService.java * /hadoop/common/branches/branch-0.23/hadoop-mapreduce-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-nodemanager/src/main/java/org/apache/hadoop/yarn/server/nodemanager/containermanager/monitor/ContainersMonitorImpl.java * /hadoop/common/branches/branch-0.23/hadoop-mapreduce-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-nodemanager/src/main/resources/container-log4j.properties * /hadoop/common/branches/branch-0.23/hadoop-mapreduce-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager/src/main/java/org/apache/hadoop/yarn/server/resourcemanager/ResourceManager.java * /hadoop/common/branches/branch-0.23/hadoop-mapreduce-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager/src/main/java/org/apache/hadoop/yarn/server/resourcemanager/amlauncher/ApplicationMasterLauncher.java * /hadoop/common/branches/branch-0.23/hadoop-mapreduce-project/src/test/log4j.properties Add names for various unnamed threads in MR2 Key: MAPREDUCE-3187 URL: https://issues.apache.org/jira/browse/MAPREDUCE-3187 Project: Hadoop Map/Reduce Issue Type: Improvement Components: mrv2 Affects Versions: 0.23.0 Reporter: Todd Lipcon
[jira] [Commented] (MAPREDUCE-2840) mr279 TestUberAM.testSleepJob test fails
[ https://issues.apache.org/jira/browse/MAPREDUCE-2840?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13128862#comment-13128862 ] Hudson commented on MAPREDUCE-2840: --- Integrated in Hadoop-Mapreduce-0.23-Build #54 (See [https://builds.apache.org/job/Hadoop-Mapreduce-0.23-Build/54/]) MAPREDUCE-2840. mr279 TestUberAM.testSleepJob test fails. (jonathan eagles via mahadev) - Merging r1184971 from trunk mahadev : http://svn.apache.org/viewcvs.cgi/?root=Apache-SVNview=revrev=1184972 Files : * /hadoop/common/branches/branch-0.23/hadoop-mapreduce-project/CHANGES.txt * /hadoop/common/branches/branch-0.23/hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-jobclient/src/test/java/org/apache/hadoop/mapreduce/v2/TestMRJobs.java * /hadoop/common/branches/branch-0.23/hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-jobclient/src/test/java/org/apache/hadoop/mapreduce/v2/TestUberAM.java mr279 TestUberAM.testSleepJob test fails Key: MAPREDUCE-2840 URL: https://issues.apache.org/jira/browse/MAPREDUCE-2840 Project: Hadoop Map/Reduce Issue Type: Bug Components: mrv2 Affects Versions: 0.23.0 Reporter: Thomas Graves Assignee: Jonathan Eagles Priority: Minor Fix For: 0.23.0 Attachments: MAPREDUCE-2840.patch Currently the TestUberAM.testSleepJob is failing on the mr279 branch. snippet of failure: junit.framework.AssertionFailedError: null at junit.framework.Assert.fail(Assert.java:47) at junit.framework.Assert.assertTrue(Assert.java:20) at junit.framework.Assert.assertTrue(Assert.java:27) at org.apache.hadoop.mapreduce.v2.TestMRJobs.testSleepJob(TestMRJobs.java:150) at org.apache.hadoop.mapreduce.v2.TestUberAM.testSleepJob(TestUberAM.java:58) at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method) at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:39) at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:25) at java.lang.reflect.Method.invoke(Method.java:597) -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (MAPREDUCE-3136) Add docs for setting up real-world MRv2 clusters
[ https://issues.apache.org/jira/browse/MAPREDUCE-3136?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13128859#comment-13128859 ] Hudson commented on MAPREDUCE-3136: --- Integrated in Hadoop-Mapreduce-0.23-Build #54 (See [https://builds.apache.org/job/Hadoop-Mapreduce-0.23-Build/54/]) Merge -c 1184981 from trunk to branch-0.23 to complete fix for MAPREDUCE-3136. Merge -c 1184977 from trunk to branch-0.23 to fix MAPREDUCE-3136. acmurthy : http://svn.apache.org/viewcvs.cgi/?root=Apache-SVNview=revrev=1184982 Files : * /hadoop/common/branches/branch-0.23/hadoop-mapreduce-project/hadoop-yarn/hadoop-yarn-site/src/site/apt/index.apt.vm acmurthy : http://svn.apache.org/viewcvs.cgi/?root=Apache-SVNview=revrev=1184978 Files : * /hadoop/common/branches/branch-0.23/hadoop-mapreduce-project/CHANGES.txt * /hadoop/common/branches/branch-0.23/hadoop-mapreduce-project/hadoop-yarn/hadoop-yarn-site/src/site/apt/CapacityScheduler.apt.vm * /hadoop/common/branches/branch-0.23/hadoop-mapreduce-project/hadoop-yarn/hadoop-yarn-site/src/site/apt/ClusterSetup.apt.vm Add docs for setting up real-world MRv2 clusters Key: MAPREDUCE-3136 URL: https://issues.apache.org/jira/browse/MAPREDUCE-3136 Project: Hadoop Map/Reduce Issue Type: Sub-task Components: documentation, mrv2 Reporter: Arun C Murthy Assignee: Arun C Murthy Priority: Blocker Fix For: 0.23.0 Attachments: MAPREDUCE-3136.patch, MAPREDUCE-3136.patch, MAPREDUCE-3136.patch, MAPREDUCE-3136.patch, MAPREDUCE-3136.patch_1 Add docs for setting up real-world MRv2 clusters - MR portion of http://hadoop.apache.org/common/docs/stable/cluster_setup.html -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (MAPREDUCE-3189) Add link decoration back to MR2's CSS
[ https://issues.apache.org/jira/browse/MAPREDUCE-3189?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13128860#comment-13128860 ] Hudson commented on MAPREDUCE-3189: --- Integrated in Hadoop-Mapreduce-0.23-Build #54 (See [https://builds.apache.org/job/Hadoop-Mapreduce-0.23-Build/54/]) MAPREDUCE-3189. Add link decoration back to MR2's CSS. (Todd Lipcon via mahadev) - Merging r1184979 from trunk mahadev : http://svn.apache.org/viewcvs.cgi/?root=Apache-SVNview=revrev=1184980 Files : * /hadoop/common/branches/branch-0.23/hadoop-mapreduce-project/CHANGES.txt * /hadoop/common/branches/branch-0.23/hadoop-mapreduce-project/hadoop-yarn/hadoop-yarn-common/src/main/resources/webapps/static/yarn.css Add link decoration back to MR2's CSS - Key: MAPREDUCE-3189 URL: https://issues.apache.org/jira/browse/MAPREDUCE-3189 Project: Hadoop Map/Reduce Issue Type: Improvement Components: mrv2 Affects Versions: 0.23.0 Reporter: Todd Lipcon Assignee: Todd Lipcon Fix For: 0.23.0 Attachments: mr-3189.txt I found the MRv2 web UI very difficult to use because it's not clear which items are links and which aren't. I'd like to change the CSS so that links are underlined, making it easier to see them (since they're also not in any different color) -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (MAPREDUCE-3101) Security issues in YARN
[ https://issues.apache.org/jira/browse/MAPREDUCE-3101?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13128863#comment-13128863 ] Hudson commented on MAPREDUCE-3101: --- Integrated in Hadoop-Mapreduce-0.23-Build #54 (See [https://builds.apache.org/job/Hadoop-Mapreduce-0.23-Build/54/]) Merge -c 1184989 from trunk to branch-0.23 to complete fix for MAPREDUCE-3101. acmurthy : http://svn.apache.org/viewcvs.cgi/?root=Apache-SVNview=revrev=1184990 Files : * /hadoop/common/branches/branch-0.23/hadoop-mapreduce-project/CHANGES.txt * /hadoop/common/branches/branch-0.23/hadoop-mapreduce-project/hadoop-yarn/hadoop-yarn-common/src/main/java/org/apache/hadoop/yarn/conf/YarnConfiguration.java * /hadoop/common/branches/branch-0.23/hadoop-mapreduce-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-common/src/main/resources/yarn-default.xml * /hadoop/common/branches/branch-0.23/hadoop-mapreduce-project/hadoop-yarn/hadoop-yarn-site/src/site/apt/ClusterSetup.apt.vm Security issues in YARN --- Key: MAPREDUCE-3101 URL: https://issues.apache.org/jira/browse/MAPREDUCE-3101 Project: Hadoop Map/Reduce Issue Type: Bug Components: security Affects Versions: 0.23.0 Reporter: Vinod Kumar Vavilapalli Assignee: Vinod Kumar Vavilapalli Fix For: 0.23.0 Most of the chassis for security in YARN is set up and is working. There are known bugs and security holes though. This JIRA is an umbrella ticket for tracking those. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (MAPREDUCE-3136) Add docs for setting up real-world MRv2 clusters
[ https://issues.apache.org/jira/browse/MAPREDUCE-3136?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13128870#comment-13128870 ] Hudson commented on MAPREDUCE-3136: --- Integrated in Hadoop-Hdfs-trunk #833 (See [https://builds.apache.org/job/Hadoop-Hdfs-trunk/833/]) MAPREDUCE-3136. Added the missing link to index.html. MAPREDUCE-3136. Added documentation for setting up Hadoop clusters in both non-secure and secure mode for both HDFS YARN. acmurthy : http://svn.apache.org/viewcvs.cgi/?root=Apache-SVNview=revrev=1184981 Files : * /hadoop/common/trunk/hadoop-mapreduce-project/hadoop-yarn/hadoop-yarn-site/src/site/apt/index.apt.vm acmurthy : http://svn.apache.org/viewcvs.cgi/?root=Apache-SVNview=revrev=1184977 Files : * /hadoop/common/trunk/hadoop-mapreduce-project/CHANGES.txt * /hadoop/common/trunk/hadoop-mapreduce-project/hadoop-yarn/hadoop-yarn-site/src/site/apt/CapacityScheduler.apt.vm * /hadoop/common/trunk/hadoop-mapreduce-project/hadoop-yarn/hadoop-yarn-site/src/site/apt/ClusterSetup.apt.vm Add docs for setting up real-world MRv2 clusters Key: MAPREDUCE-3136 URL: https://issues.apache.org/jira/browse/MAPREDUCE-3136 Project: Hadoop Map/Reduce Issue Type: Sub-task Components: documentation, mrv2 Reporter: Arun C Murthy Assignee: Arun C Murthy Priority: Blocker Fix For: 0.23.0 Attachments: MAPREDUCE-3136.patch, MAPREDUCE-3136.patch, MAPREDUCE-3136.patch, MAPREDUCE-3136.patch, MAPREDUCE-3136.patch_1 Add docs for setting up real-world MRv2 clusters - MR portion of http://hadoop.apache.org/common/docs/stable/cluster_setup.html -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (MAPREDUCE-3190) bin/yarn should barf early if HADOOP_COMMON_HOME or HADOOP_HDFS_HOME are not set
[
https://issues.apache.org/jira/browse/MAPREDUCE-3190?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13128869#comment-13128869
]
Hudson commented on MAPREDUCE-3190:
---
Integrated in Hadoop-Hdfs-trunk #833 (See
[https://builds.apache.org/job/Hadoop-Hdfs-trunk/833/])
MAPREDUCE-3190. Ensure bin/yarn fails early with a clear error message when
HADOOP_COMMON_HOME or HADOOP_HDFS_HOME are not set. Contributed by todd
acmurthy.
acmurthy :
http://svn.apache.org/viewcvs.cgi/?root=Apache-SVNview=revrev=1184975
Files :
* /hadoop/common/trunk/hadoop-mapreduce-project/CHANGES.txt
* /hadoop/common/trunk/hadoop-mapreduce-project/hadoop-yarn/bin/yarn
bin/yarn should barf early if HADOOP_COMMON_HOME or HADOOP_HDFS_HOME are not
set
Key: MAPREDUCE-3190
URL: https://issues.apache.org/jira/browse/MAPREDUCE-3190
Project: Hadoop Map/Reduce
Issue Type: Improvement
Components: mrv2
Affects Versions: 0.23.0
Reporter: Todd Lipcon
Assignee: Todd Lipcon
Fix For: 0.23.0
Attachments: mr-3190.txt, mr-3190.txt
Currently, if these env vars are not set when you run bin/yarn, it will crash
with various ClassNotFoundExceptions, having added {{/share/hadoop/hdfs}} to
the classpath. Rather, we should check for these env vars in the wrapper
script and display a reasonable error message.
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators:
https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (MAPREDUCE-3189) Add link decoration back to MR2's CSS
[ https://issues.apache.org/jira/browse/MAPREDUCE-3189?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13128871#comment-13128871 ] Hudson commented on MAPREDUCE-3189: --- Integrated in Hadoop-Hdfs-trunk #833 (See [https://builds.apache.org/job/Hadoop-Hdfs-trunk/833/]) MAPREDUCE-3189. Add link decoration back to MR2's CSS. (Todd Lipcon via mahadev) mahadev : http://svn.apache.org/viewcvs.cgi/?root=Apache-SVNview=revrev=1184979 Files : * /hadoop/common/trunk/hadoop-mapreduce-project/CHANGES.txt * /hadoop/common/trunk/hadoop-mapreduce-project/hadoop-yarn/hadoop-yarn-common/src/main/resources/webapps/static/yarn.css Add link decoration back to MR2's CSS - Key: MAPREDUCE-3189 URL: https://issues.apache.org/jira/browse/MAPREDUCE-3189 Project: Hadoop Map/Reduce Issue Type: Improvement Components: mrv2 Affects Versions: 0.23.0 Reporter: Todd Lipcon Assignee: Todd Lipcon Fix For: 0.23.0 Attachments: mr-3189.txt I found the MRv2 web UI very difficult to use because it's not clear which items are links and which aren't. I'd like to change the CSS so that links are underlined, making it easier to see them (since they're also not in any different color) -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (MAPREDUCE-2840) mr279 TestUberAM.testSleepJob test fails
[ https://issues.apache.org/jira/browse/MAPREDUCE-2840?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13128873#comment-13128873 ] Hudson commented on MAPREDUCE-2840: --- Integrated in Hadoop-Hdfs-trunk #833 (See [https://builds.apache.org/job/Hadoop-Hdfs-trunk/833/]) MAPREDUCE-2840. mr279 TestUberAM.testSleepJob test fails. (jonathan eagles via mahadev) mahadev : http://svn.apache.org/viewcvs.cgi/?root=Apache-SVNview=revrev=1184971 Files : * /hadoop/common/trunk/hadoop-mapreduce-project/CHANGES.txt * /hadoop/common/trunk/hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-jobclient/src/test/java/org/apache/hadoop/mapreduce/v2/TestMRJobs.java * /hadoop/common/trunk/hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-jobclient/src/test/java/org/apache/hadoop/mapreduce/v2/TestUberAM.java mr279 TestUberAM.testSleepJob test fails Key: MAPREDUCE-2840 URL: https://issues.apache.org/jira/browse/MAPREDUCE-2840 Project: Hadoop Map/Reduce Issue Type: Bug Components: mrv2 Affects Versions: 0.23.0 Reporter: Thomas Graves Assignee: Jonathan Eagles Priority: Minor Fix For: 0.23.0 Attachments: MAPREDUCE-2840.patch Currently the TestUberAM.testSleepJob is failing on the mr279 branch. snippet of failure: junit.framework.AssertionFailedError: null at junit.framework.Assert.fail(Assert.java:47) at junit.framework.Assert.assertTrue(Assert.java:20) at junit.framework.Assert.assertTrue(Assert.java:27) at org.apache.hadoop.mapreduce.v2.TestMRJobs.testSleepJob(TestMRJobs.java:150) at org.apache.hadoop.mapreduce.v2.TestUberAM.testSleepJob(TestUberAM.java:58) at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method) at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:39) at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:25) at java.lang.reflect.Method.invoke(Method.java:597) -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (MAPREDUCE-3187) Add names for various unnamed threads in MR2
[ https://issues.apache.org/jira/browse/MAPREDUCE-3187?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13128872#comment-13128872 ] Hudson commented on MAPREDUCE-3187: --- Integrated in Hadoop-Hdfs-trunk #833 (See [https://builds.apache.org/job/Hadoop-Hdfs-trunk/833/]) MAPREDUCE-3187. Add names for various unnamed threads in MR2. (Todd Lipcon and Siddharth Seth via mahadev) mahadev : http://svn.apache.org/viewcvs.cgi/?root=Apache-SVNview=revrev=1184904 Files : * /hadoop/common/trunk/hadoop-mapreduce-project/CHANGES.txt * /hadoop/common/trunk/hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-app/src/main/java/org/apache/hadoop/mapreduce/v2/app/TaskHeartbeatHandler.java * /hadoop/common/trunk/hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-app/src/main/java/org/apache/hadoop/mapreduce/v2/app/launcher/ContainerLauncherImpl.java * /hadoop/common/trunk/hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-app/src/main/java/org/apache/hadoop/mapreduce/v2/app/rm/RMCommunicator.java * /hadoop/common/trunk/hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-app/src/main/java/org/apache/hadoop/mapreduce/v2/app/taskclean/TaskCleanerImpl.java * /hadoop/common/trunk/hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-hs/src/main/java/org/apache/hadoop/mapreduce/v2/hs/JobHistory.java * /hadoop/common/trunk/hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-shuffle/src/main/java/org/apache/hadoop/mapred/ShuffleHandler.java * /hadoop/common/trunk/hadoop-mapreduce-project/hadoop-yarn/hadoop-yarn-common/src/main/java/org/apache/hadoop/yarn/event/AsyncDispatcher.java * /hadoop/common/trunk/hadoop-mapreduce-project/hadoop-yarn/hadoop-yarn-common/src/main/java/org/apache/hadoop/yarn/service/CompositeService.java * /hadoop/common/trunk/hadoop-mapreduce-project/hadoop-yarn/hadoop-yarn-common/src/main/java/org/apache/hadoop/yarn/util/AbstractLivelinessMonitor.java * /hadoop/common/trunk/hadoop-mapreduce-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-nodemanager/src/main/java/org/apache/hadoop/yarn/server/nodemanager/DeletionService.java * /hadoop/common/trunk/hadoop-mapreduce-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-nodemanager/src/main/java/org/apache/hadoop/yarn/server/nodemanager/NodeStatusUpdaterImpl.java * /hadoop/common/trunk/hadoop-mapreduce-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-nodemanager/src/main/java/org/apache/hadoop/yarn/server/nodemanager/containermanager/launcher/ContainersLauncher.java * /hadoop/common/trunk/hadoop-mapreduce-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-nodemanager/src/main/java/org/apache/hadoop/yarn/server/nodemanager/containermanager/localizer/ContainerLocalizer.java * /hadoop/common/trunk/hadoop-mapreduce-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-nodemanager/src/main/java/org/apache/hadoop/yarn/server/nodemanager/containermanager/localizer/ResourceLocalizationService.java * /hadoop/common/trunk/hadoop-mapreduce-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-nodemanager/src/main/java/org/apache/hadoop/yarn/server/nodemanager/containermanager/logaggregation/LogAggregationService.java * /hadoop/common/trunk/hadoop-mapreduce-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-nodemanager/src/main/java/org/apache/hadoop/yarn/server/nodemanager/containermanager/monitor/ContainersMonitorImpl.java * /hadoop/common/trunk/hadoop-mapreduce-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-nodemanager/src/main/resources/container-log4j.properties * /hadoop/common/trunk/hadoop-mapreduce-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager/src/main/java/org/apache/hadoop/yarn/server/resourcemanager/ResourceManager.java * /hadoop/common/trunk/hadoop-mapreduce-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager/src/main/java/org/apache/hadoop/yarn/server/resourcemanager/amlauncher/ApplicationMasterLauncher.java * /hadoop/common/trunk/hadoop-mapreduce-project/src/test/log4j.properties Add names for various unnamed threads in MR2 Key: MAPREDUCE-3187 URL: https://issues.apache.org/jira/browse/MAPREDUCE-3187 Project: Hadoop Map/Reduce Issue Type: Improvement Components: mrv2 Affects Versions: 0.23.0 Reporter: Todd Lipcon Assignee: Todd Lipcon Priority: Minor Fix For: 0.23.0 Attachments: mr-3187.txt, mr-3187.txt, mr3187_increment.txt Simple patch to add thread names for all the places we use Executors, etc. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators:
[jira] [Commented] (MAPREDUCE-3101) Security issues in YARN
[ https://issues.apache.org/jira/browse/MAPREDUCE-3101?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13128874#comment-13128874 ] Hudson commented on MAPREDUCE-3101: --- Integrated in Hadoop-Hdfs-trunk #833 (See [https://builds.apache.org/job/Hadoop-Hdfs-trunk/833/]) MAPREDUCE-3101. Changed default value of yarn.resourcemanager.acl.enable to true and added some more documentation. acmurthy : http://svn.apache.org/viewcvs.cgi/?root=Apache-SVNview=revrev=1184989 Files : * /hadoop/common/trunk/hadoop-mapreduce-project/CHANGES.txt * /hadoop/common/trunk/hadoop-mapreduce-project/hadoop-yarn/hadoop-yarn-common/src/main/java/org/apache/hadoop/yarn/conf/YarnConfiguration.java * /hadoop/common/trunk/hadoop-mapreduce-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-common/src/main/resources/yarn-default.xml * /hadoop/common/trunk/hadoop-mapreduce-project/hadoop-yarn/hadoop-yarn-site/src/site/apt/ClusterSetup.apt.vm Security issues in YARN --- Key: MAPREDUCE-3101 URL: https://issues.apache.org/jira/browse/MAPREDUCE-3101 Project: Hadoop Map/Reduce Issue Type: Bug Components: security Affects Versions: 0.23.0 Reporter: Vinod Kumar Vavilapalli Assignee: Vinod Kumar Vavilapalli Fix For: 0.23.0 Most of the chassis for security in YARN is set up and is working. There are known bugs and security holes though. This JIRA is an umbrella ticket for tracking those. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (MAPREDUCE-3127) Unable to restrict users based on resourcemanager.admin.acls value set
[ https://issues.apache.org/jira/browse/MAPREDUCE-3127?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13128875#comment-13128875 ] Hudson commented on MAPREDUCE-3127: --- Integrated in Hadoop-Hdfs-trunk #833 (See [https://builds.apache.org/job/Hadoop-Hdfs-trunk/833/]) MAPREDUCE-3127. Fixed CHANGES.txt to reflect correct jira. acmurthy : http://svn.apache.org/viewcvs.cgi/?root=Apache-SVNview=revrev=1184992 Files : * /hadoop/common/trunk/hadoop-mapreduce-project/CHANGES.txt Unable to restrict users based on resourcemanager.admin.acls value set -- Key: MAPREDUCE-3127 URL: https://issues.apache.org/jira/browse/MAPREDUCE-3127 Project: Hadoop Map/Reduce Issue Type: Sub-task Components: mrv2, resourcemanager Affects Versions: 0.23.0 Reporter: Amol Kekre Assignee: Arun C Murthy Priority: Blocker Fix For: 0.23.0 Attachments: MAPREDUCE-3101.patch, MAPREDUCE-3101.patch, MAPREDUCE-3101.patch Setting the following property in yarn-site.xml with user ids to restrict ability to run 'rmadmin -refreshQueues is not honoured property nameyarn.server.resourcemanager.admin.acls/name valuehadoop1/value description/description final/final /property Should it be the same for rmadmin -refreshNodes? -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (MAPREDUCE-3057) Job History Server goes of OutOfMemory with 1200 Jobs and Heap Size set to 10 GB
[ https://issues.apache.org/jira/browse/MAPREDUCE-3057?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13128878#comment-13128878 ] Eric Payne commented on MAPREDUCE-3057: --- Great! Thanks Mahadev. Job History Server goes of OutOfMemory with 1200 Jobs and Heap Size set to 10 GB Key: MAPREDUCE-3057 URL: https://issues.apache.org/jira/browse/MAPREDUCE-3057 Project: Hadoop Map/Reduce Issue Type: Bug Components: jobhistoryserver, mrv2 Affects Versions: 0.23.0 Reporter: Karam Singh Assignee: Eric Payne Priority: Blocker Fix For: 0.23.0 Attachments: MAPREDUCE-3057.v1.txt History server was started with -Xmx1m Ran GridMix V3 with 1200 Jobs trace in STRESS mode on 350 nodes with each node 4 NMS. All jobs finished as reported by RM Web UI and HADOOP_MAPRED_HOME/bin/mapred job -list all But found that GridMix job client was stuck while trying connect to HistoryServer Then tried to do HADOOP_MAPRED_HOME/bin/mapred job -status jobid JobClient also got stuck while looking for token to connect to History server Then looked at History Server logs and found History is trowing java.lang.OutOfMemoryError: GC overhead limit exceeded error. With 10GB of Heap space and 1200 Jobs, History Server should not go out of memory . No matter what are the type of jobs. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Updated] (MAPREDUCE-2747) [MR-279] [Security] Cleanup LinuxContainerExecutor binary sources
[ https://issues.apache.org/jira/browse/MAPREDUCE-2747?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Robert Joseph Evans updated MAPREDUCE-2747: --- Attachment: MR-2747.txt Updated the config variable. [MR-279] [Security] Cleanup LinuxContainerExecutor binary sources - Key: MAPREDUCE-2747 URL: https://issues.apache.org/jira/browse/MAPREDUCE-2747 Project: Hadoop Map/Reduce Issue Type: Sub-task Components: mrv2, nodemanager, security Affects Versions: 0.23.0, 0.24.0 Reporter: Vinod Kumar Vavilapalli Assignee: Robert Joseph Evans Priority: Blocker Fix For: 0.23.0, 0.24.0 Attachments: MAPREDUCE-2747.patch, MR-2747.txt, MR-2747.txt There are a lot of references to the old task-controller nomenclature still, job/task refs instead of app/container. Also the configuration file is named as taskcontroller.cfg and the configured variables are also from the mapred world (mrv1). These SHOULD be fixed before we make a release. Marking this as blocker. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Updated] (MAPREDUCE-2747) [MR-279] [Security] Cleanup LinuxContainerExecutor binary sources
[ https://issues.apache.org/jira/browse/MAPREDUCE-2747?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Robert Joseph Evans updated MAPREDUCE-2747: --- Target Version/s: 0.23.0, 0.24.0 (was: 0.24.0, 0.23.0) Status: Patch Available (was: Open) [MR-279] [Security] Cleanup LinuxContainerExecutor binary sources - Key: MAPREDUCE-2747 URL: https://issues.apache.org/jira/browse/MAPREDUCE-2747 Project: Hadoop Map/Reduce Issue Type: Sub-task Components: mrv2, nodemanager, security Affects Versions: 0.23.0, 0.24.0 Reporter: Vinod Kumar Vavilapalli Assignee: Robert Joseph Evans Priority: Blocker Fix For: 0.23.0, 0.24.0 Attachments: MAPREDUCE-2747.patch, MR-2747.txt, MR-2747.txt There are a lot of references to the old task-controller nomenclature still, job/task refs instead of app/container. Also the configuration file is named as taskcontroller.cfg and the configured variables are also from the mapred world (mrv1). These SHOULD be fixed before we make a release. Marking this as blocker. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (MAPREDUCE-2747) [MR-279] [Security] Cleanup LinuxContainerExecutor binary sources
[ https://issues.apache.org/jira/browse/MAPREDUCE-2747?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13128885#comment-13128885 ] Hadoop QA commented on MAPREDUCE-2747: -- -1 overall. Here are the results of testing the latest attachment http://issues.apache.org/jira/secure/attachment/12499369/MR-2747.txt against trunk revision . +1 @author. The patch does not contain any @author tags. +1 tests included. The patch appears to include 18 new or modified tests. -1 patch. The patch command could not apply the patch. Console output: https://builds.apache.org/job/PreCommit-MAPREDUCE-Build/1037//console This message is automatically generated. [MR-279] [Security] Cleanup LinuxContainerExecutor binary sources - Key: MAPREDUCE-2747 URL: https://issues.apache.org/jira/browse/MAPREDUCE-2747 Project: Hadoop Map/Reduce Issue Type: Sub-task Components: mrv2, nodemanager, security Affects Versions: 0.23.0, 0.24.0 Reporter: Vinod Kumar Vavilapalli Assignee: Robert Joseph Evans Priority: Blocker Fix For: 0.23.0, 0.24.0 Attachments: MAPREDUCE-2747.patch, MR-2747.txt, MR-2747.txt There are a lot of references to the old task-controller nomenclature still, job/task refs instead of app/container. Also the configuration file is named as taskcontroller.cfg and the configured variables are also from the mapred world (mrv1). These SHOULD be fixed before we make a release. Marking this as blocker. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Updated] (MAPREDUCE-2747) [MR-279] [Security] Cleanup LinuxContainerExecutor binary sources
[ https://issues.apache.org/jira/browse/MAPREDUCE-2747?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Robert Joseph Evans updated MAPREDUCE-2747: --- Target Version/s: 0.23.0, 0.24.0 (was: 0.24.0, 0.23.0) Status: Open (was: Patch Available) Sorry I forgot to apply the changes that went in on trunk. [MR-279] [Security] Cleanup LinuxContainerExecutor binary sources - Key: MAPREDUCE-2747 URL: https://issues.apache.org/jira/browse/MAPREDUCE-2747 Project: Hadoop Map/Reduce Issue Type: Sub-task Components: mrv2, nodemanager, security Affects Versions: 0.23.0, 0.24.0 Reporter: Vinod Kumar Vavilapalli Assignee: Robert Joseph Evans Priority: Blocker Fix For: 0.23.0, 0.24.0 Attachments: MAPREDUCE-2747.patch, MR-2747.txt, MR-2747.txt There are a lot of references to the old task-controller nomenclature still, job/task refs instead of app/container. Also the configuration file is named as taskcontroller.cfg and the configured variables are also from the mapred world (mrv1). These SHOULD be fixed before we make a release. Marking this as blocker. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Updated] (MAPREDUCE-3104) Implement Application ACLs, Queue ACLs and their interaction
[ https://issues.apache.org/jira/browse/MAPREDUCE-3104?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Vinod Kumar Vavilapalli updated MAPREDUCE-3104: --- Attachment: MAPREDUCE-3104-20111017.3.txt Attaching patch for review. This includes ACLs checks at the Client-RM interaction, on the RM web-UI for the per-app page and on the NM UI for the containers' logs. Also has tests verifying the Client-RM acls checks. Verified the client access, RM and NM webUIs access checks on my single node secure setup. Will add the unit tests for web-UI access separately, this patch has already ballooned quite a bit now. Implement Application ACLs, Queue ACLs and their interaction Key: MAPREDUCE-3104 URL: https://issues.apache.org/jira/browse/MAPREDUCE-3104 Project: Hadoop Map/Reduce Issue Type: Sub-task Components: mrv2, resourcemanager, security Affects Versions: 0.23.0 Reporter: Vinod Kumar Vavilapalli Assignee: Vinod Kumar Vavilapalli Priority: Blocker Fix For: 0.23.0 Attachments: MAPREDUCE-3104-20111017.3.txt -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Updated] (MAPREDUCE-2747) [MR-279] [Security] Cleanup LinuxContainerExecutor binary sources
[ https://issues.apache.org/jira/browse/MAPREDUCE-2747?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Robert Joseph Evans updated MAPREDUCE-2747: --- Target Version/s: 0.23.0, 0.24.0 (was: 0.24.0, 0.23.0) Status: Patch Available (was: Open) [MR-279] [Security] Cleanup LinuxContainerExecutor binary sources - Key: MAPREDUCE-2747 URL: https://issues.apache.org/jira/browse/MAPREDUCE-2747 Project: Hadoop Map/Reduce Issue Type: Sub-task Components: mrv2, nodemanager, security Affects Versions: 0.23.0, 0.24.0 Reporter: Vinod Kumar Vavilapalli Assignee: Robert Joseph Evans Priority: Blocker Fix For: 0.23.0, 0.24.0 Attachments: MAPREDUCE-2747.patch, MR-2747-branch-0.23.txt, MR-2747.txt, MR-2747.txt, MR-2747.txt There are a lot of references to the old task-controller nomenclature still, job/task refs instead of app/container. Also the configuration file is named as taskcontroller.cfg and the configured variables are also from the mapred world (mrv1). These SHOULD be fixed before we make a release. Marking this as blocker. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Updated] (MAPREDUCE-2747) [MR-279] [Security] Cleanup LinuxContainerExecutor binary sources
[ https://issues.apache.org/jira/browse/MAPREDUCE-2747?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Robert Joseph Evans updated MAPREDUCE-2747: --- Attachment: MR-2747.txt MR-2747-branch-0.23.txt Now we have a patch for 0.23, and for trunk. The only difference between the two is that 0.23 has a .gitignore file in it, because the executables are being generated in the same dir as the source. And the directories have been updated accordingly. [MR-279] [Security] Cleanup LinuxContainerExecutor binary sources - Key: MAPREDUCE-2747 URL: https://issues.apache.org/jira/browse/MAPREDUCE-2747 Project: Hadoop Map/Reduce Issue Type: Sub-task Components: mrv2, nodemanager, security Affects Versions: 0.23.0, 0.24.0 Reporter: Vinod Kumar Vavilapalli Assignee: Robert Joseph Evans Priority: Blocker Fix For: 0.23.0, 0.24.0 Attachments: MAPREDUCE-2747.patch, MR-2747-branch-0.23.txt, MR-2747.txt, MR-2747.txt, MR-2747.txt There are a lot of references to the old task-controller nomenclature still, job/task refs instead of app/container. Also the configuration file is named as taskcontroller.cfg and the configured variables are also from the mapred world (mrv1). These SHOULD be fixed before we make a release. Marking this as blocker. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Updated] (MAPREDUCE-3104) Implement Application ACLs, Queue ACLs and their interaction
[ https://issues.apache.org/jira/browse/MAPREDUCE-3104?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Vinod Kumar Vavilapalli updated MAPREDUCE-3104: --- Status: Patch Available (was: Open) Implement Application ACLs, Queue ACLs and their interaction Key: MAPREDUCE-3104 URL: https://issues.apache.org/jira/browse/MAPREDUCE-3104 Project: Hadoop Map/Reduce Issue Type: Sub-task Components: mrv2, resourcemanager, security Affects Versions: 0.23.0 Reporter: Vinod Kumar Vavilapalli Assignee: Vinod Kumar Vavilapalli Priority: Blocker Fix For: 0.23.0 Attachments: MAPREDUCE-3104-20111017.3.txt -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Updated] (MAPREDUCE-3032) JobHistory doesn't have error information from failed tasks
[ https://issues.apache.org/jira/browse/MAPREDUCE-3032?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Devaraj K updated MAPREDUCE-3032: - Attachment: MAPREDUCE-3032-1.patch JobHistory doesn't have error information from failed tasks --- Key: MAPREDUCE-3032 URL: https://issues.apache.org/jira/browse/MAPREDUCE-3032 Project: Hadoop Map/Reduce Issue Type: Bug Components: applicationmaster, mrv2 Affects Versions: 0.23.0 Reporter: Vinod Kumar Vavilapalli Assignee: Devaraj K Priority: Blocker Fix For: 0.23.0 Attachments: Failed map attempts.jpg, JobFail.PNG, MAPREDUCE-3032-1.patch, MAPREDUCE-3032.patch -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Updated] (MAPREDUCE-3032) JobHistory doesn't have error information from failed tasks
[ https://issues.apache.org/jira/browse/MAPREDUCE-3032?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Devaraj K updated MAPREDUCE-3032: - Status: Patch Available (was: Open) Updated the patch by removing TaskAttemptStatus.diagnosticInfo and added tests. JobHistory doesn't have error information from failed tasks --- Key: MAPREDUCE-3032 URL: https://issues.apache.org/jira/browse/MAPREDUCE-3032 Project: Hadoop Map/Reduce Issue Type: Bug Components: applicationmaster, mrv2 Affects Versions: 0.23.0 Reporter: Vinod Kumar Vavilapalli Assignee: Devaraj K Priority: Blocker Fix For: 0.23.0 Attachments: Failed map attempts.jpg, JobFail.PNG, MAPREDUCE-3032-1.patch, MAPREDUCE-3032.patch -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (MAPREDUCE-3104) Implement Application ACLs, Queue ACLs and their interaction
[ https://issues.apache.org/jira/browse/MAPREDUCE-3104?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13128910#comment-13128910 ] Hadoop QA commented on MAPREDUCE-3104: -- +1 overall. Here are the results of testing the latest attachment http://issues.apache.org/jira/secure/attachment/12499378/MAPREDUCE-3104-20111017.3.txt against trunk revision . +1 @author. The patch does not contain any @author tags. +1 tests included. The patch appears to include 41 new or modified tests. +1 javadoc. The javadoc tool did not generate any warning messages. +1 javac. The applied patch does not increase the total number of javac compiler warnings. +1 findbugs. The patch does not introduce any new Findbugs (version 1.3.9) warnings. +1 release audit. The applied patch does not increase the total number of release audit warnings. +1 core tests. The patch passed unit tests in . +1 contrib tests. The patch passed contrib unit tests. Test results: https://builds.apache.org/job/PreCommit-MAPREDUCE-Build/1038//testReport/ Console output: https://builds.apache.org/job/PreCommit-MAPREDUCE-Build/1038//console This message is automatically generated. Implement Application ACLs, Queue ACLs and their interaction Key: MAPREDUCE-3104 URL: https://issues.apache.org/jira/browse/MAPREDUCE-3104 Project: Hadoop Map/Reduce Issue Type: Sub-task Components: mrv2, resourcemanager, security Affects Versions: 0.23.0 Reporter: Vinod Kumar Vavilapalli Assignee: Vinod Kumar Vavilapalli Priority: Blocker Fix For: 0.23.0 Attachments: MAPREDUCE-3104-20111017.3.txt -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (MAPREDUCE-2747) [MR-279] [Security] Cleanup LinuxContainerExecutor binary sources
[ https://issues.apache.org/jira/browse/MAPREDUCE-2747?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13128911#comment-13128911 ] Hadoop QA commented on MAPREDUCE-2747: -- +1 overall. Here are the results of testing the latest attachment http://issues.apache.org/jira/secure/attachment/12499377/MR-2747.txt against trunk revision . +1 @author. The patch does not contain any @author tags. +1 tests included. The patch appears to include 16 new or modified tests. +1 javadoc. The javadoc tool did not generate any warning messages. +1 javac. The applied patch does not increase the total number of javac compiler warnings. +1 findbugs. The patch does not introduce any new Findbugs (version 1.3.9) warnings. +1 release audit. The applied patch does not increase the total number of release audit warnings. +1 core tests. The patch passed unit tests in . +1 contrib tests. The patch passed contrib unit tests. Test results: https://builds.apache.org/job/PreCommit-MAPREDUCE-Build/1039//testReport/ Console output: https://builds.apache.org/job/PreCommit-MAPREDUCE-Build/1039//console This message is automatically generated. [MR-279] [Security] Cleanup LinuxContainerExecutor binary sources - Key: MAPREDUCE-2747 URL: https://issues.apache.org/jira/browse/MAPREDUCE-2747 Project: Hadoop Map/Reduce Issue Type: Sub-task Components: mrv2, nodemanager, security Affects Versions: 0.23.0, 0.24.0 Reporter: Vinod Kumar Vavilapalli Assignee: Robert Joseph Evans Priority: Blocker Fix For: 0.23.0, 0.24.0 Attachments: MAPREDUCE-2747.patch, MR-2747-branch-0.23.txt, MR-2747.txt, MR-2747.txt, MR-2747.txt There are a lot of references to the old task-controller nomenclature still, job/task refs instead of app/container. Also the configuration file is named as taskcontroller.cfg and the configured variables are also from the mapred world (mrv1). These SHOULD be fixed before we make a release. Marking this as blocker. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (MAPREDUCE-3032) JobHistory doesn't have error information from failed tasks
[ https://issues.apache.org/jira/browse/MAPREDUCE-3032?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13128921#comment-13128921 ] Hadoop QA commented on MAPREDUCE-3032: -- +1 overall. Here are the results of testing the latest attachment http://issues.apache.org/jira/secure/attachment/12499381/MAPREDUCE-3032-1.patch against trunk revision . +1 @author. The patch does not contain any @author tags. +1 tests included. The patch appears to include 6 new or modified tests. +1 javadoc. The javadoc tool did not generate any warning messages. +1 javac. The applied patch does not increase the total number of javac compiler warnings. +1 findbugs. The patch does not introduce any new Findbugs (version 1.3.9) warnings. +1 release audit. The applied patch does not increase the total number of release audit warnings. +1 core tests. The patch passed unit tests in . +1 contrib tests. The patch passed contrib unit tests. Test results: https://builds.apache.org/job/PreCommit-MAPREDUCE-Build/1040//testReport/ Console output: https://builds.apache.org/job/PreCommit-MAPREDUCE-Build/1040//console This message is automatically generated. JobHistory doesn't have error information from failed tasks --- Key: MAPREDUCE-3032 URL: https://issues.apache.org/jira/browse/MAPREDUCE-3032 Project: Hadoop Map/Reduce Issue Type: Bug Components: applicationmaster, mrv2 Affects Versions: 0.23.0 Reporter: Vinod Kumar Vavilapalli Assignee: Devaraj K Priority: Blocker Fix For: 0.23.0 Attachments: Failed map attempts.jpg, JobFail.PNG, MAPREDUCE-3032-1.patch, MAPREDUCE-3032.patch -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (MAPREDUCE-3176) ant mapreduce tests are timing out
[ https://issues.apache.org/jira/browse/MAPREDUCE-3176?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13128925#comment-13128925 ] Ravi Prakash commented on MAPREDUCE-3176: - Thanks Hitesh! I applied your patch and ran TestFileSystem. It passed. I'll ask RE to apply the patch and run the secondary again. ant mapreduce tests are timing out -- Key: MAPREDUCE-3176 URL: https://issues.apache.org/jira/browse/MAPREDUCE-3176 Project: Hadoop Map/Reduce Issue Type: Bug Components: mrv2, test Affects Versions: 0.23.0 Reporter: Ravi Prakash Assignee: Hitesh Shah Priority: Blocker Labels: test Fix For: 0.23.0 Attachments: MR-3176.wip.patch Secondary YARN builds started taking inordinately long and lots of tests started failing. Usually the secondary build would take ~ 2 hours. But recently even after 7 hours it wasn't done. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (MAPREDUCE-3032) JobHistory doesn't have error information from failed tasks
[ https://issues.apache.org/jira/browse/MAPREDUCE-3032?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13128932#comment-13128932 ] Vinod Kumar Vavilapalli commented on MAPREDUCE-3032: This looks good now. +1. JobHistory doesn't have error information from failed tasks --- Key: MAPREDUCE-3032 URL: https://issues.apache.org/jira/browse/MAPREDUCE-3032 Project: Hadoop Map/Reduce Issue Type: Bug Components: applicationmaster, mrv2 Affects Versions: 0.23.0 Reporter: Vinod Kumar Vavilapalli Assignee: Devaraj K Priority: Blocker Fix For: 0.23.0 Attachments: Failed map attempts.jpg, JobFail.PNG, MAPREDUCE-3032-1.patch, MAPREDUCE-3032.patch -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (MAPREDUCE-2858) MRv2 WebApp Security
[ https://issues.apache.org/jira/browse/MAPREDUCE-2858?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13128934#comment-13128934 ] Todd Lipcon commented on MAPREDUCE-2858: bq. To give you an extreme example, it's completely acceptable for the scanner to always return true for the maybeUnsafe method How do you purport to even _find_ the JavaScript? I don't believe you can write an ad-hoc parser that detects all known ways of embedding JavaScript. I think Allen's (and other's) ideas of having a whitelisted set of ApplicationMaster jars is the only real way to achieve this. Any time non-trusted code is used, the proxy should generate an interstitial page indicating untrusted content, regardless of the HTML content of the target page. If we can't do a whitelist of binaries, we should _always_ generate the interstitial when the visitor doesn't match the submitter. MRv2 WebApp Security Key: MAPREDUCE-2858 URL: https://issues.apache.org/jira/browse/MAPREDUCE-2858 Project: Hadoop Map/Reduce Issue Type: Sub-task Components: applicationmaster, mrv2, security Affects Versions: 0.23.0 Reporter: Luke Lu Assignee: Luke Lu Priority: Blocker Fix For: 0.23.0 In MRv2, while the system servers (ResourceManager (RM), NodeManager (NM) and NameNode (NN)) run as trusted system users, the application masters (AM) run as users who submit the application. While this offers great flexibility to run multiple version of mapreduce frameworks (including their UI) on the same Hadoop cluster, it has significant implication for the security of webapps (Please do not discuss company specific vulnerabilities here). Requirements: # Secure authentication for AM (for app/job level ACLs). # Webapp security should be optional via site configuration. # Support existing pluggable single sign on mechanisms. # Should not require per app/user configuration for deployment. # Should not require special site-wide DNS configuration for deployment. This the top jira for webapp security. A design doc/notes of threat-modeling and counter measures will be posted on the wiki. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (MAPREDUCE-3032) JobHistory doesn't have error information from failed tasks
[ https://issues.apache.org/jira/browse/MAPREDUCE-3032?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13128940#comment-13128940 ] Hudson commented on MAPREDUCE-3032: --- Integrated in Hadoop-Hdfs-trunk-Commit #1173 (See [https://builds.apache.org/job/Hadoop-Hdfs-trunk-Commit/1173/]) MAPREDUCE-3032. Fixed TaskAttemptImpl so that JobHistory can have error information about failed tasks. Contributed by Devaraj K. vinodkv : http://svn.apache.org/viewcvs.cgi/?root=Apache-SVNview=revrev=1185247 Files : * /hadoop/common/trunk/hadoop-mapreduce-project/CHANGES.txt * /hadoop/common/trunk/hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-app/src/main/java/org/apache/hadoop/mapred/TaskAttemptListenerImpl.java * /hadoop/common/trunk/hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-app/src/main/java/org/apache/hadoop/mapreduce/v2/app/job/event/TaskAttemptStatusUpdateEvent.java * /hadoop/common/trunk/hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-app/src/main/java/org/apache/hadoop/mapreduce/v2/app/job/impl/TaskAttemptImpl.java * /hadoop/common/trunk/hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-app/src/main/java/org/apache/hadoop/mapreduce/v2/app/recover/RecoveryService.java * /hadoop/common/trunk/hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-app/src/test/java/org/apache/hadoop/mapreduce/v2/app/TestMRClientService.java * /hadoop/common/trunk/hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-app/src/test/java/org/apache/hadoop/mapreduce/v2/app/job/impl/TestTaskAttempt.java JobHistory doesn't have error information from failed tasks --- Key: MAPREDUCE-3032 URL: https://issues.apache.org/jira/browse/MAPREDUCE-3032 Project: Hadoop Map/Reduce Issue Type: Bug Components: applicationmaster, mrv2 Affects Versions: 0.23.0 Reporter: Vinod Kumar Vavilapalli Assignee: Devaraj K Priority: Blocker Fix For: 0.23.0 Attachments: Failed map attempts.jpg, JobFail.PNG, MAPREDUCE-3032-1.patch, MAPREDUCE-3032.patch -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Updated] (MAPREDUCE-3032) JobHistory doesn't have error information from failed tasks
[ https://issues.apache.org/jira/browse/MAPREDUCE-3032?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Vinod Kumar Vavilapalli updated MAPREDUCE-3032: --- Resolution: Fixed Hadoop Flags: Reviewed Status: Resolved (was: Patch Available) i just committed this to trunk and branch-0.23. Thanks Devaraj K! JobHistory doesn't have error information from failed tasks --- Key: MAPREDUCE-3032 URL: https://issues.apache.org/jira/browse/MAPREDUCE-3032 Project: Hadoop Map/Reduce Issue Type: Bug Components: applicationmaster, mrv2 Affects Versions: 0.23.0 Reporter: Vinod Kumar Vavilapalli Assignee: Devaraj K Priority: Blocker Fix For: 0.23.0 Attachments: Failed map attempts.jpg, JobFail.PNG, MAPREDUCE-3032-1.patch, MAPREDUCE-3032.patch -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (MAPREDUCE-3032) JobHistory doesn't have error information from failed tasks
[ https://issues.apache.org/jira/browse/MAPREDUCE-3032?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13128939#comment-13128939 ] Hudson commented on MAPREDUCE-3032: --- Integrated in Hadoop-Common-trunk-Commit #1094 (See [https://builds.apache.org/job/Hadoop-Common-trunk-Commit/1094/]) MAPREDUCE-3032. Fixed TaskAttemptImpl so that JobHistory can have error information about failed tasks. Contributed by Devaraj K. vinodkv : http://svn.apache.org/viewcvs.cgi/?root=Apache-SVNview=revrev=1185247 Files : * /hadoop/common/trunk/hadoop-mapreduce-project/CHANGES.txt * /hadoop/common/trunk/hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-app/src/main/java/org/apache/hadoop/mapred/TaskAttemptListenerImpl.java * /hadoop/common/trunk/hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-app/src/main/java/org/apache/hadoop/mapreduce/v2/app/job/event/TaskAttemptStatusUpdateEvent.java * /hadoop/common/trunk/hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-app/src/main/java/org/apache/hadoop/mapreduce/v2/app/job/impl/TaskAttemptImpl.java * /hadoop/common/trunk/hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-app/src/main/java/org/apache/hadoop/mapreduce/v2/app/recover/RecoveryService.java * /hadoop/common/trunk/hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-app/src/test/java/org/apache/hadoop/mapreduce/v2/app/TestMRClientService.java * /hadoop/common/trunk/hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-app/src/test/java/org/apache/hadoop/mapreduce/v2/app/job/impl/TestTaskAttempt.java JobHistory doesn't have error information from failed tasks --- Key: MAPREDUCE-3032 URL: https://issues.apache.org/jira/browse/MAPREDUCE-3032 Project: Hadoop Map/Reduce Issue Type: Bug Components: applicationmaster, mrv2 Affects Versions: 0.23.0 Reporter: Vinod Kumar Vavilapalli Assignee: Devaraj K Priority: Blocker Fix For: 0.23.0 Attachments: Failed map attempts.jpg, JobFail.PNG, MAPREDUCE-3032-1.patch, MAPREDUCE-3032.patch -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (MAPREDUCE-3032) JobHistory doesn't have error information from failed tasks
[ https://issues.apache.org/jira/browse/MAPREDUCE-3032?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13128944#comment-13128944 ] Hudson commented on MAPREDUCE-3032: --- Integrated in Hadoop-Hdfs-0.23-Commit #12 (See [https://builds.apache.org/job/Hadoop-Hdfs-0.23-Commit/12/]) MAPREDUCE-3032. Fixed TaskAttemptImpl so that JobHistory can have error information about failed tasks. Contributed by Devaraj K. svn merge -c r1185247 --ignore-ancestry ../../trunk/ vinodkv : http://svn.apache.org/viewcvs.cgi/?root=Apache-SVNview=revrev=1185250 Files : * /hadoop/common/branches/branch-0.23/hadoop-mapreduce-project/CHANGES.txt * /hadoop/common/branches/branch-0.23/hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-app/src/main/java/org/apache/hadoop/mapred/TaskAttemptListenerImpl.java * /hadoop/common/branches/branch-0.23/hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-app/src/main/java/org/apache/hadoop/mapreduce/v2/app/job/event/TaskAttemptStatusUpdateEvent.java * /hadoop/common/branches/branch-0.23/hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-app/src/main/java/org/apache/hadoop/mapreduce/v2/app/job/impl/TaskAttemptImpl.java * /hadoop/common/branches/branch-0.23/hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-app/src/main/java/org/apache/hadoop/mapreduce/v2/app/recover/RecoveryService.java * /hadoop/common/branches/branch-0.23/hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-app/src/test/java/org/apache/hadoop/mapreduce/v2/app/TestMRClientService.java * /hadoop/common/branches/branch-0.23/hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-app/src/test/java/org/apache/hadoop/mapreduce/v2/app/job/impl/TestTaskAttempt.java JobHistory doesn't have error information from failed tasks --- Key: MAPREDUCE-3032 URL: https://issues.apache.org/jira/browse/MAPREDUCE-3032 Project: Hadoop Map/Reduce Issue Type: Bug Components: applicationmaster, mrv2 Affects Versions: 0.23.0 Reporter: Vinod Kumar Vavilapalli Assignee: Devaraj K Priority: Blocker Fix For: 0.23.0 Attachments: Failed map attempts.jpg, JobFail.PNG, MAPREDUCE-3032-1.patch, MAPREDUCE-3032.patch -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (MAPREDUCE-3032) JobHistory doesn't have error information from failed tasks
[ https://issues.apache.org/jira/browse/MAPREDUCE-3032?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13128954#comment-13128954 ] Hudson commented on MAPREDUCE-3032: --- Integrated in Hadoop-Mapreduce-0.23-Commit #13 (See [https://builds.apache.org/job/Hadoop-Mapreduce-0.23-Commit/13/]) MAPREDUCE-3032. Fixed TaskAttemptImpl so that JobHistory can have error information about failed tasks. Contributed by Devaraj K. svn merge -c r1185247 --ignore-ancestry ../../trunk/ vinodkv : http://svn.apache.org/viewcvs.cgi/?root=Apache-SVNview=revrev=1185250 Files : * /hadoop/common/branches/branch-0.23/hadoop-mapreduce-project/CHANGES.txt * /hadoop/common/branches/branch-0.23/hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-app/src/main/java/org/apache/hadoop/mapred/TaskAttemptListenerImpl.java * /hadoop/common/branches/branch-0.23/hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-app/src/main/java/org/apache/hadoop/mapreduce/v2/app/job/event/TaskAttemptStatusUpdateEvent.java * /hadoop/common/branches/branch-0.23/hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-app/src/main/java/org/apache/hadoop/mapreduce/v2/app/job/impl/TaskAttemptImpl.java * /hadoop/common/branches/branch-0.23/hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-app/src/main/java/org/apache/hadoop/mapreduce/v2/app/recover/RecoveryService.java * /hadoop/common/branches/branch-0.23/hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-app/src/test/java/org/apache/hadoop/mapreduce/v2/app/TestMRClientService.java * /hadoop/common/branches/branch-0.23/hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-app/src/test/java/org/apache/hadoop/mapreduce/v2/app/job/impl/TestTaskAttempt.java JobHistory doesn't have error information from failed tasks --- Key: MAPREDUCE-3032 URL: https://issues.apache.org/jira/browse/MAPREDUCE-3032 Project: Hadoop Map/Reduce Issue Type: Bug Components: applicationmaster, mrv2 Affects Versions: 0.23.0 Reporter: Vinod Kumar Vavilapalli Assignee: Devaraj K Priority: Blocker Fix For: 0.23.0 Attachments: Failed map attempts.jpg, JobFail.PNG, MAPREDUCE-3032-1.patch, MAPREDUCE-3032.patch -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Updated] (MAPREDUCE-3179) Incorrect exit code for hadoop-mapreduce-test tests when exception thrown
[ https://issues.apache.org/jira/browse/MAPREDUCE-3179?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Jonathan Eagles updated MAPREDUCE-3179: --- Target Version/s: 0.23.0, 0.24.0 (was: 0.24.0, 0.23.0) Status: Open (was: Patch Available) Incorrect exit code for hadoop-mapreduce-test tests when exception thrown - Key: MAPREDUCE-3179 URL: https://issues.apache.org/jira/browse/MAPREDUCE-3179 Project: Hadoop Map/Reduce Issue Type: Bug Components: mrv2, test Affects Versions: 0.20.205.0, 0.23.0, 0.24.0 Reporter: Jonathan Eagles Assignee: Jonathan Eagles Fix For: 0.23.0 Attachments: MAPREDUCE-3179.patch Exit code for test jar is 0 despite exception thrown hadoop jar hadoop-mapreduce-test-0.23.0-SNAPSHOT.jar loadgen -Dmapreduce.job.acl-view -m 18 -r 0 -outKey org.apache.hadoop.io.Text -outValue org.apache.hadoop.io.Text -indir nonexistentdir Loadgen output snippet org.apache.hadoop.mapred.InvalidInputException: Input path does not exist: hdfs://machine.name.example.com:9000/user/exampleuser/nonexistentdir at org.apache.hadoop.mapred.FileInputFormat.listStatus(FileInputFormat.java:234) at org.apache.hadoop.mapred.FileInputFormat.getSplits(FileInputFormat.java:254) at org.apache.hadoop.mapreduce.JobSubmitter.writeOldSplits(JobSubmitter.java:470) at org.apache.hadoop.mapreduce.JobSubmitter.writeSplits(JobSubmitter.java:462) at org.apache.hadoop.mapreduce.JobSubmitter.submitJobInternal(JobSubmitter.java:358) at org.apache.hadoop.mapreduce.Job$2.run(Job.java:1159) at org.apache.hadoop.mapreduce.Job$2.run(Job.java:1156) at java.security.AccessController.doPrivileged(Native Method) at javax.security.auth.Subject.doAs(Subject.java:396) at org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1135) at org.apache.hadoop.mapreduce.Job.submit(Job.java:1156) at org.apache.hadoop.mapred.JobClient.submitJob(JobClient.java:539) at org.apache.hadoop.mapred.JobClient.runJob(JobClient.java:778) at org.apache.hadoop.mapred.GenericMRLoadGenerator.run(GenericMRLoadGenerator.java:200) at org.apache.hadoop.util.ToolRunner.run(ToolRunner.java:69) at org.apache.hadoop.mapred.GenericMRLoadGenerator.main(GenericMRLoadGenerator.java:214) at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method) at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:39) at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:25) at java.lang.reflect.Method.invoke(Method.java:597) at org.apache.hadoop.util.ProgramDriver$ProgramDescription.invoke(ProgramDriver.java:72) at org.apache.hadoop.util.ProgramDriver.driver(ProgramDriver.java:144) at org.apache.hadoop.test.MapredTestDriver.run(MapredTestDriver.java:111) at org.apache.hadoop.test.MapredTestDriver.main(MapredTestDriver.java:118) at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method) at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:39) at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:25) at java.lang.reflect.Method.invoke(Method.java:597) at org.apache.hadoop.util.RunJar.main(RunJar.java:189) -bash-3.2$ echo $? 0 This differs from example jar which correctly returns the correct exit code hadoop jar hadoop-mapreduce-examples-0.23.0-SNAPSHOT.jar wordcount nonexistentdir /outputdir wordcount output snippet org.apache.hadoop.mapreduce.lib.input.InvalidInputException: Input path does not exist: hdfs://machine.name.example.com:9000/user/exampleuser/nonexistentdir at org.apache.hadoop.mapreduce.lib.input.FileInputFormat.listStatus(FileInputFormat.java:243) at org.apache.hadoop.mapreduce.lib.input.FileInputFormat.getSplits(FileInputFormat.java:269) at org.apache.hadoop.mapreduce.JobSubmitter.writeNewSplits(JobSubmitter.java:443) at org.apache.hadoop.mapreduce.JobSubmitter.writeSplits(JobSubmitter.java:460) at org.apache.hadoop.mapreduce.JobSubmitter.submitJobInternal(JobSubmitter.java:358) at org.apache.hadoop.mapreduce.Job$2.run(Job.java:1159) at org.apache.hadoop.mapreduce.Job$2.run(Job.java:1156) at java.security.AccessController.doPrivileged(Native Method) at javax.security.auth.Subject.doAs(Subject.java:396) at org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1135) at org.apache.hadoop.mapreduce.Job.submit(Job.java:1156) at
[jira] [Updated] (MAPREDUCE-3179) Incorrect exit code for hadoop-mapreduce-test tests when exception thrown
[ https://issues.apache.org/jira/browse/MAPREDUCE-3179?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Jonathan Eagles updated MAPREDUCE-3179: --- Target Version/s: 0.23.0, 0.24.0 (was: 0.24.0, 0.23.0) Status: Patch Available (was: Open) Incorrect exit code for hadoop-mapreduce-test tests when exception thrown - Key: MAPREDUCE-3179 URL: https://issues.apache.org/jira/browse/MAPREDUCE-3179 Project: Hadoop Map/Reduce Issue Type: Bug Components: mrv2, test Affects Versions: 0.20.205.0, 0.23.0, 0.24.0 Reporter: Jonathan Eagles Assignee: Jonathan Eagles Fix For: 0.23.0 Attachments: MAPREDUCE-3179.patch Exit code for test jar is 0 despite exception thrown hadoop jar hadoop-mapreduce-test-0.23.0-SNAPSHOT.jar loadgen -Dmapreduce.job.acl-view -m 18 -r 0 -outKey org.apache.hadoop.io.Text -outValue org.apache.hadoop.io.Text -indir nonexistentdir Loadgen output snippet org.apache.hadoop.mapred.InvalidInputException: Input path does not exist: hdfs://machine.name.example.com:9000/user/exampleuser/nonexistentdir at org.apache.hadoop.mapred.FileInputFormat.listStatus(FileInputFormat.java:234) at org.apache.hadoop.mapred.FileInputFormat.getSplits(FileInputFormat.java:254) at org.apache.hadoop.mapreduce.JobSubmitter.writeOldSplits(JobSubmitter.java:470) at org.apache.hadoop.mapreduce.JobSubmitter.writeSplits(JobSubmitter.java:462) at org.apache.hadoop.mapreduce.JobSubmitter.submitJobInternal(JobSubmitter.java:358) at org.apache.hadoop.mapreduce.Job$2.run(Job.java:1159) at org.apache.hadoop.mapreduce.Job$2.run(Job.java:1156) at java.security.AccessController.doPrivileged(Native Method) at javax.security.auth.Subject.doAs(Subject.java:396) at org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1135) at org.apache.hadoop.mapreduce.Job.submit(Job.java:1156) at org.apache.hadoop.mapred.JobClient.submitJob(JobClient.java:539) at org.apache.hadoop.mapred.JobClient.runJob(JobClient.java:778) at org.apache.hadoop.mapred.GenericMRLoadGenerator.run(GenericMRLoadGenerator.java:200) at org.apache.hadoop.util.ToolRunner.run(ToolRunner.java:69) at org.apache.hadoop.mapred.GenericMRLoadGenerator.main(GenericMRLoadGenerator.java:214) at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method) at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:39) at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:25) at java.lang.reflect.Method.invoke(Method.java:597) at org.apache.hadoop.util.ProgramDriver$ProgramDescription.invoke(ProgramDriver.java:72) at org.apache.hadoop.util.ProgramDriver.driver(ProgramDriver.java:144) at org.apache.hadoop.test.MapredTestDriver.run(MapredTestDriver.java:111) at org.apache.hadoop.test.MapredTestDriver.main(MapredTestDriver.java:118) at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method) at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:39) at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:25) at java.lang.reflect.Method.invoke(Method.java:597) at org.apache.hadoop.util.RunJar.main(RunJar.java:189) -bash-3.2$ echo $? 0 This differs from example jar which correctly returns the correct exit code hadoop jar hadoop-mapreduce-examples-0.23.0-SNAPSHOT.jar wordcount nonexistentdir /outputdir wordcount output snippet org.apache.hadoop.mapreduce.lib.input.InvalidInputException: Input path does not exist: hdfs://machine.name.example.com:9000/user/exampleuser/nonexistentdir at org.apache.hadoop.mapreduce.lib.input.FileInputFormat.listStatus(FileInputFormat.java:243) at org.apache.hadoop.mapreduce.lib.input.FileInputFormat.getSplits(FileInputFormat.java:269) at org.apache.hadoop.mapreduce.JobSubmitter.writeNewSplits(JobSubmitter.java:443) at org.apache.hadoop.mapreduce.JobSubmitter.writeSplits(JobSubmitter.java:460) at org.apache.hadoop.mapreduce.JobSubmitter.submitJobInternal(JobSubmitter.java:358) at org.apache.hadoop.mapreduce.Job$2.run(Job.java:1159) at org.apache.hadoop.mapreduce.Job$2.run(Job.java:1156) at java.security.AccessController.doPrivileged(Native Method) at javax.security.auth.Subject.doAs(Subject.java:396) at org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1135) at org.apache.hadoop.mapreduce.Job.submit(Job.java:1156) at
[jira] [Updated] (MAPREDUCE-3144) Augment JobHistory to include information needed for serving aggregated logs.
[ https://issues.apache.org/jira/browse/MAPREDUCE-3144?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Vinod Kumar Vavilapalli updated MAPREDUCE-3144: --- Status: Open (was: Patch Available) Sid, the patch needs minor merge to trunk, can you please do that? I cursorily looked at the patch anyways. Few things overall: - The start-time is to be set by RM instead of the client, so it should not be part of the ApplicationSubmissionContext record? - ContainerId already has the ApplicationAttemptID, so we can pass ContainerId alone. - Can rename the ContainerId env to AM_CONTAINER_ID. Looked fine overall otherwise, but will look at the patch more carefully, tests etc, once you put up the updated patch. Thanks. Augment JobHistory to include information needed for serving aggregated logs. - Key: MAPREDUCE-3144 URL: https://issues.apache.org/jira/browse/MAPREDUCE-3144 Project: Hadoop Map/Reduce Issue Type: Sub-task Components: mrv2 Affects Versions: 0.23.0 Reporter: Vinod Kumar Vavilapalli Assignee: Siddharth Seth Priority: Critical Fix For: 0.23.0 Attachments: MR3144.patch, MR3144_v2.txt, MR3144_v3.txt -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (MAPREDUCE-3179) Incorrect exit code for hadoop-mapreduce-test tests when exception thrown
[ https://issues.apache.org/jira/browse/MAPREDUCE-3179?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13128964#comment-13128964 ] Jonathan Eagles commented on MAPREDUCE-3179: Please ignore the test results of the branch security patch. Instead look at the patch results against trunk. Incorrect exit code for hadoop-mapreduce-test tests when exception thrown - Key: MAPREDUCE-3179 URL: https://issues.apache.org/jira/browse/MAPREDUCE-3179 Project: Hadoop Map/Reduce Issue Type: Bug Components: mrv2, test Affects Versions: 0.20.205.0, 0.23.0, 0.24.0 Reporter: Jonathan Eagles Assignee: Jonathan Eagles Fix For: 0.23.0 Attachments: MAPREDUCE-3179.patch Exit code for test jar is 0 despite exception thrown hadoop jar hadoop-mapreduce-test-0.23.0-SNAPSHOT.jar loadgen -Dmapreduce.job.acl-view -m 18 -r 0 -outKey org.apache.hadoop.io.Text -outValue org.apache.hadoop.io.Text -indir nonexistentdir Loadgen output snippet org.apache.hadoop.mapred.InvalidInputException: Input path does not exist: hdfs://machine.name.example.com:9000/user/exampleuser/nonexistentdir at org.apache.hadoop.mapred.FileInputFormat.listStatus(FileInputFormat.java:234) at org.apache.hadoop.mapred.FileInputFormat.getSplits(FileInputFormat.java:254) at org.apache.hadoop.mapreduce.JobSubmitter.writeOldSplits(JobSubmitter.java:470) at org.apache.hadoop.mapreduce.JobSubmitter.writeSplits(JobSubmitter.java:462) at org.apache.hadoop.mapreduce.JobSubmitter.submitJobInternal(JobSubmitter.java:358) at org.apache.hadoop.mapreduce.Job$2.run(Job.java:1159) at org.apache.hadoop.mapreduce.Job$2.run(Job.java:1156) at java.security.AccessController.doPrivileged(Native Method) at javax.security.auth.Subject.doAs(Subject.java:396) at org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1135) at org.apache.hadoop.mapreduce.Job.submit(Job.java:1156) at org.apache.hadoop.mapred.JobClient.submitJob(JobClient.java:539) at org.apache.hadoop.mapred.JobClient.runJob(JobClient.java:778) at org.apache.hadoop.mapred.GenericMRLoadGenerator.run(GenericMRLoadGenerator.java:200) at org.apache.hadoop.util.ToolRunner.run(ToolRunner.java:69) at org.apache.hadoop.mapred.GenericMRLoadGenerator.main(GenericMRLoadGenerator.java:214) at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method) at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:39) at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:25) at java.lang.reflect.Method.invoke(Method.java:597) at org.apache.hadoop.util.ProgramDriver$ProgramDescription.invoke(ProgramDriver.java:72) at org.apache.hadoop.util.ProgramDriver.driver(ProgramDriver.java:144) at org.apache.hadoop.test.MapredTestDriver.run(MapredTestDriver.java:111) at org.apache.hadoop.test.MapredTestDriver.main(MapredTestDriver.java:118) at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method) at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:39) at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:25) at java.lang.reflect.Method.invoke(Method.java:597) at org.apache.hadoop.util.RunJar.main(RunJar.java:189) -bash-3.2$ echo $? 0 This differs from example jar which correctly returns the correct exit code hadoop jar hadoop-mapreduce-examples-0.23.0-SNAPSHOT.jar wordcount nonexistentdir /outputdir wordcount output snippet org.apache.hadoop.mapreduce.lib.input.InvalidInputException: Input path does not exist: hdfs://machine.name.example.com:9000/user/exampleuser/nonexistentdir at org.apache.hadoop.mapreduce.lib.input.FileInputFormat.listStatus(FileInputFormat.java:243) at org.apache.hadoop.mapreduce.lib.input.FileInputFormat.getSplits(FileInputFormat.java:269) at org.apache.hadoop.mapreduce.JobSubmitter.writeNewSplits(JobSubmitter.java:443) at org.apache.hadoop.mapreduce.JobSubmitter.writeSplits(JobSubmitter.java:460) at org.apache.hadoop.mapreduce.JobSubmitter.submitJobInternal(JobSubmitter.java:358) at org.apache.hadoop.mapreduce.Job$2.run(Job.java:1159) at org.apache.hadoop.mapreduce.Job$2.run(Job.java:1156) at java.security.AccessController.doPrivileged(Native Method) at javax.security.auth.Subject.doAs(Subject.java:396) at org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1135)
[jira] [Created] (MAPREDUCE-3194) mapred mradmin command is broken in mrv2
mapred mradmin command is broken in mrv2 -- Key: MAPREDUCE-3194 URL: https://issues.apache.org/jira/browse/MAPREDUCE-3194 Project: Hadoop Map/Reduce Issue Type: Bug Components: mrv2 Affects Versions: 0.23.0 Reporter: Siddharth Seth $mapred mradmin Exception in thread main java.lang.NoClassDefFoundError: org/apache/hadoop/mapred/tools/MRAdmin Caused by: java.lang.ClassNotFoundException: org.apache.hadoop.mapred.tools.MRAdmin at java.net.URLClassLoader$1.run(URLClassLoader.java:202) at java.security.AccessController.doPrivileged(Native Method) at java.net.URLClassLoader.findClass(URLClassLoader.java:190) at java.lang.ClassLoader.loadClass(ClassLoader.java:307) at sun.misc.Launcher$AppClassLoader.loadClass(Launcher.java:301) at java.lang.ClassLoader.loadClass(ClassLoader.java:248) Could not find the main class: org.apache.hadoop.mapred.tools.MRAdmin. Program will exit. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Updated] (MAPREDUCE-3068) Should set MALLOC_ARENA_MAX for all YARN daemons and AMs/Containers
[ https://issues.apache.org/jira/browse/MAPREDUCE-3068?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Chris Riccomini updated MAPREDUCE-3068: --- Status: Patch Available (was: Open) Should set MALLOC_ARENA_MAX for all YARN daemons and AMs/Containers --- Key: MAPREDUCE-3068 URL: https://issues.apache.org/jira/browse/MAPREDUCE-3068 Project: Hadoop Map/Reduce Issue Type: Bug Components: mrv2 Affects Versions: 0.23.0 Reporter: Vinod Kumar Vavilapalli Assignee: Chris Riccomini Priority: Blocker Fix For: 0.23.0 Attachments: MAPREDUCE-3068.diff, MAPREDUCE-3068.diff, MAPREDUCE-3068.diff, MAPREDUCE-3068.diff This is same as HADOOP-7154 but for yarn. RM, NM, AM and containers should all have this. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Updated] (MAPREDUCE-3068) Should set MALLOC_ARENA_MAX for all YARN daemons and AMs/Containers
[ https://issues.apache.org/jira/browse/MAPREDUCE-3068?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Chris Riccomini updated MAPREDUCE-3068: --- Attachment: MAPREDUCE-3068.diff changing back to putIfAbsent Should set MALLOC_ARENA_MAX for all YARN daemons and AMs/Containers --- Key: MAPREDUCE-3068 URL: https://issues.apache.org/jira/browse/MAPREDUCE-3068 Project: Hadoop Map/Reduce Issue Type: Bug Components: mrv2 Affects Versions: 0.23.0 Reporter: Vinod Kumar Vavilapalli Assignee: Chris Riccomini Priority: Blocker Fix For: 0.23.0 Attachments: MAPREDUCE-3068.diff, MAPREDUCE-3068.diff, MAPREDUCE-3068.diff, MAPREDUCE-3068.diff This is same as HADOOP-7154 but for yarn. RM, NM, AM and containers should all have this. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (MAPREDUCE-3176) ant mapreduce tests are timing out
[ https://issues.apache.org/jira/browse/MAPREDUCE-3176?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13128975#comment-13128975 ] Vinod Kumar Vavilapalli commented on MAPREDUCE-3176: I can also confirm that TestFileSystem is passing with the patch. Am currently running all the tests. That will take a while, can commit this once that passes. But before that. Hitesh, there are TODOs that you added in TestFileSytem, you intending to fix them? ant mapreduce tests are timing out -- Key: MAPREDUCE-3176 URL: https://issues.apache.org/jira/browse/MAPREDUCE-3176 Project: Hadoop Map/Reduce Issue Type: Bug Components: mrv2, test Affects Versions: 0.23.0 Reporter: Ravi Prakash Assignee: Hitesh Shah Priority: Blocker Labels: test Fix For: 0.23.0 Attachments: MR-3176.wip.patch Secondary YARN builds started taking inordinately long and lots of tests started failing. Usually the secondary build would take ~ 2 hours. But recently even after 7 hours it wasn't done. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (MAPREDUCE-2858) MRv2 WebApp Security
[ https://issues.apache.org/jira/browse/MAPREDUCE-2858?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13128984#comment-13128984 ] Luke Lu commented on MAPREDUCE-2858: bq. How do you purport to even find the JavaScript? I don't believe you can write an ad-hoc parser that detects all known ways of embedding JavaScript. This is a good point. The scanner needs to be aggressive (false positive is OK) to cover all script cases: anything in between script/object/embed tags, any on*|codebase|src|href attributes, while considering charset/encoding of the response. That's why typical HTML parsers are not suitable for this task. The scanner needs to be carefully reviewed. Fortunately, it's pretty short as it doesn't try to do any remotely as ambitious as Caja, which ha numerous security bugs over the years. As I mentioned, a trivial scanner that always return true for #maybeUnsafe is acceptable. bq. having a whitelisted set of ApplicationMaster jars is the only real way to achieve this. Again, whitelisting AM jars is complementary to the proxy. But it's not sufficient for real world usage, as there'll be too many jars to review and extremely inconvenient/counter-productive for cluster/cloud/grid usage. bq. If we can't do a whitelist of binaries, we should always generate the interstitial when the visitor doesn't match the submitter. +1. I personally would recommend this even with whitelisted binaries and log every action (visitor username, webapp username, visitor choice (proceed, view source, strip (non-whitelisted) scripts (may break content) etc). MRv2 WebApp Security Key: MAPREDUCE-2858 URL: https://issues.apache.org/jira/browse/MAPREDUCE-2858 Project: Hadoop Map/Reduce Issue Type: Sub-task Components: applicationmaster, mrv2, security Affects Versions: 0.23.0 Reporter: Luke Lu Assignee: Luke Lu Priority: Blocker Fix For: 0.23.0 In MRv2, while the system servers (ResourceManager (RM), NodeManager (NM) and NameNode (NN)) run as trusted system users, the application masters (AM) run as users who submit the application. While this offers great flexibility to run multiple version of mapreduce frameworks (including their UI) on the same Hadoop cluster, it has significant implication for the security of webapps (Please do not discuss company specific vulnerabilities here). Requirements: # Secure authentication for AM (for app/job level ACLs). # Webapp security should be optional via site configuration. # Support existing pluggable single sign on mechanisms. # Should not require per app/user configuration for deployment. # Should not require special site-wide DNS configuration for deployment. This the top jira for webapp security. A design doc/notes of threat-modeling and counter measures will be posted on the wiki. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (MAPREDUCE-3176) ant mapreduce tests are timing out
[ https://issues.apache.org/jira/browse/MAPREDUCE-3176?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13128989#comment-13128989 ] Hitesh Shah commented on MAPREDUCE-3176: bq. But before that. Hitesh, there are TODOs that you added in TestFileSytem, you intending to fix them? Thanks Vinod for verifying. Yes, I planned to look at the test failure and fix it hence had marked the patch as a work in progress. There was a fix done to the command parser earlier which seems to have broken this particular test. ant mapreduce tests are timing out -- Key: MAPREDUCE-3176 URL: https://issues.apache.org/jira/browse/MAPREDUCE-3176 Project: Hadoop Map/Reduce Issue Type: Bug Components: mrv2, test Affects Versions: 0.23.0 Reporter: Ravi Prakash Assignee: Hitesh Shah Priority: Blocker Labels: test Fix For: 0.23.0 Attachments: MR-3176.wip.patch Secondary YARN builds started taking inordinately long and lots of tests started failing. Usually the secondary build would take ~ 2 hours. But recently even after 7 hours it wasn't done. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (MAPREDUCE-3068) Should set MALLOC_ARENA_MAX for all YARN daemons and AMs/Containers
[ https://issues.apache.org/jira/browse/MAPREDUCE-3068?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13128988#comment-13128988 ] Hadoop QA commented on MAPREDUCE-3068: -- +1 overall. Here are the results of testing the latest attachment http://issues.apache.org/jira/secure/attachment/12499394/MAPREDUCE-3068.diff against trunk revision . +1 @author. The patch does not contain any @author tags. +1 tests included. The patch appears to include 3 new or modified tests. +1 javadoc. The javadoc tool did not generate any warning messages. +1 javac. The applied patch does not increase the total number of javac compiler warnings. +1 findbugs. The patch does not introduce any new Findbugs (version 1.3.9) warnings. +1 release audit. The applied patch does not increase the total number of release audit warnings. +1 core tests. The patch passed unit tests in . +1 contrib tests. The patch passed contrib unit tests. Test results: https://builds.apache.org/job/PreCommit-MAPREDUCE-Build/1041//testReport/ Console output: https://builds.apache.org/job/PreCommit-MAPREDUCE-Build/1041//console This message is automatically generated. Should set MALLOC_ARENA_MAX for all YARN daemons and AMs/Containers --- Key: MAPREDUCE-3068 URL: https://issues.apache.org/jira/browse/MAPREDUCE-3068 Project: Hadoop Map/Reduce Issue Type: Bug Components: mrv2 Affects Versions: 0.23.0 Reporter: Vinod Kumar Vavilapalli Assignee: Chris Riccomini Priority: Blocker Fix For: 0.23.0 Attachments: MAPREDUCE-3068.diff, MAPREDUCE-3068.diff, MAPREDUCE-3068.diff, MAPREDUCE-3068.diff This is same as HADOOP-7154 but for yarn. RM, NM, AM and containers should all have this. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Updated] (MAPREDUCE-2693) NPE in AM causes it to lose containers which are never returned back to RM
[ https://issues.apache.org/jira/browse/MAPREDUCE-2693?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Hitesh Shah updated MAPREDUCE-2693: --- Affects Version/s: 0.23.0 NPE in AM causes it to lose containers which are never returned back to RM -- Key: MAPREDUCE-2693 URL: https://issues.apache.org/jira/browse/MAPREDUCE-2693 Project: Hadoop Map/Reduce Issue Type: Bug Components: mrv2 Affects Versions: 0.23.0 Reporter: Amol Kekre Assignee: Hitesh Shah Priority: Critical Fix For: 0.23.0 Attachments: MR-2693.1.patch The following exception in AM of an application at the top of queue causes this. Once this happens, AM keeps obtaining containers from RM and simply loses them. Eventually on a cluster with multiple jobs, no more scheduling happens because of these lost containers. It happens when there are blacklisted nodes at the app level in AM. A bug in AM (RMContainerRequestor.containerFailedOnHost(hostName)) is causing this - nodes are simply getting removed from the request-table. We should make sure RM also knows about this update. 11/06/17 06:11:18 INFO rm.RMContainerAllocator: Assigned based on host match 98.138.163.34 11/06/17 06:11:18 INFO rm.RMContainerRequestor: BEFORE decResourceRequest: applicationId=30 priority=20 resourceName=... numContainers=4978 #asks=5 11/06/17 06:11:18 INFO rm.RMContainerRequestor: AFTER decResourceRequest: applicationId=30 priority=20 resourceName=... numContainers=4977 #asks=5 11/06/17 06:11:18 INFO rm.RMContainerRequestor: BEFORE decResourceRequest: applicationId=30 priority=20 resourceName=... numContainers=1540 #asks=5 11/06/17 06:11:18 INFO rm.RMContainerRequestor: AFTER decResourceRequest: applicationId=30 priority=20 resourceName=... numContainers=1539 #asks=6 11/06/17 06:11:18 ERROR rm.RMContainerAllocator: ERROR IN CONTACTING RM. java.lang.NullPointerException at org.apache.hadoop.mapreduce.v2.app.rm.RMContainerRequestor.decResourceRequest(RMContainerRequestor.java:246) at org.apache.hadoop.mapreduce.v2.app.rm.RMContainerRequestor.decContainerReq(RMContainerRequestor.java:198) at org.apache.hadoop.mapreduce.v2.app.rm.RMContainerAllocator$ScheduledRequests.assign(RMContainerAllocator.java:523) at org.apache.hadoop.mapreduce.v2.app.rm.RMContainerAllocator$ScheduledRequests.access$200(RMContainerAllocator.java:433) at org.apache.hadoop.mapreduce.v2.app.rm.RMContainerAllocator.heartbeat(RMContainerAllocator.java:151) at org.apache.hadoop.mapreduce.v2.app.rm.RMCommunicator$1.run(RMCommunicator.java:220) at java.lang.Thread.run(Thread.java:619) -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Updated] (MAPREDUCE-3146) Add a MR specific command line to dump logs for a given TaskAttemptID
[ https://issues.apache.org/jira/browse/MAPREDUCE-3146?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Arun C Murthy updated MAPREDUCE-3146: - Priority: Critical (was: Major) Add a MR specific command line to dump logs for a given TaskAttemptID - Key: MAPREDUCE-3146 URL: https://issues.apache.org/jira/browse/MAPREDUCE-3146 Project: Hadoop Map/Reduce Issue Type: Sub-task Components: mrv2, nodemanager Affects Versions: 0.23.0 Reporter: Vinod Kumar Vavilapalli Assignee: Siddharth Seth Priority: Critical Fix For: 0.23.0 -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (MAPREDUCE-3193) NextGen Mapreduce framework is not able to read the job input recursively.Input is read only for one folder level deep
[ https://issues.apache.org/jira/browse/MAPREDUCE-3193?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13129001#comment-13129001 ] Mahadev konar commented on MAPREDUCE-3193: -- Ramgopal, What input format are you using? NextGen Mapreduce framework is not able to read the job input recursively.Input is read only for one folder level deep -- Key: MAPREDUCE-3193 URL: https://issues.apache.org/jira/browse/MAPREDUCE-3193 Project: Hadoop Map/Reduce Issue Type: Bug Components: mrv2 Affects Versions: 0.23.0 Reporter: Ramgopal N java.io.FileNotFoundException is thrown,if input file is more than one folder level deep and the job is getting failed. Example:Input file is /r1/r2/input.txt -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (MAPREDUCE-3003) Publish Yarn and MapReduce artifacts to Maven snapshot repository
[ https://issues.apache.org/jira/browse/MAPREDUCE-3003?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13129000#comment-13129000 ] Tom White commented on MAPREDUCE-3003: -- +1 Publish Yarn and MapReduce artifacts to Maven snapshot repository - Key: MAPREDUCE-3003 URL: https://issues.apache.org/jira/browse/MAPREDUCE-3003 Project: Hadoop Map/Reduce Issue Type: Bug Components: build Affects Versions: 0.23.0, 0.24.0 Reporter: Tom White Assignee: Alejandro Abdelnur Fix For: 0.23.0, 0.24.0 Attachments: MAPREDUCE-3003-0.23.patch, MAPREDUCE-3003.patch, MAPREDUCE-3003.patch, MAPREDUCE-3003.patch, MAPREDUCE-3003.patch, MAPREDUCE-3003.patch, MAPREDUCE-3003.patch, MAPREDUCE-3003.patch Currently this is failing since no distribution management section is defined in the POM. https://builds.apache.org/view/G-L/view/Hadoop/job/Hadoop-Common-trunk-Commit/883/consoleFull -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Resolved] (MAPREDUCE-3145) Fix NM UI to serve logs from DFS once application finishes
[ https://issues.apache.org/jira/browse/MAPREDUCE-3145?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Vinod Kumar Vavilapalli resolved MAPREDUCE-3145. Resolution: Duplicate MAPREDUCE-2989 already is giving access to logs from the JobHistory. Adding to NM only adds to load to NMs when someone accesses very old logs. Closing this as duplicate. Fix NM UI to serve logs from DFS once application finishes -- Key: MAPREDUCE-3145 URL: https://issues.apache.org/jira/browse/MAPREDUCE-3145 Project: Hadoop Map/Reduce Issue Type: Sub-task Components: mrv2, nodemanager Affects Versions: 0.23.0 Reporter: Vinod Kumar Vavilapalli Assignee: Siddharth Seth Fix For: 0.23.0 -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (MAPREDUCE-2762) [MR-279] - Cleanup staging dir after job completion
[
https://issues.apache.org/jira/browse/MAPREDUCE-2762?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13129004#comment-13129004
]
Mahadev konar commented on MAPREDUCE-2762:
--
Yeah had thought abt that. I was trying to keep the MRAppMaster code clean and
did not want to inject any code from Jobs in the MRAppMaster.
As Vinod tells me, the code is already tainted with the uber AM, so I'll go
ahead and make the quick fix for cleaning the staging directories.
We'll have to rethink the MRAppMaster code when we do DAG of jobs in the AM.
[MR-279] - Cleanup staging dir after job completion
---
Key: MAPREDUCE-2762
URL: https://issues.apache.org/jira/browse/MAPREDUCE-2762
Project: Hadoop Map/Reduce
Issue Type: Bug
Components: mrv2
Affects Versions: 0.23.0
Reporter: Ramya Sunil
Assignee: Mahadev konar
Priority: Blocker
Fix For: 0.23.0
Attachments: MAPREDUCE-2762.patch, MAPREDUCE-2762.patch,
MAPREDUCE-2762.patch, MAPREDUCE-2762.patch
The files created under the staging dir have to be deleted after job
completion. Currently, all job.* files remain forever in the
${yarn.apps.stagingDir}
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators:
https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Updated] (MAPREDUCE-3003) Publish MR JARs to Maven snapshot repository
[ https://issues.apache.org/jira/browse/MAPREDUCE-3003?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Alejandro Abdelnur updated MAPREDUCE-3003: -- Summary: Publish MR JARs to Maven snapshot repository (was: Publish Yarn and MapReduce artifacts to Maven snapshot repository) Publish MR JARs to Maven snapshot repository Key: MAPREDUCE-3003 URL: https://issues.apache.org/jira/browse/MAPREDUCE-3003 Project: Hadoop Map/Reduce Issue Type: Bug Components: build Affects Versions: 0.23.0, 0.24.0 Reporter: Tom White Assignee: Alejandro Abdelnur Fix For: 0.23.0, 0.24.0 Attachments: MAPREDUCE-3003-0.23.patch, MAPREDUCE-3003.patch, MAPREDUCE-3003.patch, MAPREDUCE-3003.patch, MAPREDUCE-3003.patch, MAPREDUCE-3003.patch, MAPREDUCE-3003.patch, MAPREDUCE-3003.patch Currently this is failing since no distribution management section is defined in the POM. https://builds.apache.org/view/G-L/view/Hadoop/job/Hadoop-Common-trunk-Commit/883/consoleFull -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Updated] (MAPREDUCE-3003) Publish MR JARs to Maven snapshot repository
[ https://issues.apache.org/jira/browse/MAPREDUCE-3003?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Alejandro Abdelnur updated MAPREDUCE-3003: -- Resolution: Fixed Status: Resolved (was: Patch Available) I've just committed this to trunk. It will go to 0.23 together with other related Maven patches in a couple of days. Publish MR JARs to Maven snapshot repository Key: MAPREDUCE-3003 URL: https://issues.apache.org/jira/browse/MAPREDUCE-3003 Project: Hadoop Map/Reduce Issue Type: Bug Components: build Affects Versions: 0.23.0, 0.24.0 Reporter: Tom White Assignee: Alejandro Abdelnur Fix For: 0.23.0, 0.24.0 Attachments: MAPREDUCE-3003-0.23.patch, MAPREDUCE-3003.patch, MAPREDUCE-3003.patch, MAPREDUCE-3003.patch, MAPREDUCE-3003.patch, MAPREDUCE-3003.patch, MAPREDUCE-3003.patch, MAPREDUCE-3003.patch Currently this is failing since no distribution management section is defined in the POM. https://builds.apache.org/view/G-L/view/Hadoop/job/Hadoop-Common-trunk-Commit/883/consoleFull -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (MAPREDUCE-3003) Publish MR JARs to Maven snapshot repository
[ https://issues.apache.org/jira/browse/MAPREDUCE-3003?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13129010#comment-13129010 ] Hudson commented on MAPREDUCE-3003: --- Integrated in Hadoop-Hdfs-trunk-Commit #1174 (See [https://builds.apache.org/job/Hadoop-Hdfs-trunk-Commit/1174/]) MAPREDUCE-3003. Publish MR JARs to Maven snapshot repository. (tucu) tucu : http://svn.apache.org/viewcvs.cgi/?root=Apache-SVNview=revrev=1185286 Files : * /hadoop/common/trunk/BUILDING.txt * /hadoop/common/trunk/hadoop-mapreduce-project/CHANGES.txt * /hadoop/common/trunk/hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-app/pom.xml * /hadoop/common/trunk/hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-common/pom.xml * /hadoop/common/trunk/hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-core/pom.xml * /hadoop/common/trunk/hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-hs/pom.xml * /hadoop/common/trunk/hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-jobclient/pom.xml * /hadoop/common/trunk/hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-shuffle/pom.xml * /hadoop/common/trunk/hadoop-mapreduce-project/hadoop-mapreduce-client/pom.xml * /hadoop/common/trunk/hadoop-mapreduce-project/hadoop-yarn/hadoop-yarn-api/pom.xml * /hadoop/common/trunk/hadoop-mapreduce-project/hadoop-yarn/hadoop-yarn-applications/hadoop-yarn-applications-distributedshell/pom.xml * /hadoop/common/trunk/hadoop-mapreduce-project/hadoop-yarn/hadoop-yarn-applications/pom.xml * /hadoop/common/trunk/hadoop-mapreduce-project/hadoop-yarn/hadoop-yarn-common/pom.xml * /hadoop/common/trunk/hadoop-mapreduce-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-common/pom.xml * /hadoop/common/trunk/hadoop-mapreduce-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-nodemanager/pom.xml * /hadoop/common/trunk/hadoop-mapreduce-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager/pom.xml * /hadoop/common/trunk/hadoop-mapreduce-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-tests/pom.xml * /hadoop/common/trunk/hadoop-mapreduce-project/hadoop-yarn/hadoop-yarn-server/pom.xml * /hadoop/common/trunk/hadoop-mapreduce-project/hadoop-yarn/hadoop-yarn-site/pom.xml * /hadoop/common/trunk/hadoop-mapreduce-project/hadoop-yarn/pom.xml * /hadoop/common/trunk/hadoop-mapreduce-project/ivy.xml * /hadoop/common/trunk/hadoop-mapreduce-project/pom.xml * /hadoop/common/trunk/hadoop-project-dist/pom.xml * /hadoop/common/trunk/hadoop-project/pom.xml * /hadoop/common/trunk/pom.xml Publish MR JARs to Maven snapshot repository Key: MAPREDUCE-3003 URL: https://issues.apache.org/jira/browse/MAPREDUCE-3003 Project: Hadoop Map/Reduce Issue Type: Bug Components: build Affects Versions: 0.23.0, 0.24.0 Reporter: Tom White Assignee: Alejandro Abdelnur Fix For: 0.23.0, 0.24.0 Attachments: MAPREDUCE-3003-0.23.patch, MAPREDUCE-3003.patch, MAPREDUCE-3003.patch, MAPREDUCE-3003.patch, MAPREDUCE-3003.patch, MAPREDUCE-3003.patch, MAPREDUCE-3003.patch, MAPREDUCE-3003.patch Currently this is failing since no distribution management section is defined in the POM. https://builds.apache.org/view/G-L/view/Hadoop/job/Hadoop-Common-trunk-Commit/883/consoleFull -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (MAPREDUCE-3003) Publish MR JARs to Maven snapshot repository
[ https://issues.apache.org/jira/browse/MAPREDUCE-3003?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13129011#comment-13129011 ] Hudson commented on MAPREDUCE-3003: --- Integrated in Hadoop-Common-trunk-Commit #1095 (See [https://builds.apache.org/job/Hadoop-Common-trunk-Commit/1095/]) MAPREDUCE-3003. Publish MR JARs to Maven snapshot repository. (tucu) tucu : http://svn.apache.org/viewcvs.cgi/?root=Apache-SVNview=revrev=1185286 Files : * /hadoop/common/trunk/BUILDING.txt * /hadoop/common/trunk/hadoop-mapreduce-project/CHANGES.txt * /hadoop/common/trunk/hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-app/pom.xml * /hadoop/common/trunk/hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-common/pom.xml * /hadoop/common/trunk/hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-core/pom.xml * /hadoop/common/trunk/hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-hs/pom.xml * /hadoop/common/trunk/hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-jobclient/pom.xml * /hadoop/common/trunk/hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-shuffle/pom.xml * /hadoop/common/trunk/hadoop-mapreduce-project/hadoop-mapreduce-client/pom.xml * /hadoop/common/trunk/hadoop-mapreduce-project/hadoop-yarn/hadoop-yarn-api/pom.xml * /hadoop/common/trunk/hadoop-mapreduce-project/hadoop-yarn/hadoop-yarn-applications/hadoop-yarn-applications-distributedshell/pom.xml * /hadoop/common/trunk/hadoop-mapreduce-project/hadoop-yarn/hadoop-yarn-applications/pom.xml * /hadoop/common/trunk/hadoop-mapreduce-project/hadoop-yarn/hadoop-yarn-common/pom.xml * /hadoop/common/trunk/hadoop-mapreduce-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-common/pom.xml * /hadoop/common/trunk/hadoop-mapreduce-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-nodemanager/pom.xml * /hadoop/common/trunk/hadoop-mapreduce-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager/pom.xml * /hadoop/common/trunk/hadoop-mapreduce-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-tests/pom.xml * /hadoop/common/trunk/hadoop-mapreduce-project/hadoop-yarn/hadoop-yarn-server/pom.xml * /hadoop/common/trunk/hadoop-mapreduce-project/hadoop-yarn/hadoop-yarn-site/pom.xml * /hadoop/common/trunk/hadoop-mapreduce-project/hadoop-yarn/pom.xml * /hadoop/common/trunk/hadoop-mapreduce-project/ivy.xml * /hadoop/common/trunk/hadoop-mapreduce-project/pom.xml * /hadoop/common/trunk/hadoop-project-dist/pom.xml * /hadoop/common/trunk/hadoop-project/pom.xml * /hadoop/common/trunk/pom.xml Publish MR JARs to Maven snapshot repository Key: MAPREDUCE-3003 URL: https://issues.apache.org/jira/browse/MAPREDUCE-3003 Project: Hadoop Map/Reduce Issue Type: Bug Components: build Affects Versions: 0.23.0, 0.24.0 Reporter: Tom White Assignee: Alejandro Abdelnur Fix For: 0.23.0, 0.24.0 Attachments: MAPREDUCE-3003-0.23.patch, MAPREDUCE-3003.patch, MAPREDUCE-3003.patch, MAPREDUCE-3003.patch, MAPREDUCE-3003.patch, MAPREDUCE-3003.patch, MAPREDUCE-3003.patch, MAPREDUCE-3003.patch Currently this is failing since no distribution management section is defined in the POM. https://builds.apache.org/view/G-L/view/Hadoop/job/Hadoop-Common-trunk-Commit/883/consoleFull -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Assigned] (MAPREDUCE-3058) Sometimes task keeps on running while its Syslog says that it is shutdown
[
https://issues.apache.org/jira/browse/MAPREDUCE-3058?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Arun C Murthy reassigned MAPREDUCE-3058:
Assignee: Hitesh Shah (was: Vinod Kumar Vavilapalli)
Hitesh - can you please take a look since you are already working on related
areas?
Currently the TT sends SIGTERM followed by SIGKILL...
Sometimes task keeps on running while its Syslog says that it is shutdown
-
Key: MAPREDUCE-3058
URL: https://issues.apache.org/jira/browse/MAPREDUCE-3058
Project: Hadoop Map/Reduce
Issue Type: Bug
Components: contrib/gridmix, mrv2
Affects Versions: 0.23.0
Reporter: Karam Singh
Assignee: Hitesh Shah
Priority: Critical
Fix For: 0.23.0
Attachments: MAPREDUCE-3058-20110923.txt
While running GridMixV3, one of the jobs got stuck for 15 hrs. After clicking
on the Job-page, found one of its reduces to be stuck. Looking at syslog of
the stuck reducer, found this:
Task-logs' head:
{code}
2011-09-19 17:57:22,002 INFO
org.apache.hadoop.metrics2.impl.MetricsSystemImpl: Scheduled snapshot period
at 10 second(s).
2011-09-19 17:57:22,002 INFO
org.apache.hadoop.metrics2.impl.MetricsSystemImpl: ReduceTask metrics system
started
{code}
Task-logs' tail:
{code}
2011-09-19 18:06:49,818 INFO org.apache.hadoop.hdfs.DFSClient: Exception in
createBlockOutputStream java.io.IOException: Bad connect ack with
firstBadLink as DATANODE1
2011-09-19 18:06:49,818 WARN org.apache.hadoop.hdfs.DFSClient: Error Recovery
for block
BP-1405370709-NAMENODE-1316452621953:blk_-7004355226367468317_79871 in
pipeline DATANODE2, DATANODE1: bad datanode DATANODE1
2011-09-19 18:06:49,818 DEBUG
org.apache.hadoop.hdfs.protocol.datatransfer.DataTransferProtocol:
lastAckedSeqno = 26870
2011-09-19 18:06:49,820 DEBUG org.apache.hadoop.ipc.Client: IPC Client
(26613121) connection to NAMENODE from gridperf sending #454
2011-09-19 18:06:49,826 DEBUG org.apache.hadoop.ipc.Client: IPC Client
(26613121) connection to NAMENODE from gridperf got value #454
2011-09-19 18:06:49,827 DEBUG org.apache.hadoop.ipc.RPC: Call:
getAdditionalDatanode 8
2011-09-19 18:06:49,827 DEBUG org.apache.hadoop.hdfs.DFSClient: Connecting to
datanode DATANODE2
2011-09-19 18:06:49,827 DEBUG org.apache.hadoop.hdfs.DFSClient: Send buf size
131071
2011-09-19 18:06:49,833 WARN org.apache.hadoop.hdfs.DFSClient: DataStreamer
Exception
java.io.EOFException: Premature EOF: no length prefix available
at
org.apache.hadoop.hdfs.protocol.HdfsProtoUtil.vintPrefixed(HdfsProtoUtil.java:158)
at
org.apache.hadoop.hdfs.DFSOutputStream$DataStreamer.transfer(DFSOutputStream.java:860)
at
org.apache.hadoop.hdfs.DFSOutputStream$DataStreamer.addDatanode2ExistingPipeline(DFSOutputStream.java:838)
at
org.apache.hadoop.hdfs.DFSOutputStream$DataStreamer.setupPipelineForAppendOrRecovery(DFSOutputStream.java:929)
at
org.apache.hadoop.hdfs.DFSOutputStream$DataStreamer.processDatanodeError(DFSOutputStream.java:740)
at
org.apache.hadoop.hdfs.DFSOutputStream$DataStreamer.run(DFSOutputStream.java:415)
2011-09-19 18:06:49,837 WARN org.apache.hadoop.mapred.YarnChild: Exception
running child : java.io.EOFException: Premature EOF: no length prefix
available
at
org.apache.hadoop.hdfs.protocol.HdfsProtoUtil.vintPrefixed(HdfsProtoUtil.java:158)
at
org.apache.hadoop.hdfs.DFSOutputStream$DataStreamer.transfer(DFSOutputStream.java:860)
at
org.apache.hadoop.hdfs.DFSOutputStream$DataStreamer.addDatanode2ExistingPipeline(DFSOutputStream.java:838)
at
org.apache.hadoop.hdfs.DFSOutputStream$DataStreamer.setupPipelineForAppendOrRecovery(DFSOutputStream.java:929)
at
org.apache.hadoop.hdfs.DFSOutputStream$DataStreamer.processDatanodeError(DFSOutputStream.java:740)
at
org.apache.hadoop.hdfs.DFSOutputStream$DataStreamer.run(DFSOutputStream.java:415)
2011-09-19 18:06:49,837 DEBUG org.apache.hadoop.ipc.Client: IPC Client
(26613121) connection to APPMASTER from job_1316452677984_0862 sending #455
2011-09-19 18:06:49,839 DEBUG org.apache.hadoop.ipc.Client: IPC Client
(26613121) connection to APPMASTER from job_1316452677984_0862 got value
#455
2011-09-19 18:06:49,840 DEBUG org.apache.hadoop.ipc.RPC: Call: statusUpdate 3
2011-09-19 18:06:49,840 INFO org.apache.hadoop.mapred.Task: Runnning cleanup
for the task
2011-09-19 18:06:49,840 DEBUG org.apache.hadoop.ipc.Client: IPC Client
(26613121) connection to NAMENODE from gridperf sending #456
2011-09-19 18:06:49,858 DEBUG org.apache.hadoop.ipc.Client: IPC Client
(26613121) connection to NAMENODE from gridperf got value #456
2011-09-19 18:06:49,858 DEBUG
[jira] [Commented] (MAPREDUCE-3195) MapReduce PreCommit Trunk build is silently failing when running tests
[ https://issues.apache.org/jira/browse/MAPREDUCE-3195?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13129048#comment-13129048 ] Chris Riccomini commented on MAPREDUCE-3195: Yea, I just noticed this as well. It seems to have been going on for quite some time. I should also note that the -1 I got in https://issues.apache.org/jira/browse/MAPREDUCE-3068 for javadocs seems to be transient, and appears in a lot of tests. MapReduce PreCommit Trunk build is silently failing when running tests -- Key: MAPREDUCE-3195 URL: https://issues.apache.org/jira/browse/MAPREDUCE-3195 Project: Hadoop Map/Reduce Issue Type: Bug Components: build Affects Versions: 0.23.0, 0.24.0 Reporter: Jonathan Eagles Priority: Blocker MapReduce PreCommit build is silently failing only running a very small portion of tests. The build then errors out, yet +1 it given to the patch. Last known Success build - 307 tests run and passed https://builds.apache.org/view/G-L/view/Hadoop/job/PreCommit-MAPREDUCE-Build/990/testReport/ First known Error build - 69 tests run and passed https://builds.apache.org/view/G-L/view/Hadoop/job/PreCommit-MAPREDUCE-Build/994/testReport/ Snippet from failed build log - Errors out and then +1 the patch https://builds.apache.org/view/G-L/view/Hadoop/job/PreCommit-MAPREDUCE-Build/994/console [INFO] [INFO] Reactor Summary: [INFO] [INFO] hadoop-yarn-api ... SUCCESS [19.512s] [INFO] hadoop-yarn-common FAILURE [13.835s] [INFO] hadoop-yarn-server-common . SKIPPED [INFO] hadoop-yarn-server-nodemanager SKIPPED [INFO] hadoop-yarn-server-resourcemanager SKIPPED [INFO] hadoop-yarn-server-tests .. SKIPPED [INFO] hadoop-yarn-server SKIPPED [INFO] hadoop-yarn-applications-distributedshell . SKIPPED [INFO] hadoop-yarn-applications .. SKIPPED [INFO] hadoop-yarn-site .. SKIPPED [INFO] hadoop-yarn ... SKIPPED [INFO] hadoop-mapreduce-client-core .. SKIPPED [INFO] hadoop-mapreduce-client-common SKIPPED [INFO] hadoop-mapreduce-client-shuffle ... SKIPPED [INFO] hadoop-mapreduce-client-app ... SKIPPED [INFO] hadoop-mapreduce-client-hs SKIPPED [INFO] hadoop-mapreduce-client-jobclient . SKIPPED [INFO] hadoop-mapreduce-client ... SKIPPED [INFO] hadoop-mapreduce .. SKIPPED [INFO] [INFO] BUILD FAILURE [INFO] [INFO] Total time: 33.784s [INFO] Finished at: Wed Oct 12 12:03:22 UTC 2011 [INFO] Final Memory: 40M/630M [INFO] [ERROR] Failed to execute goal org.apache.maven.plugins:maven-assembly-plugin:2.2-beta-5:single (tar) on project hadoop-yarn-common: Failed to create assembly: Error adding file 'org.apache.hadoop:hadoop-yarn-api:jar:0.24.0-SNAPSHOT' to archive: /home/jenkins/jenkins-slave/workspace/PreCommit-MAPREDUCE-Build/trunk/hadoop-mapreduce-project/hadoop-yarn/hadoop-yarn-api/target/classes isn't a file. - [Help 1] [ERROR] [ERROR] To see the full stack trace of the errors, re-run Maven with the -e switch. [ERROR] Re-run Maven using the -X switch to enable full debug logging. [ERROR] [ERROR] For more information about the errors and possible solutions, please read the following articles: [ERROR] [Help 1] http://cwiki.apache.org/confluence/display/MAVEN/MojoExecutionException [ERROR] [ERROR] After correcting the problems, you can resume the build with the command [ERROR] mvn goals -rf :hadoop-yarn-common == == Running contrib tests. == == /bin/kill -9 27543 kill: No such process NOP == == Checking the integrity of system test framework code. ==
[jira] [Commented] (MAPREDUCE-3068) Should set MALLOC_ARENA_MAX for all YARN daemons and AMs/Containers
[ https://issues.apache.org/jira/browse/MAPREDUCE-3068?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13129058#comment-13129058 ] Chris Riccomini commented on MAPREDUCE-3068: Hey Arun, Do you want just the putIfAbsent vars specified in the latest patch, or do you want to modify setEnvFromInputString to putIfAbsent for ALL env variables specified in admin-env? I have some concern about doing the latter, as it changes the existing behavior of the setEnv.. method, which is used in MR, not just YARN. Thanks! Chris Should set MALLOC_ARENA_MAX for all YARN daemons and AMs/Containers --- Key: MAPREDUCE-3068 URL: https://issues.apache.org/jira/browse/MAPREDUCE-3068 Project: Hadoop Map/Reduce Issue Type: Bug Components: mrv2 Affects Versions: 0.23.0 Reporter: Vinod Kumar Vavilapalli Assignee: Chris Riccomini Priority: Blocker Fix For: 0.23.0 Attachments: MAPREDUCE-3068.diff, MAPREDUCE-3068.diff, MAPREDUCE-3068.diff, MAPREDUCE-3068.diff This is same as HADOOP-7154 but for yarn. RM, NM, AM and containers should all have this. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Updated] (MAPREDUCE-3068) Should set MALLOC_ARENA_MAX for all YARN daemons and AMs/Containers
[ https://issues.apache.org/jira/browse/MAPREDUCE-3068?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Chris Riccomini updated MAPREDUCE-3068: --- Status: Open (was: Patch Available) Waiting on Arun to figure out behavior of addToEnvironment.. whether we should change to addToEnvironmentIfBasent Should set MALLOC_ARENA_MAX for all YARN daemons and AMs/Containers --- Key: MAPREDUCE-3068 URL: https://issues.apache.org/jira/browse/MAPREDUCE-3068 Project: Hadoop Map/Reduce Issue Type: Bug Components: mrv2 Affects Versions: 0.23.0 Reporter: Vinod Kumar Vavilapalli Assignee: Chris Riccomini Priority: Blocker Fix For: 0.23.0 Attachments: MAPREDUCE-3068.diff, MAPREDUCE-3068.diff, MAPREDUCE-3068.diff, MAPREDUCE-3068.diff This is same as HADOOP-7154 but for yarn. RM, NM, AM and containers should all have this. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (MAPREDUCE-1380) Adaptive Scheduler
[ https://issues.apache.org/jira/browse/MAPREDUCE-1380?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13129066#comment-13129066 ] shivam tiwari commented on MAPREDUCE-1380: -- Where I can download this adaptive scheduler. I am not able to find, please help me Adaptive Scheduler -- Key: MAPREDUCE-1380 URL: https://issues.apache.org/jira/browse/MAPREDUCE-1380 Project: Hadoop Map/Reduce Issue Type: New Feature Reporter: Jordà Polo Priority: Minor Attachments: MAPREDUCE-1380_0.1.patch, MAPREDUCE-1380_1.1.patch, MAPREDUCE-1380_1.1.pdf The Adaptive Scheduler is a pluggable Hadoop scheduler that automatically adjusts the amount of used resources depending on the performance of jobs and on user-defined high-level business goals. Existing Hadoop schedulers are focused on managing large, static clusters in which nodes are added or removed manually. On the other hand, the goal of this scheduler is to improve the integration of Hadoop and the applications that run on top of it with environments that allow a more dynamic provisioning of resources. The current implementation is quite straightforward. Users specify a deadline at job submission time, and the scheduler adjusts the resources to meet that deadline (at the moment, the scheduler can be configured to either minimize or maximize the amount of resources). If multiple jobs are run simultaneously, the scheduler prioritizes them by deadline. Note that the current approach to estimate the completion time of jobs is quite simplistic: it is based on the time it takes to finish each task, so it works well with regular jobs, but there is still room for improvement for unpredictable jobs. The idea is to further integrate it with cloud-like and virtual environments (such as Amazon EC2, Emotive, etc.) so that if, for instance, a job isn't able to meet its deadline, the scheduler automatically requests more resources. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Updated] (MAPREDUCE-3068) Should set MALLOC_ARENA_MAX for all YARN daemons and AMs/Containers
[ https://issues.apache.org/jira/browse/MAPREDUCE-3068?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Chris Riccomini updated MAPREDUCE-3068: --- Attachment: MAPREDUCE-3068.diff Latest attachment now uses putIfAbsent logic for container environment variables that are supplied via admin-env property. Should set MALLOC_ARENA_MAX for all YARN daemons and AMs/Containers --- Key: MAPREDUCE-3068 URL: https://issues.apache.org/jira/browse/MAPREDUCE-3068 Project: Hadoop Map/Reduce Issue Type: Bug Components: mrv2 Affects Versions: 0.23.0 Reporter: Vinod Kumar Vavilapalli Assignee: Chris Riccomini Priority: Blocker Fix For: 0.23.0 Attachments: MAPREDUCE-3068.diff, MAPREDUCE-3068.diff, MAPREDUCE-3068.diff, MAPREDUCE-3068.diff, MAPREDUCE-3068.diff This is same as HADOOP-7154 but for yarn. RM, NM, AM and containers should all have this. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Updated] (MAPREDUCE-3068) Should set MALLOC_ARENA_MAX for all YARN daemons and AMs/Containers
[ https://issues.apache.org/jira/browse/MAPREDUCE-3068?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Chris Riccomini updated MAPREDUCE-3068: --- Status: Patch Available (was: Open) Submitting patch. Should set MALLOC_ARENA_MAX for all YARN daemons and AMs/Containers --- Key: MAPREDUCE-3068 URL: https://issues.apache.org/jira/browse/MAPREDUCE-3068 Project: Hadoop Map/Reduce Issue Type: Bug Components: mrv2 Affects Versions: 0.23.0 Reporter: Vinod Kumar Vavilapalli Assignee: Chris Riccomini Priority: Blocker Fix For: 0.23.0 Attachments: MAPREDUCE-3068.diff, MAPREDUCE-3068.diff, MAPREDUCE-3068.diff, MAPREDUCE-3068.diff, MAPREDUCE-3068.diff This is same as HADOOP-7154 but for yarn. RM, NM, AM and containers should all have this. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (MAPREDUCE-3195) MapReduce PreCommit Trunk build is silently failing when running tests
[ https://issues.apache.org/jira/browse/MAPREDUCE-3195?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13129127#comment-13129127 ] Chris Riccomini commented on MAPREDUCE-3195: Based on the hudson chart, it appears to have started between 993 and 994: https://builds.apache.org/job/PreCommit-MAPREDUCE-Build/993/console https://builds.apache.org/job/PreCommit-MAPREDUCE-Build/994/console 994 contains: MAPREDUCE-3157. [Rumen] Fix TraceBuilder to handle 0.20 history file names also. (detail/ViewSVN) MAPREDUCE-3125. Modified TaskImpl to consider only non-failed, non-killed task-attempts for obtaining task's progress. Contributed by Hitesh Shah. (detail/ViewSVN) Move line for HDFS-2422 under 0.23 instead of 0.24. (detail/ViewSVN) Move the HADOOP-6490 Changes note to the right section. Accidentally placed it under Bugs in r1182189. (detail/ViewSVN) MAPREDUCE-3140. Fixed the invalid JobHistory URL for failed applications. Contributed by Subroto Sanyal. (detail/ViewSVN) HADOOP-7736. Remove duplicate Path#normalizePath call. (detail/ViewSVN) HDFS-2349. Corruption detected during block transfers should log WARNs. (detail/ViewSVN) HADOOP-6490. Use StringUtils over String#replace in Path#normalizePath. Contributed by Uma Maheswara Rao G. (detail/ViewSVN) HADOOP-7642. create hadoop-dist module where TAR stitching would happen. Contributed by Thomas White. (detail/ViewSVN) Not sure if any of this is relevant/helps. The Hudson test chart has been running flat at 69 tests ever since. MapReduce PreCommit Trunk build is silently failing when running tests -- Key: MAPREDUCE-3195 URL: https://issues.apache.org/jira/browse/MAPREDUCE-3195 Project: Hadoop Map/Reduce Issue Type: Bug Components: build Affects Versions: 0.23.0, 0.24.0 Reporter: Jonathan Eagles Priority: Blocker MapReduce PreCommit build is silently failing only running a very small portion of tests. The build then errors out, yet +1 it given to the patch. Last known Success build - 307 tests run and passed https://builds.apache.org/view/G-L/view/Hadoop/job/PreCommit-MAPREDUCE-Build/990/testReport/ First known Error build - 69 tests run and passed https://builds.apache.org/view/G-L/view/Hadoop/job/PreCommit-MAPREDUCE-Build/994/testReport/ Snippet from failed build log - Errors out and then +1 the patch https://builds.apache.org/view/G-L/view/Hadoop/job/PreCommit-MAPREDUCE-Build/994/console [INFO] [INFO] Reactor Summary: [INFO] [INFO] hadoop-yarn-api ... SUCCESS [19.512s] [INFO] hadoop-yarn-common FAILURE [13.835s] [INFO] hadoop-yarn-server-common . SKIPPED [INFO] hadoop-yarn-server-nodemanager SKIPPED [INFO] hadoop-yarn-server-resourcemanager SKIPPED [INFO] hadoop-yarn-server-tests .. SKIPPED [INFO] hadoop-yarn-server SKIPPED [INFO] hadoop-yarn-applications-distributedshell . SKIPPED [INFO] hadoop-yarn-applications .. SKIPPED [INFO] hadoop-yarn-site .. SKIPPED [INFO] hadoop-yarn ... SKIPPED [INFO] hadoop-mapreduce-client-core .. SKIPPED [INFO] hadoop-mapreduce-client-common SKIPPED [INFO] hadoop-mapreduce-client-shuffle ... SKIPPED [INFO] hadoop-mapreduce-client-app ... SKIPPED [INFO] hadoop-mapreduce-client-hs SKIPPED [INFO] hadoop-mapreduce-client-jobclient . SKIPPED [INFO] hadoop-mapreduce-client ... SKIPPED [INFO] hadoop-mapreduce .. SKIPPED [INFO] [INFO] BUILD FAILURE [INFO] [INFO] Total time: 33.784s [INFO] Finished at: Wed Oct 12 12:03:22 UTC 2011 [INFO] Final Memory: 40M/630M [INFO] [ERROR] Failed to execute goal org.apache.maven.plugins:maven-assembly-plugin:2.2-beta-5:single (tar) on project hadoop-yarn-common: Failed to create assembly: Error adding file 'org.apache.hadoop:hadoop-yarn-api:jar:0.24.0-SNAPSHOT' to archive: /home/jenkins/jenkins-slave/workspace/PreCommit-MAPREDUCE-Build/trunk/hadoop-mapreduce-project/hadoop-yarn/hadoop-yarn-api/target/classes isn't a file. - [Help 1] [ERROR] [ERROR] To see the full stack trace of the errors, re-run Maven with the -e switch.
[jira] [Commented] (MAPREDUCE-3195) MapReduce PreCommit Trunk build is silently failing when running tests
[ https://issues.apache.org/jira/browse/MAPREDUCE-3195?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13129132#comment-13129132 ] Jonathan Eagles commented on MAPREDUCE-3195: Guessing ... but I see HADOOP-7642 made some changes to the version of the assemblies plugin as well some other assemblies plugin changes. I will investigate this a little, perhaps by submitting the reverse patch for experimentation. MapReduce PreCommit Trunk build is silently failing when running tests -- Key: MAPREDUCE-3195 URL: https://issues.apache.org/jira/browse/MAPREDUCE-3195 Project: Hadoop Map/Reduce Issue Type: Bug Components: build Affects Versions: 0.23.0, 0.24.0 Reporter: Jonathan Eagles Priority: Blocker MapReduce PreCommit build is silently failing only running a very small portion of tests. The build then errors out, yet +1 it given to the patch. Last known Success build - 307 tests run and passed https://builds.apache.org/view/G-L/view/Hadoop/job/PreCommit-MAPREDUCE-Build/990/testReport/ First known Error build - 69 tests run and passed https://builds.apache.org/view/G-L/view/Hadoop/job/PreCommit-MAPREDUCE-Build/994/testReport/ Snippet from failed build log - Errors out and then +1 the patch https://builds.apache.org/view/G-L/view/Hadoop/job/PreCommit-MAPREDUCE-Build/994/console [INFO] [INFO] Reactor Summary: [INFO] [INFO] hadoop-yarn-api ... SUCCESS [19.512s] [INFO] hadoop-yarn-common FAILURE [13.835s] [INFO] hadoop-yarn-server-common . SKIPPED [INFO] hadoop-yarn-server-nodemanager SKIPPED [INFO] hadoop-yarn-server-resourcemanager SKIPPED [INFO] hadoop-yarn-server-tests .. SKIPPED [INFO] hadoop-yarn-server SKIPPED [INFO] hadoop-yarn-applications-distributedshell . SKIPPED [INFO] hadoop-yarn-applications .. SKIPPED [INFO] hadoop-yarn-site .. SKIPPED [INFO] hadoop-yarn ... SKIPPED [INFO] hadoop-mapreduce-client-core .. SKIPPED [INFO] hadoop-mapreduce-client-common SKIPPED [INFO] hadoop-mapreduce-client-shuffle ... SKIPPED [INFO] hadoop-mapreduce-client-app ... SKIPPED [INFO] hadoop-mapreduce-client-hs SKIPPED [INFO] hadoop-mapreduce-client-jobclient . SKIPPED [INFO] hadoop-mapreduce-client ... SKIPPED [INFO] hadoop-mapreduce .. SKIPPED [INFO] [INFO] BUILD FAILURE [INFO] [INFO] Total time: 33.784s [INFO] Finished at: Wed Oct 12 12:03:22 UTC 2011 [INFO] Final Memory: 40M/630M [INFO] [ERROR] Failed to execute goal org.apache.maven.plugins:maven-assembly-plugin:2.2-beta-5:single (tar) on project hadoop-yarn-common: Failed to create assembly: Error adding file 'org.apache.hadoop:hadoop-yarn-api:jar:0.24.0-SNAPSHOT' to archive: /home/jenkins/jenkins-slave/workspace/PreCommit-MAPREDUCE-Build/trunk/hadoop-mapreduce-project/hadoop-yarn/hadoop-yarn-api/target/classes isn't a file. - [Help 1] [ERROR] [ERROR] To see the full stack trace of the errors, re-run Maven with the -e switch. [ERROR] Re-run Maven using the -X switch to enable full debug logging. [ERROR] [ERROR] For more information about the errors and possible solutions, please read the following articles: [ERROR] [Help 1] http://cwiki.apache.org/confluence/display/MAVEN/MojoExecutionException [ERROR] [ERROR] After correcting the problems, you can resume the build with the command [ERROR] mvn goals -rf :hadoop-yarn-common == == Running contrib tests. == == /bin/kill -9 27543 kill: No such process NOP == == Checking the integrity of system test framework code. ==
[jira] [Commented] (MAPREDUCE-3068) Should set MALLOC_ARENA_MAX for all YARN daemons and AMs/Containers
[ https://issues.apache.org/jira/browse/MAPREDUCE-3068?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13129134#comment-13129134 ] Hadoop QA commented on MAPREDUCE-3068: -- -1 overall. Here are the results of testing the latest attachment http://issues.apache.org/jira/secure/attachment/12499422/MAPREDUCE-3068.diff against trunk revision . +1 @author. The patch does not contain any @author tags. +1 tests included. The patch appears to include 3 new or modified tests. +1 javadoc. The javadoc tool did not generate any warning messages. -1 javac. The patch appears to cause tar ant target to fail. -1 findbugs. The patch appears to cause Findbugs (version 1.3.9) to fail. +1 release audit. The applied patch does not increase the total number of release audit warnings. -1 core tests. The patch failed these unit tests: org.apache.hadoop.mapreduce.TestJobMonitorAndPrint +1 contrib tests. The patch passed contrib unit tests. Test results: https://builds.apache.org/job/PreCommit-MAPREDUCE-Build/1042//testReport/ Console output: https://builds.apache.org/job/PreCommit-MAPREDUCE-Build/1042//console This message is automatically generated. Should set MALLOC_ARENA_MAX for all YARN daemons and AMs/Containers --- Key: MAPREDUCE-3068 URL: https://issues.apache.org/jira/browse/MAPREDUCE-3068 Project: Hadoop Map/Reduce Issue Type: Bug Components: mrv2 Affects Versions: 0.23.0 Reporter: Vinod Kumar Vavilapalli Assignee: Chris Riccomini Priority: Blocker Fix For: 0.23.0 Attachments: MAPREDUCE-3068.diff, MAPREDUCE-3068.diff, MAPREDUCE-3068.diff, MAPREDUCE-3068.diff, MAPREDUCE-3068.diff This is same as HADOOP-7154 but for yarn. RM, NM, AM and containers should all have this. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (MAPREDUCE-3195) MapReduce PreCommit Trunk build is silently failing when running tests
[ https://issues.apache.org/jira/browse/MAPREDUCE-3195?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13129136#comment-13129136 ] Chris Riccomini commented on MAPREDUCE-3195: Yea, that's the one that made me nervous as well. Look forward to your findings. :) MapReduce PreCommit Trunk build is silently failing when running tests -- Key: MAPREDUCE-3195 URL: https://issues.apache.org/jira/browse/MAPREDUCE-3195 Project: Hadoop Map/Reduce Issue Type: Bug Components: build Affects Versions: 0.23.0, 0.24.0 Reporter: Jonathan Eagles Priority: Blocker MapReduce PreCommit build is silently failing only running a very small portion of tests. The build then errors out, yet +1 it given to the patch. Last known Success build - 307 tests run and passed https://builds.apache.org/view/G-L/view/Hadoop/job/PreCommit-MAPREDUCE-Build/990/testReport/ First known Error build - 69 tests run and passed https://builds.apache.org/view/G-L/view/Hadoop/job/PreCommit-MAPREDUCE-Build/994/testReport/ Snippet from failed build log - Errors out and then +1 the patch https://builds.apache.org/view/G-L/view/Hadoop/job/PreCommit-MAPREDUCE-Build/994/console [INFO] [INFO] Reactor Summary: [INFO] [INFO] hadoop-yarn-api ... SUCCESS [19.512s] [INFO] hadoop-yarn-common FAILURE [13.835s] [INFO] hadoop-yarn-server-common . SKIPPED [INFO] hadoop-yarn-server-nodemanager SKIPPED [INFO] hadoop-yarn-server-resourcemanager SKIPPED [INFO] hadoop-yarn-server-tests .. SKIPPED [INFO] hadoop-yarn-server SKIPPED [INFO] hadoop-yarn-applications-distributedshell . SKIPPED [INFO] hadoop-yarn-applications .. SKIPPED [INFO] hadoop-yarn-site .. SKIPPED [INFO] hadoop-yarn ... SKIPPED [INFO] hadoop-mapreduce-client-core .. SKIPPED [INFO] hadoop-mapreduce-client-common SKIPPED [INFO] hadoop-mapreduce-client-shuffle ... SKIPPED [INFO] hadoop-mapreduce-client-app ... SKIPPED [INFO] hadoop-mapreduce-client-hs SKIPPED [INFO] hadoop-mapreduce-client-jobclient . SKIPPED [INFO] hadoop-mapreduce-client ... SKIPPED [INFO] hadoop-mapreduce .. SKIPPED [INFO] [INFO] BUILD FAILURE [INFO] [INFO] Total time: 33.784s [INFO] Finished at: Wed Oct 12 12:03:22 UTC 2011 [INFO] Final Memory: 40M/630M [INFO] [ERROR] Failed to execute goal org.apache.maven.plugins:maven-assembly-plugin:2.2-beta-5:single (tar) on project hadoop-yarn-common: Failed to create assembly: Error adding file 'org.apache.hadoop:hadoop-yarn-api:jar:0.24.0-SNAPSHOT' to archive: /home/jenkins/jenkins-slave/workspace/PreCommit-MAPREDUCE-Build/trunk/hadoop-mapreduce-project/hadoop-yarn/hadoop-yarn-api/target/classes isn't a file. - [Help 1] [ERROR] [ERROR] To see the full stack trace of the errors, re-run Maven with the -e switch. [ERROR] Re-run Maven using the -X switch to enable full debug logging. [ERROR] [ERROR] For more information about the errors and possible solutions, please read the following articles: [ERROR] [Help 1] http://cwiki.apache.org/confluence/display/MAVEN/MojoExecutionException [ERROR] [ERROR] After correcting the problems, you can resume the build with the command [ERROR] mvn goals -rf :hadoop-yarn-common == == Running contrib tests. == == /bin/kill -9 27543 kill: No such process NOP == == Checking the integrity of system test framework code. == == /bin/kill -9 27548 kill: No such process NOP +1 overall. Here are the results of testing the latest
[jira] [Updated] (MAPREDUCE-3068) Should set MALLOC_ARENA_MAX for all YARN daemons and AMs/Containers
[ https://issues.apache.org/jira/browse/MAPREDUCE-3068?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Chris Riccomini updated MAPREDUCE-3068: --- Status: Open (was: Patch Available) Should set MALLOC_ARENA_MAX for all YARN daemons and AMs/Containers --- Key: MAPREDUCE-3068 URL: https://issues.apache.org/jira/browse/MAPREDUCE-3068 Project: Hadoop Map/Reduce Issue Type: Bug Components: mrv2 Affects Versions: 0.23.0 Reporter: Vinod Kumar Vavilapalli Assignee: Chris Riccomini Priority: Blocker Fix For: 0.23.0 Attachments: MAPREDUCE-3068.diff, MAPREDUCE-3068.diff, MAPREDUCE-3068.diff, MAPREDUCE-3068.diff, MAPREDUCE-3068.diff This is same as HADOOP-7154 but for yarn. RM, NM, AM and containers should all have this. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Updated] (MAPREDUCE-3068) Should set MALLOC_ARENA_MAX for all YARN daemons and AMs/Containers
[ https://issues.apache.org/jira/browse/MAPREDUCE-3068?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Chris Riccomini updated MAPREDUCE-3068: --- Attachment: MAPREDUCE-3068.diff Turns out MR was using addToEnvironment, so enabled old API plus new API with overwrite param. Should set MALLOC_ARENA_MAX for all YARN daemons and AMs/Containers --- Key: MAPREDUCE-3068 URL: https://issues.apache.org/jira/browse/MAPREDUCE-3068 Project: Hadoop Map/Reduce Issue Type: Bug Components: mrv2 Affects Versions: 0.23.0 Reporter: Vinod Kumar Vavilapalli Assignee: Chris Riccomini Priority: Blocker Fix For: 0.23.0 Attachments: MAPREDUCE-3068.diff, MAPREDUCE-3068.diff, MAPREDUCE-3068.diff, MAPREDUCE-3068.diff, MAPREDUCE-3068.diff, MAPREDUCE-3068.diff This is same as HADOOP-7154 but for yarn. RM, NM, AM and containers should all have this. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Updated] (MAPREDUCE-3068) Should set MALLOC_ARENA_MAX for all YARN daemons and AMs/Containers
[ https://issues.apache.org/jira/browse/MAPREDUCE-3068?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Chris Riccomini updated MAPREDUCE-3068: --- Status: Patch Available (was: Open) Should set MALLOC_ARENA_MAX for all YARN daemons and AMs/Containers --- Key: MAPREDUCE-3068 URL: https://issues.apache.org/jira/browse/MAPREDUCE-3068 Project: Hadoop Map/Reduce Issue Type: Bug Components: mrv2 Affects Versions: 0.23.0 Reporter: Vinod Kumar Vavilapalli Assignee: Chris Riccomini Priority: Blocker Fix For: 0.23.0 Attachments: MAPREDUCE-3068.diff, MAPREDUCE-3068.diff, MAPREDUCE-3068.diff, MAPREDUCE-3068.diff, MAPREDUCE-3068.diff, MAPREDUCE-3068.diff This is same as HADOOP-7154 but for yarn. RM, NM, AM and containers should all have this. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (MAPREDUCE-3068) Should set MALLOC_ARENA_MAX for all YARN daemons and AMs/Containers
[ https://issues.apache.org/jira/browse/MAPREDUCE-3068?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13129139#comment-13129139 ] Arun C Murthy commented on MAPREDUCE-3068: -- Chris - we should allow applications to override JAVA_HOME, HADOOP_COMMON_HOME, HADOOP_HDFS_HOME and YARN_HOME. Thus, we want the putIfAbsent semantics when we are setting them from either the NodeManager's environment or from the admin.env. Does that make sense? Thanks. Should set MALLOC_ARENA_MAX for all YARN daemons and AMs/Containers --- Key: MAPREDUCE-3068 URL: https://issues.apache.org/jira/browse/MAPREDUCE-3068 Project: Hadoop Map/Reduce Issue Type: Bug Components: mrv2 Affects Versions: 0.23.0 Reporter: Vinod Kumar Vavilapalli Assignee: Chris Riccomini Priority: Blocker Fix For: 0.23.0 Attachments: MAPREDUCE-3068.diff, MAPREDUCE-3068.diff, MAPREDUCE-3068.diff, MAPREDUCE-3068.diff, MAPREDUCE-3068.diff, MAPREDUCE-3068.diff This is same as HADOOP-7154 but for yarn. RM, NM, AM and containers should all have this. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (MAPREDUCE-3068) Should set MALLOC_ARENA_MAX for all YARN daemons and AMs/Containers
[ https://issues.apache.org/jira/browse/MAPREDUCE-3068?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13129142#comment-13129142 ] Chris Riccomini commented on MAPREDUCE-3068: Hey Arun, Could you have a look at the patch I just put up? It uses putIfAbsent from sanitizeEnv - setEnvFromInputStringIfAbsent - addToEnvironment. That way, any env variables set via admin-env will only be applied if they aren't already set in the environment. This should allow apps to override the variables, I believe. I thought it was a little goofy to have putIfAbsent specifically called out for certain env variables and not for others, so I put them all in admin-env, and made all admin-env variables putIfAbsent. That way it covers both the env variables that you list out, as well as any other variables that someone may add. Is this OK? Cheers, Chris Should set MALLOC_ARENA_MAX for all YARN daemons and AMs/Containers --- Key: MAPREDUCE-3068 URL: https://issues.apache.org/jira/browse/MAPREDUCE-3068 Project: Hadoop Map/Reduce Issue Type: Bug Components: mrv2 Affects Versions: 0.23.0 Reporter: Vinod Kumar Vavilapalli Assignee: Chris Riccomini Priority: Blocker Fix For: 0.23.0 Attachments: MAPREDUCE-3068.diff, MAPREDUCE-3068.diff, MAPREDUCE-3068.diff, MAPREDUCE-3068.diff, MAPREDUCE-3068.diff, MAPREDUCE-3068.diff This is same as HADOOP-7154 but for yarn. RM, NM, AM and containers should all have this. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (MAPREDUCE-3068) Should set MALLOC_ARENA_MAX for all YARN daemons and AMs/Containers
[ https://issues.apache.org/jira/browse/MAPREDUCE-3068?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13129149#comment-13129149 ] Arun C Murthy commented on MAPREDUCE-3068: -- I think that makes sense. Thanks. I'll review your patch once more and commit. Should set MALLOC_ARENA_MAX for all YARN daemons and AMs/Containers --- Key: MAPREDUCE-3068 URL: https://issues.apache.org/jira/browse/MAPREDUCE-3068 Project: Hadoop Map/Reduce Issue Type: Bug Components: mrv2 Affects Versions: 0.23.0 Reporter: Vinod Kumar Vavilapalli Assignee: Chris Riccomini Priority: Blocker Fix For: 0.23.0 Attachments: MAPREDUCE-3068.diff, MAPREDUCE-3068.diff, MAPREDUCE-3068.diff, MAPREDUCE-3068.diff, MAPREDUCE-3068.diff, MAPREDUCE-3068.diff This is same as HADOOP-7154 but for yarn. RM, NM, AM and containers should all have this. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (MAPREDUCE-3009) RM UI - Applications - Application(Job History) - Map Tasks - Task ID - Node link is not working
[ https://issues.apache.org/jira/browse/MAPREDUCE-3009?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13129153#comment-13129153 ] Milind Bhandarkar commented on MAPREDUCE-3009: -- @Vinod, bq. I wanted to say I am amused that someone is using IE Just checked the date on that comment :p RM UI - Applications - Application(Job History) - Map Tasks - Task ID - Node link is not working - Key: MAPREDUCE-3009 URL: https://issues.apache.org/jira/browse/MAPREDUCE-3009 Project: Hadoop Map/Reduce Issue Type: Bug Components: jobhistoryserver, mrv2 Affects Versions: 0.23.0, 0.24.0 Reporter: chackaravarthy Assignee: chackaravarthy Fix For: 0.24.0 Attachments: IE_node_link_not_working.png, MAPREDUCE-3009-1.patch, MAPREDUCE-3009.patch, chrome_node_link_working.png, invalid_url.png, invalid_url_latest_code.png RM UI - Applications - Application(Job History) - Map Tasks - Task ID - Node link is not working. The URL contains extra '/' which is causing the problem. Please find in the attached screen shots. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (MAPREDUCE-3068) Should set MALLOC_ARENA_MAX for all YARN daemons and AMs/Containers
[ https://issues.apache.org/jira/browse/MAPREDUCE-3068?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13129157#comment-13129157 ] Arun C Murthy commented on MAPREDUCE-3068: -- Oh wait, on second thoughts... (sorry, you aren't going to like it) I think we want a whitelist of stuff apps can override (MAPREDUCE-2784). Thus we allow only those to be overridable and everything else on admin.env shouldn't be. Hope that makes sense, sorry again! Should set MALLOC_ARENA_MAX for all YARN daemons and AMs/Containers --- Key: MAPREDUCE-3068 URL: https://issues.apache.org/jira/browse/MAPREDUCE-3068 Project: Hadoop Map/Reduce Issue Type: Bug Components: mrv2 Affects Versions: 0.23.0 Reporter: Vinod Kumar Vavilapalli Assignee: Chris Riccomini Priority: Blocker Fix For: 0.23.0 Attachments: MAPREDUCE-3068.diff, MAPREDUCE-3068.diff, MAPREDUCE-3068.diff, MAPREDUCE-3068.diff, MAPREDUCE-3068.diff, MAPREDUCE-3068.diff This is same as HADOOP-7154 but for yarn. RM, NM, AM and containers should all have this. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (MAPREDUCE-3068) Should set MALLOC_ARENA_MAX for all YARN daemons and AMs/Containers
[ https://issues.apache.org/jira/browse/MAPREDUCE-3068?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13129159#comment-13129159 ] Arun C Murthy commented on MAPREDUCE-3068: -- Sorry, that ref. should have been MAPREDUCE-2748 which tracks a 'whitelist'. As Vinod mentioned earlier, we can close that as a dup of this one once we add the whitelist too. Makes sense? Should set MALLOC_ARENA_MAX for all YARN daemons and AMs/Containers --- Key: MAPREDUCE-3068 URL: https://issues.apache.org/jira/browse/MAPREDUCE-3068 Project: Hadoop Map/Reduce Issue Type: Bug Components: mrv2 Affects Versions: 0.23.0 Reporter: Vinod Kumar Vavilapalli Assignee: Chris Riccomini Priority: Blocker Fix For: 0.23.0 Attachments: MAPREDUCE-3068.diff, MAPREDUCE-3068.diff, MAPREDUCE-3068.diff, MAPREDUCE-3068.diff, MAPREDUCE-3068.diff, MAPREDUCE-3068.diff This is same as HADOOP-7154 but for yarn. RM, NM, AM and containers should all have this. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (MAPREDUCE-3068) Should set MALLOC_ARENA_MAX for all YARN daemons and AMs/Containers
[ https://issues.apache.org/jira/browse/MAPREDUCE-3068?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13129165#comment-13129165 ] Chris Riccomini commented on MAPREDUCE-3068: Hey Arun, Just to clarify, basically, you want what I had before, right? Essentially, admin-env never over-writes, and we have hard coded putIfAbsent calls in sanitizeEnv for specific variables (JAVA_HOME, HADOOP_COMMON_HOME, HADOOP_HDFS_HOME and YARN_HOME) Cheers, Chris Should set MALLOC_ARENA_MAX for all YARN daemons and AMs/Containers --- Key: MAPREDUCE-3068 URL: https://issues.apache.org/jira/browse/MAPREDUCE-3068 Project: Hadoop Map/Reduce Issue Type: Bug Components: mrv2 Affects Versions: 0.23.0 Reporter: Vinod Kumar Vavilapalli Assignee: Chris Riccomini Priority: Blocker Fix For: 0.23.0 Attachments: MAPREDUCE-3068.diff, MAPREDUCE-3068.diff, MAPREDUCE-3068.diff, MAPREDUCE-3068.diff, MAPREDUCE-3068.diff, MAPREDUCE-3068.diff This is same as HADOOP-7154 but for yarn. RM, NM, AM and containers should all have this. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (MAPREDUCE-3068) Should set MALLOC_ARENA_MAX for all YARN daemons and AMs/Containers
[ https://issues.apache.org/jira/browse/MAPREDUCE-3068?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13129172#comment-13129172 ] Arun C Murthy commented on MAPREDUCE-3068: -- Yep, that will work. Even better - add a whitelist config key instead of hardcoding the 4 env vars... I'd prefer that if you don't mind. Should set MALLOC_ARENA_MAX for all YARN daemons and AMs/Containers --- Key: MAPREDUCE-3068 URL: https://issues.apache.org/jira/browse/MAPREDUCE-3068 Project: Hadoop Map/Reduce Issue Type: Bug Components: mrv2 Affects Versions: 0.23.0 Reporter: Vinod Kumar Vavilapalli Assignee: Chris Riccomini Priority: Blocker Fix For: 0.23.0 Attachments: MAPREDUCE-3068.diff, MAPREDUCE-3068.diff, MAPREDUCE-3068.diff, MAPREDUCE-3068.diff, MAPREDUCE-3068.diff, MAPREDUCE-3068.diff This is same as HADOOP-7154 but for yarn. RM, NM, AM and containers should all have this. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (MAPREDUCE-3068) Should set MALLOC_ARENA_MAX for all YARN daemons and AMs/Containers
[ https://issues.apache.org/jira/browse/MAPREDUCE-3068?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13129171#comment-13129171 ] Hadoop QA commented on MAPREDUCE-3068: -- -1 overall. Here are the results of testing the latest attachment http://issues.apache.org/jira/secure/attachment/12499426/MAPREDUCE-3068.diff against trunk revision . +1 @author. The patch does not contain any @author tags. +1 tests included. The patch appears to include 3 new or modified tests. +1 javadoc. The javadoc tool did not generate any warning messages. -1 javac. The applied patch generated 1711 javac compiler warnings (more than the trunk's current 1710 warnings). -1 findbugs. The patch appears to introduce 161 new Findbugs (version 1.3.9) warnings. +1 release audit. The applied patch does not increase the total number of release audit warnings. -1 core tests. The patch failed these unit tests: org.apache.hadoop.mapreduce.TestJobMonitorAndPrint +1 contrib tests. The patch passed contrib unit tests. Test results: https://builds.apache.org/job/PreCommit-MAPREDUCE-Build/1043//testReport/ Findbugs warnings: https://builds.apache.org/job/PreCommit-MAPREDUCE-Build/1043//artifact/trunk/hadoop-mapreduce-project/patchprocess/newPatchFindbugsWarningshadoop-mapreduce-client-common.html Findbugs warnings: https://builds.apache.org/job/PreCommit-MAPREDUCE-Build/1043//artifact/trunk/hadoop-mapreduce-project/patchprocess/newPatchFindbugsWarningshadoop-mapreduce-client-core.html Findbugs warnings: https://builds.apache.org/job/PreCommit-MAPREDUCE-Build/1043//artifact/trunk/hadoop-mapreduce-project/patchprocess/newPatchFindbugsWarningshadoop-mapreduce-client-app.html Console output: https://builds.apache.org/job/PreCommit-MAPREDUCE-Build/1043//console This message is automatically generated. Should set MALLOC_ARENA_MAX for all YARN daemons and AMs/Containers --- Key: MAPREDUCE-3068 URL: https://issues.apache.org/jira/browse/MAPREDUCE-3068 Project: Hadoop Map/Reduce Issue Type: Bug Components: mrv2 Affects Versions: 0.23.0 Reporter: Vinod Kumar Vavilapalli Assignee: Chris Riccomini Priority: Blocker Fix For: 0.23.0 Attachments: MAPREDUCE-3068.diff, MAPREDUCE-3068.diff, MAPREDUCE-3068.diff, MAPREDUCE-3068.diff, MAPREDUCE-3068.diff, MAPREDUCE-3068.diff This is same as HADOOP-7154 but for yarn. RM, NM, AM and containers should all have this. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (MAPREDUCE-3068) Should set MALLOC_ARENA_MAX for all YARN daemons and AMs/Containers
[ https://issues.apache.org/jira/browse/MAPREDUCE-3068?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13129176#comment-13129176 ] Chris Riccomini commented on MAPREDUCE-3068: I was worried you would say that. ;) Give me 10 mins. Should set MALLOC_ARENA_MAX for all YARN daemons and AMs/Containers --- Key: MAPREDUCE-3068 URL: https://issues.apache.org/jira/browse/MAPREDUCE-3068 Project: Hadoop Map/Reduce Issue Type: Bug Components: mrv2 Affects Versions: 0.23.0 Reporter: Vinod Kumar Vavilapalli Assignee: Chris Riccomini Priority: Blocker Fix For: 0.23.0 Attachments: MAPREDUCE-3068.diff, MAPREDUCE-3068.diff, MAPREDUCE-3068.diff, MAPREDUCE-3068.diff, MAPREDUCE-3068.diff, MAPREDUCE-3068.diff This is same as HADOOP-7154 but for yarn. RM, NM, AM and containers should all have this. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (MAPREDUCE-3068) Should set MALLOC_ARENA_MAX for all YARN daemons and AMs/Containers
[ https://issues.apache.org/jira/browse/MAPREDUCE-3068?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13129187#comment-13129187 ] Arun C Murthy commented on MAPREDUCE-3068: -- Thanks Chris! Really appreciate your patience... :) Should set MALLOC_ARENA_MAX for all YARN daemons and AMs/Containers --- Key: MAPREDUCE-3068 URL: https://issues.apache.org/jira/browse/MAPREDUCE-3068 Project: Hadoop Map/Reduce Issue Type: Bug Components: mrv2 Affects Versions: 0.23.0 Reporter: Vinod Kumar Vavilapalli Assignee: Chris Riccomini Priority: Blocker Fix For: 0.23.0 Attachments: MAPREDUCE-3068.diff, MAPREDUCE-3068.diff, MAPREDUCE-3068.diff, MAPREDUCE-3068.diff, MAPREDUCE-3068.diff, MAPREDUCE-3068.diff This is same as HADOOP-7154 but for yarn. RM, NM, AM and containers should all have this. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa For more information on JIRA, see: http://www.atlassian.com/software/jira
