[jira] [Created] (MAPREDUCE-3829) [Gridmix] Gridmix should give better error message when input-data directory already exists and -generate option is given
[Gridmix] Gridmix should give better error message when input-data directory already exists and -generate option is given - Key: MAPREDUCE-3829 URL: https://issues.apache.org/jira/browse/MAPREDUCE-3829 Project: Hadoop Map/Reduce Issue Type: Bug Components: contrib/gridmix Reporter: Ravi Gummadi Assignee: Ravi Gummadi Instead of throwing exception messages on to the console, Gridmix should give better error message when input-data directory already exists and -generate option is given. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Updated] (MAPREDUCE-3796) Scheduler.normalizeRequest does not account for allocation requests that exceed maximumAllocation limits
[ https://issues.apache.org/jira/browse/MAPREDUCE-3796?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Hitesh Shah updated MAPREDUCE-3796: --- Target Version/s: 0.23.1, 0.24.0 (was: 0.24.0, 0.23.1) Status: Open (was: Patch Available) Scheduler.normalizeRequest does not account for allocation requests that exceed maximumAllocation limits - Key: MAPREDUCE-3796 URL: https://issues.apache.org/jira/browse/MAPREDUCE-3796 Project: Hadoop Map/Reduce Issue Type: Bug Components: mrv2 Affects Versions: 0.23.0, 0.24.0 Reporter: Hitesh Shah Assignee: Hitesh Shah Priority: Critical Attachments: MR-3796.1.patch, MR-3796.wip.patch -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Created] (MAPREDUCE-3830) DBOutputFormat can't support the update operation to DB
DBOutputFormat can't support the update operation to DB --- Key: MAPREDUCE-3830 URL: https://issues.apache.org/jira/browse/MAPREDUCE-3830 Project: Hadoop Map/Reduce Issue Type: Improvement Affects Versions: 0.20.203.0 Reporter: xsuyu Fix For: 0.20.203.0 After checking the DBOutputFormat class file and found it can't support the update operation to DB, It only support the insert operation. why it can't provide the update operation? Is it an improvement action? -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Updated] (MAPREDUCE-3770) [Rumen] Zombie.getJobConf() results into NPE
[
https://issues.apache.org/jira/browse/MAPREDUCE-3770?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Karam Singh updated MAPREDUCE-3770:
---
Priority: Critical (was: Major)
[Rumen] Zombie.getJobConf() results into NPE
Key: MAPREDUCE-3770
URL: https://issues.apache.org/jira/browse/MAPREDUCE-3770
Project: Hadoop Map/Reduce
Issue Type: Bug
Components: tools/rumen
Affects Versions: 0.23.0, 0.24.0
Reporter: Amar Kamat
Assignee: Amar Kamat
Priority: Critical
Labels: job-name, rumen
Fix For: 0.23.1, 0.24.0
Attachments: GridmixJobNameBug-v1.0.patch
The error trace is as follows
{code}
java.lang.NullPointerException
at java.util.Hashtable.put(Hashtable.java:394)
at java.util.Properties.setProperty(Properties.java:143)
at org.apache.hadoop.conf.Configuration.set(Configuration.java:623)
at org.apache.hadoop.mapred.JobConf.setJobName(JobConf.java:1322)
at
org.apache.hadoop.tools.rumen.ZombieJob.getJobConf(ZombieJob.java:139)
at
org.apache.hadoop.mapred.gridmix.DistributedCacheEmulator.updateHDFSDistCacheFilesList(DistributedCacheEmulator.java:315)
at
org.apache.hadoop.mapred.gridmix.DistributedCacheEmulator.buildDistCacheFilesList(DistributedCacheEmulator.java:280)
at
org.apache.hadoop.mapred.gridmix.DistributedCacheEmulator.setupGenerateDistCacheData(DistributedCacheEmulator.java:253)
at
org.apache.hadoop.mapred.gridmix.Gridmix.setupDistCacheEmulation(Gridmix.java:528)
at
org.apache.hadoop.mapred.gridmix.Gridmix.setupEmulation(Gridmix.java:501)
at org.apache.hadoop.mapred.gridmix.Gridmix.start(Gridmix.java:433)
at org.apache.hadoop.mapred.gridmix.Gridmix.runJob(Gridmix.java:380)
at
org.apache.hadoop.mapred.gridmix.Gridmix.access$000(Gridmix.java:56)
at org.apache.hadoop.mapred.gridmix.Gridmix$1.run(Gridmix.java:313)
at org.apache.hadoop.mapred.gridmix.Gridmix$1.run(Gridmix.java:311)
at java.security.AccessController.doPrivileged(Native Method)
at javax.security.auth.Subject.doAs(Subject.java:396)
at
org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1157)
at org.apache.hadoop.mapred.gridmix.Gridmix.run(Gridmix.java:311)
at org.apache.hadoop.util.ToolRunner.run(ToolRunner.java:69)
at org.apache.hadoop.mapred.gridmix.Gridmix.main(Gridmix.java:606)
at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
at
sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:39)
at
sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:25)
at java.lang.reflect.Method.invoke(Method.java:597)
at org.apache.hadoop.util.RunJar.main(RunJar.java:200)
{code}
The bug seems to be in {{ZombieJob#getName()}} where a not-null check for
jobName.getValue() is missing.
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators:
https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (MAPREDUCE-3770) [Rumen] Zombie.getJobConf() results into NPE
[
https://issues.apache.org/jira/browse/MAPREDUCE-3770?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13202231#comment-13202231
]
Ravi Gummadi commented on MAPREDUCE-3770:
-
Code change looks fine to me.
+1, though it would be nice to have some testcase.
[Rumen] Zombie.getJobConf() results into NPE
Key: MAPREDUCE-3770
URL: https://issues.apache.org/jira/browse/MAPREDUCE-3770
Project: Hadoop Map/Reduce
Issue Type: Bug
Components: tools/rumen
Affects Versions: 0.23.0, 0.24.0
Reporter: Amar Kamat
Assignee: Amar Kamat
Priority: Critical
Labels: job-name, rumen
Fix For: 0.23.1, 0.24.0
Attachments: GridmixJobNameBug-v1.0.patch
The error trace is as follows
{code}
java.lang.NullPointerException
at java.util.Hashtable.put(Hashtable.java:394)
at java.util.Properties.setProperty(Properties.java:143)
at org.apache.hadoop.conf.Configuration.set(Configuration.java:623)
at org.apache.hadoop.mapred.JobConf.setJobName(JobConf.java:1322)
at
org.apache.hadoop.tools.rumen.ZombieJob.getJobConf(ZombieJob.java:139)
at
org.apache.hadoop.mapred.gridmix.DistributedCacheEmulator.updateHDFSDistCacheFilesList(DistributedCacheEmulator.java:315)
at
org.apache.hadoop.mapred.gridmix.DistributedCacheEmulator.buildDistCacheFilesList(DistributedCacheEmulator.java:280)
at
org.apache.hadoop.mapred.gridmix.DistributedCacheEmulator.setupGenerateDistCacheData(DistributedCacheEmulator.java:253)
at
org.apache.hadoop.mapred.gridmix.Gridmix.setupDistCacheEmulation(Gridmix.java:528)
at
org.apache.hadoop.mapred.gridmix.Gridmix.setupEmulation(Gridmix.java:501)
at org.apache.hadoop.mapred.gridmix.Gridmix.start(Gridmix.java:433)
at org.apache.hadoop.mapred.gridmix.Gridmix.runJob(Gridmix.java:380)
at
org.apache.hadoop.mapred.gridmix.Gridmix.access$000(Gridmix.java:56)
at org.apache.hadoop.mapred.gridmix.Gridmix$1.run(Gridmix.java:313)
at org.apache.hadoop.mapred.gridmix.Gridmix$1.run(Gridmix.java:311)
at java.security.AccessController.doPrivileged(Native Method)
at javax.security.auth.Subject.doAs(Subject.java:396)
at
org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1157)
at org.apache.hadoop.mapred.gridmix.Gridmix.run(Gridmix.java:311)
at org.apache.hadoop.util.ToolRunner.run(ToolRunner.java:69)
at org.apache.hadoop.mapred.gridmix.Gridmix.main(Gridmix.java:606)
at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
at
sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:39)
at
sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:25)
at java.lang.reflect.Method.invoke(Method.java:597)
at org.apache.hadoop.util.RunJar.main(RunJar.java:200)
{code}
The bug seems to be in {{ZombieJob#getName()}} where a not-null check for
jobName.getValue() is missing.
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators:
https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Created] (MAPREDUCE-3831) RM scalability runtime is worse than 0.20.204 by 14.2%
RM scalability runtime is worse than 0.20.204 by 14.2% -- Key: MAPREDUCE-3831 URL: https://issues.apache.org/jira/browse/MAPREDUCE-3831 Project: Hadoop Map/Reduce Issue Type: Task Components: benchmarks, resourcemanager Affects Versions: 0.23.1 Reporter: Vinay Kumar Thota RM scalability runtime is worse than 0.20.204 by 14.2% Overall runtime against Fred in a 350 nodes cluster is 2155 secs. Overall runtime against .23 in 350 nodes cluster is 2462 secs. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (MAPREDUCE-3709) TestDistributedShell is failing
[ https://issues.apache.org/jira/browse/MAPREDUCE-3709?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13202300#comment-13202300 ] Hudson commented on MAPREDUCE-3709: --- Integrated in Hadoop-Hdfs-trunk #949 (See [https://builds.apache.org/job/Hadoop-Hdfs-trunk/949/]) MAPREDUCE-3709. TestDistributedShell is failing. (Hitesh Shah via mahadev) mahadev : http://svn.apache.org/viewcvs.cgi/?root=Apache-SVNview=revrev=1241325 Files : * /hadoop/common/trunk/hadoop-mapreduce-project/CHANGES.txt * /hadoop/common/trunk/hadoop-mapreduce-project/hadoop-yarn/hadoop-yarn-applications/hadoop-yarn-applications-distributedshell/pom.xml * /hadoop/common/trunk/hadoop-mapreduce-project/hadoop-yarn/hadoop-yarn-applications/hadoop-yarn-applications-distributedshell/src/main/java/org/apache/hadoop/yarn/applications/distributedshell/Client.java * /hadoop/common/trunk/hadoop-mapreduce-project/hadoop-yarn/hadoop-yarn-applications/hadoop-yarn-applications-distributedshell/src/test/java/org/apache/hadoop/yarn/applications/distributedshell/TestDistributedShell.java TestDistributedShell is failing --- Key: MAPREDUCE-3709 URL: https://issues.apache.org/jira/browse/MAPREDUCE-3709 Project: Hadoop Map/Reduce Issue Type: Bug Components: mrv2, test Affects Versions: 0.23.1 Reporter: Eli Collins Assignee: Hitesh Shah Fix For: 0.23.1 Attachments: MR-3709.1.patch, MR-3709.2.patch TestDistributedShell#testDSShell is failing the assert on line 90 on branch-23. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (MAPREDUCE-3794) Support mapred.Task.Counter and mapred.JobInProgress.Counter enums for compatibility
[ https://issues.apache.org/jira/browse/MAPREDUCE-3794?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13202301#comment-13202301 ] Hudson commented on MAPREDUCE-3794: --- Integrated in Hadoop-Hdfs-trunk #949 (See [https://builds.apache.org/job/Hadoop-Hdfs-trunk/949/]) MAPREDUCE-3794. Support mapred.Task.Counter and mapred.JobInProgress.Counter enums for compatibility (Tom White via mahadev) mahadev : http://svn.apache.org/viewcvs.cgi/?root=Apache-SVNview=revrev=1241299 Files : * /hadoop/common/trunk/hadoop-mapreduce-project/CHANGES.txt * /hadoop/common/trunk/hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-core/src/main/java/org/apache/hadoop/mapred/JobInProgress.java * /hadoop/common/trunk/hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-core/src/main/java/org/apache/hadoop/mapred/Task.java * /hadoop/common/trunk/hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-core/src/test/java/org/apache/hadoop/mapred/TestCounters.java Support mapred.Task.Counter and mapred.JobInProgress.Counter enums for compatibility Key: MAPREDUCE-3794 URL: https://issues.apache.org/jira/browse/MAPREDUCE-3794 Project: Hadoop Map/Reduce Issue Type: Bug Components: mrv2 Affects Versions: 0.23.0 Reporter: Tom White Assignee: Tom White Fix For: 0.23.1 Attachments: MAPREDUCE-3794.patch, MAPREDUCE-3794.patch The new counters are mapreduce.TaskCounter and mapreduce.JobCounter, but we should support the old ones too since they are public in Hadoop 1.x. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (MAPREDUCE-3810) MR AM's ContainerAllocator is assigning the allocated containers very slowly
[ https://issues.apache.org/jira/browse/MAPREDUCE-3810?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13202303#comment-13202303 ] Hudson commented on MAPREDUCE-3810: --- Integrated in Hadoop-Hdfs-trunk #949 (See [https://builds.apache.org/job/Hadoop-Hdfs-trunk/949/]) MAPREDUCE-3810. Performance tweaks - reduced logging in AM and defined hascode/equals for ResourceRequest Priority. Contributed by Vinod K V. acmurthy : http://svn.apache.org/viewcvs.cgi/?root=Apache-SVNview=revrev=1241205 Files : * /hadoop/common/trunk/hadoop-mapreduce-project/CHANGES.txt * /hadoop/common/trunk/hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-app/src/main/java/org/apache/hadoop/mapreduce/v2/app/job/impl/TaskAttemptImpl.java * /hadoop/common/trunk/hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-app/src/main/java/org/apache/hadoop/mapreduce/v2/app/job/impl/TaskImpl.java * /hadoop/common/trunk/hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-app/src/main/java/org/apache/hadoop/mapreduce/v2/app/rm/RMCommunicator.java * /hadoop/common/trunk/hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-app/src/main/java/org/apache/hadoop/mapreduce/v2/app/rm/RMContainerAllocator.java * /hadoop/common/trunk/hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-app/src/main/java/org/apache/hadoop/mapreduce/v2/app/rm/RMContainerRequestor.java * /hadoop/common/trunk/hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-app/src/test/java/org/apache/hadoop/mapreduce/v2/app/MRApp.java * /hadoop/common/trunk/hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-app/src/test/java/org/apache/hadoop/mapreduce/v2/app/MRAppBenchmark.java * /hadoop/common/trunk/hadoop-mapreduce-project/hadoop-yarn/hadoop-yarn-api/src/main/java/org/apache/hadoop/yarn/api/records/Priority.java * /hadoop/common/trunk/hadoop-mapreduce-project/hadoop-yarn/hadoop-yarn-api/src/main/java/org/apache/hadoop/yarn/api/records/ResourceRequest.java * /hadoop/common/trunk/hadoop-mapreduce-project/hadoop-yarn/hadoop-yarn-api/src/main/java/org/apache/hadoop/yarn/api/records/impl/pb/PriorityPBImpl.java * /hadoop/common/trunk/hadoop-mapreduce-project/hadoop-yarn/hadoop-yarn-api/src/main/java/org/apache/hadoop/yarn/api/records/impl/pb/ResourceRequestPBImpl.java MR AM's ContainerAllocator is assigning the allocated containers very slowly Key: MAPREDUCE-3810 URL: https://issues.apache.org/jira/browse/MAPREDUCE-3810 Project: Hadoop Map/Reduce Issue Type: Sub-task Components: mrv2, performance Affects Versions: 0.23.0 Reporter: Vinod Kumar Vavilapalli Assignee: Vinod Kumar Vavilapalli Priority: Blocker Fix For: 0.23.1 Attachments: MAPREDUCE-3810-20120205.txt This is mostly due to logging and other not-so-cheap operations we are doing as part of the AM-RM heartbeat cycle. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (MAPREDUCE-3811) Make the Client-AM IPC retry count configurable
[ https://issues.apache.org/jira/browse/MAPREDUCE-3811?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13202305#comment-13202305 ] Hudson commented on MAPREDUCE-3811: --- Integrated in Hadoop-Hdfs-trunk #949 (See [https://builds.apache.org/job/Hadoop-Hdfs-trunk/949/]) MAPREDUCE-3811. Made jobclient-to-AM retries configurable. Contributed by Siddharth Seth. acmurthy : http://svn.apache.org/viewcvs.cgi/?root=Apache-SVNview=revrev=1241285 Files : * /hadoop/common/trunk/hadoop-mapreduce-project/CHANGES.txt * /hadoop/common/trunk/hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-core/src/main/java/org/apache/hadoop/mapreduce/MRJobConfig.java * /hadoop/common/trunk/hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-core/src/main/resources/mapred-default.xml * /hadoop/common/trunk/hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-jobclient/src/main/java/org/apache/hadoop/mapred/ClientServiceDelegate.java Make the Client-AM IPC retry count configurable --- Key: MAPREDUCE-3811 URL: https://issues.apache.org/jira/browse/MAPREDUCE-3811 Project: Hadoop Map/Reduce Issue Type: Task Components: mrv2 Affects Versions: 0.23.0 Reporter: Siddharth Seth Assignee: Siddharth Seth Priority: Critical Fix For: 0.23.1 Attachments: MR3811.txt -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (MAPREDUCE-3697) Hadoop Counters API limits Oozie's working across different hadoop versions
[ https://issues.apache.org/jira/browse/MAPREDUCE-3697?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13202304#comment-13202304 ] Hudson commented on MAPREDUCE-3697: --- Integrated in Hadoop-Hdfs-trunk #949 (See [https://builds.apache.org/job/Hadoop-Hdfs-trunk/949/]) MAPREDUCE-3697. Support binary compatibility for Counters after MAPREDUCE-901. acmurthy : http://svn.apache.org/viewcvs.cgi/?root=Apache-SVNview=revrev=1241319 Files : * /hadoop/common/trunk/hadoop-mapreduce-project/CHANGES.txt * /hadoop/common/trunk/hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-core/src/main/java/org/apache/hadoop/mapred/Counters.java * /hadoop/common/trunk/hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-core/src/main/java/org/apache/hadoop/mapreduce/Counter.java * /hadoop/common/trunk/hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-core/src/main/java/org/apache/hadoop/mapreduce/Counters.java * /hadoop/common/trunk/hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-core/src/main/java/org/apache/hadoop/mapreduce/counters/AbstractCounters.java * /hadoop/common/trunk/hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-core/src/main/java/org/apache/hadoop/mapreduce/counters/CounterGroupBase.java * /hadoop/common/trunk/hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-core/src/main/java/org/apache/hadoop/mapreduce/counters/FileSystemCounterGroup.java * /hadoop/common/trunk/hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-core/src/main/java/org/apache/hadoop/mapreduce/counters/FrameworkCounterGroup.java * /hadoop/common/trunk/hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-core/src/main/java/org/apache/hadoop/mapreduce/counters/GenericCounter.java * /hadoop/common/trunk/hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-core/src/test/java/org/apache/hadoop/mapred/TestCounters.java * /hadoop/common/trunk/hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-jobclient/src/test/java/org/apache/hadoop/mapred/TestJobCounters.java Hadoop Counters API limits Oozie's working across different hadoop versions --- Key: MAPREDUCE-3697 URL: https://issues.apache.org/jira/browse/MAPREDUCE-3697 Project: Hadoop Map/Reduce Issue Type: Bug Components: mrv2 Affects Versions: 0.23.1 Reporter: John George Assignee: Mahadev konar Priority: Blocker Fix For: 0.23.1 Attachments: MAPREDUCE-3697-facade.patch, MAPREDUCE-3697-facadeV2.patch, MAPREDUCE-3697-facadeV3.patch, MAPREDUCE-3697.patch, MAPREDUCE-3697.patch, MAPREDUCE-3697.patch, MAPREDUCE-3697.patch, MAPREDUCE-3697.patch, MAPREDUCE-3697.patch, apicheck-hadoop-1.0.0-0.24.0-SNAPSHOT.sh Oozie uses Hadoop Counters API, by invoking Counters.getGroup(). However, in hadoop 23, org.apache.hadoop.mapred.Counters does not implement getGroup(). Its parent class AbstractCounters implements it. This is different from hadoop20X. As a result, Oozie compiled with either hadoop version does not work with the other version. A specific scenario, Oozie compiled with .23 and run against 205, does not update job status owing to a Counters API exception. Will explicit re-compilation against the relevant hadoop jars be required each time? This will prevent launching a uniform Oozie version across different clusters. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (MAPREDUCE-3813) RackResolver should maintain a cache to avoid repetitive lookups.
[ https://issues.apache.org/jira/browse/MAPREDUCE-3813?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13202307#comment-13202307 ] Hudson commented on MAPREDUCE-3813: --- Integrated in Hadoop-Hdfs-trunk #949 (See [https://builds.apache.org/job/Hadoop-Hdfs-trunk/949/]) MAPREDUCE-3813. Added a cache for resolved racks. Contributed by Vinod K V. acmurthy : http://svn.apache.org/viewcvs.cgi/?root=Apache-SVNview=revrev=1241209 Files : * /hadoop/common/trunk/hadoop-mapreduce-project/CHANGES.txt * /hadoop/common/trunk/hadoop-mapreduce-project/hadoop-yarn/hadoop-yarn-common/src/main/java/org/apache/hadoop/yarn/util/RackResolver.java * /hadoop/common/trunk/hadoop-mapreduce-project/hadoop-yarn/hadoop-yarn-common/src/test/java/org/apache/hadoop/yarn/util/TestRackResolver.java RackResolver should maintain a cache to avoid repetitive lookups. - Key: MAPREDUCE-3813 URL: https://issues.apache.org/jira/browse/MAPREDUCE-3813 Project: Hadoop Map/Reduce Issue Type: Sub-task Components: mrv2, performance Affects Versions: 0.23.0 Reporter: Vinod Kumar Vavilapalli Assignee: Vinod Kumar Vavilapalli Fix For: 0.23.1 Attachments: MAPREDUCE-3813-20120205.txt With the current code, during task creation, we repeatedly resolve hosts and RackResolver doesn't cache any of the results. Caching will improve performance. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (MAPREDUCE-3817) bin/mapred command cannot run distcp and archive jobs
[ https://issues.apache.org/jira/browse/MAPREDUCE-3817?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13202311#comment-13202311 ] Hudson commented on MAPREDUCE-3817: --- Integrated in Hadoop-Hdfs-trunk #949 (See [https://builds.apache.org/job/Hadoop-Hdfs-trunk/949/]) MAPREDUCE-3817. Fixed bin/mapred to allow running of distcp and archive jobs. Contributed by Arpit Gupta. acmurthy : http://svn.apache.org/viewcvs.cgi/?root=Apache-SVNview=revrev=1241323 Files : * /hadoop/common/trunk/hadoop-mapreduce-project/CHANGES.txt * /hadoop/common/trunk/hadoop-mapreduce-project/bin/mapred bin/mapred command cannot run distcp and archive jobs - Key: MAPREDUCE-3817 URL: https://issues.apache.org/jira/browse/MAPREDUCE-3817 Project: Hadoop Map/Reduce Issue Type: Bug Components: mrv2 Affects Versions: 0.23.1, 0.24.0 Reporter: Arpit Gupta Assignee: Arpit Gupta Fix For: 0.23.1 Attachments: MAPREDUCE-3817.patch, MAPREDUCE-3817.patch -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (MAPREDUCE-3809) Tasks may take upto 3 seconds to exit after completion
[ https://issues.apache.org/jira/browse/MAPREDUCE-3809?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13202312#comment-13202312 ] Hudson commented on MAPREDUCE-3809: --- Integrated in Hadoop-Hdfs-trunk #949 (See [https://builds.apache.org/job/Hadoop-Hdfs-trunk/949/]) MAPREDUCE-3809. Ensure that there is no needless sleep in Task at the end of the task. Contributed by Siddharth Seth. acmurthy : http://svn.apache.org/viewcvs.cgi/?root=Apache-SVNview=revrev=1241282 Files : * /hadoop/common/trunk/hadoop-mapreduce-project/CHANGES.txt * /hadoop/common/trunk/hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-core/src/main/java/org/apache/hadoop/mapred/Task.java Tasks may take upto 3 seconds to exit after completion -- Key: MAPREDUCE-3809 URL: https://issues.apache.org/jira/browse/MAPREDUCE-3809 Project: Hadoop Map/Reduce Issue Type: Sub-task Components: mrv2 Affects Versions: 0.23.1, 1.0.0 Reporter: Siddharth Seth Assignee: Siddharth Seth Priority: Blocker Fix For: 0.23.1 Attachments: MR3809.txt, MR3809.txt Task.TaskReporter.stopCommunicationThread can end up waiting for a thread.sleep(3000) before stopping the thread. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (MAPREDUCE-3818) Trunk MRV1 compilation is broken.
[
https://issues.apache.org/jira/browse/MAPREDUCE-3818?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13202310#comment-13202310
]
Hudson commented on MAPREDUCE-3818:
---
Integrated in Hadoop-Hdfs-trunk #949 (See
[https://builds.apache.org/job/Hadoop-Hdfs-trunk/949/])
MAPREDUCE-3818. Fixed broken compilation in TestSubmitJob after the patch
for HDFS-2895. Contributed by Suresh Srinivas.
vinodkv :
http://svn.apache.org/viewcvs.cgi/?root=Apache-SVNview=revrev=1241253
Files :
* /hadoop/common/trunk/hadoop-mapreduce-project/CHANGES.txt
*
/hadoop/common/trunk/hadoop-mapreduce-project/src/test/mapred/org/apache/hadoop/mapred/TestSubmitJob.java
Trunk MRV1 compilation is broken.
-
Key: MAPREDUCE-3818
URL: https://issues.apache.org/jira/browse/MAPREDUCE-3818
Project: Hadoop Map/Reduce
Issue Type: Bug
Components: build, test
Affects Versions: 0.24.0
Reporter: Vinod Kumar Vavilapalli
Assignee: Suresh Srinivas
Priority: Blocker
Fix For: 0.24.0
Attachments: MAPREDUCE-3818.txt
Seeing this:
{code}
[javac]
/Users/vinodkv/Workspace/eclipse-workspace/apache-git/hadoop-common/hadoop-mapreduce-project/src/test/mapred/org/apache/hadoop/mapred/TestSubmitJob.java:155:
cannot find symbol
[javac] symbol : class ClientNamenodeWireProtocol
[javac] location: class org.apache.hadoop.mapred.TestSubmitJob
[javac] RPC.getProxy(ClientNamenodeWireProtocol.class,
[javac]^
{code}
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators:
https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (MAPREDUCE-3814) MR1 compile fails
[ https://issues.apache.org/jira/browse/MAPREDUCE-3814?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13202306#comment-13202306 ] Hudson commented on MAPREDUCE-3814: --- Integrated in Hadoop-Hdfs-trunk #949 (See [https://builds.apache.org/job/Hadoop-Hdfs-trunk/949/]) MAPREDUCE-3814. Fixed MRV1 compilation. (Arun C Murthy via vinodkv) vinodkv : http://svn.apache.org/viewcvs.cgi/?root=Apache-SVNview=revrev=1241177 Files : * /hadoop/common/trunk/hadoop-mapreduce-project/CHANGES.txt * /hadoop/common/trunk/hadoop-mapreduce-project/build.xml MR1 compile fails - Key: MAPREDUCE-3814 URL: https://issues.apache.org/jira/browse/MAPREDUCE-3814 Project: Hadoop Map/Reduce Issue Type: Bug Components: mrv1, mrv2 Affects Versions: 0.23.1 Reporter: Arun C Murthy Assignee: Arun C Murthy Fix For: 0.23.1 Attachments: MAPREDUCE-3814.patch $ ant veryclean all-jars -Dversion=0.23.1 -Dresolvers=internal BUILD FAILED /grid/0/dev/acm/hadoop-0.23/hadoop-mapreduce-project/build.xml:537: srcdir /grid/0/dev/acm/hadoop-0.23/hadoop-mapreduce-project/src/test/mapred/testjar does not exist! -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (MAPREDUCE-901) Move Framework Counters into a TaskMetric structure
[ https://issues.apache.org/jira/browse/MAPREDUCE-901?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13202308#comment-13202308 ] Hudson commented on MAPREDUCE-901: -- Integrated in Hadoop-Hdfs-trunk #949 (See [https://builds.apache.org/job/Hadoop-Hdfs-trunk/949/]) MAPREDUCE-3697. Support binary compatibility for Counters after MAPREDUCE-901. acmurthy : http://svn.apache.org/viewcvs.cgi/?root=Apache-SVNview=revrev=1241319 Files : * /hadoop/common/trunk/hadoop-mapreduce-project/CHANGES.txt * /hadoop/common/trunk/hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-core/src/main/java/org/apache/hadoop/mapred/Counters.java * /hadoop/common/trunk/hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-core/src/main/java/org/apache/hadoop/mapreduce/Counter.java * /hadoop/common/trunk/hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-core/src/main/java/org/apache/hadoop/mapreduce/Counters.java * /hadoop/common/trunk/hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-core/src/main/java/org/apache/hadoop/mapreduce/counters/AbstractCounters.java * /hadoop/common/trunk/hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-core/src/main/java/org/apache/hadoop/mapreduce/counters/CounterGroupBase.java * /hadoop/common/trunk/hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-core/src/main/java/org/apache/hadoop/mapreduce/counters/FileSystemCounterGroup.java * /hadoop/common/trunk/hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-core/src/main/java/org/apache/hadoop/mapreduce/counters/FrameworkCounterGroup.java * /hadoop/common/trunk/hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-core/src/main/java/org/apache/hadoop/mapreduce/counters/GenericCounter.java * /hadoop/common/trunk/hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-core/src/test/java/org/apache/hadoop/mapred/TestCounters.java * /hadoop/common/trunk/hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-jobclient/src/test/java/org/apache/hadoop/mapred/TestJobCounters.java Move Framework Counters into a TaskMetric structure --- Key: MAPREDUCE-901 URL: https://issues.apache.org/jira/browse/MAPREDUCE-901 Project: Hadoop Map/Reduce Issue Type: Improvement Components: task Affects Versions: 0.21.0 Reporter: Owen O'Malley Assignee: Luke Lu Fix For: 0.23.0 Attachments: 901_1.patch, 901_1.patch, FrameworkCounterGroup.java, MAPREDUCE-901.patch, MAPREDUCE-901.patch, MAPREDUCE-901.patch, MAPREDUCE-901.patch_2, mr-901-trunk-v1.patch I think we should move all of the Counters that the framework updates into a single class called TaskMetrics. TaskMetrics would have specific fields for each of the metrics like input records, input bytes, output records, etc. It would both reduce the serialized size of the heartbeats (by shrinking the Counters down to just the user's counters) and decrease the latency for updates to the JobTracker (since Counters are sent at most 1/minute instead of 1/heartbeat). -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (MAPREDUCE-3808) NPE in FileOutputCommitter when running a 0 reduce job
[
https://issues.apache.org/jira/browse/MAPREDUCE-3808?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13202309#comment-13202309
]
Hudson commented on MAPREDUCE-3808:
---
Integrated in Hadoop-Hdfs-trunk #949 (See
[https://builds.apache.org/job/Hadoop-Hdfs-trunk/949/])
MAPREDUCE-3808. Fixed an NPE in FileOutputCommitter for jobs with maps but
no reduces. Contributed by Robert Joseph Evans.
vinodkv :
http://svn.apache.org/viewcvs.cgi/?root=Apache-SVNview=revrev=1241217
Files :
* /hadoop/common/trunk/hadoop-mapreduce-project/CHANGES.txt
*
/hadoop/common/trunk/hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-core/src/main/java/org/apache/hadoop/mapred/FileOutputCommitter.java
*
/hadoop/common/trunk/hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-core/src/main/java/org/apache/hadoop/mapreduce/lib/output/FileOutputCommitter.java
*
/hadoop/common/trunk/hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-core/src/test/java/org/apache/hadoop/mapred/TestFileOutputCommitter.java
NPE in FileOutputCommitter when running a 0 reduce job
--
Key: MAPREDUCE-3808
URL: https://issues.apache.org/jira/browse/MAPREDUCE-3808
Project: Hadoop Map/Reduce
Issue Type: Bug
Components: mrv2
Affects Versions: 0.23.0, 0.24.0
Reporter: Siddharth Seth
Assignee: Robert Joseph Evans
Priority: Blocker
Fix For: 0.23.1
Attachments: MR-3808.txt
This was while running LoadGen.
{noformat}
Error: java.lang.NullPointerException at
org.apache.hadoop.fs.Path.init(Path.java:67)
at org.apache.hadoop.fs.Path.init(Path.java:56)
at
org.apache.hadoop.mapreduce.lib.output.FileOutputCommitter.getPendingJobAttemptsPath(FileOutputCommitter.java:118)
at
org.apache.hadoop.mapreduce.lib.output.FileOutputCommitter.getJobAttemptPath(FileOutputCommitter.java:167)
at
org.apache.hadoop.mapreduce.lib.output.FileOutputCommitter.getJobAttemptPath(FileOutputCommitter.java:149)
at
org.apache.hadoop.mapreduce.lib.output.FileOutputCommitter.getPendingTaskAttemptsPath(FileOutputCommitter.java:185)
at
org.apache.hadoop.mapreduce.lib.output.FileOutputCommitter.getTaskAttemptPath(FileOutputCommitter.java:209)
at
org.apache.hadoop.mapred.FileOutputCommitter.getTaskAttemptPath(FileOutputCommitter.java:100)
at
org.apache.hadoop.mapred.FileOutputCommitter.getTaskAttemptPath(FileOutputCommitter.java:94)
at
org.apache.hadoop.mapred.FileOutputCommitter.needsTaskCommit(FileOutputCommitter.java:176)
at
org.apache.hadoop.mapred.OutputCommitter.needsTaskCommit(OutputCommitter.java:248)
at org.apache.hadoop.mapred.Task.isCommitRequired(Task.java:955)
at org.apache.hadoop.mapred.Task.done(Task.java:912)
at org.apache.hadoop.mapred.MapTask.run(MapTask.java:331)
at org.apache.hadoop.mapred.YarnChild$2.run(YarnChild.java:147)
at java.security.AccessController.doPrivileged(Native Method)
at javax.security.auth.Subject.doAs(Subject.java:396)
at
org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1157)
at org.apache.hadoop.mapred.YarnChild.main(YarnChild.java:142)
{noformat}
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators:
https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (MAPREDUCE-3354) JobHistoryServer should be started by bin/mapred and not by bin/yarn
[ https://issues.apache.org/jira/browse/MAPREDUCE-3354?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13202313#comment-13202313 ] Hudson commented on MAPREDUCE-3354: --- Integrated in Hadoop-Hdfs-trunk #949 (See [https://builds.apache.org/job/Hadoop-Hdfs-trunk/949/]) MAPREDUCE-3354. Changed scripts so that jobhistory server is started by bin/mapred instead of bin/yarn. Contributed by Jonathan Eagles. acmurthy : http://svn.apache.org/viewcvs.cgi/?root=Apache-SVNview=revrev=1241250 Files : * /hadoop/common/trunk/hadoop-assemblies/src/main/resources/assemblies/hadoop-mapreduce-dist.xml * /hadoop/common/trunk/hadoop-mapreduce-project/CHANGES.txt * /hadoop/common/trunk/hadoop-mapreduce-project/INSTALL * /hadoop/common/trunk/hadoop-mapreduce-project/bin/mapred * /hadoop/common/trunk/hadoop-mapreduce-project/bin/mr-jobhistory-daemon.sh * /hadoop/common/trunk/hadoop-mapreduce-project/hadoop-yarn/bin/start-yarn.sh * /hadoop/common/trunk/hadoop-mapreduce-project/hadoop-yarn/bin/stop-yarn.sh * /hadoop/common/trunk/hadoop-mapreduce-project/hadoop-yarn/bin/yarn * /hadoop/common/trunk/hadoop-mapreduce-project/hadoop-yarn/bin/yarn-daemon.sh * /hadoop/common/trunk/hadoop-mapreduce-project/hadoop-yarn/hadoop-yarn-site/src/site/apt/ClusterSetup.apt.vm JobHistoryServer should be started by bin/mapred and not by bin/yarn Key: MAPREDUCE-3354 URL: https://issues.apache.org/jira/browse/MAPREDUCE-3354 Project: Hadoop Map/Reduce Issue Type: Bug Components: jobhistoryserver, mrv2 Affects Versions: 0.23.0 Reporter: Vinod Kumar Vavilapalli Assignee: Jonathan Eagles Priority: Blocker Fix For: 0.23.1 Attachments: MAPREDUCE-3354.patch, MAPREDUCE-3354.patch, MAPREDUCE-3354.patch, MAPREDUCE-3354.patch, MAPREDUCE-3354.patch, MAPREDUCE-3354.patch, MAPREDUCE-3354.patch, MAPREDUCE-3354.patch, MAPREDUCE-3354.patch, MAPREDUCE-3354.patch, MAPREDUCE-3354.patch, MAPREDUCE-3354.patch JobHistoryServer belongs to mapreduce land. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (MAPREDUCE-3804) yarn webapp interface vulnerable to cross scripting attacks
[ https://issues.apache.org/jira/browse/MAPREDUCE-3804?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13202314#comment-13202314 ] Hudson commented on MAPREDUCE-3804: --- Integrated in Hadoop-Hdfs-trunk #949 (See [https://builds.apache.org/job/Hadoop-Hdfs-trunk/949/]) MAPREDUCE-3804. yarn webapp interface vulnerable to cross scripting attacks (Dave Thompson via bobby) bobby : http://svn.apache.org/viewcvs.cgi/?root=Apache-SVNview=revrev=1241225 Files : * /hadoop/common/trunk/hadoop-mapreduce-project/CHANGES.txt * /hadoop/common/trunk/hadoop-mapreduce-project/hadoop-yarn/hadoop-yarn-common/src/main/java/org/apache/hadoop/yarn/webapp/Dispatcher.java * /hadoop/common/trunk/hadoop-mapreduce-project/hadoop-yarn/hadoop-yarn-common/src/main/java/org/apache/hadoop/yarn/webapp/hamlet/HamletImpl.java yarn webapp interface vulnerable to cross scripting attacks --- Key: MAPREDUCE-3804 URL: https://issues.apache.org/jira/browse/MAPREDUCE-3804 Project: Hadoop Map/Reduce Issue Type: Bug Components: jobhistoryserver, mrv2, resourcemanager Affects Versions: 0.23.0 Reporter: Dave Thompson Assignee: Dave Thompson Fix For: 0.23.1 Attachments: MAPREDUCE-3804.patch, MAPREDUCE-3804.patch, MAPREDUCE_3804_br_0.23.0.patch Yarn webapp interface may be vulnerable to certain cross scripting attacks, injected through URL request. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (MAPREDUCE-3709) TestDistributedShell is failing
[ https://issues.apache.org/jira/browse/MAPREDUCE-3709?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13202319#comment-13202319 ] Hudson commented on MAPREDUCE-3709: --- Integrated in Hadoop-Hdfs-0.23-Build #162 (See [https://builds.apache.org/job/Hadoop-Hdfs-0.23-Build/162/]) MAPREDUCE-3709. TestDistributedShell is failing. (Hitesh Shah via mahadev) - Merging r1241325 from trunk. mahadev : http://svn.apache.org/viewcvs.cgi/?root=Apache-SVNview=revrev=1241327 Files : * /hadoop/common/branches/branch-0.23/hadoop-mapreduce-project/CHANGES.txt * /hadoop/common/branches/branch-0.23/hadoop-mapreduce-project/hadoop-yarn/hadoop-yarn-applications/hadoop-yarn-applications-distributedshell/pom.xml * /hadoop/common/branches/branch-0.23/hadoop-mapreduce-project/hadoop-yarn/hadoop-yarn-applications/hadoop-yarn-applications-distributedshell/src/main/java/org/apache/hadoop/yarn/applications/distributedshell/Client.java * /hadoop/common/branches/branch-0.23/hadoop-mapreduce-project/hadoop-yarn/hadoop-yarn-applications/hadoop-yarn-applications-distributedshell/src/test/java/org/apache/hadoop/yarn/applications/distributedshell/TestDistributedShell.java TestDistributedShell is failing --- Key: MAPREDUCE-3709 URL: https://issues.apache.org/jira/browse/MAPREDUCE-3709 Project: Hadoop Map/Reduce Issue Type: Bug Components: mrv2, test Affects Versions: 0.23.1 Reporter: Eli Collins Assignee: Hitesh Shah Fix For: 0.23.1 Attachments: MR-3709.1.patch, MR-3709.2.patch TestDistributedShell#testDSShell is failing the assert on line 90 on branch-23. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (MAPREDUCE-3697) Hadoop Counters API limits Oozie's working across different hadoop versions
[ https://issues.apache.org/jira/browse/MAPREDUCE-3697?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13202322#comment-13202322 ] Hudson commented on MAPREDUCE-3697: --- Integrated in Hadoop-Hdfs-0.23-Build #162 (See [https://builds.apache.org/job/Hadoop-Hdfs-0.23-Build/162/]) Merge -c 1241319 from trunk to branch-0.23 to fix MAPREDUCE-3697. Support binary compatibility for Counters after MAPREDUCE-901. acmurthy : http://svn.apache.org/viewcvs.cgi/?root=Apache-SVNview=revrev=1241321 Files : * /hadoop/common/branches/branch-0.23/hadoop-mapreduce-project/CHANGES.txt * /hadoop/common/branches/branch-0.23/hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-core/src/main/java/org/apache/hadoop/mapred/Counters.java * /hadoop/common/branches/branch-0.23/hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-core/src/main/java/org/apache/hadoop/mapreduce/Counter.java * /hadoop/common/branches/branch-0.23/hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-core/src/main/java/org/apache/hadoop/mapreduce/Counters.java * /hadoop/common/branches/branch-0.23/hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-core/src/main/java/org/apache/hadoop/mapreduce/counters/AbstractCounters.java * /hadoop/common/branches/branch-0.23/hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-core/src/main/java/org/apache/hadoop/mapreduce/counters/CounterGroupBase.java * /hadoop/common/branches/branch-0.23/hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-core/src/main/java/org/apache/hadoop/mapreduce/counters/FileSystemCounterGroup.java * /hadoop/common/branches/branch-0.23/hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-core/src/main/java/org/apache/hadoop/mapreduce/counters/FrameworkCounterGroup.java * /hadoop/common/branches/branch-0.23/hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-core/src/main/java/org/apache/hadoop/mapreduce/counters/GenericCounter.java * /hadoop/common/branches/branch-0.23/hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-core/src/test/java/org/apache/hadoop/mapred/TestCounters.java * /hadoop/common/branches/branch-0.23/hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-jobclient/src/test/java/org/apache/hadoop/mapred/TestJobCounters.java Hadoop Counters API limits Oozie's working across different hadoop versions --- Key: MAPREDUCE-3697 URL: https://issues.apache.org/jira/browse/MAPREDUCE-3697 Project: Hadoop Map/Reduce Issue Type: Bug Components: mrv2 Affects Versions: 0.23.1 Reporter: John George Assignee: Mahadev konar Priority: Blocker Fix For: 0.23.1 Attachments: MAPREDUCE-3697-facade.patch, MAPREDUCE-3697-facadeV2.patch, MAPREDUCE-3697-facadeV3.patch, MAPREDUCE-3697.patch, MAPREDUCE-3697.patch, MAPREDUCE-3697.patch, MAPREDUCE-3697.patch, MAPREDUCE-3697.patch, MAPREDUCE-3697.patch, apicheck-hadoop-1.0.0-0.24.0-SNAPSHOT.sh Oozie uses Hadoop Counters API, by invoking Counters.getGroup(). However, in hadoop 23, org.apache.hadoop.mapred.Counters does not implement getGroup(). Its parent class AbstractCounters implements it. This is different from hadoop20X. As a result, Oozie compiled with either hadoop version does not work with the other version. A specific scenario, Oozie compiled with .23 and run against 205, does not update job status owing to a Counters API exception. Will explicit re-compilation against the relevant hadoop jars be required each time? This will prevent launching a uniform Oozie version across different clusters. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (MAPREDUCE-3794) Support mapred.Task.Counter and mapred.JobInProgress.Counter enums for compatibility
[ https://issues.apache.org/jira/browse/MAPREDUCE-3794?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13202320#comment-13202320 ] Hudson commented on MAPREDUCE-3794: --- Integrated in Hadoop-Hdfs-0.23-Build #162 (See [https://builds.apache.org/job/Hadoop-Hdfs-0.23-Build/162/]) MAPREDUCE-3794. Support mapred.Task.Counter and mapred.JobInProgress.Counter enums for compatibility (Tom White via mahadev) - Merging r1241299 from trunk. mahadev : http://svn.apache.org/viewcvs.cgi/?root=Apache-SVNview=revrev=1241300 Files : * /hadoop/common/branches/branch-0.23/hadoop-mapreduce-project/CHANGES.txt * /hadoop/common/branches/branch-0.23/hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-core/src/main/java/org/apache/hadoop/mapred/JobInProgress.java * /hadoop/common/branches/branch-0.23/hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-core/src/main/java/org/apache/hadoop/mapred/Task.java * /hadoop/common/branches/branch-0.23/hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-core/src/test/java/org/apache/hadoop/mapred/TestCounters.java Support mapred.Task.Counter and mapred.JobInProgress.Counter enums for compatibility Key: MAPREDUCE-3794 URL: https://issues.apache.org/jira/browse/MAPREDUCE-3794 Project: Hadoop Map/Reduce Issue Type: Bug Components: mrv2 Affects Versions: 0.23.0 Reporter: Tom White Assignee: Tom White Fix For: 0.23.1 Attachments: MAPREDUCE-3794.patch, MAPREDUCE-3794.patch The new counters are mapreduce.TaskCounter and mapreduce.JobCounter, but we should support the old ones too since they are public in Hadoop 1.x. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (MAPREDUCE-3810) MR AM's ContainerAllocator is assigning the allocated containers very slowly
[ https://issues.apache.org/jira/browse/MAPREDUCE-3810?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13202321#comment-13202321 ] Hudson commented on MAPREDUCE-3810: --- Integrated in Hadoop-Hdfs-0.23-Build #162 (See [https://builds.apache.org/job/Hadoop-Hdfs-0.23-Build/162/]) Merge -c 1241205 from trunk to branch-0.23 to fix MAPREDUCE-3810. Performance tweaks - reduced logging in AM and defined hascode/equals for ResourceRequest Priority. acmurthy : http://svn.apache.org/viewcvs.cgi/?root=Apache-SVNview=revrev=1241208 Files : * /hadoop/common/branches/branch-0.23/hadoop-mapreduce-project/CHANGES.txt * /hadoop/common/branches/branch-0.23/hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-app/src/main/java/org/apache/hadoop/mapreduce/v2/app/job/impl/TaskAttemptImpl.java * /hadoop/common/branches/branch-0.23/hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-app/src/main/java/org/apache/hadoop/mapreduce/v2/app/job/impl/TaskImpl.java * /hadoop/common/branches/branch-0.23/hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-app/src/main/java/org/apache/hadoop/mapreduce/v2/app/rm/RMCommunicator.java * /hadoop/common/branches/branch-0.23/hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-app/src/main/java/org/apache/hadoop/mapreduce/v2/app/rm/RMContainerAllocator.java * /hadoop/common/branches/branch-0.23/hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-app/src/main/java/org/apache/hadoop/mapreduce/v2/app/rm/RMContainerRequestor.java * /hadoop/common/branches/branch-0.23/hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-app/src/test/java/org/apache/hadoop/mapreduce/v2/app/MRApp.java * /hadoop/common/branches/branch-0.23/hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-app/src/test/java/org/apache/hadoop/mapreduce/v2/app/MRAppBenchmark.java * /hadoop/common/branches/branch-0.23/hadoop-mapreduce-project/hadoop-yarn/hadoop-yarn-api/src/main/java/org/apache/hadoop/yarn/api/records/Priority.java * /hadoop/common/branches/branch-0.23/hadoop-mapreduce-project/hadoop-yarn/hadoop-yarn-api/src/main/java/org/apache/hadoop/yarn/api/records/ResourceRequest.java * /hadoop/common/branches/branch-0.23/hadoop-mapreduce-project/hadoop-yarn/hadoop-yarn-api/src/main/java/org/apache/hadoop/yarn/api/records/impl/pb/PriorityPBImpl.java * /hadoop/common/branches/branch-0.23/hadoop-mapreduce-project/hadoop-yarn/hadoop-yarn-api/src/main/java/org/apache/hadoop/yarn/api/records/impl/pb/ResourceRequestPBImpl.java MR AM's ContainerAllocator is assigning the allocated containers very slowly Key: MAPREDUCE-3810 URL: https://issues.apache.org/jira/browse/MAPREDUCE-3810 Project: Hadoop Map/Reduce Issue Type: Sub-task Components: mrv2, performance Affects Versions: 0.23.0 Reporter: Vinod Kumar Vavilapalli Assignee: Vinod Kumar Vavilapalli Priority: Blocker Fix For: 0.23.1 Attachments: MAPREDUCE-3810-20120205.txt This is mostly due to logging and other not-so-cheap operations we are doing as part of the AM-RM heartbeat cycle. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (MAPREDUCE-3813) RackResolver should maintain a cache to avoid repetitive lookups.
[ https://issues.apache.org/jira/browse/MAPREDUCE-3813?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13202324#comment-13202324 ] Hudson commented on MAPREDUCE-3813: --- Integrated in Hadoop-Hdfs-0.23-Build #162 (See [https://builds.apache.org/job/Hadoop-Hdfs-0.23-Build/162/]) Merge -c 1241209 from trunk to branch-0.23 to fix MAPREDUCE-3813. Added a cache for resolved racks. acmurthy : http://svn.apache.org/viewcvs.cgi/?root=Apache-SVNview=revrev=1241210 Files : * /hadoop/common/branches/branch-0.23/hadoop-mapreduce-project/CHANGES.txt * /hadoop/common/branches/branch-0.23/hadoop-mapreduce-project/hadoop-yarn/hadoop-yarn-common/src/main/java/org/apache/hadoop/yarn/util/RackResolver.java * /hadoop/common/branches/branch-0.23/hadoop-mapreduce-project/hadoop-yarn/hadoop-yarn-common/src/test/java/org/apache/hadoop/yarn/util/TestRackResolver.java RackResolver should maintain a cache to avoid repetitive lookups. - Key: MAPREDUCE-3813 URL: https://issues.apache.org/jira/browse/MAPREDUCE-3813 Project: Hadoop Map/Reduce Issue Type: Sub-task Components: mrv2, performance Affects Versions: 0.23.0 Reporter: Vinod Kumar Vavilapalli Assignee: Vinod Kumar Vavilapalli Fix For: 0.23.1 Attachments: MAPREDUCE-3813-20120205.txt With the current code, during task creation, we repeatedly resolve hosts and RackResolver doesn't cache any of the results. Caching will improve performance. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (MAPREDUCE-3354) JobHistoryServer should be started by bin/mapred and not by bin/yarn
[ https://issues.apache.org/jira/browse/MAPREDUCE-3354?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13202329#comment-13202329 ] Hudson commented on MAPREDUCE-3354: --- Integrated in Hadoop-Hdfs-0.23-Build #162 (See [https://builds.apache.org/job/Hadoop-Hdfs-0.23-Build/162/]) Merge -c 1241250 from trunk to branch-0.23 to fix MAPREDUCE-3354. Changed scripts so that jobhistory server is started by bin/mapred instead of bin/yarn. Contributed by Jonathan Eagles. acmurthy : http://svn.apache.org/viewcvs.cgi/?root=Apache-SVNview=revrev=1241252 Files : * /hadoop/common/branches/branch-0.23/hadoop-assemblies/src/main/resources/assemblies/hadoop-mapreduce-dist.xml * /hadoop/common/branches/branch-0.23/hadoop-mapreduce-project/CHANGES.txt * /hadoop/common/branches/branch-0.23/hadoop-mapreduce-project/INSTALL * /hadoop/common/branches/branch-0.23/hadoop-mapreduce-project/bin/mapred * /hadoop/common/branches/branch-0.23/hadoop-mapreduce-project/bin/mr-jobhistory-daemon.sh * /hadoop/common/branches/branch-0.23/hadoop-mapreduce-project/hadoop-yarn/bin/start-yarn.sh * /hadoop/common/branches/branch-0.23/hadoop-mapreduce-project/hadoop-yarn/bin/stop-yarn.sh * /hadoop/common/branches/branch-0.23/hadoop-mapreduce-project/hadoop-yarn/bin/yarn * /hadoop/common/branches/branch-0.23/hadoop-mapreduce-project/hadoop-yarn/bin/yarn-daemon.sh * /hadoop/common/branches/branch-0.23/hadoop-mapreduce-project/hadoop-yarn/hadoop-yarn-site/src/site/apt/ClusterSetup.apt.vm JobHistoryServer should be started by bin/mapred and not by bin/yarn Key: MAPREDUCE-3354 URL: https://issues.apache.org/jira/browse/MAPREDUCE-3354 Project: Hadoop Map/Reduce Issue Type: Bug Components: jobhistoryserver, mrv2 Affects Versions: 0.23.0 Reporter: Vinod Kumar Vavilapalli Assignee: Jonathan Eagles Priority: Blocker Fix For: 0.23.1 Attachments: MAPREDUCE-3354.patch, MAPREDUCE-3354.patch, MAPREDUCE-3354.patch, MAPREDUCE-3354.patch, MAPREDUCE-3354.patch, MAPREDUCE-3354.patch, MAPREDUCE-3354.patch, MAPREDUCE-3354.patch, MAPREDUCE-3354.patch, MAPREDUCE-3354.patch, MAPREDUCE-3354.patch, MAPREDUCE-3354.patch JobHistoryServer belongs to mapreduce land. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (MAPREDUCE-3804) yarn webapp interface vulnerable to cross scripting attacks
[ https://issues.apache.org/jira/browse/MAPREDUCE-3804?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13202330#comment-13202330 ] Hudson commented on MAPREDUCE-3804: --- Integrated in Hadoop-Hdfs-0.23-Build #162 (See [https://builds.apache.org/job/Hadoop-Hdfs-0.23-Build/162/]) svn merge -c 1241225 fixes MAPREDUCE-3804 bobby : http://svn.apache.org/viewcvs.cgi/?root=Apache-SVNview=revrev=1241230 Files : * /hadoop/common/branches/branch-0.23/hadoop-mapreduce-project/CHANGES.txt * /hadoop/common/branches/branch-0.23/hadoop-mapreduce-project/hadoop-yarn/hadoop-yarn-common/src/main/java/org/apache/hadoop/yarn/webapp/Dispatcher.java * /hadoop/common/branches/branch-0.23/hadoop-mapreduce-project/hadoop-yarn/hadoop-yarn-common/src/main/java/org/apache/hadoop/yarn/webapp/hamlet/HamletImpl.java yarn webapp interface vulnerable to cross scripting attacks --- Key: MAPREDUCE-3804 URL: https://issues.apache.org/jira/browse/MAPREDUCE-3804 Project: Hadoop Map/Reduce Issue Type: Bug Components: jobhistoryserver, mrv2, resourcemanager Affects Versions: 0.23.0 Reporter: Dave Thompson Assignee: Dave Thompson Fix For: 0.23.1 Attachments: MAPREDUCE-3804.patch, MAPREDUCE-3804.patch, MAPREDUCE_3804_br_0.23.0.patch Yarn webapp interface may be vulnerable to certain cross scripting attacks, injected through URL request. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (MAPREDUCE-3814) MR1 compile fails
[ https://issues.apache.org/jira/browse/MAPREDUCE-3814?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13202323#comment-13202323 ] Hudson commented on MAPREDUCE-3814: --- Integrated in Hadoop-Hdfs-0.23-Build #162 (See [https://builds.apache.org/job/Hadoop-Hdfs-0.23-Build/162/]) MAPREDUCE-3814. Fixed MRV1 compilation. (Arun C Murthy via vinodkv) svn merge --ignore-ancestry -c 1241177 ../../trunk/ vinodkv : http://svn.apache.org/viewcvs.cgi/?root=Apache-SVNview=revrev=1241181 Files : * /hadoop/common/branches/branch-0.23/hadoop-mapreduce-project/CHANGES.txt * /hadoop/common/branches/branch-0.23/hadoop-mapreduce-project/build.xml MR1 compile fails - Key: MAPREDUCE-3814 URL: https://issues.apache.org/jira/browse/MAPREDUCE-3814 Project: Hadoop Map/Reduce Issue Type: Bug Components: mrv1, mrv2 Affects Versions: 0.23.1 Reporter: Arun C Murthy Assignee: Arun C Murthy Fix For: 0.23.1 Attachments: MAPREDUCE-3814.patch $ ant veryclean all-jars -Dversion=0.23.1 -Dresolvers=internal BUILD FAILED /grid/0/dev/acm/hadoop-0.23/hadoop-mapreduce-project/build.xml:537: srcdir /grid/0/dev/acm/hadoop-0.23/hadoop-mapreduce-project/src/test/mapred/testjar does not exist! -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (MAPREDUCE-3817) bin/mapred command cannot run distcp and archive jobs
[ https://issues.apache.org/jira/browse/MAPREDUCE-3817?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13202327#comment-13202327 ] Hudson commented on MAPREDUCE-3817: --- Integrated in Hadoop-Hdfs-0.23-Build #162 (See [https://builds.apache.org/job/Hadoop-Hdfs-0.23-Build/162/]) Merge -c 1241323 from trunk to branch-0.23 to fix MAPREDUCE-3817. Fixed bin/mapred to allow running of distcp and archive jobs. Contributed by Arpit Gupta. acmurthy : http://svn.apache.org/viewcvs.cgi/?root=Apache-SVNview=revrev=1241324 Files : * /hadoop/common/branches/branch-0.23/hadoop-mapreduce-project/CHANGES.txt * /hadoop/common/branches/branch-0.23/hadoop-mapreduce-project/bin/mapred bin/mapred command cannot run distcp and archive jobs - Key: MAPREDUCE-3817 URL: https://issues.apache.org/jira/browse/MAPREDUCE-3817 Project: Hadoop Map/Reduce Issue Type: Bug Components: mrv2 Affects Versions: 0.23.1, 0.24.0 Reporter: Arpit Gupta Assignee: Arpit Gupta Fix For: 0.23.1 Attachments: MAPREDUCE-3817.patch, MAPREDUCE-3817.patch -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (MAPREDUCE-3809) Tasks may take upto 3 seconds to exit after completion
[ https://issues.apache.org/jira/browse/MAPREDUCE-3809?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13202328#comment-13202328 ] Hudson commented on MAPREDUCE-3809: --- Integrated in Hadoop-Hdfs-0.23-Build #162 (See [https://builds.apache.org/job/Hadoop-Hdfs-0.23-Build/162/]) Merge -c 1241282 from trunk to branch-0.23 to fix MAPREDUCE-3809. Ensure that there is no needless sleep in Task at the end of the task. Contributed by Siddharth Seth. acmurthy : http://svn.apache.org/viewcvs.cgi/?root=Apache-SVNview=revrev=1241283 Files : * /hadoop/common/branches/branch-0.23/hadoop-mapreduce-project/CHANGES.txt * /hadoop/common/branches/branch-0.23/hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-core/src/main/java/org/apache/hadoop/mapred/Task.java Tasks may take upto 3 seconds to exit after completion -- Key: MAPREDUCE-3809 URL: https://issues.apache.org/jira/browse/MAPREDUCE-3809 Project: Hadoop Map/Reduce Issue Type: Sub-task Components: mrv2 Affects Versions: 0.23.1, 1.0.0 Reporter: Siddharth Seth Assignee: Siddharth Seth Priority: Blocker Fix For: 0.23.1 Attachments: MR3809.txt, MR3809.txt Task.TaskReporter.stopCommunicationThread can end up waiting for a thread.sleep(3000) before stopping the thread. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (MAPREDUCE-3808) NPE in FileOutputCommitter when running a 0 reduce job
[
https://issues.apache.org/jira/browse/MAPREDUCE-3808?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13202326#comment-13202326
]
Hudson commented on MAPREDUCE-3808:
---
Integrated in Hadoop-Hdfs-0.23-Build #162 (See
[https://builds.apache.org/job/Hadoop-Hdfs-0.23-Build/162/])
MAPREDUCE-3808. Fixed an NPE in FileOutputCommitter for jobs with maps but
no reduces. Contributed by Robert Joseph Evans.
svn merge --ignore-ancestry -c 1241217 ../../trunk/
vinodkv :
http://svn.apache.org/viewcvs.cgi/?root=Apache-SVNview=revrev=1241218
Files :
* /hadoop/common/branches/branch-0.23/hadoop-mapreduce-project/CHANGES.txt
*
/hadoop/common/branches/branch-0.23/hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-core/src/main/java/org/apache/hadoop/mapred/FileOutputCommitter.java
*
/hadoop/common/branches/branch-0.23/hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-core/src/main/java/org/apache/hadoop/mapreduce/lib/output/FileOutputCommitter.java
*
/hadoop/common/branches/branch-0.23/hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-core/src/test/java/org/apache/hadoop/mapred/TestFileOutputCommitter.java
NPE in FileOutputCommitter when running a 0 reduce job
--
Key: MAPREDUCE-3808
URL: https://issues.apache.org/jira/browse/MAPREDUCE-3808
Project: Hadoop Map/Reduce
Issue Type: Bug
Components: mrv2
Affects Versions: 0.23.0, 0.24.0
Reporter: Siddharth Seth
Assignee: Robert Joseph Evans
Priority: Blocker
Fix For: 0.23.1
Attachments: MR-3808.txt
This was while running LoadGen.
{noformat}
Error: java.lang.NullPointerException at
org.apache.hadoop.fs.Path.init(Path.java:67)
at org.apache.hadoop.fs.Path.init(Path.java:56)
at
org.apache.hadoop.mapreduce.lib.output.FileOutputCommitter.getPendingJobAttemptsPath(FileOutputCommitter.java:118)
at
org.apache.hadoop.mapreduce.lib.output.FileOutputCommitter.getJobAttemptPath(FileOutputCommitter.java:167)
at
org.apache.hadoop.mapreduce.lib.output.FileOutputCommitter.getJobAttemptPath(FileOutputCommitter.java:149)
at
org.apache.hadoop.mapreduce.lib.output.FileOutputCommitter.getPendingTaskAttemptsPath(FileOutputCommitter.java:185)
at
org.apache.hadoop.mapreduce.lib.output.FileOutputCommitter.getTaskAttemptPath(FileOutputCommitter.java:209)
at
org.apache.hadoop.mapred.FileOutputCommitter.getTaskAttemptPath(FileOutputCommitter.java:100)
at
org.apache.hadoop.mapred.FileOutputCommitter.getTaskAttemptPath(FileOutputCommitter.java:94)
at
org.apache.hadoop.mapred.FileOutputCommitter.needsTaskCommit(FileOutputCommitter.java:176)
at
org.apache.hadoop.mapred.OutputCommitter.needsTaskCommit(OutputCommitter.java:248)
at org.apache.hadoop.mapred.Task.isCommitRequired(Task.java:955)
at org.apache.hadoop.mapred.Task.done(Task.java:912)
at org.apache.hadoop.mapred.MapTask.run(MapTask.java:331)
at org.apache.hadoop.mapred.YarnChild$2.run(YarnChild.java:147)
at java.security.AccessController.doPrivileged(Native Method)
at javax.security.auth.Subject.doAs(Subject.java:396)
at
org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1157)
at org.apache.hadoop.mapred.YarnChild.main(YarnChild.java:142)
{noformat}
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators:
https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (MAPREDUCE-901) Move Framework Counters into a TaskMetric structure
[ https://issues.apache.org/jira/browse/MAPREDUCE-901?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13202325#comment-13202325 ] Hudson commented on MAPREDUCE-901: -- Integrated in Hadoop-Hdfs-0.23-Build #162 (See [https://builds.apache.org/job/Hadoop-Hdfs-0.23-Build/162/]) Merge -c 1241319 from trunk to branch-0.23 to fix MAPREDUCE-3697. Support binary compatibility for Counters after MAPREDUCE-901. acmurthy : http://svn.apache.org/viewcvs.cgi/?root=Apache-SVNview=revrev=1241321 Files : * /hadoop/common/branches/branch-0.23/hadoop-mapreduce-project/CHANGES.txt * /hadoop/common/branches/branch-0.23/hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-core/src/main/java/org/apache/hadoop/mapred/Counters.java * /hadoop/common/branches/branch-0.23/hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-core/src/main/java/org/apache/hadoop/mapreduce/Counter.java * /hadoop/common/branches/branch-0.23/hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-core/src/main/java/org/apache/hadoop/mapreduce/Counters.java * /hadoop/common/branches/branch-0.23/hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-core/src/main/java/org/apache/hadoop/mapreduce/counters/AbstractCounters.java * /hadoop/common/branches/branch-0.23/hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-core/src/main/java/org/apache/hadoop/mapreduce/counters/CounterGroupBase.java * /hadoop/common/branches/branch-0.23/hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-core/src/main/java/org/apache/hadoop/mapreduce/counters/FileSystemCounterGroup.java * /hadoop/common/branches/branch-0.23/hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-core/src/main/java/org/apache/hadoop/mapreduce/counters/FrameworkCounterGroup.java * /hadoop/common/branches/branch-0.23/hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-core/src/main/java/org/apache/hadoop/mapreduce/counters/GenericCounter.java * /hadoop/common/branches/branch-0.23/hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-core/src/test/java/org/apache/hadoop/mapred/TestCounters.java * /hadoop/common/branches/branch-0.23/hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-jobclient/src/test/java/org/apache/hadoop/mapred/TestJobCounters.java Move Framework Counters into a TaskMetric structure --- Key: MAPREDUCE-901 URL: https://issues.apache.org/jira/browse/MAPREDUCE-901 Project: Hadoop Map/Reduce Issue Type: Improvement Components: task Affects Versions: 0.21.0 Reporter: Owen O'Malley Assignee: Luke Lu Fix For: 0.23.0 Attachments: 901_1.patch, 901_1.patch, FrameworkCounterGroup.java, MAPREDUCE-901.patch, MAPREDUCE-901.patch, MAPREDUCE-901.patch, MAPREDUCE-901.patch_2, mr-901-trunk-v1.patch I think we should move all of the Counters that the framework updates into a single class called TaskMetrics. TaskMetrics would have specific fields for each of the metrics like input records, input bytes, output records, etc. It would both reduce the serialized size of the heartbeats (by shrinking the Counters down to just the user's counters) and decrease the latency for updates to the JobTracker (since Counters are sent at most 1/minute instead of 1/heartbeat). -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (MAPREDUCE-3709) TestDistributedShell is failing
[ https://issues.apache.org/jira/browse/MAPREDUCE-3709?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13202346#comment-13202346 ] Hudson commented on MAPREDUCE-3709: --- Integrated in Hadoop-Mapreduce-0.23-Build #184 (See [https://builds.apache.org/job/Hadoop-Mapreduce-0.23-Build/184/]) MAPREDUCE-3709. TestDistributedShell is failing. (Hitesh Shah via mahadev) - Merging r1241325 from trunk. mahadev : http://svn.apache.org/viewcvs.cgi/?root=Apache-SVNview=revrev=1241327 Files : * /hadoop/common/branches/branch-0.23/hadoop-mapreduce-project/CHANGES.txt * /hadoop/common/branches/branch-0.23/hadoop-mapreduce-project/hadoop-yarn/hadoop-yarn-applications/hadoop-yarn-applications-distributedshell/pom.xml * /hadoop/common/branches/branch-0.23/hadoop-mapreduce-project/hadoop-yarn/hadoop-yarn-applications/hadoop-yarn-applications-distributedshell/src/main/java/org/apache/hadoop/yarn/applications/distributedshell/Client.java * /hadoop/common/branches/branch-0.23/hadoop-mapreduce-project/hadoop-yarn/hadoop-yarn-applications/hadoop-yarn-applications-distributedshell/src/test/java/org/apache/hadoop/yarn/applications/distributedshell/TestDistributedShell.java TestDistributedShell is failing --- Key: MAPREDUCE-3709 URL: https://issues.apache.org/jira/browse/MAPREDUCE-3709 Project: Hadoop Map/Reduce Issue Type: Bug Components: mrv2, test Affects Versions: 0.23.1 Reporter: Eli Collins Assignee: Hitesh Shah Fix For: 0.23.1 Attachments: MR-3709.1.patch, MR-3709.2.patch TestDistributedShell#testDSShell is failing the assert on line 90 on branch-23. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (MAPREDUCE-3810) MR AM's ContainerAllocator is assigning the allocated containers very slowly
[ https://issues.apache.org/jira/browse/MAPREDUCE-3810?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13202348#comment-13202348 ] Hudson commented on MAPREDUCE-3810: --- Integrated in Hadoop-Mapreduce-0.23-Build #184 (See [https://builds.apache.org/job/Hadoop-Mapreduce-0.23-Build/184/]) Merge -c 1241205 from trunk to branch-0.23 to fix MAPREDUCE-3810. Performance tweaks - reduced logging in AM and defined hascode/equals for ResourceRequest Priority. acmurthy : http://svn.apache.org/viewcvs.cgi/?root=Apache-SVNview=revrev=1241208 Files : * /hadoop/common/branches/branch-0.23/hadoop-mapreduce-project/CHANGES.txt * /hadoop/common/branches/branch-0.23/hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-app/src/main/java/org/apache/hadoop/mapreduce/v2/app/job/impl/TaskAttemptImpl.java * /hadoop/common/branches/branch-0.23/hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-app/src/main/java/org/apache/hadoop/mapreduce/v2/app/job/impl/TaskImpl.java * /hadoop/common/branches/branch-0.23/hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-app/src/main/java/org/apache/hadoop/mapreduce/v2/app/rm/RMCommunicator.java * /hadoop/common/branches/branch-0.23/hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-app/src/main/java/org/apache/hadoop/mapreduce/v2/app/rm/RMContainerAllocator.java * /hadoop/common/branches/branch-0.23/hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-app/src/main/java/org/apache/hadoop/mapreduce/v2/app/rm/RMContainerRequestor.java * /hadoop/common/branches/branch-0.23/hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-app/src/test/java/org/apache/hadoop/mapreduce/v2/app/MRApp.java * /hadoop/common/branches/branch-0.23/hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-app/src/test/java/org/apache/hadoop/mapreduce/v2/app/MRAppBenchmark.java * /hadoop/common/branches/branch-0.23/hadoop-mapreduce-project/hadoop-yarn/hadoop-yarn-api/src/main/java/org/apache/hadoop/yarn/api/records/Priority.java * /hadoop/common/branches/branch-0.23/hadoop-mapreduce-project/hadoop-yarn/hadoop-yarn-api/src/main/java/org/apache/hadoop/yarn/api/records/ResourceRequest.java * /hadoop/common/branches/branch-0.23/hadoop-mapreduce-project/hadoop-yarn/hadoop-yarn-api/src/main/java/org/apache/hadoop/yarn/api/records/impl/pb/PriorityPBImpl.java * /hadoop/common/branches/branch-0.23/hadoop-mapreduce-project/hadoop-yarn/hadoop-yarn-api/src/main/java/org/apache/hadoop/yarn/api/records/impl/pb/ResourceRequestPBImpl.java MR AM's ContainerAllocator is assigning the allocated containers very slowly Key: MAPREDUCE-3810 URL: https://issues.apache.org/jira/browse/MAPREDUCE-3810 Project: Hadoop Map/Reduce Issue Type: Sub-task Components: mrv2, performance Affects Versions: 0.23.0 Reporter: Vinod Kumar Vavilapalli Assignee: Vinod Kumar Vavilapalli Priority: Blocker Fix For: 0.23.1 Attachments: MAPREDUCE-3810-20120205.txt This is mostly due to logging and other not-so-cheap operations we are doing as part of the AM-RM heartbeat cycle. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (MAPREDUCE-3697) Hadoop Counters API limits Oozie's working across different hadoop versions
[ https://issues.apache.org/jira/browse/MAPREDUCE-3697?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13202349#comment-13202349 ] Hudson commented on MAPREDUCE-3697: --- Integrated in Hadoop-Mapreduce-0.23-Build #184 (See [https://builds.apache.org/job/Hadoop-Mapreduce-0.23-Build/184/]) Merge -c 1241319 from trunk to branch-0.23 to fix MAPREDUCE-3697. Support binary compatibility for Counters after MAPREDUCE-901. acmurthy : http://svn.apache.org/viewcvs.cgi/?root=Apache-SVNview=revrev=1241321 Files : * /hadoop/common/branches/branch-0.23/hadoop-mapreduce-project/CHANGES.txt * /hadoop/common/branches/branch-0.23/hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-core/src/main/java/org/apache/hadoop/mapred/Counters.java * /hadoop/common/branches/branch-0.23/hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-core/src/main/java/org/apache/hadoop/mapreduce/Counter.java * /hadoop/common/branches/branch-0.23/hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-core/src/main/java/org/apache/hadoop/mapreduce/Counters.java * /hadoop/common/branches/branch-0.23/hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-core/src/main/java/org/apache/hadoop/mapreduce/counters/AbstractCounters.java * /hadoop/common/branches/branch-0.23/hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-core/src/main/java/org/apache/hadoop/mapreduce/counters/CounterGroupBase.java * /hadoop/common/branches/branch-0.23/hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-core/src/main/java/org/apache/hadoop/mapreduce/counters/FileSystemCounterGroup.java * /hadoop/common/branches/branch-0.23/hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-core/src/main/java/org/apache/hadoop/mapreduce/counters/FrameworkCounterGroup.java * /hadoop/common/branches/branch-0.23/hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-core/src/main/java/org/apache/hadoop/mapreduce/counters/GenericCounter.java * /hadoop/common/branches/branch-0.23/hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-core/src/test/java/org/apache/hadoop/mapred/TestCounters.java * /hadoop/common/branches/branch-0.23/hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-jobclient/src/test/java/org/apache/hadoop/mapred/TestJobCounters.java Hadoop Counters API limits Oozie's working across different hadoop versions --- Key: MAPREDUCE-3697 URL: https://issues.apache.org/jira/browse/MAPREDUCE-3697 Project: Hadoop Map/Reduce Issue Type: Bug Components: mrv2 Affects Versions: 0.23.1 Reporter: John George Assignee: Mahadev konar Priority: Blocker Fix For: 0.23.1 Attachments: MAPREDUCE-3697-facade.patch, MAPREDUCE-3697-facadeV2.patch, MAPREDUCE-3697-facadeV3.patch, MAPREDUCE-3697.patch, MAPREDUCE-3697.patch, MAPREDUCE-3697.patch, MAPREDUCE-3697.patch, MAPREDUCE-3697.patch, MAPREDUCE-3697.patch, apicheck-hadoop-1.0.0-0.24.0-SNAPSHOT.sh Oozie uses Hadoop Counters API, by invoking Counters.getGroup(). However, in hadoop 23, org.apache.hadoop.mapred.Counters does not implement getGroup(). Its parent class AbstractCounters implements it. This is different from hadoop20X. As a result, Oozie compiled with either hadoop version does not work with the other version. A specific scenario, Oozie compiled with .23 and run against 205, does not update job status owing to a Counters API exception. Will explicit re-compilation against the relevant hadoop jars be required each time? This will prevent launching a uniform Oozie version across different clusters. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (MAPREDUCE-3794) Support mapred.Task.Counter and mapred.JobInProgress.Counter enums for compatibility
[ https://issues.apache.org/jira/browse/MAPREDUCE-3794?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13202347#comment-13202347 ] Hudson commented on MAPREDUCE-3794: --- Integrated in Hadoop-Mapreduce-0.23-Build #184 (See [https://builds.apache.org/job/Hadoop-Mapreduce-0.23-Build/184/]) MAPREDUCE-3794. Support mapred.Task.Counter and mapred.JobInProgress.Counter enums for compatibility (Tom White via mahadev) - Merging r1241299 from trunk. mahadev : http://svn.apache.org/viewcvs.cgi/?root=Apache-SVNview=revrev=1241300 Files : * /hadoop/common/branches/branch-0.23/hadoop-mapreduce-project/CHANGES.txt * /hadoop/common/branches/branch-0.23/hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-core/src/main/java/org/apache/hadoop/mapred/JobInProgress.java * /hadoop/common/branches/branch-0.23/hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-core/src/main/java/org/apache/hadoop/mapred/Task.java * /hadoop/common/branches/branch-0.23/hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-core/src/test/java/org/apache/hadoop/mapred/TestCounters.java Support mapred.Task.Counter and mapred.JobInProgress.Counter enums for compatibility Key: MAPREDUCE-3794 URL: https://issues.apache.org/jira/browse/MAPREDUCE-3794 Project: Hadoop Map/Reduce Issue Type: Bug Components: mrv2 Affects Versions: 0.23.0 Reporter: Tom White Assignee: Tom White Fix For: 0.23.1 Attachments: MAPREDUCE-3794.patch, MAPREDUCE-3794.patch The new counters are mapreduce.TaskCounter and mapreduce.JobCounter, but we should support the old ones too since they are public in Hadoop 1.x. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (MAPREDUCE-3814) MR1 compile fails
[ https://issues.apache.org/jira/browse/MAPREDUCE-3814?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13202350#comment-13202350 ] Hudson commented on MAPREDUCE-3814: --- Integrated in Hadoop-Mapreduce-0.23-Build #184 (See [https://builds.apache.org/job/Hadoop-Mapreduce-0.23-Build/184/]) MAPREDUCE-3814. Fixed MRV1 compilation. (Arun C Murthy via vinodkv) svn merge --ignore-ancestry -c 1241177 ../../trunk/ vinodkv : http://svn.apache.org/viewcvs.cgi/?root=Apache-SVNview=revrev=1241181 Files : * /hadoop/common/branches/branch-0.23/hadoop-mapreduce-project/CHANGES.txt * /hadoop/common/branches/branch-0.23/hadoop-mapreduce-project/build.xml MR1 compile fails - Key: MAPREDUCE-3814 URL: https://issues.apache.org/jira/browse/MAPREDUCE-3814 Project: Hadoop Map/Reduce Issue Type: Bug Components: mrv1, mrv2 Affects Versions: 0.23.1 Reporter: Arun C Murthy Assignee: Arun C Murthy Fix For: 0.23.1 Attachments: MAPREDUCE-3814.patch $ ant veryclean all-jars -Dversion=0.23.1 -Dresolvers=internal BUILD FAILED /grid/0/dev/acm/hadoop-0.23/hadoop-mapreduce-project/build.xml:537: srcdir /grid/0/dev/acm/hadoop-0.23/hadoop-mapreduce-project/src/test/mapred/testjar does not exist! -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (MAPREDUCE-3808) NPE in FileOutputCommitter when running a 0 reduce job
[
https://issues.apache.org/jira/browse/MAPREDUCE-3808?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13202353#comment-13202353
]
Hudson commented on MAPREDUCE-3808:
---
Integrated in Hadoop-Mapreduce-0.23-Build #184 (See
[https://builds.apache.org/job/Hadoop-Mapreduce-0.23-Build/184/])
MAPREDUCE-3808. Fixed an NPE in FileOutputCommitter for jobs with maps but
no reduces. Contributed by Robert Joseph Evans.
svn merge --ignore-ancestry -c 1241217 ../../trunk/
vinodkv :
http://svn.apache.org/viewcvs.cgi/?root=Apache-SVNview=revrev=1241218
Files :
* /hadoop/common/branches/branch-0.23/hadoop-mapreduce-project/CHANGES.txt
*
/hadoop/common/branches/branch-0.23/hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-core/src/main/java/org/apache/hadoop/mapred/FileOutputCommitter.java
*
/hadoop/common/branches/branch-0.23/hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-core/src/main/java/org/apache/hadoop/mapreduce/lib/output/FileOutputCommitter.java
*
/hadoop/common/branches/branch-0.23/hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-core/src/test/java/org/apache/hadoop/mapred/TestFileOutputCommitter.java
NPE in FileOutputCommitter when running a 0 reduce job
--
Key: MAPREDUCE-3808
URL: https://issues.apache.org/jira/browse/MAPREDUCE-3808
Project: Hadoop Map/Reduce
Issue Type: Bug
Components: mrv2
Affects Versions: 0.23.0, 0.24.0
Reporter: Siddharth Seth
Assignee: Robert Joseph Evans
Priority: Blocker
Fix For: 0.23.1
Attachments: MR-3808.txt
This was while running LoadGen.
{noformat}
Error: java.lang.NullPointerException at
org.apache.hadoop.fs.Path.init(Path.java:67)
at org.apache.hadoop.fs.Path.init(Path.java:56)
at
org.apache.hadoop.mapreduce.lib.output.FileOutputCommitter.getPendingJobAttemptsPath(FileOutputCommitter.java:118)
at
org.apache.hadoop.mapreduce.lib.output.FileOutputCommitter.getJobAttemptPath(FileOutputCommitter.java:167)
at
org.apache.hadoop.mapreduce.lib.output.FileOutputCommitter.getJobAttemptPath(FileOutputCommitter.java:149)
at
org.apache.hadoop.mapreduce.lib.output.FileOutputCommitter.getPendingTaskAttemptsPath(FileOutputCommitter.java:185)
at
org.apache.hadoop.mapreduce.lib.output.FileOutputCommitter.getTaskAttemptPath(FileOutputCommitter.java:209)
at
org.apache.hadoop.mapred.FileOutputCommitter.getTaskAttemptPath(FileOutputCommitter.java:100)
at
org.apache.hadoop.mapred.FileOutputCommitter.getTaskAttemptPath(FileOutputCommitter.java:94)
at
org.apache.hadoop.mapred.FileOutputCommitter.needsTaskCommit(FileOutputCommitter.java:176)
at
org.apache.hadoop.mapred.OutputCommitter.needsTaskCommit(OutputCommitter.java:248)
at org.apache.hadoop.mapred.Task.isCommitRequired(Task.java:955)
at org.apache.hadoop.mapred.Task.done(Task.java:912)
at org.apache.hadoop.mapred.MapTask.run(MapTask.java:331)
at org.apache.hadoop.mapred.YarnChild$2.run(YarnChild.java:147)
at java.security.AccessController.doPrivileged(Native Method)
at javax.security.auth.Subject.doAs(Subject.java:396)
at
org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1157)
at org.apache.hadoop.mapred.YarnChild.main(YarnChild.java:142)
{noformat}
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators:
https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (MAPREDUCE-901) Move Framework Counters into a TaskMetric structure
[ https://issues.apache.org/jira/browse/MAPREDUCE-901?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13202352#comment-13202352 ] Hudson commented on MAPREDUCE-901: -- Integrated in Hadoop-Mapreduce-0.23-Build #184 (See [https://builds.apache.org/job/Hadoop-Mapreduce-0.23-Build/184/]) Merge -c 1241319 from trunk to branch-0.23 to fix MAPREDUCE-3697. Support binary compatibility for Counters after MAPREDUCE-901. acmurthy : http://svn.apache.org/viewcvs.cgi/?root=Apache-SVNview=revrev=1241321 Files : * /hadoop/common/branches/branch-0.23/hadoop-mapreduce-project/CHANGES.txt * /hadoop/common/branches/branch-0.23/hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-core/src/main/java/org/apache/hadoop/mapred/Counters.java * /hadoop/common/branches/branch-0.23/hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-core/src/main/java/org/apache/hadoop/mapreduce/Counter.java * /hadoop/common/branches/branch-0.23/hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-core/src/main/java/org/apache/hadoop/mapreduce/Counters.java * /hadoop/common/branches/branch-0.23/hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-core/src/main/java/org/apache/hadoop/mapreduce/counters/AbstractCounters.java * /hadoop/common/branches/branch-0.23/hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-core/src/main/java/org/apache/hadoop/mapreduce/counters/CounterGroupBase.java * /hadoop/common/branches/branch-0.23/hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-core/src/main/java/org/apache/hadoop/mapreduce/counters/FileSystemCounterGroup.java * /hadoop/common/branches/branch-0.23/hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-core/src/main/java/org/apache/hadoop/mapreduce/counters/FrameworkCounterGroup.java * /hadoop/common/branches/branch-0.23/hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-core/src/main/java/org/apache/hadoop/mapreduce/counters/GenericCounter.java * /hadoop/common/branches/branch-0.23/hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-core/src/test/java/org/apache/hadoop/mapred/TestCounters.java * /hadoop/common/branches/branch-0.23/hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-jobclient/src/test/java/org/apache/hadoop/mapred/TestJobCounters.java Move Framework Counters into a TaskMetric structure --- Key: MAPREDUCE-901 URL: https://issues.apache.org/jira/browse/MAPREDUCE-901 Project: Hadoop Map/Reduce Issue Type: Improvement Components: task Affects Versions: 0.21.0 Reporter: Owen O'Malley Assignee: Luke Lu Fix For: 0.23.0 Attachments: 901_1.patch, 901_1.patch, FrameworkCounterGroup.java, MAPREDUCE-901.patch, MAPREDUCE-901.patch, MAPREDUCE-901.patch, MAPREDUCE-901.patch_2, mr-901-trunk-v1.patch I think we should move all of the Counters that the framework updates into a single class called TaskMetrics. TaskMetrics would have specific fields for each of the metrics like input records, input bytes, output records, etc. It would both reduce the serialized size of the heartbeats (by shrinking the Counters down to just the user's counters) and decrease the latency for updates to the JobTracker (since Counters are sent at most 1/minute instead of 1/heartbeat). -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (MAPREDUCE-3813) RackResolver should maintain a cache to avoid repetitive lookups.
[ https://issues.apache.org/jira/browse/MAPREDUCE-3813?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13202351#comment-13202351 ] Hudson commented on MAPREDUCE-3813: --- Integrated in Hadoop-Mapreduce-0.23-Build #184 (See [https://builds.apache.org/job/Hadoop-Mapreduce-0.23-Build/184/]) Merge -c 1241209 from trunk to branch-0.23 to fix MAPREDUCE-3813. Added a cache for resolved racks. acmurthy : http://svn.apache.org/viewcvs.cgi/?root=Apache-SVNview=revrev=1241210 Files : * /hadoop/common/branches/branch-0.23/hadoop-mapreduce-project/CHANGES.txt * /hadoop/common/branches/branch-0.23/hadoop-mapreduce-project/hadoop-yarn/hadoop-yarn-common/src/main/java/org/apache/hadoop/yarn/util/RackResolver.java * /hadoop/common/branches/branch-0.23/hadoop-mapreduce-project/hadoop-yarn/hadoop-yarn-common/src/test/java/org/apache/hadoop/yarn/util/TestRackResolver.java RackResolver should maintain a cache to avoid repetitive lookups. - Key: MAPREDUCE-3813 URL: https://issues.apache.org/jira/browse/MAPREDUCE-3813 Project: Hadoop Map/Reduce Issue Type: Sub-task Components: mrv2, performance Affects Versions: 0.23.0 Reporter: Vinod Kumar Vavilapalli Assignee: Vinod Kumar Vavilapalli Fix For: 0.23.1 Attachments: MAPREDUCE-3813-20120205.txt With the current code, during task creation, we repeatedly resolve hosts and RackResolver doesn't cache any of the results. Caching will improve performance. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (MAPREDUCE-3817) bin/mapred command cannot run distcp and archive jobs
[ https://issues.apache.org/jira/browse/MAPREDUCE-3817?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13202354#comment-13202354 ] Hudson commented on MAPREDUCE-3817: --- Integrated in Hadoop-Mapreduce-0.23-Build #184 (See [https://builds.apache.org/job/Hadoop-Mapreduce-0.23-Build/184/]) Merge -c 1241323 from trunk to branch-0.23 to fix MAPREDUCE-3817. Fixed bin/mapred to allow running of distcp and archive jobs. Contributed by Arpit Gupta. acmurthy : http://svn.apache.org/viewcvs.cgi/?root=Apache-SVNview=revrev=1241324 Files : * /hadoop/common/branches/branch-0.23/hadoop-mapreduce-project/CHANGES.txt * /hadoop/common/branches/branch-0.23/hadoop-mapreduce-project/bin/mapred bin/mapred command cannot run distcp and archive jobs - Key: MAPREDUCE-3817 URL: https://issues.apache.org/jira/browse/MAPREDUCE-3817 Project: Hadoop Map/Reduce Issue Type: Bug Components: mrv2 Affects Versions: 0.23.1, 0.24.0 Reporter: Arpit Gupta Assignee: Arpit Gupta Fix For: 0.23.1 Attachments: MAPREDUCE-3817.patch, MAPREDUCE-3817.patch -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (MAPREDUCE-3809) Tasks may take upto 3 seconds to exit after completion
[ https://issues.apache.org/jira/browse/MAPREDUCE-3809?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13202355#comment-13202355 ] Hudson commented on MAPREDUCE-3809: --- Integrated in Hadoop-Mapreduce-0.23-Build #184 (See [https://builds.apache.org/job/Hadoop-Mapreduce-0.23-Build/184/]) Merge -c 1241282 from trunk to branch-0.23 to fix MAPREDUCE-3809. Ensure that there is no needless sleep in Task at the end of the task. Contributed by Siddharth Seth. acmurthy : http://svn.apache.org/viewcvs.cgi/?root=Apache-SVNview=revrev=1241283 Files : * /hadoop/common/branches/branch-0.23/hadoop-mapreduce-project/CHANGES.txt * /hadoop/common/branches/branch-0.23/hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-core/src/main/java/org/apache/hadoop/mapred/Task.java Tasks may take upto 3 seconds to exit after completion -- Key: MAPREDUCE-3809 URL: https://issues.apache.org/jira/browse/MAPREDUCE-3809 Project: Hadoop Map/Reduce Issue Type: Sub-task Components: mrv2 Affects Versions: 0.23.1, 1.0.0 Reporter: Siddharth Seth Assignee: Siddharth Seth Priority: Blocker Fix For: 0.23.1 Attachments: MR3809.txt, MR3809.txt Task.TaskReporter.stopCommunicationThread can end up waiting for a thread.sleep(3000) before stopping the thread. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (MAPREDUCE-3804) yarn webapp interface vulnerable to cross scripting attacks
[ https://issues.apache.org/jira/browse/MAPREDUCE-3804?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13202357#comment-13202357 ] Hudson commented on MAPREDUCE-3804: --- Integrated in Hadoop-Mapreduce-0.23-Build #184 (See [https://builds.apache.org/job/Hadoop-Mapreduce-0.23-Build/184/]) svn merge -c 1241225 fixes MAPREDUCE-3804 bobby : http://svn.apache.org/viewcvs.cgi/?root=Apache-SVNview=revrev=1241230 Files : * /hadoop/common/branches/branch-0.23/hadoop-mapreduce-project/CHANGES.txt * /hadoop/common/branches/branch-0.23/hadoop-mapreduce-project/hadoop-yarn/hadoop-yarn-common/src/main/java/org/apache/hadoop/yarn/webapp/Dispatcher.java * /hadoop/common/branches/branch-0.23/hadoop-mapreduce-project/hadoop-yarn/hadoop-yarn-common/src/main/java/org/apache/hadoop/yarn/webapp/hamlet/HamletImpl.java yarn webapp interface vulnerable to cross scripting attacks --- Key: MAPREDUCE-3804 URL: https://issues.apache.org/jira/browse/MAPREDUCE-3804 Project: Hadoop Map/Reduce Issue Type: Bug Components: jobhistoryserver, mrv2, resourcemanager Affects Versions: 0.23.0 Reporter: Dave Thompson Assignee: Dave Thompson Fix For: 0.23.1 Attachments: MAPREDUCE-3804.patch, MAPREDUCE-3804.patch, MAPREDUCE_3804_br_0.23.0.patch Yarn webapp interface may be vulnerable to certain cross scripting attacks, injected through URL request. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (MAPREDUCE-3810) MR AM's ContainerAllocator is assigning the allocated containers very slowly
[ https://issues.apache.org/jira/browse/MAPREDUCE-3810?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13202384#comment-13202384 ] Hudson commented on MAPREDUCE-3810: --- Integrated in Hadoop-Mapreduce-trunk #982 (See [https://builds.apache.org/job/Hadoop-Mapreduce-trunk/982/]) MAPREDUCE-3810. Performance tweaks - reduced logging in AM and defined hascode/equals for ResourceRequest Priority. Contributed by Vinod K V. acmurthy : http://svn.apache.org/viewcvs.cgi/?root=Apache-SVNview=revrev=1241205 Files : * /hadoop/common/trunk/hadoop-mapreduce-project/CHANGES.txt * /hadoop/common/trunk/hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-app/src/main/java/org/apache/hadoop/mapreduce/v2/app/job/impl/TaskAttemptImpl.java * /hadoop/common/trunk/hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-app/src/main/java/org/apache/hadoop/mapreduce/v2/app/job/impl/TaskImpl.java * /hadoop/common/trunk/hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-app/src/main/java/org/apache/hadoop/mapreduce/v2/app/rm/RMCommunicator.java * /hadoop/common/trunk/hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-app/src/main/java/org/apache/hadoop/mapreduce/v2/app/rm/RMContainerAllocator.java * /hadoop/common/trunk/hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-app/src/main/java/org/apache/hadoop/mapreduce/v2/app/rm/RMContainerRequestor.java * /hadoop/common/trunk/hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-app/src/test/java/org/apache/hadoop/mapreduce/v2/app/MRApp.java * /hadoop/common/trunk/hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-app/src/test/java/org/apache/hadoop/mapreduce/v2/app/MRAppBenchmark.java * /hadoop/common/trunk/hadoop-mapreduce-project/hadoop-yarn/hadoop-yarn-api/src/main/java/org/apache/hadoop/yarn/api/records/Priority.java * /hadoop/common/trunk/hadoop-mapreduce-project/hadoop-yarn/hadoop-yarn-api/src/main/java/org/apache/hadoop/yarn/api/records/ResourceRequest.java * /hadoop/common/trunk/hadoop-mapreduce-project/hadoop-yarn/hadoop-yarn-api/src/main/java/org/apache/hadoop/yarn/api/records/impl/pb/PriorityPBImpl.java * /hadoop/common/trunk/hadoop-mapreduce-project/hadoop-yarn/hadoop-yarn-api/src/main/java/org/apache/hadoop/yarn/api/records/impl/pb/ResourceRequestPBImpl.java MR AM's ContainerAllocator is assigning the allocated containers very slowly Key: MAPREDUCE-3810 URL: https://issues.apache.org/jira/browse/MAPREDUCE-3810 Project: Hadoop Map/Reduce Issue Type: Sub-task Components: mrv2, performance Affects Versions: 0.23.0 Reporter: Vinod Kumar Vavilapalli Assignee: Vinod Kumar Vavilapalli Priority: Blocker Fix For: 0.23.1 Attachments: MAPREDUCE-3810-20120205.txt This is mostly due to logging and other not-so-cheap operations we are doing as part of the AM-RM heartbeat cycle. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (MAPREDUCE-3709) TestDistributedShell is failing
[ https://issues.apache.org/jira/browse/MAPREDUCE-3709?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13202381#comment-13202381 ] Hudson commented on MAPREDUCE-3709: --- Integrated in Hadoop-Mapreduce-trunk #982 (See [https://builds.apache.org/job/Hadoop-Mapreduce-trunk/982/]) MAPREDUCE-3709. TestDistributedShell is failing. (Hitesh Shah via mahadev) mahadev : http://svn.apache.org/viewcvs.cgi/?root=Apache-SVNview=revrev=1241325 Files : * /hadoop/common/trunk/hadoop-mapreduce-project/CHANGES.txt * /hadoop/common/trunk/hadoop-mapreduce-project/hadoop-yarn/hadoop-yarn-applications/hadoop-yarn-applications-distributedshell/pom.xml * /hadoop/common/trunk/hadoop-mapreduce-project/hadoop-yarn/hadoop-yarn-applications/hadoop-yarn-applications-distributedshell/src/main/java/org/apache/hadoop/yarn/applications/distributedshell/Client.java * /hadoop/common/trunk/hadoop-mapreduce-project/hadoop-yarn/hadoop-yarn-applications/hadoop-yarn-applications-distributedshell/src/test/java/org/apache/hadoop/yarn/applications/distributedshell/TestDistributedShell.java TestDistributedShell is failing --- Key: MAPREDUCE-3709 URL: https://issues.apache.org/jira/browse/MAPREDUCE-3709 Project: Hadoop Map/Reduce Issue Type: Bug Components: mrv2, test Affects Versions: 0.23.1 Reporter: Eli Collins Assignee: Hitesh Shah Fix For: 0.23.1 Attachments: MR-3709.1.patch, MR-3709.2.patch TestDistributedShell#testDSShell is failing the assert on line 90 on branch-23. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (MAPREDUCE-3794) Support mapred.Task.Counter and mapred.JobInProgress.Counter enums for compatibility
[ https://issues.apache.org/jira/browse/MAPREDUCE-3794?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13202382#comment-13202382 ] Hudson commented on MAPREDUCE-3794: --- Integrated in Hadoop-Mapreduce-trunk #982 (See [https://builds.apache.org/job/Hadoop-Mapreduce-trunk/982/]) MAPREDUCE-3794. Support mapred.Task.Counter and mapred.JobInProgress.Counter enums for compatibility (Tom White via mahadev) mahadev : http://svn.apache.org/viewcvs.cgi/?root=Apache-SVNview=revrev=1241299 Files : * /hadoop/common/trunk/hadoop-mapreduce-project/CHANGES.txt * /hadoop/common/trunk/hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-core/src/main/java/org/apache/hadoop/mapred/JobInProgress.java * /hadoop/common/trunk/hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-core/src/main/java/org/apache/hadoop/mapred/Task.java * /hadoop/common/trunk/hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-core/src/test/java/org/apache/hadoop/mapred/TestCounters.java Support mapred.Task.Counter and mapred.JobInProgress.Counter enums for compatibility Key: MAPREDUCE-3794 URL: https://issues.apache.org/jira/browse/MAPREDUCE-3794 Project: Hadoop Map/Reduce Issue Type: Bug Components: mrv2 Affects Versions: 0.23.0 Reporter: Tom White Assignee: Tom White Fix For: 0.23.1 Attachments: MAPREDUCE-3794.patch, MAPREDUCE-3794.patch The new counters are mapreduce.TaskCounter and mapreduce.JobCounter, but we should support the old ones too since they are public in Hadoop 1.x. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (MAPREDUCE-3814) MR1 compile fails
[ https://issues.apache.org/jira/browse/MAPREDUCE-3814?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13202387#comment-13202387 ] Hudson commented on MAPREDUCE-3814: --- Integrated in Hadoop-Mapreduce-trunk #982 (See [https://builds.apache.org/job/Hadoop-Mapreduce-trunk/982/]) MAPREDUCE-3814. Fixed MRV1 compilation. (Arun C Murthy via vinodkv) vinodkv : http://svn.apache.org/viewcvs.cgi/?root=Apache-SVNview=revrev=1241177 Files : * /hadoop/common/trunk/hadoop-mapreduce-project/CHANGES.txt * /hadoop/common/trunk/hadoop-mapreduce-project/build.xml MR1 compile fails - Key: MAPREDUCE-3814 URL: https://issues.apache.org/jira/browse/MAPREDUCE-3814 Project: Hadoop Map/Reduce Issue Type: Bug Components: mrv1, mrv2 Affects Versions: 0.23.1 Reporter: Arun C Murthy Assignee: Arun C Murthy Fix For: 0.23.1 Attachments: MAPREDUCE-3814.patch $ ant veryclean all-jars -Dversion=0.23.1 -Dresolvers=internal BUILD FAILED /grid/0/dev/acm/hadoop-0.23/hadoop-mapreduce-project/build.xml:537: srcdir /grid/0/dev/acm/hadoop-0.23/hadoop-mapreduce-project/src/test/mapred/testjar does not exist! -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (MAPREDUCE-3813) RackResolver should maintain a cache to avoid repetitive lookups.
[ https://issues.apache.org/jira/browse/MAPREDUCE-3813?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13202388#comment-13202388 ] Hudson commented on MAPREDUCE-3813: --- Integrated in Hadoop-Mapreduce-trunk #982 (See [https://builds.apache.org/job/Hadoop-Mapreduce-trunk/982/]) MAPREDUCE-3813. Added a cache for resolved racks. Contributed by Vinod K V. acmurthy : http://svn.apache.org/viewcvs.cgi/?root=Apache-SVNview=revrev=1241209 Files : * /hadoop/common/trunk/hadoop-mapreduce-project/CHANGES.txt * /hadoop/common/trunk/hadoop-mapreduce-project/hadoop-yarn/hadoop-yarn-common/src/main/java/org/apache/hadoop/yarn/util/RackResolver.java * /hadoop/common/trunk/hadoop-mapreduce-project/hadoop-yarn/hadoop-yarn-common/src/test/java/org/apache/hadoop/yarn/util/TestRackResolver.java RackResolver should maintain a cache to avoid repetitive lookups. - Key: MAPREDUCE-3813 URL: https://issues.apache.org/jira/browse/MAPREDUCE-3813 Project: Hadoop Map/Reduce Issue Type: Sub-task Components: mrv2, performance Affects Versions: 0.23.0 Reporter: Vinod Kumar Vavilapalli Assignee: Vinod Kumar Vavilapalli Fix For: 0.23.1 Attachments: MAPREDUCE-3813-20120205.txt With the current code, during task creation, we repeatedly resolve hosts and RackResolver doesn't cache any of the results. Caching will improve performance. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (MAPREDUCE-3697) Hadoop Counters API limits Oozie's working across different hadoop versions
[ https://issues.apache.org/jira/browse/MAPREDUCE-3697?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13202385#comment-13202385 ] Hudson commented on MAPREDUCE-3697: --- Integrated in Hadoop-Mapreduce-trunk #982 (See [https://builds.apache.org/job/Hadoop-Mapreduce-trunk/982/]) MAPREDUCE-3697. Support binary compatibility for Counters after MAPREDUCE-901. acmurthy : http://svn.apache.org/viewcvs.cgi/?root=Apache-SVNview=revrev=1241319 Files : * /hadoop/common/trunk/hadoop-mapreduce-project/CHANGES.txt * /hadoop/common/trunk/hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-core/src/main/java/org/apache/hadoop/mapred/Counters.java * /hadoop/common/trunk/hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-core/src/main/java/org/apache/hadoop/mapreduce/Counter.java * /hadoop/common/trunk/hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-core/src/main/java/org/apache/hadoop/mapreduce/Counters.java * /hadoop/common/trunk/hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-core/src/main/java/org/apache/hadoop/mapreduce/counters/AbstractCounters.java * /hadoop/common/trunk/hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-core/src/main/java/org/apache/hadoop/mapreduce/counters/CounterGroupBase.java * /hadoop/common/trunk/hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-core/src/main/java/org/apache/hadoop/mapreduce/counters/FileSystemCounterGroup.java * /hadoop/common/trunk/hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-core/src/main/java/org/apache/hadoop/mapreduce/counters/FrameworkCounterGroup.java * /hadoop/common/trunk/hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-core/src/main/java/org/apache/hadoop/mapreduce/counters/GenericCounter.java * /hadoop/common/trunk/hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-core/src/test/java/org/apache/hadoop/mapred/TestCounters.java * /hadoop/common/trunk/hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-jobclient/src/test/java/org/apache/hadoop/mapred/TestJobCounters.java Hadoop Counters API limits Oozie's working across different hadoop versions --- Key: MAPREDUCE-3697 URL: https://issues.apache.org/jira/browse/MAPREDUCE-3697 Project: Hadoop Map/Reduce Issue Type: Bug Components: mrv2 Affects Versions: 0.23.1 Reporter: John George Assignee: Mahadev konar Priority: Blocker Fix For: 0.23.1 Attachments: MAPREDUCE-3697-facade.patch, MAPREDUCE-3697-facadeV2.patch, MAPREDUCE-3697-facadeV3.patch, MAPREDUCE-3697.patch, MAPREDUCE-3697.patch, MAPREDUCE-3697.patch, MAPREDUCE-3697.patch, MAPREDUCE-3697.patch, MAPREDUCE-3697.patch, apicheck-hadoop-1.0.0-0.24.0-SNAPSHOT.sh Oozie uses Hadoop Counters API, by invoking Counters.getGroup(). However, in hadoop 23, org.apache.hadoop.mapred.Counters does not implement getGroup(). Its parent class AbstractCounters implements it. This is different from hadoop20X. As a result, Oozie compiled with either hadoop version does not work with the other version. A specific scenario, Oozie compiled with .23 and run against 205, does not update job status owing to a Counters API exception. Will explicit re-compilation against the relevant hadoop jars be required each time? This will prevent launching a uniform Oozie version across different clusters. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (MAPREDUCE-3818) Trunk MRV1 compilation is broken.
[
https://issues.apache.org/jira/browse/MAPREDUCE-3818?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13202391#comment-13202391
]
Hudson commented on MAPREDUCE-3818:
---
Integrated in Hadoop-Mapreduce-trunk #982 (See
[https://builds.apache.org/job/Hadoop-Mapreduce-trunk/982/])
MAPREDUCE-3818. Fixed broken compilation in TestSubmitJob after the patch
for HDFS-2895. Contributed by Suresh Srinivas.
vinodkv :
http://svn.apache.org/viewcvs.cgi/?root=Apache-SVNview=revrev=1241253
Files :
* /hadoop/common/trunk/hadoop-mapreduce-project/CHANGES.txt
*
/hadoop/common/trunk/hadoop-mapreduce-project/src/test/mapred/org/apache/hadoop/mapred/TestSubmitJob.java
Trunk MRV1 compilation is broken.
-
Key: MAPREDUCE-3818
URL: https://issues.apache.org/jira/browse/MAPREDUCE-3818
Project: Hadoop Map/Reduce
Issue Type: Bug
Components: build, test
Affects Versions: 0.24.0
Reporter: Vinod Kumar Vavilapalli
Assignee: Suresh Srinivas
Priority: Blocker
Fix For: 0.24.0
Attachments: MAPREDUCE-3818.txt
Seeing this:
{code}
[javac]
/Users/vinodkv/Workspace/eclipse-workspace/apache-git/hadoop-common/hadoop-mapreduce-project/src/test/mapred/org/apache/hadoop/mapred/TestSubmitJob.java:155:
cannot find symbol
[javac] symbol : class ClientNamenodeWireProtocol
[javac] location: class org.apache.hadoop.mapred.TestSubmitJob
[javac] RPC.getProxy(ClientNamenodeWireProtocol.class,
[javac]^
{code}
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators:
https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (MAPREDUCE-3817) bin/mapred command cannot run distcp and archive jobs
[ https://issues.apache.org/jira/browse/MAPREDUCE-3817?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13202392#comment-13202392 ] Hudson commented on MAPREDUCE-3817: --- Integrated in Hadoop-Mapreduce-trunk #982 (See [https://builds.apache.org/job/Hadoop-Mapreduce-trunk/982/]) MAPREDUCE-3817. Fixed bin/mapred to allow running of distcp and archive jobs. Contributed by Arpit Gupta. acmurthy : http://svn.apache.org/viewcvs.cgi/?root=Apache-SVNview=revrev=1241323 Files : * /hadoop/common/trunk/hadoop-mapreduce-project/CHANGES.txt * /hadoop/common/trunk/hadoop-mapreduce-project/bin/mapred bin/mapred command cannot run distcp and archive jobs - Key: MAPREDUCE-3817 URL: https://issues.apache.org/jira/browse/MAPREDUCE-3817 Project: Hadoop Map/Reduce Issue Type: Bug Components: mrv2 Affects Versions: 0.23.1, 0.24.0 Reporter: Arpit Gupta Assignee: Arpit Gupta Fix For: 0.23.1 Attachments: MAPREDUCE-3817.patch, MAPREDUCE-3817.patch -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (MAPREDUCE-3811) Make the Client-AM IPC retry count configurable
[ https://issues.apache.org/jira/browse/MAPREDUCE-3811?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13202386#comment-13202386 ] Hudson commented on MAPREDUCE-3811: --- Integrated in Hadoop-Mapreduce-trunk #982 (See [https://builds.apache.org/job/Hadoop-Mapreduce-trunk/982/]) MAPREDUCE-3811. Made jobclient-to-AM retries configurable. Contributed by Siddharth Seth. acmurthy : http://svn.apache.org/viewcvs.cgi/?root=Apache-SVNview=revrev=1241285 Files : * /hadoop/common/trunk/hadoop-mapreduce-project/CHANGES.txt * /hadoop/common/trunk/hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-core/src/main/java/org/apache/hadoop/mapreduce/MRJobConfig.java * /hadoop/common/trunk/hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-core/src/main/resources/mapred-default.xml * /hadoop/common/trunk/hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-jobclient/src/main/java/org/apache/hadoop/mapred/ClientServiceDelegate.java Make the Client-AM IPC retry count configurable --- Key: MAPREDUCE-3811 URL: https://issues.apache.org/jira/browse/MAPREDUCE-3811 Project: Hadoop Map/Reduce Issue Type: Task Components: mrv2 Affects Versions: 0.23.0 Reporter: Siddharth Seth Assignee: Siddharth Seth Priority: Critical Fix For: 0.23.1 Attachments: MR3811.txt -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (MAPREDUCE-3808) NPE in FileOutputCommitter when running a 0 reduce job
[
https://issues.apache.org/jira/browse/MAPREDUCE-3808?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13202390#comment-13202390
]
Hudson commented on MAPREDUCE-3808:
---
Integrated in Hadoop-Mapreduce-trunk #982 (See
[https://builds.apache.org/job/Hadoop-Mapreduce-trunk/982/])
MAPREDUCE-3808. Fixed an NPE in FileOutputCommitter for jobs with maps but
no reduces. Contributed by Robert Joseph Evans.
vinodkv :
http://svn.apache.org/viewcvs.cgi/?root=Apache-SVNview=revrev=1241217
Files :
* /hadoop/common/trunk/hadoop-mapreduce-project/CHANGES.txt
*
/hadoop/common/trunk/hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-core/src/main/java/org/apache/hadoop/mapred/FileOutputCommitter.java
*
/hadoop/common/trunk/hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-core/src/main/java/org/apache/hadoop/mapreduce/lib/output/FileOutputCommitter.java
*
/hadoop/common/trunk/hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-core/src/test/java/org/apache/hadoop/mapred/TestFileOutputCommitter.java
NPE in FileOutputCommitter when running a 0 reduce job
--
Key: MAPREDUCE-3808
URL: https://issues.apache.org/jira/browse/MAPREDUCE-3808
Project: Hadoop Map/Reduce
Issue Type: Bug
Components: mrv2
Affects Versions: 0.23.0, 0.24.0
Reporter: Siddharth Seth
Assignee: Robert Joseph Evans
Priority: Blocker
Fix For: 0.23.1
Attachments: MR-3808.txt
This was while running LoadGen.
{noformat}
Error: java.lang.NullPointerException at
org.apache.hadoop.fs.Path.init(Path.java:67)
at org.apache.hadoop.fs.Path.init(Path.java:56)
at
org.apache.hadoop.mapreduce.lib.output.FileOutputCommitter.getPendingJobAttemptsPath(FileOutputCommitter.java:118)
at
org.apache.hadoop.mapreduce.lib.output.FileOutputCommitter.getJobAttemptPath(FileOutputCommitter.java:167)
at
org.apache.hadoop.mapreduce.lib.output.FileOutputCommitter.getJobAttemptPath(FileOutputCommitter.java:149)
at
org.apache.hadoop.mapreduce.lib.output.FileOutputCommitter.getPendingTaskAttemptsPath(FileOutputCommitter.java:185)
at
org.apache.hadoop.mapreduce.lib.output.FileOutputCommitter.getTaskAttemptPath(FileOutputCommitter.java:209)
at
org.apache.hadoop.mapred.FileOutputCommitter.getTaskAttemptPath(FileOutputCommitter.java:100)
at
org.apache.hadoop.mapred.FileOutputCommitter.getTaskAttemptPath(FileOutputCommitter.java:94)
at
org.apache.hadoop.mapred.FileOutputCommitter.needsTaskCommit(FileOutputCommitter.java:176)
at
org.apache.hadoop.mapred.OutputCommitter.needsTaskCommit(OutputCommitter.java:248)
at org.apache.hadoop.mapred.Task.isCommitRequired(Task.java:955)
at org.apache.hadoop.mapred.Task.done(Task.java:912)
at org.apache.hadoop.mapred.MapTask.run(MapTask.java:331)
at org.apache.hadoop.mapred.YarnChild$2.run(YarnChild.java:147)
at java.security.AccessController.doPrivileged(Native Method)
at javax.security.auth.Subject.doAs(Subject.java:396)
at
org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1157)
at org.apache.hadoop.mapred.YarnChild.main(YarnChild.java:142)
{noformat}
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators:
https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (MAPREDUCE-3809) Tasks may take upto 3 seconds to exit after completion
[ https://issues.apache.org/jira/browse/MAPREDUCE-3809?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13202393#comment-13202393 ] Hudson commented on MAPREDUCE-3809: --- Integrated in Hadoop-Mapreduce-trunk #982 (See [https://builds.apache.org/job/Hadoop-Mapreduce-trunk/982/]) MAPREDUCE-3809. Ensure that there is no needless sleep in Task at the end of the task. Contributed by Siddharth Seth. acmurthy : http://svn.apache.org/viewcvs.cgi/?root=Apache-SVNview=revrev=1241282 Files : * /hadoop/common/trunk/hadoop-mapreduce-project/CHANGES.txt * /hadoop/common/trunk/hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-core/src/main/java/org/apache/hadoop/mapred/Task.java Tasks may take upto 3 seconds to exit after completion -- Key: MAPREDUCE-3809 URL: https://issues.apache.org/jira/browse/MAPREDUCE-3809 Project: Hadoop Map/Reduce Issue Type: Sub-task Components: mrv2 Affects Versions: 0.23.1, 1.0.0 Reporter: Siddharth Seth Assignee: Siddharth Seth Priority: Blocker Fix For: 0.23.1 Attachments: MR3809.txt, MR3809.txt Task.TaskReporter.stopCommunicationThread can end up waiting for a thread.sleep(3000) before stopping the thread. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (MAPREDUCE-3354) JobHistoryServer should be started by bin/mapred and not by bin/yarn
[ https://issues.apache.org/jira/browse/MAPREDUCE-3354?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13202394#comment-13202394 ] Hudson commented on MAPREDUCE-3354: --- Integrated in Hadoop-Mapreduce-trunk #982 (See [https://builds.apache.org/job/Hadoop-Mapreduce-trunk/982/]) MAPREDUCE-3354. Changed scripts so that jobhistory server is started by bin/mapred instead of bin/yarn. Contributed by Jonathan Eagles. acmurthy : http://svn.apache.org/viewcvs.cgi/?root=Apache-SVNview=revrev=1241250 Files : * /hadoop/common/trunk/hadoop-assemblies/src/main/resources/assemblies/hadoop-mapreduce-dist.xml * /hadoop/common/trunk/hadoop-mapreduce-project/CHANGES.txt * /hadoop/common/trunk/hadoop-mapreduce-project/INSTALL * /hadoop/common/trunk/hadoop-mapreduce-project/bin/mapred * /hadoop/common/trunk/hadoop-mapreduce-project/bin/mr-jobhistory-daemon.sh * /hadoop/common/trunk/hadoop-mapreduce-project/hadoop-yarn/bin/start-yarn.sh * /hadoop/common/trunk/hadoop-mapreduce-project/hadoop-yarn/bin/stop-yarn.sh * /hadoop/common/trunk/hadoop-mapreduce-project/hadoop-yarn/bin/yarn * /hadoop/common/trunk/hadoop-mapreduce-project/hadoop-yarn/bin/yarn-daemon.sh * /hadoop/common/trunk/hadoop-mapreduce-project/hadoop-yarn/hadoop-yarn-site/src/site/apt/ClusterSetup.apt.vm JobHistoryServer should be started by bin/mapred and not by bin/yarn Key: MAPREDUCE-3354 URL: https://issues.apache.org/jira/browse/MAPREDUCE-3354 Project: Hadoop Map/Reduce Issue Type: Bug Components: jobhistoryserver, mrv2 Affects Versions: 0.23.0 Reporter: Vinod Kumar Vavilapalli Assignee: Jonathan Eagles Priority: Blocker Fix For: 0.23.1 Attachments: MAPREDUCE-3354.patch, MAPREDUCE-3354.patch, MAPREDUCE-3354.patch, MAPREDUCE-3354.patch, MAPREDUCE-3354.patch, MAPREDUCE-3354.patch, MAPREDUCE-3354.patch, MAPREDUCE-3354.patch, MAPREDUCE-3354.patch, MAPREDUCE-3354.patch, MAPREDUCE-3354.patch, MAPREDUCE-3354.patch JobHistoryServer belongs to mapreduce land. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (MAPREDUCE-901) Move Framework Counters into a TaskMetric structure
[ https://issues.apache.org/jira/browse/MAPREDUCE-901?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13202389#comment-13202389 ] Hudson commented on MAPREDUCE-901: -- Integrated in Hadoop-Mapreduce-trunk #982 (See [https://builds.apache.org/job/Hadoop-Mapreduce-trunk/982/]) MAPREDUCE-3697. Support binary compatibility for Counters after MAPREDUCE-901. acmurthy : http://svn.apache.org/viewcvs.cgi/?root=Apache-SVNview=revrev=1241319 Files : * /hadoop/common/trunk/hadoop-mapreduce-project/CHANGES.txt * /hadoop/common/trunk/hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-core/src/main/java/org/apache/hadoop/mapred/Counters.java * /hadoop/common/trunk/hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-core/src/main/java/org/apache/hadoop/mapreduce/Counter.java * /hadoop/common/trunk/hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-core/src/main/java/org/apache/hadoop/mapreduce/Counters.java * /hadoop/common/trunk/hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-core/src/main/java/org/apache/hadoop/mapreduce/counters/AbstractCounters.java * /hadoop/common/trunk/hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-core/src/main/java/org/apache/hadoop/mapreduce/counters/CounterGroupBase.java * /hadoop/common/trunk/hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-core/src/main/java/org/apache/hadoop/mapreduce/counters/FileSystemCounterGroup.java * /hadoop/common/trunk/hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-core/src/main/java/org/apache/hadoop/mapreduce/counters/FrameworkCounterGroup.java * /hadoop/common/trunk/hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-core/src/main/java/org/apache/hadoop/mapreduce/counters/GenericCounter.java * /hadoop/common/trunk/hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-core/src/test/java/org/apache/hadoop/mapred/TestCounters.java * /hadoop/common/trunk/hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-jobclient/src/test/java/org/apache/hadoop/mapred/TestJobCounters.java Move Framework Counters into a TaskMetric structure --- Key: MAPREDUCE-901 URL: https://issues.apache.org/jira/browse/MAPREDUCE-901 Project: Hadoop Map/Reduce Issue Type: Improvement Components: task Affects Versions: 0.21.0 Reporter: Owen O'Malley Assignee: Luke Lu Fix For: 0.23.0 Attachments: 901_1.patch, 901_1.patch, FrameworkCounterGroup.java, MAPREDUCE-901.patch, MAPREDUCE-901.patch, MAPREDUCE-901.patch, MAPREDUCE-901.patch_2, mr-901-trunk-v1.patch I think we should move all of the Counters that the framework updates into a single class called TaskMetrics. TaskMetrics would have specific fields for each of the metrics like input records, input bytes, output records, etc. It would both reduce the serialized size of the heartbeats (by shrinking the Counters down to just the user's counters) and decrease the latency for updates to the JobTracker (since Counters are sent at most 1/minute instead of 1/heartbeat). -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (MAPREDUCE-3804) yarn webapp interface vulnerable to cross scripting attacks
[ https://issues.apache.org/jira/browse/MAPREDUCE-3804?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13202395#comment-13202395 ] Hudson commented on MAPREDUCE-3804: --- Integrated in Hadoop-Mapreduce-trunk #982 (See [https://builds.apache.org/job/Hadoop-Mapreduce-trunk/982/]) MAPREDUCE-3804. yarn webapp interface vulnerable to cross scripting attacks (Dave Thompson via bobby) bobby : http://svn.apache.org/viewcvs.cgi/?root=Apache-SVNview=revrev=1241225 Files : * /hadoop/common/trunk/hadoop-mapreduce-project/CHANGES.txt * /hadoop/common/trunk/hadoop-mapreduce-project/hadoop-yarn/hadoop-yarn-common/src/main/java/org/apache/hadoop/yarn/webapp/Dispatcher.java * /hadoop/common/trunk/hadoop-mapreduce-project/hadoop-yarn/hadoop-yarn-common/src/main/java/org/apache/hadoop/yarn/webapp/hamlet/HamletImpl.java yarn webapp interface vulnerable to cross scripting attacks --- Key: MAPREDUCE-3804 URL: https://issues.apache.org/jira/browse/MAPREDUCE-3804 Project: Hadoop Map/Reduce Issue Type: Bug Components: jobhistoryserver, mrv2, resourcemanager Affects Versions: 0.23.0 Reporter: Dave Thompson Assignee: Dave Thompson Fix For: 0.23.1 Attachments: MAPREDUCE-3804.patch, MAPREDUCE-3804.patch, MAPREDUCE_3804_br_0.23.0.patch Yarn webapp interface may be vulnerable to certain cross scripting attacks, injected through URL request. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (MAPREDUCE-3831) RM scalability runtime is worse than 0.20.204 by 14.2%
[
https://issues.apache.org/jira/browse/MAPREDUCE-3831?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13202431#comment-13202431
]
Karam Singh commented on MAPREDUCE-3831:
Settings for For RM scalability comparison :-
GridMix settings were for both Hadoop-0.20.204 and Hadoop-0.23 :
{code}
gridmix.client.submit.threads=10
gridmix.submit.multiplier=0.01
gridmix.client.pending.queue.depth=100
gridmix.job.type=SLEEPJOB
gridmix.sleep.maptask-only=false
gridmix.sleep.max-map-time=30
gridmix.sleep.max-reduce-tim=30
gridmix.sleep.fake-locations=3
gridmix.distributed-cache-emulation.enable=falseemulation.enable
gridmix.compression-emulation.enable=false
{code}
From JobHistroy Parsing and GridMix client found that :
Runtime (seconds): 2473
GridMix Simulation Time Spent: 41mins 8sec
Workflow End: 2046 (From histroy parsing)
While looking at GridMix log and JobHistory files :
1. Found that according to gm client last was completed was
{code}
12/02/07 08:32:26 INFO gridmix.JobMonitor: GRIDMIX29
job_1328600848949_1182) success.
Whereas the
EndTime of Job is : 1328602818684 Tue, 07 Feb 2012 08:20:18
{code}
Which means somehow GridMix got Job completion event 12 minutes after
the the actual job got completed .
2. Similarly acc. to JobHistory last Jo completed was :
{code}
job_1328600848949_1162: 1328603121882 Tue, 07 Feb 2012 08:25:21
Whereas according to GridMix client log:
12/02/07 08:32:08 INFO gridmix.JobMonitor: GRIDMIX29
(job_1328600848949_1162) success
{code}
Which again means GridMix got job completion event nearly 7 minutes
after the actual job got finished.
Whereas this problem does not exists with Hadoop-0.20.204
Seems that, Somehow in Hadoop-0.23, GridMix is getting job completion events
long after the actual job getting completed
RM scalability runtime is worse than 0.20.204 by 14.2%
--
Key: MAPREDUCE-3831
URL: https://issues.apache.org/jira/browse/MAPREDUCE-3831
Project: Hadoop Map/Reduce
Issue Type: Task
Components: benchmarks, resourcemanager
Affects Versions: 0.23.1
Reporter: Vinay Kumar Thota
RM scalability runtime is worse than 0.20.204 by 14.2%
Overall runtime against Fred in a 350 nodes cluster is 2155 secs.
Overall runtime against .23 in 350 nodes cluster is 2462 secs.
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators:
https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Updated] (MAPREDUCE-3502) Review all Service.stop() operations and make sure that they work before a service is started
[ https://issues.apache.org/jira/browse/MAPREDUCE-3502?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Steve Loughran updated MAPREDUCE-3502: -- Target Version/s: 0.23.1, 0.24.0 (was: 0.24.0, 0.23.1) Status: Open (was: Patch Available) Review all Service.stop() operations and make sure that they work before a service is started - Key: MAPREDUCE-3502 URL: https://issues.apache.org/jira/browse/MAPREDUCE-3502 Project: Hadoop Map/Reduce Issue Type: Task Components: mrv2 Affects Versions: 0.23.0, 0.24.0 Reporter: Steve Loughran Assignee: Steve Loughran Attachments: MAPREDUCE-3502.patch, MAPREDUCE-3502.patch Original Estimate: 24h Time Spent: 2.5h Remaining Estimate: 21.5h MAPREDUCE-3431 has shown that some of the key services's shutdown operations are not robust against being invoked before the service is started. They need to be by # not calling other things if the other things are null # not being re-entrant (i.e. make synchronized if possible), Maybe # have a StopService operation that only stops a service if it is live # factor out the is-running test from the base service class and make it a pre-check for all the child services, so they bail out sooner rather than later. This would be the best as it would be the one guaranteed to work consistently across all instances, so only one or two would need testing my first iteration will skip the sync though it's something to consider. Testing: try to create each instance; call stop() straight after construction. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Work started] (MAPREDUCE-3502) Review all Service.stop() operations and make sure that they work before a service is started
[ https://issues.apache.org/jira/browse/MAPREDUCE-3502?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Work on MAPREDUCE-3502 started by Steve Loughran. Review all Service.stop() operations and make sure that they work before a service is started - Key: MAPREDUCE-3502 URL: https://issues.apache.org/jira/browse/MAPREDUCE-3502 Project: Hadoop Map/Reduce Issue Type: Task Components: mrv2 Affects Versions: 0.23.0, 0.24.0 Reporter: Steve Loughran Assignee: Steve Loughran Attachments: MAPREDUCE-3502.patch, MAPREDUCE-3502.patch Original Estimate: 24h Time Spent: 2.5h Remaining Estimate: 21.5h MAPREDUCE-3431 has shown that some of the key services's shutdown operations are not robust against being invoked before the service is started. They need to be by # not calling other things if the other things are null # not being re-entrant (i.e. make synchronized if possible), Maybe # have a StopService operation that only stops a service if it is live # factor out the is-running test from the base service class and make it a pre-check for all the child services, so they bail out sooner rather than later. This would be the best as it would be the one guaranteed to work consistently across all instances, so only one or two would need testing my first iteration will skip the sync though it's something to consider. Testing: try to create each instance; call stop() straight after construction. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (MAPREDUCE-3502) Review all Service.stop() operations and make sure that they work before a service is started
[ https://issues.apache.org/jira/browse/MAPREDUCE-3502?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13202452#comment-13202452 ] Steve Loughran commented on MAPREDUCE-3502: --- Plan #Â submit Service w/ better shutdown (and helper) methods first # submit patches to specific sections of the yarn codebase Review all Service.stop() operations and make sure that they work before a service is started - Key: MAPREDUCE-3502 URL: https://issues.apache.org/jira/browse/MAPREDUCE-3502 Project: Hadoop Map/Reduce Issue Type: Task Components: mrv2 Affects Versions: 0.23.0, 0.24.0 Reporter: Steve Loughran Assignee: Steve Loughran Attachments: MAPREDUCE-3502.patch, MAPREDUCE-3502.patch Original Estimate: 24h Time Spent: 2.5h Remaining Estimate: 21.5h MAPREDUCE-3431 has shown that some of the key services's shutdown operations are not robust against being invoked before the service is started. They need to be by # not calling other things if the other things are null # not being re-entrant (i.e. make synchronized if possible), Maybe # have a StopService operation that only stops a service if it is live # factor out the is-running test from the base service class and make it a pre-check for all the child services, so they bail out sooner rather than later. This would be the best as it would be the one guaranteed to work consistently across all instances, so only one or two would need testing my first iteration will skip the sync though it's something to consider. Testing: try to create each instance; call stop() straight after construction. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (MAPREDUCE-3824) Distributed caches are not removed properly
[ https://issues.apache.org/jira/browse/MAPREDUCE-3824?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13202496#comment-13202496 ] Robert Joseph Evans commented on MAPREDUCE-3824: I like the concept of the patch. Volatile is definitely needed here, my bad on that one. I also like that you are doing a DU to update the size of the cached objects if they are 0. I do have some issues with the patch though. The first is that even though the DU size update is being done on a separate thread it is being done with the cachedArchives lock held. The amount of time it takes to do a DU could be significant. Nothing new can be added to the cache while the cachedArchives lock is held, so it could be blocking other new tasks from making progress. I would really prefer to see this done in two passes, similar to how we delete out entries. The first pass would go through all entries and identify those that need to be updated, the second pass would be to update those entries without the lock held. Then once we have all of the entries updated we can look at cleaning up the distributed cache. The second is that we are updating the size too late. We decide how much space needs to be deleted to get us back under the desired amount based totally on the size reported by BaseDirManager, which in turn gets its data from the CacheStatus object. The issue is that in the current patch we first calculate how much needs to be removed, then we update the size of the archives, then we delete them. This is not that critical, because it just means that in the next pass they would be deleted, so this is really very minor, but should be covered by doing the update in two passes. I am not sure exactly what are the situations that the size is not being set. I would like to know exactly which situations the current code is missing, because like I said previously the code that computes the used size goes completely off of what is reported to BaseDirManager, unfortunately there are some issues with BaseDirManger where if we are too aggressive with setting the size we might double count some archives, which eventually would make it so that the BaseDirManager thinks it is full all the time, which would be very bad. Distributed caches are not removed properly --- Key: MAPREDUCE-3824 URL: https://issues.apache.org/jira/browse/MAPREDUCE-3824 Project: Hadoop Map/Reduce Issue Type: Bug Components: distributed-cache Affects Versions: 1.0.0 Reporter: Allen Wittenauer Priority: Critical Attachments: MAPREDUCE-3824-branch-1.0.txt Distributed caches are not being properly removed by the TaskTracker when they are expected to be expired. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (MAPREDUCE-3697) Hadoop Counters API limits Oozie's working across different hadoop versions
[ https://issues.apache.org/jira/browse/MAPREDUCE-3697?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13202543#comment-13202543 ] Alejandro Abdelnur commented on MAPREDUCE-3697: --- Arun was too quick for me to review the patch :) It looks good, only thing is that the getUnderlaying*() methods for Counter Group should be annotated as project private. Hadoop Counters API limits Oozie's working across different hadoop versions --- Key: MAPREDUCE-3697 URL: https://issues.apache.org/jira/browse/MAPREDUCE-3697 Project: Hadoop Map/Reduce Issue Type: Bug Components: mrv2 Affects Versions: 0.23.1 Reporter: John George Assignee: Mahadev konar Priority: Blocker Fix For: 0.23.1 Attachments: MAPREDUCE-3697-facade.patch, MAPREDUCE-3697-facadeV2.patch, MAPREDUCE-3697-facadeV3.patch, MAPREDUCE-3697.patch, MAPREDUCE-3697.patch, MAPREDUCE-3697.patch, MAPREDUCE-3697.patch, MAPREDUCE-3697.patch, MAPREDUCE-3697.patch, apicheck-hadoop-1.0.0-0.24.0-SNAPSHOT.sh Oozie uses Hadoop Counters API, by invoking Counters.getGroup(). However, in hadoop 23, org.apache.hadoop.mapred.Counters does not implement getGroup(). Its parent class AbstractCounters implements it. This is different from hadoop20X. As a result, Oozie compiled with either hadoop version does not work with the other version. A specific scenario, Oozie compiled with .23 and run against 205, does not update job status owing to a Counters API exception. Will explicit re-compilation against the relevant hadoop jars be required each time? This will prevent launching a uniform Oozie version across different clusters. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Created] (MAPREDUCE-3832) mapred Counters API should not extend mapreduce Counters API
mapred Counters API should not extend mapreduce Counters API Key: MAPREDUCE-3832 URL: https://issues.apache.org/jira/browse/MAPREDUCE-3832 Project: Hadoop Map/Reduce Issue Type: Bug Components: mrv2 Affects Versions: 0.23.1 Reporter: Alejandro Abdelnur Priority: Critical The mapred (old) Counters API extends the mapreduce (new) Counters API, this effectively makes the old Counters API dependent on the new Counters API. This may affect stability and backwards/forward compatibility in the old Counters API. The new API should not pop up as dependency in the old API. Instead it should only be used in the implementation when necessary. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (MAPREDUCE-3832) mapred Counters API should not extend mapreduce Counters API
[ https://issues.apache.org/jira/browse/MAPREDUCE-3832?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13202571#comment-13202571 ] Mahadev konar commented on MAPREDUCE-3832: -- @Alejandro, I am not sure I get the issue here. How does it affect backwards/forward compatibility? mapred Counters API should not extend mapreduce Counters API Key: MAPREDUCE-3832 URL: https://issues.apache.org/jira/browse/MAPREDUCE-3832 Project: Hadoop Map/Reduce Issue Type: Bug Components: mrv2 Affects Versions: 0.23.1 Reporter: Alejandro Abdelnur Priority: Critical The mapred (old) Counters API extends the mapreduce (new) Counters API, this effectively makes the old Counters API dependent on the new Counters API. This may affect stability and backwards/forward compatibility in the old Counters API. The new API should not pop up as dependency in the old API. Instead it should only be used in the implementation when necessary. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Updated] (MAPREDUCE-3831) RM scalability runtime is worse than 0.20.204 by 14.2%
[ https://issues.apache.org/jira/browse/MAPREDUCE-3831?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Mahadev konar updated MAPREDUCE-3831: - Component/s: mrv2 RM scalability runtime is worse than 0.20.204 by 14.2% -- Key: MAPREDUCE-3831 URL: https://issues.apache.org/jira/browse/MAPREDUCE-3831 Project: Hadoop Map/Reduce Issue Type: Task Components: benchmarks, mrv2, resourcemanager Affects Versions: 0.23.1 Reporter: Vinay Kumar Thota RM scalability runtime is worse than 0.20.204 by 14.2% Overall runtime against Fred in a 350 nodes cluster is 2155 secs. Overall runtime against .23 in 350 nodes cluster is 2462 secs. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (MAPREDUCE-3802) If an MR AM dies twice it looks like the process freezes
[
https://issues.apache.org/jira/browse/MAPREDUCE-3802?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13202585#comment-13202585
]
Robert Joseph Evans commented on MAPREDUCE-3802:
OK I found the issue, sort of, and it has nothing to do with order. The issue
is with the name of the task attempt. If the task was completed by the first
AM and recovered by the second AM the name of the task attempt in the jhist
file will look like attempt_1328637230353_0001_m_00_0, but the Recovery
Service is trying to recover a task with attempt id
attempt_1328637230353_0001_m_00_1000, which appears to be the format for
attempts that completed successfully with the second AM.
I need to understand a little bit better how these names are determined, and
where they are set, so I can determine how to fix the issue. I don't see how
this could only be a problem for a single node cluster.
If an MR AM dies twice it looks like the process freezes
-
Key: MAPREDUCE-3802
URL: https://issues.apache.org/jira/browse/MAPREDUCE-3802
Project: Hadoop Map/Reduce
Issue Type: Sub-task
Components: applicationmaster, mrv2
Affects Versions: 0.23.1, 0.24.0
Reporter: Robert Joseph Evans
Assignee: Robert Joseph Evans
Priority: Critical
Attachments: syslog
It looks like recovering from an RM AM dieing works very well on a single
failure. But if it fails multiple times we appear to get into a live lock
situation.
{noformat}
yarn jar
hadoop/share/hadoop/mapreduce/hadoop-mapreduce-examples-*-SNAPSHOT.jar
wordcount -Dyarn.app.mapreduce.am.log.level=DEBUG -Dmapreduce.job.reduces=30
input output
12/02/03 21:06:57 WARN conf.Configuration: fs.default.name is deprecated.
Instead, use fs.defaultFS
12/02/03 21:06:57 WARN conf.Configuration: mapred.used.genericoptionsparser
is deprecated. Instead, use mapreduce.client.genericoptionsparser.used
12/02/03 21:06:57 INFO input.FileInputFormat: Total input paths to process :
17
12/02/03 21:06:57 INFO util.NativeCodeLoader: Loaded the native-hadoop library
12/02/03 21:06:57 WARN snappy.LoadSnappy: Snappy native library not loaded
12/02/03 21:06:57 INFO mapreduce.JobSubmitter: number of splits:17
12/02/03 21:06:57 INFO mapred.ResourceMgrDelegate: Submitted application
application_1328302034486_0003 to ResourceManager at HOST/IP:8040
12/02/03 21:06:57 INFO mapreduce.Job: The url to track the job:
http://HOST:8088/proxy/application_1328302034486_0003/
12/02/03 21:06:57 INFO mapreduce.Job: Running job: job_1328302034486_0003
12/02/03 21:07:03 INFO mapreduce.Job: Job job_1328302034486_0003 running in
uber mode : false
12/02/03 21:07:03 INFO mapreduce.Job: map 0% reduce 0%
12/02/03 21:07:09 INFO mapreduce.Job: map 5% reduce 0%
12/02/03 21:07:10 INFO mapreduce.Job: map 17% reduce 0%
#KILLED AM with kill -9 here
12/02/03 21:07:16 INFO mapreduce.Job: map 29% reduce 0%
12/02/03 21:07:17 INFO mapreduce.Job: map 35% reduce 0%
12/02/03 21:07:30 INFO mapreduce.Job: map 52% reduce 0%
12/02/03 21:07:35 INFO mapreduce.Job: map 58% reduce 0%
12/02/03 21:07:37 INFO mapreduce.Job: map 70% reduce 0%
12/02/03 21:07:41 INFO mapreduce.Job: map 76% reduce 0%
12/02/03 21:07:43 INFO mapreduce.Job: map 82% reduce 0%
12/02/03 21:07:44 INFO mapreduce.Job: map 88% reduce 0%
12/02/03 21:07:47 INFO mapreduce.Job: map 94% reduce 0%
12/02/03 21:07:49 INFO mapreduce.Job: map 100% reduce 0%
12/02/03 21:07:53 INFO mapreduce.Job: map 100% reduce 3%
12/02/03 21:08:00 INFO mapreduce.Job: map 100% reduce 6%
12/02/03 21:08:06 INFO mapreduce.Job: map 100% reduce 10%
12/02/03 21:08:12 INFO mapreduce.Job: map 100% reduce 13%
12/02/03 21:08:18 INFO mapreduce.Job: map 100% reduce 16%
#killed AM with kill -9 here
12/02/03 21:08:20 INFO ipc.Client: Retrying connect to server: HOST/IP:44223.
Already tried 0 time(s).
12/02/03 21:08:21 INFO ipc.Client: Retrying connect to server: HOST/IP:44223.
Already tried 1 time(s).
12/02/03 21:08:22 INFO ipc.Client: Retrying connect to server: HOST/IP:44223.
Already tried 2 time(s).
12/02/03 21:08:26 INFO mapreduce.Job: map 64% reduce 16%
#It never makes any more progress...
{noformat}
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators:
https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (MAPREDUCE-3824) Distributed caches are not removed properly
[ https://issues.apache.org/jira/browse/MAPREDUCE-3824?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13202612#comment-13202612 ] Allen Wittenauer commented on MAPREDUCE-3824: - There is no doubt the patch is a hack, but it solved my immediate problems because as it stands, distributed caches are really broken at scale. Some background. I have a team of users that have several 36GB distributed caches. When these caches are in play, most of the system is basically locked while these caches get built. This patch was really geared towards making sure that these massive caches at least get deleted. Without these patches in place, the mapred tmp spaces fill and tasks fail, eventually leading to mapred framework collapse. There are a lot of other problems that show up with caches this large: * Hadoop doesn't have a size limit check on caches as part of the job submission process [So any hand waving about don't use caches that big! are null and void since there is no way to actually stop a user from doing that!] * the setup and cleanup tasks also trigger cache downloads. * tasktrackers appear to be frozen for *all* tasks during cache downloads, with the task stuck in the extremely unhelpful unassigned state. * the methodology of updating the private cache as a different step seems unnecessary given the permissions at the file system level. What really needs to happen is a massive overhaul of the entire distributed cache system. But that's a bigger project, preferably for someone who gets paid to do hadoop development full time. So, like all of the patches I've been submitting lately, I'm not expecting them to get committed. But this is enough of a patch for someone who needs a useable system until a working release ships. Distributed caches are not removed properly --- Key: MAPREDUCE-3824 URL: https://issues.apache.org/jira/browse/MAPREDUCE-3824 Project: Hadoop Map/Reduce Issue Type: Bug Components: distributed-cache Affects Versions: 1.0.0 Reporter: Allen Wittenauer Priority: Critical Attachments: MAPREDUCE-3824-branch-1.0.txt Distributed caches are not being properly removed by the TaskTracker when they are expected to be expired. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (MAPREDUCE-3719) Make gridmix performance on YARN+MR to match or exceed that on 1.0
[ https://issues.apache.org/jira/browse/MAPREDUCE-3719?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13202614#comment-13202614 ] Amol Kekre commented on MAPREDUCE-3719: --- First formal number for .23 is slightly better than 1.0 line (actually .204). Make gridmix performance on YARN+MR to match or exceed that on 1.0 -- Key: MAPREDUCE-3719 URL: https://issues.apache.org/jira/browse/MAPREDUCE-3719 Project: Hadoop Map/Reduce Issue Type: Sub-task Components: mrv2, performance Reporter: Vinod Kumar Vavilapalli -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (MAPREDUCE-3824) Distributed caches are not removed properly
[ https://issues.apache.org/jira/browse/MAPREDUCE-3824?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13202621#comment-13202621 ] Allen Wittenauer commented on MAPREDUCE-3824: - PS, if the tasktracker can survive disk failure in 1.0, it must not work under these conditions. So many failures... Distributed caches are not removed properly --- Key: MAPREDUCE-3824 URL: https://issues.apache.org/jira/browse/MAPREDUCE-3824 Project: Hadoop Map/Reduce Issue Type: Bug Components: distributed-cache Affects Versions: 1.0.0 Reporter: Allen Wittenauer Priority: Critical Attachments: MAPREDUCE-3824-branch-1.0.txt Distributed caches are not being properly removed by the TaskTracker when they are expected to be expired. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (MAPREDUCE-3824) Distributed caches are not removed properly
[ https://issues.apache.org/jira/browse/MAPREDUCE-3824?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13202622#comment-13202622 ] Allen Wittenauer commented on MAPREDUCE-3824: - (well the tasks that the tasks launch, not the TT itself) Distributed caches are not removed properly --- Key: MAPREDUCE-3824 URL: https://issues.apache.org/jira/browse/MAPREDUCE-3824 Project: Hadoop Map/Reduce Issue Type: Bug Components: distributed-cache Affects Versions: 1.0.0 Reporter: Allen Wittenauer Priority: Critical Attachments: MAPREDUCE-3824-branch-1.0.txt Distributed caches are not being properly removed by the TaskTracker when they are expected to be expired. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Created] (MAPREDUCE-3833) Capacity scheduler queue refresh doesn't recompute queue capacities properly
Capacity scheduler queue refresh doesn't recompute queue capacities properly Key: MAPREDUCE-3833 URL: https://issues.apache.org/jira/browse/MAPREDUCE-3833 Project: Hadoop Map/Reduce Issue Type: Bug Components: mrv2 Affects Versions: 0.23.1, 0.24.0 Reporter: Jason Lowe Refreshing the capacity scheduler configuration (e.g.: via yarn rmadmin -refreshQueues) can fail to compute the proper absolute capacity for leaf queues. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (MAPREDUCE-3831) RM scalability runtime is worse than 0.20.204 by 14.2%
[ https://issues.apache.org/jira/browse/MAPREDUCE-3831?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13202656#comment-13202656 ] Arun C Murthy commented on MAPREDUCE-3831: -- This looks like a bug in GridMix. Amar, can you please take a look? Thanks. RM scalability runtime is worse than 0.20.204 by 14.2% -- Key: MAPREDUCE-3831 URL: https://issues.apache.org/jira/browse/MAPREDUCE-3831 Project: Hadoop Map/Reduce Issue Type: Task Components: benchmarks, mrv2, resourcemanager Affects Versions: 0.23.1 Reporter: Vinay Kumar Thota RM scalability runtime is worse than 0.20.204 by 14.2% Overall runtime against Fred in a 350 nodes cluster is 2155 secs. Overall runtime against .23 in 350 nodes cluster is 2462 secs. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Assigned] (MAPREDUCE-3826) RM UI when loaded throws a message stating Data Tables warning and then the column sorting stops working
[ https://issues.apache.org/jira/browse/MAPREDUCE-3826?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Jonathan Eagles reassigned MAPREDUCE-3826: -- Assignee: Jonathan Eagles RM UI when loaded throws a message stating Data Tables warning and then the column sorting stops working Key: MAPREDUCE-3826 URL: https://issues.apache.org/jira/browse/MAPREDUCE-3826 Project: Hadoop Map/Reduce Issue Type: Bug Components: mrv2 Reporter: Arpit Gupta Assignee: Jonathan Eagles Attachments: image.tiff -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Updated] (MAPREDUCE-3833) Capacity scheduler queue refresh doesn't recompute queue capacities properly
[ https://issues.apache.org/jira/browse/MAPREDUCE-3833?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Jason Lowe updated MAPREDUCE-3833: -- Attachment: MAPREDUCE-3833-testcase.patch Attached a test case. Capacity scheduler queue refresh doesn't recompute queue capacities properly Key: MAPREDUCE-3833 URL: https://issues.apache.org/jira/browse/MAPREDUCE-3833 Project: Hadoop Map/Reduce Issue Type: Bug Components: mrv2 Affects Versions: 0.23.1, 0.24.0 Reporter: Jason Lowe Attachments: MAPREDUCE-3833-testcase.patch Refreshing the capacity scheduler configuration (e.g.: via yarn rmadmin -refreshQueues) can fail to compute the proper absolute capacity for leaf queues. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Updated] (MAPREDUCE-3815) Data Locality suffers if the AM asks for containers using IPs instead of hostnames
[ https://issues.apache.org/jira/browse/MAPREDUCE-3815?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Siddharth Seth updated MAPREDUCE-3815: -- Status: Patch Available (was: Open) Data Locality suffers if the AM asks for containers using IPs instead of hostnames -- Key: MAPREDUCE-3815 URL: https://issues.apache.org/jira/browse/MAPREDUCE-3815 Project: Hadoop Map/Reduce Issue Type: Sub-task Components: mrv2 Affects Versions: 0.23.0 Reporter: Siddharth Seth Assignee: Siddharth Seth Priority: Critical Attachments: MR3815.txt BlockLocation.getHosts() returns IP addresses occasionally. Data locality is affected - since the RM requires hostnames. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (MAPREDUCE-3815) Data Locality suffers if HDFS returns IPs in getFileBlockLocations
[
https://issues.apache.org/jira/browse/MAPREDUCE-3815?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13202685#comment-13202685
]
Siddharth Seth commented on MAPREDUCE-3815:
---
Looked at this a little more.
This shows up when a split spans across multiple blocks.
{{getFileBlockLocations}} always returns hostnames. In case of multiple blocks,
mapred.FileInputFormat ends up using {{BlockLocations.getTopologyPaths}}
instead of getFileBlockLocations - which returns an IP address.
Will open a MR / HDFS jira once I can find out how this API behaves in the 1.0
line. Anyone happen to know ?
Meanwhile, changing the description and posting a patch to have the AM resolve
IPs if they show up.
Data Locality suffers if HDFS returns IPs in getFileBlockLocations
--
Key: MAPREDUCE-3815
URL: https://issues.apache.org/jira/browse/MAPREDUCE-3815
Project: Hadoop Map/Reduce
Issue Type: Sub-task
Components: mrv2
Affects Versions: 0.23.0
Reporter: Siddharth Seth
Assignee: Siddharth Seth
Priority: Critical
Attachments: MR3815.txt
BlockLocation.getHosts() returns IP addresses occasionally. Data locality is
affected - since the RM requires hostnames.
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators:
https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Updated] (MAPREDUCE-3815) Data Locality suffers if the AM asks for containers using IPs instead of hostnames
[ https://issues.apache.org/jira/browse/MAPREDUCE-3815?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Siddharth Seth updated MAPREDUCE-3815: -- Summary: Data Locality suffers if the AM asks for containers using IPs instead of hostnames (was: Data Locality suffers if HDFS returns IPs in getFileBlockLocations) Data Locality suffers if the AM asks for containers using IPs instead of hostnames -- Key: MAPREDUCE-3815 URL: https://issues.apache.org/jira/browse/MAPREDUCE-3815 Project: Hadoop Map/Reduce Issue Type: Sub-task Components: mrv2 Affects Versions: 0.23.0 Reporter: Siddharth Seth Assignee: Siddharth Seth Priority: Critical Attachments: MR3815.txt BlockLocation.getHosts() returns IP addresses occasionally. Data locality is affected - since the RM requires hostnames. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Updated] (MAPREDUCE-3815) Data Locality suffers if the AM asks for containers using IPs instead of hostnames
[ https://issues.apache.org/jira/browse/MAPREDUCE-3815?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Siddharth Seth updated MAPREDUCE-3815: -- Attachment: MR3815.txt Data Locality suffers if the AM asks for containers using IPs instead of hostnames -- Key: MAPREDUCE-3815 URL: https://issues.apache.org/jira/browse/MAPREDUCE-3815 Project: Hadoop Map/Reduce Issue Type: Sub-task Components: mrv2 Affects Versions: 0.23.0 Reporter: Siddharth Seth Assignee: Siddharth Seth Priority: Critical Attachments: MR3815.txt BlockLocation.getHosts() returns IP addresses occasionally. Data locality is affected - since the RM requires hostnames. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Created] (MAPREDUCE-3834) If multiple hosts for a split belong to the same rack, the rack is added multiple times in the AM request table
If multiple hosts for a split belong to the same rack, the rack is added multiple times in the AM request table --- Key: MAPREDUCE-3834 URL: https://issues.apache.org/jira/browse/MAPREDUCE-3834 Project: Hadoop Map/Reduce Issue Type: Bug Components: mrv2 Affects Versions: 0.23.0 Reporter: Siddharth Seth Priority: Critical Should be added only once - so that the RM doesn't think there's multiple rack local requests for that particular rack. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (MAPREDUCE-3834) If multiple hosts for a split belong to the same rack, the rack is added multiple times in the AM request table
[ https://issues.apache.org/jira/browse/MAPREDUCE-3834?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13202702#comment-13202702 ] Siddharth Seth commented on MAPREDUCE-3834: --- Applies on top of 3815. Will submit patch after that's done. If multiple hosts for a split belong to the same rack, the rack is added multiple times in the AM request table --- Key: MAPREDUCE-3834 URL: https://issues.apache.org/jira/browse/MAPREDUCE-3834 Project: Hadoop Map/Reduce Issue Type: Bug Components: mrv2 Affects Versions: 0.23.0 Reporter: Siddharth Seth Priority: Critical Attachments: MR3834.txt Should be added only once - so that the RM doesn't think there's multiple rack local requests for that particular rack. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Updated] (MAPREDUCE-3834) If multiple hosts for a split belong to the same rack, the rack is added multiple times in the AM request table
[ https://issues.apache.org/jira/browse/MAPREDUCE-3834?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Siddharth Seth updated MAPREDUCE-3834: -- Attachment: MR3834.txt If multiple hosts for a split belong to the same rack, the rack is added multiple times in the AM request table --- Key: MAPREDUCE-3834 URL: https://issues.apache.org/jira/browse/MAPREDUCE-3834 Project: Hadoop Map/Reduce Issue Type: Bug Components: mrv2 Affects Versions: 0.23.0 Reporter: Siddharth Seth Priority: Critical Attachments: MR3834.txt Should be added only once - so that the RM doesn't think there's multiple rack local requests for that particular rack. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Updated] (MAPREDUCE-3833) Capacity scheduler queue refresh doesn't recompute queue capacities properly
[ https://issues.apache.org/jira/browse/MAPREDUCE-3833?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Arun C Murthy updated MAPREDUCE-3833: - Attachment: MAPREDUCE-3833.patch Jason, the test helped tremendously - here is the fix. Thanks! Capacity scheduler queue refresh doesn't recompute queue capacities properly Key: MAPREDUCE-3833 URL: https://issues.apache.org/jira/browse/MAPREDUCE-3833 Project: Hadoop Map/Reduce Issue Type: Bug Components: mrv2 Affects Versions: 0.23.1, 0.24.0 Reporter: Jason Lowe Assignee: Arun C Murthy Attachments: MAPREDUCE-3833-testcase.patch, MAPREDUCE-3833.patch Refreshing the capacity scheduler configuration (e.g.: via yarn rmadmin -refreshQueues) can fail to compute the proper absolute capacity for leaf queues. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Assigned] (MAPREDUCE-3833) Capacity scheduler queue refresh doesn't recompute queue capacities properly
[ https://issues.apache.org/jira/browse/MAPREDUCE-3833?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Arun C Murthy reassigned MAPREDUCE-3833: Assignee: Jason Lowe (was: Arun C Murthy) Jason deserves credit for the hardwork here... Capacity scheduler queue refresh doesn't recompute queue capacities properly Key: MAPREDUCE-3833 URL: https://issues.apache.org/jira/browse/MAPREDUCE-3833 Project: Hadoop Map/Reduce Issue Type: Bug Components: mrv2 Affects Versions: 0.23.1, 0.24.0 Reporter: Jason Lowe Assignee: Jason Lowe Attachments: MAPREDUCE-3833-testcase.patch, MAPREDUCE-3833.patch Refreshing the capacity scheduler configuration (e.g.: via yarn rmadmin -refreshQueues) can fail to compute the proper absolute capacity for leaf queues. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Updated] (MAPREDUCE-3833) Capacity scheduler queue refresh doesn't recompute queue capacities properly
[ https://issues.apache.org/jira/browse/MAPREDUCE-3833?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Arun C Murthy updated MAPREDUCE-3833: - Status: Patch Available (was: Open) Capacity scheduler queue refresh doesn't recompute queue capacities properly Key: MAPREDUCE-3833 URL: https://issues.apache.org/jira/browse/MAPREDUCE-3833 Project: Hadoop Map/Reduce Issue Type: Bug Components: mrv2 Affects Versions: 0.23.1, 0.24.0 Reporter: Jason Lowe Assignee: Jason Lowe Attachments: MAPREDUCE-3833-testcase.patch, MAPREDUCE-3833.patch Refreshing the capacity scheduler configuration (e.g.: via yarn rmadmin -refreshQueues) can fail to compute the proper absolute capacity for leaf queues. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Updated] (MAPREDUCE-3823) Counters are getting calculated twice at job-finish and delaying clients.
[ https://issues.apache.org/jira/browse/MAPREDUCE-3823?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Vinod Kumar Vavilapalli updated MAPREDUCE-3823: --- Attachment: MAPREDUCE-3823-20120207.txt Here we go again. Test bugs and findbugs warnings should be gone after this. Counters are getting calculated twice at job-finish and delaying clients. - Key: MAPREDUCE-3823 URL: https://issues.apache.org/jira/browse/MAPREDUCE-3823 Project: Hadoop Map/Reduce Issue Type: Sub-task Components: mrv2, performance Affects Versions: 0.23.0 Reporter: Vinod Kumar Vavilapalli Assignee: Vinod Kumar Vavilapalli Fix For: 0.23.1 Attachments: MAPREDUCE-3823-20120206.1.txt, MAPREDUCE-3823-20120206.txt, MAPREDUCE-3823-20120207.txt -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Updated] (MAPREDUCE-3823) Counters are getting calculated twice at job-finish and delaying clients.
[ https://issues.apache.org/jira/browse/MAPREDUCE-3823?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Vinod Kumar Vavilapalli updated MAPREDUCE-3823: --- Status: Open (was: Patch Available) Good catch on the bug, Jenkins! Counters are getting calculated twice at job-finish and delaying clients. - Key: MAPREDUCE-3823 URL: https://issues.apache.org/jira/browse/MAPREDUCE-3823 Project: Hadoop Map/Reduce Issue Type: Sub-task Components: mrv2, performance Affects Versions: 0.23.0 Reporter: Vinod Kumar Vavilapalli Assignee: Vinod Kumar Vavilapalli Fix For: 0.23.1 Attachments: MAPREDUCE-3823-20120206.1.txt, MAPREDUCE-3823-20120206.txt, MAPREDUCE-3823-20120207.txt -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Updated] (MAPREDUCE-3823) Counters are getting calculated twice at job-finish and delaying clients.
[ https://issues.apache.org/jira/browse/MAPREDUCE-3823?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Vinod Kumar Vavilapalli updated MAPREDUCE-3823: --- Status: Patch Available (was: Open) Counters are getting calculated twice at job-finish and delaying clients. - Key: MAPREDUCE-3823 URL: https://issues.apache.org/jira/browse/MAPREDUCE-3823 Project: Hadoop Map/Reduce Issue Type: Sub-task Components: mrv2, performance Affects Versions: 0.23.0 Reporter: Vinod Kumar Vavilapalli Assignee: Vinod Kumar Vavilapalli Fix For: 0.23.1 Attachments: MAPREDUCE-3823-20120206.1.txt, MAPREDUCE-3823-20120206.txt, MAPREDUCE-3823-20120207.txt -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (MAPREDUCE-3833) Capacity scheduler queue refresh doesn't recompute queue capacities properly
[ https://issues.apache.org/jira/browse/MAPREDUCE-3833?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13202731#comment-13202731 ] Jason Lowe commented on MAPREDUCE-3833: --- +1 lgtm. Thanks for the quick turnaround Arun! Capacity scheduler queue refresh doesn't recompute queue capacities properly Key: MAPREDUCE-3833 URL: https://issues.apache.org/jira/browse/MAPREDUCE-3833 Project: Hadoop Map/Reduce Issue Type: Bug Components: mrv2 Affects Versions: 0.23.1, 0.24.0 Reporter: Jason Lowe Assignee: Jason Lowe Attachments: MAPREDUCE-3833-testcase.patch, MAPREDUCE-3833.patch Refreshing the capacity scheduler configuration (e.g.: via yarn rmadmin -refreshQueues) can fail to compute the proper absolute capacity for leaf queues. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Updated] (MAPREDUCE-3803) HDFS-2864 broke ant compilation
[
https://issues.apache.org/jira/browse/MAPREDUCE-3803?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Tsz Wo (Nicholas), SZE updated MAPREDUCE-3803:
--
Target Version/s: 0.23.1, 0.24.0 (was: 0.24.0, 0.23.1)
Affects Version/s: (was: 0.23.0)
0.23.1
Fix Version/s: 0.24.0
0.23.1
HDFS-2864 broke ant compilation
---
Key: MAPREDUCE-3803
URL: https://issues.apache.org/jira/browse/MAPREDUCE-3803
Project: Hadoop Map/Reduce
Issue Type: Test
Components: build
Affects Versions: 0.23.1, 0.24.0
Reporter: Ravi Prakash
Assignee: Ravi Prakash
Fix For: 0.23.1, 0.24.0
Attachments: HDFS-2888.patch, MAPREDUCE-3803.patch
compile:
[echo] contrib: raid
[javac]
somePath/hadoop-mapreduce-project/src/contrib/build-contrib.xml:194:
warning: 'includeantruntime' was not set, defaulting to
build.sysclasspath=last; set to false for repeatable builds
[javac] Compiling 28 source files to
somepath/hadoop-mapreduce-project/build/contrib/raid/classes
[javac]
somepath/hadoop-mapreduce-project/src/contrib/raid/src/java/org/apache/hadoop/hdfs/server/datanode/RaidBlockSender.java:111:
cannot find symbol
[javac] symbol : variable METADATA_VERSION
[javac] location: class org.apache.hadoop.hdfs.server.datanode.FSDataset
[javac] if (version != FSDataset.METADATA_VERSION) {
[javac] ^
[javac]
somepath/hadoop-mapreduce-project/src/contrib/raid/src/java/org/apache/hadoop/raid/BlockFixer.java:649:
cannot find symbol
[javac] symbol : variable METADATA_VERSION
[javac] location: class org.apache.hadoop.hdfs.server.datanode.FSDataset
[javac] mdOut.writeShort(FSDataset.METADATA_VERSION);
[javac] ^
[javac] Note: Some input files use or override a deprecated API.
[javac] Note: Recompile with -Xlint:deprecation for details.
[javac] Note: Some input files use unchecked or unsafe operations.
[javac] Note: Recompile with -Xlint:unchecked for details.
[javac] 2 errors
BUILD FAILED
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators:
https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (MAPREDUCE-3825) Need generalized multi-token filesystem support
[ https://issues.apache.org/jira/browse/MAPREDUCE-3825?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13202742#comment-13202742 ] Sanjay Radia commented on MAPREDUCE-3825: - Note the FileSystem interface was changed specifically to deal with multiple-filesystem file systems like viewfs (ie it returns an arrays of tokens not a single token). So the question is: what is broken? * The fact that the token cache is keyed by file system uri or * That FileSystem has a method called getDelegationTokens and not a method called getEmbeddedFileSystems(). When we changed from getDelegationToken() to getDelegationTokens() we had dismissed the alternate you are proposing since we needed a method to get delegation token from a file system anyway. Need generalized multi-token filesystem support --- Key: MAPREDUCE-3825 URL: https://issues.apache.org/jira/browse/MAPREDUCE-3825 Project: Hadoop Map/Reduce Issue Type: Bug Components: security Affects Versions: 0.23.1, 0.24.0 Reporter: Daryn Sharp Assignee: Daryn Sharp This is the counterpart to HADOOP-7967. The token cache currently tries to assume a filesystem's token service key. The assumption generally worked while there was a one to one mapping of filesystem to token. With the advent of multi-token filesystems like viewfs, the token cache will try to use a service key (ie. for viewfs) that will never exist (because it really gets the mounted fs tokens). -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Updated] (MAPREDUCE-3827) Counters aggregation slowed down significantly after MAPREDUCE-3749
[ https://issues.apache.org/jira/browse/MAPREDUCE-3827?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Vinod Kumar Vavilapalli updated MAPREDUCE-3827: --- Status: Patch Available (was: Open) Counters aggregation slowed down significantly after MAPREDUCE-3749 --- Key: MAPREDUCE-3827 URL: https://issues.apache.org/jira/browse/MAPREDUCE-3827 Project: Hadoop Map/Reduce Issue Type: Sub-task Components: mrv2, performance Affects Versions: 0.23.0 Reporter: Vinod Kumar Vavilapalli Assignee: Vinod Kumar Vavilapalli Priority: Blocker Fix For: 0.23.1 Attachments: MAPREDUCE-3827-20120207.txt -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Updated] (MAPREDUCE-3827) Counters aggregation slowed down significantly after MAPREDUCE-3749
[ https://issues.apache.org/jira/browse/MAPREDUCE-3827?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Vinod Kumar Vavilapalli updated MAPREDUCE-3827: --- Attachment: MAPREDUCE-3827-20120207.txt Using ConcurrentSkipListMap instead of TreeMap and simply returning its iterator which is fail-safe but only weakly consistent. No new test, the one added at MAPREDUCE-3749 should continue to validate the changes to the iterator. Counters aggregation slowed down significantly after MAPREDUCE-3749 --- Key: MAPREDUCE-3827 URL: https://issues.apache.org/jira/browse/MAPREDUCE-3827 Project: Hadoop Map/Reduce Issue Type: Sub-task Components: mrv2, performance Affects Versions: 0.23.0 Reporter: Vinod Kumar Vavilapalli Assignee: Vinod Kumar Vavilapalli Priority: Blocker Fix For: 0.23.1 Attachments: MAPREDUCE-3827-20120207.txt -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (MAPREDUCE-3815) Data Locality suffers if the AM asks for containers using IPs instead of hostnames
[ https://issues.apache.org/jira/browse/MAPREDUCE-3815?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13202754#comment-13202754 ] Vinod Kumar Vavilapalli commented on MAPREDUCE-3815: Looking at the patch for a review. Data Locality suffers if the AM asks for containers using IPs instead of hostnames -- Key: MAPREDUCE-3815 URL: https://issues.apache.org/jira/browse/MAPREDUCE-3815 Project: Hadoop Map/Reduce Issue Type: Sub-task Components: mrv2 Affects Versions: 0.23.0 Reporter: Siddharth Seth Assignee: Siddharth Seth Priority: Critical Attachments: MR3815.txt BlockLocation.getHosts() returns IP addresses occasionally. Data locality is affected - since the RM requires hostnames. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (MAPREDUCE-3815) Data Locality suffers if the AM asks for containers using IPs instead of hostnames
[ https://issues.apache.org/jira/browse/MAPREDUCE-3815?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13202757#comment-13202757 ] Hadoop QA commented on MAPREDUCE-3815: -- -1 overall. Here are the results of testing the latest attachment http://issues.apache.org/jira/secure/attachment/12513668/MR3815.txt against trunk revision . +1 @author. The patch does not contain any @author tags. +1 tests included. The patch appears to include 6 new or modified tests. +1 javadoc. The javadoc tool did not generate any warning messages. +1 javac. The applied patch does not increase the total number of javac compiler warnings. +1 eclipse:eclipse. The patch built with eclipse:eclipse. +1 findbugs. The patch does not introduce any new Findbugs (version 1.3.9) warnings. +1 release audit. The applied patch does not increase the total number of release audit warnings. -1 core tests. The patch failed these unit tests: org.apache.hadoop.mapred.TestJobCounters +1 contrib tests. The patch passed contrib unit tests. Test results: https://builds.apache.org/job/PreCommit-MAPREDUCE-Build/1811//testReport/ Console output: https://builds.apache.org/job/PreCommit-MAPREDUCE-Build/1811//console This message is automatically generated. Data Locality suffers if the AM asks for containers using IPs instead of hostnames -- Key: MAPREDUCE-3815 URL: https://issues.apache.org/jira/browse/MAPREDUCE-3815 Project: Hadoop Map/Reduce Issue Type: Sub-task Components: mrv2 Affects Versions: 0.23.0 Reporter: Siddharth Seth Assignee: Siddharth Seth Priority: Critical Attachments: MR3815.txt BlockLocation.getHosts() returns IP addresses occasionally. Data locality is affected - since the RM requires hostnames. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Updated] (MAPREDUCE-3436) JobHistory webapp address should use the host from the jobhistory address
[
https://issues.apache.org/jira/browse/MAPREDUCE-3436?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Siddharth Seth updated MAPREDUCE-3436:
--
Summary: JobHistory webapp address should use the host from the jobhistory
address (was: jobhistory link may be broken depending on the interface it is
listening on)
Changing the description to reflect the change. Will commit this shortly.
JobHistory webapp address should use the host from the jobhistory address
-
Key: MAPREDUCE-3436
URL: https://issues.apache.org/jira/browse/MAPREDUCE-3436
Project: Hadoop Map/Reduce
Issue Type: Bug
Components: mrv2, webapps
Affects Versions: 0.23.0, 0.23.1
Reporter: Bruno Mahé
Assignee: Ahmed Radwan
Labels: bigtop
Attachments: MAPREDUCE-3436.patch, MAPREDUCE-3436_rev2.patch
On the following page : http://RESOURCE_MANAGER:8088/cluster/apps
There are links to the history for each application. None of them can be
reached since they all point to the ip 0.0.0.0. For instance:
http://0.0.0.0:8088/proxy/application_1321658790349_0002/jobhistory/job/job_1321658790349_2_2
Am I missing something?
[root@bigtop-fedora-15 ~]# jps
9968 ResourceManager
1495 NameNode
1645 DataNode
12935 Jps
11140 -- process information unavailable
5309 JobHistoryServer
10237 NodeManager
[root@bigtop-fedora-15 ~]# netstat -tlpn | grep 8088
tcp0 0 :::8088 :::*
LISTEN 9968/java
For reference, here is my configuration:
root@bigtop-fedora-15 ~]# cat /etc/yarn/conf/yarn-site.xml
?xml version=1.0?
configuration
!-- Site specific YARN configuration properties --
property
nameyarn.nodemanager.aux-services/name
valuemapreduce.shuffle/value
/property
property
nameyarn.nodemanager.aux-services.mapreduce.shuffle.class/name
valueorg.apache.hadoop.mapred.ShuffleHandler/value
/property
property
namemapreduce.admin.user.env/name
valueCLASSPATH=/etc/hadoop/conf/*:/usr/lib/hadoop/*:/usr/lib/hadoop/lib/*/value
/property
/configuration
[root@bigtop-fedora-15 ~]# cat /etc/hadoop/conf/hdfs-site.xml
?xml version=1.0?
configuration
property
namedfs.replication/name
value1/value
/property
property
namedfs.permissions/name
valuefalse/value
/property
property
!-- specify this so that running 'hadoop namenode -format' formats the
right dir --
namedfs.name.dir/name
value/var/lib/hadoop/cache/hadoop/dfs/name/value
/property
/configuration
[root@bigtop-fedora-15 ~]# cat /etc/hadoop/conf/core-site.xml
?xml version=1.0?
configuration
property
namefs.default.name/name
valuehdfs://localhost:8020/value
/property
property
namehadoop.tmp.dir/name
value/var/lib/hadoop/cache/${user.name}/value
/property
!-- OOZIE proxy user setting --
property
namehadoop.proxyuser.oozie.hosts/name
value*/value
/property
property
namehadoop.proxyuser.oozie.groups/name
value*/value
/property
/configuration
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators:
https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (MAPREDUCE-3825) Need generalized multi-token filesystem support
[ https://issues.apache.org/jira/browse/MAPREDUCE-3825?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13202762#comment-13202762 ] Sanjay Radia commented on MAPREDUCE-3825: - The general idea is that that when a job is submitted one has a list of paths (input, output, defaultfs, etc). From that list of paths one gets a set of file systems, eliminates duplicates and then get delegation tokens for each. This works except that it may not be efficient in some cases. Eg: * Input path is hdfs://foo/bar * default fs is viewfs:/// which has mounted hdfs://foo/ In this case one will obtain the delegation tokens for hdfs://foo *twice*. The Jira description seems to suggest that the current implementation does not work (Bug/Major) while what i am concluding is that it is not optimal. Need generalized multi-token filesystem support --- Key: MAPREDUCE-3825 URL: https://issues.apache.org/jira/browse/MAPREDUCE-3825 Project: Hadoop Map/Reduce Issue Type: Bug Components: security Affects Versions: 0.23.1, 0.24.0 Reporter: Daryn Sharp Assignee: Daryn Sharp This is the counterpart to HADOOP-7967. The token cache currently tries to assume a filesystem's token service key. The assumption generally worked while there was a one to one mapping of filesystem to token. With the advent of multi-token filesystems like viewfs, the token cache will try to use a service key (ie. for viewfs) that will never exist (because it really gets the mounted fs tokens). -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (MAPREDUCE-3826) RM UI when loaded throws a message stating Data Tables warning and then the column sorting stops working
[ https://issues.apache.org/jira/browse/MAPREDUCE-3826?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13202765#comment-13202765 ] Jonathan Eagles commented on MAPREDUCE-3826: This stems from MAPREDUCE-3746 where the HTML columns were updated but the JSON columns were not RM UI when loaded throws a message stating Data Tables warning and then the column sorting stops working Key: MAPREDUCE-3826 URL: https://issues.apache.org/jira/browse/MAPREDUCE-3826 Project: Hadoop Map/Reduce Issue Type: Bug Components: mrv2 Reporter: Arpit Gupta Assignee: Jonathan Eagles Attachments: image.tiff -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (MAPREDUCE-3833) Capacity scheduler queue refresh doesn't recompute queue capacities properly
[ https://issues.apache.org/jira/browse/MAPREDUCE-3833?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13202773#comment-13202773 ] Hadoop QA commented on MAPREDUCE-3833: -- -1 overall. Here are the results of testing the latest attachment http://issues.apache.org/jira/secure/attachment/12513675/MAPREDUCE-3833.patch against trunk revision . +1 @author. The patch does not contain any @author tags. +1 tests included. The patch appears to include 3 new or modified tests. +1 javadoc. The javadoc tool did not generate any warning messages. +1 javac. The applied patch does not increase the total number of javac compiler warnings. +1 eclipse:eclipse. The patch built with eclipse:eclipse. +1 findbugs. The patch does not introduce any new Findbugs (version 1.3.9) warnings. +1 release audit. The applied patch does not increase the total number of release audit warnings. -1 core tests. The patch failed these unit tests: org.apache.hadoop.mapred.TestJobCounters +1 contrib tests. The patch passed contrib unit tests. Test results: https://builds.apache.org/job/PreCommit-MAPREDUCE-Build/1812//testReport/ Console output: https://builds.apache.org/job/PreCommit-MAPREDUCE-Build/1812//console This message is automatically generated. Capacity scheduler queue refresh doesn't recompute queue capacities properly Key: MAPREDUCE-3833 URL: https://issues.apache.org/jira/browse/MAPREDUCE-3833 Project: Hadoop Map/Reduce Issue Type: Bug Components: mrv2 Affects Versions: 0.23.1, 0.24.0 Reporter: Jason Lowe Assignee: Jason Lowe Attachments: MAPREDUCE-3833-testcase.patch, MAPREDUCE-3833.patch Refreshing the capacity scheduler configuration (e.g.: via yarn rmadmin -refreshQueues) can fail to compute the proper absolute capacity for leaf queues. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Updated] (MAPREDUCE-3436) JobHistory webapp address should use the host from the jobhistory address
[
https://issues.apache.org/jira/browse/MAPREDUCE-3436?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Siddharth Seth updated MAPREDUCE-3436:
--
Resolution: Fixed
Fix Version/s: 0.23.1
Hadoop Flags: Reviewed
Status: Resolved (was: Patch Available)
Committed to trunk and branch-0.23. Thanks Ahmed.
Will try posting a patch to MAPREDUCE-3828 later today.
JobHistory webapp address should use the host from the jobhistory address
-
Key: MAPREDUCE-3436
URL: https://issues.apache.org/jira/browse/MAPREDUCE-3436
Project: Hadoop Map/Reduce
Issue Type: Bug
Components: mrv2, webapps
Affects Versions: 0.23.0, 0.23.1
Reporter: Bruno Mahé
Assignee: Ahmed Radwan
Labels: bigtop
Fix For: 0.23.1
Attachments: MAPREDUCE-3436.patch, MAPREDUCE-3436_rev2.patch
On the following page : http://RESOURCE_MANAGER:8088/cluster/apps
There are links to the history for each application. None of them can be
reached since they all point to the ip 0.0.0.0. For instance:
http://0.0.0.0:8088/proxy/application_1321658790349_0002/jobhistory/job/job_1321658790349_2_2
Am I missing something?
[root@bigtop-fedora-15 ~]# jps
9968 ResourceManager
1495 NameNode
1645 DataNode
12935 Jps
11140 -- process information unavailable
5309 JobHistoryServer
10237 NodeManager
[root@bigtop-fedora-15 ~]# netstat -tlpn | grep 8088
tcp0 0 :::8088 :::*
LISTEN 9968/java
For reference, here is my configuration:
root@bigtop-fedora-15 ~]# cat /etc/yarn/conf/yarn-site.xml
?xml version=1.0?
configuration
!-- Site specific YARN configuration properties --
property
nameyarn.nodemanager.aux-services/name
valuemapreduce.shuffle/value
/property
property
nameyarn.nodemanager.aux-services.mapreduce.shuffle.class/name
valueorg.apache.hadoop.mapred.ShuffleHandler/value
/property
property
namemapreduce.admin.user.env/name
valueCLASSPATH=/etc/hadoop/conf/*:/usr/lib/hadoop/*:/usr/lib/hadoop/lib/*/value
/property
/configuration
[root@bigtop-fedora-15 ~]# cat /etc/hadoop/conf/hdfs-site.xml
?xml version=1.0?
configuration
property
namedfs.replication/name
value1/value
/property
property
namedfs.permissions/name
valuefalse/value
/property
property
!-- specify this so that running 'hadoop namenode -format' formats the
right dir --
namedfs.name.dir/name
value/var/lib/hadoop/cache/hadoop/dfs/name/value
/property
/configuration
[root@bigtop-fedora-15 ~]# cat /etc/hadoop/conf/core-site.xml
?xml version=1.0?
configuration
property
namefs.default.name/name
valuehdfs://localhost:8020/value
/property
property
namehadoop.tmp.dir/name
value/var/lib/hadoop/cache/${user.name}/value
/property
!-- OOZIE proxy user setting --
property
namehadoop.proxyuser.oozie.hosts/name
value*/value
/property
property
namehadoop.proxyuser.oozie.groups/name
value*/value
/property
/configuration
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators:
https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira
