[jira] [Updated] (MAPREDUCE-3806) [Gridmix] TestGridmixSubmission fails due to incorrect version of jackson
[
https://issues.apache.org/jira/browse/MAPREDUCE-3806?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Allen Wittenauer updated MAPREDUCE-3806:
Affects Version/s: (was: 0.24.0)
3.0.0
[Gridmix] TestGridmixSubmission fails due to incorrect version of jackson
-
Key: MAPREDUCE-3806
URL: https://issues.apache.org/jira/browse/MAPREDUCE-3806
Project: Hadoop Map/Reduce
Issue Type: Bug
Components: contrib/gridmix
Affects Versions: 3.0.0
Reporter: Amar Kamat
Labels: error, gridmix, junit
{{TestGridmixSubmission}} fails with the following error
{code}
org.codehaus.jackson.type.JavaType.isMapLikeType()Z
java.lang.NoSuchMethodError:
org.codehaus.jackson.type.JavaType.isMapLikeType()Z
at
org.codehaus.jackson.map.deser.StdDeserializerProvider._createDeserializer(StdDeserializerProvider.java:374)
at
org.codehaus.jackson.map.deser.StdDeserializerProvider._createAndCache2(StdDeserializerProvider.java:307)
at
org.codehaus.jackson.map.deser.StdDeserializerProvider._createAndCacheValueDeserializer(StdDeserializerProvider.java:287)
at
org.codehaus.jackson.map.deser.StdDeserializerProvider.findValueDeserializer(StdDeserializerProvider.java:136)
at
org.codehaus.jackson.map.deser.StdDeserializer.findDeserializer(StdDeserializer.java:551)
at
org.codehaus.jackson.map.deser.BeanDeserializer.resolve(BeanDeserializer.java:268)
at
org.codehaus.jackson.map.deser.StdDeserializerProvider._resolveDeserializer(StdDeserializerProvider.java:404)
at
org.codehaus.jackson.map.deser.StdDeserializerProvider._createAndCache2(StdDeserializerProvider.java:349)
at
org.codehaus.jackson.map.deser.StdDeserializerProvider._createAndCacheValueDeserializer(StdDeserializerProvider.java:287)
at
org.codehaus.jackson.map.deser.StdDeserializerProvider.findValueDeserializer(StdDeserializerProvider.java:136)
at
org.codehaus.jackson.map.deser.StdDeserializerProvider.findTypedValueDeserializer(StdDeserializerProvider.java:157)
at
org.codehaus.jackson.map.ObjectMapper._findRootDeserializer(ObjectMapper.java:2468)
at
org.codehaus.jackson.map.ObjectMapper._readValue(ObjectMapper.java:2383)
at
org.codehaus.jackson.map.ObjectMapper.readValue(ObjectMapper.java:1094)
at
org.apache.hadoop.tools.rumen.JsonObjectMapperParser.getNext(JsonObjectMapperParser.java:84)
at
org.apache.hadoop.tools.rumen.ZombieJobProducer.getNextJob(ZombieJobProducer.java:117)
at
org.apache.hadoop.tools.rumen.ZombieJobProducer.getNextJob(ZombieJobProducer.java:29)
at
org.apache.hadoop.mapred.gridmix.TestGridmixSubmission.testTraceReader(TestGridmixSubmission.java:440)
{code}
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Updated] (MAPREDUCE-3213) AM should kill all running tasks on a node when it gets marked as blacklisted
[ https://issues.apache.org/jira/browse/MAPREDUCE-3213?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Allen Wittenauer updated MAPREDUCE-3213: Fix Version/s: (was: 0.24.0) AM should kill all running tasks on a node when it gets marked as blacklisted - Key: MAPREDUCE-3213 URL: https://issues.apache.org/jira/browse/MAPREDUCE-3213 Project: Hadoop Map/Reduce Issue Type: Bug Components: mrv2 Affects Versions: 0.23.0 Reporter: Hitesh Shah -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Updated] (MAPREDUCE-3658) Improvements to CapacityScheduler documentation
[ https://issues.apache.org/jira/browse/MAPREDUCE-3658?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Allen Wittenauer updated MAPREDUCE-3658: Fix Version/s: (was: 0.24.0) Improvements to CapacityScheduler documentation --- Key: MAPREDUCE-3658 URL: https://issues.apache.org/jira/browse/MAPREDUCE-3658 Project: Hadoop Map/Reduce Issue Type: Improvement Components: mrv2 Affects Versions: 0.23.0 Reporter: Yoram Arnon Assignee: Yoram Arnon Priority: Minor Labels: documentation Attachments: MAPREDUCE-3658, MAPREDUCE-3658 Original Estimate: 3h Remaining Estimate: 3h There are some typos and some cases of incorrect English. Also, the descriptions of yarn.scheduler.capacity.queue-path.capacity, yarn.scheduler.capacity.queue-path.maximum-capacity, yarn.scheduler.capacity.queue-path.user-limit-factor, yarn.scheduler.capacity.maximum-applications are not very clear to the uninitiated. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Updated] (MAPREDUCE-2842) Maven build issues in MR2
[ https://issues.apache.org/jira/browse/MAPREDUCE-2842?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Allen Wittenauer updated MAPREDUCE-2842: Fix Version/s: (was: 0.24.0) Maven build issues in MR2 -- Key: MAPREDUCE-2842 URL: https://issues.apache.org/jira/browse/MAPREDUCE-2842 Project: Hadoop Map/Reduce Issue Type: Bug Components: build, mrv2 Affects Versions: 0.23.0 Reporter: Alejandro Abdelnur Labels: maven * mapreduce has not been rebased on top of trunk * mapreduce dir/module should be named hadoop-mapreduce (following convention of common, hdfs) * there is lot of stuff under mapreduce that seems stale (bin/, conf, ivy/, lib/ src/) * yarn* dirs/modules should be named hadoop-yarn* (following convention of other Hadoop artifacts) * yarn/bin/ scripts should be under yarn/src/main/bin * yarn/conf/ scripts should be under yarn/src/main/conf * JAR POM files do not use hadoop-project POM as parent * some POM files have version parameterized and this will break things for people consuming JARs from Maven repos * mapreduce is not using assembly from hadoop-assemblies (the changes introduced by HDFS-2096 make the assembly/packaging reusable across different components) -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Updated] (MAPREDUCE-3340) Deprecate Job.setJobSetupCleanupNeeded()
[ https://issues.apache.org/jira/browse/MAPREDUCE-3340?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Allen Wittenauer updated MAPREDUCE-3340: Fix Version/s: (was: 0.24.0) Deprecate Job.setJobSetupCleanupNeeded() Key: MAPREDUCE-3340 URL: https://issues.apache.org/jira/browse/MAPREDUCE-3340 Project: Hadoop Map/Reduce Issue Type: Bug Components: mrv2 Affects Versions: 0.23.0 Reporter: Vinod Kumar Vavilapalli Assignee: Vinod Kumar Vavilapalli We should deprecate the setJobSetupCleanupNeeded() API. It was originally added for performance reasons to avoid launching new JVMs altogether for job-setup and job-cleanup. With Yarn and MRAppMaster, setup and cleanup are run inside the AM itself and so nothing much can be gained by making them optional. Before 0.23, we could disable set up and cleanup, yet obtain the output when using FileOutputCommitter in the job-output directory. But post 0.23.0, that won't be the case because of the nested temporary directories to support AM recoverability. So it makes sense to *not* have cleanupJob optional. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Updated] (MAPREDUCE-3351) TaskAttempt's state string is not consumed by MR AM web-UI
[ https://issues.apache.org/jira/browse/MAPREDUCE-3351?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Allen Wittenauer updated MAPREDUCE-3351: Fix Version/s: (was: 0.24.0) TaskAttempt's state string is not consumed by MR AM web-UI -- Key: MAPREDUCE-3351 URL: https://issues.apache.org/jira/browse/MAPREDUCE-3351 Project: Hadoop Map/Reduce Issue Type: Bug Components: applicationmaster, mrv2 Affects Versions: 0.23.0 Reporter: Vinod Kumar Vavilapalli Jobs like random-writer use the state string to report the amount of work they have completed. JT used to print this on UI, AM webapp should do the same. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Updated] (MAPREDUCE-3390) NPE while submitting job
[ https://issues.apache.org/jira/browse/MAPREDUCE-3390?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Allen Wittenauer updated MAPREDUCE-3390: Affects Version/s: (was: 0.24.0) NPE while submitting job Key: MAPREDUCE-3390 URL: https://issues.apache.org/jira/browse/MAPREDUCE-3390 Project: Hadoop Map/Reduce Issue Type: Bug Components: mrv2 Affects Versions: 0.23.0 Environment: This happens only when there is something weird going on like stale jars or something similar. Reporter: John George Assignee: John George Priority: Minor Attachments: MAPREDUCE-3390-20111220.txt, MAPREDUCE-3390.branch-0.23.patch Caused by: java.lang.NullPointerException at java.io.Reader.init(Reader.java:61) at java.io.InputStreamReader.init(InputStreamReader.java:55) at org.apache.hadoop.mapreduce.v2.util.MRApps.setMRFrameworkClasspath(MRApps.java:183) at org.apache.hadoop.mapreduce.v2.util.MRApps.setClasspath(MRApps.java:220) at org.apache.hadoop.mapred.YARNRunner.createApplicationSubmissionContext(YARNRunner.java:360) at org.apache.hadoop.mapred.YARNRunner.submitJob(YARNRunner.java:237) at org.apache.hadoop.mapreduce.JobSubmitter.submitJobInternal(JobSubmitter.java:377) at org.apache.hadoop.mapreduce.Job$2.run(Job.java:1159) at org.apache.hadoop.mapreduce.Job$2.run(Job.java:1156) at java.security.AccessController.doPrivileged(Native Method) at javax.security.auth.Subject.doAs(Subject.java:396) at org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1152) at org.apache.hadoop.mapreduce.Job.submit(Job.java:1156) at org.apache.hadoop.mapred.JobClient.submitJob(JobClient.java:571) -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Resolved] (MAPREDUCE-3491) TestContainerManagerWithLCE is failing
[ https://issues.apache.org/jira/browse/MAPREDUCE-3491?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Allen Wittenauer resolved MAPREDUCE-3491. - Resolution: Fixed stale TestContainerManagerWithLCE is failing -- Key: MAPREDUCE-3491 URL: https://issues.apache.org/jira/browse/MAPREDUCE-3491 Project: Hadoop Map/Reduce Issue Type: Bug Components: task-controller Affects Versions: 0.23.0, 0.24.0 Reporter: Ravi Gummadi $ mvn test -Dtest=TestContainerManagerWithLCE -Dapplication.submitter=nobody -Dyarn.nodemanager.linux-container-executor.path=path of container-executor binary TestContainerManagerWithLCE is failing with the error: Test set: org.apache.hadoop.yarn.server.nodemanager.TestContainerManagerWithLCE --- Tests run: 6, Failures: 5, Errors: 0, Skipped: 0, Time elapsed: 26.219 sec FAILURE! testContainerSetup(org.apache.hadoop.yarn.server.nodemanager.TestContainerManagerWithLCE) Time elapsed: 2.476 sec FAILURE! junit.framework.AssertionFailedError: workspace/gitTrunk/hadoop-common/hadoop-mapreduce-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-nodemanager/target/org.apache.hadoop.yarn.server.nodemanager.TestContainerManagerWithLCE-localDir/usercache/nobody/appcache/application_0_ doesn't exist!! at junit.framework.Assert.fail(Assert.java:47) at junit.framework.Assert.assertTrue(Assert.java:20) at org.apache.hadoop.yarn.server.nodemanager.containermanager.TestContainerManager.testContainerSetup(TestContainerManager.java:179) at org.apache.hadoop.yarn.server.nodemanager.TestContainerManagerWithLCE.testContainerSetup(TestContainerManagerWithLCE.java:83) -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Updated] (MAPREDUCE-3401) Make single node secure cluster setup documentation for 0.23
[ https://issues.apache.org/jira/browse/MAPREDUCE-3401?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Allen Wittenauer updated MAPREDUCE-3401: Fix Version/s: (was: 0.24.0) Make single node secure cluster setup documentation for 0.23 Key: MAPREDUCE-3401 URL: https://issues.apache.org/jira/browse/MAPREDUCE-3401 Project: Hadoop Map/Reduce Issue Type: Task Components: documentation Reporter: Anupam Seth Assignee: Anupam Seth Priority: Minor This JIRA is to track creation of documentation for the setup of a secure single node cluster using 0.23. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Resolved] (MAPREDUCE-3781) Fix history for apps which were terminated before the AM launch
[ https://issues.apache.org/jira/browse/MAPREDUCE-3781?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Allen Wittenauer resolved MAPREDUCE-3781. - Resolution: Incomplete probably stale. Fix history for apps which were terminated before the AM launch --- Key: MAPREDUCE-3781 URL: https://issues.apache.org/jira/browse/MAPREDUCE-3781 Project: Hadoop Map/Reduce Issue Type: Bug Components: mrv2 Affects Versions: 0.23.1 Reporter: Ramya Sunil Currently the history for applications which were terminated/killed/failed before the AM was launched redirects to a page that does not exist. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Resolved] (MAPREDUCE-3777) used mem and util have negative values after a queue addition
[ https://issues.apache.org/jira/browse/MAPREDUCE-3777?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Allen Wittenauer resolved MAPREDUCE-3777. - Resolution: Fixed closing as stale. used mem and util have negative values after a queue addition - Key: MAPREDUCE-3777 URL: https://issues.apache.org/jira/browse/MAPREDUCE-3777 Project: Hadoop Map/Reduce Issue Type: Bug Components: mrv2 Affects Versions: 0.23.1 Reporter: Ramya Sunil Assignee: Arun C Murthy After a queue addition to capacity scheduler and submission of an application, root queue utilization and used memory have negative values. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Updated] (MAPREDUCE-3439) [Gridmix] Support addons in Gridmix
[ https://issues.apache.org/jira/browse/MAPREDUCE-3439?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Allen Wittenauer updated MAPREDUCE-3439: Fix Version/s: (was: 0.24.0) [Gridmix] Support addons in Gridmix --- Key: MAPREDUCE-3439 URL: https://issues.apache.org/jira/browse/MAPREDUCE-3439 Project: Hadoop Map/Reduce Issue Type: New Feature Components: contrib/gridmix Affects Versions: 0.24.0 Reporter: Amar Kamat Assignee: Amar Kamat Labels: addons, gridmix, gridmix3 At times there is a need to benchmark certain Hadoop client APIs. Often, this is done by running simple standard sort-like programs on Hadoop and then using an external utility to benchmark the APIs. But then the benchmarking results tend to be off from reality as the load on the cluster doesn't match the actual load. We believe that Gridmix3 - which is a Hadoop workload simulator - can prove useful here. Gridmix3 already provides a mechanism to load the cluster - often called as a 'test cluster' - using a real trace thus mimicking the real-life workload. Currently, Gridmix3 consumes a representative workload trace and loads the Hadoop cluster to match what is seen in the trace. Gridmix3 can be enhanced to also support user scripts (hereby referred as 'addons') which will be loaded within Gridmix3 and will get updates like 1. Job submission 2. Job completion 3. Cluster status These addons can also ping/access a live, close-to-real-life Hadoop cluster. This will allow users to benchmark the Hadoop cluster while it is running. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Updated] (MAPREDUCE-2659) MR-279: ShuffleHandler should use Protocol Buffers for ServiceData
[ https://issues.apache.org/jira/browse/MAPREDUCE-2659?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Allen Wittenauer updated MAPREDUCE-2659: Fix Version/s: (was: 0.24.0) MR-279: ShuffleHandler should use Protocol Buffers for ServiceData -- Key: MAPREDUCE-2659 URL: https://issues.apache.org/jira/browse/MAPREDUCE-2659 Project: Hadoop Map/Reduce Issue Type: Improvement Components: mrv2 Affects Versions: 0.23.0 Reporter: Robert Joseph Evans Assignee: Robert Joseph Evans Attachments: MR-2659-preview.patch Auxiliary Services (Specifically ShuffleHandler) should use ProtocolBuffers for storing/retrieving data in the ByteBuffer. Right now there are TODOs to have the format include a version number, but if we want true wire compatibility we should use the same system we are using elsewhere in the code for messages, not something invented as we go along. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Updated] (MAPREDUCE-2758) MR-279: Log info about the location of dist cache
[ https://issues.apache.org/jira/browse/MAPREDUCE-2758?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Allen Wittenauer updated MAPREDUCE-2758: Fix Version/s: (was: 0.24.0) MR-279: Log info about the location of dist cache - Key: MAPREDUCE-2758 URL: https://issues.apache.org/jira/browse/MAPREDUCE-2758 Project: Hadoop Map/Reduce Issue Type: Bug Components: mrv2 Affects Versions: 0.23.0 Reporter: Ramya Sunil Assignee: Siddharth Seth Priority: Minor Currently, there is no log info available about the actual location of the file/archive in dist cache being used by the task except for the ln command in task.sh. We need to log this information to help in debugging esp in those cases where there are more than one archive with the same name. In 0.20.x, in task logs, one could find log info such as the following: INFO org.apache.hadoop.mapred.TaskRunner: Creating symlink: distcache location/archive - mapred.local.dir/archive -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Updated] (MAPREDUCE-3293) Reason for application failure is not correctly reported
[ https://issues.apache.org/jira/browse/MAPREDUCE-3293?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Allen Wittenauer updated MAPREDUCE-3293: Fix Version/s: (was: 0.24.0) Reason for application failure is not correctly reported Key: MAPREDUCE-3293 URL: https://issues.apache.org/jira/browse/MAPREDUCE-3293 Project: Hadoop Map/Reduce Issue Type: Bug Components: mrv2 Affects Versions: 0.23.0 Reporter: Ramya Sunil When apps fail, the reason for failure is not correctly reflected in the UI. For one such app failure, the UI reports Application appID failed 1 times due to . Failing the application. which is not very helpful. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Updated] (MAPREDUCE-3294) Log the reason for killing a task during speculative execution
[ https://issues.apache.org/jira/browse/MAPREDUCE-3294?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Allen Wittenauer updated MAPREDUCE-3294: Fix Version/s: (was: 0.24.0) Log the reason for killing a task during speculative execution -- Key: MAPREDUCE-3294 URL: https://issues.apache.org/jira/browse/MAPREDUCE-3294 Project: Hadoop Map/Reduce Issue Type: Improvement Components: mrv2 Affects Versions: 0.23.0 Reporter: Ramya Sunil The reason for killing a speculated task has to be logged. Currently, a speculated task is killed with a note of Container killed by the ApplicationMaster. Container killed on request. Exit code is 137 which is not very useful. Better logging of this message stating the task was killed due to completion of its speculative task would be useful. Also, this message is lost once the app is moved to history. All we are left with is a list of killed tasks without a reason being notified to the user. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Resolved] (MAPREDUCE-3293) Reason for application failure is not correctly reported
[ https://issues.apache.org/jira/browse/MAPREDUCE-3293?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Allen Wittenauer resolved MAPREDUCE-3293. - Resolution: Incomplete closing this as stale/incomplete. Reason for application failure is not correctly reported Key: MAPREDUCE-3293 URL: https://issues.apache.org/jira/browse/MAPREDUCE-3293 Project: Hadoop Map/Reduce Issue Type: Bug Components: mrv2 Affects Versions: 0.23.0 Reporter: Ramya Sunil When apps fail, the reason for failure is not correctly reflected in the UI. For one such app failure, the UI reports Application appID failed 1 times due to . Failing the application. which is not very helpful. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Updated] (MAPREDUCE-3245) Write an integration test for validating MR AM restart and recovery
[ https://issues.apache.org/jira/browse/MAPREDUCE-3245?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Allen Wittenauer updated MAPREDUCE-3245: Issue Type: Test (was: Bug) Write an integration test for validating MR AM restart and recovery --- Key: MAPREDUCE-3245 URL: https://issues.apache.org/jira/browse/MAPREDUCE-3245 Project: Hadoop Map/Reduce Issue Type: Test Components: test Reporter: Vinod Kumar Vavilapalli This, so that we can catch bugs like MAPREDUCE-3233. We need one with recovery disabled i.e. for only restart and one for restart+recovery. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Updated] (MAPREDUCE-3335) rat check seems to be broken
[ https://issues.apache.org/jira/browse/MAPREDUCE-3335?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Allen Wittenauer updated MAPREDUCE-3335: Fix Version/s: (was: 0.24.0) rat check seems to be broken Key: MAPREDUCE-3335 URL: https://issues.apache.org/jira/browse/MAPREDUCE-3335 Project: Hadoop Map/Reduce Issue Type: Bug Components: build Affects Versions: 0.23.0 Reporter: Arun C Murthy The rat check seems broken, we don't get warned for files without license headers. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Updated] (MAPREDUCE-3401) Make single node secure cluster setup documentation for 0.23
[ https://issues.apache.org/jira/browse/MAPREDUCE-3401?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Allen Wittenauer updated MAPREDUCE-3401: Component/s: (was: mrv2) documentation Make single node secure cluster setup documentation for 0.23 Key: MAPREDUCE-3401 URL: https://issues.apache.org/jira/browse/MAPREDUCE-3401 Project: Hadoop Map/Reduce Issue Type: Task Components: documentation Reporter: Anupam Seth Assignee: Anupam Seth Priority: Minor This JIRA is to track creation of documentation for the setup of a secure single node cluster using 0.23. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Updated] (MAPREDUCE-3491) TestContainerManagerWithLCE is failing
[ https://issues.apache.org/jira/browse/MAPREDUCE-3491?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Allen Wittenauer updated MAPREDUCE-3491: Fix Version/s: (was: 0.24.0) TestContainerManagerWithLCE is failing -- Key: MAPREDUCE-3491 URL: https://issues.apache.org/jira/browse/MAPREDUCE-3491 Project: Hadoop Map/Reduce Issue Type: Bug Components: task-controller Affects Versions: 0.23.0, 0.24.0 Reporter: Ravi Gummadi $ mvn test -Dtest=TestContainerManagerWithLCE -Dapplication.submitter=nobody -Dyarn.nodemanager.linux-container-executor.path=path of container-executor binary TestContainerManagerWithLCE is failing with the error: Test set: org.apache.hadoop.yarn.server.nodemanager.TestContainerManagerWithLCE --- Tests run: 6, Failures: 5, Errors: 0, Skipped: 0, Time elapsed: 26.219 sec FAILURE! testContainerSetup(org.apache.hadoop.yarn.server.nodemanager.TestContainerManagerWithLCE) Time elapsed: 2.476 sec FAILURE! junit.framework.AssertionFailedError: workspace/gitTrunk/hadoop-common/hadoop-mapreduce-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-nodemanager/target/org.apache.hadoop.yarn.server.nodemanager.TestContainerManagerWithLCE-localDir/usercache/nobody/appcache/application_0_ doesn't exist!! at junit.framework.Assert.fail(Assert.java:47) at junit.framework.Assert.assertTrue(Assert.java:20) at org.apache.hadoop.yarn.server.nodemanager.containermanager.TestContainerManager.testContainerSetup(TestContainerManager.java:179) at org.apache.hadoop.yarn.server.nodemanager.TestContainerManagerWithLCE.testContainerSetup(TestContainerManagerWithLCE.java:83) -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Resolved] (MAPREDUCE-3390) NPE while submitting job
[ https://issues.apache.org/jira/browse/MAPREDUCE-3390?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Allen Wittenauer resolved MAPREDUCE-3390. - Resolution: Fixed stale NPE while submitting job Key: MAPREDUCE-3390 URL: https://issues.apache.org/jira/browse/MAPREDUCE-3390 Project: Hadoop Map/Reduce Issue Type: Bug Components: mrv2 Affects Versions: 0.23.0 Environment: This happens only when there is something weird going on like stale jars or something similar. Reporter: John George Assignee: John George Priority: Minor Attachments: MAPREDUCE-3390-20111220.txt, MAPREDUCE-3390.branch-0.23.patch Caused by: java.lang.NullPointerException at java.io.Reader.init(Reader.java:61) at java.io.InputStreamReader.init(InputStreamReader.java:55) at org.apache.hadoop.mapreduce.v2.util.MRApps.setMRFrameworkClasspath(MRApps.java:183) at org.apache.hadoop.mapreduce.v2.util.MRApps.setClasspath(MRApps.java:220) at org.apache.hadoop.mapred.YARNRunner.createApplicationSubmissionContext(YARNRunner.java:360) at org.apache.hadoop.mapred.YARNRunner.submitJob(YARNRunner.java:237) at org.apache.hadoop.mapreduce.JobSubmitter.submitJobInternal(JobSubmitter.java:377) at org.apache.hadoop.mapreduce.Job$2.run(Job.java:1159) at org.apache.hadoop.mapreduce.Job$2.run(Job.java:1156) at java.security.AccessController.doPrivileged(Native Method) at javax.security.auth.Subject.doAs(Subject.java:396) at org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1152) at org.apache.hadoop.mapreduce.Job.submit(Job.java:1156) at org.apache.hadoop.mapred.JobClient.submitJob(JobClient.java:571) -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Updated] (MAPREDUCE-3805) MR AM not respecting MaxReduceRampUpLimit
[ https://issues.apache.org/jira/browse/MAPREDUCE-3805?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Allen Wittenauer updated MAPREDUCE-3805: Fix Version/s: (was: 0.24.0) MR AM not respecting MaxReduceRampUpLimit - Key: MAPREDUCE-3805 URL: https://issues.apache.org/jira/browse/MAPREDUCE-3805 Project: Hadoop Map/Reduce Issue Type: Bug Components: mr-am, mrv2 Affects Versions: 0.23.0 Reporter: Vinod Kumar Vavilapalli Assignee: Vinod Kumar Vavilapalli Attachments: MAPREDUCE-3805-20120203.txt While running GridMixV3 with high memory reduces, we ran into issues where for jobs with significant number of maps and reduces, when the map progress hits 98-99% but still there are maps pending, reduces get every new container that RM allocates. And the job takes much longer time than with usual reduces. For addressing precisely these issues, a configurable limit was introduced to limit the reduce ramp up. Unfortunately this limit is not working correctly. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Resolved] (MAPREDUCE-2888) saveVersion.sh doesn't work when svn copy is staged
[ https://issues.apache.org/jira/browse/MAPREDUCE-2888?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Allen Wittenauer resolved MAPREDUCE-2888. - Resolution: Fixed switched to git. closing as stale. saveVersion.sh doesn't work when svn copy is staged --- Key: MAPREDUCE-2888 URL: https://issues.apache.org/jira/browse/MAPREDUCE-2888 Project: Hadoop Map/Reduce Issue Type: Bug Components: build Affects Versions: 0.23.0 Reporter: Todd Lipcon Assignee: Todd Lipcon Priority: Trivial Attachments: saveVersion.patch The build fails with an error on the sed command, since saveVersion.sh doesn't correctly grab the URL. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Updated] (MAPREDUCE-3580) [Rumen] Rumen anonymizer should also parse and anonymize other job properties like 'mapreduce.output.fileoutputformat.compress.codec' etc
[ https://issues.apache.org/jira/browse/MAPREDUCE-3580?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Allen Wittenauer updated MAPREDUCE-3580: Affects Version/s: 3.0.0 [Rumen] Rumen anonymizer should also parse and anonymize other job properties like 'mapreduce.output.fileoutputformat.compress.codec' etc - Key: MAPREDUCE-3580 URL: https://issues.apache.org/jira/browse/MAPREDUCE-3580 Project: Hadoop Map/Reduce Issue Type: Improvement Components: security, tools/rumen Affects Versions: 3.0.0 Reporter: Amar Kamat Assignee: Amar Kamat Labels: anonymization, job-properties, rumen Currently, the Rumen anonymizer only parses and anonymizes job properties enumerated in MRJobConfig. Other properties like 1. compression codecs 2. output files 3. Task and Attempt level properties should also be considered. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Updated] (MAPREDUCE-2745) [MR-279] NM UI should get a read-only view instead of the actual NMContext
[ https://issues.apache.org/jira/browse/MAPREDUCE-2745?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Allen Wittenauer updated MAPREDUCE-2745: Fix Version/s: (was: 0.24.0) [MR-279] NM UI should get a read-only view instead of the actual NMContext --- Key: MAPREDUCE-2745 URL: https://issues.apache.org/jira/browse/MAPREDUCE-2745 Project: Hadoop Map/Reduce Issue Type: Bug Components: mrv2 Affects Versions: 0.23.0 Reporter: Vinod Kumar Vavilapalli Assignee: Anupam Seth Priority: Trivial Labels: newbie Attachments: MAPREDUCE-2745-branch-0_23.patch, MAPREDUCE-2745-branch-0_23_v2.patch NMContext is modifiable, the UI should only get read-only access. Just like the AM web-ui. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Updated] (MAPREDUCE-3074) add location to web UI so you know where you are - cluster, node, AM, job history
[ https://issues.apache.org/jira/browse/MAPREDUCE-3074?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Allen Wittenauer updated MAPREDUCE-3074: Fix Version/s: (was: 0.24.0) add location to web UI so you know where you are - cluster, node, AM, job history - Key: MAPREDUCE-3074 URL: https://issues.apache.org/jira/browse/MAPREDUCE-3074 Project: Hadoop Map/Reduce Issue Type: Improvement Affects Versions: 3.0.0 Reporter: Thomas Graves Right now if you go to any of the web UIs for resource manager, node manager, app master, or job history, they look very similar but sometimes it hard to tell which page you are. Adding a title or something that lets you know would be helpful. Or somehow make them more seemless so one doesn't have to know. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Resolved] (MAPREDUCE-2842) Maven build issues in MR2
[ https://issues.apache.org/jira/browse/MAPREDUCE-2842?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Allen Wittenauer resolved MAPREDUCE-2842. - Resolution: Fixed Stale or fixed. take your pick. still closing this. Maven build issues in MR2 -- Key: MAPREDUCE-2842 URL: https://issues.apache.org/jira/browse/MAPREDUCE-2842 Project: Hadoop Map/Reduce Issue Type: Bug Components: build, mrv2 Affects Versions: 0.23.0 Reporter: Alejandro Abdelnur Labels: maven * mapreduce has not been rebased on top of trunk * mapreduce dir/module should be named hadoop-mapreduce (following convention of common, hdfs) * there is lot of stuff under mapreduce that seems stale (bin/, conf, ivy/, lib/ src/) * yarn* dirs/modules should be named hadoop-yarn* (following convention of other Hadoop artifacts) * yarn/bin/ scripts should be under yarn/src/main/bin * yarn/conf/ scripts should be under yarn/src/main/conf * JAR POM files do not use hadoop-project POM as parent * some POM files have version parameterized and this will break things for people consuming JARs from Maven repos * mapreduce is not using assembly from hadoop-assemblies (the changes introduced by HDFS-2096 make the assembly/packaging reusable across different components) -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Updated] (MAPREDUCE-2967) PB implementations of all records should have getters, setters and getProto() properly synchronized
[ https://issues.apache.org/jira/browse/MAPREDUCE-2967?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Allen Wittenauer updated MAPREDUCE-2967: Fix Version/s: (was: 0.24.0) PB implementations of all records should have getters, setters and getProto() properly synchronized --- Key: MAPREDUCE-2967 URL: https://issues.apache.org/jira/browse/MAPREDUCE-2967 Project: Hadoop Map/Reduce Issue Type: Bug Components: mrv2 Affects Versions: 0.23.0, 3.0.0 Reporter: Vinod Kumar Vavilapalli Assignee: Vinod Kumar Vavilapalli In the past, I've been bitten by multiple, very hard-to-debug race conditions with YARN+MR which all boiled down to locking bugs in PB implementation of various records. The main reason seems to be that while the _getProto()_ method in each record rebuilds the protocol object, if someone accesses a filed using a getter, it returns zero or a null object. Because of this, while _getProto()_ is in progress, _hashCode()_, _equals()_, _toString()_ etc are all affected, leading to hard-to-debug races. I corned this down after much logging in almost all the cases. Over time, we've fixed this for most of the ID records. This JIRA should fix it for *all* the records once and for all. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Updated] (MAPREDUCE-2967) PB implementations of all records should have getters, setters and getProto() properly synchronized
[ https://issues.apache.org/jira/browse/MAPREDUCE-2967?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Allen Wittenauer updated MAPREDUCE-2967: Affects Version/s: 3.0.0 PB implementations of all records should have getters, setters and getProto() properly synchronized --- Key: MAPREDUCE-2967 URL: https://issues.apache.org/jira/browse/MAPREDUCE-2967 Project: Hadoop Map/Reduce Issue Type: Bug Components: mrv2 Affects Versions: 0.23.0, 3.0.0 Reporter: Vinod Kumar Vavilapalli Assignee: Vinod Kumar Vavilapalli In the past, I've been bitten by multiple, very hard-to-debug race conditions with YARN+MR which all boiled down to locking bugs in PB implementation of various records. The main reason seems to be that while the _getProto()_ method in each record rebuilds the protocol object, if someone accesses a filed using a getter, it returns zero or a null object. Because of this, while _getProto()_ is in progress, _hashCode()_, _equals()_, _toString()_ etc are all affected, leading to hard-to-debug races. I corned this down after much logging in almost all the cases. Over time, we've fixed this for most of the ID records. This JIRA should fix it for *all* the records once and for all. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Updated] (MAPREDUCE-3074) add location to web UI so you know where you are - cluster, node, AM, job history
[ https://issues.apache.org/jira/browse/MAPREDUCE-3074?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Allen Wittenauer updated MAPREDUCE-3074: Affects Version/s: 3.0.0 add location to web UI so you know where you are - cluster, node, AM, job history - Key: MAPREDUCE-3074 URL: https://issues.apache.org/jira/browse/MAPREDUCE-3074 Project: Hadoop Map/Reduce Issue Type: Improvement Affects Versions: 3.0.0 Reporter: Thomas Graves Right now if you go to any of the web UIs for resource manager, node manager, app master, or job history, they look very similar but sometimes it hard to tell which page you are. Adding a title or something that lets you know would be helpful. Or somehow make them more seemless so one doesn't have to know. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Updated] (MAPREDUCE-2967) PB implementations of all records should have getters, setters and getProto() properly synchronized
[ https://issues.apache.org/jira/browse/MAPREDUCE-2967?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Allen Wittenauer updated MAPREDUCE-2967: Labels: newbie (was: ) PB implementations of all records should have getters, setters and getProto() properly synchronized --- Key: MAPREDUCE-2967 URL: https://issues.apache.org/jira/browse/MAPREDUCE-2967 Project: Hadoop Map/Reduce Issue Type: Bug Components: mrv2 Affects Versions: 0.23.0, 3.0.0 Reporter: Vinod Kumar Vavilapalli Assignee: Vinod Kumar Vavilapalli Labels: newbie In the past, I've been bitten by multiple, very hard-to-debug race conditions with YARN+MR which all boiled down to locking bugs in PB implementation of various records. The main reason seems to be that while the _getProto()_ method in each record rebuilds the protocol object, if someone accesses a filed using a getter, it returns zero or a null object. Because of this, while _getProto()_ is in progress, _hashCode()_, _equals()_, _toString()_ etc are all affected, leading to hard-to-debug races. I corned this down after much logging in almost all the cases. Over time, we've fixed this for most of the ID records. This JIRA should fix it for *all* the records once and for all. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Updated] (MAPREDUCE-3777) used mem and util have negative values after a queue addition
[ https://issues.apache.org/jira/browse/MAPREDUCE-3777?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Allen Wittenauer updated MAPREDUCE-3777: Fix Version/s: (was: 0.24.0) used mem and util have negative values after a queue addition - Key: MAPREDUCE-3777 URL: https://issues.apache.org/jira/browse/MAPREDUCE-3777 Project: Hadoop Map/Reduce Issue Type: Bug Components: mrv2 Affects Versions: 0.23.1 Reporter: Ramya Sunil Assignee: Arun C Murthy After a queue addition to capacity scheduler and submission of an application, root queue utilization and used memory have negative values. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Resolved] (MAPREDUCE-3632) Need better error message on the Web UI when NM can't find the container logs instead of NPEno
[ https://issues.apache.org/jira/browse/MAPREDUCE-3632?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Allen Wittenauer resolved MAPREDUCE-3632. - Resolution: Fixed probably stale/fixed Need better error message on the Web UI when NM can't find the container logs instead of NPEno -- Key: MAPREDUCE-3632 URL: https://issues.apache.org/jira/browse/MAPREDUCE-3632 Project: Hadoop Map/Reduce Issue Type: Bug Components: nodemanager Affects Versions: 0.23.0 Reporter: Ravi Gummadi Assignee: Ravi Gummadi Attachments: 3632.patch If for some reason NM could not find container logs, then an NPE is seen while trying to access from web UI. Instead an error message should be displayed. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Updated] (MAPREDUCE-3632) Need better error message on the Web UI when NM can't find the container logs instead of NPEno
[ https://issues.apache.org/jira/browse/MAPREDUCE-3632?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Allen Wittenauer updated MAPREDUCE-3632: Fix Version/s: (was: 0.24.0) Need better error message on the Web UI when NM can't find the container logs instead of NPEno -- Key: MAPREDUCE-3632 URL: https://issues.apache.org/jira/browse/MAPREDUCE-3632 Project: Hadoop Map/Reduce Issue Type: Bug Components: nodemanager Affects Versions: 0.23.0 Reporter: Ravi Gummadi Assignee: Ravi Gummadi Attachments: 3632.patch If for some reason NM could not find container logs, then an NPE is seen while trying to access from web UI. Instead an error message should be displayed. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Updated] (MAPREDUCE-2888) saveVersion.sh doesn't work when svn copy is staged
[ https://issues.apache.org/jira/browse/MAPREDUCE-2888?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Allen Wittenauer updated MAPREDUCE-2888: Fix Version/s: (was: 0.24.0) saveVersion.sh doesn't work when svn copy is staged --- Key: MAPREDUCE-2888 URL: https://issues.apache.org/jira/browse/MAPREDUCE-2888 Project: Hadoop Map/Reduce Issue Type: Bug Components: build Affects Versions: 0.23.0 Reporter: Todd Lipcon Assignee: Todd Lipcon Priority: Trivial Attachments: saveVersion.patch The build fails with an error on the sed command, since saveVersion.sh doesn't correctly grab the URL. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Updated] (MAPREDUCE-3781) Fix history for apps which were terminated before the AM launch
[ https://issues.apache.org/jira/browse/MAPREDUCE-3781?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Allen Wittenauer updated MAPREDUCE-3781: Fix Version/s: (was: 0.24.0) Fix history for apps which were terminated before the AM launch --- Key: MAPREDUCE-3781 URL: https://issues.apache.org/jira/browse/MAPREDUCE-3781 Project: Hadoop Map/Reduce Issue Type: Bug Components: mrv2 Affects Versions: 0.23.1 Reporter: Ramya Sunil Currently the history for applications which were terminated/killed/failed before the AM was launched redirects to a page that does not exist. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Updated] (MAPREDUCE-3581) [Rumen] Rumen anonymizer should handle composite string data
[
https://issues.apache.org/jira/browse/MAPREDUCE-3581?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Allen Wittenauer updated MAPREDUCE-3581:
Component/s: security
[Rumen] Rumen anonymizer should handle composite string data
Key: MAPREDUCE-3581
URL: https://issues.apache.org/jira/browse/MAPREDUCE-3581
Project: Hadoop Map/Reduce
Issue Type: Improvement
Components: security, tools/rumen
Affects Versions: 3.0.0
Reporter: Amar Kamat
Assignee: Amar Kamat
Labels: anonymization, chunking, rumen
Rumen's Anonymizer currently considers string as a single entity. At times,
strings can be composed of smaller sub-strings which can be anonymized
individually. Anonymizing sub-strings separately will result in retaining
certain statistics like frequency ('daily', 'weekly' etc). This was brought
up by Chris while developing the Anonymizer.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Updated] (MAPREDUCE-3580) [Rumen] Rumen anonymizer should also parse and anonymize other job properties like 'mapreduce.output.fileoutputformat.compress.codec' etc
[ https://issues.apache.org/jira/browse/MAPREDUCE-3580?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Allen Wittenauer updated MAPREDUCE-3580: Fix Version/s: (was: 0.24.0) [Rumen] Rumen anonymizer should also parse and anonymize other job properties like 'mapreduce.output.fileoutputformat.compress.codec' etc - Key: MAPREDUCE-3580 URL: https://issues.apache.org/jira/browse/MAPREDUCE-3580 Project: Hadoop Map/Reduce Issue Type: Improvement Components: security, tools/rumen Reporter: Amar Kamat Assignee: Amar Kamat Labels: anonymization, job-properties, rumen Currently, the Rumen anonymizer only parses and anonymizes job properties enumerated in MRJobConfig. Other properties like 1. compression codecs 2. output files 3. Task and Attempt level properties should also be considered. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Updated] (MAPREDUCE-3581) [Rumen] Rumen anonymizer should handle composite string data
[
https://issues.apache.org/jira/browse/MAPREDUCE-3581?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Allen Wittenauer updated MAPREDUCE-3581:
Affects Version/s: (was: 0.24.0)
3.0.0
[Rumen] Rumen anonymizer should handle composite string data
Key: MAPREDUCE-3581
URL: https://issues.apache.org/jira/browse/MAPREDUCE-3581
Project: Hadoop Map/Reduce
Issue Type: Improvement
Components: security, tools/rumen
Affects Versions: 3.0.0
Reporter: Amar Kamat
Assignee: Amar Kamat
Labels: anonymization, chunking, rumen
Rumen's Anonymizer currently considers string as a single entity. At times,
strings can be composed of smaller sub-strings which can be anonymized
individually. Anonymizing sub-strings separately will result in retaining
certain statistics like frequency ('daily', 'weekly' etc). This was brought
up by Chris while developing the Anonymizer.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Updated] (MAPREDUCE-3508) [Rumen] Rumen should provide simple trace filtering capabilities
[ https://issues.apache.org/jira/browse/MAPREDUCE-3508?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Allen Wittenauer updated MAPREDUCE-3508: Fix Version/s: (was: 0.24.0) [Rumen] Rumen should provide simple trace filtering capabilities Key: MAPREDUCE-3508 URL: https://issues.apache.org/jira/browse/MAPREDUCE-3508 Project: Hadoop Map/Reduce Issue Type: New Feature Components: tools/rumen Reporter: Amar Kamat Assignee: Amar Kamat Labels: job-filter, rumen Rumen should provide inbuilt tools to filter jobs from a given trace. Following are the usecases: 1. Select only first k jobs. 2. Select jobs with certain configuration keys set or available 3. Select jobs where the original job id matches the specified list 4. Select jobs which have at-least/at-most x map tasks and/or at-least/at-most y reduce tasks. 5. Select jobs belonging to a specific user(s) 6. Select jobs having specific name(s) and so on. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Updated] (MAPREDUCE-3439) [Gridmix] Support addons in Gridmix
[ https://issues.apache.org/jira/browse/MAPREDUCE-3439?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Allen Wittenauer updated MAPREDUCE-3439: Target Version/s: (was: 0.24.0) [Gridmix] Support addons in Gridmix --- Key: MAPREDUCE-3439 URL: https://issues.apache.org/jira/browse/MAPREDUCE-3439 Project: Hadoop Map/Reduce Issue Type: New Feature Components: contrib/gridmix Affects Versions: 0.24.0 Reporter: Amar Kamat Assignee: Amar Kamat Labels: addons, gridmix, gridmix3 At times there is a need to benchmark certain Hadoop client APIs. Often, this is done by running simple standard sort-like programs on Hadoop and then using an external utility to benchmark the APIs. But then the benchmarking results tend to be off from reality as the load on the cluster doesn't match the actual load. We believe that Gridmix3 - which is a Hadoop workload simulator - can prove useful here. Gridmix3 already provides a mechanism to load the cluster - often called as a 'test cluster' - using a real trace thus mimicking the real-life workload. Currently, Gridmix3 consumes a representative workload trace and loads the Hadoop cluster to match what is seen in the trace. Gridmix3 can be enhanced to also support user scripts (hereby referred as 'addons') which will be loaded within Gridmix3 and will get updates like 1. Job submission 2. Job completion 3. Cluster status These addons can also ping/access a live, close-to-real-life Hadoop cluster. This will allow users to benchmark the Hadoop cluster while it is running. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Resolved] (MAPREDUCE-3806) [Gridmix] TestGridmixSubmission fails due to incorrect version of jackson
[
https://issues.apache.org/jira/browse/MAPREDUCE-3806?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Allen Wittenauer resolved MAPREDUCE-3806.
-
Resolution: Cannot Reproduce
stale
[Gridmix] TestGridmixSubmission fails due to incorrect version of jackson
-
Key: MAPREDUCE-3806
URL: https://issues.apache.org/jira/browse/MAPREDUCE-3806
Project: Hadoop Map/Reduce
Issue Type: Bug
Components: contrib/gridmix
Affects Versions: 0.24.0
Reporter: Amar Kamat
Labels: error, gridmix, junit
{{TestGridmixSubmission}} fails with the following error
{code}
org.codehaus.jackson.type.JavaType.isMapLikeType()Z
java.lang.NoSuchMethodError:
org.codehaus.jackson.type.JavaType.isMapLikeType()Z
at
org.codehaus.jackson.map.deser.StdDeserializerProvider._createDeserializer(StdDeserializerProvider.java:374)
at
org.codehaus.jackson.map.deser.StdDeserializerProvider._createAndCache2(StdDeserializerProvider.java:307)
at
org.codehaus.jackson.map.deser.StdDeserializerProvider._createAndCacheValueDeserializer(StdDeserializerProvider.java:287)
at
org.codehaus.jackson.map.deser.StdDeserializerProvider.findValueDeserializer(StdDeserializerProvider.java:136)
at
org.codehaus.jackson.map.deser.StdDeserializer.findDeserializer(StdDeserializer.java:551)
at
org.codehaus.jackson.map.deser.BeanDeserializer.resolve(BeanDeserializer.java:268)
at
org.codehaus.jackson.map.deser.StdDeserializerProvider._resolveDeserializer(StdDeserializerProvider.java:404)
at
org.codehaus.jackson.map.deser.StdDeserializerProvider._createAndCache2(StdDeserializerProvider.java:349)
at
org.codehaus.jackson.map.deser.StdDeserializerProvider._createAndCacheValueDeserializer(StdDeserializerProvider.java:287)
at
org.codehaus.jackson.map.deser.StdDeserializerProvider.findValueDeserializer(StdDeserializerProvider.java:136)
at
org.codehaus.jackson.map.deser.StdDeserializerProvider.findTypedValueDeserializer(StdDeserializerProvider.java:157)
at
org.codehaus.jackson.map.ObjectMapper._findRootDeserializer(ObjectMapper.java:2468)
at
org.codehaus.jackson.map.ObjectMapper._readValue(ObjectMapper.java:2383)
at
org.codehaus.jackson.map.ObjectMapper.readValue(ObjectMapper.java:1094)
at
org.apache.hadoop.tools.rumen.JsonObjectMapperParser.getNext(JsonObjectMapperParser.java:84)
at
org.apache.hadoop.tools.rumen.ZombieJobProducer.getNextJob(ZombieJobProducer.java:117)
at
org.apache.hadoop.tools.rumen.ZombieJobProducer.getNextJob(ZombieJobProducer.java:29)
at
org.apache.hadoop.mapred.gridmix.TestGridmixSubmission.testTraceReader(TestGridmixSubmission.java:440)
{code}
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Updated] (MAPREDUCE-3075) Web UI menu inconsistencies
[ https://issues.apache.org/jira/browse/MAPREDUCE-3075?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Allen Wittenauer updated MAPREDUCE-3075: Fix Version/s: (was: 0.24.0) Web UI menu inconsistencies --- Key: MAPREDUCE-3075 URL: https://issues.apache.org/jira/browse/MAPREDUCE-3075 Project: Hadoop Map/Reduce Issue Type: Improvement Affects Versions: 3.0.0 Reporter: Thomas Graves When you go to the various web UI's the menus on the left are inconsistent and (atleast to me) sometimes confusing. For instance if you go to the application master UI, one of the menus is Cluster. If you click on one of the Cluster links it takes you back to the RM ui and you lose the app master UI altogether. Maybe its just me but that is confusing. I like having a link back to the cluster from AM but the way the UI is setup I would have expected it to just open that page in the middle div/frame and leave the AM menus there. Perhaps a different type of link or menu to indicate this is going to take you away from AM page. Also, the nodes and job history UI don't have the Cluster menus at all. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Updated] (MAPREDUCE-3245) Write an integration test for validating MR AM restart and recovery
[ https://issues.apache.org/jira/browse/MAPREDUCE-3245?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Allen Wittenauer updated MAPREDUCE-3245: Fix Version/s: (was: 0.24.0) Write an integration test for validating MR AM restart and recovery --- Key: MAPREDUCE-3245 URL: https://issues.apache.org/jira/browse/MAPREDUCE-3245 Project: Hadoop Map/Reduce Issue Type: Test Components: test Reporter: Vinod Kumar Vavilapalli This, so that we can catch bugs like MAPREDUCE-3233. We need one with recovery disabled i.e. for only restart and one for restart+recovery. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Updated] (MAPREDUCE-3340) Deprecate Job.setJobSetupCleanupNeeded()
[ https://issues.apache.org/jira/browse/MAPREDUCE-3340?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Allen Wittenauer updated MAPREDUCE-3340: Affects Version/s: 3.0.0 Deprecate Job.setJobSetupCleanupNeeded() Key: MAPREDUCE-3340 URL: https://issues.apache.org/jira/browse/MAPREDUCE-3340 Project: Hadoop Map/Reduce Issue Type: Bug Components: mrv2 Affects Versions: 0.23.0, 3.0.0 Reporter: Vinod Kumar Vavilapalli Assignee: Vinod Kumar Vavilapalli We should deprecate the setJobSetupCleanupNeeded() API. It was originally added for performance reasons to avoid launching new JVMs altogether for job-setup and job-cleanup. With Yarn and MRAppMaster, setup and cleanup are run inside the AM itself and so nothing much can be gained by making them optional. Before 0.23, we could disable set up and cleanup, yet obtain the output when using FileOutputCommitter in the job-output directory. But post 0.23.0, that won't be the case because of the nested temporary directories to support AM recoverability. So it makes sense to *not* have cleanupJob optional. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Resolved] (MAPREDUCE-3335) rat check seems to be broken
[ https://issues.apache.org/jira/browse/MAPREDUCE-3335?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Allen Wittenauer resolved MAPREDUCE-3335. - Resolution: Won't Fix closing as stale rat check seems to be broken Key: MAPREDUCE-3335 URL: https://issues.apache.org/jira/browse/MAPREDUCE-3335 Project: Hadoop Map/Reduce Issue Type: Bug Components: build Affects Versions: 0.23.0 Reporter: Arun C Murthy The rat check seems broken, we don't get warned for files without license headers. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Updated] (MAPREDUCE-3587) The deployment tarball should have different directories for yarn jars and mapreduce jars.
[ https://issues.apache.org/jira/browse/MAPREDUCE-3587?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Allen Wittenauer updated MAPREDUCE-3587: Fix Version/s: (was: 0.24.0) The deployment tarball should have different directories for yarn jars and mapreduce jars. -- Key: MAPREDUCE-3587 URL: https://issues.apache.org/jira/browse/MAPREDUCE-3587 Project: Hadoop Map/Reduce Issue Type: Bug Components: mrv2 Reporter: Mahadev konar Currently all the jars in the mr tarball go to share/hadoop/mapreduce. The jars should be split into: share/hadoop/yarn and share/hadoop/mapreduce for clear seperation between yarn framework and mr. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Resolved] (MAPREDUCE-3587) The deployment tarball should have different directories for yarn jars and mapreduce jars.
[ https://issues.apache.org/jira/browse/MAPREDUCE-3587?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Allen Wittenauer resolved MAPREDUCE-3587. - Resolution: Fixed stale/fixed The deployment tarball should have different directories for yarn jars and mapreduce jars. -- Key: MAPREDUCE-3587 URL: https://issues.apache.org/jira/browse/MAPREDUCE-3587 Project: Hadoop Map/Reduce Issue Type: Bug Components: mrv2 Reporter: Mahadev konar Currently all the jars in the mr tarball go to share/hadoop/mapreduce. The jars should be split into: share/hadoop/yarn and share/hadoop/mapreduce for clear seperation between yarn framework and mr. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Updated] (MAPREDUCE-3887) Jenkins mapred commit build tries an unknown target
[
https://issues.apache.org/jira/browse/MAPREDUCE-3887?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Allen Wittenauer updated MAPREDUCE-3887:
Fix Version/s: (was: 0.24.0)
Jenkins mapred commit build tries an unknown target
---
Key: MAPREDUCE-3887
URL: https://issues.apache.org/jira/browse/MAPREDUCE-3887
Project: Hadoop Map/Reduce
Issue Type: Bug
Components: build
Reporter: Kihwal Lee
I saw the following in the mrv1 ant build portion of
Hadoop-Mapreduce-trunk-Commit. The 0.23 build might have the same thing.
{panel}
+ /home/jenkins/tools/ant/latest/bin/ant -Dversion=0.24.0-SNAPSHOT
-Dresolvers=internal -Declipse.home=/home/jenkins/tools/eclipse/latest
-Dfindbugs.home=/home/jenkins/tools/findbugs/latest
-Dforrest.home=/home/jenkins/tools/forrest/latest -Dcompile.c++=true
-Dcompile.native=true create-c++-configure binary
BUILD FAILED
Target binary does not exist in the project Hadoop.
{panel}
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Updated] (MAPREDUCE-3925) [Gridmix] Gridmix stress mode should be queue aware
[ https://issues.apache.org/jira/browse/MAPREDUCE-3925?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Allen Wittenauer updated MAPREDUCE-3925: Target Version/s: 3.0.0 (was: 0.24.0) [Gridmix] Gridmix stress mode should be queue aware --- Key: MAPREDUCE-3925 URL: https://issues.apache.org/jira/browse/MAPREDUCE-3925 Project: Hadoop Map/Reduce Issue Type: Improvement Components: contrib/gridmix Affects Versions: 3.0.0 Reporter: Amar Kamat Labels: gridmix, multi-queue, stress Currently, the Gridmix stress mode submits jobs in the same order as seen in the trace. When Gridmix is configured to run with multiple queues, the stress mode might end up queuing lot of jobs in a single queue without really stressing the entire cluster. The goal is to make sure that each queue is loaded thus keeping the entire cluster busy. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Updated] (MAPREDUCE-3925) [Gridmix] Gridmix stress mode should be queue aware
[ https://issues.apache.org/jira/browse/MAPREDUCE-3925?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Allen Wittenauer updated MAPREDUCE-3925: Fix Version/s: (was: 0.24.0) [Gridmix] Gridmix stress mode should be queue aware --- Key: MAPREDUCE-3925 URL: https://issues.apache.org/jira/browse/MAPREDUCE-3925 Project: Hadoop Map/Reduce Issue Type: Improvement Components: contrib/gridmix Affects Versions: 0.24.0 Reporter: Amar Kamat Labels: gridmix, multi-queue, stress Currently, the Gridmix stress mode submits jobs in the same order as seen in the trace. When Gridmix is configured to run with multiple queues, the stress mode might end up queuing lot of jobs in a single queue without really stressing the entire cluster. The goal is to make sure that each queue is loaded thus keeping the entire cluster busy. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Updated] (MAPREDUCE-2798) Make the ResourceManager, NodeManager and HistoryServer run from Eclipse.
[ https://issues.apache.org/jira/browse/MAPREDUCE-2798?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Allen Wittenauer updated MAPREDUCE-2798: Fix Version/s: (was: 0.24.0) Make the ResourceManager, NodeManager and HistoryServer run from Eclipse. - Key: MAPREDUCE-2798 URL: https://issues.apache.org/jira/browse/MAPREDUCE-2798 Project: Hadoop Map/Reduce Issue Type: New Feature Affects Versions: 3.0.0 Reporter: praveen sripati Priority: Minor Make the ResourceManager, NodeManager and HistoryServer run from Eclipse, so that it would be easy for development. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Updated] (MAPREDUCE-3581) [Rumen] Rumen anonymizer should handle composite string data
[
https://issues.apache.org/jira/browse/MAPREDUCE-3581?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Allen Wittenauer updated MAPREDUCE-3581:
Target Version/s: 3.0.0 (was: 0.24.0)
[Rumen] Rumen anonymizer should handle composite string data
Key: MAPREDUCE-3581
URL: https://issues.apache.org/jira/browse/MAPREDUCE-3581
Project: Hadoop Map/Reduce
Issue Type: Improvement
Components: security, tools/rumen
Affects Versions: 3.0.0
Reporter: Amar Kamat
Assignee: Amar Kamat
Labels: anonymization, chunking, rumen
Rumen's Anonymizer currently considers string as a single entity. At times,
strings can be composed of smaller sub-strings which can be anonymized
individually. Anonymizing sub-strings separately will result in retaining
certain statistics like frequency ('daily', 'weekly' etc). This was brought
up by Chris while developing the Anonymizer.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Updated] (MAPREDUCE-3581) [Rumen] Rumen anonymizer should handle composite string data
[
https://issues.apache.org/jira/browse/MAPREDUCE-3581?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Allen Wittenauer updated MAPREDUCE-3581:
Fix Version/s: (was: 0.24.0)
[Rumen] Rumen anonymizer should handle composite string data
Key: MAPREDUCE-3581
URL: https://issues.apache.org/jira/browse/MAPREDUCE-3581
Project: Hadoop Map/Reduce
Issue Type: Improvement
Components: security, tools/rumen
Affects Versions: 3.0.0
Reporter: Amar Kamat
Assignee: Amar Kamat
Labels: anonymization, chunking, rumen
Rumen's Anonymizer currently considers string as a single entity. At times,
strings can be composed of smaller sub-strings which can be anonymized
individually. Anonymizing sub-strings separately will result in retaining
certain statistics like frequency ('daily', 'weekly' etc). This was brought
up by Chris while developing the Anonymizer.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Updated] (MAPREDUCE-2553) missing space in the error message of distcp command
[ https://issues.apache.org/jira/browse/MAPREDUCE-2553?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Allen Wittenauer updated MAPREDUCE-2553: Resolution: Won't Fix Status: Resolved (was: Patch Available) distcp v1 is no longer in trunk. closing as won't fix. missing space in the error message of distcp command Key: MAPREDUCE-2553 URL: https://issues.apache.org/jira/browse/MAPREDUCE-2553 Project: Hadoop Map/Reduce Issue Type: Improvement Components: distcp Affects Versions: 0.23.0 Reporter: Wei Yongjun Priority: Minor Attachments: MAPREDUCE-2553.patch Error message of distcp command missing space, here's an example output: $ hadoop distcp /user/test/input /user/test/ouput file:/root/temp ... Copy failed: java.io.IOException: Failed to createfile:/root/temp ... Generated from this line of source. src/tools/org/apache/hadoop/tools/DistCp.java: throw new IOException(Failed to create + args.dst); Failed to create should end with a ' '. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Updated] (MAPREDUCE-3131) Docs and Scripts for setting up single node MRV2 cluster.
[ https://issues.apache.org/jira/browse/MAPREDUCE-3131?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Allen Wittenauer updated MAPREDUCE-3131: Affects Version/s: (was: 0.24.0) 3.0.0 Docs and Scripts for setting up single node MRV2 cluster. -- Key: MAPREDUCE-3131 URL: https://issues.apache.org/jira/browse/MAPREDUCE-3131 Project: Hadoop Map/Reduce Issue Type: Improvement Components: documentation, mrv2, scripts Affects Versions: 3.0.0 Reporter: Prashant Sharma Priority: Trivial Labels: documentation, hadoop Attachments: MAPREDUCE-3131.patch, MAPREDUCE-3131.patch, MAPREDUCE-3131.patch, MAPREDUCE-3131.patch Original Estimate: 168h Time Spent: 96h Remaining Estimate: 72h Scripts to run a single node cluster with a default configuration. Takes care of running all the daemons including hdfs and yarn. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Updated] (MAPREDUCE-2585) Add dumpConfiguration option in hadoop help message
[ https://issues.apache.org/jira/browse/MAPREDUCE-2585?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Allen Wittenauer updated MAPREDUCE-2585: Fix Version/s: (was: 0.24.0) Add dumpConfiguration option in hadoop help message --- Key: MAPREDUCE-2585 URL: https://issues.apache.org/jira/browse/MAPREDUCE-2585 Project: Hadoop Map/Reduce Issue Type: Improvement Affects Versions: 0.20.1 Reporter: Ramya Sunil Assignee: V.V.Chaitanya Krishna Priority: Trivial Attachments: HADOOP-6184-ydist.patch, HADOOP-6219-ydist.patch, MAPREDUCE-919.patch, MAPREDUCE-919.patch Execution of bin/hadoop should show the -dumpConfiguration option introduced in MAPREDUCE-768 -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Updated] (MAPREDUCE-1506) Assertion failure in TestTaskTrackerMemoryManager
[
https://issues.apache.org/jira/browse/MAPREDUCE-1506?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Allen Wittenauer updated MAPREDUCE-1506:
Fix Version/s: (was: 3.0.0)
0.23.0
Assertion failure in TestTaskTrackerMemoryManager
-
Key: MAPREDUCE-1506
URL: https://issues.apache.org/jira/browse/MAPREDUCE-1506
Project: Hadoop Map/Reduce
Issue Type: Bug
Components: tasktracker
Reporter: Aaron Kimball
Fix For: 0.23.0
Attachments:
TEST-org.apache.hadoop.mapred.TestTaskTrackerMemoryManager.txt
With asserts enabled, TestTaskTrackerMemoryManager sometimes fails. From what
I've inspected, it's because some tasks are marked as FAILED/TIPFAILED while
others are marked SUCCEEDED.
This can be reproduced by applying MAPREDUCE-1092 and then running {{ant
clean test -Dtestcase=TestTaskTrackerMemoryManager}}
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Updated] (MAPREDUCE-1506) Assertion failure in TestTaskTrackerMemoryManager
[
https://issues.apache.org/jira/browse/MAPREDUCE-1506?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Allen Wittenauer updated MAPREDUCE-1506:
Fix Version/s: (was: 0.24.0)
3.0.0
Assertion failure in TestTaskTrackerMemoryManager
-
Key: MAPREDUCE-1506
URL: https://issues.apache.org/jira/browse/MAPREDUCE-1506
Project: Hadoop Map/Reduce
Issue Type: Bug
Components: tasktracker
Reporter: Aaron Kimball
Fix For: 3.0.0
Attachments:
TEST-org.apache.hadoop.mapred.TestTaskTrackerMemoryManager.txt
With asserts enabled, TestTaskTrackerMemoryManager sometimes fails. From what
I've inspected, it's because some tasks are marked as FAILED/TIPFAILED while
others are marked SUCCEEDED.
This can be reproduced by applying MAPREDUCE-1092 and then running {{ant
clean test -Dtestcase=TestTaskTrackerMemoryManager}}
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Updated] (MAPREDUCE-3629) Remove sleep from MRAppMaster during app-finish.
[ https://issues.apache.org/jira/browse/MAPREDUCE-3629?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Allen Wittenauer updated MAPREDUCE-3629: Fix Version/s: (was: 0.24.0) Remove sleep from MRAppMaster during app-finish. Key: MAPREDUCE-3629 URL: https://issues.apache.org/jira/browse/MAPREDUCE-3629 Project: Hadoop Map/Reduce Issue Type: Task Components: mrv2 Affects Versions: 0.23.0 Reporter: Amol Kekre MRAppMaster waits for 5 secs during app-finish, this was needed before we had client-side redirection. This affects the app execution in that, AppMaster will killed by the NM once NM gets confirmation from RM. AppMaster should go away immediately. Also, the done call to RM from AM should be the last thing AM ever does. Otherwise, today, JobHistory writing gets interrupted if AM gets killed by the NM. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Updated] (MAPREDUCE-3745) mapred/yarn scripts should use lib/'*' instead looping the dir for jar to create the classpath
[ https://issues.apache.org/jira/browse/MAPREDUCE-3745?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Allen Wittenauer updated MAPREDUCE-3745: Fix Version/s: (was: 0.24.0) mapred/yarn scripts should use lib/'*' instead looping the dir for jar to create the classpath -- Key: MAPREDUCE-3745 URL: https://issues.apache.org/jira/browse/MAPREDUCE-3745 Project: Hadoop Map/Reduce Issue Type: Improvement Affects Versions: 0.23.1, 0.24.0 Reporter: Alejandro Abdelnur The scripts do a for loop on the JAR contents of the directory, they should just use '*' in the classpath, ie lib/'*' This will reduce the length of the generated classpath significantly -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Updated] (MAPREDUCE-3089) Augment TestRMContainerAllocator to verify MAPREDUCE-2646
[ https://issues.apache.org/jira/browse/MAPREDUCE-3089?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Allen Wittenauer updated MAPREDUCE-3089: Fix Version/s: (was: 0.24.0) Augment TestRMContainerAllocator to verify MAPREDUCE-2646 - Key: MAPREDUCE-3089 URL: https://issues.apache.org/jira/browse/MAPREDUCE-3089 Project: Hadoop Map/Reduce Issue Type: Bug Components: applicationmaster, mrv2 Affects Versions: 0.23.0 Reporter: Arun C Murthy Assignee: Vinod Kumar Vavilapalli -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Updated] (MAPREDUCE-3043) Missing containers info on the nodes page
[ https://issues.apache.org/jira/browse/MAPREDUCE-3043?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Allen Wittenauer updated MAPREDUCE-3043: Fix Version/s: (was: 0.24.0) Missing containers info on the nodes page - Key: MAPREDUCE-3043 URL: https://issues.apache.org/jira/browse/MAPREDUCE-3043 Project: Hadoop Map/Reduce Issue Type: Bug Components: resourcemanager Affects Versions: 0.23.0 Reporter: Ramya Sunil Assignee: Subroto Sanyal Attachments: MAPREDUCE-3043.patch The containers info on the nodes page on the RM seems to be missing. This was useful in understanding the usage on each of the nodemanagers. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Updated] (MAPREDUCE-2498) TestRaidShellFsck failing on trunk
[ https://issues.apache.org/jira/browse/MAPREDUCE-2498?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Allen Wittenauer updated MAPREDUCE-2498: Fix Version/s: (was: 0.24.0) TestRaidShellFsck failing on trunk -- Key: MAPREDUCE-2498 URL: https://issues.apache.org/jira/browse/MAPREDUCE-2498 Project: Hadoop Map/Reduce Issue Type: Bug Components: contrib/raid Affects Versions: 0.23.0 Reporter: Todd Lipcon Assignee: Ramkumar Vadali Attachments: mapreduce-2498.txt TestRaidShellFsck.testFileBlockAndParityBlockMissingHar2 has been failing the last several builds: Error Message: parity file not HARed after 40s java.io.IOException: parity file not HARed after 40s at org.apache.hadoop.raid.TestRaidShellFsck.raidTestFiles(TestRaidShellFsck.java:281) at org.apache.hadoop.raid.TestRaidShellFsck.setUp(TestRaidShellFsck.java:181) at org.apache.hadoop.raid.TestRaidShellFsck.testFileBlockAndParityBlockMissingHar2(TestRaidShellFsck.java:666) -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Updated] (MAPREDUCE-2884) tmpjars not working when default filesystem mismatches between client and server
[ https://issues.apache.org/jira/browse/MAPREDUCE-2884?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Allen Wittenauer updated MAPREDUCE-2884: Fix Version/s: (was: 0.24.0) tmpjars not working when default filesystem mismatches between client and server Key: MAPREDUCE-2884 URL: https://issues.apache.org/jira/browse/MAPREDUCE-2884 Project: Hadoop Map/Reduce Issue Type: Bug Components: mrv1 Affects Versions: 0.23.0 Reporter: Todd Lipcon Assignee: Todd Lipcon One of the HBase tests is failing which tries to add a local file to the distributed cache using the tmpjars configuration variable. The first half of the distributedcache setup decides not to copy it to the JT, because the JT is apparently using the same filesystem, but the second half of distributedcache setup tries to check timestamps on a different filesystem where the file does not exist. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Resolved] (MAPREDUCE-2884) tmpjars not working when default filesystem mismatches between client and server
[ https://issues.apache.org/jira/browse/MAPREDUCE-2884?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Allen Wittenauer resolved MAPREDUCE-2884. - Resolution: Won't Fix stale tmpjars not working when default filesystem mismatches between client and server Key: MAPREDUCE-2884 URL: https://issues.apache.org/jira/browse/MAPREDUCE-2884 Project: Hadoop Map/Reduce Issue Type: Bug Components: mrv1 Affects Versions: 0.23.0 Reporter: Todd Lipcon Assignee: Todd Lipcon One of the HBase tests is failing which tries to add a local file to the distributed cache using the tmpjars configuration variable. The first half of the distributedcache setup decides not to copy it to the JT, because the JT is apparently using the same filesystem, but the second half of distributedcache setup tries to check timestamps on a different filesystem where the file does not exist. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Updated] (MAPREDUCE-3315) Master-Worker Application on YARN
[ https://issues.apache.org/jira/browse/MAPREDUCE-3315?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Allen Wittenauer updated MAPREDUCE-3315: Affects Version/s: 3.0.0 Master-Worker Application on YARN - Key: MAPREDUCE-3315 URL: https://issues.apache.org/jira/browse/MAPREDUCE-3315 Project: Hadoop Map/Reduce Issue Type: New Feature Affects Versions: 3.0.0 Reporter: Sharad Agarwal Assignee: Sharad Agarwal Attachments: MAPREDUCE-3315-1.patch, MAPREDUCE-3315-2.patch, MAPREDUCE-3315-3.patch, MAPREDUCE-3315.patch Currently master worker scenarios are forced fit into Map-Reduce. Now with YARN, these can be first class and would benefit real/near realtime workloads and be more effective in using the cluster resources. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Updated] (MAPREDUCE-2538) InputSampler.writePartitionFile() may write duplicate keys
[
https://issues.apache.org/jira/browse/MAPREDUCE-2538?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Allen Wittenauer updated MAPREDUCE-2538:
Fix Version/s: (was: 0.24.0)
Status: Patch Available (was: Open)
InputSampler.writePartitionFile() may write duplicate keys
--
Key: MAPREDUCE-2538
URL: https://issues.apache.org/jira/browse/MAPREDUCE-2538
Project: Hadoop Map/Reduce
Issue Type: Bug
Affects Versions: 0.20.2
Environment: EMR.
Reporter: Michael White
Assignee: Devaraj K
Priority: Minor
Attachments: MAPREDUCE-2538.patch
InputSampler.writePartitionFile() outputs the same key multiple times if the
input samples have enough of a given key to span multiple partitions. There
is logic in the code that appears to try to avoid this, but seems incorrect:
for(int i = 1; i numPartitions; ++i) {
int k = Math.round(stepSize * i);
while (last = k comparator.compare(samples[last], samples[k]) == 0) {
++k;
}
writer.append(samples[k], nullValue);
last = k;
}
The while loop condition last = k is always false. The sample comparison
after the never occurs.
It's not entirely clear what the correct fix is. The current behavior is
arguably correct mathematically, though the while loop could be elided for
clarity. If bug MAPREDUCE-1987 were fixed, it would be less of a problem
(for me at least), since that is where the non-uniqueness causes me problems.
Alternatively, changing the while to:
if( last = 0) {
while (comparator.compare(samples[last], samples[k]) = 0)) {
or, optimized for skipping over many duplicates (but arguably less clear):
if (last = 0) {
while (last = k || comparator.compare(samples[last], samples[k]) = 0)) {
would probably achieve what the original author intended.
Perhaps the behavior could be selected by a parameter, e.g. boolean unique.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Updated] (MAPREDUCE-2553) missing space in the error message of distcp command
[ https://issues.apache.org/jira/browse/MAPREDUCE-2553?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Allen Wittenauer updated MAPREDUCE-2553: Fix Version/s: (was: 0.24.0) missing space in the error message of distcp command Key: MAPREDUCE-2553 URL: https://issues.apache.org/jira/browse/MAPREDUCE-2553 Project: Hadoop Map/Reduce Issue Type: Improvement Components: distcp Affects Versions: 0.23.0 Reporter: Wei Yongjun Priority: Minor Attachments: MAPREDUCE-2553.patch Error message of distcp command missing space, here's an example output: $ hadoop distcp /user/test/input /user/test/ouput file:/root/temp ... Copy failed: java.io.IOException: Failed to createfile:/root/temp ... Generated from this line of source. src/tools/org/apache/hadoop/tools/DistCp.java: throw new IOException(Failed to create + args.dst); Failed to create should end with a ' '. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Updated] (MAPREDUCE-2354) Shuffle should be optimized
[ https://issues.apache.org/jira/browse/MAPREDUCE-2354?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Allen Wittenauer updated MAPREDUCE-2354: Fix Version/s: (was: 0.24.0) Shuffle should be optimized --- Key: MAPREDUCE-2354 URL: https://issues.apache.org/jira/browse/MAPREDUCE-2354 Project: Hadoop Map/Reduce Issue Type: Improvement Components: task, tasktracker Affects Versions: 0.20.1 Reporter: MengWang Labels: mapreduce, shuffle Our study shows that shuffle is a performance bottleneck of mapreduce computing. There are some problems of shuffle: (1)Shuffle and reduce are tightly-coupled, usually shuffle phase doesn't consume too much memory and CPU, so theoretically, reducetasks's slot can be used for other computing tasks when copying data from maps. This method will enhance cluster utilization. Furthermore, should shuffle be separated from reduce? Then shuffle will not use reduce's slot,we need't distinguish between map slots and reduce slots at all. (2)For large jobs, shuffle will use too many network connections, Data transmitted by each network connection is very little, which is inefficient. From 0.21.0 one connection can transfer several map outputs, but i think this is not enough. Maybe we can use a per node shuffle client progress(like tasktracker) to shuffle data for all reduce tasks on this node, then we can shuffle more data trough one connection. (3)Too many concurrent connections will cause shuffle server do massive random IO, which is inefficient. Maybe we can aggregate http request(like delay scheduler), then random IO will be sequential. (4)How to manage memory used by shuffle efficiently. We use buddy memory allocation, which will waste a considerable amount of memory. (5)If shuffle separated from reduce, then we must figure out how to do reduce locality? (6)Can we store map outputs in a Storage system(like hdfs)? (7)Can shuffle be a general data transfer service, which not only for map/reduce paradigm? -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Updated] (MAPREDUCE-3207) TestMRCLI failing on trunk
[ https://issues.apache.org/jira/browse/MAPREDUCE-3207?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Allen Wittenauer updated MAPREDUCE-3207: Fix Version/s: (was: 0.24.0) TestMRCLI failing on trunk Key: MAPREDUCE-3207 URL: https://issues.apache.org/jira/browse/MAPREDUCE-3207 Project: Hadoop Map/Reduce Issue Type: Sub-task Components: mrv2 Affects Versions: 0.23.0 Reporter: Hitesh Shah Priority: Minor Attachments: TEST-org.apache.hadoop.cli.TestMRCLI.txt Failing tests: 7: Archive: Deleting a file in archive 8: Archive: Renaming a file in archive -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Updated] (MAPREDUCE-3047) FileOutputCommitter throws wrong type of exception when calling abortTask() to handle a directory without permission
[ https://issues.apache.org/jira/browse/MAPREDUCE-3047?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Allen Wittenauer updated MAPREDUCE-3047: Status: Patch Available (was: Open) FileOutputCommitter throws wrong type of exception when calling abortTask() to handle a directory without permission Key: MAPREDUCE-3047 URL: https://issues.apache.org/jira/browse/MAPREDUCE-3047 Project: Hadoop Map/Reduce Issue Type: Improvement Reporter: JiangKai Priority: Trivial Attachments: MAPREDUCE-3047-1.patch, MAPREDUCE-3047-2.patch, MAPREDUCE-3047.patch When FileOutputCommitter calls abortTask() to create a temp directory, if the user has no permission to access the directory, or a file with the same name has existed, of course it will fail, however the system will output the error information into the log file instead of throwing an exception.As a result, when the temp directory is needed later, since the temp directory hasn't been created yet, system will throw an exception to tell user that the temp directory doesn't exist.In my opinion, the exception is not exact and the error infomation will confuse users. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Updated] (MAPREDUCE-3047) FileOutputCommitter throws wrong type of exception when calling abortTask() to handle a directory without permission
[ https://issues.apache.org/jira/browse/MAPREDUCE-3047?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Allen Wittenauer updated MAPREDUCE-3047: Fix Version/s: (was: 0.24.0) FileOutputCommitter throws wrong type of exception when calling abortTask() to handle a directory without permission Key: MAPREDUCE-3047 URL: https://issues.apache.org/jira/browse/MAPREDUCE-3047 Project: Hadoop Map/Reduce Issue Type: Improvement Reporter: JiangKai Priority: Trivial Attachments: MAPREDUCE-3047-1.patch, MAPREDUCE-3047-2.patch, MAPREDUCE-3047.patch When FileOutputCommitter calls abortTask() to create a temp directory, if the user has no permission to access the directory, or a file with the same name has existed, of course it will fail, however the system will output the error information into the log file instead of throwing an exception.As a result, when the temp directory is needed later, since the temp directory hasn't been created yet, system will throw an exception to tell user that the temp directory doesn't exist.In my opinion, the exception is not exact and the error infomation will confuse users. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Resolved] (MAPREDUCE-2461) Hudson jobs failing because mapred staging directory is full
[ https://issues.apache.org/jira/browse/MAPREDUCE-2461?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Allen Wittenauer resolved MAPREDUCE-2461. - Resolution: Fixed stale Hudson jobs failing because mapred staging directory is full Key: MAPREDUCE-2461 URL: https://issues.apache.org/jira/browse/MAPREDUCE-2461 Project: Hadoop Map/Reduce Issue Type: Bug Affects Versions: 0.23.0, 1.0.2 Reporter: Todd Lipcon Fix For: 0.24.0 All of the tests that submit MR jobs are failing on the h7 build machine. This is because the staging directory is entirely full: hudson@h7:/tmp/mr/mr$ ls -l /tmp/hadoop-hudson/mapred/staging/ | wc -l 31999 This makes me think that there's some bug where we're leaking things in the staging directory. I will manually clean this for now, but we should investigate. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Updated] (MAPREDUCE-3047) FileOutputCommitter throws wrong type of exception when calling abortTask() to handle a directory without permission
[ https://issues.apache.org/jira/browse/MAPREDUCE-3047?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Allen Wittenauer updated MAPREDUCE-3047: Affects Version/s: (was: 0.24.0) FileOutputCommitter throws wrong type of exception when calling abortTask() to handle a directory without permission Key: MAPREDUCE-3047 URL: https://issues.apache.org/jira/browse/MAPREDUCE-3047 Project: Hadoop Map/Reduce Issue Type: Improvement Reporter: JiangKai Priority: Trivial Attachments: MAPREDUCE-3047-1.patch, MAPREDUCE-3047-2.patch, MAPREDUCE-3047.patch When FileOutputCommitter calls abortTask() to create a temp directory, if the user has no permission to access the directory, or a file with the same name has existed, of course it will fail, however the system will output the error information into the log file instead of throwing an exception.As a result, when the temp directory is needed later, since the temp directory hasn't been created yet, system will throw an exception to tell user that the temp directory doesn't exist.In my opinion, the exception is not exact and the error infomation will confuse users. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Updated] (MAPREDUCE-2461) Hudson jobs failing because mapred staging directory is full
[ https://issues.apache.org/jira/browse/MAPREDUCE-2461?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Allen Wittenauer updated MAPREDUCE-2461: Fix Version/s: (was: 0.24.0) Hudson jobs failing because mapred staging directory is full Key: MAPREDUCE-2461 URL: https://issues.apache.org/jira/browse/MAPREDUCE-2461 Project: Hadoop Map/Reduce Issue Type: Bug Affects Versions: 0.23.0, 1.0.2 Reporter: Todd Lipcon All of the tests that submit MR jobs are failing on the h7 build machine. This is because the staging directory is entirely full: hudson@h7:/tmp/mr/mr$ ls -l /tmp/hadoop-hudson/mapred/staging/ | wc -l 31999 This makes me think that there's some bug where we're leaking things in the staging directory. I will manually clean this for now, but we should investigate. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Updated] (MAPREDUCE-3315) Master-Worker Application on YARN
[ https://issues.apache.org/jira/browse/MAPREDUCE-3315?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Allen Wittenauer updated MAPREDUCE-3315: Fix Version/s: (was: 0.24.0) Master-Worker Application on YARN - Key: MAPREDUCE-3315 URL: https://issues.apache.org/jira/browse/MAPREDUCE-3315 Project: Hadoop Map/Reduce Issue Type: New Feature Affects Versions: 3.0.0 Reporter: Sharad Agarwal Assignee: Sharad Agarwal Attachments: MAPREDUCE-3315-1.patch, MAPREDUCE-3315-2.patch, MAPREDUCE-3315-3.patch, MAPREDUCE-3315.patch Currently master worker scenarios are forced fit into Map-Reduce. Now with YARN, these can be first class and would benefit real/near realtime workloads and be more effective in using the cluster resources. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Updated] (MAPREDUCE-3131) Docs and Scripts for setting up single node MRV2 cluster.
[ https://issues.apache.org/jira/browse/MAPREDUCE-3131?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Allen Wittenauer updated MAPREDUCE-3131: Fix Version/s: (was: 0.24.0) Docs and Scripts for setting up single node MRV2 cluster. -- Key: MAPREDUCE-3131 URL: https://issues.apache.org/jira/browse/MAPREDUCE-3131 Project: Hadoop Map/Reduce Issue Type: Improvement Components: documentation, mrv2, scripts Affects Versions: 0.24.0 Reporter: Prashant Sharma Priority: Trivial Labels: documentation, hadoop Attachments: MAPREDUCE-3131.patch, MAPREDUCE-3131.patch, MAPREDUCE-3131.patch, MAPREDUCE-3131.patch Original Estimate: 168h Time Spent: 96h Remaining Estimate: 72h Scripts to run a single node cluster with a default configuration. Takes care of running all the daemons including hdfs and yarn. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Resolved] (MAPREDUCE-3222) ant test TestTaskContext failing on trunk
[ https://issues.apache.org/jira/browse/MAPREDUCE-3222?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Allen Wittenauer resolved MAPREDUCE-3222. - Resolution: Won't Fix stale ant test TestTaskContext failing on trunk - Key: MAPREDUCE-3222 URL: https://issues.apache.org/jira/browse/MAPREDUCE-3222 Project: Hadoop Map/Reduce Issue Type: Sub-task Components: mrv2 Affects Versions: 0.23.0 Reporter: Hitesh Shah Priority: Minor Testcase: testContextStatus took 29.977 sec FAILED null expected:map[ sort] but was:map[] junit.framework.ComparisonFailure: null expected:map[ sort] but was:map[] at org.apache.hadoop.mapreduce.TestTaskContext.testContextStatus(TestTaskContext.java:120) Testcase: testMapContextProgress took 17.371 sec Testcase: testReduceContextProgress took 16.267 sec -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Resolved] (MAPREDUCE-3207) TestMRCLI failing on trunk
[ https://issues.apache.org/jira/browse/MAPREDUCE-3207?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Allen Wittenauer resolved MAPREDUCE-3207. - Resolution: Won't Fix Stale TestMRCLI failing on trunk Key: MAPREDUCE-3207 URL: https://issues.apache.org/jira/browse/MAPREDUCE-3207 Project: Hadoop Map/Reduce Issue Type: Sub-task Components: mrv2 Affects Versions: 0.23.0 Reporter: Hitesh Shah Priority: Minor Attachments: TEST-org.apache.hadoop.cli.TestMRCLI.txt Failing tests: 7: Archive: Deleting a file in archive 8: Archive: Renaming a file in archive -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Updated] (MAPREDUCE-3131) Docs and Scripts for setting up single node MRV2 cluster.
[ https://issues.apache.org/jira/browse/MAPREDUCE-3131?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Allen Wittenauer updated MAPREDUCE-3131: Target Version/s: 3.0.0 (was: 0.24.0) Docs and Scripts for setting up single node MRV2 cluster. -- Key: MAPREDUCE-3131 URL: https://issues.apache.org/jira/browse/MAPREDUCE-3131 Project: Hadoop Map/Reduce Issue Type: Improvement Components: documentation, mrv2, scripts Affects Versions: 3.0.0 Reporter: Prashant Sharma Priority: Trivial Labels: documentation, hadoop Attachments: MAPREDUCE-3131.patch, MAPREDUCE-3131.patch, MAPREDUCE-3131.patch, MAPREDUCE-3131.patch Original Estimate: 168h Time Spent: 96h Remaining Estimate: 72h Scripts to run a single node cluster with a default configuration. Takes care of running all the daemons including hdfs and yarn. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Resolved] (MAPREDUCE-2498) TestRaidShellFsck failing on trunk
[ https://issues.apache.org/jira/browse/MAPREDUCE-2498?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Allen Wittenauer resolved MAPREDUCE-2498. - Resolution: Won't Fix stale TestRaidShellFsck failing on trunk -- Key: MAPREDUCE-2498 URL: https://issues.apache.org/jira/browse/MAPREDUCE-2498 Project: Hadoop Map/Reduce Issue Type: Bug Components: contrib/raid Affects Versions: 0.23.0 Reporter: Todd Lipcon Assignee: Ramkumar Vadali Attachments: mapreduce-2498.txt TestRaidShellFsck.testFileBlockAndParityBlockMissingHar2 has been failing the last several builds: Error Message: parity file not HARed after 40s java.io.IOException: parity file not HARed after 40s at org.apache.hadoop.raid.TestRaidShellFsck.raidTestFiles(TestRaidShellFsck.java:281) at org.apache.hadoop.raid.TestRaidShellFsck.setUp(TestRaidShellFsck.java:181) at org.apache.hadoop.raid.TestRaidShellFsck.testFileBlockAndParityBlockMissingHar2(TestRaidShellFsck.java:666) -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Resolved] (MAPREDUCE-2883) MR FI tests failing to build
[ https://issues.apache.org/jira/browse/MAPREDUCE-2883?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Allen Wittenauer resolved MAPREDUCE-2883. - Resolution: Won't Fix stale MR FI tests failing to build Key: MAPREDUCE-2883 URL: https://issues.apache.org/jira/browse/MAPREDUCE-2883 Project: Hadoop Map/Reduce Issue Type: Bug Components: test Affects Versions: 0.23.0 Reporter: Todd Lipcon Attachments: kill-fi.txt running ant mvn-install in hadoop-mapreduce-project on branch-0.23 fails in the fault injection compilation -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Updated] (MAPREDUCE-3222) ant test TestTaskContext failing on trunk
[ https://issues.apache.org/jira/browse/MAPREDUCE-3222?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Allen Wittenauer updated MAPREDUCE-3222: Fix Version/s: (was: 0.24.0) ant test TestTaskContext failing on trunk - Key: MAPREDUCE-3222 URL: https://issues.apache.org/jira/browse/MAPREDUCE-3222 Project: Hadoop Map/Reduce Issue Type: Sub-task Components: mrv2 Affects Versions: 0.23.0 Reporter: Hitesh Shah Priority: Minor Testcase: testContextStatus took 29.977 sec FAILED null expected:map[ sort] but was:map[] junit.framework.ComparisonFailure: null expected:map[ sort] but was:map[] at org.apache.hadoop.mapreduce.TestTaskContext.testContextStatus(TestTaskContext.java:120) Testcase: testMapContextProgress took 17.371 sec Testcase: testReduceContextProgress took 16.267 sec -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Updated] (MAPREDUCE-3079) usercache/user/appcache/appid directory not removed when using DefaultContainerExecutor
[ https://issues.apache.org/jira/browse/MAPREDUCE-3079?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Allen Wittenauer updated MAPREDUCE-3079: Fix Version/s: (was: 0.24.0) usercache/user/appcache/appid directory not removed when using DefaultContainerExecutor --- Key: MAPREDUCE-3079 URL: https://issues.apache.org/jira/browse/MAPREDUCE-3079 Project: Hadoop Map/Reduce Issue Type: Bug Components: mrv2 Affects Versions: 0.23.0 Reporter: Thomas Graves Running with the DefaultContainerExecutor it appears that the usercache/user/appcache/appid directory itself is not removed when the app finishes. All the directories under it are properly removed though. The nodemanager log file indicates that it tries to delete it: 11/09/23 15:17:56 INFO nodemanager.DefaultContainerExecutor: Deleting absolute path : /home/hadoop/mapred/tmp/mapred-local/usercache/tgraves/appcache/application_1316722920862_0003 This doesn't appear to happen with the LinuxContainerExecutor. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Resolved] (MAPREDUCE-3741) Conflicting dependency in hadoop-mapreduce-examples
[
https://issues.apache.org/jira/browse/MAPREDUCE-3741?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Allen Wittenauer resolved MAPREDUCE-3741.
-
Resolution: Fixed
stale
Conflicting dependency in hadoop-mapreduce-examples
---
Key: MAPREDUCE-3741
URL: https://issues.apache.org/jira/browse/MAPREDUCE-3741
Project: Hadoop Map/Reduce
Issue Type: Bug
Components: build
Affects Versions: 0.23.1
Reporter: Kihwal Lee
{code:xml}
dependency
groupIdorg.apache.hadoop/groupId
artifactIdhadoop-mapreduce-client-hs/artifactId
scopeprovided/scope
/dependency
dependency
groupIdorg.apache.hadoop/groupId
artifactIdhadoop-mapreduce-client-hs/artifactId
scopetest/scope
/dependency
{code}
Are we missing type here?
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Updated] (MAPREDUCE-3806) [Gridmix] TestGridmixSubmission fails due to incorrect version of jackson
[
https://issues.apache.org/jira/browse/MAPREDUCE-3806?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Allen Wittenauer updated MAPREDUCE-3806:
Target Version/s: (was: 0.24.0)
[Gridmix] TestGridmixSubmission fails due to incorrect version of jackson
-
Key: MAPREDUCE-3806
URL: https://issues.apache.org/jira/browse/MAPREDUCE-3806
Project: Hadoop Map/Reduce
Issue Type: Bug
Components: contrib/gridmix
Affects Versions: 0.24.0
Reporter: Amar Kamat
Labels: error, gridmix, junit
{{TestGridmixSubmission}} fails with the following error
{code}
org.codehaus.jackson.type.JavaType.isMapLikeType()Z
java.lang.NoSuchMethodError:
org.codehaus.jackson.type.JavaType.isMapLikeType()Z
at
org.codehaus.jackson.map.deser.StdDeserializerProvider._createDeserializer(StdDeserializerProvider.java:374)
at
org.codehaus.jackson.map.deser.StdDeserializerProvider._createAndCache2(StdDeserializerProvider.java:307)
at
org.codehaus.jackson.map.deser.StdDeserializerProvider._createAndCacheValueDeserializer(StdDeserializerProvider.java:287)
at
org.codehaus.jackson.map.deser.StdDeserializerProvider.findValueDeserializer(StdDeserializerProvider.java:136)
at
org.codehaus.jackson.map.deser.StdDeserializer.findDeserializer(StdDeserializer.java:551)
at
org.codehaus.jackson.map.deser.BeanDeserializer.resolve(BeanDeserializer.java:268)
at
org.codehaus.jackson.map.deser.StdDeserializerProvider._resolveDeserializer(StdDeserializerProvider.java:404)
at
org.codehaus.jackson.map.deser.StdDeserializerProvider._createAndCache2(StdDeserializerProvider.java:349)
at
org.codehaus.jackson.map.deser.StdDeserializerProvider._createAndCacheValueDeserializer(StdDeserializerProvider.java:287)
at
org.codehaus.jackson.map.deser.StdDeserializerProvider.findValueDeserializer(StdDeserializerProvider.java:136)
at
org.codehaus.jackson.map.deser.StdDeserializerProvider.findTypedValueDeserializer(StdDeserializerProvider.java:157)
at
org.codehaus.jackson.map.ObjectMapper._findRootDeserializer(ObjectMapper.java:2468)
at
org.codehaus.jackson.map.ObjectMapper._readValue(ObjectMapper.java:2383)
at
org.codehaus.jackson.map.ObjectMapper.readValue(ObjectMapper.java:1094)
at
org.apache.hadoop.tools.rumen.JsonObjectMapperParser.getNext(JsonObjectMapperParser.java:84)
at
org.apache.hadoop.tools.rumen.ZombieJobProducer.getNextJob(ZombieJobProducer.java:117)
at
org.apache.hadoop.tools.rumen.ZombieJobProducer.getNextJob(ZombieJobProducer.java:29)
at
org.apache.hadoop.mapred.gridmix.TestGridmixSubmission.testTraceReader(TestGridmixSubmission.java:440)
{code}
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Updated] (MAPREDUCE-3806) [Gridmix] TestGridmixSubmission fails due to incorrect version of jackson
[
https://issues.apache.org/jira/browse/MAPREDUCE-3806?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Allen Wittenauer updated MAPREDUCE-3806:
Fix Version/s: (was: 0.24.0)
[Gridmix] TestGridmixSubmission fails due to incorrect version of jackson
-
Key: MAPREDUCE-3806
URL: https://issues.apache.org/jira/browse/MAPREDUCE-3806
Project: Hadoop Map/Reduce
Issue Type: Bug
Components: contrib/gridmix
Affects Versions: 0.24.0
Reporter: Amar Kamat
Labels: error, gridmix, junit
{{TestGridmixSubmission}} fails with the following error
{code}
org.codehaus.jackson.type.JavaType.isMapLikeType()Z
java.lang.NoSuchMethodError:
org.codehaus.jackson.type.JavaType.isMapLikeType()Z
at
org.codehaus.jackson.map.deser.StdDeserializerProvider._createDeserializer(StdDeserializerProvider.java:374)
at
org.codehaus.jackson.map.deser.StdDeserializerProvider._createAndCache2(StdDeserializerProvider.java:307)
at
org.codehaus.jackson.map.deser.StdDeserializerProvider._createAndCacheValueDeserializer(StdDeserializerProvider.java:287)
at
org.codehaus.jackson.map.deser.StdDeserializerProvider.findValueDeserializer(StdDeserializerProvider.java:136)
at
org.codehaus.jackson.map.deser.StdDeserializer.findDeserializer(StdDeserializer.java:551)
at
org.codehaus.jackson.map.deser.BeanDeserializer.resolve(BeanDeserializer.java:268)
at
org.codehaus.jackson.map.deser.StdDeserializerProvider._resolveDeserializer(StdDeserializerProvider.java:404)
at
org.codehaus.jackson.map.deser.StdDeserializerProvider._createAndCache2(StdDeserializerProvider.java:349)
at
org.codehaus.jackson.map.deser.StdDeserializerProvider._createAndCacheValueDeserializer(StdDeserializerProvider.java:287)
at
org.codehaus.jackson.map.deser.StdDeserializerProvider.findValueDeserializer(StdDeserializerProvider.java:136)
at
org.codehaus.jackson.map.deser.StdDeserializerProvider.findTypedValueDeserializer(StdDeserializerProvider.java:157)
at
org.codehaus.jackson.map.ObjectMapper._findRootDeserializer(ObjectMapper.java:2468)
at
org.codehaus.jackson.map.ObjectMapper._readValue(ObjectMapper.java:2383)
at
org.codehaus.jackson.map.ObjectMapper.readValue(ObjectMapper.java:1094)
at
org.apache.hadoop.tools.rumen.JsonObjectMapperParser.getNext(JsonObjectMapperParser.java:84)
at
org.apache.hadoop.tools.rumen.ZombieJobProducer.getNextJob(ZombieJobProducer.java:117)
at
org.apache.hadoop.tools.rumen.ZombieJobProducer.getNextJob(ZombieJobProducer.java:29)
at
org.apache.hadoop.mapred.gridmix.TestGridmixSubmission.testTraceReader(TestGridmixSubmission.java:440)
{code}
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Updated] (MAPREDUCE-2540) [Gridmix] Gridmix should faithfully emulate old and new mapred(uce) APIs
[ https://issues.apache.org/jira/browse/MAPREDUCE-2540?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Allen Wittenauer updated MAPREDUCE-2540: Fix Version/s: (was: 0.24.0) [Gridmix] Gridmix should faithfully emulate old and new mapred(uce) APIs Key: MAPREDUCE-2540 URL: https://issues.apache.org/jira/browse/MAPREDUCE-2540 Project: Hadoop Map/Reduce Issue Type: Improvement Components: contrib/gridmix Reporter: Amar Kamat Gridmix currently submits jobs to the simulated cluster using the new MapReduce API. Since the old and new mapred(uce) APIs have different code paths, it would be useful if Gridmix faithfully emulates this behavior. Information regarding whether the original job used old or new API is captured in the job's configuration. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Updated] (MAPREDUCE-2435) MR-279: Modularize web framework and webapps
[ https://issues.apache.org/jira/browse/MAPREDUCE-2435?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Allen Wittenauer updated MAPREDUCE-2435: Fix Version/s: (was: 0.24.0) MR-279: Modularize web framework and webapps Key: MAPREDUCE-2435 URL: https://issues.apache.org/jira/browse/MAPREDUCE-2435 Project: Hadoop Map/Reduce Issue Type: Improvement Components: mrv2 Reporter: Luke Lu Assignee: Luke Lu The patch moves the web framework out of yarn-common into a separate module: yarn-web. It also decouple webapps into separate modules/jars from their respective server modules/jars to allow webapp updates independent of servers. Servers use ServiceLoader to discover its webapp modules. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Updated] (MAPREDUCE-2612) The result returned by the wrong usage of the command job -counterjob-id group-name counter-name is not appropriate
[ https://issues.apache.org/jira/browse/MAPREDUCE-2612?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Allen Wittenauer updated MAPREDUCE-2612: Fix Version/s: (was: 0.24.0) The result returned by the wrong usage of the command job -counterjob-id group-name counter-name is not appropriate --- Key: MAPREDUCE-2612 URL: https://issues.apache.org/jira/browse/MAPREDUCE-2612 Project: Hadoop Map/Reduce Issue Type: Improvement Affects Versions: 0.23.0 Reporter: XieXianshan Priority: Minor Attachments: MAPREDUCE-2612-0627, Mapreduce_2612 The result returned by Map/Reduce command job -counterjob-id group-name counter-name is always zero ,when use this command with incorrect group name or counter name. It is very easy to be misunderstood if users just make a spelling mistake. So,for more comprehensible information, more detailed results should be displayed as the following: [Could not find group FileSystemCounters_err] --incorrect group name or [Could not find counter FILE_BYTES_READ_err in the group FileSystemCounters] --incorrect counter name -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Updated] (MAPREDUCE-2371) TaskLogsTruncater does not need to check log ownership when running as Child
[ https://issues.apache.org/jira/browse/MAPREDUCE-2371?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Allen Wittenauer updated MAPREDUCE-2371: Fix Version/s: (was: 0.24.0) TaskLogsTruncater does not need to check log ownership when running as Child Key: MAPREDUCE-2371 URL: https://issues.apache.org/jira/browse/MAPREDUCE-2371 Project: Hadoop Map/Reduce Issue Type: Improvement Affects Versions: 0.23.0 Reporter: Todd Lipcon Assignee: Todd Lipcon Attachments: mapreduce-2371-on-0.20.txt Before MAPREDUCE-2178, it used to be that the TaskLogsTruncater had to use the SecureIO API to open the task logs before truncation, to avoid an attack where the user would symlink in something that the TT had access to but not the user. After MAPREDUCE-2178, this truncation is done as the user rather than as the TT, so we don't need to perform this check. Not performing the check avoids a fork() call which we've found to be troublesome since it doubles vmem consumption and thus requires that users bump mapred.child.ulimit to 2x the expected child heap size. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Updated] (MAPREDUCE-2853) Add teraread example
[ https://issues.apache.org/jira/browse/MAPREDUCE-2853?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Allen Wittenauer updated MAPREDUCE-2853: Fix Version/s: (was: 0.24.0) Add teraread example -- Key: MAPREDUCE-2853 URL: https://issues.apache.org/jira/browse/MAPREDUCE-2853 Project: Hadoop Map/Reduce Issue Type: New Feature Components: benchmarks, examples Affects Versions: 0.23.0 Reporter: Todd Lipcon Assignee: Todd Lipcon Attachments: mapreduce-2853.txt Teragen is a good benchmark of raw DFS write throughput. Terasort is a good benchmark of the whole MR system (input, shuffle, output). I've added a simple teraread example which reads through the terasort input data without performing any processing: this acts as a good benchmark of a read-only workload (similar to real-life find a needle in a haystack MR jobs) -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Resolved] (MAPREDUCE-2728) Update Mapreduce dependency of Java for deb package
[ https://issues.apache.org/jira/browse/MAPREDUCE-2728?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Allen Wittenauer resolved MAPREDUCE-2728. - Resolution: Won't Fix closing as won't fix. no longer building deb files Update Mapreduce dependency of Java for deb package --- Key: MAPREDUCE-2728 URL: https://issues.apache.org/jira/browse/MAPREDUCE-2728 Project: Hadoop Map/Reduce Issue Type: Bug Components: build Affects Versions: 0.23.0 Environment: Java 6, Ubuntu Reporter: Eric Yang Assignee: Eric Yang Attachments: MAPREDUCE-2728.patch Java dependency for Debian package is specified as open JDK, but it should depends on Sun version of Java. This dependency can be implicitly defined by hadoop-common dependency. Hence, there is no need to explicitly defined in hadoop-mapreduce. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Updated] (MAPREDUCE-2728) Update Mapreduce dependency of Java for deb package
[ https://issues.apache.org/jira/browse/MAPREDUCE-2728?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Allen Wittenauer updated MAPREDUCE-2728: Fix Version/s: (was: 0.24.0) Update Mapreduce dependency of Java for deb package --- Key: MAPREDUCE-2728 URL: https://issues.apache.org/jira/browse/MAPREDUCE-2728 Project: Hadoop Map/Reduce Issue Type: Bug Components: build Affects Versions: 0.23.0 Environment: Java 6, Ubuntu Reporter: Eric Yang Assignee: Eric Yang Attachments: MAPREDUCE-2728.patch Java dependency for Debian package is specified as open JDK, but it should depends on Sun version of Java. This dependency can be implicitly defined by hadoop-common dependency. Hence, there is no need to explicitly defined in hadoop-mapreduce. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Updated] (MAPREDUCE-2865) MRV2 Job.java and others in MRv2 client need javadocs in it.
[ https://issues.apache.org/jira/browse/MAPREDUCE-2865?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Allen Wittenauer updated MAPREDUCE-2865: Fix Version/s: (was: 0.24.0) MRV2 Job.java and others in MRv2 client need javadocs in it. Key: MAPREDUCE-2865 URL: https://issues.apache.org/jira/browse/MAPREDUCE-2865 Project: Hadoop Map/Reduce Issue Type: Bug Components: mrv2 Affects Versions: 0.23.0 Reporter: Robert Joseph Evans This may fall under another JIRA already filed, but Job.java and other files in the MRv2 client need to have javadocs in them. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Updated] (MAPREDUCE-2616) [Gridmix] InputStriper should smartly switch between compressed and uncompressed files based on the simulated job's input data characteristics
[ https://issues.apache.org/jira/browse/MAPREDUCE-2616?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Allen Wittenauer updated MAPREDUCE-2616: Fix Version/s: (was: 0.24.0) [Gridmix] InputStriper should smartly switch between compressed and uncompressed files based on the simulated job's input data characteristics -- Key: MAPREDUCE-2616 URL: https://issues.apache.org/jira/browse/MAPREDUCE-2616 Project: Hadoop Map/Reduce Issue Type: Improvement Components: contrib/gridmix Affects Versions: 0.23.0 Reporter: Amar Kamat Assignee: Amar Kamat Priority: Minor Labels: compression-emulation, gridmix Currently, all the Gridmix input data files are located at gridmix-io-dir/input (gridmix-io-dir is expected as a CLI parameter). When compression emulation is enabled, Gridmix will check for compressed files (based on suffixes) in the input folder. Gridmix will bail out if there are no compressed input files. If the input folder consists of a mix of compressed and uncompressed input files, then Gridmix will only use compressed input files for all the jobs. Gridmix should smartly assign 1. uncompressed input files for jobs the don't need input decompression 2. compressed input files for jobs that need input decompression -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Resolved] (MAPREDUCE-1824) JobTracker should reuse file system handle for delegation token renewal
[ https://issues.apache.org/jira/browse/MAPREDUCE-1824?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Allen Wittenauer resolved MAPREDUCE-1824. - Resolution: Later given the comments, closing as stale/fixed/whatevered. JobTracker should reuse file system handle for delegation token renewal --- Key: MAPREDUCE-1824 URL: https://issues.apache.org/jira/browse/MAPREDUCE-1824 Project: Hadoop Map/Reduce Issue Type: Improvement Reporter: Jitendra Nath Pandey Assignee: Daryn Sharp Attachments: MR-1824.1.patch In trunk, the DelegationTokenRenewal obtains the file system handle by creating the uri out of service in the token, which is ip:port. The intention of this jira is to use host name of the namenode so that fils system handle in the cache on jobtracker could be re-used. This jira is created because such an optimization is there in 20 code and the patch attached is the direct port of the code in 20. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Updated] (MAPREDUCE-2604) Delegation token renewal over https in JobTracker
[ https://issues.apache.org/jira/browse/MAPREDUCE-2604?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Allen Wittenauer updated MAPREDUCE-2604: Fix Version/s: (was: 0.24.0) Delegation token renewal over https in JobTracker -- Key: MAPREDUCE-2604 URL: https://issues.apache.org/jira/browse/MAPREDUCE-2604 Project: Hadoop Map/Reduce Issue Type: Bug Affects Versions: 0.23.0 Reporter: Jitendra Nath Pandey Assignee: Jitendra Nath Pandey Delegation token renewal in JobTracker uses https if there is a failure in creation of DistributedFileSystem. This handles the case when server is using a different version. With the changes in HADOOP-7227 the, creation of DistributedFileSystem will not fail because the creation of file system object doesn't make a connection or an rpc call. Therefore, the switchover to https should happen after an attempt to renew over rpc fails. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
