[jira] [Commented] (MAPREDUCE-2517) Porting Gridmix v3 system tests into trunk branch.
[ https://issues.apache.org/jira/browse/MAPREDUCE-2517?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13039551#comment-13039551 ] Amar Kamat commented on MAPREDUCE-2517: --- The latest patch looks good to me. +1. I will commit this once Vinay reverts back with the system test results. > Porting Gridmix v3 system tests into trunk branch. > -- > > Key: MAPREDUCE-2517 > URL: https://issues.apache.org/jira/browse/MAPREDUCE-2517 > Project: Hadoop Map/Reduce > Issue Type: Task > Components: contrib/gridmix >Reporter: Vinay Kumar Thota >Assignee: Vinay Kumar Thota > Attachments: MAPREDUCE-2517-v2.patch, MAPREDUCE-2517-v3.patch, > MAPREDUCE-2517.patch > > > Porting of girdmix v3 system tests into trunk branch. -- This message is automatically generated by JIRA. For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (MAPREDUCE-2521) Mapreduce RPM integration project
[ https://issues.apache.org/jira/browse/MAPREDUCE-2521?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13039599#comment-13039599 ] Hadoop QA commented on MAPREDUCE-2521: -- -1 overall. Here are the results of testing the latest attachment http://issues.apache.org/jira/secure/attachment/12480477/MAPREDUCE-2521-6.patch against trunk revision 1127444. +1 @author. The patch does not contain any @author tags. +1 tests included. The patch appears to include 16 new or modified tests. +1 javadoc. The javadoc tool did not generate any warning messages. +1 javac. The applied patch does not increase the total number of javac compiler warnings. -1 findbugs. The patch appears to introduce 1 new Findbugs (version 1.3.9) warnings. +1 release audit. The applied patch does not increase the total number of release audit warnings. -1 core tests. The patch failed these core unit tests: org.apache.hadoop.cli.TestMRCLI +1 contrib tests. The patch passed contrib unit tests. +1 system test framework. The patch passed system test framework compile. Test results: https://builds.apache.org/hudson/job/PreCommit-MAPREDUCE-Build/304//testReport/ Findbugs warnings: https://builds.apache.org/hudson/job/PreCommit-MAPREDUCE-Build/304//artifact/trunk/build/test/findbugs/newPatchFindbugsWarnings.html Console output: https://builds.apache.org/hudson/job/PreCommit-MAPREDUCE-Build/304//console This message is automatically generated. > Mapreduce RPM integration project > - > > Key: MAPREDUCE-2521 > URL: https://issues.apache.org/jira/browse/MAPREDUCE-2521 > Project: Hadoop Map/Reduce > Issue Type: New Feature > Components: build > Environment: Java 6, RHEL 5.5 >Reporter: Eric Yang >Assignee: Eric Yang > Attachments: MAPREDUCE-2521-1.patch, MAPREDUCE-2521-2.patch, > MAPREDUCE-2521-3.patch, MAPREDUCE-2521-4.patch, MAPREDUCE-2521-5.patch, > MAPREDUCE-2521-6.patch, MAPREDUCE-2521.patch > > > This jira is corresponding to HADOOP-6255 and associated directory layout > change. The patch for creating Mapreduce rpm packaging should be posted here > for patch test build to verify against mapreduce svn trunk. -- This message is automatically generated by JIRA. For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Updated] (MAPREDUCE-2517) Porting Gridmix v3 system tests into trunk branch.
[ https://issues.apache.org/jira/browse/MAPREDUCE-2517?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Vinay Kumar Thota updated MAPREDUCE-2517: - Attachment: MAPREDUCE-2517-v4.patch Uploaded the new patch with minor changes. > Porting Gridmix v3 system tests into trunk branch. > -- > > Key: MAPREDUCE-2517 > URL: https://issues.apache.org/jira/browse/MAPREDUCE-2517 > Project: Hadoop Map/Reduce > Issue Type: Task > Components: contrib/gridmix >Reporter: Vinay Kumar Thota >Assignee: Vinay Kumar Thota > Attachments: MAPREDUCE-2517-v2.patch, MAPREDUCE-2517-v3.patch, > MAPREDUCE-2517-v4.patch, MAPREDUCE-2517.patch > > > Porting of girdmix v3 system tests into trunk branch. -- This message is automatically generated by JIRA. For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Updated] (MAPREDUCE-2517) Porting Gridmix v3 system tests into trunk branch.
[ https://issues.apache.org/jira/browse/MAPREDUCE-2517?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Vinay Kumar Thota updated MAPREDUCE-2517: - Fix Version/s: 0.23.0 Hadoop Flags: [Reviewed] Status: Patch Available (was: Open) > Porting Gridmix v3 system tests into trunk branch. > -- > > Key: MAPREDUCE-2517 > URL: https://issues.apache.org/jira/browse/MAPREDUCE-2517 > Project: Hadoop Map/Reduce > Issue Type: Task > Components: contrib/gridmix >Reporter: Vinay Kumar Thota >Assignee: Vinay Kumar Thota > Fix For: 0.23.0 > > Attachments: MAPREDUCE-2517-v2.patch, MAPREDUCE-2517-v3.patch, > MAPREDUCE-2517-v4.patch, MAPREDUCE-2517.patch > > > Porting of girdmix v3 system tests into trunk branch. -- This message is automatically generated by JIRA. For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (MAPREDUCE-2137) Mapping between Gridmix jobs and the corresponding original MR jobs is needed
[ https://issues.apache.org/jira/browse/MAPREDUCE-2137?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13039651#comment-13039651 ] Amar Kamat commented on MAPREDUCE-2137: --- The latest patch looks good to me. +1. We can commit this. > Mapping between Gridmix jobs and the corresponding original MR jobs is needed > - > > Key: MAPREDUCE-2137 > URL: https://issues.apache.org/jira/browse/MAPREDUCE-2137 > Project: Hadoop Map/Reduce > Issue Type: Bug > Components: contrib/gridmix >Affects Versions: 0.23.0 >Reporter: Ravi Gummadi >Assignee: Ravi Gummadi > Fix For: 0.23.0 > > Attachments: 2137.patch, 2137.v1.patch, 2137.v2.1.patch > > > Consider a trace file "trace1" obtained by running Rumen on a set of MR jobs' > history logs. When gridmix runs simulated jobs from "trace1", it may skip > some of the jobs from the trace file for some reason like out-of-order-jobs. > Now use Rumen to generate trace2 from the history logs of gridmix's simulated > jobs. > Now, to compare and analyze the gridmix's simulated jobs with original MR > jobs, we need a mapping between them. -- This message is automatically generated by JIRA. For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (MAPREDUCE-2517) Porting Gridmix v3 system tests into trunk branch.
[ https://issues.apache.org/jira/browse/MAPREDUCE-2517?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13039655#comment-13039655 ] Vinay Kumar Thota commented on MAPREDUCE-2517: -- Ran system tests with new patch and no issues found. > Porting Gridmix v3 system tests into trunk branch. > -- > > Key: MAPREDUCE-2517 > URL: https://issues.apache.org/jira/browse/MAPREDUCE-2517 > Project: Hadoop Map/Reduce > Issue Type: Task > Components: contrib/gridmix >Reporter: Vinay Kumar Thota >Assignee: Vinay Kumar Thota > Fix For: 0.23.0 > > Attachments: MAPREDUCE-2517-v2.patch, MAPREDUCE-2517-v3.patch, > MAPREDUCE-2517-v4.patch, MAPREDUCE-2517.patch > > > Porting of girdmix v3 system tests into trunk branch. -- This message is automatically generated by JIRA. For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (MAPREDUCE-2137) Mapping between Gridmix jobs and the corresponding original MR jobs is needed
[ https://issues.apache.org/jira/browse/MAPREDUCE-2137?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13039696#comment-13039696 ] Hadoop QA commented on MAPREDUCE-2137: -- -1 overall. Here are the results of testing the latest attachment http://issues.apache.org/jira/secure/attachment/12480513/2137.v2.1.patch against trunk revision 1127444. +1 @author. The patch does not contain any @author tags. +1 tests included. The patch appears to include 6 new or modified tests. +1 javadoc. The javadoc tool did not generate any warning messages. +1 javac. The applied patch does not increase the total number of javac compiler warnings. -1 findbugs. The patch appears to introduce 1 new Findbugs (version 1.3.9) warnings. +1 release audit. The applied patch does not increase the total number of release audit warnings. -1 core tests. The patch failed these core unit tests: org.apache.hadoop.cli.TestMRCLI +1 contrib tests. The patch passed contrib unit tests. +1 system test framework. The patch passed system test framework compile. Test results: https://builds.apache.org/hudson/job/PreCommit-MAPREDUCE-Build/305//testReport/ Findbugs warnings: https://builds.apache.org/hudson/job/PreCommit-MAPREDUCE-Build/305//artifact/trunk/build/test/findbugs/newPatchFindbugsWarnings.html Console output: https://builds.apache.org/hudson/job/PreCommit-MAPREDUCE-Build/305//console This message is automatically generated. > Mapping between Gridmix jobs and the corresponding original MR jobs is needed > - > > Key: MAPREDUCE-2137 > URL: https://issues.apache.org/jira/browse/MAPREDUCE-2137 > Project: Hadoop Map/Reduce > Issue Type: Bug > Components: contrib/gridmix >Affects Versions: 0.23.0 >Reporter: Ravi Gummadi >Assignee: Ravi Gummadi > Fix For: 0.23.0 > > Attachments: 2137.patch, 2137.v1.patch, 2137.v2.1.patch > > > Consider a trace file "trace1" obtained by running Rumen on a set of MR jobs' > history logs. When gridmix runs simulated jobs from "trace1", it may skip > some of the jobs from the trace file for some reason like out-of-order-jobs. > Now use Rumen to generate trace2 from the history logs of gridmix's simulated > jobs. > Now, to compare and analyze the gridmix's simulated jobs with original MR > jobs, we need a mapping between them. -- This message is automatically generated by JIRA. For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (MAPREDUCE-2531) org.apache.hadoop.mapred.jobcontrol.getAssignedJobID throw class cast exception
[ https://issues.apache.org/jira/browse/MAPREDUCE-2531?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13039710#comment-13039710 ] Robert Joseph Evans commented on MAPREDUCE-2531: I will work on that. > org.apache.hadoop.mapred.jobcontrol.getAssignedJobID throw class cast > exception > > > Key: MAPREDUCE-2531 > URL: https://issues.apache.org/jira/browse/MAPREDUCE-2531 > Project: Hadoop Map/Reduce > Issue Type: Bug > Components: client >Affects Versions: 0.22.0 >Reporter: Robert Joseph Evans >Assignee: Robert Joseph Evans > Fix For: 0.23.0 > > Attachments: MR-2531-V1-trunk.patch > > > When using a combination of the mapred and mapreduce APIs (PIG) it is > possible to have the following exception > Caused by: java.lang.ClassCastException: org.apache.hadoop.mapreduce.JobID > cannot be cast to > org.apache.hadoop.mapred.JobID > at > org.apache.hadoop.mapred.jobcontrol.Job.getAssignedJobID(Job.java:71) > at > org.apache.pig.backend.hadoop.executionengine.mapReduceLayer.MapReduceLauncher.launchPig(MapReduceLauncher.java:239) > at org.apache.pig.PigServer.launchPlan(PigServer.java:1325) > ... 29 more > This is because the JobID is just downcast. It should be calling > JobID.downgrade -- This message is automatically generated by JIRA. For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Created] (MAPREDUCE-2535) JobClient creates a RunningJob with null status and profile
JobClient creates a RunningJob with null status and profile
---
Key: MAPREDUCE-2535
URL: https://issues.apache.org/jira/browse/MAPREDUCE-2535
Project: Hadoop Map/Reduce
Issue Type: Bug
Components: client
Affects Versions: 0.20.204.0
Reporter: Robert Joseph Evans
Assignee: Robert Joseph Evans
Exception occurred because the job was retired and is removed from
RetireJobCcahe and CompletedJobStatusStore. But, the
JobClient creates a RunningJob with null status and profile, if getJob(JobID)
is called again.
So, Even-though not null check is there in the following user code, it did not
help.
466 runningJob = jobClient.getJob(mapRedJobID);
467 if(runningJob != null) {
JobClient.getJob() should return null if status is null.
In trunk this is fixed by validating that the job status is not null every time
it is updated, and also verifying that that the profile data is not null when
created.
--
This message is automatically generated by JIRA.
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (MAPREDUCE-2510) TaskTracker throw OutOfMemoryError after upgrade to jetty6
[ https://issues.apache.org/jira/browse/MAPREDUCE-2510?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13039736#comment-13039736 ] Liyin Liang commented on MAPREDUCE-2510: After upgrading our product cluster's Jetty version to 6.1.26. The checkpoint become very slow. fsimage size download time Before upgrading 10G2 mins After upgrading9.95G 15 mins What's more, there are many "JVM BUG(s)" logs in NN's log file: 2011-05-26 22:46:48,807 INFO org.mortbay.log: org.mortbay.io.nio.SelectorManager$SelectSet@173ab5e JVM BUG(s) - injecting delay59 times 2011-05-26 22:46:48,807 INFO org.mortbay.log: org.mortbay.io.nio.SelectorManager$SelectSet@173ab5e JVM BUG(s) - recreating selector 59 times, canceled keys 944 times According to Jetty 6.1.26's code, Jetty's Selector sleep some time when print above logs. > TaskTracker throw OutOfMemoryError after upgrade to jetty6 > -- > > Key: MAPREDUCE-2510 > URL: https://issues.apache.org/jira/browse/MAPREDUCE-2510 > Project: Hadoop Map/Reduce > Issue Type: Bug >Reporter: Liyin Liang > > Our product cluster's TaskTracker sometimes throw OutOfMemoryError after > upgrade to jetty6. The exception in TT's log is as follows: > 2011-05-17 19:16:40,756 ERROR org.mortbay.log: Error for /mapOutput > java.lang.OutOfMemoryError: Java heap space > at java.io.BufferedInputStream.(BufferedInputStream.java:178) > at > org.apache.hadoop.fs.BufferedFSInputStream.(BufferedFSInputStream.java:44) > at > org.apache.hadoop.fs.RawLocalFileSystem.open(RawLocalFileSystem.java:176) > at org.apache.hadoop.fs.FileSystem.open(FileSystem.java:359) > at > org.apache.hadoop.mapred.TaskTracker$MapOutputServlet.doGet(TaskTracker.java:3040) > at javax.servlet.http.HttpServlet.service(HttpServlet.java:707) > at javax.servlet.http.HttpServlet.service(HttpServlet.java:820) > at > org.mortbay.jetty.servlet.ServletHolder.handle(ServletHolder.java:502) > at > org.mortbay.jetty.servlet.ServletHandler.handle(ServletHandler.java:363) > at > org.mortbay.jetty.security.SecurityHandler.handle(SecurityHandler.java:216) > at > org.mortbay.jetty.servlet.SessionHandler.handle(SessionHandler.java:181) > at > org.mortbay.jetty.handler.ContextHandler.handle(ContextHandler.java:766) > at > org.mortbay.jetty.webapp.WebAppContext.handle(WebAppContext.java:417) > at > org.mortbay.jetty.handler.ContextHandlerCollection.handle(ContextHandlerCollection.java:230) > at > org.mortbay.jetty.handler.HandlerWrapper.handle(HandlerWrapper.java:152) > at org.mortbay.jetty.Server.handle(Server.java:324) > at > org.mortbay.jetty.HttpConnection.handleRequest(HttpConnection.java:534) > at > org.mortbay.jetty.HttpConnection$RequestHandler.headerComplete(HttpConnection.java:864) > at org.mortbay.jetty.HttpParser.parseNext(HttpParser.java:533) > at org.mortbay.jetty.HttpParser.parseAvailable(HttpParser.java:207) > at org.mortbay.jetty.HttpConnection.handle(HttpConnection.java:403) > at > org.mortbay.io.nio.SelectChannelEndPoint.run(SelectChannelEndPoint.java:409) > at > org.mortbay.thread.QueuedThreadPool$PoolThread.run(QueuedThreadPool.java:522) > Exceptions in .out file: > java.lang.OutOfMemoryError: Java heap space > Exception in thread "process reaper" java.lang.OutOfMemoryError: Java heap > space > Exception in thread "pool-1-thread-1" java.lang.OutOfMemoryError: Java heap > space > java.lang.OutOfMemoryError: Java heap space > java.lang.reflect.InvocationTargetException > Exception in thread "IPC Server handler 6 on 50050" at > sun.reflect.GeneratedMethodAccessor15.invoke(Unknown Source) > at > sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:25) > at java.lang.reflect.Method.invoke(Method.java:597) > at org.mortbay.log.Slf4jLog.warn(Slf4jLog.java:126) > at org.mortbay.log.Log.warn(Log.java:181) > at > org.mortbay.jetty.servlet.ServletHandler.handle(ServletHandler.java:449) > at > org.mortbay.jetty.security.SecurityHandler.handle(SecurityHandler.java:216) > at > org.mortbay.jetty.servlet.SessionHandler.handle(SessionHandler.java:181) > at > org.mortbay.jetty.handler.ContextHandler.handle(ContextHandler.java:766) > at > org.mortbay.jetty.webapp.WebAppContext.handle(WebAppContext.java:417) > at > org.mortbay.jetty.handler.ContextHandlerCollection.handle(ContextHandlerCollection.java:230) > at > org.mortbay.jetty.handler.HandlerWrapper.handle(HandlerWrapper.java:152) > at org.mortbay.jetty.Server.handle(Server.java:324) >
[jira] [Updated] (MAPREDUCE-2535) JobClient creates a RunningJob with null status and profile
[
https://issues.apache.org/jira/browse/MAPREDUCE-2535?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Robert Joseph Evans updated MAPREDUCE-2535:
---
Status: Patch Available (was: Open)
> JobClient creates a RunningJob with null status and profile
> ---
>
> Key: MAPREDUCE-2535
> URL: https://issues.apache.org/jira/browse/MAPREDUCE-2535
> Project: Hadoop Map/Reduce
> Issue Type: Bug
> Components: client
>Affects Versions: 0.20.204.0
>Reporter: Robert Joseph Evans
>Assignee: Robert Joseph Evans
> Attachments: MR-2535-0.20.20X-V1.patch
>
>
> Exception occurred because the job was retired and is removed from
> RetireJobCcahe and CompletedJobStatusStore. But, the
> JobClient creates a RunningJob with null status and profile, if getJob(JobID)
> is called again.
> So, Even-though not null check is there in the following user code, it did
> not help.
> 466 runningJob = jobClient.getJob(mapRedJobID);
> 467 if(runningJob != null) {
> JobClient.getJob() should return null if status is null.
> In trunk this is fixed by validating that the job status is not null every
> time it is updated, and also verifying that that the profile data is not null
> when created.
--
This message is automatically generated by JIRA.
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Updated] (MAPREDUCE-2535) JobClient creates a RunningJob with null status and profile
[
https://issues.apache.org/jira/browse/MAPREDUCE-2535?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Robert Joseph Evans updated MAPREDUCE-2535:
---
Attachment: MR-2535-0.20.20X-V1.patch
Because this is not targeted for trunk, and Jenkins will try to run this
against trunk here are the test-patch results
[exec] -1 overall.
[exec]
[exec] +1 @author. The patch does not contain any @author tags.
[exec]
[exec] +1 tests included. The patch appears to include 4 new or
modified tests.
[exec]
[exec] -1 javadoc. The javadoc tool appears to have generated 1
warning messages.
[exec]
[exec] +1 javac. The applied patch does not increase the total number
of javac compiler warnings.
[exec]
[exec] +1 findbugs. The patch does not introduce any new Findbugs
warnings.
[exec]
[exec] -1 Eclipse classpath. The patch causes the Eclipse classpath to
differ from the contents of the lib directories.
I don't know why it complains about the eclipse classpath or javadocs. Both of
those are false positives. The eclipse classpath did not change, and it is
still 6 javadoc warnings.
> JobClient creates a RunningJob with null status and profile
> ---
>
> Key: MAPREDUCE-2535
> URL: https://issues.apache.org/jira/browse/MAPREDUCE-2535
> Project: Hadoop Map/Reduce
> Issue Type: Bug
> Components: client
>Affects Versions: 0.20.204.0
>Reporter: Robert Joseph Evans
>Assignee: Robert Joseph Evans
> Attachments: MR-2535-0.20.20X-V1.patch
>
>
> Exception occurred because the job was retired and is removed from
> RetireJobCcahe and CompletedJobStatusStore. But, the
> JobClient creates a RunningJob with null status and profile, if getJob(JobID)
> is called again.
> So, Even-though not null check is there in the following user code, it did
> not help.
> 466 runningJob = jobClient.getJob(mapRedJobID);
> 467 if(runningJob != null) {
> JobClient.getJob() should return null if status is null.
> In trunk this is fixed by validating that the job status is not null every
> time it is updated, and also verifying that that the profile data is not null
> when created.
--
This message is automatically generated by JIRA.
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (MAPREDUCE-2535) JobClient creates a RunningJob with null status and profile
[
https://issues.apache.org/jira/browse/MAPREDUCE-2535?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13039755#comment-13039755
]
Hadoop QA commented on MAPREDUCE-2535:
--
-1 overall. Here are the results of testing the latest attachment
http://issues.apache.org/jira/secure/attachment/12480551/MR-2535-0.20.20X-V1.patch
against trunk revision 1127444.
+1 @author. The patch does not contain any @author tags.
+1 tests included. The patch appears to include 4 new or modified tests.
-1 patch. The patch command could not apply the patch.
Console output:
https://builds.apache.org/hudson/job/PreCommit-MAPREDUCE-Build/307//console
This message is automatically generated.
> JobClient creates a RunningJob with null status and profile
> ---
>
> Key: MAPREDUCE-2535
> URL: https://issues.apache.org/jira/browse/MAPREDUCE-2535
> Project: Hadoop Map/Reduce
> Issue Type: Bug
> Components: client
>Affects Versions: 0.20.204.0
>Reporter: Robert Joseph Evans
>Assignee: Robert Joseph Evans
> Attachments: MR-2535-0.20.20X-V1.patch
>
>
> Exception occurred because the job was retired and is removed from
> RetireJobCcahe and CompletedJobStatusStore. But, the
> JobClient creates a RunningJob with null status and profile, if getJob(JobID)
> is called again.
> So, Even-though not null check is there in the following user code, it did
> not help.
> 466 runningJob = jobClient.getJob(mapRedJobID);
> 467 if(runningJob != null) {
> JobClient.getJob() should return null if status is null.
> In trunk this is fixed by validating that the job status is not null every
> time it is updated, and also verifying that that the profile data is not null
> when created.
--
This message is automatically generated by JIRA.
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (MAPREDUCE-2517) Porting Gridmix v3 system tests into trunk branch.
[ https://issues.apache.org/jira/browse/MAPREDUCE-2517?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13039756#comment-13039756 ] Hadoop QA commented on MAPREDUCE-2517: -- -1 overall. Here are the results of testing the latest attachment http://issues.apache.org/jira/secure/attachment/12480528/MAPREDUCE-2517-v4.patch against trunk revision 1127444. +1 @author. The patch does not contain any @author tags. +1 tests included. The patch appears to include 55 new or modified tests. +1 javadoc. The javadoc tool did not generate any warning messages. +1 javac. The applied patch does not increase the total number of javac compiler warnings. -1 findbugs. The patch appears to introduce 1 new Findbugs (version 1.3.9) warnings. +1 release audit. The applied patch does not increase the total number of release audit warnings. -1 core tests. The patch failed these core unit tests: org.apache.hadoop.cli.TestMRCLI org.apache.hadoop.tools.TestHadoopArchives org.apache.hadoop.tools.TestHarFileSystem -1 contrib tests. The patch failed contrib unit tests. +1 system test framework. The patch passed system test framework compile. Test results: https://builds.apache.org/hudson/job/PreCommit-MAPREDUCE-Build/306//testReport/ Findbugs warnings: https://builds.apache.org/hudson/job/PreCommit-MAPREDUCE-Build/306//artifact/trunk/build/test/findbugs/newPatchFindbugsWarnings.html Console output: https://builds.apache.org/hudson/job/PreCommit-MAPREDUCE-Build/306//console This message is automatically generated. > Porting Gridmix v3 system tests into trunk branch. > -- > > Key: MAPREDUCE-2517 > URL: https://issues.apache.org/jira/browse/MAPREDUCE-2517 > Project: Hadoop Map/Reduce > Issue Type: Task > Components: contrib/gridmix >Reporter: Vinay Kumar Thota >Assignee: Vinay Kumar Thota > Fix For: 0.23.0 > > Attachments: MAPREDUCE-2517-v2.patch, MAPREDUCE-2517-v3.patch, > MAPREDUCE-2517-v4.patch, MAPREDUCE-2517.patch > > > Porting of girdmix v3 system tests into trunk branch. -- This message is automatically generated by JIRA. For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Updated] (MAPREDUCE-2094) org.apache.hadoop.mapreduce.lib.input.FileInputFormat: isSplitable implements unsafe default behaviour that is different from the documented behaviour.
[ https://issues.apache.org/jira/browse/MAPREDUCE-2094?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Todd Lipcon updated MAPREDUCE-2094: --- Status: Open (was: Patch Available) cancelling patch for discussion > org.apache.hadoop.mapreduce.lib.input.FileInputFormat: isSplitable implements > unsafe default behaviour that is different from the documented behaviour. > --- > > Key: MAPREDUCE-2094 > URL: https://issues.apache.org/jira/browse/MAPREDUCE-2094 > Project: Hadoop Map/Reduce > Issue Type: Bug > Components: task >Reporter: Niels Basjes >Assignee: Niels Basjes > Attachments: MAPREDUCE-2094-2011-05-19.patch > > > When implementing a custom derivative of FileInputFormat we ran into the > effect that a large Gzipped input file would be processed several times. > A near 1GiB file would be processed around 36 times in its entirety. Thus > producing garbage results and taking up a lot more CPU time than needed. > It took a while to figure out and what we found is that the default > implementation of the isSplittable method in > [org.apache.hadoop.mapreduce.lib.input.FileInputFormat | > http://svn.apache.org/viewvc/hadoop/mapreduce/trunk/src/java/org/apache/hadoop/mapreduce/lib/input/FileInputFormat.java?view=markup > ] is simply "return true;". > This is a very unsafe default and is in contradiction with the JavaDoc of the > method which states: "Is the given filename splitable? Usually, true, but if > the file is stream compressed, it will not be. " . The actual implementation > effectively does "Is the given filename splitable? Always true, even if the > file is stream compressed using an unsplittable compression codec. " > For our situation (where we always have Gzipped input) we took the easy way > out and simply implemented an isSplittable in our class that does "return > false; " > Now there are essentially 3 ways I can think of for fixing this (in order of > what I would find preferable): > # Implement something that looks at the used compression of the file (i.e. do > migrate the implementation from TextInputFormat to FileInputFormat). This > would make the method do what the JavaDoc describes. > # "Force" developers to think about it and make this method abstract. > # Use a "safe" default (i.e. return false) -- This message is automatically generated by JIRA. For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (MAPREDUCE-2408) Make Gridmix emulate usage of data compression
[ https://issues.apache.org/jira/browse/MAPREDUCE-2408?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13039821#comment-13039821 ] Amar Kamat commented on MAPREDUCE-2408: --- The goal of this jira is to emulate the compression characteristics of a MapReduce job. Emulating compression characteristics involves the following 1. Generating compressible data. The compression characteristics (e.g compression ratio) of the data (map input, map output and reduce output) should be configurable. 2. Extract compression related properties from original job's configuration and history files. Configure the simulated job to mimic the compression behavior using the original job's configuration and history. > Make Gridmix emulate usage of data compression > -- > > Key: MAPREDUCE-2408 > URL: https://issues.apache.org/jira/browse/MAPREDUCE-2408 > Project: Hadoop Map/Reduce > Issue Type: New Feature > Components: contrib/gridmix >Reporter: Ravi Gummadi >Assignee: Amar Kamat > > Currently Gridmix emulates disk IO load only. This JIRA is to make Gridmix > emulate load due to data compression as defined by the job-trace. -- This message is automatically generated by JIRA. For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Updated] (MAPREDUCE-2408) Make Gridmix emulate usage of data compression
[ https://issues.apache.org/jira/browse/MAPREDUCE-2408?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Amar Kamat updated MAPREDUCE-2408: -- Status: Patch Available (was: Open) Running through Hudson. > Make Gridmix emulate usage of data compression > -- > > Key: MAPREDUCE-2408 > URL: https://issues.apache.org/jira/browse/MAPREDUCE-2408 > Project: Hadoop Map/Reduce > Issue Type: New Feature > Components: contrib/gridmix >Reporter: Ravi Gummadi >Assignee: Amar Kamat > Attachments: MR-2408-gridmix-compression-emulation-v1.1.patch > > > Currently Gridmix emulates disk IO load only. This JIRA is to make Gridmix > emulate load due to data compression as defined by the job-trace. -- This message is automatically generated by JIRA. For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Updated] (MAPREDUCE-2408) Make Gridmix emulate usage of data compression
[ https://issues.apache.org/jira/browse/MAPREDUCE-2408?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Amar Kamat updated MAPREDUCE-2408: -- Attachment: MR-2408-gridmix-compression-emulation-v1.1.patch Attaching a patch implementing compression emulation support in Gridmix. test-patch and ant tests passed. Manually tested the patch. > Make Gridmix emulate usage of data compression > -- > > Key: MAPREDUCE-2408 > URL: https://issues.apache.org/jira/browse/MAPREDUCE-2408 > Project: Hadoop Map/Reduce > Issue Type: New Feature > Components: contrib/gridmix >Reporter: Ravi Gummadi >Assignee: Amar Kamat > Attachments: MR-2408-gridmix-compression-emulation-v1.1.patch > > > Currently Gridmix emulates disk IO load only. This JIRA is to make Gridmix > emulate load due to data compression as defined by the job-trace. -- This message is automatically generated by JIRA. For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (MAPREDUCE-1347) Missing synchronization in MultipleOutputFormat
[ https://issues.apache.org/jira/browse/MAPREDUCE-1347?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13039830#comment-13039830 ] Todd Lipcon commented on MAPREDUCE-1347: - for the guava dependency, it should probably be conf="mapred->default" rather than releaseaudit (otherwise the jars won't end up in the built package) - the constant for version should be "guava.version" to match the naming convention - this test case seems unlikely to actually fail even if the bug persisted -- if you revert the fix and keep the test case, does it fail more than 50% of the time? Would it be possible to directly instantiate the outputformat with some mocks without having to go through running a whole job? Then you could hit it harder since you wouldn't have to touch disk. - maybe add a comment explaining that the use of the computing map is to make sure this code is thread-safe? > Missing synchronization in MultipleOutputFormat > --- > > Key: MAPREDUCE-1347 > URL: https://issues.apache.org/jira/browse/MAPREDUCE-1347 > Project: Hadoop Map/Reduce > Issue Type: Bug >Affects Versions: 0.20.2, 0.21.0, 0.22.0 >Reporter: Todd Lipcon >Assignee: Harsh J Chouraria > Attachments: MAPREDUCE-1347.r2.diff, MAPREDUCE-1347.r3.diff, > mapreduce.1347.r1.diff > > > MultipleOutputFormat's RecordWriter implementation doesn't use > synchronization when accessing the recordWriters member. When using > multithreaded mappers or reducers, this can result in problems where two > threads will both try to create the same file, causing > AlreadyBeingCreatedException. Doing this more fine-grained than just > synchronizing the whole method is probably a good idea, so that multithreaded > mappers can actually achieve parallelism writing into separate output streams. > From what I can tell, the new API's MultipleOutputs seems not to have this > issue. -- This message is automatically generated by JIRA. For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Updated] (MAPREDUCE-2531) org.apache.hadoop.mapred.jobcontrol.getAssignedJobID throw class cast exception
[ https://issues.apache.org/jira/browse/MAPREDUCE-2531?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Robert Joseph Evans updated MAPREDUCE-2531: --- Attachment: MR-2531-yarn-v1.patch Attached is the yarn patch. The test is still under ant, even though the code has been moved to mvn. Not sure if this is an issue or not. Also not all of the ant contrib tests even compile. My test runs and passes though. > org.apache.hadoop.mapred.jobcontrol.getAssignedJobID throw class cast > exception > > > Key: MAPREDUCE-2531 > URL: https://issues.apache.org/jira/browse/MAPREDUCE-2531 > Project: Hadoop Map/Reduce > Issue Type: Bug > Components: client >Affects Versions: 0.22.0 >Reporter: Robert Joseph Evans >Assignee: Robert Joseph Evans > Fix For: 0.23.0 > > Attachments: MR-2531-V1-trunk.patch, MR-2531-yarn-v1.patch > > > When using a combination of the mapred and mapreduce APIs (PIG) it is > possible to have the following exception > Caused by: java.lang.ClassCastException: org.apache.hadoop.mapreduce.JobID > cannot be cast to > org.apache.hadoop.mapred.JobID > at > org.apache.hadoop.mapred.jobcontrol.Job.getAssignedJobID(Job.java:71) > at > org.apache.pig.backend.hadoop.executionengine.mapReduceLayer.MapReduceLauncher.launchPig(MapReduceLauncher.java:239) > at org.apache.pig.PigServer.launchPlan(PigServer.java:1325) > ... 29 more > This is because the JobID is just downcast. It should be calling > JobID.downgrade -- This message is automatically generated by JIRA. For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (MAPREDUCE-2531) org.apache.hadoop.mapred.jobcontrol.getAssignedJobID throw class cast exception
[ https://issues.apache.org/jira/browse/MAPREDUCE-2531?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13039856#comment-13039856 ] Hadoop QA commented on MAPREDUCE-2531: -- -1 overall. Here are the results of testing the latest attachment http://issues.apache.org/jira/secure/attachment/12480569/MR-2531-yarn-v1.patch against trunk revision 1127444. +1 @author. The patch does not contain any @author tags. +1 tests included. The patch appears to include 3 new or modified tests. -1 patch. The patch command could not apply the patch. Console output: https://builds.apache.org/hudson/job/PreCommit-MAPREDUCE-Build/309//console This message is automatically generated. > org.apache.hadoop.mapred.jobcontrol.getAssignedJobID throw class cast > exception > > > Key: MAPREDUCE-2531 > URL: https://issues.apache.org/jira/browse/MAPREDUCE-2531 > Project: Hadoop Map/Reduce > Issue Type: Bug > Components: client >Affects Versions: 0.22.0 >Reporter: Robert Joseph Evans >Assignee: Robert Joseph Evans > Fix For: 0.23.0 > > Attachments: MR-2531-V1-trunk.patch, MR-2531-yarn-v1.patch > > > When using a combination of the mapred and mapreduce APIs (PIG) it is > possible to have the following exception > Caused by: java.lang.ClassCastException: org.apache.hadoop.mapreduce.JobID > cannot be cast to > org.apache.hadoop.mapred.JobID > at > org.apache.hadoop.mapred.jobcontrol.Job.getAssignedJobID(Job.java:71) > at > org.apache.pig.backend.hadoop.executionengine.mapReduceLayer.MapReduceLauncher.launchPig(MapReduceLauncher.java:239) > at org.apache.pig.PigServer.launchPlan(PigServer.java:1325) > ... 29 more > This is because the JobID is just downcast. It should be calling > JobID.downgrade -- This message is automatically generated by JIRA. For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Created] (MAPREDUCE-2536) TestMRCLI broke due to change in usage output
TestMRCLI broke due to change in usage output - Key: MAPREDUCE-2536 URL: https://issues.apache.org/jira/browse/MAPREDUCE-2536 Project: Hadoop Map/Reduce Issue Type: Test Components: test Affects Versions: 0.23.0 Reporter: Daryn Sharp One of the tests broke because it checks the FsShell mv usage line that is emitted after an error. The usage was updated to from "-mv " to "-mv ... ", so the "..." broke the test. -- This message is automatically generated by JIRA. For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Assigned] (MAPREDUCE-2536) TestMRCLI broke due to change in usage output
[ https://issues.apache.org/jira/browse/MAPREDUCE-2536?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Tsz Wo (Nicholas), SZE reassigned MAPREDUCE-2536: - Assignee: Tsz Wo (Nicholas), SZE > TestMRCLI broke due to change in usage output > - > > Key: MAPREDUCE-2536 > URL: https://issues.apache.org/jira/browse/MAPREDUCE-2536 > Project: Hadoop Map/Reduce > Issue Type: Test > Components: test >Affects Versions: 0.23.0 >Reporter: Daryn Sharp >Assignee: Tsz Wo (Nicholas), SZE > > One of the tests broke because it checks the FsShell mv usage line that is > emitted after an error. The usage was updated to from "-mv " to > "-mv ... ", so the "..." broke the test. -- This message is automatically generated by JIRA. For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Updated] (MAPREDUCE-2536) TestMRCLI broke due to change in usage output
[ https://issues.apache.org/jira/browse/MAPREDUCE-2536?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Tsz Wo (Nicholas), SZE updated MAPREDUCE-2536: -- Priority: Minor (was: Major) Assignee: Daryn Sharp (was: Tsz Wo (Nicholas), SZE) > TestMRCLI broke due to change in usage output > - > > Key: MAPREDUCE-2536 > URL: https://issues.apache.org/jira/browse/MAPREDUCE-2536 > Project: Hadoop Map/Reduce > Issue Type: Test > Components: test >Affects Versions: 0.23.0 >Reporter: Daryn Sharp >Assignee: Daryn Sharp >Priority: Minor > > One of the tests broke because it checks the FsShell mv usage line that is > emitted after an error. The usage was updated to from "-mv " to > "-mv ... ", so the "..." broke the test. -- This message is automatically generated by JIRA. For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Updated] (MAPREDUCE-2536) TestMRCLI broke due to change in usage output
[ https://issues.apache.org/jira/browse/MAPREDUCE-2536?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Daryn Sharp updated MAPREDUCE-2536: --- Attachment: MAPREDUCE-2536.patch Add "..." to the expected usage for mv. > TestMRCLI broke due to change in usage output > - > > Key: MAPREDUCE-2536 > URL: https://issues.apache.org/jira/browse/MAPREDUCE-2536 > Project: Hadoop Map/Reduce > Issue Type: Test > Components: test >Affects Versions: 0.23.0 >Reporter: Daryn Sharp >Assignee: Daryn Sharp >Priority: Minor > Attachments: MAPREDUCE-2536.patch > > > One of the tests broke because it checks the FsShell mv usage line that is > emitted after an error. The usage was updated to from "-mv " to > "-mv ... ", so the "..." broke the test. -- This message is automatically generated by JIRA. For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Updated] (MAPREDUCE-2536) TestMRCLI broke due to change in usage output
[ https://issues.apache.org/jira/browse/MAPREDUCE-2536?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Daryn Sharp updated MAPREDUCE-2536: --- Status: Patch Available (was: Open) > TestMRCLI broke due to change in usage output > - > > Key: MAPREDUCE-2536 > URL: https://issues.apache.org/jira/browse/MAPREDUCE-2536 > Project: Hadoop Map/Reduce > Issue Type: Test > Components: test >Affects Versions: 0.23.0 >Reporter: Daryn Sharp >Assignee: Daryn Sharp >Priority: Minor > Attachments: MAPREDUCE-2536.patch > > > One of the tests broke because it checks the FsShell mv usage line that is > emitted after an error. The usage was updated to from "-mv " to > "-mv ... ", so the "..." broke the test. -- This message is automatically generated by JIRA. For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Created] (MAPREDUCE-2537) MR-279: The RM writes its log to yarn-mapred-resourcemanager-.out
MR-279: The RM writes its log to yarn-mapred-resourcemanager-.out -- Key: MAPREDUCE-2537 URL: https://issues.apache.org/jira/browse/MAPREDUCE-2537 Project: Hadoop Map/Reduce Issue Type: Bug Components: mrv2 Reporter: Robert Joseph Evans Assignee: Robert Joseph Evans Priority: Minor -- This message is automatically generated by JIRA. For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (MAPREDUCE-2529) Recognize Jetty bug 1342 and handle it
[ https://issues.apache.org/jira/browse/MAPREDUCE-2529?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13039909#comment-13039909 ] Thomas Graves commented on MAPREDUCE-2529: -- I'm proposing to add a new metric to the shuffle output metrics and increment it when it sees a configurable regex in the IOexception in the MapOutputServlet. This metric can then be viewed by external systems or potentially the health_check script (HADOOP-7144 should make that easier). Making it configurable will make it more useful in the future in case we see other Jetty/JVM exceptions/issues that need to be worked around. > Recognize Jetty bug 1342 and handle it > -- > > Key: MAPREDUCE-2529 > URL: https://issues.apache.org/jira/browse/MAPREDUCE-2529 > Project: Hadoop Map/Reduce > Issue Type: Bug > Components: tasktracker >Affects Versions: 0.20.204.0, 0.23.0 >Reporter: Thomas Graves >Assignee: Thomas Graves > > We are seeing many instances of the Jetty-1342 > (http://jira.codehaus.org/browse/JETTY-1342). The bug doesn't cause Jetty to > stop responding altogether, some fetches go through but a lot of them throw > exceptions and eventually fail. The only way we have found to get the TT out > of this state is to restart the TT. This jira is to catch this particular > exception (or perhaps a configurable regex) and handle it in an automated way > to either blacklist or shutdown the TT after seeing it a configurable number > of them. -- This message is automatically generated by JIRA. For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (MAPREDUCE-2537) MR-279: The RM writes its log to yarn-mapred-resourcemanager-.out
[ https://issues.apache.org/jira/browse/MAPREDUCE-2537?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13039953#comment-13039953 ] Robert Joseph Evans commented on MAPREDUCE-2537: Output should end in .log > MR-279: The RM writes its log to yarn-mapred-resourcemanager-.out > -- > > Key: MAPREDUCE-2537 > URL: https://issues.apache.org/jira/browse/MAPREDUCE-2537 > Project: Hadoop Map/Reduce > Issue Type: Bug > Components: mrv2 >Reporter: Robert Joseph Evans >Assignee: Robert Joseph Evans >Priority: Minor > -- This message is automatically generated by JIRA. For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (MAPREDUCE-2537) MR-279: The RM writes its log to yarn-mapred-resourcemanager-.out
[ https://issues.apache.org/jira/browse/MAPREDUCE-2537?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13039957#comment-13039957 ] Luke Lu commented on MAPREDUCE-2537: I tried to fix this pre yarn move to MR-279 branch. The problem is that we don't set -Dhadoop.root.logger for log4j.properties and it defaults to INFO,console which is redirected to .out. I think just fixing the scripts to define hadoop.root.logger to as the same as yarn.root.logger and hadoop.log.dir, yarn.log.dir would fix the problem. > MR-279: The RM writes its log to yarn-mapred-resourcemanager-.out > -- > > Key: MAPREDUCE-2537 > URL: https://issues.apache.org/jira/browse/MAPREDUCE-2537 > Project: Hadoop Map/Reduce > Issue Type: Bug > Components: mrv2 >Reporter: Robert Joseph Evans >Assignee: Robert Joseph Evans >Priority: Minor > -- This message is automatically generated by JIRA. For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Updated] (MAPREDUCE-2537) MR-279: The RM writes its log to yarn-mapred-resourcemanager-.out
[ https://issues.apache.org/jira/browse/MAPREDUCE-2537?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Robert Joseph Evans updated MAPREDUCE-2537: --- Attachment: MR-2537-v1.patch Actually it is set in the shell script yarn/bin/yarn-daemon.sh > MR-279: The RM writes its log to yarn-mapred-resourcemanager-.out > -- > > Key: MAPREDUCE-2537 > URL: https://issues.apache.org/jira/browse/MAPREDUCE-2537 > Project: Hadoop Map/Reduce > Issue Type: Bug > Components: mrv2 >Reporter: Robert Joseph Evans >Assignee: Robert Joseph Evans >Priority: Minor > Attachments: MR-2537-v1.patch > > -- This message is automatically generated by JIRA. For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Updated] (MAPREDUCE-2537) MR-279: The RM writes its log to yarn-mapred-resourcemanager-.out
[ https://issues.apache.org/jira/browse/MAPREDUCE-2537?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Robert Joseph Evans updated MAPREDUCE-2537: --- Status: Patch Available (was: Open) No unit tests, because this is a very simple shell script fix. > MR-279: The RM writes its log to yarn-mapred-resourcemanager-.out > -- > > Key: MAPREDUCE-2537 > URL: https://issues.apache.org/jira/browse/MAPREDUCE-2537 > Project: Hadoop Map/Reduce > Issue Type: Bug > Components: mrv2 >Reporter: Robert Joseph Evans >Assignee: Robert Joseph Evans >Priority: Minor > Attachments: MR-2537-v1.patch > > -- This message is automatically generated by JIRA. For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (MAPREDUCE-2537) MR-279: The RM writes its log to yarn-mapred-resourcemanager-.out
[ https://issues.apache.org/jira/browse/MAPREDUCE-2537?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13039965#comment-13039965 ] Hadoop QA commented on MAPREDUCE-2537: -- -1 overall. Here are the results of testing the latest attachment http://issues.apache.org/jira/secure/attachment/12480589/MR-2537-v1.patch against trunk revision 1127444. +1 @author. The patch does not contain any @author tags. -1 tests included. The patch doesn't appear to include any new or modified tests. Please justify why no new tests are needed for this patch. Also please list what manual steps were performed to verify this patch. -1 patch. The patch command could not apply the patch. Console output: https://builds.apache.org/hudson/job/PreCommit-MAPREDUCE-Build/311//console This message is automatically generated. > MR-279: The RM writes its log to yarn-mapred-resourcemanager-.out > -- > > Key: MAPREDUCE-2537 > URL: https://issues.apache.org/jira/browse/MAPREDUCE-2537 > Project: Hadoop Map/Reduce > Issue Type: Bug > Components: mrv2 >Reporter: Robert Joseph Evans >Assignee: Robert Joseph Evans >Priority: Minor > Attachments: MR-2537-v1.patch > > -- This message is automatically generated by JIRA. For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Updated] (MAPREDUCE-2521) Mapreduce RPM integration project
[ https://issues.apache.org/jira/browse/MAPREDUCE-2521?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Eric Yang updated MAPREDUCE-2521: - Attachment: MAPREDUCE-2521-7.patch Minor bug fixes: - Removed *.debian and *.redhat script from /usr/sbin - Renamed package from hadoop-mapred to hadoop-mapreduce - Renamed dependency from hadoop to hadoop-common > Mapreduce RPM integration project > - > > Key: MAPREDUCE-2521 > URL: https://issues.apache.org/jira/browse/MAPREDUCE-2521 > Project: Hadoop Map/Reduce > Issue Type: New Feature > Components: build > Environment: Java 6, RHEL 5.5 >Reporter: Eric Yang >Assignee: Eric Yang > Attachments: MAPREDUCE-2521-1.patch, MAPREDUCE-2521-2.patch, > MAPREDUCE-2521-3.patch, MAPREDUCE-2521-4.patch, MAPREDUCE-2521-5.patch, > MAPREDUCE-2521-6.patch, MAPREDUCE-2521-7.patch, MAPREDUCE-2521.patch > > > This jira is corresponding to HADOOP-6255 and associated directory layout > change. The patch for creating Mapreduce rpm packaging should be posted here > for patch test build to verify against mapreduce svn trunk. -- This message is automatically generated by JIRA. For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (MAPREDUCE-2537) MR-279: The RM writes its log to yarn-mapred-resourcemanager-.out
[ https://issues.apache.org/jira/browse/MAPREDUCE-2537?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13039974#comment-13039974 ] Luke Lu commented on MAPREDUCE-2537: This is actually not the right fix. It basically redirect stdout to .log which is not the right thing to do. Currently in our logs dir, you'll see both .log and .out files. You'll notice that .log rolls daily but .out doesn't. Although yarn-daemon.sh defines YARN_ROOT_LOGGER as INFO,DRFA it's not being used to define -Dhadoop.root.logger which can then be picked up by log4j.properties. > MR-279: The RM writes its log to yarn-mapred-resourcemanager-.out > -- > > Key: MAPREDUCE-2537 > URL: https://issues.apache.org/jira/browse/MAPREDUCE-2537 > Project: Hadoop Map/Reduce > Issue Type: Bug > Components: mrv2 >Reporter: Robert Joseph Evans >Assignee: Robert Joseph Evans >Priority: Minor > Attachments: MR-2537-v1.patch > > -- This message is automatically generated by JIRA. For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (MAPREDUCE-2408) Make Gridmix emulate usage of data compression
[ https://issues.apache.org/jira/browse/MAPREDUCE-2408?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13039977#comment-13039977 ] Hadoop QA commented on MAPREDUCE-2408: -- -1 overall. Here are the results of testing the latest attachment http://issues.apache.org/jira/secure/attachment/12480563/MR-2408-gridmix-compression-emulation-v1.1.patch against trunk revision 1127444. +1 @author. The patch does not contain any @author tags. +1 tests included. The patch appears to include 6 new or modified tests. +1 javadoc. The javadoc tool did not generate any warning messages. +1 javac. The applied patch does not increase the total number of javac compiler warnings. -1 findbugs. The patch appears to introduce 1 new Findbugs (version 1.3.9) warnings. +1 release audit. The applied patch does not increase the total number of release audit warnings. -1 core tests. The patch failed these core unit tests: org.apache.hadoop.cli.TestMRCLI org.apache.hadoop.tools.TestHadoopArchives org.apache.hadoop.tools.TestHarFileSystem -1 contrib tests. The patch failed contrib unit tests. +1 system test framework. The patch passed system test framework compile. Test results: https://builds.apache.org/hudson/job/PreCommit-MAPREDUCE-Build/308//testReport/ Findbugs warnings: https://builds.apache.org/hudson/job/PreCommit-MAPREDUCE-Build/308//artifact/trunk/build/test/findbugs/newPatchFindbugsWarnings.html Console output: https://builds.apache.org/hudson/job/PreCommit-MAPREDUCE-Build/308//console This message is automatically generated. > Make Gridmix emulate usage of data compression > -- > > Key: MAPREDUCE-2408 > URL: https://issues.apache.org/jira/browse/MAPREDUCE-2408 > Project: Hadoop Map/Reduce > Issue Type: New Feature > Components: contrib/gridmix >Reporter: Ravi Gummadi >Assignee: Amar Kamat > Attachments: MR-2408-gridmix-compression-emulation-v1.1.patch > > > Currently Gridmix emulates disk IO load only. This JIRA is to make Gridmix > emulate load due to data compression as defined by the job-trace. -- This message is automatically generated by JIRA. For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Updated] (MAPREDUCE-2524) Backport trunk heuristics for failing maps when we get fetch failures retrieving map output during shuffle
[ https://issues.apache.org/jira/browse/MAPREDUCE-2524?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Thomas Graves updated MAPREDUCE-2524: - Attachment: MAPREDUCE2524-patch-20security.txt reattaching wit license granted. > Backport trunk heuristics for failing maps when we get fetch failures > retrieving map output during shuffle > -- > > Key: MAPREDUCE-2524 > URL: https://issues.apache.org/jira/browse/MAPREDUCE-2524 > Project: Hadoop Map/Reduce > Issue Type: Improvement > Components: tasktracker >Affects Versions: 0.20.204.0 >Reporter: Thomas Graves >Assignee: Thomas Graves >Priority: Minor > Fix For: 0.20.205.0 > > Attachments: MAPREDUCE2524-patch-20security.txt, > MAPREDUCE2524-patch-20security.txt > > > The heuristics for failing maps when we get map output fetch failures during > the shuffle is pretty conservative in 20. Backport the heuristics from trunk > which are more aggressive, simpler, and configurable. -- This message is automatically generated by JIRA. For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Updated] (MAPREDUCE-2524) Backport trunk heuristics for failing maps when we get fetch failures retrieving map output during shuffle
[ https://issues.apache.org/jira/browse/MAPREDUCE-2524?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Thomas Graves updated MAPREDUCE-2524: - Attachment: MAPREDUCE2524-20security-v2.patch upload new patch with changes in mapred-default.xml > Backport trunk heuristics for failing maps when we get fetch failures > retrieving map output during shuffle > -- > > Key: MAPREDUCE-2524 > URL: https://issues.apache.org/jira/browse/MAPREDUCE-2524 > Project: Hadoop Map/Reduce > Issue Type: Improvement > Components: tasktracker >Affects Versions: 0.20.204.0 >Reporter: Thomas Graves >Assignee: Thomas Graves >Priority: Minor > Fix For: 0.20.205.0 > > Attachments: MAPREDUCE2524-20security-v2.patch, > MAPREDUCE2524-patch-20security.txt, MAPREDUCE2524-patch-20security.txt > > > The heuristics for failing maps when we get map output fetch failures during > the shuffle is pretty conservative in 20. Backport the heuristics from trunk > which are more aggressive, simpler, and configurable. -- This message is automatically generated by JIRA. For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Updated] (MAPREDUCE-2524) Backport trunk heuristics for failing maps when we get fetch failures retrieving map output during shuffle
[ https://issues.apache.org/jira/browse/MAPREDUCE-2524?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Thomas Graves updated MAPREDUCE-2524: - Release Note: Added a new configuration option: mapreduce.reduce.shuffle.maxfetchfailures, and removed a no longer used option: mapred.reduce.copy.backoff. Status: Patch Available (was: Open) > Backport trunk heuristics for failing maps when we get fetch failures > retrieving map output during shuffle > -- > > Key: MAPREDUCE-2524 > URL: https://issues.apache.org/jira/browse/MAPREDUCE-2524 > Project: Hadoop Map/Reduce > Issue Type: Improvement > Components: tasktracker >Affects Versions: 0.20.204.0 >Reporter: Thomas Graves >Assignee: Thomas Graves >Priority: Minor > Fix For: 0.20.205.0 > > Attachments: MAPREDUCE2524-20security-v2.patch, > MAPREDUCE2524-patch-20security.txt, MAPREDUCE2524-patch-20security.txt > > > The heuristics for failing maps when we get map output fetch failures during > the shuffle is pretty conservative in 20. Backport the heuristics from trunk > which are more aggressive, simpler, and configurable. -- This message is automatically generated by JIRA. For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (MAPREDUCE-2536) TestMRCLI broke due to change in usage output
[ https://issues.apache.org/jira/browse/MAPREDUCE-2536?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13039985#comment-13039985 ] Hadoop QA commented on MAPREDUCE-2536: -- -1 overall. Here are the results of testing the latest attachment http://issues.apache.org/jira/secure/attachment/12480579/MAPREDUCE-2536.patch against trunk revision 1127444. +1 @author. The patch does not contain any @author tags. +1 tests included. The patch appears to include 4 new or modified tests. +1 javadoc. The javadoc tool did not generate any warning messages. +1 javac. The applied patch does not increase the total number of javac compiler warnings. -1 findbugs. The patch appears to introduce 1 new Findbugs (version 1.3.9) warnings. +1 release audit. The applied patch does not increase the total number of release audit warnings. +1 core tests. The patch passed core unit tests. -1 contrib tests. The patch failed contrib unit tests. +1 system test framework. The patch passed system test framework compile. Test results: https://builds.apache.org/hudson/job/PreCommit-MAPREDUCE-Build/310//testReport/ Findbugs warnings: https://builds.apache.org/hudson/job/PreCommit-MAPREDUCE-Build/310//artifact/trunk/build/test/findbugs/newPatchFindbugsWarnings.html Console output: https://builds.apache.org/hudson/job/PreCommit-MAPREDUCE-Build/310//console This message is automatically generated. > TestMRCLI broke due to change in usage output > - > > Key: MAPREDUCE-2536 > URL: https://issues.apache.org/jira/browse/MAPREDUCE-2536 > Project: Hadoop Map/Reduce > Issue Type: Test > Components: test >Affects Versions: 0.23.0 >Reporter: Daryn Sharp >Assignee: Daryn Sharp >Priority: Minor > Attachments: MAPREDUCE-2536.patch > > > One of the tests broke because it checks the FsShell mv usage line that is > emitted after an error. The usage was updated to from "-mv " to > "-mv ... ", so the "..." broke the test. -- This message is automatically generated by JIRA. For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (MAPREDUCE-2524) Backport trunk heuristics for failing maps when we get fetch failures retrieving map output during shuffle
[ https://issues.apache.org/jira/browse/MAPREDUCE-2524?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13039987#comment-13039987 ] Hadoop QA commented on MAPREDUCE-2524: -- -1 overall. Here are the results of testing the latest attachment http://issues.apache.org/jira/secure/attachment/12480599/MAPREDUCE2524-20security-v2.patch against trunk revision 1127444. +1 @author. The patch does not contain any @author tags. +1 tests included. The patch appears to include 3 new or modified tests. -1 patch. The patch command could not apply the patch. Console output: https://builds.apache.org/hudson/job/PreCommit-MAPREDUCE-Build/313//console This message is automatically generated. > Backport trunk heuristics for failing maps when we get fetch failures > retrieving map output during shuffle > -- > > Key: MAPREDUCE-2524 > URL: https://issues.apache.org/jira/browse/MAPREDUCE-2524 > Project: Hadoop Map/Reduce > Issue Type: Improvement > Components: tasktracker >Affects Versions: 0.20.204.0 >Reporter: Thomas Graves >Assignee: Thomas Graves >Priority: Minor > Fix For: 0.20.205.0 > > Attachments: MAPREDUCE2524-20security-v2.patch, > MAPREDUCE2524-patch-20security.txt, MAPREDUCE2524-patch-20security.txt > > > The heuristics for failing maps when we get map output fetch failures during > the shuffle is pretty conservative in 20. Backport the heuristics from trunk > which are more aggressive, simpler, and configurable. -- This message is automatically generated by JIRA. For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (MAPREDUCE-2524) Backport trunk heuristics for failing maps when we get fetch failures retrieving map output during shuffle
[ https://issues.apache.org/jira/browse/MAPREDUCE-2524?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13039988#comment-13039988 ] Thomas Graves commented on MAPREDUCE-2524: -- Note, this failed because patch is for branch-0.20-security > Backport trunk heuristics for failing maps when we get fetch failures > retrieving map output during shuffle > -- > > Key: MAPREDUCE-2524 > URL: https://issues.apache.org/jira/browse/MAPREDUCE-2524 > Project: Hadoop Map/Reduce > Issue Type: Improvement > Components: tasktracker >Affects Versions: 0.20.204.0 >Reporter: Thomas Graves >Assignee: Thomas Graves >Priority: Minor > Fix For: 0.20.205.0 > > Attachments: MAPREDUCE2524-20security-v2.patch, > MAPREDUCE2524-patch-20security.txt, MAPREDUCE2524-patch-20security.txt > > > The heuristics for failing maps when we get map output fetch failures during > the shuffle is pretty conservative in 20. Backport the heuristics from trunk > which are more aggressive, simpler, and configurable. -- This message is automatically generated by JIRA. For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (MAPREDUCE-2536) TestMRCLI broke due to change in usage output
[ https://issues.apache.org/jira/browse/MAPREDUCE-2536?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13039992#comment-13039992 ] Daryn Sharp commented on MAPREDUCE-2536: The patch changes text in only one xml file used an input for a test. The test that failed is completely unrelated. Also, something is wrong with findbugs because the changed xml file is not flagged, and the findbugs report is identical to earlier builds. Requesting review since this patch fixes the test it is intended to fix. > TestMRCLI broke due to change in usage output > - > > Key: MAPREDUCE-2536 > URL: https://issues.apache.org/jira/browse/MAPREDUCE-2536 > Project: Hadoop Map/Reduce > Issue Type: Test > Components: test >Affects Versions: 0.23.0 >Reporter: Daryn Sharp >Assignee: Daryn Sharp >Priority: Minor > Attachments: MAPREDUCE-2536.patch > > > One of the tests broke because it checks the FsShell mv usage line that is > emitted after an error. The usage was updated to from "-mv " to > "-mv ... ", so the "..." broke the test. -- This message is automatically generated by JIRA. For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (MAPREDUCE-279) Map-Reduce 2.0
[ https://issues.apache.org/jira/browse/MAPREDUCE-279?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13039993#comment-13039993 ] Tom White commented on MAPREDUCE-279: - I'm wondering what the maven modules might look like for this when integrated into trunk. Something like: * api - containing the user-facing public API of MapReduce (from org.apache.hadoop.mapred(uce)). When MAPREDUCE-1638 is done it will be possible to split the API into a self-contained tree (no dependencies on other parts of MapReduce). * lib - containing the user-facing public MapReduce libraries (from org.apache.hadoop.mapred and org.apache.hadoop.mapred(uce).lib). There's a patch in MAPREDUCE-1478 to perform this separation. * classic-impl - containing the implementation classes for MapReduce. This is what's left over after doing MAPREDUCE-1638 and MAPREDUCE-1478. * nextgen-impl - this is mr-client in the MR-279 branch, which I think should be renamed, since it's not immediately clear what it's a client of in the context of the whole MapReduce project. It has submodules app, common, hs, jobclient, shuffle. * yarn - the yarn framework from the MR-279 branch. Yarn is broken into submodules too. Given the progress on mavenizing common (HADOOP-6671), is it worth integrating MAPREDUCE-279 at the same time as doing the full Mavenization of MapReduce? That would seem ideal, but perhaps there's an alternative I haven't considered. > Map-Reduce 2.0 > -- > > Key: MAPREDUCE-279 > URL: https://issues.apache.org/jira/browse/MAPREDUCE-279 > Project: Hadoop Map/Reduce > Issue Type: Improvement > Components: jobtracker, tasktracker >Reporter: Arun C Murthy >Assignee: Arun C Murthy > Fix For: 0.23.0 > > Attachments: MR-279.patch, MR-279.patch, MR-279.sh, > MR-279_MR_files_to_move.txt, capacity-scheduler-dark-theme.png, > multi-column-stable-sort-default-theme.png, yarn-state-machine.job.dot, > yarn-state-machine.job.png, yarn-state-machine.task-attempt.dot, > yarn-state-machine.task-attempt.png, yarn-state-machine.task.dot, > yarn-state-machine.task.png > > > Re-factor MapReduce into a generic resource scheduler and a per-job, > user-defined component that manages the application execution. -- This message is automatically generated by JIRA. For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Updated] (MAPREDUCE-2529) Recognize Jetty bug 1342 and handle it
[ https://issues.apache.org/jira/browse/MAPREDUCE-2529?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Thomas Graves updated MAPREDUCE-2529: - Attachment: jetty1342-20security.patch patch for branch-0.20-security > Recognize Jetty bug 1342 and handle it > -- > > Key: MAPREDUCE-2529 > URL: https://issues.apache.org/jira/browse/MAPREDUCE-2529 > Project: Hadoop Map/Reduce > Issue Type: Bug > Components: tasktracker >Affects Versions: 0.20.204.0, 0.23.0 >Reporter: Thomas Graves >Assignee: Thomas Graves > Attachments: jetty1342-20security.patch > > > We are seeing many instances of the Jetty-1342 > (http://jira.codehaus.org/browse/JETTY-1342). The bug doesn't cause Jetty to > stop responding altogether, some fetches go through but a lot of them throw > exceptions and eventually fail. The only way we have found to get the TT out > of this state is to restart the TT. This jira is to catch this particular > exception (or perhaps a configurable regex) and handle it in an automated way > to either blacklist or shutdown the TT after seeing it a configurable number > of them. -- This message is automatically generated by JIRA. For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Updated] (MAPREDUCE-2536) TestMRCLI broke due to change in usage output
[ https://issues.apache.org/jira/browse/MAPREDUCE-2536?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Tsz Wo (Nicholas), SZE updated MAPREDUCE-2536: -- Hadoop Flags: [Reviewed] +1 patch looks good. > TestMRCLI broke due to change in usage output > - > > Key: MAPREDUCE-2536 > URL: https://issues.apache.org/jira/browse/MAPREDUCE-2536 > Project: Hadoop Map/Reduce > Issue Type: Test > Components: test >Affects Versions: 0.23.0 >Reporter: Daryn Sharp >Assignee: Daryn Sharp >Priority: Minor > Fix For: 0.23.0 > > Attachments: MAPREDUCE-2536.patch > > > One of the tests broke because it checks the FsShell mv usage line that is > emitted after an error. The usage was updated to from "-mv " to > "-mv ... ", so the "..." broke the test. -- This message is automatically generated by JIRA. For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Updated] (MAPREDUCE-2536) TestMRCLI broke due to change in usage output
[ https://issues.apache.org/jira/browse/MAPREDUCE-2536?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Tsz Wo (Nicholas), SZE updated MAPREDUCE-2536: -- Resolution: Fixed Fix Version/s: 0.23.0 Status: Resolved (was: Patch Available) I have committed this. Thanks, Daryn! > TestMRCLI broke due to change in usage output > - > > Key: MAPREDUCE-2536 > URL: https://issues.apache.org/jira/browse/MAPREDUCE-2536 > Project: Hadoop Map/Reduce > Issue Type: Test > Components: test >Affects Versions: 0.23.0 >Reporter: Daryn Sharp >Assignee: Daryn Sharp >Priority: Minor > Fix For: 0.23.0 > > Attachments: MAPREDUCE-2536.patch > > > One of the tests broke because it checks the FsShell mv usage line that is > emitted after an error. The usage was updated to from "-mv " to > "-mv ... ", so the "..." broke the test. -- This message is automatically generated by JIRA. For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (MAPREDUCE-2536) TestMRCLI broke due to change in usage output
[ https://issues.apache.org/jira/browse/MAPREDUCE-2536?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13040004#comment-13040004 ] Hudson commented on MAPREDUCE-2536: --- Integrated in Hadoop-Mapreduce-trunk-Commit #701 (See [https://builds.apache.org/hudson/job/Hadoop-Mapreduce-trunk-Commit/701/]) Fix MAPREDUCE-2536 entry in CHANGES.txt. MAPREDUCE-2536. Update FsShell -mv command usage in TestMRCLI. Contributed by Daryn Sharp szetszwo : http://svn.apache.org/viewcvs.cgi/?root=Apache-SVN&view=rev&rev=1128125 Files : * /hadoop/mapreduce/trunk/CHANGES.txt szetszwo : http://svn.apache.org/viewcvs.cgi/?root=Apache-SVN&view=rev&rev=1128124 Files : * /hadoop/mapreduce/trunk/CHANGES.txt * /hadoop/mapreduce/trunk/src/test/mapred/org/apache/hadoop/cli/testMRConf.xml > TestMRCLI broke due to change in usage output > - > > Key: MAPREDUCE-2536 > URL: https://issues.apache.org/jira/browse/MAPREDUCE-2536 > Project: Hadoop Map/Reduce > Issue Type: Test > Components: test >Affects Versions: 0.23.0 >Reporter: Daryn Sharp >Assignee: Daryn Sharp >Priority: Minor > Fix For: 0.23.0 > > Attachments: MAPREDUCE-2536.patch > > > One of the tests broke because it checks the FsShell mv usage line that is > emitted after an error. The usage was updated to from "-mv " to > "-mv ... ", so the "..." broke the test. -- This message is automatically generated by JIRA. For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (MAPREDUCE-2521) Mapreduce RPM integration project
[ https://issues.apache.org/jira/browse/MAPREDUCE-2521?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13040022#comment-13040022 ] Hadoop QA commented on MAPREDUCE-2521: -- -1 overall. Here are the results of testing the latest attachment http://issues.apache.org/jira/secure/attachment/12480596/MAPREDUCE-2521-7.patch against trunk revision 1127444. +1 @author. The patch does not contain any @author tags. +1 tests included. The patch appears to include 16 new or modified tests. +1 javadoc. The javadoc tool did not generate any warning messages. +1 javac. The applied patch does not increase the total number of javac compiler warnings. -1 findbugs. The patch appears to introduce 1 new Findbugs (version 1.3.9) warnings. +1 release audit. The applied patch does not increase the total number of release audit warnings. -1 core tests. The patch failed these core unit tests: org.apache.hadoop.cli.TestMRCLI -1 contrib tests. The patch failed contrib unit tests. +1 system test framework. The patch passed system test framework compile. Test results: https://builds.apache.org/hudson/job/PreCommit-MAPREDUCE-Build/312//testReport/ Findbugs warnings: https://builds.apache.org/hudson/job/PreCommit-MAPREDUCE-Build/312//artifact/trunk/build/test/findbugs/newPatchFindbugsWarnings.html Console output: https://builds.apache.org/hudson/job/PreCommit-MAPREDUCE-Build/312//console This message is automatically generated. > Mapreduce RPM integration project > - > > Key: MAPREDUCE-2521 > URL: https://issues.apache.org/jira/browse/MAPREDUCE-2521 > Project: Hadoop Map/Reduce > Issue Type: New Feature > Components: build > Environment: Java 6, RHEL 5.5 >Reporter: Eric Yang >Assignee: Eric Yang > Attachments: MAPREDUCE-2521-1.patch, MAPREDUCE-2521-2.patch, > MAPREDUCE-2521-3.patch, MAPREDUCE-2521-4.patch, MAPREDUCE-2521-5.patch, > MAPREDUCE-2521-6.patch, MAPREDUCE-2521-7.patch, MAPREDUCE-2521.patch > > > This jira is corresponding to HADOOP-6255 and associated directory layout > change. The patch for creating Mapreduce rpm packaging should be posted here > for patch test build to verify against mapreduce svn trunk. -- This message is automatically generated by JIRA. For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Created] (MAPREDUCE-2538) InputSampler.writePartitionFile() may write duplicate keys
InputSampler.writePartitionFile() may write duplicate keys
--
Key: MAPREDUCE-2538
URL: https://issues.apache.org/jira/browse/MAPREDUCE-2538
Project: Hadoop Map/Reduce
Issue Type: Bug
Affects Versions: 0.20.2
Environment: EMR.
Reporter: Michael White
Priority: Minor
InputSampler.writePartitionFile() outputs the same key multiple times if the
input samples have enough of a given key to span multiple partitions. There is
logic in the code that appears to try to avoid this, but seems incorrect:
for(int i = 1; i < numPartitions; ++i) {
int k = Math.round(stepSize * i);
while (last >= k && comparator.compare(samples[last], samples[k]) == 0) {
++k;
}
writer.append(samples[k], nullValue);
last = k;
}
The while loop condition "last >= k" is always false. The sample comparison
after the && never occurs.
It's not entirely clear what the correct fix is. The current behavior is
arguably correct mathematically, though the while loop could be elided for
clarity. If bug MAPREDUCE-1987 were fixed, it would be less of a problem (for
me at least), since that is where the non-uniqueness causes me problems.
Alternatively, changing the while to:
"if( last >= 0) {
while (comparator.compare(samples[last], samples[k]) >= 0)) {"
or, optimized for skipping over many duplicates (but arguably less clear):
"if (last >= 0) {
while (last >= k || comparator.compare(samples[last], samples[k]) >= 0)) {"
would probably achieve what the original author intended.
Perhaps the behavior could be selected by a parameter, e.g. "boolean unique".
--
This message is automatically generated by JIRA.
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Updated] (MAPREDUCE-2421) Remove JobHistory's dependency on JobTracker
[ https://issues.apache.org/jira/browse/MAPREDUCE-2421?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Tom White updated MAPREDUCE-2421: - Attachment: MAPREDUCE-2421.patch I looked at extracting a JobsManager class, but the jobs map is deeply entangled in JobTracker, and there's not a clean separation here. More importantly, from the point of view of this JIRA it doesn't remove the dependency that JobHistory has on JobTracker. Replacing the dependency on JobTracker with one on a JobManager class, which has a map of JobInProgress objects doesn't help, since JobInProgess itself depends on JobTracker. So, unfortunately, this doesn't look like a workable approach. Instead I think the current patch is a reasonable solution - it's a standard technique of using a callback interface to facilitate decoupling (a bit like TaskTrackerManager which exists to make JobTracker more manageable for testing). I've regenerated the patch to mark the interface as @Private @Unstable and added some javadoc so it's clearer what its purpose is. > Remove JobHistory's dependency on JobTracker > > > Key: MAPREDUCE-2421 > URL: https://issues.apache.org/jira/browse/MAPREDUCE-2421 > Project: Hadoop Map/Reduce > Issue Type: Sub-task > Components: client >Reporter: Tom White >Assignee: Tom White > Attachments: MAPREDUCE-2421.patch, MAPREDUCE-2421.patch > > > The direct dependency can be removed by introducing a callback inteface. This > is for MAPREDUCE-1638. -- This message is automatically generated by JIRA. For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (MAPREDUCE-2137) Mapping between Gridmix jobs and the corresponding original MR jobs is needed
[ https://issues.apache.org/jira/browse/MAPREDUCE-2137?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13040049#comment-13040049 ] Ravi Gummadi commented on MAPREDUCE-2137: - TestMRCLI failure is a known issue and is not related to this patch. findbugs warnings shown are also not related to this patch. I will commit this patch now. > Mapping between Gridmix jobs and the corresponding original MR jobs is needed > - > > Key: MAPREDUCE-2137 > URL: https://issues.apache.org/jira/browse/MAPREDUCE-2137 > Project: Hadoop Map/Reduce > Issue Type: Bug > Components: contrib/gridmix >Affects Versions: 0.23.0 >Reporter: Ravi Gummadi >Assignee: Ravi Gummadi > Fix For: 0.23.0 > > Attachments: 2137.patch, 2137.v1.patch, 2137.v2.1.patch > > > Consider a trace file "trace1" obtained by running Rumen on a set of MR jobs' > history logs. When gridmix runs simulated jobs from "trace1", it may skip > some of the jobs from the trace file for some reason like out-of-order-jobs. > Now use Rumen to generate trace2 from the history logs of gridmix's simulated > jobs. > Now, to compare and analyze the gridmix's simulated jobs with original MR > jobs, we need a mapping between them. -- This message is automatically generated by JIRA. For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Updated] (MAPREDUCE-2137) Mapping between Gridmix jobs and the corresponding original MR jobs is needed
[ https://issues.apache.org/jira/browse/MAPREDUCE-2137?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Ravi Gummadi updated MAPREDUCE-2137: Resolution: Fixed Hadoop Flags: [Reviewed] Status: Resolved (was: Patch Available) I just committed this to trunk. > Mapping between Gridmix jobs and the corresponding original MR jobs is needed > - > > Key: MAPREDUCE-2137 > URL: https://issues.apache.org/jira/browse/MAPREDUCE-2137 > Project: Hadoop Map/Reduce > Issue Type: Bug > Components: contrib/gridmix >Affects Versions: 0.23.0 >Reporter: Ravi Gummadi >Assignee: Ravi Gummadi > Fix For: 0.23.0 > > Attachments: 2137.patch, 2137.v1.patch, 2137.v2.1.patch > > > Consider a trace file "trace1" obtained by running Rumen on a set of MR jobs' > history logs. When gridmix runs simulated jobs from "trace1", it may skip > some of the jobs from the trace file for some reason like out-of-order-jobs. > Now use Rumen to generate trace2 from the history logs of gridmix's simulated > jobs. > Now, to compare and analyze the gridmix's simulated jobs with original MR > jobs, we need a mapping between them. -- This message is automatically generated by JIRA. For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (MAPREDUCE-2137) Mapping between Gridmix jobs and the corresponding original MR jobs is needed
[ https://issues.apache.org/jira/browse/MAPREDUCE-2137?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13040054#comment-13040054 ] Hudson commented on MAPREDUCE-2137: --- Integrated in Hadoop-Mapreduce-trunk-Commit #702 (See [https://builds.apache.org/hudson/job/Hadoop-Mapreduce-trunk-Commit/702/]) MAPREDUCE-2137. Provide mapping between jobs of trace file and the corresponding simulated cluster's jobs in Gridmix. ravigummadi : http://svn.apache.org/viewcvs.cgi/?root=Apache-SVN&view=rev&rev=1128147 Files : * /hadoop/mapreduce/trunk/src/contrib/gridmix/src/java/org/apache/hadoop/mapred/gridmix/Gridmix.java * /hadoop/mapreduce/trunk/src/docs/src/documentation/content/xdocs/gridmix.xml * /hadoop/mapreduce/trunk/src/contrib/gridmix/src/test/org/apache/hadoop/mapred/gridmix/DebugJobProducer.java * /hadoop/mapreduce/trunk/src/contrib/gridmix/src/test/org/apache/hadoop/mapred/gridmix/TestGridmixSubmission.java * /hadoop/mapreduce/trunk/src/contrib/gridmix/src/java/org/apache/hadoop/mapred/gridmix/GridmixJob.java > Mapping between Gridmix jobs and the corresponding original MR jobs is needed > - > > Key: MAPREDUCE-2137 > URL: https://issues.apache.org/jira/browse/MAPREDUCE-2137 > Project: Hadoop Map/Reduce > Issue Type: Bug > Components: contrib/gridmix >Affects Versions: 0.23.0 >Reporter: Ravi Gummadi >Assignee: Ravi Gummadi > Fix For: 0.23.0 > > Attachments: 2137.patch, 2137.v1.patch, 2137.v2.1.patch > > > Consider a trace file "trace1" obtained by running Rumen on a set of MR jobs' > history logs. When gridmix runs simulated jobs from "trace1", it may skip > some of the jobs from the trace file for some reason like out-of-order-jobs. > Now use Rumen to generate trace2 from the history logs of gridmix's simulated > jobs. > Now, to compare and analyze the gridmix's simulated jobs with original MR > jobs, we need a mapping between them. -- This message is automatically generated by JIRA. For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (MAPREDUCE-2408) Make Gridmix emulate usage of data compression
[ https://issues.apache.org/jira/browse/MAPREDUCE-2408?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13040072#comment-13040072 ] Ravi Gummadi commented on MAPREDUCE-2408: - Tests failed are not related to this patch. Findbugs warnings reported by Hudson are also not related to this patch. Patch looks good to me. +1 > Make Gridmix emulate usage of data compression > -- > > Key: MAPREDUCE-2408 > URL: https://issues.apache.org/jira/browse/MAPREDUCE-2408 > Project: Hadoop Map/Reduce > Issue Type: New Feature > Components: contrib/gridmix >Reporter: Ravi Gummadi >Assignee: Amar Kamat > Attachments: MR-2408-gridmix-compression-emulation-v1.1.patch > > > Currently Gridmix emulates disk IO load only. This JIRA is to make Gridmix > emulate load due to data compression as defined by the job-trace. -- This message is automatically generated by JIRA. For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Updated] (MAPREDUCE-2408) Make Gridmix emulate usage of data compression
[ https://issues.apache.org/jira/browse/MAPREDUCE-2408?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Amar Kamat updated MAPREDUCE-2408: -- Resolution: Fixed Fix Version/s: 0.23.0 Release Note: Emulates the MapReduce compression feature in Gridmix. By default, compression emulation is turned on. Compression emulation can be disabled by setting 'gridmix.compression-emulation.enable' to 'false'. Use 'gridmix.compression-emulation.map-input.decompression-ratio', 'gridmix.compression-emulation.map-output.compression-ratio' and 'gridmix.compression-emulation.reduce-output.compression-ratio' to configure the compression ratios at map input, map output and reduce output side respectively. Currently, compression ratios in the range [0.07, 0.68] are supported. Gridmix auto detects whether map-input, map output and reduce output should emulate compression based on original job's compression related configuration parameters. Hadoop Flags: [Reviewed] Status: Resolved (was: Patch Available) I just committed this to trunk. Thanks Ravi for the review! > Make Gridmix emulate usage of data compression > -- > > Key: MAPREDUCE-2408 > URL: https://issues.apache.org/jira/browse/MAPREDUCE-2408 > Project: Hadoop Map/Reduce > Issue Type: New Feature > Components: contrib/gridmix >Reporter: Ravi Gummadi >Assignee: Amar Kamat > Fix For: 0.23.0 > > Attachments: MR-2408-gridmix-compression-emulation-v1.1.patch > > > Currently Gridmix emulates disk IO load only. This JIRA is to make Gridmix > emulate load due to data compression as defined by the job-trace. -- This message is automatically generated by JIRA. For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (MAPREDUCE-2408) Make Gridmix emulate usage of data compression
[ https://issues.apache.org/jira/browse/MAPREDUCE-2408?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13040085#comment-13040085 ] Hudson commented on MAPREDUCE-2408: --- Integrated in Hadoop-Mapreduce-trunk-Commit #703 (See [https://builds.apache.org/hudson/job/Hadoop-Mapreduce-trunk-Commit/703/]) MAPREDUCE-2408. [Gridmix] Compression emulation in Gridmix. (amarrk) amarrk : http://svn.apache.org/viewcvs.cgi/?root=Apache-SVN&view=rev&rev=1128162 Files : * /hadoop/mapreduce/trunk/CHANGES.txt * /hadoop/mapreduce/trunk/src/contrib/gridmix/src/java/org/apache/hadoop/mapred/gridmix/DistributedCacheEmulator.java * /hadoop/mapreduce/trunk/src/contrib/gridmix/src/java/org/apache/hadoop/mapred/gridmix/GridmixRecord.java * /hadoop/mapreduce/trunk/src/contrib/gridmix/src/java/org/apache/hadoop/mapred/gridmix/CompressionEmulationUtil.java * /hadoop/mapreduce/trunk/src/docs/src/documentation/content/xdocs/gridmix.xml * /hadoop/mapreduce/trunk/src/contrib/gridmix/src/test/org/apache/hadoop/mapred/gridmix/TestCompressionEmulationUtils.java * /hadoop/mapreduce/trunk/src/contrib/gridmix/src/java/org/apache/hadoop/mapred/gridmix/AvgRecordFactory.java * /hadoop/mapreduce/trunk/src/contrib/gridmix/src/java/org/apache/hadoop/mapred/gridmix/GridmixJob.java * /hadoop/mapreduce/trunk/src/contrib/gridmix/src/java/org/apache/hadoop/mapred/gridmix/SleepJob.java * /hadoop/mapreduce/trunk/src/contrib/gridmix/src/java/org/apache/hadoop/mapred/gridmix/RandomTextDataGenerator.java * /hadoop/mapreduce/trunk/src/contrib/gridmix/src/java/org/apache/hadoop/mapred/gridmix/LoadJob.java * /hadoop/mapreduce/trunk/src/contrib/gridmix/src/test/org/apache/hadoop/mapred/gridmix/TestRandomTextDataGenerator.java * /hadoop/mapreduce/trunk/src/contrib/gridmix/src/java/org/apache/hadoop/mapred/gridmix/Gridmix.java * /hadoop/mapreduce/trunk/src/contrib/gridmix/src/java/org/apache/hadoop/mapred/gridmix/FileQueue.java * /hadoop/mapreduce/trunk/src/contrib/gridmix/src/java/org/apache/hadoop/mapred/gridmix/GenerateData.java * /hadoop/mapreduce/trunk/src/contrib/gridmix/src/java/org/apache/hadoop/mapred/gridmix/GenerateDistCacheData.java * /hadoop/mapreduce/trunk/src/contrib/gridmix/src/java/org/apache/hadoop/mapred/gridmix/InputStriper.java > Make Gridmix emulate usage of data compression > -- > > Key: MAPREDUCE-2408 > URL: https://issues.apache.org/jira/browse/MAPREDUCE-2408 > Project: Hadoop Map/Reduce > Issue Type: New Feature > Components: contrib/gridmix >Reporter: Ravi Gummadi >Assignee: Amar Kamat > Fix For: 0.23.0 > > Attachments: MR-2408-gridmix-compression-emulation-v1.1.patch > > > Currently Gridmix emulates disk IO load only. This JIRA is to make Gridmix > emulate load due to data compression as defined by the job-trace. -- This message is automatically generated by JIRA. For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (MAPREDUCE-2517) Porting Gridmix v3 system tests into trunk branch.
[ https://issues.apache.org/jira/browse/MAPREDUCE-2517?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13040091#comment-13040091 ] Vinay Kumar Thota commented on MAPREDUCE-2517: -- * Checked the findbugs details and those are not related to this patch. * Checked the failed contrib tests log file and found that these failures are not related to this patch. Any how I ran the failed Gridmix unit tests manually with the patch and it got passed. * Checked the failed core tests log file and found that failures are not related to this patch. Any how I ran the failed core tests manually with the patch and it got passed. > Porting Gridmix v3 system tests into trunk branch. > -- > > Key: MAPREDUCE-2517 > URL: https://issues.apache.org/jira/browse/MAPREDUCE-2517 > Project: Hadoop Map/Reduce > Issue Type: Task > Components: contrib/gridmix >Reporter: Vinay Kumar Thota >Assignee: Vinay Kumar Thota > Fix For: 0.23.0 > > Attachments: MAPREDUCE-2517-v2.patch, MAPREDUCE-2517-v3.patch, > MAPREDUCE-2517-v4.patch, MAPREDUCE-2517.patch > > > Porting of girdmix v3 system tests into trunk branch. -- This message is automatically generated by JIRA. For more information on JIRA, see: http://www.atlassian.com/software/jira
