[jira] [Commented] (MAPREDUCE-2492) [MAPREDUCE] The new MapReduce API should make available task's progress to the task
[
https://issues.apache.org/jira/browse/MAPREDUCE-2492?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13037231#comment-13037231
]
Chris Douglas commented on MAPREDUCE-2492:
--
Adding progress to {{TaskAttemptContext}} makes sense. Minor points on the
tests:
* Please use JUnit4 annotations instead of extending {{TestCase}} for new tests
* Using {{@BeforeClass}} instead of an instance initializer block would be
easier to read and would fail appropriately
* Deleting "test.build.data" (defaulting to "/tmp") recursively when the test
runs could have bad side-effects. Please use a subdir of {{rootTestDir}} instead
* Cleaning up after the test would also be a good idea ({{@AfterClass}}). Right
now, many of the tests will not delete output when the test fails.
> [MAPREDUCE] The new MapReduce API should make available task's progress to
> the task
> ---
>
> Key: MAPREDUCE-2492
> URL: https://issues.apache.org/jira/browse/MAPREDUCE-2492
> Project: Hadoop Map/Reduce
> Issue Type: Improvement
> Components: task
>Affects Versions: 0.23.0
>Reporter: Amar Kamat
>Assignee: Amar Kamat
> Attachments: MAPREDUCE-2492-v1.3.patch
>
>
> There is no way to get the task's current progress in the new MapReduce API.
> It would be nice to make it available so that the task (map/reduce) can use
> it.
--
This message is automatically generated by JIRA.
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (MAPREDUCE-2407) Make Gridmix emulate usage of Distributed Cache files
[ https://issues.apache.org/jira/browse/MAPREDUCE-2407?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13037191#comment-13037191 ] Amar Kamat commented on MAPREDUCE-2407: --- The latest patch looks good to me. I have some minor comments (mostly alignment, refactoring and parameter naming) which I have discussed with Ravi offline. I don't want to block the patch just for some minor comments. +1. > Make Gridmix emulate usage of Distributed Cache files > - > > Key: MAPREDUCE-2407 > URL: https://issues.apache.org/jira/browse/MAPREDUCE-2407 > Project: Hadoop Map/Reduce > Issue Type: New Feature > Components: contrib/gridmix >Affects Versions: 0.23.0 >Reporter: Ravi Gummadi >Assignee: Ravi Gummadi > Fix For: 0.23.0 > > Attachments: 2407.patch, 2407.v1.patch > > > Currently Gridmix emulates disk IO load only. This JIRA is to make Gridmix > emulate Distributed Cache load as defined by the job-trace. -- This message is automatically generated by JIRA. For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (MAPREDUCE-2470) Receiving NPE occasionally on RunningJob.getCounters() call
[ https://issues.apache.org/jira/browse/MAPREDUCE-2470?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13037181#comment-13037181 ] Hudson commented on MAPREDUCE-2470: --- Integrated in Hadoop-Mapreduce-trunk-Commit #694 (See [https://builds.apache.org/hudson/job/Hadoop-Mapreduce-trunk-Commit/694/]) Revert MAPREDUCE-2470 cdouglas : http://svn.apache.org/viewcvs.cgi/?root=Apache-SVN&view=rev&rev=1125599 Files : * /hadoop/mapreduce/trunk/CHANGES.txt * /hadoop/mapreduce/trunk/src/test/mapred/org/apache/hadoop/mapred/TestNetworkedJob.java * /hadoop/mapreduce/trunk/src/java/org/apache/hadoop/mapreduce/protocol/ClientProtocol.java * /hadoop/mapreduce/trunk/src/java/org/apache/hadoop/mapred/RunningJob.java * /hadoop/mapreduce/trunk/src/java/org/apache/hadoop/mapred/JobClient.java > Receiving NPE occasionally on RunningJob.getCounters() call > --- > > Key: MAPREDUCE-2470 > URL: https://issues.apache.org/jira/browse/MAPREDUCE-2470 > Project: Hadoop Map/Reduce > Issue Type: Bug > Components: client >Affects Versions: 0.21.0 > Environment: FreeBSD, Java6, Hadoop r0.21.0 >Reporter: Aaron Baff >Assignee: Robert Joseph Evans > Fix For: 0.23.0 > > Attachments: MAPREDUCE-2470-v1.patch, MAPREDUCE-2470-v2.patch, > counters_null_data.pcap > > > This is running in a Java daemon that is used as an interface (Thrift) to get > information and data from MR Jobs. Using JobClient.getJob(JobID) I > successfully get a RunningJob object (I'm checking for NULL), and then rarely > I get an NPE when I do RunningJob.getCounters(). This seems to occur after > the daemon has been up and running for a while, and in the event of an > Exception, I close the JobClient, set it to NULL, and a new one should then > be created on the next request for data. Yet, I still seem to be unable to > fetch the Counters. Below is the stack trace. > java.lang.NullPointerException > at org.apache.hadoop.mapred.Counters.downgrade(Counters.java:77) > at > org.apache.hadoop.mapred.JobClient$NetworkedJob.getCounters(JobClient.java:381) > at > com.telescope.HadoopThrift.service.ServiceImpl.getReportResults(ServiceImpl.java:350) > at > com.telescope.HadoopThrift.gen.HadoopThrift$Processor$getReportResults.process(HadoopThrift.java:545) > at > com.telescope.HadoopThrift.gen.HadoopThrift$Processor.process(HadoopThrift.java:421) > at > org.apache.thrift.server.TNonblockingServer$FrameBuffer.invoke(TNonblockingServer.java:697) > at > org.apache.thrift.server.THsHaServer$Invocation.run(THsHaServer.java:317) > at > java.util.concurrent.ThreadPoolExecutor$Worker.runTask(ThreadPoolExecutor.java:886) > at > java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:908) > at java.lang.Thread.run(Thread.java:619) -- This message is automatically generated by JIRA. For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (MAPREDUCE-2514) ReinitTrackerAction class name misspelled RenitTrackerAction in task tracker log
[ https://issues.apache.org/jira/browse/MAPREDUCE-2514?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13037183#comment-13037183 ] Hudson commented on MAPREDUCE-2514: --- Integrated in Hadoop-Mapreduce-trunk-Commit #694 (See [https://builds.apache.org/hudson/job/Hadoop-Mapreduce-trunk-Commit/694/]) > ReinitTrackerAction class name misspelled RenitTrackerAction in task tracker > log > > > Key: MAPREDUCE-2514 > URL: https://issues.apache.org/jira/browse/MAPREDUCE-2514 > Project: Hadoop Map/Reduce > Issue Type: Bug > Components: tasktracker >Affects Versions: 0.20.205.0, 0.23.0 >Reporter: Jonathan Eagles >Assignee: Jonathan Eagles >Priority: Trivial > Fix For: 0.20.205.0, 0.23.0 > > Attachments: MAPREDUCE-2514-branch-0.20-security.patch, > MAPREDUCE-2514-trunk.patch > > -- This message is automatically generated by JIRA. For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (MAPREDUCE-2490) Log blacklist debug count
[ https://issues.apache.org/jira/browse/MAPREDUCE-2490?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13037182#comment-13037182 ] Hudson commented on MAPREDUCE-2490: --- Integrated in Hadoop-Mapreduce-trunk-Commit #694 (See [https://builds.apache.org/hudson/job/Hadoop-Mapreduce-trunk-Commit/694/]) > Log blacklist debug count > - > > Key: MAPREDUCE-2490 > URL: https://issues.apache.org/jira/browse/MAPREDUCE-2490 > Project: Hadoop Map/Reduce > Issue Type: Improvement > Components: jobtracker >Affects Versions: 0.20.204.0, 0.22.0 >Reporter: Jonathan Eagles >Assignee: Jonathan Eagles >Priority: Trivial > Fix For: 0.20.205.0, 0.23.0 > > Attachments: MAPREDUCE-2490-branch-0.20-security-v2.patch, > MAPREDUCE-2490-branch-0.20-security.patch, MAPREDUCE-2490-trunk-v2.patch, > MAPREDUCE-2490-trunk.patch > > > Gain some insight into blacklist increments/decrements by enhancing the debug > logging -- This message is automatically generated by JIRA. For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Updated] (MAPREDUCE-2527) MR-279: Metrics for MRAppMaster
[ https://issues.apache.org/jira/browse/MAPREDUCE-2527?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Luke Lu updated MAPREDUCE-2527: --- Attachment: mr-2527-am-metrics-v1.patch > MR-279: Metrics for MRAppMaster > --- > > Key: MAPREDUCE-2527 > URL: https://issues.apache.org/jira/browse/MAPREDUCE-2527 > Project: Hadoop Map/Reduce > Issue Type: New Feature > Components: mrv2 >Affects Versions: 0.23.0 >Reporter: Luke Lu >Assignee: Luke Lu > Fix For: 0.23.0 > > Attachments: mr-2527-am-metrics-v1.patch > > -- This message is automatically generated by JIRA. For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Created] (MAPREDUCE-2527) MR-279: Metrics for MRAppMaster
MR-279: Metrics for MRAppMaster --- Key: MAPREDUCE-2527 URL: https://issues.apache.org/jira/browse/MAPREDUCE-2527 Project: Hadoop Map/Reduce Issue Type: New Feature Components: mrv2 Affects Versions: 0.23.0 Reporter: Luke Lu Assignee: Luke Lu Fix For: 0.23.0 -- This message is automatically generated by JIRA. For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Reopened] (MAPREDUCE-2470) Receiving NPE occasionally on RunningJob.getCounters() call
[ https://issues.apache.org/jira/browse/MAPREDUCE-2470?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Chris Douglas reopened MAPREDUCE-2470: -- Reverted while FI breakage is investigated > Receiving NPE occasionally on RunningJob.getCounters() call > --- > > Key: MAPREDUCE-2470 > URL: https://issues.apache.org/jira/browse/MAPREDUCE-2470 > Project: Hadoop Map/Reduce > Issue Type: Bug > Components: client >Affects Versions: 0.21.0 > Environment: FreeBSD, Java6, Hadoop r0.21.0 >Reporter: Aaron Baff >Assignee: Robert Joseph Evans > Fix For: 0.23.0 > > Attachments: MAPREDUCE-2470-v1.patch, MAPREDUCE-2470-v2.patch, > counters_null_data.pcap > > > This is running in a Java daemon that is used as an interface (Thrift) to get > information and data from MR Jobs. Using JobClient.getJob(JobID) I > successfully get a RunningJob object (I'm checking for NULL), and then rarely > I get an NPE when I do RunningJob.getCounters(). This seems to occur after > the daemon has been up and running for a while, and in the event of an > Exception, I close the JobClient, set it to NULL, and a new one should then > be created on the next request for data. Yet, I still seem to be unable to > fetch the Counters. Below is the stack trace. > java.lang.NullPointerException > at org.apache.hadoop.mapred.Counters.downgrade(Counters.java:77) > at > org.apache.hadoop.mapred.JobClient$NetworkedJob.getCounters(JobClient.java:381) > at > com.telescope.HadoopThrift.service.ServiceImpl.getReportResults(ServiceImpl.java:350) > at > com.telescope.HadoopThrift.gen.HadoopThrift$Processor$getReportResults.process(HadoopThrift.java:545) > at > com.telescope.HadoopThrift.gen.HadoopThrift$Processor.process(HadoopThrift.java:421) > at > org.apache.thrift.server.TNonblockingServer$FrameBuffer.invoke(TNonblockingServer.java:697) > at > org.apache.thrift.server.THsHaServer$Invocation.run(THsHaServer.java:317) > at > java.util.concurrent.ThreadPoolExecutor$Worker.runTask(ThreadPoolExecutor.java:886) > at > java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:908) > at java.lang.Thread.run(Thread.java:619) -- This message is automatically generated by JIRA. For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (MAPREDUCE-2470) Receiving NPE occasionally on RunningJob.getCounters() call
[ https://issues.apache.org/jira/browse/MAPREDUCE-2470?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13037161#comment-13037161 ] Chris Douglas commented on MAPREDUCE-2470: -- *sigh* Yea, it looks like the fault injection is broken. I'll revert it. Not sure why Hudson didn't flag that... > Receiving NPE occasionally on RunningJob.getCounters() call > --- > > Key: MAPREDUCE-2470 > URL: https://issues.apache.org/jira/browse/MAPREDUCE-2470 > Project: Hadoop Map/Reduce > Issue Type: Bug > Components: client >Affects Versions: 0.21.0 > Environment: FreeBSD, Java6, Hadoop r0.21.0 >Reporter: Aaron Baff >Assignee: Robert Joseph Evans > Fix For: 0.23.0 > > Attachments: MAPREDUCE-2470-v1.patch, MAPREDUCE-2470-v2.patch, > counters_null_data.pcap > > > This is running in a Java daemon that is used as an interface (Thrift) to get > information and data from MR Jobs. Using JobClient.getJob(JobID) I > successfully get a RunningJob object (I'm checking for NULL), and then rarely > I get an NPE when I do RunningJob.getCounters(). This seems to occur after > the daemon has been up and running for a while, and in the event of an > Exception, I close the JobClient, set it to NULL, and a new one should then > be created on the next request for data. Yet, I still seem to be unable to > fetch the Counters. Below is the stack trace. > java.lang.NullPointerException > at org.apache.hadoop.mapred.Counters.downgrade(Counters.java:77) > at > org.apache.hadoop.mapred.JobClient$NetworkedJob.getCounters(JobClient.java:381) > at > com.telescope.HadoopThrift.service.ServiceImpl.getReportResults(ServiceImpl.java:350) > at > com.telescope.HadoopThrift.gen.HadoopThrift$Processor$getReportResults.process(HadoopThrift.java:545) > at > com.telescope.HadoopThrift.gen.HadoopThrift$Processor.process(HadoopThrift.java:421) > at > org.apache.thrift.server.TNonblockingServer$FrameBuffer.invoke(TNonblockingServer.java:697) > at > org.apache.thrift.server.THsHaServer$Invocation.run(THsHaServer.java:317) > at > java.util.concurrent.ThreadPoolExecutor$Worker.runTask(ThreadPoolExecutor.java:886) > at > java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:908) > at java.lang.Thread.run(Thread.java:619) -- This message is automatically generated by JIRA. For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (MAPREDUCE-2470) Receiving NPE occasionally on RunningJob.getCounters() call
[ https://issues.apache.org/jira/browse/MAPREDUCE-2470?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13037148#comment-13037148 ] Todd Lipcon commented on MAPREDUCE-2470: This seems to have broken the MR trunk build: https://builds.apache.org/hudson/job/Hadoop-Mapreduce-trunk-Commit/691/ > Receiving NPE occasionally on RunningJob.getCounters() call > --- > > Key: MAPREDUCE-2470 > URL: https://issues.apache.org/jira/browse/MAPREDUCE-2470 > Project: Hadoop Map/Reduce > Issue Type: Bug > Components: client >Affects Versions: 0.21.0 > Environment: FreeBSD, Java6, Hadoop r0.21.0 >Reporter: Aaron Baff >Assignee: Robert Joseph Evans > Fix For: 0.23.0 > > Attachments: MAPREDUCE-2470-v1.patch, MAPREDUCE-2470-v2.patch, > counters_null_data.pcap > > > This is running in a Java daemon that is used as an interface (Thrift) to get > information and data from MR Jobs. Using JobClient.getJob(JobID) I > successfully get a RunningJob object (I'm checking for NULL), and then rarely > I get an NPE when I do RunningJob.getCounters(). This seems to occur after > the daemon has been up and running for a while, and in the event of an > Exception, I close the JobClient, set it to NULL, and a new one should then > be created on the next request for data. Yet, I still seem to be unable to > fetch the Counters. Below is the stack trace. > java.lang.NullPointerException > at org.apache.hadoop.mapred.Counters.downgrade(Counters.java:77) > at > org.apache.hadoop.mapred.JobClient$NetworkedJob.getCounters(JobClient.java:381) > at > com.telescope.HadoopThrift.service.ServiceImpl.getReportResults(ServiceImpl.java:350) > at > com.telescope.HadoopThrift.gen.HadoopThrift$Processor$getReportResults.process(HadoopThrift.java:545) > at > com.telescope.HadoopThrift.gen.HadoopThrift$Processor.process(HadoopThrift.java:421) > at > org.apache.thrift.server.TNonblockingServer$FrameBuffer.invoke(TNonblockingServer.java:697) > at > org.apache.thrift.server.THsHaServer$Invocation.run(THsHaServer.java:317) > at > java.util.concurrent.ThreadPoolExecutor$Worker.runTask(ThreadPoolExecutor.java:886) > at > java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:908) > at java.lang.Thread.run(Thread.java:619) -- This message is automatically generated by JIRA. For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Updated] (MAPREDUCE-2490) Log blacklist debug count
[ https://issues.apache.org/jira/browse/MAPREDUCE-2490?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Chris Douglas updated MAPREDUCE-2490: - Resolution: Fixed Fix Version/s: 0.23.0 0.20.205.0 Hadoop Flags: [Reviewed] Status: Resolved (was: Patch Available) +1 I committed this. Thanks, Jon! > Log blacklist debug count > - > > Key: MAPREDUCE-2490 > URL: https://issues.apache.org/jira/browse/MAPREDUCE-2490 > Project: Hadoop Map/Reduce > Issue Type: Improvement > Components: jobtracker >Affects Versions: 0.20.204.0, 0.22.0 >Reporter: Jonathan Eagles >Assignee: Jonathan Eagles >Priority: Trivial > Fix For: 0.20.205.0, 0.23.0 > > Attachments: MAPREDUCE-2490-branch-0.20-security-v2.patch, > MAPREDUCE-2490-branch-0.20-security.patch, MAPREDUCE-2490-trunk-v2.patch, > MAPREDUCE-2490-trunk.patch > > > Gain some insight into blacklist increments/decrements by enhancing the debug > logging -- This message is automatically generated by JIRA. For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Updated] (MAPREDUCE-2514) ReinitTrackerAction class name misspelled RenitTrackerAction in task tracker log
[ https://issues.apache.org/jira/browse/MAPREDUCE-2514?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Chris Douglas updated MAPREDUCE-2514: - Resolution: Fixed Fix Version/s: 0.23.0 0.20.205.0 Hadoop Flags: [Reviewed] Status: Resolved (was: Patch Available) +1 I committed this. Thanks, Jon! > ReinitTrackerAction class name misspelled RenitTrackerAction in task tracker > log > > > Key: MAPREDUCE-2514 > URL: https://issues.apache.org/jira/browse/MAPREDUCE-2514 > Project: Hadoop Map/Reduce > Issue Type: Bug > Components: tasktracker >Affects Versions: 0.20.205.0, 0.23.0 >Reporter: Jonathan Eagles >Assignee: Jonathan Eagles >Priority: Trivial > Fix For: 0.20.205.0, 0.23.0 > > Attachments: MAPREDUCE-2514-branch-0.20-security.patch, > MAPREDUCE-2514-trunk.patch > > -- This message is automatically generated by JIRA. For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Updated] (MAPREDUCE-2470) Receiving NPE occasionally on RunningJob.getCounters() call
[ https://issues.apache.org/jira/browse/MAPREDUCE-2470?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Chris Douglas updated MAPREDUCE-2470: - Resolution: Fixed Fix Version/s: 0.23.0 Hadoop Flags: [Reviewed] Status: Resolved (was: Patch Available) bq. The difference is that with Job.getStatus it calls internally updateStatus which will throw an IOException if it is unable to get an updated status from the JobTracker. The only time you can get an NPE from that is if you catch the IOException, ignore it, and keep trying to use the Job object what was originally created. Got it. Thanks for the explanation. +1 I committed this. Thanks! > Receiving NPE occasionally on RunningJob.getCounters() call > --- > > Key: MAPREDUCE-2470 > URL: https://issues.apache.org/jira/browse/MAPREDUCE-2470 > Project: Hadoop Map/Reduce > Issue Type: Bug > Components: client >Affects Versions: 0.21.0 > Environment: FreeBSD, Java6, Hadoop r0.21.0 >Reporter: Aaron Baff >Assignee: Robert Joseph Evans > Fix For: 0.23.0 > > Attachments: MAPREDUCE-2470-v1.patch, MAPREDUCE-2470-v2.patch, > counters_null_data.pcap > > > This is running in a Java daemon that is used as an interface (Thrift) to get > information and data from MR Jobs. Using JobClient.getJob(JobID) I > successfully get a RunningJob object (I'm checking for NULL), and then rarely > I get an NPE when I do RunningJob.getCounters(). This seems to occur after > the daemon has been up and running for a while, and in the event of an > Exception, I close the JobClient, set it to NULL, and a new one should then > be created on the next request for data. Yet, I still seem to be unable to > fetch the Counters. Below is the stack trace. > java.lang.NullPointerException > at org.apache.hadoop.mapred.Counters.downgrade(Counters.java:77) > at > org.apache.hadoop.mapred.JobClient$NetworkedJob.getCounters(JobClient.java:381) > at > com.telescope.HadoopThrift.service.ServiceImpl.getReportResults(ServiceImpl.java:350) > at > com.telescope.HadoopThrift.gen.HadoopThrift$Processor$getReportResults.process(HadoopThrift.java:545) > at > com.telescope.HadoopThrift.gen.HadoopThrift$Processor.process(HadoopThrift.java:421) > at > org.apache.thrift.server.TNonblockingServer$FrameBuffer.invoke(TNonblockingServer.java:697) > at > org.apache.thrift.server.THsHaServer$Invocation.run(THsHaServer.java:317) > at > java.util.concurrent.ThreadPoolExecutor$Worker.runTask(ThreadPoolExecutor.java:886) > at > java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:908) > at java.lang.Thread.run(Thread.java:619) -- This message is automatically generated by JIRA. For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (MAPREDUCE-2495) The distributed cache cleanup thread has no monitoring to check to see if it has died for some reason
[ https://issues.apache.org/jira/browse/MAPREDUCE-2495?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13037080#comment-13037080 ] Robert Joseph Evans commented on MAPREDUCE-2495: Just like before. The contrib test issues are with RAID, and appear to be a known issue. > The distributed cache cleanup thread has no monitoring to check to see if it > has died for some reason > - > > Key: MAPREDUCE-2495 > URL: https://issues.apache.org/jira/browse/MAPREDUCE-2495 > Project: Hadoop Map/Reduce > Issue Type: Improvement > Components: distributed-cache >Affects Versions: 0.21.0 >Reporter: Robert Joseph Evans >Assignee: Robert Joseph Evans >Priority: Minor > Attachments: MAPREDUCE-2495-20.20X-V1.patch, > MAPREDUCE-2495-20.20X-V2.patch, MAPREDUCE-2495-20.20X-V3.patch, > MAPREDUCE-2495-v1.patch, MAPREDUCE-2495-v2.patch, MAPREDUCE-2495-v3.patch > > > The cleanup thread in the distributed cache handles IOExceptions and the like > correctly, but just to be a bit more defensive it would be good to monitor > the thread, and check that it is still alive regularly, so that the > distributed cache does not fill up the entire disk on the node. -- This message is automatically generated by JIRA. For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (MAPREDUCE-2490) Log blacklist debug count
[ https://issues.apache.org/jira/browse/MAPREDUCE-2490?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13037056#comment-13037056 ] Jonathan Eagles commented on MAPREDUCE-2490: Filed MAPREDUCE-2526 "Add task tracker graylisting" to aid in the discussion to forward port task tracker graylisting to trunk to address comments above. > Log blacklist debug count > - > > Key: MAPREDUCE-2490 > URL: https://issues.apache.org/jira/browse/MAPREDUCE-2490 > Project: Hadoop Map/Reduce > Issue Type: Improvement > Components: jobtracker >Affects Versions: 0.20.204.0, 0.22.0 >Reporter: Jonathan Eagles >Assignee: Jonathan Eagles >Priority: Trivial > Attachments: MAPREDUCE-2490-branch-0.20-security-v2.patch, > MAPREDUCE-2490-branch-0.20-security.patch, MAPREDUCE-2490-trunk-v2.patch, > MAPREDUCE-2490-trunk.patch > > > Gain some insight into blacklist increments/decrements by enhancing the debug > logging -- This message is automatically generated by JIRA. For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Created] (MAPREDUCE-2526) Forward port MAPREDUCE-1966 Add task tracker graylisting
Forward port MAPREDUCE-1966 Add task tracker graylisting Key: MAPREDUCE-2526 URL: https://issues.apache.org/jira/browse/MAPREDUCE-2526 Project: Hadoop Map/Reduce Issue Type: Bug Components: jobtracker Reporter: Jonathan Eagles Assignee: Greg Roelofs The current heuristic of rolling up fixed number of job failures per tracker isn't working well, we need better design/heuristics. -- This message is automatically generated by JIRA. For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Updated] (MAPREDUCE-2526) Forward port MAPREDUCE-1966 Add task tracker graylisting
[ https://issues.apache.org/jira/browse/MAPREDUCE-2526?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Jonathan Eagles updated MAPREDUCE-2526: --- Affects Version/s: 0.23.0 Assignee: (was: Greg Roelofs) Issue Type: New Feature (was: Bug) > Forward port MAPREDUCE-1966 Add task tracker graylisting > > > Key: MAPREDUCE-2526 > URL: https://issues.apache.org/jira/browse/MAPREDUCE-2526 > Project: Hadoop Map/Reduce > Issue Type: New Feature > Components: jobtracker >Affects Versions: 0.23.0 >Reporter: Jonathan Eagles > > The current heuristic of rolling up fixed number of job failures per tracker > isn't working well, we need better design/heuristics. -- This message is automatically generated by JIRA. For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (MAPREDUCE-2490) Log blacklist debug count
[ https://issues.apache.org/jira/browse/MAPREDUCE-2490?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13037042#comment-13037042 ] Jonathan Eagles commented on MAPREDUCE-2490: This most recent patch version still logs the black/gray counts, but has also added some debug info at the caller to better aid in debug the root cause in the case for debugging black/gray list failures. The gridmix contrib test failure is unrelated to my change. No new unit tests were add due to the nature of this change (only affects logging). > Log blacklist debug count > - > > Key: MAPREDUCE-2490 > URL: https://issues.apache.org/jira/browse/MAPREDUCE-2490 > Project: Hadoop Map/Reduce > Issue Type: Improvement > Components: jobtracker >Affects Versions: 0.20.204.0, 0.22.0 >Reporter: Jonathan Eagles >Assignee: Jonathan Eagles >Priority: Trivial > Attachments: MAPREDUCE-2490-branch-0.20-security-v2.patch, > MAPREDUCE-2490-branch-0.20-security.patch, MAPREDUCE-2490-trunk-v2.patch, > MAPREDUCE-2490-trunk.patch > > > Gain some insight into blacklist increments/decrements by enhancing the debug > logging -- This message is automatically generated by JIRA. For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (MAPREDUCE-2490) Log blacklist debug count
[ https://issues.apache.org/jira/browse/MAPREDUCE-2490?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13037027#comment-13037027 ] Hadoop QA commented on MAPREDUCE-2490: -- -1 overall. Here are the results of testing the latest attachment http://issues.apache.org/jira/secure/attachment/12479926/MAPREDUCE-2490-trunk-v2.patch against trunk revision 1125428. +1 @author. The patch does not contain any @author tags. -1 tests included. The patch doesn't appear to include any new or modified tests. Please justify why no new tests are needed for this patch. Also please list what manual steps were performed to verify this patch. +1 javadoc. The javadoc tool did not generate any warning messages. +1 javac. The applied patch does not increase the total number of javac compiler warnings. +1 findbugs. The patch does not introduce any new Findbugs (version 1.3.9) warnings. +1 release audit. The applied patch does not increase the total number of release audit warnings. +1 core tests. The patch passed core unit tests. -1 contrib tests. The patch failed contrib unit tests. +1 system test framework. The patch passed system test framework compile. Test results: https://builds.apache.org/hudson/job/PreCommit-MAPREDUCE-Build/287//testReport/ Findbugs warnings: https://builds.apache.org/hudson/job/PreCommit-MAPREDUCE-Build/287//artifact/trunk/build/test/findbugs/newPatchFindbugsWarnings.html Console output: https://builds.apache.org/hudson/job/PreCommit-MAPREDUCE-Build/287//console This message is automatically generated. > Log blacklist debug count > - > > Key: MAPREDUCE-2490 > URL: https://issues.apache.org/jira/browse/MAPREDUCE-2490 > Project: Hadoop Map/Reduce > Issue Type: Improvement > Components: jobtracker >Affects Versions: 0.20.204.0, 0.22.0 >Reporter: Jonathan Eagles >Assignee: Jonathan Eagles >Priority: Trivial > Attachments: MAPREDUCE-2490-branch-0.20-security-v2.patch, > MAPREDUCE-2490-branch-0.20-security.patch, MAPREDUCE-2490-trunk-v2.patch, > MAPREDUCE-2490-trunk.patch > > > Gain some insight into blacklist increments/decrements by enhancing the debug > logging -- This message is automatically generated by JIRA. For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (MAPREDUCE-2297) All map reduce tasks are failing if we give invalid path jar file for Job
[
https://issues.apache.org/jira/browse/MAPREDUCE-2297?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13037013#comment-13037013
]
Todd Lipcon commented on MAPREDUCE-2297:
bq. Consider the case of Hive, if we configure any invalid path for the
property "hive.aux.jars.path" all the jobs will fail which is not using that
jar also.
I would consider it a broken config if you configure hive.aux.jars.path to
point to a jar which doesn't exist.
In your patch, if you accidentally make a typo in your DistributedCache
entries, you'll see NoClassDefFound exceptions or other much scarier errors. I
think it's better to fail with the "File does not exist" error during
localization.
> All map reduce tasks are failing if we give invalid path jar file for Job
> -
>
> Key: MAPREDUCE-2297
> URL: https://issues.apache.org/jira/browse/MAPREDUCE-2297
> Project: Hadoop Map/Reduce
> Issue Type: Bug
> Components: tasktracker
>Affects Versions: 0.20.2
>Reporter: Devaraj K
>Assignee: Devaraj K
>Priority: Minor
> Fix For: 0.20.4
>
> Attachments: MAPREDUCE-2297.patch
>
>
> This can be reproduced by giving the invalid jar file for the Job or it can
> be reproduced from hive.
> In hive-default.xml
>
> hive.aux.jars.path
>
> Provided for adding auxillaryjarsPath
>
> If we configure an invalid path for jar file, It is making all map reduce
> tasks to fail even those jobs are not depending on this jar file and it is
> giving the below exception.
> {code:xml}
> hive> select * from a join b on(a.b=b.c);
> Total MapReduce jobs = 1
> Launching Job 1 out of 1
> Number of reduce tasks not specified. Estimated from input data size: 1
> In order to change the average load for a reducer (in bytes):
> set hive.exec.reducers.bytes.per.reducer=
> In order to limit the maximum number of reducers:
> set hive.exec.reducers.max=
> In order to set a constant number of reducers:
> set mapred.reduce.tasks=
> java.io.FileNotFoundException: File does not exist: /user/root/grade.jar
> at
> org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:495)
> at
> org.apache.hadoop.filecache.DistributedCache.getTimestamp(DistributedCache.java:509)
> at
> org.apache.hadoop.mapred.JobClient.configureCommandLineOptions(JobClient.java:651)
> at org.apache.hadoop.mapred.JobClient.submitJobInternal(JobClient.java:783)
> at org.apache.hadoop.mapred.JobClient.submitJob(JobClient.java:752)
> at org.apache.hadoop.hive.ql.exec.ExecDriver.execute(ExecDriver.java:698)
> at org.apache.hadoop.hive.ql.exec.Task.executeTask(Task.java:107)
> at org.apache.hadoop.hive.ql.exec.TaskRunner.runSequential(TaskRunner.java:64)
> {code}
--
This message is automatically generated by JIRA.
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (MAPREDUCE-2264) Job status exceeds 100% in some cases
[ https://issues.apache.org/jira/browse/MAPREDUCE-2264?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13036974#comment-13036974 ] Hadoop QA commented on MAPREDUCE-2264: -- -1 overall. Here are the results of testing the latest attachment http://issues.apache.org/jira/secure/attachment/12479914/MAPREDUCE-2264-trunk.patch against trunk revision 1125428. +1 @author. The patch does not contain any @author tags. -1 tests included. The patch doesn't appear to include any new or modified tests. Please justify why no new tests are needed for this patch. Also please list what manual steps were performed to verify this patch. +1 javadoc. The javadoc tool did not generate any warning messages. +1 javac. The applied patch does not increase the total number of javac compiler warnings. +1 findbugs. The patch does not introduce any new Findbugs (version 1.3.9) warnings. +1 release audit. The applied patch does not increase the total number of release audit warnings. -1 core tests. The patch failed these core unit tests: org.apache.hadoop.mapred.TestReduceFetch org.apache.hadoop.mapreduce.TestMapReduceLocal +1 contrib tests. The patch passed contrib unit tests. +1 system test framework. The patch passed system test framework compile. Test results: https://builds.apache.org/hudson/job/PreCommit-MAPREDUCE-Build/286//testReport/ Findbugs warnings: https://builds.apache.org/hudson/job/PreCommit-MAPREDUCE-Build/286//artifact/trunk/build/test/findbugs/newPatchFindbugsWarnings.html Console output: https://builds.apache.org/hudson/job/PreCommit-MAPREDUCE-Build/286//console This message is automatically generated. > Job status exceeds 100% in some cases > -- > > Key: MAPREDUCE-2264 > URL: https://issues.apache.org/jira/browse/MAPREDUCE-2264 > Project: Hadoop Map/Reduce > Issue Type: Bug > Components: jobtracker >Reporter: Adam Kramer >Assignee: Devaraj K > Fix For: 0.20.4, 0.23.0 > > Attachments: MAPREDUCE-2264-0.20.3.patch, MAPREDUCE-2264-trunk.patch, > more than 100%.bmp > > > I'm looking now at my jobtracker's list of running reduce tasks. One of them > is 120.05% complete, the other is 107.28% complete. > I understand that these numbers are estimates, but there is no case in which > an estimate of 100% for a non-complete task is better than an estimate of > 99.99%, nor is there any case in which an estimate greater than 100% is valid. > I suggest that whatever logic is computing these set 99.99% as a hard maximum. -- This message is automatically generated by JIRA. For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (MAPREDUCE-2463) Job History files are not moving to done folder when job history location is hdfs location
[
https://issues.apache.org/jira/browse/MAPREDUCE-2463?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13036961#comment-13036961
]
Hadoop QA commented on MAPREDUCE-2463:
--
+1 overall. Here are the results of testing the latest attachment
http://issues.apache.org/jira/secure/attachment/12479912/MAPREDUCE-2463-2.patch
against trunk revision 1125428.
+1 @author. The patch does not contain any @author tags.
+1 tests included. The patch appears to include 3 new or modified tests.
+1 javadoc. The javadoc tool did not generate any warning messages.
+1 javac. The applied patch does not increase the total number of javac
compiler warnings.
+1 findbugs. The patch does not introduce any new Findbugs (version 1.3.9)
warnings.
+1 release audit. The applied patch does not increase the total number of
release audit warnings.
+1 core tests. The patch passed core unit tests.
+1 contrib tests. The patch passed contrib unit tests.
+1 system test framework. The patch passed system test framework compile.
Test results:
https://builds.apache.org/hudson/job/PreCommit-MAPREDUCE-Build/285//testReport/
Findbugs warnings:
https://builds.apache.org/hudson/job/PreCommit-MAPREDUCE-Build/285//artifact/trunk/build/test/findbugs/newPatchFindbugsWarnings.html
Console output:
https://builds.apache.org/hudson/job/PreCommit-MAPREDUCE-Build/285//console
This message is automatically generated.
> Job History files are not moving to done folder when job history location is
> hdfs location
> --
>
> Key: MAPREDUCE-2463
> URL: https://issues.apache.org/jira/browse/MAPREDUCE-2463
> Project: Hadoop Map/Reduce
> Issue Type: Bug
> Components: jobtracker
>Affects Versions: 0.23.0
>Reporter: Devaraj K
>Assignee: Devaraj K
> Fix For: 0.23.0
>
> Attachments: MAPREDUCE-2463-1.patch, MAPREDUCE-2463-2.patch,
> MAPREDUCE-2463.patch
>
>
> If "mapreduce.jobtracker.jobhistory.location" is configured as HDFS location
> then either during initialization of Job Tracker (while moving old job
> history files) or after completion of the job, history files are not moving
> to done and giving following exception.
> {code:xml}
> 2011-04-29 15:27:27,813 ERROR
> org.apache.hadoop.mapreduce.jobhistory.JobHistory: Unable to move history
> file to DONE folder.
> java.lang.IllegalArgumentException: Wrong FS:
> hdfs://10.18.52.146:9000/history/job_201104291518_0001_root, expected:
> file:///
> at org.apache.hadoop.fs.FileSystem.checkPath(FileSystem.java:402)
> at
> org.apache.hadoop.fs.RawLocalFileSystem.pathToFile(RawLocalFileSystem.java:58)
> at
> org.apache.hadoop.fs.RawLocalFileSystem.getFileStatus(RawLocalFileSystem.java:419)
> at
> org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:294)
> at org.apache.hadoop.fs.FileUtil.copy(FileUtil.java:215)
> at
> org.apache.hadoop.fs.FileSystem.copyFromLocalFile(FileSystem.java:1516)
> at
> org.apache.hadoop.fs.FileSystem.copyFromLocalFile(FileSystem.java:1492)
> at
> org.apache.hadoop.fs.FileSystem.moveFromLocalFile(FileSystem.java:1482)
> at
> org.apache.hadoop.mapreduce.jobhistory.JobHistory.moveToDoneNow(JobHistory.java:348)
> at
> org.apache.hadoop.mapreduce.jobhistory.JobHistory.access$200(JobHistory.java:61)
> at
> org.apache.hadoop.mapreduce.jobhistory.JobHistory$1.run(JobHistory.java:439)
> at
> java.util.concurrent.ThreadPoolExecutor$Worker.runTask(ThreadPoolExecutor.java:886)
> at
> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:908)
> at java.lang.Thread.run(Thread.java:619)
> {code}
--
This message is automatically generated by JIRA.
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Updated] (MAPREDUCE-2490) Log blacklist debug count
[ https://issues.apache.org/jira/browse/MAPREDUCE-2490?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Jonathan Eagles updated MAPREDUCE-2490: --- Status: Patch Available (was: Open) > Log blacklist debug count > - > > Key: MAPREDUCE-2490 > URL: https://issues.apache.org/jira/browse/MAPREDUCE-2490 > Project: Hadoop Map/Reduce > Issue Type: Improvement > Components: jobtracker >Affects Versions: 0.20.204.0, 0.22.0 >Reporter: Jonathan Eagles >Assignee: Jonathan Eagles >Priority: Trivial > Attachments: MAPREDUCE-2490-branch-0.20-security-v2.patch, > MAPREDUCE-2490-branch-0.20-security.patch, MAPREDUCE-2490-trunk-v2.patch, > MAPREDUCE-2490-trunk.patch > > > Gain some insight into blacklist increments/decrements by enhancing the debug > logging -- This message is automatically generated by JIRA. For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Updated] (MAPREDUCE-2490) Log blacklist debug count
[ https://issues.apache.org/jira/browse/MAPREDUCE-2490?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Jonathan Eagles updated MAPREDUCE-2490: --- Status: Open (was: Patch Available) > Log blacklist debug count > - > > Key: MAPREDUCE-2490 > URL: https://issues.apache.org/jira/browse/MAPREDUCE-2490 > Project: Hadoop Map/Reduce > Issue Type: Improvement > Components: jobtracker >Affects Versions: 0.20.204.0, 0.22.0 >Reporter: Jonathan Eagles >Assignee: Jonathan Eagles >Priority: Trivial > Attachments: MAPREDUCE-2490-branch-0.20-security-v2.patch, > MAPREDUCE-2490-branch-0.20-security.patch, MAPREDUCE-2490-trunk-v2.patch, > MAPREDUCE-2490-trunk.patch > > > Gain some insight into blacklist increments/decrements by enhancing the debug > logging -- This message is automatically generated by JIRA. For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Updated] (MAPREDUCE-2490) Log blacklist debug count
[ https://issues.apache.org/jira/browse/MAPREDUCE-2490?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Jonathan Eagles updated MAPREDUCE-2490: --- Attachment: MAPREDUCE-2490-branch-0.20-security-v2.patch > Log blacklist debug count > - > > Key: MAPREDUCE-2490 > URL: https://issues.apache.org/jira/browse/MAPREDUCE-2490 > Project: Hadoop Map/Reduce > Issue Type: Improvement > Components: jobtracker >Affects Versions: 0.20.204.0, 0.22.0 >Reporter: Jonathan Eagles >Assignee: Jonathan Eagles >Priority: Trivial > Attachments: MAPREDUCE-2490-branch-0.20-security-v2.patch, > MAPREDUCE-2490-branch-0.20-security.patch, MAPREDUCE-2490-trunk.patch > > > Gain some insight into blacklist increments/decrements by enhancing the debug > logging -- This message is automatically generated by JIRA. For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Updated] (MAPREDUCE-2490) Log blacklist debug count
[ https://issues.apache.org/jira/browse/MAPREDUCE-2490?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Jonathan Eagles updated MAPREDUCE-2490: --- Attachment: MAPREDUCE-2490-trunk-v2.patch > Log blacklist debug count > - > > Key: MAPREDUCE-2490 > URL: https://issues.apache.org/jira/browse/MAPREDUCE-2490 > Project: Hadoop Map/Reduce > Issue Type: Improvement > Components: jobtracker >Affects Versions: 0.20.204.0, 0.22.0 >Reporter: Jonathan Eagles >Assignee: Jonathan Eagles >Priority: Trivial > Attachments: MAPREDUCE-2490-branch-0.20-security-v2.patch, > MAPREDUCE-2490-branch-0.20-security.patch, MAPREDUCE-2490-trunk-v2.patch, > MAPREDUCE-2490-trunk.patch > > > Gain some insight into blacklist increments/decrements by enhancing the debug > logging -- This message is automatically generated by JIRA. For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Updated] (MAPREDUCE-2490) Log blacklist debug count
[ https://issues.apache.org/jira/browse/MAPREDUCE-2490?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Jonathan Eagles updated MAPREDUCE-2490: --- Attachment: (was: MAPREDUCE-2490-branch-0.20-security-v2.patch) > Log blacklist debug count > - > > Key: MAPREDUCE-2490 > URL: https://issues.apache.org/jira/browse/MAPREDUCE-2490 > Project: Hadoop Map/Reduce > Issue Type: Improvement > Components: jobtracker >Affects Versions: 0.20.204.0, 0.22.0 >Reporter: Jonathan Eagles >Assignee: Jonathan Eagles >Priority: Trivial > Attachments: MAPREDUCE-2490-branch-0.20-security-v2.patch, > MAPREDUCE-2490-branch-0.20-security.patch, MAPREDUCE-2490-trunk.patch > > > Gain some insight into blacklist increments/decrements by enhancing the debug > logging -- This message is automatically generated by JIRA. For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Updated] (MAPREDUCE-2490) Log blacklist debug count
[ https://issues.apache.org/jira/browse/MAPREDUCE-2490?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Jonathan Eagles updated MAPREDUCE-2490: --- Attachment: MAPREDUCE-2490-branch-0.20-security-v2.patch > Log blacklist debug count > - > > Key: MAPREDUCE-2490 > URL: https://issues.apache.org/jira/browse/MAPREDUCE-2490 > Project: Hadoop Map/Reduce > Issue Type: Improvement > Components: jobtracker >Affects Versions: 0.20.204.0, 0.22.0 >Reporter: Jonathan Eagles >Assignee: Jonathan Eagles >Priority: Trivial > Attachments: MAPREDUCE-2490-branch-0.20-security-v2.patch, > MAPREDUCE-2490-branch-0.20-security.patch, MAPREDUCE-2490-trunk.patch > > > Gain some insight into blacklist increments/decrements by enhancing the debug > logging -- This message is automatically generated by JIRA. For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (MAPREDUCE-2518) missing t flag in distcp help message '-p[rbugp]'
[ https://issues.apache.org/jira/browse/MAPREDUCE-2518?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13036893#comment-13036893 ] Hudson commented on MAPREDUCE-2518: --- Integrated in Hadoop-Mapreduce-trunk #685 (See [https://builds.apache.org/hudson/job/Hadoop-Mapreduce-trunk/685/]) MAPREDUCE-2518. The t flag is missing in distcp help message. Contributed by Wei Yongjun szetszwo : http://svn.apache.org/viewcvs.cgi/?root=Apache-SVN&view=rev&rev=1125148 Files : * /hadoop/mapreduce/trunk/CHANGES.txt * /hadoop/mapreduce/trunk/src/tools/org/apache/hadoop/tools/DistCp.java > missing t flag in distcp help message '-p[rbugp]' > - > > Key: MAPREDUCE-2518 > URL: https://issues.apache.org/jira/browse/MAPREDUCE-2518 > Project: Hadoop Map/Reduce > Issue Type: Bug > Components: distcp >Affects Versions: 0.23.0 >Reporter: Wei Yongjun >Assignee: Wei Yongjun > Fix For: 0.23.0 > > Attachments: MAPREDUCE-2518.patch > > > 't: modification and access times' flag is defined but > missing in distcp help message '-p[rbugp]'. should be > changed to -p[rbugpt]. -- This message is automatically generated by JIRA. For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (MAPREDUCE-2516) option to control sensitive web actions
[ https://issues.apache.org/jira/browse/MAPREDUCE-2516?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13036894#comment-13036894 ] Hudson commented on MAPREDUCE-2516: --- Integrated in Hadoop-Mapreduce-trunk #685 (See [https://builds.apache.org/hudson/job/Hadoop-Mapreduce-trunk/685/]) MAPREDUCE-2516. Rename webinterface.private.actions to mapreduce.jobtracker.webinterface.trusted. Contributed by Ari Rabkin. todd : http://svn.apache.org/viewcvs.cgi/?root=Apache-SVN&view=rev&rev=1125223 Files : * /hadoop/mapreduce/trunk/src/java/mapred-default.xml * /hadoop/mapreduce/trunk/src/test/mapred/org/apache/hadoop/mapred/TestWebUIAuthorization.java * /hadoop/mapreduce/trunk/CHANGES.txt * /hadoop/mapreduce/trunk/src/java/org/apache/hadoop/mapreduce/server/jobtracker/JTConfig.java * /hadoop/mapreduce/trunk/src/java/org/apache/hadoop/mapred/JSPUtil.java * /hadoop/mapreduce/trunk/src/java/org/apache/hadoop/mapreduce/util/ConfigUtil.java > option to control sensitive web actions > --- > > Key: MAPREDUCE-2516 > URL: https://issues.apache.org/jira/browse/MAPREDUCE-2516 > Project: Hadoop Map/Reduce > Issue Type: Bug >Affects Versions: 0.22.0 >Reporter: Ari Rabkin >Assignee: Ari Rabkin >Priority: Minor > Fix For: 0.22.0 > > Attachments: MAPREDUCE-2516.txt > > > as per HADOOP-7302, webinterface.private.actions should not be in trunk. But > it should be here, and should have a clearer name. -- This message is automatically generated by JIRA. For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (MAPREDUCE-2372) TaskLogAppender mechanism shouldn't be set in log4j.properties
[ https://issues.apache.org/jira/browse/MAPREDUCE-2372?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13036891#comment-13036891 ] Hudson commented on MAPREDUCE-2372: --- Integrated in Hadoop-Mapreduce-trunk #685 (See [https://builds.apache.org/hudson/job/Hadoop-Mapreduce-trunk/685/]) MAPREDUCE-2372. TaskLogAppender mechanism shouldn't be set up in log4j.properties. Contributed by Todd Lipcon. todd : http://svn.apache.org/viewcvs.cgi/?root=Apache-SVN&view=rev&rev=1125082 Files : * /hadoop/mapreduce/trunk/src/java/org/apache/hadoop/mapred/TaskLogAppender.java * /hadoop/mapreduce/trunk/CHANGES.txt * /hadoop/mapreduce/trunk/src/java/org/apache/hadoop/mapred/TaskRunner.java > TaskLogAppender mechanism shouldn't be set in log4j.properties > -- > > Key: MAPREDUCE-2372 > URL: https://issues.apache.org/jira/browse/MAPREDUCE-2372 > Project: Hadoop Map/Reduce > Issue Type: Improvement > Components: task >Affects Versions: 0.22.0 >Reporter: Todd Lipcon >Assignee: Todd Lipcon > Fix For: 0.22.0 > > Attachments: mapreduce-2372.txt, mapreduce-2372.txt, > mapreduce-2372.txt > > > The TaskLogAppender log4j appender relies on using log4j.properties to pass > in some Java system properties into properties of the logger. This is > problematic since we've often found that users have customized > log4j.properties and don't upgrade it when they upgrade the version of Hadoop. > Since this is really an internal mechanism of how the task runner passes task > info to the TLA, we shouldn't rely on these settings in log4j.properties at > all. Rather, we should just get the system properties directly from > System.getProperty. -- This message is automatically generated by JIRA. For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (MAPREDUCE-2483) Clean up duplication of dependent jar files
[ https://issues.apache.org/jira/browse/MAPREDUCE-2483?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13036892#comment-13036892 ] Hudson commented on MAPREDUCE-2483: --- Integrated in Hadoop-Mapreduce-trunk #685 (See [https://builds.apache.org/hudson/job/Hadoop-Mapreduce-trunk/685/]) MAPREDUCE-2483. Remove duplication of jars between Hadoop subprojects from build artifacts. (Eric Yang via omalley) omalley : http://svn.apache.org/viewcvs.cgi/?root=Apache-SVN&view=rev&rev=1125017 Files : * /hadoop/mapreduce/trunk/CHANGES.txt * /hadoop/mapreduce/trunk/ivy.xml * /hadoop/mapreduce/trunk/src/contrib/mumak/build.xml * /hadoop/mapreduce/trunk/build.xml > Clean up duplication of dependent jar files > --- > > Key: MAPREDUCE-2483 > URL: https://issues.apache.org/jira/browse/MAPREDUCE-2483 > Project: Hadoop Map/Reduce > Issue Type: Bug > Components: build >Affects Versions: 0.23.0 >Reporter: Eric Yang >Assignee: Eric Yang > Fix For: 0.23.0 > > Attachments: MAPREDUCE-2483.patch > > > For trunk, the build and deployment tree look like this: > hadoop-common-0.2x.y > hadoop-hdfs-0.2x.y > hadoop-mapred-0.2x.y > Technically, mapred's the third party dependent jar files should be fetch > from hadoop-common and hadoop-hdfs. However, it is currently fetching from > hadoop-mapred/lib only. It would be nice to eliminate the need to repeat > duplicated jar files at build time. > There are two options to manage this dependency list, continue to enhance ant > build structure to fetch and filter jar file dependencies using ivy. On the > other hand, it would be a good opportunity to convert the build structure to > maven, and use maven to manage the provided jar files. -- This message is automatically generated by JIRA. For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (MAPREDUCE-2459) Cache HAR filesystem metadata
[ https://issues.apache.org/jira/browse/MAPREDUCE-2459?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13036890#comment-13036890 ] Hudson commented on MAPREDUCE-2459: --- Integrated in Hadoop-Mapreduce-trunk-Commit #690 (See [https://builds.apache.org/hudson/job/Hadoop-Mapreduce-trunk-Commit/690/]) MAPREDUCE-2459. Cache HAR filesystem metadata. (Mac Yang via mahadev) mahadev : http://svn.apache.org/viewcvs.cgi/?root=Apache-SVN&view=rev&rev=1125428 Files : * /hadoop/mapreduce/trunk/CHANGES.txt * /hadoop/mapreduce/trunk/src/tools/org/apache/hadoop/fs/HarFileSystem.java > Cache HAR filesystem metadata > - > > Key: MAPREDUCE-2459 > URL: https://issues.apache.org/jira/browse/MAPREDUCE-2459 > Project: Hadoop Map/Reduce > Issue Type: Improvement > Components: harchive >Reporter: Mac Yang >Assignee: Mac Yang > Fix For: 0.23.0 > > Attachments: MAPREDUCE-2459.1.patch, MAPREDUCE-2459.2.patch > > > Each HAR file system has two index files that contains information on how > files are stored in the part files. During the block location calculation, > these indexes are reread for every file in the archive. Caching the indexes > and the status of the part files will greatly reduce the number of name node > operations during the job setup time. -- This message is automatically generated by JIRA. For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Updated] (MAPREDUCE-2264) Job status exceeds 100% in some cases
[ https://issues.apache.org/jira/browse/MAPREDUCE-2264?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Devaraj K updated MAPREDUCE-2264: - Fix Version/s: 0.23.0 0.20.4 Status: Patch Available (was: Open) > Job status exceeds 100% in some cases > -- > > Key: MAPREDUCE-2264 > URL: https://issues.apache.org/jira/browse/MAPREDUCE-2264 > Project: Hadoop Map/Reduce > Issue Type: Bug > Components: jobtracker >Reporter: Adam Kramer >Assignee: Devaraj K > Fix For: 0.20.4, 0.23.0 > > Attachments: MAPREDUCE-2264-0.20.3.patch, MAPREDUCE-2264-trunk.patch, > more than 100%.bmp > > > I'm looking now at my jobtracker's list of running reduce tasks. One of them > is 120.05% complete, the other is 107.28% complete. > I understand that these numbers are estimates, but there is no case in which > an estimate of 100% for a non-complete task is better than an estimate of > 99.99%, nor is there any case in which an estimate greater than 100% is valid. > I suggest that whatever logic is computing these set 99.99% as a hard maximum. -- This message is automatically generated by JIRA. For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (MAPREDUCE-2264) Job status exceeds 100% in some cases
[ https://issues.apache.org/jira/browse/MAPREDUCE-2264?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13036887#comment-13036887 ] Devaraj K commented on MAPREDUCE-2264: -- It was considering compressed bytes size for calculating the totalBytes because of this it is showing the progress as more than 100%. Provided patch consider the uncompressed size of the data for calculating the progress. > Job status exceeds 100% in some cases > -- > > Key: MAPREDUCE-2264 > URL: https://issues.apache.org/jira/browse/MAPREDUCE-2264 > Project: Hadoop Map/Reduce > Issue Type: Bug > Components: jobtracker >Reporter: Adam Kramer >Assignee: Devaraj K > Fix For: 0.20.4, 0.23.0 > > Attachments: MAPREDUCE-2264-0.20.3.patch, MAPREDUCE-2264-trunk.patch, > more than 100%.bmp > > > I'm looking now at my jobtracker's list of running reduce tasks. One of them > is 120.05% complete, the other is 107.28% complete. > I understand that these numbers are estimates, but there is no case in which > an estimate of 100% for a non-complete task is better than an estimate of > 99.99%, nor is there any case in which an estimate greater than 100% is valid. > I suggest that whatever logic is computing these set 99.99% as a hard maximum. -- This message is automatically generated by JIRA. For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (MAPREDUCE-2463) Job History files are not moving to done folder when job history location is hdfs location
[
https://issues.apache.org/jira/browse/MAPREDUCE-2463?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13036876#comment-13036876
]
Devaraj K commented on MAPREDUCE-2463:
--
Thanks Todd for reviewing.
Provided patch by addressing the above comments.
> Job History files are not moving to done folder when job history location is
> hdfs location
> --
>
> Key: MAPREDUCE-2463
> URL: https://issues.apache.org/jira/browse/MAPREDUCE-2463
> Project: Hadoop Map/Reduce
> Issue Type: Bug
> Components: jobtracker
>Affects Versions: 0.23.0
>Reporter: Devaraj K
>Assignee: Devaraj K
> Fix For: 0.23.0
>
> Attachments: MAPREDUCE-2463-1.patch, MAPREDUCE-2463-2.patch,
> MAPREDUCE-2463.patch
>
>
> If "mapreduce.jobtracker.jobhistory.location" is configured as HDFS location
> then either during initialization of Job Tracker (while moving old job
> history files) or after completion of the job, history files are not moving
> to done and giving following exception.
> {code:xml}
> 2011-04-29 15:27:27,813 ERROR
> org.apache.hadoop.mapreduce.jobhistory.JobHistory: Unable to move history
> file to DONE folder.
> java.lang.IllegalArgumentException: Wrong FS:
> hdfs://10.18.52.146:9000/history/job_201104291518_0001_root, expected:
> file:///
> at org.apache.hadoop.fs.FileSystem.checkPath(FileSystem.java:402)
> at
> org.apache.hadoop.fs.RawLocalFileSystem.pathToFile(RawLocalFileSystem.java:58)
> at
> org.apache.hadoop.fs.RawLocalFileSystem.getFileStatus(RawLocalFileSystem.java:419)
> at
> org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:294)
> at org.apache.hadoop.fs.FileUtil.copy(FileUtil.java:215)
> at
> org.apache.hadoop.fs.FileSystem.copyFromLocalFile(FileSystem.java:1516)
> at
> org.apache.hadoop.fs.FileSystem.copyFromLocalFile(FileSystem.java:1492)
> at
> org.apache.hadoop.fs.FileSystem.moveFromLocalFile(FileSystem.java:1482)
> at
> org.apache.hadoop.mapreduce.jobhistory.JobHistory.moveToDoneNow(JobHistory.java:348)
> at
> org.apache.hadoop.mapreduce.jobhistory.JobHistory.access$200(JobHistory.java:61)
> at
> org.apache.hadoop.mapreduce.jobhistory.JobHistory$1.run(JobHistory.java:439)
> at
> java.util.concurrent.ThreadPoolExecutor$Worker.runTask(ThreadPoolExecutor.java:886)
> at
> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:908)
> at java.lang.Thread.run(Thread.java:619)
> {code}
--
This message is automatically generated by JIRA.
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Updated] (MAPREDUCE-2459) Cache HAR filesystem metadata
[ https://issues.apache.org/jira/browse/MAPREDUCE-2459?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Mahadev konar updated MAPREDUCE-2459: - Resolution: Fixed Hadoop Flags: [Reviewed] Status: Resolved (was: Patch Available) I just committed this to trunk. Thanks mac! > Cache HAR filesystem metadata > - > > Key: MAPREDUCE-2459 > URL: https://issues.apache.org/jira/browse/MAPREDUCE-2459 > Project: Hadoop Map/Reduce > Issue Type: Improvement > Components: harchive >Reporter: Mac Yang >Assignee: Mac Yang > Fix For: 0.23.0 > > Attachments: MAPREDUCE-2459.1.patch, MAPREDUCE-2459.2.patch > > > Each HAR file system has two index files that contains information on how > files are stored in the part files. During the block location calculation, > these indexes are reread for every file in the archive. Caching the indexes > and the status of the part files will greatly reduce the number of name node > operations during the job setup time. -- This message is automatically generated by JIRA. For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Assigned] (MAPREDUCE-2264) Job status exceeds 100% in some cases
[ https://issues.apache.org/jira/browse/MAPREDUCE-2264?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Devaraj K reassigned MAPREDUCE-2264: Assignee: Devaraj K > Job status exceeds 100% in some cases > -- > > Key: MAPREDUCE-2264 > URL: https://issues.apache.org/jira/browse/MAPREDUCE-2264 > Project: Hadoop Map/Reduce > Issue Type: Bug > Components: jobtracker >Reporter: Adam Kramer >Assignee: Devaraj K > Attachments: MAPREDUCE-2264-0.20.3.patch, MAPREDUCE-2264-trunk.patch, > more than 100%.bmp > > > I'm looking now at my jobtracker's list of running reduce tasks. One of them > is 120.05% complete, the other is 107.28% complete. > I understand that these numbers are estimates, but there is no case in which > an estimate of 100% for a non-complete task is better than an estimate of > 99.99%, nor is there any case in which an estimate greater than 100% is valid. > I suggest that whatever logic is computing these set 99.99% as a hard maximum. -- This message is automatically generated by JIRA. For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Updated] (MAPREDUCE-2264) Job status exceeds 100% in some cases
[ https://issues.apache.org/jira/browse/MAPREDUCE-2264?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Devaraj K updated MAPREDUCE-2264: - Attachment: MAPREDUCE-2264-trunk.patch MAPREDUCE-2264-0.20.3.patch > Job status exceeds 100% in some cases > -- > > Key: MAPREDUCE-2264 > URL: https://issues.apache.org/jira/browse/MAPREDUCE-2264 > Project: Hadoop Map/Reduce > Issue Type: Bug > Components: jobtracker >Reporter: Adam Kramer > Attachments: MAPREDUCE-2264-0.20.3.patch, MAPREDUCE-2264-trunk.patch, > more than 100%.bmp > > > I'm looking now at my jobtracker's list of running reduce tasks. One of them > is 120.05% complete, the other is 107.28% complete. > I understand that these numbers are estimates, but there is no case in which > an estimate of 100% for a non-complete task is better than an estimate of > 99.99%, nor is there any case in which an estimate greater than 100% is valid. > I suggest that whatever logic is computing these set 99.99% as a hard maximum. -- This message is automatically generated by JIRA. For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Updated] (MAPREDUCE-2463) Job History files are not moving to done folder when job history location is hdfs location
[
https://issues.apache.org/jira/browse/MAPREDUCE-2463?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Devaraj K updated MAPREDUCE-2463:
-
Attachment: MAPREDUCE-2463-2.patch
> Job History files are not moving to done folder when job history location is
> hdfs location
> --
>
> Key: MAPREDUCE-2463
> URL: https://issues.apache.org/jira/browse/MAPREDUCE-2463
> Project: Hadoop Map/Reduce
> Issue Type: Bug
> Components: jobtracker
>Affects Versions: 0.23.0
>Reporter: Devaraj K
>Assignee: Devaraj K
> Fix For: 0.23.0
>
> Attachments: MAPREDUCE-2463-1.patch, MAPREDUCE-2463-2.patch,
> MAPREDUCE-2463.patch
>
>
> If "mapreduce.jobtracker.jobhistory.location" is configured as HDFS location
> then either during initialization of Job Tracker (while moving old job
> history files) or after completion of the job, history files are not moving
> to done and giving following exception.
> {code:xml}
> 2011-04-29 15:27:27,813 ERROR
> org.apache.hadoop.mapreduce.jobhistory.JobHistory: Unable to move history
> file to DONE folder.
> java.lang.IllegalArgumentException: Wrong FS:
> hdfs://10.18.52.146:9000/history/job_201104291518_0001_root, expected:
> file:///
> at org.apache.hadoop.fs.FileSystem.checkPath(FileSystem.java:402)
> at
> org.apache.hadoop.fs.RawLocalFileSystem.pathToFile(RawLocalFileSystem.java:58)
> at
> org.apache.hadoop.fs.RawLocalFileSystem.getFileStatus(RawLocalFileSystem.java:419)
> at
> org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:294)
> at org.apache.hadoop.fs.FileUtil.copy(FileUtil.java:215)
> at
> org.apache.hadoop.fs.FileSystem.copyFromLocalFile(FileSystem.java:1516)
> at
> org.apache.hadoop.fs.FileSystem.copyFromLocalFile(FileSystem.java:1492)
> at
> org.apache.hadoop.fs.FileSystem.moveFromLocalFile(FileSystem.java:1482)
> at
> org.apache.hadoop.mapreduce.jobhistory.JobHistory.moveToDoneNow(JobHistory.java:348)
> at
> org.apache.hadoop.mapreduce.jobhistory.JobHistory.access$200(JobHistory.java:61)
> at
> org.apache.hadoop.mapreduce.jobhistory.JobHistory$1.run(JobHistory.java:439)
> at
> java.util.concurrent.ThreadPoolExecutor$Worker.runTask(ThreadPoolExecutor.java:886)
> at
> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:908)
> at java.lang.Thread.run(Thread.java:619)
> {code}
--
This message is automatically generated by JIRA.
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (MAPREDUCE-2459) Cache HAR filesystem metadata
[ https://issues.apache.org/jira/browse/MAPREDUCE-2459?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13036867#comment-13036867 ] Mahadev konar commented on MAPREDUCE-2459: -- +1 lgtm. Ill commit it to trunk. > Cache HAR filesystem metadata > - > > Key: MAPREDUCE-2459 > URL: https://issues.apache.org/jira/browse/MAPREDUCE-2459 > Project: Hadoop Map/Reduce > Issue Type: Improvement > Components: harchive >Reporter: Mac Yang >Assignee: Mac Yang > Fix For: 0.23.0 > > Attachments: MAPREDUCE-2459.1.patch, MAPREDUCE-2459.2.patch > > > Each HAR file system has two index files that contains information on how > files are stored in the part files. During the block location calculation, > these indexes are reread for every file in the archive. Caching the indexes > and the status of the part files will greatly reduce the number of name node > operations during the job setup time. -- This message is automatically generated by JIRA. For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Updated] (MAPREDUCE-2525) State of the checkboxes are not matching with Select All / Deselect All button, after refreshing jobtracker.jsp
[ https://issues.apache.org/jira/browse/MAPREDUCE-2525?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Devaraj K updated MAPREDUCE-2525: - Attachment: MR-UI.jpg > State of the checkboxes are not matching with Select All / Deselect All > button, after refreshing jobtracker.jsp > --- > > Key: MAPREDUCE-2525 > URL: https://issues.apache.org/jira/browse/MAPREDUCE-2525 > Project: Hadoop Map/Reduce > Issue Type: Bug > Components: jobtracker >Affects Versions: 0.23.0 > Environment: *Mozilla/5.0 (Windows; U; Windows NT 5.1; en-US; > rv:1.9.0.3) Gecko/2008092417 Firefox/3.0.3* >Reporter: Devaraj K >Assignee: Devaraj K >Priority: Minor > Attachments: MR-UI.jpg > > > These are the steps to reproduce, > # Select all the running jobs. > # Refresh the jobtracker.jsp page. > # After refreshing, *Deselect All* button becomes *Select All*, however all > the selected check boxes remain in the select state only. -- This message is automatically generated by JIRA. For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (MAPREDUCE-2489) Jobsplits with random hostnames can make the queue unusable
[ https://issues.apache.org/jira/browse/MAPREDUCE-2489?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13036847#comment-13036847 ] Jeffrey Naisbitt commented on MAPREDUCE-2489: - @Allen, apologies for not being clear in the description. Also, I expect this patch will fail in the Hudson build until HADOOP-7314 has been committed. > Jobsplits with random hostnames can make the queue unusable > --- > > Key: MAPREDUCE-2489 > URL: https://issues.apache.org/jira/browse/MAPREDUCE-2489 > Project: Hadoop Map/Reduce > Issue Type: Bug > Components: jobtracker >Reporter: Jeffrey Naisbitt >Assignee: Jeffrey Naisbitt > Attachments: MAPREDUCE-2489-mapred.patch > > > We saw an issue where a custom InputSplit was returning invalid hostnames for > the splits that were then causing the JobTracker to attempt to excessively > resolve host names. This caused a major slowdown for the JobTracker. We > should prevent invalid InputSplit hostnames from affecting everyone else. > I propose we implement some verification for the hostnames to try to ensure > that we only do DNS lookups on valid hostnames (and fail otherwise). We > could also fail the job after a certain number of failures in the resolve. -- This message is automatically generated by JIRA. For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Updated] (MAPREDUCE-2489) Jobsplits with random hostnames can make the queue unusable
[ https://issues.apache.org/jira/browse/MAPREDUCE-2489?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Jeffrey Naisbitt updated MAPREDUCE-2489: Attachment: MAPREDUCE-2489-mapred.patch > Jobsplits with random hostnames can make the queue unusable > --- > > Key: MAPREDUCE-2489 > URL: https://issues.apache.org/jira/browse/MAPREDUCE-2489 > Project: Hadoop Map/Reduce > Issue Type: Bug > Components: jobtracker >Reporter: Jeffrey Naisbitt >Assignee: Jeffrey Naisbitt > Attachments: MAPREDUCE-2489-mapred.patch > > > We saw an issue where a custom InputSplit was returning invalid hostnames for > the splits that were then causing the JobTracker to attempt to excessively > resolve host names. This caused a major slowdown for the JobTracker. We > should prevent invalid InputSplit hostnames from affecting everyone else. > I propose we implement some verification for the hostnames to try to ensure > that we only do DNS lookups on valid hostnames (and fail otherwise). We > could also fail the job after a certain number of failures in the resolve. -- This message is automatically generated by JIRA. For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Created] (MAPREDUCE-2525) State of the checkboxes are not matching with Select All / Deselect All button, after refreshing jobtracker.jsp
State of the checkboxes are not matching with Select All / Deselect All button, after refreshing jobtracker.jsp --- Key: MAPREDUCE-2525 URL: https://issues.apache.org/jira/browse/MAPREDUCE-2525 Project: Hadoop Map/Reduce Issue Type: Bug Components: jobtracker Affects Versions: 0.23.0 Environment: *Mozilla/5.0 (Windows; U; Windows NT 5.1; en-US; rv:1.9.0.3) Gecko/2008092417 Firefox/3.0.3* Reporter: Devaraj K Priority: Minor These are the steps to reproduce, # Select all the running jobs. # Refresh the jobtracker.jsp page. # After refreshing, *Deselect All* button becomes *Select All*, however all the selected check boxes remain in the select state only. -- This message is automatically generated by JIRA. For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Assigned] (MAPREDUCE-2525) State of the checkboxes are not matching with Select All / Deselect All button, after refreshing jobtracker.jsp
[ https://issues.apache.org/jira/browse/MAPREDUCE-2525?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Devaraj K reassigned MAPREDUCE-2525: Assignee: Devaraj K > State of the checkboxes are not matching with Select All / Deselect All > button, after refreshing jobtracker.jsp > --- > > Key: MAPREDUCE-2525 > URL: https://issues.apache.org/jira/browse/MAPREDUCE-2525 > Project: Hadoop Map/Reduce > Issue Type: Bug > Components: jobtracker >Affects Versions: 0.23.0 > Environment: *Mozilla/5.0 (Windows; U; Windows NT 5.1; en-US; > rv:1.9.0.3) Gecko/2008092417 Firefox/3.0.3* >Reporter: Devaraj K >Assignee: Devaraj K >Priority: Minor > > These are the steps to reproduce, > # Select all the running jobs. > # Refresh the jobtracker.jsp page. > # After refreshing, *Deselect All* button becomes *Select All*, however all > the selected check boxes remain in the select state only. -- This message is automatically generated by JIRA. For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Created] (MAPREDUCE-2524) Backport trunk heuristics for failing maps when we get fetch failures retrieving map output during shuffle
Backport trunk heuristics for failing maps when we get fetch failures retrieving map output during shuffle -- Key: MAPREDUCE-2524 URL: https://issues.apache.org/jira/browse/MAPREDUCE-2524 Project: Hadoop Map/Reduce Issue Type: Improvement Components: tasktracker Affects Versions: 0.20.204.0 Reporter: Thomas Graves Assignee: Thomas Graves Priority: Minor Fix For: 0.20.205.0 The heuristics for failing maps when we get map output fetch failures during the shuffle is pretty conservative in 20. Backport the heuristics from trunk which are more aggressive, simpler, and configurable. -- This message is automatically generated by JIRA. For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (MAPREDUCE-2297) All map reduce tasks are failing if we give invalid path jar file for Job
[
https://issues.apache.org/jira/browse/MAPREDUCE-2297?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13036831#comment-13036831
]
Devaraj K commented on MAPREDUCE-2297:
--
Hi Todd, I agree with you for job specific configuration. If we configure
archives (contains invalid path) common for all jobs then all jobs will fail.
Here all jobs may not depend on all the archives.
Consider the case of Hive, if we configure any invalid path for the property
"hive.aux.jars.path" all the jobs will fail which is not using that jar also.
> All map reduce tasks are failing if we give invalid path jar file for Job
> -
>
> Key: MAPREDUCE-2297
> URL: https://issues.apache.org/jira/browse/MAPREDUCE-2297
> Project: Hadoop Map/Reduce
> Issue Type: Bug
> Components: tasktracker
>Affects Versions: 0.20.2
>Reporter: Devaraj K
>Assignee: Devaraj K
>Priority: Minor
> Fix For: 0.20.4
>
> Attachments: MAPREDUCE-2297.patch
>
>
> This can be reproduced by giving the invalid jar file for the Job or it can
> be reproduced from hive.
> In hive-default.xml
>
> hive.aux.jars.path
>
> Provided for adding auxillaryjarsPath
>
> If we configure an invalid path for jar file, It is making all map reduce
> tasks to fail even those jobs are not depending on this jar file and it is
> giving the below exception.
> {code:xml}
> hive> select * from a join b on(a.b=b.c);
> Total MapReduce jobs = 1
> Launching Job 1 out of 1
> Number of reduce tasks not specified. Estimated from input data size: 1
> In order to change the average load for a reducer (in bytes):
> set hive.exec.reducers.bytes.per.reducer=
> In order to limit the maximum number of reducers:
> set hive.exec.reducers.max=
> In order to set a constant number of reducers:
> set mapred.reduce.tasks=
> java.io.FileNotFoundException: File does not exist: /user/root/grade.jar
> at
> org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:495)
> at
> org.apache.hadoop.filecache.DistributedCache.getTimestamp(DistributedCache.java:509)
> at
> org.apache.hadoop.mapred.JobClient.configureCommandLineOptions(JobClient.java:651)
> at org.apache.hadoop.mapred.JobClient.submitJobInternal(JobClient.java:783)
> at org.apache.hadoop.mapred.JobClient.submitJob(JobClient.java:752)
> at org.apache.hadoop.hive.ql.exec.ExecDriver.execute(ExecDriver.java:698)
> at org.apache.hadoop.hive.ql.exec.Task.executeTask(Task.java:107)
> at org.apache.hadoop.hive.ql.exec.TaskRunner.runSequential(TaskRunner.java:64)
> {code}
--
This message is automatically generated by JIRA.
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (MAPREDUCE-2470) Receiving NPE occasionally on RunningJob.getCounters() call
[ https://issues.apache.org/jira/browse/MAPREDUCE-2470?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13036814#comment-13036814 ] Robert Joseph Evans commented on MAPREDUCE-2470: Oh I forgot to add that all of the other downgrade methods in JobClient.java are similar to Job.getStatus, either they return values from the cached JobStatus which should never be null except if an IOException was caught and ignored, so they are not likely to cause an NPE. > Receiving NPE occasionally on RunningJob.getCounters() call > --- > > Key: MAPREDUCE-2470 > URL: https://issues.apache.org/jira/browse/MAPREDUCE-2470 > Project: Hadoop Map/Reduce > Issue Type: Bug > Components: client >Affects Versions: 0.21.0 > Environment: FreeBSD, Java6, Hadoop r0.21.0 >Reporter: Aaron Baff >Assignee: Robert Joseph Evans > Attachments: MAPREDUCE-2470-v1.patch, MAPREDUCE-2470-v2.patch, > counters_null_data.pcap > > > This is running in a Java daemon that is used as an interface (Thrift) to get > information and data from MR Jobs. Using JobClient.getJob(JobID) I > successfully get a RunningJob object (I'm checking for NULL), and then rarely > I get an NPE when I do RunningJob.getCounters(). This seems to occur after > the daemon has been up and running for a while, and in the event of an > Exception, I close the JobClient, set it to NULL, and a new one should then > be created on the next request for data. Yet, I still seem to be unable to > fetch the Counters. Below is the stack trace. > java.lang.NullPointerException > at org.apache.hadoop.mapred.Counters.downgrade(Counters.java:77) > at > org.apache.hadoop.mapred.JobClient$NetworkedJob.getCounters(JobClient.java:381) > at > com.telescope.HadoopThrift.service.ServiceImpl.getReportResults(ServiceImpl.java:350) > at > com.telescope.HadoopThrift.gen.HadoopThrift$Processor$getReportResults.process(HadoopThrift.java:545) > at > com.telescope.HadoopThrift.gen.HadoopThrift$Processor.process(HadoopThrift.java:421) > at > org.apache.thrift.server.TNonblockingServer$FrameBuffer.invoke(TNonblockingServer.java:697) > at > org.apache.thrift.server.THsHaServer$Invocation.run(THsHaServer.java:317) > at > java.util.concurrent.ThreadPoolExecutor$Worker.runTask(ThreadPoolExecutor.java:886) > at > java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:908) > at java.lang.Thread.run(Thread.java:619) -- This message is automatically generated by JIRA. For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (MAPREDUCE-2470) Receiving NPE occasionally on RunningJob.getCounters() call
[ https://issues.apache.org/jira/browse/MAPREDUCE-2470?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13036811#comment-13036811 ] Robert Joseph Evans commented on MAPREDUCE-2470: The difference is that with Job.getStatus it calls internally updateStatus which will throw an IOException if it is unable to get an updated status from the JobTracker. The only time you can get an NPE from that is if you catch the IOException, ignore it, and keep trying to use the Job object what was originally created. We can fix that, by modifying updateStatus to not overwrite Job.status until it knows that the updated status is not null. But that was not explicitly part of this JIRA, and I could not find any other JIRA to cover it so I though it was not a big deal, and most likely worked as designed. > Receiving NPE occasionally on RunningJob.getCounters() call > --- > > Key: MAPREDUCE-2470 > URL: https://issues.apache.org/jira/browse/MAPREDUCE-2470 > Project: Hadoop Map/Reduce > Issue Type: Bug > Components: client >Affects Versions: 0.21.0 > Environment: FreeBSD, Java6, Hadoop r0.21.0 >Reporter: Aaron Baff >Assignee: Robert Joseph Evans > Attachments: MAPREDUCE-2470-v1.patch, MAPREDUCE-2470-v2.patch, > counters_null_data.pcap > > > This is running in a Java daemon that is used as an interface (Thrift) to get > information and data from MR Jobs. Using JobClient.getJob(JobID) I > successfully get a RunningJob object (I'm checking for NULL), and then rarely > I get an NPE when I do RunningJob.getCounters(). This seems to occur after > the daemon has been up and running for a while, and in the event of an > Exception, I close the JobClient, set it to NULL, and a new one should then > be created on the next request for data. Yet, I still seem to be unable to > fetch the Counters. Below is the stack trace. > java.lang.NullPointerException > at org.apache.hadoop.mapred.Counters.downgrade(Counters.java:77) > at > org.apache.hadoop.mapred.JobClient$NetworkedJob.getCounters(JobClient.java:381) > at > com.telescope.HadoopThrift.service.ServiceImpl.getReportResults(ServiceImpl.java:350) > at > com.telescope.HadoopThrift.gen.HadoopThrift$Processor$getReportResults.process(HadoopThrift.java:545) > at > com.telescope.HadoopThrift.gen.HadoopThrift$Processor.process(HadoopThrift.java:421) > at > org.apache.thrift.server.TNonblockingServer$FrameBuffer.invoke(TNonblockingServer.java:697) > at > org.apache.thrift.server.THsHaServer$Invocation.run(THsHaServer.java:317) > at > java.util.concurrent.ThreadPoolExecutor$Worker.runTask(ThreadPoolExecutor.java:886) > at > java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:908) > at java.lang.Thread.run(Thread.java:619) -- This message is automatically generated by JIRA. For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (MAPREDUCE-2516) option to control sensitive web actions
[ https://issues.apache.org/jira/browse/MAPREDUCE-2516?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13036808#comment-13036808 ] Hudson commented on MAPREDUCE-2516: --- Integrated in Hadoop-Mapreduce-22-branch #57 (See [https://builds.apache.org/hudson/job/Hadoop-Mapreduce-22-branch/57/]) MAPREDUCE-2516. Rename webinterface.private.actions to mapreduce.jobtracker.webinterface.trusted. Contributed by Ari Rabkin. todd : http://svn.apache.org/viewcvs.cgi/?root=Apache-SVN&view=rev&rev=1125222 Files : * /hadoop/mapreduce/branches/branch-0.22/src/java/org/apache/hadoop/mapreduce/util/ConfigUtil.java * /hadoop/mapreduce/branches/branch-0.22/src/java/org/apache/hadoop/mapreduce/server/jobtracker/JTConfig.java * /hadoop/mapreduce/branches/branch-0.22/src/java/mapred-default.xml * /hadoop/mapreduce/branches/branch-0.22/CHANGES.txt * /hadoop/mapreduce/branches/branch-0.22/src/java/org/apache/hadoop/mapred/JSPUtil.java * /hadoop/mapreduce/branches/branch-0.22/src/test/mapred/org/apache/hadoop/mapred/TestWebUIAuthorization.java > option to control sensitive web actions > --- > > Key: MAPREDUCE-2516 > URL: https://issues.apache.org/jira/browse/MAPREDUCE-2516 > Project: Hadoop Map/Reduce > Issue Type: Bug >Affects Versions: 0.22.0 >Reporter: Ari Rabkin >Assignee: Ari Rabkin >Priority: Minor > Fix For: 0.22.0 > > Attachments: MAPREDUCE-2516.txt > > > as per HADOOP-7302, webinterface.private.actions should not be in trunk. But > it should be here, and should have a clearer name. -- This message is automatically generated by JIRA. For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (MAPREDUCE-2407) Make Gridmix emulate usage of Distributed Cache files
[ https://issues.apache.org/jira/browse/MAPREDUCE-2407?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13036777#comment-13036777 ] Hadoop QA commented on MAPREDUCE-2407: -- -1 overall. Here are the results of testing the latest attachment http://issues.apache.org/jira/secure/attachment/12479884/2407.v1.patch against trunk revision 1125223. +1 @author. The patch does not contain any @author tags. +1 tests included. The patch appears to include 10 new or modified tests. +1 javadoc. The javadoc tool did not generate any warning messages. +1 javac. The applied patch does not increase the total number of javac compiler warnings. +1 findbugs. The patch does not introduce any new Findbugs (version 1.3.9) warnings. +1 release audit. The applied patch does not increase the total number of release audit warnings. +1 core tests. The patch passed core unit tests. -1 contrib tests. The patch failed contrib unit tests. +1 system test framework. The patch passed system test framework compile. Test results: https://builds.apache.org/hudson/job/PreCommit-MAPREDUCE-Build/284//testReport/ Findbugs warnings: https://builds.apache.org/hudson/job/PreCommit-MAPREDUCE-Build/284//artifact/trunk/build/test/findbugs/newPatchFindbugsWarnings.html Console output: https://builds.apache.org/hudson/job/PreCommit-MAPREDUCE-Build/284//console This message is automatically generated. > Make Gridmix emulate usage of Distributed Cache files > - > > Key: MAPREDUCE-2407 > URL: https://issues.apache.org/jira/browse/MAPREDUCE-2407 > Project: Hadoop Map/Reduce > Issue Type: New Feature > Components: contrib/gridmix >Affects Versions: 0.23.0 >Reporter: Ravi Gummadi >Assignee: Ravi Gummadi > Fix For: 0.23.0 > > Attachments: 2407.patch, 2407.v1.patch > > > Currently Gridmix emulates disk IO load only. This JIRA is to make Gridmix > emulate Distributed Cache load as defined by the job-trace. -- This message is automatically generated by JIRA. For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (MAPREDUCE-2512) wait(5000) and notify() mechanism can be implemented instead of sleep(5000) in reduce task when there are no copies in progress and no new copies to schedule
[
https://issues.apache.org/jira/browse/MAPREDUCE-2512?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13036751#comment-13036751
]
Devaraj K commented on MAPREDUCE-2512:
--
This try/catch is not related to the any stuck tasks. If there are no map
outputs ready to copy, this thread goes to sleep state for 5000 millis and then
continue execution after timeout. When this thread is in sleep state, if any
thread performs some operation (like interrupt (), stop ()) on this thread, it
will throw InterruptedException and comes out from the sleep state. After
getting InterruptedException, it will ignore and continue execution normally.
This thread waits for 5000 millis even If it gets new map completion events
before 5000 millis. This can be optimized such that,
{code:title=ReduceTask.java|borderStyle=solid}
reporter.progress();
- Thread.sleep(5000);
+ synchronized (lockCopy) {
+lockCopy.wait(5000);
+ }
}
} catch (InterruptedException e) { } // IGNORE
int numNewMaps = getMapCompletionEvents();
if (numNewMaps > 0) {
+ synchronized (lockCopy) {
+lockCopy.notify();
+ }
LOG.info(reduceTask.getTaskID() + ": " +
"Got " + numNewMaps + " new map-outputs");
{code}
> wait(5000) and notify() mechanism can be implemented instead of sleep(5000)
> in reduce task when there are no copies in progress and no new copies to
> schedule
> -
>
> Key: MAPREDUCE-2512
> URL: https://issues.apache.org/jira/browse/MAPREDUCE-2512
> Project: Hadoop Map/Reduce
> Issue Type: Improvement
> Components: task
>Affects Versions: 0.20.2
>Reporter: Devaraj K
>Assignee: Devaraj K
>
> {code:title=ReduceTask.java|borderStyle=solid}
>try {
> if (numInFlight == 0 && numScheduled == 0) {
> // we should indicate progress as we don't want TT to think
> // we're stuck and kill us
> reporter.progress();
> Thread.sleep(5000);
> }
> } catch (InterruptedException e) { } // IGNORE
> {code}
> Here if we have no copies in flight and we can't schedule anything new, it is
> going to wait for 5000 millis. Instead of waiting for 5000 millis, this
> thread can wait with timeout and GetMapEventsThread can notify it if gets new
> map completion events earlier than 5000 millis time.
>
--
This message is automatically generated by JIRA.
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Updated] (MAPREDUCE-2407) Make Gridmix emulate usage of Distributed Cache files
[ https://issues.apache.org/jira/browse/MAPREDUCE-2407?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Ravi Gummadi updated MAPREDUCE-2407: Status: Patch Available (was: Open) > Make Gridmix emulate usage of Distributed Cache files > - > > Key: MAPREDUCE-2407 > URL: https://issues.apache.org/jira/browse/MAPREDUCE-2407 > Project: Hadoop Map/Reduce > Issue Type: New Feature > Components: contrib/gridmix >Affects Versions: 0.23.0 >Reporter: Ravi Gummadi >Assignee: Ravi Gummadi > Fix For: 0.23.0 > > Attachments: 2407.patch, 2407.v1.patch > > > Currently Gridmix emulates disk IO load only. This JIRA is to make Gridmix > emulate Distributed Cache load as defined by the job-trace. -- This message is automatically generated by JIRA. For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Updated] (MAPREDUCE-2407) Make Gridmix emulate usage of Distributed Cache files
[ https://issues.apache.org/jira/browse/MAPREDUCE-2407?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Ravi Gummadi updated MAPREDUCE-2407: Attachment: 2407.v1.patch Attaching new patch fixing the release audit warning. > Make Gridmix emulate usage of Distributed Cache files > - > > Key: MAPREDUCE-2407 > URL: https://issues.apache.org/jira/browse/MAPREDUCE-2407 > Project: Hadoop Map/Reduce > Issue Type: New Feature > Components: contrib/gridmix >Affects Versions: 0.23.0 >Reporter: Ravi Gummadi >Assignee: Ravi Gummadi > Fix For: 0.23.0 > > Attachments: 2407.patch, 2407.v1.patch > > > Currently Gridmix emulates disk IO load only. This JIRA is to make Gridmix > emulate Distributed Cache load as defined by the job-trace. -- This message is automatically generated by JIRA. For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Updated] (MAPREDUCE-2407) Make Gridmix emulate usage of Distributed Cache files
[ https://issues.apache.org/jira/browse/MAPREDUCE-2407?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Ravi Gummadi updated MAPREDUCE-2407: Status: Open (was: Patch Available) > Make Gridmix emulate usage of Distributed Cache files > - > > Key: MAPREDUCE-2407 > URL: https://issues.apache.org/jira/browse/MAPREDUCE-2407 > Project: Hadoop Map/Reduce > Issue Type: New Feature > Components: contrib/gridmix >Affects Versions: 0.23.0 >Reporter: Ravi Gummadi >Assignee: Ravi Gummadi > Fix For: 0.23.0 > > Attachments: 2407.patch, 2407.v1.patch > > > Currently Gridmix emulates disk IO load only. This JIRA is to make Gridmix > emulate Distributed Cache load as defined by the job-trace. -- This message is automatically generated by JIRA. For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (MAPREDUCE-2407) Make Gridmix emulate usage of Distributed Cache files
[ https://issues.apache.org/jira/browse/MAPREDUCE-2407?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13036713#comment-13036713 ] Hadoop QA commented on MAPREDUCE-2407: -- -1 overall. Here are the results of testing the latest attachment http://issues.apache.org/jira/secure/attachment/12478943/2407.patch against trunk revision 1125223. +1 @author. The patch does not contain any @author tags. +1 tests included. The patch appears to include 10 new or modified tests. +1 javadoc. The javadoc tool did not generate any warning messages. +1 javac. The applied patch does not increase the total number of javac compiler warnings. +1 findbugs. The patch does not introduce any new Findbugs (version 1.3.9) warnings. -1 release audit. The applied patch generated 3 release audit warnings (more than the trunk's current 2 warnings). +1 core tests. The patch passed core unit tests. +1 contrib tests. The patch passed contrib unit tests. +1 system test framework. The patch passed system test framework compile. Test results: https://builds.apache.org/hudson/job/PreCommit-MAPREDUCE-Build/283//testReport/ Release audit warnings: https://builds.apache.org/hudson/job/PreCommit-MAPREDUCE-Build/283//artifact/trunk/patchprocess/patchReleaseAuditProblems.txt Findbugs warnings: https://builds.apache.org/hudson/job/PreCommit-MAPREDUCE-Build/283//artifact/trunk/build/test/findbugs/newPatchFindbugsWarnings.html Console output: https://builds.apache.org/hudson/job/PreCommit-MAPREDUCE-Build/283//console This message is automatically generated. > Make Gridmix emulate usage of Distributed Cache files > - > > Key: MAPREDUCE-2407 > URL: https://issues.apache.org/jira/browse/MAPREDUCE-2407 > Project: Hadoop Map/Reduce > Issue Type: New Feature > Components: contrib/gridmix >Affects Versions: 0.23.0 >Reporter: Ravi Gummadi >Assignee: Ravi Gummadi > Fix For: 0.23.0 > > Attachments: 2407.patch > > > Currently Gridmix emulates disk IO load only. This JIRA is to make Gridmix > emulate Distributed Cache load as defined by the job-trace. -- This message is automatically generated by JIRA. For more information on JIRA, see: http://www.atlassian.com/software/jira
