[jira] Commented: (MAPREDUCE-828) Provide a mechanism to pause the jobtracker
[ https://issues.apache.org/jira/browse/MAPREDUCE-828?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=12739914#action_12739914 ] Hemanth Yamijala commented on MAPREDUCE-828: Some initial thoughts on implementation: - From the time the pause command is issued, the JT will process heartbeats by sending back a special response to the TTs indicating its state. - This special command will cause the TTs to replay their message as if the original message was not received by the JT. - The above is similar to what happens today if TTs fail to communicate with the JT. - The JT will not process any data sent by the TTs during a paused state. IOW job state will not change. Since this data will be replayed until resuming, it will not be lost and can be picked up once the JT is resumed. - The ExpireLaunchingTasks thread will pause as well, since the status of tasks last launched before the pause will not be updated. - The CleanupQueue thread on the JT which deletes files from the DFS will also be paused as it might fail DFS deletes. Thoughts on the requirements / proposal ? Provide a mechanism to pause the jobtracker --- Key: MAPREDUCE-828 URL: https://issues.apache.org/jira/browse/MAPREDUCE-828 Project: Hadoop Map/Reduce Issue Type: New Feature Components: jobtracker Reporter: Hemanth Yamijala We've seen scenarios when we have needed to stop the namenode for a maintenance activity. In such scenarios, if the jobtracker (JT) continues to run, jobs would fail due to initialization or task failures (due to DFS). We could restart the JT enabling job recovery, during such scenarios. But restart has proved to be a very intrusive activity, particularly if the JT is not at fault itself and does not require a restart. The ask is for a admin-controlled feature to pause the JT which would take it to a state somewhat analogous to the safe mode of DFS. -- This message is automatically generated by JIRA. - You can reply to this email to add a comment to the issue online.
[jira] Commented: (MAPREDUCE-227) Ability to pause/resume jobs
[ https://issues.apache.org/jira/browse/MAPREDUCE-227?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=12739913#action_12739913 ] Alejandro Abdelnur commented on MAPREDUCE-227: -- Between AUG03 and AUG10 I'll be on vacations and I will not checking email. I'll reply to your message at my return. Alejandro Ability to pause/resume jobs Key: MAPREDUCE-227 URL: https://issues.apache.org/jira/browse/MAPREDUCE-227 Project: Hadoop Map/Reduce Issue Type: New Feature Reporter: Amar Kamat Assignee: Amar Kamat Attachments: HADOOP-4350-v1.2.patch, HADOOP-4350-v1.3.patch Consider a case where the user job depends on some external entity/service like a database or a web service. If the service needs restart or encounters a failure, the user should be able to pause the job and resume only when the service is up. This will be better than re-executing the whole job. Hence there should be some way to pause/resume jobs (from web-ui/command line) etc. -- This message is automatically generated by JIRA. - You can reply to this email to add a comment to the issue online.
[jira] Updated: (MAPREDUCE-659) gridmix2 not compiling under mapred module trunk/src/benchmarks/gridmix2
[ https://issues.apache.org/jira/browse/MAPREDUCE-659?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Giridharan Kesavan updated MAPREDUCE-659: - Resolution: Fixed Fix Version/s: 0.21.0 Status: Resolved (was: Patch Available) I just committed this. Thanks for the comments. gridmix2 not compiling under mapred module trunk/src/benchmarks/gridmix2 - Key: MAPREDUCE-659 URL: https://issues.apache.org/jira/browse/MAPREDUCE-659 Project: Hadoop Map/Reduce Issue Type: Bug Components: build Environment: latest trunk Reporter: Iyappan Srinivasan Assignee: Giridharan Kesavan Priority: Critical Fix For: 0.21.0 Attachments: 659-1.patch, MAPREDUCE-659-V2.PATCH, MAPREDUCE-659.PATCH When build is tried in gridmix2, it fails trunk/src/benchmarks/gridmix2 $ ant Buildfile: build.xml init: compile: [javac] Compiling 3 source files to /home/iyappans/new_trunk1/mapreduce/trunk/src/benchmarks/gridmix2/build [javac] /home/iyappans/new_trunk1/mapreduce/trunk/src/benchmarks/gridmix2/src/java/org/apache/hadoop/mapred/GridMixRunner.java:40: package org.apache.hadoop.streaming does not exist [javac] import org.apache.hadoop.streaming.StreamJob; [javac] ^ [javac] /home/iyappans/new_trunk1/mapreduce/trunk/src/benchmarks/gridmix2/src/java/org/apache/hadoop/mapred/GridMixRunner.java:123: cannot find symbol [javac] symbol: variable StreamJob [javac] JobConf jobconf = StreamJob.createJob(args); [javac] ^ [javac] Note: Some input files use or override a deprecated API. [javac] Note: Recompile with -Xlint:deprecation for details. [javac] 2 errors BUILD FAILED /home/iyappans/new_trunk1/mapreduce/trunk/src/benchmarks/gridmix2/build.xml:27: Compile failed; see the compiler error output for details. Total time: 1 second -- This message is automatically generated by JIRA. - You can reply to this email to add a comment to the issue online.
[jira] Updated: (MAPREDUCE-478) separate jvm param for mapper and reducer
[ https://issues.apache.org/jira/browse/MAPREDUCE-478?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Arun C Murthy updated MAPREDUCE-478: Fix Version/s: 0.21.0 Status: Patch Available (was: Open) separate jvm param for mapper and reducer - Key: MAPREDUCE-478 URL: https://issues.apache.org/jira/browse/MAPREDUCE-478 Project: Hadoop Map/Reduce Issue Type: Improvement Reporter: Koji Noguchi Assignee: Arun C Murthy Priority: Minor Fix For: 0.21.0 Attachments: HADOOP-5684_0_20090420.patch, MAPREDUCE-478_0_20090804.patch, MAPREDUCE-478_0_20090804_yhadoop20.patch Memory footprint of mapper and reducer can differ. It would be nice if we can pass different jvm param (mapred.child.java.opts) for mappers and reducers. -- This message is automatically generated by JIRA. - You can reply to this email to add a comment to the issue online.
[jira] Commented: (MAPREDUCE-706) Support for FIFO pools in the fair scheduler
[ https://issues.apache.org/jira/browse/MAPREDUCE-706?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=12739939#action_12739939 ] Hadoop QA commented on MAPREDUCE-706: - -1 overall. Here are the results of testing the latest attachment http://issues.apache.org/jira/secure/attachment/12415552/mapreduce-706.v3.patch against trunk revision 800693. +1 @author. The patch does not contain any @author tags. +1 tests included. The patch appears to include 9 new or modified tests. +1 javadoc. The javadoc tool did not generate any warning messages. +1 javac. The applied patch does not increase the total number of javac compiler warnings. +1 findbugs. The patch does not introduce any new Findbugs warnings. +1 release audit. The applied patch does not increase the total number of release audit warnings. +1 core tests. The patch passed core unit tests. -1 contrib tests. The patch failed contrib unit tests. Test results: http://hudson.zones.apache.org/hudson/job/Mapreduce-Patch-vesta.apache.org/443/testReport/ Findbugs warnings: http://hudson.zones.apache.org/hudson/job/Mapreduce-Patch-vesta.apache.org/443/artifact/trunk/build/test/findbugs/newPatchFindbugsWarnings.html Checkstyle results: http://hudson.zones.apache.org/hudson/job/Mapreduce-Patch-vesta.apache.org/443/artifact/trunk/build/test/checkstyle-errors.html Console output: http://hudson.zones.apache.org/hudson/job/Mapreduce-Patch-vesta.apache.org/443/console This message is automatically generated. Support for FIFO pools in the fair scheduler Key: MAPREDUCE-706 URL: https://issues.apache.org/jira/browse/MAPREDUCE-706 Project: Hadoop Map/Reduce Issue Type: New Feature Components: contrib/fair-share Reporter: Matei Zaharia Assignee: Matei Zaharia Attachments: fsdesigndoc.pdf, fsdesigndoc.tex, mapreduce-706.patch, mapreduce-706.v1.patch, mapreduce-706.v2.patch, mapreduce-706.v3.patch The fair scheduler should support making the internal scheduling algorithm for some pools be FIFO instead of fair sharing in order to work better for batch workloads. FIFO pools will behave exactly like the current default scheduler, sorting jobs by priority and then submission time. Pools will have their scheduling algorithm set through the pools config file, and it will be changeable at runtime. To support this feature, I'm also changing the internal logic of the fair scheduler to no longer use deficits. Instead, for fair sharing, we will assign tasks to the job farthest below its share as a ratio of its share. This is easier to combine with other scheduling algorithms and leads to a more stable sharing situation, avoiding unfairness issues brought up in MAPREDUCE-543 and MAPREDUCE-544 that happen when some jobs have long tasks. The new preemption (MAPREDUCE-551) will ensure that critical jobs can gain their fair share within a bounded amount of time. -- This message is automatically generated by JIRA. - You can reply to this email to add a comment to the issue online.
[jira] Created: (MAPREDUCE-829) When localizing job configuration from FS, TTs configuration on local disk is not loaded at all
When localizing job configuration from FS, TTs configuration on local disk is not loaded at all --- Key: MAPREDUCE-829 URL: https://issues.apache.org/jira/browse/MAPREDUCE-829 Project: Hadoop Map/Reduce Issue Type: Bug Components: tasktracker Reporter: Vinod K V We should first load the local configuration fConf, over which the job.xml from the JobTracker's file system should be loaded. This is needed so as to enforce settings specific to the TaskTracker if it has some. -- This message is automatically generated by JIRA. - You can reply to this email to add a comment to the issue online.
[jira] Updated: (MAPREDUCE-814) Move completed Job history files to HDFS
[ https://issues.apache.org/jira/browse/MAPREDUCE-814?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Sharad Agarwal updated MAPREDUCE-814: - Attachment: 814_v4.patch Changes from last patch: Refactored JobTracker constructor to fix an issue related to Jobtracker recovery requiring history to be inited. Move completed Job history files to HDFS Key: MAPREDUCE-814 URL: https://issues.apache.org/jira/browse/MAPREDUCE-814 Project: Hadoop Map/Reduce Issue Type: New Feature Components: jobtracker Reporter: Sharad Agarwal Assignee: Sharad Agarwal Attachments: 814_v1.patch, 814_v2.patch, 814_v3.patch, 814_v4.patch Currently completed job history files remain on the jobtracker node. Having the files available on HDFS will enable clients to access these files more easily. -- This message is automatically generated by JIRA. - You can reply to this email to add a comment to the issue online.
[jira] Updated: (MAPREDUCE-824) Support a hierarchy of queues in the capacity scheduler
[ https://issues.apache.org/jira/browse/MAPREDUCE-824?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] rahul k singh updated MAPREDUCE-824: Attachment: HADOOP-824-1.patch Uploaded the patch: Patch implements the model discussed above. This is first cut of the implementation. Some Class level details: == The hierarchial queue follows composite pattern. Component AbstractQueue : - This is abstraction for all the queues. - implements default behavior for some of the abstraction. - declares interfaces to access the children and its components. - defines an interface for accessing a queue's parent in the recursive structure. Composite - ContainerQueue: - This provides functionality mentioned as the part of Sub-Queues. - consists of composition of AbstractQueue. - provides functionality to manipulate children - delegates respective children component calls. Leaf -- JobQueue: - This provides functionality similar to Leaf Queue mentioned in the document above. - Jobs would be submitted only to this queue. - Consists of list of jobs and acts on the JobInProgressListener events. JobQueuesManager no more maintains list of runningJob and waitingJobs , they are maintained by JobQueue. It simply delegates the call to JobQueue. AbstractQueue have QueueSchedulingContext ,defined , it consists of all the queue related information required for scheduling decision. Some refactoring has been done in terms of moving out some of the inner classes from CapacityTaskScheduler , esp the QueueSchedulingInfo and TaskSchedulingInfo. I ll be uploading patch with testcases. Support a hierarchy of queues in the capacity scheduler --- Key: MAPREDUCE-824 URL: https://issues.apache.org/jira/browse/MAPREDUCE-824 Project: Hadoop Map/Reduce Issue Type: New Feature Components: contrib/capacity-sched Reporter: Hemanth Yamijala Attachments: HADOOP-824-1.patch Currently in Capacity Scheduler, cluster capacity is divided among the queues based on the queue capacity. These queues typically represent an organization and the capacity of the queue represents the capacity the organization is entitled to. Most organizations are large and need to divide their capacity among sub-organizations they have. Or they may want to divide the capacity based on a category or type of jobs they run. This JIRA covers the requirements and other details to provide the above feature. -- This message is automatically generated by JIRA. - You can reply to this email to add a comment to the issue online.
[jira] Updated: (MAPREDUCE-479) Add reduce ID to shuffle clienttrace
[ https://issues.apache.org/jira/browse/MAPREDUCE-479?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Jiaqi Tan updated MAPREDUCE-479: Release Note: Adds Reduce Attempt ID to ClientTrace log messages, and adds Reduce Attempt ID to HTTP query string sent to mapOutputServlet. Extracts partition number from attempt ID. (was: Adds Reduce Attempt ID to ClientTrace log messages, and adds Reduce Attempt ID to HTTP query string sent to mapOutputServlet.) Status: Patch Available (was: Open) Did microbenchmark of shuffle durations with and without added reduce attempt ID transmission and reduce partition number extraction; shuffle times before and after this patch are statistically comparable (chi-squared test for distribution similarity of shuffle times, p-value 0.23 = null-hypothesis of statistically different distributions not rejected); thus this patch does not cause any performance impact. Add reduce ID to shuffle clienttrace Key: MAPREDUCE-479 URL: https://issues.apache.org/jira/browse/MAPREDUCE-479 Project: Hadoop Map/Reduce Issue Type: Improvement Affects Versions: 0.21.0 Reporter: Jiaqi Tan Assignee: Jiaqi Tan Priority: Minor Fix For: 0.21.0 Attachments: HADOOP-6013.patch, MAPREDUCE-479-1.patch, MAPREDUCE-479-2.patch, MAPREDUCE-479-3.patch, MAPREDUCE-479-4.patch, MAPREDUCE-479.patch Current clienttrace messages from shuffles note only the destination map ID but not the source reduce ID. Having both source and destination ID of each shuffle enables full tracing of execution. -- This message is automatically generated by JIRA. - You can reply to this email to add a comment to the issue online.
[jira] Commented: (MAPREDUCE-805) Deadlock in Jobtracker
[
https://issues.apache.org/jira/browse/MAPREDUCE-805?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=12740074#action_12740074
]
Amar Kamat commented on MAPREDUCE-805:
--
Test failed with 2 errors TestReduceFetch FAILED (timeout) and
TestTaskTrackerMemoryManager FAILED. Doesnt seem related but will debug.
Deadlock in Jobtracker
--
Key: MAPREDUCE-805
URL: https://issues.apache.org/jira/browse/MAPREDUCE-805
Project: Hadoop Map/Reduce
Issue Type: Bug
Reporter: Michael Tamm
Attachments: MAPREDUCE-805-v1.1.patch, MAPREDUCE-805-v1.2.patch,
MAPREDUCE-805-v1.3.patch, MAPREDUCE-805-v1.6.patch
We are running a hadoop cluster (version 0.20.0) and have detected the
following deadlock on our jobtracker:
{code}
IPC Server handler 51 on 9001:
at
org.apache.hadoop.mapred.JobInProgress.getCounters(JobInProgress.java:943)
- waiting to lock 0x7f2b6fb46130 (a
org.apache.hadoop.mapred.JobInProgress)
at
org.apache.hadoop.mapred.JobTracker.getJobCounters(JobTracker.java:3102)
- locked 0x7f2b5f026000 (a org.apache.hadoop.mapred.JobTracker)
at sun.reflect.GeneratedMethodAccessor21.invoke(Unknown Source)
at
sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:25)
at java.lang.reflect.Method.invoke(Method.java:597)
at org.apache.hadoop.ipc.RPC$Server.call(RPC.java:508)
at org.apache.hadoop.ipc.Server$Handler$1.run(Server.java:959)
at org.apache.hadoop.ipc.Server$Handler$1.run(Server.java:955)
at java.security.AccessController.doPrivileged(Native Method)
at javax.security.auth.Subject.doAs(Subject.java:396)
at org.apache.hadoop.ipc.Server$Handler.run(Server.java:953)
pool-1-thread-2:
at org.apache.hadoop.mapred.JobTracker.finalizeJob(JobTracker.java:2017)
- waiting to lock 0x7f2b5f026000 (a
org.apache.hadoop.mapred.JobTracker)
at
org.apache.hadoop.mapred.JobInProgress.garbageCollect(JobInProgress.java:2483)
- locked 0x7f2b6fb46130 (a org.apache.hadoop.mapred.JobInProgress)
at
org.apache.hadoop.mapred.JobInProgress.terminateJob(JobInProgress.java:2152)
- locked 0x7f2b6fb46130 (a org.apache.hadoop.mapred.JobInProgress)
at
org.apache.hadoop.mapred.JobInProgress.terminate(JobInProgress.java:2169)
- locked 0x7f2b6fb46130 (a org.apache.hadoop.mapred.JobInProgress)
at org.apache.hadoop.mapred.JobInProgress.fail(JobInProgress.java:2245)
- locked 0x7f2b6fb46130 (a org.apache.hadoop.mapred.JobInProgress)
at
org.apache.hadoop.mapred.EagerTaskInitializationListener$InitJob.run(EagerTaskInitializationListener.java:86)
at
java.util.concurrent.ThreadPoolExecutor$Worker.runTask(ThreadPoolExecutor.java:886)
at
java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:908)
at java.lang.Thread.run(Thread.java:619)
{code}
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
[jira] Updated: (MAPREDUCE-825) JobClient completion poll interval of 5s causes slow tests in local mode
[ https://issues.apache.org/jira/browse/MAPREDUCE-825?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Aaron Kimball updated MAPREDUCE-825: Status: Open (was: Patch Available) JobClient completion poll interval of 5s causes slow tests in local mode Key: MAPREDUCE-825 URL: https://issues.apache.org/jira/browse/MAPREDUCE-825 Project: Hadoop Map/Reduce Issue Type: Bug Reporter: Aaron Kimball Assignee: Aaron Kimball Priority: Minor Attachments: completion-poll-interval.patch The JobClient.NetworkedJob.waitForCompletion() method polls for job completion every 5 seconds. When running a set of short tests in pseudo-distributed mode, this is unnecessarily slow and causes lots of wasted time. When bandwidth is not scarce, setting the poll interval to 100 ms results in a 4x speedup in some tests. This interval should be parametrized to allow users to control the interval for testing purposes. -- This message is automatically generated by JIRA. - You can reply to this email to add a comment to the issue online.
[jira] Updated: (MAPREDUCE-825) JobClient completion poll interval of 5s causes slow tests in local mode
[ https://issues.apache.org/jira/browse/MAPREDUCE-825?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Aaron Kimball updated MAPREDUCE-825: Attachment: MAPREDUCE-825.2.patch Attaching a new patch that also includes a new jobclient.progress.monitor.poll.interval setting; default is 1000 ms. Modified TestTaskFail to set the completion poll interval to 50 ms. With default (5000) ms timeout, test runtime was 3 minutes 15 seconds. Setting the timeout to 50 ms reduced test runtime to 3 minutes 8 seconds. If we expect an average of 2500 milliseconds wasted per job in the default case, then this is 2500*3 = 7500 ms expected to be wasted, so the observed speedup seems correct. To be sure, I also set the timeout to 2 ms; test runtime went up to 3 minutes 52 seconds. So there's definitely a correlation. JobClient completion poll interval of 5s causes slow tests in local mode Key: MAPREDUCE-825 URL: https://issues.apache.org/jira/browse/MAPREDUCE-825 Project: Hadoop Map/Reduce Issue Type: Bug Reporter: Aaron Kimball Assignee: Aaron Kimball Priority: Minor Attachments: completion-poll-interval.patch, MAPREDUCE-825.2.patch The JobClient.NetworkedJob.waitForCompletion() method polls for job completion every 5 seconds. When running a set of short tests in pseudo-distributed mode, this is unnecessarily slow and causes lots of wasted time. When bandwidth is not scarce, setting the poll interval to 100 ms results in a 4x speedup in some tests. This interval should be parametrized to allow users to control the interval for testing purposes. -- This message is automatically generated by JIRA. - You can reply to this email to add a comment to the issue online.
[jira] Updated: (MAPREDUCE-706) Support for FIFO pools in the fair scheduler
[ https://issues.apache.org/jira/browse/MAPREDUCE-706?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Matei Zaharia updated MAPREDUCE-706: Status: Open (was: Patch Available) Support for FIFO pools in the fair scheduler Key: MAPREDUCE-706 URL: https://issues.apache.org/jira/browse/MAPREDUCE-706 Project: Hadoop Map/Reduce Issue Type: New Feature Components: contrib/fair-share Reporter: Matei Zaharia Assignee: Matei Zaharia Attachments: fsdesigndoc.pdf, fsdesigndoc.tex, mapreduce-706.patch, mapreduce-706.v1.patch, mapreduce-706.v2.patch, mapreduce-706.v3.patch, mapreduce-706.v4.patch The fair scheduler should support making the internal scheduling algorithm for some pools be FIFO instead of fair sharing in order to work better for batch workloads. FIFO pools will behave exactly like the current default scheduler, sorting jobs by priority and then submission time. Pools will have their scheduling algorithm set through the pools config file, and it will be changeable at runtime. To support this feature, I'm also changing the internal logic of the fair scheduler to no longer use deficits. Instead, for fair sharing, we will assign tasks to the job farthest below its share as a ratio of its share. This is easier to combine with other scheduling algorithms and leads to a more stable sharing situation, avoiding unfairness issues brought up in MAPREDUCE-543 and MAPREDUCE-544 that happen when some jobs have long tasks. The new preemption (MAPREDUCE-551) will ensure that critical jobs can gain their fair share within a bounded amount of time. -- This message is automatically generated by JIRA. - You can reply to this email to add a comment to the issue online.
[jira] Updated: (MAPREDUCE-706) Support for FIFO pools in the fair scheduler
[ https://issues.apache.org/jira/browse/MAPREDUCE-706?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Matei Zaharia updated MAPREDUCE-706: Attachment: mapreduce-706.v4.patch I've made a few changes to the Web UI to show more / better info: - Pools now have their scheduling mode and fair shares displayed. - Jobs in a FIFO pool have their fair share displayed as NA instead of 0. Attaching a patch with these changes. Support for FIFO pools in the fair scheduler Key: MAPREDUCE-706 URL: https://issues.apache.org/jira/browse/MAPREDUCE-706 Project: Hadoop Map/Reduce Issue Type: New Feature Components: contrib/fair-share Reporter: Matei Zaharia Assignee: Matei Zaharia Attachments: fsdesigndoc.pdf, fsdesigndoc.tex, mapreduce-706.patch, mapreduce-706.v1.patch, mapreduce-706.v2.patch, mapreduce-706.v3.patch, mapreduce-706.v4.patch The fair scheduler should support making the internal scheduling algorithm for some pools be FIFO instead of fair sharing in order to work better for batch workloads. FIFO pools will behave exactly like the current default scheduler, sorting jobs by priority and then submission time. Pools will have their scheduling algorithm set through the pools config file, and it will be changeable at runtime. To support this feature, I'm also changing the internal logic of the fair scheduler to no longer use deficits. Instead, for fair sharing, we will assign tasks to the job farthest below its share as a ratio of its share. This is easier to combine with other scheduling algorithms and leads to a more stable sharing situation, avoiding unfairness issues brought up in MAPREDUCE-543 and MAPREDUCE-544 that happen when some jobs have long tasks. The new preemption (MAPREDUCE-551) will ensure that critical jobs can gain their fair share within a bounded amount of time. -- This message is automatically generated by JIRA. - You can reply to this email to add a comment to the issue online.
[jira] Commented: (MAPREDUCE-749) Make Sqoop unit tests more Hudson-friendly
[ https://issues.apache.org/jira/browse/MAPREDUCE-749?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=12740344#action_12740344 ] Hadoop QA commented on MAPREDUCE-749: - -1 overall. Here are the results of testing the latest attachment http://issues.apache.org/jira/secure/attachment/12415424/MAPREDUCE-749.3.patch against trunk revision 801517. +1 @author. The patch does not contain any @author tags. +1 tests included. The patch appears to include 5 new or modified tests. +1 javadoc. The javadoc tool did not generate any warning messages. +1 javac. The applied patch does not increase the total number of javac compiler warnings. +1 findbugs. The patch does not introduce any new Findbugs warnings. +1 release audit. The applied patch does not increase the total number of release audit warnings. +1 core tests. The patch passed core unit tests. -1 contrib tests. The patch failed contrib unit tests. Test results: http://hudson.zones.apache.org/hudson/job/Mapreduce-Patch-vesta.apache.org/447/testReport/ Findbugs warnings: http://hudson.zones.apache.org/hudson/job/Mapreduce-Patch-vesta.apache.org/447/artifact/trunk/build/test/findbugs/newPatchFindbugsWarnings.html Checkstyle results: http://hudson.zones.apache.org/hudson/job/Mapreduce-Patch-vesta.apache.org/447/artifact/trunk/build/test/checkstyle-errors.html Console output: http://hudson.zones.apache.org/hudson/job/Mapreduce-Patch-vesta.apache.org/447/console This message is automatically generated. Make Sqoop unit tests more Hudson-friendly -- Key: MAPREDUCE-749 URL: https://issues.apache.org/jira/browse/MAPREDUCE-749 Project: Hadoop Map/Reduce Issue Type: Improvement Components: contrib/sqoop Reporter: Aaron Kimball Assignee: Aaron Kimball Attachments: MAPREDUCE-749.2.patch, MAPREDUCE-749.3.patch, MAPREDUCE-749.patch Hudson servers (other than Apache's) need to be able to run the sqoop unit tests which depend on thirdparty JDBC drivers / database implementations. The build.xml needs some refactoring to make this happen. -- This message is automatically generated by JIRA. - You can reply to this email to add a comment to the issue online.
[jira] Commented: (MAPREDUCE-749) Make Sqoop unit tests more Hudson-friendly
[ https://issues.apache.org/jira/browse/MAPREDUCE-749?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=12740354#action_12740354 ] Aaron Kimball commented on MAPREDUCE-749: - Streaming test failures are unrelated Make Sqoop unit tests more Hudson-friendly -- Key: MAPREDUCE-749 URL: https://issues.apache.org/jira/browse/MAPREDUCE-749 Project: Hadoop Map/Reduce Issue Type: Improvement Components: contrib/sqoop Reporter: Aaron Kimball Assignee: Aaron Kimball Attachments: MAPREDUCE-749.2.patch, MAPREDUCE-749.3.patch, MAPREDUCE-749.patch Hudson servers (other than Apache's) need to be able to run the sqoop unit tests which depend on thirdparty JDBC drivers / database implementations. The build.xml needs some refactoring to make this happen. -- This message is automatically generated by JIRA. - You can reply to this email to add a comment to the issue online.
[jira] Updated: (MAPREDUCE-478) separate jvm param for mapper and reducer
[ https://issues.apache.org/jira/browse/MAPREDUCE-478?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Arun C Murthy updated MAPREDUCE-478: Status: Patch Available (was: Open) separate jvm param for mapper and reducer - Key: MAPREDUCE-478 URL: https://issues.apache.org/jira/browse/MAPREDUCE-478 Project: Hadoop Map/Reduce Issue Type: Improvement Reporter: Koji Noguchi Assignee: Arun C Murthy Priority: Minor Fix For: 0.21.0 Attachments: HADOOP-5684_0_20090420.patch, MAPREDUCE-478_0_20090804.patch, MAPREDUCE-478_0_20090804_yhadoop20.patch, MAPREDUCE-478_1_20090806.patch, MAPREDUCE-478_1_20090806_yhadoop20.patch Memory footprint of mapper and reducer can differ. It would be nice if we can pass different jvm param (mapred.child.java.opts) for mappers and reducers. -- This message is automatically generated by JIRA. - You can reply to this email to add a comment to the issue online.
[jira] Commented: (MAPREDUCE-372) Change org.apache.hadoop.mapred.lib.ChainMapper/Reducer to use new api.
[ https://issues.apache.org/jira/browse/MAPREDUCE-372?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=12740386#action_12740386 ] Amareshwari Sriramadasu commented on MAPREDUCE-372: --- Test failure TestTaskTrackerMemoryManager is not related to the patch. It is due to HADOOP-6075 Change org.apache.hadoop.mapred.lib.ChainMapper/Reducer to use new api. --- Key: MAPREDUCE-372 URL: https://issues.apache.org/jira/browse/MAPREDUCE-372 Project: Hadoop Map/Reduce Issue Type: Sub-task Reporter: Amareshwari Sriramadasu Assignee: Amareshwari Sriramadasu Fix For: 0.21.0 Attachments: patch-372-1.txt, patch-372.txt -- This message is automatically generated by JIRA. - You can reply to this email to add a comment to the issue online.
[jira] Commented: (MAPREDUCE-779) Add node health failures into JobTrackerStatistics
[
https://issues.apache.org/jira/browse/MAPREDUCE-779?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=12740387#action_12740387
]
Hemanth Yamijala commented on MAPREDUCE-779:
I looked at the patch. Overall, looking good.
Sharad and I feel that the method
JobTrackerStatistics.taskTrackerHealthCheckFailed() is not required. I think it
is assumed that the tasktracker stat will be created and available. Further, it
seems like the class JobTrackerStatistics manages the TasktrackerStatistics
instances and nothing more. So, the method seems out of place. Hence, wherever
we need to update the health check failure count, we could simply call
JobTrackerStatistics.getTaskTrackerStat().incrHealthCheckFailed(). Sreekanth,
any reason you thought the TaskTrackerStat for a TT will not be available and
hence were implicitly managing the creation of it ?
One more minor nit is we can rename the label Blacklisted due to health
Failures to Failed Health Checks just to reduce the verbosity.
Other than this +1
Add node health failures into JobTrackerStatistics
--
Key: MAPREDUCE-779
URL: https://issues.apache.org/jira/browse/MAPREDUCE-779
Project: Hadoop Map/Reduce
Issue Type: Improvement
Components: jobtracker
Reporter: Sreekanth Ramakrishnan
Assignee: Sreekanth Ramakrishnan
Attachments: mapreduce-779-1.patch, mapreduce-779-2.patch,
mapreduce-779-3.patch
Add the node health failure counts into {{JobTrackerStatistics}}.
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
[jira] Commented: (MAPREDUCE-757) JobConf will not be deleted from the logs folder if job retires from finalizeJob()
[ https://issues.apache.org/jira/browse/MAPREDUCE-757?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=12740399#action_12740399 ] Amar Kamat commented on MAPREDUCE-757: -- ant tests and contrib tests passed. JobConf will not be deleted from the logs folder if job retires from finalizeJob() -- Key: MAPREDUCE-757 URL: https://issues.apache.org/jira/browse/MAPREDUCE-757 Project: Hadoop Map/Reduce Issue Type: Bug Components: jobtracker Reporter: Amar Kamat Assignee: Amar Kamat Attachments: MAPREDUCE-757-v1.0.patch, MAPREDUCE-757-v2.0.patch MAPREDUCE-130 fixed the case where the job is retired from the retire jobs thread. But jobs can also retire when the num-job-per-user limit is exceeded. In such cases the conf file will not be deleted. -- This message is automatically generated by JIRA. - You can reply to this email to add a comment to the issue online.
[jira] Updated: (MAPREDUCE-779) Add node health failures into JobTrackerStatistics
[
https://issues.apache.org/jira/browse/MAPREDUCE-779?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Sreekanth Ramakrishnan updated MAPREDUCE-779:
-
Attachment: mapreduce-779-4.patch
Had added that method assuming that TaskTrackerStat is not created until first
task is run/failed in a lazy manner, now rechecked that the stat object is
created when tracker is added. So I have removed the method.
Renamed the heading in JSP.
Also added {{TestTaskTrackerBlacklisting}} to commit-tests list.
Add node health failures into JobTrackerStatistics
--
Key: MAPREDUCE-779
URL: https://issues.apache.org/jira/browse/MAPREDUCE-779
Project: Hadoop Map/Reduce
Issue Type: Improvement
Components: jobtracker
Reporter: Sreekanth Ramakrishnan
Assignee: Sreekanth Ramakrishnan
Attachments: mapreduce-779-1.patch, mapreduce-779-2.patch,
mapreduce-779-3.patch, mapreduce-779-4.patch
Add the node health failure counts into {{JobTrackerStatistics}}.
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
[jira] Commented: (MAPREDUCE-375) Change org.apache.hadoop.mapred.lib.NLineInputFormat and org.apache.hadoop.mapred.MapFileOutputFormat to use new api.
[ https://issues.apache.org/jira/browse/MAPREDUCE-375?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=12740409#action_12740409 ] Hadoop QA commented on MAPREDUCE-375: - -1 overall. Here are the results of testing the latest attachment http://issues.apache.org/jira/secure/attachment/12415117/patch-375-2.txt against trunk revision 801517. +1 @author. The patch does not contain any @author tags. +1 tests included. The patch appears to include 6 new or modified tests. +1 javadoc. The javadoc tool did not generate any warning messages. +1 javac. The applied patch does not increase the total number of javac compiler warnings. +1 findbugs. The patch does not introduce any new Findbugs warnings. +1 release audit. The applied patch does not increase the total number of release audit warnings. +1 core tests. The patch passed core unit tests. -1 contrib tests. The patch failed contrib unit tests. Test results: http://hudson.zones.apache.org/hudson/job/Mapreduce-Patch-vesta.apache.org/448/testReport/ Findbugs warnings: http://hudson.zones.apache.org/hudson/job/Mapreduce-Patch-vesta.apache.org/448/artifact/trunk/build/test/findbugs/newPatchFindbugsWarnings.html Checkstyle results: http://hudson.zones.apache.org/hudson/job/Mapreduce-Patch-vesta.apache.org/448/artifact/trunk/build/test/checkstyle-errors.html Console output: http://hudson.zones.apache.org/hudson/job/Mapreduce-Patch-vesta.apache.org/448/console This message is automatically generated. Change org.apache.hadoop.mapred.lib.NLineInputFormat and org.apache.hadoop.mapred.MapFileOutputFormat to use new api. -- Key: MAPREDUCE-375 URL: https://issues.apache.org/jira/browse/MAPREDUCE-375 Project: Hadoop Map/Reduce Issue Type: Sub-task Reporter: Amareshwari Sriramadasu Assignee: Amareshwari Sriramadasu Fix For: 0.21.0 Attachments: patch-375-1.txt, patch-375-2.txt, patch-375.txt -- This message is automatically generated by JIRA. - You can reply to this email to add a comment to the issue online.
