[jira] Updated: (MAPREDUCE-745) TestRecoveryManager fails sometimes
[ https://issues.apache.org/jira/browse/MAPREDUCE-745?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Amar Kamat updated MAPREDUCE-745: - Attachment: mapred-745-yahoo-internal.patch Attaching a patch for resolving conflicts when the yahoo-hadoop-distribution is rolled forwarded to MAPREDUCE-745 (aka SVN r806173). > TestRecoveryManager fails sometimes > --- > > Key: MAPREDUCE-745 > URL: https://issues.apache.org/jira/browse/MAPREDUCE-745 > Project: Hadoop Map/Reduce > Issue Type: Bug > Components: jobtracker >Reporter: Amareshwari Sriramadasu >Assignee: Amar Kamat > Fix For: 0.20.1 > > Attachments: mapred-745-yahoo-internal.patch, > MAPREDUCE-745-v1.0.patch, MAPREDUCE-745-v1.2.patch, MAPREDUCE-745-v1.3.patch, > MAPREDUCE-745-v1.7.patch, MAPREDUCE-745-v1.8-branch-0.20.patch, > MAPREDUCE-745-v1.8.1-branch-0.20.patch, MAPREDUCE-745-v1.8.patch, > TEST-org.apache.hadoop.mapred.TestRecoveryManager.txt, > TEST-org.apache.hadoop.mapred.TestRecoveryManager.txt, > TEST-org.apache.hadoop.mapred.TestRecoveryManager.txt > > -- This message is automatically generated by JIRA. - You can reply to this email to add a comment to the issue online.
[jira] Commented: (MAPREDUCE-824) Support a hierarchy of queues in the capacity scheduler
[
https://issues.apache.org/jira/browse/MAPREDUCE-824?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12745827#action_12745827
]
Hemanth Yamijala commented on MAPREDUCE-824:
Looked at the test cases:
- Code seems duplicated between CapacitySchedulerUtils and
CapacityTaskScheduler and TestContainerQueue.
- In some test cases, when we create a queue, it is already adding a child to
the parent. So, why do we need additional calls to addChildren ?
- What's the difference between testConfiguredCapacity and testMinCapacity ?
- The test cases testing scheduling are nice. The comments are out of sync a
bit, and will be hard to maintain. Instead I suggest that we assert what we are
documenting in the tests itself, so that they themselves read as comments, and
will also always be in sync.
- As discussed, getCapacity() should not return max capacity any time. It
should always return the current capacity or limit, whichever is smaller.
Otherwise, the sort order of queues would be affected.
- areTasksInQueueOverLimit should be changed to something along these lines:
{code}
if (tsi.getMaxTaskLimit() > 0) {
if (tsi.getNumSlotsOccupied() >= tsi.getCapacity()) {
return true;
}
}
if (tsi.getMaxCapacity() > 0) {
if (tsi.getNumSlotsOccupied() >= tsi.getMaxCapacity()) {
return true;
}
}
return false;
{code}
- At the same time, testMaxCapacity should be removed. I would instead
recommend a test case that sets a max capacity on a queue, and checks
scheduling honors the decision.
> Support a hierarchy of queues in the capacity scheduler
> ---
>
> Key: MAPREDUCE-824
> URL: https://issues.apache.org/jira/browse/MAPREDUCE-824
> Project: Hadoop Map/Reduce
> Issue Type: New Feature
> Components: contrib/capacity-sched
>Reporter: Hemanth Yamijala
> Attachments: HADOOP-824-1.patch, HADOOP-824-2.patch,
> HADOOP-824-3.patch
>
>
> Currently in Capacity Scheduler, cluster capacity is divided among the queues
> based on the queue capacity. These queues typically represent an organization
> and the capacity of the queue represents the capacity the organization is
> entitled to. Most organizations are large and need to divide their capacity
> among sub-organizations they have. Or they may want to divide the capacity
> based on a category or type of jobs they run. This JIRA covers the
> requirements and other details to provide the above feature.
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
[jira] Created: (MAPREDUCE-898) Change DistributedCache to use new api.
Change DistributedCache to use new api. --- Key: MAPREDUCE-898 URL: https://issues.apache.org/jira/browse/MAPREDUCE-898 Project: Hadoop Map/Reduce Issue Type: Sub-task Reporter: Amareshwari Sriramadasu -- This message is automatically generated by JIRA. - You can reply to this email to add a comment to the issue online.
[jira] Commented: (MAPREDUCE-430) Task stuck in cleanup with OutOfMemoryErrors
[ https://issues.apache.org/jira/browse/MAPREDUCE-430?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12745824#action_12745824 ] Arun C Murthy commented on MAPREDUCE-430: - Yep, also the worst case is that the TT will kill the stuck JVM after mapred.task.timeout. But, yes it would be good to watch for this. > Task stuck in cleanup with OutOfMemoryErrors > > > Key: MAPREDUCE-430 > URL: https://issues.apache.org/jira/browse/MAPREDUCE-430 > Project: Hadoop Map/Reduce > Issue Type: Bug >Reporter: Amareshwari Sriramadasu >Assignee: Amar Kamat > Fix For: 0.20.1 > > Attachments: MAPREDUCE-430-v1.6-branch-0.20.patch, > MAPREDUCE-430-v1.6.patch, MAPREDUCE-430-v1.7.patch, MAPREDUCE-430-v1.8.patch > > > Obesrved a task with OutOfMemory error, stuck in cleanup. -- This message is automatically generated by JIRA. - You can reply to this email to add a comment to the issue online.
[jira] Commented: (MAPREDUCE-430) Task stuck in cleanup with OutOfMemoryErrors
[ https://issues.apache.org/jira/browse/MAPREDUCE-430?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12745823#action_12745823 ] Devaraj Das commented on MAPREDUCE-430: --- +1. Let's keep a watch on situations where the communication gets stuck due to an OOM.. > Task stuck in cleanup with OutOfMemoryErrors > > > Key: MAPREDUCE-430 > URL: https://issues.apache.org/jira/browse/MAPREDUCE-430 > Project: Hadoop Map/Reduce > Issue Type: Bug >Reporter: Amareshwari Sriramadasu >Assignee: Amar Kamat > Fix For: 0.20.1 > > Attachments: MAPREDUCE-430-v1.6-branch-0.20.patch, > MAPREDUCE-430-v1.6.patch, MAPREDUCE-430-v1.7.patch, MAPREDUCE-430-v1.8.patch > > > Obesrved a task with OutOfMemory error, stuck in cleanup. -- This message is automatically generated by JIRA. - You can reply to this email to add a comment to the issue online.
[jira] Commented: (MAPREDUCE-430) Task stuck in cleanup with OutOfMemoryErrors
[
https://issues.apache.org/jira/browse/MAPREDUCE-430?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12745820#action_12745820
]
Arun C Murthy commented on MAPREDUCE-430:
-
I'm thinking we can do:
{noformat}
try {
// existing stuff
} catch (FSError e) {
// ...
} catch (Throwable t) {
umblical.fatalError();
}
{noformat}
Thus we can allow the Child to exit if it can't do umbilical.fatalError(). I'm
thinking that we'll just do a best-effort to call {umbilical.fatalError}, I
don't think the jvm will get 'stuck' there. Thoughts?
> Task stuck in cleanup with OutOfMemoryErrors
>
>
> Key: MAPREDUCE-430
> URL: https://issues.apache.org/jira/browse/MAPREDUCE-430
> Project: Hadoop Map/Reduce
> Issue Type: Bug
>Reporter: Amareshwari Sriramadasu
>Assignee: Amar Kamat
> Fix For: 0.20.1
>
> Attachments: MAPREDUCE-430-v1.6-branch-0.20.patch,
> MAPREDUCE-430-v1.6.patch, MAPREDUCE-430-v1.7.patch, MAPREDUCE-430-v1.8.patch
>
>
> Obesrved a task with OutOfMemory error, stuck in cleanup.
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
[jira] Commented: (MAPREDUCE-430) Task stuck in cleanup with OutOfMemoryErrors
[
https://issues.apache.org/jira/browse/MAPREDUCE-430?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12745817#action_12745817
]
Devaraj Das commented on MAPREDUCE-430:
---
I am worried that the communication to the TT might get stuck if it was a OOM.
On the other hand, System.exit has a much better probability of making the
process exit. So my thinking for Child.java is:
try {
// do the existing stuff
} catch (OutOfMemoryError) {
System.exit(-1);
} catch (FSError fse) {
// do the existing stuff
} catch (Throwable t ) {
// do the existing stuff
} finally {
// do the existing stuff
}
> Task stuck in cleanup with OutOfMemoryErrors
>
>
> Key: MAPREDUCE-430
> URL: https://issues.apache.org/jira/browse/MAPREDUCE-430
> Project: Hadoop Map/Reduce
> Issue Type: Bug
>Reporter: Amareshwari Sriramadasu
>Assignee: Amar Kamat
> Fix For: 0.20.1
>
> Attachments: MAPREDUCE-430-v1.6-branch-0.20.patch,
> MAPREDUCE-430-v1.6.patch, MAPREDUCE-430-v1.7.patch, MAPREDUCE-430-v1.8.patch
>
>
> Obesrved a task with OutOfMemory error, stuck in cleanup.
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
[jira] Commented: (MAPREDUCE-430) Task stuck in cleanup with OutOfMemoryErrors
[
https://issues.apache.org/jira/browse/MAPREDUCE-430?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12745815#action_12745815
]
Arun C Murthy commented on MAPREDUCE-430:
-
A suggestion for a test-case for this bug:
A job with 1 map and 1 reduce. {map|reduce}_0_0 should throw OOM,
{map|reduce}_0_1 should throw FSError, {map|reduce}_0_2 should throw
IOException and {map|reduce}_0_3 should succeed. We can probably add a check to
ensure that the job completed under a couple of mins or so also.
> Task stuck in cleanup with OutOfMemoryErrors
>
>
> Key: MAPREDUCE-430
> URL: https://issues.apache.org/jira/browse/MAPREDUCE-430
> Project: Hadoop Map/Reduce
> Issue Type: Bug
>Reporter: Amareshwari Sriramadasu
>Assignee: Amar Kamat
> Fix For: 0.20.1
>
> Attachments: MAPREDUCE-430-v1.6-branch-0.20.patch,
> MAPREDUCE-430-v1.6.patch, MAPREDUCE-430-v1.7.patch, MAPREDUCE-430-v1.8.patch
>
>
> Obesrved a task with OutOfMemory error, stuck in cleanup.
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
[jira] Commented: (MAPREDUCE-679) XML-based metrics as JSP servlet for JobTracker
[ https://issues.apache.org/jira/browse/MAPREDUCE-679?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12745806#action_12745806 ] Hadoop QA commented on MAPREDUCE-679: - -1 overall. Here are the results of testing the latest attachment http://issues.apache.org/jira/secure/attachment/12417204/MAPREDUCE-679.3.patch against trunk revision 806408. +1 @author. The patch does not contain any @author tags. +1 tests included. The patch appears to include 2 new or modified tests. +1 javadoc. The javadoc tool did not generate any warning messages. +1 javac. The applied patch does not increase the total number of javac compiler warnings. -1 findbugs. The patch appears to introduce 2 new Findbugs warnings. +1 release audit. The applied patch does not increase the total number of release audit warnings. -1 core tests. The patch failed core unit tests. -1 contrib tests. The patch failed contrib unit tests. Test results: http://hudson.zones.apache.org/hudson/job/Mapreduce-Patch-vesta.apache.org/500/testReport/ Findbugs warnings: http://hudson.zones.apache.org/hudson/job/Mapreduce-Patch-vesta.apache.org/500/artifact/trunk/build/test/findbugs/newPatchFindbugsWarnings.html Checkstyle results: http://hudson.zones.apache.org/hudson/job/Mapreduce-Patch-vesta.apache.org/500/artifact/trunk/build/test/checkstyle-errors.html Console output: http://hudson.zones.apache.org/hudson/job/Mapreduce-Patch-vesta.apache.org/500/console This message is automatically generated. > XML-based metrics as JSP servlet for JobTracker > --- > > Key: MAPREDUCE-679 > URL: https://issues.apache.org/jira/browse/MAPREDUCE-679 > Project: Hadoop Map/Reduce > Issue Type: New Feature > Components: jobtracker >Reporter: Aaron Kimball >Assignee: Aaron Kimball > Attachments: example-jobtracker-completed-job.xml, > example-jobtracker-running-job.xml, MAPREDUCE-679.2.patch, > MAPREDUCE-679.3.patch, MAPREDUCE-679.patch > > > In HADOOP-4559, a general REST API for reporting metrics was proposed but > work seems to have stalled. In the interim, we have a simple XML translation > of the existing JobTracker status page which provides the same metrics > (including the tables of running/completed/failed jobs) as the human-readable > page. This is a relatively lightweight addition to provide some > machine-understandable metrics reporting. -- This message is automatically generated by JIRA. - You can reply to this email to add a comment to the issue online.
[jira] Commented: (MAPREDUCE-370) Change org.apache.hadoop.mapred.lib.MultipleOutputs to use new api.
[ https://issues.apache.org/jira/browse/MAPREDUCE-370?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12745805#action_12745805 ] Amareshwari Sriramadasu commented on MAPREDUCE-370: --- -1 core tests : due to MAPREDUCE-880 -1 contrib tests : due to MAPREDUCE-699 > Change org.apache.hadoop.mapred.lib.MultipleOutputs to use new api. > --- > > Key: MAPREDUCE-370 > URL: https://issues.apache.org/jira/browse/MAPREDUCE-370 > Project: Hadoop Map/Reduce > Issue Type: Sub-task >Reporter: Amareshwari Sriramadasu >Assignee: Amareshwari Sriramadasu > Fix For: 0.21.0 > > Attachments: patch-370-1.txt, patch-370.txt > > -- This message is automatically generated by JIRA. - You can reply to this email to add a comment to the issue online.
[jira] Updated: (MAPREDUCE-834) When TaskTracker config use old memory management values its memory monitoring is diabled.
[ https://issues.apache.org/jira/browse/MAPREDUCE-834?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Hemanth Yamijala updated MAPREDUCE-834: --- Resolution: Fixed Fix Version/s: 0.20.1 Hadoop Flags: [Reviewed] Status: Resolved (was: Patch Available) I just committed this to trunk and branch 0.20. Thanks, Sreekanth ! > When TaskTracker config use old memory management values its memory > monitoring is diabled. > -- > > Key: MAPREDUCE-834 > URL: https://issues.apache.org/jira/browse/MAPREDUCE-834 > Project: Hadoop Map/Reduce > Issue Type: Bug >Reporter: Karam Singh >Assignee: Sreekanth Ramakrishnan > Fix For: 0.20.1 > > Attachments: mapred-834-20.patch, mapreduce-834-1.patch, > mapreduce-834-2.patch, mapreduce-834-3.patch, mapreduce-834-4.patch > > > TaskTracker memory config values -: > mapred.tasktracker.vmem.reserved=8589934592 > mapred.task.default.maxvmem=2147483648 > mapred.task.limit.maxvmem=4294967296 > mapred.tasktracker.pmem.reserved=2147483648 > TaskTracker start as -: >2009-08-05 12:39:03,308 WARN > org.apache.hadoop.mapred.TaskTracker: The variable > mapred.tasktracker.vmem.reserved is no longer used > 2009-08-05 12:39:03,308 WARN > org.apache.hadoop.mapred.TaskTracker: The variable > mapred.tasktracker.pmem.reserved is no longer used > 2009-08-05 12:39:03,308 WARN > org.apache.hadoop.mapred.TaskTracker: The variable > mapred.task.default.maxvmem is no longer used > 2009-08-05 12:39:03,308 WARN > org.apache.hadoop.mapred.TaskTracker: The variable mapred.task.limit.maxvmem > is no longer used > 2009-08-05 12:39:03,308 INFO > org.apache.hadoop.mapred.TaskTracker: Starting thread: Map-events fetcher for > all reduce tasks on > 2009-08-05 12:39:03,309 INFO > org.apache.hadoop.mapred.TaskTracker: Using MemoryCalculatorPlugin : > org.apache.hadoop.util.linuxmemorycalculatorplu...@19be4777 > 2009-08-05 12:39:03,311 WARN > org.apache.hadoop.mapred.TaskTracker: TaskTracker's > totalMemoryAllottedForTasks is -1. TaskMemoryManager is disabled. -- This message is automatically generated by JIRA. - You can reply to this email to add a comment to the issue online.
[jira] Updated: (MAPREDUCE-430) Task stuck in cleanup with OutOfMemoryErrors
[
https://issues.apache.org/jira/browse/MAPREDUCE-430?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Arun C Murthy updated MAPREDUCE-430:
Status: Open (was: Patch Available)
I had a chat with Amar to explain my take on this:
# {Map|Reduce}Task shouldn't catch Throwable except for the parts where they
deal with threads e.g. copier threads or merge threads in which case they
should use a Throwable variable to save the error and check for it.
# Child should catch Throwable, inform the TaskTracker via
TaskUmbilicalProtocal.fatalError or such and then exit.
Thoughts?
> Task stuck in cleanup with OutOfMemoryErrors
>
>
> Key: MAPREDUCE-430
> URL: https://issues.apache.org/jira/browse/MAPREDUCE-430
> Project: Hadoop Map/Reduce
> Issue Type: Bug
>Reporter: Amareshwari Sriramadasu
>Assignee: Amar Kamat
> Fix For: 0.20.1
>
> Attachments: MAPREDUCE-430-v1.6-branch-0.20.patch,
> MAPREDUCE-430-v1.6.patch, MAPREDUCE-430-v1.7.patch, MAPREDUCE-430-v1.8.patch
>
>
> Obesrved a task with OutOfMemory error, stuck in cleanup.
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
[jira] Resolved: (MAPREDUCE-745) TestRecoveryManager fails sometimes
[ https://issues.apache.org/jira/browse/MAPREDUCE-745?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Devaraj Das resolved MAPREDUCE-745. --- Resolution: Fixed Fix Version/s: (was: 0.21.0) 0.20.1 Assignee: Amar Kamat Hadoop Flags: [Reviewed] I committed this. Thanks, Amar! > TestRecoveryManager fails sometimes > --- > > Key: MAPREDUCE-745 > URL: https://issues.apache.org/jira/browse/MAPREDUCE-745 > Project: Hadoop Map/Reduce > Issue Type: Bug > Components: jobtracker >Reporter: Amareshwari Sriramadasu >Assignee: Amar Kamat > Fix For: 0.20.1 > > Attachments: MAPREDUCE-745-v1.0.patch, MAPREDUCE-745-v1.2.patch, > MAPREDUCE-745-v1.3.patch, MAPREDUCE-745-v1.7.patch, > MAPREDUCE-745-v1.8-branch-0.20.patch, MAPREDUCE-745-v1.8.1-branch-0.20.patch, > MAPREDUCE-745-v1.8.patch, > TEST-org.apache.hadoop.mapred.TestRecoveryManager.txt, > TEST-org.apache.hadoop.mapred.TestRecoveryManager.txt, > TEST-org.apache.hadoop.mapred.TestRecoveryManager.txt > > -- This message is automatically generated by JIRA. - You can reply to this email to add a comment to the issue online.
[jira] Commented: (MAPREDUCE-745) TestRecoveryManager fails sometimes
[ https://issues.apache.org/jira/browse/MAPREDUCE-745?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12745793#action_12745793 ] Devaraj Das commented on MAPREDUCE-745: --- Yesterday, there was a problem with the machine on the Apache that I use for commits. Wasn't sure whether it got committed. But it actually did get committed. Thanks, Amar! > TestRecoveryManager fails sometimes > --- > > Key: MAPREDUCE-745 > URL: https://issues.apache.org/jira/browse/MAPREDUCE-745 > Project: Hadoop Map/Reduce > Issue Type: Bug > Components: jobtracker >Reporter: Amareshwari Sriramadasu > Fix For: 0.21.0 > > Attachments: MAPREDUCE-745-v1.0.patch, MAPREDUCE-745-v1.2.patch, > MAPREDUCE-745-v1.3.patch, MAPREDUCE-745-v1.7.patch, > MAPREDUCE-745-v1.8-branch-0.20.patch, MAPREDUCE-745-v1.8.1-branch-0.20.patch, > MAPREDUCE-745-v1.8.patch, > TEST-org.apache.hadoop.mapred.TestRecoveryManager.txt, > TEST-org.apache.hadoop.mapred.TestRecoveryManager.txt, > TEST-org.apache.hadoop.mapred.TestRecoveryManager.txt > > -- This message is automatically generated by JIRA. - You can reply to this email to add a comment to the issue online.
[jira] Commented: (MAPREDUCE-896) Users can set non-writable permissions on temporary files for TT and can abuse disk usage.
[
https://issues.apache.org/jira/browse/MAPREDUCE-896?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12745787#action_12745787
]
Vinod K V commented on MAPREDUCE-896:
-
To quote an example, TT cannot properly cleanup the following dirs.
{code}
$mapred.local.dir
`-- taskTracker
`-- jobcache
`--- $jobid
`--- work
`-- unwritable-dir dr--r--r--
` user-file -r--r--r--
{code}
> Users can set non-writable permissions on temporary files for TT and can
> abuse disk usage.
> --
>
> Key: MAPREDUCE-896
> URL: https://issues.apache.org/jira/browse/MAPREDUCE-896
> Project: Hadoop Map/Reduce
> Issue Type: Bug
> Components: tasktracker
>Reporter: Vinod K V
>
> As of now, irrespective of the TaskController in use, TT itself does a full
> delete on local files created by itself or job tasks. This step, depending
> upon TT's umask and the permissions set by files by the user, for e.g in
> job-work/task-work or child.tmp directories, may or may not go through
> successful completion fully. Thus is left an opportunity for abusing disk
> space usage either accidentally or intentionally by TT/users.
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
[jira] Commented: (MAPREDUCE-896) Users can set non-writable permissions on temporary files for TT and can abuse disk usage.
[ https://issues.apache.org/jira/browse/MAPREDUCE-896?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12745788#action_12745788 ] Vinod K V commented on MAPREDUCE-896: - This problem gets aggravated when LinuxTaskController is in use. User can simply set group ownership of files that he/she creates to himself thereby preventing TaskTracker to clean up those files. By default, all directories have setgid bit set and so files/dirs are cleanable by the TT. > Users can set non-writable permissions on temporary files for TT and can > abuse disk usage. > -- > > Key: MAPREDUCE-896 > URL: https://issues.apache.org/jira/browse/MAPREDUCE-896 > Project: Hadoop Map/Reduce > Issue Type: Bug > Components: tasktracker >Reporter: Vinod K V > > As of now, irrespective of the TaskController in use, TT itself does a full > delete on local files created by itself or job tasks. This step, depending > upon TT's umask and the permissions set by files by the user, for e.g in > job-work/task-work or child.tmp directories, may or may not go through > successful completion fully. Thus is left an opportunity for abusing disk > space usage either accidentally or intentionally by TT/users. -- This message is automatically generated by JIRA. - You can reply to this email to add a comment to the issue online.
[jira] Created: (MAPREDUCE-897) Provide information captured as part of JobTrackerStatistics via the Hadoop metrics API
Provide information captured as part of JobTrackerStatistics via the Hadoop metrics API --- Key: MAPREDUCE-897 URL: https://issues.apache.org/jira/browse/MAPREDUCE-897 Project: Hadoop Map/Reduce Issue Type: Improvement Components: tasktracker Reporter: Hemanth Yamijala MAPREDUCE-467 introduced a framework to collect statistics per node on a fixed set of intervals. Presently there is support for collecting statistics related to number of task failures and also health check script failures per hour, day and since start of system. It is felt that this information can be made available via the tasktracker's metrics system as well. -- This message is automatically generated by JIRA. - You can reply to this email to add a comment to the issue online.
[jira] Created: (MAPREDUCE-896) Users can set non-writable permissions on temporary files for TT and can abuse disk usage.
Users can set non-writable permissions on temporary files for TT and can abuse disk usage. -- Key: MAPREDUCE-896 URL: https://issues.apache.org/jira/browse/MAPREDUCE-896 Project: Hadoop Map/Reduce Issue Type: Bug Components: tasktracker Reporter: Vinod K V As of now, irrespective of the TaskController in use, TT itself does a full delete on local files created by itself or job tasks. This step, depending upon TT's umask and the permissions set by files by the user, for e.g in job-work/task-work or child.tmp directories, may or may not go through successful completion fully. Thus is left an opportunity for abusing disk space usage either accidentally or intentionally by TT/users. -- This message is automatically generated by JIRA. - You can reply to this email to add a comment to the issue online.
[jira] Commented: (MAPREDUCE-890) After HADOOP-4491, the user who started mapred system is not able to run job.
[
https://issues.apache.org/jira/browse/MAPREDUCE-890?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12745784#action_12745784
]
Vinod K V commented on MAPREDUCE-890:
-
In other words, the problem is the following:
{code}
$ mkdir testing
$ ls -ld testing/
drwxr-xr-x 2 vinodkv vinodkv 4096 2009-08-21 09:54 testing/
$ touch testing/t1.txt
$ echo $?
0
$ chmod 0570 testing/
$ ls -ld testing/
dr-xrwx--- 2 vinodkv vinodkv 4096 2009-08-21 09:54 testing/
$ touch testing/t2.txt
touch: cannot touch `testing/t2.txt': Permission denied
{code}
I searched and hunted this down in the Linux Kernel filesystem code:
http://git.kernel.org/?p=linux/kernel/git/torvalds/linux-2.6.git;a=blob;f=fs/namei.c;h=f3c5b278895a0d3e0f23fe6fd474e2728a1c6cb6;hb=6c30c53fd5ae6a99a23ad78e90c428d2c8ffb07f
. Particularly the generic_permission() function where we check if
current_fsuid() == inode->i_uid.
> After HADOOP-4491, the user who started mapred system is not able to run job.
> -
>
> Key: MAPREDUCE-890
> URL: https://issues.apache.org/jira/browse/MAPREDUCE-890
> Project: Hadoop Map/Reduce
> Issue Type: Bug
> Components: tasktracker
>Reporter: Karam Singh
>Assignee: Vinod K V
>
> Even setup and cleanup task of job fails due exception -: It fails to create
> job and related directories under mapred.local.dir/taskTracker/jobcache
> Directories are created as -:
> [dr-xrws--- mapred hadoop ] job_200908190916_0002
> mapred is not wrtie under this. Even manually I failed to touch file.
> mapred is use of started mr cluster
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
[jira] Commented: (MAPREDUCE-323) Improve the way job history files are managed
[ https://issues.apache.org/jira/browse/MAPREDUCE-323?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12745785#action_12745785 ] Rajiv Chittajallu commented on MAPREDUCE-323: - +1 for Tim Williamson suggestion. I would also request for removing jobname from the history filename. > Improve the way job history files are managed > - > > Key: MAPREDUCE-323 > URL: https://issues.apache.org/jira/browse/MAPREDUCE-323 > Project: Hadoop Map/Reduce > Issue Type: Improvement >Reporter: Amar Kamat >Assignee: Amar Kamat > > Today all the jobhistory files are dumped in one _job-history_ folder. This > can cause problems when there is a need to search the history folder > (job-recovery etc). It would be nice if we group all the jobs under a _user_ > folder. So all the jobs for user _amar_ will go in _history-folder/amar/_. > Jobs can be categorized using various features like _jobid, date, jobname_ > etc but using _username_ will make the search much more efficient and also > will not result into namespace explosion. -- This message is automatically generated by JIRA. - You can reply to this email to add a comment to the issue online.
[jira] Commented: (MAPREDUCE-890) After HADOOP-4491, the user who started mapred system is not able to run job.
[ https://issues.apache.org/jira/browse/MAPREDUCE-890?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12745783#action_12745783 ] Vinod K V commented on MAPREDUCE-890: - As part of MAPREDUCE-842, job directories are set to have the following permissions: ||permissions||user ownership||group ownership||file/dir name|| |dr-xrws--- |$job-owner| $tt_group |job_200908190916_0002| CASE I: the $job-owner is other than the $tt_user $tt_user is part of $tt_group, and so can create attempt directories inside the job directory as part of the task-localization because job_directory is group writable. CASE II: the $job-owner is same as the $tt_user TT cannot create attempt directories inside the job directory!! Because Linux seems to check uid of the process with the fsuid of the directory and return error if the directory is not user writable! CASE II is what is causing the current bug. Alternative solutions at hand: - Leave the code as is, and live with the fact that user's cannot submit jobs as mapred user - Set drwxrws--- on $job-dir in all cases. This means user's tasks CAN potentially create unwarranted files/dirs in the $job_dir - Set drwxrws-- if the $job-owner is same as the $tt_user, set dr-xrws--- otherwise. Handles both the cases, but complicates code very slightly. Thoughts? > After HADOOP-4491, the user who started mapred system is not able to run job. > - > > Key: MAPREDUCE-890 > URL: https://issues.apache.org/jira/browse/MAPREDUCE-890 > Project: Hadoop Map/Reduce > Issue Type: Bug > Components: tasktracker >Reporter: Karam Singh > > Even setup and cleanup task of job fails due exception -: It fails to create > job and related directories under mapred.local.dir/taskTracker/jobcache > Directories are created as -: > [dr-xrws--- mapred hadoop ] job_200908190916_0002 > mapred is not wrtie under this. Even manually I failed to touch file. > mapred is use of started mr cluster -- This message is automatically generated by JIRA. - You can reply to this email to add a comment to the issue online.
[jira] Assigned: (MAPREDUCE-890) After HADOOP-4491, the user who started mapred system is not able to run job.
[ https://issues.apache.org/jira/browse/MAPREDUCE-890?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Vinod K V reassigned MAPREDUCE-890: --- Assignee: Vinod K V > After HADOOP-4491, the user who started mapred system is not able to run job. > - > > Key: MAPREDUCE-890 > URL: https://issues.apache.org/jira/browse/MAPREDUCE-890 > Project: Hadoop Map/Reduce > Issue Type: Bug > Components: tasktracker >Reporter: Karam Singh >Assignee: Vinod K V > > Even setup and cleanup task of job fails due exception -: It fails to create > job and related directories under mapred.local.dir/taskTracker/jobcache > Directories are created as -: > [dr-xrws--- mapred hadoop ] job_200908190916_0002 > mapred is not wrtie under this. Even manually I failed to touch file. > mapred is use of started mr cluster -- This message is automatically generated by JIRA. - You can reply to this email to add a comment to the issue online.
[jira] Updated: (MAPREDUCE-894) DBInputformat not working with SQLServer
[ https://issues.apache.org/jira/browse/MAPREDUCE-894?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Budianto Lie updated MAPREDUCE-894: --- Attachment: MAPREDUCE-894.patch > DBInputformat not working with SQLServer > > > Key: MAPREDUCE-894 > URL: https://issues.apache.org/jira/browse/MAPREDUCE-894 > Project: Hadoop Map/Reduce > Issue Type: Bug >Reporter: Budianto Lie > Attachments: MAPREDUCE-894.patch > > > org.apache.hadoop.mapreduce.lib.db.DBInputFormat > Microsoft SQLServer doesn't support LIMIT and OFFSET. > Fix: > Based on MAPREDUCE-716, I already implemented it. > By creating a new class > org.apache.hadoop.mapreduce.lib.db.MsSqlDBRecordReader > and modifying class org.apache.hadoop.mapreduce.lib.db.DBInputFormat > Note: this fix is working only with SQLServer 2005 or higher. -- This message is automatically generated by JIRA. - You can reply to this email to add a comment to the issue online.
[jira] Updated: (MAPREDUCE-712) RandomTextWriter example is CPU bound
[ https://issues.apache.org/jira/browse/MAPREDUCE-712?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Chris Douglas updated MAPREDUCE-712: Resolution: Fixed Fix Version/s: 0.21.0 Hadoop Flags: [Reviewed] Status: Resolved (was: Patch Available) I committed this. > RandomTextWriter example is CPU bound > - > > Key: MAPREDUCE-712 > URL: https://issues.apache.org/jira/browse/MAPREDUCE-712 > Project: Hadoop Map/Reduce > Issue Type: Improvement > Components: examples >Affects Versions: 0.20.1, 0.21.0 > Environment: ~200 nodes cluster > Each node has the following configuration: > Processors: 2 x Xeon L5420 2.50GHz (8 cores) - Harpertown C0, 64-bit, > quad-core (8 CPUs) > 4 Disks > 16 GB RAM > Linux 2.6 > Hadoop version: trunk >Reporter: Khaled Elmeleegy >Assignee: Chris Douglas > Fix For: 0.21.0 > > Attachments: MR712-0.patch, MR712-1.patch > > > Running the RandomTextWritter example job ( from the examples jar) pegs the > machiens' CPUs. -- This message is automatically generated by JIRA. - You can reply to this email to add a comment to the issue online.
[jira] Commented: (MAPREDUCE-712) RandomTextWriter example is CPU bound
[ https://issues.apache.org/jira/browse/MAPREDUCE-712?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12745762#action_12745762 ] Hong Tang commented on MAPREDUCE-712: - Patch looks good. +1. > RandomTextWriter example is CPU bound > - > > Key: MAPREDUCE-712 > URL: https://issues.apache.org/jira/browse/MAPREDUCE-712 > Project: Hadoop Map/Reduce > Issue Type: Improvement > Components: examples >Affects Versions: 0.20.1, 0.21.0 > Environment: ~200 nodes cluster > Each node has the following configuration: > Processors: 2 x Xeon L5420 2.50GHz (8 cores) - Harpertown C0, 64-bit, > quad-core (8 CPUs) > 4 Disks > 16 GB RAM > Linux 2.6 > Hadoop version: trunk >Reporter: Khaled Elmeleegy >Assignee: Chris Douglas > Attachments: MR712-0.patch, MR712-1.patch > > > Running the RandomTextWritter example job ( from the examples jar) pegs the > machiens' CPUs. -- This message is automatically generated by JIRA. - You can reply to this email to add a comment to the issue online.
[jira] Created: (MAPREDUCE-895) FileSystem::ListStatus will now throw FileNotFoundException, MapRed needs updated
FileSystem::ListStatus will now throw FileNotFoundException, MapRed needs updated - Key: MAPREDUCE-895 URL: https://issues.apache.org/jira/browse/MAPREDUCE-895 Project: Hadoop Map/Reduce Issue Type: Bug Reporter: Jakob Homan Assignee: Jakob Homan HADOOP-6201 (and HDFS-538) determined the semantics of FileSystem::ListStatus is not correct and that the actual file system class vary in their implemenations, with some throwing an exception and some returning null. Fixing this will require adjusting code that calls this method. -- This message is automatically generated by JIRA. - You can reply to this email to add a comment to the issue online.
[jira] Updated: (MAPREDUCE-679) XML-based metrics as JSP servlet for JobTracker
[ https://issues.apache.org/jira/browse/MAPREDUCE-679?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Aaron Kimball updated MAPREDUCE-679: Status: Patch Available (was: Open) > XML-based metrics as JSP servlet for JobTracker > --- > > Key: MAPREDUCE-679 > URL: https://issues.apache.org/jira/browse/MAPREDUCE-679 > Project: Hadoop Map/Reduce > Issue Type: New Feature > Components: jobtracker >Reporter: Aaron Kimball >Assignee: Aaron Kimball > Attachments: example-jobtracker-completed-job.xml, > example-jobtracker-running-job.xml, MAPREDUCE-679.2.patch, > MAPREDUCE-679.3.patch, MAPREDUCE-679.patch > > > In HADOOP-4559, a general REST API for reporting metrics was proposed but > work seems to have stalled. In the interim, we have a simple XML translation > of the existing JobTracker status page which provides the same metrics > (including the tables of running/completed/failed jobs) as the human-readable > page. This is a relatively lightweight addition to provide some > machine-understandable metrics reporting. -- This message is automatically generated by JIRA. - You can reply to this email to add a comment to the issue online.
[jira] Updated: (MAPREDUCE-679) XML-based metrics as JSP servlet for JobTracker
[ https://issues.apache.org/jira/browse/MAPREDUCE-679?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Aaron Kimball updated MAPREDUCE-679: Attachment: example-jobtracker-completed-job.xml > XML-based metrics as JSP servlet for JobTracker > --- > > Key: MAPREDUCE-679 > URL: https://issues.apache.org/jira/browse/MAPREDUCE-679 > Project: Hadoop Map/Reduce > Issue Type: New Feature > Components: jobtracker >Reporter: Aaron Kimball >Assignee: Aaron Kimball > Attachments: example-jobtracker-completed-job.xml, > example-jobtracker-running-job.xml, MAPREDUCE-679.2.patch, > MAPREDUCE-679.3.patch, MAPREDUCE-679.patch > > > In HADOOP-4559, a general REST API for reporting metrics was proposed but > work seems to have stalled. In the interim, we have a simple XML translation > of the existing JobTracker status page which provides the same metrics > (including the tables of running/completed/failed jobs) as the human-readable > page. This is a relatively lightweight addition to provide some > machine-understandable metrics reporting. -- This message is automatically generated by JIRA. - You can reply to this email to add a comment to the issue online.
[jira] Updated: (MAPREDUCE-679) XML-based metrics as JSP servlet for JobTracker
[ https://issues.apache.org/jira/browse/MAPREDUCE-679?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Aaron Kimball updated MAPREDUCE-679: Attachment: example-jobtracker-running-job.xml Attaching example xml outputs > XML-based metrics as JSP servlet for JobTracker > --- > > Key: MAPREDUCE-679 > URL: https://issues.apache.org/jira/browse/MAPREDUCE-679 > Project: Hadoop Map/Reduce > Issue Type: New Feature > Components: jobtracker >Reporter: Aaron Kimball >Assignee: Aaron Kimball > Attachments: example-jobtracker-completed-job.xml, > example-jobtracker-running-job.xml, MAPREDUCE-679.2.patch, > MAPREDUCE-679.3.patch, MAPREDUCE-679.patch > > > In HADOOP-4559, a general REST API for reporting metrics was proposed but > work seems to have stalled. In the interim, we have a simple XML translation > of the existing JobTracker status page which provides the same metrics > (including the tables of running/completed/failed jobs) as the human-readable > page. This is a relatively lightweight addition to provide some > machine-understandable metrics reporting. -- This message is automatically generated by JIRA. - You can reply to this email to add a comment to the issue online.
[jira] Updated: (MAPREDUCE-679) XML-based metrics as JSP servlet for JobTracker
[ https://issues.apache.org/jira/browse/MAPREDUCE-679?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Aaron Kimball updated MAPREDUCE-679: Status: Open (was: Patch Available) > XML-based metrics as JSP servlet for JobTracker > --- > > Key: MAPREDUCE-679 > URL: https://issues.apache.org/jira/browse/MAPREDUCE-679 > Project: Hadoop Map/Reduce > Issue Type: New Feature > Components: jobtracker >Reporter: Aaron Kimball >Assignee: Aaron Kimball > Attachments: example-jobtracker-completed-job.xml, > example-jobtracker-running-job.xml, MAPREDUCE-679.2.patch, > MAPREDUCE-679.3.patch, MAPREDUCE-679.patch > > > In HADOOP-4559, a general REST API for reporting metrics was proposed but > work seems to have stalled. In the interim, we have a simple XML translation > of the existing JobTracker status page which provides the same metrics > (including the tables of running/completed/failed jobs) as the human-readable > page. This is a relatively lightweight addition to provide some > machine-understandable metrics reporting. -- This message is automatically generated by JIRA. - You can reply to this email to add a comment to the issue online.
[jira] Updated: (MAPREDUCE-679) XML-based metrics as JSP servlet for JobTracker
[ https://issues.apache.org/jira/browse/MAPREDUCE-679?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Aaron Kimball updated MAPREDUCE-679: Attachment: MAPREDUCE-679.3.patch Attaching new patch * moves /jobtracker.xml.jsp to /jobtracker.jspx * updates test to go along with it * updates build.xml to fix bug in webapp compilation. > XML-based metrics as JSP servlet for JobTracker > --- > > Key: MAPREDUCE-679 > URL: https://issues.apache.org/jira/browse/MAPREDUCE-679 > Project: Hadoop Map/Reduce > Issue Type: New Feature > Components: jobtracker >Reporter: Aaron Kimball >Assignee: Aaron Kimball > Attachments: example-jobtracker-completed-job.xml, > example-jobtracker-running-job.xml, MAPREDUCE-679.2.patch, > MAPREDUCE-679.3.patch, MAPREDUCE-679.patch > > > In HADOOP-4559, a general REST API for reporting metrics was proposed but > work seems to have stalled. In the interim, we have a simple XML translation > of the existing JobTracker status page which provides the same metrics > (including the tables of running/completed/failed jobs) as the human-readable > page. This is a relatively lightweight addition to provide some > machine-understandable metrics reporting. -- This message is automatically generated by JIRA. - You can reply to this email to add a comment to the issue online.
[jira] Commented: (MAPREDUCE-870) Clean up the job Retire code
[ https://issues.apache.org/jira/browse/MAPREDUCE-870?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12745736#action_12745736 ] Hadoop QA commented on MAPREDUCE-870: - -1 overall. Here are the results of testing the latest attachment http://issues.apache.org/jira/secure/attachment/12417097/870_v3.patch against trunk revision 806288. +1 @author. The patch does not contain any @author tags. +1 tests included. The patch appears to include 13 new or modified tests. +1 javadoc. The javadoc tool did not generate any warning messages. +1 javac. The applied patch does not increase the total number of javac compiler warnings. +1 findbugs. The patch does not introduce any new Findbugs warnings. +1 release audit. The applied patch does not increase the total number of release audit warnings. -1 core tests. The patch failed core unit tests. -1 contrib tests. The patch failed contrib unit tests. Test results: http://hudson.zones.apache.org/hudson/job/Mapreduce-Patch-vesta.apache.org/499/testReport/ Findbugs warnings: http://hudson.zones.apache.org/hudson/job/Mapreduce-Patch-vesta.apache.org/499/artifact/trunk/build/test/findbugs/newPatchFindbugsWarnings.html Checkstyle results: http://hudson.zones.apache.org/hudson/job/Mapreduce-Patch-vesta.apache.org/499/artifact/trunk/build/test/checkstyle-errors.html Console output: http://hudson.zones.apache.org/hudson/job/Mapreduce-Patch-vesta.apache.org/499/console This message is automatically generated. > Clean up the job Retire code > > > Key: MAPREDUCE-870 > URL: https://issues.apache.org/jira/browse/MAPREDUCE-870 > Project: Hadoop Map/Reduce > Issue Type: Sub-task >Reporter: Sharad Agarwal >Assignee: Sharad Agarwal > Attachments: 870_v1.patch, 870_v2.patch, 870_v3.patch > > > Currently completed job's full data structures are kept in memory based on > mapred.jobtracker.completeuserjobs.maximum, > mapred.jobtracker.retirejob.interval.min, > mapred.jobtracker.retirejob.interval and mapred.jobtracker.retirejob.check > settings. These controls are not much useful now since MAPREDUCE-817 > introduced a cache for keeping just the very basic info of the completed job. > These settings should be removed and the job should be purged as soon as the > history files are available in HDFS. > Going forward, clients can read the history files if they need to drill down > into more information (MAPREDUCE-864). -- This message is automatically generated by JIRA. - You can reply to this email to add a comment to the issue online.
[jira] Commented: (MAPREDUCE-871) Job/Task local files have incorrect group ownership set by LinuxTaskController binary
[ https://issues.apache.org/jira/browse/MAPREDUCE-871?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12745637#action_12745637 ] Hadoop QA commented on MAPREDUCE-871: - -1 overall. Here are the results of testing the latest attachment http://issues.apache.org/jira/secure/attachment/12417121/MAPREDUCE-871-20090820.1.txt against trunk revision 806152. +1 @author. The patch does not contain any @author tags. +1 tests included. The patch appears to include 6 new or modified tests. +1 javadoc. The javadoc tool did not generate any warning messages. +1 javac. The applied patch does not increase the total number of javac compiler warnings. +1 findbugs. The patch does not introduce any new Findbugs warnings. +1 release audit. The applied patch does not increase the total number of release audit warnings. -1 core tests. The patch failed core unit tests. -1 contrib tests. The patch failed contrib unit tests. Test results: http://hudson.zones.apache.org/hudson/job/Mapreduce-Patch-vesta.apache.org/498/testReport/ Findbugs warnings: http://hudson.zones.apache.org/hudson/job/Mapreduce-Patch-vesta.apache.org/498/artifact/trunk/build/test/findbugs/newPatchFindbugsWarnings.html Checkstyle results: http://hudson.zones.apache.org/hudson/job/Mapreduce-Patch-vesta.apache.org/498/artifact/trunk/build/test/checkstyle-errors.html Console output: http://hudson.zones.apache.org/hudson/job/Mapreduce-Patch-vesta.apache.org/498/console This message is automatically generated. > Job/Task local files have incorrect group ownership set by > LinuxTaskController binary > - > > Key: MAPREDUCE-871 > URL: https://issues.apache.org/jira/browse/MAPREDUCE-871 > Project: Hadoop Map/Reduce > Issue Type: Sub-task > Components: tasktracker >Reporter: Vinod K V >Assignee: Vinod K V > Attachments: MAPREDUCE-871-20090820.1.txt > > > HADOOP-4491 fixed the secure permissions of local files on a TT. While > testing HADOOP-4491 on a cluster, [~karams] found out a bug. All the > files/dirs have should be owned by the group corresponding to the group owner > of the task-controller binary (via using getegid()) which in turn is a > special group to which *only* TT user belongs. HADOOP-4491 incorrectly set it > to primary group of the TT via getgid(), and not the special group. -- This message is automatically generated by JIRA. - You can reply to this email to add a comment to the issue online.
[jira] Updated: (MAPREDUCE-767) to remove mapreduce dependency on commons-cli2
[
https://issues.apache.org/jira/browse/MAPREDUCE-767?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Owen O'Malley updated MAPREDUCE-767:
Affects Version/s: 0.20.1
Fix Version/s: 0.20.1
+1
I've committed this to trunk with the slight change that string.split("=",2)
was missing the "2".
We need this patch for 0.20.1. Can you make one please?
> to remove mapreduce dependency on commons-cli2
> --
>
> Key: MAPREDUCE-767
> URL: https://issues.apache.org/jira/browse/MAPREDUCE-767
> Project: Hadoop Map/Reduce
> Issue Type: Improvement
> Components: contrib/streaming
>Affects Versions: 0.20.1
>Reporter: Giridharan Kesavan
>Assignee: Amar Kamat
> Fix For: 0.20.1
>
> Attachments: MAPREDUCE-767-v1.1.patch, MAPREDUCE-767-v1.2.patch
>
>
> mapreduce, streaming and eclipse plugin depends on common-cli2
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
[jira] Commented: (MAPREDUCE-892) command line tool to list all tasktrackers and their status
[ https://issues.apache.org/jira/browse/MAPREDUCE-892?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12745542#action_12745542 ] Dmytro Molkov commented on MAPREDUCE-892: - Currently some classes that hold the information relevant to this report: TaskTrackerStatus, JobInProgress are marked as package private, which restricts me from using them directly in the MRAdmin class. Two ways to solve it might be introducing public interfaces that expose information useful in reports or making the classes public. > command line tool to list all tasktrackers and their status > --- > > Key: MAPREDUCE-892 > URL: https://issues.apache.org/jira/browse/MAPREDUCE-892 > Project: Hadoop Map/Reduce > Issue Type: Improvement >Reporter: dhruba borthakur >Assignee: Dmytro Molkov > > The "hadoop mradmin -report" could list all the tasktrackers that the > JobTracker knows about. It will also list a brief status summary for each of > the TaskTracker. (This is similar to the hadop dfsadmin -report command that > lists all Datanodes) -- This message is automatically generated by JIRA. - You can reply to this email to add a comment to the issue online.
[jira] Commented: (MAPREDUCE-767) to remove mapreduce dependency on commons-cli2
[ https://issues.apache.org/jira/browse/MAPREDUCE-767?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12745487#action_12745487 ] Amar Kamat commented on MAPREDUCE-767: -- >From the documentation of [cli2|http://commons.apache.org/sandbox/cli2/] and >[cli1.2|http://commons.apache.org/cli/release_1_2.html], its clear that cli1.2 >doesnt support >[Validators|http://commons.apache.org/sandbox/cli2/manual/validators.html] and >[PropertyOption|http://commons.apache.org/sandbox/cli2/manual/options.html]. >But from my manual testing it seems like we can have a workaround for that. >Options like _-jobconf_ and _-cmdenv_ extend PropertyOption. Also _-file_ >option uses Validators which can be checked once the option is parsed. I didnt >see any difference in streaming command-line parsing with the patch. > to remove mapreduce dependency on commons-cli2 > -- > > Key: MAPREDUCE-767 > URL: https://issues.apache.org/jira/browse/MAPREDUCE-767 > Project: Hadoop Map/Reduce > Issue Type: Improvement > Components: contrib/streaming >Reporter: Giridharan Kesavan >Assignee: Amar Kamat > Attachments: MAPREDUCE-767-v1.1.patch, MAPREDUCE-767-v1.2.patch > > > mapreduce, streaming and eclipse plugin depends on common-cli2 -- This message is automatically generated by JIRA. - You can reply to this email to add a comment to the issue online.
[jira] Updated: (MAPREDUCE-318) Refactor reduce shuffle code
[ https://issues.apache.org/jira/browse/MAPREDUCE-318?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Jothi Padmanabhan updated MAPREDUCE-318: Attachment: mapred-318-20Aug.patch Some more modifications to the previous patch # The shuffle status on the web UI -- the number of maps being copied and the bandwidth -- is now present. This will make this similar to the current trunk status # Modified the condition for triggering on disk merge as numfies > (2*iosortfactor - 1), similar to the current trunk code. This ensures we do merges a little less agressively # Modified to trigger an memory merge on stall. We cannot pull the trigger only on the cross of memory threshold, that has a possibility of hang when several fetchers just return if there is not sufficient memory for the current map output, but the total memory used has not crossed the threshold. > Refactor reduce shuffle code > > > Key: MAPREDUCE-318 > URL: https://issues.apache.org/jira/browse/MAPREDUCE-318 > Project: Hadoop Map/Reduce > Issue Type: Improvement >Reporter: Owen O'Malley >Assignee: Owen O'Malley > Attachments: HADOOP-5233_api.patch, HADOOP-5233_part0.patch, > mapred-318-14Aug.patch, mapred-318-20Aug.patch, mapred-318-common.patch > > > The reduce shuffle code has become very complex and entangled. I think we > should move it out of ReduceTask and into a separate package > (org.apache.hadoop.mapred.task.reduce). Details to follow. -- This message is automatically generated by JIRA. - You can reply to this email to add a comment to the issue online.
[jira] Updated: (MAPREDUCE-873) Simplify Job Recovery
[ https://issues.apache.org/jira/browse/MAPREDUCE-873?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Sharad Agarwal updated MAPREDUCE-873: - Attachment: 873_v1.patch Early patch. Testing in progress. It: - removes the old recovery logic. - recovery is done by submitting the jobIds from the mapred.system dir to Jobtracker#submitJob > Simplify Job Recovery > - > > Key: MAPREDUCE-873 > URL: https://issues.apache.org/jira/browse/MAPREDUCE-873 > Project: Hadoop Map/Reduce > Issue Type: Improvement > Components: jobtracker >Affects Versions: 0.20.1 >Reporter: Devaraj Das >Assignee: Sharad Agarwal > Fix For: 0.21.0 > > Attachments: 873_v1.patch > > > On a couple of occasions we have seen the JobTracker not being able to handle > job recovery well, and leading to cluster downtime after a restart. The > current design for handling job recovery is complex and prone to corner cases > not being handled well enough. In retrospect, it seems like the transaction > log based approach as was proposed on HADOOP-3245 > (http://tinyurl.com/luh9hb), would have been a better/simpler model. However, > that is a big project, and it seems for the medium term, just handling job > re-submissions after a restart is a good tradeoff. That is, the JobTracker > after getting restarted, will resubmit all jobs that were running in its past > life. They will all start from the beginning (downside is completed tasks > will reexecute). In the long term, the transaction log model or some variant > of that should be pursued. > Thoughts/comments welcome. -- This message is automatically generated by JIRA. - You can reply to this email to add a comment to the issue online.
[jira] Commented: (MAPREDUCE-430) Task stuck in cleanup with OutOfMemoryErrors
[ https://issues.apache.org/jira/browse/MAPREDUCE-430?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12745463#action_12745463 ] Amar Kamat commented on MAPREDUCE-430: -- Spoke to Devaraj about this offline. So child can ignore Errors and catch only Exceptions and FSError. But ReduceTask and MapTask still catch Throwable and eat it up. This can be serious and hence we need to take care of this. There are multiple options - dont catch Throwable only catch Exceptions - catch Throwable and exit the jvm - catch OOM and exit the jvm. > Task stuck in cleanup with OutOfMemoryErrors > > > Key: MAPREDUCE-430 > URL: https://issues.apache.org/jira/browse/MAPREDUCE-430 > Project: Hadoop Map/Reduce > Issue Type: Bug >Reporter: Amareshwari Sriramadasu >Assignee: Amar Kamat > Fix For: 0.20.1 > > Attachments: MAPREDUCE-430-v1.6-branch-0.20.patch, > MAPREDUCE-430-v1.6.patch, MAPREDUCE-430-v1.7.patch, MAPREDUCE-430-v1.8.patch > > > Obesrved a task with OutOfMemory error, stuck in cleanup. -- This message is automatically generated by JIRA. - You can reply to this email to add a comment to the issue online.
[jira] Commented: (MAPREDUCE-370) Change org.apache.hadoop.mapred.lib.MultipleOutputs to use new api.
[ https://issues.apache.org/jira/browse/MAPREDUCE-370?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12745440#action_12745440 ] Hadoop QA commented on MAPREDUCE-370: - -1 overall. Here are the results of testing the latest attachment http://issues.apache.org/jira/secure/attachment/12417117/patch-370-1.txt against trunk revision 806065. +1 @author. The patch does not contain any @author tags. +1 tests included. The patch appears to include 3 new or modified tests. +1 javadoc. The javadoc tool did not generate any warning messages. +1 javac. The applied patch does not increase the total number of javac compiler warnings. +1 findbugs. The patch does not introduce any new Findbugs warnings. +1 release audit. The applied patch does not increase the total number of release audit warnings. -1 core tests. The patch failed core unit tests. -1 contrib tests. The patch failed contrib unit tests. Test results: http://hudson.zones.apache.org/hudson/job/Mapreduce-Patch-vesta.apache.org/497/testReport/ Findbugs warnings: http://hudson.zones.apache.org/hudson/job/Mapreduce-Patch-vesta.apache.org/497/artifact/trunk/build/test/findbugs/newPatchFindbugsWarnings.html Checkstyle results: http://hudson.zones.apache.org/hudson/job/Mapreduce-Patch-vesta.apache.org/497/artifact/trunk/build/test/checkstyle-errors.html Console output: http://hudson.zones.apache.org/hudson/job/Mapreduce-Patch-vesta.apache.org/497/console This message is automatically generated. > Change org.apache.hadoop.mapred.lib.MultipleOutputs to use new api. > --- > > Key: MAPREDUCE-370 > URL: https://issues.apache.org/jira/browse/MAPREDUCE-370 > Project: Hadoop Map/Reduce > Issue Type: Sub-task >Reporter: Amareshwari Sriramadasu >Assignee: Amareshwari Sriramadasu > Fix For: 0.21.0 > > Attachments: patch-370-1.txt, patch-370.txt > > -- This message is automatically generated by JIRA. - You can reply to this email to add a comment to the issue online.
[jira] Updated: (MAPREDUCE-745) TestRecoveryManager fails sometimes
[ https://issues.apache.org/jira/browse/MAPREDUCE-745?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Amar Kamat updated MAPREDUCE-745: - Attachment: MAPREDUCE-745-v1.8.1-branch-0.20.patch Patch for branch 0.20 with removed imports. > TestRecoveryManager fails sometimes > --- > > Key: MAPREDUCE-745 > URL: https://issues.apache.org/jira/browse/MAPREDUCE-745 > Project: Hadoop Map/Reduce > Issue Type: Bug > Components: jobtracker >Reporter: Amareshwari Sriramadasu > Fix For: 0.21.0 > > Attachments: MAPREDUCE-745-v1.0.patch, MAPREDUCE-745-v1.2.patch, > MAPREDUCE-745-v1.3.patch, MAPREDUCE-745-v1.7.patch, > MAPREDUCE-745-v1.8-branch-0.20.patch, MAPREDUCE-745-v1.8.1-branch-0.20.patch, > MAPREDUCE-745-v1.8.patch, > TEST-org.apache.hadoop.mapred.TestRecoveryManager.txt, > TEST-org.apache.hadoop.mapred.TestRecoveryManager.txt, > TEST-org.apache.hadoop.mapred.TestRecoveryManager.txt > > -- This message is automatically generated by JIRA. - You can reply to this email to add a comment to the issue online.
[jira] Updated: (MAPREDUCE-157) Job History log file format is not friendly for external tools.
[ https://issues.apache.org/jira/browse/MAPREDUCE-157?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Jothi Padmanabhan updated MAPREDUCE-157: Attachment: mapred-157-prelim.patch Preliminary patch as requested by Doug. I got this to produce a history file in Json format (but only after commenting lots of code), so more useful for illustration than otherwise. My guess is that this should be fairly straight forward to port this to use Avro. > Job History log file format is not friendly for external tools. > --- > > Key: MAPREDUCE-157 > URL: https://issues.apache.org/jira/browse/MAPREDUCE-157 > Project: Hadoop Map/Reduce > Issue Type: Sub-task >Reporter: Owen O'Malley >Assignee: Jothi Padmanabhan > Attachments: mapred-157-prelim.patch > > > Currently, parsing the job history logs with external tools is very difficult > because of the format. The most critical problem is that newlines aren't > escaped in the strings. That makes using tools like grep, sed, and awk very > tricky. -- This message is automatically generated by JIRA. - You can reply to this email to add a comment to the issue online.
[jira] Commented: (MAPREDUCE-807) Stray user files in mapred.system.dir with permissions other than 777 can prevent the jobtracker from starting up.
[ https://issues.apache.org/jira/browse/MAPREDUCE-807?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12745399#action_12745399 ] Amar Kamat commented on MAPREDUCE-807: -- All tests except TestRecoveryManager FAILED (timeout) and TestReduceFetch FAILED (timeout) passed. > Stray user files in mapred.system.dir with permissions other than 777 can > prevent the jobtracker from starting up. > -- > > Key: MAPREDUCE-807 > URL: https://issues.apache.org/jira/browse/MAPREDUCE-807 > Project: Hadoop Map/Reduce > Issue Type: Bug > Components: jobtracker >Reporter: Amar Kamat >Assignee: Amar Kamat >Priority: Blocker > Attachments: MAPRED-807-v1.1.patch, MAPRED-807-v1.2.patch, > MAPRED-807-v1.3.patch, MAPRED-807-v1.4.patch > > > With restart disabled, the jobtracker does a _rm -rf_ of the > mapred.system.dir. If the mapred.system.dir contains user files with > permissions other than 777 then the jobtracker gets stuck in a loop trying to > delete the mapred.system.dir (and each time failing with > AccessControlException). -- This message is automatically generated by JIRA. - You can reply to this email to add a comment to the issue online.
[jira] Commented: (MAPREDUCE-27) Jobs with 0 maps will never get removed from the default scheduler
[
https://issues.apache.org/jira/browse/MAPREDUCE-27?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12745398#action_12745398
]
Amar Kamat commented on MAPREDUCE-27:
-
MAPREDUCE-805
> Jobs with 0 maps will never get removed from the default scheduler
> --
>
> Key: MAPREDUCE-27
> URL: https://issues.apache.org/jira/browse/MAPREDUCE-27
> Project: Hadoop Map/Reduce
> Issue Type: Bug
>Reporter: Amar Kamat
>Assignee: Amar Kamat
> Attachments: HADOOP-5049-v1.1.patch
>
>
> Jobs' with 0 maps finish/succeed in the init phase i.e while the job is in
> the _PREP_ state. {{EagerTaskInitializationListener}} removes the job after
> initing but {{JobQueueJobInProgressListener}} waits for a job-state change
> event to be raised and aonly then removes the job from the queue and hence
> the job will stay forever with the {{JobQueueJobInProgressListener}}. Looks
> like {{FairScheduler}} periodically scans the job list and removes completed
> jobs. {{CapacityScheduler}} has a concept of waiting jobs and scans waiting
> queue for completed jobs and purges them.
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
[jira] Updated: (MAPREDUCE-834) When TaskTracker config use old memory management values its memory monitoring is diabled.
[ https://issues.apache.org/jira/browse/MAPREDUCE-834?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Sreekanth Ramakrishnan updated MAPREDUCE-834: - Attachment: mapred-834-20.patch Attaching patch for branch 20. > When TaskTracker config use old memory management values its memory > monitoring is diabled. > -- > > Key: MAPREDUCE-834 > URL: https://issues.apache.org/jira/browse/MAPREDUCE-834 > Project: Hadoop Map/Reduce > Issue Type: Bug >Reporter: Karam Singh >Assignee: Sreekanth Ramakrishnan > Attachments: mapred-834-20.patch, mapreduce-834-1.patch, > mapreduce-834-2.patch, mapreduce-834-3.patch, mapreduce-834-4.patch > > > TaskTracker memory config values -: > mapred.tasktracker.vmem.reserved=8589934592 > mapred.task.default.maxvmem=2147483648 > mapred.task.limit.maxvmem=4294967296 > mapred.tasktracker.pmem.reserved=2147483648 > TaskTracker start as -: >2009-08-05 12:39:03,308 WARN > org.apache.hadoop.mapred.TaskTracker: The variable > mapred.tasktracker.vmem.reserved is no longer used > 2009-08-05 12:39:03,308 WARN > org.apache.hadoop.mapred.TaskTracker: The variable > mapred.tasktracker.pmem.reserved is no longer used > 2009-08-05 12:39:03,308 WARN > org.apache.hadoop.mapred.TaskTracker: The variable > mapred.task.default.maxvmem is no longer used > 2009-08-05 12:39:03,308 WARN > org.apache.hadoop.mapred.TaskTracker: The variable mapred.task.limit.maxvmem > is no longer used > 2009-08-05 12:39:03,308 INFO > org.apache.hadoop.mapred.TaskTracker: Starting thread: Map-events fetcher for > all reduce tasks on > 2009-08-05 12:39:03,309 INFO > org.apache.hadoop.mapred.TaskTracker: Using MemoryCalculatorPlugin : > org.apache.hadoop.util.linuxmemorycalculatorplu...@19be4777 > 2009-08-05 12:39:03,311 WARN > org.apache.hadoop.mapred.TaskTracker: TaskTracker's > totalMemoryAllottedForTasks is -1. TaskMemoryManager is disabled. -- This message is automatically generated by JIRA. - You can reply to this email to add a comment to the issue online.
[jira] Updated: (MAPREDUCE-745) TestRecoveryManager fails sometimes
[ https://issues.apache.org/jira/browse/MAPREDUCE-745?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Amar Kamat updated MAPREDUCE-745: - Attachment: MAPREDUCE-745-v1.8-branch-0.20.patch MAPREDUCE-745-v1.8.patch Attaching patches for trunk and branch-0.20 > TestRecoveryManager fails sometimes > --- > > Key: MAPREDUCE-745 > URL: https://issues.apache.org/jira/browse/MAPREDUCE-745 > Project: Hadoop Map/Reduce > Issue Type: Bug > Components: jobtracker >Reporter: Amareshwari Sriramadasu > Fix For: 0.21.0 > > Attachments: MAPREDUCE-745-v1.0.patch, MAPREDUCE-745-v1.2.patch, > MAPREDUCE-745-v1.3.patch, MAPREDUCE-745-v1.7.patch, > MAPREDUCE-745-v1.8-branch-0.20.patch, MAPREDUCE-745-v1.8.patch, > TEST-org.apache.hadoop.mapred.TestRecoveryManager.txt, > TEST-org.apache.hadoop.mapred.TestRecoveryManager.txt, > TEST-org.apache.hadoop.mapred.TestRecoveryManager.txt > > -- This message is automatically generated by JIRA. - You can reply to this email to add a comment to the issue online.
[jira] Updated: (MAPREDUCE-745) TestRecoveryManager fails sometimes
[ https://issues.apache.org/jira/browse/MAPREDUCE-745?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Amar Kamat updated MAPREDUCE-745: - Attachment: (was: MAPREDUCE-745-v1.8.patch) > TestRecoveryManager fails sometimes > --- > > Key: MAPREDUCE-745 > URL: https://issues.apache.org/jira/browse/MAPREDUCE-745 > Project: Hadoop Map/Reduce > Issue Type: Bug > Components: jobtracker >Reporter: Amareshwari Sriramadasu > Fix For: 0.21.0 > > Attachments: MAPREDUCE-745-v1.0.patch, MAPREDUCE-745-v1.2.patch, > MAPREDUCE-745-v1.3.patch, MAPREDUCE-745-v1.7.patch, > TEST-org.apache.hadoop.mapred.TestRecoveryManager.txt, > TEST-org.apache.hadoop.mapred.TestRecoveryManager.txt, > TEST-org.apache.hadoop.mapred.TestRecoveryManager.txt > > -- This message is automatically generated by JIRA. - You can reply to this email to add a comment to the issue online.
[jira] Commented: (MAPREDUCE-27) Jobs with 0 maps will never get removed from the default scheduler
[
https://issues.apache.org/jira/browse/MAPREDUCE-27?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12745385#action_12745385
]
Vinod K V commented on MAPREDUCE-27:
Duplicated by?
> Jobs with 0 maps will never get removed from the default scheduler
> --
>
> Key: MAPREDUCE-27
> URL: https://issues.apache.org/jira/browse/MAPREDUCE-27
> Project: Hadoop Map/Reduce
> Issue Type: Bug
>Reporter: Amar Kamat
>Assignee: Amar Kamat
> Attachments: HADOOP-5049-v1.1.patch
>
>
> Jobs' with 0 maps finish/succeed in the init phase i.e while the job is in
> the _PREP_ state. {{EagerTaskInitializationListener}} removes the job after
> initing but {{JobQueueJobInProgressListener}} waits for a job-state change
> event to be raised and aonly then removes the job from the queue and hence
> the job will stay forever with the {{JobQueueJobInProgressListener}}. Looks
> like {{FairScheduler}} periodically scans the job list and removes completed
> jobs. {{CapacityScheduler}} has a concept of waiting jobs and scans waiting
> queue for completed jobs and purges them.
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
[jira] Updated: (MAPREDUCE-745) TestRecoveryManager fails sometimes
[ https://issues.apache.org/jira/browse/MAPREDUCE-745?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Amar Kamat updated MAPREDUCE-745: - Attachment: MAPREDUCE-745-v1.8.patch Attaching a patch for trunk with a minor change. Changes to JobTracker.getDateFormat() are reverted. > TestRecoveryManager fails sometimes > --- > > Key: MAPREDUCE-745 > URL: https://issues.apache.org/jira/browse/MAPREDUCE-745 > Project: Hadoop Map/Reduce > Issue Type: Bug > Components: jobtracker >Reporter: Amareshwari Sriramadasu > Fix For: 0.21.0 > > Attachments: MAPREDUCE-745-v1.0.patch, MAPREDUCE-745-v1.2.patch, > MAPREDUCE-745-v1.3.patch, MAPREDUCE-745-v1.7.patch, MAPREDUCE-745-v1.8.patch, > TEST-org.apache.hadoop.mapred.TestRecoveryManager.txt, > TEST-org.apache.hadoop.mapred.TestRecoveryManager.txt, > TEST-org.apache.hadoop.mapred.TestRecoveryManager.txt > > -- This message is automatically generated by JIRA. - You can reply to this email to add a comment to the issue online.
[jira] Commented: (MAPREDUCE-834) When TaskTracker config use old memory management values its memory monitoring is diabled.
[
https://issues.apache.org/jira/browse/MAPREDUCE-834?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12745384#action_12745384
]
Sreekanth Ramakrishnan commented on MAPREDUCE-834:
--
The patch does not change any part on {{JobTracker}} or streaming api's.
The issue with {{TestRecoveryManager}} timing out is reported on MAPREDUCE-880
and the streaming test case failures are also a known reported issue on hudson.
> When TaskTracker config use old memory management values its memory
> monitoring is diabled.
> --
>
> Key: MAPREDUCE-834
> URL: https://issues.apache.org/jira/browse/MAPREDUCE-834
> Project: Hadoop Map/Reduce
> Issue Type: Bug
>Reporter: Karam Singh
>Assignee: Sreekanth Ramakrishnan
> Attachments: mapreduce-834-1.patch, mapreduce-834-2.patch,
> mapreduce-834-3.patch, mapreduce-834-4.patch
>
>
> TaskTracker memory config values -:
> mapred.tasktracker.vmem.reserved=8589934592
> mapred.task.default.maxvmem=2147483648
> mapred.task.limit.maxvmem=4294967296
> mapred.tasktracker.pmem.reserved=2147483648
> TaskTracker start as -:
>2009-08-05 12:39:03,308 WARN
> org.apache.hadoop.mapred.TaskTracker: The variable
> mapred.tasktracker.vmem.reserved is no longer used
> 2009-08-05 12:39:03,308 WARN
> org.apache.hadoop.mapred.TaskTracker: The variable
> mapred.tasktracker.pmem.reserved is no longer used
> 2009-08-05 12:39:03,308 WARN
> org.apache.hadoop.mapred.TaskTracker: The variable
> mapred.task.default.maxvmem is no longer used
> 2009-08-05 12:39:03,308 WARN
> org.apache.hadoop.mapred.TaskTracker: The variable mapred.task.limit.maxvmem
> is no longer used
> 2009-08-05 12:39:03,308 INFO
> org.apache.hadoop.mapred.TaskTracker: Starting thread: Map-events fetcher for
> all reduce tasks on
> 2009-08-05 12:39:03,309 INFO
> org.apache.hadoop.mapred.TaskTracker: Using MemoryCalculatorPlugin :
> org.apache.hadoop.util.linuxmemorycalculatorplu...@19be4777
> 2009-08-05 12:39:03,311 WARN
> org.apache.hadoop.mapred.TaskTracker: TaskTracker's
> totalMemoryAllottedForTasks is -1. TaskMemoryManager is disabled.
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
[jira] Updated: (MAPREDUCE-870) Clean up the job Retire code
[ https://issues.apache.org/jira/browse/MAPREDUCE-870?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Sharad Agarwal updated MAPREDUCE-870: - Status: Patch Available (was: Open) > Clean up the job Retire code > > > Key: MAPREDUCE-870 > URL: https://issues.apache.org/jira/browse/MAPREDUCE-870 > Project: Hadoop Map/Reduce > Issue Type: Sub-task >Reporter: Sharad Agarwal >Assignee: Sharad Agarwal > Attachments: 870_v1.patch, 870_v2.patch, 870_v3.patch > > > Currently completed job's full data structures are kept in memory based on > mapred.jobtracker.completeuserjobs.maximum, > mapred.jobtracker.retirejob.interval.min, > mapred.jobtracker.retirejob.interval and mapred.jobtracker.retirejob.check > settings. These controls are not much useful now since MAPREDUCE-817 > introduced a cache for keeping just the very basic info of the completed job. > These settings should be removed and the job should be purged as soon as the > history files are available in HDFS. > Going forward, clients can read the history files if they need to drill down > into more information (MAPREDUCE-864). -- This message is automatically generated by JIRA. - You can reply to this email to add a comment to the issue online.
[jira] Commented: (MAPREDUCE-834) When TaskTracker config use old memory management values its memory monitoring is diabled.
[ https://issues.apache.org/jira/browse/MAPREDUCE-834?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12745375#action_12745375 ] Hadoop QA commented on MAPREDUCE-834: - -1 overall. Here are the results of testing the latest attachment http://issues.apache.org/jira/secure/attachment/12417108/mapreduce-834-4.patch against trunk revision 806065. +1 @author. The patch does not contain any @author tags. +1 tests included. The patch appears to include 3 new or modified tests. +1 javadoc. The javadoc tool did not generate any warning messages. +1 javac. The applied patch does not increase the total number of javac compiler warnings. +1 findbugs. The patch does not introduce any new Findbugs warnings. +1 release audit. The applied patch does not increase the total number of release audit warnings. -1 core tests. The patch failed core unit tests. -1 contrib tests. The patch failed contrib unit tests. Test results: http://hudson.zones.apache.org/hudson/job/Mapreduce-Patch-vesta.apache.org/496/testReport/ Findbugs warnings: http://hudson.zones.apache.org/hudson/job/Mapreduce-Patch-vesta.apache.org/496/artifact/trunk/build/test/findbugs/newPatchFindbugsWarnings.html Checkstyle results: http://hudson.zones.apache.org/hudson/job/Mapreduce-Patch-vesta.apache.org/496/artifact/trunk/build/test/checkstyle-errors.html Console output: http://hudson.zones.apache.org/hudson/job/Mapreduce-Patch-vesta.apache.org/496/console This message is automatically generated. > When TaskTracker config use old memory management values its memory > monitoring is diabled. > -- > > Key: MAPREDUCE-834 > URL: https://issues.apache.org/jira/browse/MAPREDUCE-834 > Project: Hadoop Map/Reduce > Issue Type: Bug >Reporter: Karam Singh >Assignee: Sreekanth Ramakrishnan > Attachments: mapreduce-834-1.patch, mapreduce-834-2.patch, > mapreduce-834-3.patch, mapreduce-834-4.patch > > > TaskTracker memory config values -: > mapred.tasktracker.vmem.reserved=8589934592 > mapred.task.default.maxvmem=2147483648 > mapred.task.limit.maxvmem=4294967296 > mapred.tasktracker.pmem.reserved=2147483648 > TaskTracker start as -: >2009-08-05 12:39:03,308 WARN > org.apache.hadoop.mapred.TaskTracker: The variable > mapred.tasktracker.vmem.reserved is no longer used > 2009-08-05 12:39:03,308 WARN > org.apache.hadoop.mapred.TaskTracker: The variable > mapred.tasktracker.pmem.reserved is no longer used > 2009-08-05 12:39:03,308 WARN > org.apache.hadoop.mapred.TaskTracker: The variable > mapred.task.default.maxvmem is no longer used > 2009-08-05 12:39:03,308 WARN > org.apache.hadoop.mapred.TaskTracker: The variable mapred.task.limit.maxvmem > is no longer used > 2009-08-05 12:39:03,308 INFO > org.apache.hadoop.mapred.TaskTracker: Starting thread: Map-events fetcher for > all reduce tasks on > 2009-08-05 12:39:03,309 INFO > org.apache.hadoop.mapred.TaskTracker: Using MemoryCalculatorPlugin : > org.apache.hadoop.util.linuxmemorycalculatorplu...@19be4777 > 2009-08-05 12:39:03,311 WARN > org.apache.hadoop.mapred.TaskTracker: TaskTracker's > totalMemoryAllottedForTasks is -1. TaskMemoryManager is disabled. -- This message is automatically generated by JIRA. - You can reply to this email to add a comment to the issue online.
[jira] Updated: (MAPREDUCE-894) DBInputformat not working with SQLServer
[ https://issues.apache.org/jira/browse/MAPREDUCE-894?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Budianto Lie updated MAPREDUCE-894: --- Hadoop Flags: (was: [Reviewed]) > DBInputformat not working with SQLServer > > > Key: MAPREDUCE-894 > URL: https://issues.apache.org/jira/browse/MAPREDUCE-894 > Project: Hadoop Map/Reduce > Issue Type: Bug >Reporter: Budianto Lie > > org.apache.hadoop.mapreduce.lib.db.DBInputFormat > Microsoft SQLServer doesn't support LIMIT and OFFSET. > Fix: > Based on MAPREDUCE-716, I already implemented it. > By creating a new class > org.apache.hadoop.mapreduce.lib.db.MsSqlDBRecordReader > and modifying class org.apache.hadoop.mapreduce.lib.db.DBInputFormat > Note: this fix is working only with SQLServer 2005 or higher. -- This message is automatically generated by JIRA. - You can reply to this email to add a comment to the issue online.
[jira] Updated: (MAPREDUCE-890) After HADOOP-4491, the user who started mapred system is not able to run job.
[ https://issues.apache.org/jira/browse/MAPREDUCE-890?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Vinod K V updated MAPREDUCE-890: Component/s: tasktracker Summary: After HADOOP-4491, the user who started mapred system is not able to run job. (was: Afer 4491, the user who started mapred system is not able to run job.) > After HADOOP-4491, the user who started mapred system is not able to run job. > - > > Key: MAPREDUCE-890 > URL: https://issues.apache.org/jira/browse/MAPREDUCE-890 > Project: Hadoop Map/Reduce > Issue Type: Bug > Components: tasktracker >Reporter: Karam Singh > > Even setup and cleanup task of job fails due exception -: It fails to create > job and related directories under mapred.local.dir/taskTracker/jobcache > Directories are created as -: > [dr-xrws--- mapred hadoop ] job_200908190916_0002 > mapred is not wrtie under this. Even manually I failed to touch file. > mapred is use of started mr cluster -- This message is automatically generated by JIRA. - You can reply to this email to add a comment to the issue online.
[jira] Updated: (MAPREDUCE-871) Job/Task local files have incorrect group ownership set by LinuxTaskController binary
[
https://issues.apache.org/jira/browse/MAPREDUCE-871?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Vinod K V updated MAPREDUCE-871:
Attachment: MAPREDUCE-871-20090820.1.txt
Here's the scenario.
{quote}
TT process user/group information: mapred
group1,group2,group3 (group1 is primary)
task-controller ownership: ---Sr-s--- 1 root group2
job submitter: userA:groupA
{quote}
The bug is the ownership of files created for the task, for e.g.
bq. $jobid dr-xrws--- userA group1
Attaching patch to fix this issue which essentially sets the permissions of
files created for the task as
bq. $jobid dr-xrws--- userA group2
Also modified the testcase to reflect this.
> Job/Task local files have incorrect group ownership set by
> LinuxTaskController binary
> -
>
> Key: MAPREDUCE-871
> URL: https://issues.apache.org/jira/browse/MAPREDUCE-871
> Project: Hadoop Map/Reduce
> Issue Type: Sub-task
> Components: tasktracker
>Reporter: Vinod K V
>Assignee: Vinod K V
> Attachments: MAPREDUCE-871-20090820.1.txt
>
>
> HADOOP-4491 fixed the secure permissions of local files on a TT. While
> testing HADOOP-4491 on a cluster, [~karams] found out a bug. All the
> files/dirs have should be owned by the group corresponding to the group owner
> of the task-controller binary (via using getegid()) which in turn is a
> special group to which *only* TT user belongs. HADOOP-4491 incorrectly set it
> to primary group of the TT via getgid(), and not the special group.
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
[jira] Updated: (MAPREDUCE-871) Job/Task local files have incorrect group ownership set by LinuxTaskController binary
[ https://issues.apache.org/jira/browse/MAPREDUCE-871?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Vinod K V updated MAPREDUCE-871: Status: Patch Available (was: Open) > Job/Task local files have incorrect group ownership set by > LinuxTaskController binary > - > > Key: MAPREDUCE-871 > URL: https://issues.apache.org/jira/browse/MAPREDUCE-871 > Project: Hadoop Map/Reduce > Issue Type: Sub-task > Components: tasktracker >Reporter: Vinod K V >Assignee: Vinod K V > Attachments: MAPREDUCE-871-20090820.1.txt > > > HADOOP-4491 fixed the secure permissions of local files on a TT. While > testing HADOOP-4491 on a cluster, [~karams] found out a bug. All the > files/dirs have should be owned by the group corresponding to the group owner > of the task-controller binary (via using getegid()) which in turn is a > special group to which *only* TT user belongs. HADOOP-4491 incorrectly set it > to primary group of the TT via getgid(), and not the special group. -- This message is automatically generated by JIRA. - You can reply to this email to add a comment to the issue online.
[jira] Created: (MAPREDUCE-894) DBInputformat not working with SQLServer
DBInputformat not working with SQLServer Key: MAPREDUCE-894 URL: https://issues.apache.org/jira/browse/MAPREDUCE-894 Project: Hadoop Map/Reduce Issue Type: Bug Reporter: Budianto Lie org.apache.hadoop.mapreduce.lib.db.DBInputFormat Microsoft SQLServer doesn't support LIMIT and OFFSET. Fix: Based on MAPREDUCE-716, I already implemented it. By creating a new class org.apache.hadoop.mapreduce.lib.db.MsSqlDBRecordReader and modifying class org.apache.hadoop.mapreduce.lib.db.DBInputFormat Note: this fix is working only with SQLServer 2005 or higher. -- This message is automatically generated by JIRA. - You can reply to this email to add a comment to the issue online.
[jira] Created: (MAPREDUCE-893) Provide an ability to refresh queue configuration without restart.
Provide an ability to refresh queue configuration without restart. -- Key: MAPREDUCE-893 URL: https://issues.apache.org/jira/browse/MAPREDUCE-893 Project: Hadoop Map/Reduce Issue Type: Improvement Components: jobtracker Reporter: Hemanth Yamijala While administering a cluster using multiple queues, administrators feel a need to refresh queue properties on the fly without needing to restart the JobTracker. This is partially supported for some properties such as queue ACLs (HADOOP-5396) and state (HADOOP-5913). The idea is to extend the facility to refresh other queue properties as well, including scheduler properties. -- This message is automatically generated by JIRA. - You can reply to this email to add a comment to the issue online.
[jira] Updated: (MAPREDUCE-886) After 4491, when task-controller exit with some error message, LinuxTaskController only ExitCodeException but does not prints the exit code of task-controller
[ https://issues.apache.org/jira/browse/MAPREDUCE-886?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Sreekanth Ramakrishnan updated MAPREDUCE-886: - Attachment: mapreduce-886-1.patch Adding exit code to the logging method, logging exit code only if the process has exited with non-zero exit code. > After 4491, when task-controller exit with some error message, > LinuxTaskController only ExitCodeException but does not prints the exit code > of task-controller > -- > > Key: MAPREDUCE-886 > URL: https://issues.apache.org/jira/browse/MAPREDUCE-886 > Project: Hadoop Map/Reduce > Issue Type: Bug > Components: tasktracker >Affects Versions: 0.21.0 >Reporter: Karam Singh > Attachments: mapreduce-886-1.patch > > -- This message is automatically generated by JIRA. - You can reply to this email to add a comment to the issue online.
[jira] Updated: (MAPREDUCE-370) Change org.apache.hadoop.mapred.lib.MultipleOutputs to use new api.
[ https://issues.apache.org/jira/browse/MAPREDUCE-370?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Amareshwari Sriramadasu updated MAPREDUCE-370: -- Fix Version/s: 0.21.0 Status: Patch Available (was: Open) > Change org.apache.hadoop.mapred.lib.MultipleOutputs to use new api. > --- > > Key: MAPREDUCE-370 > URL: https://issues.apache.org/jira/browse/MAPREDUCE-370 > Project: Hadoop Map/Reduce > Issue Type: Sub-task >Reporter: Amareshwari Sriramadasu >Assignee: Amareshwari Sriramadasu > Fix For: 0.21.0 > > Attachments: patch-370-1.txt, patch-370.txt > > -- This message is automatically generated by JIRA. - You can reply to this email to add a comment to the issue online.
[jira] Updated: (MAPREDUCE-370) Change org.apache.hadoop.mapred.lib.MultipleOutputs to use new api.
[ https://issues.apache.org/jira/browse/MAPREDUCE-370?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Amareshwari Sriramadasu updated MAPREDUCE-370: -- Attachment: patch-370-1.txt Patch with review comments from Sharad incorporated. > Change org.apache.hadoop.mapred.lib.MultipleOutputs to use new api. > --- > > Key: MAPREDUCE-370 > URL: https://issues.apache.org/jira/browse/MAPREDUCE-370 > Project: Hadoop Map/Reduce > Issue Type: Sub-task >Reporter: Amareshwari Sriramadasu >Assignee: Amareshwari Sriramadasu > Fix For: 0.21.0 > > Attachments: patch-370-1.txt, patch-370.txt > > -- This message is automatically generated by JIRA. - You can reply to this email to add a comment to the issue online.
[jira] Commented: (MAPREDUCE-818) org.apache.hadoop.mapreduce.Counters.getGroup returns null if the group name doesnt exist.
[ https://issues.apache.org/jira/browse/MAPREDUCE-818?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12745342#action_12745342 ] Amareshwari Sriramadasu commented on MAPREDUCE-818: --- -1 contrib tests. Is due to MAPREDUCE-699 > org.apache.hadoop.mapreduce.Counters.getGroup returns null if the group name > doesnt exist. > -- > > Key: MAPREDUCE-818 > URL: https://issues.apache.org/jira/browse/MAPREDUCE-818 > Project: Hadoop Map/Reduce > Issue Type: Bug >Reporter: Amareshwari Sriramadasu >Assignee: Amareshwari Sriramadasu >Priority: Minor > Fix For: 0.20.1 > > Attachments: patch-818.txt > > > org.apache.hadoop.mapreduce.Counters.getGroup returns null if the group name > doesnt exist. But the documentation says it returns an empty group if there > is none with the specified name. -- This message is automatically generated by JIRA. - You can reply to this email to add a comment to the issue online.
[jira] Updated: (MAPREDUCE-856) Localized files from DistributedCache should have right access-control
[
https://issues.apache.org/jira/browse/MAPREDUCE-856?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Vinod K V updated MAPREDUCE-856:
Attachment: MAPREDUCE-856-20090820.txt
Attaching patch. It has to be applied over the latest one at MAPREDUCE-476.
This does the following:
- Changes dir structure to the following:
{code}
$mapred.local.dir
`-- taskTracker
`-- $user
|- distcache
|
`-- jobcache
{code}
- Adds a new initializeDistributedCache(InitializationContext context) to
secure distributed cache files via TaskRunner
- Adds a new INITIALIZE_DISTRIBUTEDCACHE command for LinuxTaskController
binary which takes a user name as argument and sets private permissions for the
corresponding distcache files in $mapred.local.dir/taskTracker/$user/distcache
- Adds a new INITIALIZE_USER command to set secure permissions for the
$mapred.local.dir/taskTracker/$user directories.
- Adds user as part of Task serialization as TT needs user name even before
job-conf is localized. The user-name is needed for creating user specific
directories $mapred.local.dir/taskTracker/$user.
- Moves some of the old and newly added localization code in TaskTracker.java
to org.apache.hadoop.mapreduce.server.tasktracker.Localizer as per
MAPREDUCE-759 and MAPREDUCE-303. Had to change the scope of few things to
public, but marked them only for internal use.
- Adds functionality to clean up stale user directories whenever possible.
- Adds a new test in TestTaskTrackerLocalization to verify cleanup of task
files when needed.
- Adds a new testcase
TestTrackerDistributedCacheManagerWithLinuxTaskController to verify permissions
set on distributed cache files by LinuxTaskController
> Localized files from DistributedCache should have right access-control
> --
>
> Key: MAPREDUCE-856
> URL: https://issues.apache.org/jira/browse/MAPREDUCE-856
> Project: Hadoop Map/Reduce
> Issue Type: Sub-task
> Components: tasktracker
>Reporter: Arun C Murthy
>Assignee: Vinod K V
> Attachments: MAPREDUCE-856-20090820.txt
>
>
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
[jira] Updated: (MAPREDUCE-834) When TaskTracker config use old memory management values its memory monitoring is diabled.
[ https://issues.apache.org/jira/browse/MAPREDUCE-834?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Hemanth Yamijala updated MAPREDUCE-834: --- Assignee: Sreekanth Ramakrishnan Status: Patch Available (was: Open) Last patch looks fine to me. +1. Running through Hudson. Let's wait for the hudson run before uploading the 20 patch. > When TaskTracker config use old memory management values its memory > monitoring is diabled. > -- > > Key: MAPREDUCE-834 > URL: https://issues.apache.org/jira/browse/MAPREDUCE-834 > Project: Hadoop Map/Reduce > Issue Type: Bug >Reporter: Karam Singh >Assignee: Sreekanth Ramakrishnan > Attachments: mapreduce-834-1.patch, mapreduce-834-2.patch, > mapreduce-834-3.patch, mapreduce-834-4.patch > > > TaskTracker memory config values -: > mapred.tasktracker.vmem.reserved=8589934592 > mapred.task.default.maxvmem=2147483648 > mapred.task.limit.maxvmem=4294967296 > mapred.tasktracker.pmem.reserved=2147483648 > TaskTracker start as -: >2009-08-05 12:39:03,308 WARN > org.apache.hadoop.mapred.TaskTracker: The variable > mapred.tasktracker.vmem.reserved is no longer used > 2009-08-05 12:39:03,308 WARN > org.apache.hadoop.mapred.TaskTracker: The variable > mapred.tasktracker.pmem.reserved is no longer used > 2009-08-05 12:39:03,308 WARN > org.apache.hadoop.mapred.TaskTracker: The variable > mapred.task.default.maxvmem is no longer used > 2009-08-05 12:39:03,308 WARN > org.apache.hadoop.mapred.TaskTracker: The variable mapred.task.limit.maxvmem > is no longer used > 2009-08-05 12:39:03,308 INFO > org.apache.hadoop.mapred.TaskTracker: Starting thread: Map-events fetcher for > all reduce tasks on > 2009-08-05 12:39:03,309 INFO > org.apache.hadoop.mapred.TaskTracker: Using MemoryCalculatorPlugin : > org.apache.hadoop.util.linuxmemorycalculatorplu...@19be4777 > 2009-08-05 12:39:03,311 WARN > org.apache.hadoop.mapred.TaskTracker: TaskTracker's > totalMemoryAllottedForTasks is -1. TaskMemoryManager is disabled. -- This message is automatically generated by JIRA. - You can reply to this email to add a comment to the issue online.
