[jira] Updated: (MAPREDUCE-834) When TaskTracker config use old memory management values its memory monitoring is diabled.
[ https://issues.apache.org/jira/browse/MAPREDUCE-834?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Sreekanth Ramakrishnan updated MAPREDUCE-834: - Attachment: mapreduce-834-4.patch Removing an unused variable from the test case. > When TaskTracker config use old memory management values its memory > monitoring is diabled. > -- > > Key: MAPREDUCE-834 > URL: https://issues.apache.org/jira/browse/MAPREDUCE-834 > Project: Hadoop Map/Reduce > Issue Type: Bug >Reporter: Karam Singh > Attachments: mapreduce-834-1.patch, mapreduce-834-2.patch, > mapreduce-834-3.patch, mapreduce-834-4.patch > > > TaskTracker memory config values -: > mapred.tasktracker.vmem.reserved=8589934592 > mapred.task.default.maxvmem=2147483648 > mapred.task.limit.maxvmem=4294967296 > mapred.tasktracker.pmem.reserved=2147483648 > TaskTracker start as -: >2009-08-05 12:39:03,308 WARN > org.apache.hadoop.mapred.TaskTracker: The variable > mapred.tasktracker.vmem.reserved is no longer used > 2009-08-05 12:39:03,308 WARN > org.apache.hadoop.mapred.TaskTracker: The variable > mapred.tasktracker.pmem.reserved is no longer used > 2009-08-05 12:39:03,308 WARN > org.apache.hadoop.mapred.TaskTracker: The variable > mapred.task.default.maxvmem is no longer used > 2009-08-05 12:39:03,308 WARN > org.apache.hadoop.mapred.TaskTracker: The variable mapred.task.limit.maxvmem > is no longer used > 2009-08-05 12:39:03,308 INFO > org.apache.hadoop.mapred.TaskTracker: Starting thread: Map-events fetcher for > all reduce tasks on > 2009-08-05 12:39:03,309 INFO > org.apache.hadoop.mapred.TaskTracker: Using MemoryCalculatorPlugin : > org.apache.hadoop.util.linuxmemorycalculatorplu...@19be4777 > 2009-08-05 12:39:03,311 WARN > org.apache.hadoop.mapred.TaskTracker: TaskTracker's > totalMemoryAllottedForTasks is -1. TaskMemoryManager is disabled. -- This message is automatically generated by JIRA. - You can reply to this email to add a comment to the issue online.
[jira] Commented: (MAPREDUCE-284) Improvements to RPC between Child and TaskTracker
[ https://issues.apache.org/jira/browse/MAPREDUCE-284?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12745332#action_12745332 ] Ravi Gummadi commented on MAPREDUCE-284: unit tests passed on my local machine. ant test-patch gave: [exec] -1 overall. [exec] [exec] +1 @author. The patch does not contain any @author tags. [exec] [exec] -1 tests included. The patch doesn't appear to include any new or modified tests. [exec] Please justify why no new tests are needed for this patch. [exec] Also please list what manual steps were performed to verify this patch. [exec] [exec] +1 javadoc. The javadoc tool did not generate any warning messages. [exec] [exec] +1 javac. The applied patch does not increase the total number of javac compiler warnings. [exec] [exec] +1 findbugs. The patch does not introduce any new Findbugs warnings. [exec] [exec] +1 release audit. The applied patch does not increase the total number of release audit warnings. I don't see a simple way to add a unit testcase for this. > Improvements to RPC between Child and TaskTracker > - > > Key: MAPREDUCE-284 > URL: https://issues.apache.org/jira/browse/MAPREDUCE-284 > Project: Hadoop Map/Reduce > Issue Type: Improvement >Reporter: Arun C Murthy >Assignee: Ravi Gummadi > Fix For: 0.21.0 > > Attachments: MR-284.patch, MR-284.v1.patch, MR-284.v2.patch > > > We could improve the RPC between the Child and TaskTracker: >* Set ping interval lower by default to 5s >* Disable nagle's algorithm (tcp no-delay) -- This message is automatically generated by JIRA. - You can reply to this email to add a comment to the issue online.
[jira] Updated: (MAPREDUCE-284) Improvements to RPC between Child and TaskTracker
[ https://issues.apache.org/jira/browse/MAPREDUCE-284?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Ravi Gummadi updated MAPREDUCE-284: --- Attachment: MR-284.v2.patch As changing ipc.client.tcpnodelay affects other clients(not just child to TT) and that needs more discussion, am uploading a new patch that sets tcpnodelay only in Child.java as part of this JIRA. Please review and provide your comments. > Improvements to RPC between Child and TaskTracker > - > > Key: MAPREDUCE-284 > URL: https://issues.apache.org/jira/browse/MAPREDUCE-284 > Project: Hadoop Map/Reduce > Issue Type: Improvement >Reporter: Arun C Murthy >Assignee: Ravi Gummadi > Fix For: 0.21.0 > > Attachments: MR-284.patch, MR-284.v1.patch, MR-284.v2.patch > > > We could improve the RPC between the Child and TaskTracker: >* Set ping interval lower by default to 5s >* Disable nagle's algorithm (tcp no-delay) -- This message is automatically generated by JIRA. - You can reply to this email to add a comment to the issue online.
[jira] Resolved: (MAPREDUCE-833) Jobclient does not print any warning message when old memory config variable used with -D option from command line
[ https://issues.apache.org/jira/browse/MAPREDUCE-833?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Hemanth Yamijala resolved MAPREDUCE-833. Resolution: Duplicate This was fixed as a part of MAPREDUCE-832, which introduced a log message in JobConf's constructor. By doing so, any time a new conf object is created on the client, these warning messages would get printed. Hence resolving as duplicate. > Jobclient does not print any warning message when old memory config variable > used with -D option from command line > -- > > Key: MAPREDUCE-833 > URL: https://issues.apache.org/jira/browse/MAPREDUCE-833 > Project: Hadoop Map/Reduce > Issue Type: Bug >Affects Versions: 0.20.1 >Reporter: Karam Singh > -- This message is automatically generated by JIRA. - You can reply to this email to add a comment to the issue online.
[jira] Updated: (MAPREDUCE-832) Too many WARN messages about deprecated memorty config variables in JobTacker log
[ https://issues.apache.org/jira/browse/MAPREDUCE-832?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Hemanth Yamijala updated MAPREDUCE-832: --- Resolution: Fixed Fix Version/s: 0.20.1 Hadoop Flags: [Reviewed] Status: Resolved (was: Patch Available) I just committed this to trunk and branch 0.20. Thanks, Rahul ! > Too many WARN messages about deprecated memorty config variables in JobTacker > log > - > > Key: MAPREDUCE-832 > URL: https://issues.apache.org/jira/browse/MAPREDUCE-832 > Project: Hadoop Map/Reduce > Issue Type: Bug >Affects Versions: 0.20.1 >Reporter: Karam Singh >Assignee: rahul k singh > Fix For: 0.20.1 > > Attachments: mapreduce-832-20.patch, mapreduce-832-20.patch, > mapreduce-832.patch > > > When user submit a mapred job using old memory config vairiable > (mapred.task.maxmem) followinig message too many times in JobTracker logs -: > [ > WARN org.apache.hadoop.mapred.JobConf: The variable mapred.task.maxvmem is no > longer used instead use mapred.job.map.memory.mb and > mapred.job.reduce.memory.mb > ] -- This message is automatically generated by JIRA. - You can reply to this email to add a comment to the issue online.
[jira] Commented: (MAPREDUCE-370) Change org.apache.hadoop.mapred.lib.MultipleOutputs to use new api.
[ https://issues.apache.org/jira/browse/MAPREDUCE-370?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12745326#action_12745326 ] Sharad Agarwal commented on MAPREDUCE-370: -- bq. If we do this, this will remove generate* methods from the api proposed. I am +1 on removing generate*. This will simplify the api and overriding of MultipleOutputs is not required in case user wants to control the output path name. > Change org.apache.hadoop.mapred.lib.MultipleOutputs to use new api. > --- > > Key: MAPREDUCE-370 > URL: https://issues.apache.org/jira/browse/MAPREDUCE-370 > Project: Hadoop Map/Reduce > Issue Type: Sub-task >Reporter: Amareshwari Sriramadasu >Assignee: Amareshwari Sriramadasu > Attachments: patch-370.txt > > -- This message is automatically generated by JIRA. - You can reply to this email to add a comment to the issue online.
[jira] Updated: (MAPREDUCE-832) Too many WARN messages about deprecated memorty config variables in JobTacker log
[ https://issues.apache.org/jira/browse/MAPREDUCE-832?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Hemanth Yamijala updated MAPREDUCE-832: --- Attachment: mapreduce-832-20.patch Patch for Hadoop 0.20 branch. Only source paths have been changed. > Too many WARN messages about deprecated memorty config variables in JobTacker > log > - > > Key: MAPREDUCE-832 > URL: https://issues.apache.org/jira/browse/MAPREDUCE-832 > Project: Hadoop Map/Reduce > Issue Type: Bug >Affects Versions: 0.20.1 >Reporter: Karam Singh >Assignee: rahul k singh > Attachments: mapreduce-832-20.patch, mapreduce-832-20.patch, > mapreduce-832.patch > > > When user submit a mapred job using old memory config vairiable > (mapred.task.maxmem) followinig message too many times in JobTracker logs -: > [ > WARN org.apache.hadoop.mapred.JobConf: The variable mapred.task.maxvmem is no > longer used instead use mapred.job.map.memory.mb and > mapred.job.reduce.memory.mb > ] -- This message is automatically generated by JIRA. - You can reply to this email to add a comment to the issue online.
[jira] Commented: (MAPREDUCE-832) Too many WARN messages about deprecated memorty config variables in JobTacker log
[ https://issues.apache.org/jira/browse/MAPREDUCE-832?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12745318#action_12745318 ] Hemanth Yamijala commented on MAPREDUCE-832: bq. -1 tests included. The patch doesn't appear to include any new or modified tests. The patch is adding a debug log message which is hard to write a unit test for. Our QE team has manually verified the patch by running tests on a cluster with old memory configuration and verifying that the number of WARN messages has come down drastically. Also, the scenario in MAPREDUCE-833 has been tested and found fixed. bq. -1 core tests. The patch failed core unit tests. org.apache.hadoop.mapred.TestRecoveryManager.testRestartCount's failure is independent of this patch. It happens with trunk and is being tracked in MAPREDUCE-880 bq. -1 contrib tests. The patch failed contrib unit tests. The streaming test failures have been happening for a while on hudson alone. They pass locally. On this basis, I am going to commit the patch to trunk and an equivalent patch to branch 20. > Too many WARN messages about deprecated memorty config variables in JobTacker > log > - > > Key: MAPREDUCE-832 > URL: https://issues.apache.org/jira/browse/MAPREDUCE-832 > Project: Hadoop Map/Reduce > Issue Type: Bug >Affects Versions: 0.20.1 >Reporter: Karam Singh >Assignee: rahul k singh > Attachments: mapreduce-832-20.patch, mapreduce-832.patch > > > When user submit a mapred job using old memory config vairiable > (mapred.task.maxmem) followinig message too many times in JobTracker logs -: > [ > WARN org.apache.hadoop.mapred.JobConf: The variable mapred.task.maxvmem is no > longer used instead use mapred.job.map.memory.mb and > mapred.job.reduce.memory.mb > ] -- This message is automatically generated by JIRA. - You can reply to this email to add a comment to the issue online.
[jira] Updated: (MAPREDUCE-807) Stray user files in mapred.system.dir with permissions other than 777 can prevent the jobtracker from starting up.
[ https://issues.apache.org/jira/browse/MAPREDUCE-807?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Amar Kamat updated MAPREDUCE-807: - Attachment: MAPRED-807-v1.4.patch Attaching a patch that fixes the issue at hand. Result of test-patch [exec] -1 overall. [exec] [exec] +1 @author. The patch does not contain any @author tags. [exec] [exec] -1 tests included. The patch doesn't appear to include any new or modified tests. [exec] Please justify why no new tests are needed for this patch. [exec] Also please list what manual steps were performed to verify this patch. [exec] [exec] +1 javadoc. The javadoc tool did not generate any warning messages. [exec] [exec] +1 javac. The applied patch does not increase the total number of javac compiler warnings. [exec] [exec] +1 findbugs. The patch does not introduce any new Findbugs warnings. [exec] [exec] +1 release audit. The applied patch does not increase the total number of release audit warnings. Running ant tests. > Stray user files in mapred.system.dir with permissions other than 777 can > prevent the jobtracker from starting up. > -- > > Key: MAPREDUCE-807 > URL: https://issues.apache.org/jira/browse/MAPREDUCE-807 > Project: Hadoop Map/Reduce > Issue Type: Bug > Components: jobtracker >Reporter: Amar Kamat >Assignee: Amar Kamat >Priority: Blocker > Attachments: MAPRED-807-v1.1.patch, MAPRED-807-v1.2.patch, > MAPRED-807-v1.3.patch, MAPRED-807-v1.4.patch > > > With restart disabled, the jobtracker does a _rm -rf_ of the > mapred.system.dir. If the mapred.system.dir contains user files with > permissions other than 777 then the jobtracker gets stuck in a loop trying to > delete the mapred.system.dir (and each time failing with > AccessControlException). -- This message is automatically generated by JIRA. - You can reply to this email to add a comment to the issue online.
[jira] Commented: (MAPREDUCE-824) Support a hierarchy of queues in the capacity scheduler
[
https://issues.apache.org/jira/browse/MAPREDUCE-824?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12745312#action_12745312
]
Hemanth Yamijala commented on MAPREDUCE-824:
Some comments:
- AbstractQueue.updateContext can move into QueueSchedulingContext, as all the
state it is operating on is in QSC.
- Also prevMapClusterCapacity and prevReduceClusterCapacity can also be moved
to the context. They can be private, and renamed to prev*Capacity, dropping the
'Cluster' because for container queues, they don't reflect the entire cluster
capacity. Same naming change would apply to variables in QueueSchedulingContext
(like setMapClusterCapacity, etc)
- AbstractQueue.getOrderedJobQueues, its not very clear that this is looking
through the entire hierarchy. Also, it assumes that sorting is done before
this. So, its not a very orthogonal API. Move this to the scheduler, and
introduce a new API like AbstractQueue.getDescendentJobQueues().
- Override AbstractQueue.addChildren in JobQueue to throw an unsupported
exception.
- Make AbstractQueue.getChildren package private and document it is for tests.
- I suggest we modify the algorithm in distributeUnConfiguredCapacity to follow
this pattern to make it clearer:
{code}
for (Queue q : children) {
if (q.capacity == -1) {
unconfigured.add(q);
}
}
// distribute capacity for all unconfigured queues.
for (Queue q : children) {
q.distributeUnconfiguredCapacity();
}
{code}
- I would suggest we provide equals and hashCode in AbstractQueue to be based
on the queue Name. toString in AbstractQueue should print the queue name.
- I didn't understand the need for setting the capacity in conf in
distributeUnconfiguredCapacity. It seems like requiring the Configuration
instance to be passed to distributeUnconfiguredCapacity is creating an
undesirable dependency. Can you check if we can break this dependency.
- distributeUnConfiguredCapacity will throw a Divide by zero if there is no
queues without configured capacity.
- We don't need to pass the supportsPriority variable separately to the
JobQueue's constructor. Let's set that directly in the
JobQueue.QueueSchedulingContext which we are already passing to JobQueue.
- In JobQueue, methods like addWaitingJob etc should be private. Also, I think
some of the methods can be folded. For e.g. makeJobRunning just calls
addRunningJob, so we can refactor to remove makeJobRunning and call
addRunningJob directly.
- TaskData seems out of place in TaskSchedulingContext. The scheduling context
contains state w.r.to scheduler. TaskData is a simple abstraction that returns
a view of job information based on the task type. So, let's pull it out and
call it TaskDataView which can be extended by MapTaskDataView and
ReduceTaskDataView. There should be only one instance of these per scheduler
instance and they can be got from the scheduler itself.
- Rename TaskSchedulingContext.add to TSC.update.
- Can we pull out the whole hierarchy building logic into a separate class -
like a QueueHierarchyBuilder ? It could be given the CapacitySchedulerConf and
QueueManager and have an API like buildHierarchy - which would return the root
of the queues. Capacity scheduler can thus be abstracted from how the hierarchy
is created - it just gets the hierarchy from somewhere. For e.g. in tests, the
hierarchy can be manually created and given to the given.
- Please remove mapScheduler.initialize() and reduceScheduler.initialize().
- tsi.getMaxCapacity() < tsi.getCapacity(): this check in
areTasksInQueueOverLimit does not seem required. Because the check is already
being done in tsi.getCapacity()
- totalCapacity modification in the loadContext is a no-op, because the changes
will not be reflected in the caller method. Likewise the check for
totalCapacity > 100.0 is a no-op in createHierarchy.
- The separator char for queues is chosen to be '.' in createHierarchy. It must
be checked that this character doesn't appear anywhere else in the queue name.
- Call to root.sort() should be from TaskSchedulingMgr.assignTasks()
- JobQueuesManager.createQueue should be addQueue. Also, it can get the queue
name from the job queue object directly, and doesn't need the extra parameter.
- JobQueueManager.getQueueNames can be getJobQueueNames.
> Support a hierarchy of queues in the capacity scheduler
> ---
>
> Key: MAPREDUCE-824
> URL: https://issues.apache.org/jira/browse/MAPREDUCE-824
> Project: Hadoop Map/Reduce
> Issue Type: New Feature
> Components: contrib/capacity-sched
>Reporter: Hemanth Yamijala
> Attachments: HADOOP-824-1.patch, HADOOP-824-2.patch,
> HADOOP-824-3.patch
>
>
> Currently in Capacity Scheduler, cluster capacity is divided among the queues
> based on the queue capacity. These queues typically
[jira] Updated: (MAPREDUCE-834) When TaskTracker config use old memory management values its memory monitoring is diabled.
[ https://issues.apache.org/jira/browse/MAPREDUCE-834?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Sreekanth Ramakrishnan updated MAPREDUCE-834: - Attachment: mapreduce-834-3.patch Attaching file fixing an issue found during internal testing. Modified test case and code. > When TaskTracker config use old memory management values its memory > monitoring is diabled. > -- > > Key: MAPREDUCE-834 > URL: https://issues.apache.org/jira/browse/MAPREDUCE-834 > Project: Hadoop Map/Reduce > Issue Type: Bug >Reporter: Karam Singh > Attachments: mapreduce-834-1.patch, mapreduce-834-2.patch, > mapreduce-834-3.patch > > > TaskTracker memory config values -: > mapred.tasktracker.vmem.reserved=8589934592 > mapred.task.default.maxvmem=2147483648 > mapred.task.limit.maxvmem=4294967296 > mapred.tasktracker.pmem.reserved=2147483648 > TaskTracker start as -: >2009-08-05 12:39:03,308 WARN > org.apache.hadoop.mapred.TaskTracker: The variable > mapred.tasktracker.vmem.reserved is no longer used > 2009-08-05 12:39:03,308 WARN > org.apache.hadoop.mapred.TaskTracker: The variable > mapred.tasktracker.pmem.reserved is no longer used > 2009-08-05 12:39:03,308 WARN > org.apache.hadoop.mapred.TaskTracker: The variable > mapred.task.default.maxvmem is no longer used > 2009-08-05 12:39:03,308 WARN > org.apache.hadoop.mapred.TaskTracker: The variable mapred.task.limit.maxvmem > is no longer used > 2009-08-05 12:39:03,308 INFO > org.apache.hadoop.mapred.TaskTracker: Starting thread: Map-events fetcher for > all reduce tasks on > 2009-08-05 12:39:03,309 INFO > org.apache.hadoop.mapred.TaskTracker: Using MemoryCalculatorPlugin : > org.apache.hadoop.util.linuxmemorycalculatorplu...@19be4777 > 2009-08-05 12:39:03,311 WARN > org.apache.hadoop.mapred.TaskTracker: TaskTracker's > totalMemoryAllottedForTasks is -1. TaskMemoryManager is disabled. -- This message is automatically generated by JIRA. - You can reply to this email to add a comment to the issue online.
[jira] Updated: (MAPREDUCE-870) Clean up the job Retire code
[ https://issues.apache.org/jira/browse/MAPREDUCE-870?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Sharad Agarwal updated MAPREDUCE-870: - Attachment: 870_v3.patch > Clean up the job Retire code > > > Key: MAPREDUCE-870 > URL: https://issues.apache.org/jira/browse/MAPREDUCE-870 > Project: Hadoop Map/Reduce > Issue Type: Sub-task >Reporter: Sharad Agarwal >Assignee: Sharad Agarwal > Attachments: 870_v1.patch, 870_v2.patch, 870_v3.patch > > > Currently completed job's full data structures are kept in memory based on > mapred.jobtracker.completeuserjobs.maximum, > mapred.jobtracker.retirejob.interval.min, > mapred.jobtracker.retirejob.interval and mapred.jobtracker.retirejob.check > settings. These controls are not much useful now since MAPREDUCE-817 > introduced a cache for keeping just the very basic info of the completed job. > These settings should be removed and the job should be purged as soon as the > history files are available in HDFS. > Going forward, clients can read the history files if they need to drill down > into more information (MAPREDUCE-864). -- This message is automatically generated by JIRA. - You can reply to this email to add a comment to the issue online.
[jira] Updated: (MAPREDUCE-870) Clean up the job Retire code
[ https://issues.apache.org/jira/browse/MAPREDUCE-870?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Sharad Agarwal updated MAPREDUCE-870: - Status: Open (was: Patch Available) > Clean up the job Retire code > > > Key: MAPREDUCE-870 > URL: https://issues.apache.org/jira/browse/MAPREDUCE-870 > Project: Hadoop Map/Reduce > Issue Type: Sub-task >Reporter: Sharad Agarwal >Assignee: Sharad Agarwal > Attachments: 870_v1.patch, 870_v2.patch > > > Currently completed job's full data structures are kept in memory based on > mapred.jobtracker.completeuserjobs.maximum, > mapred.jobtracker.retirejob.interval.min, > mapred.jobtracker.retirejob.interval and mapred.jobtracker.retirejob.check > settings. These controls are not much useful now since MAPREDUCE-817 > introduced a cache for keeping just the very basic info of the completed job. > These settings should be removed and the job should be purged as soon as the > history files are available in HDFS. > Going forward, clients can read the history files if they need to drill down > into more information (MAPREDUCE-864). -- This message is automatically generated by JIRA. - You can reply to this email to add a comment to the issue online.
[jira] Resolved: (MAPREDUCE-882) TestJobHistory fails sometimes
[ https://issues.apache.org/jira/browse/MAPREDUCE-882?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Amar Kamat resolved MAPREDUCE-882. -- Resolution: Duplicate The problem comes because taskIDsToAttemptIDs structure is shared in TestJobHistory across tests. So if the job-id clash across testcases then the checks will fail. MAPREDUCE-745 should fix this. > TestJobHistory fails sometimes > -- > > Key: MAPREDUCE-882 > URL: https://issues.apache.org/jira/browse/MAPREDUCE-882 > Project: Hadoop Map/Reduce > Issue Type: Bug >Reporter: Devaraj Das >Assignee: Amar Kamat > > Testcase: testDoneFolderOnHDFS took 31.892 sec > Testcase: testJobHistoryFile took 27.901 sec > FAILED > Duplicate START_TIME seen for task task_200908161937_0001_m_03 in history > file at line 5 > junit.framework.AssertionFailedError: Duplicate START_TIME seen for task > task_200908161937_0001_m_03 in history file at line 5 > at > org.apache.hadoop.mapred.TestJobHistory$TestListener.handle(TestJobHistory.java:164) > at org.apache.hadoop.mapred.JobHistory.parseLine(JobHistory.java:505) > at > org.apache.hadoop.mapred.JobHistory.parseHistoryFromFS(JobHistory.java:469) > at > org.apache.hadoop.mapred.TestJobHistory.validateJobHistoryFileFormat(TestJobHistory.java:496) > at > org.apache.hadoop.mapred.TestJobHistory.testJobHistoryFile(TestJobHistory.java:964) > Testcase: testJobHistoryUserLogLocation took 75.161 sec > Testcase: testJobHistoryJobStatus took 156.88 sec -- This message is automatically generated by JIRA. - You can reply to this email to add a comment to the issue online.
[jira] Assigned: (MAPREDUCE-882) TestJobHistory fails sometimes
[ https://issues.apache.org/jira/browse/MAPREDUCE-882?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Amar Kamat reassigned MAPREDUCE-882: Assignee: Amar Kamat > TestJobHistory fails sometimes > -- > > Key: MAPREDUCE-882 > URL: https://issues.apache.org/jira/browse/MAPREDUCE-882 > Project: Hadoop Map/Reduce > Issue Type: Bug >Reporter: Devaraj Das >Assignee: Amar Kamat > > Testcase: testDoneFolderOnHDFS took 31.892 sec > Testcase: testJobHistoryFile took 27.901 sec > FAILED > Duplicate START_TIME seen for task task_200908161937_0001_m_03 in history > file at line 5 > junit.framework.AssertionFailedError: Duplicate START_TIME seen for task > task_200908161937_0001_m_03 in history file at line 5 > at > org.apache.hadoop.mapred.TestJobHistory$TestListener.handle(TestJobHistory.java:164) > at org.apache.hadoop.mapred.JobHistory.parseLine(JobHistory.java:505) > at > org.apache.hadoop.mapred.JobHistory.parseHistoryFromFS(JobHistory.java:469) > at > org.apache.hadoop.mapred.TestJobHistory.validateJobHistoryFileFormat(TestJobHistory.java:496) > at > org.apache.hadoop.mapred.TestJobHistory.testJobHistoryFile(TestJobHistory.java:964) > Testcase: testJobHistoryUserLogLocation took 75.161 sec > Testcase: testJobHistoryJobStatus took 156.88 sec -- This message is automatically generated by JIRA. - You can reply to this email to add a comment to the issue online.
[jira] Resolved: (MAPREDUCE-891) Streaming tests fail with NPE in MiniDFSCluster
[
https://issues.apache.org/jira/browse/MAPREDUCE-891?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Vinod K V resolved MAPREDUCE-891.
-
Resolution: Duplicate
Duplicate of MAPREDUCE-699
> Streaming tests fail with NPE in MiniDFSCluster
> ---
>
> Key: MAPREDUCE-891
> URL: https://issues.apache.org/jira/browse/MAPREDUCE-891
> Project: Hadoop Map/Reduce
> Issue Type: Bug
> Components: contrib/streaming, test
>Reporter: Arun C Murthy
> Fix For: 0.21.0
>
>
> Streaming testcases' usage of MiniDFSCluster.startDatanodes causes NPE in
> GenericOptionsParser:
> {noformat}
> java.lang.NullPointerException
> at org.apache.commons.cli.GnuParser.flatten(GnuParser.java:110)
> at org.apache.commons.cli.Parser.parse(Parser.java:143)
> at
> org.apache.hadoop.util.GenericOptionsParser.parseGeneralOptions(GenericOptionsParser.java:374)
> at
> org.apache.hadoop.util.GenericOptionsParser.(GenericOptionsParser.java:153)
> at
> org.apache.hadoop.util.GenericOptionsParser.(GenericOptionsParser.java:138)
> at
> org.apache.hadoop.hdfs.server.datanode.DataNode.instantiateDataNode(DataNode.java:1314)
> at
> org.apache.hadoop.hdfs.MiniDFSCluster.startDataNodes(MiniDFSCluster.java:414)
> at org.apache.hadoop.hdfs.MiniDFSCluster.(MiniDFSCluster.java:278)
> at org.apache.hadoop.hdfs.MiniDFSCluster.(MiniDFSCluster.java:119)
> at
> org.apache.hadoop.streaming.TestDumpTypedBytes.testDumping(TestDumpTypedBytes.java:40)
> {noformat}
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
[jira] Commented: (MAPREDUCE-157) Job History log file format is not friendly for external tools.
[ https://issues.apache.org/jira/browse/MAPREDUCE-157?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12745279#action_12745279 ] eric baldeschwieler commented on MAPREDUCE-157: --- Re: AVRO conversion Doug and I chatted. My concern is that we are working through a job history refactor with a bunch of moving parts. I want to get those all into 21 and stable. Until that is done I don't want to consider AVRO since it might put that rework at risk (since our team has already invested in JSON). That said, I'm not against binary AVRO here. It could have advantages. If someone else can put in the time to demonstrate that this will work, I think that might be a better approach. I just hope we can do that as a distinct patch that follows this one. (Or collaborate to make one patch, we just don't have the resources before 21 freeze) (For the wider context of the refactor, see MAPREDUCE-863) > Job History log file format is not friendly for external tools. > --- > > Key: MAPREDUCE-157 > URL: https://issues.apache.org/jira/browse/MAPREDUCE-157 > Project: Hadoop Map/Reduce > Issue Type: Sub-task >Reporter: Owen O'Malley >Assignee: Jothi Padmanabhan > > Currently, parsing the job history logs with external tools is very difficult > because of the format. The most critical problem is that newlines aren't > escaped in the strings. That makes using tools like grep, sed, and awk very > tricky. -- This message is automatically generated by JIRA. - You can reply to this email to add a comment to the issue online.
[jira] Resolved: (MAPREDUCE-198) Log job history events to a common dump file
[ https://issues.apache.org/jira/browse/MAPREDUCE-198?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] eric baldeschwieler resolved MAPREDUCE-198. --- Resolution: Won't Fix We've abandon this idea. We will not be working on it. > Log job history events to a common dump file > > > Key: MAPREDUCE-198 > URL: https://issues.apache.org/jira/browse/MAPREDUCE-198 > Project: Hadoop Map/Reduce > Issue Type: New Feature >Reporter: Amar Kamat >Assignee: Amar Kamat > > As of today all the jobhistory events are logged to separate files. It would > be nice to also dump all this info into a common file so that external tools > (e.g Chukwa) can harvest history info. Job configuration should also be > dumped. Whether to use a same log file for history dumps and configuration > dumps should be configurable (by default everything goes to one file). -- This message is automatically generated by JIRA. - You can reply to this email to add a comment to the issue online.
[jira] Commented: (MAPREDUCE-751) Rumen: a tool to extract job characterization data from job tracker logs
[ https://issues.apache.org/jira/browse/MAPREDUCE-751?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12745240#action_12745240 ] Hadoop QA commented on MAPREDUCE-751: - -1 overall. Here are the results of testing the latest attachment http://issues.apache.org/jira/secure/attachment/12417041/2009-08-19--1030.patch against trunk revision 805324. +1 @author. The patch does not contain any @author tags. +1 tests included. The patch appears to include 40 new or modified tests. +1 javadoc. The javadoc tool did not generate any warning messages. -1 javac. The applied patch generated 2239 javac compiler warnings (more than the trunk's current 2232 warnings). -1 findbugs. The patch appears to introduce 8 new Findbugs warnings. -1 release audit. The applied patch generated 217 release audit warnings (more than the trunk's current 202 warnings). -1 core tests. The patch failed core unit tests. -1 contrib tests. The patch failed contrib unit tests. Test results: http://hudson.zones.apache.org/hudson/job/Mapreduce-Patch-vesta.apache.org/495/testReport/ Release audit warnings: http://hudson.zones.apache.org/hudson/job/Mapreduce-Patch-vesta.apache.org/495/artifact/trunk/patchprocess/releaseAuditDiffWarnings.txt Findbugs warnings: http://hudson.zones.apache.org/hudson/job/Mapreduce-Patch-vesta.apache.org/495/artifact/trunk/build/test/findbugs/newPatchFindbugsWarnings.html Checkstyle results: http://hudson.zones.apache.org/hudson/job/Mapreduce-Patch-vesta.apache.org/495/artifact/trunk/build/test/checkstyle-errors.html Console output: http://hudson.zones.apache.org/hudson/job/Mapreduce-Patch-vesta.apache.org/495/console This message is automatically generated. > Rumen: a tool to extract job characterization data from job tracker logs > > > Key: MAPREDUCE-751 > URL: https://issues.apache.org/jira/browse/MAPREDUCE-751 > Project: Hadoop Map/Reduce > Issue Type: Bug >Affects Versions: 0.20.1, 0.21.0 >Reporter: Dick King > Fix For: 0.20.1, 0.21.0 > > Attachments: 2009-08-19--1030.patch, mapreduce-751--2009-07-23.patch > > > We propose a new map/reduce component, rumen, which can be used to process > job history logs to produce any or all of the following: > * Retrospective info describing the statistical behavior of the > amount of time it would have taken to launch a job into a certain > percentage of the number of mapper slots in the log's cluster, given the > load over the period covered by the log > * Statistical info as to the runtimes and shuffle times, etc. of > the tasks and jobs covered by the log > * files describing detailed job trace information, and the > network topology as inferred from the host locations and rack IDs that > arise in the job tracker log. In addition to this facility, rumen > includes readers for this information to return job and detailed task > information to other tools. > These other tools include a more advanced version of gridmix, and > also includes mumak: see blocked issues. -- This message is automatically generated by JIRA. - You can reply to this email to add a comment to the issue online.
[jira] Commented: (MAPREDUCE-157) Job History log file format is not friendly for external tools.
[ https://issues.apache.org/jira/browse/MAPREDUCE-157?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12745211#action_12745211 ] Doug Cutting commented on MAPREDUCE-157: Jothi, if you have an early version of this patch, please post it. That way we can better evaluate converting it to use Avro. Thanks! > Job History log file format is not friendly for external tools. > --- > > Key: MAPREDUCE-157 > URL: https://issues.apache.org/jira/browse/MAPREDUCE-157 > Project: Hadoop Map/Reduce > Issue Type: Sub-task >Reporter: Owen O'Malley >Assignee: Jothi Padmanabhan > > Currently, parsing the job history logs with external tools is very difficult > because of the format. The most critical problem is that newlines aren't > escaped in the strings. That makes using tools like grep, sed, and awk very > tricky. -- This message is automatically generated by JIRA. - You can reply to this email to add a comment to the issue online.
[jira] Updated: (MAPREDUCE-892) command line tool to list all tasktrackers and their status
[ https://issues.apache.org/jira/browse/MAPREDUCE-892?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] dhruba borthakur updated MAPREDUCE-892: --- Assignee: Dmytro Molkov Summary: command line tool to list all tasktrackers and their status (was: command line tool to list all tasktracers and their status) > command line tool to list all tasktrackers and their status > --- > > Key: MAPREDUCE-892 > URL: https://issues.apache.org/jira/browse/MAPREDUCE-892 > Project: Hadoop Map/Reduce > Issue Type: Improvement >Reporter: dhruba borthakur >Assignee: Dmytro Molkov > > The "hadoop mradmin -report" could list all the tasktrackers that the > JobTracker knows about. It will also list a brief status summary for each of > the TaskTracker. (This is similar to the hadop dfsadmin -report command that > lists all Datanodes) -- This message is automatically generated by JIRA. - You can reply to this email to add a comment to the issue online.
[jira] Created: (MAPREDUCE-892) command line tool to list all tasktracers and their status
command line tool to list all tasktracers and their status -- Key: MAPREDUCE-892 URL: https://issues.apache.org/jira/browse/MAPREDUCE-892 Project: Hadoop Map/Reduce Issue Type: Improvement Reporter: dhruba borthakur The "hadoop mradmin -report" could list all the tasktrackers that the JobTracker knows about. It will also list a brief status summary for each of the TaskTracker. (This is similar to the hadop dfsadmin -report command that lists all Datanodes) -- This message is automatically generated by JIRA. - You can reply to this email to add a comment to the issue online.
[jira] Commented: (MAPREDUCE-886) After 4491, when task-controller exit with some error message, LinuxTaskController only ExitCodeException but does not prints the exit code of task-controller
[ https://issues.apache.org/jira/browse/MAPREDUCE-886?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12745155#action_12745155 ] Tsz Wo (Nicholas), SZE commented on MAPREDUCE-886: -- Do you mean MAPREDUCE-842 for "4491"? > After 4491, when task-controller exit with some error message, > LinuxTaskController only ExitCodeException but does not prints the exit code > of task-controller > -- > > Key: MAPREDUCE-886 > URL: https://issues.apache.org/jira/browse/MAPREDUCE-886 > Project: Hadoop Map/Reduce > Issue Type: Bug > Components: tasktracker >Affects Versions: 0.21.0 >Reporter: Karam Singh > -- This message is automatically generated by JIRA. - You can reply to this email to add a comment to the issue online.
[jira] Updated: (MAPREDUCE-751) Rumen: a tool to extract job characterization data from job tracker logs
[ https://issues.apache.org/jira/browse/MAPREDUCE-751?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Dick King updated MAPREDUCE-751: Tags: rumen,mumakil,job tracker logs Fix Version/s: 0.21.0 0.20.1 Affects Version/s: 0.21.0 0.20.1 Status: Patch Available (was: Open) This patch implements Rumen as described by this issue. Rumen consumes job tracker log directories and produces the job traces that mumakil and GridMMIX cosume. > Rumen: a tool to extract job characterization data from job tracker logs > > > Key: MAPREDUCE-751 > URL: https://issues.apache.org/jira/browse/MAPREDUCE-751 > Project: Hadoop Map/Reduce > Issue Type: Bug >Affects Versions: 0.20.1, 0.21.0 >Reporter: Dick King > Fix For: 0.20.1, 0.21.0 > > Attachments: 2009-08-19--1030.patch, mapreduce-751--2009-07-23.patch > > > We propose a new map/reduce component, rumen, which can be used to process > job history logs to produce any or all of the following: > * Retrospective info describing the statistical behavior of the > amount of time it would have taken to launch a job into a certain > percentage of the number of mapper slots in the log's cluster, given the > load over the period covered by the log > * Statistical info as to the runtimes and shuffle times, etc. of > the tasks and jobs covered by the log > * files describing detailed job trace information, and the > network topology as inferred from the host locations and rack IDs that > arise in the job tracker log. In addition to this facility, rumen > includes readers for this information to return job and detailed task > information to other tools. > These other tools include a more advanced version of gridmix, and > also includes mumak: see blocked issues. -- This message is automatically generated by JIRA. - You can reply to this email to add a comment to the issue online.
[jira] Updated: (MAPREDUCE-751) Rumen: a tool to extract job characterization data from job tracker logs
[ https://issues.apache.org/jira/browse/MAPREDUCE-751?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Dick King updated MAPREDUCE-751: Attachment: 2009-08-19--1030.patch This is the patch that implements Rumen. It is licensed to Apache. > Rumen: a tool to extract job characterization data from job tracker logs > > > Key: MAPREDUCE-751 > URL: https://issues.apache.org/jira/browse/MAPREDUCE-751 > Project: Hadoop Map/Reduce > Issue Type: Bug >Reporter: Dick King > Attachments: 2009-08-19--1030.patch, mapreduce-751--2009-07-23.patch > > > We propose a new map/reduce component, rumen, which can be used to process > job history logs to produce any or all of the following: > * Retrospective info describing the statistical behavior of the > amount of time it would have taken to launch a job into a certain > percentage of the number of mapper slots in the log's cluster, given the > load over the period covered by the log > * Statistical info as to the runtimes and shuffle times, etc. of > the tasks and jobs covered by the log > * files describing detailed job trace information, and the > network topology as inferred from the host locations and rack IDs that > arise in the job tracker log. In addition to this facility, rumen > includes readers for this information to return job and detailed task > information to other tools. > These other tools include a more advanced version of gridmix, and > also includes mumak: see blocked issues. -- This message is automatically generated by JIRA. - You can reply to this email to add a comment to the issue online.
[jira] Commented: (MAPREDUCE-336) The logging level of the tasks should be configurable by the job
[ https://issues.apache.org/jira/browse/MAPREDUCE-336?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12745145#action_12745145 ] Arun C Murthy commented on MAPREDUCE-336: - TestJobHistory failure is being tracked at MAPREDUCE-882. I've opened MAPREDUCE-891 to track streaming test-failures. > The logging level of the tasks should be configurable by the job > > > Key: MAPREDUCE-336 > URL: https://issues.apache.org/jira/browse/MAPREDUCE-336 > Project: Hadoop Map/Reduce > Issue Type: Improvement >Reporter: Owen O'Malley >Assignee: Arun C Murthy > Fix For: 0.21.0 > > Attachments: MAPREDUCE-336_0_20090818.patch > > > It would be nice to be able to configure the logging level of the Task JVM's > separately from the server JVM's. Reducing logging substantially increases > performance and reduces the consumption of local disk on the task trackers. -- This message is automatically generated by JIRA. - You can reply to this email to add a comment to the issue online.
[jira] Created: (MAPREDUCE-891) Streaming tests fail with NPE in MiniDFSCluster
Streaming tests fail with NPE in MiniDFSCluster
---
Key: MAPREDUCE-891
URL: https://issues.apache.org/jira/browse/MAPREDUCE-891
Project: Hadoop Map/Reduce
Issue Type: Bug
Components: contrib/streaming, test
Reporter: Arun C Murthy
Fix For: 0.21.0
Streaming testcases' usage of MiniDFSCluster.startDatanodes causes NPE in
GenericOptionsParser:
{noformat}
java.lang.NullPointerException
at org.apache.commons.cli.GnuParser.flatten(GnuParser.java:110)
at org.apache.commons.cli.Parser.parse(Parser.java:143)
at
org.apache.hadoop.util.GenericOptionsParser.parseGeneralOptions(GenericOptionsParser.java:374)
at
org.apache.hadoop.util.GenericOptionsParser.(GenericOptionsParser.java:153)
at
org.apache.hadoop.util.GenericOptionsParser.(GenericOptionsParser.java:138)
at
org.apache.hadoop.hdfs.server.datanode.DataNode.instantiateDataNode(DataNode.java:1314)
at
org.apache.hadoop.hdfs.MiniDFSCluster.startDataNodes(MiniDFSCluster.java:414)
at org.apache.hadoop.hdfs.MiniDFSCluster.(MiniDFSCluster.java:278)
at org.apache.hadoop.hdfs.MiniDFSCluster.(MiniDFSCluster.java:119)
at
org.apache.hadoop.streaming.TestDumpTypedBytes.testDumping(TestDumpTypedBytes.java:40)
{noformat}
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
[jira] Commented: (MAPREDUCE-876) Sqoop import of large tables can time out
[ https://issues.apache.org/jira/browse/MAPREDUCE-876?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12745139#action_12745139 ] Aaron Kimball commented on MAPREDUCE-876: - Failures are in capacity scheduler and streaming; unrelated. > Sqoop import of large tables can time out > - > > Key: MAPREDUCE-876 > URL: https://issues.apache.org/jira/browse/MAPREDUCE-876 > Project: Hadoop Map/Reduce > Issue Type: Improvement > Components: contrib/sqoop >Reporter: Aaron Kimball >Assignee: Aaron Kimball > Attachments: MAPREDUCE-876.2.patch, MAPREDUCE-876.patch > > > Related to MAPREDUCE-875, Sqoop should use a background thread to ensure that > progress is being reported while a database does external work for the > MapReduce task. -- This message is automatically generated by JIRA. - You can reply to this email to add a comment to the issue online.
[jira] Commented: (MAPREDUCE-777) A method for finding and tracking jobs from the new API
[ https://issues.apache.org/jira/browse/MAPREDUCE-777?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12745124#action_12745124 ] Hadoop QA commented on MAPREDUCE-777: - -1 overall. Here are the results of testing the latest attachment http://issues.apache.org/jira/secure/attachment/12416974/patch-777-2.txt against trunk revision 805324. +1 @author. The patch does not contain any @author tags. +1 tests included. The patch appears to include 9 new or modified tests. +1 javadoc. The javadoc tool did not generate any warning messages. -1 javac. The applied patch generated 2272 javac compiler warnings (more than the trunk's current 2232 warnings). +1 findbugs. The patch does not introduce any new Findbugs warnings. +1 release audit. The applied patch does not increase the total number of release audit warnings. -1 core tests. The patch failed core unit tests. -1 contrib tests. The patch failed contrib unit tests. Test results: http://hudson.zones.apache.org/hudson/job/Mapreduce-Patch-vesta.apache.org/494/testReport/ Findbugs warnings: http://hudson.zones.apache.org/hudson/job/Mapreduce-Patch-vesta.apache.org/494/artifact/trunk/build/test/findbugs/newPatchFindbugsWarnings.html Checkstyle results: http://hudson.zones.apache.org/hudson/job/Mapreduce-Patch-vesta.apache.org/494/artifact/trunk/build/test/checkstyle-errors.html Console output: http://hudson.zones.apache.org/hudson/job/Mapreduce-Patch-vesta.apache.org/494/console This message is automatically generated. > A method for finding and tracking jobs from the new API > --- > > Key: MAPREDUCE-777 > URL: https://issues.apache.org/jira/browse/MAPREDUCE-777 > Project: Hadoop Map/Reduce > Issue Type: New Feature > Components: client >Reporter: Owen O'Malley >Assignee: Amareshwari Sriramadasu > Fix For: 0.21.0 > > Attachments: patch-777-1.txt, patch-777-2.txt, patch-777.txt > > > We need to create a replacement interface for the JobClient API in the new > interface. In particular, the user needs to be able to query and track jobs > that were launched by other processes. -- This message is automatically generated by JIRA. - You can reply to this email to add a comment to the issue online.
[jira] Created: (MAPREDUCE-890) Afer 4491, the user who started mapred system is not able to run job.
Afer 4491, the user who started mapred system is not able to run job. - Key: MAPREDUCE-890 URL: https://issues.apache.org/jira/browse/MAPREDUCE-890 Project: Hadoop Map/Reduce Issue Type: Bug Reporter: Karam Singh Even setup and cleanup task of job fails due exception -: It fails to create job and related directories under mapred.local.dir/taskTracker/jobcache Directories are created as -: [dr-xrws--- mapred hadoop ] job_200908190916_0002 mapred is not wrtie under this. Even manually I failed to touch file. mapred is use of started mr cluster -- This message is automatically generated by JIRA. - You can reply to this email to add a comment to the issue online.
[jira] Commented: (MAPREDUCE-832) Too many WARN messages about deprecated memorty config variables in JobTacker log
[ https://issues.apache.org/jira/browse/MAPREDUCE-832?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12745014#action_12745014 ] Hadoop QA commented on MAPREDUCE-832: - -1 overall. Here are the results of testing the latest attachment http://issues.apache.org/jira/secure/attachment/12416966/mapreduce-832.patch against trunk revision 805324. +1 @author. The patch does not contain any @author tags. -1 tests included. The patch doesn't appear to include any new or modified tests. Please justify why no new tests are needed for this patch. Also please list what manual steps were performed to verify this patch. +1 javadoc. The javadoc tool did not generate any warning messages. +1 javac. The applied patch does not increase the total number of javac compiler warnings. +1 findbugs. The patch does not introduce any new Findbugs warnings. +1 release audit. The applied patch does not increase the total number of release audit warnings. -1 core tests. The patch failed core unit tests. -1 contrib tests. The patch failed contrib unit tests. Test results: http://hudson.zones.apache.org/hudson/job/Mapreduce-Patch-vesta.apache.org/493/testReport/ Findbugs warnings: http://hudson.zones.apache.org/hudson/job/Mapreduce-Patch-vesta.apache.org/493/artifact/trunk/build/test/findbugs/newPatchFindbugsWarnings.html Checkstyle results: http://hudson.zones.apache.org/hudson/job/Mapreduce-Patch-vesta.apache.org/493/artifact/trunk/build/test/checkstyle-errors.html Console output: http://hudson.zones.apache.org/hudson/job/Mapreduce-Patch-vesta.apache.org/493/console This message is automatically generated. > Too many WARN messages about deprecated memorty config variables in JobTacker > log > - > > Key: MAPREDUCE-832 > URL: https://issues.apache.org/jira/browse/MAPREDUCE-832 > Project: Hadoop Map/Reduce > Issue Type: Bug >Affects Versions: 0.20.1 >Reporter: Karam Singh >Assignee: rahul k singh > Attachments: mapreduce-832-20.patch, mapreduce-832.patch > > > When user submit a mapred job using old memory config vairiable > (mapred.task.maxmem) followinig message too many times in JobTracker logs -: > [ > WARN org.apache.hadoop.mapred.JobConf: The variable mapred.task.maxvmem is no > longer used instead use mapred.job.map.memory.mb and > mapred.job.reduce.memory.mb > ] -- This message is automatically generated by JIRA. - You can reply to this email to add a comment to the issue online.
[jira] Created: (MAPREDUCE-889) binary communication formats added to Streaming by HADOOP-1722 should be documented
binary communication formats added to Streaming by HADOOP-1722 should be documented --- Key: MAPREDUCE-889 URL: https://issues.apache.org/jira/browse/MAPREDUCE-889 Project: Hadoop Map/Reduce Issue Type: Bug Components: documentation Reporter: Amareshwari Sriramadasu Priority: Blocker Fix For: 0.21.0 binary communication formats added to Streaming by HADOOP-1722 should be documented in forrest -- This message is automatically generated by JIRA. - You can reply to this email to add a comment to the issue online.
[jira] Resolved: (MAPREDUCE-888) TestJobHistory sometimes fails while validating history.
[ https://issues.apache.org/jira/browse/MAPREDUCE-888?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Amar Kamat resolved MAPREDUCE-888. -- Resolution: Duplicate MAPREDUCE-882 > TestJobHistory sometimes fails while validating history. > > > Key: MAPREDUCE-888 > URL: https://issues.apache.org/jira/browse/MAPREDUCE-888 > Project: Hadoop Map/Reduce > Issue Type: Bug > Components: test >Reporter: Amar Kamat > > Error Message > Duplicate START_TIME seen for task task_200908190021_0001_m_03 in history > file at line 5 > Stacktrace > junit.framework.AssertionFailedError: Duplicate START_TIME seen for task > task_200908190021_0001_m_03 in history file at line 5 > at > org.apache.hadoop.mapred.TestJobHistory$TestListener.handle(TestJobHistory.java:163) > at org.apache.hadoop.mapred.JobHistory.parseLine(JobHistory.java:497) > at > org.apache.hadoop.mapred.JobHistory.parseHistoryFromFS(JobHistory.java:463) > at > org.apache.hadoop.mapred.TestJobHistory.validateJobHistoryFileFormat(TestJobHistory.java:486) > at > org.apache.hadoop.mapred.TestJobHistory.testJobHistoryFile(TestJobHistory.java:955) -- This message is automatically generated by JIRA. - You can reply to this email to add a comment to the issue online.
[jira] Created: (MAPREDUCE-888) TestJobHistory sometimes fails while validating history.
TestJobHistory sometimes fails while validating history. Key: MAPREDUCE-888 URL: https://issues.apache.org/jira/browse/MAPREDUCE-888 Project: Hadoop Map/Reduce Issue Type: Bug Components: test Reporter: Amar Kamat Error Message Duplicate START_TIME seen for task task_200908190021_0001_m_03 in history file at line 5 Stacktrace junit.framework.AssertionFailedError: Duplicate START_TIME seen for task task_200908190021_0001_m_03 in history file at line 5 at org.apache.hadoop.mapred.TestJobHistory$TestListener.handle(TestJobHistory.java:163) at org.apache.hadoop.mapred.JobHistory.parseLine(JobHistory.java:497) at org.apache.hadoop.mapred.JobHistory.parseHistoryFromFS(JobHistory.java:463) at org.apache.hadoop.mapred.TestJobHistory.validateJobHistoryFileFormat(TestJobHistory.java:486) at org.apache.hadoop.mapred.TestJobHistory.testJobHistoryFile(TestJobHistory.java:955) -- This message is automatically generated by JIRA. - You can reply to this email to add a comment to the issue online.
[jira] Resolved: (MAPREDUCE-27) Jobs with 0 maps will never get removed from the default scheduler
[
https://issues.apache.org/jira/browse/MAPREDUCE-27?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Amar Kamat resolved MAPREDUCE-27.
-
Resolution: Duplicate
> Jobs with 0 maps will never get removed from the default scheduler
> --
>
> Key: MAPREDUCE-27
> URL: https://issues.apache.org/jira/browse/MAPREDUCE-27
> Project: Hadoop Map/Reduce
> Issue Type: Bug
>Reporter: Amar Kamat
>Assignee: Amar Kamat
> Attachments: HADOOP-5049-v1.1.patch
>
>
> Jobs' with 0 maps finish/succeed in the init phase i.e while the job is in
> the _PREP_ state. {{EagerTaskInitializationListener}} removes the job after
> initing but {{JobQueueJobInProgressListener}} waits for a job-state change
> event to be raised and aonly then removes the job from the queue and hence
> the job will stay forever with the {{JobQueueJobInProgressListener}}. Looks
> like {{FairScheduler}} periodically scans the job list and removes completed
> jobs. {{CapacityScheduler}} has a concept of waiting jobs and scans waiting
> queue for completed jobs and purges them.
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
[jira] Updated: (MAPREDUCE-849) Renaming of configuration property names in mapreduce
[ https://issues.apache.org/jira/browse/MAPREDUCE-849?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Amareshwari Sriramadasu updated MAPREDUCE-849: -- Attachment: Config changes.xls Attaching the document listing oldname and newname with proposed naming convention. > Renaming of configuration property names in mapreduce > - > > Key: MAPREDUCE-849 > URL: https://issues.apache.org/jira/browse/MAPREDUCE-849 > Project: Hadoop Map/Reduce > Issue Type: Improvement >Reporter: Amareshwari Sriramadasu >Assignee: Amareshwari Sriramadasu > Fix For: 0.21.0 > > Attachments: Config changes.xls > > > In-line with HDFS-531, property names in configuration files should be > standardized in MAPREDUCE. -- This message is automatically generated by JIRA. - You can reply to this email to add a comment to the issue online.
[jira] Commented: (MAPREDUCE-430) Task stuck in cleanup with OutOfMemoryErrors
[ https://issues.apache.org/jira/browse/MAPREDUCE-430?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12744986#action_12744986 ] Amar Kamat commented on MAPREDUCE-430: -- bq. ant tests (core + core) passed except TestJobHistory. I meant core + contrib > Task stuck in cleanup with OutOfMemoryErrors > > > Key: MAPREDUCE-430 > URL: https://issues.apache.org/jira/browse/MAPREDUCE-430 > Project: Hadoop Map/Reduce > Issue Type: Bug >Reporter: Amareshwari Sriramadasu >Assignee: Amar Kamat > Fix For: 0.20.1 > > Attachments: MAPREDUCE-430-v1.6-branch-0.20.patch, > MAPREDUCE-430-v1.6.patch, MAPREDUCE-430-v1.7.patch, MAPREDUCE-430-v1.8.patch > > > Obesrved a task with OutOfMemory error, stuck in cleanup. -- This message is automatically generated by JIRA. - You can reply to this email to add a comment to the issue online.
[jira] Commented: (MAPREDUCE-430) Task stuck in cleanup with OutOfMemoryErrors
[ https://issues.apache.org/jira/browse/MAPREDUCE-430?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12744985#action_12744985 ] Amar Kamat commented on MAPREDUCE-430: -- ant tests (core + core) passed except TestJobHistory. > Task stuck in cleanup with OutOfMemoryErrors > > > Key: MAPREDUCE-430 > URL: https://issues.apache.org/jira/browse/MAPREDUCE-430 > Project: Hadoop Map/Reduce > Issue Type: Bug >Reporter: Amareshwari Sriramadasu >Assignee: Amar Kamat > Fix For: 0.20.1 > > Attachments: MAPREDUCE-430-v1.6-branch-0.20.patch, > MAPREDUCE-430-v1.6.patch, MAPREDUCE-430-v1.7.patch, MAPREDUCE-430-v1.8.patch > > > Obesrved a task with OutOfMemory error, stuck in cleanup. -- This message is automatically generated by JIRA. - You can reply to this email to add a comment to the issue online.
[jira] Commented: (MAPREDUCE-336) The logging level of the tasks should be configurable by the job
[ https://issues.apache.org/jira/browse/MAPREDUCE-336?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12744960#action_12744960 ] Hadoop QA commented on MAPREDUCE-336: - -1 overall. Here are the results of testing the latest attachment http://issues.apache.org/jira/secure/attachment/12416955/MAPREDUCE-336_0_20090818.patch against trunk revision 805324. +1 @author. The patch does not contain any @author tags. +1 tests included. The patch appears to include 3 new or modified tests. +1 javadoc. The javadoc tool did not generate any warning messages. +1 javac. The applied patch does not increase the total number of javac compiler warnings. +1 findbugs. The patch does not introduce any new Findbugs warnings. +1 release audit. The applied patch does not increase the total number of release audit warnings. -1 core tests. The patch failed core unit tests. -1 contrib tests. The patch failed contrib unit tests. Test results: http://hudson.zones.apache.org/hudson/job/Mapreduce-Patch-vesta.apache.org/492/testReport/ Findbugs warnings: http://hudson.zones.apache.org/hudson/job/Mapreduce-Patch-vesta.apache.org/492/artifact/trunk/build/test/findbugs/newPatchFindbugsWarnings.html Checkstyle results: http://hudson.zones.apache.org/hudson/job/Mapreduce-Patch-vesta.apache.org/492/artifact/trunk/build/test/checkstyle-errors.html Console output: http://hudson.zones.apache.org/hudson/job/Mapreduce-Patch-vesta.apache.org/492/console This message is automatically generated. > The logging level of the tasks should be configurable by the job > > > Key: MAPREDUCE-336 > URL: https://issues.apache.org/jira/browse/MAPREDUCE-336 > Project: Hadoop Map/Reduce > Issue Type: Improvement >Reporter: Owen O'Malley >Assignee: Arun C Murthy > Fix For: 0.21.0 > > Attachments: MAPREDUCE-336_0_20090818.patch > > > It would be nice to be able to configure the logging level of the Task JVM's > separately from the server JVM's. Reducing logging substantially increases > performance and reduces the consumption of local disk on the task trackers. -- This message is automatically generated by JIRA. - You can reply to this email to add a comment to the issue online.
[jira] Updated: (MAPREDUCE-516) Fix the 'cluster drain' problem in the Capacity Scheduler wrt High RAM Jobs
[ https://issues.apache.org/jira/browse/MAPREDUCE-516?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Sreekanth Ramakrishnan updated MAPREDUCE-516: - Attachment: mr516-ydist.patch Attaching Yahoo! distribution patch. > Fix the 'cluster drain' problem in the Capacity Scheduler wrt High RAM Jobs > --- > > Key: MAPREDUCE-516 > URL: https://issues.apache.org/jira/browse/MAPREDUCE-516 > Project: Hadoop Map/Reduce > Issue Type: Bug >Affects Versions: 0.20.1 >Reporter: Arun C Murthy >Assignee: Arun C Murthy > Fix For: 0.21.0 > > Attachments: HADOOP-5964-11.patch, HADOOP-5964-12.patch, > HADOOP-5964_0_20090602.patch, HADOOP-5964_10_20090622.patch, > HADOOP-5964_11_20090623.patch, HADOOP-5964_1_20090608.patch, > HADOOP-5964_1_20090623_yhadoop.patch, HADOOP-5964_2_20090609.patch, > HADOOP-5964_2_20090629_yhadoop.patch, HADOOP-5964_2_20090629_yhadoop.patch, > HADOOP-5964_4_20090615.patch, HADOOP-5964_6_20090617.patch, > HADOOP-5964_7_20090618.patch, HADOOP-5964_8_20090618.patch, > HADOOP-5964_9_20090619.patch, HADOOP-5964_9_20090619.patch, > MAPREDUCE-516-Y20.patch, mr516-ydist.patch > > > When a HighRAMJob turns up at the head of the queue, the current > implementation of support for HighRAMJobs in the Capacity Scheduler has > problem in that the scheduler stops assigning tasks to all TaskTrackers in > the cluster until a HighRAMJob finds a suitable TaskTrackers for all its > tasks. > This causes a severe utilization problem since effectively no new tasks are > allowed to run until the HighRAMJob (at the head of the queue) gets slots. -- This message is automatically generated by JIRA. - You can reply to this email to add a comment to the issue online.
[jira] Created: (MAPREDUCE-887) After 4491, task cleaup directory some gets created under the ownershiptasktracker user instread job submitting.
After 4491, task cleaup directory some gets created under the ownershiptasktracker user instread job submitting. Key: MAPREDUCE-887 URL: https://issues.apache.org/jira/browse/MAPREDUCE-887 Project: Hadoop Map/Reduce Issue Type: Bug Components: tasktracker Affects Versions: 0.21.0 Reporter: Karam Singh Some time, when task is killed, task cleanup directory is created under the ownership tasktracker launching user instead job submitting user. dr-xrws--- karams hadoop ] job_200908170914_0020 |-- [drwxr-sr-x mapred hadoop ] attempt_200908170914_0020_m_02_0.cleanup `-- [drwxrws--- karams hadoop ] attempt_200908170914_0020_m_12_0 Here karams is user who submitted job and mapred is the use who launched TT. taskattrempt.cleanup created with mapred user not with karams user. This issue is intermittent, not always reproducible. -- This message is automatically generated by JIRA. - You can reply to this email to add a comment to the issue online.
[jira] Created: (MAPREDUCE-886) After 4491, when task-controller exit with some error message, LinuxTaskController only ExitCodeException but does not prints the exit code of task-controller
After 4491, when task-controller exit with some error message, LinuxTaskController only ExitCodeException but does not prints the exit code of task-controller -- Key: MAPREDUCE-886 URL: https://issues.apache.org/jira/browse/MAPREDUCE-886 Project: Hadoop Map/Reduce Issue Type: Bug Components: tasktracker Affects Versions: 0.21.0 Reporter: Karam Singh -- This message is automatically generated by JIRA. - You can reply to this email to add a comment to the issue online.
[jira] Commented: (MAPREDUCE-181) mapred.system.dir should be accessible only to hadoop daemons
[
https://issues.apache.org/jira/browse/MAPREDUCE-181?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12744944#action_12744944
]
Devaraj Das commented on MAPREDUCE-181:
---
Some more details on the split file handling:
1) The FileSystem used for writing the split bytes would be the same filesystem
where mapred.system.dir is located.
2) The split info (actual split bytes) would get written to the user's home
directory on that filesystem (e.g., /user//.mapreduce/jobid)
3) The split info can be cleaned up by the cleanup task of the job.
For now, let's postpone the special handling for the JobConf, and instead put a
cap on the max size (like 1 MB).
> mapred.system.dir should be accessible only to hadoop daemons
> --
>
> Key: MAPREDUCE-181
> URL: https://issues.apache.org/jira/browse/MAPREDUCE-181
> Project: Hadoop Map/Reduce
> Issue Type: Bug
>Reporter: Amar Kamat
>Assignee: Amar Kamat
> Attachments: hadoop-3578-branch-20-example-2.patch,
> hadoop-3578-branch-20-example.patch, HADOOP-3578-v2.6.patch,
> HADOOP-3578-v2.7.patch
>
>
> Currently the jobclient accesses the {{mapred.system.dir}} to add job
> details. Hence the {{mapred.system.dir}} has the permissions of
> {{rwx-wx-wx}}. This could be a security loophole where the job files might
> get overwritten/tampered after the job submission.
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
[jira] Updated: (MAPREDUCE-834) When TaskTracker config use old memory management values its memory monitoring is diabled.
[ https://issues.apache.org/jira/browse/MAPREDUCE-834?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Sreekanth Ramakrishnan updated MAPREDUCE-834: - Attachment: mapreduce-834-2.patch Attaching patch fixing using the correct key and converting the same to MB while allotting total memory alloted to the tasks. > When TaskTracker config use old memory management values its memory > monitoring is diabled. > -- > > Key: MAPREDUCE-834 > URL: https://issues.apache.org/jira/browse/MAPREDUCE-834 > Project: Hadoop Map/Reduce > Issue Type: Bug >Reporter: Karam Singh > Attachments: mapreduce-834-1.patch, mapreduce-834-2.patch > > > TaskTracker memory config values -: > mapred.tasktracker.vmem.reserved=8589934592 > mapred.task.default.maxvmem=2147483648 > mapred.task.limit.maxvmem=4294967296 > mapred.tasktracker.pmem.reserved=2147483648 > TaskTracker start as -: >2009-08-05 12:39:03,308 WARN > org.apache.hadoop.mapred.TaskTracker: The variable > mapred.tasktracker.vmem.reserved is no longer used > 2009-08-05 12:39:03,308 WARN > org.apache.hadoop.mapred.TaskTracker: The variable > mapred.tasktracker.pmem.reserved is no longer used > 2009-08-05 12:39:03,308 WARN > org.apache.hadoop.mapred.TaskTracker: The variable > mapred.task.default.maxvmem is no longer used > 2009-08-05 12:39:03,308 WARN > org.apache.hadoop.mapred.TaskTracker: The variable mapred.task.limit.maxvmem > is no longer used > 2009-08-05 12:39:03,308 INFO > org.apache.hadoop.mapred.TaskTracker: Starting thread: Map-events fetcher for > all reduce tasks on > 2009-08-05 12:39:03,309 INFO > org.apache.hadoop.mapred.TaskTracker: Using MemoryCalculatorPlugin : > org.apache.hadoop.util.linuxmemorycalculatorplu...@19be4777 > 2009-08-05 12:39:03,311 WARN > org.apache.hadoop.mapred.TaskTracker: TaskTracker's > totalMemoryAllottedForTasks is -1. TaskMemoryManager is disabled. -- This message is automatically generated by JIRA. - You can reply to this email to add a comment to the issue online.
[jira] Updated: (MAPREDUCE-430) Task stuck in cleanup with OutOfMemoryErrors
[ https://issues.apache.org/jira/browse/MAPREDUCE-430?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Amar Kamat updated MAPREDUCE-430: - Attachment: MAPREDUCE-430-v1.8.patch Attaching a patch that avoid finally being called upon errors. Result of test-patch [exec] +1 overall. [exec] [exec] +1 @author. The patch does not contain any @author tags. [exec] [exec] +1 tests included. The patch appears to include 3 new or modified tests. [exec] [exec] +1 javadoc. The javadoc tool did not generate any warning messages. [exec] [exec] +1 javac. The applied patch does not increase the total number of javac compiler warnings. [exec] [exec] +1 findbugs. The patch does not introduce any new Findbugs warnings. [exec] [exec] +1 release audit. The applied patch does not increase the total number of release audit warnings. Testing the patch. > Task stuck in cleanup with OutOfMemoryErrors > > > Key: MAPREDUCE-430 > URL: https://issues.apache.org/jira/browse/MAPREDUCE-430 > Project: Hadoop Map/Reduce > Issue Type: Bug >Reporter: Amareshwari Sriramadasu >Assignee: Amar Kamat > Fix For: 0.20.1 > > Attachments: MAPREDUCE-430-v1.6-branch-0.20.patch, > MAPREDUCE-430-v1.6.patch, MAPREDUCE-430-v1.7.patch, MAPREDUCE-430-v1.8.patch > > > Obesrved a task with OutOfMemory error, stuck in cleanup. -- This message is automatically generated by JIRA. - You can reply to this email to add a comment to the issue online.
