[jira] Updated: (MAPREDUCE-777) A method for finding and tracking jobs from the new API
[ https://issues.apache.org/jira/browse/MAPREDUCE-777?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Amareshwari Sriramadasu updated MAPREDUCE-777: -- Attachment: patch-777-12.txt Patch updated with the trunk > A method for finding and tracking jobs from the new API > --- > > Key: MAPREDUCE-777 > URL: https://issues.apache.org/jira/browse/MAPREDUCE-777 > Project: Hadoop Map/Reduce > Issue Type: New Feature > Components: client >Reporter: Owen O'Malley >Assignee: Amareshwari Sriramadasu > Fix For: 0.21.0 > > Attachments: m-777.patch, patch-777-1.txt, patch-777-10.txt, > patch-777-11.txt, patch-777-12.txt, patch-777-2.txt, patch-777-3.txt, > patch-777-4.txt, patch-777-5.txt, patch-777-6.txt, patch-777-7.txt, > patch-777-8.txt, patch-777-9.txt, patch-777.txt > > > We need to create a replacement interface for the JobClient API in the new > interface. In particular, the user needs to be able to query and track jobs > that were launched by other processes. -- This message is automatically generated by JIRA. - You can reply to this email to add a comment to the issue online.
[jira] Updated: (MAPREDUCE-988) ant package does not copy the capacity-scheduler.jar under HADOOP_HOME/build/hadoop-mapred-0.21.0-dev/contrib/capacity-scheduler
[ https://issues.apache.org/jira/browse/MAPREDUCE-988?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Hong Tang updated MAPREDUCE-988: Component/s: (was: contrib/capacity-sched) build > ant package does not copy the capacity-scheduler.jar under > HADOOP_HOME/build/hadoop-mapred-0.21.0-dev/contrib/capacity-scheduler > > > Key: MAPREDUCE-988 > URL: https://issues.apache.org/jira/browse/MAPREDUCE-988 > Project: Hadoop Map/Reduce > Issue Type: Bug > Components: build >Affects Versions: 0.21.0 >Reporter: Iyappan Srinivasan >Assignee: Hong Tang > Fix For: 0.21.0 > > Attachments: mapreduce-988-20090916.patch > > > ant package does not copy the hadoop-0.21.0-dev-capacity-scheduler.jar under > HADOOP_HOME/build/hadoop-mapred-0.21.0-dev/contrib/capacity-scheduler/. > Till yesterday it was copying it properly. Issue seems to be pointing to the > latest checkin of MAPREDUCE-776, which changes build.xml. -- This message is automatically generated by JIRA. - You can reply to this email to add a comment to the issue online.
[jira] Updated: (MAPREDUCE-988) ant package does not copy the capacity-scheduler.jar under HADOOP_HOME/build/hadoop-mapred-0.21.0-dev/contrib/capacity-scheduler
[ https://issues.apache.org/jira/browse/MAPREDUCE-988?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Hong Tang updated MAPREDUCE-988: Status: Patch Available (was: Open) Foreach target (package, bin-package, binary, tar), I ran "ant clean" followed by "ant $target", and hadoop-0.21.0-dev-capacity-scheduler.jar appears under build/hadoop-mapred-0.21.0-dev/contrib/capacity-scheduler. No test included as the fix touches the build system. > ant package does not copy the capacity-scheduler.jar under > HADOOP_HOME/build/hadoop-mapred-0.21.0-dev/contrib/capacity-scheduler > > > Key: MAPREDUCE-988 > URL: https://issues.apache.org/jira/browse/MAPREDUCE-988 > Project: Hadoop Map/Reduce > Issue Type: Bug > Components: build >Reporter: Iyappan Srinivasan >Assignee: Hong Tang > Attachments: mapreduce-988-20090916.patch > > > ant package does not copy the hadoop-0.21.0-dev-capacity-scheduler.jar under > HADOOP_HOME/build/hadoop-mapred-0.21.0-dev/contrib/capacity-scheduler/. > Till yesterday it was copying it properly. Issue seems to be pointing to the > latest checkin of MAPREDUCE-776, which changes build.xml. -- This message is automatically generated by JIRA. - You can reply to this email to add a comment to the issue online.
[jira] Updated: (MAPREDUCE-988) ant package does not copy the capacity-scheduler.jar under HADOOP_HOME/build/hadoop-mapred-0.21.0-dev/contrib/capacity-scheduler
[ https://issues.apache.org/jira/browse/MAPREDUCE-988?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Hong Tang updated MAPREDUCE-988: Affects Version/s: 0.21.0 Fix Version/s: 0.21.0 > ant package does not copy the capacity-scheduler.jar under > HADOOP_HOME/build/hadoop-mapred-0.21.0-dev/contrib/capacity-scheduler > > > Key: MAPREDUCE-988 > URL: https://issues.apache.org/jira/browse/MAPREDUCE-988 > Project: Hadoop Map/Reduce > Issue Type: Bug > Components: build >Affects Versions: 0.21.0 >Reporter: Iyappan Srinivasan >Assignee: Hong Tang > Fix For: 0.21.0 > > Attachments: mapreduce-988-20090916.patch > > > ant package does not copy the hadoop-0.21.0-dev-capacity-scheduler.jar under > HADOOP_HOME/build/hadoop-mapred-0.21.0-dev/contrib/capacity-scheduler/. > Till yesterday it was copying it properly. Issue seems to be pointing to the > latest checkin of MAPREDUCE-776, which changes build.xml. -- This message is automatically generated by JIRA. - You can reply to this email to add a comment to the issue online.
[jira] Updated: (MAPREDUCE-777) A method for finding and tracking jobs from the new API
[
https://issues.apache.org/jira/browse/MAPREDUCE-777?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Arun C Murthy updated MAPREDUCE-777:
Status: Open (was: Patch Available)
This is close...
# We should at least open a new jira for more tests: TestCluster, TestJob etc.,
I see you've converted TestJobClient
# o.a.h.mapreduce.JobStatus.{RUNNING|SUCCEEDED|...} etc. shud be an enum and it
should have a 'int getValue()' which returns values compatible with
o.a.h.mapred.JobStatus.{RUNNING|SUCCEEDED|...}
# Typo: arreyToBlackListInfo
# Job has a copy-paste javadoc which mentions NetworkedJob
> A method for finding and tracking jobs from the new API
> ---
>
> Key: MAPREDUCE-777
> URL: https://issues.apache.org/jira/browse/MAPREDUCE-777
> Project: Hadoop Map/Reduce
> Issue Type: New Feature
> Components: client
>Reporter: Owen O'Malley
>Assignee: Amareshwari Sriramadasu
> Fix For: 0.21.0
>
> Attachments: m-777.patch, patch-777-1.txt, patch-777-10.txt,
> patch-777-11.txt, patch-777-12.txt, patch-777-2.txt, patch-777-3.txt,
> patch-777-4.txt, patch-777-5.txt, patch-777-6.txt, patch-777-7.txt,
> patch-777-8.txt, patch-777-9.txt, patch-777.txt
>
>
> We need to create a replacement interface for the JobClient API in the new
> interface. In particular, the user needs to be able to query and track jobs
> that were launched by other processes.
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
[jira] Commented: (MAPREDUCE-777) A method for finding and tracking jobs from the new API
[ https://issues.apache.org/jira/browse/MAPREDUCE-777?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12756403#action_12756403 ] Arun C Murthy commented on MAPREDUCE-777: - 5. Queue.getJobQueueInfo calls Enum.name() (state.name()), it should use Enum.toString() ? > A method for finding and tracking jobs from the new API > --- > > Key: MAPREDUCE-777 > URL: https://issues.apache.org/jira/browse/MAPREDUCE-777 > Project: Hadoop Map/Reduce > Issue Type: New Feature > Components: client >Reporter: Owen O'Malley >Assignee: Amareshwari Sriramadasu > Fix For: 0.21.0 > > Attachments: m-777.patch, patch-777-1.txt, patch-777-10.txt, > patch-777-11.txt, patch-777-12.txt, patch-777-2.txt, patch-777-3.txt, > patch-777-4.txt, patch-777-5.txt, patch-777-6.txt, patch-777-7.txt, > patch-777-8.txt, patch-777-9.txt, patch-777.txt > > > We need to create a replacement interface for the JobClient API in the new > interface. In particular, the user needs to be able to query and track jobs > that were launched by other processes. -- This message is automatically generated by JIRA. - You can reply to this email to add a comment to the issue online.
[jira] Updated: (MAPREDUCE-907) Sqoop should use more intelligent splits
[ https://issues.apache.org/jira/browse/MAPREDUCE-907?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Aaron Kimball updated MAPREDUCE-907: Status: Patch Available (was: Open) Canceled patch on wrong issue by mistake. > Sqoop should use more intelligent splits > > > Key: MAPREDUCE-907 > URL: https://issues.apache.org/jira/browse/MAPREDUCE-907 > Project: Hadoop Map/Reduce > Issue Type: Improvement > Components: contrib/sqoop >Reporter: Aaron Kimball >Assignee: Aaron Kimball > Attachments: MAPREDUCE-907.2.patch, MAPREDUCE-907.patch > > > Sqoop should use the new split generation / InputFormat in MAPREDUCE-885 -- This message is automatically generated by JIRA. - You can reply to this email to add a comment to the issue online.
[jira] Updated: (MAPREDUCE-907) Sqoop should use more intelligent splits
[ https://issues.apache.org/jira/browse/MAPREDUCE-907?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Aaron Kimball updated MAPREDUCE-907: Status: Open (was: Patch Available) > Sqoop should use more intelligent splits > > > Key: MAPREDUCE-907 > URL: https://issues.apache.org/jira/browse/MAPREDUCE-907 > Project: Hadoop Map/Reduce > Issue Type: Improvement > Components: contrib/sqoop >Reporter: Aaron Kimball >Assignee: Aaron Kimball > Attachments: MAPREDUCE-907.2.patch, MAPREDUCE-907.patch > > > Sqoop should use the new split generation / InputFormat in MAPREDUCE-885 -- This message is automatically generated by JIRA. - You can reply to this email to add a comment to the issue online.
[jira] Updated: (MAPREDUCE-906) Updated Sqoop documentation
[ https://issues.apache.org/jira/browse/MAPREDUCE-906?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Aaron Kimball updated MAPREDUCE-906: Status: Open (was: Patch Available) > Updated Sqoop documentation > --- > > Key: MAPREDUCE-906 > URL: https://issues.apache.org/jira/browse/MAPREDUCE-906 > Project: Hadoop Map/Reduce > Issue Type: Improvement > Components: contrib/sqoop >Reporter: Aaron Kimball >Assignee: Aaron Kimball > Attachments: MAPREDUCE-906.2.patch, MAPREDUCE-906.3.patch, > MAPREDUCE-906.4.patch, MAPREDUCE-906.patch > > > Here's the latest documentation for Sqoop, in both user-guide and manpage > form. Built with asciidoc. -- This message is automatically generated by JIRA. - You can reply to this email to add a comment to the issue online.
[jira] Updated: (MAPREDUCE-946) Fix regression in LineRecordReader to comply with line length parameters
[ https://issues.apache.org/jira/browse/MAPREDUCE-946?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Chris Douglas updated MAPREDUCE-946: Resolution: Fixed Hadoop Flags: [Reviewed] Status: Resolved (was: Patch Available) I committed this. > Fix regression in LineRecordReader to comply with line length parameters > > > Key: MAPREDUCE-946 > URL: https://issues.apache.org/jira/browse/MAPREDUCE-946 > Project: Hadoop Map/Reduce > Issue Type: Bug >Reporter: Chris Douglas >Assignee: Chris Douglas >Priority: Blocker > Fix For: 0.21.0 > > Attachments: M946-0.patch, M946-1.patch > > > MAPREDUCE-773 accidentally changed code introduced in HADOOP-3144 controlling > max line lengths. The behavior should be restored. -- This message is automatically generated by JIRA. - You can reply to this email to add a comment to the issue online.
[jira] Commented: (MAPREDUCE-990) Making distributed cache getters in JobContext never return null
[ https://issues.apache.org/jira/browse/MAPREDUCE-990?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12756415#action_12756415 ] Hadoop QA commented on MAPREDUCE-990: - +1 overall. Here are the results of testing the latest attachment http://issues.apache.org/jira/secure/attachment/12419796/MAPREDUCE-990.patch.txt against trunk revision 816052. +1 @author. The patch does not contain any @author tags. -1 tests included. The patch doesn't appear to include any new or modified tests. Please justify why no new tests are needed for this patch. Also please list what manual steps were performed to verify this patch. +1 javadoc. The javadoc tool did not generate any warning messages. +1 javac. The applied patch does not increase the total number of javac compiler warnings. +1 findbugs. The patch does not introduce any new Findbugs warnings. +1 release audit. The applied patch does not increase the total number of release audit warnings. +1 core tests. The patch passed core unit tests. +1 contrib tests. The patch passed contrib unit tests. Test results: http://hudson.zones.apache.org/hudson/job/Mapreduce-Patch-h3.grid.sp2.yahoo.net/41/testReport/ Findbugs warnings: http://hudson.zones.apache.org/hudson/job/Mapreduce-Patch-h3.grid.sp2.yahoo.net/41/artifact/trunk/build/test/findbugs/newPatchFindbugsWarnings.html Checkstyle results: http://hudson.zones.apache.org/hudson/job/Mapreduce-Patch-h3.grid.sp2.yahoo.net/41/artifact/trunk/build/test/checkstyle-errors.html Console output: http://hudson.zones.apache.org/hudson/job/Mapreduce-Patch-h3.grid.sp2.yahoo.net/41/console This message is automatically generated. > Making distributed cache getters in JobContext never return null > > > Key: MAPREDUCE-990 > URL: https://issues.apache.org/jira/browse/MAPREDUCE-990 > Project: Hadoop Map/Reduce > Issue Type: Improvement >Reporter: Philip Zeyliger >Assignee: Philip Zeyliger >Priority: Minor > Attachments: MAPREDUCE-990.patch.txt > > > MAPREDUCE-898 moved distributed cache setters and getters into Job and > JobContext. Since the API is new, I'd like to propose that those getters > never return null, but instead always return an array, even if it's empty. > If people don't like this change, I can instead merely update the javadoc to > reflect the fact that null may be returned. -- This message is automatically generated by JIRA. - You can reply to this email to add a comment to the issue online.
[jira] Updated: (MAPREDUCE-777) A method for finding and tracking jobs from the new API
[ https://issues.apache.org/jira/browse/MAPREDUCE-777?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Amareshwari Sriramadasu updated MAPREDUCE-777: -- Status: Patch Available (was: Open) > A method for finding and tracking jobs from the new API > --- > > Key: MAPREDUCE-777 > URL: https://issues.apache.org/jira/browse/MAPREDUCE-777 > Project: Hadoop Map/Reduce > Issue Type: New Feature > Components: client >Reporter: Owen O'Malley >Assignee: Amareshwari Sriramadasu > Fix For: 0.21.0 > > Attachments: m-777.patch, patch-777-1.txt, patch-777-10.txt, > patch-777-11.txt, patch-777-12.txt, patch-777-13.txt, patch-777-2.txt, > patch-777-3.txt, patch-777-4.txt, patch-777-5.txt, patch-777-6.txt, > patch-777-7.txt, patch-777-8.txt, patch-777-9.txt, patch-777.txt > > > We need to create a replacement interface for the JobClient API in the new > interface. In particular, the user needs to be able to query and track jobs > that were launched by other processes. -- This message is automatically generated by JIRA. - You can reply to this email to add a comment to the issue online.
[jira] Updated: (MAPREDUCE-777) A method for finding and tracking jobs from the new API
[ https://issues.apache.org/jira/browse/MAPREDUCE-777?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Amareshwari Sriramadasu updated MAPREDUCE-777: -- Attachment: patch-777-13.txt Patch updated with the comments > A method for finding and tracking jobs from the new API > --- > > Key: MAPREDUCE-777 > URL: https://issues.apache.org/jira/browse/MAPREDUCE-777 > Project: Hadoop Map/Reduce > Issue Type: New Feature > Components: client >Reporter: Owen O'Malley >Assignee: Amareshwari Sriramadasu > Fix For: 0.21.0 > > Attachments: m-777.patch, patch-777-1.txt, patch-777-10.txt, > patch-777-11.txt, patch-777-12.txt, patch-777-13.txt, patch-777-2.txt, > patch-777-3.txt, patch-777-4.txt, patch-777-5.txt, patch-777-6.txt, > patch-777-7.txt, patch-777-8.txt, patch-777-9.txt, patch-777.txt > > > We need to create a replacement interface for the JobClient API in the new > interface. In particular, the user needs to be able to query and track jobs > that were launched by other processes. -- This message is automatically generated by JIRA. - You can reply to this email to add a comment to the issue online.
[jira] Commented: (MAPREDUCE-862) Modify UI to support a hierarchy of queues
[ https://issues.apache.org/jira/browse/MAPREDUCE-862?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12756421#action_12756421 ] Hadoop QA commented on MAPREDUCE-862: - +1 overall. Here are the results of testing the latest attachment http://issues.apache.org/jira/secure/attachment/12419845/MAPREDUCE-862-7.patch against trunk revision 816052. +1 @author. The patch does not contain any @author tags. +1 tests included. The patch appears to include 3 new or modified tests. +1 javadoc. The javadoc tool did not generate any warning messages. +1 javac. The applied patch does not increase the total number of javac compiler warnings. +1 findbugs. The patch does not introduce any new Findbugs warnings. +1 release audit. The applied patch does not increase the total number of release audit warnings. +1 core tests. The patch passed core unit tests. +1 contrib tests. The patch passed contrib unit tests. Test results: http://hudson.zones.apache.org/hudson/job/Mapreduce-Patch-h6.grid.sp2.yahoo.net/94/testReport/ Findbugs warnings: http://hudson.zones.apache.org/hudson/job/Mapreduce-Patch-h6.grid.sp2.yahoo.net/94/artifact/trunk/build/test/findbugs/newPatchFindbugsWarnings.html Checkstyle results: http://hudson.zones.apache.org/hudson/job/Mapreduce-Patch-h6.grid.sp2.yahoo.net/94/artifact/trunk/build/test/checkstyle-errors.html Console output: http://hudson.zones.apache.org/hudson/job/Mapreduce-Patch-h6.grid.sp2.yahoo.net/94/console This message is automatically generated. > Modify UI to support a hierarchy of queues > -- > > Key: MAPREDUCE-862 > URL: https://issues.apache.org/jira/browse/MAPREDUCE-862 > Project: Hadoop Map/Reduce > Issue Type: Sub-task >Reporter: Hemanth Yamijala >Assignee: V.V.Chaitanya Krishna > Fix For: 0.21.0 > > Attachments: clustersummarymodification.png, detailspage.png, > initialscreen.png, MAPREDUCE-862-1.patch, MAPREDUCE-862-2.patch, > MAPREDUCE-862-3.patch, MAPREDUCE-862-4.patch, MAPREDUCE-862-5.patch, > MAPREDUCE-862-6.patch, MAPREDUCE-862-7.patch, subqueue.png > > > MAPREDUCE-853 proposes to introduce a hierarchy of queues into the Map/Reduce > framework. This JIRA is for defining changes to the UI related to queues. > This includes the hadoop queue CLI and the web UI on the JobTracker. -- This message is automatically generated by JIRA. - You can reply to this email to add a comment to the issue online.
[jira] Updated: (MAPREDUCE-781) distcp overrides user-selected job name
[ https://issues.apache.org/jira/browse/MAPREDUCE-781?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Venkatesh S updated MAPREDUCE-781: -- Status: Open (was: Patch Available) > distcp overrides user-selected job name > --- > > Key: MAPREDUCE-781 > URL: https://issues.apache.org/jira/browse/MAPREDUCE-781 > Project: Hadoop Map/Reduce > Issue Type: Bug > Components: distcp >Affects Versions: 0.21.0, 0.22.0 >Reporter: Rob Weltman >Assignee: Venkatesh S > Fix For: 0.21.0, 0.22.0 > > Attachments: MAPREDUCE-781.patch, MAPREDUCE_781.patch > > > distcp hard-codes the hadoop job name to "distcp" even if the user specifies > a job name. This is a problem in general, but especially for generalized > replication services since the Job Tracker UI and history can't be made to > indicate what is being copied in the job name. -- This message is automatically generated by JIRA. - You can reply to this email to add a comment to the issue online.
[jira] Updated: (MAPREDUCE-781) distcp overrides user-selected job name
[ https://issues.apache.org/jira/browse/MAPREDUCE-781?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Venkatesh S updated MAPREDUCE-781: -- Status: Patch Available (was: Open) > distcp overrides user-selected job name > --- > > Key: MAPREDUCE-781 > URL: https://issues.apache.org/jira/browse/MAPREDUCE-781 > Project: Hadoop Map/Reduce > Issue Type: Bug > Components: distcp >Affects Versions: 0.21.0, 0.22.0 >Reporter: Rob Weltman >Assignee: Venkatesh S > Fix For: 0.21.0, 0.22.0 > > Attachments: MAPREDUCE-781.patch, MAPREDUCE_781.patch > > > distcp hard-codes the hadoop job name to "distcp" even if the user specifies > a job name. This is a problem in general, but especially for generalized > replication services since the Job Tracker UI and history can't be made to > indicate what is being copied in the job name. -- This message is automatically generated by JIRA. - You can reply to this email to add a comment to the issue online.
[jira] Commented: (MAPREDUCE-946) Fix regression in LineRecordReader to comply with line length parameters
[ https://issues.apache.org/jira/browse/MAPREDUCE-946?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12756423#action_12756423 ] Hudson commented on MAPREDUCE-946: -- Integrated in Hadoop-Mapreduce-trunk-Commit #43 (See [http://hudson.zones.apache.org/hudson/job/Hadoop-Mapreduce-trunk-Commit/43/]) . Fix a regression in LineRecordReader where the maxBytesToConsume parameter is not set correctly. > Fix regression in LineRecordReader to comply with line length parameters > > > Key: MAPREDUCE-946 > URL: https://issues.apache.org/jira/browse/MAPREDUCE-946 > Project: Hadoop Map/Reduce > Issue Type: Bug >Reporter: Chris Douglas >Assignee: Chris Douglas >Priority: Blocker > Fix For: 0.21.0 > > Attachments: M946-0.patch, M946-1.patch > > > MAPREDUCE-773 accidentally changed code introduced in HADOOP-3144 controlling > max line lengths. The behavior should be restored. -- This message is automatically generated by JIRA. - You can reply to this email to add a comment to the issue online.
[jira] Commented: (MAPREDUCE-157) Job History log file format is not friendly for external tools.
[ https://issues.apache.org/jira/browse/MAPREDUCE-157?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12756424#action_12756424 ] Hudson commented on MAPREDUCE-157: -- Integrated in Hadoop-Mapreduce-trunk-Commit #43 (See [http://hudson.zones.apache.org/hudson/job/Hadoop-Mapreduce-trunk-Commit/43/]) . Refactor job history APIs and change the history format to JSON. Contributed by Jothi Padmanabhan. > Job History log file format is not friendly for external tools. > --- > > Key: MAPREDUCE-157 > URL: https://issues.apache.org/jira/browse/MAPREDUCE-157 > Project: Hadoop Map/Reduce > Issue Type: Sub-task >Affects Versions: 0.20.1 >Reporter: Owen O'Malley >Assignee: Jothi Padmanabhan > Fix For: 0.21.0 > > Attachments: mapred-157-10Sep.patch, mapred-157-15Sep-v1.patch, > mapred-157-15Sep.patch, mapred-157-16Sep-v1.patch, mapred-157-16Sep.patch, > mapred-157-4Sep.patch, mapred-157-7Sep-v1.patch, mapred-157-7Sep.patch, > mapred-157-prelim.patch, MAPREDUCE-157-avro.patch, MAPREDUCE-157-avro.patch > > > Currently, parsing the job history logs with external tools is very difficult > because of the format. The most critical problem is that newlines aren't > escaped in the strings. That makes using tools like grep, sed, and awk very > tricky. -- This message is automatically generated by JIRA. - You can reply to this email to add a comment to the issue online.
[jira] Commented: (MAPREDUCE-953) Generate configuration dump for hierarchial queue configuration
[
https://issues.apache.org/jira/browse/MAPREDUCE-953?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12756426#action_12756426
]
V.V.Chaitanya Krishna commented on MAPREDUCE-953:
-
The proposal of the JSONN string format is as follows:
{"queues":[
{"name":,"state":,"submit_job":,
"administer_jobs":,"properties","children":[
{"name":,"state":,
"submit_job":,
"administer_jobs":,"properties":,"children":[]
} ]
}
] }
Note: The will be again in the same format as its parent is.
So, it is a tree-like structure.
> Generate configuration dump for hierarchial queue configuration
> ---
>
> Key: MAPREDUCE-953
> URL: https://issues.apache.org/jira/browse/MAPREDUCE-953
> Project: Hadoop Map/Reduce
> Issue Type: Improvement
>Reporter: rahul k singh
>Priority: Blocker
>
> Generate configuration dump for hierarchial queue configuration
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
[jira] Commented: (MAPREDUCE-777) A method for finding and tracking jobs from the new API
[ https://issues.apache.org/jira/browse/MAPREDUCE-777?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12756430#action_12756430 ] Iyappan Srinivasan commented on MAPREDUCE-777: -- +1 from QA Tested all options of bin/hadoop job and checked for its correctness as well as check all options of bin/hadoop queueue and checked for its correctness. 1) JobClient [-submit ] [-status ] [-counter ] [-kill ] [-set-priority ]. Valid values for priorities are: VERY_HIGH HIGH NORMAL LOW VERY_LOW [-events <#-of-events>] [-history ] [-list [all]] [-list-active-trackers] [-list-blacklisted-trackers] [-list-attempt-ids ] [-kill-task ] [-fail-task ] Generic options supported are -conf -D -fs -jt -files -libjars -archives For scheduler, I started a capacity scheduler with linux task controller, with different queues and different permissions to different users.. 2) bin/hadoop queue [-list] [-info [-showJobs]] [-showacls] Generic options supported are -conf specify an application configuration file -D use value for given property -fs specify a namenode -jt specify a job tracker -files specify comma separated files to be copied to the map reduce cluster -libjars specify comma separated jar files to include in the classpath. -archives specify comma separated archives to be unarchived on the compute machines. Other test sceanrios were, testing it with JT restart, running job, waiting job etc. The values can be seen by making the variable mapred.job.tracker.retire.jobs to false. Raised these generic bugs/improvements, which were already found in trunk before this patch.: 1) MAPREDUCE-983 bin/hadoop job -fs file:///sdsdad -list still works. It is not picking up the latest fs input 2) MAPREDUCE-984 bin/hadoop job -kill command says" job successfully killed" even though job has retired 3) MAPREDUCE-985 job -kill-task ] and -fail-task are not task-ids they are attempt ids 4) MAPREDUCE-994 bin/hadoop job -counter help options do not give information on permissible values. 5) MAPREDUCE-993 bin/hadoop job -events <#-of-events> help message is confusing 6) MAPREDUCE-992 bin/hadoop job -events < jobid> gives event links which does not work. > A method for finding and tracking jobs from the new API > --- > > Key: MAPREDUCE-777 > URL: https://issues.apache.org/jira/browse/MAPREDUCE-777 > Project: Hadoop Map/Reduce > Issue Type: New Feature > Components: client >Reporter: Owen O'Malley >Assignee: Amareshwari Sriramadasu > Fix For: 0.21.0 > > Attachments: m-777.patch, patch-777-1.txt, patch-777-10.txt, > patch-777-11.txt, patch-777-12.txt, patch-777-13.txt, patch-777-2.txt, > patch-777-3.txt, patch-777-4.txt, patch-777-5.txt, patch-777-6.txt, > patch-777-7.txt, patch-777-8.txt, patch-777-9.txt, patch-777.txt > > > We need to create a replacement interface for the JobClient API in the new > interface. In particular, the user needs to be able to query and track jobs > that were launched by other processes. -- This message is automatically generated by JIRA. - You can reply to this email to add a comment to the issue online.
[jira] Updated: (MAPREDUCE-893) Provide an ability to refresh queue configuration without restart.
[ https://issues.apache.org/jira/browse/MAPREDUCE-893?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Vinod K V updated MAPREDUCE-893: Attachment: MAPREDUCE-893-20090917.2.txt Attaching patch that includes the above comments. In particular: - Documented QueueConfigurationParser's constructor that it also does parsing itself. - Renamed scheduler parameter of refreshQueues() to schedulerRefresher. - Inner class CapacitySchedulerQueueRefresher uses 'initializationPoller' instance. Left it as is to avoid synthetic constructors. - getDisplayInfo() is knocked off completely. TestRefreshOfQueues has minor problems and is being fixed. Everything else can be reviewed. I'll fix it as the review progresses. Some questions: - There is a need for getting a QueueState from a string. QueueState().valueOf() returns exact name i.e. RUNNING, STOPPED etc, so we cannot use them directly as display names can be different. Is maintaining the Queue.QueueState.enumMap bad? What is the standard way of handling this in enums? - I left my earlier change which assigns the state UNDEFINED to Container-Queues. This I did mainly because state doesn't make sense to Container-Queues - they cannot be stopped, started for e.g. as of now. Is this not OK? - Capacity-scheduler.xml still contains the default queue specific properties. Shouldn't we remove them? If so, where shall the documentation of scheduler properties be? - Removed the schedConf instance in scheduler because maintaining it would add extra effort to keep it consistent during refresh. Is this fine? - Removed SchedulingDisplayInfo object all together and instead passing QueueSchedulingContext to the QueueManager directly. Also, the corresponding queueInfoMap completely. Is this change OK? - Minor: I am confused about the name TestContainerQueue and what it is actually testing. Should we rename it to TestHierarchicalQueues or something like that? > Provide an ability to refresh queue configuration without restart. > -- > > Key: MAPREDUCE-893 > URL: https://issues.apache.org/jira/browse/MAPREDUCE-893 > Project: Hadoop Map/Reduce > Issue Type: Improvement > Components: jobtracker >Reporter: Hemanth Yamijala >Assignee: Vinod K V > Fix For: 0.21.0 > > Attachments: MAPREDUCE-893-20090915.1.txt, > MAPREDUCE-893-20090917.2.txt > > > While administering a cluster using multiple queues, administrators feel a > need to refresh queue properties on the fly without needing to restart the > JobTracker. This is partially supported for some properties such as queue > ACLs (HADOOP-5396) and state (HADOOP-5913). The idea is to extend the > facility to refresh other queue properties as well, including scheduler > properties. -- This message is automatically generated by JIRA. - You can reply to this email to add a comment to the issue online.
[jira] Commented: (MAPREDUCE-654) Add an option -count to distcp for displaying some info about the src files
[
https://issues.apache.org/jira/browse/MAPREDUCE-654?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12756445#action_12756445
]
Tom White commented on MAPREDUCE-654:
-
How about calling the new option {{-dryrun}}, to make it clearer that it
doesn't do a copy?
> Add an option -count to distcp for displaying some info about the src files
> ---
>
> Key: MAPREDUCE-654
> URL: https://issues.apache.org/jira/browse/MAPREDUCE-654
> Project: Hadoop Map/Reduce
> Issue Type: New Feature
> Components: distcp
>Affects Versions: 0.21.0
>Reporter: Ravi Gummadi
>Assignee: Ravi Gummadi
> Fix For: 0.21.0
>
> Attachments: d_count.patch, d_count654.patch, d_count_v1.patch
>
>
> Add an option -count to distcp for displaying metadata about src files like
> number of files to be copied and total size of src files to be copied.
> WIth -count, distcp doesn't do any copy. Just displays info and exits.
> This is useful specifically when used with -update.
> distcp -update -count *
> would display the number of files to be updated and the total size of
> copy needs to be done(by comparing the file sizes and checksums at src and
> dst). Based on this info, users could allocate the number of nodes needed for
> the actual update job.
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
[jira] Updated: (MAPREDUCE-649) distcp should validate the data copied
[ https://issues.apache.org/jira/browse/MAPREDUCE-649?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Ravi Gummadi updated MAPREDUCE-649: --- Attachment: d_verify649.v1.patch Attaching patch that makes copy of file to be retried distcp.file.retries(with default value of 3) times in case of failures. Failure could be failure of validation of copy(through file checksums mismatch). These retries are at done within the same map irrespective of -i option. Please review and provide your comments. > distcp should validate the data copied > -- > > Key: MAPREDUCE-649 > URL: https://issues.apache.org/jira/browse/MAPREDUCE-649 > Project: Hadoop Map/Reduce > Issue Type: New Feature > Components: distcp >Reporter: Ravi Gummadi >Assignee: Ravi Gummadi > Attachments: d_verify.patch, d_verify649.patch, d_verify649.v1.patch > > > distcp should validate the files copied by checking the checksums, if the > filesystem supports checksums. -- This message is automatically generated by JIRA. - You can reply to this email to add a comment to the issue online.
[jira] Commented: (MAPREDUCE-893) Provide an ability to refresh queue configuration without restart.
[
https://issues.apache.org/jira/browse/MAPREDUCE-893?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12756446#action_12756446
]
Vinod K V commented on MAPREDUCE-893:
-
Also, for those who are listening, this is the new QueueRefresher interface
that I am adding to TaskScheduler:
{code}
abstract class TaskScheduler implements Configurable {
..
..
/**
* Abstract QueueRefresher class. Scheduler's can extend this and return an
* instance of this in the {...@link #getQueueRefresher()} method. The
* {...@link #refreshQueues(List)} method of this instance will be invoked by
the
* {...@link QueueManager} whenever it gets a request from an administrator to
* refresh its own queue-configuration. This method has a documented contract
* between the {...@link QueueManager} and the {...@link TaskScheduler}.
*/
abstract class QueueRefresher {
/**
* Refresh the queue-configuration in the scheduler. This method has the
* following contract.
*
* Before this method, {...@link QueueManager} does a validation of the
new
* queue-configuration. For e.g, currently addition of new queues, or
* removal of queues at any level in the hierarchy is not supported by
* {...@link QueueManager} and so are not supported for schedulers too.
* Schedulers will be passed a list of {...@link JobQueueInfo}s of the
root
* queues i.e. the queues at the top level. All the descendants are properly
* linked from these top-level queues.
* Schedulers should use the scheduler specific queue properties from
* the newRootQueues, validate the properties themselves and apply them
* internally.
*
* Once the method returns successfully from the schedulers, it is assumed
* that the refresh of queue properties is successful throughout and will be
* 'committed' internally to {...@link QueueManager} too. It is guaranteed
that
* at no point, after successful return from the scheduler, is the queue
* refresh in QueueManager failed. If ever, such abnormalities happen, the
* queue framework will be inconsistent and will need a JT restart.
* If scheduler throws an exception during {...@link #refreshQueues()},
* {...@link QueueManager} throws away the newly read configuration, retains
* the old (consistent) configuration and informs the request issuer about
* the error appropriately.
*
*
* @param newRootQueues
*/
abstract void refreshQueues(List newRootQueues)
throws Throwable;
}
/**
* Get the {...@link QueueRefresher} for this scheduler. By default, no
* {...@link QueueRefresher} exists for a scheduler and is set to null.
* Schedulers need to return an instance of {...@link QueueRefresher} if they
* wish to refresh their queue-configuration when {...@link QueueManager}
* refreshes its own queue-configuration via an administrator request.
*
* @return
*/
QueueRefresher getQueueRefresher() {
return null;
}
}
{code}
> Provide an ability to refresh queue configuration without restart.
> --
>
> Key: MAPREDUCE-893
> URL: https://issues.apache.org/jira/browse/MAPREDUCE-893
> Project: Hadoop Map/Reduce
> Issue Type: Improvement
> Components: jobtracker
>Reporter: Hemanth Yamijala
>Assignee: Vinod K V
> Fix For: 0.21.0
>
> Attachments: MAPREDUCE-893-20090915.1.txt,
> MAPREDUCE-893-20090917.2.txt
>
>
> While administering a cluster using multiple queues, administrators feel a
> need to refresh queue properties on the fly without needing to restart the
> JobTracker. This is partially supported for some properties such as queue
> ACLs (HADOOP-5396) and state (HADOOP-5913). The idea is to extend the
> facility to refresh other queue properties as well, including scheduler
> properties.
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
[jira] Updated: (MAPREDUCE-977) Missing jackson jars from Eclipse template
[ https://issues.apache.org/jira/browse/MAPREDUCE-977?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Tom White updated MAPREDUCE-977: Resolution: Fixed Hadoop Flags: [Reviewed] Status: Resolved (was: Patch Available) I've just committed this. > Missing jackson jars from Eclipse template > -- > > Key: MAPREDUCE-977 > URL: https://issues.apache.org/jira/browse/MAPREDUCE-977 > Project: Hadoop Map/Reduce > Issue Type: Bug > Components: build >Reporter: Tom White >Assignee: Tom White > Fix For: 0.21.0 > > Attachments: MAPREDUCE-977.patch > > -- This message is automatically generated by JIRA. - You can reply to this email to add a comment to the issue online.
[jira] Created: (MAPREDUCE-995) JobHistory should handle cases where task completion events are generated after job completion event
JobHistory should handle cases where task completion events are generated after job completion event Key: MAPREDUCE-995 URL: https://issues.apache.org/jira/browse/MAPREDUCE-995 Project: Hadoop Map/Reduce Issue Type: Bug Reporter: Jothi Padmanabhan Assignee: Jothi Padmanabhan It is apparently possible, in certain circumstances (failed job, for example), for the job history to get task completion events after the job completion event. This currently causes NPE in job history. Thanks Hong for identifying this issue -- This message is automatically generated by JIRA. - You can reply to this email to add a comment to the issue online.
[jira] Commented: (MAPREDUCE-777) A method for finding and tracking jobs from the new API
[ https://issues.apache.org/jira/browse/MAPREDUCE-777?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12756459#action_12756459 ] Arun C Murthy commented on MAPREDUCE-777: - +1 I'd commit this once Hudson gives it the once-over. > A method for finding and tracking jobs from the new API > --- > > Key: MAPREDUCE-777 > URL: https://issues.apache.org/jira/browse/MAPREDUCE-777 > Project: Hadoop Map/Reduce > Issue Type: New Feature > Components: client >Reporter: Owen O'Malley >Assignee: Amareshwari Sriramadasu > Fix For: 0.21.0 > > Attachments: m-777.patch, patch-777-1.txt, patch-777-10.txt, > patch-777-11.txt, patch-777-12.txt, patch-777-13.txt, patch-777-2.txt, > patch-777-3.txt, patch-777-4.txt, patch-777-5.txt, patch-777-6.txt, > patch-777-7.txt, patch-777-8.txt, patch-777-9.txt, patch-777.txt > > > We need to create a replacement interface for the JobClient API in the new > interface. In particular, the user needs to be able to query and track jobs > that were launched by other processes. -- This message is automatically generated by JIRA. - You can reply to this email to add a comment to the issue online.
[jira] Updated: (MAPREDUCE-954) The new interface's Context objects should be interfaces
[ https://issues.apache.org/jira/browse/MAPREDUCE-954?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Arun C Murthy updated MAPREDUCE-954: Attachment: MAPREDUCE-954.patch Closer, not quite. > The new interface's Context objects should be interfaces > > > Key: MAPREDUCE-954 > URL: https://issues.apache.org/jira/browse/MAPREDUCE-954 > Project: Hadoop Map/Reduce > Issue Type: Improvement > Components: client >Reporter: Owen O'Malley >Assignee: Arun C Murthy > Fix For: 0.21.0 > > Attachments: MAPREDUCE-954.patch, MAPREDUCE-954.patch, > MAPREDUCE-954.patch > > > When I was doing HADOOP-1230, I was persuaded to make the Context objects as > classes. I think that was a serious mistake. It caused a lot of information > leakage into the public classes. -- This message is automatically generated by JIRA. - You can reply to this email to add a comment to the issue online.
[jira] Updated: (MAPREDUCE-849) Renaming of configuration property names in mapreduce
[ https://issues.apache.org/jira/browse/MAPREDUCE-849?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Amareshwari Sriramadasu updated MAPREDUCE-849: -- Attachment: patch-849.txt Patch doing the suggested renaming > Renaming of configuration property names in mapreduce > - > > Key: MAPREDUCE-849 > URL: https://issues.apache.org/jira/browse/MAPREDUCE-849 > Project: Hadoop Map/Reduce > Issue Type: Improvement >Reporter: Amareshwari Sriramadasu >Assignee: Amareshwari Sriramadasu > Fix For: 0.21.0 > > Attachments: Config changes.xls, Config changes.xls, patch-849.txt > > > In-line with HDFS-531, property names in configuration files should be > standardized in MAPREDUCE. -- This message is automatically generated by JIRA. - You can reply to this email to add a comment to the issue online.
[jira] Commented: (MAPREDUCE-995) JobHistory should handle cases where task completion events are generated after job completion event
[
https://issues.apache.org/jira/browse/MAPREDUCE-995?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12756466#action_12756466
]
Hong Tang commented on MAPREDUCE-995:
-
I found this issue while preparing patch for Mumak (Hadoop Simulator
MAPREDUCE-728). The backtrace of the problem is as follows:
{noformat}
java.lang.NullPointerException
at
org.codehaus.jackson.impl.WriterBasedGenerator._verifyValueWrite(WriterBasedGenerator.java:553)
at
org.codehaus.jackson.impl.JsonGeneratorBase.writeStartObject(JsonGeneratorBase.java:142)
at
org.apache.hadoop.mapreduce.jobhistory.EventWriter.writeEventType(EventWriter.java:81)
at
org.apache.hadoop.mapreduce.jobhistory.EventWriter.write(EventWriter.java:60)
at
org.apache.hadoop.mapreduce.jobhistory.JobHistory.logEvent(JobHistory.java:339)
at
org.apache.hadoop.mapred.JobInProgress.completedTask(JobInProgress.java:2519)
at
org.apache.hadoop.mapred.JobInProgress.updateTaskStatus(JobInProgress.java:1130)
at
org.apache.hadoop.mapred.SimulatorJobTracker.updateTaskStatuses(SimulatorJobTracker.java:621)
{noformat}
The last statement in the backtrace occurs in the context when the job has
failed (one task fails 4 times), and the job is marked as complete. Then a
lingering task that belongs to the same job completes, and JT tries to update
its status.
> JobHistory should handle cases where task completion events are generated
> after job completion event
>
>
> Key: MAPREDUCE-995
> URL: https://issues.apache.org/jira/browse/MAPREDUCE-995
> Project: Hadoop Map/Reduce
> Issue Type: Bug
>Reporter: Jothi Padmanabhan
>Assignee: Jothi Padmanabhan
>
> It is apparently possible, in certain circumstances (failed job, for
> example), for the job history to get task completion events after the job
> completion event. This currently causes NPE in job history.
> Thanks Hong for identifying this issue
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
[jira] Updated: (MAPREDUCE-564) Provide a way for the client to get the number of currently running maps/reduces
[ https://issues.apache.org/jira/browse/MAPREDUCE-564?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Ravi Gummadi updated MAPREDUCE-564: --- Attachment: MR-564.v4.2.patch Sorry. Previous patch has a minor bug. Attaching new patch. > Provide a way for the client to get the number of currently running > maps/reduces > > > Key: MAPREDUCE-564 > URL: https://issues.apache.org/jira/browse/MAPREDUCE-564 > Project: Hadoop Map/Reduce > Issue Type: Improvement > Components: jobtracker >Affects Versions: 0.21.0 >Reporter: Ravi Gummadi >Assignee: Ravi Gummadi > Fix For: 0.21.0 > > Attachments: MR-564.patch, MR-564.v1.patch, MR-564.v2.patch, > MR-564.v3.patch, MR-564.v4.1.patch, MR-564.v4.2.patch, MR-564.v4.patch > > > Add counters for Number of Succeeded Maps and Number of Succeeded Reduces so > that client can get this number without iterating through all the task > reports while the job is in progress. -- This message is automatically generated by JIRA. - You can reply to this email to add a comment to the issue online.
[jira] Commented: (MAPREDUCE-893) Provide an ability to refresh queue configuration without restart.
[ https://issues.apache.org/jira/browse/MAPREDUCE-893?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12756468#action_12756468 ] Hemanth Yamijala commented on MAPREDUCE-893: bq. Capacity-scheduler.xml still contains the default queue specific properties. Shouldn't we remove them? If so, where shall the documentation of scheduler properties be? I think we should remove the properties that are no longer supposed to be capacity-scheduler.xml and add a new file - something like mapred-queues-for-cs.xml.template. This would only be for documentation purposes and we could comment that users of capacity scheduler can rename this to mapred-queues.xml adding more queues etc, based on the documentation provided. Works ? bq. Removed SchedulingDisplayInfo object all together and instead passing QueueSchedulingContext to the QueueManager directly. Also, the corresponding queueInfoMap completely. Is this change OK? As discussed offline, I am not completely comfortable with this idea. I see the QueueSchedulingContext as a core object of the CS, and wouldn't want the framework to hold an instance of it (that too, in ways that are not very easily detectable). I think the existing model of providing just a 'view' of the scheduler state to the framework is better. Regarding the rest of the questions, I am not really too strong on an opinion. Let's try and minimize change to the extent possible at this stage in the interest of caution. *smile* > Provide an ability to refresh queue configuration without restart. > -- > > Key: MAPREDUCE-893 > URL: https://issues.apache.org/jira/browse/MAPREDUCE-893 > Project: Hadoop Map/Reduce > Issue Type: Improvement > Components: jobtracker >Reporter: Hemanth Yamijala >Assignee: Vinod K V > Fix For: 0.21.0 > > Attachments: MAPREDUCE-893-20090915.1.txt, > MAPREDUCE-893-20090917.2.txt > > > While administering a cluster using multiple queues, administrators feel a > need to refresh queue properties on the fly without needing to restart the > JobTracker. This is partially supported for some properties such as queue > ACLs (HADOOP-5396) and state (HADOOP-5913). The idea is to extend the > facility to refresh other queue properties as well, including scheduler > properties. -- This message is automatically generated by JIRA. - You can reply to this email to add a comment to the issue online.
[jira] Updated: (MAPREDUCE-995) JobHistory should handle cases where task completion events are generated after job completion event
[ https://issues.apache.org/jira/browse/MAPREDUCE-995?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Jothi Padmanabhan updated MAPREDUCE-995: Attachment: mapred-995.patch Straight Forward Patch. Also, made fileMap a ConcurrentHashMap. > JobHistory should handle cases where task completion events are generated > after job completion event > > > Key: MAPREDUCE-995 > URL: https://issues.apache.org/jira/browse/MAPREDUCE-995 > Project: Hadoop Map/Reduce > Issue Type: Bug >Reporter: Jothi Padmanabhan >Assignee: Jothi Padmanabhan > Attachments: mapred-995.patch > > > It is apparently possible, in certain circumstances (failed job, for > example), for the job history to get task completion events after the job > completion event. This currently causes NPE in job history. > Thanks Hong for identifying this issue -- This message is automatically generated by JIRA. - You can reply to this email to add a comment to the issue online.
[jira] Updated: (MAPREDUCE-995) JobHistory should handle cases where task completion events are generated after job completion event
[ https://issues.apache.org/jira/browse/MAPREDUCE-995?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Jothi Padmanabhan updated MAPREDUCE-995: Status: Patch Available (was: Open) > JobHistory should handle cases where task completion events are generated > after job completion event > > > Key: MAPREDUCE-995 > URL: https://issues.apache.org/jira/browse/MAPREDUCE-995 > Project: Hadoop Map/Reduce > Issue Type: Bug >Reporter: Jothi Padmanabhan >Assignee: Jothi Padmanabhan > Attachments: mapred-995.patch > > > It is apparently possible, in certain circumstances (failed job, for > example), for the job history to get task completion events after the job > completion event. This currently causes NPE in job history. > Thanks Hong for identifying this issue -- This message is automatically generated by JIRA. - You can reply to this email to add a comment to the issue online.
[jira] Updated: (MAPREDUCE-988) ant package does not copy the capacity-scheduler.jar under HADOOP_HOME/build/hadoop-mapred-0.21.0-dev/contrib/capacity-scheduler
[ https://issues.apache.org/jira/browse/MAPREDUCE-988?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Chris Douglas updated MAPREDUCE-988: Resolution: Fixed Hadoop Flags: [Reviewed] Status: Resolved (was: Patch Available) +1 I committed this. Thanks, Hong! > ant package does not copy the capacity-scheduler.jar under > HADOOP_HOME/build/hadoop-mapred-0.21.0-dev/contrib/capacity-scheduler > > > Key: MAPREDUCE-988 > URL: https://issues.apache.org/jira/browse/MAPREDUCE-988 > Project: Hadoop Map/Reduce > Issue Type: Bug > Components: build >Affects Versions: 0.21.0 >Reporter: Iyappan Srinivasan >Assignee: Hong Tang > Fix For: 0.21.0 > > Attachments: mapreduce-988-20090916.patch > > > ant package does not copy the hadoop-0.21.0-dev-capacity-scheduler.jar under > HADOOP_HOME/build/hadoop-mapred-0.21.0-dev/contrib/capacity-scheduler/. > Till yesterday it was copying it properly. Issue seems to be pointing to the > latest checkin of MAPREDUCE-776, which changes build.xml. -- This message is automatically generated by JIRA. - You can reply to this email to add a comment to the issue online.
[jira] Updated: (MAPREDUCE-653) distcp can support bandwidth limiting
[ https://issues.apache.org/jira/browse/MAPREDUCE-653?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Ravi Gummadi updated MAPREDUCE-653: --- Attachment: d_bw.v2.patch Attaching new patch as the dependent patch of MAPREDUCE-564 is changed and that provdes api to access the number of running maps/reduces and not new counters. > distcp can support bandwidth limiting > - > > Key: MAPREDUCE-653 > URL: https://issues.apache.org/jira/browse/MAPREDUCE-653 > Project: Hadoop Map/Reduce > Issue Type: New Feature > Components: distcp >Reporter: Ravi Gummadi >Assignee: Ravi Gummadi > Attachments: d_bw.patch, d_bw.v1.patch, d_bw.v2.patch > > > distcp should support an option for user to specify the bandwidth limit for > the distcp job. -- This message is automatically generated by JIRA. - You can reply to this email to add a comment to the issue online.
[jira] Commented: (MAPREDUCE-995) JobHistory should handle cases where task completion events are generated after job completion event
[
https://issues.apache.org/jira/browse/MAPREDUCE-995?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12756475#action_12756475
]
Hong Tang commented on MAPREDUCE-995:
-
The bug seems trickier than I thought...
There is an obvious race condition:
Thread 1:
{code}
writer.close();
fileMap.get(id).clearEventWriter();
{code}
Thread 2:
{code}
MetaInfo mi = fileMap.get(jobId);
if (mi == null || (writer = mi.getEventWriter()) == null) {
{code}
writer.close() should be inside clearEventWriter(), and clearEventWriter()
should close the writer and set it to null atomically. Another problem is that
thread 2 may get a valid writer instance, but may later be closed by thread 1.
So I think the right fix would have to synchronize on the MetaInfo for both
clearing event writer and writing logs - logEvent would have to be implemented
on MetaInfo, and synchornized with clearEventWriter().
> JobHistory should handle cases where task completion events are generated
> after job completion event
>
>
> Key: MAPREDUCE-995
> URL: https://issues.apache.org/jira/browse/MAPREDUCE-995
> Project: Hadoop Map/Reduce
> Issue Type: Bug
>Reporter: Jothi Padmanabhan
>Assignee: Jothi Padmanabhan
> Attachments: mapred-995.patch
>
>
> It is apparently possible, in certain circumstances (failed job, for
> example), for the job history to get task completion events after the job
> completion event. This currently causes NPE in job history.
> Thanks Hong for identifying this issue
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
[jira] Commented: (MAPREDUCE-777) A method for finding and tracking jobs from the new API
[
https://issues.apache.org/jira/browse/MAPREDUCE-777?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12756478#action_12756478
]
Tom White commented on MAPREDUCE-777:
-
More comments, mainly naming:
* There are occurrences of both "blacklist" and "blackList" in the public API
(e.g. TaskTrackerInfo#getReasonForBlackList() and getBlacklistReport()). Either
is correct since the word may be spelled as "blacklist" or "black list", but we
need to be consistent throughout.
* Cluster#getFs() would be better as getFileSystem() (particularly with the
debate in HADOOP-6223). Also it would be good to have javadoc describing the
fact it is returning the file system where job-specific files are placed.
* JobStatus#{setup,map,reduce,cleanup}Progress() would be better as getters to
be consistent with the rest of the class.
* TaskCompletionEvent#getTaskAttemptID() should be getTaskAttemptId() to be
consistent with getEventId().
* TaskCompletionEvent#setTaskID() should be setTaskAttemptId().
* TaskReport's method names should be made consistent with this convention too.
> A method for finding and tracking jobs from the new API
> ---
>
> Key: MAPREDUCE-777
> URL: https://issues.apache.org/jira/browse/MAPREDUCE-777
> Project: Hadoop Map/Reduce
> Issue Type: New Feature
> Components: client
>Reporter: Owen O'Malley
>Assignee: Amareshwari Sriramadasu
> Fix For: 0.21.0
>
> Attachments: m-777.patch, patch-777-1.txt, patch-777-10.txt,
> patch-777-11.txt, patch-777-12.txt, patch-777-13.txt, patch-777-2.txt,
> patch-777-3.txt, patch-777-4.txt, patch-777-5.txt, patch-777-6.txt,
> patch-777-7.txt, patch-777-8.txt, patch-777-9.txt, patch-777.txt
>
>
> We need to create a replacement interface for the JobClient API in the new
> interface. In particular, the user needs to be able to query and track jobs
> that were launched by other processes.
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
[jira] Updated: (MAPREDUCE-995) JobHistory should handle cases where task completion events are generated after job completion event
[ https://issues.apache.org/jira/browse/MAPREDUCE-995?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Jothi Padmanabhan updated MAPREDUCE-995: Status: Open (was: Patch Available) > JobHistory should handle cases where task completion events are generated > after job completion event > > > Key: MAPREDUCE-995 > URL: https://issues.apache.org/jira/browse/MAPREDUCE-995 > Project: Hadoop Map/Reduce > Issue Type: Bug >Reporter: Jothi Padmanabhan >Assignee: Jothi Padmanabhan > Attachments: mapred-995.patch > > > It is apparently possible, in certain circumstances (failed job, for > example), for the job history to get task completion events after the job > completion event. This currently causes NPE in job history. > Thanks Hong for identifying this issue -- This message is automatically generated by JIRA. - You can reply to this email to add a comment to the issue online.
[jira] Commented: (MAPREDUCE-988) ant package does not copy the capacity-scheduler.jar under HADOOP_HOME/build/hadoop-mapred-0.21.0-dev/contrib/capacity-scheduler
[ https://issues.apache.org/jira/browse/MAPREDUCE-988?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12756480#action_12756480 ] Hadoop QA commented on MAPREDUCE-988: - +1 overall. Here are the results of testing the latest attachment http://issues.apache.org/jira/secure/attachment/12419851/mapreduce-988-20090916.patch against trunk revision 816088. +1 @author. The patch does not contain any @author tags. -1 tests included. The patch doesn't appear to include any new or modified tests. Please justify why no new tests are needed for this patch. Also please list what manual steps were performed to verify this patch. +1 javadoc. The javadoc tool did not generate any warning messages. +1 javac. The applied patch does not increase the total number of javac compiler warnings. +1 findbugs. The patch does not introduce any new Findbugs warnings. +1 release audit. The applied patch does not increase the total number of release audit warnings. +1 core tests. The patch passed core unit tests. +1 contrib tests. The patch passed contrib unit tests. Test results: http://hudson.zones.apache.org/hudson/job/Mapreduce-Patch-h3.grid.sp2.yahoo.net/42/testReport/ Findbugs warnings: http://hudson.zones.apache.org/hudson/job/Mapreduce-Patch-h3.grid.sp2.yahoo.net/42/artifact/trunk/build/test/findbugs/newPatchFindbugsWarnings.html Checkstyle results: http://hudson.zones.apache.org/hudson/job/Mapreduce-Patch-h3.grid.sp2.yahoo.net/42/artifact/trunk/build/test/checkstyle-errors.html Console output: http://hudson.zones.apache.org/hudson/job/Mapreduce-Patch-h3.grid.sp2.yahoo.net/42/console This message is automatically generated. > ant package does not copy the capacity-scheduler.jar under > HADOOP_HOME/build/hadoop-mapred-0.21.0-dev/contrib/capacity-scheduler > > > Key: MAPREDUCE-988 > URL: https://issues.apache.org/jira/browse/MAPREDUCE-988 > Project: Hadoop Map/Reduce > Issue Type: Bug > Components: build >Affects Versions: 0.21.0 >Reporter: Iyappan Srinivasan >Assignee: Hong Tang > Fix For: 0.21.0 > > Attachments: mapreduce-988-20090916.patch > > > ant package does not copy the hadoop-0.21.0-dev-capacity-scheduler.jar under > HADOOP_HOME/build/hadoop-mapred-0.21.0-dev/contrib/capacity-scheduler/. > Till yesterday it was copying it properly. Issue seems to be pointing to the > latest checkin of MAPREDUCE-776, which changes build.xml. -- This message is automatically generated by JIRA. - You can reply to this email to add a comment to the issue online.
[jira] Commented: (MAPREDUCE-654) Add an option -count to distcp for displaying some info about the src files
[ https://issues.apache.org/jira/browse/MAPREDUCE-654?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12756481#action_12756481 ] Ravi Gummadi commented on MAPREDUCE-654: Yes. Venkatesh is working on the new option -dryrun that displays the files to be copied by distcp also. This option will be renamed to -dryrun. > Add an option -count to distcp for displaying some info about the src files > --- > > Key: MAPREDUCE-654 > URL: https://issues.apache.org/jira/browse/MAPREDUCE-654 > Project: Hadoop Map/Reduce > Issue Type: New Feature > Components: distcp >Affects Versions: 0.21.0 >Reporter: Ravi Gummadi >Assignee: Ravi Gummadi > Fix For: 0.21.0 > > Attachments: d_count.patch, d_count654.patch, d_count_v1.patch > > > Add an option -count to distcp for displaying metadata about src files like > number of files to be copied and total size of src files to be copied. > WIth -count, distcp doesn't do any copy. Just displays info and exits. > This is useful specifically when used with -update. > distcp -update -count * > would display the number of files to be updated and the total size of > copy needs to be done(by comparing the file sizes and checksums at src and > dst). Based on this info, users could allocate the number of nodes needed for > the actual update job. -- This message is automatically generated by JIRA. - You can reply to this email to add a comment to the issue online.
[jira] Updated: (MAPREDUCE-953) Generate configuration dump for hierarchial queue configuration
[ https://issues.apache.org/jira/browse/MAPREDUCE-953?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] V.V.Chaitanya Krishna updated MAPREDUCE-953: Attachment: MAPREDUCE-953-1.patch Uploading patch taking the above mentioned format. Proper test cases are also included. > Generate configuration dump for hierarchial queue configuration > --- > > Key: MAPREDUCE-953 > URL: https://issues.apache.org/jira/browse/MAPREDUCE-953 > Project: Hadoop Map/Reduce > Issue Type: Improvement >Reporter: rahul k singh >Priority: Blocker > Attachments: MAPREDUCE-953-1.patch > > > Generate configuration dump for hierarchial queue configuration -- This message is automatically generated by JIRA. - You can reply to this email to add a comment to the issue online.
[jira] Commented: (MAPREDUCE-907) Sqoop should use more intelligent splits
[ https://issues.apache.org/jira/browse/MAPREDUCE-907?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12756485#action_12756485 ] Hadoop QA commented on MAPREDUCE-907: - +1 overall. Here are the results of testing the latest attachment http://issues.apache.org/jira/secure/attachment/12419565/MAPREDUCE-907.2.patch against trunk revision 816088. +1 @author. The patch does not contain any @author tags. +1 tests included. The patch appears to include 46 new or modified tests. +1 javadoc. The javadoc tool did not generate any warning messages. +1 javac. The applied patch does not increase the total number of javac compiler warnings. +1 findbugs. The patch does not introduce any new Findbugs warnings. +1 release audit. The applied patch does not increase the total number of release audit warnings. +1 core tests. The patch passed core unit tests. +1 contrib tests. The patch passed contrib unit tests. Test results: http://hudson.zones.apache.org/hudson/job/Mapreduce-Patch-h6.grid.sp2.yahoo.net/95/testReport/ Findbugs warnings: http://hudson.zones.apache.org/hudson/job/Mapreduce-Patch-h6.grid.sp2.yahoo.net/95/artifact/trunk/build/test/findbugs/newPatchFindbugsWarnings.html Checkstyle results: http://hudson.zones.apache.org/hudson/job/Mapreduce-Patch-h6.grid.sp2.yahoo.net/95/artifact/trunk/build/test/checkstyle-errors.html Console output: http://hudson.zones.apache.org/hudson/job/Mapreduce-Patch-h6.grid.sp2.yahoo.net/95/console This message is automatically generated. > Sqoop should use more intelligent splits > > > Key: MAPREDUCE-907 > URL: https://issues.apache.org/jira/browse/MAPREDUCE-907 > Project: Hadoop Map/Reduce > Issue Type: Improvement > Components: contrib/sqoop >Reporter: Aaron Kimball >Assignee: Aaron Kimball > Attachments: MAPREDUCE-907.2.patch, MAPREDUCE-907.patch > > > Sqoop should use the new split generation / InputFormat in MAPREDUCE-885 -- This message is automatically generated by JIRA. - You can reply to this email to add a comment to the issue online.
[jira] Updated: (MAPREDUCE-995) JobHistory should handle cases where task completion events are generated after job completion event
[ https://issues.apache.org/jira/browse/MAPREDUCE-995?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Jothi Padmanabhan updated MAPREDUCE-995: Status: Patch Available (was: Open) > JobHistory should handle cases where task completion events are generated > after job completion event > > > Key: MAPREDUCE-995 > URL: https://issues.apache.org/jira/browse/MAPREDUCE-995 > Project: Hadoop Map/Reduce > Issue Type: Bug >Reporter: Jothi Padmanabhan >Assignee: Jothi Padmanabhan > Attachments: mapred-995-v1.patch, mapred-995.patch > > > It is apparently possible, in certain circumstances (failed job, for > example), for the job history to get task completion events after the job > completion event. This currently causes NPE in job history. > Thanks Hong for identifying this issue -- This message is automatically generated by JIRA. - You can reply to this email to add a comment to the issue online.
[jira] Created: (MAPREDUCE-996) Queue Scheduling Information is lost from Ui when we run mapred mradmin -refreshQueues after mapreduce 861
Queue Scheduling Information is lost from Ui when we run mapred mradmin -refreshQueues after mapreduce 861 -- Key: MAPREDUCE-996 URL: https://issues.apache.org/jira/browse/MAPREDUCE-996 Project: Hadoop Map/Reduce Issue Type: Bug Components: contrib/capacity-sched Affects Versions: 0.21.0 Reporter: Karam Singh Fix For: 0.21.0 -- This message is automatically generated by JIRA. - You can reply to this email to add a comment to the issue online.
[jira] Updated: (MAPREDUCE-995) JobHistory should handle cases where task completion events are generated after job completion event
[ https://issues.apache.org/jira/browse/MAPREDUCE-995?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Jothi Padmanabhan updated MAPREDUCE-995: Attachment: mapred-995-v1.patch Patch incorporating Hongs' comments. > JobHistory should handle cases where task completion events are generated > after job completion event > > > Key: MAPREDUCE-995 > URL: https://issues.apache.org/jira/browse/MAPREDUCE-995 > Project: Hadoop Map/Reduce > Issue Type: Bug >Reporter: Jothi Padmanabhan >Assignee: Jothi Padmanabhan > Attachments: mapred-995-v1.patch, mapred-995.patch > > > It is apparently possible, in certain circumstances (failed job, for > example), for the job history to get task completion events after the job > completion event. This currently causes NPE in job history. > Thanks Hong for identifying this issue -- This message is automatically generated by JIRA. - You can reply to this email to add a comment to the issue online.
[jira] Commented: (MAPREDUCE-781) distcp overrides user-selected job name
[ https://issues.apache.org/jira/browse/MAPREDUCE-781?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12756492#action_12756492 ] Hadoop QA commented on MAPREDUCE-781: - -1 overall. Here are the results of testing the latest attachment http://issues.apache.org/jira/secure/attachment/12419740/MAPREDUCE-781.patch against trunk revision 816147. +1 @author. The patch does not contain any @author tags. -1 tests included. The patch doesn't appear to include any new or modified tests. Please justify why no new tests are needed for this patch. Also please list what manual steps were performed to verify this patch. -1 patch. The patch command could not apply the patch. Console output: http://hudson.zones.apache.org/hudson/job/Mapreduce-Patch-h6.grid.sp2.yahoo.net/96/console This message is automatically generated. > distcp overrides user-selected job name > --- > > Key: MAPREDUCE-781 > URL: https://issues.apache.org/jira/browse/MAPREDUCE-781 > Project: Hadoop Map/Reduce > Issue Type: Bug > Components: distcp >Affects Versions: 0.21.0, 0.22.0 >Reporter: Rob Weltman >Assignee: Venkatesh S > Fix For: 0.21.0, 0.22.0 > > Attachments: MAPREDUCE-781.patch, MAPREDUCE_781.patch > > > distcp hard-codes the hadoop job name to "distcp" even if the user specifies > a job name. This is a problem in general, but especially for generalized > replication services since the Job Tracker UI and history can't be made to > indicate what is being copied in the job name. -- This message is automatically generated by JIRA. - You can reply to this email to add a comment to the issue online.
[jira] Commented: (MAPREDUCE-995) JobHistory should handle cases where task completion events are generated after job completion event
[ https://issues.apache.org/jira/browse/MAPREDUCE-995?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12756491#action_12756491 ] Hong Tang commented on MAPREDUCE-995: - +1. > JobHistory should handle cases where task completion events are generated > after job completion event > > > Key: MAPREDUCE-995 > URL: https://issues.apache.org/jira/browse/MAPREDUCE-995 > Project: Hadoop Map/Reduce > Issue Type: Bug >Reporter: Jothi Padmanabhan >Assignee: Jothi Padmanabhan > Attachments: mapred-995-v1.patch, mapred-995.patch > > > It is apparently possible, in certain circumstances (failed job, for > example), for the job history to get task completion events after the job > completion event. This currently causes NPE in job history. > Thanks Hong for identifying this issue -- This message is automatically generated by JIRA. - You can reply to this email to add a comment to the issue online.
[jira] Created: (MAPREDUCE-997) Acls are working properly when they are to user groups
Acls are working properly when they are to user groups --- Key: MAPREDUCE-997 URL: https://issues.apache.org/jira/browse/MAPREDUCE-997 Project: Hadoop Map/Reduce Issue Type: Bug Affects Versions: 0.21.0 Reporter: Karam Singh When submit-job-acl set usergroup (ug1). if user submits a using hadoop.job.ugi=u1,ug2 it is also gets accepted. (user u1 is also part ug1). In hadoop 0.20.0, job gets rejected. Its a regression issue. -- This message is automatically generated by JIRA. - You can reply to this email to add a comment to the issue online.
[jira] Commented: (MAPREDUCE-977) Missing jackson jars from Eclipse template
[ https://issues.apache.org/jira/browse/MAPREDUCE-977?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12756495#action_12756495 ] Hudson commented on MAPREDUCE-977: -- Integrated in Hadoop-Mapreduce-trunk-Commit #44 (See [http://hudson.zones.apache.org/hudson/job/Hadoop-Mapreduce-trunk-Commit/44/]) . Missing jackson jars from Eclipse template. > Missing jackson jars from Eclipse template > -- > > Key: MAPREDUCE-977 > URL: https://issues.apache.org/jira/browse/MAPREDUCE-977 > Project: Hadoop Map/Reduce > Issue Type: Bug > Components: build >Reporter: Tom White >Assignee: Tom White > Fix For: 0.21.0 > > Attachments: MAPREDUCE-977.patch > > -- This message is automatically generated by JIRA. - You can reply to this email to add a comment to the issue online.
[jira] Commented: (MAPREDUCE-988) ant package does not copy the capacity-scheduler.jar under HADOOP_HOME/build/hadoop-mapred-0.21.0-dev/contrib/capacity-scheduler
[ https://issues.apache.org/jira/browse/MAPREDUCE-988?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12756496#action_12756496 ] Hudson commented on MAPREDUCE-988: -- Integrated in Hadoop-Mapreduce-trunk-Commit #44 (See [http://hudson.zones.apache.org/hudson/job/Hadoop-Mapreduce-trunk-Commit/44/]) . Fix a packaging issue in the contrib modules. Contributed by Hong Tang > ant package does not copy the capacity-scheduler.jar under > HADOOP_HOME/build/hadoop-mapred-0.21.0-dev/contrib/capacity-scheduler > > > Key: MAPREDUCE-988 > URL: https://issues.apache.org/jira/browse/MAPREDUCE-988 > Project: Hadoop Map/Reduce > Issue Type: Bug > Components: build >Affects Versions: 0.21.0 >Reporter: Iyappan Srinivasan >Assignee: Hong Tang > Fix For: 0.21.0 > > Attachments: mapreduce-988-20090916.patch > > > ant package does not copy the hadoop-0.21.0-dev-capacity-scheduler.jar under > HADOOP_HOME/build/hadoop-mapred-0.21.0-dev/contrib/capacity-scheduler/. > Till yesterday it was copying it properly. Issue seems to be pointing to the > latest checkin of MAPREDUCE-776, which changes build.xml. -- This message is automatically generated by JIRA. - You can reply to this email to add a comment to the issue online.
[jira] Updated: (MAPREDUCE-862) Modify UI to support a hierarchy of queues
[ https://issues.apache.org/jira/browse/MAPREDUCE-862?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Hemanth Yamijala updated MAPREDUCE-862: --- Resolution: Fixed Release Note: - The command line of hadoop queue -list and -info was changed to support hierarchical queues. So, they would now print information about child queues, wherever relevant. - The Web UI of the JobTracker was changed to list queues and queue information in a separate page. Hadoop Flags: [Incompatible change, Reviewed] Status: Resolved (was: Patch Available) I just committed this. Thanks, Chaitanya ! > Modify UI to support a hierarchy of queues > -- > > Key: MAPREDUCE-862 > URL: https://issues.apache.org/jira/browse/MAPREDUCE-862 > Project: Hadoop Map/Reduce > Issue Type: Sub-task >Reporter: Hemanth Yamijala >Assignee: V.V.Chaitanya Krishna > Fix For: 0.21.0 > > Attachments: clustersummarymodification.png, detailspage.png, > initialscreen.png, MAPREDUCE-862-1.patch, MAPREDUCE-862-2.patch, > MAPREDUCE-862-3.patch, MAPREDUCE-862-4.patch, MAPREDUCE-862-5.patch, > MAPREDUCE-862-6.patch, MAPREDUCE-862-7.patch, subqueue.png > > > MAPREDUCE-853 proposes to introduce a hierarchy of queues into the Map/Reduce > framework. This JIRA is for defining changes to the UI related to queues. > This includes the hadoop queue CLI and the web UI on the JobTracker. -- This message is automatically generated by JIRA. - You can reply to this email to add a comment to the issue online.
[jira] Commented: (MAPREDUCE-777) A method for finding and tracking jobs from the new API
[ https://issues.apache.org/jira/browse/MAPREDUCE-777?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12756498#action_12756498 ] Hadoop QA commented on MAPREDUCE-777: - -1 overall. Here are the results of testing the latest attachment http://issues.apache.org/jira/secure/attachment/12419858/patch-777-13.txt against trunk revision 816147. +1 @author. The patch does not contain any @author tags. +1 tests included. The patch appears to include 39 new or modified tests. +1 javadoc. The javadoc tool did not generate any warning messages. -1 javac. The patch appears to cause tar ant target to fail. -1 findbugs. The patch appears to cause Findbugs to fail. +1 release audit. The applied patch does not increase the total number of release audit warnings. -1 core tests. The patch failed core unit tests. -1 contrib tests. The patch failed contrib unit tests. Test results: http://hudson.zones.apache.org/hudson/job/Mapreduce-Patch-h3.grid.sp2.yahoo.net/43/testReport/ Checkstyle results: http://hudson.zones.apache.org/hudson/job/Mapreduce-Patch-h3.grid.sp2.yahoo.net/43/artifact/trunk/build/test/checkstyle-errors.html Console output: http://hudson.zones.apache.org/hudson/job/Mapreduce-Patch-h3.grid.sp2.yahoo.net/43/console This message is automatically generated. > A method for finding and tracking jobs from the new API > --- > > Key: MAPREDUCE-777 > URL: https://issues.apache.org/jira/browse/MAPREDUCE-777 > Project: Hadoop Map/Reduce > Issue Type: New Feature > Components: client >Reporter: Owen O'Malley >Assignee: Amareshwari Sriramadasu > Fix For: 0.21.0 > > Attachments: m-777.patch, patch-777-1.txt, patch-777-10.txt, > patch-777-11.txt, patch-777-12.txt, patch-777-13.txt, patch-777-2.txt, > patch-777-3.txt, patch-777-4.txt, patch-777-5.txt, patch-777-6.txt, > patch-777-7.txt, patch-777-8.txt, patch-777-9.txt, patch-777.txt > > > We need to create a replacement interface for the JobClient API in the new > interface. In particular, the user needs to be able to query and track jobs > that were launched by other processes. -- This message is automatically generated by JIRA. - You can reply to this email to add a comment to the issue online.
[jira] Created: (MAPREDUCE-998) Wrong error message thrown when we try submit to container queue.
Wrong error message thrown when we try submit to container queue. - Key: MAPREDUCE-998 URL: https://issues.apache.org/jira/browse/MAPREDUCE-998 Project: Hadoop Map/Reduce Issue Type: Bug Components: contrib/capacity-sched Affects Versions: 0.21.0 Reporter: Karam Singh Setup have multilevel queue. parant queues a,b and has two child queues a11, a12. If we try sub queue "a" the following error is thrown -": [ org.apache.hadoop.ipc.RemoteException: java.io.IOException: Queue "a" does not exist at org.apache.hadoop.mapred.JobTracker.submitJob(JobTracker.java:2758) at org.apache.hadoop.mapred.JobTracker.submitJob(JobTracker.java:2740) ] where it should have proper like user cannot submit job to container queue. -- This message is automatically generated by JIRA. - You can reply to this email to add a comment to the issue online.
[jira] Updated: (MAPREDUCE-998) Wrong error message thrown when we try submit to container queue.
[ https://issues.apache.org/jira/browse/MAPREDUCE-998?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Karam Singh updated MAPREDUCE-998: -- Description: Setup have multilevel queue. parent queues a,b and has two child queues a11, a12. If we try sub queue "a" the following error is thrown -": [ org.apache.hadoop.ipc.RemoteException: java.io.IOException: Queue "a" does not exist at org.apache.hadoop.mapred.JobTracker.submitJob(JobTracker.java:2758) at org.apache.hadoop.mapred.JobTracker.submitJob(JobTracker.java:2740) ] where it should have proper like user cannot submit job to container queue. was: Setup have multilevel queue. parant queues a,b and has two child queues a11, a12. If we try sub queue "a" the following error is thrown -": [ org.apache.hadoop.ipc.RemoteException: java.io.IOException: Queue "a" does not exist at org.apache.hadoop.mapred.JobTracker.submitJob(JobTracker.java:2758) at org.apache.hadoop.mapred.JobTracker.submitJob(JobTracker.java:2740) ] where it should have proper like user cannot submit job to container queue. > Wrong error message thrown when we try submit to container queue. > - > > Key: MAPREDUCE-998 > URL: https://issues.apache.org/jira/browse/MAPREDUCE-998 > Project: Hadoop Map/Reduce > Issue Type: Bug > Components: contrib/capacity-sched >Affects Versions: 0.21.0 >Reporter: Karam Singh > > Setup have multilevel queue. > parent queues a,b and has two child queues a11, a12. If we try sub queue "a" > the following error is thrown -": > [ > org.apache.hadoop.ipc.RemoteException: java.io.IOException: Queue "a" does > not exist > at org.apache.hadoop.mapred.JobTracker.submitJob(JobTracker.java:2758) > at org.apache.hadoop.mapred.JobTracker.submitJob(JobTracker.java:2740) > ] > where it should have proper like user cannot submit job to container queue. -- This message is automatically generated by JIRA. - You can reply to this email to add a comment to the issue online.
[jira] Updated: (MAPREDUCE-953) Generate configuration dump for hierarchial queue configuration
[ https://issues.apache.org/jira/browse/MAPREDUCE-953?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Hemanth Yamijala updated MAPREDUCE-953: --- Component/s: jobtracker Affects Version/s: 0.21.0 > Generate configuration dump for hierarchial queue configuration > --- > > Key: MAPREDUCE-953 > URL: https://issues.apache.org/jira/browse/MAPREDUCE-953 > Project: Hadoop Map/Reduce > Issue Type: Improvement > Components: jobtracker >Affects Versions: 0.21.0 >Reporter: rahul k singh >Priority: Blocker > Fix For: 0.21.0 > > Attachments: MAPREDUCE-953-1.patch > > > Generate configuration dump for hierarchial queue configuration -- This message is automatically generated by JIRA. - You can reply to this email to add a comment to the issue online.
[jira] Updated: (MAPREDUCE-953) Generate configuration dump for hierarchial queue configuration
[ https://issues.apache.org/jira/browse/MAPREDUCE-953?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Hemanth Yamijala updated MAPREDUCE-953: --- Fix Version/s: 0.21.0 > Generate configuration dump for hierarchial queue configuration > --- > > Key: MAPREDUCE-953 > URL: https://issues.apache.org/jira/browse/MAPREDUCE-953 > Project: Hadoop Map/Reduce > Issue Type: Improvement > Components: jobtracker >Affects Versions: 0.21.0 >Reporter: rahul k singh >Priority: Blocker > Fix For: 0.21.0 > > Attachments: MAPREDUCE-953-1.patch > > > Generate configuration dump for hierarchial queue configuration -- This message is automatically generated by JIRA. - You can reply to this email to add a comment to the issue online.
[jira] Commented: (MAPREDUCE-862) Modify UI to support a hierarchy of queues
[ https://issues.apache.org/jira/browse/MAPREDUCE-862?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12756502#action_12756502 ] Hudson commented on MAPREDUCE-862: -- Integrated in Hadoop-Mapreduce-trunk-Commit #45 (See [http://hudson.zones.apache.org/hudson/job/Hadoop-Mapreduce-trunk-Commit/45/]) . Enhance JobTracker UI to display hierarchical queues. Contributed by V.V.Chaitanya Krishna. > Modify UI to support a hierarchy of queues > -- > > Key: MAPREDUCE-862 > URL: https://issues.apache.org/jira/browse/MAPREDUCE-862 > Project: Hadoop Map/Reduce > Issue Type: Sub-task >Reporter: Hemanth Yamijala >Assignee: V.V.Chaitanya Krishna > Fix For: 0.21.0 > > Attachments: clustersummarymodification.png, detailspage.png, > initialscreen.png, MAPREDUCE-862-1.patch, MAPREDUCE-862-2.patch, > MAPREDUCE-862-3.patch, MAPREDUCE-862-4.patch, MAPREDUCE-862-5.patch, > MAPREDUCE-862-6.patch, MAPREDUCE-862-7.patch, subqueue.png > > > MAPREDUCE-853 proposes to introduce a hierarchy of queues into the Map/Reduce > framework. This JIRA is for defining changes to the UI related to queues. > This includes the hadoop queue CLI and the web UI on the JobTracker. -- This message is automatically generated by JIRA. - You can reply to this email to add a comment to the issue online.
[jira] Assigned: (MAPREDUCE-975) Add an API in job client to get the history file url for a given job id
[ https://issues.apache.org/jira/browse/MAPREDUCE-975?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Sharad Agarwal reassigned MAPREDUCE-975: Assignee: Sharad Agarwal > Add an API in job client to get the history file url for a given job id > --- > > Key: MAPREDUCE-975 > URL: https://issues.apache.org/jira/browse/MAPREDUCE-975 > Project: Hadoop Map/Reduce > Issue Type: Sub-task > Components: client >Reporter: Sharad Agarwal >Assignee: Sharad Agarwal > > MAPREDUCE-817 added an API to get history url in RunningJob. Similar API > should be added in job client to get the history file given a job id. > Something like: > String getHistoryFile(JobId jobid); -- This message is automatically generated by JIRA. - You can reply to this email to add a comment to the issue online.
[jira] Updated: (MAPREDUCE-975) Add an API in job client to get the history file url for a given job id
[ https://issues.apache.org/jira/browse/MAPREDUCE-975?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Sharad Agarwal updated MAPREDUCE-975: - Component/s: jobtracker Fix Version/s: 0.21.0 > Add an API in job client to get the history file url for a given job id > --- > > Key: MAPREDUCE-975 > URL: https://issues.apache.org/jira/browse/MAPREDUCE-975 > Project: Hadoop Map/Reduce > Issue Type: Sub-task > Components: client, jobtracker >Reporter: Sharad Agarwal >Assignee: Sharad Agarwal > Fix For: 0.21.0 > > Attachments: 975_v1.patch > > > MAPREDUCE-817 added an API to get history url in RunningJob. Similar API > should be added in job client to get the history file given a job id. > Something like: > String getHistoryFile(JobId jobid); -- This message is automatically generated by JIRA. - You can reply to this email to add a comment to the issue online.
[jira] Updated: (MAPREDUCE-975) Add an API in job client to get the history file url for a given job id
[ https://issues.apache.org/jira/browse/MAPREDUCE-975?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Sharad Agarwal updated MAPREDUCE-975: - Attachment: 975_v1.patch Patch on top of MAPREDUCE-777 patch. This patch also removes some redundant methods from JobHistory, used only by the test case. > Add an API in job client to get the history file url for a given job id > --- > > Key: MAPREDUCE-975 > URL: https://issues.apache.org/jira/browse/MAPREDUCE-975 > Project: Hadoop Map/Reduce > Issue Type: Sub-task > Components: client, jobtracker >Reporter: Sharad Agarwal >Assignee: Sharad Agarwal > Fix For: 0.21.0 > > Attachments: 975_v1.patch > > > MAPREDUCE-817 added an API to get history url in RunningJob. Similar API > should be added in job client to get the history file given a job id. > Something like: > String getHistoryFile(JobId jobid); -- This message is automatically generated by JIRA. - You can reply to this email to add a comment to the issue online.
[jira] Commented: (MAPREDUCE-997) Acls are working properly when they are to user groups
[ https://issues.apache.org/jira/browse/MAPREDUCE-997?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12756504#action_12756504 ] Hemanth Yamijala commented on MAPREDUCE-997: Karam, did you use -Dhadoop.job.ugi to submit the job ? In which case, can we see if in Hadoop 0.21 maybe we resolve the user to the right group and override the specified value ? Or maybe the resolution of user to group is done on the server, which actually is the right thing to do ? > Acls are working properly when they are to user groups > --- > > Key: MAPREDUCE-997 > URL: https://issues.apache.org/jira/browse/MAPREDUCE-997 > Project: Hadoop Map/Reduce > Issue Type: Bug >Affects Versions: 0.21.0 >Reporter: Karam Singh > > When submit-job-acl set usergroup (ug1). > if user submits a using hadoop.job.ugi=u1,ug2 it is also gets accepted. (user > u1 is also part ug1). > In hadoop 0.20.0, job gets rejected. Its a regression issue. -- This message is automatically generated by JIRA. - You can reply to this email to add a comment to the issue online.
[jira] Assigned: (MAPREDUCE-953) Generate configuration dump for hierarchial queue configuration
[ https://issues.apache.org/jira/browse/MAPREDUCE-953?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] V.V.Chaitanya Krishna reassigned MAPREDUCE-953: --- Assignee: V.V.Chaitanya Krishna > Generate configuration dump for hierarchial queue configuration > --- > > Key: MAPREDUCE-953 > URL: https://issues.apache.org/jira/browse/MAPREDUCE-953 > Project: Hadoop Map/Reduce > Issue Type: Improvement > Components: jobtracker >Affects Versions: 0.21.0 >Reporter: rahul k singh >Assignee: V.V.Chaitanya Krishna >Priority: Blocker > Fix For: 0.21.0 > > Attachments: MAPREDUCE-953-1.patch > > > Generate configuration dump for hierarchial queue configuration -- This message is automatically generated by JIRA. - You can reply to this email to add a comment to the issue online.
[jira] Updated: (MAPREDUCE-649) distcp should validate the data copied
[ https://issues.apache.org/jira/browse/MAPREDUCE-649?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Chris Douglas updated MAPREDUCE-649: Status: Patch Available (was: Open) > distcp should validate the data copied > -- > > Key: MAPREDUCE-649 > URL: https://issues.apache.org/jira/browse/MAPREDUCE-649 > Project: Hadoop Map/Reduce > Issue Type: New Feature > Components: distcp >Reporter: Ravi Gummadi >Assignee: Ravi Gummadi > Attachments: d_verify.patch, d_verify649.patch, d_verify649.v1.patch > > > distcp should validate the files copied by checking the checksums, if the > filesystem supports checksums. -- This message is automatically generated by JIRA. - You can reply to this email to add a comment to the issue online.
[jira] Updated: (MAPREDUCE-777) A method for finding and tracking jobs from the new API
[ https://issues.apache.org/jira/browse/MAPREDUCE-777?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Amareshwari Sriramadasu updated MAPREDUCE-777: -- Status: Open (was: Patch Available) > A method for finding and tracking jobs from the new API > --- > > Key: MAPREDUCE-777 > URL: https://issues.apache.org/jira/browse/MAPREDUCE-777 > Project: Hadoop Map/Reduce > Issue Type: New Feature > Components: client >Reporter: Owen O'Malley >Assignee: Amareshwari Sriramadasu > Fix For: 0.21.0 > > Attachments: m-777.patch, patch-777-1.txt, patch-777-10.txt, > patch-777-11.txt, patch-777-12.txt, patch-777-13.txt, patch-777-2.txt, > patch-777-3.txt, patch-777-4.txt, patch-777-5.txt, patch-777-6.txt, > patch-777-7.txt, patch-777-8.txt, patch-777-9.txt, patch-777.txt > > > We need to create a replacement interface for the JobClient API in the new > interface. In particular, the user needs to be able to query and track jobs > that were launched by other processes. -- This message is automatically generated by JIRA. - You can reply to this email to add a comment to the issue online.
[jira] Updated: (MAPREDUCE-781) distcp overrides user-selected job name
[ https://issues.apache.org/jira/browse/MAPREDUCE-781?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Chris Douglas updated MAPREDUCE-781: Fix Version/s: (was: 0.22.0) (was: 0.21.0) Status: Open (was: Patch Available) > distcp overrides user-selected job name > --- > > Key: MAPREDUCE-781 > URL: https://issues.apache.org/jira/browse/MAPREDUCE-781 > Project: Hadoop Map/Reduce > Issue Type: Bug > Components: distcp >Affects Versions: 0.21.0, 0.22.0 >Reporter: Rob Weltman >Assignee: Venkatesh S > Attachments: MAPREDUCE-781.patch, MAPREDUCE-781.patch, > MAPREDUCE_781.patch > > > distcp hard-codes the hadoop job name to "distcp" even if the user specifies > a job name. This is a problem in general, but especially for generalized > replication services since the Job Tracker UI and history can't be made to > indicate what is being copied in the job name. -- This message is automatically generated by JIRA. - You can reply to this email to add a comment to the issue online.
[jira] Updated: (MAPREDUCE-781) distcp overrides user-selected job name
[ https://issues.apache.org/jira/browse/MAPREDUCE-781?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Chris Douglas updated MAPREDUCE-781: Status: Patch Available (was: Open) > distcp overrides user-selected job name > --- > > Key: MAPREDUCE-781 > URL: https://issues.apache.org/jira/browse/MAPREDUCE-781 > Project: Hadoop Map/Reduce > Issue Type: Bug > Components: distcp >Affects Versions: 0.21.0, 0.22.0 >Reporter: Rob Weltman >Assignee: Venkatesh S > Attachments: MAPREDUCE-781.patch, MAPREDUCE-781.patch, > MAPREDUCE_781.patch > > > distcp hard-codes the hadoop job name to "distcp" even if the user specifies > a job name. This is a problem in general, but especially for generalized > replication services since the Job Tracker UI and history can't be made to > indicate what is being copied in the job name. -- This message is automatically generated by JIRA. - You can reply to this email to add a comment to the issue online.
[jira] Updated: (MAPREDUCE-781) distcp overrides user-selected job name
[ https://issues.apache.org/jira/browse/MAPREDUCE-781?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Chris Douglas updated MAPREDUCE-781: Attachment: MAPREDUCE-781.patch Corrected path > distcp overrides user-selected job name > --- > > Key: MAPREDUCE-781 > URL: https://issues.apache.org/jira/browse/MAPREDUCE-781 > Project: Hadoop Map/Reduce > Issue Type: Bug > Components: distcp >Affects Versions: 0.21.0, 0.22.0 >Reporter: Rob Weltman >Assignee: Venkatesh S > Attachments: MAPREDUCE-781.patch, MAPREDUCE-781.patch, > MAPREDUCE_781.patch > > > distcp hard-codes the hadoop job name to "distcp" even if the user specifies > a job name. This is a problem in general, but especially for generalized > replication services since the Job Tracker UI and history can't be made to > indicate what is being copied in the job name. -- This message is automatically generated by JIRA. - You can reply to this email to add a comment to the issue online.
[jira] Updated: (MAPREDUCE-777) A method for finding and tracking jobs from the new API
[ https://issues.apache.org/jira/browse/MAPREDUCE-777?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Amareshwari Sriramadasu updated MAPREDUCE-777: -- Status: Patch Available (was: Open) > A method for finding and tracking jobs from the new API > --- > > Key: MAPREDUCE-777 > URL: https://issues.apache.org/jira/browse/MAPREDUCE-777 > Project: Hadoop Map/Reduce > Issue Type: New Feature > Components: client >Reporter: Owen O'Malley >Assignee: Amareshwari Sriramadasu > Fix For: 0.21.0 > > Attachments: m-777.patch, patch-777-1.txt, patch-777-10.txt, > patch-777-11.txt, patch-777-12.txt, patch-777-13.txt, patch-777-14.txt, > patch-777-2.txt, patch-777-3.txt, patch-777-4.txt, patch-777-5.txt, > patch-777-6.txt, patch-777-7.txt, patch-777-8.txt, patch-777-9.txt, > patch-777.txt > > > We need to create a replacement interface for the JobClient API in the new > interface. In particular, the user needs to be able to query and track jobs > that were launched by other processes. -- This message is automatically generated by JIRA. - You can reply to this email to add a comment to the issue online.
[jira] Commented: (MAPREDUCE-997) Acls are working properly when they are to user groups
[ https://issues.apache.org/jira/browse/MAPREDUCE-997?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12756519#action_12756519 ] Karam Singh commented on MAPREDUCE-997: --- yes I used hadoop.job.ugi > Acls are working properly when they are to user groups > --- > > Key: MAPREDUCE-997 > URL: https://issues.apache.org/jira/browse/MAPREDUCE-997 > Project: Hadoop Map/Reduce > Issue Type: Bug >Affects Versions: 0.21.0 >Reporter: Karam Singh > > When submit-job-acl set usergroup (ug1). > if user submits a using hadoop.job.ugi=u1,ug2 it is also gets accepted. (user > u1 is also part ug1). > In hadoop 0.20.0, job gets rejected. Its a regression issue. -- This message is automatically generated by JIRA. - You can reply to this email to add a comment to the issue online.
[jira] Updated: (MAPREDUCE-777) A method for finding and tracking jobs from the new API
[ https://issues.apache.org/jira/browse/MAPREDUCE-777?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Amareshwari Sriramadasu updated MAPREDUCE-777: -- Attachment: patch-777-14.txt Patch incorporating comments. > A method for finding and tracking jobs from the new API > --- > > Key: MAPREDUCE-777 > URL: https://issues.apache.org/jira/browse/MAPREDUCE-777 > Project: Hadoop Map/Reduce > Issue Type: New Feature > Components: client >Reporter: Owen O'Malley >Assignee: Amareshwari Sriramadasu > Fix For: 0.21.0 > > Attachments: m-777.patch, patch-777-1.txt, patch-777-10.txt, > patch-777-11.txt, patch-777-12.txt, patch-777-13.txt, patch-777-14.txt, > patch-777-2.txt, patch-777-3.txt, patch-777-4.txt, patch-777-5.txt, > patch-777-6.txt, patch-777-7.txt, patch-777-8.txt, patch-777-9.txt, > patch-777.txt > > > We need to create a replacement interface for the JobClient API in the new > interface. In particular, the user needs to be able to query and track jobs > that were launched by other processes. -- This message is automatically generated by JIRA. - You can reply to this email to add a comment to the issue online.
[jira] Updated: (MAPREDUCE-849) Renaming of configuration property names in mapreduce
[ https://issues.apache.org/jira/browse/MAPREDUCE-849?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Amareshwari Sriramadasu updated MAPREDUCE-849: -- Attachment: patch-849-1.txt Patch incorporating offline comments from Sharad. > Renaming of configuration property names in mapreduce > - > > Key: MAPREDUCE-849 > URL: https://issues.apache.org/jira/browse/MAPREDUCE-849 > Project: Hadoop Map/Reduce > Issue Type: Improvement >Reporter: Amareshwari Sriramadasu >Assignee: Amareshwari Sriramadasu > Fix For: 0.21.0 > > Attachments: Config changes.xls, Config changes.xls, patch-849-1.txt, > patch-849.txt > > > In-line with HDFS-531, property names in configuration files should be > standardized in MAPREDUCE. -- This message is automatically generated by JIRA. - You can reply to this email to add a comment to the issue online.
[jira] Updated: (MAPREDUCE-849) Renaming of configuration property names in mapreduce
[ https://issues.apache.org/jira/browse/MAPREDUCE-849?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Amareshwari Sriramadasu updated MAPREDUCE-849: -- Status: Patch Available (was: Open) > Renaming of configuration property names in mapreduce > - > > Key: MAPREDUCE-849 > URL: https://issues.apache.org/jira/browse/MAPREDUCE-849 > Project: Hadoop Map/Reduce > Issue Type: Improvement >Reporter: Amareshwari Sriramadasu >Assignee: Amareshwari Sriramadasu > Fix For: 0.21.0 > > Attachments: Config changes.xls, Config changes.xls, patch-849-1.txt, > patch-849.txt > > > In-line with HDFS-531, property names in configuration files should be > standardized in MAPREDUCE. -- This message is automatically generated by JIRA. - You can reply to this email to add a comment to the issue online.
[jira] Commented: (MAPREDUCE-953) Generate configuration dump for hierarchial queue configuration
[ https://issues.apache.org/jira/browse/MAPREDUCE-953?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12756544#action_12756544 ] Hemanth Yamijala commented on MAPREDUCE-953: - QueueManager.dumpConfiguration currently assumes that the new configuration is being used because it will call getQueueConfigurationParser with a null configuration, that loads the new QueueConfigurationParser. It should take the Configuration instance from JobTracker, and check if the queues are configured there. If yes, it should return silently as the config would have been dumped in the JT itself. Otherwise, it can use a QueueConfigurationParser - We are not dumping whether acls are enabled or not. - To get the ACLs from the ACLs map, you should use the API toFullPropertyName. - We have no way to separate ACL user names and groups - it will be difficult for admins to separate them, Why not use the existing format itself: u1,u2 g1,g2 - will having a comma be a problem for json ? - At any rate the code to build the list of users and groups should be pulled into a separate utility API and called for both the ACLs. - Use acl_submit_job and acl_administer_jobs as keys. > Generate configuration dump for hierarchial queue configuration > --- > > Key: MAPREDUCE-953 > URL: https://issues.apache.org/jira/browse/MAPREDUCE-953 > Project: Hadoop Map/Reduce > Issue Type: Improvement > Components: jobtracker >Affects Versions: 0.21.0 >Reporter: rahul k singh >Assignee: V.V.Chaitanya Krishna >Priority: Blocker > Fix For: 0.21.0 > > Attachments: MAPREDUCE-953-1.patch > > > Generate configuration dump for hierarchial queue configuration -- This message is automatically generated by JIRA. - You can reply to this email to add a comment to the issue online.
[jira] Commented: (MAPREDUCE-995) JobHistory should handle cases where task completion events are generated after job completion event
[ https://issues.apache.org/jira/browse/MAPREDUCE-995?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12756558#action_12756558 ] Hadoop QA commented on MAPREDUCE-995: - +1 overall. Here are the results of testing the latest attachment http://issues.apache.org/jira/secure/attachment/12419875/mapred-995-v1.patch against trunk revision 816147. +1 @author. The patch does not contain any @author tags. +1 tests included. The patch appears to include 3 new or modified tests. +1 javadoc. The javadoc tool did not generate any warning messages. +1 javac. The applied patch does not increase the total number of javac compiler warnings. +1 findbugs. The patch does not introduce any new Findbugs warnings. +1 release audit. The applied patch does not increase the total number of release audit warnings. +1 core tests. The patch passed core unit tests. +1 contrib tests. The patch passed contrib unit tests. Test results: http://hudson.zones.apache.org/hudson/job/Mapreduce-Patch-h6.grid.sp2.yahoo.net/97/testReport/ Findbugs warnings: http://hudson.zones.apache.org/hudson/job/Mapreduce-Patch-h6.grid.sp2.yahoo.net/97/artifact/trunk/build/test/findbugs/newPatchFindbugsWarnings.html Checkstyle results: http://hudson.zones.apache.org/hudson/job/Mapreduce-Patch-h6.grid.sp2.yahoo.net/97/artifact/trunk/build/test/checkstyle-errors.html Console output: http://hudson.zones.apache.org/hudson/job/Mapreduce-Patch-h6.grid.sp2.yahoo.net/97/console This message is automatically generated. > JobHistory should handle cases where task completion events are generated > after job completion event > > > Key: MAPREDUCE-995 > URL: https://issues.apache.org/jira/browse/MAPREDUCE-995 > Project: Hadoop Map/Reduce > Issue Type: Bug >Reporter: Jothi Padmanabhan >Assignee: Jothi Padmanabhan > Attachments: mapred-995-v1.patch, mapred-995.patch > > > It is apparently possible, in certain circumstances (failed job, for > example), for the job history to get task completion events after the job > completion event. This currently causes NPE in job history. > Thanks Hong for identifying this issue -- This message is automatically generated by JIRA. - You can reply to this email to add a comment to the issue online.
[jira] Commented: (MAPREDUCE-954) The new interface's Context objects should be interfaces
[ https://issues.apache.org/jira/browse/MAPREDUCE-954?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12756559#action_12756559 ] Tom White commented on MAPREDUCE-954: - I've gone back to the cases that have motivated this in order to better understand why this change is needed. The first motivating case is MAPREDUCE-901, which needs to change the constructor to ReduceContext due to an internal type change. As noted above, user applications should never create context objects, so the constructor could be marked as Private Evolving to solve this problem, couldn't it? The other motivating case is MAPREDUCE-372 (chain MapReduce), where various fields of the context need to be replaced with custom versions. The approach described in [this comment|https://issues.apache.org/jira/browse/MAPREDUCE-372?focusedCommentId=12752452&page=com.atlassian.jira.plugin.system.issuetabpanels%3Acomment-tabpanel#action_12752452] which is to add a new constructor to ReduceContext (a "pseudo copy constructor"), could again be made Private Evolving to mark its scope. This approach didn't cause any information leakage as far as I can tell (the previous version had to add extra getters, which did cause leakage). Another approach would be to create package private constructors and have a Private Evolving ContextFactory in the same package if the intent is to keep the constructors further away from user code (in IDE autocomplete, for example). The point is that I think we can avoid the problem of leakage with some changes to visibility annotations. > The new interface's Context objects should be interfaces > > > Key: MAPREDUCE-954 > URL: https://issues.apache.org/jira/browse/MAPREDUCE-954 > Project: Hadoop Map/Reduce > Issue Type: Improvement > Components: client >Reporter: Owen O'Malley >Assignee: Arun C Murthy > Fix For: 0.21.0 > > Attachments: MAPREDUCE-954.patch, MAPREDUCE-954.patch, > MAPREDUCE-954.patch > > > When I was doing HADOOP-1230, I was persuaded to make the Context objects as > classes. I think that was a serious mistake. It caused a lot of information > leakage into the public classes. -- This message is automatically generated by JIRA. - You can reply to this email to add a comment to the issue online.
[jira] Commented: (MAPREDUCE-954) The new interface's Context objects should be interfaces
[
https://issues.apache.org/jira/browse/MAPREDUCE-954?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12756565#action_12756565
]
Tom White commented on MAPREDUCE-954:
-
Only marginally relevant to this discussion, but I noticed that the (existing)
Mapper.Context object in MapTask is constructed via reflection. I think it's
possible to create one directly like this:
{code}
Mapper mapper = ...
Mapper.Context mapperContext = mapper.new Context(...);
{code}
Same for Reducer.Context.
> The new interface's Context objects should be interfaces
>
>
> Key: MAPREDUCE-954
> URL: https://issues.apache.org/jira/browse/MAPREDUCE-954
> Project: Hadoop Map/Reduce
> Issue Type: Improvement
> Components: client
>Reporter: Owen O'Malley
>Assignee: Arun C Murthy
> Fix For: 0.21.0
>
> Attachments: MAPREDUCE-954.patch, MAPREDUCE-954.patch,
> MAPREDUCE-954.patch
>
>
> When I was doing HADOOP-1230, I was persuaded to make the Context objects as
> classes. I think that was a serious mistake. It caused a lot of information
> leakage into the public classes.
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
[jira] Commented: (MAPREDUCE-893) Provide an ability to refresh queue configuration without restart.
[ https://issues.apache.org/jira/browse/MAPREDUCE-893?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12756567#action_12756567 ] rahul k singh commented on MAPREDUCE-893: - comments : CapacityTestUtils: 1. ((FakeQueueManager) qm).getQueue(q.queueName).setProperties(p); No need for casting qm is already FakeQueueManager. TestQueueManagerRefresh 1.testRefreshFailureWithChangeOfHierarchy(JobQueueInfo[] queues => This parameter is not used. QueueManager 1.Check if QueueRefresher parameter is null in refreshQueues method. CapacityTaskScheduler. 1.Check for supportsPriority , it should not be updated as part of refresh. 2.JobInitializationPoller entries are part of capacity-scheduler.xml , but we are not refreshing capacity-scheduler.xml so we need to queue specific JobInitializationPoller entries to QueueSchedulingContext. AbstractQueue 1.addChild method should be documented as being only used for testing. Queue 1. there is no need for maintaining the fullyQualifiedName and name separatly . Instead we shuold always use fullyQualified name everywhere and can provide method to get the normal queueName. > Provide an ability to refresh queue configuration without restart. > -- > > Key: MAPREDUCE-893 > URL: https://issues.apache.org/jira/browse/MAPREDUCE-893 > Project: Hadoop Map/Reduce > Issue Type: Improvement > Components: jobtracker >Reporter: Hemanth Yamijala >Assignee: Vinod K V > Fix For: 0.21.0 > > Attachments: MAPREDUCE-893-20090915.1.txt, > MAPREDUCE-893-20090917.2.txt > > > While administering a cluster using multiple queues, administrators feel a > need to refresh queue properties on the fly without needing to restart the > JobTracker. This is partially supported for some properties such as queue > ACLs (HADOOP-5396) and state (HADOOP-5913). The idea is to extend the > facility to refresh other queue properties as well, including scheduler > properties. -- This message is automatically generated by JIRA. - You can reply to this email to add a comment to the issue online.
[jira] Commented: (MAPREDUCE-953) Generate configuration dump for hierarchial queue configuration
[ https://issues.apache.org/jira/browse/MAPREDUCE-953?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12756569#action_12756569 ] Hemanth Yamijala commented on MAPREDUCE-953: Looked at the test case also. One comment I have is that we should verify the ACLs, properties and children in a different manner I think. Because right now, we are constructing the ACLs in the test case using similar code as in the actual code. This leads to a chance that we could make the same mistake at both places. I think it may be OK to hardcode what we are verifying, rather than writing generically. For e.g. we know what the queue configuration is. We know which queues have what ACLs and what properties, we can hardcode that verification. That way, the test will be simpler and it will be correct. For e.g., now because both test and code make the same mistake about the ACL key, the test doesn't fail, but it doesn't catch the bug as well. > Generate configuration dump for hierarchial queue configuration > --- > > Key: MAPREDUCE-953 > URL: https://issues.apache.org/jira/browse/MAPREDUCE-953 > Project: Hadoop Map/Reduce > Issue Type: Improvement > Components: jobtracker >Affects Versions: 0.21.0 >Reporter: rahul k singh >Assignee: V.V.Chaitanya Krishna >Priority: Blocker > Fix For: 0.21.0 > > Attachments: MAPREDUCE-953-1.patch > > > Generate configuration dump for hierarchial queue configuration -- This message is automatically generated by JIRA. - You can reply to this email to add a comment to the issue online.
[jira] Updated: (MAPREDUCE-728) Mumak: Map-Reduce Simulator
[ https://issues.apache.org/jira/browse/MAPREDUCE-728?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Hong Tang updated MAPREDUCE-728: Attachment: mapreduce-728-20090917.patch Ok, this is the patch that is close to the final form. The unit tests currently would fail due to MAPREDUCE-995. Also, the end-to-end unit test requires two trace files in gzip format. I will upload them separately. > Mumak: Map-Reduce Simulator > --- > > Key: MAPREDUCE-728 > URL: https://issues.apache.org/jira/browse/MAPREDUCE-728 > Project: Hadoop Map/Reduce > Issue Type: New Feature >Reporter: Arun C Murthy >Assignee: Arun C Murthy > Fix For: 0.21.0 > > Attachments: mapreduce-728-20090917.patch, mumak.png > > > h3. Vision: > We want to build a Simulator to simulate large-scale Hadoop clusters, > applications and workloads. This would be invaluable in furthering Hadoop by > providing a tool for researchers and developers to prototype features (e.g. > pluggable block-placement for HDFS, Map-Reduce schedulers etc.) and predict > their behaviour and performance with reasonable amount of confidence, > there-by aiding rapid innovation. > > h3. First Cut: Simulator for the Map-Reduce Scheduler > The Map-Reduce Scheduler is a fertile area of interest with at least four > schedulers, each with their own set of features, currently in existence: > Default Scheduler, Capacity Scheduler, Fairshare Scheduler & Priority > Scheduler. > Each scheduler's scheduling decisions are driven by many factors, such as > fairness, capacity guarantee, resource availability, data-locality etc. > Given that, it is non-trivial to accurately choose a single scheduler or even > a set of desired features to predict the right scheduler (or features) for a > given workload. Hence a simulator which can predict how well a particular > scheduler works for some specific workload by quickly iterating over > schedulers and/or scheduler features would be quite useful. > So, the first cut is to implement a simulator for the Map-Reduce scheduler > which take as input a job trace derived from production workload and a > cluster definition, and simulates the execution of the jobs in as defined in > the trace in this virtual cluster. As output, the detailed job execution > trace (recorded in relation to virtual simulated time) could then be analyzed > to understand various traits of individual schedulers (individual jobs turn > around time, throughput, faireness, capacity guarantee, etc). To support > this, we would need a simulator which could accurately model the conditions > of the actual system which would affect a schedulers decisions. These include > very large-scale clusters (thousands of nodes), the detailed characteristics > of the workload thrown at the clusters, job or task failures, data locality, > and cluster hardware (cpu, memory, disk i/o, network i/o, network topology) > etc. -- This message is automatically generated by JIRA. - You can reply to this email to add a comment to the issue online.
[jira] Updated: (MAPREDUCE-728) Mumak: Map-Reduce Simulator
[ https://issues.apache.org/jira/browse/MAPREDUCE-728?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Hong Tang updated MAPREDUCE-728: Attachment: 19-jobs.trace.json.gz 19-jobs.topology.json.gz These two trace files should go to src/contrib/mumak/src/test/data. > Mumak: Map-Reduce Simulator > --- > > Key: MAPREDUCE-728 > URL: https://issues.apache.org/jira/browse/MAPREDUCE-728 > Project: Hadoop Map/Reduce > Issue Type: New Feature >Reporter: Arun C Murthy >Assignee: Arun C Murthy > Fix For: 0.21.0 > > Attachments: 19-jobs.topology.json.gz, 19-jobs.trace.json.gz, > mapreduce-728-20090917.patch, mumak.png > > > h3. Vision: > We want to build a Simulator to simulate large-scale Hadoop clusters, > applications and workloads. This would be invaluable in furthering Hadoop by > providing a tool for researchers and developers to prototype features (e.g. > pluggable block-placement for HDFS, Map-Reduce schedulers etc.) and predict > their behaviour and performance with reasonable amount of confidence, > there-by aiding rapid innovation. > > h3. First Cut: Simulator for the Map-Reduce Scheduler > The Map-Reduce Scheduler is a fertile area of interest with at least four > schedulers, each with their own set of features, currently in existence: > Default Scheduler, Capacity Scheduler, Fairshare Scheduler & Priority > Scheduler. > Each scheduler's scheduling decisions are driven by many factors, such as > fairness, capacity guarantee, resource availability, data-locality etc. > Given that, it is non-trivial to accurately choose a single scheduler or even > a set of desired features to predict the right scheduler (or features) for a > given workload. Hence a simulator which can predict how well a particular > scheduler works for some specific workload by quickly iterating over > schedulers and/or scheduler features would be quite useful. > So, the first cut is to implement a simulator for the Map-Reduce scheduler > which take as input a job trace derived from production workload and a > cluster definition, and simulates the execution of the jobs in as defined in > the trace in this virtual cluster. As output, the detailed job execution > trace (recorded in relation to virtual simulated time) could then be analyzed > to understand various traits of individual schedulers (individual jobs turn > around time, throughput, faireness, capacity guarantee, etc). To support > this, we would need a simulator which could accurately model the conditions > of the actual system which would affect a schedulers decisions. These include > very large-scale clusters (thousands of nodes), the detailed characteristics > of the workload thrown at the clusters, job or task failures, data locality, > and cluster hardware (cpu, memory, disk i/o, network i/o, network topology) > etc. -- This message is automatically generated by JIRA. - You can reply to this email to add a comment to the issue online.
[jira] Assigned: (MAPREDUCE-728) Mumak: Map-Reduce Simulator
[ https://issues.apache.org/jira/browse/MAPREDUCE-728?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Hong Tang reassigned MAPREDUCE-728: --- Assignee: Hong Tang (was: Arun C Murthy) > Mumak: Map-Reduce Simulator > --- > > Key: MAPREDUCE-728 > URL: https://issues.apache.org/jira/browse/MAPREDUCE-728 > Project: Hadoop Map/Reduce > Issue Type: New Feature >Reporter: Arun C Murthy >Assignee: Hong Tang > Fix For: 0.21.0 > > Attachments: 19-jobs.topology.json.gz, 19-jobs.trace.json.gz, > mapreduce-728-20090917.patch, mumak.png > > > h3. Vision: > We want to build a Simulator to simulate large-scale Hadoop clusters, > applications and workloads. This would be invaluable in furthering Hadoop by > providing a tool for researchers and developers to prototype features (e.g. > pluggable block-placement for HDFS, Map-Reduce schedulers etc.) and predict > their behaviour and performance with reasonable amount of confidence, > there-by aiding rapid innovation. > > h3. First Cut: Simulator for the Map-Reduce Scheduler > The Map-Reduce Scheduler is a fertile area of interest with at least four > schedulers, each with their own set of features, currently in existence: > Default Scheduler, Capacity Scheduler, Fairshare Scheduler & Priority > Scheduler. > Each scheduler's scheduling decisions are driven by many factors, such as > fairness, capacity guarantee, resource availability, data-locality etc. > Given that, it is non-trivial to accurately choose a single scheduler or even > a set of desired features to predict the right scheduler (or features) for a > given workload. Hence a simulator which can predict how well a particular > scheduler works for some specific workload by quickly iterating over > schedulers and/or scheduler features would be quite useful. > So, the first cut is to implement a simulator for the Map-Reduce scheduler > which take as input a job trace derived from production workload and a > cluster definition, and simulates the execution of the jobs in as defined in > the trace in this virtual cluster. As output, the detailed job execution > trace (recorded in relation to virtual simulated time) could then be analyzed > to understand various traits of individual schedulers (individual jobs turn > around time, throughput, faireness, capacity guarantee, etc). To support > this, we would need a simulator which could accurately model the conditions > of the actual system which would affect a schedulers decisions. These include > very large-scale clusters (thousands of nodes), the detailed characteristics > of the workload thrown at the clusters, job or task failures, data locality, > and cluster hardware (cpu, memory, disk i/o, network i/o, network topology) > etc. -- This message is automatically generated by JIRA. - You can reply to this email to add a comment to the issue online.
[jira] Commented: (MAPREDUCE-649) distcp should validate the data copied
[ https://issues.apache.org/jira/browse/MAPREDUCE-649?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12756577#action_12756577 ] Hadoop QA commented on MAPREDUCE-649: - +1 overall. Here are the results of testing the latest attachment http://issues.apache.org/jira/secure/attachment/12419864/d_verify649.v1.patch against trunk revision 816147. +1 @author. The patch does not contain any @author tags. -1 tests included. The patch doesn't appear to include any new or modified tests. Please justify why no new tests are needed for this patch. Also please list what manual steps were performed to verify this patch. +1 javadoc. The javadoc tool did not generate any warning messages. +1 javac. The applied patch does not increase the total number of javac compiler warnings. +1 findbugs. The patch does not introduce any new Findbugs warnings. +1 release audit. The applied patch does not increase the total number of release audit warnings. +1 core tests. The patch passed core unit tests. +1 contrib tests. The patch passed contrib unit tests. Test results: http://hudson.zones.apache.org/hudson/job/Mapreduce-Patch-h3.grid.sp2.yahoo.net/44/testReport/ Findbugs warnings: http://hudson.zones.apache.org/hudson/job/Mapreduce-Patch-h3.grid.sp2.yahoo.net/44/artifact/trunk/build/test/findbugs/newPatchFindbugsWarnings.html Checkstyle results: http://hudson.zones.apache.org/hudson/job/Mapreduce-Patch-h3.grid.sp2.yahoo.net/44/artifact/trunk/build/test/checkstyle-errors.html Console output: http://hudson.zones.apache.org/hudson/job/Mapreduce-Patch-h3.grid.sp2.yahoo.net/44/console This message is automatically generated. > distcp should validate the data copied > -- > > Key: MAPREDUCE-649 > URL: https://issues.apache.org/jira/browse/MAPREDUCE-649 > Project: Hadoop Map/Reduce > Issue Type: New Feature > Components: distcp >Reporter: Ravi Gummadi >Assignee: Ravi Gummadi > Attachments: d_verify.patch, d_verify649.patch, d_verify649.v1.patch > > > distcp should validate the files copied by checking the checksums, if the > filesystem supports checksums. -- This message is automatically generated by JIRA. - You can reply to this email to add a comment to the issue online.
[jira] Commented: (MAPREDUCE-990) Making distributed cache getters in JobContext never return null
[ https://issues.apache.org/jira/browse/MAPREDUCE-990?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12756579#action_12756579 ] Tom White commented on MAPREDUCE-990: - bq. I don't think they should be part of the user API--can I delete them entirely until they actually get used? +1 > Making distributed cache getters in JobContext never return null > > > Key: MAPREDUCE-990 > URL: https://issues.apache.org/jira/browse/MAPREDUCE-990 > Project: Hadoop Map/Reduce > Issue Type: Improvement >Reporter: Philip Zeyliger >Assignee: Philip Zeyliger >Priority: Minor > Attachments: MAPREDUCE-990.patch.txt > > > MAPREDUCE-898 moved distributed cache setters and getters into Job and > JobContext. Since the API is new, I'd like to propose that those getters > never return null, but instead always return an array, even if it's empty. > If people don't like this change, I can instead merely update the javadoc to > reflect the fact that null may be returned. -- This message is automatically generated by JIRA. - You can reply to this email to add a comment to the issue online.
[jira] Updated: (MAPREDUCE-971) distcp does not always remove distcp.tmp.dir
[ https://issues.apache.org/jira/browse/MAPREDUCE-971?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Tom White updated MAPREDUCE-971: Resolution: Fixed Fix Version/s: 0.21.0 Hadoop Flags: [Reviewed] Status: Resolved (was: Patch Available) I've just committed this. Thanks Aaron! > distcp does not always remove distcp.tmp.dir > > > Key: MAPREDUCE-971 > URL: https://issues.apache.org/jira/browse/MAPREDUCE-971 > Project: Hadoop Map/Reduce > Issue Type: Bug > Components: distcp >Reporter: Aaron Kimball >Assignee: Aaron Kimball > Fix For: 0.21.0 > > Attachments: MAPREDUCE-971.patch > > > Sometimes distcp leaves behind its tmpdir when the target filesystem is s3n. -- This message is automatically generated by JIRA. - You can reply to this email to add a comment to the issue online.
[jira] Updated: (MAPREDUCE-907) Sqoop should use more intelligent splits
[ https://issues.apache.org/jira/browse/MAPREDUCE-907?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Tom White updated MAPREDUCE-907: Resolution: Fixed Fix Version/s: 0.21.0 Hadoop Flags: [Reviewed] Status: Resolved (was: Patch Available) I've just committed this. Thanks Aaron! > Sqoop should use more intelligent splits > > > Key: MAPREDUCE-907 > URL: https://issues.apache.org/jira/browse/MAPREDUCE-907 > Project: Hadoop Map/Reduce > Issue Type: Improvement > Components: contrib/sqoop >Reporter: Aaron Kimball >Assignee: Aaron Kimball > Fix For: 0.21.0 > > Attachments: MAPREDUCE-907.2.patch, MAPREDUCE-907.patch > > > Sqoop should use the new split generation / InputFormat in MAPREDUCE-885 -- This message is automatically generated by JIRA. - You can reply to this email to add a comment to the issue online.
[jira] Updated: (MAPREDUCE-997) Acls are not working properly when they are to user groups
[ https://issues.apache.org/jira/browse/MAPREDUCE-997?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Tsz Wo (Nicholas), SZE updated MAPREDUCE-997: - Summary: Acls are not working properly when they are to user groups (was: Acls are working properly when they are to user groups ) I think you mean "Acls are *not* working properly when they are to user groups " in the Summary. > Acls are not working properly when they are to user groups > --- > > Key: MAPREDUCE-997 > URL: https://issues.apache.org/jira/browse/MAPREDUCE-997 > Project: Hadoop Map/Reduce > Issue Type: Bug >Affects Versions: 0.21.0 >Reporter: Karam Singh > > When submit-job-acl set usergroup (ug1). > if user submits a using hadoop.job.ugi=u1,ug2 it is also gets accepted. (user > u1 is also part ug1). > In hadoop 0.20.0, job gets rejected. Its a regression issue. -- This message is automatically generated by JIRA. - You can reply to this email to add a comment to the issue online.
[jira] Updated: (MAPREDUCE-997) Acls are not working properly when they are set to user groups
[ https://issues.apache.org/jira/browse/MAPREDUCE-997?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Hemanth Yamijala updated MAPREDUCE-997: --- Summary: Acls are not working properly when they are set to user groups (was: Acls are not working properly when they are to user groups ) Probably, closer to the intent: "Acls are not working properly when they are set to user groups" > Acls are not working properly when they are set to user groups > -- > > Key: MAPREDUCE-997 > URL: https://issues.apache.org/jira/browse/MAPREDUCE-997 > Project: Hadoop Map/Reduce > Issue Type: Bug >Affects Versions: 0.21.0 >Reporter: Karam Singh > > When submit-job-acl set usergroup (ug1). > if user submits a using hadoop.job.ugi=u1,ug2 it is also gets accepted. (user > u1 is also part ug1). > In hadoop 0.20.0, job gets rejected. Its a regression issue. -- This message is automatically generated by JIRA. - You can reply to this email to add a comment to the issue online.
[jira] Updated: (MAPREDUCE-893) Provide an ability to refresh queue configuration without restart.
[ https://issues.apache.org/jira/browse/MAPREDUCE-893?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Vinod K V updated MAPREDUCE-893: Attachment: MAPREDUCE-893-20090917.4.txt Attaching patch that addresses the above comments. Test cases are also fixed now. Will run it through Hudson. > Provide an ability to refresh queue configuration without restart. > -- > > Key: MAPREDUCE-893 > URL: https://issues.apache.org/jira/browse/MAPREDUCE-893 > Project: Hadoop Map/Reduce > Issue Type: Improvement > Components: jobtracker >Reporter: Hemanth Yamijala >Assignee: Vinod K V > Fix For: 0.21.0 > > Attachments: MAPREDUCE-893-20090915.1.txt, > MAPREDUCE-893-20090917.2.txt, MAPREDUCE-893-20090917.4.txt > > > While administering a cluster using multiple queues, administrators feel a > need to refresh queue properties on the fly without needing to restart the > JobTracker. This is partially supported for some properties such as queue > ACLs (HADOOP-5396) and state (HADOOP-5913). The idea is to extend the > facility to refresh other queue properties as well, including scheduler > properties. -- This message is automatically generated by JIRA. - You can reply to this email to add a comment to the issue online.
[jira] Updated: (MAPREDUCE-893) Provide an ability to refresh queue configuration without restart.
[ https://issues.apache.org/jira/browse/MAPREDUCE-893?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Vinod K V updated MAPREDUCE-893: Status: Patch Available (was: Open) > Provide an ability to refresh queue configuration without restart. > -- > > Key: MAPREDUCE-893 > URL: https://issues.apache.org/jira/browse/MAPREDUCE-893 > Project: Hadoop Map/Reduce > Issue Type: Improvement > Components: jobtracker >Reporter: Hemanth Yamijala >Assignee: Vinod K V > Fix For: 0.21.0 > > Attachments: MAPREDUCE-893-20090915.1.txt, > MAPREDUCE-893-20090917.2.txt, MAPREDUCE-893-20090917.4.txt > > > While administering a cluster using multiple queues, administrators feel a > need to refresh queue properties on the fly without needing to restart the > JobTracker. This is partially supported for some properties such as queue > ACLs (HADOOP-5396) and state (HADOOP-5913). The idea is to extend the > facility to refresh other queue properties as well, including scheduler > properties. -- This message is automatically generated by JIRA. - You can reply to this email to add a comment to the issue online.
[jira] Commented: (MAPREDUCE-893) Provide an ability to refresh queue configuration without restart.
[ https://issues.apache.org/jira/browse/MAPREDUCE-893?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12756608#action_12756608 ] Vinod K V commented on MAPREDUCE-893: - bq. I think we should remove the properties that are no longer supposed to be capacity-scheduler.xml and add a new file - something like mapred-queues-for-cs.xml.template. This would only be for documentation purposes and we could comment that users of capacity scheduler can rename this to mapred-queues.xml adding more queues etc, based on the documentation provided. Works ? I am not taking this up in this JIRA. Can do it in a follow up. > Provide an ability to refresh queue configuration without restart. > -- > > Key: MAPREDUCE-893 > URL: https://issues.apache.org/jira/browse/MAPREDUCE-893 > Project: Hadoop Map/Reduce > Issue Type: Improvement > Components: jobtracker >Reporter: Hemanth Yamijala >Assignee: Vinod K V > Fix For: 0.21.0 > > Attachments: MAPREDUCE-893-20090915.1.txt, > MAPREDUCE-893-20090917.2.txt, MAPREDUCE-893-20090917.4.txt > > > While administering a cluster using multiple queues, administrators feel a > need to refresh queue properties on the fly without needing to restart the > JobTracker. This is partially supported for some properties such as queue > ACLs (HADOOP-5396) and state (HADOOP-5913). The idea is to extend the > facility to refresh other queue properties as well, including scheduler > properties. -- This message is automatically generated by JIRA. - You can reply to this email to add a comment to the issue online.
[jira] Commented: (MAPREDUCE-781) distcp overrides user-selected job name
[ https://issues.apache.org/jira/browse/MAPREDUCE-781?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12756624#action_12756624 ] Hadoop QA commented on MAPREDUCE-781: - +1 overall. Here are the results of testing the latest attachment http://issues.apache.org/jira/secure/attachment/12419877/MAPREDUCE-781.patch against trunk revision 816147. +1 @author. The patch does not contain any @author tags. -1 tests included. The patch doesn't appear to include any new or modified tests. Please justify why no new tests are needed for this patch. Also please list what manual steps were performed to verify this patch. +1 javadoc. The javadoc tool did not generate any warning messages. +1 javac. The applied patch does not increase the total number of javac compiler warnings. +1 findbugs. The patch does not introduce any new Findbugs warnings. +1 release audit. The applied patch does not increase the total number of release audit warnings. +1 core tests. The patch passed core unit tests. +1 contrib tests. The patch passed contrib unit tests. Test results: http://hudson.zones.apache.org/hudson/job/Mapreduce-Patch-h6.grid.sp2.yahoo.net/98/testReport/ Findbugs warnings: http://hudson.zones.apache.org/hudson/job/Mapreduce-Patch-h6.grid.sp2.yahoo.net/98/artifact/trunk/build/test/findbugs/newPatchFindbugsWarnings.html Checkstyle results: http://hudson.zones.apache.org/hudson/job/Mapreduce-Patch-h6.grid.sp2.yahoo.net/98/artifact/trunk/build/test/checkstyle-errors.html Console output: http://hudson.zones.apache.org/hudson/job/Mapreduce-Patch-h6.grid.sp2.yahoo.net/98/console This message is automatically generated. > distcp overrides user-selected job name > --- > > Key: MAPREDUCE-781 > URL: https://issues.apache.org/jira/browse/MAPREDUCE-781 > Project: Hadoop Map/Reduce > Issue Type: Bug > Components: distcp >Affects Versions: 0.21.0, 0.22.0 >Reporter: Rob Weltman >Assignee: Venkatesh S > Attachments: MAPREDUCE-781.patch, MAPREDUCE-781.patch, > MAPREDUCE_781.patch > > > distcp hard-codes the hadoop job name to "distcp" even if the user specifies > a job name. This is a problem in general, but especially for generalized > replication services since the Job Tracker UI and history can't be made to > indicate what is being copied in the job name. -- This message is automatically generated by JIRA. - You can reply to this email to add a comment to the issue online.
[jira] Updated: (MAPREDUCE-953) Generate configuration dump for hierarchial queue configuration
[
https://issues.apache.org/jira/browse/MAPREDUCE-953?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
V.V.Chaitanya Krishna updated MAPREDUCE-953:
Attachment: MAPREDUCE-953-2.patch
Uploading the patch with the above suggested modifications done.
In order to dump acls_enabled, the json format is changed to the following:
{"acls_enabled":true, "queues":[
{"name":,"state":,"submit_job":,
"administer_jobs":,"properties","children":[
{"name":,"state":,
"submit_job":,
"administer_jobs":,"properties":,"children":[] } ]
}
] }
> Generate configuration dump for hierarchial queue configuration
> ---
>
> Key: MAPREDUCE-953
> URL: https://issues.apache.org/jira/browse/MAPREDUCE-953
> Project: Hadoop Map/Reduce
> Issue Type: Improvement
> Components: jobtracker
>Affects Versions: 0.21.0
>Reporter: rahul k singh
>Assignee: V.V.Chaitanya Krishna
>Priority: Blocker
> Fix For: 0.21.0
>
> Attachments: MAPREDUCE-953-1.patch, MAPREDUCE-953-2.patch
>
>
> Generate configuration dump for hierarchial queue configuration
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
[jira] Commented: (MAPREDUCE-679) XML-based metrics as JSP servlet for JobTracker
[ https://issues.apache.org/jira/browse/MAPREDUCE-679?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12756640#action_12756640 ] Aaron Kimball commented on MAPREDUCE-679: - I don't see why using some additional XML writing library is necessary. I have already provided a unit test which ensures that the output of this page is well-formed XML. (For what it's worth, none of the other JSPs have any unit tests at all.) Perhaps there is a broader problem of refactoring the existing JSPs to ensure that all generated HTML/XML is well-formed, but that is outside the scope of this issue. I'll get rid of the buffered job history and pass in a writer; expect a new patch for that soon. > XML-based metrics as JSP servlet for JobTracker > --- > > Key: MAPREDUCE-679 > URL: https://issues.apache.org/jira/browse/MAPREDUCE-679 > Project: Hadoop Map/Reduce > Issue Type: New Feature > Components: jobtracker >Reporter: Aaron Kimball >Assignee: Aaron Kimball > Attachments: example-jobtracker-completed-job.xml, > example-jobtracker-running-job.xml, MAPREDUCE-679.2.patch, > MAPREDUCE-679.3.patch, MAPREDUCE-679.4.patch, MAPREDUCE-679.5.patch, > MAPREDUCE-679.patch > > > In HADOOP-4559, a general REST API for reporting metrics was proposed but > work seems to have stalled. In the interim, we have a simple XML translation > of the existing JobTracker status page which provides the same metrics > (including the tables of running/completed/failed jobs) as the human-readable > page. This is a relatively lightweight addition to provide some > machine-understandable metrics reporting. -- This message is automatically generated by JIRA. - You can reply to this email to add a comment to the issue online.
[jira] Commented: (MAPREDUCE-977) Missing jackson jars from Eclipse template
[ https://issues.apache.org/jira/browse/MAPREDUCE-977?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12756644#action_12756644 ] Hudson commented on MAPREDUCE-977: -- Integrated in Hadoop-Mapreduce-trunk #85 (See [http://hudson.zones.apache.org/hudson/job/Hadoop-Mapreduce-trunk/85/]) . Missing jackson jars from Eclipse template. > Missing jackson jars from Eclipse template > -- > > Key: MAPREDUCE-977 > URL: https://issues.apache.org/jira/browse/MAPREDUCE-977 > Project: Hadoop Map/Reduce > Issue Type: Bug > Components: build >Reporter: Tom White >Assignee: Tom White > Fix For: 0.21.0 > > Attachments: MAPREDUCE-977.patch > > -- This message is automatically generated by JIRA. - You can reply to this email to add a comment to the issue online.
[jira] Commented: (MAPREDUCE-988) ant package does not copy the capacity-scheduler.jar under HADOOP_HOME/build/hadoop-mapred-0.21.0-dev/contrib/capacity-scheduler
[ https://issues.apache.org/jira/browse/MAPREDUCE-988?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12756646#action_12756646 ] Hudson commented on MAPREDUCE-988: -- Integrated in Hadoop-Mapreduce-trunk #85 (See [http://hudson.zones.apache.org/hudson/job/Hadoop-Mapreduce-trunk/85/]) . Fix a packaging issue in the contrib modules. Contributed by Hong Tang > ant package does not copy the capacity-scheduler.jar under > HADOOP_HOME/build/hadoop-mapred-0.21.0-dev/contrib/capacity-scheduler > > > Key: MAPREDUCE-988 > URL: https://issues.apache.org/jira/browse/MAPREDUCE-988 > Project: Hadoop Map/Reduce > Issue Type: Bug > Components: build >Affects Versions: 0.21.0 >Reporter: Iyappan Srinivasan >Assignee: Hong Tang > Fix For: 0.21.0 > > Attachments: mapreduce-988-20090916.patch > > > ant package does not copy the hadoop-0.21.0-dev-capacity-scheduler.jar under > HADOOP_HOME/build/hadoop-mapred-0.21.0-dev/contrib/capacity-scheduler/. > Till yesterday it was copying it properly. Issue seems to be pointing to the > latest checkin of MAPREDUCE-776, which changes build.xml. -- This message is automatically generated by JIRA. - You can reply to this email to add a comment to the issue online.
[jira] Commented: (MAPREDUCE-862) Modify UI to support a hierarchy of queues
[ https://issues.apache.org/jira/browse/MAPREDUCE-862?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12756648#action_12756648 ] Hudson commented on MAPREDUCE-862: -- Integrated in Hadoop-Mapreduce-trunk #85 (See [http://hudson.zones.apache.org/hudson/job/Hadoop-Mapreduce-trunk/85/]) . Enhance JobTracker UI to display hierarchical queues. Contributed by V.V.Chaitanya Krishna. > Modify UI to support a hierarchy of queues > -- > > Key: MAPREDUCE-862 > URL: https://issues.apache.org/jira/browse/MAPREDUCE-862 > Project: Hadoop Map/Reduce > Issue Type: Sub-task >Reporter: Hemanth Yamijala >Assignee: V.V.Chaitanya Krishna > Fix For: 0.21.0 > > Attachments: clustersummarymodification.png, detailspage.png, > initialscreen.png, MAPREDUCE-862-1.patch, MAPREDUCE-862-2.patch, > MAPREDUCE-862-3.patch, MAPREDUCE-862-4.patch, MAPREDUCE-862-5.patch, > MAPREDUCE-862-6.patch, MAPREDUCE-862-7.patch, subqueue.png > > > MAPREDUCE-853 proposes to introduce a hierarchy of queues into the Map/Reduce > framework. This JIRA is for defining changes to the UI related to queues. > This includes the hadoop queue CLI and the web UI on the JobTracker. -- This message is automatically generated by JIRA. - You can reply to this email to add a comment to the issue online.
[jira] Commented: (MAPREDUCE-157) Job History log file format is not friendly for external tools.
[ https://issues.apache.org/jira/browse/MAPREDUCE-157?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12756647#action_12756647 ] Hudson commented on MAPREDUCE-157: -- Integrated in Hadoop-Mapreduce-trunk #85 (See [http://hudson.zones.apache.org/hudson/job/Hadoop-Mapreduce-trunk/85/]) . Refactor job history APIs and change the history format to JSON. Contributed by Jothi Padmanabhan. > Job History log file format is not friendly for external tools. > --- > > Key: MAPREDUCE-157 > URL: https://issues.apache.org/jira/browse/MAPREDUCE-157 > Project: Hadoop Map/Reduce > Issue Type: Sub-task >Affects Versions: 0.20.1 >Reporter: Owen O'Malley >Assignee: Jothi Padmanabhan > Fix For: 0.21.0 > > Attachments: mapred-157-10Sep.patch, mapred-157-15Sep-v1.patch, > mapred-157-15Sep.patch, mapred-157-16Sep-v1.patch, mapred-157-16Sep.patch, > mapred-157-4Sep.patch, mapred-157-7Sep-v1.patch, mapred-157-7Sep.patch, > mapred-157-prelim.patch, MAPREDUCE-157-avro.patch, MAPREDUCE-157-avro.patch > > > Currently, parsing the job history logs with external tools is very difficult > because of the format. The most critical problem is that newlines aren't > escaped in the strings. That makes using tools like grep, sed, and awk very > tricky. -- This message is automatically generated by JIRA. - You can reply to this email to add a comment to the issue online.
[jira] Commented: (MAPREDUCE-946) Fix regression in LineRecordReader to comply with line length parameters
[ https://issues.apache.org/jira/browse/MAPREDUCE-946?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12756645#action_12756645 ] Hudson commented on MAPREDUCE-946: -- Integrated in Hadoop-Mapreduce-trunk #85 (See [http://hudson.zones.apache.org/hudson/job/Hadoop-Mapreduce-trunk/85/]) . Fix a regression in LineRecordReader where the maxBytesToConsume parameter is not set correctly. > Fix regression in LineRecordReader to comply with line length parameters > > > Key: MAPREDUCE-946 > URL: https://issues.apache.org/jira/browse/MAPREDUCE-946 > Project: Hadoop Map/Reduce > Issue Type: Bug >Reporter: Chris Douglas >Assignee: Chris Douglas >Priority: Blocker > Fix For: 0.21.0 > > Attachments: M946-0.patch, M946-1.patch > > > MAPREDUCE-773 accidentally changed code introduced in HADOOP-3144 controlling > max line lengths. The behavior should be restored. -- This message is automatically generated by JIRA. - You can reply to this email to add a comment to the issue online.
[jira] Commented: (MAPREDUCE-954) The new interface's Context objects should be interfaces
[ https://issues.apache.org/jira/browse/MAPREDUCE-954?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12756649#action_12756649 ] Owen O'Malley commented on MAPREDUCE-954: - Thanks Tom, for that. I hadn't seen that syntax for creating nested objects before. The use cases you found are indicative of the problem, but it is really wider than that. The common theme was that the details, especially in the constructors and fields were far more specific than the interfaces. My goal with this jira was to protect ourselves from future needs where a copy constructor is not sufficient. Does that make sense? > The new interface's Context objects should be interfaces > > > Key: MAPREDUCE-954 > URL: https://issues.apache.org/jira/browse/MAPREDUCE-954 > Project: Hadoop Map/Reduce > Issue Type: Improvement > Components: client >Reporter: Owen O'Malley >Assignee: Arun C Murthy > Fix For: 0.21.0 > > Attachments: MAPREDUCE-954.patch, MAPREDUCE-954.patch, > MAPREDUCE-954.patch > > > When I was doing HADOOP-1230, I was persuaded to make the Context objects as > classes. I think that was a serious mistake. It caused a lot of information > leakage into the public classes. -- This message is automatically generated by JIRA. - You can reply to this email to add a comment to the issue online.
