[jira] Commented: (MAPREDUCE-766) Enhance -list-blacklisted-trackers to display host name, blacklisted reason and blacklist report.
[ https://issues.apache.org/jira/browse/MAPREDUCE-766?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=12737035#action_12737035 ] Hemanth Yamijala commented on MAPREDUCE-766: +1 except for a minor nit. A System.out.println in the test case should be removed. Please run it through hudson or equivalent locally. Enhance -list-blacklisted-trackers to display host name, blacklisted reason and blacklist report. - Key: MAPREDUCE-766 URL: https://issues.apache.org/jira/browse/MAPREDUCE-766 Project: Hadoop Map/Reduce Issue Type: Improvement Reporter: Sreekanth Ramakrishnan Assignee: Sreekanth Ramakrishnan Attachments: blacklist3.png, mapreduce-766-1.patch, mapreduce-766-2.patch, mapreduce-766-3.patch, mapreduce-766-4.patch, mapreduce-766-5.patch Currently, the -list-blacklisted-trackers in the mapred job option list only tracker name. We should enhance it to display as hostname, reason for blacklisting and blacklist report. -- This message is automatically generated by JIRA. - You can reply to this email to add a comment to the issue online.
[jira] Commented: (MAPREDUCE-706) Support for FIFO pools in the fair scheduler
[ https://issues.apache.org/jira/browse/MAPREDUCE-706?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=12737034#action_12737034 ] Hadoop QA commented on MAPREDUCE-706: - -1 overall. Here are the results of testing the latest attachment http://issues.apache.org/jira/secure/attachment/12414820/mapreduce-706.v1.patch against trunk revision 799126. +1 @author. The patch does not contain any @author tags. +1 tests included. The patch appears to include 9 new or modified tests. +1 javadoc. The javadoc tool did not generate any warning messages. +1 javac. The applied patch does not increase the total number of javac compiler warnings. +1 findbugs. The patch does not introduce any new Findbugs warnings. -1 release audit. The applied patch generated 210 release audit warnings (more than the trunk's current 203 warnings). +1 core tests. The patch passed core unit tests. -1 contrib tests. The patch failed contrib unit tests. Test results: http://hudson.zones.apache.org/hudson/job/Mapreduce-Patch-vesta.apache.org/427/testReport/ Release audit warnings: http://hudson.zones.apache.org/hudson/job/Mapreduce-Patch-vesta.apache.org/427/artifact/trunk/patchprocess/releaseAuditDiffWarnings.txt Findbugs warnings: http://hudson.zones.apache.org/hudson/job/Mapreduce-Patch-vesta.apache.org/427/artifact/trunk/build/test/findbugs/newPatchFindbugsWarnings.html Checkstyle results: http://hudson.zones.apache.org/hudson/job/Mapreduce-Patch-vesta.apache.org/427/artifact/trunk/build/test/checkstyle-errors.html Console output: http://hudson.zones.apache.org/hudson/job/Mapreduce-Patch-vesta.apache.org/427/console This message is automatically generated. Support for FIFO pools in the fair scheduler Key: MAPREDUCE-706 URL: https://issues.apache.org/jira/browse/MAPREDUCE-706 Project: Hadoop Map/Reduce Issue Type: New Feature Components: contrib/fair-share Reporter: Matei Zaharia Assignee: Matei Zaharia Attachments: fsdesigndoc.pdf, fsdesigndoc.tex, mapreduce-706.patch, mapreduce-706.v1.patch The fair scheduler should support making the internal scheduling algorithm for some pools be FIFO instead of fair sharing in order to work better for batch workloads. FIFO pools will behave exactly like the current default scheduler, sorting jobs by priority and then submission time. Pools will have their scheduling algorithm set through the pools config file, and it will be changeable at runtime. To support this feature, I'm also changing the internal logic of the fair scheduler to no longer use deficits. Instead, for fair sharing, we will assign tasks to the job farthest below its share as a ratio of its share. This is easier to combine with other scheduling algorithms and leads to a more stable sharing situation, avoiding unfairness issues brought up in MAPREDUCE-543 and MAPREDUCE-544 that happen when some jobs have long tasks. The new preemption (MAPREDUCE-551) will ensure that critical jobs can gain their fair share within a bounded amount of time. -- This message is automatically generated by JIRA. - You can reply to this email to add a comment to the issue online.
[jira] Updated: (MAPREDUCE-766) Enhance -list-blacklisted-trackers to display host name, blacklisted reason and blacklist report.
[
https://issues.apache.org/jira/browse/MAPREDUCE-766?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Sreekanth Ramakrishnan updated MAPREDUCE-766:
-
Attachment: mapreduce-766-6.patch
Attaching patch removing the {{System.out.println()}} statement.
Running test patch and tests.
Enhance -list-blacklisted-trackers to display host name, blacklisted reason
and blacklist report.
-
Key: MAPREDUCE-766
URL: https://issues.apache.org/jira/browse/MAPREDUCE-766
Project: Hadoop Map/Reduce
Issue Type: Improvement
Reporter: Sreekanth Ramakrishnan
Assignee: Sreekanth Ramakrishnan
Attachments: blacklist3.png, mapreduce-766-1.patch,
mapreduce-766-2.patch, mapreduce-766-3.patch, mapreduce-766-4.patch,
mapreduce-766-5.patch, mapreduce-766-6.patch
Currently, the -list-blacklisted-trackers in the mapred job option list only
tracker name. We should enhance it to display as hostname, reason for
blacklisting and blacklist report.
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
[jira] Updated: (MAPREDUCE-766) Enhance -list-blacklisted-trackers to display host name, blacklisted reason and blacklist report.
[ https://issues.apache.org/jira/browse/MAPREDUCE-766?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Sreekanth Ramakrishnan updated MAPREDUCE-766: - Status: Patch Available (was: Open) Enhance -list-blacklisted-trackers to display host name, blacklisted reason and blacklist report. - Key: MAPREDUCE-766 URL: https://issues.apache.org/jira/browse/MAPREDUCE-766 Project: Hadoop Map/Reduce Issue Type: Improvement Reporter: Sreekanth Ramakrishnan Assignee: Sreekanth Ramakrishnan Attachments: blacklist3.png, mapreduce-766-1.patch, mapreduce-766-2.patch, mapreduce-766-3.patch, mapreduce-766-4.patch, mapreduce-766-5.patch, mapreduce-766-6.patch Currently, the -list-blacklisted-trackers in the mapred job option list only tracker name. We should enhance it to display as hostname, reason for blacklisting and blacklist report. -- This message is automatically generated by JIRA. - You can reply to this email to add a comment to the issue online.
[jira] Commented: (MAPREDUCE-766) Enhance -list-blacklisted-trackers to display host name, blacklisted reason and blacklist report.
[
https://issues.apache.org/jira/browse/MAPREDUCE-766?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=12737050#action_12737050
]
Sreekanth Ramakrishnan commented on MAPREDUCE-766:
--
output from ant test-patch
{noformat}
[exec] +1 overall.
[exec]
[exec] +1 @author. The patch does not contain any @author tags.
[exec]
[exec] +1 tests included. The patch appears to include 3 new or
modified tests.
[exec]
[exec] +1 javadoc. The javadoc tool did not generate any warning
messages.
[exec]
[exec] +1 javac. The applied patch does not increase the total number
of javac compiler warnings.
[exec]
[exec] +1 findbugs. The patch does not introduce any new Findbugs
warnings.
[exec]
[exec] +1 release audit. The applied patch does not increase the
total number of release audit warnings.
{noformat}
Enhance -list-blacklisted-trackers to display host name, blacklisted reason
and blacklist report.
-
Key: MAPREDUCE-766
URL: https://issues.apache.org/jira/browse/MAPREDUCE-766
Project: Hadoop Map/Reduce
Issue Type: Improvement
Reporter: Sreekanth Ramakrishnan
Assignee: Sreekanth Ramakrishnan
Attachments: blacklist3.png, mapreduce-766-1.patch,
mapreduce-766-2.patch, mapreduce-766-3.patch, mapreduce-766-4.patch,
mapreduce-766-5.patch, mapreduce-766-6.patch
Currently, the -list-blacklisted-trackers in the mapred job option list only
tracker name. We should enhance it to display as hostname, reason for
blacklisting and blacklist report.
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
[jira] Assigned: (MAPREDUCE-763) Capacity scheduler should clean up reservations if it runs tasks on nodes other than where it has made reservations
[ https://issues.apache.org/jira/browse/MAPREDUCE-763?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Hemanth Yamijala reassigned MAPREDUCE-763: -- Assignee: Sreekanth Ramakrishnan Capacity scheduler should clean up reservations if it runs tasks on nodes other than where it has made reservations --- Key: MAPREDUCE-763 URL: https://issues.apache.org/jira/browse/MAPREDUCE-763 Project: Hadoop Map/Reduce Issue Type: Bug Components: contrib/capacity-sched Reporter: Hemanth Yamijala Assignee: Sreekanth Ramakrishnan Currently capacity scheduler makes a reservation on nodes for high memory jobs that cannot currently run at the time. It could happen that in the meantime other tasktrackers become free to run the tasks of this job. Ideally in the next heartbeat from the reserved TTs the reservation should be removed. Otherwise it could unnecessarily block capacity for a while (until the TT has enough slots free to run a task of this job). -- This message is automatically generated by JIRA. - You can reply to this email to add a comment to the issue online.
[jira] Resolved: (MAPREDUCE-757) JobConf will not be deleted from the logs folder if job retires from finalizeJob()
[ https://issues.apache.org/jira/browse/MAPREDUCE-757?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Amar Kamat resolved MAPREDUCE-757. -- Resolution: Duplicate Should be addressed in MAPREDUCE-814. JobConf will not be deleted from the logs folder if job retires from finalizeJob() -- Key: MAPREDUCE-757 URL: https://issues.apache.org/jira/browse/MAPREDUCE-757 Project: Hadoop Map/Reduce Issue Type: Bug Components: jobtracker Reporter: Amar Kamat Assignee: Amar Kamat Attachments: MAPREDUCE-757-v1.0.patch MAPREDUCE-130 fixed the case where the job is retired from the retire jobs thread. But jobs can also retire when the num-job-per-user limit is exceeded. In such cases the conf file will not be deleted. -- This message is automatically generated by JIRA. - You can reply to this email to add a comment to the issue online.
[jira] Commented: (MAPREDUCE-792) javac warnings in DBInputFormat
[ https://issues.apache.org/jira/browse/MAPREDUCE-792?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=12737069#action_12737069 ] Hadoop QA commented on MAPREDUCE-792: - -1 overall. Here are the results of testing the latest attachment http://issues.apache.org/jira/secure/attachment/12414303/MAPREDUCE-792.2.patch against trunk revision 799126. +1 @author. The patch does not contain any @author tags. -1 tests included. The patch doesn't appear to include any new or modified tests. Please justify why no new tests are needed for this patch. Also please list what manual steps were performed to verify this patch. +1 javadoc. The javadoc tool did not generate any warning messages. +1 javac. The applied patch does not increase the total number of javac compiler warnings. +1 findbugs. The patch does not introduce any new Findbugs warnings. +1 release audit. The applied patch does not increase the total number of release audit warnings. -1 core tests. The patch failed core unit tests. -1 contrib tests. The patch failed contrib unit tests. Test results: http://hudson.zones.apache.org/hudson/job/Mapreduce-Patch-vesta.apache.org/428/testReport/ Findbugs warnings: http://hudson.zones.apache.org/hudson/job/Mapreduce-Patch-vesta.apache.org/428/artifact/trunk/build/test/findbugs/newPatchFindbugsWarnings.html Checkstyle results: http://hudson.zones.apache.org/hudson/job/Mapreduce-Patch-vesta.apache.org/428/artifact/trunk/build/test/checkstyle-errors.html Console output: http://hudson.zones.apache.org/hudson/job/Mapreduce-Patch-vesta.apache.org/428/console This message is automatically generated. javac warnings in DBInputFormat --- Key: MAPREDUCE-792 URL: https://issues.apache.org/jira/browse/MAPREDUCE-792 Project: Hadoop Map/Reduce Issue Type: Bug Reporter: Aaron Kimball Assignee: Aaron Kimball Priority: Minor Attachments: MAPREDUCE-792.2.patch, MAPREDUCE-792.patch MAPREDUCE-716 introduces javac warnings -- This message is automatically generated by JIRA. - You can reply to this email to add a comment to the issue online.
[jira] Resolved: (MAPREDUCE-796) Encountered ClassCastException on tasktracker while running wordcount with MultithreadedMapRunner
[ https://issues.apache.org/jira/browse/MAPREDUCE-796?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Devaraj Das resolved MAPREDUCE-796. --- Resolution: Won't Fix Looks like this is a problem caused by a combination of MultiThreadedMapRunner and enabling Native Compression. Encountered ClassCastException on tasktracker while running wordcount with MultithreadedMapRunner --- Key: MAPREDUCE-796 URL: https://issues.apache.org/jira/browse/MAPREDUCE-796 Project: Hadoop Map/Reduce Issue Type: Bug Components: examples Affects Versions: 0.20.1 Reporter: Suman Sehgal ClassCastException for OutOfMemoryError is encountered on tasktracker while running wordcount example with MultithreadedMapRunner. Stack trace : = java.lang.ClassCastException: java.lang.OutOfMemoryError cannot be cast to java.lang.RuntimeException at org.apache.hadoop.mapreduce.lib.map.MultithreadedMapper.run(MultithreadedMapper.java:149) at org.apache.hadoop.mapred.MapTask.runNewMapper(MapTask.java:581) at org.apache.hadoop.mapred.MapTask.run(MapTask.java:303) at org.apache.hadoop.mapred.Child.main(Child.java:170) -- This message is automatically generated by JIRA. - You can reply to this email to add a comment to the issue online.
[jira] Commented: (MAPREDUCE-796) Encountered ClassCastException on tasktracker while running wordcount with MultithreadedMapRunner
[ https://issues.apache.org/jira/browse/MAPREDUCE-796?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=12737093#action_12737093 ] Devaraj Das commented on MAPREDUCE-796: --- Sorry, forgot to add that an increased heap size is a solution to the problem. Encountered ClassCastException on tasktracker while running wordcount with MultithreadedMapRunner --- Key: MAPREDUCE-796 URL: https://issues.apache.org/jira/browse/MAPREDUCE-796 Project: Hadoop Map/Reduce Issue Type: Bug Components: examples Affects Versions: 0.20.1 Reporter: Suman Sehgal ClassCastException for OutOfMemoryError is encountered on tasktracker while running wordcount example with MultithreadedMapRunner. Stack trace : = java.lang.ClassCastException: java.lang.OutOfMemoryError cannot be cast to java.lang.RuntimeException at org.apache.hadoop.mapreduce.lib.map.MultithreadedMapper.run(MultithreadedMapper.java:149) at org.apache.hadoop.mapred.MapTask.runNewMapper(MapTask.java:581) at org.apache.hadoop.mapred.MapTask.run(MapTask.java:303) at org.apache.hadoop.mapred.Child.main(Child.java:170) -- This message is automatically generated by JIRA. - You can reply to this email to add a comment to the issue online.
[jira] Commented: (MAPREDUCE-797) MRUnit MapReduceDriver should support combiners
[ https://issues.apache.org/jira/browse/MAPREDUCE-797?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=12737138#action_12737138 ] Hadoop QA commented on MAPREDUCE-797: - -1 overall. Here are the results of testing the latest attachment http://issues.apache.org/jira/secure/attachment/12414376/MAPREDUCE-797.patch against trunk revision 799126. +1 @author. The patch does not contain any @author tags. +1 tests included. The patch appears to include 3 new or modified tests. +1 javadoc. The javadoc tool did not generate any warning messages. +1 javac. The applied patch does not increase the total number of javac compiler warnings. +1 findbugs. The patch does not introduce any new Findbugs warnings. +1 release audit. The applied patch does not increase the total number of release audit warnings. +1 core tests. The patch passed core unit tests. -1 contrib tests. The patch failed contrib unit tests. Test results: http://hudson.zones.apache.org/hudson/job/Mapreduce-Patch-vesta.apache.org/429/testReport/ Findbugs warnings: http://hudson.zones.apache.org/hudson/job/Mapreduce-Patch-vesta.apache.org/429/artifact/trunk/build/test/findbugs/newPatchFindbugsWarnings.html Checkstyle results: http://hudson.zones.apache.org/hudson/job/Mapreduce-Patch-vesta.apache.org/429/artifact/trunk/build/test/checkstyle-errors.html Console output: http://hudson.zones.apache.org/hudson/job/Mapreduce-Patch-vesta.apache.org/429/console This message is automatically generated. MRUnit MapReduceDriver should support combiners --- Key: MAPREDUCE-797 URL: https://issues.apache.org/jira/browse/MAPREDUCE-797 Project: Hadoop Map/Reduce Issue Type: Improvement Reporter: Aaron Kimball Assignee: Aaron Kimball Attachments: MAPREDUCE-797.patch The MapReduceDriver allows you to specify a mapper and a reducer class with a simple sort/shuffle between the passes. It would be nice to also support another Reducer implementation being used as a combiner in the middle. -- This message is automatically generated by JIRA. - You can reply to this email to add a comment to the issue online.
[jira] Commented: (MAPREDUCE-792) javac warnings in DBInputFormat
[ https://issues.apache.org/jira/browse/MAPREDUCE-792?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=12737156#action_12737156 ] Aaron Kimball commented on MAPREDUCE-792: - unrelated test failures javac warnings in DBInputFormat --- Key: MAPREDUCE-792 URL: https://issues.apache.org/jira/browse/MAPREDUCE-792 Project: Hadoop Map/Reduce Issue Type: Bug Reporter: Aaron Kimball Assignee: Aaron Kimball Priority: Minor Attachments: MAPREDUCE-792.2.patch, MAPREDUCE-792.patch MAPREDUCE-716 introduces javac warnings -- This message is automatically generated by JIRA. - You can reply to this email to add a comment to the issue online.
[jira] Commented: (MAPREDUCE-797) MRUnit MapReduceDriver should support combiners
[ https://issues.apache.org/jira/browse/MAPREDUCE-797?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=12737155#action_12737155 ] Aaron Kimball commented on MAPREDUCE-797: - unrelated test failures MRUnit MapReduceDriver should support combiners --- Key: MAPREDUCE-797 URL: https://issues.apache.org/jira/browse/MAPREDUCE-797 Project: Hadoop Map/Reduce Issue Type: Improvement Reporter: Aaron Kimball Assignee: Aaron Kimball Attachments: MAPREDUCE-797.patch The MapReduceDriver allows you to specify a mapper and a reducer class with a simple sort/shuffle between the passes. It would be nice to also support another Reducer implementation being used as a combiner in the middle. -- This message is automatically generated by JIRA. - You can reply to this email to add a comment to the issue online.
[jira] Updated: (MAPREDUCE-797) MRUnit MapReduceDriver should support combiners
[ https://issues.apache.org/jira/browse/MAPREDUCE-797?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Johan Oskarsson updated MAPREDUCE-797: -- Resolution: Fixed Fix Version/s: 0.21.0 Hadoop Flags: [Reviewed] Status: Resolved (was: Patch Available) I just committed this. Thanks, Aaron! MRUnit MapReduceDriver should support combiners --- Key: MAPREDUCE-797 URL: https://issues.apache.org/jira/browse/MAPREDUCE-797 Project: Hadoop Map/Reduce Issue Type: Improvement Reporter: Aaron Kimball Assignee: Aaron Kimball Fix For: 0.21.0 Attachments: MAPREDUCE-797.patch The MapReduceDriver allows you to specify a mapper and a reducer class with a simple sort/shuffle between the passes. It would be nice to also support another Reducer implementation being used as a combiner in the middle. -- This message is automatically generated by JIRA. - You can reply to this email to add a comment to the issue online.
[jira] Updated: (MAPREDUCE-587) Stream test TestStreamingExitStatus fails with Out of Memory
[ https://issues.apache.org/jira/browse/MAPREDUCE-587?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Devaraj Das updated MAPREDUCE-587: -- Assignee: Amar Kamat Status: Patch Available (was: Open) Passing through hudson (on Amar's behalf).. Stream test TestStreamingExitStatus fails with Out of Memory Key: MAPREDUCE-587 URL: https://issues.apache.org/jira/browse/MAPREDUCE-587 Project: Hadoop Map/Reduce Issue Type: Bug Components: contrib/streaming Environment: OS/X, 64-bit x86 imac, 4GB RAM. Reporter: Steve Loughran Assignee: Amar Kamat Priority: Minor Attachments: MAPREDUCE-587-v1.0.patch contrib/streaming tests are failing a test with an Out of Memory error on an OS/X Mac -same problem does not surface on Linux. -- This message is automatically generated by JIRA. - You can reply to this email to add a comment to the issue online.
[jira] Commented: (MAPREDUCE-670) Create target for 10 minute patch test build for mapreduce
[
https://issues.apache.org/jira/browse/MAPREDUCE-670?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=12737226#action_12737226
]
Konstantin Boudnik commented on MAPREDUCE-670:
--
I'm still seeing two more misaligned spots:
1. Everything below sequential is a body of this element and suppose to be
indented.
{noformat}
+ macrodef name=macro-test-runner
+attribute name=test.file /
+sequential
delete dir=${test.build.data}/
mkdir dir=${test.build.data}/
delete dir=${test.log.dir}/
{noformat}
2. Same is here: body of patternset isn't indented.
{noformat}
+ patternset
+ includesfile name=@{test.file}/
+ /patternset
{noformat}
Looks good otherwise.
Create target for 10 minute patch test build for mapreduce
---
Key: MAPREDUCE-670
URL: https://issues.apache.org/jira/browse/MAPREDUCE-670
Project: Hadoop Map/Reduce
Issue Type: Test
Components: build
Reporter: Jothi Padmanabhan
Assignee: Jothi Padmanabhan
Attachments: FastTestsInfo.xls, mapred-670-v1.patch, mapred-670.patch
Creating a new Jira to track HADOOP-5628 for MapReduce
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
[jira] Commented: (MAPREDUCE-369) Change org.apache.hadoop.mapred.lib.MultipleInputs to use new api.
[ https://issues.apache.org/jira/browse/MAPREDUCE-369?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=12737280#action_12737280 ] Hudson commented on MAPREDUCE-369: -- Integrated in Hadoop-Mapreduce-trunk #33 (See [http://hudson.zones.apache.org/hudson/job/Hadoop-Mapreduce-trunk/33/]) . Change org.apache.hadoop.mapred.lib.MultipleInputs to use new mapreduce API. Contributed by Amareshwari Sriramadasu. Change org.apache.hadoop.mapred.lib.MultipleInputs to use new api. -- Key: MAPREDUCE-369 URL: https://issues.apache.org/jira/browse/MAPREDUCE-369 Project: Hadoop Map/Reduce Issue Type: Sub-task Reporter: Amareshwari Sriramadasu Assignee: Amareshwari Sriramadasu Fix For: 0.21.0 Attachments: patch-369.txt -- This message is automatically generated by JIRA. - You can reply to this email to add a comment to the issue online.
[jira] Commented: (MAPREDUCE-797) MRUnit MapReduceDriver should support combiners
[ https://issues.apache.org/jira/browse/MAPREDUCE-797?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=12737279#action_12737279 ] Hudson commented on MAPREDUCE-797: -- Integrated in Hadoop-Mapreduce-trunk #33 (See [http://hudson.zones.apache.org/hudson/job/Hadoop-Mapreduce-trunk/33/]) . Adds combiner support to MRUnit MapReduceDriver. (Aaron Kimball via johan) MRUnit MapReduceDriver should support combiners --- Key: MAPREDUCE-797 URL: https://issues.apache.org/jira/browse/MAPREDUCE-797 Project: Hadoop Map/Reduce Issue Type: Improvement Reporter: Aaron Kimball Assignee: Aaron Kimball Fix For: 0.21.0 Attachments: MAPREDUCE-797.patch The MapReduceDriver allows you to specify a mapper and a reducer class with a simple sort/shuffle between the passes. It would be nice to also support another Reducer implementation being used as a combiner in the middle. -- This message is automatically generated by JIRA. - You can reply to this email to add a comment to the issue online.
[jira] Commented: (MAPREDUCE-809) Job summary logs show status of completed jobs as RUNNING
[ https://issues.apache.org/jira/browse/MAPREDUCE-809?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=12737281#action_12737281 ] Hudson commented on MAPREDUCE-809: -- Integrated in Hadoop-Mapreduce-trunk #33 (See [http://hudson.zones.apache.org/hudson/job/Hadoop-Mapreduce-trunk/33/]) . Fix job-summary logs to correctly record status of FAILED and KILLED jobs. Job summary logs show status of completed jobs as RUNNING -- Key: MAPREDUCE-809 URL: https://issues.apache.org/jira/browse/MAPREDUCE-809 Project: Hadoop Map/Reduce Issue Type: Bug Components: jobtracker Affects Versions: 0.21.0 Reporter: Arun C Murthy Assignee: Arun C Murthy Fix For: 0.21.0 Attachments: MAPREDUCE-809_0_20090728.patch, MAPREDUCE-809_0_20090728_yhadoop20.patch MAPREDUCE-740 added job summary logs. During testing our QA folks noticed that completed jobs show up as RUNNING in the logs. -- This message is automatically generated by JIRA. - You can reply to this email to add a comment to the issue online.
[jira] Commented: (MAPREDUCE-801) MAPREDUCE framework should issue warning with too many locations for a split
[ https://issues.apache.org/jira/browse/MAPREDUCE-801?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=12737292#action_12737292 ] Arun C Murthy commented on MAPREDUCE-801: - bq. Truncating is probably fine. The original proposal was to discard location information completely. Hmm... truncating is probably fine, but a couple of points to ponder: # The #locations per split to keep should probably be a cluster-wide config limit? # Should we pick first n locations or pick randomly? I'd lean towards randomly picked splits in light of features such as HADOOP-548 # We should do truncation on both the JobClient _and_ JobTracker to be wary of DOS if a malicious client submits too many locations per split... Thoughts? MAPREDUCE framework should issue warning with too many locations for a split Key: MAPREDUCE-801 URL: https://issues.apache.org/jira/browse/MAPREDUCE-801 Project: Hadoop Map/Reduce Issue Type: New Feature Reporter: Hong Tang Customized input-format may be buggy and report misleading locations through input-split, an example of which is PIG-878. When an input split returns too many locations, it would not only artificially inflate the percentage of data local or rack local maps, but also force scheduler to use more memory and work harder to conduct task assignment. -- This message is automatically generated by JIRA. - You can reply to this email to add a comment to the issue online.
[jira] Updated: (MAPREDUCE-813) Streaming Doc and M/R-Tutorial Doc - updates
[ https://issues.apache.org/jira/browse/MAPREDUCE-813?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Corinne Chandel updated MAPREDUCE-813: -- Attachment: MAPREDUCE-813-2.patch Patch file #2. (1) replaced map/reduce with MAP/REDUCE (m/r tutorial doc, streaming doc) (2) removed tabs (m/r tutorial doc) Streaming Doc and M/R-Tutorial Doc - updates - Key: MAPREDUCE-813 URL: https://issues.apache.org/jira/browse/MAPREDUCE-813 Project: Hadoop Map/Reduce Issue Type: Task Components: documentation Affects Versions: 0.21.0 Reporter: Corinne Chandel Priority: Minor Fix For: 0.21.0 Attachments: mapred_tutorial.pdf, MAPREDUCE-813-2.patch, MAPREDUCE-813.patch, streaming.pdf This JIRA addresses issues in the Streaming doc that also require a cross-link to and update in the M/R Tutorial doc. All changes approved by the reviewing engineer. Streaming issues: 1. During the execution of a streaming job, the names of the mapred parameters are transformed. The dots ( . ) become underscores ( _ ). Docs affected: streaming and m/r tutorial (new sub-sections added under Task Execution Environment section) 2. For -files and -archives options, Hadoop now creates symlink with same name as file (user-defined symlinks, #mysymlink, currently not supported) Docs affected:streaming 3. Streaming supports streaming command options and generic command options. Generic options must be placed before streaming options, otherwise command fails. Docs affected: streaming (reorganized the streaming doc to make distinctions between 2 sets of command options more clear) -- This message is automatically generated by JIRA. - You can reply to this email to add a comment to the issue online.
[jira] Updated: (MAPREDUCE-706) Support for FIFO pools in the fair scheduler
[ https://issues.apache.org/jira/browse/MAPREDUCE-706?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Matei Zaharia updated MAPREDUCE-706: Attachment: mapreduce-706.v2.patch I've fixed the release audit warnings by adding Apache license headers to the files in question. The contrib test failures are unrelated to this patch. Support for FIFO pools in the fair scheduler Key: MAPREDUCE-706 URL: https://issues.apache.org/jira/browse/MAPREDUCE-706 Project: Hadoop Map/Reduce Issue Type: New Feature Components: contrib/fair-share Reporter: Matei Zaharia Assignee: Matei Zaharia Attachments: fsdesigndoc.pdf, fsdesigndoc.tex, mapreduce-706.patch, mapreduce-706.v1.patch, mapreduce-706.v2.patch The fair scheduler should support making the internal scheduling algorithm for some pools be FIFO instead of fair sharing in order to work better for batch workloads. FIFO pools will behave exactly like the current default scheduler, sorting jobs by priority and then submission time. Pools will have their scheduling algorithm set through the pools config file, and it will be changeable at runtime. To support this feature, I'm also changing the internal logic of the fair scheduler to no longer use deficits. Instead, for fair sharing, we will assign tasks to the job farthest below its share as a ratio of its share. This is easier to combine with other scheduling algorithms and leads to a more stable sharing situation, avoiding unfairness issues brought up in MAPREDUCE-543 and MAPREDUCE-544 that happen when some jobs have long tasks. The new preemption (MAPREDUCE-551) will ensure that critical jobs can gain their fair share within a bounded amount of time. -- This message is automatically generated by JIRA. - You can reply to this email to add a comment to the issue online.
[jira] Updated: (MAPREDUCE-706) Support for FIFO pools in the fair scheduler
[ https://issues.apache.org/jira/browse/MAPREDUCE-706?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Matei Zaharia updated MAPREDUCE-706: Status: Patch Available (was: Open) Support for FIFO pools in the fair scheduler Key: MAPREDUCE-706 URL: https://issues.apache.org/jira/browse/MAPREDUCE-706 Project: Hadoop Map/Reduce Issue Type: New Feature Components: contrib/fair-share Reporter: Matei Zaharia Assignee: Matei Zaharia Attachments: fsdesigndoc.pdf, fsdesigndoc.tex, mapreduce-706.patch, mapreduce-706.v1.patch, mapreduce-706.v2.patch The fair scheduler should support making the internal scheduling algorithm for some pools be FIFO instead of fair sharing in order to work better for batch workloads. FIFO pools will behave exactly like the current default scheduler, sorting jobs by priority and then submission time. Pools will have their scheduling algorithm set through the pools config file, and it will be changeable at runtime. To support this feature, I'm also changing the internal logic of the fair scheduler to no longer use deficits. Instead, for fair sharing, we will assign tasks to the job farthest below its share as a ratio of its share. This is easier to combine with other scheduling algorithms and leads to a more stable sharing situation, avoiding unfairness issues brought up in MAPREDUCE-543 and MAPREDUCE-544 that happen when some jobs have long tasks. The new preemption (MAPREDUCE-551) will ensure that critical jobs can gain their fair share within a bounded amount of time. -- This message is automatically generated by JIRA. - You can reply to this email to add a comment to the issue online.
[jira] Commented: (MAPREDUCE-476) extend DistributedCache to work locally (LocalJobRunner)
[ https://issues.apache.org/jira/browse/MAPREDUCE-476?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=12737385#action_12737385 ] Philip Zeyliger commented on MAPREDUCE-476: --- Vinod, Yes. I've been hacking away at it today. Please ignore those last two updated diffs: while getting rid of some 80+ character lines, I fumbled some git stuff and produced bad patches. I'll be producing good ones after some more sanity checking either late today or tomorrow morning. -- Philip extend DistributedCache to work locally (LocalJobRunner) Key: MAPREDUCE-476 URL: https://issues.apache.org/jira/browse/MAPREDUCE-476 Project: Hadoop Map/Reduce Issue Type: Improvement Reporter: sam rash Assignee: Philip Zeyliger Priority: Minor Attachments: HADOOP-2914-v1-full.patch, HADOOP-2914-v1-since-4041.patch, HADOOP-2914-v2.patch, HADOOP-2914-v3.patch, MAPREDUCE-476-v2-vs-v3.patch, MAPREDUCE-476-v2.patch, MAPREDUCE-476-v3.patch, MAPREDUCE-476.patch The DistributedCache does not work locally when using the outlined recipe at http://hadoop.apache.org/core/docs/r0.16.0/api/org/apache/hadoop/filecache/DistributedCache.html Ideally, LocalJobRunner would take care of populating the JobConf and copying remote files to the local file sytem (http, assume hdfs = default fs = local fs when doing local development. -- This message is automatically generated by JIRA. - You can reply to this email to add a comment to the issue online.
[jira] Commented: (MAPREDUCE-476) extend DistributedCache to work locally (LocalJobRunner)
[ https://issues.apache.org/jira/browse/MAPREDUCE-476?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=12737391#action_12737391 ] Philip Zeyliger commented on MAPREDUCE-476: --- Never mind, trying to rush before leaving the office, and the tests fail here. Back tomorrow. extend DistributedCache to work locally (LocalJobRunner) Key: MAPREDUCE-476 URL: https://issues.apache.org/jira/browse/MAPREDUCE-476 Project: Hadoop Map/Reduce Issue Type: Improvement Reporter: sam rash Assignee: Philip Zeyliger Priority: Minor Attachments: HADOOP-2914-v1-full.patch, HADOOP-2914-v1-since-4041.patch, HADOOP-2914-v2.patch, HADOOP-2914-v3.patch, MAPREDUCE-476-v2-vs-v3.patch, MAPREDUCE-476-v2-vs-v3.try2.patch, MAPREDUCE-476-v2.patch, MAPREDUCE-476-v3.patch, MAPREDUCE-476-v3.try2.patch, MAPREDUCE-476.patch The DistributedCache does not work locally when using the outlined recipe at http://hadoop.apache.org/core/docs/r0.16.0/api/org/apache/hadoop/filecache/DistributedCache.html Ideally, LocalJobRunner would take care of populating the JobConf and copying remote files to the local file sytem (http, assume hdfs = default fs = local fs when doing local development. -- This message is automatically generated by JIRA. - You can reply to this email to add a comment to the issue online.
[jira] Updated: (MAPREDUCE-670) Create target for 10 minute patch test build for mapreduce
[ https://issues.apache.org/jira/browse/MAPREDUCE-670?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Jothi Padmanabhan updated MAPREDUCE-670: Status: Open (was: Patch Available) Create target for 10 minute patch test build for mapreduce --- Key: MAPREDUCE-670 URL: https://issues.apache.org/jira/browse/MAPREDUCE-670 Project: Hadoop Map/Reduce Issue Type: Test Components: build Reporter: Jothi Padmanabhan Assignee: Jothi Padmanabhan Attachments: FastTestsInfo.xls, mapred-670-v1.patch, mapred-670.patch Creating a new Jira to track HADOOP-5628 for MapReduce -- This message is automatically generated by JIRA. - You can reply to this email to add a comment to the issue online.
[jira] Updated: (MAPREDUCE-670) Create target for 10 minute patch test build for mapreduce
[ https://issues.apache.org/jira/browse/MAPREDUCE-670?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Jothi Padmanabhan updated MAPREDUCE-670: Status: Patch Available (was: Open) Create target for 10 minute patch test build for mapreduce --- Key: MAPREDUCE-670 URL: https://issues.apache.org/jira/browse/MAPREDUCE-670 Project: Hadoop Map/Reduce Issue Type: Test Components: build Reporter: Jothi Padmanabhan Assignee: Jothi Padmanabhan Attachments: FastTestsInfo.xls, mapred-670-v1.patch, mapred-670-v2.patch, mapred-670.patch Creating a new Jira to track HADOOP-5628 for MapReduce -- This message is automatically generated by JIRA. - You can reply to this email to add a comment to the issue online.
[jira] Updated: (MAPREDUCE-670) Create target for 10 minute patch test build for mapreduce
[ https://issues.apache.org/jira/browse/MAPREDUCE-670?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Jothi Padmanabhan updated MAPREDUCE-670: Attachment: mapred-670-v2.patch Fixing the indentation (again !) The original source also had this mismatch and was carried over in the previous patch too :( Create target for 10 minute patch test build for mapreduce --- Key: MAPREDUCE-670 URL: https://issues.apache.org/jira/browse/MAPREDUCE-670 Project: Hadoop Map/Reduce Issue Type: Test Components: build Reporter: Jothi Padmanabhan Assignee: Jothi Padmanabhan Attachments: FastTestsInfo.xls, mapred-670-v1.patch, mapred-670-v2.patch, mapred-670.patch Creating a new Jira to track HADOOP-5628 for MapReduce -- This message is automatically generated by JIRA. - You can reply to this email to add a comment to the issue online.
[jira] Commented: (MAPREDUCE-706) Support for FIFO pools in the fair scheduler
[ https://issues.apache.org/jira/browse/MAPREDUCE-706?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=12737425#action_12737425 ] Aaron Kimball commented on MAPREDUCE-706: - Matei, Great documentation -- that really helps! :) Also good that you added a lot of tests. +1 overall on this patch, subject to the following (relatively minor) questions and suggestions: TestFairScheduler.obtainNewReduceTask(): Task task = new ReduceTask(, attemptId, 0, maps.length, 1) -- shouldn't this be reduces.length ? TestFairScheduler.getLocalityLevel(): These locality level constants are used throughout the FairScheduler; they should be converted to an Enum. (Magic constants are evil.) TestComputeFairShares.testEmptyList() -- should this call verifyShares() after computeFairShares() to assert that the list length is zero? PoolManager.parseSchedulingMode(): why case sensitive 'fifo' and 'fair' ? maybe use toLower() ? PoolSchedulable c'tor: scheduler.getClock().getTime() should be called only once to guarantee this.lastTimeAtMinShare == this.lastTimeAtHalfFairShare on start? assignTask(): Is SchedulingMode guaranteed to never be extended by another internal algorithm? If not, turn else into else if and have an else throw InvalidArgumentException at the end of the case. JobSchedulable.updateDemand(): why does this use System.currentTimeMillis() instead of getting the time from a Clock object? Schedulable's class javadoc: typo algoirthms SchuldingAlgorithms.LOG: rather than use a string, use SchedulingAlgorithms.class.getName() FairScheduler.UpdateThread.run(): why is preemptTasksIfNecessary() commented out? Needs a comment for rationale. FairScheduler.assignTasks() -- Should convert System.out.println to log msg. This method is also getting pretty long. Consider refactoring the inner loop into shorter methods if you need to add anything else to it in the future. getAllowedLocalityLevel(): You have the comment: // Job not in infos (shouldn't happen)- ... So throw an exception if it does, or at least log this event with level ERROR, rather than returning an in-bounds value? When you get to switch(info.lastMapLocalityLevel), you'll naturally throw an NPE, so the caller should just deal with that and clean up its own mess. Support for FIFO pools in the fair scheduler Key: MAPREDUCE-706 URL: https://issues.apache.org/jira/browse/MAPREDUCE-706 Project: Hadoop Map/Reduce Issue Type: New Feature Components: contrib/fair-share Reporter: Matei Zaharia Assignee: Matei Zaharia Attachments: fsdesigndoc.pdf, fsdesigndoc.tex, mapreduce-706.patch, mapreduce-706.v1.patch, mapreduce-706.v2.patch The fair scheduler should support making the internal scheduling algorithm for some pools be FIFO instead of fair sharing in order to work better for batch workloads. FIFO pools will behave exactly like the current default scheduler, sorting jobs by priority and then submission time. Pools will have their scheduling algorithm set through the pools config file, and it will be changeable at runtime. To support this feature, I'm also changing the internal logic of the fair scheduler to no longer use deficits. Instead, for fair sharing, we will assign tasks to the job farthest below its share as a ratio of its share. This is easier to combine with other scheduling algorithms and leads to a more stable sharing situation, avoiding unfairness issues brought up in MAPREDUCE-543 and MAPREDUCE-544 that happen when some jobs have long tasks. The new preemption (MAPREDUCE-551) will ensure that critical jobs can gain their fair share within a bounded amount of time. -- This message is automatically generated by JIRA. - You can reply to this email to add a comment to the issue online.
