[jira] Updated: (MAPREDUCE-945) Test programs support only default queue.
[ https://issues.apache.org/jira/browse/MAPREDUCE-945?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Sreekanth Ramakrishnan updated MAPREDUCE-945: - Attachment: mapreduce-945-1.patch > Test programs support only default queue. > - > > Key: MAPREDUCE-945 > URL: https://issues.apache.org/jira/browse/MAPREDUCE-945 > Project: Hadoop Map/Reduce > Issue Type: Bug > Components: test >Reporter: Suman Sehgal > Attachments: mapreduce-945-1.patch > > > None of the test program seems to be supporting queue's concept. These > programs looks for the default queue only even if some other queue is > specified to run these programs. -- This message is automatically generated by JIRA. - You can reply to this email to add a comment to the issue online.
[jira] Commented: (MAPREDUCE-945) Test programs support only default queue.
[ https://issues.apache.org/jira/browse/MAPREDUCE-945?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12750281#action_12750281 ] Sreekanth Ramakrishnan commented on MAPREDUCE-945: -- After the project split the issue has to be split into two parts. Created a new HDFS JIRA: HDFS-587 > Test programs support only default queue. > - > > Key: MAPREDUCE-945 > URL: https://issues.apache.org/jira/browse/MAPREDUCE-945 > Project: Hadoop Map/Reduce > Issue Type: Bug > Components: test >Reporter: Suman Sehgal > > None of the test program seems to be supporting queue's concept. These > programs looks for the default queue only even if some other queue is > specified to run these programs. -- This message is automatically generated by JIRA. - You can reply to this email to add a comment to the issue online.
[jira] Updated: (MAPREDUCE-884) TestReduceFetchFromPartialMem fails sometimes
[
https://issues.apache.org/jira/browse/MAPREDUCE-884?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Jothi Padmanabhan updated MAPREDUCE-884:
Fix Version/s: 0.21.0
Assignee: Jothi Padmanabhan
Affects Version/s: 0.20.1
Status: Patch Available (was: Open)
> TestReduceFetchFromPartialMem fails sometimes
> -
>
> Key: MAPREDUCE-884
> URL: https://issues.apache.org/jira/browse/MAPREDUCE-884
> Project: Hadoop Map/Reduce
> Issue Type: Bug
> Components: test
>Affects Versions: 0.20.1
>Reporter: Amar Kamat
>Assignee: Jothi Padmanabhan
> Fix For: 0.21.0
>
> Attachments: mapred-884.patch
>
>
> TestReduceFetchFromPartialMem failed with the following exception trace :
> {code}
> Expected some records not spilled during reduce40980)
> junit.framework.AssertionFailedError: Expected some records not spilled
> during reduce40980)
> at
> org.apache.hadoop.mapred.TestReduceFetchFromPartialMem.testReduceFromPartialMem(TestReduceFetchFromPartialMem.java:94)
> at junit.extensions.TestDecorator.basicRun(TestDecorator.java:24)
> at junit.extensions.TestSetup$1.protect(TestSetup.java:23)
> at junit.extensions.TestSetup.run(TestSetup.java:27)
> {code}
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
[jira] Updated: (MAPREDUCE-884) TestReduceFetchFromPartialMem fails sometimes
[
https://issues.apache.org/jira/browse/MAPREDUCE-884?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Jothi Padmanabhan updated MAPREDUCE-884:
Attachment: mapred-884.patch
Simple patch to increase the number of map tasks
> TestReduceFetchFromPartialMem fails sometimes
> -
>
> Key: MAPREDUCE-884
> URL: https://issues.apache.org/jira/browse/MAPREDUCE-884
> Project: Hadoop Map/Reduce
> Issue Type: Bug
> Components: test
>Affects Versions: 0.20.1
>Reporter: Amar Kamat
> Fix For: 0.21.0
>
> Attachments: mapred-884.patch
>
>
> TestReduceFetchFromPartialMem failed with the following exception trace :
> {code}
> Expected some records not spilled during reduce40980)
> junit.framework.AssertionFailedError: Expected some records not spilled
> during reduce40980)
> at
> org.apache.hadoop.mapred.TestReduceFetchFromPartialMem.testReduceFromPartialMem(TestReduceFetchFromPartialMem.java:94)
> at junit.extensions.TestDecorator.basicRun(TestDecorator.java:24)
> at junit.extensions.TestSetup$1.protect(TestSetup.java:23)
> at junit.extensions.TestSetup.run(TestSetup.java:27)
> {code}
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
[jira] Moved: (MAPREDUCE-945) Test programs support only default queue.
[ https://issues.apache.org/jira/browse/MAPREDUCE-945?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Sreekanth Ramakrishnan moved HADOOP-5536 to MAPREDUCE-945: -- Component/s: (was: test) test Affects Version/s: (was: 0.20.0) Key: MAPREDUCE-945 (was: HADOOP-5536) Project: Hadoop Map/Reduce (was: Hadoop Common) > Test programs support only default queue. > - > > Key: MAPREDUCE-945 > URL: https://issues.apache.org/jira/browse/MAPREDUCE-945 > Project: Hadoop Map/Reduce > Issue Type: Bug > Components: test >Reporter: Suman Sehgal > > None of the test program seems to be supporting queue's concept. These > programs looks for the default queue only even if some other queue is > specified to run these programs. -- This message is automatically generated by JIRA. - You can reply to this email to add a comment to the issue online.
[jira] Created: (MAPREDUCE-944) Extend FairShare scheduler to fair-share memory usage in the cluster
Extend FairShare scheduler to fair-share memory usage in the cluster Key: MAPREDUCE-944 URL: https://issues.apache.org/jira/browse/MAPREDUCE-944 Project: Hadoop Map/Reduce Issue Type: Improvement Components: contrib/fair-share Reporter: dhruba borthakur The FairShare Scheduler has an extensible LoadManager API to regulate allocating new tasks on a particular TaskTracker. In similar lines, it would be nice if the FairShare Scheduler can have a pluggable policy to regulate new tasks from a particular job. This will allow one to skip scheduling tasks of a job that is eating a large percentage of memory in the cluster, i.e. fair-share of memory resources among jobs. -- This message is automatically generated by JIRA. - You can reply to this email to add a comment to the issue online.
[jira] Commented: (MAPREDUCE-944) Extend FairShare scheduler to fair-share memory usage in the cluster
[ https://issues.apache.org/jira/browse/MAPREDUCE-944?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12750221#action_12750221 ] dhruba borthakur commented on MAPREDUCE-944: One simple way to do this would be to pass in the LoadManager object into the Schedulable.assignTask(). > Extend FairShare scheduler to fair-share memory usage in the cluster > > > Key: MAPREDUCE-944 > URL: https://issues.apache.org/jira/browse/MAPREDUCE-944 > Project: Hadoop Map/Reduce > Issue Type: Improvement > Components: contrib/fair-share >Reporter: dhruba borthakur > > The FairShare Scheduler has an extensible LoadManager API to regulate > allocating new tasks on a particular TaskTracker. In similar lines, it would > be nice if the FairShare Scheduler can have a pluggable policy to regulate > new tasks from a particular job. This will allow one to skip scheduling tasks > of a job that is eating a large percentage of memory in the cluster, i.e. > fair-share of memory resources among jobs. -- This message is automatically generated by JIRA. - You can reply to this email to add a comment to the issue online.
[jira] Updated: (MAPREDUCE-830) Providing BZip2 splitting support for Text data
[ https://issues.apache.org/jira/browse/MAPREDUCE-830?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Chris Douglas updated MAPREDUCE-830: Attachment: M830-2.patch Corresponding changes in 4012-12 reflected here, including merge with MAPREDUCE-773 > Providing BZip2 splitting support for Text data > --- > > Key: MAPREDUCE-830 > URL: https://issues.apache.org/jira/browse/MAPREDUCE-830 > Project: Hadoop Map/Reduce > Issue Type: Improvement >Affects Versions: 0.21.0 >Reporter: Abdul Qadeer >Assignee: Abdul Qadeer > Fix For: 0.21.0 > > Attachments: M830-2.patch, MapReduce-830-version1.patch > > > HADOOP-4012 (https://issues.apache.org/jira/browse/HADOOP-4012) is providing > support to handle BZip2 compressed data such that the input compressed file > is split at arbitrary points. This JIRA uses that functionality in > LineRecordReader. The benefit of this work is that, if user provides > compressed BZip2 Text data, it will be split by Hadoop and hence will be > processed by multiple mappers. So BZip2 compressed data will be able to > fully utilize the cluster power. Currently BZip2 compressed Text file goes > to one mapper and is not split. So the enhancement in this JIRA provides > splitting support and a considerable performance gains. -- This message is automatically generated by JIRA. - You can reply to this email to add a comment to the issue online.
[jira] Commented: (MAPREDUCE-773) LineRecordReader can report non-zero progress while it is processing a compressed stream
[ https://issues.apache.org/jira/browse/MAPREDUCE-773?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12750183#action_12750183 ] Hong Tang commented on MAPREDUCE-773: - My bad. I overlooked the difference between these two similar places. > LineRecordReader can report non-zero progress while it is processing a > compressed stream > > > Key: MAPREDUCE-773 > URL: https://issues.apache.org/jira/browse/MAPREDUCE-773 > Project: Hadoop Map/Reduce > Issue Type: Bug > Components: task >Reporter: Devaraj Das >Assignee: Devaraj Das > Fix For: 0.21.0 > > Attachments: 773.2.patch, 773.3.patch, 773.patch, 773.patch > > > Currently, the LineRecordReader returns 0.0 from getProgress() for most > inputs (since the "end" of the filesplit is set to Long.MAX_VALUE for > compressed inputs). This can be improved to return a non-zero progress even > for compressed streams (though it may not be very reflective of the actual > progress). -- This message is automatically generated by JIRA. - You can reply to this email to add a comment to the issue online.
[jira] Commented: (MAPREDUCE-773) LineRecordReader can report non-zero progress while it is processing a compressed stream
[
https://issues.apache.org/jira/browse/MAPREDUCE-773?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12750179#action_12750179
]
Chris Douglas commented on MAPREDUCE-773:
-
This change does not preserve the existing behavior:
{noformat}
-Math.max((int)Math.min(Integer.MAX_VALUE, end-pos),
- maxLineLength));
+Math.max(maxBytesToConsume(), maxLineLength));
{noformat}
{noformat}
+ private int maxBytesToConsume() {
+return (isCompressedInput()) ? Integer.MAX_VALUE
+ : (int) Math.min(Integer.MAX_VALUE, (end - start));
+ }
{noformat}
Instead of {{end - pos}}, this uses {{end - start}} if less than maxint. This
is a regression in HADOOP-3144
> LineRecordReader can report non-zero progress while it is processing a
> compressed stream
>
>
> Key: MAPREDUCE-773
> URL: https://issues.apache.org/jira/browse/MAPREDUCE-773
> Project: Hadoop Map/Reduce
> Issue Type: Bug
> Components: task
>Reporter: Devaraj Das
>Assignee: Devaraj Das
> Fix For: 0.21.0
>
> Attachments: 773.2.patch, 773.3.patch, 773.patch, 773.patch
>
>
> Currently, the LineRecordReader returns 0.0 from getProgress() for most
> inputs (since the "end" of the filesplit is set to Long.MAX_VALUE for
> compressed inputs). This can be improved to return a non-zero progress even
> for compressed streams (though it may not be very reflective of the actual
> progress).
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
[jira] Commented: (MAPREDUCE-421) mapred pipes might return exit code 0 even when failing
[ https://issues.apache.org/jira/browse/MAPREDUCE-421?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12750070#action_12750070 ] Christian Kunz commented on MAPREDUCE-421: -- 1) Tested by submitting 'mapred pipes' commands with incorrect parameters. Obviously the patch does not impact correct pipes jobs, otherwise we would have seen test failures. 2) I would interpret Amareshwari's '+1' as an indication that the tests failed for unrelated reasons (unfortunately, cannot be verified, as the link to the testReport no longer exists). That test fail for unrelated reasons is not unusual -- it is my subjective impression that it happens often. > mapred pipes might return exit code 0 even when failing > --- > > Key: MAPREDUCE-421 > URL: https://issues.apache.org/jira/browse/MAPREDUCE-421 > Project: Hadoop Map/Reduce > Issue Type: Bug > Components: pipes >Reporter: Christian Kunz >Assignee: Christian Kunz > Fix For: 0.20.1 > > Attachments: MAPREDUCE-421.patch > > > up to hadoop 0.18.3 org.apache.hadoop.mapred.JobShell ensured that 'hadoop > jar' returns non-zero exit code when the job fails. > This is no longer true after moving this to org.apache.hadoop.util.RunJar. > Pipes jobs submitted through cli never returned proper exit code. > The main methods in org.apache.hadoop.util.RunJar. and > org.apache.hadoop.mapred.pipes.Submitter should be modified to return an exit > code similar to how org.apache.hadoop.mapred.JobShell did it. -- This message is automatically generated by JIRA. - You can reply to this email to add a comment to the issue online.
[jira] Commented: (MAPREDUCE-943) TestNodeRefresh timesout occasionally
[
https://issues.apache.org/jira/browse/MAPREDUCE-943?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12750054#action_12750054
]
Konstantin Boudnik commented on MAPREDUCE-943:
--
It seems to be caused the unhandled crash in MiniMRCluster.startTaskTracker
{noformat}
[exec] [junit] 2009-08-31 11:59:24,583 INFO mapred.TaskTracker
(TaskTracker.java:initialize(567)) - TaskTracker up at:
localhost/127.0.0.1:45234
[exec] [junit] 2009-08-31 11:59:24,583 INFO mapred.TaskTracker
(TaskTracker.java:initialize(570)) - Starting tracker
tracker_host2.com:localhost/127.0.0.1:45234
[exec] [junit] 2009-08-31 11:59:24,584 ERROR mapred.MiniMRCluster
(MiniMRCluster.java:(194)) - task tracker 2 crashed
[exec] [junit] java.io.IOException: Call to localhost/127.0.0.1:47063
failed on local exception: java.io.EOFException
[exec] [junit] at
org.apache.hadoop.ipc.Client.wrapException(Client.java:801)
[exec] [junit] at
org.apache.hadoop.ipc.Client.call(Client.java:769)
...
[exec] [junit] at
org.apache.hadoop.mapred.MiniMRCluster$TaskTrackerRunner.(MiniMRCluster.java:189)
[exec] [junit] at
org.apache.hadoop.mapred.MiniMRCluster.startTaskTracker(MiniMRCluster.java:675)
[exec] [junit] at
org.apache.hadoop.mapred.TestNodeRefresh.testMRExcludeHostsAcrossRestarts(TestNodeRefresh.java:455)
{noformat}
> TestNodeRefresh timesout occasionally
> -
>
> Key: MAPREDUCE-943
> URL: https://issues.apache.org/jira/browse/MAPREDUCE-943
> Project: Hadoop Map/Reduce
> Issue Type: Bug
> Components: jobtracker
>Reporter: Amareshwari Sriramadasu
> Fix For: 0.21.0
>
>
> TestNodeRefresh timesout occasionally.
> One of the hudson patch build with timeout
> @http://hudson.zones.apache.org/hudson/job/Mapreduce-Patch-h6.grid.sp2.yahoo.net/26/testReport/org.apache.hadoop.mapred/TestNodeRefresh/testMRExcludeHostsAcrossRestarts/
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
[jira] Commented: (MAPREDUCE-421) mapred pipes might return exit code 0 even when failing
[ https://issues.apache.org/jira/browse/MAPREDUCE-421?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12750025#action_12750025 ] gary murry commented on MAPREDUCE-421: -- Hey, even if this hard to write new unit tests for, could we get a note on how it was tested? Also a note on why we were not concerned by the failed core and contrib tests? Thanks. > mapred pipes might return exit code 0 even when failing > --- > > Key: MAPREDUCE-421 > URL: https://issues.apache.org/jira/browse/MAPREDUCE-421 > Project: Hadoop Map/Reduce > Issue Type: Bug > Components: pipes >Reporter: Christian Kunz >Assignee: Christian Kunz > Fix For: 0.20.1 > > Attachments: MAPREDUCE-421.patch > > > up to hadoop 0.18.3 org.apache.hadoop.mapred.JobShell ensured that 'hadoop > jar' returns non-zero exit code when the job fails. > This is no longer true after moving this to org.apache.hadoop.util.RunJar. > Pipes jobs submitted through cli never returned proper exit code. > The main methods in org.apache.hadoop.util.RunJar. and > org.apache.hadoop.mapred.pipes.Submitter should be modified to return an exit > code similar to how org.apache.hadoop.mapred.JobShell did it. -- This message is automatically generated by JIRA. - You can reply to this email to add a comment to the issue online.
[jira] Commented: (MAPREDUCE-839) unit test TestMiniMRChildTask fails on mac os-x
[ https://issues.apache.org/jira/browse/MAPREDUCE-839?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12750017#action_12750017 ] Hong Tang commented on MAPREDUCE-839: - bq. I'd be happier driving this from JVM properties than env variables, as its easier to set up JVM properties under a debugger or from build.properties. The reason I choose to inspect env variables TEMP and TMP is because if they are set, we should try to honor those locations. But I am fine to support using JVM properties (in case altering the global TMP/TEMP would affect other test cases). bq. Assuming is forking, you can set up env variables to pass down Not quite clear what you mean, do you suggest to test whether the aforementioned properties are set, and if so, set env variables TEMP and TMP to those values? > unit test TestMiniMRChildTask fails on mac os-x > --- > > Key: MAPREDUCE-839 > URL: https://issues.apache.org/jira/browse/MAPREDUCE-839 > Project: Hadoop Map/Reduce > Issue Type: Bug >Affects Versions: 0.21.0 >Reporter: Hong Tang >Assignee: Hong Tang >Priority: Minor > Fix For: 0.21.0 > > Attachments: mapreduce-839-20090828.patch > > > The unit test TestMiniMRChildTask fails on Mac OS-X (10.5.8) -- This message is automatically generated by JIRA. - You can reply to this email to add a comment to the issue online.
[jira] Updated: (MAPREDUCE-941) vaidya script calls awk instead of nawk
[
https://issues.apache.org/jira/browse/MAPREDUCE-941?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Chad Metcalf updated MAPREDUCE-941:
---
Attachment: MapReduce941.patch
This is perhaps a better alternative. The script current already makes an
assumption that the version information is in the second half of the first
line. This changes the assumption to second half of a line starting with
Hadoop. So it isn't any less maintainable. It improves readability. And doesn't
require a user to manipulate paths or have nawk installed.
The patch uses the most basic (read portable) functionality from both grep and
awk.
> vaidya script calls awk instead of nawk
> ---
>
> Key: MAPREDUCE-941
> URL: https://issues.apache.org/jira/browse/MAPREDUCE-941
> Project: Hadoop Map/Reduce
> Issue Type: Bug
> Environment: Solaris
>Reporter: Allen Wittenauer
>Priority: Trivial
> Attachments: MapReduce941.patch
>
>
> The vaidya script uses this construction:
> awk 'BEGIN { RS = "" ; FS = "\n" } ; { print $1 }'
> Under "old awk", this fails. Under "new awk", this works. Depending upon
> the OS, awk may point to either old or new awk. In the Solaris case, awk
> defaults to oawk.
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
[jira] Commented: (MAPREDUCE-924) TestPipes crashes on trunk
[ https://issues.apache.org/jira/browse/MAPREDUCE-924?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12750013#action_12750013 ] Hudson commented on MAPREDUCE-924: -- Integrated in Hadoop-Mapreduce-trunk-Commit #10 (See [http://hudson.zones.apache.org/hudson/job/Hadoop-Mapreduce-trunk-Commit/10/]) . Fixes the TestPipes testcase to use Tool. Contributed by Amareshwari Sriramadasu. > TestPipes crashes on trunk > -- > > Key: MAPREDUCE-924 > URL: https://issues.apache.org/jira/browse/MAPREDUCE-924 > Project: Hadoop Map/Reduce > Issue Type: Bug > Components: pipes >Affects Versions: 0.20.1 >Reporter: Amareshwari Sriramadasu >Assignee: Amareshwari Sriramadasu > Fix For: 0.20.1 > > Attachments: patch-924-0.20.txt, patch-924.txt > > > TestPipes crashes on trunk due to MAPREDUCE-421. > Testcase should be modified to use Tool insteadof running main directly. -- This message is automatically generated by JIRA. - You can reply to this email to add a comment to the issue online.
[jira] Updated: (MAPREDUCE-924) TestPipes crashes on trunk
[ https://issues.apache.org/jira/browse/MAPREDUCE-924?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Devaraj Das updated MAPREDUCE-924: -- Resolution: Fixed Status: Resolved (was: Patch Available) I just committed this. Thanks, Amareshwari! > TestPipes crashes on trunk > -- > > Key: MAPREDUCE-924 > URL: https://issues.apache.org/jira/browse/MAPREDUCE-924 > Project: Hadoop Map/Reduce > Issue Type: Bug > Components: pipes >Affects Versions: 0.20.1 >Reporter: Amareshwari Sriramadasu >Assignee: Amareshwari Sriramadasu > Fix For: 0.20.1 > > Attachments: patch-924-0.20.txt, patch-924.txt > > > TestPipes crashes on trunk due to MAPREDUCE-421. > Testcase should be modified to use Tool insteadof running main directly. -- This message is automatically generated by JIRA. - You can reply to this email to add a comment to the issue online.
[jira] Commented: (MAPREDUCE-932) Rumen needs a job trace sorter
[ https://issues.apache.org/jira/browse/MAPREDUCE-932?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12749899#action_12749899 ] Dick King commented on MAPREDUCE-932: - The findbugs warnings will be relatively routine to fix. The release audit is two test cases which cannot reasonably get a banner ASF license comment. The core tests that fail fail also in the base. > Rumen needs a job trace sorter > -- > > Key: MAPREDUCE-932 > URL: https://issues.apache.org/jira/browse/MAPREDUCE-932 > Project: Hadoop Map/Reduce > Issue Type: New Feature >Reporter: Dick King >Assignee: Dick King > Attachments: patch-932--2009-08-31--1702.patch > > > Rumen reads job history logs and produces job traces. The jobs in a job > trace do not occur in any promised order. Certain tools need the jobs to be > ordered by job submission time. We should include, in Rumen, a tool to sort > job traces. -- This message is automatically generated by JIRA. - You can reply to this email to add a comment to the issue online.
[jira] Commented: (MAPREDUCE-931) rumen should use its own interpolation classes to create runtimes for simulated tasks
[ https://issues.apache.org/jira/browse/MAPREDUCE-931?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12749896#action_12749896 ] Dick King commented on MAPREDUCE-931: - The core tests are those failed without the patch. I'll look into the new warnings Wednesday. > rumen should use its own interpolation classes to create runtimes for > simulated tasks > - > > Key: MAPREDUCE-931 > URL: https://issues.apache.org/jira/browse/MAPREDUCE-931 > Project: Hadoop Map/Reduce > Issue Type: Improvement >Reporter: Dick King >Assignee: Dick King >Priority: Minor > Attachments: patch-931-b.patch > > > Currently, when a simulator or benchmark is running and simulating hadoop > jobs using rumen data, and rumen's runtime system is used to get execution > times for the tasks in the simulated jobs, rumen would use some ad hoc code, > despite the fact that rumen has a perfectly good interpolation framework to > generate random variables that fit discrete CDFs. > We should use the interpolation framework. -- This message is automatically generated by JIRA. - You can reply to this email to add a comment to the issue online.
[jira] Commented: (MAPREDUCE-930) rumen should interpret job history log input paths with respect to default FS, not local FS
[ https://issues.apache.org/jira/browse/MAPREDUCE-930?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12749894#action_12749894 ] Dick King commented on MAPREDUCE-930: - The core tests are probably tests that are failed even without the patch. The findbugs warning link does not serve. I'll look into this Wednesday. > rumen should interpret job history log input paths with respect to default > FS, not local FS > --- > > Key: MAPREDUCE-930 > URL: https://issues.apache.org/jira/browse/MAPREDUCE-930 > Project: Hadoop Map/Reduce > Issue Type: Improvement >Reporter: Dick King >Assignee: Dick King >Priority: Minor > Attachments: patch-930.patch > > > This allows job history log file/directory names that don't specify a file > system to use the configured default FS instead of the local FS [when the > configured default is not the local]. -- This message is automatically generated by JIRA. - You can reply to this email to add a comment to the issue online.
[jira] Commented: (MAPREDUCE-873) Simplify Job Recovery
[ https://issues.apache.org/jira/browse/MAPREDUCE-873?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12749849#action_12749849 ] Hudson commented on MAPREDUCE-873: -- Integrated in Hadoop-Mapreduce-trunk-Commit #9 (See [http://hudson.zones.apache.org/hudson/job/Hadoop-Mapreduce-trunk-Commit/9/]) . Moving the CHANGES.txt comment to Incompatible section. . Simplify job recovery. InComplete jobs are resubmitted on jobtracker restart. Contributed by Sharad Agarwal. > Simplify Job Recovery > - > > Key: MAPREDUCE-873 > URL: https://issues.apache.org/jira/browse/MAPREDUCE-873 > Project: Hadoop Map/Reduce > Issue Type: Improvement > Components: jobtracker >Affects Versions: 0.20.1 >Reporter: Devaraj Das >Assignee: Sharad Agarwal > Fix For: 0.21.0 > > Attachments: 873_v1.patch, 873_v2.patch, 873_v3.patch > > > On a couple of occasions we have seen the JobTracker not being able to handle > job recovery well, and leading to cluster downtime after a restart. The > current design for handling job recovery is complex and prone to corner cases > not being handled well enough. In retrospect, it seems like the transaction > log based approach as was proposed on HADOOP-3245 > (http://tinyurl.com/luh9hb), would have been a better/simpler model. However, > that is a big project, and it seems for the medium term, just handling job > re-submissions after a restart is a good tradeoff. That is, the JobTracker > after getting restarted, will resubmit all jobs that were running in its past > life. They will all start from the beginning (downside is completed tasks > will reexecute). In the long term, the transaction log model or some variant > of that should be pursued. > Thoughts/comments welcome. -- This message is automatically generated by JIRA. - You can reply to this email to add a comment to the issue online.
[jira] Commented: (MAPREDUCE-839) unit test TestMiniMRChildTask fails on mac os-x
[ https://issues.apache.org/jira/browse/MAPREDUCE-839?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12749843#action_12749843 ] Steve Loughran commented on MAPREDUCE-839: -- # I'd be happier driving this from JVM properties than env variables, as its easier to set up JVM properties under a debugger or from build.properties. # Assuming is forking, you can set up env variables to pass down > unit test TestMiniMRChildTask fails on mac os-x > --- > > Key: MAPREDUCE-839 > URL: https://issues.apache.org/jira/browse/MAPREDUCE-839 > Project: Hadoop Map/Reduce > Issue Type: Bug >Affects Versions: 0.21.0 >Reporter: Hong Tang >Assignee: Hong Tang >Priority: Minor > Fix For: 0.21.0 > > Attachments: mapreduce-839-20090828.patch > > > The unit test TestMiniMRChildTask fails on Mac OS-X (10.5.8) -- This message is automatically generated by JIRA. - You can reply to this email to add a comment to the issue online.
[jira] Commented: (MAPREDUCE-931) rumen should use its own interpolation classes to create runtimes for simulated tasks
[ https://issues.apache.org/jira/browse/MAPREDUCE-931?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12749839#action_12749839 ] Hadoop QA commented on MAPREDUCE-931: - -1 overall. Here are the results of testing the latest attachment http://issues.apache.org/jira/secure/attachment/12418199/patch-931-b.patch against trunk revision 809843. +1 @author. The patch does not contain any @author tags. -1 tests included. The patch doesn't appear to include any new or modified tests. Please justify why no new tests are needed for this patch. Also please list what manual steps were performed to verify this patch. +1 javadoc. The javadoc tool did not generate any warning messages. -1 javac. The applied patch generated 2228 javac compiler warnings (more than the trunk's current 2226 warnings). +1 findbugs. The patch does not introduce any new Findbugs warnings. +1 release audit. The applied patch does not increase the total number of release audit warnings. -1 core tests. The patch failed core unit tests. +1 contrib tests. The patch passed contrib unit tests. Test results: http://hudson.zones.apache.org/hudson/job/Mapreduce-Patch-h3.grid.sp2.yahoo.net/1/testReport/ Findbugs warnings: http://hudson.zones.apache.org/hudson/job/Mapreduce-Patch-h3.grid.sp2.yahoo.net/1/artifact/trunk/build/test/findbugs/newPatchFindbugsWarnings.html Checkstyle results: http://hudson.zones.apache.org/hudson/job/Mapreduce-Patch-h3.grid.sp2.yahoo.net/1/artifact/trunk/build/test/checkstyle-errors.html Console output: http://hudson.zones.apache.org/hudson/job/Mapreduce-Patch-h3.grid.sp2.yahoo.net/1/console This message is automatically generated. > rumen should use its own interpolation classes to create runtimes for > simulated tasks > - > > Key: MAPREDUCE-931 > URL: https://issues.apache.org/jira/browse/MAPREDUCE-931 > Project: Hadoop Map/Reduce > Issue Type: Improvement >Reporter: Dick King >Assignee: Dick King >Priority: Minor > Attachments: patch-931-b.patch > > > Currently, when a simulator or benchmark is running and simulating hadoop > jobs using rumen data, and rumen's runtime system is used to get execution > times for the tasks in the simulated jobs, rumen would use some ad hoc code, > despite the fact that rumen has a perfectly good interpolation framework to > generate random variables that fit discrete CDFs. > We should use the interpolation framework. -- This message is automatically generated by JIRA. - You can reply to this email to add a comment to the issue online.
[jira] Commented: (MAPREDUCE-930) rumen should interpret job history log input paths with respect to default FS, not local FS
[ https://issues.apache.org/jira/browse/MAPREDUCE-930?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12749833#action_12749833 ] Hadoop QA commented on MAPREDUCE-930: - -1 overall. Here are the results of testing the latest attachment http://issues.apache.org/jira/secure/attachment/12418196/patch-930.patch against trunk revision 809843. +1 @author. The patch does not contain any @author tags. -1 tests included. The patch doesn't appear to include any new or modified tests. Please justify why no new tests are needed for this patch. Also please list what manual steps were performed to verify this patch. +1 javadoc. The javadoc tool did not generate any warning messages. +1 javac. The applied patch does not increase the total number of javac compiler warnings. -1 findbugs. The patch appears to introduce new Findbugs warnings. +1 release audit. The applied patch does not increase the total number of release audit warnings. -1 core tests. The patch failed core unit tests. +1 contrib tests. The patch passed contrib unit tests. Test results: http://hudson.zones.apache.org/hudson/job/Mapreduce-Patch-h6.grid.sp2.yahoo.net/32/testReport/ Findbugs warnings: http://hudson.zones.apache.org/hudson/job/Mapreduce-Patch-h6.grid.sp2.yahoo.net/32/artifact/trunk/build/test/findbugs/newPatchFindbugsWarnings.html Checkstyle results: http://hudson.zones.apache.org/hudson/job/Mapreduce-Patch-h6.grid.sp2.yahoo.net/32/artifact/trunk/build/test/checkstyle-errors.html Console output: http://hudson.zones.apache.org/hudson/job/Mapreduce-Patch-h6.grid.sp2.yahoo.net/32/console This message is automatically generated. > rumen should interpret job history log input paths with respect to default > FS, not local FS > --- > > Key: MAPREDUCE-930 > URL: https://issues.apache.org/jira/browse/MAPREDUCE-930 > Project: Hadoop Map/Reduce > Issue Type: Improvement >Reporter: Dick King >Assignee: Dick King >Priority: Minor > Attachments: patch-930.patch > > > This allows job history log file/directory names that don't specify a file > system to use the configured default FS instead of the local FS [when the > configured default is not the local]. -- This message is automatically generated by JIRA. - You can reply to this email to add a comment to the issue online.
[jira] Updated: (MAPREDUCE-873) Simplify Job Recovery
[ https://issues.apache.org/jira/browse/MAPREDUCE-873?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Sharad Agarwal updated MAPREDUCE-873: - Resolution: Fixed Release Note: Simplifies job recovery. On jobtracker restart, incomplete jobs are resubmitted and all tasks reexecute. This JIRA removes a public constructor in JobInProgress. Hadoop Flags: [Incompatible change, Reviewed] (was: [Incompatible change]) Status: Resolved (was: Patch Available) I just committed this. > Simplify Job Recovery > - > > Key: MAPREDUCE-873 > URL: https://issues.apache.org/jira/browse/MAPREDUCE-873 > Project: Hadoop Map/Reduce > Issue Type: Improvement > Components: jobtracker >Affects Versions: 0.20.1 >Reporter: Devaraj Das >Assignee: Sharad Agarwal > Fix For: 0.21.0 > > Attachments: 873_v1.patch, 873_v2.patch, 873_v3.patch > > > On a couple of occasions we have seen the JobTracker not being able to handle > job recovery well, and leading to cluster downtime after a restart. The > current design for handling job recovery is complex and prone to corner cases > not being handled well enough. In retrospect, it seems like the transaction > log based approach as was proposed on HADOOP-3245 > (http://tinyurl.com/luh9hb), would have been a better/simpler model. However, > that is a big project, and it seems for the medium term, just handling job > re-submissions after a restart is a good tradeoff. That is, the JobTracker > after getting restarted, will resubmit all jobs that were running in its past > life. They will all start from the beginning (downside is completed tasks > will reexecute). In the long term, the transaction log model or some variant > of that should be pursued. > Thoughts/comments welcome. -- This message is automatically generated by JIRA. - You can reply to this email to add a comment to the issue online.
[jira] Commented: (MAPREDUCE-873) Simplify Job Recovery
[ https://issues.apache.org/jira/browse/MAPREDUCE-873?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12749805#action_12749805 ] Devaraj Das commented on MAPREDUCE-873: --- +1 > Simplify Job Recovery > - > > Key: MAPREDUCE-873 > URL: https://issues.apache.org/jira/browse/MAPREDUCE-873 > Project: Hadoop Map/Reduce > Issue Type: Improvement > Components: jobtracker >Affects Versions: 0.20.1 >Reporter: Devaraj Das >Assignee: Sharad Agarwal > Fix For: 0.21.0 > > Attachments: 873_v1.patch, 873_v2.patch, 873_v3.patch > > > On a couple of occasions we have seen the JobTracker not being able to handle > job recovery well, and leading to cluster downtime after a restart. The > current design for handling job recovery is complex and prone to corner cases > not being handled well enough. In retrospect, it seems like the transaction > log based approach as was proposed on HADOOP-3245 > (http://tinyurl.com/luh9hb), would have been a better/simpler model. However, > that is a big project, and it seems for the medium term, just handling job > re-submissions after a restart is a good tradeoff. That is, the JobTracker > after getting restarted, will resubmit all jobs that were running in its past > life. They will all start from the beginning (downside is completed tasks > will reexecute). In the long term, the transaction log model or some variant > of that should be pursued. > Thoughts/comments welcome. -- This message is automatically generated by JIRA. - You can reply to this email to add a comment to the issue online.
[jira] Commented: (MAPREDUCE-687) TestMiniMRMapRedDebugScript fails sometimes
[ https://issues.apache.org/jira/browse/MAPREDUCE-687?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12749801#action_12749801 ] Hudson commented on MAPREDUCE-687: -- Integrated in Hadoop-Mapreduce-trunk-Commit #8 (See [http://hudson.zones.apache.org/hudson/job/Hadoop-Mapreduce-trunk-Commit/8/]) . Fix an assertion in TestMiniMRMapRedDebugScript. Contributed by Amareshwari Sriramadasu. > TestMiniMRMapRedDebugScript fails sometimes > --- > > Key: MAPREDUCE-687 > URL: https://issues.apache.org/jira/browse/MAPREDUCE-687 > Project: Hadoop Map/Reduce > Issue Type: Bug > Components: test >Reporter: Amar Kamat >Assignee: Amareshwari Sriramadasu > Fix For: 0.20.1 > > Attachments: patch-687.txt > > > Testcase: testMapDebugScript took 149.994 sec > FAILED > null expected:<...t Script > Bailing out[]> but was:<...t Script > Bailing out[ > log4j:WARN No appenders could be found for logger > (org.apache.hadoop.mapred.Task). > log4j:WARN Please initialize the log4j system properly.]> > junit.framework.ComparisonFailure: null expected:<...t Script > Bailing out[]> but was:<...t Script > Bailing out[ > log4j:WARN No appenders could be found for logger > (org.apache.hadoop.mapred.Task). > log4j:WARN Please initialize the log4j system properly.]> > at > org.apache.hadoop.mapred.TestMiniMRMapRedDebugScript.testMapDebugScript(TestMiniMRMapRedDebugScript.java:212) -- This message is automatically generated by JIRA. - You can reply to this email to add a comment to the issue online.
