[jira] Updated: (MAPREDUCE-1972) TestUserLogCleanup test cant clean up the toBeDeleted
[ https://issues.apache.org/jira/browse/MAPREDUCE-1972?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Amareshwari Sriramadasu updated MAPREDUCE-1972: --- Attachment: TestCompressedEmptyMapOutputs.log Test log which created the non-deletable user-log directory > TestUserLogCleanup test cant clean up the toBeDeleted > - > > Key: MAPREDUCE-1972 > URL: https://issues.apache.org/jira/browse/MAPREDUCE-1972 > Project: Hadoop Map/Reduce > Issue Type: Bug > Environment: > http://hudson.zones.apache.org/hudson/view/Mapreduce/job/Mapreduce-Patch-h4.grid.sp2.yahoo.net/300 >Reporter: Giridharan Kesavan > Attachments: TestCompressedEmptyMapOutputs.log, TestUserLogCleanup.log > > > All the hudson patch test builds are failing as the > Mapreduce-Patch-h4.grid.sp2.yahoo.net/trunk/build/test/logs/userlogs has a > folder created by the following test which doesnt seem to have read > permission. > Running org.apache.hadoop.mapred.TestUserLogCleanup > [exec] [junit] 2010-07-14 22:24:54,027 INFO mapred.UserLogCleaner > (UserLogCleaner.java:markJobLogsForDeletion(174)) - Adding job_test_0001 for > user-log deletion with retainTimeStamp:720 > ... > [exec] [junit] 2010-07-14 22:24:54,373 WARN util.MRAsyncDiskService > (MRAsyncDiskService.java:run(214)) - Failure in deletion of > toBeDeleted/2010-07-14_22-24-54.372_6 on > /grid/0/hudson/hudson-slave/workspace/Mapreduce-Patch-h4.grid.sp2.yahoo.net/trunk/build/test/logs/userlogs > with original name job_20100714203911410_0002 > [exec] [junit] 2010-07-14 22:24:54,374 WARN util.MRAsyncDiskService > (MRAsyncDiskService.java:run(214)) - Failure in deletion of > toBeDeleted/2010-07-14_22-24-54.373_7 on > /grid/0/hudson/hudson-slave/workspace/Mapreduce-Patch-h4.grid.sp2.yahoo.net/trunk/build/test/logs/userlogs > with original name job_test_0003 > [exec] [junit] 2010-07-14 22:24:54,391 WARN util.MRAsyncDiskService > (MRAsyncDiskService.java:run(214)) - Failure in deletion of > toBeDeleted/2010-07-14_22-24-54.372_6 on > /grid/0/hudson/hudson-slave/workspace/Mapreduce-Patch-h4.grid.sp2.yahoo.net/trunk/build/test/logs/userlogs > with original name > /grid/0/hudson/hudson-slave/workspace/Mapreduce-Patch-h4.grid.sp2.yahoo.net/trunk/build/test/logs/userlogs/toBeDeleted/2010-07-14_22-24-54.372_6 > [exec] [junit] 2010-07-14 22:24:54,405 INFO mapred.UserLogCleaner > (UserLogCleaner.java:markJobLogsForDeletion(174)) - Adding job_test_0001 for > user-log deletion with retainTimeStamp:720 > .. > [exec] [junit] 2010-07-14 22:24:54,441 WARN util.MRAsyncDiskService > (MRAsyncDiskService.java:run(214)) - Failure in deletion of > toBeDeleted/2010-07-14_22-2 -- This message is automatically generated by JIRA. - You can reply to this email to add a comment to the issue online.
[jira] Commented: (MAPREDUCE-1959) Should use long name for token renewer on the client side
[ https://issues.apache.org/jira/browse/MAPREDUCE-1959?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12893936#action_12893936 ] Hadoop QA commented on MAPREDUCE-1959: -- -1 overall. Here are the results of testing the latest attachment http://issues.apache.org/jira/secure/attachment/12450193/m1959-01.patch against trunk revision 980316. +1 @author. The patch does not contain any @author tags. -1 tests included. The patch doesn't appear to include any new or modified tests. Please justify why no new tests are needed for this patch. Also please list what manual steps were performed to verify this patch. -1 patch. The patch command could not apply the patch. Console output: http://hudson.zones.apache.org/hudson/job/Mapreduce-Patch-h4.grid.sp2.yahoo.net/339/console This message is automatically generated. > Should use long name for token renewer on the client side > - > > Key: MAPREDUCE-1959 > URL: https://issues.apache.org/jira/browse/MAPREDUCE-1959 > Project: Hadoop Map/Reduce > Issue Type: Bug > Components: job submission, security >Reporter: Kan Zhang >Assignee: Kan Zhang > Attachments: m1959-01.patch > > > When getting a delegation token from a NN, a client needs to specify the > renewer for the token. For use on a MapRed cluster, JT should be specified as > the renewer. However, in the current code, the client maps JT's long name > (Kerberos principal name) to cluster-internal short name and then sets the > short name as the renewer. This is undesirable for 2 reasons. 1) It's > unnecessary since NN (or JT) converts client-supplied renewer from long to > short name anyway. 2) In principle, the mapping from long to short name > should be done on the server. This is consistent with the authentication > case, where the client uses the same long name to authenticate to multiple > servers and servers map client's long name to their own internal short names. > It facilitates using the same job client to get delegation tokens from > multiple NN's, which may have different mapping rules for JT. -- This message is automatically generated by JIRA. - You can reply to this email to add a comment to the issue online.
[jira] Updated: (MAPREDUCE-1972) TestUserLogCleanup test cant clean up the toBeDeleted
[ https://issues.apache.org/jira/browse/MAPREDUCE-1972?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Amareshwari Sriramadasu updated MAPREDUCE-1972: --- Attachment: TestUserLogCleanup.log Attaching complete log for TestUserLogCleanup from the console output. > TestUserLogCleanup test cant clean up the toBeDeleted > - > > Key: MAPREDUCE-1972 > URL: https://issues.apache.org/jira/browse/MAPREDUCE-1972 > Project: Hadoop Map/Reduce > Issue Type: Bug > Environment: > http://hudson.zones.apache.org/hudson/view/Mapreduce/job/Mapreduce-Patch-h4.grid.sp2.yahoo.net/300 >Reporter: Giridharan Kesavan > Attachments: TestUserLogCleanup.log > > > All the hudson patch test builds are failing as the > Mapreduce-Patch-h4.grid.sp2.yahoo.net/trunk/build/test/logs/userlogs has a > folder created by the following test which doesnt seem to have read > permission. > Running org.apache.hadoop.mapred.TestUserLogCleanup > [exec] [junit] 2010-07-14 22:24:54,027 INFO mapred.UserLogCleaner > (UserLogCleaner.java:markJobLogsForDeletion(174)) - Adding job_test_0001 for > user-log deletion with retainTimeStamp:720 > ... > [exec] [junit] 2010-07-14 22:24:54,373 WARN util.MRAsyncDiskService > (MRAsyncDiskService.java:run(214)) - Failure in deletion of > toBeDeleted/2010-07-14_22-24-54.372_6 on > /grid/0/hudson/hudson-slave/workspace/Mapreduce-Patch-h4.grid.sp2.yahoo.net/trunk/build/test/logs/userlogs > with original name job_20100714203911410_0002 > [exec] [junit] 2010-07-14 22:24:54,374 WARN util.MRAsyncDiskService > (MRAsyncDiskService.java:run(214)) - Failure in deletion of > toBeDeleted/2010-07-14_22-24-54.373_7 on > /grid/0/hudson/hudson-slave/workspace/Mapreduce-Patch-h4.grid.sp2.yahoo.net/trunk/build/test/logs/userlogs > with original name job_test_0003 > [exec] [junit] 2010-07-14 22:24:54,391 WARN util.MRAsyncDiskService > (MRAsyncDiskService.java:run(214)) - Failure in deletion of > toBeDeleted/2010-07-14_22-24-54.372_6 on > /grid/0/hudson/hudson-slave/workspace/Mapreduce-Patch-h4.grid.sp2.yahoo.net/trunk/build/test/logs/userlogs > with original name > /grid/0/hudson/hudson-slave/workspace/Mapreduce-Patch-h4.grid.sp2.yahoo.net/trunk/build/test/logs/userlogs/toBeDeleted/2010-07-14_22-24-54.372_6 > [exec] [junit] 2010-07-14 22:24:54,405 INFO mapred.UserLogCleaner > (UserLogCleaner.java:markJobLogsForDeletion(174)) - Adding job_test_0001 for > user-log deletion with retainTimeStamp:720 > .. > [exec] [junit] 2010-07-14 22:24:54,441 WARN util.MRAsyncDiskService > (MRAsyncDiskService.java:run(214)) - Failure in deletion of > toBeDeleted/2010-07-14_22-2 -- This message is automatically generated by JIRA. - You can reply to this email to add a comment to the issue online.
[jira] Commented: (MAPREDUCE-1986) History files fail to move to DONE folder when hadoop.job.history.location is configured to a HDFS path
[
https://issues.apache.org/jira/browse/MAPREDUCE-1986?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12893929#action_12893929
]
Ramya R commented on MAPREDUCE-1986:
I have the following setup on my cluster:
hadoop.job.history.location
/mapred/history1
If job tracker is static the history files are stored in this
single well known place. If No
value is set here, by default,
it is in the local file system at ${hadoop.log.dir}/history.
mapred.job.tracker.history.completed.location
/mapred/history2/done
The completed job history files are stored at this single well
known location. If nothing is
specified, the files are stored at
${hadoop.job.history.location}/done.
i.e. both hadoop.job.history.location and
mapred.job.tracker.history.completed.location are configured to HDFS path.
The jobtracker successfully comes up without any error. However when the job
completes and it retires, I get the following exception:
{noformat}
ERROR org.apache.hadoop.mapred.JobHistory: Unable to move history file to DONE
canonical subfolder.
java.io.FileNotFoundException: File
/mapred/history1/job_201007290838_0001_1280392750589_rramya_Sleep+job does not
exist.
at
org.apache.hadoop.fs.RawLocalFileSystem.getFileStatus(RawLocalFileSystem.java:361)
at
org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:245)
at org.apache.hadoop.fs.FileUtil.copy(FileUtil.java:207)
at
org.apache.hadoop.fs.FileSystem.copyFromLocalFile(FileSystem.java:1118)
at
org.apache.hadoop.fs.FileSystem.copyFromLocalFile(FileSystem.java:1094)
at
org.apache.hadoop.fs.FileSystem.moveFromLocalFile(FileSystem.java:1084)
at
org.apache.hadoop.mapred.JobHistory$JobHistoryFilesManager$1.run(JobHistory.java:300)
at
java.util.concurrent.ThreadPoolExecutor$Worker.runTask(ThreadPoolExecutor.java:886)
at
java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:908)
at java.lang.Thread.run(Thread.java:619)
{noformat}
However, /mapred/history1 is created and the history files are very much
present on HDFS.
Thus the history files remain forever in hadoop.job.history.location and are
never moved to mapred.job.tracker.history.completed.location. The jobhistory is
never viewable thereafter.
> History files fail to move to DONE folder when hadoop.job.history.location is
> configured to a HDFS path
> ---
>
> Key: MAPREDUCE-1986
> URL: https://issues.apache.org/jira/browse/MAPREDUCE-1986
> Project: Hadoop Map/Reduce
> Issue Type: Bug
> Components: jobtracker
>Affects Versions: 0.20.2
>Reporter: Ramya R
>
> When hadoop.job.history.location is configured to a HDFS path and when the
> job retires, the history files are never moved to
> mapred.job.tracker.history.completed.location and they remain forever in
> hadoop.job.history.location . The jobhistory is never viewable thereafter.
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
[jira] Created: (MAPREDUCE-1986) History files fail to move to DONE folder when hadoop.job.history.location is configured to a HDFS path
History files fail to move to DONE folder when hadoop.job.history.location is configured to a HDFS path --- Key: MAPREDUCE-1986 URL: https://issues.apache.org/jira/browse/MAPREDUCE-1986 Project: Hadoop Map/Reduce Issue Type: Bug Components: jobtracker Affects Versions: 0.20.2 Reporter: Ramya R When hadoop.job.history.location is configured to a HDFS path and when the job retires, the history files are never moved to mapred.job.tracker.history.completed.location and they remain forever in hadoop.job.history.location . The jobhistory is never viewable thereafter. -- This message is automatically generated by JIRA. - You can reply to this email to add a comment to the issue online.
[jira] Commented: (MAPREDUCE-1933) Create automated testcase for tasktracker dealing with corrupted disk.
[
https://issues.apache.org/jira/browse/MAPREDUCE-1933?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12893905#action_12893905
]
Konstantin Boudnik commented on MAPREDUCE-1933:
---
- I still see that string "mapred.local.dir" is used all over the place in the
trunk patch. First, don't use string literals where there's a named constant
for the same; Second: this is a deprecated parameter: javac will curse and
raise the level of warnings.
- Shall these two be swapped, actually?
{noformat}
+cluster.tearDown();
+cluster.restart();
{noformat}
Perhaps, cleaning the folders should also be done before the {{tearDown()}}
call?
- Is there defined constants for something like
"test.randomwrite.bytes_per_map" ? If not - define a private member of the test
class instead of using string literals.
> Create automated testcase for tasktracker dealing with corrupted disk.
> --
>
> Key: MAPREDUCE-1933
> URL: https://issues.apache.org/jira/browse/MAPREDUCE-1933
> Project: Hadoop Map/Reduce
> Issue Type: Test
> Components: test
>Reporter: Iyappan Srinivasan
>Assignee: Iyappan Srinivasan
> Attachments: 1933-ydist-security-patch.txt,
> 1933-ydist-security-patch.txt, MAPREDUCE-1933.patch, MAPREDUCE-1933.patch,
> MAPREDUCE-1933.patch, TestCorruptedDiskJob.java
>
>
> After the TaskTracker has already run some tasks successfully, "corrupt" a
> disk by making the corresponding mapred.local.dir unreadable/unwritable.
> Make sure that jobs continue to succeed even though some tasks scheduled
> there fail.
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
[jira] Updated: (MAPREDUCE-1985) java.lang.ArrayIndexOutOfBoundsException in analysejobhistory.jsp of jobs with 0 maps
[ https://issues.apache.org/jira/browse/MAPREDUCE-1985?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Todd Lipcon updated MAPREDUCE-1985: --- Attachment: mapreduce-1985.txt Patch from YDH > java.lang.ArrayIndexOutOfBoundsException in analysejobhistory.jsp of jobs > with 0 maps > - > > Key: MAPREDUCE-1985 > URL: https://issues.apache.org/jira/browse/MAPREDUCE-1985 > Project: Hadoop Map/Reduce > Issue Type: Bug > Components: jobtracker >Affects Versions: 0.20.2 >Reporter: Todd Lipcon > Attachments: mapreduce-1985.txt > > > This is Yahoo bug #3460762 included in ydist, but couldn't find a public JIRA > for it. Uploading patch from YDH in case anyone else has run into this on the > 0.20 branch. -- This message is automatically generated by JIRA. - You can reply to this email to add a comment to the issue online.
[jira] Resolved: (MAPREDUCE-1985) java.lang.ArrayIndexOutOfBoundsException in analysejobhistory.jsp of jobs with 0 maps
[ https://issues.apache.org/jira/browse/MAPREDUCE-1985?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Todd Lipcon resolved MAPREDUCE-1985. Resolution: Won't Fix Resolving won't fix, since this is already fixed in trunk as a side effect of a later patch (MAPREDUCE-157 I think) > java.lang.ArrayIndexOutOfBoundsException in analysejobhistory.jsp of jobs > with 0 maps > - > > Key: MAPREDUCE-1985 > URL: https://issues.apache.org/jira/browse/MAPREDUCE-1985 > Project: Hadoop Map/Reduce > Issue Type: Bug > Components: jobtracker >Affects Versions: 0.20.2 >Reporter: Todd Lipcon > Attachments: mapreduce-1985.txt > > > This is Yahoo bug #3460762 included in ydist, but couldn't find a public JIRA > for it. Uploading patch from YDH in case anyone else has run into this on the > 0.20 branch. -- This message is automatically generated by JIRA. - You can reply to this email to add a comment to the issue online.
[jira] Created: (MAPREDUCE-1985) java.lang.ArrayIndexOutOfBoundsException in analysejobhistory.jsp of jobs with 0 maps
java.lang.ArrayIndexOutOfBoundsException in analysejobhistory.jsp of jobs with 0 maps - Key: MAPREDUCE-1985 URL: https://issues.apache.org/jira/browse/MAPREDUCE-1985 Project: Hadoop Map/Reduce Issue Type: Bug Components: jobtracker Affects Versions: 0.20.2 Reporter: Todd Lipcon Attachments: mapreduce-1985.txt This is Yahoo bug #3460762 included in ydist, but couldn't find a public JIRA for it. Uploading patch from YDH in case anyone else has run into this on the 0.20 branch. -- This message is automatically generated by JIRA. - You can reply to this email to add a comment to the issue online.
[jira] Commented: (MAPREDUCE-1790) Automatic resolution of Lzo codecs is needed.
[
https://issues.apache.org/jira/browse/MAPREDUCE-1790?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12893901#action_12893901
]
Konstantin Boudnik commented on MAPREDUCE-1790:
---
I don't see from where {{lib.file.path}} is coming
I have applied this patch to 0.20S and all my tests are failing with
{{ClassNotFoundException}}
> Automatic resolution of Lzo codecs is needed.
> -
>
> Key: MAPREDUCE-1790
> URL: https://issues.apache.org/jira/browse/MAPREDUCE-1790
> Project: Hadoop Map/Reduce
> Issue Type: Improvement
> Components: build
> Environment: Herriot system test case automation
>Reporter: Balaji Rajagopalan
>Assignee: Giridharan Kesavan
> Attachments: ivy_lzcodec.patch, ivy_lzcodec_1.patch, lzcodec_fix.txt
>
>
> The test cases are failing due to non-availablity of the jar
> hadoop-gpl-compression-0.1.0-1005060043.jar, need changes to aop xml to fix
> this.
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
[jira] Commented: (MAPREDUCE-1882) Use Jsch instead of Shell.java
[ https://issues.apache.org/jira/browse/MAPREDUCE-1882?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12893898#action_12893898 ] Konstantin Boudnik commented on MAPREDUCE-1882: --- Patch looks good... However, this is common functionality and I assume HDFS test might need it as well? Shall it be moved to Common instead? > Use Jsch instead of Shell.java > --- > > Key: MAPREDUCE-1882 > URL: https://issues.apache.org/jira/browse/MAPREDUCE-1882 > Project: Hadoop Map/Reduce > Issue Type: New Feature > Components: test > Environment: herriot framework >Reporter: Balaji Rajagopalan >Assignee: Iyappan Srinivasan > Attachments: 1882-ydist-security-patch.txt, MAPREDUCE-1882.patch, > RemoteExecution.patch > > Original Estimate: 24h > Remaining Estimate: 24h > > In herriot ( hadoop system test case dev) we often find that we are resorted > to habit of ssh to remote node execute a shell command, and come out. It is > wise to use Jsch instead of doing this through Shell.java ( hadoop code), > since Jsch provides nice Java abstraction, the JIRA will only close after we > import Jsch input hadoop build system and also fix all the existing test > cases. -- This message is automatically generated by JIRA. - You can reply to this email to add a comment to the issue online.
[jira] Updated: (MAPREDUCE-1900) MapReduce daemons should close FileSystems that are not needed anymore
[ https://issues.apache.org/jira/browse/MAPREDUCE-1900?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Kan Zhang updated MAPREDUCE-1900: - Status: Patch Available (was: Open) > MapReduce daemons should close FileSystems that are not needed anymore > -- > > Key: MAPREDUCE-1900 > URL: https://issues.apache.org/jira/browse/MAPREDUCE-1900 > Project: Hadoop Map/Reduce > Issue Type: Bug > Components: jobtracker, tasktracker >Reporter: Devaraj Das > Fix For: 0.22.0 > > Attachments: fs-close-delta.patch, m1900-04.patch, > mapred-fs-close.patch > > > Related to HADOOP-6843, this jira is to make MapReduce behave better with > respect to closing FileSystems when they are not needed anymore. -- This message is automatically generated by JIRA. - You can reply to this email to add a comment to the issue online.
[jira] Updated: (MAPREDUCE-1900) MapReduce daemons should close FileSystems that are not needed anymore
[ https://issues.apache.org/jira/browse/MAPREDUCE-1900?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Kan Zhang updated MAPREDUCE-1900: - Attachment: m1900-04.patch a port for trunk. > MapReduce daemons should close FileSystems that are not needed anymore > -- > > Key: MAPREDUCE-1900 > URL: https://issues.apache.org/jira/browse/MAPREDUCE-1900 > Project: Hadoop Map/Reduce > Issue Type: Bug > Components: jobtracker, tasktracker >Reporter: Devaraj Das > Fix For: 0.22.0 > > Attachments: fs-close-delta.patch, m1900-04.patch, > mapred-fs-close.patch > > > Related to HADOOP-6843, this jira is to make MapReduce behave better with > respect to closing FileSystems when they are not needed anymore. -- This message is automatically generated by JIRA. - You can reply to this email to add a comment to the issue online.
[jira] Commented: (MAPREDUCE-1958) using delegation token over hftp for long running clients (part of hdfs 1296)
[ https://issues.apache.org/jira/browse/MAPREDUCE-1958?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12893842#action_12893842 ] Jitendra Nath Pandey commented on MAPREDUCE-1958: - +1 for the patch. > using delegation token over hftp for long running clients (part of hdfs 1296) > - > > Key: MAPREDUCE-1958 > URL: https://issues.apache.org/jira/browse/MAPREDUCE-1958 > Project: Hadoop Map/Reduce > Issue Type: Bug >Reporter: Boris Shkolnik >Assignee: Boris Shkolnik > Attachments: MAPREDUCE-1958-1.patch > > -- This message is automatically generated by JIRA. - You can reply to this email to add a comment to the issue online.
[jira] Updated: (MAPREDUCE-1981) Improve getSplits performance by using listFiles, the new FileSystem API
[
https://issues.apache.org/jira/browse/MAPREDUCE-1981?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Hairong Kuang updated MAPREDUCE-1981:
-
Attachment: mapredListFiles.patch
This patch makes FileInputFormat & CombineFileInputFormat to use the new
listFiles() API introduced by HADOOP-6870.
Ideally FileInputFormat#listStatus should have the following syntax:
{code}
Iterator listStatus(JobConf job) throws IOException;
{code}
But since this is a public interface, I keep it as it is now to keep it
backward compatible.
This patch also changes the semantics of listStatus a little bit. When
recursive is false, listStatus used to return every child (including
subdirectories) of the input directories. But with the new API, it returns only
the file children. So it is not able to support this case: throwing an
exception when an input directory has a subdirectory but recursive is false. I
removed this test case from TestFileInputFormat. If we really want to support
this scenario, I could make FileContext#listFiles to throw an exception when
recursive is false but there is sbudirectory.
> Improve getSplits performance by using listFiles, the new FileSystem API
>
>
> Key: MAPREDUCE-1981
> URL: https://issues.apache.org/jira/browse/MAPREDUCE-1981
> Project: Hadoop Map/Reduce
> Issue Type: Improvement
> Components: job submission
>Reporter: Hairong Kuang
>Assignee: Hairong Kuang
> Fix For: 0.22.0
>
> Attachments: mapredListFiles.patch
>
>
> This jira will make FileInputFormat and CombinedFileInputForm to use the new
> API, thus reducing the number of RPCs to HDFS NameNode.
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
[jira] Updated: (MAPREDUCE-1920) Job.getCounters() returns null when using a cluster
[ https://issues.apache.org/jira/browse/MAPREDUCE-1920?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Tom White updated MAPREDUCE-1920: - Status: Patch Available (was: Open) Re-running through Hudson. > Job.getCounters() returns null when using a cluster > --- > > Key: MAPREDUCE-1920 > URL: https://issues.apache.org/jira/browse/MAPREDUCE-1920 > Project: Hadoop Map/Reduce > Issue Type: Bug >Affects Versions: 0.21.0 >Reporter: Aaron Kimball >Assignee: Tom White >Priority: Critical > Attachments: MAPREDUCE-1920.patch > > > Calling Job.getCounters() after the job has completed (successfully) returns > null. -- This message is automatically generated by JIRA. - You can reply to this email to add a comment to the issue online.
[jira] Updated: (MAPREDUCE-1920) Job.getCounters() returns null when using a cluster
[ https://issues.apache.org/jira/browse/MAPREDUCE-1920?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Tom White updated MAPREDUCE-1920: - Status: Open (was: Patch Available) > Job.getCounters() returns null when using a cluster > --- > > Key: MAPREDUCE-1920 > URL: https://issues.apache.org/jira/browse/MAPREDUCE-1920 > Project: Hadoop Map/Reduce > Issue Type: Bug >Affects Versions: 0.21.0 >Reporter: Aaron Kimball >Assignee: Tom White >Priority: Critical > Attachments: MAPREDUCE-1920.patch > > > Calling Job.getCounters() after the job has completed (successfully) returns > null. -- This message is automatically generated by JIRA. - You can reply to this email to add a comment to the issue online.
[jira] Updated: (MAPREDUCE-1926) MapReduce distribution is missing build-utils.xml
[ https://issues.apache.org/jira/browse/MAPREDUCE-1926?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Tom White updated MAPREDUCE-1926: - Attachment: MAPREDUCE-1926.patch This patch includes build-utils.xml in the distribution. Also, since template files are not included in the distribution, it doesn't fail if certain ones are not present. > MapReduce distribution is missing build-utils.xml > - > > Key: MAPREDUCE-1926 > URL: https://issues.apache.org/jira/browse/MAPREDUCE-1926 > Project: Hadoop Map/Reduce > Issue Type: Bug > Components: build >Reporter: Tom White >Priority: Blocker > Fix For: 0.21.0 > > Attachments: MAPREDUCE-1926.patch > > > The tarball should be able to build itself. -- This message is automatically generated by JIRA. - You can reply to this email to add a comment to the issue online.
[jira] Updated: (MAPREDUCE-1926) MapReduce distribution is missing build-utils.xml
[ https://issues.apache.org/jira/browse/MAPREDUCE-1926?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Tom White updated MAPREDUCE-1926: - Status: Patch Available (was: Open) Assignee: Tom White > MapReduce distribution is missing build-utils.xml > - > > Key: MAPREDUCE-1926 > URL: https://issues.apache.org/jira/browse/MAPREDUCE-1926 > Project: Hadoop Map/Reduce > Issue Type: Bug > Components: build >Reporter: Tom White >Assignee: Tom White >Priority: Blocker > Fix For: 0.21.0 > > Attachments: MAPREDUCE-1926.patch > > > The tarball should be able to build itself. -- This message is automatically generated by JIRA. - You can reply to this email to add a comment to the issue online.
[jira] Updated: (MAPREDUCE-1686) ClassNotFoundException for custom format classes provided in libjars
[
https://issues.apache.org/jira/browse/MAPREDUCE-1686?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Paul Burkhardt updated MAPREDUCE-1686:
--
Attachment: HADOOP-1686-2.patch
New patch against the trunk. The previous patches were against the tag
release-0.20.1 since it was our intent to get it in the 0.20.x baseline. Also
removed the extra assert as suggested.
> ClassNotFoundException for custom format classes provided in libjars
>
>
> Key: MAPREDUCE-1686
> URL: https://issues.apache.org/jira/browse/MAPREDUCE-1686
> Project: Hadoop Map/Reduce
> Issue Type: Bug
> Components: contrib/streaming
>Affects Versions: 0.20.1
>Reporter: Paul Burkhardt
>Priority: Minor
> Attachments: HADOOP-1686-1.patch, HADOOP-1686-2.patch,
> HADOOP-1686.patch
>
>
> The StreamUtil::goodClassOrNull method assumes user-provided classes have
> package names and if not, they are part of the Hadoop Streaming package. For
> example, using custom InputFormat or OutputFormat classes without package
> names will fail with a ClassNotFound exception which is not indicative given
> the classes are provided in the libjars option. Admittedly, most Java
> packages should have a package name so this should rarely come up.
> Possible resolution options:
> 1) modify the error message to include the actual classname that was
> attempted in the goodClassOrNull method
> 2) call the Configuration::getClassByName method first and if class not found
> check for default package name and try the call again
> {code}
> public static Class goodClassOrNull(Configuration conf, String className,
> String defaultPackage) {
> Class clazz = null;
> try {
> clazz = conf.getClassByName(className);
> } catch (ClassNotFoundException cnf) {
> }
> if (clazz == null) {
> if (className.indexOf('.') == -1 && defaultPackage != null) {
> className = defaultPackage + "." + className;
> try {
> clazz = conf.getClassByName(className);
> } catch (ClassNotFoundException cnf) {
> }
> }
> }
> return clazz;
> }
> {code}
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
[jira] Updated: (MAPREDUCE-1686) ClassNotFoundException for custom format classes provided in libjars
[
https://issues.apache.org/jira/browse/MAPREDUCE-1686?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Paul Burkhardt updated MAPREDUCE-1686:
--
Status: Patch Available (was: Open)
New patch against the trunk.
> ClassNotFoundException for custom format classes provided in libjars
>
>
> Key: MAPREDUCE-1686
> URL: https://issues.apache.org/jira/browse/MAPREDUCE-1686
> Project: Hadoop Map/Reduce
> Issue Type: Bug
> Components: contrib/streaming
>Affects Versions: 0.20.1
>Reporter: Paul Burkhardt
>Priority: Minor
> Attachments: HADOOP-1686-1.patch, HADOOP-1686.patch
>
>
> The StreamUtil::goodClassOrNull method assumes user-provided classes have
> package names and if not, they are part of the Hadoop Streaming package. For
> example, using custom InputFormat or OutputFormat classes without package
> names will fail with a ClassNotFound exception which is not indicative given
> the classes are provided in the libjars option. Admittedly, most Java
> packages should have a package name so this should rarely come up.
> Possible resolution options:
> 1) modify the error message to include the actual classname that was
> attempted in the goodClassOrNull method
> 2) call the Configuration::getClassByName method first and if class not found
> check for default package name and try the call again
> {code}
> public static Class goodClassOrNull(Configuration conf, String className,
> String defaultPackage) {
> Class clazz = null;
> try {
> clazz = conf.getClassByName(className);
> } catch (ClassNotFoundException cnf) {
> }
> if (clazz == null) {
> if (className.indexOf('.') == -1 && defaultPackage != null) {
> className = defaultPackage + "." + className;
> try {
> clazz = conf.getClassByName(className);
> } catch (ClassNotFoundException cnf) {
> }
> }
> }
> return clazz;
> }
> {code}
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
[jira] Updated: (MAPREDUCE-1959) Should use long name for token renewer on the client side
[ https://issues.apache.org/jira/browse/MAPREDUCE-1959?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Kan Zhang updated MAPREDUCE-1959: - Component/s: job submission > Should use long name for token renewer on the client side > - > > Key: MAPREDUCE-1959 > URL: https://issues.apache.org/jira/browse/MAPREDUCE-1959 > Project: Hadoop Map/Reduce > Issue Type: Bug > Components: job submission, security >Reporter: Kan Zhang >Assignee: Kan Zhang > Attachments: m1959-01.patch > > > When getting a delegation token from a NN, a client needs to specify the > renewer for the token. For use on a MapRed cluster, JT should be specified as > the renewer. However, in the current code, the client maps JT's long name > (Kerberos principal name) to cluster-internal short name and then sets the > short name as the renewer. This is undesirable for 2 reasons. 1) It's > unnecessary since NN (or JT) converts client-supplied renewer from long to > short name anyway. 2) In principle, the mapping from long to short name > should be done on the server. This is consistent with the authentication > case, where the client uses the same long name to authenticate to multiple > servers and servers map client's long name to their own internal short names. > It facilitates using the same job client to get delegation tokens from > multiple NN's, which may have different mapping rules for JT. -- This message is automatically generated by JIRA. - You can reply to this email to add a comment to the issue online.
[jira] Updated: (MAPREDUCE-1959) Should use long name for token renewer on the client side
[ https://issues.apache.org/jira/browse/MAPREDUCE-1959?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Kan Zhang updated MAPREDUCE-1959: - Status: Patch Available (was: Open) > Should use long name for token renewer on the client side > - > > Key: MAPREDUCE-1959 > URL: https://issues.apache.org/jira/browse/MAPREDUCE-1959 > Project: Hadoop Map/Reduce > Issue Type: Bug > Components: job submission, security >Reporter: Kan Zhang >Assignee: Kan Zhang > Attachments: m1959-01.patch > > > When getting a delegation token from a NN, a client needs to specify the > renewer for the token. For use on a MapRed cluster, JT should be specified as > the renewer. However, in the current code, the client maps JT's long name > (Kerberos principal name) to cluster-internal short name and then sets the > short name as the renewer. This is undesirable for 2 reasons. 1) It's > unnecessary since NN (or JT) converts client-supplied renewer from long to > short name anyway. 2) In principle, the mapping from long to short name > should be done on the server. This is consistent with the authentication > case, where the client uses the same long name to authenticate to multiple > servers and servers map client's long name to their own internal short names. > It facilitates using the same job client to get delegation tokens from > multiple NN's, which may have different mapping rules for JT. -- This message is automatically generated by JIRA. - You can reply to this email to add a comment to the issue online.
[jira] Updated: (MAPREDUCE-1959) Should use long name for token renewer on the client side
[ https://issues.apache.org/jira/browse/MAPREDUCE-1959?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Kan Zhang updated MAPREDUCE-1959: - Status: Open (was: Patch Available) > Should use long name for token renewer on the client side > - > > Key: MAPREDUCE-1959 > URL: https://issues.apache.org/jira/browse/MAPREDUCE-1959 > Project: Hadoop Map/Reduce > Issue Type: Bug > Components: security >Reporter: Kan Zhang >Assignee: Kan Zhang > Attachments: m1959-01.patch > > > When getting a delegation token from a NN, a client needs to specify the > renewer for the token. For use on a MapRed cluster, JT should be specified as > the renewer. However, in the current code, the client maps JT's long name > (Kerberos principal name) to cluster-internal short name and then sets the > short name as the renewer. This is undesirable for 2 reasons. 1) It's > unnecessary since NN (or JT) converts client-supplied renewer from long to > short name anyway. 2) In principle, the mapping from long to short name > should be done on the server. This is consistent with the authentication > case, where the client uses the same long name to authenticate to multiple > servers and servers map client's long name to their own internal short names. > It facilitates using the same job client to get delegation tokens from > multiple NN's, which may have different mapping rules for JT. -- This message is automatically generated by JIRA. - You can reply to this email to add a comment to the issue online.
[jira] Issue Comment Edited: (MAPREDUCE-1982) [Rumen] TraceBuilder's output shows jobname as NULL for jobhistory files with valid jobnames
[
https://issues.apache.org/jira/browse/MAPREDUCE-1982?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12893552#action_12893552
]
Owen O'Malley edited comment on MAPREDUCE-1982 at 7/29/10 12:15 PM:
An example fragment is:
--- 8<
---
{
"priority" : "NORMAL",
"jobID" : "job_201007280625_0001",
"user" : "gridperf",
"jobName" : null,
"mapTasks" : [ {
"startTime" : 1280298532016,
"taskID" : "task_201007280625_0001_m_000323",
"taskType" : "MAP",
"attempts" : [ {
"location" : {
"layers" : [ "98\\.137\\.102\\.64", "node1\\.yahoo\\.com" ]
},
"hostName" : "/98\\.137\\.102\\.64/node1\\.yahoo\\.com",
"startTime" : 1280298532028,
"finishTime" : 1280298597501,
"result" : "SUCCESS",
--- 8<
---
This is because Rumen directly uses Properties instead of Configuration - the
new key-name is "mapreduce.job.name", while the old key-name was
"mapred.job.name". Using Configuration insulates against such changes.
was (Author: ranjit):
An example fragment is:
--- 8<
---
{
"priority" : "NORMAL",
"jobID" : "job_201007280625_0001",
"user" : "gridperf",
"jobName" : null,
"mapTasks" : [ {
"startTime" : 1280298532016,
"taskID" : "task_201007280625_0001_m_000323",
"taskType" : "MAP",
"attempts" : [ {
"location" : {
"layers" : [ "98\\.137\\.102\\.64",
"gsbl91438\\.blue\\.ygrid\\.yahoo\\.com" ]
},
"hostName" :
"/98\\.137\\.102\\.64/gsbl91438\\.blue\\.ygrid\\.yahoo\\.com",
"startTime" : 1280298532028,
"finishTime" : 1280298597501,
"result" : "SUCCESS",
--- 8<
---
This is because Rumen directly uses Properties instead of Configuration - the
new key-name is "mapreduce.job.name", while the old key-name was
"mapred.job.name". Using Configuration insulates against such changes.
> [Rumen] TraceBuilder's output shows jobname as NULL for jobhistory files with
> valid jobnames
>
>
> Key: MAPREDUCE-1982
> URL: https://issues.apache.org/jira/browse/MAPREDUCE-1982
> Project: Hadoop Map/Reduce
> Issue Type: Bug
> Components: tools/rumen
>Reporter: Amar Kamat
>Assignee: Ravi Gummadi
> Attachments: 1982.v1.patch
>
>
> {{TraceBuilder}} fails to extract configuration properties (like job-name)
> from the job-conf if the job-conf has the properties stored using the
> deprecated keys.
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
[jira] Commented: (MAPREDUCE-1834) TestSimulatorDeterministicReplay timesout on trunk
[ https://issues.apache.org/jira/browse/MAPREDUCE-1834?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12893666#action_12893666 ] Hadoop QA commented on MAPREDUCE-1834: -- -1 overall. Here are the results of testing the latest attachment http://issues.apache.org/jira/secure/attachment/12450573/mr-1834-20100727.patch against trunk revision 980316. +1 @author. The patch does not contain any @author tags. +1 tests included. The patch appears to include 3 new or modified tests. +1 javadoc. The javadoc tool did not generate any warning messages. +1 javac. The applied patch does not increase the total number of javac compiler warnings. -1 findbugs. The patch appears to introduce 2 new Findbugs warnings. +1 release audit. The applied patch does not increase the total number of release audit warnings. +1 core tests. The patch passed core unit tests. +1 contrib tests. The patch passed contrib unit tests. Test results: http://hudson.zones.apache.org/hudson/job/Mapreduce-Patch-h4.grid.sp2.yahoo.net/338/testReport/ Findbugs warnings: http://hudson.zones.apache.org/hudson/job/Mapreduce-Patch-h4.grid.sp2.yahoo.net/338/artifact/trunk/build/test/findbugs/newPatchFindbugsWarnings.html Checkstyle results: http://hudson.zones.apache.org/hudson/job/Mapreduce-Patch-h4.grid.sp2.yahoo.net/338/artifact/trunk/build/test/checkstyle-errors.html Console output: http://hudson.zones.apache.org/hudson/job/Mapreduce-Patch-h4.grid.sp2.yahoo.net/338/console This message is automatically generated. > TestSimulatorDeterministicReplay timesout on trunk > -- > > Key: MAPREDUCE-1834 > URL: https://issues.apache.org/jira/browse/MAPREDUCE-1834 > Project: Hadoop Map/Reduce > Issue Type: Bug > Components: contrib/mumak >Affects Versions: 0.21.0 >Reporter: Amareshwari Sriramadasu >Assignee: Hong Tang > Attachments: MAPREDUCE-1834.patch, mr-1834-20100727.patch, > TestSimulatorDeterministicReplay.log > > > TestSimulatorDeterministicReplay timesout on trunk. > See hudson patch build > http://hudson.zones.apache.org/hudson/job/Mapreduce-Patch-h4.grid.sp2.yahoo.net/216/testReport/org.apache.hadoop.mapred/TestSimulatorDeterministicReplay/testMain/ -- This message is automatically generated by JIRA. - You can reply to this email to add a comment to the issue online.
[jira] Updated: (MAPREDUCE-1982) [Rumen] TraceBuilder's output shows jobname as NULL for jobhistory files with valid jobnames
[
https://issues.apache.org/jira/browse/MAPREDUCE-1982?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Ravi Gummadi updated MAPREDUCE-1982:
Attachment: 1982.v1.patch
Attaching patch fixing the issue. Added testcase that fails without the fix and
passes with the fix.
> [Rumen] TraceBuilder's output shows jobname as NULL for jobhistory files with
> valid jobnames
>
>
> Key: MAPREDUCE-1982
> URL: https://issues.apache.org/jira/browse/MAPREDUCE-1982
> Project: Hadoop Map/Reduce
> Issue Type: Bug
> Components: tools/rumen
>Reporter: Amar Kamat
>Assignee: Ravi Gummadi
> Attachments: 1982.v1.patch
>
>
> {{TraceBuilder}} fails to extract configuration properties (like job-name)
> from the job-conf if the job-conf has the properties stored using the
> deprecated keys.
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
[jira] Updated: (MAPREDUCE-1933) Create automated testcase for tasktracker dealing with corrupted disk.
[ https://issues.apache.org/jira/browse/MAPREDUCE-1933?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Iyappan Srinivasan updated MAPREDUCE-1933: -- Attachment: MAPREDUCE-1933.patch trunk patch > Create automated testcase for tasktracker dealing with corrupted disk. > -- > > Key: MAPREDUCE-1933 > URL: https://issues.apache.org/jira/browse/MAPREDUCE-1933 > Project: Hadoop Map/Reduce > Issue Type: Test > Components: test >Reporter: Iyappan Srinivasan >Assignee: Iyappan Srinivasan > Attachments: 1933-ydist-security-patch.txt, > 1933-ydist-security-patch.txt, MAPREDUCE-1933.patch, MAPREDUCE-1933.patch, > MAPREDUCE-1933.patch, TestCorruptedDiskJob.java > > > After the TaskTracker has already run some tasks successfully, "corrupt" a > disk by making the corresponding mapred.local.dir unreadable/unwritable. > Make sure that jobs continue to succeed even though some tasks scheduled > there fail. -- This message is automatically generated by JIRA. - You can reply to this email to add a comment to the issue online.
[jira] Assigned: (MAPREDUCE-1890) Create automated test scenarios for decommissioning of task trackers
[ https://issues.apache.org/jira/browse/MAPREDUCE-1890?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Iyappan Srinivasan reassigned MAPREDUCE-1890: - Assignee: Iyappan Srinivasan > Create automated test scenarios for decommissioning of task trackers > > > Key: MAPREDUCE-1890 > URL: https://issues.apache.org/jira/browse/MAPREDUCE-1890 > Project: Hadoop Map/Reduce > Issue Type: Test > Components: test >Reporter: Iyappan Srinivasan >Assignee: Iyappan Srinivasan > Attachments: 1890-ydist-security-patch.txt, > 1890-ydist-security-patch.txt, TestDecomissioning.patch > > > Test scenarios : > 1) Put a healthy slave task tracker in the dfs.exclude file. > 2) As a valid user, decommission a node in the cluster by issuing the > command "hadoop mradmin -refreshNodes" > 3) Make sure that the node is decommissioned. > 4) Now take the task tracker out of the file. > 5) As a valid user, again issue the command "hadoop mradmin -refreshNodes" > 6) Make sure that the node is not in the decommiossion list. > 7) Bring back that node. -- This message is automatically generated by JIRA. - You can reply to this email to add a comment to the issue online.
[jira] Updated: (MAPREDUCE-1366) Tests should not timeout if TaskTracker/JobTracker crashes in MiniMRCluster
[ https://issues.apache.org/jira/browse/MAPREDUCE-1366?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Amareshwari Sriramadasu updated MAPREDUCE-1366: --- Status: Open (was: Patch Available) Test failure TestMapredSystemDir should be fixed > Tests should not timeout if TaskTracker/JobTracker crashes in MiniMRCluster > --- > > Key: MAPREDUCE-1366 > URL: https://issues.apache.org/jira/browse/MAPREDUCE-1366 > Project: Hadoop Map/Reduce > Issue Type: Bug > Components: test >Reporter: Amareshwari Sriramadasu > Fix For: 0.22.0 > > Attachments: M1366-0.patch, M1366-1.patch > > > Currently tests timeout if there is any problem bringing up JobTracker or > TaskTracker in MiniMRCluster. Instead tests should fail saying JT/TT crashed. > See test timeout on MAPREDUCE-1365 -- This message is automatically generated by JIRA. - You can reply to this email to add a comment to the issue online.
[jira] Updated: (MAPREDUCE-1834) TestSimulatorDeterministicReplay timesout on trunk
[ https://issues.apache.org/jira/browse/MAPREDUCE-1834?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Amareshwari Sriramadasu updated MAPREDUCE-1834: --- Status: Open (was: Patch Available) Resubmitting the patch for hudson > TestSimulatorDeterministicReplay timesout on trunk > -- > > Key: MAPREDUCE-1834 > URL: https://issues.apache.org/jira/browse/MAPREDUCE-1834 > Project: Hadoop Map/Reduce > Issue Type: Bug > Components: contrib/mumak >Affects Versions: 0.21.0 >Reporter: Amareshwari Sriramadasu >Assignee: Hong Tang > Attachments: MAPREDUCE-1834.patch, mr-1834-20100727.patch, > TestSimulatorDeterministicReplay.log > > > TestSimulatorDeterministicReplay timesout on trunk. > See hudson patch build > http://hudson.zones.apache.org/hudson/job/Mapreduce-Patch-h4.grid.sp2.yahoo.net/216/testReport/org.apache.hadoop.mapred/TestSimulatorDeterministicReplay/testMain/ -- This message is automatically generated by JIRA. - You can reply to this email to add a comment to the issue online.
[jira] Updated: (MAPREDUCE-1834) TestSimulatorDeterministicReplay timesout on trunk
[ https://issues.apache.org/jira/browse/MAPREDUCE-1834?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Amareshwari Sriramadasu updated MAPREDUCE-1834: --- Status: Patch Available (was: Open) > TestSimulatorDeterministicReplay timesout on trunk > -- > > Key: MAPREDUCE-1834 > URL: https://issues.apache.org/jira/browse/MAPREDUCE-1834 > Project: Hadoop Map/Reduce > Issue Type: Bug > Components: contrib/mumak >Affects Versions: 0.21.0 >Reporter: Amareshwari Sriramadasu >Assignee: Hong Tang > Attachments: MAPREDUCE-1834.patch, mr-1834-20100727.patch, > TestSimulatorDeterministicReplay.log > > > TestSimulatorDeterministicReplay timesout on trunk. > See hudson patch build > http://hudson.zones.apache.org/hudson/job/Mapreduce-Patch-h4.grid.sp2.yahoo.net/216/testReport/org.apache.hadoop.mapred/TestSimulatorDeterministicReplay/testMain/ -- This message is automatically generated by JIRA. - You can reply to this email to add a comment to the issue online.
[jira] Commented: (MAPREDUCE-1366) Tests should not timeout if TaskTracker/JobTracker crashes in MiniMRCluster
[ https://issues.apache.org/jira/browse/MAPREDUCE-1366?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12893583#action_12893583 ] Hadoop QA commented on MAPREDUCE-1366: -- -1 overall. Here are the results of testing the latest attachment http://issues.apache.org/jira/secure/attachment/12450801/M1366-1.patch against trunk revision 980316. +1 @author. The patch does not contain any @author tags. +1 tests included. The patch appears to include 3 new or modified tests. +1 javadoc. The javadoc tool did not generate any warning messages. +1 javac. The applied patch does not increase the total number of javac compiler warnings. +1 findbugs. The patch does not introduce any new Findbugs warnings. +1 release audit. The applied patch does not increase the total number of release audit warnings. -1 core tests. The patch failed core unit tests. -1 contrib tests. The patch failed contrib unit tests. Test results: http://hudson.zones.apache.org/hudson/job/Mapreduce-Patch-h4.grid.sp2.yahoo.net/337/testReport/ Findbugs warnings: http://hudson.zones.apache.org/hudson/job/Mapreduce-Patch-h4.grid.sp2.yahoo.net/337/artifact/trunk/build/test/findbugs/newPatchFindbugsWarnings.html Checkstyle results: http://hudson.zones.apache.org/hudson/job/Mapreduce-Patch-h4.grid.sp2.yahoo.net/337/artifact/trunk/build/test/checkstyle-errors.html Console output: http://hudson.zones.apache.org/hudson/job/Mapreduce-Patch-h4.grid.sp2.yahoo.net/337/console This message is automatically generated. > Tests should not timeout if TaskTracker/JobTracker crashes in MiniMRCluster > --- > > Key: MAPREDUCE-1366 > URL: https://issues.apache.org/jira/browse/MAPREDUCE-1366 > Project: Hadoop Map/Reduce > Issue Type: Bug > Components: test >Reporter: Amareshwari Sriramadasu > Fix For: 0.22.0 > > Attachments: M1366-0.patch, M1366-1.patch > > > Currently tests timeout if there is any problem bringing up JobTracker or > TaskTracker in MiniMRCluster. Instead tests should fail saying JT/TT crashed. > See test timeout on MAPREDUCE-1365 -- This message is automatically generated by JIRA. - You can reply to this email to add a comment to the issue online.
[jira] Updated: (MAPREDUCE-1790) Automatic resolution of Lzo codecs is needed.
[ https://issues.apache.org/jira/browse/MAPREDUCE-1790?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Balaji Rajagopalan updated MAPREDUCE-1790: -- Attachment: ivy_lzcodec_1.patch latest patch from giri. > Automatic resolution of Lzo codecs is needed. > - > > Key: MAPREDUCE-1790 > URL: https://issues.apache.org/jira/browse/MAPREDUCE-1790 > Project: Hadoop Map/Reduce > Issue Type: Improvement > Components: build > Environment: Herriot system test case automation >Reporter: Balaji Rajagopalan >Assignee: Giridharan Kesavan > Attachments: ivy_lzcodec.patch, ivy_lzcodec_1.patch, lzcodec_fix.txt > > > The test cases are failing due to non-availablity of the jar > hadoop-gpl-compression-0.1.0-1005060043.jar, need changes to aop xml to fix > this. -- This message is automatically generated by JIRA. - You can reply to this email to add a comment to the issue online.
[jira] Commented: (MAPREDUCE-1984) herriot TestCluster fails because expclusion is not there
[ https://issues.apache.org/jira/browse/MAPREDUCE-1984?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12893576#action_12893576 ] Vinay Kumar Thota commented on MAPREDUCE-1984: -- patch looks good. +1 > herriot TestCluster fails because expclusion is not there > - > > Key: MAPREDUCE-1984 > URL: https://issues.apache.org/jira/browse/MAPREDUCE-1984 > Project: Hadoop Map/Reduce > Issue Type: Bug > Environment: herriot >Reporter: Balaji Rajagopalan > Attachments: testcluster.patch > > > restart is part of the test case which causes ioexceptions, and this needs to > be ignored. The test case should not be incorrectly failed. -- This message is automatically generated by JIRA. - You can reply to this email to add a comment to the issue online.
[jira] Updated: (MAPREDUCE-1984) herriot TestCluster fails because expclusion is not there
[ https://issues.apache.org/jira/browse/MAPREDUCE-1984?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Balaji Rajagopalan updated MAPREDUCE-1984: -- Attachment: testcluster.patch adds expclusion list. > herriot TestCluster fails because expclusion is not there > - > > Key: MAPREDUCE-1984 > URL: https://issues.apache.org/jira/browse/MAPREDUCE-1984 > Project: Hadoop Map/Reduce > Issue Type: Bug > Environment: herriot >Reporter: Balaji Rajagopalan > Attachments: testcluster.patch > > > restart is part of the test case which causes ioexceptions, and this needs to > be ignored. The test case should not be incorrectly failed. -- This message is automatically generated by JIRA. - You can reply to this email to add a comment to the issue online.
[jira] Created: (MAPREDUCE-1984) herriot TestCluster fails because expclusion is not there
herriot TestCluster fails because expclusion is not there - Key: MAPREDUCE-1984 URL: https://issues.apache.org/jira/browse/MAPREDUCE-1984 Project: Hadoop Map/Reduce Issue Type: Bug Environment: herriot Reporter: Balaji Rajagopalan restart is part of the test case which causes ioexceptions, and this needs to be ignored. The test case should not be incorrectly failed. -- This message is automatically generated by JIRA. - You can reply to this email to add a comment to the issue online.
[jira] Commented: (MAPREDUCE-1983) [Herriot] Test linux task controller with invalid information in conf.
[ https://issues.apache.org/jira/browse/MAPREDUCE-1983?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12893573#action_12893573 ] Balaji Rajagopalan commented on MAPREDUCE-1983: --- Instead of restarting the whole cluster you can restart just the job tracket using restartDaemon with role as JT, else the code looks good. > [Herriot] Test linux task controller with invalid information in conf. > -- > > Key: MAPREDUCE-1983 > URL: https://issues.apache.org/jira/browse/MAPREDUCE-1983 > Project: Hadoop Map/Reduce > Issue Type: Task > Components: test >Reporter: Vinay Kumar Thota >Assignee: Vinay Kumar Thota > Attachments: 1983-ydist-security.patch > > > This case is about scenarios where the taskcontroller file is absent, or > contains incorrect values for mapred.local.dir.Submit a job and verify > whether job fails or not. -- This message is automatically generated by JIRA. - You can reply to this email to add a comment to the issue online.
[jira] Updated: (MAPREDUCE-1983) [Herriot] Test linux task controller with invalid information in conf.
[ https://issues.apache.org/jira/browse/MAPREDUCE-1983?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Vinay Kumar Thota updated MAPREDUCE-1983: - Attachment: 1983-ydist-security.patch patch for yahoo security dist branch. > [Herriot] Test linux task controller with invalid information in conf. > -- > > Key: MAPREDUCE-1983 > URL: https://issues.apache.org/jira/browse/MAPREDUCE-1983 > Project: Hadoop Map/Reduce > Issue Type: Task > Components: test >Reporter: Vinay Kumar Thota >Assignee: Vinay Kumar Thota > Attachments: 1983-ydist-security.patch > > > This case is about scenarios where the taskcontroller file is absent, or > contains incorrect values for mapred.local.dir.Submit a job and verify > whether job fails or not. -- This message is automatically generated by JIRA. - You can reply to this email to add a comment to the issue online.
[jira] Created: (MAPREDUCE-1983) [Herriot] Test linux task controller with invalid information in conf.
[Herriot] Test linux task controller with invalid information in conf. -- Key: MAPREDUCE-1983 URL: https://issues.apache.org/jira/browse/MAPREDUCE-1983 Project: Hadoop Map/Reduce Issue Type: Task Components: test Reporter: Vinay Kumar Thota Assignee: Vinay Kumar Thota This case is about scenarios where the taskcontroller file is absent, or contains incorrect values for mapred.local.dir.Submit a job and verify whether job fails or not. -- This message is automatically generated by JIRA. - You can reply to this email to add a comment to the issue online.
[jira] Commented: (MAPREDUCE-1982) [Rumen] TraceBuilder's output shows jobname as NULL for jobhistory files with valid jobnames
[
https://issues.apache.org/jira/browse/MAPREDUCE-1982?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12893560#action_12893560
]
Amar Kamat commented on MAPREDUCE-1982:
---
Also {{TraceBuilder}} extracts various job configuration properties like
# job-name
# queue-names
# mapred-child task jvm settings etc
from job's conf. So this bug is seen only when there is a jobconf present in
the jobhistory folder.
> [Rumen] TraceBuilder's output shows jobname as NULL for jobhistory files with
> valid jobnames
>
>
> Key: MAPREDUCE-1982
> URL: https://issues.apache.org/jira/browse/MAPREDUCE-1982
> Project: Hadoop Map/Reduce
> Issue Type: Bug
> Components: tools/rumen
>Reporter: Amar Kamat
>Assignee: Ravi Gummadi
>
> {{TraceBuilder}} fails to extract configuration properties (like job-name)
> from the job-conf if the job-conf has the properties stored using the
> deprecated keys.
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
[jira] Assigned: (MAPREDUCE-1982) [Rumen] TraceBuilder's output shows jobname as NULL for jobhistory files with valid jobnames
[
https://issues.apache.org/jira/browse/MAPREDUCE-1982?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Ranjit Mathew reassigned MAPREDUCE-1982:
Assignee: Ravi Gummadi
An example fragment is:
--- 8<
---
{
"priority" : "NORMAL",
"jobID" : "job_201007280625_0001",
"user" : "gridperf",
"jobName" : null,
"mapTasks" : [ {
"startTime" : 1280298532016,
"taskID" : "task_201007280625_0001_m_000323",
"taskType" : "MAP",
"attempts" : [ {
"location" : {
"layers" : [ "98\\.137\\.102\\.64",
"gsbl91438\\.blue\\.ygrid\\.yahoo\\.com" ]
},
"hostName" :
"/98\\.137\\.102\\.64/gsbl91438\\.blue\\.ygrid\\.yahoo\\.com",
"startTime" : 1280298532028,
"finishTime" : 1280298597501,
"result" : "SUCCESS",
--- 8<
---
This is because Rumen directly uses Properties instead of Configuration - the
new key-name is "mapreduce.job.name", while the old key-name was
"mapred.job.name". Using Configuration insulates against such changes.
> [Rumen] TraceBuilder's output shows jobname as NULL for jobhistory files with
> valid jobnames
>
>
> Key: MAPREDUCE-1982
> URL: https://issues.apache.org/jira/browse/MAPREDUCE-1982
> Project: Hadoop Map/Reduce
> Issue Type: Bug
> Components: tools/rumen
>Reporter: Amar Kamat
>Assignee: Ravi Gummadi
>
> {{TraceBuilder}} fails to extract configuration properties (like job-name)
> from the job-conf if the job-conf has the properties stored using the
> deprecated keys.
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
[jira] Created: (MAPREDUCE-1982) [Rumen] TraceBuilder's output shows jobname as NULL for jobhistory files with valid jobnames
[Rumen] TraceBuilder's output shows jobname as NULL for jobhistory files with
valid jobnames
Key: MAPREDUCE-1982
URL: https://issues.apache.org/jira/browse/MAPREDUCE-1982
Project: Hadoop Map/Reduce
Issue Type: Bug
Components: tools/rumen
Reporter: Amar Kamat
{{TraceBuilder}} fails to extract configuration properties (like job-name) from
the job-conf if the job-conf has the properties stored using the deprecated
keys.
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
