[jira] [Commented] (MAPREDUCE-5094) Disable mem monitoring by default in MiniMRYarnCluster
[
https://issues.apache.org/jira/browse/MAPREDUCE-5094?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13628716#comment-13628716
]
Hadoop QA commented on MAPREDUCE-5094:
--
{color:green}+1 overall{color}. Here are the results of testing the latest
attachment
http://issues.apache.org/jira/secure/attachment/12578146/MAPREDUCE-5094.txt
against trunk revision .
{color:green}+1 @author{color}. The patch does not contain any @author
tags.
{color:green}+1 tests included{color}. The patch appears to include 1 new
or modified test files.
{color:green}+1 javac{color}. The applied patch does not increase the
total number of javac compiler warnings.
{color:green}+1 javadoc{color}. The javadoc tool did not generate any
warning messages.
{color:green}+1 eclipse:eclipse{color}. The patch built with
eclipse:eclipse.
{color:green}+1 findbugs{color}. The patch does not introduce any new
Findbugs (version 1.3.9) warnings.
{color:green}+1 release audit{color}. The applied patch does not increase
the total number of release audit warnings.
{color:green}+1 core tests{color}. The patch passed unit tests in
hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-core
hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-jobclient.
{color:green}+1 contrib tests{color}. The patch passed contrib unit tests.
Test results:
https://builds.apache.org/job/PreCommit-MAPREDUCE-Build/3517//testReport/
Console output:
https://builds.apache.org/job/PreCommit-MAPREDUCE-Build/3517//console
This message is automatically generated.
> Disable mem monitoring by default in MiniMRYarnCluster
> --
>
> Key: MAPREDUCE-5094
> URL: https://issues.apache.org/jira/browse/MAPREDUCE-5094
> Project: Hadoop Map/Reduce
> Issue Type: Bug
>Affects Versions: 2.0.3-alpha
>Reporter: Siddharth Seth
>Assignee: Siddharth Seth
> Attachments: MAPREDUCE-5094.txt, MAPREDUCE-5094.txt,
> MAPREDUCE-5094.txt
>
>
> YARN-449. Some hbase tests were failing since containers were getting killed.
> I believe these checks are disabled by default on the branch-1 MiniMRCluster.
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (MAPREDUCE-4974) Optimising the LineRecordReader initialize() method
[ https://issues.apache.org/jira/browse/MAPREDUCE-4974?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13628715#comment-13628715 ] Gera Shegalov commented on MAPREDUCE-4974: -- [~gelesh], I am not a committer. It looks fine to me. While waiting for committers' review, if you have not done so yet you probably should rerun locally the inputformat-related test suites to make sure nothing is broken. And you should maybe also try to quantify the performance improvement using some local MR job. > Optimising the LineRecordReader initialize() method > --- > > Key: MAPREDUCE-4974 > URL: https://issues.apache.org/jira/browse/MAPREDUCE-4974 > Project: Hadoop Map/Reduce > Issue Type: Improvement > Components: mrv1, mrv2, performance >Affects Versions: 2.0.2-alpha, 0.23.5 > Environment: Hadoop Linux >Reporter: Arun A K >Assignee: Gelesh > Labels: patch, performance > Fix For: 0.23.7, 2.0.5-beta > > Attachments: MAPREDUCE-4974.2.patch, MAPREDUCE-4974.3.patch, > MAPREDUCE-4974.4.patch, MAPREDUCE-4974.5.patch > > Original Estimate: 1h > Remaining Estimate: 1h > > I found there is a a scope of optimizing the code, over initialize() if we > have compressionCodecs & codec instantiated only if its a compressed input. > Mean while Gelesh George Omathil, added if we could avoid the null check of > key & value. This would time save, since for every next key value generation, > null check is done. The intention being to instantiate only once and avoid > NPE as well. Hope both could be met if initialize key & value over > initialize() method. We both have worked on it. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (MAPREDUCE-5059) Job overview shows average merge time larger than for any reduce attempt
[
https://issues.apache.org/jira/browse/MAPREDUCE-5059?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13628694#comment-13628694
]
Hadoop QA commented on MAPREDUCE-5059:
--
{color:green}+1 overall{color}. Here are the results of testing the latest
attachment
http://issues.apache.org/jira/secure/attachment/12578138/MAPREDUCE-5059-20130410.patch
against trunk revision .
{color:green}+1 @author{color}. The patch does not contain any @author
tags.
{color:green}+1 tests included{color}. The patch appears to include 3 new
or modified test files.
{color:green}+1 javac{color}. The applied patch does not increase the
total number of javac compiler warnings.
{color:green}+1 javadoc{color}. The javadoc tool did not generate any
warning messages.
{color:green}+1 eclipse:eclipse{color}. The patch built with
eclipse:eclipse.
{color:green}+1 findbugs{color}. The patch does not introduce any new
Findbugs (version 1.3.9) warnings.
{color:green}+1 release audit{color}. The applied patch does not increase
the total number of release audit warnings.
{color:green}+1 core tests{color}. The patch passed unit tests in
hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-hs.
{color:green}+1 contrib tests{color}. The patch passed contrib unit tests.
Test results:
https://builds.apache.org/job/PreCommit-MAPREDUCE-Build/3516//testReport/
Console output:
https://builds.apache.org/job/PreCommit-MAPREDUCE-Build/3516//console
This message is automatically generated.
> Job overview shows average merge time larger than for any reduce attempt
>
>
> Key: MAPREDUCE-5059
> URL: https://issues.apache.org/jira/browse/MAPREDUCE-5059
> Project: Hadoop Map/Reduce
> Issue Type: Bug
> Components: jobhistoryserver, webapps
>Reporter: Jason Lowe
>Assignee: Omkar Vinit Joshi
> Attachments: MAPREDUCE-5059-20130325.patch,
> MAPREDUCE-5059-20130410.patch
>
>
> When looking at a job overview page on the history server, the Average Merge
> Time is often reported with a value that is far larger than the Elapsed Merge
> Time shown for any reduce task attempt. The job overview page calculates the
> merge time as the time delta between the sort finishing and the job launching
> while the attempts page calculates it as the time delta between the sort
> finishing and the shuffle finishing.
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (MAPREDUCE-5053) java.lang.InternalError from decompression codec cause reducer to fail
[ https://issues.apache.org/jira/browse/MAPREDUCE-5053?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13628687#comment-13628687 ] Siddharth Seth commented on MAPREDUCE-5053: --- [~jeagles] , Is the fix version correct ? - did this go into 2.0.4-alpha, or to branch-2 (2.0.5-beta) > java.lang.InternalError from decompression codec cause reducer to fail > -- > > Key: MAPREDUCE-5053 > URL: https://issues.apache.org/jira/browse/MAPREDUCE-5053 > Project: Hadoop Map/Reduce > Issue Type: Bug >Affects Versions: trunk, 2.0.3-alpha, 0.23.5 >Reporter: Robert Parker >Assignee: Robert Parker > Fix For: 3.0.0, 0.23.7, 2.0.4-alpha > > Attachments: MAPREDUCE-5053-1.patch, MAPREDUCE-5053-2.patch, > MAPREDUCE-5053-b023-1.patch > > > lz4, snappy, zlib, and lzo Decompressor's only throw java.lang.InternalError. > This exception will cause the reducer to fail and bypass all of the fetch > failure logic. The decompressing errors should be treated as fetch failures. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (MAPREDUCE-5079) Recovery should restore task state from job history info directly
[ https://issues.apache.org/jira/browse/MAPREDUCE-5079?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13628685#comment-13628685 ] Hudson commented on MAPREDUCE-5079: --- Integrated in Hadoop-trunk-Commit #3592 (See [https://builds.apache.org/job/Hadoop-trunk-Commit/3592/]) MAPREDUCE-5079. Changes job recovery to restore state directly from job history, instaed of simulating state machine events. Contributed by Jason Lowe and Robert Parker. (Revision 1466767) Result = SUCCESS sseth : http://svn.apache.org/viewcvs.cgi/?root=Apache-SVN&view=rev&rev=1466767 Files : * /hadoop/common/trunk/hadoop-mapreduce-project/CHANGES.txt * /hadoop/common/trunk/hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-app/src/main/java/org/apache/hadoop/mapreduce/v2/app/MRAppMaster.java * /hadoop/common/trunk/hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-app/src/main/java/org/apache/hadoop/mapreduce/v2/app/job/event/JobStartEvent.java * /hadoop/common/trunk/hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-app/src/main/java/org/apache/hadoop/mapreduce/v2/app/job/event/TaskAttemptEventType.java * /hadoop/common/trunk/hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-app/src/main/java/org/apache/hadoop/mapreduce/v2/app/job/event/TaskAttemptRecoverEvent.java * /hadoop/common/trunk/hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-app/src/main/java/org/apache/hadoop/mapreduce/v2/app/job/event/TaskEventType.java * /hadoop/common/trunk/hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-app/src/main/java/org/apache/hadoop/mapreduce/v2/app/job/event/TaskRecoverEvent.java * /hadoop/common/trunk/hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-app/src/main/java/org/apache/hadoop/mapreduce/v2/app/job/impl/JobImpl.java * /hadoop/common/trunk/hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-app/src/main/java/org/apache/hadoop/mapreduce/v2/app/job/impl/MapTaskImpl.java * /hadoop/common/trunk/hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-app/src/main/java/org/apache/hadoop/mapreduce/v2/app/job/impl/ReduceTaskImpl.java * /hadoop/common/trunk/hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-app/src/main/java/org/apache/hadoop/mapreduce/v2/app/job/impl/TaskAttemptImpl.java * /hadoop/common/trunk/hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-app/src/main/java/org/apache/hadoop/mapreduce/v2/app/job/impl/TaskImpl.java * /hadoop/common/trunk/hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-app/src/main/java/org/apache/hadoop/mapreduce/v2/app/recover * /hadoop/common/trunk/hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-app/src/test/java/org/apache/hadoop/mapreduce/v2/app/MRApp.java * /hadoop/common/trunk/hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-app/src/test/java/org/apache/hadoop/mapreduce/v2/app/TestRecovery.java * /hadoop/common/trunk/hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-app/src/test/java/org/apache/hadoop/mapreduce/v2/app/TestStagingCleanup.java * /hadoop/common/trunk/hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-app/src/test/java/org/apache/hadoop/mapreduce/v2/app/job/impl/TestJobImpl.java * /hadoop/common/trunk/hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-app/src/test/java/org/apache/hadoop/mapreduce/v2/app/job/impl/TestTaskImpl.java * /hadoop/common/trunk/hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-common/src/main/java/org/apache/hadoop/mapreduce/v2/jobhistory/JobHistoryUtils.java * /hadoop/common/trunk/hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-core/src/main/java/org/apache/hadoop/mapreduce/MRJobConfig.java > Recovery should restore task state from job history info directly > - > > Key: MAPREDUCE-5079 > URL: https://issues.apache.org/jira/browse/MAPREDUCE-5079 > Project: Hadoop Map/Reduce > Issue Type: Improvement > Components: mr-am >Affects Versions: 0.23.7 >Reporter: Jason Lowe >Assignee: Jason Lowe >Priority: Critical > Fix For: 0.23.7, 2.0.5-beta > > Attachments: MAPREDUCE-5079-branch-0.23.patch, > MAPREDUCE-5079-branch-0.23.patch, MAPREDUCE-5079.patch, MAPREDUCE-5079.patch, > MAPREDUCE-5079.patch, MAPREDUCE-5079.patch > > > We've encountered a lot of hanging issues during MR-AM recovery because the > state machines don't always end up in the same states after recovery. This > is especially true when speculative execution is enabled. I
[jira] [Updated] (MAPREDUCE-5079) Recovery should restore task state from job history info directly
[ https://issues.apache.org/jira/browse/MAPREDUCE-5079?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Siddharth Seth updated MAPREDUCE-5079: -- Resolution: Fixed Fix Version/s: 2.0.5-beta 0.23.7 Hadoop Flags: Reviewed Status: Resolved (was: Patch Available) Committed to trunk, branch-2 and branch-0.23. > Recovery should restore task state from job history info directly > - > > Key: MAPREDUCE-5079 > URL: https://issues.apache.org/jira/browse/MAPREDUCE-5079 > Project: Hadoop Map/Reduce > Issue Type: Improvement > Components: mr-am >Affects Versions: 0.23.7 >Reporter: Jason Lowe >Assignee: Jason Lowe >Priority: Critical > Fix For: 0.23.7, 2.0.5-beta > > Attachments: MAPREDUCE-5079-branch-0.23.patch, > MAPREDUCE-5079-branch-0.23.patch, MAPREDUCE-5079.patch, MAPREDUCE-5079.patch, > MAPREDUCE-5079.patch, MAPREDUCE-5079.patch > > > We've encountered a lot of hanging issues during MR-AM recovery because the > state machines don't always end up in the same states after recovery. This > is especially true when speculative execution is enabled. It should be > straightforward to restore task and task attempt states directly from the > TaskInfo and TaskAttemptInfo records in the job history file to avoid relying > on the task state machines ending up in the proper states with the proper > number of attempts. > This should be a more robust solution that would also give us the option of > recovering start time and log locations for tasks that were in-progress when > the AM crashed. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (MAPREDUCE-5079) Recovery should restore task state from job history info directly
[ https://issues.apache.org/jira/browse/MAPREDUCE-5079?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13628661#comment-13628661 ] Siddharth Seth commented on MAPREDUCE-5079: --- +1. Thanks Jason, and Robert for the unit tests. Committing this. > Recovery should restore task state from job history info directly > - > > Key: MAPREDUCE-5079 > URL: https://issues.apache.org/jira/browse/MAPREDUCE-5079 > Project: Hadoop Map/Reduce > Issue Type: Improvement > Components: mr-am >Affects Versions: 0.23.7 >Reporter: Jason Lowe >Assignee: Jason Lowe >Priority: Critical > Attachments: MAPREDUCE-5079-branch-0.23.patch, > MAPREDUCE-5079-branch-0.23.patch, MAPREDUCE-5079.patch, MAPREDUCE-5079.patch, > MAPREDUCE-5079.patch, MAPREDUCE-5079.patch > > > We've encountered a lot of hanging issues during MR-AM recovery because the > state machines don't always end up in the same states after recovery. This > is especially true when speculative execution is enabled. It should be > straightforward to restore task and task attempt states directly from the > TaskInfo and TaskAttemptInfo records in the job history file to avoid relying > on the task state machines ending up in the proper states with the proper > number of attempts. > This should be a more robust solution that would also give us the option of > recovering start time and log locations for tasks that were in-progress when > the AM crashed. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Updated] (MAPREDUCE-5094) Disable mem monitoring by default in MiniMRYarnCluster
[ https://issues.apache.org/jira/browse/MAPREDUCE-5094?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Siddharth Seth updated MAPREDUCE-5094: -- Status: Patch Available (was: Open) > Disable mem monitoring by default in MiniMRYarnCluster > -- > > Key: MAPREDUCE-5094 > URL: https://issues.apache.org/jira/browse/MAPREDUCE-5094 > Project: Hadoop Map/Reduce > Issue Type: Bug >Affects Versions: 2.0.3-alpha >Reporter: Siddharth Seth >Assignee: Siddharth Seth > Attachments: MAPREDUCE-5094.txt, MAPREDUCE-5094.txt, > MAPREDUCE-5094.txt > > > YARN-449. Some hbase tests were failing since containers were getting killed. > I believe these checks are disabled by default on the branch-1 MiniMRCluster. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Updated] (MAPREDUCE-5094) Disable mem monitoring by default in MiniMRYarnCluster
[ https://issues.apache.org/jira/browse/MAPREDUCE-5094?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Siddharth Seth updated MAPREDUCE-5094: -- Attachment: MAPREDUCE-5094.txt Oops, looks like i uploaded the wrong file last time. Correct patch this time with the pmem check disabled. > Disable mem monitoring by default in MiniMRYarnCluster > -- > > Key: MAPREDUCE-5094 > URL: https://issues.apache.org/jira/browse/MAPREDUCE-5094 > Project: Hadoop Map/Reduce > Issue Type: Bug >Affects Versions: 2.0.3-alpha >Reporter: Siddharth Seth >Assignee: Siddharth Seth > Attachments: MAPREDUCE-5094.txt, MAPREDUCE-5094.txt, > MAPREDUCE-5094.txt > > > YARN-449. Some hbase tests were failing since containers were getting killed. > I believe these checks are disabled by default on the branch-1 MiniMRCluster. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Updated] (MAPREDUCE-5094) Disable mem monitoring by default in MiniMRYarnCluster
[ https://issues.apache.org/jira/browse/MAPREDUCE-5094?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Siddharth Seth updated MAPREDUCE-5094: -- Status: Open (was: Patch Available) > Disable mem monitoring by default in MiniMRYarnCluster > -- > > Key: MAPREDUCE-5094 > URL: https://issues.apache.org/jira/browse/MAPREDUCE-5094 > Project: Hadoop Map/Reduce > Issue Type: Bug >Affects Versions: 2.0.3-alpha >Reporter: Siddharth Seth >Assignee: Siddharth Seth > Attachments: MAPREDUCE-5094.txt, MAPREDUCE-5094.txt > > > YARN-449. Some hbase tests were failing since containers were getting killed. > I believe these checks are disabled by default on the branch-1 MiniMRCluster. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (MAPREDUCE-5094) Disable mem monitoring by default in MiniMRYarnCluster
[ https://issues.apache.org/jira/browse/MAPREDUCE-5094?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13628623#comment-13628623 ] Vinod Kumar Vavilapalli commented on MAPREDUCE-5094: Looks like this is the same patch the previous one, can you post an updated patch addressing my comment? > Disable mem monitoring by default in MiniMRYarnCluster > -- > > Key: MAPREDUCE-5094 > URL: https://issues.apache.org/jira/browse/MAPREDUCE-5094 > Project: Hadoop Map/Reduce > Issue Type: Bug >Affects Versions: 2.0.3-alpha >Reporter: Siddharth Seth >Assignee: Siddharth Seth > Attachments: MAPREDUCE-5094.txt, MAPREDUCE-5094.txt > > > YARN-449. Some hbase tests were failing since containers were getting killed. > I believe these checks are disabled by default on the branch-1 MiniMRCluster. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Updated] (MAPREDUCE-5059) Job overview shows average merge time larger than for any reduce attempt
[ https://issues.apache.org/jira/browse/MAPREDUCE-5059?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Omkar Vinit Joshi updated MAPREDUCE-5059: - Attachment: MAPREDUCE-5059-20130410.patch > Job overview shows average merge time larger than for any reduce attempt > > > Key: MAPREDUCE-5059 > URL: https://issues.apache.org/jira/browse/MAPREDUCE-5059 > Project: Hadoop Map/Reduce > Issue Type: Bug > Components: jobhistoryserver, webapps >Reporter: Jason Lowe >Assignee: Omkar Vinit Joshi > Attachments: MAPREDUCE-5059-20130325.patch, > MAPREDUCE-5059-20130410.patch > > > When looking at a job overview page on the history server, the Average Merge > Time is often reported with a value that is far larger than the Elapsed Merge > Time shown for any reduce task attempt. The job overview page calculates the > merge time as the time delta between the sort finishing and the job launching > while the attempts page calculates it as the time delta between the sort > finishing and the shuffle finishing. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Updated] (MAPREDUCE-5059) Job overview shows average merge time larger than for any reduce attempt
[ https://issues.apache.org/jira/browse/MAPREDUCE-5059?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Omkar Vinit Joshi updated MAPREDUCE-5059: - Status: Patch Available (was: Open) > Job overview shows average merge time larger than for any reduce attempt > > > Key: MAPREDUCE-5059 > URL: https://issues.apache.org/jira/browse/MAPREDUCE-5059 > Project: Hadoop Map/Reduce > Issue Type: Bug > Components: jobhistoryserver, webapps >Reporter: Jason Lowe >Assignee: Omkar Vinit Joshi > Attachments: MAPREDUCE-5059-20130325.patch, > MAPREDUCE-5059-20130410.patch > > > When looking at a job overview page on the history server, the Average Merge > Time is often reported with a value that is far larger than the Elapsed Merge > Time shown for any reduce task attempt. The job overview page calculates the > merge time as the time delta between the sort finishing and the job launching > while the attempts page calculates it as the time delta between the sort > finishing and the shuffle finishing. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (MAPREDUCE-5059) Job overview shows average merge time larger than for any reduce attempt
[ https://issues.apache.org/jira/browse/MAPREDUCE-5059?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13628598#comment-13628598 ] Omkar Vinit Joshi commented on MAPREDUCE-5059: -- * removed commented code * Yes individual merge time is 45 and 55. I have added a comment in the TestJobInfo. > Job overview shows average merge time larger than for any reduce attempt > > > Key: MAPREDUCE-5059 > URL: https://issues.apache.org/jira/browse/MAPREDUCE-5059 > Project: Hadoop Map/Reduce > Issue Type: Bug > Components: jobhistoryserver, webapps >Reporter: Jason Lowe >Assignee: Omkar Vinit Joshi > Attachments: MAPREDUCE-5059-20130325.patch, > MAPREDUCE-5059-20130410.patch > > > When looking at a job overview page on the history server, the Average Merge > Time is often reported with a value that is far larger than the Elapsed Merge > Time shown for any reduce task attempt. The job overview page calculates the > merge time as the time delta between the sort finishing and the job launching > while the attempts page calculates it as the time delta between the sort > finishing and the shuffle finishing. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Updated] (MAPREDUCE-5059) Job overview shows average merge time larger than for any reduce attempt
[ https://issues.apache.org/jira/browse/MAPREDUCE-5059?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Vinod Kumar Vavilapalli updated MAPREDUCE-5059: --- Status: Open (was: Patch Available) bq. Personally I think it'd be more useful to calculate merge time as the time delta between the end of the shuffle and the start of the reduce phase (i.e.: what the task attempts page is already showing for each reduce task attempt). I understand that the shuffle and merge phases overlap in practice, but really what we're looking for here is excessive time spent merging even after the data has been shuffled to the reducer. +1 for doing this, I too agree that it is more useful this way. The patch looks mostly good to me. - Can you remove the old code completely, instead of commenting it out? - In the test, I couldn't figure out why the expected time was 50, can you put a small comment saying it's an average of 45 and 55, or something of that sort? Also the shown time is sort+merge time from shuffleEnd, not sure how we can make that clear. > Job overview shows average merge time larger than for any reduce attempt > > > Key: MAPREDUCE-5059 > URL: https://issues.apache.org/jira/browse/MAPREDUCE-5059 > Project: Hadoop Map/Reduce > Issue Type: Bug > Components: jobhistoryserver, webapps >Reporter: Jason Lowe >Assignee: Omkar Vinit Joshi > Attachments: MAPREDUCE-5059-20130325.patch > > > When looking at a job overview page on the history server, the Average Merge > Time is often reported with a value that is far larger than the Elapsed Merge > Time shown for any reduce task attempt. The job overview page calculates the > merge time as the time delta between the sort finishing and the job launching > while the attempts page calculates it as the time delta between the sort > finishing and the shuffle finishing. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (MAPREDUCE-5079) Recovery should restore task state from job history info directly
[
https://issues.apache.org/jira/browse/MAPREDUCE-5079?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13628386#comment-13628386
]
Hadoop QA commented on MAPREDUCE-5079:
--
{color:red}-1 overall{color}. Here are the results of testing the latest
attachment
http://issues.apache.org/jira/secure/attachment/12578101/MAPREDUCE-5079-branch-0.23.patch
against trunk revision .
{color:red}-1 patch{color}. The patch command could not apply the patch.
Console output:
https://builds.apache.org/job/PreCommit-MAPREDUCE-Build/3515//console
This message is automatically generated.
> Recovery should restore task state from job history info directly
> -
>
> Key: MAPREDUCE-5079
> URL: https://issues.apache.org/jira/browse/MAPREDUCE-5079
> Project: Hadoop Map/Reduce
> Issue Type: Improvement
> Components: mr-am
>Affects Versions: 0.23.7
>Reporter: Jason Lowe
>Assignee: Jason Lowe
>Priority: Critical
> Attachments: MAPREDUCE-5079-branch-0.23.patch,
> MAPREDUCE-5079-branch-0.23.patch, MAPREDUCE-5079.patch, MAPREDUCE-5079.patch,
> MAPREDUCE-5079.patch, MAPREDUCE-5079.patch
>
>
> We've encountered a lot of hanging issues during MR-AM recovery because the
> state machines don't always end up in the same states after recovery. This
> is especially true when speculative execution is enabled. It should be
> straightforward to restore task and task attempt states directly from the
> TaskInfo and TaskAttemptInfo records in the job history file to avoid relying
> on the task state machines ending up in the proper states with the proper
> number of attempts.
> This should be a more robust solution that would also give us the option of
> recovering start time and log locations for tasks that were in-progress when
> the AM crashed.
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Updated] (MAPREDUCE-5079) Recovery should restore task state from job history info directly
[ https://issues.apache.org/jira/browse/MAPREDUCE-5079?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Jason Lowe updated MAPREDUCE-5079: -- Attachment: MAPREDUCE-5079-branch-0.23.patch Updated branch-0.23 patch with the same changes. > Recovery should restore task state from job history info directly > - > > Key: MAPREDUCE-5079 > URL: https://issues.apache.org/jira/browse/MAPREDUCE-5079 > Project: Hadoop Map/Reduce > Issue Type: Improvement > Components: mr-am >Affects Versions: 0.23.7 >Reporter: Jason Lowe >Assignee: Jason Lowe >Priority: Critical > Attachments: MAPREDUCE-5079-branch-0.23.patch, > MAPREDUCE-5079-branch-0.23.patch, MAPREDUCE-5079.patch, MAPREDUCE-5079.patch, > MAPREDUCE-5079.patch, MAPREDUCE-5079.patch > > > We've encountered a lot of hanging issues during MR-AM recovery because the > state machines don't always end up in the same states after recovery. This > is especially true when speculative execution is enabled. It should be > straightforward to restore task and task attempt states directly from the > TaskInfo and TaskAttemptInfo records in the job history file to avoid relying > on the task state machines ending up in the proper states with the proper > number of attempts. > This should be a more robust solution that would also give us the option of > recovering start time and log locations for tasks that were in-progress when > the AM crashed. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (MAPREDUCE-5079) Recovery should restore task state from job history info directly
[
https://issues.apache.org/jira/browse/MAPREDUCE-5079?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13628378#comment-13628378
]
Hadoop QA commented on MAPREDUCE-5079:
--
{color:green}+1 overall{color}. Here are the results of testing the latest
attachment
http://issues.apache.org/jira/secure/attachment/12578094/MAPREDUCE-5079.patch
against trunk revision .
{color:green}+1 @author{color}. The patch does not contain any @author
tags.
{color:green}+1 tests included{color}. The patch appears to include 5 new
or modified test files.
{color:green}+1 javac{color}. The applied patch does not increase the
total number of javac compiler warnings.
{color:green}+1 javadoc{color}. The javadoc tool did not generate any
warning messages.
{color:green}+1 eclipse:eclipse{color}. The patch built with
eclipse:eclipse.
{color:green}+1 findbugs{color}. The patch does not introduce any new
Findbugs (version 1.3.9) warnings.
{color:green}+1 release audit{color}. The applied patch does not increase
the total number of release audit warnings.
{color:green}+1 core tests{color}. The patch passed unit tests in
hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-app
hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-common
hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-core.
{color:green}+1 contrib tests{color}. The patch passed contrib unit tests.
Test results:
https://builds.apache.org/job/PreCommit-MAPREDUCE-Build/3514//testReport/
Console output:
https://builds.apache.org/job/PreCommit-MAPREDUCE-Build/3514//console
This message is automatically generated.
> Recovery should restore task state from job history info directly
> -
>
> Key: MAPREDUCE-5079
> URL: https://issues.apache.org/jira/browse/MAPREDUCE-5079
> Project: Hadoop Map/Reduce
> Issue Type: Improvement
> Components: mr-am
>Affects Versions: 0.23.7
>Reporter: Jason Lowe
>Assignee: Jason Lowe
>Priority: Critical
> Attachments: MAPREDUCE-5079-branch-0.23.patch, MAPREDUCE-5079.patch,
> MAPREDUCE-5079.patch, MAPREDUCE-5079.patch, MAPREDUCE-5079.patch
>
>
> We've encountered a lot of hanging issues during MR-AM recovery because the
> state machines don't always end up in the same states after recovery. This
> is especially true when speculative execution is enabled. It should be
> straightforward to restore task and task attempt states directly from the
> TaskInfo and TaskAttemptInfo records in the job history file to avoid relying
> on the task state machines ending up in the proper states with the proper
> number of attempts.
> This should be a more robust solution that would also give us the option of
> recovering start time and log locations for tasks that were in-progress when
> the AM crashed.
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Updated] (MAPREDUCE-5079) Recovery should restore task state from job history info directly
[ https://issues.apache.org/jira/browse/MAPREDUCE-5079?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Jason Lowe updated MAPREDUCE-5079: -- Attachment: MAPREDUCE-5079.patch Thanks for the in-depth review, Sidd! I addressed most of the review comments, with a couple of exceptions: * getPreviousJobHistoryFileStream had a useful log message in it, and since JobHistoryUtils doesn't have a logger I refactored it into two parts: JobHistoryUtils.getPreviousJobHistoryPath which gets the path and MRAppMaster logs the result and opens it to get the stream. * testRecoveryTaskSuccessAllAttemptsFail already was checking for a new attempt being created by checking for an extra, third attempt in the NEW state. I added a comment to the test to call that out. * testRecoveryTaskSuccessAllAttemptsSucceed already is checking that the JobEventType is returning a SUCCEEDED state in the checkRecovery method. > Recovery should restore task state from job history info directly > - > > Key: MAPREDUCE-5079 > URL: https://issues.apache.org/jira/browse/MAPREDUCE-5079 > Project: Hadoop Map/Reduce > Issue Type: Improvement > Components: mr-am >Affects Versions: 0.23.7 >Reporter: Jason Lowe >Assignee: Jason Lowe >Priority: Critical > Attachments: MAPREDUCE-5079-branch-0.23.patch, MAPREDUCE-5079.patch, > MAPREDUCE-5079.patch, MAPREDUCE-5079.patch, MAPREDUCE-5079.patch > > > We've encountered a lot of hanging issues during MR-AM recovery because the > state machines don't always end up in the same states after recovery. This > is especially true when speculative execution is enabled. It should be > straightforward to restore task and task attempt states directly from the > TaskInfo and TaskAttemptInfo records in the job history file to avoid relying > on the task state machines ending up in the proper states with the proper > number of attempts. > This should be a more robust solution that would also give us the option of > recovering start time and log locations for tasks that were in-progress when > the AM crashed. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Updated] (MAPREDUCE-5142) MR AM unregisters with state KILLED when an error causes dispatcher to shutdown
[
https://issues.apache.org/jira/browse/MAPREDUCE-5142?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Hitesh Shah updated MAPREDUCE-5142:
---
Description:
RMCommunicator sets final state to KILLED if the job is in a running state and
isSignalled is set to true.
{code}
} else if (jobImpl.getInternalState() == JobStateInternal.KILLED
|| (jobImpl.getInternalState() == JobStateInternal.RUNNING &&
isSignalled)) {
finishState = FinalApplicationStatus.KILLED;
} else if (jobImpl.getInternalState() == JobStateInternal.FAILED
|| jobImpl.getInternalState() == JobStateInternal.ERROR) {
finishState = FinalApplicationStatus.FAILED;
{code}
This happens when any uncaught exception in any event handler ends up causing
the AsyncDispatcher to trigger a shutdown. In such a scenario, even though the
AM actually failed due to some error, its actual state ends up as KILLED.
was:
RMCommunicator sets final state to KILLED if the job is in a running state and
isSignalled is set to true.
{code}
} else if (jobImpl.getInternalState() == JobStateInternal.KILLED
|| (jobImpl.getInternalState() == JobStateInternal.RUNNING &&
isSignalled)) {
finishState = FinalApplicationStatus.KILLED;
} else if (jobImpl.getInternalState() == JobStateInternal.FAILED
|| jobImpl.getInternalState() == JobStateInternal.ERROR) {
finishState = FinalApplicationStatus.FAILED;
{code}
This happens when for some reason, there is an exception in a state machine's
event handler causing AsyncDispatcher to trigger a shutdown. In such a
scenario, even though the AM actually failed due to some error, its actual
state ends up as KILLED.
> MR AM unregisters with state KILLED when an error causes dispatcher to
> shutdown
> ---
>
> Key: MAPREDUCE-5142
> URL: https://issues.apache.org/jira/browse/MAPREDUCE-5142
> Project: Hadoop Map/Reduce
> Issue Type: Bug
>Affects Versions: 2.0.3-alpha, 0.23.5
>Reporter: Hitesh Shah
>
> RMCommunicator sets final state to KILLED if the job is in a running state
> and isSignalled is set to true.
> {code}
> } else if (jobImpl.getInternalState() == JobStateInternal.KILLED
> || (jobImpl.getInternalState() == JobStateInternal.RUNNING &&
> isSignalled)) {
> finishState = FinalApplicationStatus.KILLED;
> } else if (jobImpl.getInternalState() == JobStateInternal.FAILED
> || jobImpl.getInternalState() == JobStateInternal.ERROR) {
> finishState = FinalApplicationStatus.FAILED;
> {code}
> This happens when any uncaught exception in any event handler ends up causing
> the AsyncDispatcher to trigger a shutdown. In such a scenario, even though
> the AM actually failed due to some error, its actual state ends up as KILLED.
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Updated] (MAPREDUCE-5142) MR AM unregisters with state KILLED when an error causes dispatcher to shutdown
[
https://issues.apache.org/jira/browse/MAPREDUCE-5142?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Hitesh Shah updated MAPREDUCE-5142:
---
Affects Version/s: 2.0.3-alpha
0.23.5
> MR AM unregisters with state KILLED when an error causes dispatcher to
> shutdown
> ---
>
> Key: MAPREDUCE-5142
> URL: https://issues.apache.org/jira/browse/MAPREDUCE-5142
> Project: Hadoop Map/Reduce
> Issue Type: Bug
>Affects Versions: 2.0.3-alpha, 0.23.5
>Reporter: Hitesh Shah
>
> RMCommunicator sets final state to KILLED if the job is in a running state
> and isSignalled is set to true.
> {code}
> } else if (jobImpl.getInternalState() == JobStateInternal.KILLED
> || (jobImpl.getInternalState() == JobStateInternal.RUNNING &&
> isSignalled)) {
> finishState = FinalApplicationStatus.KILLED;
> } else if (jobImpl.getInternalState() == JobStateInternal.FAILED
> || jobImpl.getInternalState() == JobStateInternal.ERROR) {
> finishState = FinalApplicationStatus.FAILED;
> {code}
> This happens when for some reason, there is an exception in a state machine's
> event handler causing AsyncDispatcher to trigger a shutdown. In such a
> scenario, even though the AM actually failed due to some error, its actual
> state ends up as KILLED.
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (MAPREDUCE-5142) MR AM unregisters with state KILLED when an error causes dispatcher to shutdown
[
https://issues.apache.org/jira/browse/MAPREDUCE-5142?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13628187#comment-13628187
]
Hitesh Shah commented on MAPREDUCE-5142:
@Jason, yes - definitely the same underlying issue. Addressing the CLC creation
would address a part of the issue but currently all uncaught exceptions will
end up with the AM in a KILLED state.
> MR AM unregisters with state KILLED when an error causes dispatcher to
> shutdown
> ---
>
> Key: MAPREDUCE-5142
> URL: https://issues.apache.org/jira/browse/MAPREDUCE-5142
> Project: Hadoop Map/Reduce
> Issue Type: Bug
>Reporter: Hitesh Shah
>
> RMCommunicator sets final state to KILLED if the job is in a running state
> and isSignalled is set to true.
> {code}
> } else if (jobImpl.getInternalState() == JobStateInternal.KILLED
> || (jobImpl.getInternalState() == JobStateInternal.RUNNING &&
> isSignalled)) {
> finishState = FinalApplicationStatus.KILLED;
> } else if (jobImpl.getInternalState() == JobStateInternal.FAILED
> || jobImpl.getInternalState() == JobStateInternal.ERROR) {
> finishState = FinalApplicationStatus.FAILED;
> {code}
> This happens when for some reason, there is an exception in a state machine's
> event handler causing AsyncDispatcher to trigger a shutdown. In such a
> scenario, even though the AM actually failed due to some error, its actual
> state ends up as KILLED.
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (MAPREDUCE-5142) MR AM unregisters with state KILLED when an error causes dispatcher to shutdown
[
https://issues.apache.org/jira/browse/MAPREDUCE-5142?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13628177#comment-13628177
]
Jason Lowe commented on MAPREDUCE-5142:
---
This looks like the general case of MAPREDUCE-4993.
> MR AM unregisters with state KILLED when an error causes dispatcher to
> shutdown
> ---
>
> Key: MAPREDUCE-5142
> URL: https://issues.apache.org/jira/browse/MAPREDUCE-5142
> Project: Hadoop Map/Reduce
> Issue Type: Bug
>Reporter: Hitesh Shah
>
> RMCommunicator sets final state to KILLED if the job is in a running state
> and isSignalled is set to true.
> {code}
> } else if (jobImpl.getInternalState() == JobStateInternal.KILLED
> || (jobImpl.getInternalState() == JobStateInternal.RUNNING &&
> isSignalled)) {
> finishState = FinalApplicationStatus.KILLED;
> } else if (jobImpl.getInternalState() == JobStateInternal.FAILED
> || jobImpl.getInternalState() == JobStateInternal.ERROR) {
> finishState = FinalApplicationStatus.FAILED;
> {code}
> This happens when for some reason, there is an exception in a state machine's
> event handler causing AsyncDispatcher to trigger a shutdown. In such a
> scenario, even though the AM actually failed due to some error, its actual
> state ends up as KILLED.
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Created] (MAPREDUCE-5142) MR AM unregisters with state KILLED when an error causes dispatcher to shutdown
Hitesh Shah created MAPREDUCE-5142:
--
Summary: MR AM unregisters with state KILLED when an error causes
dispatcher to shutdown
Key: MAPREDUCE-5142
URL: https://issues.apache.org/jira/browse/MAPREDUCE-5142
Project: Hadoop Map/Reduce
Issue Type: Bug
Reporter: Hitesh Shah
RMCommunicator sets final state to KILLED if the job is in a running state and
isSignalled is set to true.
{code}
} else if (jobImpl.getInternalState() == JobStateInternal.KILLED
|| (jobImpl.getInternalState() == JobStateInternal.RUNNING &&
isSignalled)) {
finishState = FinalApplicationStatus.KILLED;
} else if (jobImpl.getInternalState() == JobStateInternal.FAILED
|| jobImpl.getInternalState() == JobStateInternal.ERROR) {
finishState = FinalApplicationStatus.FAILED;
{code}
This happens when for some reason, there is an exception in a state machine's
event handler causing AsyncDispatcher to trigger a shutdown. In such a
scenario, even though the AM actually failed due to some error, its actual
state ends up as KILLED.
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (MAPREDUCE-5137) AM web UI: clicking on Map Task results in 500 error
[
https://issues.apache.org/jira/browse/MAPREDUCE-5137?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13628060#comment-13628060
]
Hadoop QA commented on MAPREDUCE-5137:
--
{color:red}-1 overall{color}. Here are the results of testing the latest
attachment
http://issues.apache.org/jira/secure/attachment/12578042/MAPREDUCE-5137.patch
against trunk revision .
{color:green}+1 @author{color}. The patch does not contain any @author
tags.
{color:red}-1 tests included{color}. The patch doesn't appear to include
any new or modified tests.
Please justify why no new tests are needed for this
patch.
Also please list what manual steps were performed to
verify this patch.
{color:green}+1 javac{color}. The applied patch does not increase the
total number of javac compiler warnings.
{color:green}+1 javadoc{color}. The javadoc tool did not generate any
warning messages.
{color:green}+1 eclipse:eclipse{color}. The patch built with
eclipse:eclipse.
{color:green}+1 findbugs{color}. The patch does not introduce any new
Findbugs (version 1.3.9) warnings.
{color:green}+1 release audit{color}. The applied patch does not increase
the total number of release audit warnings.
{color:green}+1 core tests{color}. The patch passed unit tests in
hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-app.
{color:green}+1 contrib tests{color}. The patch passed contrib unit tests.
Test results:
https://builds.apache.org/job/PreCommit-MAPREDUCE-Build/3513//testReport/
Console output:
https://builds.apache.org/job/PreCommit-MAPREDUCE-Build/3513//console
This message is automatically generated.
> AM web UI: clicking on Map Task results in 500 error
>
>
> Key: MAPREDUCE-5137
> URL: https://issues.apache.org/jira/browse/MAPREDUCE-5137
> Project: Hadoop Map/Reduce
> Issue Type: Bug
> Components: applicationmaster
>Affects Versions: 0.23.7, 2.0.5-beta
>Reporter: Thomas Graves
>Assignee: Thomas Graves
> Attachments: MAPREDUCE-5137.patch
>
>
> Go to a running mapreduce app master web UI. Click on the job, then click on
> the MAP task type to bring up the list of maps, then try to click on a
> particular map task. It fails with a 500 error. Note this doesn't exist in
> 0.23.6.
> Exception in the log looks like:
> 2013-04-09 13:53:01,587 DEBUG [1088374@qtp-13877033-2 -
> /mapreduce/task/task_1365457322543_0004_m_00]
> org.apache.hadoop.yarn.webapp.GenericExceptionHandler: GOT EXCEPITION
> com.sun.jersey.api.NotFoundException: null for uri:
> http://host.com:38158/mapreduce/task/task_1365457322543_0004_m_00
> at
> com.sun.jersey.server.impl.application.WebApplicationImpl._handleRequest(WebApplicationImpl.java:1470)
> at
> com.sun.jersey.server.impl.application.WebApplicationImpl._handleRequest(WebApplicationImpl.java:1400)
> at
> com.sun.jersey.server.impl.application.WebApplicationImpl.handleRequest(WebApplicationImpl.java:1349)
> at
> com.sun.jersey.server.impl.application.WebApplicationImpl.handleRequest(WebApplicationImpl.java:1339)
> at
> com.sun.jersey.spi.container.servlet.WebComponent.service(WebComponent.java:416)
> at
> com.sun.jersey.spi.container.servlet.ServletContainer.service(ServletContainer.java:537)
> at
> com.sun.jersey.spi.container.servlet.ServletContainer.doFilter(ServletContainer.java:886)
> at
> com.sun.jersey.spi.container.servlet.ServletContainer.doFilter(ServletContainer.java:834)
> at
> com.sun.jersey.spi.container.servlet.ServletContainer.doFilter(ServletContainer.java:795)
> at
> com.google.inject.servlet.FilterDefinition.doFilter(FilterDefinition.java:163)
> at
> com.google.inject.servlet.FilterChainInvocation.doFilter(FilterChainInvocation.java:58)
> at
> com.google.inject.servlet.ManagedFilterPipeline.dispatch(ManagedFilterPipeline.java:118)
> at com.google.inject.servlet.GuiceFilter.doFilter(GuiceFilter.java:113)
> at
> org.mortbay.jetty.servlet.ServletHandler$CachedChain.doFilter(ServletHandler.java:1212)
> at
> org.apache.hadoop.yarn.server.webproxy.amfilter.AmIpFilter.doFilter(AmIpFilter.java:123)
> at
> org.mortbay.jetty.servlet.ServletHandler$CachedChain.doFilter(ServletHandler.java:1212)
> at
> org.apache.hadoop.http.HttpServer$QuotingInputFilter.doFilter(HttpServer.java:1069)
> at
> org.mortbay.jetty.servlet.ServletHandler$CachedChain.doFilter(ServletHandler.java:1212)
> ...
> ...
> ...
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Updated] (MAPREDUCE-5137) AM web UI: clicking on Map Task results in 500 error
[ https://issues.apache.org/jira/browse/MAPREDUCE-5137?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Thomas Graves updated MAPREDUCE-5137: - Target Version/s: 0.23.7, 2.0.5-beta (was: 2.0.5-beta, 0.23.8) > AM web UI: clicking on Map Task results in 500 error > > > Key: MAPREDUCE-5137 > URL: https://issues.apache.org/jira/browse/MAPREDUCE-5137 > Project: Hadoop Map/Reduce > Issue Type: Bug > Components: applicationmaster >Affects Versions: 0.23.7, 2.0.5-beta >Reporter: Thomas Graves >Assignee: Thomas Graves > Attachments: MAPREDUCE-5137.patch > > > Go to a running mapreduce app master web UI. Click on the job, then click on > the MAP task type to bring up the list of maps, then try to click on a > particular map task. It fails with a 500 error. Note this doesn't exist in > 0.23.6. > Exception in the log looks like: > 2013-04-09 13:53:01,587 DEBUG [1088374@qtp-13877033-2 - > /mapreduce/task/task_1365457322543_0004_m_00] > org.apache.hadoop.yarn.webapp.GenericExceptionHandler: GOT EXCEPITION > com.sun.jersey.api.NotFoundException: null for uri: > http://host.com:38158/mapreduce/task/task_1365457322543_0004_m_00 > at > com.sun.jersey.server.impl.application.WebApplicationImpl._handleRequest(WebApplicationImpl.java:1470) > at > com.sun.jersey.server.impl.application.WebApplicationImpl._handleRequest(WebApplicationImpl.java:1400) > at > com.sun.jersey.server.impl.application.WebApplicationImpl.handleRequest(WebApplicationImpl.java:1349) > at > com.sun.jersey.server.impl.application.WebApplicationImpl.handleRequest(WebApplicationImpl.java:1339) > at > com.sun.jersey.spi.container.servlet.WebComponent.service(WebComponent.java:416) > at > com.sun.jersey.spi.container.servlet.ServletContainer.service(ServletContainer.java:537) > at > com.sun.jersey.spi.container.servlet.ServletContainer.doFilter(ServletContainer.java:886) > at > com.sun.jersey.spi.container.servlet.ServletContainer.doFilter(ServletContainer.java:834) > at > com.sun.jersey.spi.container.servlet.ServletContainer.doFilter(ServletContainer.java:795) > at > com.google.inject.servlet.FilterDefinition.doFilter(FilterDefinition.java:163) > at > com.google.inject.servlet.FilterChainInvocation.doFilter(FilterChainInvocation.java:58) > at > com.google.inject.servlet.ManagedFilterPipeline.dispatch(ManagedFilterPipeline.java:118) > at com.google.inject.servlet.GuiceFilter.doFilter(GuiceFilter.java:113) > at > org.mortbay.jetty.servlet.ServletHandler$CachedChain.doFilter(ServletHandler.java:1212) > at > org.apache.hadoop.yarn.server.webproxy.amfilter.AmIpFilter.doFilter(AmIpFilter.java:123) > at > org.mortbay.jetty.servlet.ServletHandler$CachedChain.doFilter(ServletHandler.java:1212) > at > org.apache.hadoop.http.HttpServer$QuotingInputFilter.doFilter(HttpServer.java:1069) > at > org.mortbay.jetty.servlet.ServletHandler$CachedChain.doFilter(ServletHandler.java:1212) > ... > ... > ... -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Updated] (MAPREDUCE-5137) AM web UI: clicking on Map Task results in 500 error
[ https://issues.apache.org/jira/browse/MAPREDUCE-5137?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Thomas Graves updated MAPREDUCE-5137: - Attachment: MAPREDUCE-5137.patch Patch tested manually since its the web UI. tested on trunk, branch-2, and branch-0.23. > AM web UI: clicking on Map Task results in 500 error > > > Key: MAPREDUCE-5137 > URL: https://issues.apache.org/jira/browse/MAPREDUCE-5137 > Project: Hadoop Map/Reduce > Issue Type: Bug > Components: applicationmaster >Affects Versions: 0.23.7, 2.0.5-beta >Reporter: Thomas Graves >Assignee: Thomas Graves > Attachments: MAPREDUCE-5137.patch > > > Go to a running mapreduce app master web UI. Click on the job, then click on > the MAP task type to bring up the list of maps, then try to click on a > particular map task. It fails with a 500 error. Note this doesn't exist in > 0.23.6. > Exception in the log looks like: > 2013-04-09 13:53:01,587 DEBUG [1088374@qtp-13877033-2 - > /mapreduce/task/task_1365457322543_0004_m_00] > org.apache.hadoop.yarn.webapp.GenericExceptionHandler: GOT EXCEPITION > com.sun.jersey.api.NotFoundException: null for uri: > http://host.com:38158/mapreduce/task/task_1365457322543_0004_m_00 > at > com.sun.jersey.server.impl.application.WebApplicationImpl._handleRequest(WebApplicationImpl.java:1470) > at > com.sun.jersey.server.impl.application.WebApplicationImpl._handleRequest(WebApplicationImpl.java:1400) > at > com.sun.jersey.server.impl.application.WebApplicationImpl.handleRequest(WebApplicationImpl.java:1349) > at > com.sun.jersey.server.impl.application.WebApplicationImpl.handleRequest(WebApplicationImpl.java:1339) > at > com.sun.jersey.spi.container.servlet.WebComponent.service(WebComponent.java:416) > at > com.sun.jersey.spi.container.servlet.ServletContainer.service(ServletContainer.java:537) > at > com.sun.jersey.spi.container.servlet.ServletContainer.doFilter(ServletContainer.java:886) > at > com.sun.jersey.spi.container.servlet.ServletContainer.doFilter(ServletContainer.java:834) > at > com.sun.jersey.spi.container.servlet.ServletContainer.doFilter(ServletContainer.java:795) > at > com.google.inject.servlet.FilterDefinition.doFilter(FilterDefinition.java:163) > at > com.google.inject.servlet.FilterChainInvocation.doFilter(FilterChainInvocation.java:58) > at > com.google.inject.servlet.ManagedFilterPipeline.dispatch(ManagedFilterPipeline.java:118) > at com.google.inject.servlet.GuiceFilter.doFilter(GuiceFilter.java:113) > at > org.mortbay.jetty.servlet.ServletHandler$CachedChain.doFilter(ServletHandler.java:1212) > at > org.apache.hadoop.yarn.server.webproxy.amfilter.AmIpFilter.doFilter(AmIpFilter.java:123) > at > org.mortbay.jetty.servlet.ServletHandler$CachedChain.doFilter(ServletHandler.java:1212) > at > org.apache.hadoop.http.HttpServer$QuotingInputFilter.doFilter(HttpServer.java:1069) > at > org.mortbay.jetty.servlet.ServletHandler$CachedChain.doFilter(ServletHandler.java:1212) > ... > ... > ... -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Updated] (MAPREDUCE-5137) AM web UI: clicking on Map Task results in 500 error
[ https://issues.apache.org/jira/browse/MAPREDUCE-5137?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Thomas Graves updated MAPREDUCE-5137: - Status: Patch Available (was: Open) > AM web UI: clicking on Map Task results in 500 error > > > Key: MAPREDUCE-5137 > URL: https://issues.apache.org/jira/browse/MAPREDUCE-5137 > Project: Hadoop Map/Reduce > Issue Type: Bug > Components: applicationmaster >Affects Versions: 0.23.7, 2.0.5-beta >Reporter: Thomas Graves >Assignee: Thomas Graves > Attachments: MAPREDUCE-5137.patch > > > Go to a running mapreduce app master web UI. Click on the job, then click on > the MAP task type to bring up the list of maps, then try to click on a > particular map task. It fails with a 500 error. Note this doesn't exist in > 0.23.6. > Exception in the log looks like: > 2013-04-09 13:53:01,587 DEBUG [1088374@qtp-13877033-2 - > /mapreduce/task/task_1365457322543_0004_m_00] > org.apache.hadoop.yarn.webapp.GenericExceptionHandler: GOT EXCEPITION > com.sun.jersey.api.NotFoundException: null for uri: > http://host.com:38158/mapreduce/task/task_1365457322543_0004_m_00 > at > com.sun.jersey.server.impl.application.WebApplicationImpl._handleRequest(WebApplicationImpl.java:1470) > at > com.sun.jersey.server.impl.application.WebApplicationImpl._handleRequest(WebApplicationImpl.java:1400) > at > com.sun.jersey.server.impl.application.WebApplicationImpl.handleRequest(WebApplicationImpl.java:1349) > at > com.sun.jersey.server.impl.application.WebApplicationImpl.handleRequest(WebApplicationImpl.java:1339) > at > com.sun.jersey.spi.container.servlet.WebComponent.service(WebComponent.java:416) > at > com.sun.jersey.spi.container.servlet.ServletContainer.service(ServletContainer.java:537) > at > com.sun.jersey.spi.container.servlet.ServletContainer.doFilter(ServletContainer.java:886) > at > com.sun.jersey.spi.container.servlet.ServletContainer.doFilter(ServletContainer.java:834) > at > com.sun.jersey.spi.container.servlet.ServletContainer.doFilter(ServletContainer.java:795) > at > com.google.inject.servlet.FilterDefinition.doFilter(FilterDefinition.java:163) > at > com.google.inject.servlet.FilterChainInvocation.doFilter(FilterChainInvocation.java:58) > at > com.google.inject.servlet.ManagedFilterPipeline.dispatch(ManagedFilterPipeline.java:118) > at com.google.inject.servlet.GuiceFilter.doFilter(GuiceFilter.java:113) > at > org.mortbay.jetty.servlet.ServletHandler$CachedChain.doFilter(ServletHandler.java:1212) > at > org.apache.hadoop.yarn.server.webproxy.amfilter.AmIpFilter.doFilter(AmIpFilter.java:123) > at > org.mortbay.jetty.servlet.ServletHandler$CachedChain.doFilter(ServletHandler.java:1212) > at > org.apache.hadoop.http.HttpServer$QuotingInputFilter.doFilter(HttpServer.java:1069) > at > org.mortbay.jetty.servlet.ServletHandler$CachedChain.doFilter(ServletHandler.java:1212) > ... > ... > ... -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Updated] (MAPREDUCE-5137) AM web UI: clicking on Map Task results in 500 error
[ https://issues.apache.org/jira/browse/MAPREDUCE-5137?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Thomas Graves updated MAPREDUCE-5137: - Assignee: Thomas Graves > AM web UI: clicking on Map Task results in 500 error > > > Key: MAPREDUCE-5137 > URL: https://issues.apache.org/jira/browse/MAPREDUCE-5137 > Project: Hadoop Map/Reduce > Issue Type: Bug > Components: applicationmaster >Affects Versions: 0.23.7, 2.0.5-beta >Reporter: Thomas Graves >Assignee: Thomas Graves > > Go to a running mapreduce app master web UI. Click on the job, then click on > the MAP task type to bring up the list of maps, then try to click on a > particular map task. It fails with a 500 error. Note this doesn't exist in > 0.23.6. > Exception in the log looks like: > 2013-04-09 13:53:01,587 DEBUG [1088374@qtp-13877033-2 - > /mapreduce/task/task_1365457322543_0004_m_00] > org.apache.hadoop.yarn.webapp.GenericExceptionHandler: GOT EXCEPITION > com.sun.jersey.api.NotFoundException: null for uri: > http://host.com:38158/mapreduce/task/task_1365457322543_0004_m_00 > at > com.sun.jersey.server.impl.application.WebApplicationImpl._handleRequest(WebApplicationImpl.java:1470) > at > com.sun.jersey.server.impl.application.WebApplicationImpl._handleRequest(WebApplicationImpl.java:1400) > at > com.sun.jersey.server.impl.application.WebApplicationImpl.handleRequest(WebApplicationImpl.java:1349) > at > com.sun.jersey.server.impl.application.WebApplicationImpl.handleRequest(WebApplicationImpl.java:1339) > at > com.sun.jersey.spi.container.servlet.WebComponent.service(WebComponent.java:416) > at > com.sun.jersey.spi.container.servlet.ServletContainer.service(ServletContainer.java:537) > at > com.sun.jersey.spi.container.servlet.ServletContainer.doFilter(ServletContainer.java:886) > at > com.sun.jersey.spi.container.servlet.ServletContainer.doFilter(ServletContainer.java:834) > at > com.sun.jersey.spi.container.servlet.ServletContainer.doFilter(ServletContainer.java:795) > at > com.google.inject.servlet.FilterDefinition.doFilter(FilterDefinition.java:163) > at > com.google.inject.servlet.FilterChainInvocation.doFilter(FilterChainInvocation.java:58) > at > com.google.inject.servlet.ManagedFilterPipeline.dispatch(ManagedFilterPipeline.java:118) > at com.google.inject.servlet.GuiceFilter.doFilter(GuiceFilter.java:113) > at > org.mortbay.jetty.servlet.ServletHandler$CachedChain.doFilter(ServletHandler.java:1212) > at > org.apache.hadoop.yarn.server.webproxy.amfilter.AmIpFilter.doFilter(AmIpFilter.java:123) > at > org.mortbay.jetty.servlet.ServletHandler$CachedChain.doFilter(ServletHandler.java:1212) > at > org.apache.hadoop.http.HttpServer$QuotingInputFilter.doFilter(HttpServer.java:1069) > at > org.mortbay.jetty.servlet.ServletHandler$CachedChain.doFilter(ServletHandler.java:1212) > ... > ... > ... -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (MAPREDUCE-5136) TestJobImpl->testJobNoTasks fails with IBM JAVA ..
[
https://issues.apache.org/jira/browse/MAPREDUCE-5136?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13627949#comment-13627949
]
Billie Rinaldi commented on MAPREDUCE-5136:
---
Yes, that's correct. Thanks for testing that!
> TestJobImpl->testJobNoTasks fails with IBM JAVA ..
> --
>
> Key: MAPREDUCE-5136
> URL: https://issues.apache.org/jira/browse/MAPREDUCE-5136
> Project: Hadoop Map/Reduce
> Issue Type: Bug
>Affects Versions: 2.0.3-alpha
> Environment: Linux + IBM JAVA 6
>Reporter: Amir Sanjar
>Assignee: Amir Sanjar
> Attachments: MAPREDUCE-5136.patch
>
>
> I am not sure if this is a testcase or a design issue. During execution of
> TestJobImpl->testJobNoTasks() there is an assertion made based on the order
> of key/value pairs stored in adjacency list. However adjacency list was
> created by Configuration->getValByRegex() as a HashMap (order is not
> guaranteed):
> Testcase:
> JobSubmittedEventHandler jseHandler = new
> JobSubmittedEventHandler("testId",
> "testName", "testNodeName", "\"key2\"=\"value2\" \"key1\"=\"value1\"
> ");
>
>
> try {
> Assert.assertTrue(jseHandler.getAssertValue()); <===
> Configuration->getValByRegex():
> public Map getValByRegex(String regex) {
> Pattern p = Pattern.compile(regex);
> Map result = new HashMap(); <===
>
> as we all know, HashMap makes absolutely no guarantees about the iteration
> order. It can (and will) even change completely when new elements are added.
> Changing HashMap to LinkedHashMap fixes the ordering inconsistency, however
> with a small performance side effect.
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (MAPREDUCE-5136) TestJobImpl->testJobNoTasks fails with IBM JAVA ..
[
https://issues.apache.org/jira/browse/MAPREDUCE-5136?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13627914#comment-13627914
]
Amir Sanjar commented on MAPREDUCE-5136:
Billi, I assumed the intended testcase in branch_1 was TestJobHistory, was that
correct? All JUnit tests, including validateJobHistoryFileContent() passed
successfully with IBM JAVA.
> TestJobImpl->testJobNoTasks fails with IBM JAVA ..
> --
>
> Key: MAPREDUCE-5136
> URL: https://issues.apache.org/jira/browse/MAPREDUCE-5136
> Project: Hadoop Map/Reduce
> Issue Type: Bug
>Affects Versions: 2.0.3-alpha
> Environment: Linux + IBM JAVA 6
>Reporter: Amir Sanjar
>Assignee: Amir Sanjar
> Attachments: MAPREDUCE-5136.patch
>
>
> I am not sure if this is a testcase or a design issue. During execution of
> TestJobImpl->testJobNoTasks() there is an assertion made based on the order
> of key/value pairs stored in adjacency list. However adjacency list was
> created by Configuration->getValByRegex() as a HashMap (order is not
> guaranteed):
> Testcase:
> JobSubmittedEventHandler jseHandler = new
> JobSubmittedEventHandler("testId",
> "testName", "testNodeName", "\"key2\"=\"value2\" \"key1\"=\"value1\"
> ");
>
>
> try {
> Assert.assertTrue(jseHandler.getAssertValue()); <===
> Configuration->getValByRegex():
> public Map getValByRegex(String regex) {
> Pattern p = Pattern.compile(regex);
> Map result = new HashMap(); <===
>
> as we all know, HashMap makes absolutely no guarantees about the iteration
> order. It can (and will) even change completely when new elements are added.
> Changing HashMap to LinkedHashMap fixes the ordering inconsistency, however
> with a small performance side effect.
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Updated] (MAPREDUCE-5136) TestJobImpl->testJobNoTasks fails with IBM JAVA ..
[
https://issues.apache.org/jira/browse/MAPREDUCE-5136?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Amir Sanjar updated MAPREDUCE-5136:
---
Summary: TestJobImpl->testJobNoTasks fails with IBM JAVA .. (was:
TestJobImpl->testJobNoTasks fails ..)
> TestJobImpl->testJobNoTasks fails with IBM JAVA ..
> --
>
> Key: MAPREDUCE-5136
> URL: https://issues.apache.org/jira/browse/MAPREDUCE-5136
> Project: Hadoop Map/Reduce
> Issue Type: Bug
>Affects Versions: 2.0.3-alpha
> Environment: Linux + IBM JAVA 6
>Reporter: Amir Sanjar
>Assignee: Amir Sanjar
> Attachments: MAPREDUCE-5136.patch
>
>
> I am not sure if this is a testcase or a design issue. During execution of
> TestJobImpl->testJobNoTasks() there is an assertion made based on the order
> of key/value pairs stored in adjacency list. However adjacency list was
> created by Configuration->getValByRegex() as a HashMap (order is not
> guaranteed):
> Testcase:
> JobSubmittedEventHandler jseHandler = new
> JobSubmittedEventHandler("testId",
> "testName", "testNodeName", "\"key2\"=\"value2\" \"key1\"=\"value1\"
> ");
>
>
> try {
> Assert.assertTrue(jseHandler.getAssertValue()); <===
> Configuration->getValByRegex():
> public Map getValByRegex(String regex) {
> Pattern p = Pattern.compile(regex);
> Map result = new HashMap(); <===
>
> as we all know, HashMap makes absolutely no guarantees about the iteration
> order. It can (and will) even change completely when new elements are added.
> Changing HashMap to LinkedHashMap fixes the ordering inconsistency, however
> with a small performance side effect.
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (MAPREDUCE-5138) Fix LocalDistributedCacheManager after YARN-112
[ https://issues.apache.org/jira/browse/MAPREDUCE-5138?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13627812#comment-13627812 ] Hudson commented on MAPREDUCE-5138: --- Integrated in Hadoop-Mapreduce-trunk #1395 (See [https://builds.apache.org/job/Hadoop-Mapreduce-trunk/1395/]) YARN-112. Fixed a race condition during localization that fails containers. Contributed by Omkar Vinit Joshi. MAPREDUCE-5138. Fix LocalDistributedCacheManager after YARN-112. Contributed by Omkar Vinit Joshi. (Revision 1466196) Result = SUCCESS vinodkv : http://svn.apache.org/viewcvs.cgi/?root=Apache-SVN&view=rev&rev=1466196 Files : * /hadoop/common/trunk/hadoop-mapreduce-project/CHANGES.txt * /hadoop/common/trunk/hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-common/src/main/java/org/apache/hadoop/mapred/LocalDistributedCacheManager.java * /hadoop/common/trunk/hadoop-yarn-project/CHANGES.txt * /hadoop/common/trunk/hadoop-yarn-project/hadoop-yarn/hadoop-yarn-common/src/main/java/org/apache/hadoop/yarn/util/FSDownload.java * /hadoop/common/trunk/hadoop-yarn-project/hadoop-yarn/hadoop-yarn-common/src/test/java/org/apache/hadoop/yarn/util/TestFSDownload.java * /hadoop/common/trunk/hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-nodemanager/src/main/java/org/apache/hadoop/yarn/server/nodemanager/containermanager/localizer/ContainerLocalizer.java * /hadoop/common/trunk/hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-nodemanager/src/main/java/org/apache/hadoop/yarn/server/nodemanager/containermanager/localizer/LocalResourcesTracker.java * /hadoop/common/trunk/hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-nodemanager/src/main/java/org/apache/hadoop/yarn/server/nodemanager/containermanager/localizer/LocalResourcesTrackerImpl.java * /hadoop/common/trunk/hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-nodemanager/src/main/java/org/apache/hadoop/yarn/server/nodemanager/containermanager/localizer/ResourceLocalizationService.java * /hadoop/common/trunk/hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-nodemanager/src/test/java/org/apache/hadoop/yarn/server/nodemanager/containermanager/localizer/TestResourceLocalizationService.java > Fix LocalDistributedCacheManager after YARN-112 > --- > > Key: MAPREDUCE-5138 > URL: https://issues.apache.org/jira/browse/MAPREDUCE-5138 > Project: Hadoop Map/Reduce > Issue Type: Bug >Reporter: Vinod Kumar Vavilapalli >Assignee: Omkar Vinit Joshi > Fix For: 2.0.5-beta > > Attachments: MAPREDUCE-5138.txt > > > LocalDistributedCacheManager uses FSDownload which is changing in YARN-112. > Need to fix it. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (MAPREDUCE-5141) Time strings are formated in different timezone
[
https://issues.apache.org/jira/browse/MAPREDUCE-5141?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13627769#comment-13627769
]
Hadoop QA commented on MAPREDUCE-5141:
--
{color:red}-1 overall{color}. Here are the results of testing the latest
attachment
http://issues.apache.org/jira/secure/attachment/12577997/MAPREDUCE-5141.patch
against trunk revision .
{color:green}+1 @author{color}. The patch does not contain any @author
tags.
{color:red}-1 tests included{color}. The patch doesn't appear to include
any new or modified tests.
Please justify why no new tests are needed for this
patch.
Also please list what manual steps were performed to
verify this patch.
{color:green}+1 javac{color}. The applied patch does not increase the
total number of javac compiler warnings.
{color:green}+1 javadoc{color}. The javadoc tool did not generate any
warning messages.
{color:green}+1 eclipse:eclipse{color}. The patch built with
eclipse:eclipse.
{color:green}+1 findbugs{color}. The patch does not introduce any new
Findbugs (version 1.3.9) warnings.
{color:green}+1 release audit{color}. The applied patch does not increase
the total number of release audit warnings.
{color:green}+1 core tests{color}. The patch passed unit tests in
hadoop-yarn-project/hadoop-yarn/hadoop-yarn-common.
{color:green}+1 contrib tests{color}. The patch passed contrib unit tests.
Test results:
https://builds.apache.org/job/PreCommit-MAPREDUCE-Build/3512//testReport/
Console output:
https://builds.apache.org/job/PreCommit-MAPREDUCE-Build/3512//console
This message is automatically generated.
> Time strings are formated in different timezone
> ---
>
> Key: MAPREDUCE-5141
> URL: https://issues.apache.org/jira/browse/MAPREDUCE-5141
> Project: Hadoop Map/Reduce
> Issue Type: Bug
>Reporter: PengZhang
> Attachments: MAPREDUCE-5141.patch
>
>
> Time strings on different page are displayed in different timezone.
> If it is rendered by renderHadoopDate() in yarn.dt.plugins.js, it appears as
> "Wed, 10 Apr 2013 08:29:56 GMT"
> If it is formatted by format() in yarn.util.Times, it appears as "10-Apr-2013
> 16:29:56"
> Same value, but different timezone.
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (MAPREDUCE-5138) Fix LocalDistributedCacheManager after YARN-112
[ https://issues.apache.org/jira/browse/MAPREDUCE-5138?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13627757#comment-13627757 ] Hudson commented on MAPREDUCE-5138: --- Integrated in Hadoop-Hdfs-trunk #1368 (See [https://builds.apache.org/job/Hadoop-Hdfs-trunk/1368/]) YARN-112. Fixed a race condition during localization that fails containers. Contributed by Omkar Vinit Joshi. MAPREDUCE-5138. Fix LocalDistributedCacheManager after YARN-112. Contributed by Omkar Vinit Joshi. (Revision 1466196) Result = FAILURE vinodkv : http://svn.apache.org/viewcvs.cgi/?root=Apache-SVN&view=rev&rev=1466196 Files : * /hadoop/common/trunk/hadoop-mapreduce-project/CHANGES.txt * /hadoop/common/trunk/hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-common/src/main/java/org/apache/hadoop/mapred/LocalDistributedCacheManager.java * /hadoop/common/trunk/hadoop-yarn-project/CHANGES.txt * /hadoop/common/trunk/hadoop-yarn-project/hadoop-yarn/hadoop-yarn-common/src/main/java/org/apache/hadoop/yarn/util/FSDownload.java * /hadoop/common/trunk/hadoop-yarn-project/hadoop-yarn/hadoop-yarn-common/src/test/java/org/apache/hadoop/yarn/util/TestFSDownload.java * /hadoop/common/trunk/hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-nodemanager/src/main/java/org/apache/hadoop/yarn/server/nodemanager/containermanager/localizer/ContainerLocalizer.java * /hadoop/common/trunk/hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-nodemanager/src/main/java/org/apache/hadoop/yarn/server/nodemanager/containermanager/localizer/LocalResourcesTracker.java * /hadoop/common/trunk/hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-nodemanager/src/main/java/org/apache/hadoop/yarn/server/nodemanager/containermanager/localizer/LocalResourcesTrackerImpl.java * /hadoop/common/trunk/hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-nodemanager/src/main/java/org/apache/hadoop/yarn/server/nodemanager/containermanager/localizer/ResourceLocalizationService.java * /hadoop/common/trunk/hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-nodemanager/src/test/java/org/apache/hadoop/yarn/server/nodemanager/containermanager/localizer/TestResourceLocalizationService.java > Fix LocalDistributedCacheManager after YARN-112 > --- > > Key: MAPREDUCE-5138 > URL: https://issues.apache.org/jira/browse/MAPREDUCE-5138 > Project: Hadoop Map/Reduce > Issue Type: Bug >Reporter: Vinod Kumar Vavilapalli >Assignee: Omkar Vinit Joshi > Fix For: 2.0.5-beta > > Attachments: MAPREDUCE-5138.txt > > > LocalDistributedCacheManager uses FSDownload which is changing in YARN-112. > Need to fix it. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Updated] (MAPREDUCE-5141) Time strings are formated in different timezone
[ https://issues.apache.org/jira/browse/MAPREDUCE-5141?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] PengZhang updated MAPREDUCE-5141: - Status: Patch Available (was: Open) > Time strings are formated in different timezone > --- > > Key: MAPREDUCE-5141 > URL: https://issues.apache.org/jira/browse/MAPREDUCE-5141 > Project: Hadoop Map/Reduce > Issue Type: Bug >Reporter: PengZhang > Attachments: MAPREDUCE-5141.patch > > > Time strings on different page are displayed in different timezone. > If it is rendered by renderHadoopDate() in yarn.dt.plugins.js, it appears as > "Wed, 10 Apr 2013 08:29:56 GMT" > If it is formatted by format() in yarn.util.Times, it appears as "10-Apr-2013 > 16:29:56" > Same value, but different timezone. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Updated] (MAPREDUCE-5141) Time strings are formated in different timezone
[ https://issues.apache.org/jira/browse/MAPREDUCE-5141?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] PengZhang updated MAPREDUCE-5141: - Attachment: MAPREDUCE-5141.patch simple patch, change js render to locale time > Time strings are formated in different timezone > --- > > Key: MAPREDUCE-5141 > URL: https://issues.apache.org/jira/browse/MAPREDUCE-5141 > Project: Hadoop Map/Reduce > Issue Type: Bug >Reporter: PengZhang > Attachments: MAPREDUCE-5141.patch > > > Time strings on different page are displayed in different timezone. > If it is rendered by renderHadoopDate() in yarn.dt.plugins.js, it appears as > "Wed, 10 Apr 2013 08:29:56 GMT" > If it is formatted by format() in yarn.util.Times, it appears as "10-Apr-2013 > 16:29:56" > Same value, but different timezone. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Created] (MAPREDUCE-5141) Time strings are formated in different timezone
PengZhang created MAPREDUCE-5141: Summary: Time strings are formated in different timezone Key: MAPREDUCE-5141 URL: https://issues.apache.org/jira/browse/MAPREDUCE-5141 Project: Hadoop Map/Reduce Issue Type: Bug Reporter: PengZhang Time strings on different page are displayed in different timezone. If it is rendered by renderHadoopDate() in yarn.dt.plugins.js, it appears as "Wed, 10 Apr 2013 08:29:56 GMT" If it is formatted by format() in yarn.util.Times, it appears as "10-Apr-2013 16:29:56" Same value, but different timezone. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (MAPREDUCE-5138) Fix LocalDistributedCacheManager after YARN-112
[ https://issues.apache.org/jira/browse/MAPREDUCE-5138?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13627668#comment-13627668 ] Hudson commented on MAPREDUCE-5138: --- Integrated in Hadoop-Yarn-trunk #179 (See [https://builds.apache.org/job/Hadoop-Yarn-trunk/179/]) YARN-112. Fixed a race condition during localization that fails containers. Contributed by Omkar Vinit Joshi. MAPREDUCE-5138. Fix LocalDistributedCacheManager after YARN-112. Contributed by Omkar Vinit Joshi. (Revision 1466196) Result = SUCCESS vinodkv : http://svn.apache.org/viewcvs.cgi/?root=Apache-SVN&view=rev&rev=1466196 Files : * /hadoop/common/trunk/hadoop-mapreduce-project/CHANGES.txt * /hadoop/common/trunk/hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-common/src/main/java/org/apache/hadoop/mapred/LocalDistributedCacheManager.java * /hadoop/common/trunk/hadoop-yarn-project/CHANGES.txt * /hadoop/common/trunk/hadoop-yarn-project/hadoop-yarn/hadoop-yarn-common/src/main/java/org/apache/hadoop/yarn/util/FSDownload.java * /hadoop/common/trunk/hadoop-yarn-project/hadoop-yarn/hadoop-yarn-common/src/test/java/org/apache/hadoop/yarn/util/TestFSDownload.java * /hadoop/common/trunk/hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-nodemanager/src/main/java/org/apache/hadoop/yarn/server/nodemanager/containermanager/localizer/ContainerLocalizer.java * /hadoop/common/trunk/hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-nodemanager/src/main/java/org/apache/hadoop/yarn/server/nodemanager/containermanager/localizer/LocalResourcesTracker.java * /hadoop/common/trunk/hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-nodemanager/src/main/java/org/apache/hadoop/yarn/server/nodemanager/containermanager/localizer/LocalResourcesTrackerImpl.java * /hadoop/common/trunk/hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-nodemanager/src/main/java/org/apache/hadoop/yarn/server/nodemanager/containermanager/localizer/ResourceLocalizationService.java * /hadoop/common/trunk/hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-nodemanager/src/test/java/org/apache/hadoop/yarn/server/nodemanager/containermanager/localizer/TestResourceLocalizationService.java > Fix LocalDistributedCacheManager after YARN-112 > --- > > Key: MAPREDUCE-5138 > URL: https://issues.apache.org/jira/browse/MAPREDUCE-5138 > Project: Hadoop Map/Reduce > Issue Type: Bug >Reporter: Vinod Kumar Vavilapalli >Assignee: Omkar Vinit Joshi > Fix For: 2.0.5-beta > > Attachments: MAPREDUCE-5138.txt > > > LocalDistributedCacheManager uses FSDownload which is changing in YARN-112. > Need to fix it. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators For more information on JIRA, see: http://www.atlassian.com/software/jira
