[jira] [Commented] (MAPREDUCE-6096) SummarizedJob class NPEs with some jhist files
[
https://issues.apache.org/jira/browse/MAPREDUCE-6096?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14144353#comment-14144353
]
Hadoop QA commented on MAPREDUCE-6096:
--
{color:green}+1 overall{color}. Here are the results of testing the latest
attachment
http://issues.apache.org/jira/secure/attachment/12670596/MAPREDUCE-6096-v2.patch
against trunk revision 7b8df93.
{color:green}+1 @author{color}. The patch does not contain any @author
tags.
{color:green}+1 tests included{color}. The patch appears to include 1 new
or modified test files.
{color:green}+1 javac{color}. The applied patch does not increase the
total number of javac compiler warnings.
{color:green}+1 javadoc{color}. There were no new javadoc warning messages.
{color:green}+1 eclipse:eclipse{color}. The patch built with
eclipse:eclipse.
{color:green}+1 findbugs{color}. The patch does not introduce any new
Findbugs (version 2.0.3) warnings.
{color:green}+1 release audit{color}. The applied patch does not increase
the total number of release audit warnings.
{color:green}+1 core tests{color}. The patch passed unit tests in
hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-core.
{color:green}+1 contrib tests{color}. The patch passed contrib unit tests.
Test results:
https://builds.apache.org/job/PreCommit-MAPREDUCE-Build/4908//testReport/
Console output:
https://builds.apache.org/job/PreCommit-MAPREDUCE-Build/4908//console
This message is automatically generated.
> SummarizedJob class NPEs with some jhist files
> --
>
> Key: MAPREDUCE-6096
> URL: https://issues.apache.org/jira/browse/MAPREDUCE-6096
> Project: Hadoop Map/Reduce
> Issue Type: Bug
> Components: jobhistoryserver
>Affects Versions: trunk
>Reporter: zhangyubiao
>Assignee: zhangyubiao
> Labels: easyfix, patch
> Attachments: MAPREDUCE-6096-v2.patch, MAPREDUCE-6096.patch
>
>
> When I Parse the JobHistory in the HistoryFile,I use the Hadoop System's
> map-reduce-client-core project
> org.apache.hadoop.mapreduce.jobhistory.JobHistoryParser class and
> HistoryViewer$SummarizedJob to Parse the JobHistoryFile(Just Like
> job_1408862281971_489761-1410883171851_XXX.jhist)
> and it throw an Exception Just Like
> Exception in thread "pool-1-thread-1" java.lang.NullPointerException
> at
> org.apache.hadoop.mapreduce.jobhistory.HistoryViewer$SummarizedJob.(HistoryViewer.java:626)
> at
> com.jd.hadoop.log.parse.ParseLogService.getJobDetail(ParseLogService.java:70)
> After I'm see the SummarizedJob class I find that attempt.getTaskStatus() is
> NULL ， So I change the order of
> attempt.getTaskStatus().equals (TaskStatus.State.FAILED.toString()) to
> TaskStatus.State.FAILED.toString().equals(attempt.getTaskStatus())
> and it works well .
> So I wonder If we can change all attempt.getTaskStatus() after
> TaskStatus.State.XXX.toString() ?
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Updated] (MAPREDUCE-6096) SummarizedJob class NPEs with some jhist files
[ https://issues.apache.org/jira/browse/MAPREDUCE-6096?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] zhangyubiao updated MAPREDUCE-6096: --- Hadoop Flags: Reviewed Status: Patch Available (was: Open) > SummarizedJob class NPEs with some jhist files > -- > > Key: MAPREDUCE-6096 > URL: https://issues.apache.org/jira/browse/MAPREDUCE-6096 > Project: Hadoop Map/Reduce > Issue Type: Bug > Components: jobhistoryserver >Affects Versions: trunk >Reporter: zhangyubiao >Assignee: zhangyubiao > Labels: easyfix, patch > Attachments: MAPREDUCE-6096-v2.patch, MAPREDUCE-6096.patch > > > When I Parse the JobHistory in the HistoryFile,I use the Hadoop System's > map-reduce-client-core project > org.apache.hadoop.mapreduce.jobhistory.JobHistoryParser class and > HistoryViewer$SummarizedJob to Parse the JobHistoryFile(Just Like > job_1408862281971_489761-1410883171851_XXX.jhist) > and it throw an Exception Just Like > Exception in thread "pool-1-thread-1" java.lang.NullPointerException > at > org.apache.hadoop.mapreduce.jobhistory.HistoryViewer$SummarizedJob.(HistoryViewer.java:626) > at > com.jd.hadoop.log.parse.ParseLogService.getJobDetail(ParseLogService.java:70) > After I'm see the SummarizedJob class I find that attempt.getTaskStatus() is > NULL ， So I change the order of > attempt.getTaskStatus().equals (TaskStatus.State.FAILED.toString()) to > TaskStatus.State.FAILED.toString().equals(attempt.getTaskStatus()) > and it works well . > So I wonder If we can change all attempt.getTaskStatus() after > TaskStatus.State.XXX.toString() ? -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (MAPREDUCE-6096) SummarizedJob class NPEs with some jhist files
[ https://issues.apache.org/jira/browse/MAPREDUCE-6096?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14144283#comment-14144283 ] zhangyubiao commented on MAPREDUCE-6096: MAPREDUCE-6096-v2.patch for review > SummarizedJob class NPEs with some jhist files > -- > > Key: MAPREDUCE-6096 > URL: https://issues.apache.org/jira/browse/MAPREDUCE-6096 > Project: Hadoop Map/Reduce > Issue Type: Bug > Components: jobhistoryserver >Affects Versions: trunk >Reporter: zhangyubiao >Assignee: zhangyubiao > Labels: easyfix, patch > Attachments: MAPREDUCE-6096-v2.patch, MAPREDUCE-6096.patch > > > When I Parse the JobHistory in the HistoryFile,I use the Hadoop System's > map-reduce-client-core project > org.apache.hadoop.mapreduce.jobhistory.JobHistoryParser class and > HistoryViewer$SummarizedJob to Parse the JobHistoryFile(Just Like > job_1408862281971_489761-1410883171851_XXX.jhist) > and it throw an Exception Just Like > Exception in thread "pool-1-thread-1" java.lang.NullPointerException > at > org.apache.hadoop.mapreduce.jobhistory.HistoryViewer$SummarizedJob.(HistoryViewer.java:626) > at > com.jd.hadoop.log.parse.ParseLogService.getJobDetail(ParseLogService.java:70) > After I'm see the SummarizedJob class I find that attempt.getTaskStatus() is > NULL ， So I change the order of > attempt.getTaskStatus().equals (TaskStatus.State.FAILED.toString()) to > TaskStatus.State.FAILED.toString().equals(attempt.getTaskStatus()) > and it works well . > So I wonder If we can change all attempt.getTaskStatus() after > TaskStatus.State.XXX.toString() ? -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Updated] (MAPREDUCE-6096) SummarizedJob class NPEs with some jhist files
[ https://issues.apache.org/jira/browse/MAPREDUCE-6096?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] zhangyubiao updated MAPREDUCE-6096: --- Attachment: MAPREDUCE-6096-v2.patch > SummarizedJob class NPEs with some jhist files > -- > > Key: MAPREDUCE-6096 > URL: https://issues.apache.org/jira/browse/MAPREDUCE-6096 > Project: Hadoop Map/Reduce > Issue Type: Bug > Components: jobhistoryserver >Affects Versions: trunk >Reporter: zhangyubiao >Assignee: zhangyubiao > Labels: easyfix, patch > Attachments: MAPREDUCE-6096-v2.patch, MAPREDUCE-6096.patch > > > When I Parse the JobHistory in the HistoryFile,I use the Hadoop System's > map-reduce-client-core project > org.apache.hadoop.mapreduce.jobhistory.JobHistoryParser class and > HistoryViewer$SummarizedJob to Parse the JobHistoryFile(Just Like > job_1408862281971_489761-1410883171851_XXX.jhist) > and it throw an Exception Just Like > Exception in thread "pool-1-thread-1" java.lang.NullPointerException > at > org.apache.hadoop.mapreduce.jobhistory.HistoryViewer$SummarizedJob.(HistoryViewer.java:626) > at > com.jd.hadoop.log.parse.ParseLogService.getJobDetail(ParseLogService.java:70) > After I'm see the SummarizedJob class I find that attempt.getTaskStatus() is > NULL ， So I change the order of > attempt.getTaskStatus().equals (TaskStatus.State.FAILED.toString()) to > TaskStatus.State.FAILED.toString().equals(attempt.getTaskStatus()) > and it works well . > So I wonder If we can change all attempt.getTaskStatus() after > TaskStatus.State.XXX.toString() ? -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Updated] (MAPREDUCE-6096) SummarizedJob class NPEs with some jhist files
[ https://issues.apache.org/jira/browse/MAPREDUCE-6096?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] zhangyubiao updated MAPREDUCE-6096: --- Issue Type: Bug (was: Improvement) > SummarizedJob class NPEs with some jhist files > -- > > Key: MAPREDUCE-6096 > URL: https://issues.apache.org/jira/browse/MAPREDUCE-6096 > Project: Hadoop Map/Reduce > Issue Type: Bug > Components: jobhistoryserver >Affects Versions: trunk >Reporter: zhangyubiao >Assignee: zhangyubiao > Labels: easyfix, patch > Attachments: MAPREDUCE-6096.patch > > > When I Parse the JobHistory in the HistoryFile,I use the Hadoop System's > map-reduce-client-core project > org.apache.hadoop.mapreduce.jobhistory.JobHistoryParser class and > HistoryViewer$SummarizedJob to Parse the JobHistoryFile(Just Like > job_1408862281971_489761-1410883171851_XXX.jhist) > and it throw an Exception Just Like > Exception in thread "pool-1-thread-1" java.lang.NullPointerException > at > org.apache.hadoop.mapreduce.jobhistory.HistoryViewer$SummarizedJob.(HistoryViewer.java:626) > at > com.jd.hadoop.log.parse.ParseLogService.getJobDetail(ParseLogService.java:70) > After I'm see the SummarizedJob class I find that attempt.getTaskStatus() is > NULL ， So I change the order of > attempt.getTaskStatus().equals (TaskStatus.State.FAILED.toString()) to > TaskStatus.State.FAILED.toString().equals(attempt.getTaskStatus()) > and it works well . > So I wonder If we can change all attempt.getTaskStatus() after > TaskStatus.State.XXX.toString() ? -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (MAPREDUCE-6096) SummarizedJob class NPEs with some jhist files
[ https://issues.apache.org/jira/browse/MAPREDUCE-6096?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14144257#comment-14144257 ] zhangyubiao commented on MAPREDUCE-6096: Path jobFile = new Path(jobHistoryFilePath); > SummarizedJob class NPEs with some jhist files > -- > > Key: MAPREDUCE-6096 > URL: https://issues.apache.org/jira/browse/MAPREDUCE-6096 > Project: Hadoop Map/Reduce > Issue Type: Improvement > Components: jobhistoryserver >Affects Versions: trunk >Reporter: zhangyubiao >Assignee: zhangyubiao > Labels: easyfix, patch > Attachments: MAPREDUCE-6096.patch > > > When I Parse the JobHistory in the HistoryFile,I use the Hadoop System's > map-reduce-client-core project > org.apache.hadoop.mapreduce.jobhistory.JobHistoryParser class and > HistoryViewer$SummarizedJob to Parse the JobHistoryFile(Just Like > job_1408862281971_489761-1410883171851_XXX.jhist) > and it throw an Exception Just Like > Exception in thread "pool-1-thread-1" java.lang.NullPointerException > at > org.apache.hadoop.mapreduce.jobhistory.HistoryViewer$SummarizedJob.(HistoryViewer.java:626) > at > com.jd.hadoop.log.parse.ParseLogService.getJobDetail(ParseLogService.java:70) > After I'm see the SummarizedJob class I find that attempt.getTaskStatus() is > NULL ， So I change the order of > attempt.getTaskStatus().equals (TaskStatus.State.FAILED.toString()) to > TaskStatus.State.FAILED.toString().equals(attempt.getTaskStatus()) > and it works well . > So I wonder If we can change all attempt.getTaskStatus() after > TaskStatus.State.XXX.toString() ? -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (MAPREDUCE-6096) SummarizedJob class NPEs with some jhist files
[
https://issues.apache.org/jira/browse/MAPREDUCE-6096?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14144248#comment-14144248
]
zhangyubiao commented on MAPREDUCE-6096:
Thank you very much for review,It's my first time to Commit a patch.
No new tests is because this test need jhist File and the exception file is
large size. You can tests this patch as follow
1、get jhist File done into Local File
2、Test use this method By use jhist File Path into jobHistoryFilePath
public static SummarizedJob getSummarizedJob(String jobHistoryFilePath) {
JobInfo jobInfo = null;
SummarizedJob ts = null;
Path jobFile = new Path(jobHistoryFile);
try {
Configuration conf = new Configuration();
FileSystem fs = jobFile.getFileSystem(conf);
JobHistoryParser parser = new JobHistoryParser(fs,
jobHistoryFile);
jobInfo = parser.parse();
ts = new SummarizedJob(jobInfo);
} catch (IOException e) {
e.printStackTrace();
}
return ts;
}
3、if you use the current SummarizedJob ,It will throw NullPointerException in
some jhist File. If you change the the order of
attempt.getTaskStatus().equals (TaskStatus.State.FAILED.toString()) ,
It works well.
> SummarizedJob class NPEs with some jhist files
> --
>
> Key: MAPREDUCE-6096
> URL: https://issues.apache.org/jira/browse/MAPREDUCE-6096
> Project: Hadoop Map/Reduce
> Issue Type: Improvement
> Components: jobhistoryserver
>Affects Versions: trunk
>Reporter: zhangyubiao
>Assignee: zhangyubiao
> Labels: easyfix, patch
> Attachments: MAPREDUCE-6096.patch
>
>
> When I Parse the JobHistory in the HistoryFile,I use the Hadoop System's
> map-reduce-client-core project
> org.apache.hadoop.mapreduce.jobhistory.JobHistoryParser class and
> HistoryViewer$SummarizedJob to Parse the JobHistoryFile(Just Like
> job_1408862281971_489761-1410883171851_XXX.jhist)
> and it throw an Exception Just Like
> Exception in thread "pool-1-thread-1" java.lang.NullPointerException
> at
> org.apache.hadoop.mapreduce.jobhistory.HistoryViewer$SummarizedJob.(HistoryViewer.java:626)
> at
> com.jd.hadoop.log.parse.ParseLogService.getJobDetail(ParseLogService.java:70)
> After I'm see the SummarizedJob class I find that attempt.getTaskStatus() is
> NULL ， So I change the order of
> attempt.getTaskStatus().equals (TaskStatus.State.FAILED.toString()) to
> TaskStatus.State.FAILED.toString().equals(attempt.getTaskStatus())
> and it works well .
> So I wonder If we can change all attempt.getTaskStatus() after
> TaskStatus.State.XXX.toString() ?
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Commented] (MAPREDUCE-6103) Adding reservation APIs to resource manager delegate
[ https://issues.apache.org/jira/browse/MAPREDUCE-6103?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14144171#comment-14144171 ] Chris Douglas commented on MAPREDUCE-6103: -- +1 straightforward update to the branch > Adding reservation APIs to resource manager delegate > > > Key: MAPREDUCE-6103 > URL: https://issues.apache.org/jira/browse/MAPREDUCE-6103 > Project: Hadoop Map/Reduce > Issue Type: Improvement >Reporter: Subru Krishnan >Assignee: Subru Krishnan > Attachments: MR-6103.patch > > > YARN-1051 introduces the ReservationSystem and the corresponding APIs for > create/update/delete ops. The MR resource manager delegate needs to to be > updated with the APIs. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (MAPREDUCE-6103) Adding reservation APIs to resource manager delegate
[
https://issues.apache.org/jira/browse/MAPREDUCE-6103?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14144027#comment-14144027
]
Hadoop QA commented on MAPREDUCE-6103:
--
{color:red}-1 overall{color}. Here are the results of testing the latest
attachment
http://issues.apache.org/jira/secure/attachment/12670531/MR-6103.patch
against trunk revision 43efdd3.
{color:green}+1 @author{color}. The patch does not contain any @author
tags.
{color:green}+1 tests included{color}. The patch appears to include 1 new
or modified test files.
{color:red}-1 javac{color:red}. The patch appears to cause the build to
fail.
Console output:
https://builds.apache.org/job/PreCommit-MAPREDUCE-Build/4907//console
This message is automatically generated.
> Adding reservation APIs to resource manager delegate
>
>
> Key: MAPREDUCE-6103
> URL: https://issues.apache.org/jira/browse/MAPREDUCE-6103
> Project: Hadoop Map/Reduce
> Issue Type: Improvement
>Reporter: Subru Krishnan
>Assignee: Subru Krishnan
> Attachments: MR-6103.patch
>
>
> YARN-1051 introduces the ReservationSystem and the corresponding APIs for
> create/update/delete ops. The MR resource manager delegate needs to to be
> updated with the APIs.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Updated] (MAPREDUCE-6103) Adding reservation APIs to resource manager delegate
[ https://issues.apache.org/jira/browse/MAPREDUCE-6103?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Subru Krishnan updated MAPREDUCE-6103: -- Status: Patch Available (was: Open) The idea is to commit the patch to branch yarn-1051 and then merge to trunk as part of YARN-1051. > Adding reservation APIs to resource manager delegate > > > Key: MAPREDUCE-6103 > URL: https://issues.apache.org/jira/browse/MAPREDUCE-6103 > Project: Hadoop Map/Reduce > Issue Type: Improvement >Reporter: Subru Krishnan >Assignee: Subru Krishnan > Attachments: MR-6103.patch > > > YARN-1051 introduces the ReservationSystem and the corresponding APIs for > create/update/delete ops. The MR resource manager delegate needs to to be > updated with the APIs. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Updated] (MAPREDUCE-6103) Adding reservation APIs to resource manager delegate
[ https://issues.apache.org/jira/browse/MAPREDUCE-6103?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Subru Krishnan updated MAPREDUCE-6103: -- Attachment: MR-6103.patch This patch is baslined againts branch yarn-1051. > Adding reservation APIs to resource manager delegate > > > Key: MAPREDUCE-6103 > URL: https://issues.apache.org/jira/browse/MAPREDUCE-6103 > Project: Hadoop Map/Reduce > Issue Type: Improvement >Reporter: Subru Krishnan >Assignee: Subru Krishnan > Attachments: MR-6103.patch > > > YARN-1051 introduces the ReservationSystem and the corresponding APIs for > create/update/delete ops. The MR resource manager delegate needs to to be > updated with the APIs. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (MAPREDUCE-5933) Enable MR AM to post history events to the timeline server
[ https://issues.apache.org/jira/browse/MAPREDUCE-5933?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14143949#comment-14143949 ] Robert Kanter commented on MAPREDUCE-5933: -- Thanks for taking a look [~zjshen]. Here are my responses: 1. TimelineClient should do nothing in the case where the ATS is not enabled. Or did you mean that users may want to disable posting to the ATS for a specific job, even when the ATS is enabled? 2. I think we should revisit this after YARN-2517 is done. No reason to duplicate work here. 3. The current behavior is to throw a YarnRuntimeException when there's a problem processing the events for the JHS, so I was just following that for the TimelineClient. Unless you think we should do it differently for the TimelineClient and log a warning instead? 4. I think we want to publish each event as it comes in. This should make the Job available in the ATS sooner than it is in the JHS. And in the case of a failure where the ATS has partial information, the ATS should be able to handle partial job info (obviously it probably can't display every piece of information about a job and/or some info might be inaccurate, but at least there's a record of the job); the JHS doesn't even know about many of the jobs that fail part way through. 5. The countersToJSON method already turns the Counters into JSON so the ATS can display them. 6. As before, I was trying to mirror what has been done for the JHS. I'm publishing to the ATS the same events and information that the JHS gets. The JHS doesn't get the TaskFailedEvent, so I didn't do it here. Should publish every type of event? > Enable MR AM to post history events to the timeline server > -- > > Key: MAPREDUCE-5933 > URL: https://issues.apache.org/jira/browse/MAPREDUCE-5933 > Project: Hadoop Map/Reduce > Issue Type: Sub-task > Components: mr-am >Reporter: Zhijie Shen >Assignee: Robert Kanter > Attachments: MAPREDUCE-5933.patch, MAPREDUCE-5933.patch, > MAPREDUCE-5933.patch, mr_timelineserver_response.txt > > > Nowadays, MR AM collects the history events and writes it to HDFS for JHS to > source. With the timeline server, MR AM can put these events there. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Created] (MAPREDUCE-6103) Adding reservation APIs to resource manager delegate
Subru Krishnan created MAPREDUCE-6103: - Summary: Adding reservation APIs to resource manager delegate Key: MAPREDUCE-6103 URL: https://issues.apache.org/jira/browse/MAPREDUCE-6103 Project: Hadoop Map/Reduce Issue Type: Improvement Reporter: Subru Krishnan Assignee: Subru Krishnan YARN-1051 introduces the ReservationSystem and the corresponding APIs for create/update/delete ops. The MR resource manager delegate needs to to be updated with the APIs. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Updated] (MAPREDUCE-5848) MapReduce counts forcibly preempted containers as FAILED
[ https://issues.apache.org/jira/browse/MAPREDUCE-5848?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Subru Krishnan updated MAPREDUCE-5848: -- Resolution: Duplicate Status: Resolved (was: Patch Available) Marking it as fixed as it is subsumed by the later MAPREDUCE-5900. > MapReduce counts forcibly preempted containers as FAILED > > > Key: MAPREDUCE-5848 > URL: https://issues.apache.org/jira/browse/MAPREDUCE-5848 > Project: Hadoop Map/Reduce > Issue Type: Sub-task >Affects Versions: 2.1.0-beta >Reporter: Carlo Curino >Assignee: Subru Krishnan > Attachments: MR-5848.patch, MR-5848.patch, YARN-1958.patch > > > The MapReduce AM is considering a forcibly preempted container as FAILED, > while I think it should be considered as KILLED (i.e., not count against the > maximum number of failures). -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (MAPREDUCE-6096) SummarizedJob class NPEs with some jhist files
[ https://issues.apache.org/jira/browse/MAPREDUCE-6096?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14143805#comment-14143805 ] Allen Wittenauer commented on MAPREDUCE-6096: - This should probably have a test associated with it. I don't think one exists. > SummarizedJob class NPEs with some jhist files > -- > > Key: MAPREDUCE-6096 > URL: https://issues.apache.org/jira/browse/MAPREDUCE-6096 > Project: Hadoop Map/Reduce > Issue Type: Improvement > Components: jobhistoryserver >Affects Versions: trunk >Reporter: zhangyubiao >Assignee: zhangyubiao > Labels: easyfix, patch > Attachments: MAPREDUCE-6096.patch > > > When I Parse the JobHistory in the HistoryFile,I use the Hadoop System's > map-reduce-client-core project > org.apache.hadoop.mapreduce.jobhistory.JobHistoryParser class and > HistoryViewer$SummarizedJob to Parse the JobHistoryFile(Just Like > job_1408862281971_489761-1410883171851_XXX.jhist) > and it throw an Exception Just Like > Exception in thread "pool-1-thread-1" java.lang.NullPointerException > at > org.apache.hadoop.mapreduce.jobhistory.HistoryViewer$SummarizedJob.(HistoryViewer.java:626) > at > com.jd.hadoop.log.parse.ParseLogService.getJobDetail(ParseLogService.java:70) > After I'm see the SummarizedJob class I find that attempt.getTaskStatus() is > NULL ， So I change the order of > attempt.getTaskStatus().equals (TaskStatus.State.FAILED.toString()) to > TaskStatus.State.FAILED.toString().equals(attempt.getTaskStatus()) > and it works well . > So I wonder If we can change all attempt.getTaskStatus() after > TaskStatus.State.XXX.toString() ? -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Updated] (MAPREDUCE-6096) SummarizedJob class NPEs with some jhist files
[ https://issues.apache.org/jira/browse/MAPREDUCE-6096?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Allen Wittenauer updated MAPREDUCE-6096: Summary: SummarizedJob class NPEs with some jhist files (was: SummarizedJob class should be Improvment ) > SummarizedJob class NPEs with some jhist files > -- > > Key: MAPREDUCE-6096 > URL: https://issues.apache.org/jira/browse/MAPREDUCE-6096 > Project: Hadoop Map/Reduce > Issue Type: Improvement > Components: jobhistoryserver >Affects Versions: trunk >Reporter: zhangyubiao >Assignee: zhangyubiao > Labels: easyfix, patch > Attachments: MAPREDUCE-6096.patch > > > When I Parse the JobHistory in the HistoryFile,I use the Hadoop System's > map-reduce-client-core project > org.apache.hadoop.mapreduce.jobhistory.JobHistoryParser class and > HistoryViewer$SummarizedJob to Parse the JobHistoryFile(Just Like > job_1408862281971_489761-1410883171851_XXX.jhist) > and it throw an Exception Just Like > Exception in thread "pool-1-thread-1" java.lang.NullPointerException > at > org.apache.hadoop.mapreduce.jobhistory.HistoryViewer$SummarizedJob.(HistoryViewer.java:626) > at > com.jd.hadoop.log.parse.ParseLogService.getJobDetail(ParseLogService.java:70) > After I'm see the SummarizedJob class I find that attempt.getTaskStatus() is > NULL ， So I change the order of > attempt.getTaskStatus().equals (TaskStatus.State.FAILED.toString()) to > TaskStatus.State.FAILED.toString().equals(attempt.getTaskStatus()) > and it works well . > So I wonder If we can change all attempt.getTaskStatus() after > TaskStatus.State.XXX.toString() ? -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (MAPREDUCE-6095) Enable DistributedCache for uber-mode Jobs
[ https://issues.apache.org/jira/browse/MAPREDUCE-6095?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14143604#comment-14143604 ] Gera Shegalov commented on MAPREDUCE-6095: -- Thanks for review, Jason! > Enable DistributedCache for uber-mode Jobs > -- > > Key: MAPREDUCE-6095 > URL: https://issues.apache.org/jira/browse/MAPREDUCE-6095 > Project: Hadoop Map/Reduce > Issue Type: Bug > Components: applicationmaster, distributed-cache >Affects Versions: 2.0.5-alpha >Reporter: Gera Shegalov >Assignee: Gera Shegalov > Fix For: 2.6.0 > > Attachments: MAPREDUCE-6095.v01.patch, MAPREDUCE-6095.v02.patch > > > mapreduce.job.cache.local.* is not set for uber-mode jobs. TestUberAM hides > the fact that DC files are not available for uber-mode jobs by overriding > testDistributedCache with a nop method. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (MAPREDUCE-6087) MRJobConfig#MR_CLIENT_TO_AM_IPC_MAX_RETRIES_ON_TIMEOUTS config name is wrong
[
https://issues.apache.org/jira/browse/MAPREDUCE-6087?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14143583#comment-14143583
]
Jian He commented on MAPREDUCE-6087:
mapred-default.xml has the correct name, which is good.
{code}
yarn.app.mapreduce.client-am.ipc.max-retries-on-timeouts
3
{code}
Thanks [~ajisakaa] for working on the issue !
> MRJobConfig#MR_CLIENT_TO_AM_IPC_MAX_RETRIES_ON_TIMEOUTS config name is wrong
>
>
> Key: MAPREDUCE-6087
> URL: https://issues.apache.org/jira/browse/MAPREDUCE-6087
> Project: Hadoop Map/Reduce
> Issue Type: Bug
>Affects Versions: 2.3.0
>Reporter: Jian He
>Assignee: Akira AJISAKA
> Labels: newbie
> Attachments: MAPREDUCE-6087.2.patch, MAPREDUCE-6087.patch
>
>
> The config name for MRJobConfig#MR_CLIENT_TO_AM_IPC_MAX_RETRIES_ON_TIMEOUTS
> now has double prefix as "yarn.app.mapreduce." +
> "yarn.app.mapreduce.client-am.ipc.max-retries-on-timeouts"
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Updated] (MAPREDUCE-5279) Jobs can deadlock if headroom is limited by cpu instead of memory
[ https://issues.apache.org/jira/browse/MAPREDUCE-5279?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Zhijie Shen updated MAPREDUCE-5279: --- Resolution: Fixed Fix Version/s: 2.6.0 Hadoop Flags: Reviewed Status: Resolved (was: Patch Available) > Jobs can deadlock if headroom is limited by cpu instead of memory > - > > Key: MAPREDUCE-5279 > URL: https://issues.apache.org/jira/browse/MAPREDUCE-5279 > Project: Hadoop Map/Reduce > Issue Type: Bug > Components: mrv2, scheduler >Affects Versions: 2.0.3-alpha >Reporter: Peng Zhang >Assignee: Peng Zhang >Priority: Critical > Fix For: 2.6.0 > > Attachments: MAPREDUCE-5279-branch-2.patch, MAPREDUCE-5279-v2.patch, > MAPREDUCE-5279.patch, apache-mapreduce-5279.3.patch, > apache-mapreduce-5279.4.patch, apache-mapreduce-5279.5.patch, > apache-mapreduce-5279.6.patch, apache-mapreduce-5279.7.patch > > > YARN-2 imported cpu dimension scheduling, but MR RMContainerAllocator doesn't > take into account virtual cores while scheduling reduce tasks. > This may cause more reduce tasks to be scheduled because memory is enough. > And on a small cluster, this will end with deadlock, all running containers > are reduce tasks but map phase is not finished. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (MAPREDUCE-5279) Jobs can deadlock if headroom is limited by cpu instead of memory
[ https://issues.apache.org/jira/browse/MAPREDUCE-5279?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14143516#comment-14143516 ] Zhijie Shen commented on MAPREDUCE-5279: Commit the patch to trunk and branch-2. Thanks, [~peng.zhang] and [~vvasudev]! > Jobs can deadlock if headroom is limited by cpu instead of memory > - > > Key: MAPREDUCE-5279 > URL: https://issues.apache.org/jira/browse/MAPREDUCE-5279 > Project: Hadoop Map/Reduce > Issue Type: Bug > Components: mrv2, scheduler >Affects Versions: 2.0.3-alpha >Reporter: Peng Zhang >Assignee: Peng Zhang >Priority: Critical > Fix For: 2.6.0 > > Attachments: MAPREDUCE-5279-branch-2.patch, MAPREDUCE-5279-v2.patch, > MAPREDUCE-5279.patch, apache-mapreduce-5279.3.patch, > apache-mapreduce-5279.4.patch, apache-mapreduce-5279.5.patch, > apache-mapreduce-5279.6.patch, apache-mapreduce-5279.7.patch > > > YARN-2 imported cpu dimension scheduling, but MR RMContainerAllocator doesn't > take into account virtual cores while scheduling reduce tasks. > This may cause more reduce tasks to be scheduled because memory is enough. > And on a small cluster, this will end with deadlock, all running containers > are reduce tasks but map phase is not finished. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Updated] (MAPREDUCE-6096) SummarizedJob class should be Improvment
[ https://issues.apache.org/jira/browse/MAPREDUCE-6096?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Allen Wittenauer updated MAPREDUCE-6096: Hadoop Flags: (was: Reviewed) > SummarizedJob class should be Improvment > - > > Key: MAPREDUCE-6096 > URL: https://issues.apache.org/jira/browse/MAPREDUCE-6096 > Project: Hadoop Map/Reduce > Issue Type: Improvement > Components: jobhistoryserver >Affects Versions: trunk >Reporter: zhangyubiao >Assignee: zhangyubiao > Labels: easyfix, patch > Attachments: MAPREDUCE-6096.patch > > > When I Parse the JobHistory in the HistoryFile,I use the Hadoop System's > map-reduce-client-core project > org.apache.hadoop.mapreduce.jobhistory.JobHistoryParser class and > HistoryViewer$SummarizedJob to Parse the JobHistoryFile(Just Like > job_1408862281971_489761-1410883171851_XXX.jhist) > and it throw an Exception Just Like > Exception in thread "pool-1-thread-1" java.lang.NullPointerException > at > org.apache.hadoop.mapreduce.jobhistory.HistoryViewer$SummarizedJob.(HistoryViewer.java:626) > at > com.jd.hadoop.log.parse.ParseLogService.getJobDetail(ParseLogService.java:70) > After I'm see the SummarizedJob class I find that attempt.getTaskStatus() is > NULL ， So I change the order of > attempt.getTaskStatus().equals (TaskStatus.State.FAILED.toString()) to > TaskStatus.State.FAILED.toString().equals(attempt.getTaskStatus()) > and it works well . > So I wonder If we can change all attempt.getTaskStatus() after > TaskStatus.State.XXX.toString() ? -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Updated] (MAPREDUCE-6096) SummarizedJob class should be Improvment
[ https://issues.apache.org/jira/browse/MAPREDUCE-6096?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Allen Wittenauer updated MAPREDUCE-6096: Fix Version/s: (was: trunk) > SummarizedJob class should be Improvment > - > > Key: MAPREDUCE-6096 > URL: https://issues.apache.org/jira/browse/MAPREDUCE-6096 > Project: Hadoop Map/Reduce > Issue Type: Improvement > Components: jobhistoryserver >Affects Versions: trunk >Reporter: zhangyubiao >Assignee: zhangyubiao > Labels: easyfix, patch > Attachments: MAPREDUCE-6096.patch > > > When I Parse the JobHistory in the HistoryFile,I use the Hadoop System's > map-reduce-client-core project > org.apache.hadoop.mapreduce.jobhistory.JobHistoryParser class and > HistoryViewer$SummarizedJob to Parse the JobHistoryFile(Just Like > job_1408862281971_489761-1410883171851_XXX.jhist) > and it throw an Exception Just Like > Exception in thread "pool-1-thread-1" java.lang.NullPointerException > at > org.apache.hadoop.mapreduce.jobhistory.HistoryViewer$SummarizedJob.(HistoryViewer.java:626) > at > com.jd.hadoop.log.parse.ParseLogService.getJobDetail(ParseLogService.java:70) > After I'm see the SummarizedJob class I find that attempt.getTaskStatus() is > NULL ， So I change the order of > attempt.getTaskStatus().equals (TaskStatus.State.FAILED.toString()) to > TaskStatus.State.FAILED.toString().equals(attempt.getTaskStatus()) > and it works well . > So I wonder If we can change all attempt.getTaskStatus() after > TaskStatus.State.XXX.toString() ? -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (MAPREDUCE-5279) Jobs can deadlock if headroom is limited by cpu instead of memory
[
https://issues.apache.org/jira/browse/MAPREDUCE-5279?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14143424#comment-14143424
]
Hadoop QA commented on MAPREDUCE-5279:
--
{color:red}-1 overall{color}. Here are the results of testing the latest
attachment
http://issues.apache.org/jira/secure/attachment/12670438/MAPREDUCE-5279-branch-2.patch
against trunk revision 376233c.
{color:red}-1 patch{color}. The patch command could not apply the patch.
Console output:
https://builds.apache.org/job/PreCommit-MAPREDUCE-Build/4906//console
This message is automatically generated.
> Jobs can deadlock if headroom is limited by cpu instead of memory
> -
>
> Key: MAPREDUCE-5279
> URL: https://issues.apache.org/jira/browse/MAPREDUCE-5279
> Project: Hadoop Map/Reduce
> Issue Type: Bug
> Components: mrv2, scheduler
>Affects Versions: 2.0.3-alpha
>Reporter: Peng Zhang
>Assignee: Peng Zhang
>Priority: Critical
> Attachments: MAPREDUCE-5279-branch-2.patch, MAPREDUCE-5279-v2.patch,
> MAPREDUCE-5279.patch, apache-mapreduce-5279.3.patch,
> apache-mapreduce-5279.4.patch, apache-mapreduce-5279.5.patch,
> apache-mapreduce-5279.6.patch, apache-mapreduce-5279.7.patch
>
>
> YARN-2 imported cpu dimension scheduling, but MR RMContainerAllocator doesn't
> take into account virtual cores while scheduling reduce tasks.
> This may cause more reduce tasks to be scheduled because memory is enough.
> And on a small cluster, this will end with deadlock, all running containers
> are reduce tasks but map phase is not finished.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Updated] (MAPREDUCE-5279) Jobs can deadlock if headroom is limited by cpu instead of memory
[ https://issues.apache.org/jira/browse/MAPREDUCE-5279?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Zhijie Shen updated MAPREDUCE-5279: --- Attachment: MAPREDUCE-5279-branch-2.patch Conflict happens on branch-2. Create patch for branch-2. > Jobs can deadlock if headroom is limited by cpu instead of memory > - > > Key: MAPREDUCE-5279 > URL: https://issues.apache.org/jira/browse/MAPREDUCE-5279 > Project: Hadoop Map/Reduce > Issue Type: Bug > Components: mrv2, scheduler >Affects Versions: 2.0.3-alpha >Reporter: Peng Zhang >Assignee: Peng Zhang >Priority: Critical > Attachments: MAPREDUCE-5279-branch-2.patch, MAPREDUCE-5279-v2.patch, > MAPREDUCE-5279.patch, apache-mapreduce-5279.3.patch, > apache-mapreduce-5279.4.patch, apache-mapreduce-5279.5.patch, > apache-mapreduce-5279.6.patch, apache-mapreduce-5279.7.patch > > > YARN-2 imported cpu dimension scheduling, but MR RMContainerAllocator doesn't > take into account virtual cores while scheduling reduce tasks. > This may cause more reduce tasks to be scheduled because memory is enough. > And on a small cluster, this will end with deadlock, all running containers > are reduce tasks but map phase is not finished. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Created] (MAPREDUCE-6102) MR Task UI, sort by name doesn't work
Thomas Graves created MAPREDUCE-6102: Summary: MR Task UI, sort by name doesn't work Key: MAPREDUCE-6102 URL: https://issues.apache.org/jira/browse/MAPREDUCE-6102 Project: Hadoop Map/Reduce Issue Type: Bug Components: webapps Affects Versions: 2.5.1 Reporter: Thomas Graves If you go to the MapReduce ApplicationMaster or HistoryServer UI and open the list of tasks, then try to sort by the task name/id, it does nothing. Note that if you go to the task attempts, that seem to sort fine. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Updated] (MAPREDUCE-6095) Enable DistributedCache for uber-mode Jobs
[ https://issues.apache.org/jira/browse/MAPREDUCE-6095?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Jason Lowe updated MAPREDUCE-6095: -- Resolution: Fixed Fix Version/s: 2.6.0 Hadoop Flags: Reviewed Status: Resolved (was: Patch Available) Thanks, Gera! I committed this to trunk and branch-2. > Enable DistributedCache for uber-mode Jobs > -- > > Key: MAPREDUCE-6095 > URL: https://issues.apache.org/jira/browse/MAPREDUCE-6095 > Project: Hadoop Map/Reduce > Issue Type: Bug > Components: applicationmaster, distributed-cache >Affects Versions: 2.0.5-alpha >Reporter: Gera Shegalov >Assignee: Gera Shegalov > Fix For: 2.6.0 > > Attachments: MAPREDUCE-6095.v01.patch, MAPREDUCE-6095.v02.patch > > > mapreduce.job.cache.local.* is not set for uber-mode jobs. TestUberAM hides > the fact that DC files are not available for uber-mode jobs by overriding > testDistributedCache with a nop method. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (MAPREDUCE-6095) Enable DistributedCache for uber-mode Jobs
[ https://issues.apache.org/jira/browse/MAPREDUCE-6095?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14143296#comment-14143296 ] Jason Lowe commented on MAPREDUCE-6095: --- +1 lgtm. Committing this. > Enable DistributedCache for uber-mode Jobs > -- > > Key: MAPREDUCE-6095 > URL: https://issues.apache.org/jira/browse/MAPREDUCE-6095 > Project: Hadoop Map/Reduce > Issue Type: Bug > Components: applicationmaster, distributed-cache >Affects Versions: 2.0.5-alpha >Reporter: Gera Shegalov >Assignee: Gera Shegalov > Attachments: MAPREDUCE-6095.v01.patch, MAPREDUCE-6095.v02.patch > > > mapreduce.job.cache.local.* is not set for uber-mode jobs. TestUberAM hides > the fact that DC files are not available for uber-mode jobs by overriding > testDistributedCache with a nop method. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (MAPREDUCE-6096) SummarizedJob class should be Improvment
[
https://issues.apache.org/jira/browse/MAPREDUCE-6096?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14143206#comment-14143206
]
Hadoop QA commented on MAPREDUCE-6096:
--
{color:red}-1 overall{color}. Here are the results of testing the latest
attachment
http://issues.apache.org/jira/secure/attachment/12669678/MAPREDUCE-6096.patch
against trunk revision 9721e2c.
{color:green}+1 @author{color}. The patch does not contain any @author
tags.
{color:red}-1 tests included{color}. The patch doesn't appear to include
any new or modified tests.
Please justify why no new tests are needed for this
patch.
Also please list what manual steps were performed to
verify this patch.
{color:green}+1 javac{color}. The applied patch does not increase the
total number of javac compiler warnings.
{color:green}+1 javadoc{color}. There were no new javadoc warning messages.
{color:green}+1 eclipse:eclipse{color}. The patch built with
eclipse:eclipse.
{color:green}+1 findbugs{color}. The patch does not introduce any new
Findbugs (version 2.0.3) warnings.
{color:green}+1 release audit{color}. The applied patch does not increase
the total number of release audit warnings.
{color:green}+1 core tests{color}. The patch passed unit tests in
hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-core.
{color:green}+1 contrib tests{color}. The patch passed contrib unit tests.
Test results:
https://builds.apache.org/job/PreCommit-MAPREDUCE-Build/4905//testReport/
Console output:
https://builds.apache.org/job/PreCommit-MAPREDUCE-Build/4905//console
This message is automatically generated.
> SummarizedJob class should be Improvment
> -
>
> Key: MAPREDUCE-6096
> URL: https://issues.apache.org/jira/browse/MAPREDUCE-6096
> Project: Hadoop Map/Reduce
> Issue Type: Improvement
> Components: jobhistoryserver
>Affects Versions: trunk
>Reporter: zhangyubiao
>Assignee: zhangyubiao
> Labels: easyfix, patch
> Fix For: trunk
>
> Attachments: MAPREDUCE-6096.patch
>
>
> When I Parse the JobHistory in the HistoryFile,I use the Hadoop System's
> map-reduce-client-core project
> org.apache.hadoop.mapreduce.jobhistory.JobHistoryParser class and
> HistoryViewer$SummarizedJob to Parse the JobHistoryFile(Just Like
> job_1408862281971_489761-1410883171851_XXX.jhist)
> and it throw an Exception Just Like
> Exception in thread "pool-1-thread-1" java.lang.NullPointerException
> at
> org.apache.hadoop.mapreduce.jobhistory.HistoryViewer$SummarizedJob.(HistoryViewer.java:626)
> at
> com.jd.hadoop.log.parse.ParseLogService.getJobDetail(ParseLogService.java:70)
> After I'm see the SummarizedJob class I find that attempt.getTaskStatus() is
> NULL ， So I change the order of
> attempt.getTaskStatus().equals (TaskStatus.State.FAILED.toString()) to
> TaskStatus.State.FAILED.toString().equals(attempt.getTaskStatus())
> and it works well .
> So I wonder If we can change all attempt.getTaskStatus() after
> TaskStatus.State.XXX.toString() ?
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Updated] (MAPREDUCE-6096) SummarizedJob class should be Improvment
[ https://issues.apache.org/jira/browse/MAPREDUCE-6096?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] zhangyubiao updated MAPREDUCE-6096: --- Status: Open (was: Patch Available) MAPREDUCE-6096.pat > SummarizedJob class should be Improvment > - > > Key: MAPREDUCE-6096 > URL: https://issues.apache.org/jira/browse/MAPREDUCE-6096 > Project: Hadoop Map/Reduce > Issue Type: Improvement > Components: jobhistoryserver >Affects Versions: trunk >Reporter: zhangyubiao >Assignee: zhangyubiao > Labels: easyfix, patch > Fix For: trunk > > Attachments: MAPREDUCE-6096.patch > > > When I Parse the JobHistory in the HistoryFile,I use the Hadoop System's > map-reduce-client-core project > org.apache.hadoop.mapreduce.jobhistory.JobHistoryParser class and > HistoryViewer$SummarizedJob to Parse the JobHistoryFile(Just Like > job_1408862281971_489761-1410883171851_XXX.jhist) > and it throw an Exception Just Like > Exception in thread "pool-1-thread-1" java.lang.NullPointerException > at > org.apache.hadoop.mapreduce.jobhistory.HistoryViewer$SummarizedJob.(HistoryViewer.java:626) > at > com.jd.hadoop.log.parse.ParseLogService.getJobDetail(ParseLogService.java:70) > After I'm see the SummarizedJob class I find that attempt.getTaskStatus() is > NULL ， So I change the order of > attempt.getTaskStatus().equals (TaskStatus.State.FAILED.toString()) to > TaskStatus.State.FAILED.toString().equals(attempt.getTaskStatus()) > and it works well . > So I wonder If we can change all attempt.getTaskStatus() after > TaskStatus.State.XXX.toString() ? -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Updated] (MAPREDUCE-6096) SummarizedJob class should be Improvment
[ https://issues.apache.org/jira/browse/MAPREDUCE-6096?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] zhangyubiao updated MAPREDUCE-6096: --- Status: Patch Available (was: Open) > SummarizedJob class should be Improvment > - > > Key: MAPREDUCE-6096 > URL: https://issues.apache.org/jira/browse/MAPREDUCE-6096 > Project: Hadoop Map/Reduce > Issue Type: Improvement > Components: jobhistoryserver >Affects Versions: trunk >Reporter: zhangyubiao >Assignee: zhangyubiao > Labels: easyfix, patch > Fix For: trunk > > Attachments: MAPREDUCE-6096.patch > > > When I Parse the JobHistory in the HistoryFile,I use the Hadoop System's > map-reduce-client-core project > org.apache.hadoop.mapreduce.jobhistory.JobHistoryParser class and > HistoryViewer$SummarizedJob to Parse the JobHistoryFile(Just Like > job_1408862281971_489761-1410883171851_XXX.jhist) > and it throw an Exception Just Like > Exception in thread "pool-1-thread-1" java.lang.NullPointerException > at > org.apache.hadoop.mapreduce.jobhistory.HistoryViewer$SummarizedJob.(HistoryViewer.java:626) > at > com.jd.hadoop.log.parse.ParseLogService.getJobDetail(ParseLogService.java:70) > After I'm see the SummarizedJob class I find that attempt.getTaskStatus() is > NULL ， So I change the order of > attempt.getTaskStatus().equals (TaskStatus.State.FAILED.toString()) to > TaskStatus.State.FAILED.toString().equals(attempt.getTaskStatus()) > and it works well . > So I wonder If we can change all attempt.getTaskStatus() after > TaskStatus.State.XXX.toString() ? -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (MAPREDUCE-6096) SummarizedJob class should be Improvment
[ https://issues.apache.org/jira/browse/MAPREDUCE-6096?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14143166#comment-14143166 ] zhangyubiao commented on MAPREDUCE-6096: MAPREDUCE-6096.patch for review > SummarizedJob class should be Improvment > - > > Key: MAPREDUCE-6096 > URL: https://issues.apache.org/jira/browse/MAPREDUCE-6096 > Project: Hadoop Map/Reduce > Issue Type: Improvement > Components: jobhistoryserver >Affects Versions: trunk >Reporter: zhangyubiao >Assignee: zhangyubiao > Labels: easyfix, patch > Fix For: trunk > > Attachments: MAPREDUCE-6096.patch > > > When I Parse the JobHistory in the HistoryFile,I use the Hadoop System's > map-reduce-client-core project > org.apache.hadoop.mapreduce.jobhistory.JobHistoryParser class and > HistoryViewer$SummarizedJob to Parse the JobHistoryFile(Just Like > job_1408862281971_489761-1410883171851_XXX.jhist) > and it throw an Exception Just Like > Exception in thread "pool-1-thread-1" java.lang.NullPointerException > at > org.apache.hadoop.mapreduce.jobhistory.HistoryViewer$SummarizedJob.(HistoryViewer.java:626) > at > com.jd.hadoop.log.parse.ParseLogService.getJobDetail(ParseLogService.java:70) > After I'm see the SummarizedJob class I find that attempt.getTaskStatus() is > NULL ， So I change the order of > attempt.getTaskStatus().equals (TaskStatus.State.FAILED.toString()) to > TaskStatus.State.FAILED.toString().equals(attempt.getTaskStatus()) > and it works well . > So I wonder If we can change all attempt.getTaskStatus() after > TaskStatus.State.XXX.toString() ? -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (MAPREDUCE-5279) Jobs can deadlock if headroom is limited by cpu instead of memory
[ https://issues.apache.org/jira/browse/MAPREDUCE-5279?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14142966#comment-14142966 ] Peng Zhang commented on MAPREDUCE-5279: --- LGTM > Jobs can deadlock if headroom is limited by cpu instead of memory > - > > Key: MAPREDUCE-5279 > URL: https://issues.apache.org/jira/browse/MAPREDUCE-5279 > Project: Hadoop Map/Reduce > Issue Type: Bug > Components: mrv2, scheduler >Affects Versions: 2.0.3-alpha >Reporter: Peng Zhang >Assignee: Peng Zhang >Priority: Critical > Attachments: MAPREDUCE-5279-v2.patch, MAPREDUCE-5279.patch, > apache-mapreduce-5279.3.patch, apache-mapreduce-5279.4.patch, > apache-mapreduce-5279.5.patch, apache-mapreduce-5279.6.patch, > apache-mapreduce-5279.7.patch > > > YARN-2 imported cpu dimension scheduling, but MR RMContainerAllocator doesn't > take into account virtual cores while scheduling reduce tasks. > This may cause more reduce tasks to be scheduled because memory is enough. > And on a small cluster, this will end with deadlock, all running containers > are reduce tasks but map phase is not finished. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
