[jira] [Commented] (MAPREDUCE-5849) ReduceContextImpl.ValueIterable can only be iterated once
[
https://issues.apache.org/jira/browse/MAPREDUCE-5849?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14182412#comment-14182412
]
Morgany Mendes commented on MAPREDUCE-5849:
---
I think that is not a bug, but, can be a design mistake on the signature of
reduce method. Thinking about the following (simplified) scenario:
1. The reduce fase can start reducer tasks if there is some slot available in
some task tracker.
2. So, if some reducer for a specific key (in some specific partition) has
started, then, the iterator within the values can only reach the end (false
hasNext) after the map fase has reached the end.
3. Therefore, if you try to iterate the values again. That will only happens
after all the mappers has ended up its work.
Because receiving an Iterable make us naturally to think that iterator()
method will return an new instance of Iterator positioned at the
beginning. I would like to propose that the reduce method receive the
Iterator (just like reduce on MRv1), instead of an Iterable. It
can be done by implementing a new Reducer class which extends and deprecates
the existent one and overriding the run() method calling the new reduce.
@Deprecated
public class Reducer{
...
}
public class NewReducer extends
Reducer{
protected void reduce(KEYIN key, Iterator values, Context context
) throws IOException, InterruptedException {
for (;iterator.hasNext();) {
VALUEIN value = iterator.next();
context.write((KEYOUT) key, (VALUEOUT) value);
}
}
@Overrides
public void run(Context context) throws IOException, InterruptedException {
setup(context);
while (context.nextKey()) {
Iterator iter = context.getValues().iterator();
reduce(context.getCurrentKey(), iter, context);
// If a back up store is used, reset it
if(iter instanceof ReduceContext.ValueIterator) {
((ReduceContext.ValueIterator)iter).resetBackupStore();
}
}
cleanup(context);
}
}
> ReduceContextImpl.ValueIterable can only be iterated once
> -
>
> Key: MAPREDUCE-5849
> URL: https://issues.apache.org/jira/browse/MAPREDUCE-5849
> Project: Hadoop Map/Reduce
> Issue Type: Bug
>Affects Versions: 2.3.0
>Reporter: Ivan Balashov
>Priority: Minor
>
> Since ValueIterable reuses same iterator object it cannot be iterated more
> than once. This behaviour is counter-intuitive (with regards to Iterable
> usage) and thus difficult to spot.
> Unless there are particular performance concerns, new instance of Iterator
> should be returned.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Commented] (MAPREDUCE-5933) Enable MR AM to post history events to the timeline server
[
https://issues.apache.org/jira/browse/MAPREDUCE-5933?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14182316#comment-14182316
]
Hadoop QA commented on MAPREDUCE-5933:
--
{color:green}+1 overall{color}. Here are the results of testing the latest
attachment
http://issues.apache.org/jira/secure/attachment/12676742/MAPREDUCE-5933.patch
against trunk revision 670879e.
{color:green}+1 @author{color}. The patch does not contain any @author
tags.
{color:green}+1 tests included{color}. The patch appears to include 3 new
or modified test files.
{color:green}+1 javac{color}. The applied patch does not increase the
total number of javac compiler warnings.
{color:green}+1 javadoc{color}. There were no new javadoc warning messages.
{color:green}+1 eclipse:eclipse{color}. The patch built with
eclipse:eclipse.
{color:green}+1 findbugs{color}. The patch does not introduce any new
Findbugs (version 2.0.3) warnings.
{color:green}+1 release audit{color}. The applied patch does not increase
the total number of release audit warnings.
{color:green}+1 core tests{color}. The patch passed unit tests in
hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-app
hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-jobclient.
{color:green}+1 contrib tests{color}. The patch passed contrib unit tests.
Test results:
https://builds.apache.org/job/PreCommit-MAPREDUCE-Build/4978//testReport/
Console output:
https://builds.apache.org/job/PreCommit-MAPREDUCE-Build/4978//console
This message is automatically generated.
> Enable MR AM to post history events to the timeline server
> --
>
> Key: MAPREDUCE-5933
> URL: https://issues.apache.org/jira/browse/MAPREDUCE-5933
> Project: Hadoop Map/Reduce
> Issue Type: Sub-task
> Components: mr-am
>Reporter: Zhijie Shen
>Assignee: Robert Kanter
> Attachments: MAPREDUCE-5933.patch, MAPREDUCE-5933.patch,
> MAPREDUCE-5933.patch, MAPREDUCE-5933.patch, MAPREDUCE-5933.patch,
> MAPREDUCE-5933.patch, MAPREDUCE-5933.patch, MAPREDUCE-5933.patch,
> mr_timelineserver_response.txt
>
>
> Nowadays, MR AM collects the history events and writes it to HDFS for JHS to
> source. With the timeline server, MR AM can put these events there.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Commented] (MAPREDUCE-5933) Enable MR AM to post history events to the timeline server
[ https://issues.apache.org/jira/browse/MAPREDUCE-5933?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14182210#comment-14182210 ] Robert Kanter commented on MAPREDUCE-5933: -- Looks like something funny happened with the tests; I've kicked off another run. > Enable MR AM to post history events to the timeline server > -- > > Key: MAPREDUCE-5933 > URL: https://issues.apache.org/jira/browse/MAPREDUCE-5933 > Project: Hadoop Map/Reduce > Issue Type: Sub-task > Components: mr-am >Reporter: Zhijie Shen >Assignee: Robert Kanter > Attachments: MAPREDUCE-5933.patch, MAPREDUCE-5933.patch, > MAPREDUCE-5933.patch, MAPREDUCE-5933.patch, MAPREDUCE-5933.patch, > MAPREDUCE-5933.patch, MAPREDUCE-5933.patch, MAPREDUCE-5933.patch, > mr_timelineserver_response.txt > > > Nowadays, MR AM collects the history events and writes it to HDFS for JHS to > source. With the timeline server, MR AM can put these events there. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (MAPREDUCE-5933) Enable MR AM to post history events to the timeline server
[
https://issues.apache.org/jira/browse/MAPREDUCE-5933?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14182143#comment-14182143
]
Hadoop QA commented on MAPREDUCE-5933:
--
{color:red}-1 overall{color}. Here are the results of testing the latest
attachment
http://issues.apache.org/jira/secure/attachment/12676742/MAPREDUCE-5933.patch
against trunk revision 828429d.
{color:green}+1 @author{color}. The patch does not contain any @author
tags.
{color:green}+1 tests included{color}. The patch appears to include 3 new
or modified test files.
{color:green}+1 javac{color}. The applied patch does not increase the
total number of javac compiler warnings.
{color:green}+1 javadoc{color}. There were no new javadoc warning messages.
{color:green}+1 eclipse:eclipse{color}. The patch built with
eclipse:eclipse.
{color:green}+1 findbugs{color}. The patch does not introduce any new
Findbugs (version 2.0.3) warnings.
{color:green}+1 release audit{color}. The applied patch does not increase
the total number of release audit warnings.
{color:red}-1 core tests{color}. The test build failed in
hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-jobclient
{color:green}+1 contrib tests{color}. The patch passed contrib unit tests.
Test results:
https://builds.apache.org/job/PreCommit-MAPREDUCE-Build/4977//testReport/
Console output:
https://builds.apache.org/job/PreCommit-MAPREDUCE-Build/4977//console
This message is automatically generated.
> Enable MR AM to post history events to the timeline server
> --
>
> Key: MAPREDUCE-5933
> URL: https://issues.apache.org/jira/browse/MAPREDUCE-5933
> Project: Hadoop Map/Reduce
> Issue Type: Sub-task
> Components: mr-am
>Reporter: Zhijie Shen
>Assignee: Robert Kanter
> Attachments: MAPREDUCE-5933.patch, MAPREDUCE-5933.patch,
> MAPREDUCE-5933.patch, MAPREDUCE-5933.patch, MAPREDUCE-5933.patch,
> MAPREDUCE-5933.patch, MAPREDUCE-5933.patch, MAPREDUCE-5933.patch,
> mr_timelineserver_response.txt
>
>
> Nowadays, MR AM collects the history events and writes it to HDFS for JHS to
> source. With the timeline server, MR AM can put these events there.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Updated] (MAPREDUCE-5933) Enable MR AM to post history events to the timeline server
[ https://issues.apache.org/jira/browse/MAPREDUCE-5933?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Robert Kanter updated MAPREDUCE-5933: - Attachment: MAPREDUCE-5933.patch Same patch but I fixed a missing import. > Enable MR AM to post history events to the timeline server > -- > > Key: MAPREDUCE-5933 > URL: https://issues.apache.org/jira/browse/MAPREDUCE-5933 > Project: Hadoop Map/Reduce > Issue Type: Sub-task > Components: mr-am >Reporter: Zhijie Shen >Assignee: Robert Kanter > Attachments: MAPREDUCE-5933.patch, MAPREDUCE-5933.patch, > MAPREDUCE-5933.patch, MAPREDUCE-5933.patch, MAPREDUCE-5933.patch, > MAPREDUCE-5933.patch, MAPREDUCE-5933.patch, MAPREDUCE-5933.patch, > mr_timelineserver_response.txt > > > Nowadays, MR AM collects the history events and writes it to HDFS for JHS to > source. With the timeline server, MR AM can put these events there. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Resolved] (MAPREDUCE-6135) Job staging directory remains if MRAppMaster is OOM
[ https://issues.apache.org/jira/browse/MAPREDUCE-6135?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Ming Ma resolved MAPREDUCE-6135. Resolution: Duplicate Thanks, Jason. Resolve this as dup. Will continue the discussion over at MAPREDUCE-5502. It looks like Robert in MAPREDUCE-4428 also mentioned the approach of rerun AM for cleanup. > Job staging directory remains if MRAppMaster is OOM > --- > > Key: MAPREDUCE-6135 > URL: https://issues.apache.org/jira/browse/MAPREDUCE-6135 > Project: Hadoop Map/Reduce > Issue Type: Bug >Reporter: Ming Ma > > If MRAppMaster attempts run out of memory, it won't go through the normal job > clean up process to move history files to history server location. When > customers try to find out why the job failed, the data won't be available on > history server webUI. > The work around is to extract the container id and NM id from the jhist file > in the job staging directory; then use "yarn logs" command to get the AM logs. > It would be great the platform can take care of it by moving these hist files > automatically to history server if AM attempts don't exit properly. > We discuss ideas on how to address this and would like get suggestions from > others. Not sure if timeline server design covers this scenario. > 1. Define some protocol for YARN to tell AppMaster "you have exceeded AM max > attempt, please clean up". For example, YARN can launch AppMaster one more > time after AM max attempt and MRAppMaster use that as the indication this is > clean-up-only attempt. > 2. Have some program periodically check job statuses and move files from job > staging directory to history server for those finished jobs. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (MAPREDUCE-5933) Enable MR AM to post history events to the timeline server
[
https://issues.apache.org/jira/browse/MAPREDUCE-5933?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14182000#comment-14182000
]
Hadoop QA commented on MAPREDUCE-5933:
--
{color:red}-1 overall{color}. Here are the results of testing the latest
attachment
http://issues.apache.org/jira/secure/attachment/12676727/MAPREDUCE-5933.patch
against trunk revision 828429d.
{color:green}+1 @author{color}. The patch does not contain any @author
tags.
{color:green}+1 tests included{color}. The patch appears to include 3 new
or modified test files.
{color:red}-1 javac{color:red}. The patch appears to cause the build to
fail.
Console output:
https://builds.apache.org/job/PreCommit-MAPREDUCE-Build/4976//console
This message is automatically generated.
> Enable MR AM to post history events to the timeline server
> --
>
> Key: MAPREDUCE-5933
> URL: https://issues.apache.org/jira/browse/MAPREDUCE-5933
> Project: Hadoop Map/Reduce
> Issue Type: Sub-task
> Components: mr-am
>Reporter: Zhijie Shen
>Assignee: Robert Kanter
> Attachments: MAPREDUCE-5933.patch, MAPREDUCE-5933.patch,
> MAPREDUCE-5933.patch, MAPREDUCE-5933.patch, MAPREDUCE-5933.patch,
> MAPREDUCE-5933.patch, MAPREDUCE-5933.patch, mr_timelineserver_response.txt
>
>
> Nowadays, MR AM collects the history events and writes it to HDFS for JHS to
> source. With the timeline server, MR AM can put these events there.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Updated] (MAPREDUCE-5933) Enable MR AM to post history events to the timeline server
[ https://issues.apache.org/jira/browse/MAPREDUCE-5933?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Robert Kanter updated MAPREDUCE-5933: - Attachment: MAPREDUCE-5933.patch The new patch addresses the latest comments. I see that you've already created MAPREDUCE-6134 for handling the primary filters; thanks! > Enable MR AM to post history events to the timeline server > -- > > Key: MAPREDUCE-5933 > URL: https://issues.apache.org/jira/browse/MAPREDUCE-5933 > Project: Hadoop Map/Reduce > Issue Type: Sub-task > Components: mr-am >Reporter: Zhijie Shen >Assignee: Robert Kanter > Attachments: MAPREDUCE-5933.patch, MAPREDUCE-5933.patch, > MAPREDUCE-5933.patch, MAPREDUCE-5933.patch, MAPREDUCE-5933.patch, > MAPREDUCE-5933.patch, MAPREDUCE-5933.patch, mr_timelineserver_response.txt > > > Nowadays, MR AM collects the history events and writes it to HDFS for JHS to > source. With the timeline server, MR AM can put these events there. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (MAPREDUCE-6018) Create a framework specific config to enable timeline server
[ https://issues.apache.org/jira/browse/MAPREDUCE-6018?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14181868#comment-14181868 ] Zhijie Shen commented on MAPREDUCE-6018: [~rkanter], thanks for taking care of this issue. We could have a MR framework-level config, such as "mapreduce.jobhistory.timeline-service.enabled", and it can also be overwritten by each individual MR job by specifying it in CLI args. Only when this flag is set, the logic that history events are posted to the timeline server will be executed. > Create a framework specific config to enable timeline server > > > Key: MAPREDUCE-6018 > URL: https://issues.apache.org/jira/browse/MAPREDUCE-6018 > Project: Hadoop Map/Reduce > Issue Type: Sub-task >Reporter: Jonathan Eagles >Assignee: Robert Kanter > -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (MAPREDUCE-6135) Job staging directory remains if MRAppMaster is OOM
[ https://issues.apache.org/jira/browse/MAPREDUCE-6135?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14181818#comment-14181818 ] Jason Lowe commented on MAPREDUCE-6135: --- This is a well known issue in YARN covered by a number of other JIRAs and can also happen even if the AM never even has a chance to run, see MAPREDUCE-5502. Other related JIRAs include MAPREDUCE-4559 and MAPREDUCE-4428. > Job staging directory remains if MRAppMaster is OOM > --- > > Key: MAPREDUCE-6135 > URL: https://issues.apache.org/jira/browse/MAPREDUCE-6135 > Project: Hadoop Map/Reduce > Issue Type: Bug >Reporter: Ming Ma > > If MRAppMaster attempts run out of memory, it won't go through the normal job > clean up process to move history files to history server location. When > customers try to find out why the job failed, the data won't be available on > history server webUI. > The work around is to extract the container id and NM id from the jhist file > in the job staging directory; then use "yarn logs" command to get the AM logs. > It would be great the platform can take care of it by moving these hist files > automatically to history server if AM attempts don't exit properly. > We discuss ideas on how to address this and would like get suggestions from > others. Not sure if timeline server design covers this scenario. > 1. Define some protocol for YARN to tell AppMaster "you have exceeded AM max > attempt, please clean up". For example, YARN can launch AppMaster one more > time after AM max attempt and MRAppMaster use that as the indication this is > clean-up-only attempt. > 2. Have some program periodically check job statuses and move files from job > staging directory to history server for those finished jobs. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Assigned] (MAPREDUCE-6018) Create a framework specific config to enable timeline server
[ https://issues.apache.org/jira/browse/MAPREDUCE-6018?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Robert Kanter reassigned MAPREDUCE-6018: Assignee: Robert Kanter > Create a framework specific config to enable timeline server > > > Key: MAPREDUCE-6018 > URL: https://issues.apache.org/jira/browse/MAPREDUCE-6018 > Project: Hadoop Map/Reduce > Issue Type: Sub-task >Reporter: Jonathan Eagles >Assignee: Robert Kanter > -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Resolved] (MAPREDUCE-6138) Unable to run start-all.sh script, getting error
[ https://issues.apache.org/jira/browse/MAPREDUCE-6138?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Akira AJISAKA resolved MAPREDUCE-6138. -- Resolution: Invalid Fix Version/s: (was: 2.5.0) Hadoop Flags: (was: Reviewed) > Unable to run start-all.sh script, getting error > > > Key: MAPREDUCE-6138 > URL: https://issues.apache.org/jira/browse/MAPREDUCE-6138 > Project: Hadoop Map/Reduce > Issue Type: Bug >Affects Versions: 2.5.0 >Reporter: Jagadeesh A Y >Priority: Blocker > > Present working Shell: /bin/bash > Trying to run start-all.sh, but getting below error. > hduser@jagadeesh-Aspire-E5-511:/usr/local/hadoop/hadoop-2.5.0/sbin$ sh > start-all.sh > This script is Deprecated. Instead use start-dfs.sh and start-yarn.sh > start-all.sh: 82: > /usr/local/hadoop/hadoop-2.5.0/sbin/../libexec/hadoop-config.sh: Syntax > error: word unexpected (expecting ")") > Kindly help. > Regards > Jagadeesh A Y -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Reopened] (MAPREDUCE-6138) Unable to run start-all.sh script, getting error
[ https://issues.apache.org/jira/browse/MAPREDUCE-6138?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Akira AJISAKA reopened MAPREDUCE-6138: -- Assignee: (was: Jagadeesh A Y) > Unable to run start-all.sh script, getting error > > > Key: MAPREDUCE-6138 > URL: https://issues.apache.org/jira/browse/MAPREDUCE-6138 > Project: Hadoop Map/Reduce > Issue Type: Bug >Affects Versions: 2.5.0 >Reporter: Jagadeesh A Y >Priority: Blocker > Fix For: 2.5.0 > > > Present working Shell: /bin/bash > Trying to run start-all.sh, but getting below error. > hduser@jagadeesh-Aspire-E5-511:/usr/local/hadoop/hadoop-2.5.0/sbin$ sh > start-all.sh > This script is Deprecated. Instead use start-dfs.sh and start-yarn.sh > start-all.sh: 82: > /usr/local/hadoop/hadoop-2.5.0/sbin/../libexec/hadoop-config.sh: Syntax > error: word unexpected (expecting ")") > Kindly help. > Regards > Jagadeesh A Y -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (MAPREDUCE-6138) Unable to run start-all.sh script, getting error
[ https://issues.apache.org/jira/browse/MAPREDUCE-6138?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14181736#comment-14181736 ] Akira AJISAKA commented on MAPREDUCE-6138: -- bq. Works fine... I'm glad to hear that :) Changing to the correct flag for the users who write a query to search the fixed issues. > Unable to run start-all.sh script, getting error > > > Key: MAPREDUCE-6138 > URL: https://issues.apache.org/jira/browse/MAPREDUCE-6138 > Project: Hadoop Map/Reduce > Issue Type: Bug >Affects Versions: 2.5.0 >Reporter: Jagadeesh A Y >Assignee: Jagadeesh A Y >Priority: Blocker > Fix For: 2.5.0 > > > Present working Shell: /bin/bash > Trying to run start-all.sh, but getting below error. > hduser@jagadeesh-Aspire-E5-511:/usr/local/hadoop/hadoop-2.5.0/sbin$ sh > start-all.sh > This script is Deprecated. Instead use start-dfs.sh and start-yarn.sh > start-all.sh: 82: > /usr/local/hadoop/hadoop-2.5.0/sbin/../libexec/hadoop-config.sh: Syntax > error: word unexpected (expecting ")") > Kindly help. > Regards > Jagadeesh A Y -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (MAPREDUCE-3202) Integrating Hadoop Vaidya with Job History UI in Hadoop 2.0
[
https://issues.apache.org/jira/browse/MAPREDUCE-3202?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14181583#comment-14181583
]
Hadoop QA commented on MAPREDUCE-3202:
--
{color:red}-1 overall{color}. Here are the results of testing the latest
attachment
http://issues.apache.org/jira/secure/attachment/12607698/MAPREDUCE-3202.patch
against trunk revision d71d40a.
{color:red}-1 patch{color}. The patch command could not apply the patch.
Console output:
https://builds.apache.org/job/PreCommit-MAPREDUCE-Build/4975//console
This message is automatically generated.
> Integrating Hadoop Vaidya with Job History UI in Hadoop 2.0
>
>
> Key: MAPREDUCE-3202
> URL: https://issues.apache.org/jira/browse/MAPREDUCE-3202
> Project: Hadoop Map/Reduce
> Issue Type: New Feature
> Components: jobhistoryserver
>Affects Versions: 2.0.0-alpha
>Reporter: vitthal (Suhas) Gogate
>Assignee: vitthal (Suhas) Gogate
> Fix For: trunk
>
> Attachments: MAPREDUCE-3202.patch, MAPREDUCE-3202.patch
>
>
> Hadoop Vaidya provides a detailed analysis of the M/R job in terms of various
> execution inefficiencies and the associated remedies that user can easily
> understand and fix. This Jira patch integrates it with Job History UI under
> Hadoop 2.0 branch.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Assigned] (MAPREDUCE-3202) Integrating Hadoop Vaidya with Job History UI in Hadoop 2.0
[ https://issues.apache.org/jira/browse/MAPREDUCE-3202?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] vitthal (Suhas) Gogate reassigned MAPREDUCE-3202: - Assignee: vitthal (Suhas) Gogate (was: Evan (Yiming) Meng) > Integrating Hadoop Vaidya with Job History UI in Hadoop 2.0 > > > Key: MAPREDUCE-3202 > URL: https://issues.apache.org/jira/browse/MAPREDUCE-3202 > Project: Hadoop Map/Reduce > Issue Type: New Feature > Components: jobhistoryserver >Affects Versions: 2.0.0-alpha >Reporter: vitthal (Suhas) Gogate >Assignee: vitthal (Suhas) Gogate > Fix For: trunk > > Attachments: MAPREDUCE-3202.patch, MAPREDUCE-3202.patch > > > Hadoop Vaidya provides a detailed analysis of the M/R job in terms of various > execution inefficiencies and the associated remedies that user can easily > understand and fix. This Jira patch integrates it with Job History UI under > Hadoop 2.0 branch. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (MAPREDUCE-6126) (Rumen) Rumen tool returns error "ava.lang.IllegalArgumentException: JobBuilder.process(HistoryEvent): unknown event type"
[ https://issues.apache.org/jira/browse/MAPREDUCE-6126?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14181441#comment-14181441 ] Hudson commented on MAPREDUCE-6126: --- FAILURE: Integrated in Hadoop-Mapreduce-trunk #1935 (See [https://builds.apache.org/job/Hadoop-Mapreduce-trunk/1935/]) MAPREDUCE-6126. Fixed Rumen JobBuilder to ignore NormalizedResourceEvent. Contributed by Junping Du (jianhe: rev b8f7966c7a0d6aa0c0835fc0c4a4254420ab26a6) * hadoop-tools/hadoop-rumen/src/main/java/org/apache/hadoop/tools/rumen/JobBuilder.java * hadoop-mapreduce-project/CHANGES.txt Updated CHANGES.txt to move MAPREDUCE-6126 to 2.6 (jianhe: rev d67214fd69f44391b7b43f7b7d18d8eefc3dd2da) * hadoop-mapreduce-project/CHANGES.txt > (Rumen) Rumen tool returns error "ava.lang.IllegalArgumentException: > JobBuilder.process(HistoryEvent): unknown event type" > -- > > Key: MAPREDUCE-6126 > URL: https://issues.apache.org/jira/browse/MAPREDUCE-6126 > Project: Hadoop Map/Reduce > Issue Type: Bug >Reporter: Junping Du >Assignee: Junping Du > Fix For: 2.6.0 > > Attachments: MAPREDUCE-6126-v2.patch, MAPREDUCE-6126.patch > > > java.lang.IllegalArgumentException: JobBuilder.process(HistoryEvent): unknown > event type > at org.apache.hadoop.tools.rumen.JobBuilder.process(JobBuilder.java:172) > at > org.apache.hadoop.tools.rumen.TraceBuilder.processJobHistory(TraceBuilder.java:305) > at org.apache.hadoop.tools.rumen.TraceBuilder.run(TraceBuilder.java:259) > at org.apache.hadoop.util.ToolRunner.run(ToolRunner.java:70) > at org.apache.hadoop.util.ToolRunner.run(ToolRunner.java:84) > at org.apache.hadoop.tools.rumen.TraceBuilder.main(TraceBuilder.java:186) -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (MAPREDUCE-6126) (Rumen) Rumen tool returns error "ava.lang.IllegalArgumentException: JobBuilder.process(HistoryEvent): unknown event type"
[ https://issues.apache.org/jira/browse/MAPREDUCE-6126?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14181379#comment-14181379 ] Hudson commented on MAPREDUCE-6126: --- SUCCESS: Integrated in Hadoop-Hdfs-trunk #1910 (See [https://builds.apache.org/job/Hadoop-Hdfs-trunk/1910/]) MAPREDUCE-6126. Fixed Rumen JobBuilder to ignore NormalizedResourceEvent. Contributed by Junping Du (jianhe: rev b8f7966c7a0d6aa0c0835fc0c4a4254420ab26a6) * hadoop-tools/hadoop-rumen/src/main/java/org/apache/hadoop/tools/rumen/JobBuilder.java * hadoop-mapreduce-project/CHANGES.txt Updated CHANGES.txt to move MAPREDUCE-6126 to 2.6 (jianhe: rev d67214fd69f44391b7b43f7b7d18d8eefc3dd2da) * hadoop-mapreduce-project/CHANGES.txt > (Rumen) Rumen tool returns error "ava.lang.IllegalArgumentException: > JobBuilder.process(HistoryEvent): unknown event type" > -- > > Key: MAPREDUCE-6126 > URL: https://issues.apache.org/jira/browse/MAPREDUCE-6126 > Project: Hadoop Map/Reduce > Issue Type: Bug >Reporter: Junping Du >Assignee: Junping Du > Fix For: 2.6.0 > > Attachments: MAPREDUCE-6126-v2.patch, MAPREDUCE-6126.patch > > > java.lang.IllegalArgumentException: JobBuilder.process(HistoryEvent): unknown > event type > at org.apache.hadoop.tools.rumen.JobBuilder.process(JobBuilder.java:172) > at > org.apache.hadoop.tools.rumen.TraceBuilder.processJobHistory(TraceBuilder.java:305) > at org.apache.hadoop.tools.rumen.TraceBuilder.run(TraceBuilder.java:259) > at org.apache.hadoop.util.ToolRunner.run(ToolRunner.java:70) > at org.apache.hadoop.util.ToolRunner.run(ToolRunner.java:84) > at org.apache.hadoop.tools.rumen.TraceBuilder.main(TraceBuilder.java:186) -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Resolved] (MAPREDUCE-6138) Unable to run start-all.sh script, getting error
[ https://issues.apache.org/jira/browse/MAPREDUCE-6138?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Jagadeesh A Y resolved MAPREDUCE-6138. -- Resolution: Fixed Fix Version/s: 2.5.0 Assignee: Jagadeesh A Y Hadoop Flags: Reviewed > Unable to run start-all.sh script, getting error > > > Key: MAPREDUCE-6138 > URL: https://issues.apache.org/jira/browse/MAPREDUCE-6138 > Project: Hadoop Map/Reduce > Issue Type: Bug >Affects Versions: 2.5.0 >Reporter: Jagadeesh A Y >Assignee: Jagadeesh A Y >Priority: Blocker > Fix For: 2.5.0 > > > Present working Shell: /bin/bash > Trying to run start-all.sh, but getting below error. > hduser@jagadeesh-Aspire-E5-511:/usr/local/hadoop/hadoop-2.5.0/sbin$ sh > start-all.sh > This script is Deprecated. Instead use start-dfs.sh and start-yarn.sh > start-all.sh: 82: > /usr/local/hadoop/hadoop-2.5.0/sbin/../libexec/hadoop-config.sh: Syntax > error: word unexpected (expecting ")") > Kindly help. > Regards > Jagadeesh A Y -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (MAPREDUCE-6138) Unable to run start-all.sh script, getting error
[ https://issues.apache.org/jira/browse/MAPREDUCE-6138?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14181303#comment-14181303 ] Jagadeesh A Y commented on MAPREDUCE-6138: -- Way to execute start-all.sh is hduser@jagadeesh-Aspire-E5-511:/usr/local/hadoop/hadoop-2.5.0/sbin$ bash start-all.sh Works fine... Regards Jagadeesh A Y > Unable to run start-all.sh script, getting error > > > Key: MAPREDUCE-6138 > URL: https://issues.apache.org/jira/browse/MAPREDUCE-6138 > Project: Hadoop Map/Reduce > Issue Type: Bug >Affects Versions: 2.5.0 >Reporter: Jagadeesh A Y >Priority: Blocker > > Present working Shell: /bin/bash > Trying to run start-all.sh, but getting below error. > hduser@jagadeesh-Aspire-E5-511:/usr/local/hadoop/hadoop-2.5.0/sbin$ sh > start-all.sh > This script is Deprecated. Instead use start-dfs.sh and start-yarn.sh > start-all.sh: 82: > /usr/local/hadoop/hadoop-2.5.0/sbin/../libexec/hadoop-config.sh: Syntax > error: word unexpected (expecting ")") > Kindly help. > Regards > Jagadeesh A Y -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Created] (MAPREDUCE-6138) Unable to run start-all.sh script, getting error
Jagadeesh A Y created MAPREDUCE-6138: Summary: Unable to run start-all.sh script, getting error Key: MAPREDUCE-6138 URL: https://issues.apache.org/jira/browse/MAPREDUCE-6138 Project: Hadoop Map/Reduce Issue Type: Bug Affects Versions: 2.5.0 Reporter: Jagadeesh A Y Priority: Blocker Present working Shell: /bin/bash Trying to run start-all.sh, but getting below error. hduser@jagadeesh-Aspire-E5-511:/usr/local/hadoop/hadoop-2.5.0/sbin$ sh start-all.sh This script is Deprecated. Instead use start-dfs.sh and start-yarn.sh start-all.sh: 82: /usr/local/hadoop/hadoop-2.5.0/sbin/../libexec/hadoop-config.sh: Syntax error: word unexpected (expecting ")") Kindly help. Regards Jagadeesh A Y -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Created] (MAPREDUCE-6137) Unable to run start-all.sh script, getting error
Jagadeesh A Y created MAPREDUCE-6137: Summary: Unable to run start-all.sh script, getting error Key: MAPREDUCE-6137 URL: https://issues.apache.org/jira/browse/MAPREDUCE-6137 Project: Hadoop Map/Reduce Issue Type: Bug Affects Versions: 2.5.0 Reporter: Jagadeesh A Y Priority: Blocker Present working Shell: /bin/bash Trying to run start-all.sh, but getting below error. hduser@jagadeesh-Aspire-E5-511:/usr/local/hadoop/hadoop-2.5.0/sbin$ sh start-all.sh This script is Deprecated. Instead use start-dfs.sh and start-yarn.sh start-all.sh: 82: /usr/local/hadoop/hadoop-2.5.0/sbin/../libexec/hadoop-config.sh: Syntax error: word unexpected (expecting ")") Kindly help. Regards Jagadeesh A Y -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (MAPREDUCE-6126) (Rumen) Rumen tool returns error "ava.lang.IllegalArgumentException: JobBuilder.process(HistoryEvent): unknown event type"
[ https://issues.apache.org/jira/browse/MAPREDUCE-6126?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14181253#comment-14181253 ] Hudson commented on MAPREDUCE-6126: --- SUCCESS: Integrated in Hadoop-Yarn-trunk #721 (See [https://builds.apache.org/job/Hadoop-Yarn-trunk/721/]) MAPREDUCE-6126. Fixed Rumen JobBuilder to ignore NormalizedResourceEvent. Contributed by Junping Du (jianhe: rev b8f7966c7a0d6aa0c0835fc0c4a4254420ab26a6) * hadoop-tools/hadoop-rumen/src/main/java/org/apache/hadoop/tools/rumen/JobBuilder.java * hadoop-mapreduce-project/CHANGES.txt Updated CHANGES.txt to move MAPREDUCE-6126 to 2.6 (jianhe: rev d67214fd69f44391b7b43f7b7d18d8eefc3dd2da) * hadoop-mapreduce-project/CHANGES.txt > (Rumen) Rumen tool returns error "ava.lang.IllegalArgumentException: > JobBuilder.process(HistoryEvent): unknown event type" > -- > > Key: MAPREDUCE-6126 > URL: https://issues.apache.org/jira/browse/MAPREDUCE-6126 > Project: Hadoop Map/Reduce > Issue Type: Bug >Reporter: Junping Du >Assignee: Junping Du > Fix For: 2.6.0 > > Attachments: MAPREDUCE-6126-v2.patch, MAPREDUCE-6126.patch > > > java.lang.IllegalArgumentException: JobBuilder.process(HistoryEvent): unknown > event type > at org.apache.hadoop.tools.rumen.JobBuilder.process(JobBuilder.java:172) > at > org.apache.hadoop.tools.rumen.TraceBuilder.processJobHistory(TraceBuilder.java:305) > at org.apache.hadoop.tools.rumen.TraceBuilder.run(TraceBuilder.java:259) > at org.apache.hadoop.util.ToolRunner.run(ToolRunner.java:70) > at org.apache.hadoop.util.ToolRunner.run(ToolRunner.java:84) > at org.apache.hadoop.tools.rumen.TraceBuilder.main(TraceBuilder.java:186) -- This message was sent by Atlassian JIRA (v6.3.4#6332)
