[jira] [Updated] (MAPREDUCE-6515) Update Application priority in AM side from AM-RM heartbeat
[ https://issues.apache.org/jira/browse/MAPREDUCE-6515?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Sunil G updated MAPREDUCE-6515: --- Attachment: 0002-MAPREDUCE-6515.patch Updating a cleaner patch by adding more information in test case. cc/[~jlowe], [~jianhe] and [~rohithsharma]. Kindly help to check this patch. > Update Application priority in AM side from AM-RM heartbeat > --- > > Key: MAPREDUCE-6515 > URL: https://issues.apache.org/jira/browse/MAPREDUCE-6515 > Project: Hadoop Map/Reduce > Issue Type: Bug > Components: applicationmaster >Reporter: Sunil G >Assignee: Sunil G > Attachments: 0001-MAPREDUCE-6515.patch, 0002-MAPREDUCE-6515.patch > > > After YARN-4170, Application Priority is available via heartbeat call. Update > this information in AM sothat client can fetch this information via JobStatus > (JobReport) call. > This is as per the discussion happened in MAPREDUCE-5870. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (MAPREDUCE-6521) MiniMRYarnCluster should not create /tmp/hadoop-yarn/staging on local filesystem in unit test
[
https://issues.apache.org/jira/browse/MAPREDUCE-6521?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=14968794#comment-14968794
]
Masatake Iwasaki commented on MAPREDUCE-6521:
-
{code}
if (conf.get(MRJobConfig.MR_AM_STAGING_DIR) == null) {
conf.set(MRJobConfig.MR_AM_STAGING_DIR, new File(getTestWorkDir(),
"apps_staging_dir/").getAbsolutePath());
}
{code}
The intent of the if statement seems to be "if
yarn.app.mapreduce.am.staging-dir is not specified in configuration" but the
{{conf.get(MRJobConfig.MR_AM_STAGING_DIR)}} always returns non-null value
because the default value is defined in mapred-default.xml as
{{/tmp/hadoop-yarn/staging}}.
> MiniMRYarnCluster should not create /tmp/hadoop-yarn/staging on local
> filesystem in unit test
> -
>
> Key: MAPREDUCE-6521
> URL: https://issues.apache.org/jira/browse/MAPREDUCE-6521
> Project: Hadoop Map/Reduce
> Issue Type: Bug
> Components: test
>Reporter: Masatake Iwasaki
>Assignee: Masatake Iwasaki
>
> MiniMRYarnCluster create /tmp/hadoop-yarn/staging/history/done by default. It
> should be under {{testWorkDir}} if the file system is localFs in order to
> make it to be removed by {{mvn clean}}. It would also avoid issues under
> parallel unit testing.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Commented] (MAPREDUCE-6515) Update Application priority in AM side from AM-RM heartbeat
[
https://issues.apache.org/jira/browse/MAPREDUCE-6515?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=14968799#comment-14968799
]
Hadoop QA commented on MAPREDUCE-6515:
--
\\
\\
| (x) *{color:red}-1 overall{color}* |
\\
\\
|| Vote || Subsystem || Runtime || Comment ||
| {color:blue}0{color} | pre-patch | 20m 3s | Pre-patch trunk compilation is
healthy. |
| {color:green}+1{color} | @author | 0m 0s | The patch does not contain any
@author tags. |
| {color:green}+1{color} | tests included | 0m 0s | The patch appears to
include 5 new or modified test files. |
| {color:green}+1{color} | javac | 8m 4s | There were no new javac warning
messages. |
| {color:green}+1{color} | javadoc | 10m 36s | There were no new javadoc
warning messages. |
| {color:green}+1{color} | release audit | 0m 25s | The applied patch does
not increase the total number of release audit warnings. |
| {color:green}+1{color} | checkstyle | 2m 7s | There were no new checkstyle
issues. |
| {color:red}-1{color} | whitespace | 0m 7s | The patch has 3 line(s) that
end in whitespace. Use git apply --whitespace=fix. |
| {color:green}+1{color} | install | 1m 32s | mvn install still works. |
| {color:green}+1{color} | eclipse:eclipse | 0m 37s | The patch built with
eclipse:eclipse. |
| {color:red}-1{color} | findbugs | 4m 38s | The patch appears to introduce 3
new Findbugs (version 3.0.0) warnings. |
| {color:green}+1{color} | mapreduce tests | 9m 54s | Tests passed in
hadoop-mapreduce-client-app. |
| {color:green}+1{color} | mapreduce tests | 0m 49s | Tests passed in
hadoop-mapreduce-client-common. |
| {color:green}+1{color} | mapreduce tests | 1m 57s | Tests passed in
hadoop-mapreduce-client-core. |
| {color:green}+1{color} | mapreduce tests | 6m 4s | Tests passed in
hadoop-mapreduce-client-hs. |
| | | 66m 58s | |
\\
\\
|| Reason || Tests ||
| FindBugs | module:hadoop-mapreduce-client-common |
\\
\\
|| Subsystem || Report/Notes ||
| Patch URL |
http://issues.apache.org/jira/secure/attachment/12767989/0002-MAPREDUCE-6515.patch
|
| Optional Tests | javadoc javac unit findbugs checkstyle |
| git revision | trunk / 381610d |
| whitespace |
https://builds.apache.org/job/PreCommit-MAPREDUCE-Build/6082/artifact/patchprocess/whitespace.txt
|
| Findbugs warnings |
https://builds.apache.org/job/PreCommit-MAPREDUCE-Build/6082/artifact/patchprocess/newPatchFindbugsWarningshadoop-mapreduce-client-common.html
|
| hadoop-mapreduce-client-app test log |
https://builds.apache.org/job/PreCommit-MAPREDUCE-Build/6082/artifact/patchprocess/testrun_hadoop-mapreduce-client-app.txt
|
| hadoop-mapreduce-client-common test log |
https://builds.apache.org/job/PreCommit-MAPREDUCE-Build/6082/artifact/patchprocess/testrun_hadoop-mapreduce-client-common.txt
|
| hadoop-mapreduce-client-core test log |
https://builds.apache.org/job/PreCommit-MAPREDUCE-Build/6082/artifact/patchprocess/testrun_hadoop-mapreduce-client-core.txt
|
| hadoop-mapreduce-client-hs test log |
https://builds.apache.org/job/PreCommit-MAPREDUCE-Build/6082/artifact/patchprocess/testrun_hadoop-mapreduce-client-hs.txt
|
| Test Results |
https://builds.apache.org/job/PreCommit-MAPREDUCE-Build/6082/testReport/ |
| Java | 1.7.0_55 |
| uname | Linux asf905.gq1.ygridcore.net 3.13.0-36-lowlatency #63-Ubuntu SMP
PREEMPT Wed Sep 3 21:56:12 UTC 2014 x86_64 x86_64 x86_64 GNU/Linux |
| Console output |
https://builds.apache.org/job/PreCommit-MAPREDUCE-Build/6082/console |
This message was automatically generated.
> Update Application priority in AM side from AM-RM heartbeat
> ---
>
> Key: MAPREDUCE-6515
> URL: https://issues.apache.org/jira/browse/MAPREDUCE-6515
> Project: Hadoop Map/Reduce
> Issue Type: Bug
> Components: applicationmaster
>Reporter: Sunil G
>Assignee: Sunil G
> Attachments: 0001-MAPREDUCE-6515.patch, 0002-MAPREDUCE-6515.patch
>
>
> After YARN-4170, Application Priority is available via heartbeat call. Update
> this information in AM sothat client can fetch this information via JobStatus
> (JobReport) call.
> This is as per the discussion happened in MAPREDUCE-5870.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Created] (MAPREDUCE-6521) MiniMRYarnCluster should not create /tmp/hadoop-yarn/staging on local filesystem in unit test
Masatake Iwasaki created MAPREDUCE-6521:
---
Summary: MiniMRYarnCluster should not create
/tmp/hadoop-yarn/staging on local filesystem in unit test
Key: MAPREDUCE-6521
URL: https://issues.apache.org/jira/browse/MAPREDUCE-6521
Project: Hadoop Map/Reduce
Issue Type: Bug
Components: test
Reporter: Masatake Iwasaki
Assignee: Masatake Iwasaki
MiniMRYarnCluster create /tmp/hadoop-yarn/staging/history/done by default. It
should be under {{testWorkDir}} if the file system is localFs in order to make
it to be removed by {{mvn clean}}. It would also avoid issues under parallel
unit testing.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Created] (MAPREDUCE-6522) Need improved WARN or ERROR when token based auth fails for kmsclient request
Todd Grayson created MAPREDUCE-6522:
---
Summary: Need improved WARN or ERROR when token based auth fails
for kmsclient request
Key: MAPREDUCE-6522
URL: https://issues.apache.org/jira/browse/MAPREDUCE-6522
Project: Hadoop Map/Reduce
Issue Type: Improvement
Components: client, security
Reporter: Todd Grayson
When token based authentication fails, it would be helpful to have a WARN event
of the failure, as well as a WARN event that alternative forms of
authentication are being attempted.
For example if token based authentication has failed; it appears that there is
a fallback to attempting kerberos authentication. At that point the most
prominent logging is a kerberos GSS error, when the actual issue was a failure
at the token evaluation of a client access request to an HDFS encrypted zone.
In the example below we are presented with a kerberos error, but the actual
error was a failure of token authorization in an unexpected way.
{code}
15/08/27 07:35:35 INFO mapreduce.Job: Task Id :
attempt_1440594773177_0021_m_09_0, Status : FAILED
org.apache.hadoop.security.authentication.client.AuthenticationException:
GSSException: No valid credentials provided (Mechanism level: Failed to find
any Kerberos tgt)
java.io.IOException:
org.apache.hadoop.security.authentication.client.AuthenticationException:
GSSException: No valid credentials provided (Mechanism level: Failed to find
any Kerberos tgt)
at
{code}
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Commented] (MAPREDUCE-6499) Add More Informations Of Jobs In JobHistory Main Page
[ https://issues.apache.org/jira/browse/MAPREDUCE-6499?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=14969227#comment-14969227 ] Lin Yiqun commented on MAPREDUCE-6499: -- Steve Loughran,can you have time review my patch, test is all passed,thanks > Add More Informations Of Jobs In JobHistory Main Page > - > > Key: MAPREDUCE-6499 > URL: https://issues.apache.org/jira/browse/MAPREDUCE-6499 > Project: Hadoop Map/Reduce > Issue Type: Improvement >Affects Versions: 2.7.1 >Reporter: Lin Yiqun >Assignee: Lin Yiqun >Priority: Trivial > Attachments: HADOOP-MAPREDUCE-6499.2.patch, > HADOOP-MAPREDUCE-6499.3.patch, HADOOP-MAPREDUCE-6499.4.patch, > HADOOP-MAPREDUCE-6499.patch > > > Now in JobHistory Main Page show too little information about finished > jobs,even don't have the job's running total time,only > startTime,finishedTime.So,in jobHistory main page,we can add more > informations about jobs, that we can better analyze jobs. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (MAPREDUCE-6499) Add More Informations Of Jobs In JobHistory Main Page
[ https://issues.apache.org/jira/browse/MAPREDUCE-6499?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=14969228#comment-14969228 ] Lin Yiqun commented on MAPREDUCE-6499: -- [~ste...@apache.org] > Add More Informations Of Jobs In JobHistory Main Page > - > > Key: MAPREDUCE-6499 > URL: https://issues.apache.org/jira/browse/MAPREDUCE-6499 > Project: Hadoop Map/Reduce > Issue Type: Improvement >Affects Versions: 2.7.1 >Reporter: Lin Yiqun >Assignee: Lin Yiqun >Priority: Trivial > Attachments: HADOOP-MAPREDUCE-6499.2.patch, > HADOOP-MAPREDUCE-6499.3.patch, HADOOP-MAPREDUCE-6499.4.patch, > HADOOP-MAPREDUCE-6499.patch > > > Now in JobHistory Main Page show too little information about finished > jobs,even don't have the job's running total time,only > startTime,finishedTime.So,in jobHistory main page,we can add more > informations about jobs, that we can better analyze jobs. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Updated] (MAPREDUCE-5485) Allow repeating job commit by extending OutputCommitter API
[ https://issues.apache.org/jira/browse/MAPREDUCE-5485?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Junping Du updated MAPREDUCE-5485: -- Attachment: MAPREDUCE-5485-demo.patch Upload a demo patch first. More unit test will be added later. BTW, it adopt some code in MAPREDUCE-5718 with similar purpose, so please share the credit to the contributors of MAPREDUCE-5718 if we want to commit the following patches of this JIRA in future. > Allow repeating job commit by extending OutputCommitter API > --- > > Key: MAPREDUCE-5485 > URL: https://issues.apache.org/jira/browse/MAPREDUCE-5485 > Project: Hadoop Map/Reduce > Issue Type: Improvement >Affects Versions: 2.1.0-beta >Reporter: Nemon Lou >Assignee: Junping Du > Attachments: MAPREDUCE-5485-demo.patch > > > There are chances MRAppMaster crush during job committing,or NodeManager > restart cause the committing AM exit due to container expire.In these cases > ,the job will fail. > However,some jobs can redo commit so failing the job becomes unnecessary. > Let clients tell AM to allow redo commit or not is a better choice. > This idea comes from Jason Lowe's comments in MAPREDUCE-4819 -- This message was sent by Atlassian JIRA (v6.3.4#6332)
