[jira] [Commented] (MAPREDUCE-6316) Task Attempt List entries should link to the task overview
[ https://issues.apache.org/jira/browse/MAPREDUCE-6316?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14573833#comment-14573833 ] Gera Shegalov commented on MAPREDUCE-6316: -- The section is empty in Chrome as well but I did not look at the debugger. > Task Attempt List entries should link to the task overview > -- > > Key: MAPREDUCE-6316 > URL: https://issues.apache.org/jira/browse/MAPREDUCE-6316 > Project: Hadoop Map/Reduce > Issue Type: Improvement >Reporter: Siqi Li >Assignee: Siqi Li > Labels: BB2015-05-TBR > Attachments: AM attempt page.png, AM task page.png, All Attempts > page.png, MAPREDUCE-6316.v1.patch, MAPREDUCE-6316.v2.patch, > MAPREDUCE-6316.v3.patch, MAPREDUCE-6316.v4.patch, MAPREDUCE-6316.v5.patch, > MAPREDUCE-6316.v6.patch, Task Overview page.png > > > Typical workflow is to click on the list of failed attempts. Then you want to > look at the counters, or the list of attempts of just one task in general. If > each entry task attempt id linked the task id portion of it back to the task, > we would not have to go through the list of tasks to search for the task. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (MAPREDUCE-6316) Task Attempt List entries should link to the task overview
[ https://issues.apache.org/jira/browse/MAPREDUCE-6316?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14573811#comment-14573811 ] Siqi Li commented on MAPREDUCE-6316: [~jira.shegalov], Is it also happening to other web browsers ? > Task Attempt List entries should link to the task overview > -- > > Key: MAPREDUCE-6316 > URL: https://issues.apache.org/jira/browse/MAPREDUCE-6316 > Project: Hadoop Map/Reduce > Issue Type: Improvement >Reporter: Siqi Li >Assignee: Siqi Li > Labels: BB2015-05-TBR > Attachments: AM attempt page.png, AM task page.png, All Attempts > page.png, MAPREDUCE-6316.v1.patch, MAPREDUCE-6316.v2.patch, > MAPREDUCE-6316.v3.patch, MAPREDUCE-6316.v4.patch, MAPREDUCE-6316.v5.patch, > MAPREDUCE-6316.v6.patch, Task Overview page.png > > > Typical workflow is to click on the list of failed attempts. Then you want to > look at the counters, or the list of attempts of just one task in general. If > each entry task attempt id linked the task id portion of it back to the task, > we would not have to go through the list of tasks to search for the task. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Updated] (MAPREDUCE-6316) Task Attempt List entries should link to the task overview
[
https://issues.apache.org/jira/browse/MAPREDUCE-6316?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Gera Shegalov updated MAPREDUCE-6316:
-
Status: Open (was: Patch Available)
Thanks for v6 [~l201514]. The JHS pages look good. But AM's "running map
attempts" is corrupt. In the Safari debugger:
{code}
ReferenceError: Can't find variable: attemptsTableData
{code}
> Task Attempt List entries should link to the task overview
> --
>
> Key: MAPREDUCE-6316
> URL: https://issues.apache.org/jira/browse/MAPREDUCE-6316
> Project: Hadoop Map/Reduce
> Issue Type: Improvement
>Reporter: Siqi Li
>Assignee: Siqi Li
> Labels: BB2015-05-TBR
> Attachments: AM attempt page.png, AM task page.png, All Attempts
> page.png, MAPREDUCE-6316.v1.patch, MAPREDUCE-6316.v2.patch,
> MAPREDUCE-6316.v3.patch, MAPREDUCE-6316.v4.patch, MAPREDUCE-6316.v5.patch,
> MAPREDUCE-6316.v6.patch, Task Overview page.png
>
>
> Typical workflow is to click on the list of failed attempts. Then you want to
> look at the counters, or the list of attempts of just one task in general. If
> each entry task attempt id linked the task id portion of it back to the task,
> we would not have to go through the list of tasks to search for the task.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Commented] (MAPREDUCE-6387) Fix Task object to allow Jobs not submitted via distributed cache to survive rolling upgrade
[
https://issues.apache.org/jira/browse/MAPREDUCE-6387?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14573650#comment-14573650
]
Karthik Kambatla commented on MAPREDUCE-6387:
-
The fix looks safe to me. +1. Will go ahead and commit this first thing
tomorrow if no one objects.
> Fix Task object to allow Jobs not submitted via distributed cache to survive
> rolling upgrade
>
>
> Key: MAPREDUCE-6387
> URL: https://issues.apache.org/jira/browse/MAPREDUCE-6387
> Project: Hadoop Map/Reduce
> Issue Type: Bug
>Reporter: Arun Suresh
>Assignee: Arun Suresh
>Priority: Minor
> Attachments: MAPREDUCE-6387.1.patch
>
>
> There was a recent addition of an {{encryptedSpillKey}} to the Task object.
> And when serialized, this field was written out somewhere in the middle. This
> caused deployments that do not use DistributedCache to push job jars before
> running the job to fail rolling upgrade.
> Although deploying via Distributed Cache is the recommended method, there
> might still be deployments that use the node local classpath to pick up the
> MR framework classes (eg. for efficiency purposes, since this does not
> require the jar being copied to hdfs and then to all the nodes)
> Ensuring that it is the last field written and read when the Task object is
> serialized would alleviate this issue.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Updated] (MAPREDUCE-6354) ShuffleHandler should be able to log shuffle connections
[ https://issues.apache.org/jira/browse/MAPREDUCE-6354?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Jason Lowe updated MAPREDUCE-6354: -- Issue Type: Improvement (was: Bug) Summary: ShuffleHandler should be able to log shuffle connections (was: A) Hadoop Flags: Reviewed +1 latest patch lgtm. Will commit this tomorrow if there are no objections. > ShuffleHandler should be able to log shuffle connections > > > Key: MAPREDUCE-6354 > URL: https://issues.apache.org/jira/browse/MAPREDUCE-6354 > Project: Hadoop Map/Reduce > Issue Type: Improvement >Reporter: Chang Li >Assignee: Chang Li > Attachments: MAPREDUCE-6354.2.patch, MAPREDUCE-6354.3.patch, > MAPREDUCE-6354.4.patch, MAPREDUCE-6354.5.patch, MAPREDUCE-6354.6.patch, > MAPREDUCE-6354.7.patch, MAPREDUCE-6354.8.patch, MAPREDUCE-6354.patch > > > currently, shuffle handler only log connection info in debug mode, we want to > log that info in a more concise way -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Updated] (MAPREDUCE-6354) A
[ https://issues.apache.org/jira/browse/MAPREDUCE-6354?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Jason Lowe updated MAPREDUCE-6354: -- Summary: A (was: shuffle handler should log connection info) > A > - > > Key: MAPREDUCE-6354 > URL: https://issues.apache.org/jira/browse/MAPREDUCE-6354 > Project: Hadoop Map/Reduce > Issue Type: Bug >Reporter: Chang Li >Assignee: Chang Li > Attachments: MAPREDUCE-6354.2.patch, MAPREDUCE-6354.3.patch, > MAPREDUCE-6354.4.patch, MAPREDUCE-6354.5.patch, MAPREDUCE-6354.6.patch, > MAPREDUCE-6354.7.patch, MAPREDUCE-6354.8.patch, MAPREDUCE-6354.patch > > > currently, shuffle handler only log connection info in debug mode, we want to > log that info in a more concise way -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (MAPREDUCE-6354) shuffle handler should log connection info
[
https://issues.apache.org/jira/browse/MAPREDUCE-6354?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14573522#comment-14573522
]
Hadoop QA commented on MAPREDUCE-6354:
--
\\
\\
| (x) *{color:red}-1 overall{color}* |
\\
\\
|| Vote || Subsystem || Runtime || Comment ||
| {color:blue}0{color} | pre-patch | 16m 58s | Pre-patch trunk compilation is
healthy. |
| {color:green}+1{color} | @author | 0m 0s | The patch does not contain any
@author tags. |
| {color:red}-1{color} | tests included | 0m 0s | The patch doesn't appear
to include any new or modified tests. Please justify why no new tests are
needed for this patch. Also please list what manual steps were performed to
verify this patch. |
| {color:green}+1{color} | javac | 7m 34s | There were no new javac warning
messages. |
| {color:green}+1{color} | javadoc | 9m 34s | There were no new javadoc
warning messages. |
| {color:green}+1{color} | release audit | 0m 23s | The applied patch does
not increase the total number of release audit warnings. |
| {color:green}+1{color} | checkstyle | 1m 26s | There were no new checkstyle
issues. |
| {color:green}+1{color} | whitespace | 0m 0s | The patch has no lines that
end in whitespace. |
| {color:green}+1{color} | install | 1m 35s | mvn install still works. |
| {color:green}+1{color} | eclipse:eclipse | 0m 32s | The patch built with
eclipse:eclipse. |
| {color:green}+1{color} | findbugs | 2m 34s | The patch does not introduce
any new Findbugs (version 3.0.0) warnings. |
| {color:green}+1{color} | common tests | 23m 2s | Tests passed in
hadoop-common. |
| {color:green}+1{color} | mapreduce tests | 0m 20s | Tests passed in
hadoop-mapreduce-client-shuffle. |
| | | 64m 2s | |
\\
\\
|| Subsystem || Report/Notes ||
| Patch URL |
http://issues.apache.org/jira/secure/attachment/12737665/MAPREDUCE-6354.8.patch
|
| Optional Tests | javadoc javac unit findbugs checkstyle |
| git revision | trunk / 9f6d678 |
| hadoop-common test log |
https://builds.apache.org/job/PreCommit-MAPREDUCE-Build/5774/artifact/patchprocess/testrun_hadoop-common.txt
|
| hadoop-mapreduce-client-shuffle test log |
https://builds.apache.org/job/PreCommit-MAPREDUCE-Build/5774/artifact/patchprocess/testrun_hadoop-mapreduce-client-shuffle.txt
|
| Test Results |
https://builds.apache.org/job/PreCommit-MAPREDUCE-Build/5774/testReport/ |
| Java | 1.7.0_55 |
| uname | Linux asf905.gq1.ygridcore.net 3.13.0-36-lowlatency #63-Ubuntu SMP
PREEMPT Wed Sep 3 21:56:12 UTC 2014 x86_64 x86_64 x86_64 GNU/Linux |
| Console output |
https://builds.apache.org/job/PreCommit-MAPREDUCE-Build/5774/console |
This message was automatically generated.
> shuffle handler should log connection info
> --
>
> Key: MAPREDUCE-6354
> URL: https://issues.apache.org/jira/browse/MAPREDUCE-6354
> Project: Hadoop Map/Reduce
> Issue Type: Bug
>Reporter: Chang Li
>Assignee: Chang Li
> Attachments: MAPREDUCE-6354.2.patch, MAPREDUCE-6354.3.patch,
> MAPREDUCE-6354.4.patch, MAPREDUCE-6354.5.patch, MAPREDUCE-6354.6.patch,
> MAPREDUCE-6354.7.patch, MAPREDUCE-6354.8.patch, MAPREDUCE-6354.patch
>
>
> currently, shuffle handler only log connection info in debug mode, we want to
> log that info in a more concise way
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Commented] (MAPREDUCE-6387) Fix Task object to allow Jobs not submitted via distributed cache to survive rolling upgrade
[
https://issues.apache.org/jira/browse/MAPREDUCE-6387?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14573519#comment-14573519
]
Hadoop QA commented on MAPREDUCE-6387:
--
\\
\\
| (x) *{color:red}-1 overall{color}* |
\\
\\
|| Vote || Subsystem || Runtime || Comment ||
| {color:blue}0{color} | pre-patch | 16m 58s | Pre-patch trunk compilation is
healthy. |
| {color:green}+1{color} | @author | 0m 0s | The patch does not contain any
@author tags. |
| {color:red}-1{color} | tests included | 0m 0s | The patch doesn't appear
to include any new or modified tests. Please justify why no new tests are
needed for this patch. Also please list what manual steps were performed to
verify this patch. |
| {color:green}+1{color} | javac | 7m 59s | There were no new javac warning
messages. |
| {color:green}+1{color} | javadoc | 9m 39s | There were no new javadoc
warning messages. |
| {color:green}+1{color} | release audit | 0m 21s | The applied patch does
not increase the total number of release audit warnings. |
| {color:green}+1{color} | checkstyle | 0m 48s | There were no new checkstyle
issues. |
| {color:green}+1{color} | whitespace | 0m 0s | The patch has no lines that
end in whitespace. |
| {color:green}+1{color} | install | 1m 33s | mvn install still works. |
| {color:green}+1{color} | eclipse:eclipse | 0m 32s | The patch built with
eclipse:eclipse. |
| {color:green}+1{color} | findbugs | 1m 25s | The patch does not introduce
any new Findbugs (version 3.0.0) warnings. |
| {color:green}+1{color} | mapreduce tests | 1m 38s | Tests passed in
hadoop-mapreduce-client-core. |
| | | 40m 56s | |
\\
\\
|| Subsystem || Report/Notes ||
| Patch URL |
http://issues.apache.org/jira/secure/attachment/12737621/MAPREDUCE-6387.1.patch
|
| Optional Tests | javadoc javac unit findbugs checkstyle |
| git revision | trunk / 9f6d678 |
| hadoop-mapreduce-client-core test log |
https://builds.apache.org/job/PreCommit-MAPREDUCE-Build/5775/artifact/patchprocess/testrun_hadoop-mapreduce-client-core.txt
|
| Test Results |
https://builds.apache.org/job/PreCommit-MAPREDUCE-Build/5775/testReport/ |
| Java | 1.7.0_55 |
| uname | Linux asf903.gq1.ygridcore.net 3.13.0-36-lowlatency #63-Ubuntu SMP
PREEMPT Wed Sep 3 21:56:12 UTC 2014 x86_64 x86_64 x86_64 GNU/Linux |
| Console output |
https://builds.apache.org/job/PreCommit-MAPREDUCE-Build/5775/console |
This message was automatically generated.
> Fix Task object to allow Jobs not submitted via distributed cache to survive
> rolling upgrade
>
>
> Key: MAPREDUCE-6387
> URL: https://issues.apache.org/jira/browse/MAPREDUCE-6387
> Project: Hadoop Map/Reduce
> Issue Type: Bug
>Reporter: Arun Suresh
>Assignee: Arun Suresh
>Priority: Minor
> Attachments: MAPREDUCE-6387.1.patch
>
>
> There was a recent addition of an {{encryptedSpillKey}} to the Task object.
> And when serialized, this field was written out somewhere in the middle. This
> caused deployments that do not use DistributedCache to push job jars before
> running the job to fail rolling upgrade.
> Although deploying via Distributed Cache is the recommended method, there
> might still be deployments that use the node local classpath to pick up the
> MR framework classes (eg. for efficiency purposes, since this does not
> require the jar being copied to hdfs and then to all the nodes)
> Ensuring that it is the last field written and read when the Task object is
> serialized would alleviate this issue.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Commented] (MAPREDUCE-5232) log classpath and other key properties on child JVM start
[ https://issues.apache.org/jira/browse/MAPREDUCE-5232?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14573462#comment-14573462 ] Sangjin Lee commented on MAPREDUCE-5232: Thanks [~jira.shegalov]! > log classpath and other key properties on child JVM start > - > > Key: MAPREDUCE-5232 > URL: https://issues.apache.org/jira/browse/MAPREDUCE-5232 > Project: Hadoop Map/Reduce > Issue Type: Improvement > Components: mrv1, mrv2 >Affects Versions: 2.0.4-alpha >Reporter: Sangjin Lee >Assignee: Sangjin Lee > Fix For: 2.8.0 > > Attachments: MAPREDUCE-5232-branch-2.patch, > MAPREDUCE-5232-branch-2.patch, MAPREDUCE-5232-branch-2.patch, > MAPREDUCE-5232.patch, MAPREDUCE-5232.patch, MAPREDUCE-5232.patch > > > It would be great if we log vital information such as classpath, etc. upon a > mapreduce child JVM start. This would help a great deal in terms of > troubleshooting classpath issues, etc. Today it is pretty difficult to debug > this unless you preserve the container script. > Maybe it can log things like classpath, os name/version, java version, etc. > at the beginning of the child JVM start. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Updated] (MAPREDUCE-6387) Fix Task object to allow Jobs not submitted via distributed cache to survive rolling upgrade
[
https://issues.apache.org/jira/browse/MAPREDUCE-6387?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Arun Suresh updated MAPREDUCE-6387:
---
Status: Patch Available (was: Open)
> Fix Task object to allow Jobs not submitted via distributed cache to survive
> rolling upgrade
>
>
> Key: MAPREDUCE-6387
> URL: https://issues.apache.org/jira/browse/MAPREDUCE-6387
> Project: Hadoop Map/Reduce
> Issue Type: Bug
>Reporter: Arun Suresh
>Assignee: Arun Suresh
>Priority: Minor
> Attachments: MAPREDUCE-6387.1.patch
>
>
> There was a recent addition of an {{encryptedSpillKey}} to the Task object.
> And when serialized, this field was written out somewhere in the middle. This
> caused deployments that do not use DistributedCache to push job jars before
> running the job to fail rolling upgrade.
> Although deploying via Distributed Cache is the recommended method, there
> might still be deployments that use the node local classpath to pick up the
> MR framework classes (eg. for efficiency purposes, since this does not
> require the jar being copied to hdfs and then to all the nodes)
> Ensuring that it is the last field written and read when the Task object is
> serialized would alleviate this issue.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Updated] (MAPREDUCE-6354) shuffle handler should log connection info
[ https://issues.apache.org/jira/browse/MAPREDUCE-6354?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Chang Li updated MAPREDUCE-6354: Attachment: MAPREDUCE-6354.8.patch > shuffle handler should log connection info > -- > > Key: MAPREDUCE-6354 > URL: https://issues.apache.org/jira/browse/MAPREDUCE-6354 > Project: Hadoop Map/Reduce > Issue Type: Bug >Reporter: Chang Li >Assignee: Chang Li > Attachments: MAPREDUCE-6354.2.patch, MAPREDUCE-6354.3.patch, > MAPREDUCE-6354.4.patch, MAPREDUCE-6354.5.patch, MAPREDUCE-6354.6.patch, > MAPREDUCE-6354.7.patch, MAPREDUCE-6354.8.patch, MAPREDUCE-6354.patch > > > currently, shuffle handler only log connection info in debug mode, we want to > log that info in a more concise way -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (MAPREDUCE-6354) shuffle handler should log connection info
[ https://issues.apache.org/jira/browse/MAPREDUCE-6354?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14573328#comment-14573328 ] Jason Lowe commented on MAPREDUCE-6354: --- Thanks for updating the patch, Chang. The debug log should be wrapped in AUDITLOG.isDebugEnabled() so we don't perform the string processing if we aren't going to log it. Also, even though this wasn't technically part of this fix, please go ahead and fixup the trailing whitespace in log4j.properties that checkstyle is whining about. Otherwise it will continue to whine for any other patch that happens to touch nearby lines. > shuffle handler should log connection info > -- > > Key: MAPREDUCE-6354 > URL: https://issues.apache.org/jira/browse/MAPREDUCE-6354 > Project: Hadoop Map/Reduce > Issue Type: Bug >Reporter: Chang Li >Assignee: Chang Li > Attachments: MAPREDUCE-6354.2.patch, MAPREDUCE-6354.3.patch, > MAPREDUCE-6354.4.patch, MAPREDUCE-6354.5.patch, MAPREDUCE-6354.6.patch, > MAPREDUCE-6354.7.patch, MAPREDUCE-6354.patch > > > currently, shuffle handler only log connection info in debug mode, we want to > log that info in a more concise way -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Updated] (MAPREDUCE-5232) log classpath and other key properties on child JVM start
[ https://issues.apache.org/jira/browse/MAPREDUCE-5232?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Gera Shegalov updated MAPREDUCE-5232: - Fix Version/s: (was: 3.0.0) 2.8.0 > log classpath and other key properties on child JVM start > - > > Key: MAPREDUCE-5232 > URL: https://issues.apache.org/jira/browse/MAPREDUCE-5232 > Project: Hadoop Map/Reduce > Issue Type: Improvement > Components: mrv1, mrv2 >Affects Versions: 2.0.4-alpha >Reporter: Sangjin Lee >Assignee: Sangjin Lee > Fix For: 2.8.0 > > Attachments: MAPREDUCE-5232-branch-2.patch, > MAPREDUCE-5232-branch-2.patch, MAPREDUCE-5232-branch-2.patch, > MAPREDUCE-5232.patch, MAPREDUCE-5232.patch, MAPREDUCE-5232.patch > > > It would be great if we log vital information such as classpath, etc. upon a > mapreduce child JVM start. This would help a great deal in terms of > troubleshooting classpath issues, etc. Today it is pretty difficult to debug > this unless you preserve the container script. > Maybe it can log things like classpath, os name/version, java version, etc. > at the beginning of the child JVM start. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (MAPREDUCE-5232) log classpath and other key properties on child JVM start
[ https://issues.apache.org/jira/browse/MAPREDUCE-5232?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14573252#comment-14573252 ] Gera Shegalov commented on MAPREDUCE-5232: -- Merge conflicts are minor. I'll push it to branch-2. > log classpath and other key properties on child JVM start > - > > Key: MAPREDUCE-5232 > URL: https://issues.apache.org/jira/browse/MAPREDUCE-5232 > Project: Hadoop Map/Reduce > Issue Type: Improvement > Components: mrv1, mrv2 >Affects Versions: 2.0.4-alpha >Reporter: Sangjin Lee >Assignee: Sangjin Lee > Fix For: 3.0.0 > > Attachments: MAPREDUCE-5232-branch-2.patch, > MAPREDUCE-5232-branch-2.patch, MAPREDUCE-5232-branch-2.patch, > MAPREDUCE-5232.patch, MAPREDUCE-5232.patch, MAPREDUCE-5232.patch > > > It would be great if we log vital information such as classpath, etc. upon a > mapreduce child JVM start. This would help a great deal in terms of > troubleshooting classpath issues, etc. Today it is pretty difficult to debug > this unless you preserve the container script. > Maybe it can log things like classpath, os name/version, java version, etc. > at the beginning of the child JVM start. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (MAPREDUCE-3971) Job History web services need to have limits on the number of items they can return.
[ https://issues.apache.org/jira/browse/MAPREDUCE-3971?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14573204#comment-14573204 ] Ray Chiang commented on MAPREDUCE-3971: --- For .jhist parsing, I've added MAPREDUCE-6376. > Job History web services need to have limits on the number of items they can > return. > > > Key: MAPREDUCE-3971 > URL: https://issues.apache.org/jira/browse/MAPREDUCE-3971 > Project: Hadoop Map/Reduce > Issue Type: Sub-task > Components: mrv2 >Affects Versions: 0.23.2 >Reporter: Robert Joseph Evans > > The Job History web services canput a very large load on the job history > server. We should put in a limit on the number of entries that can be > returned by the web service, and also add in the ability to modify the > starting location in the list, so that all entries can still be downlaoded. > Just not all at once. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (MAPREDUCE-6384) add the last reporting reducer host info and attempt id on the map error message due to too many fetch failure
[
https://issues.apache.org/jira/browse/MAPREDUCE-6384?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14573150#comment-14573150
]
Hadoop QA commented on MAPREDUCE-6384:
--
\\
\\
| (x) *{color:red}-1 overall{color}* |
\\
\\
|| Vote || Subsystem || Runtime || Comment ||
| {color:red}-1{color} | pre-patch | 15m 47s | Pre-patch trunk has 1 extant
Findbugs (version 3.0.0) warnings. |
| {color:green}+1{color} | @author | 0m 0s | The patch does not contain any
@author tags. |
| {color:green}+1{color} | tests included | 0m 0s | The patch appears to
include 2 new or modified test files. |
| {color:red}-1{color} | javac | 7m 32s | The applied patch generated 1
additional warning messages. |
| {color:green}+1{color} | javadoc | 9m 35s | There were no new javadoc
warning messages. |
| {color:green}+1{color} | release audit | 0m 23s | The applied patch does
not increase the total number of release audit warnings. |
| {color:red}-1{color} | checkstyle | 0m 33s | The applied patch generated 2
new checkstyle issues (total was 357, now 356). |
| {color:red}-1{color} | whitespace | 0m 1s | The patch has 2 line(s) that
end in whitespace. Use git apply --whitespace=fix. |
| {color:green}+1{color} | install | 1m 34s | mvn install still works. |
| {color:green}+1{color} | eclipse:eclipse | 0m 33s | The patch built with
eclipse:eclipse. |
| {color:green}+1{color} | findbugs | 1m 6s | The patch does not introduce
any new Findbugs (version 3.0.0) warnings. |
| {color:green}+1{color} | mapreduce tests | 9m 36s | Tests passed in
hadoop-mapreduce-client-app. |
| | | 46m 47s | |
\\
\\
|| Subsystem || Report/Notes ||
| Patch URL |
http://issues.apache.org/jira/secure/attachment/12737611/MAPREDUCE-6384.4.2.patch
|
| Optional Tests | javadoc javac unit findbugs checkstyle |
| git revision | trunk / d7e7f6a |
| Pre-patch Findbugs warnings |
https://builds.apache.org/job/PreCommit-MAPREDUCE-Build/5773/artifact/patchprocess/trunkFindbugsWarningshadoop-mapreduce-client-app.html
|
| javac |
https://builds.apache.org/job/PreCommit-MAPREDUCE-Build/5773/artifact/patchprocess/diffJavacWarnings.txt
|
| checkstyle |
https://builds.apache.org/job/PreCommit-MAPREDUCE-Build/5773/artifact/patchprocess/diffcheckstylehadoop-mapreduce-client-app.txt
|
| whitespace |
https://builds.apache.org/job/PreCommit-MAPREDUCE-Build/5773/artifact/patchprocess/whitespace.txt
|
| hadoop-mapreduce-client-app test log |
https://builds.apache.org/job/PreCommit-MAPREDUCE-Build/5773/artifact/patchprocess/testrun_hadoop-mapreduce-client-app.txt
|
| Test Results |
https://builds.apache.org/job/PreCommit-MAPREDUCE-Build/5773/testReport/ |
| Java | 1.7.0_55 |
| uname | Linux asf903.gq1.ygridcore.net 3.13.0-36-lowlatency #63-Ubuntu SMP
PREEMPT Wed Sep 3 21:56:12 UTC 2014 x86_64 x86_64 x86_64 GNU/Linux |
| Console output |
https://builds.apache.org/job/PreCommit-MAPREDUCE-Build/5773/console |
This message was automatically generated.
> add the last reporting reducer host info and attempt id on the map error
> message due to too many fetch failure
> --
>
> Key: MAPREDUCE-6384
> URL: https://issues.apache.org/jira/browse/MAPREDUCE-6384
> Project: Hadoop Map/Reduce
> Issue Type: Bug
>Reporter: Chang Li
>Assignee: Chang Li
> Attachments: MAPREDUCE-6384.2.patch, MAPREDUCE-6384.3.patch,
> MAPREDUCE-6384.4.2.patch, MAPREDUCE-6384.4.patch
>
>
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Commented] (MAPREDUCE-6354) shuffle handler should log connection info
[
https://issues.apache.org/jira/browse/MAPREDUCE-6354?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14573146#comment-14573146
]
Hadoop QA commented on MAPREDUCE-6354:
--
\\
\\
| (x) *{color:red}-1 overall{color}* |
\\
\\
|| Vote || Subsystem || Runtime || Comment ||
| {color:blue}0{color} | pre-patch | 17m 0s | Pre-patch trunk compilation is
healthy. |
| {color:green}+1{color} | @author | 0m 0s | The patch does not contain any
@author tags. |
| {color:red}-1{color} | tests included | 0m 0s | The patch doesn't appear
to include any new or modified tests. Please justify why no new tests are
needed for this patch. Also please list what manual steps were performed to
verify this patch. |
| {color:green}+1{color} | javac | 7m 32s | There were no new javac warning
messages. |
| {color:green}+1{color} | javadoc | 9m 32s | There were no new javadoc
warning messages. |
| {color:green}+1{color} | release audit | 0m 23s | The applied patch does
not increase the total number of release audit warnings. |
| {color:green}+1{color} | checkstyle | 1m 27s | There were no new checkstyle
issues. |
| {color:red}-1{color} | whitespace | 0m 0s | The patch has 1 line(s) that
end in whitespace. Use git apply --whitespace=fix. |
| {color:green}+1{color} | install | 1m 32s | mvn install still works. |
| {color:green}+1{color} | eclipse:eclipse | 0m 33s | The patch built with
eclipse:eclipse. |
| {color:green}+1{color} | findbugs | 2m 34s | The patch does not introduce
any new Findbugs (version 3.0.0) warnings. |
| {color:green}+1{color} | common tests | 23m 1s | Tests passed in
hadoop-common. |
| {color:green}+1{color} | mapreduce tests | 0m 20s | Tests passed in
hadoop-mapreduce-client-shuffle. |
| | | 63m 58s | |
\\
\\
|| Subsystem || Report/Notes ||
| Patch URL |
http://issues.apache.org/jira/secure/attachment/12737602/MAPREDUCE-6354.7.patch
|
| Optional Tests | javadoc javac unit findbugs checkstyle |
| git revision | trunk / d7e7f6a |
| whitespace |
https://builds.apache.org/job/PreCommit-MAPREDUCE-Build/5772/artifact/patchprocess/whitespace.txt
|
| hadoop-common test log |
https://builds.apache.org/job/PreCommit-MAPREDUCE-Build/5772/artifact/patchprocess/testrun_hadoop-common.txt
|
| hadoop-mapreduce-client-shuffle test log |
https://builds.apache.org/job/PreCommit-MAPREDUCE-Build/5772/artifact/patchprocess/testrun_hadoop-mapreduce-client-shuffle.txt
|
| Test Results |
https://builds.apache.org/job/PreCommit-MAPREDUCE-Build/5772/testReport/ |
| Java | 1.7.0_55 |
| uname | Linux asf905.gq1.ygridcore.net 3.13.0-36-lowlatency #63-Ubuntu SMP
PREEMPT Wed Sep 3 21:56:12 UTC 2014 x86_64 x86_64 x86_64 GNU/Linux |
| Console output |
https://builds.apache.org/job/PreCommit-MAPREDUCE-Build/5772/console |
This message was automatically generated.
> shuffle handler should log connection info
> --
>
> Key: MAPREDUCE-6354
> URL: https://issues.apache.org/jira/browse/MAPREDUCE-6354
> Project: Hadoop Map/Reduce
> Issue Type: Bug
>Reporter: Chang Li
>Assignee: Chang Li
> Attachments: MAPREDUCE-6354.2.patch, MAPREDUCE-6354.3.patch,
> MAPREDUCE-6354.4.patch, MAPREDUCE-6354.5.patch, MAPREDUCE-6354.6.patch,
> MAPREDUCE-6354.7.patch, MAPREDUCE-6354.patch
>
>
> currently, shuffle handler only log connection info in debug mode, we want to
> log that info in a more concise way
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Updated] (MAPREDUCE-6387) Fix Task object to allow Jobs not submitted via distributed cache to survive rolling upgrade
[
https://issues.apache.org/jira/browse/MAPREDUCE-6387?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Arun Suresh updated MAPREDUCE-6387:
---
Attachment: MAPREDUCE-6387.1.patch
Attaching trivial patch to fix this.
Not adding test-cases since it is just reordering of a few fields
> Fix Task object to allow Jobs not submitted via distributed cache to survive
> rolling upgrade
>
>
> Key: MAPREDUCE-6387
> URL: https://issues.apache.org/jira/browse/MAPREDUCE-6387
> Project: Hadoop Map/Reduce
> Issue Type: Bug
>Reporter: Arun Suresh
>Assignee: Arun Suresh
>Priority: Minor
> Attachments: MAPREDUCE-6387.1.patch
>
>
> There was a recent addition of an {{encryptedSpillKey}} to the Task object.
> And when serialized, this field was written out somewhere in the middle. This
> caused deployments that do not use DistributedCache to push job jars before
> running the job to fail rolling upgrade.
> Although deploying via Distributed Cache is the recommended method, there
> might still be deployments that use the node local classpath to pick up the
> MR framework classes (eg. for efficiency purposes, since this does not
> require the jar being copied to hdfs and then to all the nodes)
> Ensuring that it is the last field written and read when the Task object is
> serialized would alleviate this issue.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Updated] (MAPREDUCE-6387) Fix Task object to allow Jobs not submitted via distributed cache to survive rolling upgrade
[
https://issues.apache.org/jira/browse/MAPREDUCE-6387?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Arun Suresh updated MAPREDUCE-6387:
---
Summary: Fix Task object to allow Jobs not submitted via distributed cache
to survive rolling upgrade (was: Fix Task object to serialize
encryptedSpillKey at the end to to allow rolling upgrades without distributed
cache to work)
> Fix Task object to allow Jobs not submitted via distributed cache to survive
> rolling upgrade
>
>
> Key: MAPREDUCE-6387
> URL: https://issues.apache.org/jira/browse/MAPREDUCE-6387
> Project: Hadoop Map/Reduce
> Issue Type: Bug
>Reporter: Arun Suresh
>Assignee: Arun Suresh
>Priority: Minor
>
> There was a recent addition of an {{encryptedSpillKey}} to the Task object.
> And when serialized, this field was written out somewhere in the middle. This
> caused deployments that do not use DistributedCache to push job jars before
> running the job to fail rolling upgrade.
> Although deploying via Distributed Cache is the recommended method, there
> might still be deployments that use the node local classpath to pick up the
> MR framework classes (eg. for efficiency purposes, since this does not
> require the jar being copied to hdfs and then to all the nodes)
> Ensuring that it is the last field written and read when the Task object is
> serialized would alleviate this issue.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Created] (MAPREDUCE-6387) Fix Task.java to serialize encryptedSpillKey at the end to to allow rolling upgrades without distributed cache
Arun Suresh created MAPREDUCE-6387:
--
Summary: Fix Task.java to serialize encryptedSpillKey at the end
to to allow rolling upgrades without distributed cache
Key: MAPREDUCE-6387
URL: https://issues.apache.org/jira/browse/MAPREDUCE-6387
Project: Hadoop Map/Reduce
Issue Type: Bug
Reporter: Arun Suresh
Assignee: Arun Suresh
Priority: Minor
There was a recent addition of an {{encryptedSpillKey}} to the Task object. And
when serialized, this field was written out somewhere in the middle. This
caused deployments that do not use DistributedCache to push job jars before
running the job to fail rolling upgrade.
Although deploying via Distributed Cache is the recommended method, there might
still be deployments that use the node local classpath to pick up the MR
framework classes (eg. for efficiency purposes, since this does not require the
jar being copied to hdfs and then to all the nodes)
Ensuring that it is the last field written and read when the Task object is
serialized would alleviate this issue.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Updated] (MAPREDUCE-6387) Fix Task object to serialize encryptedSpillKey at the end to to allow rolling upgrades without distributed cache to work
[
https://issues.apache.org/jira/browse/MAPREDUCE-6387?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Arun Suresh updated MAPREDUCE-6387:
---
Summary: Fix Task object to serialize encryptedSpillKey at the end to to
allow rolling upgrades without distributed cache to work (was: Fix Task.java
to serialize encryptedSpillKey at the end to to allow rolling upgrades without
distributed cache)
> Fix Task object to serialize encryptedSpillKey at the end to to allow rolling
> upgrades without distributed cache to work
>
>
> Key: MAPREDUCE-6387
> URL: https://issues.apache.org/jira/browse/MAPREDUCE-6387
> Project: Hadoop Map/Reduce
> Issue Type: Bug
>Reporter: Arun Suresh
>Assignee: Arun Suresh
>Priority: Minor
>
> There was a recent addition of an {{encryptedSpillKey}} to the Task object.
> And when serialized, this field was written out somewhere in the middle. This
> caused deployments that do not use DistributedCache to push job jars before
> running the job to fail rolling upgrade.
> Although deploying via Distributed Cache is the recommended method, there
> might still be deployments that use the node local classpath to pick up the
> MR framework classes (eg. for efficiency purposes, since this does not
> require the jar being copied to hdfs and then to all the nodes)
> Ensuring that it is the last field written and read when the Task object is
> serialized would alleviate this issue.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Updated] (MAPREDUCE-6384) add the last reporting reducer host info and attempt id on the map error message due to too many fetch failure
[ https://issues.apache.org/jira/browse/MAPREDUCE-6384?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Chang Li updated MAPREDUCE-6384: Attachment: MAPREDUCE-6384.4.2.patch > add the last reporting reducer host info and attempt id on the map error > message due to too many fetch failure > -- > > Key: MAPREDUCE-6384 > URL: https://issues.apache.org/jira/browse/MAPREDUCE-6384 > Project: Hadoop Map/Reduce > Issue Type: Bug >Reporter: Chang Li >Assignee: Chang Li > Attachments: MAPREDUCE-6384.2.patch, MAPREDUCE-6384.3.patch, > MAPREDUCE-6384.4.2.patch, MAPREDUCE-6384.4.patch > > -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (MAPREDUCE-6384) add the last reporting reducer host info and attempt id on the map error message due to too many fetch failure
[
https://issues.apache.org/jira/browse/MAPREDUCE-6384?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14573055#comment-14573055
]
Hadoop QA commented on MAPREDUCE-6384:
--
\\
\\
| (x) *{color:red}-1 overall{color}* |
\\
\\
|| Vote || Subsystem || Runtime || Comment ||
| {color:red}-1{color} | pre-patch | 15m 40s | Pre-patch trunk has 1 extant
Findbugs (version 3.0.0) warnings. |
| {color:green}+1{color} | @author | 0m 0s | The patch does not contain any
@author tags. |
| {color:green}+1{color} | tests included | 0m 0s | The patch appears to
include 2 new or modified test files. |
| {color:red}-1{color} | javac | 7m 33s | The applied patch generated 1
additional warning messages. |
| {color:green}+1{color} | javadoc | 9m 36s | There were no new javadoc
warning messages. |
| {color:green}+1{color} | release audit | 0m 23s | The applied patch does
not increase the total number of release audit warnings. |
| {color:red}-1{color} | checkstyle | 0m 34s | The applied patch generated 3
new checkstyle issues (total was 357, now 357). |
| {color:red}-1{color} | whitespace | 0m 1s | The patch has 2 line(s) that
end in whitespace. Use git apply --whitespace=fix. |
| {color:green}+1{color} | install | 1m 35s | mvn install still works. |
| {color:green}+1{color} | eclipse:eclipse | 0m 32s | The patch built with
eclipse:eclipse. |
| {color:green}+1{color} | findbugs | 1m 7s | The patch does not introduce
any new Findbugs (version 3.0.0) warnings. |
| {color:green}+1{color} | mapreduce tests | 8m 59s | Tests passed in
hadoop-mapreduce-client-app. |
| | | 46m 4s | |
\\
\\
|| Subsystem || Report/Notes ||
| Patch URL |
http://issues.apache.org/jira/secure/attachment/12737592/MAPREDUCE-6384.4.patch
|
| Optional Tests | javadoc javac unit findbugs checkstyle |
| git revision | trunk / d7e7f6a |
| Pre-patch Findbugs warnings |
https://builds.apache.org/job/PreCommit-MAPREDUCE-Build/5771/artifact/patchprocess/trunkFindbugsWarningshadoop-mapreduce-client-app.html
|
| javac |
https://builds.apache.org/job/PreCommit-MAPREDUCE-Build/5771/artifact/patchprocess/diffJavacWarnings.txt
|
| checkstyle |
https://builds.apache.org/job/PreCommit-MAPREDUCE-Build/5771/artifact/patchprocess/diffcheckstylehadoop-mapreduce-client-app.txt
|
| whitespace |
https://builds.apache.org/job/PreCommit-MAPREDUCE-Build/5771/artifact/patchprocess/whitespace.txt
|
| hadoop-mapreduce-client-app test log |
https://builds.apache.org/job/PreCommit-MAPREDUCE-Build/5771/artifact/patchprocess/testrun_hadoop-mapreduce-client-app.txt
|
| Test Results |
https://builds.apache.org/job/PreCommit-MAPREDUCE-Build/5771/testReport/ |
| Java | 1.7.0_55 |
| uname | Linux asf901.gq1.ygridcore.net 3.13.0-36-lowlatency #63-Ubuntu SMP
PREEMPT Wed Sep 3 21:56:12 UTC 2014 x86_64 x86_64 x86_64 GNU/Linux |
| Console output |
https://builds.apache.org/job/PreCommit-MAPREDUCE-Build/5771/console |
This message was automatically generated.
> add the last reporting reducer host info and attempt id on the map error
> message due to too many fetch failure
> --
>
> Key: MAPREDUCE-6384
> URL: https://issues.apache.org/jira/browse/MAPREDUCE-6384
> Project: Hadoop Map/Reduce
> Issue Type: Bug
>Reporter: Chang Li
>Assignee: Chang Li
> Attachments: MAPREDUCE-6384.2.patch, MAPREDUCE-6384.3.patch,
> MAPREDUCE-6384.4.patch
>
>
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Commented] (MAPREDUCE-6354) shuffle handler should log connection info
[ https://issues.apache.org/jira/browse/MAPREDUCE-6354?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14573047#comment-14573047 ] Chang Li commented on MAPREDUCE-6354: - [~jlowe] thanks a lot for thoughtful review and hearty discussion of how to make this logging more efficient! I have made changes of debug level logging. As for the trace level logging, I think we could wait till this get committed and file another jira to address that issue. Let me know what you think of the latest patch. Thanks! > shuffle handler should log connection info > -- > > Key: MAPREDUCE-6354 > URL: https://issues.apache.org/jira/browse/MAPREDUCE-6354 > Project: Hadoop Map/Reduce > Issue Type: Bug >Reporter: Chang Li >Assignee: Chang Li > Attachments: MAPREDUCE-6354.2.patch, MAPREDUCE-6354.3.patch, > MAPREDUCE-6354.4.patch, MAPREDUCE-6354.5.patch, MAPREDUCE-6354.6.patch, > MAPREDUCE-6354.7.patch, MAPREDUCE-6354.patch > > > currently, shuffle handler only log connection info in debug mode, we want to > log that info in a more concise way -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (MAPREDUCE-6382) Don't escape HTML links in Diagnostics in JHS job overview
[ https://issues.apache.org/jira/browse/MAPREDUCE-6382?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14573035#comment-14573035 ] Hudson commented on MAPREDUCE-6382: --- FAILURE: Integrated in Hadoop-Mapreduce-trunk #2164 (See [https://builds.apache.org/job/Hadoop-Mapreduce-trunk/2164/]) MAPREDUCE-6382. Don't escape HTML links in Diagnostics in JHS job overview. (Siqi Li via gera) (gera: rev e8302071ff16247a3d0cfc2dcca02da1bbf4db18) * hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-hs/src/main/java/org/apache/hadoop/mapreduce/v2/hs/webapp/HsJobBlock.java * hadoop-mapreduce-project/CHANGES.txt > Don't escape HTML links in Diagnostics in JHS job overview > -- > > Key: MAPREDUCE-6382 > URL: https://issues.apache.org/jira/browse/MAPREDUCE-6382 > Project: Hadoop Map/Reduce > Issue Type: Bug >Affects Versions: 2.8.0 >Reporter: Siqi Li >Assignee: Siqi Li > Fix For: 2.8.0 > > Attachments: MAPREDUCE-6382.v1.patch, links are escaped.jpg > > > for some reason, links are working properly in 2.4, but they are escaped in > 2.6 -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (MAPREDUCE-6174) Combine common stream code into parent class for InMemoryMapOutput and OnDiskMapOutput.
[ https://issues.apache.org/jira/browse/MAPREDUCE-6174?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14573037#comment-14573037 ] Hudson commented on MAPREDUCE-6174: --- FAILURE: Integrated in Hadoop-Mapreduce-trunk #2164 (See [https://builds.apache.org/job/Hadoop-Mapreduce-trunk/2164/]) MAPREDUCE-6174. Combine common stream code into parent class for InMemoryMapOutput and OnDiskMapOutput. (Eric Payne via gera) (gera: rev d90c13e2da8867661bf19a802add70145ab9a462) * hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-core/src/main/java/org/apache/hadoop/mapreduce/task/reduce/MergeManagerImpl.java * hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-core/src/main/java/org/apache/hadoop/mapreduce/task/reduce/OnDiskMapOutput.java * hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-core/src/main/java/org/apache/hadoop/mapreduce/task/reduce/InMemoryMapOutput.java * hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-core/src/main/java/org/apache/hadoop/mapreduce/task/reduce/IFileWrappedMapOutput.java * hadoop-mapreduce-project/CHANGES.txt * hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-core/src/test/java/org/apache/hadoop/mapreduce/task/reduce/TestFetcher.java > Combine common stream code into parent class for InMemoryMapOutput and > OnDiskMapOutput. > --- > > Key: MAPREDUCE-6174 > URL: https://issues.apache.org/jira/browse/MAPREDUCE-6174 > Project: Hadoop Map/Reduce > Issue Type: Improvement > Components: mrv2 >Affects Versions: 3.0.0, 2.6.0 >Reporter: Eric Payne >Assignee: Eric Payne > Labels: BB2015-05-RFC > Fix For: 2.8.0 > > Attachments: MAPREDUCE-6174.002.patch, MAPREDUCE-6174.003.patch, > MAPREDUCE-6174.004.patch, MAPREDUCE-6174.005.patch, MAPREDUCE-6174.006.patch, > MAPREDUCE-6174.007.patch, MAPREDUCE-6174.v1.txt > > > Per MAPREDUCE-6166, both InMemoryMapOutput and OnDiskMapOutput will be doing > similar things with regards to IFile streams. > In order to make it explicit that InMemoryMapOutput and OnDiskMapOutput are > different from 3rd-party implementations, this JIRA will make them subclass a > common class (see > https://issues.apache.org/jira/browse/MAPREDUCE-6166?focusedCommentId=14223368&page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel#comment-14223368) -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (MAPREDUCE-6374) Distributed Cache File visibility should check permission of full path
[ https://issues.apache.org/jira/browse/MAPREDUCE-6374?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14573038#comment-14573038 ] Hudson commented on MAPREDUCE-6374: --- FAILURE: Integrated in Hadoop-Mapreduce-trunk #2164 (See [https://builds.apache.org/job/Hadoop-Mapreduce-trunk/2164/]) MAPREDUCE-6374. Distributed Cache File visibility should check permission of full path. Contributed by Chang Li (jlowe: rev 107da29ff9b6efd02b21c96df60c58ac51590ebd) * hadoop-mapreduce-project/CHANGES.txt * hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-core/src/main/java/org/apache/hadoop/mapreduce/filecache/ClientDistributedCacheManager.java * hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-core/src/test/java/org/apache/hadoop/mapreduce/filecache/TestClientDistributedCacheManager.java > Distributed Cache File visibility should check permission of full path > -- > > Key: MAPREDUCE-6374 > URL: https://issues.apache.org/jira/browse/MAPREDUCE-6374 > Project: Hadoop Map/Reduce > Issue Type: Bug >Reporter: Chang Li >Assignee: Chang Li > Fix For: 2.8.0 > > Attachments: MAPREDUCE-6374.1.patch > > > should do full ancestor permission check for a relative cache file input -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (MAPREDUCE-5965) Hadoop streaming throws error if list of input files is high. Error is: "error=7, Argument list too long at if number of input file is high"
[
https://issues.apache.org/jira/browse/MAPREDUCE-5965?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14573036#comment-14573036
]
Hudson commented on MAPREDUCE-5965:
---
FAILURE: Integrated in Hadoop-Mapreduce-trunk #2164 (See
[https://builds.apache.org/job/Hadoop-Mapreduce-trunk/2164/])
MAPREDUCE-5965. Hadoop streaming throws error if list of input files is high.
Error is: "error=7, Argument list too long at if number of input file is high"
(wilfreds via rkanter) (rkanter: rev cc70df98e74142331043a611a3bd8a53ff6a2242)
*
hadoop-tools/hadoop-streaming/src/main/java/org/apache/hadoop/streaming/StreamJob.java
* hadoop-tools/hadoop-streaming/src/site/markdown/HadoopStreaming.md.vm
* hadoop-mapreduce-project/CHANGES.txt
*
hadoop-tools/hadoop-streaming/src/main/java/org/apache/hadoop/streaming/PipeMapRed.java
> Hadoop streaming throws error if list of input files is high. Error is:
> "error=7, Argument list too long at if number of input file is high"
>
>
> Key: MAPREDUCE-5965
> URL: https://issues.apache.org/jira/browse/MAPREDUCE-5965
> Project: Hadoop Map/Reduce
> Issue Type: Bug
>Reporter: Arup Malakar
>Assignee: Wilfred Spiegelenburg
> Fix For: 2.8.0
>
> Attachments: MAPREDUCE-5965.1.patch, MAPREDUCE-5965.2.patch,
> MAPREDUCE-5965.3.patch, MAPREDUCE-5965.patch
>
>
> Hadoop streaming exposes all the key values in job conf as environment
> variables when it forks a process for streaming code to run. Unfortunately
> the variable mapreduce_input_fileinputformat_inputdir contains the list of
> input files, and Linux has a limit on size of environment variables +
> arguments.
> Based on how long the list of files and their full path is this could be
> pretty huge. And given all of these variables are not even used it stops user
> from running hadoop job with large number of files, even though it could be
> run.
> Linux throws E2BIG if the size is greater than certain size which is error
> code 7. And java translates that to "error=7, Argument list too long". More:
> http://man7.org/linux/man-pages/man2/execve.2.html I suggest skipping
> variables if it is greater than certain length. That way if user code
> requires the environment variable it would fail. It should also introduce a
> config variable to skip long variables, and set it to false by default. That
> way user has to specifically set it to true to invoke this feature.
> Here is the exception:
> {code}
> Error: java.lang.RuntimeException: Error in configuring object at
> org.apache.hadoop.util.ReflectionUtils.setJobConf(ReflectionUtils.java:109)
> at org.apache.hadoop.util.ReflectionUtils.setConf(ReflectionUtils.java:75) at
> org.apache.hadoop.util.ReflectionUtils.newInstance(ReflectionUtils.java:133)
> at org.apache.hadoop.mapred.MapTask.runOldMapper(MapTask.java:426) at
> org.apache.hadoop.mapred.MapTask.run(MapTask.java:342) at
> org.apache.hadoop.mapred.YarnChild$2.run(YarnChild.java:168) at
> java.security.AccessController.doPrivileged(Native Method) at
> javax.security.auth.Subject.doAs(Subject.java:415) at
> org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1548)
> at org.apache.hadoop.mapred.YarnChild.main(YarnChild.java:163) Caused by:
> java.lang.reflect.InvocationTargetException at
> sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method) at
> sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:57)
> at
> sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
> at java.lang.reflect.Method.invoke(Method.java:606) at
> org.apache.hadoop.util.ReflectionUtils.setJobConf(ReflectionUtils.java:106)
> ... 9 more Caused by: java.lang.RuntimeException: Error in configuring object
> at
> org.apache.hadoop.util.ReflectionUtils.setJobConf(ReflectionUtils.java:109)
> at org.apache.hadoop.util.ReflectionUtils.setConf(ReflectionUtils.java:75) at
> org.apache.hadoop.util.ReflectionUtils.newInstance(ReflectionUtils.java:133)
> at org.apache.hadoop.mapred.MapRunner.configure(MapRunner.java:38) ... 14
> more Caused by: java.lang.reflect.InvocationTargetException at
> sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method) at
> sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:57)
> at
> sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
> at java.lang.reflect.Method.invoke(Method.java:606) at
> org.apache.hadoop.util.ReflectionUtils.setJobConf(ReflectionUtils.java:106)
> ... 17 more Caused by: java.lang.RuntimeException: configuration exception at
> org.apache.hadoop.streaming.PipeMapRed.configure(PipeMapRed.java:222) at
> org.apache.hadoop.streaming
[jira] [Updated] (MAPREDUCE-6354) shuffle handler should log connection info
[ https://issues.apache.org/jira/browse/MAPREDUCE-6354?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Chang Li updated MAPREDUCE-6354: Attachment: MAPREDUCE-6354.7.patch > shuffle handler should log connection info > -- > > Key: MAPREDUCE-6354 > URL: https://issues.apache.org/jira/browse/MAPREDUCE-6354 > Project: Hadoop Map/Reduce > Issue Type: Bug >Reporter: Chang Li >Assignee: Chang Li > Attachments: MAPREDUCE-6354.2.patch, MAPREDUCE-6354.3.patch, > MAPREDUCE-6354.4.patch, MAPREDUCE-6354.5.patch, MAPREDUCE-6354.6.patch, > MAPREDUCE-6354.7.patch, MAPREDUCE-6354.patch > > > currently, shuffle handler only log connection info in debug mode, we want to > log that info in a more concise way -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (MAPREDUCE-6382) Don't escape HTML links in Diagnostics in JHS job overview
[ https://issues.apache.org/jira/browse/MAPREDUCE-6382?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14573007#comment-14573007 ] Hudson commented on MAPREDUCE-6382: --- FAILURE: Integrated in Hadoop-Mapreduce-trunk-Java8 #216 (See [https://builds.apache.org/job/Hadoop-Mapreduce-trunk-Java8/216/]) MAPREDUCE-6382. Don't escape HTML links in Diagnostics in JHS job overview. (Siqi Li via gera) (gera: rev e8302071ff16247a3d0cfc2dcca02da1bbf4db18) * hadoop-mapreduce-project/CHANGES.txt * hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-hs/src/main/java/org/apache/hadoop/mapreduce/v2/hs/webapp/HsJobBlock.java > Don't escape HTML links in Diagnostics in JHS job overview > -- > > Key: MAPREDUCE-6382 > URL: https://issues.apache.org/jira/browse/MAPREDUCE-6382 > Project: Hadoop Map/Reduce > Issue Type: Bug >Affects Versions: 2.8.0 >Reporter: Siqi Li >Assignee: Siqi Li > Fix For: 2.8.0 > > Attachments: MAPREDUCE-6382.v1.patch, links are escaped.jpg > > > for some reason, links are working properly in 2.4, but they are escaped in > 2.6 -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (MAPREDUCE-5965) Hadoop streaming throws error if list of input files is high. Error is: "error=7, Argument list too long at if number of input file is high"
[
https://issues.apache.org/jira/browse/MAPREDUCE-5965?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14573008#comment-14573008
]
Hudson commented on MAPREDUCE-5965:
---
FAILURE: Integrated in Hadoop-Mapreduce-trunk-Java8 #216 (See
[https://builds.apache.org/job/Hadoop-Mapreduce-trunk-Java8/216/])
MAPREDUCE-5965. Hadoop streaming throws error if list of input files is high.
Error is: "error=7, Argument list too long at if number of input file is high"
(wilfreds via rkanter) (rkanter: rev cc70df98e74142331043a611a3bd8a53ff6a2242)
* hadoop-tools/hadoop-streaming/src/site/markdown/HadoopStreaming.md.vm
*
hadoop-tools/hadoop-streaming/src/main/java/org/apache/hadoop/streaming/StreamJob.java
*
hadoop-tools/hadoop-streaming/src/main/java/org/apache/hadoop/streaming/PipeMapRed.java
* hadoop-mapreduce-project/CHANGES.txt
> Hadoop streaming throws error if list of input files is high. Error is:
> "error=7, Argument list too long at if number of input file is high"
>
>
> Key: MAPREDUCE-5965
> URL: https://issues.apache.org/jira/browse/MAPREDUCE-5965
> Project: Hadoop Map/Reduce
> Issue Type: Bug
>Reporter: Arup Malakar
>Assignee: Wilfred Spiegelenburg
> Fix For: 2.8.0
>
> Attachments: MAPREDUCE-5965.1.patch, MAPREDUCE-5965.2.patch,
> MAPREDUCE-5965.3.patch, MAPREDUCE-5965.patch
>
>
> Hadoop streaming exposes all the key values in job conf as environment
> variables when it forks a process for streaming code to run. Unfortunately
> the variable mapreduce_input_fileinputformat_inputdir contains the list of
> input files, and Linux has a limit on size of environment variables +
> arguments.
> Based on how long the list of files and their full path is this could be
> pretty huge. And given all of these variables are not even used it stops user
> from running hadoop job with large number of files, even though it could be
> run.
> Linux throws E2BIG if the size is greater than certain size which is error
> code 7. And java translates that to "error=7, Argument list too long". More:
> http://man7.org/linux/man-pages/man2/execve.2.html I suggest skipping
> variables if it is greater than certain length. That way if user code
> requires the environment variable it would fail. It should also introduce a
> config variable to skip long variables, and set it to false by default. That
> way user has to specifically set it to true to invoke this feature.
> Here is the exception:
> {code}
> Error: java.lang.RuntimeException: Error in configuring object at
> org.apache.hadoop.util.ReflectionUtils.setJobConf(ReflectionUtils.java:109)
> at org.apache.hadoop.util.ReflectionUtils.setConf(ReflectionUtils.java:75) at
> org.apache.hadoop.util.ReflectionUtils.newInstance(ReflectionUtils.java:133)
> at org.apache.hadoop.mapred.MapTask.runOldMapper(MapTask.java:426) at
> org.apache.hadoop.mapred.MapTask.run(MapTask.java:342) at
> org.apache.hadoop.mapred.YarnChild$2.run(YarnChild.java:168) at
> java.security.AccessController.doPrivileged(Native Method) at
> javax.security.auth.Subject.doAs(Subject.java:415) at
> org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1548)
> at org.apache.hadoop.mapred.YarnChild.main(YarnChild.java:163) Caused by:
> java.lang.reflect.InvocationTargetException at
> sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method) at
> sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:57)
> at
> sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
> at java.lang.reflect.Method.invoke(Method.java:606) at
> org.apache.hadoop.util.ReflectionUtils.setJobConf(ReflectionUtils.java:106)
> ... 9 more Caused by: java.lang.RuntimeException: Error in configuring object
> at
> org.apache.hadoop.util.ReflectionUtils.setJobConf(ReflectionUtils.java:109)
> at org.apache.hadoop.util.ReflectionUtils.setConf(ReflectionUtils.java:75) at
> org.apache.hadoop.util.ReflectionUtils.newInstance(ReflectionUtils.java:133)
> at org.apache.hadoop.mapred.MapRunner.configure(MapRunner.java:38) ... 14
> more Caused by: java.lang.reflect.InvocationTargetException at
> sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method) at
> sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:57)
> at
> sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
> at java.lang.reflect.Method.invoke(Method.java:606) at
> org.apache.hadoop.util.ReflectionUtils.setJobConf(ReflectionUtils.java:106)
> ... 17 more Caused by: java.lang.RuntimeException: configuration exception at
> org.apache.hadoop.streaming.PipeMapRed.configure(PipeMapRed.java:222) at
> org.apache.hadoop
[jira] [Commented] (MAPREDUCE-6374) Distributed Cache File visibility should check permission of full path
[ https://issues.apache.org/jira/browse/MAPREDUCE-6374?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14573010#comment-14573010 ] Hudson commented on MAPREDUCE-6374: --- FAILURE: Integrated in Hadoop-Mapreduce-trunk-Java8 #216 (See [https://builds.apache.org/job/Hadoop-Mapreduce-trunk-Java8/216/]) MAPREDUCE-6374. Distributed Cache File visibility should check permission of full path. Contributed by Chang Li (jlowe: rev 107da29ff9b6efd02b21c96df60c58ac51590ebd) * hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-core/src/main/java/org/apache/hadoop/mapreduce/filecache/ClientDistributedCacheManager.java * hadoop-mapreduce-project/CHANGES.txt * hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-core/src/test/java/org/apache/hadoop/mapreduce/filecache/TestClientDistributedCacheManager.java > Distributed Cache File visibility should check permission of full path > -- > > Key: MAPREDUCE-6374 > URL: https://issues.apache.org/jira/browse/MAPREDUCE-6374 > Project: Hadoop Map/Reduce > Issue Type: Bug >Reporter: Chang Li >Assignee: Chang Li > Fix For: 2.8.0 > > Attachments: MAPREDUCE-6374.1.patch > > > should do full ancestor permission check for a relative cache file input -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (MAPREDUCE-6174) Combine common stream code into parent class for InMemoryMapOutput and OnDiskMapOutput.
[ https://issues.apache.org/jira/browse/MAPREDUCE-6174?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14573009#comment-14573009 ] Hudson commented on MAPREDUCE-6174: --- FAILURE: Integrated in Hadoop-Mapreduce-trunk-Java8 #216 (See [https://builds.apache.org/job/Hadoop-Mapreduce-trunk-Java8/216/]) MAPREDUCE-6174. Combine common stream code into parent class for InMemoryMapOutput and OnDiskMapOutput. (Eric Payne via gera) (gera: rev d90c13e2da8867661bf19a802add70145ab9a462) * hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-core/src/main/java/org/apache/hadoop/mapreduce/task/reduce/MergeManagerImpl.java * hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-core/src/main/java/org/apache/hadoop/mapreduce/task/reduce/OnDiskMapOutput.java * hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-core/src/main/java/org/apache/hadoop/mapreduce/task/reduce/InMemoryMapOutput.java * hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-core/src/test/java/org/apache/hadoop/mapreduce/task/reduce/TestFetcher.java * hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-core/src/main/java/org/apache/hadoop/mapreduce/task/reduce/IFileWrappedMapOutput.java * hadoop-mapreduce-project/CHANGES.txt > Combine common stream code into parent class for InMemoryMapOutput and > OnDiskMapOutput. > --- > > Key: MAPREDUCE-6174 > URL: https://issues.apache.org/jira/browse/MAPREDUCE-6174 > Project: Hadoop Map/Reduce > Issue Type: Improvement > Components: mrv2 >Affects Versions: 3.0.0, 2.6.0 >Reporter: Eric Payne >Assignee: Eric Payne > Labels: BB2015-05-RFC > Fix For: 2.8.0 > > Attachments: MAPREDUCE-6174.002.patch, MAPREDUCE-6174.003.patch, > MAPREDUCE-6174.004.patch, MAPREDUCE-6174.005.patch, MAPREDUCE-6174.006.patch, > MAPREDUCE-6174.007.patch, MAPREDUCE-6174.v1.txt > > > Per MAPREDUCE-6166, both InMemoryMapOutput and OnDiskMapOutput will be doing > similar things with regards to IFile streams. > In order to make it explicit that InMemoryMapOutput and OnDiskMapOutput are > different from 3rd-party implementations, this JIRA will make them subclass a > common class (see > https://issues.apache.org/jira/browse/MAPREDUCE-6166?focusedCommentId=14223368&page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel#comment-14223368) -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (MAPREDUCE-3418) If map output is not found, shuffle runs in tight loop
[ https://issues.apache.org/jira/browse/MAPREDUCE-3418?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14572996#comment-14572996 ] sahitya pavurala commented on MAPREDUCE-3418: - Is this issue resolved ? If not, i will look into this issue. > If map output is not found, shuffle runs in tight loop > -- > > Key: MAPREDUCE-3418 > URL: https://issues.apache.org/jira/browse/MAPREDUCE-3418 > Project: Hadoop Map/Reduce > Issue Type: Bug >Affects Versions: 0.23.0, 2.3.0 >Reporter: John George >Assignee: Chen He > > Sharad Agarwal bumped into this while simulating fetch failures. > Removed the map output directory. Shuffle runs in tight loop throwing > : > 2011-06-01 09:02:20,511 WARN org.apache.hadoop.mapreduce.task.reduce.Fetcher: > Invalid map id > java.lang.IllegalArgumentException: TaskAttemptId string : TTP/1.1 500 > Internal Server Error > Content-Type: text/plain; charset=UTF is not properly formed > at > org.apache.hadoop.mapreduce.TaskAttemptID.forName(TaskAttemptID.java:174) > at > org.apache.hadoop.mapreduce.task.reduce.Fetcher.copyMapOutput(Fetcher.java:284) > at > org.apache.hadoop.mapreduce.task.reduce.Fetcher.copyFromHost(Fetcher.java:251) > at org.apache.hadoop.mapreduce.task.reduce.Fetcher.run(Fetcher.java:149) > Fetch failure is not triggered. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Updated] (MAPREDUCE-6384) add the last reporting reducer host info and attempt id on the map error message due to too many fetch failure
[ https://issues.apache.org/jira/browse/MAPREDUCE-6384?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Chang Li updated MAPREDUCE-6384: Attachment: MAPREDUCE-6384.4.patch > add the last reporting reducer host info and attempt id on the map error > message due to too many fetch failure > -- > > Key: MAPREDUCE-6384 > URL: https://issues.apache.org/jira/browse/MAPREDUCE-6384 > Project: Hadoop Map/Reduce > Issue Type: Bug >Reporter: Chang Li >Assignee: Chang Li > Attachments: MAPREDUCE-6384.2.patch, MAPREDUCE-6384.3.patch, > MAPREDUCE-6384.4.patch > > -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (MAPREDUCE-6382) Don't escape HTML links in Diagnostics in JHS job overview
[ https://issues.apache.org/jira/browse/MAPREDUCE-6382?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14572890#comment-14572890 ] Hudson commented on MAPREDUCE-6382: --- SUCCESS: Integrated in Hadoop-Hdfs-trunk-Java8 #207 (See [https://builds.apache.org/job/Hadoop-Hdfs-trunk-Java8/207/]) MAPREDUCE-6382. Don't escape HTML links in Diagnostics in JHS job overview. (Siqi Li via gera) (gera: rev e8302071ff16247a3d0cfc2dcca02da1bbf4db18) * hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-hs/src/main/java/org/apache/hadoop/mapreduce/v2/hs/webapp/HsJobBlock.java * hadoop-mapreduce-project/CHANGES.txt > Don't escape HTML links in Diagnostics in JHS job overview > -- > > Key: MAPREDUCE-6382 > URL: https://issues.apache.org/jira/browse/MAPREDUCE-6382 > Project: Hadoop Map/Reduce > Issue Type: Bug >Affects Versions: 2.8.0 >Reporter: Siqi Li >Assignee: Siqi Li > Fix For: 2.8.0 > > Attachments: MAPREDUCE-6382.v1.patch, links are escaped.jpg > > > for some reason, links are working properly in 2.4, but they are escaped in > 2.6 -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (MAPREDUCE-6174) Combine common stream code into parent class for InMemoryMapOutput and OnDiskMapOutput.
[ https://issues.apache.org/jira/browse/MAPREDUCE-6174?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14572892#comment-14572892 ] Hudson commented on MAPREDUCE-6174: --- SUCCESS: Integrated in Hadoop-Hdfs-trunk-Java8 #207 (See [https://builds.apache.org/job/Hadoop-Hdfs-trunk-Java8/207/]) MAPREDUCE-6174. Combine common stream code into parent class for InMemoryMapOutput and OnDiskMapOutput. (Eric Payne via gera) (gera: rev d90c13e2da8867661bf19a802add70145ab9a462) * hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-core/src/main/java/org/apache/hadoop/mapreduce/task/reduce/IFileWrappedMapOutput.java * hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-core/src/test/java/org/apache/hadoop/mapreduce/task/reduce/TestFetcher.java * hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-core/src/main/java/org/apache/hadoop/mapreduce/task/reduce/InMemoryMapOutput.java * hadoop-mapreduce-project/CHANGES.txt * hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-core/src/main/java/org/apache/hadoop/mapreduce/task/reduce/MergeManagerImpl.java * hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-core/src/main/java/org/apache/hadoop/mapreduce/task/reduce/OnDiskMapOutput.java > Combine common stream code into parent class for InMemoryMapOutput and > OnDiskMapOutput. > --- > > Key: MAPREDUCE-6174 > URL: https://issues.apache.org/jira/browse/MAPREDUCE-6174 > Project: Hadoop Map/Reduce > Issue Type: Improvement > Components: mrv2 >Affects Versions: 3.0.0, 2.6.0 >Reporter: Eric Payne >Assignee: Eric Payne > Labels: BB2015-05-RFC > Fix For: 2.8.0 > > Attachments: MAPREDUCE-6174.002.patch, MAPREDUCE-6174.003.patch, > MAPREDUCE-6174.004.patch, MAPREDUCE-6174.005.patch, MAPREDUCE-6174.006.patch, > MAPREDUCE-6174.007.patch, MAPREDUCE-6174.v1.txt > > > Per MAPREDUCE-6166, both InMemoryMapOutput and OnDiskMapOutput will be doing > similar things with regards to IFile streams. > In order to make it explicit that InMemoryMapOutput and OnDiskMapOutput are > different from 3rd-party implementations, this JIRA will make them subclass a > common class (see > https://issues.apache.org/jira/browse/MAPREDUCE-6166?focusedCommentId=14223368&page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel#comment-14223368) -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (MAPREDUCE-6374) Distributed Cache File visibility should check permission of full path
[ https://issues.apache.org/jira/browse/MAPREDUCE-6374?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14572893#comment-14572893 ] Hudson commented on MAPREDUCE-6374: --- SUCCESS: Integrated in Hadoop-Hdfs-trunk-Java8 #207 (See [https://builds.apache.org/job/Hadoop-Hdfs-trunk-Java8/207/]) MAPREDUCE-6374. Distributed Cache File visibility should check permission of full path. Contributed by Chang Li (jlowe: rev 107da29ff9b6efd02b21c96df60c58ac51590ebd) * hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-core/src/main/java/org/apache/hadoop/mapreduce/filecache/ClientDistributedCacheManager.java * hadoop-mapreduce-project/CHANGES.txt * hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-core/src/test/java/org/apache/hadoop/mapreduce/filecache/TestClientDistributedCacheManager.java > Distributed Cache File visibility should check permission of full path > -- > > Key: MAPREDUCE-6374 > URL: https://issues.apache.org/jira/browse/MAPREDUCE-6374 > Project: Hadoop Map/Reduce > Issue Type: Bug >Reporter: Chang Li >Assignee: Chang Li > Fix For: 2.8.0 > > Attachments: MAPREDUCE-6374.1.patch > > > should do full ancestor permission check for a relative cache file input -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (MAPREDUCE-5965) Hadoop streaming throws error if list of input files is high. Error is: "error=7, Argument list too long at if number of input file is high"
[
https://issues.apache.org/jira/browse/MAPREDUCE-5965?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14572891#comment-14572891
]
Hudson commented on MAPREDUCE-5965:
---
SUCCESS: Integrated in Hadoop-Hdfs-trunk-Java8 #207 (See
[https://builds.apache.org/job/Hadoop-Hdfs-trunk-Java8/207/])
MAPREDUCE-5965. Hadoop streaming throws error if list of input files is high.
Error is: "error=7, Argument list too long at if number of input file is high"
(wilfreds via rkanter) (rkanter: rev cc70df98e74142331043a611a3bd8a53ff6a2242)
*
hadoop-tools/hadoop-streaming/src/main/java/org/apache/hadoop/streaming/StreamJob.java
* hadoop-tools/hadoop-streaming/src/site/markdown/HadoopStreaming.md.vm
* hadoop-mapreduce-project/CHANGES.txt
*
hadoop-tools/hadoop-streaming/src/main/java/org/apache/hadoop/streaming/PipeMapRed.java
> Hadoop streaming throws error if list of input files is high. Error is:
> "error=7, Argument list too long at if number of input file is high"
>
>
> Key: MAPREDUCE-5965
> URL: https://issues.apache.org/jira/browse/MAPREDUCE-5965
> Project: Hadoop Map/Reduce
> Issue Type: Bug
>Reporter: Arup Malakar
>Assignee: Wilfred Spiegelenburg
> Fix For: 2.8.0
>
> Attachments: MAPREDUCE-5965.1.patch, MAPREDUCE-5965.2.patch,
> MAPREDUCE-5965.3.patch, MAPREDUCE-5965.patch
>
>
> Hadoop streaming exposes all the key values in job conf as environment
> variables when it forks a process for streaming code to run. Unfortunately
> the variable mapreduce_input_fileinputformat_inputdir contains the list of
> input files, and Linux has a limit on size of environment variables +
> arguments.
> Based on how long the list of files and their full path is this could be
> pretty huge. And given all of these variables are not even used it stops user
> from running hadoop job with large number of files, even though it could be
> run.
> Linux throws E2BIG if the size is greater than certain size which is error
> code 7. And java translates that to "error=7, Argument list too long". More:
> http://man7.org/linux/man-pages/man2/execve.2.html I suggest skipping
> variables if it is greater than certain length. That way if user code
> requires the environment variable it would fail. It should also introduce a
> config variable to skip long variables, and set it to false by default. That
> way user has to specifically set it to true to invoke this feature.
> Here is the exception:
> {code}
> Error: java.lang.RuntimeException: Error in configuring object at
> org.apache.hadoop.util.ReflectionUtils.setJobConf(ReflectionUtils.java:109)
> at org.apache.hadoop.util.ReflectionUtils.setConf(ReflectionUtils.java:75) at
> org.apache.hadoop.util.ReflectionUtils.newInstance(ReflectionUtils.java:133)
> at org.apache.hadoop.mapred.MapTask.runOldMapper(MapTask.java:426) at
> org.apache.hadoop.mapred.MapTask.run(MapTask.java:342) at
> org.apache.hadoop.mapred.YarnChild$2.run(YarnChild.java:168) at
> java.security.AccessController.doPrivileged(Native Method) at
> javax.security.auth.Subject.doAs(Subject.java:415) at
> org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1548)
> at org.apache.hadoop.mapred.YarnChild.main(YarnChild.java:163) Caused by:
> java.lang.reflect.InvocationTargetException at
> sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method) at
> sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:57)
> at
> sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
> at java.lang.reflect.Method.invoke(Method.java:606) at
> org.apache.hadoop.util.ReflectionUtils.setJobConf(ReflectionUtils.java:106)
> ... 9 more Caused by: java.lang.RuntimeException: Error in configuring object
> at
> org.apache.hadoop.util.ReflectionUtils.setJobConf(ReflectionUtils.java:109)
> at org.apache.hadoop.util.ReflectionUtils.setConf(ReflectionUtils.java:75) at
> org.apache.hadoop.util.ReflectionUtils.newInstance(ReflectionUtils.java:133)
> at org.apache.hadoop.mapred.MapRunner.configure(MapRunner.java:38) ... 14
> more Caused by: java.lang.reflect.InvocationTargetException at
> sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method) at
> sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:57)
> at
> sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
> at java.lang.reflect.Method.invoke(Method.java:606) at
> org.apache.hadoop.util.ReflectionUtils.setJobConf(ReflectionUtils.java:106)
> ... 17 more Caused by: java.lang.RuntimeException: configuration exception at
> org.apache.hadoop.streaming.PipeMapRed.configure(PipeMapRed.java:222) at
> org.apache.hadoop.streaming
[jira] [Commented] (MAPREDUCE-5965) Hadoop streaming throws error if list of input files is high. Error is: "error=7, Argument list too long at if number of input file is high"
[
https://issues.apache.org/jira/browse/MAPREDUCE-5965?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14572863#comment-14572863
]
Hudson commented on MAPREDUCE-5965:
---
SUCCESS: Integrated in Hadoop-Hdfs-trunk #2146 (See
[https://builds.apache.org/job/Hadoop-Hdfs-trunk/2146/])
MAPREDUCE-5965. Hadoop streaming throws error if list of input files is high.
Error is: "error=7, Argument list too long at if number of input file is high"
(wilfreds via rkanter) (rkanter: rev cc70df98e74142331043a611a3bd8a53ff6a2242)
*
hadoop-tools/hadoop-streaming/src/main/java/org/apache/hadoop/streaming/PipeMapRed.java
*
hadoop-tools/hadoop-streaming/src/main/java/org/apache/hadoop/streaming/StreamJob.java
* hadoop-tools/hadoop-streaming/src/site/markdown/HadoopStreaming.md.vm
* hadoop-mapreduce-project/CHANGES.txt
> Hadoop streaming throws error if list of input files is high. Error is:
> "error=7, Argument list too long at if number of input file is high"
>
>
> Key: MAPREDUCE-5965
> URL: https://issues.apache.org/jira/browse/MAPREDUCE-5965
> Project: Hadoop Map/Reduce
> Issue Type: Bug
>Reporter: Arup Malakar
>Assignee: Wilfred Spiegelenburg
> Fix For: 2.8.0
>
> Attachments: MAPREDUCE-5965.1.patch, MAPREDUCE-5965.2.patch,
> MAPREDUCE-5965.3.patch, MAPREDUCE-5965.patch
>
>
> Hadoop streaming exposes all the key values in job conf as environment
> variables when it forks a process for streaming code to run. Unfortunately
> the variable mapreduce_input_fileinputformat_inputdir contains the list of
> input files, and Linux has a limit on size of environment variables +
> arguments.
> Based on how long the list of files and their full path is this could be
> pretty huge. And given all of these variables are not even used it stops user
> from running hadoop job with large number of files, even though it could be
> run.
> Linux throws E2BIG if the size is greater than certain size which is error
> code 7. And java translates that to "error=7, Argument list too long". More:
> http://man7.org/linux/man-pages/man2/execve.2.html I suggest skipping
> variables if it is greater than certain length. That way if user code
> requires the environment variable it would fail. It should also introduce a
> config variable to skip long variables, and set it to false by default. That
> way user has to specifically set it to true to invoke this feature.
> Here is the exception:
> {code}
> Error: java.lang.RuntimeException: Error in configuring object at
> org.apache.hadoop.util.ReflectionUtils.setJobConf(ReflectionUtils.java:109)
> at org.apache.hadoop.util.ReflectionUtils.setConf(ReflectionUtils.java:75) at
> org.apache.hadoop.util.ReflectionUtils.newInstance(ReflectionUtils.java:133)
> at org.apache.hadoop.mapred.MapTask.runOldMapper(MapTask.java:426) at
> org.apache.hadoop.mapred.MapTask.run(MapTask.java:342) at
> org.apache.hadoop.mapred.YarnChild$2.run(YarnChild.java:168) at
> java.security.AccessController.doPrivileged(Native Method) at
> javax.security.auth.Subject.doAs(Subject.java:415) at
> org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1548)
> at org.apache.hadoop.mapred.YarnChild.main(YarnChild.java:163) Caused by:
> java.lang.reflect.InvocationTargetException at
> sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method) at
> sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:57)
> at
> sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
> at java.lang.reflect.Method.invoke(Method.java:606) at
> org.apache.hadoop.util.ReflectionUtils.setJobConf(ReflectionUtils.java:106)
> ... 9 more Caused by: java.lang.RuntimeException: Error in configuring object
> at
> org.apache.hadoop.util.ReflectionUtils.setJobConf(ReflectionUtils.java:109)
> at org.apache.hadoop.util.ReflectionUtils.setConf(ReflectionUtils.java:75) at
> org.apache.hadoop.util.ReflectionUtils.newInstance(ReflectionUtils.java:133)
> at org.apache.hadoop.mapred.MapRunner.configure(MapRunner.java:38) ... 14
> more Caused by: java.lang.reflect.InvocationTargetException at
> sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method) at
> sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:57)
> at
> sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
> at java.lang.reflect.Method.invoke(Method.java:606) at
> org.apache.hadoop.util.ReflectionUtils.setJobConf(ReflectionUtils.java:106)
> ... 17 more Caused by: java.lang.RuntimeException: configuration exception at
> org.apache.hadoop.streaming.PipeMapRed.configure(PipeMapRed.java:222) at
> org.apache.hadoop.streaming.PipeMappe
[jira] [Commented] (MAPREDUCE-6374) Distributed Cache File visibility should check permission of full path
[ https://issues.apache.org/jira/browse/MAPREDUCE-6374?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14572865#comment-14572865 ] Hudson commented on MAPREDUCE-6374: --- SUCCESS: Integrated in Hadoop-Hdfs-trunk #2146 (See [https://builds.apache.org/job/Hadoop-Hdfs-trunk/2146/]) MAPREDUCE-6374. Distributed Cache File visibility should check permission of full path. Contributed by Chang Li (jlowe: rev 107da29ff9b6efd02b21c96df60c58ac51590ebd) * hadoop-mapreduce-project/CHANGES.txt * hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-core/src/main/java/org/apache/hadoop/mapreduce/filecache/ClientDistributedCacheManager.java * hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-core/src/test/java/org/apache/hadoop/mapreduce/filecache/TestClientDistributedCacheManager.java > Distributed Cache File visibility should check permission of full path > -- > > Key: MAPREDUCE-6374 > URL: https://issues.apache.org/jira/browse/MAPREDUCE-6374 > Project: Hadoop Map/Reduce > Issue Type: Bug >Reporter: Chang Li >Assignee: Chang Li > Fix For: 2.8.0 > > Attachments: MAPREDUCE-6374.1.patch > > > should do full ancestor permission check for a relative cache file input -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (MAPREDUCE-6174) Combine common stream code into parent class for InMemoryMapOutput and OnDiskMapOutput.
[ https://issues.apache.org/jira/browse/MAPREDUCE-6174?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14572864#comment-14572864 ] Hudson commented on MAPREDUCE-6174: --- SUCCESS: Integrated in Hadoop-Hdfs-trunk #2146 (See [https://builds.apache.org/job/Hadoop-Hdfs-trunk/2146/]) MAPREDUCE-6174. Combine common stream code into parent class for InMemoryMapOutput and OnDiskMapOutput. (Eric Payne via gera) (gera: rev d90c13e2da8867661bf19a802add70145ab9a462) * hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-core/src/main/java/org/apache/hadoop/mapreduce/task/reduce/IFileWrappedMapOutput.java * hadoop-mapreduce-project/CHANGES.txt * hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-core/src/main/java/org/apache/hadoop/mapreduce/task/reduce/InMemoryMapOutput.java * hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-core/src/main/java/org/apache/hadoop/mapreduce/task/reduce/OnDiskMapOutput.java * hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-core/src/test/java/org/apache/hadoop/mapreduce/task/reduce/TestFetcher.java * hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-core/src/main/java/org/apache/hadoop/mapreduce/task/reduce/MergeManagerImpl.java > Combine common stream code into parent class for InMemoryMapOutput and > OnDiskMapOutput. > --- > > Key: MAPREDUCE-6174 > URL: https://issues.apache.org/jira/browse/MAPREDUCE-6174 > Project: Hadoop Map/Reduce > Issue Type: Improvement > Components: mrv2 >Affects Versions: 3.0.0, 2.6.0 >Reporter: Eric Payne >Assignee: Eric Payne > Labels: BB2015-05-RFC > Fix For: 2.8.0 > > Attachments: MAPREDUCE-6174.002.patch, MAPREDUCE-6174.003.patch, > MAPREDUCE-6174.004.patch, MAPREDUCE-6174.005.patch, MAPREDUCE-6174.006.patch, > MAPREDUCE-6174.007.patch, MAPREDUCE-6174.v1.txt > > > Per MAPREDUCE-6166, both InMemoryMapOutput and OnDiskMapOutput will be doing > similar things with regards to IFile streams. > In order to make it explicit that InMemoryMapOutput and OnDiskMapOutput are > different from 3rd-party implementations, this JIRA will make them subclass a > common class (see > https://issues.apache.org/jira/browse/MAPREDUCE-6166?focusedCommentId=14223368&page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel#comment-14223368) -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (MAPREDUCE-6382) Don't escape HTML links in Diagnostics in JHS job overview
[ https://issues.apache.org/jira/browse/MAPREDUCE-6382?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14572862#comment-14572862 ] Hudson commented on MAPREDUCE-6382: --- SUCCESS: Integrated in Hadoop-Hdfs-trunk #2146 (See [https://builds.apache.org/job/Hadoop-Hdfs-trunk/2146/]) MAPREDUCE-6382. Don't escape HTML links in Diagnostics in JHS job overview. (Siqi Li via gera) (gera: rev e8302071ff16247a3d0cfc2dcca02da1bbf4db18) * hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-hs/src/main/java/org/apache/hadoop/mapreduce/v2/hs/webapp/HsJobBlock.java * hadoop-mapreduce-project/CHANGES.txt > Don't escape HTML links in Diagnostics in JHS job overview > -- > > Key: MAPREDUCE-6382 > URL: https://issues.apache.org/jira/browse/MAPREDUCE-6382 > Project: Hadoop Map/Reduce > Issue Type: Bug >Affects Versions: 2.8.0 >Reporter: Siqi Li >Assignee: Siqi Li > Fix For: 2.8.0 > > Attachments: MAPREDUCE-6382.v1.patch, links are escaped.jpg > > > for some reason, links are working properly in 2.4, but they are escaped in > 2.6 -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Assigned] (MAPREDUCE-6386) refreshLogRetentionSettings is invalid in secure cluster
[ https://issues.apache.org/jira/browse/MAPREDUCE-6386?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Varun Saxena reassigned MAPREDUCE-6386: --- Assignee: Varun Saxena > refreshLogRetentionSettings is invalid in secure cluster > > > Key: MAPREDUCE-6386 > URL: https://issues.apache.org/jira/browse/MAPREDUCE-6386 > Project: Hadoop Map/Reduce > Issue Type: Bug > Components: jobhistoryserver > Environment: mrV2, secure mode >Reporter: Zhang Wei >Assignee: Varun Saxena > > bin/mapred hsadmin -refreshLogRetentionSettings is not available in secure > cluster. > reproduce steps: > 1. startup historyserver in secure cluster. > 2. login the secure user with kinit.(the same user with startup step) > 3. execute bin/mapred hsadmin -refreshLogRetentionSettings to refresh. > there will be exceptions as follows in the historyserver's log if log > deletion is enabled. > 2015-06-04 14:14:40,070 | ERROR | Timer-3 | Error reading root log dir this > deletion attempt is being aborted | AggregatedLogDeletionService.java:127 > java.io.IOException: Failed on local exception: java.io.IOException: > javax.security.sasl.SaslException: GSS initiate failed [Caused by > GSSException: No valid credentials provided (Mechanism level: Failed to find > any Kerberos tgt)]; Host Details : local host is: "vm-31/9.91.12.31"; > destination host is: "vm-33":25000; > at org.apache.hadoop.net.NetUtils.wrapException(NetUtils.java:764) > at org.apache.hadoop.ipc.Client.call(Client.java:1414) > at org.apache.hadoop.ipc.Client.call(Client.java:1363) > at > org.apache.hadoop.ipc.ProtobufRpcEngine$Invoker.invoke(ProtobufRpcEngine.java:206) > at com.sun.proxy.$Proxy9.getListing(Unknown Source) > at > org.apache.hadoop.hdfs.protocolPB.ClientNamenodeProtocolTranslatorPB.getListing(ClientNamenodeProtocolTranslatorPB.java:519) > at sun.reflect.GeneratedMethodAccessor16.invoke(Unknown Source) > at > sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) > at java.lang.reflect.Method.invoke(Method.java:606) > at > org.apache.hadoop.io.retry.RetryInvocationHandler.invokeMethod(RetryInvocationHandler.java:187) > at > org.apache.hadoop.io.retry.RetryInvocationHandler.invoke(RetryInvocationHandler.java:102) > at com.sun.proxy.$Proxy10.getListing(Unknown Source) > at org.apache.hadoop.hdfs.DFSClient.listPaths(DFSClient.java:1767) > at org.apache.hadoop.hdfs.DFSClient.listPaths(DFSClient.java:1750) > at > org.apache.hadoop.hdfs.DistributedFileSystem.listStatusInternal(DistributedFileSystem.java:691) > at > org.apache.hadoop.hdfs.DistributedFileSystem.access$600(DistributedFileSystem.java:102) > at > org.apache.hadoop.hdfs.DistributedFileSystem$15.doCall(DistributedFileSystem.java:753) > at > org.apache.hadoop.hdfs.DistributedFileSystem$15.doCall(DistributedFileSystem.java:749) > at > org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) > at > org.apache.hadoop.hdfs.DistributedFileSystem.listStatus(DistributedFileSystem.java:749) > at > org.apache.hadoop.yarn.logaggregation.AggregatedLogDeletionService$LogDeletionTask.run(AggregatedLogDeletionService.java:68) > at java.util.TimerThread.mainLoop(Timer.java:555) > at java.util.TimerThread.run(Timer.java:505) > Caused by: java.io.IOException: javax.security.sasl.SaslException: GSS > initiate failed [Caused by GSSException: No valid credentials provided > (Mechanism level: Failed to find any Kerberos tgt)] > at org.apache.hadoop.ipc.Client$Connection$1.run(Client.java:677) > at java.security.AccessController.doPrivileged(Native Method) > at javax.security.auth.Subject.doAs(Subject.java:415) > at > org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1641) > at > org.apache.hadoop.ipc.Client$Connection.handleSaslConnectionFailure(Client.java:640) > at > org.apache.hadoop.ipc.Client$Connection.setupIOstreams(Client.java:724) > at > org.apache.hadoop.ipc.Client$Connection.access$2800(Client.java:367) > at org.apache.hadoop.ipc.Client.getConnection(Client.java:1462) > at org.apache.hadoop.ipc.Client.call(Client.java:1381) > ... 21 more > Caused by: javax.security.sasl.SaslException: GSS initiate failed [Caused by > GSSException: No valid credentials provided (Mechanism level: Failed to find > any Kerberos tgt)] > at > com.sun.security.sasl.gsskerb.GssKrb5Client.evaluateChallenge(GssKrb5Client.java:212) > at > org.apache.hadoop.security.SaslRpcClient.saslConnect(SaslRpcClient.java:411) > at > org.apache.ha
[jira] [Commented] (MAPREDUCE-6382) Don't escape HTML links in Diagnostics in JHS job overview
[ https://issues.apache.org/jira/browse/MAPREDUCE-6382?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14572612#comment-14572612 ] Hudson commented on MAPREDUCE-6382: --- FAILURE: Integrated in Hadoop-Yarn-trunk #948 (See [https://builds.apache.org/job/Hadoop-Yarn-trunk/948/]) MAPREDUCE-6382. Don't escape HTML links in Diagnostics in JHS job overview. (Siqi Li via gera) (gera: rev e8302071ff16247a3d0cfc2dcca02da1bbf4db18) * hadoop-mapreduce-project/CHANGES.txt * hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-hs/src/main/java/org/apache/hadoop/mapreduce/v2/hs/webapp/HsJobBlock.java > Don't escape HTML links in Diagnostics in JHS job overview > -- > > Key: MAPREDUCE-6382 > URL: https://issues.apache.org/jira/browse/MAPREDUCE-6382 > Project: Hadoop Map/Reduce > Issue Type: Bug >Affects Versions: 2.8.0 >Reporter: Siqi Li >Assignee: Siqi Li > Fix For: 2.8.0 > > Attachments: MAPREDUCE-6382.v1.patch, links are escaped.jpg > > > for some reason, links are working properly in 2.4, but they are escaped in > 2.6 -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (MAPREDUCE-6374) Distributed Cache File visibility should check permission of full path
[ https://issues.apache.org/jira/browse/MAPREDUCE-6374?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14572616#comment-14572616 ] Hudson commented on MAPREDUCE-6374: --- FAILURE: Integrated in Hadoop-Yarn-trunk #948 (See [https://builds.apache.org/job/Hadoop-Yarn-trunk/948/]) MAPREDUCE-6374. Distributed Cache File visibility should check permission of full path. Contributed by Chang Li (jlowe: rev 107da29ff9b6efd02b21c96df60c58ac51590ebd) * hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-core/src/test/java/org/apache/hadoop/mapreduce/filecache/TestClientDistributedCacheManager.java * hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-core/src/main/java/org/apache/hadoop/mapreduce/filecache/ClientDistributedCacheManager.java * hadoop-mapreduce-project/CHANGES.txt > Distributed Cache File visibility should check permission of full path > -- > > Key: MAPREDUCE-6374 > URL: https://issues.apache.org/jira/browse/MAPREDUCE-6374 > Project: Hadoop Map/Reduce > Issue Type: Bug >Reporter: Chang Li >Assignee: Chang Li > Fix For: 2.8.0 > > Attachments: MAPREDUCE-6374.1.patch > > > should do full ancestor permission check for a relative cache file input -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (MAPREDUCE-6174) Combine common stream code into parent class for InMemoryMapOutput and OnDiskMapOutput.
[ https://issues.apache.org/jira/browse/MAPREDUCE-6174?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14572615#comment-14572615 ] Hudson commented on MAPREDUCE-6174: --- FAILURE: Integrated in Hadoop-Yarn-trunk #948 (See [https://builds.apache.org/job/Hadoop-Yarn-trunk/948/]) MAPREDUCE-6174. Combine common stream code into parent class for InMemoryMapOutput and OnDiskMapOutput. (Eric Payne via gera) (gera: rev d90c13e2da8867661bf19a802add70145ab9a462) * hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-core/src/main/java/org/apache/hadoop/mapreduce/task/reduce/MergeManagerImpl.java * hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-core/src/main/java/org/apache/hadoop/mapreduce/task/reduce/OnDiskMapOutput.java * hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-core/src/main/java/org/apache/hadoop/mapreduce/task/reduce/IFileWrappedMapOutput.java * hadoop-mapreduce-project/CHANGES.txt * hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-core/src/main/java/org/apache/hadoop/mapreduce/task/reduce/InMemoryMapOutput.java * hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-core/src/test/java/org/apache/hadoop/mapreduce/task/reduce/TestFetcher.java > Combine common stream code into parent class for InMemoryMapOutput and > OnDiskMapOutput. > --- > > Key: MAPREDUCE-6174 > URL: https://issues.apache.org/jira/browse/MAPREDUCE-6174 > Project: Hadoop Map/Reduce > Issue Type: Improvement > Components: mrv2 >Affects Versions: 3.0.0, 2.6.0 >Reporter: Eric Payne >Assignee: Eric Payne > Labels: BB2015-05-RFC > Fix For: 2.8.0 > > Attachments: MAPREDUCE-6174.002.patch, MAPREDUCE-6174.003.patch, > MAPREDUCE-6174.004.patch, MAPREDUCE-6174.005.patch, MAPREDUCE-6174.006.patch, > MAPREDUCE-6174.007.patch, MAPREDUCE-6174.v1.txt > > > Per MAPREDUCE-6166, both InMemoryMapOutput and OnDiskMapOutput will be doing > similar things with regards to IFile streams. > In order to make it explicit that InMemoryMapOutput and OnDiskMapOutput are > different from 3rd-party implementations, this JIRA will make them subclass a > common class (see > https://issues.apache.org/jira/browse/MAPREDUCE-6166?focusedCommentId=14223368&page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel#comment-14223368) -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (MAPREDUCE-5965) Hadoop streaming throws error if list of input files is high. Error is: "error=7, Argument list too long at if number of input file is high"
[
https://issues.apache.org/jira/browse/MAPREDUCE-5965?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14572614#comment-14572614
]
Hudson commented on MAPREDUCE-5965:
---
FAILURE: Integrated in Hadoop-Yarn-trunk #948 (See
[https://builds.apache.org/job/Hadoop-Yarn-trunk/948/])
MAPREDUCE-5965. Hadoop streaming throws error if list of input files is high.
Error is: "error=7, Argument list too long at if number of input file is high"
(wilfreds via rkanter) (rkanter: rev cc70df98e74142331043a611a3bd8a53ff6a2242)
*
hadoop-tools/hadoop-streaming/src/main/java/org/apache/hadoop/streaming/StreamJob.java
* hadoop-mapreduce-project/CHANGES.txt
* hadoop-tools/hadoop-streaming/src/site/markdown/HadoopStreaming.md.vm
*
hadoop-tools/hadoop-streaming/src/main/java/org/apache/hadoop/streaming/PipeMapRed.java
> Hadoop streaming throws error if list of input files is high. Error is:
> "error=7, Argument list too long at if number of input file is high"
>
>
> Key: MAPREDUCE-5965
> URL: https://issues.apache.org/jira/browse/MAPREDUCE-5965
> Project: Hadoop Map/Reduce
> Issue Type: Bug
>Reporter: Arup Malakar
>Assignee: Wilfred Spiegelenburg
> Fix For: 2.8.0
>
> Attachments: MAPREDUCE-5965.1.patch, MAPREDUCE-5965.2.patch,
> MAPREDUCE-5965.3.patch, MAPREDUCE-5965.patch
>
>
> Hadoop streaming exposes all the key values in job conf as environment
> variables when it forks a process for streaming code to run. Unfortunately
> the variable mapreduce_input_fileinputformat_inputdir contains the list of
> input files, and Linux has a limit on size of environment variables +
> arguments.
> Based on how long the list of files and their full path is this could be
> pretty huge. And given all of these variables are not even used it stops user
> from running hadoop job with large number of files, even though it could be
> run.
> Linux throws E2BIG if the size is greater than certain size which is error
> code 7. And java translates that to "error=7, Argument list too long". More:
> http://man7.org/linux/man-pages/man2/execve.2.html I suggest skipping
> variables if it is greater than certain length. That way if user code
> requires the environment variable it would fail. It should also introduce a
> config variable to skip long variables, and set it to false by default. That
> way user has to specifically set it to true to invoke this feature.
> Here is the exception:
> {code}
> Error: java.lang.RuntimeException: Error in configuring object at
> org.apache.hadoop.util.ReflectionUtils.setJobConf(ReflectionUtils.java:109)
> at org.apache.hadoop.util.ReflectionUtils.setConf(ReflectionUtils.java:75) at
> org.apache.hadoop.util.ReflectionUtils.newInstance(ReflectionUtils.java:133)
> at org.apache.hadoop.mapred.MapTask.runOldMapper(MapTask.java:426) at
> org.apache.hadoop.mapred.MapTask.run(MapTask.java:342) at
> org.apache.hadoop.mapred.YarnChild$2.run(YarnChild.java:168) at
> java.security.AccessController.doPrivileged(Native Method) at
> javax.security.auth.Subject.doAs(Subject.java:415) at
> org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1548)
> at org.apache.hadoop.mapred.YarnChild.main(YarnChild.java:163) Caused by:
> java.lang.reflect.InvocationTargetException at
> sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method) at
> sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:57)
> at
> sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
> at java.lang.reflect.Method.invoke(Method.java:606) at
> org.apache.hadoop.util.ReflectionUtils.setJobConf(ReflectionUtils.java:106)
> ... 9 more Caused by: java.lang.RuntimeException: Error in configuring object
> at
> org.apache.hadoop.util.ReflectionUtils.setJobConf(ReflectionUtils.java:109)
> at org.apache.hadoop.util.ReflectionUtils.setConf(ReflectionUtils.java:75) at
> org.apache.hadoop.util.ReflectionUtils.newInstance(ReflectionUtils.java:133)
> at org.apache.hadoop.mapred.MapRunner.configure(MapRunner.java:38) ... 14
> more Caused by: java.lang.reflect.InvocationTargetException at
> sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method) at
> sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:57)
> at
> sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
> at java.lang.reflect.Method.invoke(Method.java:606) at
> org.apache.hadoop.util.ReflectionUtils.setJobConf(ReflectionUtils.java:106)
> ... 17 more Caused by: java.lang.RuntimeException: configuration exception at
> org.apache.hadoop.streaming.PipeMapRed.configure(PipeMapRed.java:222) at
> org.apache.hadoop.streaming.PipeMapper.
[jira] [Commented] (MAPREDUCE-5965) Hadoop streaming throws error if list of input files is high. Error is: "error=7, Argument list too long at if number of input file is high"
[
https://issues.apache.org/jira/browse/MAPREDUCE-5965?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14572593#comment-14572593
]
Hudson commented on MAPREDUCE-5965:
---
FAILURE: Integrated in Hadoop-Yarn-trunk-Java8 #218 (See
[https://builds.apache.org/job/Hadoop-Yarn-trunk-Java8/218/])
MAPREDUCE-5965. Hadoop streaming throws error if list of input files is high.
Error is: "error=7, Argument list too long at if number of input file is high"
(wilfreds via rkanter) (rkanter: rev cc70df98e74142331043a611a3bd8a53ff6a2242)
*
hadoop-tools/hadoop-streaming/src/main/java/org/apache/hadoop/streaming/PipeMapRed.java
* hadoop-mapreduce-project/CHANGES.txt
*
hadoop-tools/hadoop-streaming/src/main/java/org/apache/hadoop/streaming/StreamJob.java
* hadoop-tools/hadoop-streaming/src/site/markdown/HadoopStreaming.md.vm
> Hadoop streaming throws error if list of input files is high. Error is:
> "error=7, Argument list too long at if number of input file is high"
>
>
> Key: MAPREDUCE-5965
> URL: https://issues.apache.org/jira/browse/MAPREDUCE-5965
> Project: Hadoop Map/Reduce
> Issue Type: Bug
>Reporter: Arup Malakar
>Assignee: Wilfred Spiegelenburg
> Fix For: 2.8.0
>
> Attachments: MAPREDUCE-5965.1.patch, MAPREDUCE-5965.2.patch,
> MAPREDUCE-5965.3.patch, MAPREDUCE-5965.patch
>
>
> Hadoop streaming exposes all the key values in job conf as environment
> variables when it forks a process for streaming code to run. Unfortunately
> the variable mapreduce_input_fileinputformat_inputdir contains the list of
> input files, and Linux has a limit on size of environment variables +
> arguments.
> Based on how long the list of files and their full path is this could be
> pretty huge. And given all of these variables are not even used it stops user
> from running hadoop job with large number of files, even though it could be
> run.
> Linux throws E2BIG if the size is greater than certain size which is error
> code 7. And java translates that to "error=7, Argument list too long". More:
> http://man7.org/linux/man-pages/man2/execve.2.html I suggest skipping
> variables if it is greater than certain length. That way if user code
> requires the environment variable it would fail. It should also introduce a
> config variable to skip long variables, and set it to false by default. That
> way user has to specifically set it to true to invoke this feature.
> Here is the exception:
> {code}
> Error: java.lang.RuntimeException: Error in configuring object at
> org.apache.hadoop.util.ReflectionUtils.setJobConf(ReflectionUtils.java:109)
> at org.apache.hadoop.util.ReflectionUtils.setConf(ReflectionUtils.java:75) at
> org.apache.hadoop.util.ReflectionUtils.newInstance(ReflectionUtils.java:133)
> at org.apache.hadoop.mapred.MapTask.runOldMapper(MapTask.java:426) at
> org.apache.hadoop.mapred.MapTask.run(MapTask.java:342) at
> org.apache.hadoop.mapred.YarnChild$2.run(YarnChild.java:168) at
> java.security.AccessController.doPrivileged(Native Method) at
> javax.security.auth.Subject.doAs(Subject.java:415) at
> org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1548)
> at org.apache.hadoop.mapred.YarnChild.main(YarnChild.java:163) Caused by:
> java.lang.reflect.InvocationTargetException at
> sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method) at
> sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:57)
> at
> sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
> at java.lang.reflect.Method.invoke(Method.java:606) at
> org.apache.hadoop.util.ReflectionUtils.setJobConf(ReflectionUtils.java:106)
> ... 9 more Caused by: java.lang.RuntimeException: Error in configuring object
> at
> org.apache.hadoop.util.ReflectionUtils.setJobConf(ReflectionUtils.java:109)
> at org.apache.hadoop.util.ReflectionUtils.setConf(ReflectionUtils.java:75) at
> org.apache.hadoop.util.ReflectionUtils.newInstance(ReflectionUtils.java:133)
> at org.apache.hadoop.mapred.MapRunner.configure(MapRunner.java:38) ... 14
> more Caused by: java.lang.reflect.InvocationTargetException at
> sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method) at
> sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:57)
> at
> sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
> at java.lang.reflect.Method.invoke(Method.java:606) at
> org.apache.hadoop.util.ReflectionUtils.setJobConf(ReflectionUtils.java:106)
> ... 17 more Caused by: java.lang.RuntimeException: configuration exception at
> org.apache.hadoop.streaming.PipeMapRed.configure(PipeMapRed.java:222) at
> org.apache.hadoop.streaming
[jira] [Commented] (MAPREDUCE-6374) Distributed Cache File visibility should check permission of full path
[ https://issues.apache.org/jira/browse/MAPREDUCE-6374?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14572595#comment-14572595 ] Hudson commented on MAPREDUCE-6374: --- FAILURE: Integrated in Hadoop-Yarn-trunk-Java8 #218 (See [https://builds.apache.org/job/Hadoop-Yarn-trunk-Java8/218/]) MAPREDUCE-6374. Distributed Cache File visibility should check permission of full path. Contributed by Chang Li (jlowe: rev 107da29ff9b6efd02b21c96df60c58ac51590ebd) * hadoop-mapreduce-project/CHANGES.txt * hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-core/src/test/java/org/apache/hadoop/mapreduce/filecache/TestClientDistributedCacheManager.java * hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-core/src/main/java/org/apache/hadoop/mapreduce/filecache/ClientDistributedCacheManager.java > Distributed Cache File visibility should check permission of full path > -- > > Key: MAPREDUCE-6374 > URL: https://issues.apache.org/jira/browse/MAPREDUCE-6374 > Project: Hadoop Map/Reduce > Issue Type: Bug >Reporter: Chang Li >Assignee: Chang Li > Fix For: 2.8.0 > > Attachments: MAPREDUCE-6374.1.patch > > > should do full ancestor permission check for a relative cache file input -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (MAPREDUCE-6382) Don't escape HTML links in Diagnostics in JHS job overview
[ https://issues.apache.org/jira/browse/MAPREDUCE-6382?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14572592#comment-14572592 ] Hudson commented on MAPREDUCE-6382: --- FAILURE: Integrated in Hadoop-Yarn-trunk-Java8 #218 (See [https://builds.apache.org/job/Hadoop-Yarn-trunk-Java8/218/]) MAPREDUCE-6382. Don't escape HTML links in Diagnostics in JHS job overview. (Siqi Li via gera) (gera: rev e8302071ff16247a3d0cfc2dcca02da1bbf4db18) * hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-hs/src/main/java/org/apache/hadoop/mapreduce/v2/hs/webapp/HsJobBlock.java * hadoop-mapreduce-project/CHANGES.txt > Don't escape HTML links in Diagnostics in JHS job overview > -- > > Key: MAPREDUCE-6382 > URL: https://issues.apache.org/jira/browse/MAPREDUCE-6382 > Project: Hadoop Map/Reduce > Issue Type: Bug >Affects Versions: 2.8.0 >Reporter: Siqi Li >Assignee: Siqi Li > Fix For: 2.8.0 > > Attachments: MAPREDUCE-6382.v1.patch, links are escaped.jpg > > > for some reason, links are working properly in 2.4, but they are escaped in > 2.6 -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (MAPREDUCE-6174) Combine common stream code into parent class for InMemoryMapOutput and OnDiskMapOutput.
[ https://issues.apache.org/jira/browse/MAPREDUCE-6174?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14572594#comment-14572594 ] Hudson commented on MAPREDUCE-6174: --- FAILURE: Integrated in Hadoop-Yarn-trunk-Java8 #218 (See [https://builds.apache.org/job/Hadoop-Yarn-trunk-Java8/218/]) MAPREDUCE-6174. Combine common stream code into parent class for InMemoryMapOutput and OnDiskMapOutput. (Eric Payne via gera) (gera: rev d90c13e2da8867661bf19a802add70145ab9a462) * hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-core/src/main/java/org/apache/hadoop/mapreduce/task/reduce/OnDiskMapOutput.java * hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-core/src/main/java/org/apache/hadoop/mapreduce/task/reduce/MergeManagerImpl.java * hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-core/src/main/java/org/apache/hadoop/mapreduce/task/reduce/InMemoryMapOutput.java * hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-core/src/test/java/org/apache/hadoop/mapreduce/task/reduce/TestFetcher.java * hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-core/src/main/java/org/apache/hadoop/mapreduce/task/reduce/IFileWrappedMapOutput.java * hadoop-mapreduce-project/CHANGES.txt > Combine common stream code into parent class for InMemoryMapOutput and > OnDiskMapOutput. > --- > > Key: MAPREDUCE-6174 > URL: https://issues.apache.org/jira/browse/MAPREDUCE-6174 > Project: Hadoop Map/Reduce > Issue Type: Improvement > Components: mrv2 >Affects Versions: 3.0.0, 2.6.0 >Reporter: Eric Payne >Assignee: Eric Payne > Labels: BB2015-05-RFC > Fix For: 2.8.0 > > Attachments: MAPREDUCE-6174.002.patch, MAPREDUCE-6174.003.patch, > MAPREDUCE-6174.004.patch, MAPREDUCE-6174.005.patch, MAPREDUCE-6174.006.patch, > MAPREDUCE-6174.007.patch, MAPREDUCE-6174.v1.txt > > > Per MAPREDUCE-6166, both InMemoryMapOutput and OnDiskMapOutput will be doing > similar things with regards to IFile streams. > In order to make it explicit that InMemoryMapOutput and OnDiskMapOutput are > different from 3rd-party implementations, this JIRA will make them subclass a > common class (see > https://issues.apache.org/jira/browse/MAPREDUCE-6166?focusedCommentId=14223368&page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel#comment-14223368) -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (MAPREDUCE-6382) Don't escape HTML links in Diagnostics in JHS job overview
[ https://issues.apache.org/jira/browse/MAPREDUCE-6382?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14572288#comment-14572288 ] Hudson commented on MAPREDUCE-6382: --- FAILURE: Integrated in Hadoop-trunk-Commit #7962 (See [https://builds.apache.org/job/Hadoop-trunk-Commit/7962/]) MAPREDUCE-6382. Don't escape HTML links in Diagnostics in JHS job overview. (Siqi Li via gera) (gera: rev e8302071ff16247a3d0cfc2dcca02da1bbf4db18) * hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-hs/src/main/java/org/apache/hadoop/mapreduce/v2/hs/webapp/HsJobBlock.java * hadoop-mapreduce-project/CHANGES.txt > Don't escape HTML links in Diagnostics in JHS job overview > -- > > Key: MAPREDUCE-6382 > URL: https://issues.apache.org/jira/browse/MAPREDUCE-6382 > Project: Hadoop Map/Reduce > Issue Type: Bug >Affects Versions: 2.8.0 >Reporter: Siqi Li >Assignee: Siqi Li > Fix For: 2.8.0 > > Attachments: MAPREDUCE-6382.v1.patch, links are escaped.jpg > > > for some reason, links are working properly in 2.4, but they are escaped in > 2.6 -- This message was sent by Atlassian JIRA (v6.3.4#6332)
