[jira] [Commented] (MAPREDUCE-6387) Fix Task object to allow Jobs not submitted via distributed cache to survive rolling upgrade
[
https://issues.apache.org/jira/browse/MAPREDUCE-6387?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14573650#comment-14573650
]
Karthik Kambatla commented on MAPREDUCE-6387:
-
The fix looks safe to me. +1. Will go ahead and commit this first thing
tomorrow if no one objects.
Fix Task object to allow Jobs not submitted via distributed cache to survive
rolling upgrade
Key: MAPREDUCE-6387
URL: https://issues.apache.org/jira/browse/MAPREDUCE-6387
Project: Hadoop Map/Reduce
Issue Type: Bug
Reporter: Arun Suresh
Assignee: Arun Suresh
Priority: Minor
Attachments: MAPREDUCE-6387.1.patch
There was a recent addition of an {{encryptedSpillKey}} to the Task object.
And when serialized, this field was written out somewhere in the middle. This
caused deployments that do not use DistributedCache to push job jars before
running the job to fail rolling upgrade.
Although deploying via Distributed Cache is the recommended method, there
might still be deployments that use the node local classpath to pick up the
MR framework classes (eg. for efficiency purposes, since this does not
require the jar being copied to hdfs and then to all the nodes)
Ensuring that it is the last field written and read when the Task object is
serialized would alleviate this issue.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Updated] (MAPREDUCE-6354) A
[ https://issues.apache.org/jira/browse/MAPREDUCE-6354?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Jason Lowe updated MAPREDUCE-6354: -- Summary: A (was: shuffle handler should log connection info) A - Key: MAPREDUCE-6354 URL: https://issues.apache.org/jira/browse/MAPREDUCE-6354 Project: Hadoop Map/Reduce Issue Type: Bug Reporter: Chang Li Assignee: Chang Li Attachments: MAPREDUCE-6354.2.patch, MAPREDUCE-6354.3.patch, MAPREDUCE-6354.4.patch, MAPREDUCE-6354.5.patch, MAPREDUCE-6354.6.patch, MAPREDUCE-6354.7.patch, MAPREDUCE-6354.8.patch, MAPREDUCE-6354.patch currently, shuffle handler only log connection info in debug mode, we want to log that info in a more concise way -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Updated] (MAPREDUCE-6316) Task Attempt List entries should link to the task overview
[
https://issues.apache.org/jira/browse/MAPREDUCE-6316?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Gera Shegalov updated MAPREDUCE-6316:
-
Status: Open (was: Patch Available)
Thanks for v6 [~l201514]. The JHS pages look good. But AM's running map
attempts is corrupt. In the Safari debugger:
{code}
ReferenceError: Can't find variable: attemptsTableData
{code}
Task Attempt List entries should link to the task overview
--
Key: MAPREDUCE-6316
URL: https://issues.apache.org/jira/browse/MAPREDUCE-6316
Project: Hadoop Map/Reduce
Issue Type: Improvement
Reporter: Siqi Li
Assignee: Siqi Li
Labels: BB2015-05-TBR
Attachments: AM attempt page.png, AM task page.png, All Attempts
page.png, MAPREDUCE-6316.v1.patch, MAPREDUCE-6316.v2.patch,
MAPREDUCE-6316.v3.patch, MAPREDUCE-6316.v4.patch, MAPREDUCE-6316.v5.patch,
MAPREDUCE-6316.v6.patch, Task Overview page.png
Typical workflow is to click on the list of failed attempts. Then you want to
look at the counters, or the list of attempts of just one task in general. If
each entry task attempt id linked the task id portion of it back to the task,
we would not have to go through the list of tasks to search for the task.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Updated] (MAPREDUCE-6354) ShuffleHandler should be able to log shuffle connections
[ https://issues.apache.org/jira/browse/MAPREDUCE-6354?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Jason Lowe updated MAPREDUCE-6354: -- Issue Type: Improvement (was: Bug) Summary: ShuffleHandler should be able to log shuffle connections (was: A) Hadoop Flags: Reviewed +1 latest patch lgtm. Will commit this tomorrow if there are no objections. ShuffleHandler should be able to log shuffle connections Key: MAPREDUCE-6354 URL: https://issues.apache.org/jira/browse/MAPREDUCE-6354 Project: Hadoop Map/Reduce Issue Type: Improvement Reporter: Chang Li Assignee: Chang Li Attachments: MAPREDUCE-6354.2.patch, MAPREDUCE-6354.3.patch, MAPREDUCE-6354.4.patch, MAPREDUCE-6354.5.patch, MAPREDUCE-6354.6.patch, MAPREDUCE-6354.7.patch, MAPREDUCE-6354.8.patch, MAPREDUCE-6354.patch currently, shuffle handler only log connection info in debug mode, we want to log that info in a more concise way -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (MAPREDUCE-6316) Task Attempt List entries should link to the task overview
[ https://issues.apache.org/jira/browse/MAPREDUCE-6316?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14573811#comment-14573811 ] Siqi Li commented on MAPREDUCE-6316: [~jira.shegalov], Is it also happening to other web browsers ? Task Attempt List entries should link to the task overview -- Key: MAPREDUCE-6316 URL: https://issues.apache.org/jira/browse/MAPREDUCE-6316 Project: Hadoop Map/Reduce Issue Type: Improvement Reporter: Siqi Li Assignee: Siqi Li Labels: BB2015-05-TBR Attachments: AM attempt page.png, AM task page.png, All Attempts page.png, MAPREDUCE-6316.v1.patch, MAPREDUCE-6316.v2.patch, MAPREDUCE-6316.v3.patch, MAPREDUCE-6316.v4.patch, MAPREDUCE-6316.v5.patch, MAPREDUCE-6316.v6.patch, Task Overview page.png Typical workflow is to click on the list of failed attempts. Then you want to look at the counters, or the list of attempts of just one task in general. If each entry task attempt id linked the task id portion of it back to the task, we would not have to go through the list of tasks to search for the task. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (MAPREDUCE-6384) add the last reporting reducer host info and attempt id on the map error message due to too many fetch failure
[
https://issues.apache.org/jira/browse/MAPREDUCE-6384?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14573150#comment-14573150
]
Hadoop QA commented on MAPREDUCE-6384:
--
\\
\\
| (x) *{color:red}-1 overall{color}* |
\\
\\
|| Vote || Subsystem || Runtime || Comment ||
| {color:red}-1{color} | pre-patch | 15m 47s | Pre-patch trunk has 1 extant
Findbugs (version 3.0.0) warnings. |
| {color:green}+1{color} | @author | 0m 0s | The patch does not contain any
@author tags. |
| {color:green}+1{color} | tests included | 0m 0s | The patch appears to
include 2 new or modified test files. |
| {color:red}-1{color} | javac | 7m 32s | The applied patch generated 1
additional warning messages. |
| {color:green}+1{color} | javadoc | 9m 35s | There were no new javadoc
warning messages. |
| {color:green}+1{color} | release audit | 0m 23s | The applied patch does
not increase the total number of release audit warnings. |
| {color:red}-1{color} | checkstyle | 0m 33s | The applied patch generated 2
new checkstyle issues (total was 357, now 356). |
| {color:red}-1{color} | whitespace | 0m 1s | The patch has 2 line(s) that
end in whitespace. Use git apply --whitespace=fix. |
| {color:green}+1{color} | install | 1m 34s | mvn install still works. |
| {color:green}+1{color} | eclipse:eclipse | 0m 33s | The patch built with
eclipse:eclipse. |
| {color:green}+1{color} | findbugs | 1m 6s | The patch does not introduce
any new Findbugs (version 3.0.0) warnings. |
| {color:green}+1{color} | mapreduce tests | 9m 36s | Tests passed in
hadoop-mapreduce-client-app. |
| | | 46m 47s | |
\\
\\
|| Subsystem || Report/Notes ||
| Patch URL |
http://issues.apache.org/jira/secure/attachment/12737611/MAPREDUCE-6384.4.2.patch
|
| Optional Tests | javadoc javac unit findbugs checkstyle |
| git revision | trunk / d7e7f6a |
| Pre-patch Findbugs warnings |
https://builds.apache.org/job/PreCommit-MAPREDUCE-Build/5773/artifact/patchprocess/trunkFindbugsWarningshadoop-mapreduce-client-app.html
|
| javac |
https://builds.apache.org/job/PreCommit-MAPREDUCE-Build/5773/artifact/patchprocess/diffJavacWarnings.txt
|
| checkstyle |
https://builds.apache.org/job/PreCommit-MAPREDUCE-Build/5773/artifact/patchprocess/diffcheckstylehadoop-mapreduce-client-app.txt
|
| whitespace |
https://builds.apache.org/job/PreCommit-MAPREDUCE-Build/5773/artifact/patchprocess/whitespace.txt
|
| hadoop-mapreduce-client-app test log |
https://builds.apache.org/job/PreCommit-MAPREDUCE-Build/5773/artifact/patchprocess/testrun_hadoop-mapreduce-client-app.txt
|
| Test Results |
https://builds.apache.org/job/PreCommit-MAPREDUCE-Build/5773/testReport/ |
| Java | 1.7.0_55 |
| uname | Linux asf903.gq1.ygridcore.net 3.13.0-36-lowlatency #63-Ubuntu SMP
PREEMPT Wed Sep 3 21:56:12 UTC 2014 x86_64 x86_64 x86_64 GNU/Linux |
| Console output |
https://builds.apache.org/job/PreCommit-MAPREDUCE-Build/5773/console |
This message was automatically generated.
add the last reporting reducer host info and attempt id on the map error
message due to too many fetch failure
--
Key: MAPREDUCE-6384
URL: https://issues.apache.org/jira/browse/MAPREDUCE-6384
Project: Hadoop Map/Reduce
Issue Type: Bug
Reporter: Chang Li
Assignee: Chang Li
Attachments: MAPREDUCE-6384.2.patch, MAPREDUCE-6384.3.patch,
MAPREDUCE-6384.4.2.patch, MAPREDUCE-6384.4.patch
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Commented] (MAPREDUCE-3971) Job History web services need to have limits on the number of items they can return.
[ https://issues.apache.org/jira/browse/MAPREDUCE-3971?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14573204#comment-14573204 ] Ray Chiang commented on MAPREDUCE-3971: --- For .jhist parsing, I've added MAPREDUCE-6376. Job History web services need to have limits on the number of items they can return. Key: MAPREDUCE-3971 URL: https://issues.apache.org/jira/browse/MAPREDUCE-3971 Project: Hadoop Map/Reduce Issue Type: Sub-task Components: mrv2 Affects Versions: 0.23.2 Reporter: Robert Joseph Evans The Job History web services canput a very large load on the job history server. We should put in a limit on the number of entries that can be returned by the web service, and also add in the ability to modify the starting location in the list, so that all entries can still be downlaoded. Just not all at once. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (MAPREDUCE-6354) shuffle handler should log connection info
[ https://issues.apache.org/jira/browse/MAPREDUCE-6354?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14573328#comment-14573328 ] Jason Lowe commented on MAPREDUCE-6354: --- Thanks for updating the patch, Chang. The debug log should be wrapped in AUDITLOG.isDebugEnabled() so we don't perform the string processing if we aren't going to log it. Also, even though this wasn't technically part of this fix, please go ahead and fixup the trailing whitespace in log4j.properties that checkstyle is whining about. Otherwise it will continue to whine for any other patch that happens to touch nearby lines. shuffle handler should log connection info -- Key: MAPREDUCE-6354 URL: https://issues.apache.org/jira/browse/MAPREDUCE-6354 Project: Hadoop Map/Reduce Issue Type: Bug Reporter: Chang Li Assignee: Chang Li Attachments: MAPREDUCE-6354.2.patch, MAPREDUCE-6354.3.patch, MAPREDUCE-6354.4.patch, MAPREDUCE-6354.5.patch, MAPREDUCE-6354.6.patch, MAPREDUCE-6354.7.patch, MAPREDUCE-6354.patch currently, shuffle handler only log connection info in debug mode, we want to log that info in a more concise way -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (MAPREDUCE-5232) log classpath and other key properties on child JVM start
[ https://issues.apache.org/jira/browse/MAPREDUCE-5232?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14573252#comment-14573252 ] Gera Shegalov commented on MAPREDUCE-5232: -- Merge conflicts are minor. I'll push it to branch-2. log classpath and other key properties on child JVM start - Key: MAPREDUCE-5232 URL: https://issues.apache.org/jira/browse/MAPREDUCE-5232 Project: Hadoop Map/Reduce Issue Type: Improvement Components: mrv1, mrv2 Affects Versions: 2.0.4-alpha Reporter: Sangjin Lee Assignee: Sangjin Lee Fix For: 3.0.0 Attachments: MAPREDUCE-5232-branch-2.patch, MAPREDUCE-5232-branch-2.patch, MAPREDUCE-5232-branch-2.patch, MAPREDUCE-5232.patch, MAPREDUCE-5232.patch, MAPREDUCE-5232.patch It would be great if we log vital information such as classpath, etc. upon a mapreduce child JVM start. This would help a great deal in terms of troubleshooting classpath issues, etc. Today it is pretty difficult to debug this unless you preserve the container script. Maybe it can log things like classpath, os name/version, java version, etc. at the beginning of the child JVM start. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Updated] (MAPREDUCE-5232) log classpath and other key properties on child JVM start
[ https://issues.apache.org/jira/browse/MAPREDUCE-5232?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Gera Shegalov updated MAPREDUCE-5232: - Fix Version/s: (was: 3.0.0) 2.8.0 log classpath and other key properties on child JVM start - Key: MAPREDUCE-5232 URL: https://issues.apache.org/jira/browse/MAPREDUCE-5232 Project: Hadoop Map/Reduce Issue Type: Improvement Components: mrv1, mrv2 Affects Versions: 2.0.4-alpha Reporter: Sangjin Lee Assignee: Sangjin Lee Fix For: 2.8.0 Attachments: MAPREDUCE-5232-branch-2.patch, MAPREDUCE-5232-branch-2.patch, MAPREDUCE-5232-branch-2.patch, MAPREDUCE-5232.patch, MAPREDUCE-5232.patch, MAPREDUCE-5232.patch It would be great if we log vital information such as classpath, etc. upon a mapreduce child JVM start. This would help a great deal in terms of troubleshooting classpath issues, etc. Today it is pretty difficult to debug this unless you preserve the container script. Maybe it can log things like classpath, os name/version, java version, etc. at the beginning of the child JVM start. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (MAPREDUCE-6387) Fix Task object to allow Jobs not submitted via distributed cache to survive rolling upgrade
[
https://issues.apache.org/jira/browse/MAPREDUCE-6387?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14573519#comment-14573519
]
Hadoop QA commented on MAPREDUCE-6387:
--
\\
\\
| (x) *{color:red}-1 overall{color}* |
\\
\\
|| Vote || Subsystem || Runtime || Comment ||
| {color:blue}0{color} | pre-patch | 16m 58s | Pre-patch trunk compilation is
healthy. |
| {color:green}+1{color} | @author | 0m 0s | The patch does not contain any
@author tags. |
| {color:red}-1{color} | tests included | 0m 0s | The patch doesn't appear
to include any new or modified tests. Please justify why no new tests are
needed for this patch. Also please list what manual steps were performed to
verify this patch. |
| {color:green}+1{color} | javac | 7m 59s | There were no new javac warning
messages. |
| {color:green}+1{color} | javadoc | 9m 39s | There were no new javadoc
warning messages. |
| {color:green}+1{color} | release audit | 0m 21s | The applied patch does
not increase the total number of release audit warnings. |
| {color:green}+1{color} | checkstyle | 0m 48s | There were no new checkstyle
issues. |
| {color:green}+1{color} | whitespace | 0m 0s | The patch has no lines that
end in whitespace. |
| {color:green}+1{color} | install | 1m 33s | mvn install still works. |
| {color:green}+1{color} | eclipse:eclipse | 0m 32s | The patch built with
eclipse:eclipse. |
| {color:green}+1{color} | findbugs | 1m 25s | The patch does not introduce
any new Findbugs (version 3.0.0) warnings. |
| {color:green}+1{color} | mapreduce tests | 1m 38s | Tests passed in
hadoop-mapreduce-client-core. |
| | | 40m 56s | |
\\
\\
|| Subsystem || Report/Notes ||
| Patch URL |
http://issues.apache.org/jira/secure/attachment/12737621/MAPREDUCE-6387.1.patch
|
| Optional Tests | javadoc javac unit findbugs checkstyle |
| git revision | trunk / 9f6d678 |
| hadoop-mapreduce-client-core test log |
https://builds.apache.org/job/PreCommit-MAPREDUCE-Build/5775/artifact/patchprocess/testrun_hadoop-mapreduce-client-core.txt
|
| Test Results |
https://builds.apache.org/job/PreCommit-MAPREDUCE-Build/5775/testReport/ |
| Java | 1.7.0_55 |
| uname | Linux asf903.gq1.ygridcore.net 3.13.0-36-lowlatency #63-Ubuntu SMP
PREEMPT Wed Sep 3 21:56:12 UTC 2014 x86_64 x86_64 x86_64 GNU/Linux |
| Console output |
https://builds.apache.org/job/PreCommit-MAPREDUCE-Build/5775/console |
This message was automatically generated.
Fix Task object to allow Jobs not submitted via distributed cache to survive
rolling upgrade
Key: MAPREDUCE-6387
URL: https://issues.apache.org/jira/browse/MAPREDUCE-6387
Project: Hadoop Map/Reduce
Issue Type: Bug
Reporter: Arun Suresh
Assignee: Arun Suresh
Priority: Minor
Attachments: MAPREDUCE-6387.1.patch
There was a recent addition of an {{encryptedSpillKey}} to the Task object.
And when serialized, this field was written out somewhere in the middle. This
caused deployments that do not use DistributedCache to push job jars before
running the job to fail rolling upgrade.
Although deploying via Distributed Cache is the recommended method, there
might still be deployments that use the node local classpath to pick up the
MR framework classes (eg. for efficiency purposes, since this does not
require the jar being copied to hdfs and then to all the nodes)
Ensuring that it is the last field written and read when the Task object is
serialized would alleviate this issue.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Commented] (MAPREDUCE-6354) shuffle handler should log connection info
[
https://issues.apache.org/jira/browse/MAPREDUCE-6354?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14573522#comment-14573522
]
Hadoop QA commented on MAPREDUCE-6354:
--
\\
\\
| (x) *{color:red}-1 overall{color}* |
\\
\\
|| Vote || Subsystem || Runtime || Comment ||
| {color:blue}0{color} | pre-patch | 16m 58s | Pre-patch trunk compilation is
healthy. |
| {color:green}+1{color} | @author | 0m 0s | The patch does not contain any
@author tags. |
| {color:red}-1{color} | tests included | 0m 0s | The patch doesn't appear
to include any new or modified tests. Please justify why no new tests are
needed for this patch. Also please list what manual steps were performed to
verify this patch. |
| {color:green}+1{color} | javac | 7m 34s | There were no new javac warning
messages. |
| {color:green}+1{color} | javadoc | 9m 34s | There were no new javadoc
warning messages. |
| {color:green}+1{color} | release audit | 0m 23s | The applied patch does
not increase the total number of release audit warnings. |
| {color:green}+1{color} | checkstyle | 1m 26s | There were no new checkstyle
issues. |
| {color:green}+1{color} | whitespace | 0m 0s | The patch has no lines that
end in whitespace. |
| {color:green}+1{color} | install | 1m 35s | mvn install still works. |
| {color:green}+1{color} | eclipse:eclipse | 0m 32s | The patch built with
eclipse:eclipse. |
| {color:green}+1{color} | findbugs | 2m 34s | The patch does not introduce
any new Findbugs (version 3.0.0) warnings. |
| {color:green}+1{color} | common tests | 23m 2s | Tests passed in
hadoop-common. |
| {color:green}+1{color} | mapreduce tests | 0m 20s | Tests passed in
hadoop-mapreduce-client-shuffle. |
| | | 64m 2s | |
\\
\\
|| Subsystem || Report/Notes ||
| Patch URL |
http://issues.apache.org/jira/secure/attachment/12737665/MAPREDUCE-6354.8.patch
|
| Optional Tests | javadoc javac unit findbugs checkstyle |
| git revision | trunk / 9f6d678 |
| hadoop-common test log |
https://builds.apache.org/job/PreCommit-MAPREDUCE-Build/5774/artifact/patchprocess/testrun_hadoop-common.txt
|
| hadoop-mapreduce-client-shuffle test log |
https://builds.apache.org/job/PreCommit-MAPREDUCE-Build/5774/artifact/patchprocess/testrun_hadoop-mapreduce-client-shuffle.txt
|
| Test Results |
https://builds.apache.org/job/PreCommit-MAPREDUCE-Build/5774/testReport/ |
| Java | 1.7.0_55 |
| uname | Linux asf905.gq1.ygridcore.net 3.13.0-36-lowlatency #63-Ubuntu SMP
PREEMPT Wed Sep 3 21:56:12 UTC 2014 x86_64 x86_64 x86_64 GNU/Linux |
| Console output |
https://builds.apache.org/job/PreCommit-MAPREDUCE-Build/5774/console |
This message was automatically generated.
shuffle handler should log connection info
--
Key: MAPREDUCE-6354
URL: https://issues.apache.org/jira/browse/MAPREDUCE-6354
Project: Hadoop Map/Reduce
Issue Type: Bug
Reporter: Chang Li
Assignee: Chang Li
Attachments: MAPREDUCE-6354.2.patch, MAPREDUCE-6354.3.patch,
MAPREDUCE-6354.4.patch, MAPREDUCE-6354.5.patch, MAPREDUCE-6354.6.patch,
MAPREDUCE-6354.7.patch, MAPREDUCE-6354.8.patch, MAPREDUCE-6354.patch
currently, shuffle handler only log connection info in debug mode, we want to
log that info in a more concise way
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Commented] (MAPREDUCE-5965) Hadoop streaming throws error if list of input files is high. Error is: error=7, Argument list too long at if number of input file is high
[
https://issues.apache.org/jira/browse/MAPREDUCE-5965?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14573008#comment-14573008
]
Hudson commented on MAPREDUCE-5965:
---
FAILURE: Integrated in Hadoop-Mapreduce-trunk-Java8 #216 (See
[https://builds.apache.org/job/Hadoop-Mapreduce-trunk-Java8/216/])
MAPREDUCE-5965. Hadoop streaming throws error if list of input files is high.
Error is: error=7, Argument list too long at if number of input file is high
(wilfreds via rkanter) (rkanter: rev cc70df98e74142331043a611a3bd8a53ff6a2242)
* hadoop-tools/hadoop-streaming/src/site/markdown/HadoopStreaming.md.vm
*
hadoop-tools/hadoop-streaming/src/main/java/org/apache/hadoop/streaming/StreamJob.java
*
hadoop-tools/hadoop-streaming/src/main/java/org/apache/hadoop/streaming/PipeMapRed.java
* hadoop-mapreduce-project/CHANGES.txt
Hadoop streaming throws error if list of input files is high. Error is:
error=7, Argument list too long at if number of input file is high
Key: MAPREDUCE-5965
URL: https://issues.apache.org/jira/browse/MAPREDUCE-5965
Project: Hadoop Map/Reduce
Issue Type: Bug
Reporter: Arup Malakar
Assignee: Wilfred Spiegelenburg
Fix For: 2.8.0
Attachments: MAPREDUCE-5965.1.patch, MAPREDUCE-5965.2.patch,
MAPREDUCE-5965.3.patch, MAPREDUCE-5965.patch
Hadoop streaming exposes all the key values in job conf as environment
variables when it forks a process for streaming code to run. Unfortunately
the variable mapreduce_input_fileinputformat_inputdir contains the list of
input files, and Linux has a limit on size of environment variables +
arguments.
Based on how long the list of files and their full path is this could be
pretty huge. And given all of these variables are not even used it stops user
from running hadoop job with large number of files, even though it could be
run.
Linux throws E2BIG if the size is greater than certain size which is error
code 7. And java translates that to error=7, Argument list too long. More:
http://man7.org/linux/man-pages/man2/execve.2.html I suggest skipping
variables if it is greater than certain length. That way if user code
requires the environment variable it would fail. It should also introduce a
config variable to skip long variables, and set it to false by default. That
way user has to specifically set it to true to invoke this feature.
Here is the exception:
{code}
Error: java.lang.RuntimeException: Error in configuring object at
org.apache.hadoop.util.ReflectionUtils.setJobConf(ReflectionUtils.java:109)
at org.apache.hadoop.util.ReflectionUtils.setConf(ReflectionUtils.java:75) at
org.apache.hadoop.util.ReflectionUtils.newInstance(ReflectionUtils.java:133)
at org.apache.hadoop.mapred.MapTask.runOldMapper(MapTask.java:426) at
org.apache.hadoop.mapred.MapTask.run(MapTask.java:342) at
org.apache.hadoop.mapred.YarnChild$2.run(YarnChild.java:168) at
java.security.AccessController.doPrivileged(Native Method) at
javax.security.auth.Subject.doAs(Subject.java:415) at
org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1548)
at org.apache.hadoop.mapred.YarnChild.main(YarnChild.java:163) Caused by:
java.lang.reflect.InvocationTargetException at
sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method) at
sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:57)
at
sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
at java.lang.reflect.Method.invoke(Method.java:606) at
org.apache.hadoop.util.ReflectionUtils.setJobConf(ReflectionUtils.java:106)
... 9 more Caused by: java.lang.RuntimeException: Error in configuring object
at
org.apache.hadoop.util.ReflectionUtils.setJobConf(ReflectionUtils.java:109)
at org.apache.hadoop.util.ReflectionUtils.setConf(ReflectionUtils.java:75) at
org.apache.hadoop.util.ReflectionUtils.newInstance(ReflectionUtils.java:133)
at org.apache.hadoop.mapred.MapRunner.configure(MapRunner.java:38) ... 14
more Caused by: java.lang.reflect.InvocationTargetException at
sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method) at
sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:57)
at
sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
at java.lang.reflect.Method.invoke(Method.java:606) at
org.apache.hadoop.util.ReflectionUtils.setJobConf(ReflectionUtils.java:106)
... 17 more Caused by: java.lang.RuntimeException: configuration exception at
org.apache.hadoop.streaming.PipeMapRed.configure(PipeMapRed.java:222) at
org.apache.hadoop.streaming.PipeMapper.configure(PipeMapper.java:66) ... 22
more Caused
[jira] [Commented] (MAPREDUCE-6382) Don't escape HTML links in Diagnostics in JHS job overview
[ https://issues.apache.org/jira/browse/MAPREDUCE-6382?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14573007#comment-14573007 ] Hudson commented on MAPREDUCE-6382: --- FAILURE: Integrated in Hadoop-Mapreduce-trunk-Java8 #216 (See [https://builds.apache.org/job/Hadoop-Mapreduce-trunk-Java8/216/]) MAPREDUCE-6382. Don't escape HTML links in Diagnostics in JHS job overview. (Siqi Li via gera) (gera: rev e8302071ff16247a3d0cfc2dcca02da1bbf4db18) * hadoop-mapreduce-project/CHANGES.txt * hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-hs/src/main/java/org/apache/hadoop/mapreduce/v2/hs/webapp/HsJobBlock.java Don't escape HTML links in Diagnostics in JHS job overview -- Key: MAPREDUCE-6382 URL: https://issues.apache.org/jira/browse/MAPREDUCE-6382 Project: Hadoop Map/Reduce Issue Type: Bug Affects Versions: 2.8.0 Reporter: Siqi Li Assignee: Siqi Li Fix For: 2.8.0 Attachments: MAPREDUCE-6382.v1.patch, links are escaped.jpg for some reason, links are working properly in 2.4, but they are escaped in 2.6 -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (MAPREDUCE-6384) add the last reporting reducer host info and attempt id on the map error message due to too many fetch failure
[
https://issues.apache.org/jira/browse/MAPREDUCE-6384?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14573055#comment-14573055
]
Hadoop QA commented on MAPREDUCE-6384:
--
\\
\\
| (x) *{color:red}-1 overall{color}* |
\\
\\
|| Vote || Subsystem || Runtime || Comment ||
| {color:red}-1{color} | pre-patch | 15m 40s | Pre-patch trunk has 1 extant
Findbugs (version 3.0.0) warnings. |
| {color:green}+1{color} | @author | 0m 0s | The patch does not contain any
@author tags. |
| {color:green}+1{color} | tests included | 0m 0s | The patch appears to
include 2 new or modified test files. |
| {color:red}-1{color} | javac | 7m 33s | The applied patch generated 1
additional warning messages. |
| {color:green}+1{color} | javadoc | 9m 36s | There were no new javadoc
warning messages. |
| {color:green}+1{color} | release audit | 0m 23s | The applied patch does
not increase the total number of release audit warnings. |
| {color:red}-1{color} | checkstyle | 0m 34s | The applied patch generated 3
new checkstyle issues (total was 357, now 357). |
| {color:red}-1{color} | whitespace | 0m 1s | The patch has 2 line(s) that
end in whitespace. Use git apply --whitespace=fix. |
| {color:green}+1{color} | install | 1m 35s | mvn install still works. |
| {color:green}+1{color} | eclipse:eclipse | 0m 32s | The patch built with
eclipse:eclipse. |
| {color:green}+1{color} | findbugs | 1m 7s | The patch does not introduce
any new Findbugs (version 3.0.0) warnings. |
| {color:green}+1{color} | mapreduce tests | 8m 59s | Tests passed in
hadoop-mapreduce-client-app. |
| | | 46m 4s | |
\\
\\
|| Subsystem || Report/Notes ||
| Patch URL |
http://issues.apache.org/jira/secure/attachment/12737592/MAPREDUCE-6384.4.patch
|
| Optional Tests | javadoc javac unit findbugs checkstyle |
| git revision | trunk / d7e7f6a |
| Pre-patch Findbugs warnings |
https://builds.apache.org/job/PreCommit-MAPREDUCE-Build/5771/artifact/patchprocess/trunkFindbugsWarningshadoop-mapreduce-client-app.html
|
| javac |
https://builds.apache.org/job/PreCommit-MAPREDUCE-Build/5771/artifact/patchprocess/diffJavacWarnings.txt
|
| checkstyle |
https://builds.apache.org/job/PreCommit-MAPREDUCE-Build/5771/artifact/patchprocess/diffcheckstylehadoop-mapreduce-client-app.txt
|
| whitespace |
https://builds.apache.org/job/PreCommit-MAPREDUCE-Build/5771/artifact/patchprocess/whitespace.txt
|
| hadoop-mapreduce-client-app test log |
https://builds.apache.org/job/PreCommit-MAPREDUCE-Build/5771/artifact/patchprocess/testrun_hadoop-mapreduce-client-app.txt
|
| Test Results |
https://builds.apache.org/job/PreCommit-MAPREDUCE-Build/5771/testReport/ |
| Java | 1.7.0_55 |
| uname | Linux asf901.gq1.ygridcore.net 3.13.0-36-lowlatency #63-Ubuntu SMP
PREEMPT Wed Sep 3 21:56:12 UTC 2014 x86_64 x86_64 x86_64 GNU/Linux |
| Console output |
https://builds.apache.org/job/PreCommit-MAPREDUCE-Build/5771/console |
This message was automatically generated.
add the last reporting reducer host info and attempt id on the map error
message due to too many fetch failure
--
Key: MAPREDUCE-6384
URL: https://issues.apache.org/jira/browse/MAPREDUCE-6384
Project: Hadoop Map/Reduce
Issue Type: Bug
Reporter: Chang Li
Assignee: Chang Li
Attachments: MAPREDUCE-6384.2.patch, MAPREDUCE-6384.3.patch,
MAPREDUCE-6384.4.patch
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Updated] (MAPREDUCE-6387) Fix Task object to allow Jobs not submitted via distributed cache to survive rolling upgrade
[
https://issues.apache.org/jira/browse/MAPREDUCE-6387?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Arun Suresh updated MAPREDUCE-6387:
---
Summary: Fix Task object to allow Jobs not submitted via distributed cache
to survive rolling upgrade (was: Fix Task object to serialize
encryptedSpillKey at the end to to allow rolling upgrades without distributed
cache to work)
Fix Task object to allow Jobs not submitted via distributed cache to survive
rolling upgrade
Key: MAPREDUCE-6387
URL: https://issues.apache.org/jira/browse/MAPREDUCE-6387
Project: Hadoop Map/Reduce
Issue Type: Bug
Reporter: Arun Suresh
Assignee: Arun Suresh
Priority: Minor
There was a recent addition of an {{encryptedSpillKey}} to the Task object.
And when serialized, this field was written out somewhere in the middle. This
caused deployments that do not use DistributedCache to push job jars before
running the job to fail rolling upgrade.
Although deploying via Distributed Cache is the recommended method, there
might still be deployments that use the node local classpath to pick up the
MR framework classes (eg. for efficiency purposes, since this does not
require the jar being copied to hdfs and then to all the nodes)
Ensuring that it is the last field written and read when the Task object is
serialized would alleviate this issue.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Commented] (MAPREDUCE-6354) shuffle handler should log connection info
[
https://issues.apache.org/jira/browse/MAPREDUCE-6354?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14573146#comment-14573146
]
Hadoop QA commented on MAPREDUCE-6354:
--
\\
\\
| (x) *{color:red}-1 overall{color}* |
\\
\\
|| Vote || Subsystem || Runtime || Comment ||
| {color:blue}0{color} | pre-patch | 17m 0s | Pre-patch trunk compilation is
healthy. |
| {color:green}+1{color} | @author | 0m 0s | The patch does not contain any
@author tags. |
| {color:red}-1{color} | tests included | 0m 0s | The patch doesn't appear
to include any new or modified tests. Please justify why no new tests are
needed for this patch. Also please list what manual steps were performed to
verify this patch. |
| {color:green}+1{color} | javac | 7m 32s | There were no new javac warning
messages. |
| {color:green}+1{color} | javadoc | 9m 32s | There were no new javadoc
warning messages. |
| {color:green}+1{color} | release audit | 0m 23s | The applied patch does
not increase the total number of release audit warnings. |
| {color:green}+1{color} | checkstyle | 1m 27s | There were no new checkstyle
issues. |
| {color:red}-1{color} | whitespace | 0m 0s | The patch has 1 line(s) that
end in whitespace. Use git apply --whitespace=fix. |
| {color:green}+1{color} | install | 1m 32s | mvn install still works. |
| {color:green}+1{color} | eclipse:eclipse | 0m 33s | The patch built with
eclipse:eclipse. |
| {color:green}+1{color} | findbugs | 2m 34s | The patch does not introduce
any new Findbugs (version 3.0.0) warnings. |
| {color:green}+1{color} | common tests | 23m 1s | Tests passed in
hadoop-common. |
| {color:green}+1{color} | mapreduce tests | 0m 20s | Tests passed in
hadoop-mapreduce-client-shuffle. |
| | | 63m 58s | |
\\
\\
|| Subsystem || Report/Notes ||
| Patch URL |
http://issues.apache.org/jira/secure/attachment/12737602/MAPREDUCE-6354.7.patch
|
| Optional Tests | javadoc javac unit findbugs checkstyle |
| git revision | trunk / d7e7f6a |
| whitespace |
https://builds.apache.org/job/PreCommit-MAPREDUCE-Build/5772/artifact/patchprocess/whitespace.txt
|
| hadoop-common test log |
https://builds.apache.org/job/PreCommit-MAPREDUCE-Build/5772/artifact/patchprocess/testrun_hadoop-common.txt
|
| hadoop-mapreduce-client-shuffle test log |
https://builds.apache.org/job/PreCommit-MAPREDUCE-Build/5772/artifact/patchprocess/testrun_hadoop-mapreduce-client-shuffle.txt
|
| Test Results |
https://builds.apache.org/job/PreCommit-MAPREDUCE-Build/5772/testReport/ |
| Java | 1.7.0_55 |
| uname | Linux asf905.gq1.ygridcore.net 3.13.0-36-lowlatency #63-Ubuntu SMP
PREEMPT Wed Sep 3 21:56:12 UTC 2014 x86_64 x86_64 x86_64 GNU/Linux |
| Console output |
https://builds.apache.org/job/PreCommit-MAPREDUCE-Build/5772/console |
This message was automatically generated.
shuffle handler should log connection info
--
Key: MAPREDUCE-6354
URL: https://issues.apache.org/jira/browse/MAPREDUCE-6354
Project: Hadoop Map/Reduce
Issue Type: Bug
Reporter: Chang Li
Assignee: Chang Li
Attachments: MAPREDUCE-6354.2.patch, MAPREDUCE-6354.3.patch,
MAPREDUCE-6354.4.patch, MAPREDUCE-6354.5.patch, MAPREDUCE-6354.6.patch,
MAPREDUCE-6354.7.patch, MAPREDUCE-6354.patch
currently, shuffle handler only log connection info in debug mode, we want to
log that info in a more concise way
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Updated] (MAPREDUCE-6387) Fix Task object to serialize encryptedSpillKey at the end to to allow rolling upgrades without distributed cache to work
[
https://issues.apache.org/jira/browse/MAPREDUCE-6387?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Arun Suresh updated MAPREDUCE-6387:
---
Summary: Fix Task object to serialize encryptedSpillKey at the end to to
allow rolling upgrades without distributed cache to work (was: Fix Task.java
to serialize encryptedSpillKey at the end to to allow rolling upgrades without
distributed cache)
Fix Task object to serialize encryptedSpillKey at the end to to allow rolling
upgrades without distributed cache to work
Key: MAPREDUCE-6387
URL: https://issues.apache.org/jira/browse/MAPREDUCE-6387
Project: Hadoop Map/Reduce
Issue Type: Bug
Reporter: Arun Suresh
Assignee: Arun Suresh
Priority: Minor
There was a recent addition of an {{encryptedSpillKey}} to the Task object.
And when serialized, this field was written out somewhere in the middle. This
caused deployments that do not use DistributedCache to push job jars before
running the job to fail rolling upgrade.
Although deploying via Distributed Cache is the recommended method, there
might still be deployments that use the node local classpath to pick up the
MR framework classes (eg. for efficiency purposes, since this does not
require the jar being copied to hdfs and then to all the nodes)
Ensuring that it is the last field written and read when the Task object is
serialized would alleviate this issue.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Created] (MAPREDUCE-6387) Fix Task.java to serialize encryptedSpillKey at the end to to allow rolling upgrades without distributed cache
Arun Suresh created MAPREDUCE-6387:
--
Summary: Fix Task.java to serialize encryptedSpillKey at the end
to to allow rolling upgrades without distributed cache
Key: MAPREDUCE-6387
URL: https://issues.apache.org/jira/browse/MAPREDUCE-6387
Project: Hadoop Map/Reduce
Issue Type: Bug
Reporter: Arun Suresh
Assignee: Arun Suresh
Priority: Minor
There was a recent addition of an {{encryptedSpillKey}} to the Task object. And
when serialized, this field was written out somewhere in the middle. This
caused deployments that do not use DistributedCache to push job jars before
running the job to fail rolling upgrade.
Although deploying via Distributed Cache is the recommended method, there might
still be deployments that use the node local classpath to pick up the MR
framework classes (eg. for efficiency purposes, since this does not require the
jar being copied to hdfs and then to all the nodes)
Ensuring that it is the last field written and read when the Task object is
serialized would alleviate this issue.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Updated] (MAPREDUCE-6387) Fix Task object to allow Jobs not submitted via distributed cache to survive rolling upgrade
[
https://issues.apache.org/jira/browse/MAPREDUCE-6387?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Arun Suresh updated MAPREDUCE-6387:
---
Attachment: MAPREDUCE-6387.1.patch
Attaching trivial patch to fix this.
Not adding test-cases since it is just reordering of a few fields
Fix Task object to allow Jobs not submitted via distributed cache to survive
rolling upgrade
Key: MAPREDUCE-6387
URL: https://issues.apache.org/jira/browse/MAPREDUCE-6387
Project: Hadoop Map/Reduce
Issue Type: Bug
Reporter: Arun Suresh
Assignee: Arun Suresh
Priority: Minor
Attachments: MAPREDUCE-6387.1.patch
There was a recent addition of an {{encryptedSpillKey}} to the Task object.
And when serialized, this field was written out somewhere in the middle. This
caused deployments that do not use DistributedCache to push job jars before
running the job to fail rolling upgrade.
Although deploying via Distributed Cache is the recommended method, there
might still be deployments that use the node local classpath to pick up the
MR framework classes (eg. for efficiency purposes, since this does not
require the jar being copied to hdfs and then to all the nodes)
Ensuring that it is the last field written and read when the Task object is
serialized would alleviate this issue.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Commented] (MAPREDUCE-3418) If map output is not found, shuffle runs in tight loop
[ https://issues.apache.org/jira/browse/MAPREDUCE-3418?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14572996#comment-14572996 ] sahitya pavurala commented on MAPREDUCE-3418: - Is this issue resolved ? If not, i will look into this issue. If map output is not found, shuffle runs in tight loop -- Key: MAPREDUCE-3418 URL: https://issues.apache.org/jira/browse/MAPREDUCE-3418 Project: Hadoop Map/Reduce Issue Type: Bug Affects Versions: 0.23.0, 2.3.0 Reporter: John George Assignee: Chen He Sharad Agarwal bumped into this while simulating fetch failures. Removed the map output directory. Shuffle runs in tight loop throwing : 2011-06-01 09:02:20,511 WARN org.apache.hadoop.mapreduce.task.reduce.Fetcher: Invalid map id java.lang.IllegalArgumentException: TaskAttemptId string : TTP/1.1 500 Internal Server Error Content-Type: text/plain; charset=UTF is not properly formed at org.apache.hadoop.mapreduce.TaskAttemptID.forName(TaskAttemptID.java:174) at org.apache.hadoop.mapreduce.task.reduce.Fetcher.copyMapOutput(Fetcher.java:284) at org.apache.hadoop.mapreduce.task.reduce.Fetcher.copyFromHost(Fetcher.java:251) at org.apache.hadoop.mapreduce.task.reduce.Fetcher.run(Fetcher.java:149) Fetch failure is not triggered. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (MAPREDUCE-6174) Combine common stream code into parent class for InMemoryMapOutput and OnDiskMapOutput.
[ https://issues.apache.org/jira/browse/MAPREDUCE-6174?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14573009#comment-14573009 ] Hudson commented on MAPREDUCE-6174: --- FAILURE: Integrated in Hadoop-Mapreduce-trunk-Java8 #216 (See [https://builds.apache.org/job/Hadoop-Mapreduce-trunk-Java8/216/]) MAPREDUCE-6174. Combine common stream code into parent class for InMemoryMapOutput and OnDiskMapOutput. (Eric Payne via gera) (gera: rev d90c13e2da8867661bf19a802add70145ab9a462) * hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-core/src/main/java/org/apache/hadoop/mapreduce/task/reduce/MergeManagerImpl.java * hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-core/src/main/java/org/apache/hadoop/mapreduce/task/reduce/OnDiskMapOutput.java * hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-core/src/main/java/org/apache/hadoop/mapreduce/task/reduce/InMemoryMapOutput.java * hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-core/src/test/java/org/apache/hadoop/mapreduce/task/reduce/TestFetcher.java * hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-core/src/main/java/org/apache/hadoop/mapreduce/task/reduce/IFileWrappedMapOutput.java * hadoop-mapreduce-project/CHANGES.txt Combine common stream code into parent class for InMemoryMapOutput and OnDiskMapOutput. --- Key: MAPREDUCE-6174 URL: https://issues.apache.org/jira/browse/MAPREDUCE-6174 Project: Hadoop Map/Reduce Issue Type: Improvement Components: mrv2 Affects Versions: 3.0.0, 2.6.0 Reporter: Eric Payne Assignee: Eric Payne Labels: BB2015-05-RFC Fix For: 2.8.0 Attachments: MAPREDUCE-6174.002.patch, MAPREDUCE-6174.003.patch, MAPREDUCE-6174.004.patch, MAPREDUCE-6174.005.patch, MAPREDUCE-6174.006.patch, MAPREDUCE-6174.007.patch, MAPREDUCE-6174.v1.txt Per MAPREDUCE-6166, both InMemoryMapOutput and OnDiskMapOutput will be doing similar things with regards to IFile streams. In order to make it explicit that InMemoryMapOutput and OnDiskMapOutput are different from 3rd-party implementations, this JIRA will make them subclass a common class (see https://issues.apache.org/jira/browse/MAPREDUCE-6166?focusedCommentId=14223368page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel#comment-14223368) -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (MAPREDUCE-6374) Distributed Cache File visibility should check permission of full path
[ https://issues.apache.org/jira/browse/MAPREDUCE-6374?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14573010#comment-14573010 ] Hudson commented on MAPREDUCE-6374: --- FAILURE: Integrated in Hadoop-Mapreduce-trunk-Java8 #216 (See [https://builds.apache.org/job/Hadoop-Mapreduce-trunk-Java8/216/]) MAPREDUCE-6374. Distributed Cache File visibility should check permission of full path. Contributed by Chang Li (jlowe: rev 107da29ff9b6efd02b21c96df60c58ac51590ebd) * hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-core/src/main/java/org/apache/hadoop/mapreduce/filecache/ClientDistributedCacheManager.java * hadoop-mapreduce-project/CHANGES.txt * hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-core/src/test/java/org/apache/hadoop/mapreduce/filecache/TestClientDistributedCacheManager.java Distributed Cache File visibility should check permission of full path -- Key: MAPREDUCE-6374 URL: https://issues.apache.org/jira/browse/MAPREDUCE-6374 Project: Hadoop Map/Reduce Issue Type: Bug Reporter: Chang Li Assignee: Chang Li Fix For: 2.8.0 Attachments: MAPREDUCE-6374.1.patch should do full ancestor permission check for a relative cache file input -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Updated] (MAPREDUCE-6354) shuffle handler should log connection info
[ https://issues.apache.org/jira/browse/MAPREDUCE-6354?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Chang Li updated MAPREDUCE-6354: Attachment: MAPREDUCE-6354.7.patch shuffle handler should log connection info -- Key: MAPREDUCE-6354 URL: https://issues.apache.org/jira/browse/MAPREDUCE-6354 Project: Hadoop Map/Reduce Issue Type: Bug Reporter: Chang Li Assignee: Chang Li Attachments: MAPREDUCE-6354.2.patch, MAPREDUCE-6354.3.patch, MAPREDUCE-6354.4.patch, MAPREDUCE-6354.5.patch, MAPREDUCE-6354.6.patch, MAPREDUCE-6354.7.patch, MAPREDUCE-6354.patch currently, shuffle handler only log connection info in debug mode, we want to log that info in a more concise way -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (MAPREDUCE-5965) Hadoop streaming throws error if list of input files is high. Error is: error=7, Argument list too long at if number of input file is high
[
https://issues.apache.org/jira/browse/MAPREDUCE-5965?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14573036#comment-14573036
]
Hudson commented on MAPREDUCE-5965:
---
FAILURE: Integrated in Hadoop-Mapreduce-trunk #2164 (See
[https://builds.apache.org/job/Hadoop-Mapreduce-trunk/2164/])
MAPREDUCE-5965. Hadoop streaming throws error if list of input files is high.
Error is: error=7, Argument list too long at if number of input file is high
(wilfreds via rkanter) (rkanter: rev cc70df98e74142331043a611a3bd8a53ff6a2242)
*
hadoop-tools/hadoop-streaming/src/main/java/org/apache/hadoop/streaming/StreamJob.java
* hadoop-tools/hadoop-streaming/src/site/markdown/HadoopStreaming.md.vm
* hadoop-mapreduce-project/CHANGES.txt
*
hadoop-tools/hadoop-streaming/src/main/java/org/apache/hadoop/streaming/PipeMapRed.java
Hadoop streaming throws error if list of input files is high. Error is:
error=7, Argument list too long at if number of input file is high
Key: MAPREDUCE-5965
URL: https://issues.apache.org/jira/browse/MAPREDUCE-5965
Project: Hadoop Map/Reduce
Issue Type: Bug
Reporter: Arup Malakar
Assignee: Wilfred Spiegelenburg
Fix For: 2.8.0
Attachments: MAPREDUCE-5965.1.patch, MAPREDUCE-5965.2.patch,
MAPREDUCE-5965.3.patch, MAPREDUCE-5965.patch
Hadoop streaming exposes all the key values in job conf as environment
variables when it forks a process for streaming code to run. Unfortunately
the variable mapreduce_input_fileinputformat_inputdir contains the list of
input files, and Linux has a limit on size of environment variables +
arguments.
Based on how long the list of files and their full path is this could be
pretty huge. And given all of these variables are not even used it stops user
from running hadoop job with large number of files, even though it could be
run.
Linux throws E2BIG if the size is greater than certain size which is error
code 7. And java translates that to error=7, Argument list too long. More:
http://man7.org/linux/man-pages/man2/execve.2.html I suggest skipping
variables if it is greater than certain length. That way if user code
requires the environment variable it would fail. It should also introduce a
config variable to skip long variables, and set it to false by default. That
way user has to specifically set it to true to invoke this feature.
Here is the exception:
{code}
Error: java.lang.RuntimeException: Error in configuring object at
org.apache.hadoop.util.ReflectionUtils.setJobConf(ReflectionUtils.java:109)
at org.apache.hadoop.util.ReflectionUtils.setConf(ReflectionUtils.java:75) at
org.apache.hadoop.util.ReflectionUtils.newInstance(ReflectionUtils.java:133)
at org.apache.hadoop.mapred.MapTask.runOldMapper(MapTask.java:426) at
org.apache.hadoop.mapred.MapTask.run(MapTask.java:342) at
org.apache.hadoop.mapred.YarnChild$2.run(YarnChild.java:168) at
java.security.AccessController.doPrivileged(Native Method) at
javax.security.auth.Subject.doAs(Subject.java:415) at
org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1548)
at org.apache.hadoop.mapred.YarnChild.main(YarnChild.java:163) Caused by:
java.lang.reflect.InvocationTargetException at
sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method) at
sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:57)
at
sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
at java.lang.reflect.Method.invoke(Method.java:606) at
org.apache.hadoop.util.ReflectionUtils.setJobConf(ReflectionUtils.java:106)
... 9 more Caused by: java.lang.RuntimeException: Error in configuring object
at
org.apache.hadoop.util.ReflectionUtils.setJobConf(ReflectionUtils.java:109)
at org.apache.hadoop.util.ReflectionUtils.setConf(ReflectionUtils.java:75) at
org.apache.hadoop.util.ReflectionUtils.newInstance(ReflectionUtils.java:133)
at org.apache.hadoop.mapred.MapRunner.configure(MapRunner.java:38) ... 14
more Caused by: java.lang.reflect.InvocationTargetException at
sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method) at
sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:57)
at
sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
at java.lang.reflect.Method.invoke(Method.java:606) at
org.apache.hadoop.util.ReflectionUtils.setJobConf(ReflectionUtils.java:106)
... 17 more Caused by: java.lang.RuntimeException: configuration exception at
org.apache.hadoop.streaming.PipeMapRed.configure(PipeMapRed.java:222) at
org.apache.hadoop.streaming.PipeMapper.configure(PipeMapper.java:66) ... 22
more Caused by:
[jira] [Commented] (MAPREDUCE-6374) Distributed Cache File visibility should check permission of full path
[ https://issues.apache.org/jira/browse/MAPREDUCE-6374?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14573038#comment-14573038 ] Hudson commented on MAPREDUCE-6374: --- FAILURE: Integrated in Hadoop-Mapreduce-trunk #2164 (See [https://builds.apache.org/job/Hadoop-Mapreduce-trunk/2164/]) MAPREDUCE-6374. Distributed Cache File visibility should check permission of full path. Contributed by Chang Li (jlowe: rev 107da29ff9b6efd02b21c96df60c58ac51590ebd) * hadoop-mapreduce-project/CHANGES.txt * hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-core/src/main/java/org/apache/hadoop/mapreduce/filecache/ClientDistributedCacheManager.java * hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-core/src/test/java/org/apache/hadoop/mapreduce/filecache/TestClientDistributedCacheManager.java Distributed Cache File visibility should check permission of full path -- Key: MAPREDUCE-6374 URL: https://issues.apache.org/jira/browse/MAPREDUCE-6374 Project: Hadoop Map/Reduce Issue Type: Bug Reporter: Chang Li Assignee: Chang Li Fix For: 2.8.0 Attachments: MAPREDUCE-6374.1.patch should do full ancestor permission check for a relative cache file input -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (MAPREDUCE-6382) Don't escape HTML links in Diagnostics in JHS job overview
[ https://issues.apache.org/jira/browse/MAPREDUCE-6382?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14573035#comment-14573035 ] Hudson commented on MAPREDUCE-6382: --- FAILURE: Integrated in Hadoop-Mapreduce-trunk #2164 (See [https://builds.apache.org/job/Hadoop-Mapreduce-trunk/2164/]) MAPREDUCE-6382. Don't escape HTML links in Diagnostics in JHS job overview. (Siqi Li via gera) (gera: rev e8302071ff16247a3d0cfc2dcca02da1bbf4db18) * hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-hs/src/main/java/org/apache/hadoop/mapreduce/v2/hs/webapp/HsJobBlock.java * hadoop-mapreduce-project/CHANGES.txt Don't escape HTML links in Diagnostics in JHS job overview -- Key: MAPREDUCE-6382 URL: https://issues.apache.org/jira/browse/MAPREDUCE-6382 Project: Hadoop Map/Reduce Issue Type: Bug Affects Versions: 2.8.0 Reporter: Siqi Li Assignee: Siqi Li Fix For: 2.8.0 Attachments: MAPREDUCE-6382.v1.patch, links are escaped.jpg for some reason, links are working properly in 2.4, but they are escaped in 2.6 -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (MAPREDUCE-6174) Combine common stream code into parent class for InMemoryMapOutput and OnDiskMapOutput.
[ https://issues.apache.org/jira/browse/MAPREDUCE-6174?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14573037#comment-14573037 ] Hudson commented on MAPREDUCE-6174: --- FAILURE: Integrated in Hadoop-Mapreduce-trunk #2164 (See [https://builds.apache.org/job/Hadoop-Mapreduce-trunk/2164/]) MAPREDUCE-6174. Combine common stream code into parent class for InMemoryMapOutput and OnDiskMapOutput. (Eric Payne via gera) (gera: rev d90c13e2da8867661bf19a802add70145ab9a462) * hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-core/src/main/java/org/apache/hadoop/mapreduce/task/reduce/MergeManagerImpl.java * hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-core/src/main/java/org/apache/hadoop/mapreduce/task/reduce/OnDiskMapOutput.java * hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-core/src/main/java/org/apache/hadoop/mapreduce/task/reduce/InMemoryMapOutput.java * hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-core/src/main/java/org/apache/hadoop/mapreduce/task/reduce/IFileWrappedMapOutput.java * hadoop-mapreduce-project/CHANGES.txt * hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-core/src/test/java/org/apache/hadoop/mapreduce/task/reduce/TestFetcher.java Combine common stream code into parent class for InMemoryMapOutput and OnDiskMapOutput. --- Key: MAPREDUCE-6174 URL: https://issues.apache.org/jira/browse/MAPREDUCE-6174 Project: Hadoop Map/Reduce Issue Type: Improvement Components: mrv2 Affects Versions: 3.0.0, 2.6.0 Reporter: Eric Payne Assignee: Eric Payne Labels: BB2015-05-RFC Fix For: 2.8.0 Attachments: MAPREDUCE-6174.002.patch, MAPREDUCE-6174.003.patch, MAPREDUCE-6174.004.patch, MAPREDUCE-6174.005.patch, MAPREDUCE-6174.006.patch, MAPREDUCE-6174.007.patch, MAPREDUCE-6174.v1.txt Per MAPREDUCE-6166, both InMemoryMapOutput and OnDiskMapOutput will be doing similar things with regards to IFile streams. In order to make it explicit that InMemoryMapOutput and OnDiskMapOutput are different from 3rd-party implementations, this JIRA will make them subclass a common class (see https://issues.apache.org/jira/browse/MAPREDUCE-6166?focusedCommentId=14223368page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel#comment-14223368) -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (MAPREDUCE-6354) shuffle handler should log connection info
[ https://issues.apache.org/jira/browse/MAPREDUCE-6354?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14573047#comment-14573047 ] Chang Li commented on MAPREDUCE-6354: - [~jlowe] thanks a lot for thoughtful review and hearty discussion of how to make this logging more efficient! I have made changes of debug level logging. As for the trace level logging, I think we could wait till this get committed and file another jira to address that issue. Let me know what you think of the latest patch. Thanks! shuffle handler should log connection info -- Key: MAPREDUCE-6354 URL: https://issues.apache.org/jira/browse/MAPREDUCE-6354 Project: Hadoop Map/Reduce Issue Type: Bug Reporter: Chang Li Assignee: Chang Li Attachments: MAPREDUCE-6354.2.patch, MAPREDUCE-6354.3.patch, MAPREDUCE-6354.4.patch, MAPREDUCE-6354.5.patch, MAPREDUCE-6354.6.patch, MAPREDUCE-6354.7.patch, MAPREDUCE-6354.patch currently, shuffle handler only log connection info in debug mode, we want to log that info in a more concise way -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Updated] (MAPREDUCE-6384) add the last reporting reducer host info and attempt id on the map error message due to too many fetch failure
[ https://issues.apache.org/jira/browse/MAPREDUCE-6384?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Chang Li updated MAPREDUCE-6384: Attachment: MAPREDUCE-6384.4.2.patch add the last reporting reducer host info and attempt id on the map error message due to too many fetch failure -- Key: MAPREDUCE-6384 URL: https://issues.apache.org/jira/browse/MAPREDUCE-6384 Project: Hadoop Map/Reduce Issue Type: Bug Reporter: Chang Li Assignee: Chang Li Attachments: MAPREDUCE-6384.2.patch, MAPREDUCE-6384.3.patch, MAPREDUCE-6384.4.2.patch, MAPREDUCE-6384.4.patch -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (MAPREDUCE-6174) Combine common stream code into parent class for InMemoryMapOutput and OnDiskMapOutput.
[ https://issues.apache.org/jira/browse/MAPREDUCE-6174?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14572892#comment-14572892 ] Hudson commented on MAPREDUCE-6174: --- SUCCESS: Integrated in Hadoop-Hdfs-trunk-Java8 #207 (See [https://builds.apache.org/job/Hadoop-Hdfs-trunk-Java8/207/]) MAPREDUCE-6174. Combine common stream code into parent class for InMemoryMapOutput and OnDiskMapOutput. (Eric Payne via gera) (gera: rev d90c13e2da8867661bf19a802add70145ab9a462) * hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-core/src/main/java/org/apache/hadoop/mapreduce/task/reduce/IFileWrappedMapOutput.java * hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-core/src/test/java/org/apache/hadoop/mapreduce/task/reduce/TestFetcher.java * hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-core/src/main/java/org/apache/hadoop/mapreduce/task/reduce/InMemoryMapOutput.java * hadoop-mapreduce-project/CHANGES.txt * hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-core/src/main/java/org/apache/hadoop/mapreduce/task/reduce/MergeManagerImpl.java * hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-core/src/main/java/org/apache/hadoop/mapreduce/task/reduce/OnDiskMapOutput.java Combine common stream code into parent class for InMemoryMapOutput and OnDiskMapOutput. --- Key: MAPREDUCE-6174 URL: https://issues.apache.org/jira/browse/MAPREDUCE-6174 Project: Hadoop Map/Reduce Issue Type: Improvement Components: mrv2 Affects Versions: 3.0.0, 2.6.0 Reporter: Eric Payne Assignee: Eric Payne Labels: BB2015-05-RFC Fix For: 2.8.0 Attachments: MAPREDUCE-6174.002.patch, MAPREDUCE-6174.003.patch, MAPREDUCE-6174.004.patch, MAPREDUCE-6174.005.patch, MAPREDUCE-6174.006.patch, MAPREDUCE-6174.007.patch, MAPREDUCE-6174.v1.txt Per MAPREDUCE-6166, both InMemoryMapOutput and OnDiskMapOutput will be doing similar things with regards to IFile streams. In order to make it explicit that InMemoryMapOutput and OnDiskMapOutput are different from 3rd-party implementations, this JIRA will make them subclass a common class (see https://issues.apache.org/jira/browse/MAPREDUCE-6166?focusedCommentId=14223368page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel#comment-14223368) -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (MAPREDUCE-6374) Distributed Cache File visibility should check permission of full path
[ https://issues.apache.org/jira/browse/MAPREDUCE-6374?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14572893#comment-14572893 ] Hudson commented on MAPREDUCE-6374: --- SUCCESS: Integrated in Hadoop-Hdfs-trunk-Java8 #207 (See [https://builds.apache.org/job/Hadoop-Hdfs-trunk-Java8/207/]) MAPREDUCE-6374. Distributed Cache File visibility should check permission of full path. Contributed by Chang Li (jlowe: rev 107da29ff9b6efd02b21c96df60c58ac51590ebd) * hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-core/src/main/java/org/apache/hadoop/mapreduce/filecache/ClientDistributedCacheManager.java * hadoop-mapreduce-project/CHANGES.txt * hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-core/src/test/java/org/apache/hadoop/mapreduce/filecache/TestClientDistributedCacheManager.java Distributed Cache File visibility should check permission of full path -- Key: MAPREDUCE-6374 URL: https://issues.apache.org/jira/browse/MAPREDUCE-6374 Project: Hadoop Map/Reduce Issue Type: Bug Reporter: Chang Li Assignee: Chang Li Fix For: 2.8.0 Attachments: MAPREDUCE-6374.1.patch should do full ancestor permission check for a relative cache file input -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (MAPREDUCE-5965) Hadoop streaming throws error if list of input files is high. Error is: error=7, Argument list too long at if number of input file is high
[
https://issues.apache.org/jira/browse/MAPREDUCE-5965?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14572891#comment-14572891
]
Hudson commented on MAPREDUCE-5965:
---
SUCCESS: Integrated in Hadoop-Hdfs-trunk-Java8 #207 (See
[https://builds.apache.org/job/Hadoop-Hdfs-trunk-Java8/207/])
MAPREDUCE-5965. Hadoop streaming throws error if list of input files is high.
Error is: error=7, Argument list too long at if number of input file is high
(wilfreds via rkanter) (rkanter: rev cc70df98e74142331043a611a3bd8a53ff6a2242)
*
hadoop-tools/hadoop-streaming/src/main/java/org/apache/hadoop/streaming/StreamJob.java
* hadoop-tools/hadoop-streaming/src/site/markdown/HadoopStreaming.md.vm
* hadoop-mapreduce-project/CHANGES.txt
*
hadoop-tools/hadoop-streaming/src/main/java/org/apache/hadoop/streaming/PipeMapRed.java
Hadoop streaming throws error if list of input files is high. Error is:
error=7, Argument list too long at if number of input file is high
Key: MAPREDUCE-5965
URL: https://issues.apache.org/jira/browse/MAPREDUCE-5965
Project: Hadoop Map/Reduce
Issue Type: Bug
Reporter: Arup Malakar
Assignee: Wilfred Spiegelenburg
Fix For: 2.8.0
Attachments: MAPREDUCE-5965.1.patch, MAPREDUCE-5965.2.patch,
MAPREDUCE-5965.3.patch, MAPREDUCE-5965.patch
Hadoop streaming exposes all the key values in job conf as environment
variables when it forks a process for streaming code to run. Unfortunately
the variable mapreduce_input_fileinputformat_inputdir contains the list of
input files, and Linux has a limit on size of environment variables +
arguments.
Based on how long the list of files and their full path is this could be
pretty huge. And given all of these variables are not even used it stops user
from running hadoop job with large number of files, even though it could be
run.
Linux throws E2BIG if the size is greater than certain size which is error
code 7. And java translates that to error=7, Argument list too long. More:
http://man7.org/linux/man-pages/man2/execve.2.html I suggest skipping
variables if it is greater than certain length. That way if user code
requires the environment variable it would fail. It should also introduce a
config variable to skip long variables, and set it to false by default. That
way user has to specifically set it to true to invoke this feature.
Here is the exception:
{code}
Error: java.lang.RuntimeException: Error in configuring object at
org.apache.hadoop.util.ReflectionUtils.setJobConf(ReflectionUtils.java:109)
at org.apache.hadoop.util.ReflectionUtils.setConf(ReflectionUtils.java:75) at
org.apache.hadoop.util.ReflectionUtils.newInstance(ReflectionUtils.java:133)
at org.apache.hadoop.mapred.MapTask.runOldMapper(MapTask.java:426) at
org.apache.hadoop.mapred.MapTask.run(MapTask.java:342) at
org.apache.hadoop.mapred.YarnChild$2.run(YarnChild.java:168) at
java.security.AccessController.doPrivileged(Native Method) at
javax.security.auth.Subject.doAs(Subject.java:415) at
org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1548)
at org.apache.hadoop.mapred.YarnChild.main(YarnChild.java:163) Caused by:
java.lang.reflect.InvocationTargetException at
sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method) at
sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:57)
at
sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
at java.lang.reflect.Method.invoke(Method.java:606) at
org.apache.hadoop.util.ReflectionUtils.setJobConf(ReflectionUtils.java:106)
... 9 more Caused by: java.lang.RuntimeException: Error in configuring object
at
org.apache.hadoop.util.ReflectionUtils.setJobConf(ReflectionUtils.java:109)
at org.apache.hadoop.util.ReflectionUtils.setConf(ReflectionUtils.java:75) at
org.apache.hadoop.util.ReflectionUtils.newInstance(ReflectionUtils.java:133)
at org.apache.hadoop.mapred.MapRunner.configure(MapRunner.java:38) ... 14
more Caused by: java.lang.reflect.InvocationTargetException at
sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method) at
sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:57)
at
sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
at java.lang.reflect.Method.invoke(Method.java:606) at
org.apache.hadoop.util.ReflectionUtils.setJobConf(ReflectionUtils.java:106)
... 17 more Caused by: java.lang.RuntimeException: configuration exception at
org.apache.hadoop.streaming.PipeMapRed.configure(PipeMapRed.java:222) at
org.apache.hadoop.streaming.PipeMapper.configure(PipeMapper.java:66) ... 22
more Caused by:
[jira] [Commented] (MAPREDUCE-6382) Don't escape HTML links in Diagnostics in JHS job overview
[ https://issues.apache.org/jira/browse/MAPREDUCE-6382?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14572862#comment-14572862 ] Hudson commented on MAPREDUCE-6382: --- SUCCESS: Integrated in Hadoop-Hdfs-trunk #2146 (See [https://builds.apache.org/job/Hadoop-Hdfs-trunk/2146/]) MAPREDUCE-6382. Don't escape HTML links in Diagnostics in JHS job overview. (Siqi Li via gera) (gera: rev e8302071ff16247a3d0cfc2dcca02da1bbf4db18) * hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-hs/src/main/java/org/apache/hadoop/mapreduce/v2/hs/webapp/HsJobBlock.java * hadoop-mapreduce-project/CHANGES.txt Don't escape HTML links in Diagnostics in JHS job overview -- Key: MAPREDUCE-6382 URL: https://issues.apache.org/jira/browse/MAPREDUCE-6382 Project: Hadoop Map/Reduce Issue Type: Bug Affects Versions: 2.8.0 Reporter: Siqi Li Assignee: Siqi Li Fix For: 2.8.0 Attachments: MAPREDUCE-6382.v1.patch, links are escaped.jpg for some reason, links are working properly in 2.4, but they are escaped in 2.6 -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (MAPREDUCE-6174) Combine common stream code into parent class for InMemoryMapOutput and OnDiskMapOutput.
[ https://issues.apache.org/jira/browse/MAPREDUCE-6174?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14572864#comment-14572864 ] Hudson commented on MAPREDUCE-6174: --- SUCCESS: Integrated in Hadoop-Hdfs-trunk #2146 (See [https://builds.apache.org/job/Hadoop-Hdfs-trunk/2146/]) MAPREDUCE-6174. Combine common stream code into parent class for InMemoryMapOutput and OnDiskMapOutput. (Eric Payne via gera) (gera: rev d90c13e2da8867661bf19a802add70145ab9a462) * hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-core/src/main/java/org/apache/hadoop/mapreduce/task/reduce/IFileWrappedMapOutput.java * hadoop-mapreduce-project/CHANGES.txt * hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-core/src/main/java/org/apache/hadoop/mapreduce/task/reduce/InMemoryMapOutput.java * hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-core/src/main/java/org/apache/hadoop/mapreduce/task/reduce/OnDiskMapOutput.java * hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-core/src/test/java/org/apache/hadoop/mapreduce/task/reduce/TestFetcher.java * hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-core/src/main/java/org/apache/hadoop/mapreduce/task/reduce/MergeManagerImpl.java Combine common stream code into parent class for InMemoryMapOutput and OnDiskMapOutput. --- Key: MAPREDUCE-6174 URL: https://issues.apache.org/jira/browse/MAPREDUCE-6174 Project: Hadoop Map/Reduce Issue Type: Improvement Components: mrv2 Affects Versions: 3.0.0, 2.6.0 Reporter: Eric Payne Assignee: Eric Payne Labels: BB2015-05-RFC Fix For: 2.8.0 Attachments: MAPREDUCE-6174.002.patch, MAPREDUCE-6174.003.patch, MAPREDUCE-6174.004.patch, MAPREDUCE-6174.005.patch, MAPREDUCE-6174.006.patch, MAPREDUCE-6174.007.patch, MAPREDUCE-6174.v1.txt Per MAPREDUCE-6166, both InMemoryMapOutput and OnDiskMapOutput will be doing similar things with regards to IFile streams. In order to make it explicit that InMemoryMapOutput and OnDiskMapOutput are different from 3rd-party implementations, this JIRA will make them subclass a common class (see https://issues.apache.org/jira/browse/MAPREDUCE-6166?focusedCommentId=14223368page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel#comment-14223368) -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Updated] (MAPREDUCE-6384) add the last reporting reducer host info and attempt id on the map error message due to too many fetch failure
[ https://issues.apache.org/jira/browse/MAPREDUCE-6384?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Chang Li updated MAPREDUCE-6384: Attachment: MAPREDUCE-6384.4.patch add the last reporting reducer host info and attempt id on the map error message due to too many fetch failure -- Key: MAPREDUCE-6384 URL: https://issues.apache.org/jira/browse/MAPREDUCE-6384 Project: Hadoop Map/Reduce Issue Type: Bug Reporter: Chang Li Assignee: Chang Li Attachments: MAPREDUCE-6384.2.patch, MAPREDUCE-6384.3.patch, MAPREDUCE-6384.4.patch -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (MAPREDUCE-6374) Distributed Cache File visibility should check permission of full path
[ https://issues.apache.org/jira/browse/MAPREDUCE-6374?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14572865#comment-14572865 ] Hudson commented on MAPREDUCE-6374: --- SUCCESS: Integrated in Hadoop-Hdfs-trunk #2146 (See [https://builds.apache.org/job/Hadoop-Hdfs-trunk/2146/]) MAPREDUCE-6374. Distributed Cache File visibility should check permission of full path. Contributed by Chang Li (jlowe: rev 107da29ff9b6efd02b21c96df60c58ac51590ebd) * hadoop-mapreduce-project/CHANGES.txt * hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-core/src/main/java/org/apache/hadoop/mapreduce/filecache/ClientDistributedCacheManager.java * hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-core/src/test/java/org/apache/hadoop/mapreduce/filecache/TestClientDistributedCacheManager.java Distributed Cache File visibility should check permission of full path -- Key: MAPREDUCE-6374 URL: https://issues.apache.org/jira/browse/MAPREDUCE-6374 Project: Hadoop Map/Reduce Issue Type: Bug Reporter: Chang Li Assignee: Chang Li Fix For: 2.8.0 Attachments: MAPREDUCE-6374.1.patch should do full ancestor permission check for a relative cache file input -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (MAPREDUCE-5965) Hadoop streaming throws error if list of input files is high. Error is: error=7, Argument list too long at if number of input file is high
[
https://issues.apache.org/jira/browse/MAPREDUCE-5965?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14572863#comment-14572863
]
Hudson commented on MAPREDUCE-5965:
---
SUCCESS: Integrated in Hadoop-Hdfs-trunk #2146 (See
[https://builds.apache.org/job/Hadoop-Hdfs-trunk/2146/])
MAPREDUCE-5965. Hadoop streaming throws error if list of input files is high.
Error is: error=7, Argument list too long at if number of input file is high
(wilfreds via rkanter) (rkanter: rev cc70df98e74142331043a611a3bd8a53ff6a2242)
*
hadoop-tools/hadoop-streaming/src/main/java/org/apache/hadoop/streaming/PipeMapRed.java
*
hadoop-tools/hadoop-streaming/src/main/java/org/apache/hadoop/streaming/StreamJob.java
* hadoop-tools/hadoop-streaming/src/site/markdown/HadoopStreaming.md.vm
* hadoop-mapreduce-project/CHANGES.txt
Hadoop streaming throws error if list of input files is high. Error is:
error=7, Argument list too long at if number of input file is high
Key: MAPREDUCE-5965
URL: https://issues.apache.org/jira/browse/MAPREDUCE-5965
Project: Hadoop Map/Reduce
Issue Type: Bug
Reporter: Arup Malakar
Assignee: Wilfred Spiegelenburg
Fix For: 2.8.0
Attachments: MAPREDUCE-5965.1.patch, MAPREDUCE-5965.2.patch,
MAPREDUCE-5965.3.patch, MAPREDUCE-5965.patch
Hadoop streaming exposes all the key values in job conf as environment
variables when it forks a process for streaming code to run. Unfortunately
the variable mapreduce_input_fileinputformat_inputdir contains the list of
input files, and Linux has a limit on size of environment variables +
arguments.
Based on how long the list of files and their full path is this could be
pretty huge. And given all of these variables are not even used it stops user
from running hadoop job with large number of files, even though it could be
run.
Linux throws E2BIG if the size is greater than certain size which is error
code 7. And java translates that to error=7, Argument list too long. More:
http://man7.org/linux/man-pages/man2/execve.2.html I suggest skipping
variables if it is greater than certain length. That way if user code
requires the environment variable it would fail. It should also introduce a
config variable to skip long variables, and set it to false by default. That
way user has to specifically set it to true to invoke this feature.
Here is the exception:
{code}
Error: java.lang.RuntimeException: Error in configuring object at
org.apache.hadoop.util.ReflectionUtils.setJobConf(ReflectionUtils.java:109)
at org.apache.hadoop.util.ReflectionUtils.setConf(ReflectionUtils.java:75) at
org.apache.hadoop.util.ReflectionUtils.newInstance(ReflectionUtils.java:133)
at org.apache.hadoop.mapred.MapTask.runOldMapper(MapTask.java:426) at
org.apache.hadoop.mapred.MapTask.run(MapTask.java:342) at
org.apache.hadoop.mapred.YarnChild$2.run(YarnChild.java:168) at
java.security.AccessController.doPrivileged(Native Method) at
javax.security.auth.Subject.doAs(Subject.java:415) at
org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1548)
at org.apache.hadoop.mapred.YarnChild.main(YarnChild.java:163) Caused by:
java.lang.reflect.InvocationTargetException at
sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method) at
sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:57)
at
sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
at java.lang.reflect.Method.invoke(Method.java:606) at
org.apache.hadoop.util.ReflectionUtils.setJobConf(ReflectionUtils.java:106)
... 9 more Caused by: java.lang.RuntimeException: Error in configuring object
at
org.apache.hadoop.util.ReflectionUtils.setJobConf(ReflectionUtils.java:109)
at org.apache.hadoop.util.ReflectionUtils.setConf(ReflectionUtils.java:75) at
org.apache.hadoop.util.ReflectionUtils.newInstance(ReflectionUtils.java:133)
at org.apache.hadoop.mapred.MapRunner.configure(MapRunner.java:38) ... 14
more Caused by: java.lang.reflect.InvocationTargetException at
sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method) at
sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:57)
at
sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
at java.lang.reflect.Method.invoke(Method.java:606) at
org.apache.hadoop.util.ReflectionUtils.setJobConf(ReflectionUtils.java:106)
... 17 more Caused by: java.lang.RuntimeException: configuration exception at
org.apache.hadoop.streaming.PipeMapRed.configure(PipeMapRed.java:222) at
org.apache.hadoop.streaming.PipeMapper.configure(PipeMapper.java:66) ... 22
more Caused by:
[jira] [Commented] (MAPREDUCE-6382) Don't escape HTML links in Diagnostics in JHS job overview
[ https://issues.apache.org/jira/browse/MAPREDUCE-6382?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14572890#comment-14572890 ] Hudson commented on MAPREDUCE-6382: --- SUCCESS: Integrated in Hadoop-Hdfs-trunk-Java8 #207 (See [https://builds.apache.org/job/Hadoop-Hdfs-trunk-Java8/207/]) MAPREDUCE-6382. Don't escape HTML links in Diagnostics in JHS job overview. (Siqi Li via gera) (gera: rev e8302071ff16247a3d0cfc2dcca02da1bbf4db18) * hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-hs/src/main/java/org/apache/hadoop/mapreduce/v2/hs/webapp/HsJobBlock.java * hadoop-mapreduce-project/CHANGES.txt Don't escape HTML links in Diagnostics in JHS job overview -- Key: MAPREDUCE-6382 URL: https://issues.apache.org/jira/browse/MAPREDUCE-6382 Project: Hadoop Map/Reduce Issue Type: Bug Affects Versions: 2.8.0 Reporter: Siqi Li Assignee: Siqi Li Fix For: 2.8.0 Attachments: MAPREDUCE-6382.v1.patch, links are escaped.jpg for some reason, links are working properly in 2.4, but they are escaped in 2.6 -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Updated] (MAPREDUCE-6382) Don't escape HTML links in Diagnostics in JHS job overview
[ https://issues.apache.org/jira/browse/MAPREDUCE-6382?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Gera Shegalov updated MAPREDUCE-6382: - Resolution: Fixed Fix Version/s: 2.8.0 Target Version/s: 2.8.0 Hadoop Flags: Reviewed Status: Resolved (was: Patch Available) Thanks [~l201514] for contribution! Committed to trunk and branch-2. Don't escape HTML links in Diagnostics in JHS job overview -- Key: MAPREDUCE-6382 URL: https://issues.apache.org/jira/browse/MAPREDUCE-6382 Project: Hadoop Map/Reduce Issue Type: Bug Affects Versions: 2.8.0 Reporter: Siqi Li Assignee: Siqi Li Fix For: 2.8.0 Attachments: MAPREDUCE-6382.v1.patch, links are escaped.jpg for some reason, links are working properly in 2.4, but they are escaped in 2.6 -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (MAPREDUCE-6382) Don't escape HTML links in Diagnostics in JHS job overview
[ https://issues.apache.org/jira/browse/MAPREDUCE-6382?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14572288#comment-14572288 ] Hudson commented on MAPREDUCE-6382: --- FAILURE: Integrated in Hadoop-trunk-Commit #7962 (See [https://builds.apache.org/job/Hadoop-trunk-Commit/7962/]) MAPREDUCE-6382. Don't escape HTML links in Diagnostics in JHS job overview. (Siqi Li via gera) (gera: rev e8302071ff16247a3d0cfc2dcca02da1bbf4db18) * hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-hs/src/main/java/org/apache/hadoop/mapreduce/v2/hs/webapp/HsJobBlock.java * hadoop-mapreduce-project/CHANGES.txt Don't escape HTML links in Diagnostics in JHS job overview -- Key: MAPREDUCE-6382 URL: https://issues.apache.org/jira/browse/MAPREDUCE-6382 Project: Hadoop Map/Reduce Issue Type: Bug Affects Versions: 2.8.0 Reporter: Siqi Li Assignee: Siqi Li Fix For: 2.8.0 Attachments: MAPREDUCE-6382.v1.patch, links are escaped.jpg for some reason, links are working properly in 2.4, but they are escaped in 2.6 -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Updated] (MAPREDUCE-6332) Provide facility to users for writting custom MergeManager implementation when custom shuffleconsumerPluggin is used
[ https://issues.apache.org/jira/browse/MAPREDUCE-6332?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Rohith updated MAPREDUCE-6332: -- Summary: Provide facility to users for writting custom MergeManager implementation when custom shuffleconsumerPluggin is used (was: Add more required API's to MergeManager interface ) Provide facility to users for writting custom MergeManager implementation when custom shuffleconsumerPluggin is used Key: MAPREDUCE-6332 URL: https://issues.apache.org/jira/browse/MAPREDUCE-6332 Project: Hadoop Map/Reduce Issue Type: New Feature Affects Versions: 2.5.0, 2.6.0, 2.7.0 Reporter: Rohith Assignee: Rohith Labels: BB2015-05-TBR Attachments: 0001-MAPREDUCE-6332.patch, 0002-MAPREDUCE-6332.patch MR provides ability to the user for plugin custom ShuffleConsumerPlugin using *mapreduce.job.reduce.shuffle.consumer.plugin.class*. When the user is allowed to use this configuration as plugin, user also interest in implementing his own MergeManagerImpl. But now , user is forced to use MR provided MergeManagerImpl instead of custom MergeManagerImpl when user is using shuffle.consumer.plugin class. There should be well defined API's in MergeManager that can be used for any implementation without much effort to user for custom implementation. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Created] (MAPREDUCE-6386) refreshLogRetentionSettings is invalid in secure cluster
Zhang Wei created MAPREDUCE-6386: Summary: refreshLogRetentionSettings is invalid in secure cluster Key: MAPREDUCE-6386 URL: https://issues.apache.org/jira/browse/MAPREDUCE-6386 Project: Hadoop Map/Reduce Issue Type: Bug Components: jobhistoryserver Environment: mrV2, secure mode Reporter: Zhang Wei bin/mapred hsadmin -refreshLogRetentionSettings is not available in secure cluster. reproduce steps: 1. startup historyserver in secure cluster. 2. login the secure user with kinit.(the same user with startup step) 3. execute bin/mapred hsadmin -refreshLogRetentionSettings to refresh. there will be exceptions as follows in the historyserver's log if log deletion is enabled. 2015-06-04 14:14:40,070 | ERROR | Timer-3 | Error reading root log dir this deletion attempt is being aborted | AggregatedLogDeletionService.java:127 java.io.IOException: Failed on local exception: java.io.IOException: javax.security.sasl.SaslException: GSS initiate failed [Caused by GSSException: No valid credentials provided (Mechanism level: Failed to find any Kerberos tgt)]; Host Details : local host is: vm-31/9.91.12.31; destination host is: vm-33:25000; at org.apache.hadoop.net.NetUtils.wrapException(NetUtils.java:764) at org.apache.hadoop.ipc.Client.call(Client.java:1414) at org.apache.hadoop.ipc.Client.call(Client.java:1363) at org.apache.hadoop.ipc.ProtobufRpcEngine$Invoker.invoke(ProtobufRpcEngine.java:206) at com.sun.proxy.$Proxy9.getListing(Unknown Source) at org.apache.hadoop.hdfs.protocolPB.ClientNamenodeProtocolTranslatorPB.getListing(ClientNamenodeProtocolTranslatorPB.java:519) at sun.reflect.GeneratedMethodAccessor16.invoke(Unknown Source) at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) at java.lang.reflect.Method.invoke(Method.java:606) at org.apache.hadoop.io.retry.RetryInvocationHandler.invokeMethod(RetryInvocationHandler.java:187) at org.apache.hadoop.io.retry.RetryInvocationHandler.invoke(RetryInvocationHandler.java:102) at com.sun.proxy.$Proxy10.getListing(Unknown Source) at org.apache.hadoop.hdfs.DFSClient.listPaths(DFSClient.java:1767) at org.apache.hadoop.hdfs.DFSClient.listPaths(DFSClient.java:1750) at org.apache.hadoop.hdfs.DistributedFileSystem.listStatusInternal(DistributedFileSystem.java:691) at org.apache.hadoop.hdfs.DistributedFileSystem.access$600(DistributedFileSystem.java:102) at org.apache.hadoop.hdfs.DistributedFileSystem$15.doCall(DistributedFileSystem.java:753) at org.apache.hadoop.hdfs.DistributedFileSystem$15.doCall(DistributedFileSystem.java:749) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.listStatus(DistributedFileSystem.java:749) at org.apache.hadoop.yarn.logaggregation.AggregatedLogDeletionService$LogDeletionTask.run(AggregatedLogDeletionService.java:68) at java.util.TimerThread.mainLoop(Timer.java:555) at java.util.TimerThread.run(Timer.java:505) Caused by: java.io.IOException: javax.security.sasl.SaslException: GSS initiate failed [Caused by GSSException: No valid credentials provided (Mechanism level: Failed to find any Kerberos tgt)] at org.apache.hadoop.ipc.Client$Connection$1.run(Client.java:677) at java.security.AccessController.doPrivileged(Native Method) at javax.security.auth.Subject.doAs(Subject.java:415) at org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1641) at org.apache.hadoop.ipc.Client$Connection.handleSaslConnectionFailure(Client.java:640) at org.apache.hadoop.ipc.Client$Connection.setupIOstreams(Client.java:724) at org.apache.hadoop.ipc.Client$Connection.access$2800(Client.java:367) at org.apache.hadoop.ipc.Client.getConnection(Client.java:1462) at org.apache.hadoop.ipc.Client.call(Client.java:1381) ... 21 more Caused by: javax.security.sasl.SaslException: GSS initiate failed [Caused by GSSException: No valid credentials provided (Mechanism level: Failed to find any Kerberos tgt)] at com.sun.security.sasl.gsskerb.GssKrb5Client.evaluateChallenge(GssKrb5Client.java:212) at org.apache.hadoop.security.SaslRpcClient.saslConnect(SaslRpcClient.java:411) at org.apache.hadoop.ipc.Client$Connection.setupSaslConnection(Client.java:550) at org.apache.hadoop.ipc.Client$Connection.access$1800(Client.java:367) at org.apache.hadoop.ipc.Client$Connection$2.run(Client.java:716) at org.apache.hadoop.ipc.Client$Connection$2.run(Client.java:712) at java.security.AccessController.doPrivileged(Native Method) at
[jira] [Updated] (MAPREDUCE-6382) Don't escape HTML links in Diagnostics in JHS job overview
[ https://issues.apache.org/jira/browse/MAPREDUCE-6382?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Gera Shegalov updated MAPREDUCE-6382: - Summary: Don't escape HTML links in Diagnostics in JHS job overview (was: Links in Diagnostics are escaped in 2.6) Don't escape HTML links in Diagnostics in JHS job overview -- Key: MAPREDUCE-6382 URL: https://issues.apache.org/jira/browse/MAPREDUCE-6382 Project: Hadoop Map/Reduce Issue Type: Bug Affects Versions: 2.8.0 Reporter: Siqi Li Assignee: Siqi Li Attachments: MAPREDUCE-6382.v1.patch, links are escaped.jpg for some reason, links are working properly in 2.4, but they are escaped in 2.6 -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Updated] (MAPREDUCE-6354) shuffle handler should log connection info
[ https://issues.apache.org/jira/browse/MAPREDUCE-6354?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Chang Li updated MAPREDUCE-6354: Attachment: MAPREDUCE-6354.8.patch shuffle handler should log connection info -- Key: MAPREDUCE-6354 URL: https://issues.apache.org/jira/browse/MAPREDUCE-6354 Project: Hadoop Map/Reduce Issue Type: Bug Reporter: Chang Li Assignee: Chang Li Attachments: MAPREDUCE-6354.2.patch, MAPREDUCE-6354.3.patch, MAPREDUCE-6354.4.patch, MAPREDUCE-6354.5.patch, MAPREDUCE-6354.6.patch, MAPREDUCE-6354.7.patch, MAPREDUCE-6354.8.patch, MAPREDUCE-6354.patch currently, shuffle handler only log connection info in debug mode, we want to log that info in a more concise way -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (MAPREDUCE-5965) Hadoop streaming throws error if list of input files is high. Error is: error=7, Argument list too long at if number of input file is high
[
https://issues.apache.org/jira/browse/MAPREDUCE-5965?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14572593#comment-14572593
]
Hudson commented on MAPREDUCE-5965:
---
FAILURE: Integrated in Hadoop-Yarn-trunk-Java8 #218 (See
[https://builds.apache.org/job/Hadoop-Yarn-trunk-Java8/218/])
MAPREDUCE-5965. Hadoop streaming throws error if list of input files is high.
Error is: error=7, Argument list too long at if number of input file is high
(wilfreds via rkanter) (rkanter: rev cc70df98e74142331043a611a3bd8a53ff6a2242)
*
hadoop-tools/hadoop-streaming/src/main/java/org/apache/hadoop/streaming/PipeMapRed.java
* hadoop-mapreduce-project/CHANGES.txt
*
hadoop-tools/hadoop-streaming/src/main/java/org/apache/hadoop/streaming/StreamJob.java
* hadoop-tools/hadoop-streaming/src/site/markdown/HadoopStreaming.md.vm
Hadoop streaming throws error if list of input files is high. Error is:
error=7, Argument list too long at if number of input file is high
Key: MAPREDUCE-5965
URL: https://issues.apache.org/jira/browse/MAPREDUCE-5965
Project: Hadoop Map/Reduce
Issue Type: Bug
Reporter: Arup Malakar
Assignee: Wilfred Spiegelenburg
Fix For: 2.8.0
Attachments: MAPREDUCE-5965.1.patch, MAPREDUCE-5965.2.patch,
MAPREDUCE-5965.3.patch, MAPREDUCE-5965.patch
Hadoop streaming exposes all the key values in job conf as environment
variables when it forks a process for streaming code to run. Unfortunately
the variable mapreduce_input_fileinputformat_inputdir contains the list of
input files, and Linux has a limit on size of environment variables +
arguments.
Based on how long the list of files and their full path is this could be
pretty huge. And given all of these variables are not even used it stops user
from running hadoop job with large number of files, even though it could be
run.
Linux throws E2BIG if the size is greater than certain size which is error
code 7. And java translates that to error=7, Argument list too long. More:
http://man7.org/linux/man-pages/man2/execve.2.html I suggest skipping
variables if it is greater than certain length. That way if user code
requires the environment variable it would fail. It should also introduce a
config variable to skip long variables, and set it to false by default. That
way user has to specifically set it to true to invoke this feature.
Here is the exception:
{code}
Error: java.lang.RuntimeException: Error in configuring object at
org.apache.hadoop.util.ReflectionUtils.setJobConf(ReflectionUtils.java:109)
at org.apache.hadoop.util.ReflectionUtils.setConf(ReflectionUtils.java:75) at
org.apache.hadoop.util.ReflectionUtils.newInstance(ReflectionUtils.java:133)
at org.apache.hadoop.mapred.MapTask.runOldMapper(MapTask.java:426) at
org.apache.hadoop.mapred.MapTask.run(MapTask.java:342) at
org.apache.hadoop.mapred.YarnChild$2.run(YarnChild.java:168) at
java.security.AccessController.doPrivileged(Native Method) at
javax.security.auth.Subject.doAs(Subject.java:415) at
org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1548)
at org.apache.hadoop.mapred.YarnChild.main(YarnChild.java:163) Caused by:
java.lang.reflect.InvocationTargetException at
sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method) at
sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:57)
at
sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
at java.lang.reflect.Method.invoke(Method.java:606) at
org.apache.hadoop.util.ReflectionUtils.setJobConf(ReflectionUtils.java:106)
... 9 more Caused by: java.lang.RuntimeException: Error in configuring object
at
org.apache.hadoop.util.ReflectionUtils.setJobConf(ReflectionUtils.java:109)
at org.apache.hadoop.util.ReflectionUtils.setConf(ReflectionUtils.java:75) at
org.apache.hadoop.util.ReflectionUtils.newInstance(ReflectionUtils.java:133)
at org.apache.hadoop.mapred.MapRunner.configure(MapRunner.java:38) ... 14
more Caused by: java.lang.reflect.InvocationTargetException at
sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method) at
sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:57)
at
sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
at java.lang.reflect.Method.invoke(Method.java:606) at
org.apache.hadoop.util.ReflectionUtils.setJobConf(ReflectionUtils.java:106)
... 17 more Caused by: java.lang.RuntimeException: configuration exception at
org.apache.hadoop.streaming.PipeMapRed.configure(PipeMapRed.java:222) at
org.apache.hadoop.streaming.PipeMapper.configure(PipeMapper.java:66) ... 22
more Caused by:
[jira] [Commented] (MAPREDUCE-6374) Distributed Cache File visibility should check permission of full path
[ https://issues.apache.org/jira/browse/MAPREDUCE-6374?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14572595#comment-14572595 ] Hudson commented on MAPREDUCE-6374: --- FAILURE: Integrated in Hadoop-Yarn-trunk-Java8 #218 (See [https://builds.apache.org/job/Hadoop-Yarn-trunk-Java8/218/]) MAPREDUCE-6374. Distributed Cache File visibility should check permission of full path. Contributed by Chang Li (jlowe: rev 107da29ff9b6efd02b21c96df60c58ac51590ebd) * hadoop-mapreduce-project/CHANGES.txt * hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-core/src/test/java/org/apache/hadoop/mapreduce/filecache/TestClientDistributedCacheManager.java * hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-core/src/main/java/org/apache/hadoop/mapreduce/filecache/ClientDistributedCacheManager.java Distributed Cache File visibility should check permission of full path -- Key: MAPREDUCE-6374 URL: https://issues.apache.org/jira/browse/MAPREDUCE-6374 Project: Hadoop Map/Reduce Issue Type: Bug Reporter: Chang Li Assignee: Chang Li Fix For: 2.8.0 Attachments: MAPREDUCE-6374.1.patch should do full ancestor permission check for a relative cache file input -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (MAPREDUCE-6382) Don't escape HTML links in Diagnostics in JHS job overview
[ https://issues.apache.org/jira/browse/MAPREDUCE-6382?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14572592#comment-14572592 ] Hudson commented on MAPREDUCE-6382: --- FAILURE: Integrated in Hadoop-Yarn-trunk-Java8 #218 (See [https://builds.apache.org/job/Hadoop-Yarn-trunk-Java8/218/]) MAPREDUCE-6382. Don't escape HTML links in Diagnostics in JHS job overview. (Siqi Li via gera) (gera: rev e8302071ff16247a3d0cfc2dcca02da1bbf4db18) * hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-hs/src/main/java/org/apache/hadoop/mapreduce/v2/hs/webapp/HsJobBlock.java * hadoop-mapreduce-project/CHANGES.txt Don't escape HTML links in Diagnostics in JHS job overview -- Key: MAPREDUCE-6382 URL: https://issues.apache.org/jira/browse/MAPREDUCE-6382 Project: Hadoop Map/Reduce Issue Type: Bug Affects Versions: 2.8.0 Reporter: Siqi Li Assignee: Siqi Li Fix For: 2.8.0 Attachments: MAPREDUCE-6382.v1.patch, links are escaped.jpg for some reason, links are working properly in 2.4, but they are escaped in 2.6 -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (MAPREDUCE-6174) Combine common stream code into parent class for InMemoryMapOutput and OnDiskMapOutput.
[ https://issues.apache.org/jira/browse/MAPREDUCE-6174?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14572594#comment-14572594 ] Hudson commented on MAPREDUCE-6174: --- FAILURE: Integrated in Hadoop-Yarn-trunk-Java8 #218 (See [https://builds.apache.org/job/Hadoop-Yarn-trunk-Java8/218/]) MAPREDUCE-6174. Combine common stream code into parent class for InMemoryMapOutput and OnDiskMapOutput. (Eric Payne via gera) (gera: rev d90c13e2da8867661bf19a802add70145ab9a462) * hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-core/src/main/java/org/apache/hadoop/mapreduce/task/reduce/OnDiskMapOutput.java * hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-core/src/main/java/org/apache/hadoop/mapreduce/task/reduce/MergeManagerImpl.java * hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-core/src/main/java/org/apache/hadoop/mapreduce/task/reduce/InMemoryMapOutput.java * hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-core/src/test/java/org/apache/hadoop/mapreduce/task/reduce/TestFetcher.java * hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-core/src/main/java/org/apache/hadoop/mapreduce/task/reduce/IFileWrappedMapOutput.java * hadoop-mapreduce-project/CHANGES.txt Combine common stream code into parent class for InMemoryMapOutput and OnDiskMapOutput. --- Key: MAPREDUCE-6174 URL: https://issues.apache.org/jira/browse/MAPREDUCE-6174 Project: Hadoop Map/Reduce Issue Type: Improvement Components: mrv2 Affects Versions: 3.0.0, 2.6.0 Reporter: Eric Payne Assignee: Eric Payne Labels: BB2015-05-RFC Fix For: 2.8.0 Attachments: MAPREDUCE-6174.002.patch, MAPREDUCE-6174.003.patch, MAPREDUCE-6174.004.patch, MAPREDUCE-6174.005.patch, MAPREDUCE-6174.006.patch, MAPREDUCE-6174.007.patch, MAPREDUCE-6174.v1.txt Per MAPREDUCE-6166, both InMemoryMapOutput and OnDiskMapOutput will be doing similar things with regards to IFile streams. In order to make it explicit that InMemoryMapOutput and OnDiskMapOutput are different from 3rd-party implementations, this JIRA will make them subclass a common class (see https://issues.apache.org/jira/browse/MAPREDUCE-6166?focusedCommentId=14223368page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel#comment-14223368) -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (MAPREDUCE-5965) Hadoop streaming throws error if list of input files is high. Error is: error=7, Argument list too long at if number of input file is high
[
https://issues.apache.org/jira/browse/MAPREDUCE-5965?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14572614#comment-14572614
]
Hudson commented on MAPREDUCE-5965:
---
FAILURE: Integrated in Hadoop-Yarn-trunk #948 (See
[https://builds.apache.org/job/Hadoop-Yarn-trunk/948/])
MAPREDUCE-5965. Hadoop streaming throws error if list of input files is high.
Error is: error=7, Argument list too long at if number of input file is high
(wilfreds via rkanter) (rkanter: rev cc70df98e74142331043a611a3bd8a53ff6a2242)
*
hadoop-tools/hadoop-streaming/src/main/java/org/apache/hadoop/streaming/StreamJob.java
* hadoop-mapreduce-project/CHANGES.txt
* hadoop-tools/hadoop-streaming/src/site/markdown/HadoopStreaming.md.vm
*
hadoop-tools/hadoop-streaming/src/main/java/org/apache/hadoop/streaming/PipeMapRed.java
Hadoop streaming throws error if list of input files is high. Error is:
error=7, Argument list too long at if number of input file is high
Key: MAPREDUCE-5965
URL: https://issues.apache.org/jira/browse/MAPREDUCE-5965
Project: Hadoop Map/Reduce
Issue Type: Bug
Reporter: Arup Malakar
Assignee: Wilfred Spiegelenburg
Fix For: 2.8.0
Attachments: MAPREDUCE-5965.1.patch, MAPREDUCE-5965.2.patch,
MAPREDUCE-5965.3.patch, MAPREDUCE-5965.patch
Hadoop streaming exposes all the key values in job conf as environment
variables when it forks a process for streaming code to run. Unfortunately
the variable mapreduce_input_fileinputformat_inputdir contains the list of
input files, and Linux has a limit on size of environment variables +
arguments.
Based on how long the list of files and their full path is this could be
pretty huge. And given all of these variables are not even used it stops user
from running hadoop job with large number of files, even though it could be
run.
Linux throws E2BIG if the size is greater than certain size which is error
code 7. And java translates that to error=7, Argument list too long. More:
http://man7.org/linux/man-pages/man2/execve.2.html I suggest skipping
variables if it is greater than certain length. That way if user code
requires the environment variable it would fail. It should also introduce a
config variable to skip long variables, and set it to false by default. That
way user has to specifically set it to true to invoke this feature.
Here is the exception:
{code}
Error: java.lang.RuntimeException: Error in configuring object at
org.apache.hadoop.util.ReflectionUtils.setJobConf(ReflectionUtils.java:109)
at org.apache.hadoop.util.ReflectionUtils.setConf(ReflectionUtils.java:75) at
org.apache.hadoop.util.ReflectionUtils.newInstance(ReflectionUtils.java:133)
at org.apache.hadoop.mapred.MapTask.runOldMapper(MapTask.java:426) at
org.apache.hadoop.mapred.MapTask.run(MapTask.java:342) at
org.apache.hadoop.mapred.YarnChild$2.run(YarnChild.java:168) at
java.security.AccessController.doPrivileged(Native Method) at
javax.security.auth.Subject.doAs(Subject.java:415) at
org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1548)
at org.apache.hadoop.mapred.YarnChild.main(YarnChild.java:163) Caused by:
java.lang.reflect.InvocationTargetException at
sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method) at
sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:57)
at
sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
at java.lang.reflect.Method.invoke(Method.java:606) at
org.apache.hadoop.util.ReflectionUtils.setJobConf(ReflectionUtils.java:106)
... 9 more Caused by: java.lang.RuntimeException: Error in configuring object
at
org.apache.hadoop.util.ReflectionUtils.setJobConf(ReflectionUtils.java:109)
at org.apache.hadoop.util.ReflectionUtils.setConf(ReflectionUtils.java:75) at
org.apache.hadoop.util.ReflectionUtils.newInstance(ReflectionUtils.java:133)
at org.apache.hadoop.mapred.MapRunner.configure(MapRunner.java:38) ... 14
more Caused by: java.lang.reflect.InvocationTargetException at
sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method) at
sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:57)
at
sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
at java.lang.reflect.Method.invoke(Method.java:606) at
org.apache.hadoop.util.ReflectionUtils.setJobConf(ReflectionUtils.java:106)
... 17 more Caused by: java.lang.RuntimeException: configuration exception at
org.apache.hadoop.streaming.PipeMapRed.configure(PipeMapRed.java:222) at
org.apache.hadoop.streaming.PipeMapper.configure(PipeMapper.java:66) ... 22
more Caused by:
[jira] [Commented] (MAPREDUCE-6174) Combine common stream code into parent class for InMemoryMapOutput and OnDiskMapOutput.
[ https://issues.apache.org/jira/browse/MAPREDUCE-6174?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14572615#comment-14572615 ] Hudson commented on MAPREDUCE-6174: --- FAILURE: Integrated in Hadoop-Yarn-trunk #948 (See [https://builds.apache.org/job/Hadoop-Yarn-trunk/948/]) MAPREDUCE-6174. Combine common stream code into parent class for InMemoryMapOutput and OnDiskMapOutput. (Eric Payne via gera) (gera: rev d90c13e2da8867661bf19a802add70145ab9a462) * hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-core/src/main/java/org/apache/hadoop/mapreduce/task/reduce/MergeManagerImpl.java * hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-core/src/main/java/org/apache/hadoop/mapreduce/task/reduce/OnDiskMapOutput.java * hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-core/src/main/java/org/apache/hadoop/mapreduce/task/reduce/IFileWrappedMapOutput.java * hadoop-mapreduce-project/CHANGES.txt * hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-core/src/main/java/org/apache/hadoop/mapreduce/task/reduce/InMemoryMapOutput.java * hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-core/src/test/java/org/apache/hadoop/mapreduce/task/reduce/TestFetcher.java Combine common stream code into parent class for InMemoryMapOutput and OnDiskMapOutput. --- Key: MAPREDUCE-6174 URL: https://issues.apache.org/jira/browse/MAPREDUCE-6174 Project: Hadoop Map/Reduce Issue Type: Improvement Components: mrv2 Affects Versions: 3.0.0, 2.6.0 Reporter: Eric Payne Assignee: Eric Payne Labels: BB2015-05-RFC Fix For: 2.8.0 Attachments: MAPREDUCE-6174.002.patch, MAPREDUCE-6174.003.patch, MAPREDUCE-6174.004.patch, MAPREDUCE-6174.005.patch, MAPREDUCE-6174.006.patch, MAPREDUCE-6174.007.patch, MAPREDUCE-6174.v1.txt Per MAPREDUCE-6166, both InMemoryMapOutput and OnDiskMapOutput will be doing similar things with regards to IFile streams. In order to make it explicit that InMemoryMapOutput and OnDiskMapOutput are different from 3rd-party implementations, this JIRA will make them subclass a common class (see https://issues.apache.org/jira/browse/MAPREDUCE-6166?focusedCommentId=14223368page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel#comment-14223368) -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Assigned] (MAPREDUCE-6386) refreshLogRetentionSettings is invalid in secure cluster
[ https://issues.apache.org/jira/browse/MAPREDUCE-6386?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Varun Saxena reassigned MAPREDUCE-6386: --- Assignee: Varun Saxena refreshLogRetentionSettings is invalid in secure cluster Key: MAPREDUCE-6386 URL: https://issues.apache.org/jira/browse/MAPREDUCE-6386 Project: Hadoop Map/Reduce Issue Type: Bug Components: jobhistoryserver Environment: mrV2, secure mode Reporter: Zhang Wei Assignee: Varun Saxena bin/mapred hsadmin -refreshLogRetentionSettings is not available in secure cluster. reproduce steps: 1. startup historyserver in secure cluster. 2. login the secure user with kinit.(the same user with startup step) 3. execute bin/mapred hsadmin -refreshLogRetentionSettings to refresh. there will be exceptions as follows in the historyserver's log if log deletion is enabled. 2015-06-04 14:14:40,070 | ERROR | Timer-3 | Error reading root log dir this deletion attempt is being aborted | AggregatedLogDeletionService.java:127 java.io.IOException: Failed on local exception: java.io.IOException: javax.security.sasl.SaslException: GSS initiate failed [Caused by GSSException: No valid credentials provided (Mechanism level: Failed to find any Kerberos tgt)]; Host Details : local host is: vm-31/9.91.12.31; destination host is: vm-33:25000; at org.apache.hadoop.net.NetUtils.wrapException(NetUtils.java:764) at org.apache.hadoop.ipc.Client.call(Client.java:1414) at org.apache.hadoop.ipc.Client.call(Client.java:1363) at org.apache.hadoop.ipc.ProtobufRpcEngine$Invoker.invoke(ProtobufRpcEngine.java:206) at com.sun.proxy.$Proxy9.getListing(Unknown Source) at org.apache.hadoop.hdfs.protocolPB.ClientNamenodeProtocolTranslatorPB.getListing(ClientNamenodeProtocolTranslatorPB.java:519) at sun.reflect.GeneratedMethodAccessor16.invoke(Unknown Source) at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) at java.lang.reflect.Method.invoke(Method.java:606) at org.apache.hadoop.io.retry.RetryInvocationHandler.invokeMethod(RetryInvocationHandler.java:187) at org.apache.hadoop.io.retry.RetryInvocationHandler.invoke(RetryInvocationHandler.java:102) at com.sun.proxy.$Proxy10.getListing(Unknown Source) at org.apache.hadoop.hdfs.DFSClient.listPaths(DFSClient.java:1767) at org.apache.hadoop.hdfs.DFSClient.listPaths(DFSClient.java:1750) at org.apache.hadoop.hdfs.DistributedFileSystem.listStatusInternal(DistributedFileSystem.java:691) at org.apache.hadoop.hdfs.DistributedFileSystem.access$600(DistributedFileSystem.java:102) at org.apache.hadoop.hdfs.DistributedFileSystem$15.doCall(DistributedFileSystem.java:753) at org.apache.hadoop.hdfs.DistributedFileSystem$15.doCall(DistributedFileSystem.java:749) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.listStatus(DistributedFileSystem.java:749) at org.apache.hadoop.yarn.logaggregation.AggregatedLogDeletionService$LogDeletionTask.run(AggregatedLogDeletionService.java:68) at java.util.TimerThread.mainLoop(Timer.java:555) at java.util.TimerThread.run(Timer.java:505) Caused by: java.io.IOException: javax.security.sasl.SaslException: GSS initiate failed [Caused by GSSException: No valid credentials provided (Mechanism level: Failed to find any Kerberos tgt)] at org.apache.hadoop.ipc.Client$Connection$1.run(Client.java:677) at java.security.AccessController.doPrivileged(Native Method) at javax.security.auth.Subject.doAs(Subject.java:415) at org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1641) at org.apache.hadoop.ipc.Client$Connection.handleSaslConnectionFailure(Client.java:640) at org.apache.hadoop.ipc.Client$Connection.setupIOstreams(Client.java:724) at org.apache.hadoop.ipc.Client$Connection.access$2800(Client.java:367) at org.apache.hadoop.ipc.Client.getConnection(Client.java:1462) at org.apache.hadoop.ipc.Client.call(Client.java:1381) ... 21 more Caused by: javax.security.sasl.SaslException: GSS initiate failed [Caused by GSSException: No valid credentials provided (Mechanism level: Failed to find any Kerberos tgt)] at com.sun.security.sasl.gsskerb.GssKrb5Client.evaluateChallenge(GssKrb5Client.java:212) at org.apache.hadoop.security.SaslRpcClient.saslConnect(SaslRpcClient.java:411) at org.apache.hadoop.ipc.Client$Connection.setupSaslConnection(Client.java:550) at
[jira] [Commented] (MAPREDUCE-6382) Don't escape HTML links in Diagnostics in JHS job overview
[ https://issues.apache.org/jira/browse/MAPREDUCE-6382?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14572612#comment-14572612 ] Hudson commented on MAPREDUCE-6382: --- FAILURE: Integrated in Hadoop-Yarn-trunk #948 (See [https://builds.apache.org/job/Hadoop-Yarn-trunk/948/]) MAPREDUCE-6382. Don't escape HTML links in Diagnostics in JHS job overview. (Siqi Li via gera) (gera: rev e8302071ff16247a3d0cfc2dcca02da1bbf4db18) * hadoop-mapreduce-project/CHANGES.txt * hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-hs/src/main/java/org/apache/hadoop/mapreduce/v2/hs/webapp/HsJobBlock.java Don't escape HTML links in Diagnostics in JHS job overview -- Key: MAPREDUCE-6382 URL: https://issues.apache.org/jira/browse/MAPREDUCE-6382 Project: Hadoop Map/Reduce Issue Type: Bug Affects Versions: 2.8.0 Reporter: Siqi Li Assignee: Siqi Li Fix For: 2.8.0 Attachments: MAPREDUCE-6382.v1.patch, links are escaped.jpg for some reason, links are working properly in 2.4, but they are escaped in 2.6 -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (MAPREDUCE-6374) Distributed Cache File visibility should check permission of full path
[ https://issues.apache.org/jira/browse/MAPREDUCE-6374?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14572616#comment-14572616 ] Hudson commented on MAPREDUCE-6374: --- FAILURE: Integrated in Hadoop-Yarn-trunk #948 (See [https://builds.apache.org/job/Hadoop-Yarn-trunk/948/]) MAPREDUCE-6374. Distributed Cache File visibility should check permission of full path. Contributed by Chang Li (jlowe: rev 107da29ff9b6efd02b21c96df60c58ac51590ebd) * hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-core/src/test/java/org/apache/hadoop/mapreduce/filecache/TestClientDistributedCacheManager.java * hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-core/src/main/java/org/apache/hadoop/mapreduce/filecache/ClientDistributedCacheManager.java * hadoop-mapreduce-project/CHANGES.txt Distributed Cache File visibility should check permission of full path -- Key: MAPREDUCE-6374 URL: https://issues.apache.org/jira/browse/MAPREDUCE-6374 Project: Hadoop Map/Reduce Issue Type: Bug Reporter: Chang Li Assignee: Chang Li Fix For: 2.8.0 Attachments: MAPREDUCE-6374.1.patch should do full ancestor permission check for a relative cache file input -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Updated] (MAPREDUCE-6387) Fix Task object to allow Jobs not submitted via distributed cache to survive rolling upgrade
[
https://issues.apache.org/jira/browse/MAPREDUCE-6387?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Arun Suresh updated MAPREDUCE-6387:
---
Status: Patch Available (was: Open)
Fix Task object to allow Jobs not submitted via distributed cache to survive
rolling upgrade
Key: MAPREDUCE-6387
URL: https://issues.apache.org/jira/browse/MAPREDUCE-6387
Project: Hadoop Map/Reduce
Issue Type: Bug
Reporter: Arun Suresh
Assignee: Arun Suresh
Priority: Minor
Attachments: MAPREDUCE-6387.1.patch
There was a recent addition of an {{encryptedSpillKey}} to the Task object.
And when serialized, this field was written out somewhere in the middle. This
caused deployments that do not use DistributedCache to push job jars before
running the job to fail rolling upgrade.
Although deploying via Distributed Cache is the recommended method, there
might still be deployments that use the node local classpath to pick up the
MR framework classes (eg. for efficiency purposes, since this does not
require the jar being copied to hdfs and then to all the nodes)
Ensuring that it is the last field written and read when the Task object is
serialized would alleviate this issue.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Commented] (MAPREDUCE-5232) log classpath and other key properties on child JVM start
[ https://issues.apache.org/jira/browse/MAPREDUCE-5232?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14573462#comment-14573462 ] Sangjin Lee commented on MAPREDUCE-5232: Thanks [~jira.shegalov]! log classpath and other key properties on child JVM start - Key: MAPREDUCE-5232 URL: https://issues.apache.org/jira/browse/MAPREDUCE-5232 Project: Hadoop Map/Reduce Issue Type: Improvement Components: mrv1, mrv2 Affects Versions: 2.0.4-alpha Reporter: Sangjin Lee Assignee: Sangjin Lee Fix For: 2.8.0 Attachments: MAPREDUCE-5232-branch-2.patch, MAPREDUCE-5232-branch-2.patch, MAPREDUCE-5232-branch-2.patch, MAPREDUCE-5232.patch, MAPREDUCE-5232.patch, MAPREDUCE-5232.patch It would be great if we log vital information such as classpath, etc. upon a mapreduce child JVM start. This would help a great deal in terms of troubleshooting classpath issues, etc. Today it is pretty difficult to debug this unless you preserve the container script. Maybe it can log things like classpath, os name/version, java version, etc. at the beginning of the child JVM start. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
