[jira] [Commented] (MAPREDUCE-6045) need close the DataInputStream after open it in TestMapReduce.java
[
https://issues.apache.org/jira/browse/MAPREDUCE-6045?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14209507#comment-14209507
]
Hadoop QA commented on MAPREDUCE-6045:
--
{color:red}-1 overall{color}. Here are the results of testing the latest
attachment
http://issues.apache.org/jira/secure/attachment/12681280/MAPREDUCE-6045.001.patch
against trunk revision 7dae5b5.
{color:green}+1 @author{color}. The patch does not contain any @author
tags.
{color:green}+1 tests included{color}. The patch appears to include 1 new
or modified test files.
{color:green}+1 javac{color}. The applied patch does not increase the
total number of javac compiler warnings.
{color:green}+1 javadoc{color}. There were no new javadoc warning messages.
{color:green}+1 eclipse:eclipse{color}. The patch built with
eclipse:eclipse.
{color:green}+1 findbugs{color}. The patch does not introduce any new
Findbugs (version 2.0.3) warnings.
{color:green}+1 release audit{color}. The applied patch does not increase
the total number of release audit warnings.
{color:red}-1 core tests{color}. The test build failed in
hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-jobclient
{color:green}+1 contrib tests{color}. The patch passed contrib unit tests.
Test results:
https://builds.apache.org/job/PreCommit-MAPREDUCE-Build/5017//testReport/
Console output:
https://builds.apache.org/job/PreCommit-MAPREDUCE-Build/5017//console
This message is automatically generated.
need close the DataInputStream after open it in TestMapReduce.java
--
Key: MAPREDUCE-6045
URL: https://issues.apache.org/jira/browse/MAPREDUCE-6045
Project: Hadoop Map/Reduce
Issue Type: Bug
Components: test
Affects Versions: 2.5.0
Reporter: zhihai xu
Assignee: zhihai xu
Priority: Minor
Attachments: MAPREDUCE-6045.000.patch, MAPREDUCE-6045.001.patch
In TestMapReduce.java, we didn't close the DataInputStream after open it in
isSequenceFile.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Updated] (MAPREDUCE-5918) LineRecordReader can return the same decompressor to CodecPool multiple times
[ https://issues.apache.org/jira/browse/MAPREDUCE-5918?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Sergey Murylev updated MAPREDUCE-5918: -- Attachment: MAPREDUCE-5918.4.patch LineRecordReader can return the same decompressor to CodecPool multiple times - Key: MAPREDUCE-5918 URL: https://issues.apache.org/jira/browse/MAPREDUCE-5918 Project: Hadoop Map/Reduce Issue Type: Bug Affects Versions: 2.3.0 Reporter: Sergey Murylev Assignee: Sergey Murylev Fix For: trunk Attachments: MAPREDUCE-5918.1.patch, MAPREDUCE-5918.2.patch, MAPREDUCE-5918.3.patch, MAPREDUCE-5918.4.patch LineRecordReader can return the same decompressor to CodecPool multiple times if method close() called multiple times. In this case CodecPool doesn't guarantee that it always return different decompressors. This issue can cause some difficult reproducible and difficult diagnosable bugs in Hadoop based programs. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (MAPREDUCE-5918) LineRecordReader can return the same decompressor to CodecPool multiple times
[
https://issues.apache.org/jira/browse/MAPREDUCE-5918?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14209599#comment-14209599
]
Hadoop QA commented on MAPREDUCE-5918:
--
{color:green}+1 overall{color}. Here are the results of testing the latest
attachment
http://issues.apache.org/jira/secure/attachment/12681299/MAPREDUCE-5918.4.patch
against trunk revision 7dae5b5.
{color:green}+1 @author{color}. The patch does not contain any @author
tags.
{color:green}+1 tests included{color}. The patch appears to include 3 new
or modified test files.
{color:green}+1 javac{color}. The applied patch does not increase the
total number of javac compiler warnings.
{color:green}+1 javadoc{color}. There were no new javadoc warning messages.
{color:green}+1 eclipse:eclipse{color}. The patch built with
eclipse:eclipse.
{color:green}+1 findbugs{color}. The patch does not introduce any new
Findbugs (version 2.0.3) warnings.
{color:green}+1 release audit{color}. The applied patch does not increase
the total number of release audit warnings.
{color:green}+1 core tests{color}. The patch passed unit tests in
hadoop-common-project/hadoop-common
hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-core.
{color:green}+1 contrib tests{color}. The patch passed contrib unit tests.
Test results:
https://builds.apache.org/job/PreCommit-MAPREDUCE-Build/5018//testReport/
Console output:
https://builds.apache.org/job/PreCommit-MAPREDUCE-Build/5018//console
This message is automatically generated.
LineRecordReader can return the same decompressor to CodecPool multiple times
-
Key: MAPREDUCE-5918
URL: https://issues.apache.org/jira/browse/MAPREDUCE-5918
Project: Hadoop Map/Reduce
Issue Type: Bug
Affects Versions: 2.3.0
Reporter: Sergey Murylev
Assignee: Sergey Murylev
Fix For: trunk
Attachments: MAPREDUCE-5918.1.patch, MAPREDUCE-5918.2.patch,
MAPREDUCE-5918.3.patch, MAPREDUCE-5918.4.patch
LineRecordReader can return the same decompressor to CodecPool multiple times
if method close() called multiple times. In this case CodecPool doesn't
guarantee that it always return different decompressors. This issue can cause
some difficult reproducible and difficult diagnosable bugs in Hadoop based
programs.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Commented] (MAPREDUCE-5918) LineRecordReader can return the same decompressor to CodecPool multiple times
[ https://issues.apache.org/jira/browse/MAPREDUCE-5918?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14209603#comment-14209603 ] Sergey Murylev commented on MAPREDUCE-5918: --- Hi Ravi, I've made changes that you mentioned above. I've attached single patch with all changes as [^MAPREDUCE-5918.4.patch]. LineRecordReader can return the same decompressor to CodecPool multiple times - Key: MAPREDUCE-5918 URL: https://issues.apache.org/jira/browse/MAPREDUCE-5918 Project: Hadoop Map/Reduce Issue Type: Bug Affects Versions: 2.3.0 Reporter: Sergey Murylev Assignee: Sergey Murylev Fix For: trunk Attachments: MAPREDUCE-5918.1.patch, MAPREDUCE-5918.2.patch, MAPREDUCE-5918.3.patch, MAPREDUCE-5918.4.patch LineRecordReader can return the same decompressor to CodecPool multiple times if method close() called multiple times. In this case CodecPool doesn't guarantee that it always return different decompressors. This issue can cause some difficult reproducible and difficult diagnosable bugs in Hadoop based programs. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (MAPREDUCE-6156) Fetcher - connect() doesn't handle connection refused correctly
[
https://issues.apache.org/jira/browse/MAPREDUCE-6156?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14209914#comment-14209914
]
Jason Lowe commented on MAPREDUCE-6156:
---
+1 lgtm. Committing this.
Fetcher - connect() doesn't handle connection refused correctly
Key: MAPREDUCE-6156
URL: https://issues.apache.org/jira/browse/MAPREDUCE-6156
Project: Hadoop Map/Reduce
Issue Type: Bug
Reporter: Sidharta Seethana
Assignee: Junping Du
Priority: Blocker
Attachments: MAPREDUCE-6156-v2.patch, MAPREDUCE-6156-v3.patch,
MAPREDUCE-6156.patch
The connect() function in the fetcher assumes that whenever an IOException is
thrown, the amount of time passed equals connectionTimeout ( see code
snippet below ). This is incorrect. For example, in case the NM is down, an
ConnectException is thrown immediately - and the catch block assumes a minute
has passed when it is not the case.
{code}
if (connectionTimeout 0) {
throw new IOException(Invalid timeout
+ [timeout = + connectionTimeout + ms]);
} else if (connectionTimeout 0) {
unit = Math.min(UNIT_CONNECT_TIMEOUT, connectionTimeout);
}
// set the connect timeout to the unit-connect-timeout
connection.setConnectTimeout(unit);
while (true) {
try {
connection.connect();
break;
} catch (IOException ioe) {
// update the total remaining connect-timeout
connectionTimeout -= unit;
// throw an exception if we have waited for timeout amount of time
// note that the updated value if timeout is used here
if (connectionTimeout == 0) {
throw ioe;
}
// reset the connect timeout for the last try
if (connectionTimeout unit) {
unit = connectionTimeout;
// reset the connect time out for the final connect
connection.setConnectTimeout(unit);
}
}
}
{code}
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Commented] (MAPREDUCE-6153) Apply `mapreduce.admin.user.env' to AM
[ https://issues.apache.org/jira/browse/MAPREDUCE-6153?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14209956#comment-14209956 ] Allen Wittenauer commented on MAPREDUCE-6153: - By include snappy what exactly do you mean? Isn't CLASSPATH, java.library.path, and LD_LIBRARY_PATH being set already to include the appropriate dirs? Apply `mapreduce.admin.user.env' to AM -- Key: MAPREDUCE-6153 URL: https://issues.apache.org/jira/browse/MAPREDUCE-6153 Project: Hadoop Map/Reduce Issue Type: Improvement Components: applicationmaster Affects Versions: 2.5.0 Reporter: bc Wong Would be nice to be able to manipulate the AM's library path to include snappy. The `mapreduce.admin.user.env' seems perfect for this, except that it only affects the tasks. I think it's useful to have it apply to the AM as well. Use case: I have a job that uses the output committer (which runs in the AM) to read the output file of the reducer, which is compressed using snappy. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Resolved] (MAPREDUCE-6153) Apply `mapreduce.admin.user.env' to AM
[ https://issues.apache.org/jira/browse/MAPREDUCE-6153?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Vinod Kumar Vavilapalli resolved MAPREDUCE-6153. Resolution: Not a Problem There is a config for this: yarn.app.mapreduce.am.admin.user.env besides yarn.app.mapreduce.am.env Inconsistent naming due to multiple developers adding configs, but it's there. Closing this. Apply `mapreduce.admin.user.env' to AM -- Key: MAPREDUCE-6153 URL: https://issues.apache.org/jira/browse/MAPREDUCE-6153 Project: Hadoop Map/Reduce Issue Type: Improvement Components: applicationmaster Affects Versions: 2.5.0 Reporter: bc Wong Would be nice to be able to manipulate the AM's library path to include snappy. The `mapreduce.admin.user.env' seems perfect for this, except that it only affects the tasks. I think it's useful to have it apply to the AM as well. Use case: I have a job that uses the output committer (which runs in the AM) to read the output file of the reducer, which is compressed using snappy. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (MAPREDUCE-6045) need close the DataInputStream after open it in TestMapReduce.java
[ https://issues.apache.org/jira/browse/MAPREDUCE-6045?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14210007#comment-14210007 ] zhihai xu commented on MAPREDUCE-6045: -- I didn't see any test failure about hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-jobclient in the console. [INFO] hadoop-mapreduce-client-jobclient . SUCCESS [ 0.335 s] need close the DataInputStream after open it in TestMapReduce.java -- Key: MAPREDUCE-6045 URL: https://issues.apache.org/jira/browse/MAPREDUCE-6045 Project: Hadoop Map/Reduce Issue Type: Bug Components: test Affects Versions: 2.5.0 Reporter: zhihai xu Assignee: zhihai xu Priority: Minor Attachments: MAPREDUCE-6045.000.patch, MAPREDUCE-6045.001.patch In TestMapReduce.java, we didn't close the DataInputStream after open it in isSequenceFile. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Updated] (MAPREDUCE-6045) need close the DataInputStream after open it in TestMapReduce.java
[ https://issues.apache.org/jira/browse/MAPREDUCE-6045?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] zhihai xu updated MAPREDUCE-6045: - Attachment: MAPREDUCE-6045.001.patch need close the DataInputStream after open it in TestMapReduce.java -- Key: MAPREDUCE-6045 URL: https://issues.apache.org/jira/browse/MAPREDUCE-6045 Project: Hadoop Map/Reduce Issue Type: Bug Components: test Affects Versions: 2.5.0 Reporter: zhihai xu Assignee: zhihai xu Priority: Minor Attachments: MAPREDUCE-6045.000.patch, MAPREDUCE-6045.001.patch In TestMapReduce.java, we didn't close the DataInputStream after open it in isSequenceFile. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Updated] (MAPREDUCE-6045) need close the DataInputStream after open it in TestMapReduce.java
[ https://issues.apache.org/jira/browse/MAPREDUCE-6045?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] zhihai xu updated MAPREDUCE-6045: - Attachment: (was: MAPREDUCE-6045.001.patch) need close the DataInputStream after open it in TestMapReduce.java -- Key: MAPREDUCE-6045 URL: https://issues.apache.org/jira/browse/MAPREDUCE-6045 Project: Hadoop Map/Reduce Issue Type: Bug Components: test Affects Versions: 2.5.0 Reporter: zhihai xu Assignee: zhihai xu Priority: Minor Attachments: MAPREDUCE-6045.000.patch, MAPREDUCE-6045.001.patch In TestMapReduce.java, we didn't close the DataInputStream after open it in isSequenceFile. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (MAPREDUCE-6139) ConcurrentModificationException in ContainerLogAppender
[
https://issues.apache.org/jira/browse/MAPREDUCE-6139?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14210024#comment-14210024
]
Vinod Kumar Vavilapalli commented on MAPREDUCE-6139:
This belongs to YARN. Moving..
ConcurrentModificationException in ContainerLogAppender
---
Key: MAPREDUCE-6139
URL: https://issues.apache.org/jira/browse/MAPREDUCE-6139
Project: Hadoop Map/Reduce
Issue Type: Bug
Reporter: Mohammad Kamrul Islam
Assignee: Mohammad Kamrul Islam
Priority: Critical
Attachments: ContainerLogAppender.java, MAPREDUCE-6139-test.01.patch,
MAPREDUCE-6139.1.patch, MAPREDUCE-6139.2.patch, MAPREDUCE-6139.3.patch
Context:
* Hadoop-2.3.0
* Using Oozie 4.0.1
* Pig version 0.11.x
The job is submitted by Oozie to launch Pig script.
The following exception traces were found on MR task log:
In syslog:
{noformat}
2014-10-24 20:37:29,317 WARN [Thread-5]
org.apache.hadoop.util.ShutdownHookManager: ShutdownHook '' failed,
java.util.ConcurrentModificationException
java.util.ConcurrentModificationException
at
java.util.LinkedList$ListItr.checkForComodification(LinkedList.java:966)
at java.util.LinkedList$ListItr.next(LinkedList.java:888)
at
org.apache.hadoop.yarn.ContainerLogAppender.close(ContainerLogAppender.java:94)
at
org.apache.log4j.helpers.AppenderAttachableImpl.removeAllAppenders(AppenderAttachableImpl.java:141)
at org.apache.log4j.Category.removeAllAppenders(Category.java:891)
at org.apache.log4j.Hierarchy.shutdown(Hierarchy.java:471)
at org.apache.log4j.LogManager.shutdown(LogManager.java:267)
at org.apache.hadoop.mapred.TaskLog.syncLogsShutdown(TaskLog.java:286)
at org.apache.hadoop.mapred.TaskLog$2.run(TaskLog.java:339)
at
org.apache.hadoop.util.ShutdownHookManager$1.run(ShutdownHookManager.java:54)
2014-10-24 20:37:29,395 INFO [main]
org.apache.hadoop.metrics2.impl.MetricsSystemImpl: Stopping MapTask metrics
system...
{noformat}
in stderr:
{noformat}
java.util.ConcurrentModificationException
at
java.util.LinkedList$ListItr.checkForComodification(LinkedList.java:966)
at java.util.LinkedList$ListItr.next(LinkedList.java:888)
at
org.apache.hadoop.yarn.ContainerLogAppender.close(ContainerLogAppender.java:94)
at
org.apache.log4j.helpers.AppenderAttachableImpl.removeAllAppenders(AppenderAttachableImpl.java:141)
at org.apache.log4j.Category.removeAllAppenders(Category.java:891)
at
org.apache.log4j.PropertyConfigurator.parseCategory(PropertyConfigurator.java:759)
at
org.apache.log4j.PropertyConfigurator.configureRootCategory(PropertyConfigurator.java:648)
at
org.apache.log4j.PropertyConfigurator.doConfigure(PropertyConfigurator.java:514)
at
org.apache.log4j.PropertyConfigurator.configure(PropertyConfigurator.java:440)
at org.apache.pig.Main.configureLog4J(Main.java:740)
at org.apache.pig.Main.run(Main.java:384)
at org.apache.pig.PigRunner.run(PigRunner.java:49)
at org.apache.oozie.action.hadoop.PigMain.runPigJob(PigMain.java:283)
at org.apache.oozie.action.hadoop.PigMain.run(PigMain.java:223)
at org.apache.oozie.action.hadoop.LauncherMain.run(LauncherMain.java:37)
at org.apache.oozie.action.hadoop.PigMain.main(PigMain.java:76)
at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
at
sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
at
sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
at java.lang.reflect.Method.invoke(Method.java:483)
at
org.apache.oozie.action.hadoop.LauncherMapper.map(LauncherMapper.java:226)
at org.apache.hadoop.mapred.MapRunner.run(MapRunner.java:54)
at org.apache.hadoop.mapred.MapTask.runOldMapper(MapTask.java:430)
at org.apache.hadoop.mapred.MapTask.run(MapTask.java:342)
at org.apache.hadoop.mapred.YarnChild$2.run(YarnChild.java:167)
at java.security.AccessController.doPrivileged(Native Method)
at javax.security.auth.Subject.doAs(Subject.java:422)
at
org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1548)
at org.apache.hadoop.mapred.YarnChild.main(YarnChild.java:162)
{noformat}
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Updated] (MAPREDUCE-6139) ConcurrentModificationException in ContainerLogAppender
[
https://issues.apache.org/jira/browse/MAPREDUCE-6139?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Vinod Kumar Vavilapalli updated MAPREDUCE-6139:
---
Target Version/s: 2.7.0
ConcurrentModificationException in ContainerLogAppender
---
Key: MAPREDUCE-6139
URL: https://issues.apache.org/jira/browse/MAPREDUCE-6139
Project: Hadoop Map/Reduce
Issue Type: Bug
Reporter: Mohammad Kamrul Islam
Assignee: Mohammad Kamrul Islam
Priority: Critical
Attachments: ContainerLogAppender.java, MAPREDUCE-6139-test.01.patch,
MAPREDUCE-6139.1.patch, MAPREDUCE-6139.2.patch, MAPREDUCE-6139.3.patch
Context:
* Hadoop-2.3.0
* Using Oozie 4.0.1
* Pig version 0.11.x
The job is submitted by Oozie to launch Pig script.
The following exception traces were found on MR task log:
In syslog:
{noformat}
2014-10-24 20:37:29,317 WARN [Thread-5]
org.apache.hadoop.util.ShutdownHookManager: ShutdownHook '' failed,
java.util.ConcurrentModificationException
java.util.ConcurrentModificationException
at
java.util.LinkedList$ListItr.checkForComodification(LinkedList.java:966)
at java.util.LinkedList$ListItr.next(LinkedList.java:888)
at
org.apache.hadoop.yarn.ContainerLogAppender.close(ContainerLogAppender.java:94)
at
org.apache.log4j.helpers.AppenderAttachableImpl.removeAllAppenders(AppenderAttachableImpl.java:141)
at org.apache.log4j.Category.removeAllAppenders(Category.java:891)
at org.apache.log4j.Hierarchy.shutdown(Hierarchy.java:471)
at org.apache.log4j.LogManager.shutdown(LogManager.java:267)
at org.apache.hadoop.mapred.TaskLog.syncLogsShutdown(TaskLog.java:286)
at org.apache.hadoop.mapred.TaskLog$2.run(TaskLog.java:339)
at
org.apache.hadoop.util.ShutdownHookManager$1.run(ShutdownHookManager.java:54)
2014-10-24 20:37:29,395 INFO [main]
org.apache.hadoop.metrics2.impl.MetricsSystemImpl: Stopping MapTask metrics
system...
{noformat}
in stderr:
{noformat}
java.util.ConcurrentModificationException
at
java.util.LinkedList$ListItr.checkForComodification(LinkedList.java:966)
at java.util.LinkedList$ListItr.next(LinkedList.java:888)
at
org.apache.hadoop.yarn.ContainerLogAppender.close(ContainerLogAppender.java:94)
at
org.apache.log4j.helpers.AppenderAttachableImpl.removeAllAppenders(AppenderAttachableImpl.java:141)
at org.apache.log4j.Category.removeAllAppenders(Category.java:891)
at
org.apache.log4j.PropertyConfigurator.parseCategory(PropertyConfigurator.java:759)
at
org.apache.log4j.PropertyConfigurator.configureRootCategory(PropertyConfigurator.java:648)
at
org.apache.log4j.PropertyConfigurator.doConfigure(PropertyConfigurator.java:514)
at
org.apache.log4j.PropertyConfigurator.configure(PropertyConfigurator.java:440)
at org.apache.pig.Main.configureLog4J(Main.java:740)
at org.apache.pig.Main.run(Main.java:384)
at org.apache.pig.PigRunner.run(PigRunner.java:49)
at org.apache.oozie.action.hadoop.PigMain.runPigJob(PigMain.java:283)
at org.apache.oozie.action.hadoop.PigMain.run(PigMain.java:223)
at org.apache.oozie.action.hadoop.LauncherMain.run(LauncherMain.java:37)
at org.apache.oozie.action.hadoop.PigMain.main(PigMain.java:76)
at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
at
sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
at
sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
at java.lang.reflect.Method.invoke(Method.java:483)
at
org.apache.oozie.action.hadoop.LauncherMapper.map(LauncherMapper.java:226)
at org.apache.hadoop.mapred.MapRunner.run(MapRunner.java:54)
at org.apache.hadoop.mapred.MapTask.runOldMapper(MapTask.java:430)
at org.apache.hadoop.mapred.MapTask.run(MapTask.java:342)
at org.apache.hadoop.mapred.YarnChild$2.run(YarnChild.java:167)
at java.security.AccessController.doPrivileged(Native Method)
at javax.security.auth.Subject.doAs(Subject.java:422)
at
org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1548)
at org.apache.hadoop.mapred.YarnChild.main(YarnChild.java:162)
{noformat}
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Commented] (MAPREDUCE-6156) Fetcher - connect() doesn't handle connection refused correctly
[
https://issues.apache.org/jira/browse/MAPREDUCE-6156?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14210145#comment-14210145
]
Arun C Murthy commented on MAPREDUCE-6156:
--
[~jlowe] Thanks. I've merged this into branch-2.6.0 also for hadoop-2.6.0-rc1.
Fetcher - connect() doesn't handle connection refused correctly
Key: MAPREDUCE-6156
URL: https://issues.apache.org/jira/browse/MAPREDUCE-6156
Project: Hadoop Map/Reduce
Issue Type: Bug
Reporter: Sidharta Seethana
Assignee: Junping Du
Priority: Blocker
Fix For: 2.6.0
Attachments: MAPREDUCE-6156-v2.patch, MAPREDUCE-6156-v3.patch,
MAPREDUCE-6156.patch
The connect() function in the fetcher assumes that whenever an IOException is
thrown, the amount of time passed equals connectionTimeout ( see code
snippet below ). This is incorrect. For example, in case the NM is down, an
ConnectException is thrown immediately - and the catch block assumes a minute
has passed when it is not the case.
{code}
if (connectionTimeout 0) {
throw new IOException(Invalid timeout
+ [timeout = + connectionTimeout + ms]);
} else if (connectionTimeout 0) {
unit = Math.min(UNIT_CONNECT_TIMEOUT, connectionTimeout);
}
// set the connect timeout to the unit-connect-timeout
connection.setConnectTimeout(unit);
while (true) {
try {
connection.connect();
break;
} catch (IOException ioe) {
// update the total remaining connect-timeout
connectionTimeout -= unit;
// throw an exception if we have waited for timeout amount of time
// note that the updated value if timeout is used here
if (connectionTimeout == 0) {
throw ioe;
}
// reset the connect timeout for the last try
if (connectionTimeout unit) {
unit = connectionTimeout;
// reset the connect time out for the final connect
connection.setConnectTimeout(unit);
}
}
}
{code}
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Created] (MAPREDUCE-6161) mapred hsadmin command missing from trunk
Jason Lowe created MAPREDUCE-6161: - Summary: mapred hsadmin command missing from trunk Key: MAPREDUCE-6161 URL: https://issues.apache.org/jira/browse/MAPREDUCE-6161 Project: Hadoop Map/Reduce Issue Type: Bug Components: scripts Affects Versions: trunk Reporter: Jason Lowe The hsadmin subcommand of the mapred script is no longer present in trunk. It is present in branch-2. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (MAPREDUCE-6161) mapred hsadmin command missing from trunk
[ https://issues.apache.org/jira/browse/MAPREDUCE-6161?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14210162#comment-14210162 ] Jason Lowe commented on MAPREDUCE-6161: --- Appears to be a casualty of HADOOP-9902. mapred hsadmin command missing from trunk - Key: MAPREDUCE-6161 URL: https://issues.apache.org/jira/browse/MAPREDUCE-6161 Project: Hadoop Map/Reduce Issue Type: Bug Components: scripts Affects Versions: trunk Reporter: Jason Lowe The hsadmin subcommand of the mapred script is no longer present in trunk. It is present in branch-2. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (MAPREDUCE-6045) need close the DataInputStream after open it in TestMapReduce.java
[
https://issues.apache.org/jira/browse/MAPREDUCE-6045?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14210172#comment-14210172
]
Hadoop QA commented on MAPREDUCE-6045:
--
{color:green}+1 overall{color}. Here are the results of testing the latest
attachment
http://issues.apache.org/jira/secure/attachment/12681362/MAPREDUCE-6045.001.patch
against trunk revision 3651fe1.
{color:green}+1 @author{color}. The patch does not contain any @author
tags.
{color:green}+1 tests included{color}. The patch appears to include 1 new
or modified test files.
{color:green}+1 javac{color}. The applied patch does not increase the
total number of javac compiler warnings.
{color:green}+1 javadoc{color}. There were no new javadoc warning messages.
{color:green}+1 eclipse:eclipse{color}. The patch built with
eclipse:eclipse.
{color:green}+1 findbugs{color}. The patch does not introduce any new
Findbugs (version 2.0.3) warnings.
{color:green}+1 release audit{color}. The applied patch does not increase
the total number of release audit warnings.
{color:green}+1 core tests{color}. The patch passed unit tests in
hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-jobclient.
{color:green}+1 contrib tests{color}. The patch passed contrib unit tests.
Test results:
https://builds.apache.org/job/PreCommit-MAPREDUCE-Build/5019//testReport/
Console output:
https://builds.apache.org/job/PreCommit-MAPREDUCE-Build/5019//console
This message is automatically generated.
need close the DataInputStream after open it in TestMapReduce.java
--
Key: MAPREDUCE-6045
URL: https://issues.apache.org/jira/browse/MAPREDUCE-6045
Project: Hadoop Map/Reduce
Issue Type: Bug
Components: test
Affects Versions: 2.5.0
Reporter: zhihai xu
Assignee: zhihai xu
Priority: Minor
Attachments: MAPREDUCE-6045.000.patch, MAPREDUCE-6045.001.patch
In TestMapReduce.java, we didn't close the DataInputStream after open it in
isSequenceFile.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Assigned] (MAPREDUCE-6161) mapred hsadmin command missing from trunk
[ https://issues.apache.org/jira/browse/MAPREDUCE-6161?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Allen Wittenauer reassigned MAPREDUCE-6161: --- Assignee: Allen Wittenauer mapred hsadmin command missing from trunk - Key: MAPREDUCE-6161 URL: https://issues.apache.org/jira/browse/MAPREDUCE-6161 Project: Hadoop Map/Reduce Issue Type: Bug Components: scripts Affects Versions: trunk Reporter: Jason Lowe Assignee: Allen Wittenauer Attachments: MAPREDUCE-6161.patch The hsadmin subcommand of the mapred script is no longer present in trunk. It is present in branch-2. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Updated] (MAPREDUCE-6161) mapred hsadmin command missing from trunk
[ https://issues.apache.org/jira/browse/MAPREDUCE-6161?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Allen Wittenauer updated MAPREDUCE-6161: Attachment: MAPREDUCE-6161.patch -0: * fixes hsadmin * fixes sort order of job in usage * adds HADOOP_CLIENT_OPTS to job subcommand mapred hsadmin command missing from trunk - Key: MAPREDUCE-6161 URL: https://issues.apache.org/jira/browse/MAPREDUCE-6161 Project: Hadoop Map/Reduce Issue Type: Bug Components: scripts Affects Versions: trunk Reporter: Jason Lowe Attachments: MAPREDUCE-6161.patch The hsadmin subcommand of the mapred script is no longer present in trunk. It is present in branch-2. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Updated] (MAPREDUCE-6161) mapred hsadmin command missing from trunk
[ https://issues.apache.org/jira/browse/MAPREDUCE-6161?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Allen Wittenauer updated MAPREDUCE-6161: Status: Patch Available (was: Open) mapred hsadmin command missing from trunk - Key: MAPREDUCE-6161 URL: https://issues.apache.org/jira/browse/MAPREDUCE-6161 Project: Hadoop Map/Reduce Issue Type: Bug Components: scripts Affects Versions: trunk Reporter: Jason Lowe Assignee: Allen Wittenauer Attachments: MAPREDUCE-6161.patch The hsadmin subcommand of the mapred script is no longer present in trunk. It is present in branch-2. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (MAPREDUCE-6161) mapred hsadmin command missing from trunk
[
https://issues.apache.org/jira/browse/MAPREDUCE-6161?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14211285#comment-14211285
]
Hadoop QA commented on MAPREDUCE-6161:
--
{color:red}-1 overall{color}. Here are the results of testing the latest
attachment
http://issues.apache.org/jira/secure/attachment/12681388/MAPREDUCE-6161.patch
against trunk revision 394ba94.
{color:green}+1 @author{color}. The patch does not contain any @author
tags.
{color:red}-1 tests included{color}. The patch doesn't appear to include
any new or modified tests.
Please justify why no new tests are needed for this
patch.
Also please list what manual steps were performed to
verify this patch.
{color:green}+1 javac{color}. The applied patch does not increase the
total number of javac compiler warnings.
{color:green}+1 javadoc{color}. There were no new javadoc warning messages.
{color:green}+1 eclipse:eclipse{color}. The patch built with
eclipse:eclipse.
{color:green}+1 findbugs{color}. The patch does not introduce any new
Findbugs (version 2.0.3) warnings.
{color:green}+1 release audit{color}. The applied patch does not increase
the total number of release audit warnings.
{color:green}+1 core tests{color}. The patch passed unit tests in .
{color:green}+1 contrib tests{color}. The patch passed contrib unit tests.
Test results:
https://builds.apache.org/job/PreCommit-MAPREDUCE-Build/5020//testReport/
Console output:
https://builds.apache.org/job/PreCommit-MAPREDUCE-Build/5020//console
This message is automatically generated.
mapred hsadmin command missing from trunk
-
Key: MAPREDUCE-6161
URL: https://issues.apache.org/jira/browse/MAPREDUCE-6161
Project: Hadoop Map/Reduce
Issue Type: Bug
Components: scripts
Affects Versions: trunk
Reporter: Jason Lowe
Assignee: Allen Wittenauer
Attachments: MAPREDUCE-6161.patch
The hsadmin subcommand of the mapred script is no longer present in trunk. It
is present in branch-2.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Commented] (MAPREDUCE-6128) Automatic addition of bundled jars to distributed cache
[ https://issues.apache.org/jira/browse/MAPREDUCE-6128?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14211295#comment-14211295 ] Jason Lowe commented on MAPREDUCE-6128: --- Thanks for updating the patch, Gera. The application classloader feature with '$' should be a separate JIRA and deserving of its own unit test. The filter test classes feature also seems separate (although I could see the two classloader features done in conjunction). But do we really need the test filter feature? Seems like a hack to work around a very specific case with the minicluster. Can we configure a minicluster to run without the jobclient tests in the classpath instead? Speaking of the '$' behavior, would it be better if users simply listed the class and internally isSystemClass checked not only for equality but also starting with the classname + '$'? Trying to think of when a user would ever want a particular class to be a system class but _not_ have the inner classes of that class loaded from the same place. In other words, if I'm listing a specific class then when wouldn't I want to tack on the '$'? JobSubmitter is only checking files specified by -libjars for conflicts, but it needs to check all other distributed cache files. $PWD/* is added to the classpath, so one doesn't need to use -libjars to add a jar, and a distributed cache conflict is bad even if the file isn't in the classpath. Also need to watch out for renames during localization (i.e.: the fragment of the URI). USE_MANIFEST_CLASSES should be JOB_USE_MANIFEST_CLASSES and it should be mapreduce.job.include.manifest.classpath since this is a job-level concept not a task-level concept. Also USE_MANIFEST_CLASSES should be JOB_USE_MANIFEST_CLASSES or JOB_INCLUDE_MANIFEST_CLASSES. Nit: USE_MANIFEST_CLASSES_DEFAULT should be grouped next to the property it defaults, whitespace-wise, rather than something else. names ending with a '$' are treated as inner classes is a bit confusing. What does it mean to treat it as an inner class? Probably clearer to state that any inner classes of the specified class are also included. Automatic addition of bundled jars to distributed cache Key: MAPREDUCE-6128 URL: https://issues.apache.org/jira/browse/MAPREDUCE-6128 Project: Hadoop Map/Reduce Issue Type: Improvement Components: client Affects Versions: 2.5.1 Reporter: Gera Shegalov Assignee: Gera Shegalov Attachments: MAPREDUCE-6128.v01.patch, MAPREDUCE-6128.v02.patch, MAPREDUCE-6128.v03.patch, MAPREDUCE-6128.v04.patch, MAPREDUCE-6128.v05.patch On the client side, JDK adds Class-Path elements from the job jar manifest on the classpath. In theory there could be many bundled jars in many directories such that adding them manually via libjars or similar means to task classpaths is cumbersome. If this property is enabled, the same jars are added to the task classpaths automatically. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
