[jira] [Updated] (MAPREDUCE-3366) Mapreduce component should use consistent directory structure layout as HDFS/common
[
https://issues.apache.org/jira/browse/MAPREDUCE-3366?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Eric Yang updated MAPREDUCE-3366:
-
Attachment: MAPREDUCE-3366-4-trunk.patch
Include updated documentation.
Mapreduce component should use consistent directory structure layout as
HDFS/common
---
Key: MAPREDUCE-3366
URL: https://issues.apache.org/jira/browse/MAPREDUCE-3366
Project: Hadoop Map/Reduce
Issue Type: Bug
Components: mrv2
Affects Versions: 0.23.0
Environment: Java, Maven
Reporter: Eric Yang
Assignee: Eric Yang
Labels: bigtop
Attachments: MAPREDUCE-3366-1-trunk.patch,
MAPREDUCE-3366-2-trunk.patch, MAPREDUCE-3366-3-trunk.patch,
MAPREDUCE-3366-4-trunk.patch, MAPREDUCE-3366.patch,
svn-rename-MAPREDUCE-3366-3.sh
Directory structure for MRv2 layout looks like:
{noformat}
hadoop-mapreduce-0.23.0-SNAPSHOT/bin
/conf
/lib
/modules
{noformat}
The directory structure layout should be updated to reflect changes
implemented in HADOOP-6255.
{noformat}
hadoop-mapreduce-0.23.0-SNAPSHOT/bin
/etc/hadoop
/lib
/libexec
/sbin
/share/hadoop
/share/hadoop/lib
{noformat}
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators:
https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Updated] (MAPREDUCE-3366) Mapreduce component should use consistent directory structure layout as HDFS/common
[
https://issues.apache.org/jira/browse/MAPREDUCE-3366?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Eric Yang updated MAPREDUCE-3366:
-
Status: Patch Available (was: Open)
Mapreduce component should use consistent directory structure layout as
HDFS/common
---
Key: MAPREDUCE-3366
URL: https://issues.apache.org/jira/browse/MAPREDUCE-3366
Project: Hadoop Map/Reduce
Issue Type: Bug
Components: mrv2
Affects Versions: 0.23.0
Environment: Java, Maven
Reporter: Eric Yang
Assignee: Eric Yang
Attachments: MAPREDUCE-3366-1-trunk.patch,
MAPREDUCE-3366-2-trunk.patch, MAPREDUCE-3366-3-trunk.patch,
MAPREDUCE-3366.patch, svn-rename-MAPREDUCE-3366-3.sh
Directory structure for MRv2 layout looks like:
{noformat}
hadoop-mapreduce-0.23.0-SNAPSHOT/bin
/conf
/lib
/modules
{noformat}
The directory structure layout should be updated to reflect changes
implemented in HADOOP-6255.
{noformat}
hadoop-mapreduce-0.23.0-SNAPSHOT/bin
/etc/hadoop
/lib
/libexec
/sbin
/share/hadoop
/share/hadoop/lib
{noformat}
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators:
https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Updated] (MAPREDUCE-3366) Mapreduce component should use consistent directory structure layout as HDFS/common
[
https://issues.apache.org/jira/browse/MAPREDUCE-3366?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Eric Yang updated MAPREDUCE-3366:
-
Attachment: MAPREDUCE-3366-3-trunk.patch
svn-rename-MAPREDUCE-3366-3.sh
- Revised patch to use proper libexec path.
- Renamed start-all.sh/stop-all.sh to start-yarn.sh/stop-yarn.sh.
Apply instruction:
{noformat}
svn-rename-MAPREDUCE-3366-3.sh
patch -p0 MAPREDUCE-3366-3-trunk.patch
{noformat}
Mapreduce component should use consistent directory structure layout as
HDFS/common
---
Key: MAPREDUCE-3366
URL: https://issues.apache.org/jira/browse/MAPREDUCE-3366
Project: Hadoop Map/Reduce
Issue Type: Bug
Components: mrv2
Affects Versions: 0.23.0
Environment: Java, Maven
Reporter: Eric Yang
Assignee: Eric Yang
Attachments: MAPREDUCE-3366-1-trunk.patch,
MAPREDUCE-3366-2-trunk.patch, MAPREDUCE-3366-3-trunk.patch,
MAPREDUCE-3366.patch, svn-rename-MAPREDUCE-3366-3.sh
Directory structure for MRv2 layout looks like:
{noformat}
hadoop-mapreduce-0.23.0-SNAPSHOT/bin
/conf
/lib
/modules
{noformat}
The directory structure layout should be updated to reflect changes
implemented in HADOOP-6255.
{noformat}
hadoop-mapreduce-0.23.0-SNAPSHOT/bin
/etc/hadoop
/lib
/libexec
/sbin
/share/hadoop
/share/hadoop/lib
{noformat}
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators:
https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Updated] (MAPREDUCE-3366) Mapreduce component should use consistent directory structure layout as HDFS/common
[
https://issues.apache.org/jira/browse/MAPREDUCE-3366?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Eric Yang updated MAPREDUCE-3366:
-
Attachment: MAPREDUCE-3366-1-trunk.patch
Updated patch for trunk with ApplicationConstants.java fixed.
Mapreduce component should use consistent directory structure layout as
HDFS/common
---
Key: MAPREDUCE-3366
URL: https://issues.apache.org/jira/browse/MAPREDUCE-3366
Project: Hadoop Map/Reduce
Issue Type: Bug
Components: mrv2
Affects Versions: 0.23.0
Environment: Java, Maven
Reporter: Eric Yang
Assignee: Eric Yang
Attachments: MAPREDUCE-3366-1-trunk.patch, MAPREDUCE-3366.patch
Directory structure for MRv2 layout looks like:
{noformat}
hadoop-mapreduce-0.23.0-SNAPSHOT/bin
/conf
/lib
/modules
{noformat}
The directory structure layout should be updated to reflect changes
implemented in HADOOP-6255.
{noformat}
hadoop-mapreduce-0.23.0-SNAPSHOT/bin
/etc/hadoop
/lib
/libexec
/sbin
/share/hadoop
/share/hadoop/lib
{noformat}
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators:
https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Updated] (MAPREDUCE-3366) Mapreduce component should use consistent directory structure layout as HDFS/common
[
https://issues.apache.org/jira/browse/MAPREDUCE-3366?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Eric Yang updated MAPREDUCE-3366:
-
Attachment: MAPREDUCE-3366-2-trunk.patch
Updated LIBEXEC to match what was done in MAPREDUCE-3373.
Mapreduce component should use consistent directory structure layout as
HDFS/common
---
Key: MAPREDUCE-3366
URL: https://issues.apache.org/jira/browse/MAPREDUCE-3366
Project: Hadoop Map/Reduce
Issue Type: Bug
Components: mrv2
Affects Versions: 0.23.0
Environment: Java, Maven
Reporter: Eric Yang
Assignee: Eric Yang
Attachments: MAPREDUCE-3366-1-trunk.patch,
MAPREDUCE-3366-2-trunk.patch, MAPREDUCE-3366.patch
Directory structure for MRv2 layout looks like:
{noformat}
hadoop-mapreduce-0.23.0-SNAPSHOT/bin
/conf
/lib
/modules
{noformat}
The directory structure layout should be updated to reflect changes
implemented in HADOOP-6255.
{noformat}
hadoop-mapreduce-0.23.0-SNAPSHOT/bin
/etc/hadoop
/lib
/libexec
/sbin
/share/hadoop
/share/hadoop/lib
{noformat}
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators:
https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Updated] (MAPREDUCE-3366) Mapreduce component should use consistent directory structure layout as HDFS/common
[
https://issues.apache.org/jira/browse/MAPREDUCE-3366?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Eric Yang updated MAPREDUCE-3366:
-
Status: Open (was: Patch Available)
Patch build didn't work for some other reasons, cancel patch and retest.
Mapreduce component should use consistent directory structure layout as
HDFS/common
---
Key: MAPREDUCE-3366
URL: https://issues.apache.org/jira/browse/MAPREDUCE-3366
Project: Hadoop Map/Reduce
Issue Type: Bug
Components: mrv2
Affects Versions: 0.23.0
Environment: Java, Maven
Reporter: Eric Yang
Assignee: Eric Yang
Attachments: MAPREDUCE-3366.patch
Directory structure for MRv2 layout looks like:
{noformat}
hadoop-mapreduce-0.23.0-SNAPSHOT/bin
/conf
/lib
/modules
{noformat}
The directory structure layout should be updated to reflect changes
implemented in HADOOP-6255.
{noformat}
hadoop-mapreduce-0.23.0-SNAPSHOT/bin
/etc/hadoop
/lib
/libexec
/sbin
/share/hadoop
/share/hadoop/lib
{noformat}
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators:
https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Updated] (MAPREDUCE-3366) Mapreduce component should use consistent directory structure layout as HDFS/common
[
https://issues.apache.org/jira/browse/MAPREDUCE-3366?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Eric Yang updated MAPREDUCE-3366:
-
Status: Patch Available (was: Open)
Mapreduce component should use consistent directory structure layout as
HDFS/common
---
Key: MAPREDUCE-3366
URL: https://issues.apache.org/jira/browse/MAPREDUCE-3366
Project: Hadoop Map/Reduce
Issue Type: Bug
Components: mrv2
Affects Versions: 0.23.0
Environment: Java, Maven
Reporter: Eric Yang
Assignee: Eric Yang
Attachments: MAPREDUCE-3366.patch
Directory structure for MRv2 layout looks like:
{noformat}
hadoop-mapreduce-0.23.0-SNAPSHOT/bin
/conf
/lib
/modules
{noformat}
The directory structure layout should be updated to reflect changes
implemented in HADOOP-6255.
{noformat}
hadoop-mapreduce-0.23.0-SNAPSHOT/bin
/etc/hadoop
/lib
/libexec
/sbin
/share/hadoop
/share/hadoop/lib
{noformat}
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators:
https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Updated] (MAPREDUCE-3366) Mapreduce component should use consistent directory structure layout as HDFS/common
[
https://issues.apache.org/jira/browse/MAPREDUCE-3366?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Eric Yang updated MAPREDUCE-3366:
-
Status: Patch Available (was: Open)
Mapreduce component should use consistent directory structure layout as
HDFS/common
---
Key: MAPREDUCE-3366
URL: https://issues.apache.org/jira/browse/MAPREDUCE-3366
Project: Hadoop Map/Reduce
Issue Type: Bug
Components: mrv2
Affects Versions: 0.23.0
Environment: Java, Maven
Reporter: Eric Yang
Assignee: Eric Yang
Attachments: MAPREDUCE-3366.patch
Directory structure for MRv2 layout looks like:
{noformat}
hadoop-mapreduce-0.23.0-SNAPSHOT/bin
/conf
/lib
/modules
{noformat}
The directory structure layout should be updated to reflect changes
implemented in HADOOP-6255.
{noformat}
hadoop-mapreduce-0.23.0-SNAPSHOT/bin
/etc/hadoop
/lib
/libexec
/sbin
/share/hadoop
/share/hadoop/lib
{noformat}
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators:
https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Updated] (MAPREDUCE-3366) Mapreduce component should use consistent directory structure layout as HDFS/common
[
https://issues.apache.org/jira/browse/MAPREDUCE-3366?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Eric Yang updated MAPREDUCE-3366:
-
Attachment: MAPREDUCE-3366.patch
Map YARN directory structure to the same layout as HDFS/common.
Mapreduce component should use consistent directory structure layout as
HDFS/common
---
Key: MAPREDUCE-3366
URL: https://issues.apache.org/jira/browse/MAPREDUCE-3366
Project: Hadoop Map/Reduce
Issue Type: Bug
Components: mrv2
Affects Versions: 0.23.0
Environment: Java, Maven
Reporter: Eric Yang
Assignee: Eric Yang
Attachments: MAPREDUCE-3366.patch
Directory structure for MRv2 layout looks like:
{noformat}
hadoop-mapreduce-0.23.0-SNAPSHOT/bin
/conf
/lib
/modules
{noformat}
The directory structure layout should be updated to reflect changes
implemented in HADOOP-6255.
{noformat}
hadoop-mapreduce-0.23.0-SNAPSHOT/bin
/etc/hadoop
/lib
/libexec
/sbin
/share/hadoop
/share/hadoop/lib
{noformat}
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators:
https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Updated] (MAPREDUCE-3112) Calling hadoop cli inside mapreduce job leads to errors
[
https://issues.apache.org/jira/browse/MAPREDUCE-3112?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Eric Yang updated MAPREDUCE-3112:
-
Resolution: Fixed
Status: Resolved (was: Patch Available)
I just committed this to 0.23 and trunk, thanks Rayma and Matt.
Calling hadoop cli inside mapreduce job leads to errors
---
Key: MAPREDUCE-3112
URL: https://issues.apache.org/jira/browse/MAPREDUCE-3112
Project: Hadoop Map/Reduce
Issue Type: Bug
Components: contrib/streaming
Affects Versions: 0.20.205.0, 0.23.0
Environment: Java, Linux
Reporter: Eric Yang
Assignee: Eric Yang
Fix For: 0.20.205.0, 0.23.0
Attachments: HAPREDUCE-3112-1.patch, MAPREDUCE-3112-trunk-2.patch,
MAPREDUCE-3112-trunk.patch, MAPREDUCE-3112.patch
When running a streaming job with mapper
bin/hadoop --config /etc/hadoop/ jar
contrib/streaming/hadoop-streaming-0.20.205.0.jar -mapper hadoop --config
/etc/hadoop/ dfs -help -reducer NONE -input /tmp/input.txt -output NONE
Task log shows:
{noformat}
Exception in thread main java.lang.ExceptionInInitializerError
at org.apache.hadoop.util.ToolRunner.run(ToolRunner.java:57)
at org.apache.hadoop.util.ToolRunner.run(ToolRunner.java:79)
at org.apache.hadoop.fs.FsShell.main(FsShell.java:1895)
Caused by: org.apache.commons.logging.LogConfigurationException:
User-specified log class 'org.apache.commons.logging.impl.Log4JLogger' cannot
be found or is not useable.
at
org.apache.commons.logging.impl.LogFactoryImpl.discoverLogImplementation(LogFactoryImpl.java:874)
at
org.apache.commons.logging.impl.LogFactoryImpl.newInstance(LogFactoryImpl.java:604)
at
org.apache.commons.logging.impl.LogFactoryImpl.getInstance(LogFactoryImpl.java:336)
at
org.apache.commons.logging.impl.LogFactoryImpl.getInstance(LogFactoryImpl.java:310)
at org.apache.commons.logging.LogFactory.getLog(LogFactory.java:685)
at org.apache.hadoop.conf.Configuration.clinit(Configuration.java:142)
... 3 more
java.lang.RuntimeException: PipeMapRed.waitOutputThreads(): subprocess failed
with code 1
at
org.apache.hadoop.streaming.PipeMapRed.waitOutputThreads(PipeMapRed.java:311)
at
org.apache.hadoop.streaming.PipeMapRed.mapRedFinished(PipeMapRed.java:545)
at org.apache.hadoop.streaming.PipeMapper.close(PipeMapper.java:132)
at org.apache.hadoop.mapred.MapRunner.run(MapRunner.java:57)
at org.apache.hadoop.streaming.PipeMapRunner.run(PipeMapRunner.java:36)
at org.apache.hadoop.mapred.MapTask.runOldMapper(MapTask.java:436)
at org.apache.hadoop.mapred.MapTask.run(MapTask.java:372)
at org.apache.hadoop.mapred.Child$4.run(Child.java:261)
at java.security.AccessController.doPrivileged(Native Method)
at javax.security.auth.Subject.doAs(Subject.java:396)
at
org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1059)
at org.apache.hadoop.mapred.Child.main(Child.java:255)
{noformat}
Upon inspection, there are two problems in the inherited from environment
which prevent the logger initialization to work properly. In hadoop-env.sh,
the HADOOP_OPTS is inherited from the parent process. This configuration was
requested by user to have a way to override HADOOP environment in the
configuration template:
{noformat}
export HADOOP_OPTS=-Djava.net.preferIPv4Stack=true $HADOOP_OPTS
{noformat}
-Dhadoop.log.dir=$HADOOP_LOG_DIR/task_tracker_user is injected into
HADOOP_OPTS in the tasktracker environment. Hence, the running task would
inherit the wrong logging directory, which the end user might not have
sufficient access to write. Second, $HADOOP_ROOT_LOGGER is override to:
-Dhadoop.root.logger=INFO,TLA by the task controller, therefore, the
bin/hadoop script will attempt to use hadoop.root.logger=INFO,TLA, but fail
to initialize.
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators:
https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Updated] (MAPREDUCE-3112) Calling hadoop cli inside mapreduce job leads to errors
[
https://issues.apache.org/jira/browse/MAPREDUCE-3112?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Eric Yang updated MAPREDUCE-3112:
-
Attachment: MAPREDUCE-3112.patch
Patch for brach-20-security.
Calling hadoop cli inside mapreduce job leads to errors
---
Key: MAPREDUCE-3112
URL: https://issues.apache.org/jira/browse/MAPREDUCE-3112
Project: Hadoop Map/Reduce
Issue Type: Bug
Components: contrib/streaming
Affects Versions: 0.20.205.0
Environment: Java, Linux
Reporter: Eric Yang
Assignee: Eric Yang
Fix For: 0.20.205.0
Attachments: MAPREDUCE-3112.patch
When running a streaming job with mapper
bin/hadoop --config /etc/hadoop/ jar
contrib/streaming/hadoop-streaming-0.20.205.0.jar -mapper hadoop --config
/etc/hadoop/ dfs -help -reducer NONE -input /tmp/input.txt -output NONE
Task log shows:
{noformat}
Exception in thread main java.lang.ExceptionInInitializerError
at org.apache.hadoop.util.ToolRunner.run(ToolRunner.java:57)
at org.apache.hadoop.util.ToolRunner.run(ToolRunner.java:79)
at org.apache.hadoop.fs.FsShell.main(FsShell.java:1895)
Caused by: org.apache.commons.logging.LogConfigurationException:
User-specified log class 'org.apache.commons.logging.impl.Log4JLogger' cannot
be found or is not useable.
at
org.apache.commons.logging.impl.LogFactoryImpl.discoverLogImplementation(LogFactoryImpl.java:874)
at
org.apache.commons.logging.impl.LogFactoryImpl.newInstance(LogFactoryImpl.java:604)
at
org.apache.commons.logging.impl.LogFactoryImpl.getInstance(LogFactoryImpl.java:336)
at
org.apache.commons.logging.impl.LogFactoryImpl.getInstance(LogFactoryImpl.java:310)
at org.apache.commons.logging.LogFactory.getLog(LogFactory.java:685)
at org.apache.hadoop.conf.Configuration.clinit(Configuration.java:142)
... 3 more
java.lang.RuntimeException: PipeMapRed.waitOutputThreads(): subprocess failed
with code 1
at
org.apache.hadoop.streaming.PipeMapRed.waitOutputThreads(PipeMapRed.java:311)
at
org.apache.hadoop.streaming.PipeMapRed.mapRedFinished(PipeMapRed.java:545)
at org.apache.hadoop.streaming.PipeMapper.close(PipeMapper.java:132)
at org.apache.hadoop.mapred.MapRunner.run(MapRunner.java:57)
at org.apache.hadoop.streaming.PipeMapRunner.run(PipeMapRunner.java:36)
at org.apache.hadoop.mapred.MapTask.runOldMapper(MapTask.java:436)
at org.apache.hadoop.mapred.MapTask.run(MapTask.java:372)
at org.apache.hadoop.mapred.Child$4.run(Child.java:261)
at java.security.AccessController.doPrivileged(Native Method)
at javax.security.auth.Subject.doAs(Subject.java:396)
at
org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1059)
at org.apache.hadoop.mapred.Child.main(Child.java:255)
{noformat}
Upon inspection, there are two problems in the inherited from environment
which prevent the logger initialization to work properly. In hadoop-env.sh,
the HADOOP_OPTS is inherited from the parent process. This configuration was
requested by user to have a way to override HADOOP environment in the
configuration template:
{noformat}
export HADOOP_OPTS=-Djava.net.preferIPv4Stack=true $HADOOP_OPTS
{noformat}
-Dhadoop.log.dir=$HADOOP_LOG_DIR/task_tracker_user is injected into
HADOOP_OPTS in the tasktracker environment. Hence, the running task would
inherit the wrong logging directory, which the end user might not have
sufficient access to write. Second, $HADOOP_ROOT_LOGGER is override to:
-Dhadoop.root.logger=INFO,TLA by the task controller, therefore, the
bin/hadoop script will attempt to use hadoop.root.logger=INFO,TLA, but fail
to initialize.
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators:
https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Updated] (MAPREDUCE-3112) Calling hadoop cli inside mapreduce job leads to errors
[
https://issues.apache.org/jira/browse/MAPREDUCE-3112?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Eric Yang updated MAPREDUCE-3112:
-
Attachment: MAPREDUCE-3112-trunk.patch
Same patch for trunk.
Calling hadoop cli inside mapreduce job leads to errors
---
Key: MAPREDUCE-3112
URL: https://issues.apache.org/jira/browse/MAPREDUCE-3112
Project: Hadoop Map/Reduce
Issue Type: Bug
Components: contrib/streaming
Affects Versions: 0.20.205.0
Environment: Java, Linux
Reporter: Eric Yang
Assignee: Eric Yang
Fix For: 0.20.205.0
Attachments: MAPREDUCE-3112-trunk.patch, MAPREDUCE-3112.patch
When running a streaming job with mapper
bin/hadoop --config /etc/hadoop/ jar
contrib/streaming/hadoop-streaming-0.20.205.0.jar -mapper hadoop --config
/etc/hadoop/ dfs -help -reducer NONE -input /tmp/input.txt -output NONE
Task log shows:
{noformat}
Exception in thread main java.lang.ExceptionInInitializerError
at org.apache.hadoop.util.ToolRunner.run(ToolRunner.java:57)
at org.apache.hadoop.util.ToolRunner.run(ToolRunner.java:79)
at org.apache.hadoop.fs.FsShell.main(FsShell.java:1895)
Caused by: org.apache.commons.logging.LogConfigurationException:
User-specified log class 'org.apache.commons.logging.impl.Log4JLogger' cannot
be found or is not useable.
at
org.apache.commons.logging.impl.LogFactoryImpl.discoverLogImplementation(LogFactoryImpl.java:874)
at
org.apache.commons.logging.impl.LogFactoryImpl.newInstance(LogFactoryImpl.java:604)
at
org.apache.commons.logging.impl.LogFactoryImpl.getInstance(LogFactoryImpl.java:336)
at
org.apache.commons.logging.impl.LogFactoryImpl.getInstance(LogFactoryImpl.java:310)
at org.apache.commons.logging.LogFactory.getLog(LogFactory.java:685)
at org.apache.hadoop.conf.Configuration.clinit(Configuration.java:142)
... 3 more
java.lang.RuntimeException: PipeMapRed.waitOutputThreads(): subprocess failed
with code 1
at
org.apache.hadoop.streaming.PipeMapRed.waitOutputThreads(PipeMapRed.java:311)
at
org.apache.hadoop.streaming.PipeMapRed.mapRedFinished(PipeMapRed.java:545)
at org.apache.hadoop.streaming.PipeMapper.close(PipeMapper.java:132)
at org.apache.hadoop.mapred.MapRunner.run(MapRunner.java:57)
at org.apache.hadoop.streaming.PipeMapRunner.run(PipeMapRunner.java:36)
at org.apache.hadoop.mapred.MapTask.runOldMapper(MapTask.java:436)
at org.apache.hadoop.mapred.MapTask.run(MapTask.java:372)
at org.apache.hadoop.mapred.Child$4.run(Child.java:261)
at java.security.AccessController.doPrivileged(Native Method)
at javax.security.auth.Subject.doAs(Subject.java:396)
at
org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1059)
at org.apache.hadoop.mapred.Child.main(Child.java:255)
{noformat}
Upon inspection, there are two problems in the inherited from environment
which prevent the logger initialization to work properly. In hadoop-env.sh,
the HADOOP_OPTS is inherited from the parent process. This configuration was
requested by user to have a way to override HADOOP environment in the
configuration template:
{noformat}
export HADOOP_OPTS=-Djava.net.preferIPv4Stack=true $HADOOP_OPTS
{noformat}
-Dhadoop.log.dir=$HADOOP_LOG_DIR/task_tracker_user is injected into
HADOOP_OPTS in the tasktracker environment. Hence, the running task would
inherit the wrong logging directory, which the end user might not have
sufficient access to write. Second, $HADOOP_ROOT_LOGGER is override to:
-Dhadoop.root.logger=INFO,TLA by the task controller, therefore, the
bin/hadoop script will attempt to use hadoop.root.logger=INFO,TLA, but fail
to initialize.
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators:
https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Updated] (MAPREDUCE-3112) Calling hadoop cli inside mapreduce job leads to errors
[
https://issues.apache.org/jira/browse/MAPREDUCE-3112?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Eric Yang updated MAPREDUCE-3112:
-
Fix Version/s: 0.23.0
Affects Version/s: 0.23.0
Release Note:
Removed inheritance of certain server environment variables (HADOOP_OPTS and
HADOOP_ROOT_LOGGER) in task attempt process.
Status: Patch Available (was: Open)
Calling hadoop cli inside mapreduce job leads to errors
---
Key: MAPREDUCE-3112
URL: https://issues.apache.org/jira/browse/MAPREDUCE-3112
Project: Hadoop Map/Reduce
Issue Type: Bug
Components: contrib/streaming
Affects Versions: 0.20.205.0, 0.23.0
Environment: Java, Linux
Reporter: Eric Yang
Assignee: Eric Yang
Fix For: 0.20.205.0, 0.23.0
Attachments: MAPREDUCE-3112-trunk.patch, MAPREDUCE-3112.patch
When running a streaming job with mapper
bin/hadoop --config /etc/hadoop/ jar
contrib/streaming/hadoop-streaming-0.20.205.0.jar -mapper hadoop --config
/etc/hadoop/ dfs -help -reducer NONE -input /tmp/input.txt -output NONE
Task log shows:
{noformat}
Exception in thread main java.lang.ExceptionInInitializerError
at org.apache.hadoop.util.ToolRunner.run(ToolRunner.java:57)
at org.apache.hadoop.util.ToolRunner.run(ToolRunner.java:79)
at org.apache.hadoop.fs.FsShell.main(FsShell.java:1895)
Caused by: org.apache.commons.logging.LogConfigurationException:
User-specified log class 'org.apache.commons.logging.impl.Log4JLogger' cannot
be found or is not useable.
at
org.apache.commons.logging.impl.LogFactoryImpl.discoverLogImplementation(LogFactoryImpl.java:874)
at
org.apache.commons.logging.impl.LogFactoryImpl.newInstance(LogFactoryImpl.java:604)
at
org.apache.commons.logging.impl.LogFactoryImpl.getInstance(LogFactoryImpl.java:336)
at
org.apache.commons.logging.impl.LogFactoryImpl.getInstance(LogFactoryImpl.java:310)
at org.apache.commons.logging.LogFactory.getLog(LogFactory.java:685)
at org.apache.hadoop.conf.Configuration.clinit(Configuration.java:142)
... 3 more
java.lang.RuntimeException: PipeMapRed.waitOutputThreads(): subprocess failed
with code 1
at
org.apache.hadoop.streaming.PipeMapRed.waitOutputThreads(PipeMapRed.java:311)
at
org.apache.hadoop.streaming.PipeMapRed.mapRedFinished(PipeMapRed.java:545)
at org.apache.hadoop.streaming.PipeMapper.close(PipeMapper.java:132)
at org.apache.hadoop.mapred.MapRunner.run(MapRunner.java:57)
at org.apache.hadoop.streaming.PipeMapRunner.run(PipeMapRunner.java:36)
at org.apache.hadoop.mapred.MapTask.runOldMapper(MapTask.java:436)
at org.apache.hadoop.mapred.MapTask.run(MapTask.java:372)
at org.apache.hadoop.mapred.Child$4.run(Child.java:261)
at java.security.AccessController.doPrivileged(Native Method)
at javax.security.auth.Subject.doAs(Subject.java:396)
at
org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1059)
at org.apache.hadoop.mapred.Child.main(Child.java:255)
{noformat}
Upon inspection, there are two problems in the inherited from environment
which prevent the logger initialization to work properly. In hadoop-env.sh,
the HADOOP_OPTS is inherited from the parent process. This configuration was
requested by user to have a way to override HADOOP environment in the
configuration template:
{noformat}
export HADOOP_OPTS=-Djava.net.preferIPv4Stack=true $HADOOP_OPTS
{noformat}
-Dhadoop.log.dir=$HADOOP_LOG_DIR/task_tracker_user is injected into
HADOOP_OPTS in the tasktracker environment. Hence, the running task would
inherit the wrong logging directory, which the end user might not have
sufficient access to write. Second, $HADOOP_ROOT_LOGGER is override to:
-Dhadoop.root.logger=INFO,TLA by the task controller, therefore, the
bin/hadoop script will attempt to use hadoop.root.logger=INFO,TLA, but fail
to initialize.
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators:
https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Updated] (MAPREDUCE-3112) Calling hadoop cli inside mapreduce job leads to errors
[
https://issues.apache.org/jira/browse/MAPREDUCE-3112?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Eric Yang updated MAPREDUCE-3112:
-
Attachment: HAPREDUCE-3112-1.patch
There is no need of HADOOP_USER_OPTS because HADOOP_CLIENT_OPS already exists.
The patch is modified to have client ops be part of HADOOP_OPTS in the template.
Calling hadoop cli inside mapreduce job leads to errors
---
Key: MAPREDUCE-3112
URL: https://issues.apache.org/jira/browse/MAPREDUCE-3112
Project: Hadoop Map/Reduce
Issue Type: Bug
Components: contrib/streaming
Affects Versions: 0.20.205.0, 0.23.0
Environment: Java, Linux
Reporter: Eric Yang
Assignee: Eric Yang
Fix For: 0.20.205.0, 0.23.0
Attachments: HAPREDUCE-3112-1.patch, MAPREDUCE-3112-trunk.patch,
MAPREDUCE-3112.patch
When running a streaming job with mapper
bin/hadoop --config /etc/hadoop/ jar
contrib/streaming/hadoop-streaming-0.20.205.0.jar -mapper hadoop --config
/etc/hadoop/ dfs -help -reducer NONE -input /tmp/input.txt -output NONE
Task log shows:
{noformat}
Exception in thread main java.lang.ExceptionInInitializerError
at org.apache.hadoop.util.ToolRunner.run(ToolRunner.java:57)
at org.apache.hadoop.util.ToolRunner.run(ToolRunner.java:79)
at org.apache.hadoop.fs.FsShell.main(FsShell.java:1895)
Caused by: org.apache.commons.logging.LogConfigurationException:
User-specified log class 'org.apache.commons.logging.impl.Log4JLogger' cannot
be found or is not useable.
at
org.apache.commons.logging.impl.LogFactoryImpl.discoverLogImplementation(LogFactoryImpl.java:874)
at
org.apache.commons.logging.impl.LogFactoryImpl.newInstance(LogFactoryImpl.java:604)
at
org.apache.commons.logging.impl.LogFactoryImpl.getInstance(LogFactoryImpl.java:336)
at
org.apache.commons.logging.impl.LogFactoryImpl.getInstance(LogFactoryImpl.java:310)
at org.apache.commons.logging.LogFactory.getLog(LogFactory.java:685)
at org.apache.hadoop.conf.Configuration.clinit(Configuration.java:142)
... 3 more
java.lang.RuntimeException: PipeMapRed.waitOutputThreads(): subprocess failed
with code 1
at
org.apache.hadoop.streaming.PipeMapRed.waitOutputThreads(PipeMapRed.java:311)
at
org.apache.hadoop.streaming.PipeMapRed.mapRedFinished(PipeMapRed.java:545)
at org.apache.hadoop.streaming.PipeMapper.close(PipeMapper.java:132)
at org.apache.hadoop.mapred.MapRunner.run(MapRunner.java:57)
at org.apache.hadoop.streaming.PipeMapRunner.run(PipeMapRunner.java:36)
at org.apache.hadoop.mapred.MapTask.runOldMapper(MapTask.java:436)
at org.apache.hadoop.mapred.MapTask.run(MapTask.java:372)
at org.apache.hadoop.mapred.Child$4.run(Child.java:261)
at java.security.AccessController.doPrivileged(Native Method)
at javax.security.auth.Subject.doAs(Subject.java:396)
at
org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1059)
at org.apache.hadoop.mapred.Child.main(Child.java:255)
{noformat}
Upon inspection, there are two problems in the inherited from environment
which prevent the logger initialization to work properly. In hadoop-env.sh,
the HADOOP_OPTS is inherited from the parent process. This configuration was
requested by user to have a way to override HADOOP environment in the
configuration template:
{noformat}
export HADOOP_OPTS=-Djava.net.preferIPv4Stack=true $HADOOP_OPTS
{noformat}
-Dhadoop.log.dir=$HADOOP_LOG_DIR/task_tracker_user is injected into
HADOOP_OPTS in the tasktracker environment. Hence, the running task would
inherit the wrong logging directory, which the end user might not have
sufficient access to write. Second, $HADOOP_ROOT_LOGGER is override to:
-Dhadoop.root.logger=INFO,TLA by the task controller, therefore, the
bin/hadoop script will attempt to use hadoop.root.logger=INFO,TLA, but fail
to initialize.
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators:
https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Updated] (MAPREDUCE-3112) Calling hadoop cli inside mapreduce job leads to errors
[
https://issues.apache.org/jira/browse/MAPREDUCE-3112?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Eric Yang updated MAPREDUCE-3112:
-
Attachment: (was: HAPREDUCE-3112-1.patch)
Calling hadoop cli inside mapreduce job leads to errors
---
Key: MAPREDUCE-3112
URL: https://issues.apache.org/jira/browse/MAPREDUCE-3112
Project: Hadoop Map/Reduce
Issue Type: Bug
Components: contrib/streaming
Affects Versions: 0.20.205.0, 0.23.0
Environment: Java, Linux
Reporter: Eric Yang
Assignee: Eric Yang
Fix For: 0.20.205.0, 0.23.0
Attachments: HAPREDUCE-3112-1.patch, MAPREDUCE-3112-trunk.patch,
MAPREDUCE-3112.patch
When running a streaming job with mapper
bin/hadoop --config /etc/hadoop/ jar
contrib/streaming/hadoop-streaming-0.20.205.0.jar -mapper hadoop --config
/etc/hadoop/ dfs -help -reducer NONE -input /tmp/input.txt -output NONE
Task log shows:
{noformat}
Exception in thread main java.lang.ExceptionInInitializerError
at org.apache.hadoop.util.ToolRunner.run(ToolRunner.java:57)
at org.apache.hadoop.util.ToolRunner.run(ToolRunner.java:79)
at org.apache.hadoop.fs.FsShell.main(FsShell.java:1895)
Caused by: org.apache.commons.logging.LogConfigurationException:
User-specified log class 'org.apache.commons.logging.impl.Log4JLogger' cannot
be found or is not useable.
at
org.apache.commons.logging.impl.LogFactoryImpl.discoverLogImplementation(LogFactoryImpl.java:874)
at
org.apache.commons.logging.impl.LogFactoryImpl.newInstance(LogFactoryImpl.java:604)
at
org.apache.commons.logging.impl.LogFactoryImpl.getInstance(LogFactoryImpl.java:336)
at
org.apache.commons.logging.impl.LogFactoryImpl.getInstance(LogFactoryImpl.java:310)
at org.apache.commons.logging.LogFactory.getLog(LogFactory.java:685)
at org.apache.hadoop.conf.Configuration.clinit(Configuration.java:142)
... 3 more
java.lang.RuntimeException: PipeMapRed.waitOutputThreads(): subprocess failed
with code 1
at
org.apache.hadoop.streaming.PipeMapRed.waitOutputThreads(PipeMapRed.java:311)
at
org.apache.hadoop.streaming.PipeMapRed.mapRedFinished(PipeMapRed.java:545)
at org.apache.hadoop.streaming.PipeMapper.close(PipeMapper.java:132)
at org.apache.hadoop.mapred.MapRunner.run(MapRunner.java:57)
at org.apache.hadoop.streaming.PipeMapRunner.run(PipeMapRunner.java:36)
at org.apache.hadoop.mapred.MapTask.runOldMapper(MapTask.java:436)
at org.apache.hadoop.mapred.MapTask.run(MapTask.java:372)
at org.apache.hadoop.mapred.Child$4.run(Child.java:261)
at java.security.AccessController.doPrivileged(Native Method)
at javax.security.auth.Subject.doAs(Subject.java:396)
at
org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1059)
at org.apache.hadoop.mapred.Child.main(Child.java:255)
{noformat}
Upon inspection, there are two problems in the inherited from environment
which prevent the logger initialization to work properly. In hadoop-env.sh,
the HADOOP_OPTS is inherited from the parent process. This configuration was
requested by user to have a way to override HADOOP environment in the
configuration template:
{noformat}
export HADOOP_OPTS=-Djava.net.preferIPv4Stack=true $HADOOP_OPTS
{noformat}
-Dhadoop.log.dir=$HADOOP_LOG_DIR/task_tracker_user is injected into
HADOOP_OPTS in the tasktracker environment. Hence, the running task would
inherit the wrong logging directory, which the end user might not have
sufficient access to write. Second, $HADOOP_ROOT_LOGGER is override to:
-Dhadoop.root.logger=INFO,TLA by the task controller, therefore, the
bin/hadoop script will attempt to use hadoop.root.logger=INFO,TLA, but fail
to initialize.
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators:
https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Updated] (MAPREDUCE-3112) Calling hadoop cli inside mapreduce job leads to errors
[
https://issues.apache.org/jira/browse/MAPREDUCE-3112?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Eric Yang updated MAPREDUCE-3112:
-
Attachment: HAPREDUCE-3112-1.patch
Make sure HADOOP_OPTS contains HADOOP_CLIENT_OPTS in case Hadoop command is
executed inside streaming job. TaskLogAppender is streamed to the user log
file.
Calling hadoop cli inside mapreduce job leads to errors
---
Key: MAPREDUCE-3112
URL: https://issues.apache.org/jira/browse/MAPREDUCE-3112
Project: Hadoop Map/Reduce
Issue Type: Bug
Components: contrib/streaming
Affects Versions: 0.20.205.0, 0.23.0
Environment: Java, Linux
Reporter: Eric Yang
Assignee: Eric Yang
Fix For: 0.20.205.0, 0.23.0
Attachments: HAPREDUCE-3112-1.patch, MAPREDUCE-3112-trunk.patch,
MAPREDUCE-3112.patch
When running a streaming job with mapper
bin/hadoop --config /etc/hadoop/ jar
contrib/streaming/hadoop-streaming-0.20.205.0.jar -mapper hadoop --config
/etc/hadoop/ dfs -help -reducer NONE -input /tmp/input.txt -output NONE
Task log shows:
{noformat}
Exception in thread main java.lang.ExceptionInInitializerError
at org.apache.hadoop.util.ToolRunner.run(ToolRunner.java:57)
at org.apache.hadoop.util.ToolRunner.run(ToolRunner.java:79)
at org.apache.hadoop.fs.FsShell.main(FsShell.java:1895)
Caused by: org.apache.commons.logging.LogConfigurationException:
User-specified log class 'org.apache.commons.logging.impl.Log4JLogger' cannot
be found or is not useable.
at
org.apache.commons.logging.impl.LogFactoryImpl.discoverLogImplementation(LogFactoryImpl.java:874)
at
org.apache.commons.logging.impl.LogFactoryImpl.newInstance(LogFactoryImpl.java:604)
at
org.apache.commons.logging.impl.LogFactoryImpl.getInstance(LogFactoryImpl.java:336)
at
org.apache.commons.logging.impl.LogFactoryImpl.getInstance(LogFactoryImpl.java:310)
at org.apache.commons.logging.LogFactory.getLog(LogFactory.java:685)
at org.apache.hadoop.conf.Configuration.clinit(Configuration.java:142)
... 3 more
java.lang.RuntimeException: PipeMapRed.waitOutputThreads(): subprocess failed
with code 1
at
org.apache.hadoop.streaming.PipeMapRed.waitOutputThreads(PipeMapRed.java:311)
at
org.apache.hadoop.streaming.PipeMapRed.mapRedFinished(PipeMapRed.java:545)
at org.apache.hadoop.streaming.PipeMapper.close(PipeMapper.java:132)
at org.apache.hadoop.mapred.MapRunner.run(MapRunner.java:57)
at org.apache.hadoop.streaming.PipeMapRunner.run(PipeMapRunner.java:36)
at org.apache.hadoop.mapred.MapTask.runOldMapper(MapTask.java:436)
at org.apache.hadoop.mapred.MapTask.run(MapTask.java:372)
at org.apache.hadoop.mapred.Child$4.run(Child.java:261)
at java.security.AccessController.doPrivileged(Native Method)
at javax.security.auth.Subject.doAs(Subject.java:396)
at
org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1059)
at org.apache.hadoop.mapred.Child.main(Child.java:255)
{noformat}
Upon inspection, there are two problems in the inherited from environment
which prevent the logger initialization to work properly. In hadoop-env.sh,
the HADOOP_OPTS is inherited from the parent process. This configuration was
requested by user to have a way to override HADOOP environment in the
configuration template:
{noformat}
export HADOOP_OPTS=-Djava.net.preferIPv4Stack=true $HADOOP_OPTS
{noformat}
-Dhadoop.log.dir=$HADOOP_LOG_DIR/task_tracker_user is injected into
HADOOP_OPTS in the tasktracker environment. Hence, the running task would
inherit the wrong logging directory, which the end user might not have
sufficient access to write. Second, $HADOOP_ROOT_LOGGER is override to:
-Dhadoop.root.logger=INFO,TLA by the task controller, therefore, the
bin/hadoop script will attempt to use hadoop.root.logger=INFO,TLA, but fail
to initialize.
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators:
https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Updated] (MAPREDUCE-3112) Calling hadoop cli inside mapreduce job leads to errors
[
https://issues.apache.org/jira/browse/MAPREDUCE-3112?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Eric Yang updated MAPREDUCE-3112:
-
Attachment: MAPREDUCE-3112-trunk-2.patch
Updated configuration to have HADOOP_CLIENT_OPTS override.
Calling hadoop cli inside mapreduce job leads to errors
---
Key: MAPREDUCE-3112
URL: https://issues.apache.org/jira/browse/MAPREDUCE-3112
Project: Hadoop Map/Reduce
Issue Type: Bug
Components: contrib/streaming
Affects Versions: 0.20.205.0, 0.23.0
Environment: Java, Linux
Reporter: Eric Yang
Assignee: Eric Yang
Fix For: 0.20.205.0, 0.23.0
Attachments: HAPREDUCE-3112-1.patch, MAPREDUCE-3112-trunk-2.patch,
MAPREDUCE-3112-trunk.patch, MAPREDUCE-3112.patch
When running a streaming job with mapper
bin/hadoop --config /etc/hadoop/ jar
contrib/streaming/hadoop-streaming-0.20.205.0.jar -mapper hadoop --config
/etc/hadoop/ dfs -help -reducer NONE -input /tmp/input.txt -output NONE
Task log shows:
{noformat}
Exception in thread main java.lang.ExceptionInInitializerError
at org.apache.hadoop.util.ToolRunner.run(ToolRunner.java:57)
at org.apache.hadoop.util.ToolRunner.run(ToolRunner.java:79)
at org.apache.hadoop.fs.FsShell.main(FsShell.java:1895)
Caused by: org.apache.commons.logging.LogConfigurationException:
User-specified log class 'org.apache.commons.logging.impl.Log4JLogger' cannot
be found or is not useable.
at
org.apache.commons.logging.impl.LogFactoryImpl.discoverLogImplementation(LogFactoryImpl.java:874)
at
org.apache.commons.logging.impl.LogFactoryImpl.newInstance(LogFactoryImpl.java:604)
at
org.apache.commons.logging.impl.LogFactoryImpl.getInstance(LogFactoryImpl.java:336)
at
org.apache.commons.logging.impl.LogFactoryImpl.getInstance(LogFactoryImpl.java:310)
at org.apache.commons.logging.LogFactory.getLog(LogFactory.java:685)
at org.apache.hadoop.conf.Configuration.clinit(Configuration.java:142)
... 3 more
java.lang.RuntimeException: PipeMapRed.waitOutputThreads(): subprocess failed
with code 1
at
org.apache.hadoop.streaming.PipeMapRed.waitOutputThreads(PipeMapRed.java:311)
at
org.apache.hadoop.streaming.PipeMapRed.mapRedFinished(PipeMapRed.java:545)
at org.apache.hadoop.streaming.PipeMapper.close(PipeMapper.java:132)
at org.apache.hadoop.mapred.MapRunner.run(MapRunner.java:57)
at org.apache.hadoop.streaming.PipeMapRunner.run(PipeMapRunner.java:36)
at org.apache.hadoop.mapred.MapTask.runOldMapper(MapTask.java:436)
at org.apache.hadoop.mapred.MapTask.run(MapTask.java:372)
at org.apache.hadoop.mapred.Child$4.run(Child.java:261)
at java.security.AccessController.doPrivileged(Native Method)
at javax.security.auth.Subject.doAs(Subject.java:396)
at
org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1059)
at org.apache.hadoop.mapred.Child.main(Child.java:255)
{noformat}
Upon inspection, there are two problems in the inherited from environment
which prevent the logger initialization to work properly. In hadoop-env.sh,
the HADOOP_OPTS is inherited from the parent process. This configuration was
requested by user to have a way to override HADOOP environment in the
configuration template:
{noformat}
export HADOOP_OPTS=-Djava.net.preferIPv4Stack=true $HADOOP_OPTS
{noformat}
-Dhadoop.log.dir=$HADOOP_LOG_DIR/task_tracker_user is injected into
HADOOP_OPTS in the tasktracker environment. Hence, the running task would
inherit the wrong logging directory, which the end user might not have
sufficient access to write. Second, $HADOOP_ROOT_LOGGER is override to:
-Dhadoop.root.logger=INFO,TLA by the task controller, therefore, the
bin/hadoop script will attempt to use hadoop.root.logger=INFO,TLA, but fail
to initialize.
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators:
https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira
