[jira] [Commented] (MAPREDUCE-5795) Job should be marked as Failed if it is recovered from commit.
[ https://issues.apache.org/jira/browse/MAPREDUCE-5795?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13946407#comment-13946407 ] Hudson commented on MAPREDUCE-5795: --- FAILURE: Integrated in Hadoop-Yarn-trunk #520 (See [https://builds.apache.org/job/Hadoop-Yarn-trunk/520/]) MAPREDUCE-5795. Fixed MRAppMaster to record the correct job-state after it recovers from a commit during a previous attempt. Contributed by Xuan Gong. (vinodkv: http://svn.apache.org/viewcvs.cgi/?root=Apache-SVNview=revrev=1581180) * /hadoop/common/trunk/hadoop-mapreduce-project/CHANGES.txt * /hadoop/common/trunk/hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-app/src/main/java/org/apache/hadoop/mapreduce/jobhistory/JobHistoryEventHandler.java * /hadoop/common/trunk/hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-app/src/main/java/org/apache/hadoop/mapreduce/v2/app/MRAppMaster.java * /hadoop/common/trunk/hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-app/src/test/java/org/apache/hadoop/mapreduce/jobhistory/TestJobHistoryEventHandler.java * /hadoop/common/trunk/hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-app/src/test/java/org/apache/hadoop/mapreduce/v2/app/TestMRAppMaster.java * /hadoop/common/trunk/hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-core/src/main/java/org/apache/hadoop/mapreduce/jobhistory/AMStartedEvent.java Job should be marked as Failed if it is recovered from commit. -- Key: MAPREDUCE-5795 URL: https://issues.apache.org/jira/browse/MAPREDUCE-5795 Project: Hadoop Map/Reduce Issue Type: Bug Reporter: Yesha Vora Assignee: Xuan Gong Fix For: 2.4.0 Attachments: MAPREDUCE-5795.1.patch, MAPREDUCE-5795.2.patch If Resource manager is restarted when a job is in commit state, The job is not able to recovered after RM restart and it is marked as Killed. The job status should be Failed instead killed. -- This message was sent by Atlassian JIRA (v6.2#6252)
[jira] [Commented] (MAPREDUCE-5791) Shuffle phase is slow in Windows - FadviseFileRegion::transferTo does not read disks efficiently
[ https://issues.apache.org/jira/browse/MAPREDUCE-5791?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13946411#comment-13946411 ] Hudson commented on MAPREDUCE-5791: --- FAILURE: Integrated in Hadoop-Yarn-trunk #520 (See [https://builds.apache.org/job/Hadoop-Yarn-trunk/520/]) MAPREDUCE-5791. Shuffle phase is slow in Windows - FadviseFileRegion::transferTo does not read disks efficiently. Contributed by Nikola Vujic. (cnauroth: http://svn.apache.org/viewcvs.cgi/?root=Apache-SVNview=revrev=1580994) * /hadoop/common/trunk/hadoop-mapreduce-project/CHANGES.txt * /hadoop/common/trunk/hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-core/src/main/resources/mapred-default.xml * /hadoop/common/trunk/hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-shuffle/src/main/java/org/apache/hadoop/mapred/FadvisedFileRegion.java * /hadoop/common/trunk/hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-shuffle/src/main/java/org/apache/hadoop/mapred/ShuffleHandler.java * /hadoop/common/trunk/hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-shuffle/src/test/java/org/apache/hadoop/mapred/TestFadvisedFileRegion.java Shuffle phase is slow in Windows - FadviseFileRegion::transferTo does not read disks efficiently Key: MAPREDUCE-5791 URL: https://issues.apache.org/jira/browse/MAPREDUCE-5791 Project: Hadoop Map/Reduce Issue Type: Bug Components: client Affects Versions: 3.0.0, 2.3.0 Reporter: Nikola Vujic Assignee: Nikola Vujic Fix For: 3.0.0, 2.4.0 Attachments: MAPREDUCE-5791.patch, MAPREDUCE-5791.patch, MAPREDUCE-5791.patch transferTo method in org.apache.hadoop.mapred.FadvisedFileRegion is using transferTo method from a FileChannel to transfer data from a disk to socket. This is performing slow in Windows, slower than in Linux. The reason is that transferTo method for the java.nio is issuing 32K IO requests all the time. In Windows, these 32K transfers are not optimal and we don't get the best performance form the underlying IO subsystem. In order to achieve better performance when reading from the drives, we need to read data in bigger chunks, 512K for example. -- This message was sent by Atlassian JIRA (v6.2#6252)
[jira] [Commented] (MAPREDUCE-5791) Shuffle phase is slow in Windows - FadviseFileRegion::transferTo does not read disks efficiently
[ https://issues.apache.org/jira/browse/MAPREDUCE-5791?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13946543#comment-13946543 ] Hudson commented on MAPREDUCE-5791: --- FAILURE: Integrated in Hadoop-Mapreduce-trunk #1737 (See [https://builds.apache.org/job/Hadoop-Mapreduce-trunk/1737/]) MAPREDUCE-5791. Shuffle phase is slow in Windows - FadviseFileRegion::transferTo does not read disks efficiently. Contributed by Nikola Vujic. (cnauroth: http://svn.apache.org/viewcvs.cgi/?root=Apache-SVNview=revrev=1580994) * /hadoop/common/trunk/hadoop-mapreduce-project/CHANGES.txt * /hadoop/common/trunk/hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-core/src/main/resources/mapred-default.xml * /hadoop/common/trunk/hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-shuffle/src/main/java/org/apache/hadoop/mapred/FadvisedFileRegion.java * /hadoop/common/trunk/hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-shuffle/src/main/java/org/apache/hadoop/mapred/ShuffleHandler.java * /hadoop/common/trunk/hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-shuffle/src/test/java/org/apache/hadoop/mapred/TestFadvisedFileRegion.java Shuffle phase is slow in Windows - FadviseFileRegion::transferTo does not read disks efficiently Key: MAPREDUCE-5791 URL: https://issues.apache.org/jira/browse/MAPREDUCE-5791 Project: Hadoop Map/Reduce Issue Type: Bug Components: client Affects Versions: 3.0.0, 2.3.0 Reporter: Nikola Vujic Assignee: Nikola Vujic Fix For: 3.0.0, 2.4.0 Attachments: MAPREDUCE-5791.patch, MAPREDUCE-5791.patch, MAPREDUCE-5791.patch transferTo method in org.apache.hadoop.mapred.FadvisedFileRegion is using transferTo method from a FileChannel to transfer data from a disk to socket. This is performing slow in Windows, slower than in Linux. The reason is that transferTo method for the java.nio is issuing 32K IO requests all the time. In Windows, these 32K transfers are not optimal and we don't get the best performance form the underlying IO subsystem. In order to achieve better performance when reading from the drives, we need to read data in bigger chunks, 512K for example. -- This message was sent by Atlassian JIRA (v6.2#6252)
[jira] [Commented] (MAPREDUCE-5795) Job should be marked as Failed if it is recovered from commit.
[ https://issues.apache.org/jira/browse/MAPREDUCE-5795?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13946539#comment-13946539 ] Hudson commented on MAPREDUCE-5795: --- FAILURE: Integrated in Hadoop-Mapreduce-trunk #1737 (See [https://builds.apache.org/job/Hadoop-Mapreduce-trunk/1737/]) MAPREDUCE-5795. Fixed MRAppMaster to record the correct job-state after it recovers from a commit during a previous attempt. Contributed by Xuan Gong. (vinodkv: http://svn.apache.org/viewcvs.cgi/?root=Apache-SVNview=revrev=1581180) * /hadoop/common/trunk/hadoop-mapreduce-project/CHANGES.txt * /hadoop/common/trunk/hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-app/src/main/java/org/apache/hadoop/mapreduce/jobhistory/JobHistoryEventHandler.java * /hadoop/common/trunk/hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-app/src/main/java/org/apache/hadoop/mapreduce/v2/app/MRAppMaster.java * /hadoop/common/trunk/hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-app/src/test/java/org/apache/hadoop/mapreduce/jobhistory/TestJobHistoryEventHandler.java * /hadoop/common/trunk/hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-app/src/test/java/org/apache/hadoop/mapreduce/v2/app/TestMRAppMaster.java * /hadoop/common/trunk/hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-core/src/main/java/org/apache/hadoop/mapreduce/jobhistory/AMStartedEvent.java Job should be marked as Failed if it is recovered from commit. -- Key: MAPREDUCE-5795 URL: https://issues.apache.org/jira/browse/MAPREDUCE-5795 Project: Hadoop Map/Reduce Issue Type: Bug Reporter: Yesha Vora Assignee: Xuan Gong Fix For: 2.4.0 Attachments: MAPREDUCE-5795.1.patch, MAPREDUCE-5795.2.patch If Resource manager is restarted when a job is in commit state, The job is not able to recovered after RM restart and it is marked as Killed. The job status should be Failed instead killed. -- This message was sent by Atlassian JIRA (v6.2#6252)
[jira] [Commented] (MAPREDUCE-5795) Job should be marked as Failed if it is recovered from commit.
[ https://issues.apache.org/jira/browse/MAPREDUCE-5795?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13946567#comment-13946567 ] Hudson commented on MAPREDUCE-5795: --- FAILURE: Integrated in Hadoop-Hdfs-trunk #1712 (See [https://builds.apache.org/job/Hadoop-Hdfs-trunk/1712/]) MAPREDUCE-5795. Fixed MRAppMaster to record the correct job-state after it recovers from a commit during a previous attempt. Contributed by Xuan Gong. (vinodkv: http://svn.apache.org/viewcvs.cgi/?root=Apache-SVNview=revrev=1581180) * /hadoop/common/trunk/hadoop-mapreduce-project/CHANGES.txt * /hadoop/common/trunk/hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-app/src/main/java/org/apache/hadoop/mapreduce/jobhistory/JobHistoryEventHandler.java * /hadoop/common/trunk/hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-app/src/main/java/org/apache/hadoop/mapreduce/v2/app/MRAppMaster.java * /hadoop/common/trunk/hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-app/src/test/java/org/apache/hadoop/mapreduce/jobhistory/TestJobHistoryEventHandler.java * /hadoop/common/trunk/hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-app/src/test/java/org/apache/hadoop/mapreduce/v2/app/TestMRAppMaster.java * /hadoop/common/trunk/hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-core/src/main/java/org/apache/hadoop/mapreduce/jobhistory/AMStartedEvent.java Job should be marked as Failed if it is recovered from commit. -- Key: MAPREDUCE-5795 URL: https://issues.apache.org/jira/browse/MAPREDUCE-5795 Project: Hadoop Map/Reduce Issue Type: Bug Reporter: Yesha Vora Assignee: Xuan Gong Fix For: 2.4.0 Attachments: MAPREDUCE-5795.1.patch, MAPREDUCE-5795.2.patch If Resource manager is restarted when a job is in commit state, The job is not able to recovered after RM restart and it is marked as Killed. The job status should be Failed instead killed. -- This message was sent by Atlassian JIRA (v6.2#6252)
[jira] [Commented] (MAPREDUCE-5791) Shuffle phase is slow in Windows - FadviseFileRegion::transferTo does not read disks efficiently
[ https://issues.apache.org/jira/browse/MAPREDUCE-5791?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13946571#comment-13946571 ] Hudson commented on MAPREDUCE-5791: --- FAILURE: Integrated in Hadoop-Hdfs-trunk #1712 (See [https://builds.apache.org/job/Hadoop-Hdfs-trunk/1712/]) MAPREDUCE-5791. Shuffle phase is slow in Windows - FadviseFileRegion::transferTo does not read disks efficiently. Contributed by Nikola Vujic. (cnauroth: http://svn.apache.org/viewcvs.cgi/?root=Apache-SVNview=revrev=1580994) * /hadoop/common/trunk/hadoop-mapreduce-project/CHANGES.txt * /hadoop/common/trunk/hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-core/src/main/resources/mapred-default.xml * /hadoop/common/trunk/hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-shuffle/src/main/java/org/apache/hadoop/mapred/FadvisedFileRegion.java * /hadoop/common/trunk/hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-shuffle/src/main/java/org/apache/hadoop/mapred/ShuffleHandler.java * /hadoop/common/trunk/hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-shuffle/src/test/java/org/apache/hadoop/mapred/TestFadvisedFileRegion.java Shuffle phase is slow in Windows - FadviseFileRegion::transferTo does not read disks efficiently Key: MAPREDUCE-5791 URL: https://issues.apache.org/jira/browse/MAPREDUCE-5791 Project: Hadoop Map/Reduce Issue Type: Bug Components: client Affects Versions: 3.0.0, 2.3.0 Reporter: Nikola Vujic Assignee: Nikola Vujic Fix For: 3.0.0, 2.4.0 Attachments: MAPREDUCE-5791.patch, MAPREDUCE-5791.patch, MAPREDUCE-5791.patch transferTo method in org.apache.hadoop.mapred.FadvisedFileRegion is using transferTo method from a FileChannel to transfer data from a disk to socket. This is performing slow in Windows, slower than in Linux. The reason is that transferTo method for the java.nio is issuing 32K IO requests all the time. In Windows, these 32K transfers are not optimal and we don't get the best performance form the underlying IO subsystem. In order to achieve better performance when reading from the drives, we need to read data in bigger chunks, 512K for example. -- This message was sent by Atlassian JIRA (v6.2#6252)
[jira] [Work started] (MAPREDUCE-5809) Enhance distcp to support preserving HDFS ACLs.
[ https://issues.apache.org/jira/browse/MAPREDUCE-5809?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Work on MAPREDUCE-5809 started by Chris Nauroth. Enhance distcp to support preserving HDFS ACLs. --- Key: MAPREDUCE-5809 URL: https://issues.apache.org/jira/browse/MAPREDUCE-5809 Project: Hadoop Map/Reduce Issue Type: Improvement Components: distcp Affects Versions: 2.4.0 Reporter: Chris Nauroth Assignee: Chris Nauroth This issue tracks enhancing distcp to add a new command-line argument for preserving HDFS ACLs from the source at the copy destination. -- This message was sent by Atlassian JIRA (v6.2#6252)
[jira] [Moved] (MAPREDUCE-5809) Enhance distcp to support preserving HDFS ACLs.
[ https://issues.apache.org/jira/browse/MAPREDUCE-5809?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Chris Nauroth moved HADOOP-10435 to MAPREDUCE-5809: --- Component/s: (was: tools/distcp) distcp Target Version/s: 3.0.0, 2.5.0 (was: 3.0.0, 2.5.0) Affects Version/s: (was: 2.4.0) 2.4.0 Key: MAPREDUCE-5809 (was: HADOOP-10435) Project: Hadoop Map/Reduce (was: Hadoop Common) Enhance distcp to support preserving HDFS ACLs. --- Key: MAPREDUCE-5809 URL: https://issues.apache.org/jira/browse/MAPREDUCE-5809 Project: Hadoop Map/Reduce Issue Type: Improvement Components: distcp Affects Versions: 2.4.0 Reporter: Chris Nauroth Assignee: Chris Nauroth This issue tracks enhancing distcp to add a new command-line argument for preserving HDFS ACLs from the source at the copy destination. -- This message was sent by Atlassian JIRA (v6.2#6252)
[jira] [Commented] (MAPREDUCE-5809) Enhance distcp to support preserving HDFS ACLs.
[ https://issues.apache.org/jira/browse/MAPREDUCE-5809?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13947234#comment-13947234 ] Suresh Srinivas commented on MAPREDUCE-5809: [~cnauroth], I am sure you have thought about this already; we should take care of distcp from src cluster to destination, where src has ACL enabled and destination does not have it. Enhance distcp to support preserving HDFS ACLs. --- Key: MAPREDUCE-5809 URL: https://issues.apache.org/jira/browse/MAPREDUCE-5809 Project: Hadoop Map/Reduce Issue Type: Improvement Components: distcp Affects Versions: 2.4.0 Reporter: Chris Nauroth Assignee: Chris Nauroth This issue tracks enhancing distcp to add a new command-line argument for preserving HDFS ACLs from the source at the copy destination. -- This message was sent by Atlassian JIRA (v6.2#6252)
[jira] [Commented] (MAPREDUCE-5809) Enhance distcp to support preserving HDFS ACLs.
[
https://issues.apache.org/jira/browse/MAPREDUCE-5809?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13947252#comment-13947252
]
Chris Nauroth commented on MAPREDUCE-5809:
--
[~sureshms], yes, this is a very important consideration. Here is what I had
in mind for the logic:
# distcp CLI accepts a new optional flag: -pa for preserve ACLs. The
presence of -pa also implies the existing -pp flag, because ACLs are a
super-set of permissions.
# If preserving ACLs, then before submitting the job, distcp sends a canary
{{getAclStatus}} request for / on the source and target file systems. This
will detect ACL compatibility/support problems and fail fast before even
submitting the job. There are three specific sub-cases that this check catches:
## File system is HDFS 2.4, so the getAclStatus RPC endpoint doesn't exist.
## File system is HDFS = 2.4, but ACLs are not enabled.
## File system is a {{FileSystem}} subclass that doesn't override the ACL APIs.
({{UnsupportedOperationException}})
# Then, distcp map tasks call {{getAclStatus}} and {{setAcl}} instead of
{{setPermission}}.
If the operator wishes to copy files with ACLs to a target cluster and just
drop the ACLs, then they'd run without the -pa option, and distcp would
continue to work as it does today.
One additional thing I just learned while reading the code is that we support
passing the -p flag with no additional arguments, and this is assumed to
preserve replication, block size, user, group, permission and checksum type.
I'm planning on preserving this behavior. I don't think we can include -pa as
part of the defaults, because that could break existing deployments that are
running distcp -p if they start using ACLs on one cluster but not the other.
Enhance distcp to support preserving HDFS ACLs.
---
Key: MAPREDUCE-5809
URL: https://issues.apache.org/jira/browse/MAPREDUCE-5809
Project: Hadoop Map/Reduce
Issue Type: Improvement
Components: distcp
Affects Versions: 2.4.0
Reporter: Chris Nauroth
Assignee: Chris Nauroth
This issue tracks enhancing distcp to add a new command-line argument for
preserving HDFS ACLs from the source at the copy destination.
--
This message was sent by Atlassian JIRA
(v6.2#6252)
[jira] [Commented] (MAPREDUCE-5809) Enhance distcp to support preserving HDFS ACLs.
[ https://issues.apache.org/jira/browse/MAPREDUCE-5809?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13947257#comment-13947257 ] Suresh Srinivas commented on MAPREDUCE-5809: [~cnauroth], that sounds excellent. Thanks for answering. Enhance distcp to support preserving HDFS ACLs. --- Key: MAPREDUCE-5809 URL: https://issues.apache.org/jira/browse/MAPREDUCE-5809 Project: Hadoop Map/Reduce Issue Type: Improvement Components: distcp Affects Versions: 2.4.0 Reporter: Chris Nauroth Assignee: Chris Nauroth This issue tracks enhancing distcp to add a new command-line argument for preserving HDFS ACLs from the source at the copy destination. -- This message was sent by Atlassian JIRA (v6.2#6252)
[jira] [Commented] (MAPREDUCE-5799) add default value of MR_AM_ADMIN_USER_ENV
[
https://issues.apache.org/jira/browse/MAPREDUCE-5799?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13947317#comment-13947317
]
Mohammad Kamrul Islam commented on MAPREDUCE-5799:
--
+1
[~liangly] : Please assign the JIRA to yourself.
add default value of MR_AM_ADMIN_USER_ENV
-
Key: MAPREDUCE-5799
URL: https://issues.apache.org/jira/browse/MAPREDUCE-5799
Project: Hadoop Map/Reduce
Issue Type: Improvement
Affects Versions: 2.3.0
Reporter: Liyin Liang
Priority: Minor
Attachments: MAPREDUCE-5799.diff
Submit a 1 map + 1 reduce sleep job with the following config:
{code}
property
namemapreduce.map.output.compress/name
valuetrue/value
/property
property
namemapreduce.map.output.compress.codec/name
valueorg.apache.hadoop.io.compress.SnappyCodec/value
/property
property
namemapreduce.job.ubertask.enable/name
valuetrue/value
/property
{code}
And the LinuxContainerExecutor is enable on NodeManager.
This job will fail with the following error:
{code}
2014-03-18 21:28:20,153 FATAL [uber-SubtaskRunner]
org.apache.hadoop.mapred.LocalContainerLauncher: Error running local
(uberized) 'child' : java.lang.UnsatisfiedLinkError:
org.apache.hadoop.util.NativeCodeLoader.buildSupportsSnappy()Z
at org.apache.hadoop.util.NativeCodeLoader.buildSupportsSnappy(Native
Method)
at
org.apache.hadoop.io.compress.SnappyCodec.checkNativeCodeLoaded(SnappyCodec.java:63)
at
org.apache.hadoop.io.compress.SnappyCodec.getCompressorType(SnappyCodec.java:132)
at
org.apache.hadoop.io.compress.CodecPool.getCompressor(CodecPool.java:148)
at
org.apache.hadoop.io.compress.CodecPool.getCompressor(CodecPool.java:163)
at org.apache.hadoop.mapred.IFile$Writer.init(IFile.java:115)
at
org.apache.hadoop.mapred.MapTask$MapOutputBuffer.sortAndSpill(MapTask.java:1583)
at
org.apache.hadoop.mapred.MapTask$MapOutputBuffer.flush(MapTask.java:1462)
at
org.apache.hadoop.mapred.MapTask$NewOutputCollector.close(MapTask.java:700)
at org.apache.hadoop.mapred.MapTask.closeQuietly(MapTask.java:1990)
at org.apache.hadoop.mapred.MapTask.runNewMapper(MapTask.java:774)
at org.apache.hadoop.mapred.MapTask.run(MapTask.java:340)
at
org.apache.hadoop.mapred.LocalContainerLauncher$SubtaskRunner.runSubtask(LocalContainerLauncher.java:317)
at
org.apache.hadoop.mapred.LocalContainerLauncher$SubtaskRunner.run(LocalContainerLauncher.java:232)
at java.lang.Thread.run(Thread.java:662)
{code}
When create a ContainerLaunchContext for task in
TaskAttemptImpl.createCommonContainerLaunchContext(), the
DEFAULT_MAPRED_ADMIN_USER_ENV which is
LD_LIBRARY_PATH=$HADOOP_COMMON_HOME/lib/native is added to the environment.
Where when create a ContainerLaunchContext for mrappmaster in
YARNRunner.createApplicationSubmissionContext(), there is no default
environment. So the ubermode job fails to find native lib.
--
This message was sent by Atlassian JIRA
(v6.2#6252)
[jira] [Resolved] (MAPREDUCE-4976) Use the new StringUtils methods added by HADOOP-9252
[ https://issues.apache.org/jira/browse/MAPREDUCE-4976?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Tsz Wo Nicholas Sze resolved MAPREDUCE-4976. Resolution: Not A Problem This is actually not a problem. Use the new StringUtils methods added by HADOOP-9252 Key: MAPREDUCE-4976 URL: https://issues.apache.org/jira/browse/MAPREDUCE-4976 Project: Hadoop Map/Reduce Issue Type: Improvement Reporter: Tsz Wo Nicholas Sze Assignee: Tsz Wo Nicholas Sze Priority: Minor HADOOP-9252 slightly changed the format of some StringUtils outputs. Some methods were deprecated by HADOOP-9252. The use of them should be replaced with the new methods. -- This message was sent by Atlassian JIRA (v6.2#6252)
[jira] [Moved] (MAPREDUCE-5810) TestStreamingTaskLog#testStreamingTaskLogWithHadoopCmd is failing
[ https://issues.apache.org/jira/browse/MAPREDUCE-5810?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Akira AJISAKA moved HADOOP-10424 to MAPREDUCE-5810: --- Affects Version/s: (was: 2.4.0) (was: 3.0.0) 2.4.0 3.0.0 Key: MAPREDUCE-5810 (was: HADOOP-10424) Project: Hadoop Map/Reduce (was: Hadoop Common) TestStreamingTaskLog#testStreamingTaskLogWithHadoopCmd is failing - Key: MAPREDUCE-5810 URL: https://issues.apache.org/jira/browse/MAPREDUCE-5810 Project: Hadoop Map/Reduce Issue Type: Bug Affects Versions: 3.0.0, 2.4.0 Reporter: Mit Desai Assignee: Akira AJISAKA Attachments: log.txt testStreamingTaskLogWithHadoopCmd(org.apache.hadoop.streaming.TestStreamingTaskLog) Time elapsed: 44.069 sec FAILURE! java.lang.AssertionError: environment set for child is wrong at org.junit.Assert.fail(Assert.java:93) at org.junit.Assert.assertTrue(Assert.java:43) at org.apache.hadoop.streaming.TestStreamingTaskLog.runStreamJobAndValidateEnv(TestStreamingTaskLog.java:157) at org.apache.hadoop.streaming.TestStreamingTaskLog.testStreamingTaskLogWithHadoopCmd(TestStreamingTaskLog.java:107) Results : Failed tests: TestStreamingTaskLog.testStreamingTaskLogWithHadoopCmd:107-runStreamJobAndValidateEnv:157 environment set for child is wrong -- This message was sent by Atlassian JIRA (v6.2#6252)
[jira] [Updated] (MAPREDUCE-5810) TestStreamingTaskLog#testStreamingTaskLogWithHadoopCmd is failing
[ https://issues.apache.org/jira/browse/MAPREDUCE-5810?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Akira AJISAKA updated MAPREDUCE-5810: - Attachment: MAPREDUCE-5810.patch Moved to MapReduce project, and attaching a patch to delete the test. TestStreamingTaskLog#testStreamingTaskLogWithHadoopCmd is failing - Key: MAPREDUCE-5810 URL: https://issues.apache.org/jira/browse/MAPREDUCE-5810 Project: Hadoop Map/Reduce Issue Type: Bug Affects Versions: 3.0.0, 2.4.0 Reporter: Mit Desai Assignee: Akira AJISAKA Attachments: MAPREDUCE-5810.patch, log.txt testStreamingTaskLogWithHadoopCmd(org.apache.hadoop.streaming.TestStreamingTaskLog) Time elapsed: 44.069 sec FAILURE! java.lang.AssertionError: environment set for child is wrong at org.junit.Assert.fail(Assert.java:93) at org.junit.Assert.assertTrue(Assert.java:43) at org.apache.hadoop.streaming.TestStreamingTaskLog.runStreamJobAndValidateEnv(TestStreamingTaskLog.java:157) at org.apache.hadoop.streaming.TestStreamingTaskLog.testStreamingTaskLogWithHadoopCmd(TestStreamingTaskLog.java:107) Results : Failed tests: TestStreamingTaskLog.testStreamingTaskLogWithHadoopCmd:107-runStreamJobAndValidateEnv:157 environment set for child is wrong -- This message was sent by Atlassian JIRA (v6.2#6252)
[jira] [Updated] (MAPREDUCE-5810) TestStreamingTaskLog#testStreamingTaskLogWithHadoopCmd is failing
[ https://issues.apache.org/jira/browse/MAPREDUCE-5810?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Akira AJISAKA updated MAPREDUCE-5810: - Component/s: contrib/streaming TestStreamingTaskLog#testStreamingTaskLogWithHadoopCmd is failing - Key: MAPREDUCE-5810 URL: https://issues.apache.org/jira/browse/MAPREDUCE-5810 Project: Hadoop Map/Reduce Issue Type: Bug Components: contrib/streaming Affects Versions: 3.0.0, 2.4.0 Reporter: Mit Desai Assignee: Akira AJISAKA Attachments: MAPREDUCE-5810.patch, log.txt testStreamingTaskLogWithHadoopCmd(org.apache.hadoop.streaming.TestStreamingTaskLog) Time elapsed: 44.069 sec FAILURE! java.lang.AssertionError: environment set for child is wrong at org.junit.Assert.fail(Assert.java:93) at org.junit.Assert.assertTrue(Assert.java:43) at org.apache.hadoop.streaming.TestStreamingTaskLog.runStreamJobAndValidateEnv(TestStreamingTaskLog.java:157) at org.apache.hadoop.streaming.TestStreamingTaskLog.testStreamingTaskLogWithHadoopCmd(TestStreamingTaskLog.java:107) Results : Failed tests: TestStreamingTaskLog.testStreamingTaskLogWithHadoopCmd:107-runStreamJobAndValidateEnv:157 environment set for child is wrong -- This message was sent by Atlassian JIRA (v6.2#6252)
[jira] [Updated] (MAPREDUCE-5810) TestStreamingTaskLog#testStreamingTaskLogWithHadoopCmd is failing
[ https://issues.apache.org/jira/browse/MAPREDUCE-5810?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Akira AJISAKA updated MAPREDUCE-5810: - Target Version/s: 2.4.0 Status: Patch Available (was: Open) TestStreamingTaskLog#testStreamingTaskLogWithHadoopCmd is failing - Key: MAPREDUCE-5810 URL: https://issues.apache.org/jira/browse/MAPREDUCE-5810 Project: Hadoop Map/Reduce Issue Type: Bug Components: contrib/streaming Affects Versions: 3.0.0, 2.4.0 Reporter: Mit Desai Assignee: Akira AJISAKA Attachments: MAPREDUCE-5810.patch, log.txt testStreamingTaskLogWithHadoopCmd(org.apache.hadoop.streaming.TestStreamingTaskLog) Time elapsed: 44.069 sec FAILURE! java.lang.AssertionError: environment set for child is wrong at org.junit.Assert.fail(Assert.java:93) at org.junit.Assert.assertTrue(Assert.java:43) at org.apache.hadoop.streaming.TestStreamingTaskLog.runStreamJobAndValidateEnv(TestStreamingTaskLog.java:157) at org.apache.hadoop.streaming.TestStreamingTaskLog.testStreamingTaskLogWithHadoopCmd(TestStreamingTaskLog.java:107) Results : Failed tests: TestStreamingTaskLog.testStreamingTaskLogWithHadoopCmd:107-runStreamJobAndValidateEnv:157 environment set for child is wrong -- This message was sent by Atlassian JIRA (v6.2#6252)
