[jira] [Commented] (MAPREDUCE-5810) TestStreamingTaskLog#testStreamingTaskLogWithHadoopCmd is failing
[
https://issues.apache.org/jira/browse/MAPREDUCE-5810?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13947583#comment-13947583
]
Hadoop QA commented on MAPREDUCE-5810:
--
{color:red}-1 overall{color}. Here are the results of testing the latest
attachment
http://issues.apache.org/jira/secure/attachment/12636860/MAPREDUCE-5810.patch
against trunk revision .
{color:green}+1 @author{color}. The patch does not contain any @author
tags.
{color:red}-1 tests included{color}. The patch doesn't appear to include
any new or modified tests.
Please justify why no new tests are needed for this
patch.
Also please list what manual steps were performed to
verify this patch.
{color:green}+1 javac{color}. The applied patch does not increase the
total number of javac compiler warnings.
{color:green}+1 javadoc{color}. There were no new javadoc warning messages.
{color:green}+1 eclipse:eclipse{color}. The patch built with
eclipse:eclipse.
{color:green}+1 findbugs{color}. The patch does not introduce any new
Findbugs (version 1.3.9) warnings.
{color:green}+1 release audit{color}. The applied patch does not increase
the total number of release audit warnings.
{color:green}+1 core tests{color}. The patch passed unit tests in
hadoop-tools/hadoop-streaming.
{color:green}+1 contrib tests{color}. The patch passed contrib unit tests.
Test results:
https://builds.apache.org/job/PreCommit-MAPREDUCE-Build/4460//testReport/
Console output:
https://builds.apache.org/job/PreCommit-MAPREDUCE-Build/4460//console
This message is automatically generated.
> TestStreamingTaskLog#testStreamingTaskLogWithHadoopCmd is failing
> -
>
> Key: MAPREDUCE-5810
> URL: https://issues.apache.org/jira/browse/MAPREDUCE-5810
> Project: Hadoop Map/Reduce
> Issue Type: Bug
> Components: contrib/streaming
>Affects Versions: 3.0.0, 2.4.0
>Reporter: Mit Desai
>Assignee: Akira AJISAKA
> Attachments: MAPREDUCE-5810.patch, log.txt
>
>
> testStreamingTaskLogWithHadoopCmd(org.apache.hadoop.streaming.TestStreamingTaskLog)
> Time elapsed: 44.069 sec <<< FAILURE!
> java.lang.AssertionError: environment set for child is wrong
> at org.junit.Assert.fail(Assert.java:93)
> at org.junit.Assert.assertTrue(Assert.java:43)
> at
> org.apache.hadoop.streaming.TestStreamingTaskLog.runStreamJobAndValidateEnv(TestStreamingTaskLog.java:157)
> at
> org.apache.hadoop.streaming.TestStreamingTaskLog.testStreamingTaskLogWithHadoopCmd(TestStreamingTaskLog.java:107)
> Results :
> Failed tests:
>
> TestStreamingTaskLog.testStreamingTaskLogWithHadoopCmd:107->runStreamJobAndValidateEnv:157
> environment set for child is wrong
--
This message was sent by Atlassian JIRA
(v6.2#6252)
[jira] [Updated] (MAPREDUCE-5810) TestStreamingTaskLog#testStreamingTaskLogWithHadoopCmd is failing
[ https://issues.apache.org/jira/browse/MAPREDUCE-5810?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Akira AJISAKA updated MAPREDUCE-5810: - Target Version/s: 2.4.0 Status: Patch Available (was: Open) > TestStreamingTaskLog#testStreamingTaskLogWithHadoopCmd is failing > - > > Key: MAPREDUCE-5810 > URL: https://issues.apache.org/jira/browse/MAPREDUCE-5810 > Project: Hadoop Map/Reduce > Issue Type: Bug > Components: contrib/streaming >Affects Versions: 3.0.0, 2.4.0 >Reporter: Mit Desai >Assignee: Akira AJISAKA > Attachments: MAPREDUCE-5810.patch, log.txt > > > testStreamingTaskLogWithHadoopCmd(org.apache.hadoop.streaming.TestStreamingTaskLog) > Time elapsed: 44.069 sec <<< FAILURE! > java.lang.AssertionError: environment set for child is wrong > at org.junit.Assert.fail(Assert.java:93) > at org.junit.Assert.assertTrue(Assert.java:43) > at > org.apache.hadoop.streaming.TestStreamingTaskLog.runStreamJobAndValidateEnv(TestStreamingTaskLog.java:157) > at > org.apache.hadoop.streaming.TestStreamingTaskLog.testStreamingTaskLogWithHadoopCmd(TestStreamingTaskLog.java:107) > Results : > Failed tests: > > TestStreamingTaskLog.testStreamingTaskLogWithHadoopCmd:107->runStreamJobAndValidateEnv:157 > environment set for child is wrong -- This message was sent by Atlassian JIRA (v6.2#6252)
[jira] [Updated] (MAPREDUCE-5810) TestStreamingTaskLog#testStreamingTaskLogWithHadoopCmd is failing
[ https://issues.apache.org/jira/browse/MAPREDUCE-5810?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Akira AJISAKA updated MAPREDUCE-5810: - Component/s: contrib/streaming > TestStreamingTaskLog#testStreamingTaskLogWithHadoopCmd is failing > - > > Key: MAPREDUCE-5810 > URL: https://issues.apache.org/jira/browse/MAPREDUCE-5810 > Project: Hadoop Map/Reduce > Issue Type: Bug > Components: contrib/streaming >Affects Versions: 3.0.0, 2.4.0 >Reporter: Mit Desai >Assignee: Akira AJISAKA > Attachments: MAPREDUCE-5810.patch, log.txt > > > testStreamingTaskLogWithHadoopCmd(org.apache.hadoop.streaming.TestStreamingTaskLog) > Time elapsed: 44.069 sec <<< FAILURE! > java.lang.AssertionError: environment set for child is wrong > at org.junit.Assert.fail(Assert.java:93) > at org.junit.Assert.assertTrue(Assert.java:43) > at > org.apache.hadoop.streaming.TestStreamingTaskLog.runStreamJobAndValidateEnv(TestStreamingTaskLog.java:157) > at > org.apache.hadoop.streaming.TestStreamingTaskLog.testStreamingTaskLogWithHadoopCmd(TestStreamingTaskLog.java:107) > Results : > Failed tests: > > TestStreamingTaskLog.testStreamingTaskLogWithHadoopCmd:107->runStreamJobAndValidateEnv:157 > environment set for child is wrong -- This message was sent by Atlassian JIRA (v6.2#6252)
[jira] [Updated] (MAPREDUCE-5810) TestStreamingTaskLog#testStreamingTaskLogWithHadoopCmd is failing
[ https://issues.apache.org/jira/browse/MAPREDUCE-5810?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Akira AJISAKA updated MAPREDUCE-5810: - Attachment: MAPREDUCE-5810.patch Moved to MapReduce project, and attaching a patch to delete the test. > TestStreamingTaskLog#testStreamingTaskLogWithHadoopCmd is failing > - > > Key: MAPREDUCE-5810 > URL: https://issues.apache.org/jira/browse/MAPREDUCE-5810 > Project: Hadoop Map/Reduce > Issue Type: Bug >Affects Versions: 3.0.0, 2.4.0 >Reporter: Mit Desai >Assignee: Akira AJISAKA > Attachments: MAPREDUCE-5810.patch, log.txt > > > testStreamingTaskLogWithHadoopCmd(org.apache.hadoop.streaming.TestStreamingTaskLog) > Time elapsed: 44.069 sec <<< FAILURE! > java.lang.AssertionError: environment set for child is wrong > at org.junit.Assert.fail(Assert.java:93) > at org.junit.Assert.assertTrue(Assert.java:43) > at > org.apache.hadoop.streaming.TestStreamingTaskLog.runStreamJobAndValidateEnv(TestStreamingTaskLog.java:157) > at > org.apache.hadoop.streaming.TestStreamingTaskLog.testStreamingTaskLogWithHadoopCmd(TestStreamingTaskLog.java:107) > Results : > Failed tests: > > TestStreamingTaskLog.testStreamingTaskLogWithHadoopCmd:107->runStreamJobAndValidateEnv:157 > environment set for child is wrong -- This message was sent by Atlassian JIRA (v6.2#6252)
[jira] [Moved] (MAPREDUCE-5810) TestStreamingTaskLog#testStreamingTaskLogWithHadoopCmd is failing
[ https://issues.apache.org/jira/browse/MAPREDUCE-5810?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Akira AJISAKA moved HADOOP-10424 to MAPREDUCE-5810: --- Affects Version/s: (was: 2.4.0) (was: 3.0.0) 2.4.0 3.0.0 Key: MAPREDUCE-5810 (was: HADOOP-10424) Project: Hadoop Map/Reduce (was: Hadoop Common) > TestStreamingTaskLog#testStreamingTaskLogWithHadoopCmd is failing > - > > Key: MAPREDUCE-5810 > URL: https://issues.apache.org/jira/browse/MAPREDUCE-5810 > Project: Hadoop Map/Reduce > Issue Type: Bug >Affects Versions: 3.0.0, 2.4.0 >Reporter: Mit Desai >Assignee: Akira AJISAKA > Attachments: log.txt > > > testStreamingTaskLogWithHadoopCmd(org.apache.hadoop.streaming.TestStreamingTaskLog) > Time elapsed: 44.069 sec <<< FAILURE! > java.lang.AssertionError: environment set for child is wrong > at org.junit.Assert.fail(Assert.java:93) > at org.junit.Assert.assertTrue(Assert.java:43) > at > org.apache.hadoop.streaming.TestStreamingTaskLog.runStreamJobAndValidateEnv(TestStreamingTaskLog.java:157) > at > org.apache.hadoop.streaming.TestStreamingTaskLog.testStreamingTaskLogWithHadoopCmd(TestStreamingTaskLog.java:107) > Results : > Failed tests: > > TestStreamingTaskLog.testStreamingTaskLogWithHadoopCmd:107->runStreamJobAndValidateEnv:157 > environment set for child is wrong -- This message was sent by Atlassian JIRA (v6.2#6252)
[jira] [Resolved] (MAPREDUCE-4976) Use the new StringUtils methods added by HADOOP-9252
[ https://issues.apache.org/jira/browse/MAPREDUCE-4976?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Tsz Wo Nicholas Sze resolved MAPREDUCE-4976. Resolution: Not A Problem This is actually not a problem. > Use the new StringUtils methods added by HADOOP-9252 > > > Key: MAPREDUCE-4976 > URL: https://issues.apache.org/jira/browse/MAPREDUCE-4976 > Project: Hadoop Map/Reduce > Issue Type: Improvement >Reporter: Tsz Wo Nicholas Sze >Assignee: Tsz Wo Nicholas Sze >Priority: Minor > > HADOOP-9252 slightly changed the format of some StringUtils outputs. Some > methods were deprecated by HADOOP-9252. The use of them should be replaced > with the new methods. -- This message was sent by Atlassian JIRA (v6.2#6252)
[jira] [Commented] (MAPREDUCE-5799) add default value of MR_AM_ADMIN_USER_ENV
[
https://issues.apache.org/jira/browse/MAPREDUCE-5799?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13947317#comment-13947317
]
Mohammad Kamrul Islam commented on MAPREDUCE-5799:
--
+1
[~liangly] : Please assign the JIRA to yourself.
> add default value of MR_AM_ADMIN_USER_ENV
> -
>
> Key: MAPREDUCE-5799
> URL: https://issues.apache.org/jira/browse/MAPREDUCE-5799
> Project: Hadoop Map/Reduce
> Issue Type: Improvement
>Affects Versions: 2.3.0
>Reporter: Liyin Liang
>Priority: Minor
> Attachments: MAPREDUCE-5799.diff
>
>
> Submit a 1 map + 1 reduce sleep job with the following config:
> {code}
>
> mapreduce.map.output.compress
> true
>
>
> mapreduce.map.output.compress.codec
> org.apache.hadoop.io.compress.SnappyCodec
>
>
> mapreduce.job.ubertask.enable
> true
>
> {code}
> And the LinuxContainerExecutor is enable on NodeManager.
> This job will fail with the following error:
> {code}
> 2014-03-18 21:28:20,153 FATAL [uber-SubtaskRunner]
> org.apache.hadoop.mapred.LocalContainerLauncher: Error running local
> (uberized) 'child' : java.lang.UnsatisfiedLinkError:
> org.apache.hadoop.util.NativeCodeLoader.buildSupportsSnappy()Z
> at org.apache.hadoop.util.NativeCodeLoader.buildSupportsSnappy(Native
> Method)
> at
> org.apache.hadoop.io.compress.SnappyCodec.checkNativeCodeLoaded(SnappyCodec.java:63)
> at
> org.apache.hadoop.io.compress.SnappyCodec.getCompressorType(SnappyCodec.java:132)
> at
> org.apache.hadoop.io.compress.CodecPool.getCompressor(CodecPool.java:148)
> at
> org.apache.hadoop.io.compress.CodecPool.getCompressor(CodecPool.java:163)
> at org.apache.hadoop.mapred.IFile$Writer.(IFile.java:115)
> at
> org.apache.hadoop.mapred.MapTask$MapOutputBuffer.sortAndSpill(MapTask.java:1583)
> at
> org.apache.hadoop.mapred.MapTask$MapOutputBuffer.flush(MapTask.java:1462)
> at
> org.apache.hadoop.mapred.MapTask$NewOutputCollector.close(MapTask.java:700)
> at org.apache.hadoop.mapred.MapTask.closeQuietly(MapTask.java:1990)
> at org.apache.hadoop.mapred.MapTask.runNewMapper(MapTask.java:774)
> at org.apache.hadoop.mapred.MapTask.run(MapTask.java:340)
> at
> org.apache.hadoop.mapred.LocalContainerLauncher$SubtaskRunner.runSubtask(LocalContainerLauncher.java:317)
> at
> org.apache.hadoop.mapred.LocalContainerLauncher$SubtaskRunner.run(LocalContainerLauncher.java:232)
> at java.lang.Thread.run(Thread.java:662)
> {code}
> When create a ContainerLaunchContext for task in
> TaskAttemptImpl.createCommonContainerLaunchContext(), the
> DEFAULT_MAPRED_ADMIN_USER_ENV which is
> "LD_LIBRARY_PATH=$HADOOP_COMMON_HOME/lib/native" is added to the environment.
> Where when create a ContainerLaunchContext for mrappmaster in
> YARNRunner.createApplicationSubmissionContext(), there is no default
> environment. So the ubermode job fails to find native lib.
--
This message was sent by Atlassian JIRA
(v6.2#6252)
[jira] [Commented] (MAPREDUCE-5809) Enhance distcp to support preserving HDFS ACLs.
[ https://issues.apache.org/jira/browse/MAPREDUCE-5809?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13947257#comment-13947257 ] Suresh Srinivas commented on MAPREDUCE-5809: [~cnauroth], that sounds excellent. Thanks for answering. > Enhance distcp to support preserving HDFS ACLs. > --- > > Key: MAPREDUCE-5809 > URL: https://issues.apache.org/jira/browse/MAPREDUCE-5809 > Project: Hadoop Map/Reduce > Issue Type: Improvement > Components: distcp >Affects Versions: 2.4.0 >Reporter: Chris Nauroth >Assignee: Chris Nauroth > > This issue tracks enhancing distcp to add a new command-line argument for > preserving HDFS ACLs from the source at the copy destination. -- This message was sent by Atlassian JIRA (v6.2#6252)
[jira] [Commented] (MAPREDUCE-5809) Enhance distcp to support preserving HDFS ACLs.
[
https://issues.apache.org/jira/browse/MAPREDUCE-5809?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13947252#comment-13947252
]
Chris Nauroth commented on MAPREDUCE-5809:
--
[~sureshms], yes, this is a very important consideration. Here is what I had
in mind for the logic:
# distcp CLI accepts a new optional flag: -pa for "preserve ACLs". The
presence of -pa also implies the existing -pp flag, because ACLs are a
super-set of permissions.
# If preserving ACLs, then before submitting the job, distcp sends a canary
{{getAclStatus}} request for / on the source and target file systems. This
will detect ACL compatibility/support problems and fail fast before even
submitting the job. There are three specific sub-cases that this check catches:
## File system is HDFS < 2.4, so the getAclStatus RPC endpoint doesn't exist.
## File system is HDFS >= 2.4, but ACLs are not enabled.
## File system is a {{FileSystem}} subclass that doesn't override the ACL APIs.
({{UnsupportedOperationException}})
# Then, distcp map tasks call {{getAclStatus}} and {{setAcl}} instead of
{{setPermission}}.
If the operator wishes to copy files with ACLs to a target cluster and just
drop the ACLs, then they'd run without the -pa option, and distcp would
continue to work as it does today.
One additional thing I just learned while reading the code is that we support
passing the -p flag with no additional arguments, and this is assumed to
preserve replication, block size, user, group, permission and checksum type.
I'm planning on preserving this behavior. I don't think we can include -pa as
part of the defaults, because that could break existing deployments that are
running distcp -p if they start using ACLs on one cluster but not the other.
> Enhance distcp to support preserving HDFS ACLs.
> ---
>
> Key: MAPREDUCE-5809
> URL: https://issues.apache.org/jira/browse/MAPREDUCE-5809
> Project: Hadoop Map/Reduce
> Issue Type: Improvement
> Components: distcp
>Affects Versions: 2.4.0
>Reporter: Chris Nauroth
>Assignee: Chris Nauroth
>
> This issue tracks enhancing distcp to add a new command-line argument for
> preserving HDFS ACLs from the source at the copy destination.
--
This message was sent by Atlassian JIRA
(v6.2#6252)
[jira] [Commented] (MAPREDUCE-5809) Enhance distcp to support preserving HDFS ACLs.
[ https://issues.apache.org/jira/browse/MAPREDUCE-5809?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13947234#comment-13947234 ] Suresh Srinivas commented on MAPREDUCE-5809: [~cnauroth], I am sure you have thought about this already; we should take care of distcp from src cluster to destination, where src has ACL enabled and destination does not have it. > Enhance distcp to support preserving HDFS ACLs. > --- > > Key: MAPREDUCE-5809 > URL: https://issues.apache.org/jira/browse/MAPREDUCE-5809 > Project: Hadoop Map/Reduce > Issue Type: Improvement > Components: distcp >Affects Versions: 2.4.0 >Reporter: Chris Nauroth >Assignee: Chris Nauroth > > This issue tracks enhancing distcp to add a new command-line argument for > preserving HDFS ACLs from the source at the copy destination. -- This message was sent by Atlassian JIRA (v6.2#6252)
[jira] [Moved] (MAPREDUCE-5809) Enhance distcp to support preserving HDFS ACLs.
[ https://issues.apache.org/jira/browse/MAPREDUCE-5809?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Chris Nauroth moved HADOOP-10435 to MAPREDUCE-5809: --- Component/s: (was: tools/distcp) distcp Target Version/s: 3.0.0, 2.5.0 (was: 3.0.0, 2.5.0) Affects Version/s: (was: 2.4.0) 2.4.0 Key: MAPREDUCE-5809 (was: HADOOP-10435) Project: Hadoop Map/Reduce (was: Hadoop Common) > Enhance distcp to support preserving HDFS ACLs. > --- > > Key: MAPREDUCE-5809 > URL: https://issues.apache.org/jira/browse/MAPREDUCE-5809 > Project: Hadoop Map/Reduce > Issue Type: Improvement > Components: distcp >Affects Versions: 2.4.0 >Reporter: Chris Nauroth >Assignee: Chris Nauroth > > This issue tracks enhancing distcp to add a new command-line argument for > preserving HDFS ACLs from the source at the copy destination. -- This message was sent by Atlassian JIRA (v6.2#6252)
[jira] [Work started] (MAPREDUCE-5809) Enhance distcp to support preserving HDFS ACLs.
[ https://issues.apache.org/jira/browse/MAPREDUCE-5809?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Work on MAPREDUCE-5809 started by Chris Nauroth. > Enhance distcp to support preserving HDFS ACLs. > --- > > Key: MAPREDUCE-5809 > URL: https://issues.apache.org/jira/browse/MAPREDUCE-5809 > Project: Hadoop Map/Reduce > Issue Type: Improvement > Components: distcp >Affects Versions: 2.4.0 >Reporter: Chris Nauroth >Assignee: Chris Nauroth > > This issue tracks enhancing distcp to add a new command-line argument for > preserving HDFS ACLs from the source at the copy destination. -- This message was sent by Atlassian JIRA (v6.2#6252)
[jira] [Commented] (MAPREDUCE-5791) Shuffle phase is slow in Windows - FadviseFileRegion::transferTo does not read disks efficiently
[ https://issues.apache.org/jira/browse/MAPREDUCE-5791?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13946571#comment-13946571 ] Hudson commented on MAPREDUCE-5791: --- FAILURE: Integrated in Hadoop-Hdfs-trunk #1712 (See [https://builds.apache.org/job/Hadoop-Hdfs-trunk/1712/]) MAPREDUCE-5791. Shuffle phase is slow in Windows - FadviseFileRegion::transferTo does not read disks efficiently. Contributed by Nikola Vujic. (cnauroth: http://svn.apache.org/viewcvs.cgi/?root=Apache-SVN&view=rev&rev=1580994) * /hadoop/common/trunk/hadoop-mapreduce-project/CHANGES.txt * /hadoop/common/trunk/hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-core/src/main/resources/mapred-default.xml * /hadoop/common/trunk/hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-shuffle/src/main/java/org/apache/hadoop/mapred/FadvisedFileRegion.java * /hadoop/common/trunk/hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-shuffle/src/main/java/org/apache/hadoop/mapred/ShuffleHandler.java * /hadoop/common/trunk/hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-shuffle/src/test/java/org/apache/hadoop/mapred/TestFadvisedFileRegion.java > Shuffle phase is slow in Windows - FadviseFileRegion::transferTo does not > read disks efficiently > > > Key: MAPREDUCE-5791 > URL: https://issues.apache.org/jira/browse/MAPREDUCE-5791 > Project: Hadoop Map/Reduce > Issue Type: Bug > Components: client >Affects Versions: 3.0.0, 2.3.0 >Reporter: Nikola Vujic >Assignee: Nikola Vujic > Fix For: 3.0.0, 2.4.0 > > Attachments: MAPREDUCE-5791.patch, MAPREDUCE-5791.patch, > MAPREDUCE-5791.patch > > > transferTo method in org.apache.hadoop.mapred.FadvisedFileRegion is using > transferTo method from a FileChannel to transfer data from a disk to socket. > This is performing slow in Windows, slower than in Linux. The reason is that > transferTo method for the java.nio is issuing 32K IO requests all the time. > In Windows, these 32K transfers are not optimal and we don't get the best > performance form the underlying IO subsystem. In order to achieve better > performance when reading from the drives, we need to read data in bigger > chunks, 512K for example. -- This message was sent by Atlassian JIRA (v6.2#6252)
[jira] [Commented] (MAPREDUCE-5795) Job should be marked as Failed if it is recovered from commit.
[ https://issues.apache.org/jira/browse/MAPREDUCE-5795?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13946567#comment-13946567 ] Hudson commented on MAPREDUCE-5795: --- FAILURE: Integrated in Hadoop-Hdfs-trunk #1712 (See [https://builds.apache.org/job/Hadoop-Hdfs-trunk/1712/]) MAPREDUCE-5795. Fixed MRAppMaster to record the correct job-state after it recovers from a commit during a previous attempt. Contributed by Xuan Gong. (vinodkv: http://svn.apache.org/viewcvs.cgi/?root=Apache-SVN&view=rev&rev=1581180) * /hadoop/common/trunk/hadoop-mapreduce-project/CHANGES.txt * /hadoop/common/trunk/hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-app/src/main/java/org/apache/hadoop/mapreduce/jobhistory/JobHistoryEventHandler.java * /hadoop/common/trunk/hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-app/src/main/java/org/apache/hadoop/mapreduce/v2/app/MRAppMaster.java * /hadoop/common/trunk/hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-app/src/test/java/org/apache/hadoop/mapreduce/jobhistory/TestJobHistoryEventHandler.java * /hadoop/common/trunk/hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-app/src/test/java/org/apache/hadoop/mapreduce/v2/app/TestMRAppMaster.java * /hadoop/common/trunk/hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-core/src/main/java/org/apache/hadoop/mapreduce/jobhistory/AMStartedEvent.java > Job should be marked as Failed if it is recovered from commit. > -- > > Key: MAPREDUCE-5795 > URL: https://issues.apache.org/jira/browse/MAPREDUCE-5795 > Project: Hadoop Map/Reduce > Issue Type: Bug >Reporter: Yesha Vora >Assignee: Xuan Gong > Fix For: 2.4.0 > > Attachments: MAPREDUCE-5795.1.patch, MAPREDUCE-5795.2.patch > > > If Resource manager is restarted when a job is in commit state, The job is > not able to recovered after RM restart and it is marked as Killed. > The job status should be Failed instead killed. -- This message was sent by Atlassian JIRA (v6.2#6252)
[jira] [Commented] (MAPREDUCE-5791) Shuffle phase is slow in Windows - FadviseFileRegion::transferTo does not read disks efficiently
[ https://issues.apache.org/jira/browse/MAPREDUCE-5791?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13946543#comment-13946543 ] Hudson commented on MAPREDUCE-5791: --- FAILURE: Integrated in Hadoop-Mapreduce-trunk #1737 (See [https://builds.apache.org/job/Hadoop-Mapreduce-trunk/1737/]) MAPREDUCE-5791. Shuffle phase is slow in Windows - FadviseFileRegion::transferTo does not read disks efficiently. Contributed by Nikola Vujic. (cnauroth: http://svn.apache.org/viewcvs.cgi/?root=Apache-SVN&view=rev&rev=1580994) * /hadoop/common/trunk/hadoop-mapreduce-project/CHANGES.txt * /hadoop/common/trunk/hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-core/src/main/resources/mapred-default.xml * /hadoop/common/trunk/hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-shuffle/src/main/java/org/apache/hadoop/mapred/FadvisedFileRegion.java * /hadoop/common/trunk/hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-shuffle/src/main/java/org/apache/hadoop/mapred/ShuffleHandler.java * /hadoop/common/trunk/hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-shuffle/src/test/java/org/apache/hadoop/mapred/TestFadvisedFileRegion.java > Shuffle phase is slow in Windows - FadviseFileRegion::transferTo does not > read disks efficiently > > > Key: MAPREDUCE-5791 > URL: https://issues.apache.org/jira/browse/MAPREDUCE-5791 > Project: Hadoop Map/Reduce > Issue Type: Bug > Components: client >Affects Versions: 3.0.0, 2.3.0 >Reporter: Nikola Vujic >Assignee: Nikola Vujic > Fix For: 3.0.0, 2.4.0 > > Attachments: MAPREDUCE-5791.patch, MAPREDUCE-5791.patch, > MAPREDUCE-5791.patch > > > transferTo method in org.apache.hadoop.mapred.FadvisedFileRegion is using > transferTo method from a FileChannel to transfer data from a disk to socket. > This is performing slow in Windows, slower than in Linux. The reason is that > transferTo method for the java.nio is issuing 32K IO requests all the time. > In Windows, these 32K transfers are not optimal and we don't get the best > performance form the underlying IO subsystem. In order to achieve better > performance when reading from the drives, we need to read data in bigger > chunks, 512K for example. -- This message was sent by Atlassian JIRA (v6.2#6252)
[jira] [Commented] (MAPREDUCE-5795) Job should be marked as Failed if it is recovered from commit.
[ https://issues.apache.org/jira/browse/MAPREDUCE-5795?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13946539#comment-13946539 ] Hudson commented on MAPREDUCE-5795: --- FAILURE: Integrated in Hadoop-Mapreduce-trunk #1737 (See [https://builds.apache.org/job/Hadoop-Mapreduce-trunk/1737/]) MAPREDUCE-5795. Fixed MRAppMaster to record the correct job-state after it recovers from a commit during a previous attempt. Contributed by Xuan Gong. (vinodkv: http://svn.apache.org/viewcvs.cgi/?root=Apache-SVN&view=rev&rev=1581180) * /hadoop/common/trunk/hadoop-mapreduce-project/CHANGES.txt * /hadoop/common/trunk/hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-app/src/main/java/org/apache/hadoop/mapreduce/jobhistory/JobHistoryEventHandler.java * /hadoop/common/trunk/hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-app/src/main/java/org/apache/hadoop/mapreduce/v2/app/MRAppMaster.java * /hadoop/common/trunk/hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-app/src/test/java/org/apache/hadoop/mapreduce/jobhistory/TestJobHistoryEventHandler.java * /hadoop/common/trunk/hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-app/src/test/java/org/apache/hadoop/mapreduce/v2/app/TestMRAppMaster.java * /hadoop/common/trunk/hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-core/src/main/java/org/apache/hadoop/mapreduce/jobhistory/AMStartedEvent.java > Job should be marked as Failed if it is recovered from commit. > -- > > Key: MAPREDUCE-5795 > URL: https://issues.apache.org/jira/browse/MAPREDUCE-5795 > Project: Hadoop Map/Reduce > Issue Type: Bug >Reporter: Yesha Vora >Assignee: Xuan Gong > Fix For: 2.4.0 > > Attachments: MAPREDUCE-5795.1.patch, MAPREDUCE-5795.2.patch > > > If Resource manager is restarted when a job is in commit state, The job is > not able to recovered after RM restart and it is marked as Killed. > The job status should be Failed instead killed. -- This message was sent by Atlassian JIRA (v6.2#6252)
[jira] [Commented] (MAPREDUCE-5791) Shuffle phase is slow in Windows - FadviseFileRegion::transferTo does not read disks efficiently
[ https://issues.apache.org/jira/browse/MAPREDUCE-5791?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13946411#comment-13946411 ] Hudson commented on MAPREDUCE-5791: --- FAILURE: Integrated in Hadoop-Yarn-trunk #520 (See [https://builds.apache.org/job/Hadoop-Yarn-trunk/520/]) MAPREDUCE-5791. Shuffle phase is slow in Windows - FadviseFileRegion::transferTo does not read disks efficiently. Contributed by Nikola Vujic. (cnauroth: http://svn.apache.org/viewcvs.cgi/?root=Apache-SVN&view=rev&rev=1580994) * /hadoop/common/trunk/hadoop-mapreduce-project/CHANGES.txt * /hadoop/common/trunk/hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-core/src/main/resources/mapred-default.xml * /hadoop/common/trunk/hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-shuffle/src/main/java/org/apache/hadoop/mapred/FadvisedFileRegion.java * /hadoop/common/trunk/hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-shuffle/src/main/java/org/apache/hadoop/mapred/ShuffleHandler.java * /hadoop/common/trunk/hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-shuffle/src/test/java/org/apache/hadoop/mapred/TestFadvisedFileRegion.java > Shuffle phase is slow in Windows - FadviseFileRegion::transferTo does not > read disks efficiently > > > Key: MAPREDUCE-5791 > URL: https://issues.apache.org/jira/browse/MAPREDUCE-5791 > Project: Hadoop Map/Reduce > Issue Type: Bug > Components: client >Affects Versions: 3.0.0, 2.3.0 >Reporter: Nikola Vujic >Assignee: Nikola Vujic > Fix For: 3.0.0, 2.4.0 > > Attachments: MAPREDUCE-5791.patch, MAPREDUCE-5791.patch, > MAPREDUCE-5791.patch > > > transferTo method in org.apache.hadoop.mapred.FadvisedFileRegion is using > transferTo method from a FileChannel to transfer data from a disk to socket. > This is performing slow in Windows, slower than in Linux. The reason is that > transferTo method for the java.nio is issuing 32K IO requests all the time. > In Windows, these 32K transfers are not optimal and we don't get the best > performance form the underlying IO subsystem. In order to achieve better > performance when reading from the drives, we need to read data in bigger > chunks, 512K for example. -- This message was sent by Atlassian JIRA (v6.2#6252)
[jira] [Commented] (MAPREDUCE-5795) Job should be marked as Failed if it is recovered from commit.
[ https://issues.apache.org/jira/browse/MAPREDUCE-5795?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13946407#comment-13946407 ] Hudson commented on MAPREDUCE-5795: --- FAILURE: Integrated in Hadoop-Yarn-trunk #520 (See [https://builds.apache.org/job/Hadoop-Yarn-trunk/520/]) MAPREDUCE-5795. Fixed MRAppMaster to record the correct job-state after it recovers from a commit during a previous attempt. Contributed by Xuan Gong. (vinodkv: http://svn.apache.org/viewcvs.cgi/?root=Apache-SVN&view=rev&rev=1581180) * /hadoop/common/trunk/hadoop-mapreduce-project/CHANGES.txt * /hadoop/common/trunk/hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-app/src/main/java/org/apache/hadoop/mapreduce/jobhistory/JobHistoryEventHandler.java * /hadoop/common/trunk/hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-app/src/main/java/org/apache/hadoop/mapreduce/v2/app/MRAppMaster.java * /hadoop/common/trunk/hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-app/src/test/java/org/apache/hadoop/mapreduce/jobhistory/TestJobHistoryEventHandler.java * /hadoop/common/trunk/hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-app/src/test/java/org/apache/hadoop/mapreduce/v2/app/TestMRAppMaster.java * /hadoop/common/trunk/hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-core/src/main/java/org/apache/hadoop/mapreduce/jobhistory/AMStartedEvent.java > Job should be marked as Failed if it is recovered from commit. > -- > > Key: MAPREDUCE-5795 > URL: https://issues.apache.org/jira/browse/MAPREDUCE-5795 > Project: Hadoop Map/Reduce > Issue Type: Bug >Reporter: Yesha Vora >Assignee: Xuan Gong > Fix For: 2.4.0 > > Attachments: MAPREDUCE-5795.1.patch, MAPREDUCE-5795.2.patch > > > If Resource manager is restarted when a job is in commit state, The job is > not able to recovered after RM restart and it is marked as Killed. > The job status should be Failed instead killed. -- This message was sent by Atlassian JIRA (v6.2#6252)
