[jira] [Commented] (MAPREDUCE-5969) Private non-Archive Files' size add twice in Distributed Cache directory size calculation.
[
https://issues.apache.org/jira/browse/MAPREDUCE-5969?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14164773#comment-14164773
]
Hadoop QA commented on MAPREDUCE-5969:
--
{color:red}-1 overall{color}. Here are the results of testing the latest
attachment
http://issues.apache.org/jira/secure/attachment/12673831/MAPREDUCE-5969.branch1.1.patch
against trunk revision 2a51494.
{color:red}-1 patch{color}. The patch command could not apply the patch.
Console output:
https://builds.apache.org/job/PreCommit-MAPREDUCE-Build/4945//console
This message is automatically generated.
> Private non-Archive Files' size add twice in Distributed Cache directory size
> calculation.
> --
>
> Key: MAPREDUCE-5969
> URL: https://issues.apache.org/jira/browse/MAPREDUCE-5969
> Project: Hadoop Map/Reduce
> Issue Type: Bug
> Components: mrv1
>Reporter: zhihai xu
>Assignee: zhihai xu
> Attachments: MAPREDUCE-5969.branch1.1.patch,
> MAPREDUCE-5969.branch1.patch
>
>
> Private non-Archive Files' size add twice in Distributed Cache directory size
> calculation. Private non-Archive Files list is passed in by "-files" command
> line option. The Distributed Cache directory size is used to check whether
> the total cache files size exceed the cache size limitation, the default
> cache size limitation is 10G.
> I add log in addCacheInfoUpdate and setSize in
> TrackerDistributedCacheManager.java.
> I use the following command to test:
> hadoop jar ./wordcount.jar org.apache.hadoop.examples.WordCount -files
> hdfs://host:8022/tmp/zxu/WordCount.java,hdfs://host:8022/tmp/zxu/wordcount.jar
> /tmp/zxu/test_in/ /tmp/zxu/test_out
> to add two files into distributed cache:WordCount.java and wordcount.jar.
> WordCount.java file size is 2395 byes and wordcount.jar file size is 3865
> bytes. The total should be 6260.
> The log show these files size added twice:
> add one time before download to local node and add second time after download
> to local node, so total file number becomes 4 instead of 2:
> addCacheInfoUpdate size: 6260 num: 2 baseDir: /mapred/local
> addCacheInfoUpdate size: 8683 num: 3 baseDir: /mapred/local
> addCacheInfoUpdate size: 12588 num: 4 baseDir: /mapred/local
> In the code, for Private non-Archive File, the first time we add file size is
> at
> getLocalCache:
> {code}
> if (!isArchive) {
> //for private archives, the lengths come over RPC from the
> //JobLocalizer since the JobLocalizer is the one who expands
> //archives and gets the total length
> lcacheStatus.size = fileStatus.getLen();
> LOG.info("getLocalCache:" + localizedPath + " size = "
> + lcacheStatus.size);
> // Increase the size and sub directory count of the cache
> // from baseDirSize and baseDirNumberSubDir.
> baseDirManager.addCacheInfoUpdate(lcacheStatus);
> }
> {code}
> The second time we add file size is at
> setSize:
> {code}
> synchronized (status) {
> status.size = size;
> baseDirManager.addCacheInfoUpdate(status);
> }
> {code}
> The fix is not to add the file size for for Private non-Archive File after
> download(downloadCacheObject).
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Commented] (MAPREDUCE-5969) Private non-Archive Files' size add twice in Distributed Cache directory size calculation.
[
https://issues.apache.org/jira/browse/MAPREDUCE-5969?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14164772#comment-14164772
]
zhihai xu commented on MAPREDUCE-5969:
--
update the patch MAPREDUCE-5969.branch1.1.patch based on the latest branch-1
code.
> Private non-Archive Files' size add twice in Distributed Cache directory size
> calculation.
> --
>
> Key: MAPREDUCE-5969
> URL: https://issues.apache.org/jira/browse/MAPREDUCE-5969
> Project: Hadoop Map/Reduce
> Issue Type: Bug
> Components: mrv1
>Reporter: zhihai xu
>Assignee: zhihai xu
> Attachments: MAPREDUCE-5969.branch1.1.patch,
> MAPREDUCE-5969.branch1.patch
>
>
> Private non-Archive Files' size add twice in Distributed Cache directory size
> calculation. Private non-Archive Files list is passed in by "-files" command
> line option. The Distributed Cache directory size is used to check whether
> the total cache files size exceed the cache size limitation, the default
> cache size limitation is 10G.
> I add log in addCacheInfoUpdate and setSize in
> TrackerDistributedCacheManager.java.
> I use the following command to test:
> hadoop jar ./wordcount.jar org.apache.hadoop.examples.WordCount -files
> hdfs://host:8022/tmp/zxu/WordCount.java,hdfs://host:8022/tmp/zxu/wordcount.jar
> /tmp/zxu/test_in/ /tmp/zxu/test_out
> to add two files into distributed cache:WordCount.java and wordcount.jar.
> WordCount.java file size is 2395 byes and wordcount.jar file size is 3865
> bytes. The total should be 6260.
> The log show these files size added twice:
> add one time before download to local node and add second time after download
> to local node, so total file number becomes 4 instead of 2:
> addCacheInfoUpdate size: 6260 num: 2 baseDir: /mapred/local
> addCacheInfoUpdate size: 8683 num: 3 baseDir: /mapred/local
> addCacheInfoUpdate size: 12588 num: 4 baseDir: /mapred/local
> In the code, for Private non-Archive File, the first time we add file size is
> at
> getLocalCache:
> {code}
> if (!isArchive) {
> //for private archives, the lengths come over RPC from the
> //JobLocalizer since the JobLocalizer is the one who expands
> //archives and gets the total length
> lcacheStatus.size = fileStatus.getLen();
> LOG.info("getLocalCache:" + localizedPath + " size = "
> + lcacheStatus.size);
> // Increase the size and sub directory count of the cache
> // from baseDirSize and baseDirNumberSubDir.
> baseDirManager.addCacheInfoUpdate(lcacheStatus);
> }
> {code}
> The second time we add file size is at
> setSize:
> {code}
> synchronized (status) {
> status.size = size;
> baseDirManager.addCacheInfoUpdate(status);
> }
> {code}
> The fix is not to add the file size for for Private non-Archive File after
> download(downloadCacheObject).
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Updated] (MAPREDUCE-5969) Private non-Archive Files' size add twice in Distributed Cache directory size calculation.
[
https://issues.apache.org/jira/browse/MAPREDUCE-5969?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
zhihai xu updated MAPREDUCE-5969:
-
Attachment: MAPREDUCE-5969.branch1.1.patch
> Private non-Archive Files' size add twice in Distributed Cache directory size
> calculation.
> --
>
> Key: MAPREDUCE-5969
> URL: https://issues.apache.org/jira/browse/MAPREDUCE-5969
> Project: Hadoop Map/Reduce
> Issue Type: Bug
> Components: mrv1
>Reporter: zhihai xu
>Assignee: zhihai xu
> Attachments: MAPREDUCE-5969.branch1.1.patch,
> MAPREDUCE-5969.branch1.patch
>
>
> Private non-Archive Files' size add twice in Distributed Cache directory size
> calculation. Private non-Archive Files list is passed in by "-files" command
> line option. The Distributed Cache directory size is used to check whether
> the total cache files size exceed the cache size limitation, the default
> cache size limitation is 10G.
> I add log in addCacheInfoUpdate and setSize in
> TrackerDistributedCacheManager.java.
> I use the following command to test:
> hadoop jar ./wordcount.jar org.apache.hadoop.examples.WordCount -files
> hdfs://host:8022/tmp/zxu/WordCount.java,hdfs://host:8022/tmp/zxu/wordcount.jar
> /tmp/zxu/test_in/ /tmp/zxu/test_out
> to add two files into distributed cache:WordCount.java and wordcount.jar.
> WordCount.java file size is 2395 byes and wordcount.jar file size is 3865
> bytes. The total should be 6260.
> The log show these files size added twice:
> add one time before download to local node and add second time after download
> to local node, so total file number becomes 4 instead of 2:
> addCacheInfoUpdate size: 6260 num: 2 baseDir: /mapred/local
> addCacheInfoUpdate size: 8683 num: 3 baseDir: /mapred/local
> addCacheInfoUpdate size: 12588 num: 4 baseDir: /mapred/local
> In the code, for Private non-Archive File, the first time we add file size is
> at
> getLocalCache:
> {code}
> if (!isArchive) {
> //for private archives, the lengths come over RPC from the
> //JobLocalizer since the JobLocalizer is the one who expands
> //archives and gets the total length
> lcacheStatus.size = fileStatus.getLen();
> LOG.info("getLocalCache:" + localizedPath + " size = "
> + lcacheStatus.size);
> // Increase the size and sub directory count of the cache
> // from baseDirSize and baseDirNumberSubDir.
> baseDirManager.addCacheInfoUpdate(lcacheStatus);
> }
> {code}
> The second time we add file size is at
> setSize:
> {code}
> synchronized (status) {
> status.size = size;
> baseDirManager.addCacheInfoUpdate(status);
> }
> {code}
> The fix is not to add the file size for for Private non-Archive File after
> download(downloadCacheObject).
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Commented] (MAPREDUCE-5933) Enable MR AM to post history events to the timeline server
[
https://issues.apache.org/jira/browse/MAPREDUCE-5933?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14164532#comment-14164532
]
Hadoop QA commented on MAPREDUCE-5933:
--
{color:red}-1 overall{color}. Here are the results of testing the latest
attachment
http://issues.apache.org/jira/secure/attachment/12673741/MAPREDUCE-5933.patch
against trunk revision a169051.
{color:green}+1 @author{color}. The patch does not contain any @author
tags.
{color:green}+1 tests included{color}. The patch appears to include 3 new
or modified test files.
{color:green}+1 javac{color}. The applied patch does not increase the
total number of javac compiler warnings.
{color:green}+1 javadoc{color}. There were no new javadoc warning messages.
{color:green}+1 eclipse:eclipse{color}. The patch built with
eclipse:eclipse.
{color:green}+1 findbugs{color}. The patch does not introduce any new
Findbugs (version 2.0.3) warnings.
{color:red}-1 release audit{color}. The applied patch generated 1
release audit warnings.
{color:red}-1 core tests{color}. The patch failed these unit tests in
hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-app
hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-jobclient:
org.apache.hadoop.mapred.pipes.TestPipeApplication
{color:green}+1 contrib tests{color}. The patch passed contrib unit tests.
Test results:
https://builds.apache.org/job/PreCommit-MAPREDUCE-Build/4944//testReport/
Release audit warnings:
https://builds.apache.org/job/PreCommit-MAPREDUCE-Build/4944//artifact/patchprocess/patchReleaseAuditProblems.txt
Console output:
https://builds.apache.org/job/PreCommit-MAPREDUCE-Build/4944//console
This message is automatically generated.
> Enable MR AM to post history events to the timeline server
> --
>
> Key: MAPREDUCE-5933
> URL: https://issues.apache.org/jira/browse/MAPREDUCE-5933
> Project: Hadoop Map/Reduce
> Issue Type: Sub-task
> Components: mr-am
>Reporter: Zhijie Shen
>Assignee: Robert Kanter
> Attachments: MAPREDUCE-5933.patch, MAPREDUCE-5933.patch,
> MAPREDUCE-5933.patch, MAPREDUCE-5933.patch, MAPREDUCE-5933.patch,
> MAPREDUCE-5933.patch, mr_timelineserver_response.txt
>
>
> Nowadays, MR AM collects the history events and writes it to HDFS for JHS to
> source. With the timeline server, MR AM can put these events there.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Resolved] (MAPREDUCE-5718) MR AM should tolerate RM restart/failover during commit
[ https://issues.apache.org/jira/browse/MAPREDUCE-5718?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Karthik Kambatla resolved MAPREDUCE-5718. - Resolution: Won't Fix Target Version/s: (was: ) With work-preserving RM/NM restart, one can run into this only when the AM dies or the node running the AM dies. This is no longer a regression when moving from MR1 to MR2. Closing this as "Won't Fix" for now. Please re-open if needed. > MR AM should tolerate RM restart/failover during commit > --- > > Key: MAPREDUCE-5718 > URL: https://issues.apache.org/jira/browse/MAPREDUCE-5718 > Project: Hadoop Map/Reduce > Issue Type: Bug > Components: mr-am >Affects Versions: 2.3.0 >Reporter: Karthik Kambatla >Assignee: Karthik Kambatla > Labels: ha > Attachments: mr-5718-0.patch > > > While testing RM HA, we ran into this issue where if the RM fails over while > an MR AM is in the middle of a commit, the subsequent AM gets spawned but > dies with a diagnostic message - "We crashed durring a commit". -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (MAPREDUCE-6052) Support overriding log4j.properties per job
[
https://issues.apache.org/jira/browse/MAPREDUCE-6052?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14164497#comment-14164497
]
Vinod Kumar Vavilapalli commented on MAPREDUCE-6052:
bq. No matter which NM the containers will be launched, they will always get
the same container-log4j.properties. So, if we use customized log4j properties,
do we need to distribute this file to all the NMs (just like the default
container-log4j.properties) ?
Was asked to comment. This makes sense. Let's just make it a user-configuration
of a local log4j file that we accept and add it automatically to the
distributed-cache.
> Support overriding log4j.properties per job
> ---
>
> Key: MAPREDUCE-6052
> URL: https://issues.apache.org/jira/browse/MAPREDUCE-6052
> Project: Hadoop Map/Reduce
> Issue Type: Bug
>Affects Versions: 2.5.0
>Reporter: Junping Du
>Assignee: Junping Du
> Attachments: MAPREDUCE-6052.patch
>
>
> For current MR application, the "log4j.configuration" is hard coded to
> container-log4j.properties within each node. We still need flexibility to
> override it per job like what we do in MRV1.
> {code}
> public static void addLog4jSystemProperties(
> String logLevel, long logSize, int numBackups, List vargs) {
> vargs.add("-Dlog4j.configuration=container-log4j.properties");
> {code}
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Updated] (MAPREDUCE-5933) Enable MR AM to post history events to the timeline server
[ https://issues.apache.org/jira/browse/MAPREDUCE-5933?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Robert Kanter updated MAPREDUCE-5933: - Attachment: MAPREDUCE-5933.patch Updated patch addresses the latest comments. > Enable MR AM to post history events to the timeline server > -- > > Key: MAPREDUCE-5933 > URL: https://issues.apache.org/jira/browse/MAPREDUCE-5933 > Project: Hadoop Map/Reduce > Issue Type: Sub-task > Components: mr-am >Reporter: Zhijie Shen >Assignee: Robert Kanter > Attachments: MAPREDUCE-5933.patch, MAPREDUCE-5933.patch, > MAPREDUCE-5933.patch, MAPREDUCE-5933.patch, MAPREDUCE-5933.patch, > MAPREDUCE-5933.patch, mr_timelineserver_response.txt > > > Nowadays, MR AM collects the history events and writes it to HDFS for JHS to > source. With the timeline server, MR AM can put these events there. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (MAPREDUCE-6125) TestContainerLauncherImpl sometimes fails
[
https://issues.apache.org/jira/browse/MAPREDUCE-6125?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14164213#comment-14164213
]
Hadoop QA commented on MAPREDUCE-6125:
--
{color:red}-1 overall{color}. Here are the results of testing the latest
attachment
http://issues.apache.org/jira/secure/attachment/12673604/YARN-2633.patch
against trunk revision 853cb70.
{color:green}+1 @author{color}. The patch does not contain any @author
tags.
{color:green}+1 tests included{color}. The patch appears to include 1 new
or modified test files.
{color:green}+1 javac{color}. The applied patch does not increase the
total number of javac compiler warnings.
{color:green}+1 javadoc{color}. There were no new javadoc warning messages.
{color:green}+1 eclipse:eclipse{color}. The patch built with
eclipse:eclipse.
{color:green}+1 findbugs{color}. The patch does not introduce any new
Findbugs (version 2.0.3) warnings.
{color:red}-1 release audit{color}. The applied patch generated 1
release audit warnings.
{color:red}-1 core tests{color}. The patch failed these unit tests in
hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-app:
org.apache.hadoop.mapreduce.v2.app.TestRecovery
org.apache.hadoop.mapreduce.v2.app.job.impl.TestTaskAttempt
org.apache.hadoop.mapreduce.v2.app.TestMRAppMaster
{color:green}+1 contrib tests{color}. The patch passed contrib unit tests.
Test results:
https://builds.apache.org/job/PreCommit-MAPREDUCE-Build/4943//testReport/
Release audit warnings:
https://builds.apache.org/job/PreCommit-MAPREDUCE-Build/4943//artifact/patchprocess/patchReleaseAuditProblems.txt
Console output:
https://builds.apache.org/job/PreCommit-MAPREDUCE-Build/4943//console
This message is automatically generated.
> TestContainerLauncherImpl sometimes fails
> -
>
> Key: MAPREDUCE-6125
> URL: https://issues.apache.org/jira/browse/MAPREDUCE-6125
> Project: Hadoop Map/Reduce
> Issue Type: Bug
> Components: test
>Reporter: Mit Desai
>Assignee: Mit Desai
> Attachments: YARN-2633.patch, YARN-2633.patch, YARN-2633.patch
>
>
> {noformat}
> org.apache.hadoop.yarn.exceptions.YarnRuntimeException:
> java.lang.NoSuchMethodException:
> org.apache.hadoop.yarn.api.ContainerManagementProtocol$$EnhancerByMockitoWithCGLIB$$25708415.close()
> at java.lang.Class.getMethod(Class.java:1665)
> at
> org.apache.hadoop.yarn.factories.impl.pb.RpcClientFactoryPBImpl.stopClient(RpcClientFactoryPBImpl.java:90)
> at
> org.apache.hadoop.yarn.ipc.HadoopYarnProtoRPC.stopProxy(HadoopYarnProtoRPC.java:54)
> at
> org.apache.hadoop.yarn.client.api.impl.ContainerManagementProtocolProxy.mayBeCloseProxy(ContainerManagementProtocolProxy.java:79)
> at
> org.apache.hadoop.mapreduce.v2.app.launcher.ContainerLauncherImpl$Container.kill(ContainerLauncherImpl.java:225)
> at
> org.apache.hadoop.mapreduce.v2.app.launcher.ContainerLauncherImpl.shutdownAllContainers(ContainerLauncherImpl.java:320)
> at
> org.apache.hadoop.mapreduce.v2.app.launcher.ContainerLauncherImpl.serviceStop(ContainerLauncherImpl.java:331)
> at
> org.apache.hadoop.service.AbstractService.stop(AbstractService.java:221)
> at
> org.apache.hadoop.mapreduce.v2.app.launcher.TestContainerLauncherImpl.testMyShutdown(TestContainerLauncherImpl.java:315)
> {noformat}
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Updated] (MAPREDUCE-6125) TestContainerLauncherImpl sometimes fails
[
https://issues.apache.org/jira/browse/MAPREDUCE-6125?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Jason Lowe updated MAPREDUCE-6125:
--
Component/s: test
Thanks for updating the patch, Mit. Comments:
There should be a comment with ContainerManagementProtocolWithClose explaining
why it's necessary, and ContainerManagementProtocolClient might be a better
name.
ContainerManagementProtocolWithClose should be an interface rather than an
abstract class. Removes the need to write all those unused stubs. We can let
the concrete classes provide the close method.
Override annotations were removed and should be restored.
Adding Public and Stable annotations to the mock class for test is not
necessary or really applicable to a unit test
ContainerManagementProtocolPB and ContainerManagementProtocolPBClientImpl
imports were added but are unused.
> TestContainerLauncherImpl sometimes fails
> -
>
> Key: MAPREDUCE-6125
> URL: https://issues.apache.org/jira/browse/MAPREDUCE-6125
> Project: Hadoop Map/Reduce
> Issue Type: Bug
> Components: test
>Reporter: Mit Desai
>Assignee: Mit Desai
> Attachments: YARN-2633.patch, YARN-2633.patch, YARN-2633.patch
>
>
> {noformat}
> org.apache.hadoop.yarn.exceptions.YarnRuntimeException:
> java.lang.NoSuchMethodException:
> org.apache.hadoop.yarn.api.ContainerManagementProtocol$$EnhancerByMockitoWithCGLIB$$25708415.close()
> at java.lang.Class.getMethod(Class.java:1665)
> at
> org.apache.hadoop.yarn.factories.impl.pb.RpcClientFactoryPBImpl.stopClient(RpcClientFactoryPBImpl.java:90)
> at
> org.apache.hadoop.yarn.ipc.HadoopYarnProtoRPC.stopProxy(HadoopYarnProtoRPC.java:54)
> at
> org.apache.hadoop.yarn.client.api.impl.ContainerManagementProtocolProxy.mayBeCloseProxy(ContainerManagementProtocolProxy.java:79)
> at
> org.apache.hadoop.mapreduce.v2.app.launcher.ContainerLauncherImpl$Container.kill(ContainerLauncherImpl.java:225)
> at
> org.apache.hadoop.mapreduce.v2.app.launcher.ContainerLauncherImpl.shutdownAllContainers(ContainerLauncherImpl.java:320)
> at
> org.apache.hadoop.mapreduce.v2.app.launcher.ContainerLauncherImpl.serviceStop(ContainerLauncherImpl.java:331)
> at
> org.apache.hadoop.service.AbstractService.stop(AbstractService.java:221)
> at
> org.apache.hadoop.mapreduce.v2.app.launcher.TestContainerLauncherImpl.testMyShutdown(TestContainerLauncherImpl.java:315)
> {noformat}
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Moved] (MAPREDUCE-6125) TestContainerLauncherImpl sometimes fails
[
https://issues.apache.org/jira/browse/MAPREDUCE-6125?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Jason Lowe moved YARN-2633 to MAPREDUCE-6125:
-
Target Version/s: 2.6.0 (was: 2.6.0)
Key: MAPREDUCE-6125 (was: YARN-2633)
Project: Hadoop Map/Reduce (was: Hadoop YARN)
> TestContainerLauncherImpl sometimes fails
> -
>
> Key: MAPREDUCE-6125
> URL: https://issues.apache.org/jira/browse/MAPREDUCE-6125
> Project: Hadoop Map/Reduce
> Issue Type: Bug
>Reporter: Mit Desai
>Assignee: Mit Desai
> Attachments: YARN-2633.patch, YARN-2633.patch, YARN-2633.patch
>
>
> {noformat}
> org.apache.hadoop.yarn.exceptions.YarnRuntimeException:
> java.lang.NoSuchMethodException:
> org.apache.hadoop.yarn.api.ContainerManagementProtocol$$EnhancerByMockitoWithCGLIB$$25708415.close()
> at java.lang.Class.getMethod(Class.java:1665)
> at
> org.apache.hadoop.yarn.factories.impl.pb.RpcClientFactoryPBImpl.stopClient(RpcClientFactoryPBImpl.java:90)
> at
> org.apache.hadoop.yarn.ipc.HadoopYarnProtoRPC.stopProxy(HadoopYarnProtoRPC.java:54)
> at
> org.apache.hadoop.yarn.client.api.impl.ContainerManagementProtocolProxy.mayBeCloseProxy(ContainerManagementProtocolProxy.java:79)
> at
> org.apache.hadoop.mapreduce.v2.app.launcher.ContainerLauncherImpl$Container.kill(ContainerLauncherImpl.java:225)
> at
> org.apache.hadoop.mapreduce.v2.app.launcher.ContainerLauncherImpl.shutdownAllContainers(ContainerLauncherImpl.java:320)
> at
> org.apache.hadoop.mapreduce.v2.app.launcher.ContainerLauncherImpl.serviceStop(ContainerLauncherImpl.java:331)
> at
> org.apache.hadoop.service.AbstractService.stop(AbstractService.java:221)
> at
> org.apache.hadoop.mapreduce.v2.app.launcher.TestContainerLauncherImpl.testMyShutdown(TestContainerLauncherImpl.java:315)
> {noformat}
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Commented] (MAPREDUCE-5873) Measure bw of a single copy call and display the correct aggregated bw
[ https://issues.apache.org/jira/browse/MAPREDUCE-5873?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14164119#comment-14164119 ] Siqi Li commented on MAPREDUCE-5873: [~jlowe] Hi Jason, I have update the patch, can you take a look at it and see if you can commit it ? thanks > Measure bw of a single copy call and display the correct aggregated bw > -- > > Key: MAPREDUCE-5873 > URL: https://issues.apache.org/jira/browse/MAPREDUCE-5873 > Project: Hadoop Map/Reduce > Issue Type: Bug >Affects Versions: 2.3.0 >Reporter: Siqi Li >Assignee: Siqi Li > Attachments: MAPREDUCE-5873.v1.patch, MAPREDUCE-5873.v2.patch, > MAPREDUCE-5873.v3.patch, MAPREDUCE-5873.v4.patch, MAPREDUCE-5873.v5.patch, > MAPREDUCE-5873.v6.patch > > > Currently ShuffleScheduler in ReduceTask JVM status displays bandwidth. Its > definition however is confusing because it captures the time where there is > no copying because there is a pause between when new wave of map outputs is > available. > current bw is definded as (bytes copied so far) / (total time in the copy > phase so far) > It would be more useful > 1) to measure bandwidth of a single copy call. > 2) display aggregated bw as long as there is at least one fetcher is in the > copy call. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (MAPREDUCE-5873) Measure bw of a single copy call and display the correct aggregated bw
[
https://issues.apache.org/jira/browse/MAPREDUCE-5873?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14164110#comment-14164110
]
Hadoop QA commented on MAPREDUCE-5873:
--
{color:green}+1 overall{color}. Here are the results of testing the latest
attachment
http://issues.apache.org/jira/secure/attachment/12673678/MAPREDUCE-5873.v6.patch
against trunk revision e16e25a.
{color:green}+1 @author{color}. The patch does not contain any @author
tags.
{color:green}+1 tests included{color}. The patch appears to include 1 new
or modified test files.
{color:green}+1 javac{color}. The applied patch does not increase the
total number of javac compiler warnings.
{color:green}+1 javadoc{color}. There were no new javadoc warning messages.
{color:green}+1 eclipse:eclipse{color}. The patch built with
eclipse:eclipse.
{color:green}+1 findbugs{color}. The patch does not introduce any new
Findbugs (version 2.0.3) warnings.
{color:green}+1 release audit{color}. The applied patch does not increase
the total number of release audit warnings.
{color:green}+1 core tests{color}. The patch passed unit tests in
hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-core.
{color:green}+1 contrib tests{color}. The patch passed contrib unit tests.
Test results:
https://builds.apache.org/job/PreCommit-MAPREDUCE-Build/4942//testReport/
Console output:
https://builds.apache.org/job/PreCommit-MAPREDUCE-Build/4942//console
This message is automatically generated.
> Measure bw of a single copy call and display the correct aggregated bw
> --
>
> Key: MAPREDUCE-5873
> URL: https://issues.apache.org/jira/browse/MAPREDUCE-5873
> Project: Hadoop Map/Reduce
> Issue Type: Bug
>Affects Versions: 2.3.0
>Reporter: Siqi Li
>Assignee: Siqi Li
> Attachments: MAPREDUCE-5873.v1.patch, MAPREDUCE-5873.v2.patch,
> MAPREDUCE-5873.v3.patch, MAPREDUCE-5873.v4.patch, MAPREDUCE-5873.v5.patch,
> MAPREDUCE-5873.v6.patch
>
>
> Currently ShuffleScheduler in ReduceTask JVM status displays bandwidth. Its
> definition however is confusing because it captures the time where there is
> no copying because there is a pause between when new wave of map outputs is
> available.
> current bw is definded as (bytes copied so far) / (total time in the copy
> phase so far)
> It would be more useful
> 1) to measure bandwidth of a single copy call.
> 2) display aggregated bw as long as there is at least one fetcher is in the
> copy call.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Updated] (MAPREDUCE-5873) Measure bw of a single copy call and display the correct aggregated bw
[ https://issues.apache.org/jira/browse/MAPREDUCE-5873?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Siqi Li updated MAPREDUCE-5873: --- Status: Patch Available (was: Open) > Measure bw of a single copy call and display the correct aggregated bw > -- > > Key: MAPREDUCE-5873 > URL: https://issues.apache.org/jira/browse/MAPREDUCE-5873 > Project: Hadoop Map/Reduce > Issue Type: Bug >Affects Versions: 2.3.0 >Reporter: Siqi Li >Assignee: Siqi Li > Attachments: MAPREDUCE-5873.v1.patch, MAPREDUCE-5873.v2.patch, > MAPREDUCE-5873.v3.patch, MAPREDUCE-5873.v4.patch, MAPREDUCE-5873.v5.patch, > MAPREDUCE-5873.v6.patch > > > Currently ShuffleScheduler in ReduceTask JVM status displays bandwidth. Its > definition however is confusing because it captures the time where there is > no copying because there is a pause between when new wave of map outputs is > available. > current bw is definded as (bytes copied so far) / (total time in the copy > phase so far) > It would be more useful > 1) to measure bandwidth of a single copy call. > 2) display aggregated bw as long as there is at least one fetcher is in the > copy call. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Updated] (MAPREDUCE-5873) Measure bw of a single copy call and display the correct aggregated bw
[ https://issues.apache.org/jira/browse/MAPREDUCE-5873?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Siqi Li updated MAPREDUCE-5873: --- Attachment: MAPREDUCE-5873.v6.patch > Measure bw of a single copy call and display the correct aggregated bw > -- > > Key: MAPREDUCE-5873 > URL: https://issues.apache.org/jira/browse/MAPREDUCE-5873 > Project: Hadoop Map/Reduce > Issue Type: Bug >Affects Versions: 2.3.0 >Reporter: Siqi Li >Assignee: Siqi Li > Attachments: MAPREDUCE-5873.v1.patch, MAPREDUCE-5873.v2.patch, > MAPREDUCE-5873.v3.patch, MAPREDUCE-5873.v4.patch, MAPREDUCE-5873.v5.patch, > MAPREDUCE-5873.v6.patch > > > Currently ShuffleScheduler in ReduceTask JVM status displays bandwidth. Its > definition however is confusing because it captures the time where there is > no copying because there is a pause between when new wave of map outputs is > available. > current bw is definded as (bytes copied so far) / (total time in the copy > phase so far) > It would be more useful > 1) to measure bandwidth of a single copy call. > 2) display aggregated bw as long as there is at least one fetcher is in the > copy call. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Updated] (MAPREDUCE-5873) Measure bw of a single copy call and display the correct aggregated bw
[ https://issues.apache.org/jira/browse/MAPREDUCE-5873?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Siqi Li updated MAPREDUCE-5873: --- Status: Open (was: Patch Available) > Measure bw of a single copy call and display the correct aggregated bw > -- > > Key: MAPREDUCE-5873 > URL: https://issues.apache.org/jira/browse/MAPREDUCE-5873 > Project: Hadoop Map/Reduce > Issue Type: Bug >Affects Versions: 2.3.0 >Reporter: Siqi Li >Assignee: Siqi Li > Attachments: MAPREDUCE-5873.v1.patch, MAPREDUCE-5873.v2.patch, > MAPREDUCE-5873.v3.patch, MAPREDUCE-5873.v4.patch, MAPREDUCE-5873.v5.patch, > MAPREDUCE-5873.v6.patch > > > Currently ShuffleScheduler in ReduceTask JVM status displays bandwidth. Its > definition however is confusing because it captures the time where there is > no copying because there is a pause between when new wave of map outputs is > available. > current bw is definded as (bytes copied so far) / (total time in the copy > phase so far) > It would be more useful > 1) to measure bandwidth of a single copy call. > 2) display aggregated bw as long as there is at least one fetcher is in the > copy call. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (MAPREDUCE-5873) Measure bw of a single copy call and display the correct aggregated bw
[
https://issues.apache.org/jira/browse/MAPREDUCE-5873?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14164012#comment-14164012
]
Hadoop QA commented on MAPREDUCE-5873:
--
{color:red}-1 overall{color}. Here are the results of testing the latest
attachment
http://issues.apache.org/jira/secure/attachment/12673483/MAPREDUCE-5873.v5.patch
against trunk revision e16e25a.
{color:green}+1 @author{color}. The patch does not contain any @author
tags.
{color:green}+1 tests included{color}. The patch appears to include 1 new
or modified test files.
{color:green}+1 javac{color}. The applied patch does not increase the
total number of javac compiler warnings.
{color:green}+1 javadoc{color}. There were no new javadoc warning messages.
{color:green}+1 eclipse:eclipse{color}. The patch built with
eclipse:eclipse.
{color:red}-1 findbugs{color}. The patch appears to introduce 1 new
Findbugs (version 2.0.3) warnings.
{color:green}+1 release audit{color}. The applied patch does not increase
the total number of release audit warnings.
{color:green}+1 core tests{color}. The patch passed unit tests in
hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-core.
{color:green}+1 contrib tests{color}. The patch passed contrib unit tests.
Test results:
https://builds.apache.org/job/PreCommit-MAPREDUCE-Build/4941//testReport/
Findbugs warnings:
https://builds.apache.org/job/PreCommit-MAPREDUCE-Build/4941//artifact/patchprocess/newPatchFindbugsWarningshadoop-mapreduce-client-core.html
Console output:
https://builds.apache.org/job/PreCommit-MAPREDUCE-Build/4941//console
This message is automatically generated.
> Measure bw of a single copy call and display the correct aggregated bw
> --
>
> Key: MAPREDUCE-5873
> URL: https://issues.apache.org/jira/browse/MAPREDUCE-5873
> Project: Hadoop Map/Reduce
> Issue Type: Bug
>Affects Versions: 2.3.0
>Reporter: Siqi Li
>Assignee: Siqi Li
> Attachments: MAPREDUCE-5873.v1.patch, MAPREDUCE-5873.v2.patch,
> MAPREDUCE-5873.v3.patch, MAPREDUCE-5873.v4.patch, MAPREDUCE-5873.v5.patch
>
>
> Currently ShuffleScheduler in ReduceTask JVM status displays bandwidth. Its
> definition however is confusing because it captures the time where there is
> no copying because there is a pause between when new wave of map outputs is
> available.
> current bw is definded as (bytes copied so far) / (total time in the copy
> phase so far)
> It would be more useful
> 1) to measure bandwidth of a single copy call.
> 2) display aggregated bw as long as there is at least one fetcher is in the
> copy call.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Updated] (MAPREDUCE-5873) Measure bw of a single copy call and display the correct aggregated bw
[ https://issues.apache.org/jira/browse/MAPREDUCE-5873?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Siqi Li updated MAPREDUCE-5873: --- Status: Patch Available (was: Open) > Measure bw of a single copy call and display the correct aggregated bw > -- > > Key: MAPREDUCE-5873 > URL: https://issues.apache.org/jira/browse/MAPREDUCE-5873 > Project: Hadoop Map/Reduce > Issue Type: Bug >Affects Versions: 2.3.0 >Reporter: Siqi Li >Assignee: Siqi Li > Attachments: MAPREDUCE-5873.v1.patch, MAPREDUCE-5873.v2.patch, > MAPREDUCE-5873.v3.patch, MAPREDUCE-5873.v4.patch, MAPREDUCE-5873.v5.patch > > > Currently ShuffleScheduler in ReduceTask JVM status displays bandwidth. Its > definition however is confusing because it captures the time where there is > no copying because there is a pause between when new wave of map outputs is > available. > current bw is definded as (bytes copied so far) / (total time in the copy > phase so far) > It would be more useful > 1) to measure bandwidth of a single copy call. > 2) display aggregated bw as long as there is at least one fetcher is in the > copy call. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (MAPREDUCE-6029) TestCommitterEventHandler fails in trunk
[
https://issues.apache.org/jira/browse/MAPREDUCE-6029?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14163592#comment-14163592
]
Hudson commented on MAPREDUCE-6029:
---
FAILURE: Integrated in Hadoop-Mapreduce-trunk #1920 (See
[https://builds.apache.org/job/Hadoop-Mapreduce-trunk/1920/])
MAPREDUCE-6029. TestCommitterEventHandler fails in trunk. Contributed by Mit
Desai (jlowe: rev 2e789eb2261fa607489d8ae630fb77e8147d0266)
*
hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-app/src/test/java/org/apache/hadoop/mapreduce/v2/app/commit/TestCommitterEventHandler.java
* hadoop-mapreduce-project/CHANGES.txt
> TestCommitterEventHandler fails in trunk
>
>
> Key: MAPREDUCE-6029
> URL: https://issues.apache.org/jira/browse/MAPREDUCE-6029
> Project: Hadoop Map/Reduce
> Issue Type: Bug
>Affects Versions: 3.0.0
>Reporter: Ted Yu
>Assignee: Mit Desai
> Fix For: 2.6.0
>
> Attachments: MAPREDUCE-6029.patch, MAPREDUCE-6029.patch,
> MAPREDUCE-6029.patch
>
>
> From https://builds.apache.org/job/Hadoop-Mapreduce-trunk/1857/console :
> {code}
> Running org.apache.hadoop.mapreduce.v2.app.commit.TestCommitterEventHandler
> Tests run: 3, Failures: 1, Errors: 0, Skipped: 0, Time elapsed: 1.429 sec <<<
> FAILURE! - in
> org.apache.hadoop.mapreduce.v2.app.commit.TestCommitterEventHandler
> testFailure(org.apache.hadoop.mapreduce.v2.app.commit.TestCommitterEventHandler)
> Time elapsed: 1.198 sec <<< FAILURE!
> java.lang.AssertionError: null
> at org.junit.Assert.fail(Assert.java:86)
> at org.junit.Assert.assertTrue(Assert.java:41)
> at org.junit.Assert.assertNotNull(Assert.java:621)
> at org.junit.Assert.assertNotNull(Assert.java:631)
> at
> org.apache.hadoop.mapreduce.v2.app.commit.TestCommitterEventHandler.testFailure(TestCommitterEventHandler.java:314)
> {code}
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Commented] (MAPREDUCE-6029) TestCommitterEventHandler fails in trunk
[
https://issues.apache.org/jira/browse/MAPREDUCE-6029?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14163514#comment-14163514
]
Hudson commented on MAPREDUCE-6029:
---
FAILURE: Integrated in Hadoop-Hdfs-trunk #1895 (See
[https://builds.apache.org/job/Hadoop-Hdfs-trunk/1895/])
MAPREDUCE-6029. TestCommitterEventHandler fails in trunk. Contributed by Mit
Desai (jlowe: rev 2e789eb2261fa607489d8ae630fb77e8147d0266)
*
hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-app/src/test/java/org/apache/hadoop/mapreduce/v2/app/commit/TestCommitterEventHandler.java
* hadoop-mapreduce-project/CHANGES.txt
> TestCommitterEventHandler fails in trunk
>
>
> Key: MAPREDUCE-6029
> URL: https://issues.apache.org/jira/browse/MAPREDUCE-6029
> Project: Hadoop Map/Reduce
> Issue Type: Bug
>Affects Versions: 3.0.0
>Reporter: Ted Yu
>Assignee: Mit Desai
> Fix For: 2.6.0
>
> Attachments: MAPREDUCE-6029.patch, MAPREDUCE-6029.patch,
> MAPREDUCE-6029.patch
>
>
> From https://builds.apache.org/job/Hadoop-Mapreduce-trunk/1857/console :
> {code}
> Running org.apache.hadoop.mapreduce.v2.app.commit.TestCommitterEventHandler
> Tests run: 3, Failures: 1, Errors: 0, Skipped: 0, Time elapsed: 1.429 sec <<<
> FAILURE! - in
> org.apache.hadoop.mapreduce.v2.app.commit.TestCommitterEventHandler
> testFailure(org.apache.hadoop.mapreduce.v2.app.commit.TestCommitterEventHandler)
> Time elapsed: 1.198 sec <<< FAILURE!
> java.lang.AssertionError: null
> at org.junit.Assert.fail(Assert.java:86)
> at org.junit.Assert.assertTrue(Assert.java:41)
> at org.junit.Assert.assertNotNull(Assert.java:621)
> at org.junit.Assert.assertNotNull(Assert.java:631)
> at
> org.apache.hadoop.mapreduce.v2.app.commit.TestCommitterEventHandler.testFailure(TestCommitterEventHandler.java:314)
> {code}
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Commented] (MAPREDUCE-6029) TestCommitterEventHandler fails in trunk
[
https://issues.apache.org/jira/browse/MAPREDUCE-6029?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14163361#comment-14163361
]
Hudson commented on MAPREDUCE-6029:
---
FAILURE: Integrated in Hadoop-Yarn-trunk #705 (See
[https://builds.apache.org/job/Hadoop-Yarn-trunk/705/])
MAPREDUCE-6029. TestCommitterEventHandler fails in trunk. Contributed by Mit
Desai (jlowe: rev 2e789eb2261fa607489d8ae630fb77e8147d0266)
* hadoop-mapreduce-project/CHANGES.txt
*
hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-app/src/test/java/org/apache/hadoop/mapreduce/v2/app/commit/TestCommitterEventHandler.java
> TestCommitterEventHandler fails in trunk
>
>
> Key: MAPREDUCE-6029
> URL: https://issues.apache.org/jira/browse/MAPREDUCE-6029
> Project: Hadoop Map/Reduce
> Issue Type: Bug
>Affects Versions: 3.0.0
>Reporter: Ted Yu
>Assignee: Mit Desai
> Fix For: 2.6.0
>
> Attachments: MAPREDUCE-6029.patch, MAPREDUCE-6029.patch,
> MAPREDUCE-6029.patch
>
>
> From https://builds.apache.org/job/Hadoop-Mapreduce-trunk/1857/console :
> {code}
> Running org.apache.hadoop.mapreduce.v2.app.commit.TestCommitterEventHandler
> Tests run: 3, Failures: 1, Errors: 0, Skipped: 0, Time elapsed: 1.429 sec <<<
> FAILURE! - in
> org.apache.hadoop.mapreduce.v2.app.commit.TestCommitterEventHandler
> testFailure(org.apache.hadoop.mapreduce.v2.app.commit.TestCommitterEventHandler)
> Time elapsed: 1.198 sec <<< FAILURE!
> java.lang.AssertionError: null
> at org.junit.Assert.fail(Assert.java:86)
> at org.junit.Assert.assertTrue(Assert.java:41)
> at org.junit.Assert.assertNotNull(Assert.java:621)
> at org.junit.Assert.assertNotNull(Assert.java:631)
> at
> org.apache.hadoop.mapreduce.v2.app.commit.TestCommitterEventHandler.testFailure(TestCommitterEventHandler.java:314)
> {code}
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
