[jira] [Commented] (MAPREDUCE-5027) Shuffle does not limit number of outstanding connections
[ https://issues.apache.org/jira/browse/MAPREDUCE-5027?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13594728#comment-13594728 ] Hudson commented on MAPREDUCE-5027: --- Integrated in Hadoop-Mapreduce-trunk #1364 (See [https://builds.apache.org/job/Hadoop-Mapreduce-trunk/1364/]) MAPREDUCE-5027. Shuffle does not limit number of outstanding connections (Robert Parker via jeagles) (Revision 1453098) Result = SUCCESS jeagles : http://svn.apache.org/viewcvs.cgi/?root=Apache-SVN&view=rev&rev=1453098 Files : * /hadoop/common/trunk/hadoop-mapreduce-project/CHANGES.txt * /hadoop/common/trunk/hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-core/src/main/resources/mapred-default.xml * /hadoop/common/trunk/hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-shuffle/src/main/java/org/apache/hadoop/mapred/ShuffleHandler.java * /hadoop/common/trunk/hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-shuffle/src/test/java/org/apache/hadoop/mapred/TestShuffleHandler.java > Shuffle does not limit number of outstanding connections > > > Key: MAPREDUCE-5027 > URL: https://issues.apache.org/jira/browse/MAPREDUCE-5027 > Project: Hadoop Map/Reduce > Issue Type: Bug >Affects Versions: 2.0.3-alpha, 0.23.5 >Reporter: Jason Lowe >Assignee: Robert Parker > Fix For: 3.0.0, 0.23.7, 2.0.4-beta > > Attachments: MAPREDUCE-5027-2.patch, MAPREDUCE-5027-3.patch, > MAPREDUCE-5027-4.patch, MAPREDUCE-5027-b023-2.patch, > MAPREDUCE-5027-b023.patch, MAPREDUCE-5027.patch, MAPREDUCE-5027.patch > > > The ShuffleHandler does not have any configurable limits to the number of > outstanding connections allowed. Therefore a node with many map outputs and > many reducers in the cluster trying to fetch those outputs can exhaust a > nodemanager out of file descriptors. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (MAPREDUCE-5027) Shuffle does not limit number of outstanding connections
[ https://issues.apache.org/jira/browse/MAPREDUCE-5027?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13594682#comment-13594682 ] Hudson commented on MAPREDUCE-5027: --- Integrated in Hadoop-Hdfs-trunk #1336 (See [https://builds.apache.org/job/Hadoop-Hdfs-trunk/1336/]) MAPREDUCE-5027. Shuffle does not limit number of outstanding connections (Robert Parker via jeagles) (Revision 1453098) Result = SUCCESS jeagles : http://svn.apache.org/viewcvs.cgi/?root=Apache-SVN&view=rev&rev=1453098 Files : * /hadoop/common/trunk/hadoop-mapreduce-project/CHANGES.txt * /hadoop/common/trunk/hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-core/src/main/resources/mapred-default.xml * /hadoop/common/trunk/hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-shuffle/src/main/java/org/apache/hadoop/mapred/ShuffleHandler.java * /hadoop/common/trunk/hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-shuffle/src/test/java/org/apache/hadoop/mapred/TestShuffleHandler.java > Shuffle does not limit number of outstanding connections > > > Key: MAPREDUCE-5027 > URL: https://issues.apache.org/jira/browse/MAPREDUCE-5027 > Project: Hadoop Map/Reduce > Issue Type: Bug >Affects Versions: 2.0.3-alpha, 0.23.5 >Reporter: Jason Lowe >Assignee: Robert Parker > Fix For: 3.0.0, 0.23.7, 2.0.4-beta > > Attachments: MAPREDUCE-5027-2.patch, MAPREDUCE-5027-3.patch, > MAPREDUCE-5027-4.patch, MAPREDUCE-5027-b023-2.patch, > MAPREDUCE-5027-b023.patch, MAPREDUCE-5027.patch, MAPREDUCE-5027.patch > > > The ShuffleHandler does not have any configurable limits to the number of > outstanding connections allowed. Therefore a node with many map outputs and > many reducers in the cluster trying to fetch those outputs can exhaust a > nodemanager out of file descriptors. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (MAPREDUCE-5027) Shuffle does not limit number of outstanding connections
[ https://issues.apache.org/jira/browse/MAPREDUCE-5027?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13594670#comment-13594670 ] Hudson commented on MAPREDUCE-5027: --- Integrated in Hadoop-Hdfs-0.23-Build #545 (See [https://builds.apache.org/job/Hadoop-Hdfs-0.23-Build/545/]) MAPREDUCE-5027. Shuffle does not limit number of outstanding connections (Robert Parker via jeagles) (Revision 1453100) Result = FAILURE jeagles : http://svn.apache.org/viewcvs.cgi/?root=Apache-SVN&view=rev&rev=1453100 Files : * /hadoop/common/branches/branch-0.23/hadoop-mapreduce-project/CHANGES.txt * /hadoop/common/branches/branch-0.23/hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-core/src/main/resources/mapred-default.xml * /hadoop/common/branches/branch-0.23/hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-shuffle/src/main/java/org/apache/hadoop/mapred/ShuffleHandler.java * /hadoop/common/branches/branch-0.23/hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-shuffle/src/test/java/org/apache/hadoop/mapred/TestShuffleHandler.java > Shuffle does not limit number of outstanding connections > > > Key: MAPREDUCE-5027 > URL: https://issues.apache.org/jira/browse/MAPREDUCE-5027 > Project: Hadoop Map/Reduce > Issue Type: Bug >Affects Versions: 2.0.3-alpha, 0.23.5 >Reporter: Jason Lowe >Assignee: Robert Parker > Fix For: 3.0.0, 0.23.7, 2.0.4-beta > > Attachments: MAPREDUCE-5027-2.patch, MAPREDUCE-5027-3.patch, > MAPREDUCE-5027-4.patch, MAPREDUCE-5027-b023-2.patch, > MAPREDUCE-5027-b023.patch, MAPREDUCE-5027.patch, MAPREDUCE-5027.patch > > > The ShuffleHandler does not have any configurable limits to the number of > outstanding connections allowed. Therefore a node with many map outputs and > many reducers in the cluster trying to fetch those outputs can exhaust a > nodemanager out of file descriptors. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (MAPREDUCE-5027) Shuffle does not limit number of outstanding connections
[ https://issues.apache.org/jira/browse/MAPREDUCE-5027?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13594598#comment-13594598 ] Hudson commented on MAPREDUCE-5027: --- Integrated in Hadoop-Yarn-trunk #147 (See [https://builds.apache.org/job/Hadoop-Yarn-trunk/147/]) MAPREDUCE-5027. Shuffle does not limit number of outstanding connections (Robert Parker via jeagles) (Revision 1453098) Result = SUCCESS jeagles : http://svn.apache.org/viewcvs.cgi/?root=Apache-SVN&view=rev&rev=1453098 Files : * /hadoop/common/trunk/hadoop-mapreduce-project/CHANGES.txt * /hadoop/common/trunk/hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-core/src/main/resources/mapred-default.xml * /hadoop/common/trunk/hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-shuffle/src/main/java/org/apache/hadoop/mapred/ShuffleHandler.java * /hadoop/common/trunk/hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-shuffle/src/test/java/org/apache/hadoop/mapred/TestShuffleHandler.java > Shuffle does not limit number of outstanding connections > > > Key: MAPREDUCE-5027 > URL: https://issues.apache.org/jira/browse/MAPREDUCE-5027 > Project: Hadoop Map/Reduce > Issue Type: Bug >Affects Versions: 2.0.3-alpha, 0.23.5 >Reporter: Jason Lowe >Assignee: Robert Parker > Fix For: 3.0.0, 0.23.7, 2.0.4-beta > > Attachments: MAPREDUCE-5027-2.patch, MAPREDUCE-5027-3.patch, > MAPREDUCE-5027-4.patch, MAPREDUCE-5027-b023-2.patch, > MAPREDUCE-5027-b023.patch, MAPREDUCE-5027.patch, MAPREDUCE-5027.patch > > > The ShuffleHandler does not have any configurable limits to the number of > outstanding connections allowed. Therefore a node with many map outputs and > many reducers in the cluster trying to fetch those outputs can exhaust a > nodemanager out of file descriptors. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (MAPREDUCE-5027) Shuffle does not limit number of outstanding connections
[ https://issues.apache.org/jira/browse/MAPREDUCE-5027?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13594141#comment-13594141 ] Hudson commented on MAPREDUCE-5027: --- Integrated in Hadoop-trunk-Commit #3421 (See [https://builds.apache.org/job/Hadoop-trunk-Commit/3421/]) MAPREDUCE-5027. Shuffle does not limit number of outstanding connections (Robert Parker via jeagles) (Revision 1453098) Result = SUCCESS jeagles : http://svn.apache.org/viewcvs.cgi/?root=Apache-SVN&view=rev&rev=1453098 Files : * /hadoop/common/trunk/hadoop-mapreduce-project/CHANGES.txt * /hadoop/common/trunk/hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-core/src/main/resources/mapred-default.xml * /hadoop/common/trunk/hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-shuffle/src/main/java/org/apache/hadoop/mapred/ShuffleHandler.java * /hadoop/common/trunk/hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-shuffle/src/test/java/org/apache/hadoop/mapred/TestShuffleHandler.java > Shuffle does not limit number of outstanding connections > > > Key: MAPREDUCE-5027 > URL: https://issues.apache.org/jira/browse/MAPREDUCE-5027 > Project: Hadoop Map/Reduce > Issue Type: Bug >Affects Versions: 2.0.3-alpha, 0.23.5 >Reporter: Jason Lowe >Assignee: Robert Parker > Fix For: 3.0.0, 0.23.7, 2.0.4-beta > > Attachments: MAPREDUCE-5027-2.patch, MAPREDUCE-5027-3.patch, > MAPREDUCE-5027-4.patch, MAPREDUCE-5027-b023-2.patch, > MAPREDUCE-5027-b023.patch, MAPREDUCE-5027.patch, MAPREDUCE-5027.patch > > > The ShuffleHandler does not have any configurable limits to the number of > outstanding connections allowed. Therefore a node with many map outputs and > many reducers in the cluster trying to fetch those outputs can exhaust a > nodemanager out of file descriptors. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (MAPREDUCE-5027) Shuffle does not limit number of outstanding connections
[ https://issues.apache.org/jira/browse/MAPREDUCE-5027?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13594127#comment-13594127 ] Jonathan Eagles commented on MAPREDUCE-5027: +1. This patch looks good. Thanks, Rob. > Shuffle does not limit number of outstanding connections > > > Key: MAPREDUCE-5027 > URL: https://issues.apache.org/jira/browse/MAPREDUCE-5027 > Project: Hadoop Map/Reduce > Issue Type: Bug >Affects Versions: 2.0.3-alpha, 0.23.5 >Reporter: Jason Lowe >Assignee: Robert Parker > Attachments: MAPREDUCE-5027-2.patch, MAPREDUCE-5027-3.patch, > MAPREDUCE-5027-4.patch, MAPREDUCE-5027-b023-2.patch, > MAPREDUCE-5027-b023.patch, MAPREDUCE-5027.patch, MAPREDUCE-5027.patch > > > The ShuffleHandler does not have any configurable limits to the number of > outstanding connections allowed. Therefore a node with many map outputs and > many reducers in the cluster trying to fetch those outputs can exhaust a > nodemanager out of file descriptors. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (MAPREDUCE-5027) Shuffle does not limit number of outstanding connections
[
https://issues.apache.org/jira/browse/MAPREDUCE-5027?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13593919#comment-13593919
]
Hadoop QA commented on MAPREDUCE-5027:
--
{color:green}+1 overall{color}. Here are the results of testing the latest
attachment
http://issues.apache.org/jira/secure/attachment/12572165/MAPREDUCE-5027-4.patch
against trunk revision .
{color:green}+1 @author{color}. The patch does not contain any @author
tags.
{color:green}+1 tests included{color}. The patch appears to include 1 new
or modified test files.
{color:green}+1 tests included appear to have a timeout.{color}
{color:green}+1 javac{color}. The applied patch does not increase the
total number of javac compiler warnings.
{color:green}+1 javadoc{color}. The javadoc tool did not generate any
warning messages.
{color:green}+1 eclipse:eclipse{color}. The patch built with
eclipse:eclipse.
{color:green}+1 findbugs{color}. The patch does not introduce any new
Findbugs (version 1.3.9) warnings.
{color:green}+1 release audit{color}. The applied patch does not increase
the total number of release audit warnings.
{color:green}+1 core tests{color}. The patch passed unit tests in
hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-core
hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-shuffle.
{color:green}+1 contrib tests{color}. The patch passed contrib unit tests.
Test results:
https://builds.apache.org/job/PreCommit-MAPREDUCE-Build/3385//testReport/
Console output:
https://builds.apache.org/job/PreCommit-MAPREDUCE-Build/3385//console
This message is automatically generated.
> Shuffle does not limit number of outstanding connections
>
>
> Key: MAPREDUCE-5027
> URL: https://issues.apache.org/jira/browse/MAPREDUCE-5027
> Project: Hadoop Map/Reduce
> Issue Type: Bug
>Affects Versions: 2.0.3-alpha, 0.23.5
>Reporter: Jason Lowe
>Assignee: Robert Parker
> Attachments: MAPREDUCE-5027-2.patch, MAPREDUCE-5027-3.patch,
> MAPREDUCE-5027-4.patch, MAPREDUCE-5027-b023-2.patch,
> MAPREDUCE-5027-b023.patch, MAPREDUCE-5027.patch, MAPREDUCE-5027.patch
>
>
> The ShuffleHandler does not have any configurable limits to the number of
> outstanding connections allowed. Therefore a node with many map outputs and
> many reducers in the cluster trying to fetch those outputs can exhaust a
> nodemanager out of file descriptors.
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (MAPREDUCE-5027) Shuffle does not limit number of outstanding connections
[
https://issues.apache.org/jira/browse/MAPREDUCE-5027?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13593886#comment-13593886
]
Hadoop QA commented on MAPREDUCE-5027:
--
{color:red}-1 overall{color}. Here are the results of testing the latest
attachment
http://issues.apache.org/jira/secure/attachment/12572161/MAPREDUCE-5027-b023-2.patch
against trunk revision .
{color:red}-1 patch{color}. The patch command could not apply the patch.
Console output:
https://builds.apache.org/job/PreCommit-MAPREDUCE-Build/3384//console
This message is automatically generated.
> Shuffle does not limit number of outstanding connections
>
>
> Key: MAPREDUCE-5027
> URL: https://issues.apache.org/jira/browse/MAPREDUCE-5027
> Project: Hadoop Map/Reduce
> Issue Type: Bug
>Affects Versions: 2.0.3-alpha, 0.23.5
>Reporter: Jason Lowe
>Assignee: Robert Parker
> Attachments: MAPREDUCE-5027-2.patch, MAPREDUCE-5027-3.patch,
> MAPREDUCE-5027-4.patch, MAPREDUCE-5027-b023-2.patch,
> MAPREDUCE-5027-b023.patch, MAPREDUCE-5027.patch, MAPREDUCE-5027.patch
>
>
> The ShuffleHandler does not have any configurable limits to the number of
> outstanding connections allowed. Therefore a node with many map outputs and
> many reducers in the cluster trying to fetch those outputs can exhaust a
> nodemanager out of file descriptors.
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (MAPREDUCE-5027) Shuffle does not limit number of outstanding connections
[ https://issues.apache.org/jira/browse/MAPREDUCE-5027?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13593881#comment-13593881 ] Robert Parker commented on MAPREDUCE-5027: -- Jon, I have uploaded a new patch for trunk and branch 0.23, I have eliminated the timing issues in the test. > Shuffle does not limit number of outstanding connections > > > Key: MAPREDUCE-5027 > URL: https://issues.apache.org/jira/browse/MAPREDUCE-5027 > Project: Hadoop Map/Reduce > Issue Type: Bug >Affects Versions: 2.0.3-alpha, 0.23.5 >Reporter: Jason Lowe >Assignee: Robert Parker > Attachments: MAPREDUCE-5027-2.patch, MAPREDUCE-5027-3.patch, > MAPREDUCE-5027-4.patch, MAPREDUCE-5027-b023-2.patch, > MAPREDUCE-5027-b023.patch, MAPREDUCE-5027.patch, MAPREDUCE-5027.patch > > > The ShuffleHandler does not have any configurable limits to the number of > outstanding connections allowed. Therefore a node with many map outputs and > many reducers in the cluster trying to fetch those outputs can exhaust a > nodemanager out of file descriptors. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (MAPREDUCE-5027) Shuffle does not limit number of outstanding connections
[
https://issues.apache.org/jira/browse/MAPREDUCE-5027?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13592323#comment-13592323
]
Hadoop QA commented on MAPREDUCE-5027:
--
{color:red}-1 overall{color}. Here are the results of testing the latest
attachment
http://issues.apache.org/jira/secure/attachment/12571895/MAPREDUCE-5027-b023.patch
against trunk revision .
{color:red}-1 patch{color}. The patch command could not apply the patch.
Console output:
https://builds.apache.org/job/PreCommit-MAPREDUCE-Build/3381//console
This message is automatically generated.
> Shuffle does not limit number of outstanding connections
>
>
> Key: MAPREDUCE-5027
> URL: https://issues.apache.org/jira/browse/MAPREDUCE-5027
> Project: Hadoop Map/Reduce
> Issue Type: Bug
>Affects Versions: 2.0.3-alpha, 0.23.5
>Reporter: Jason Lowe
>Assignee: Robert Parker
> Attachments: MAPREDUCE-5027-2.patch, MAPREDUCE-5027-3.patch,
> MAPREDUCE-5027-b023.patch, MAPREDUCE-5027.patch, MAPREDUCE-5027.patch
>
>
> The ShuffleHandler does not have any configurable limits to the number of
> outstanding connections allowed. Therefore a node with many map outputs and
> many reducers in the cluster trying to fetch those outputs can exhaust a
> nodemanager out of file descriptors.
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (MAPREDUCE-5027) Shuffle does not limit number of outstanding connections
[
https://issues.apache.org/jira/browse/MAPREDUCE-5027?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13591943#comment-13591943
]
Hadoop QA commented on MAPREDUCE-5027:
--
{color:green}+1 overall{color}. Here are the results of testing the latest
attachment
http://issues.apache.org/jira/secure/attachment/12571838/MAPREDUCE-5027-3.patch
against trunk revision .
{color:green}+1 @author{color}. The patch does not contain any @author
tags.
{color:green}+1 tests included{color}. The patch appears to include 1 new
or modified test files.
{color:green}+1 tests included appear to have a timeout.{color}
{color:green}+1 javac{color}. The applied patch does not increase the
total number of javac compiler warnings.
{color:green}+1 javadoc{color}. The javadoc tool did not generate any
warning messages.
{color:green}+1 eclipse:eclipse{color}. The patch built with
eclipse:eclipse.
{color:green}+1 findbugs{color}. The patch does not introduce any new
Findbugs (version 1.3.9) warnings.
{color:green}+1 release audit{color}. The applied patch does not increase
the total number of release audit warnings.
{color:green}+1 core tests{color}. The patch passed unit tests in
hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-core
hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-shuffle.
{color:green}+1 contrib tests{color}. The patch passed contrib unit tests.
Test results:
https://builds.apache.org/job/PreCommit-MAPREDUCE-Build/3378//testReport/
Console output:
https://builds.apache.org/job/PreCommit-MAPREDUCE-Build/3378//console
This message is automatically generated.
> Shuffle does not limit number of outstanding connections
>
>
> Key: MAPREDUCE-5027
> URL: https://issues.apache.org/jira/browse/MAPREDUCE-5027
> Project: Hadoop Map/Reduce
> Issue Type: Bug
>Affects Versions: 2.0.3-alpha, 0.23.5
>Reporter: Jason Lowe
>Assignee: Robert Parker
> Attachments: MAPREDUCE-5027-2.patch, MAPREDUCE-5027-3.patch,
> MAPREDUCE-5027.patch, MAPREDUCE-5027.patch
>
>
> The ShuffleHandler does not have any configurable limits to the number of
> outstanding connections allowed. Therefore a node with many map outputs and
> many reducers in the cluster trying to fetch those outputs can exhaust a
> nodemanager out of file descriptors.
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (MAPREDUCE-5027) Shuffle does not limit number of outstanding connections
[ https://issues.apache.org/jira/browse/MAPREDUCE-5027?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13591034#comment-13591034 ] Jonathan Eagles commented on MAPREDUCE-5027: Thanks for the patch Rob. There seems to be a race condition in the testMaxConnections test. On my desktop I get two different results consistently. First: Failed tests: testMaxConnections(org.apache.hadoop.mapred.TestShuffleHandler): expected:<2> but was:<1> Second: Tests in error: testSerializeMeta(org.apache.hadoop.mapred.TestShuffleHandler): test timed out after 2000 milliseconds testShuffleMetrics(org.apache.hadoop.mapred.TestShuffleHandler): test timed out after 2000 milliseconds testClientClosesConnection(org.apache.hadoop.mapred.TestShuffleHandler): test timed out after 2000 milliseconds testMaxConnections(org.apache.hadoop.mapred.TestShuffleHandler): test timed out after 2000 milliseconds > Shuffle does not limit number of outstanding connections > > > Key: MAPREDUCE-5027 > URL: https://issues.apache.org/jira/browse/MAPREDUCE-5027 > Project: Hadoop Map/Reduce > Issue Type: Bug >Affects Versions: 2.0.3-alpha, 0.23.5 >Reporter: Jason Lowe >Assignee: Robert Parker > Attachments: MAPREDUCE-5027-2.patch, MAPREDUCE-5027.patch, > MAPREDUCE-5027.patch > > > The ShuffleHandler does not have any configurable limits to the number of > outstanding connections allowed. Therefore a node with many map outputs and > many reducers in the cluster trying to fetch those outputs can exhaust a > nodemanager out of file descriptors. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (MAPREDUCE-5027) Shuffle does not limit number of outstanding connections
[
https://issues.apache.org/jira/browse/MAPREDUCE-5027?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13590190#comment-13590190
]
Hadoop QA commented on MAPREDUCE-5027:
--
{color:green}+1 overall{color}. Here are the results of testing the latest
attachment
http://issues.apache.org/jira/secure/attachment/12571527/MAPREDUCE-5027-2.patch
against trunk revision .
{color:green}+1 @author{color}. The patch does not contain any @author
tags.
{color:green}+1 tests included{color}. The patch appears to include 1 new
or modified test files.
{color:green}+1 tests included appear to have a timeout.{color}
{color:green}+1 javac{color}. The applied patch does not increase the
total number of javac compiler warnings.
{color:green}+1 javadoc{color}. The javadoc tool did not generate any
warning messages.
{color:green}+1 eclipse:eclipse{color}. The patch built with
eclipse:eclipse.
{color:green}+1 findbugs{color}. The patch does not introduce any new
Findbugs (version 1.3.9) warnings.
{color:green}+1 release audit{color}. The applied patch does not increase
the total number of release audit warnings.
{color:green}+1 core tests{color}. The patch passed unit tests in
hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-core
hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-shuffle.
{color:green}+1 contrib tests{color}. The patch passed contrib unit tests.
Test results:
https://builds.apache.org/job/PreCommit-MAPREDUCE-Build/3376//testReport/
Console output:
https://builds.apache.org/job/PreCommit-MAPREDUCE-Build/3376//console
This message is automatically generated.
> Shuffle does not limit number of outstanding connections
>
>
> Key: MAPREDUCE-5027
> URL: https://issues.apache.org/jira/browse/MAPREDUCE-5027
> Project: Hadoop Map/Reduce
> Issue Type: Bug
>Affects Versions: 2.0.3-alpha, 0.23.5
>Reporter: Jason Lowe
>Assignee: Robert Parker
> Attachments: MAPREDUCE-5027-2.patch, MAPREDUCE-5027.patch,
> MAPREDUCE-5027.patch
>
>
> The ShuffleHandler does not have any configurable limits to the number of
> outstanding connections allowed. Therefore a node with many map outputs and
> many reducers in the cluster trying to fetch those outputs can exhaust a
> nodemanager out of file descriptors.
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (MAPREDUCE-5027) Shuffle does not limit number of outstanding connections
[
https://issues.apache.org/jira/browse/MAPREDUCE-5027?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13587940#comment-13587940
]
Hadoop QA commented on MAPREDUCE-5027:
--
{color:red}-1 overall{color}. Here are the results of testing the latest
attachment
http://issues.apache.org/jira/secure/attachment/12571124/MAPREDUCE-5027.patch
against trunk revision .
{color:green}+1 @author{color}. The patch does not contain any @author
tags.
{color:green}+1 tests included{color}. The patch appears to include 1 new
or modified test files.
{color:red}-1 one of tests included doesn't have a timeout.{color}
{color:green}+1 javac{color}. The applied patch does not increase the
total number of javac compiler warnings.
{color:green}+1 javadoc{color}. The javadoc tool did not generate any
warning messages.
{color:green}+1 eclipse:eclipse{color}. The patch built with
eclipse:eclipse.
{color:green}+1 findbugs{color}. The patch does not introduce any new
Findbugs (version 1.3.9) warnings.
{color:green}+1 release audit{color}. The applied patch does not increase
the total number of release audit warnings.
{color:green}+1 core tests{color}. The patch passed unit tests in
hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-core
hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-shuffle.
{color:green}+1 contrib tests{color}. The patch passed contrib unit tests.
Test results:
https://builds.apache.org/job/PreCommit-MAPREDUCE-Build/3367//testReport/
Console output:
https://builds.apache.org/job/PreCommit-MAPREDUCE-Build/3367//console
This message is automatically generated.
> Shuffle does not limit number of outstanding connections
>
>
> Key: MAPREDUCE-5027
> URL: https://issues.apache.org/jira/browse/MAPREDUCE-5027
> Project: Hadoop Map/Reduce
> Issue Type: Bug
>Affects Versions: 2.0.3-alpha, 0.23.5
>Reporter: Jason Lowe
>Assignee: Robert Parker
> Attachments: MAPREDUCE-5027.patch, MAPREDUCE-5027.patch
>
>
> The ShuffleHandler does not have any configurable limits to the number of
> outstanding connections allowed. Therefore a node with many map outputs and
> many reducers in the cluster trying to fetch those outputs can exhaust a
> nodemanager out of file descriptors.
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (MAPREDUCE-5027) Shuffle does not limit number of outstanding connections
[
https://issues.apache.org/jira/browse/MAPREDUCE-5027?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13587836#comment-13587836
]
Hadoop QA commented on MAPREDUCE-5027:
--
{color:red}-1 overall{color}. Here are the results of testing the latest
attachment
http://issues.apache.org/jira/secure/attachment/12571092/MAPREDUCE-5027.patch
against trunk revision .
{color:green}+1 @author{color}. The patch does not contain any @author
tags.
{color:green}+1 tests included{color}. The patch appears to include 1 new
or modified test files.
{color:red}-1 one of tests included doesn't have a timeout.{color}
{color:green}+1 javac{color}. The applied patch does not increase the
total number of javac compiler warnings.
{color:green}+1 javadoc{color}. The javadoc tool did not generate any
warning messages.
{color:green}+1 eclipse:eclipse{color}. The patch built with
eclipse:eclipse.
{color:green}+1 findbugs{color}. The patch does not introduce any new
Findbugs (version 1.3.9) warnings.
{color:green}+1 release audit{color}. The applied patch does not increase
the total number of release audit warnings.
{color:green}+1 core tests{color}. The patch passed unit tests in
hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-core
hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-shuffle.
{color:green}+1 contrib tests{color}. The patch passed contrib unit tests.
Test results:
https://builds.apache.org/job/PreCommit-MAPREDUCE-Build/3366//testReport/
Console output:
https://builds.apache.org/job/PreCommit-MAPREDUCE-Build/3366//console
This message is automatically generated.
> Shuffle does not limit number of outstanding connections
>
>
> Key: MAPREDUCE-5027
> URL: https://issues.apache.org/jira/browse/MAPREDUCE-5027
> Project: Hadoop Map/Reduce
> Issue Type: Bug
>Affects Versions: 2.0.3-alpha, 0.23.5
>Reporter: Jason Lowe
>Assignee: Robert Parker
> Attachments: MAPREDUCE-5027.patch
>
>
> The ShuffleHandler does not have any configurable limits to the number of
> outstanding connections allowed. Therefore a node with many map outputs and
> many reducers in the cluster trying to fetch those outputs can exhaust a
> nodemanager out of file descriptors.
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (MAPREDUCE-5027) Shuffle does not limit number of outstanding connections
[
https://issues.apache.org/jira/browse/MAPREDUCE-5027?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13587654#comment-13587654
]
Hadoop QA commented on MAPREDUCE-5027:
--
{color:red}-1 overall{color}. Here are the results of testing the latest
attachment
http://issues.apache.org/jira/secure/attachment/12571063/MAPREDUCE-5027.patch
against trunk revision .
{color:green}+1 @author{color}. The patch does not contain any @author
tags.
{color:green}+1 tests included{color}. The patch appears to include 1 new
or modified test files.
{color:red}-1 one of tests included doesn't have a timeout.{color}
{color:green}+1 javac{color}. The applied patch does not increase the
total number of javac compiler warnings.
{color:green}+1 javadoc{color}. The javadoc tool did not generate any
warning messages.
{color:green}+1 eclipse:eclipse{color}. The patch built with
eclipse:eclipse.
{color:green}+1 findbugs{color}. The patch does not introduce any new
Findbugs (version 1.3.9) warnings.
{color:green}+1 release audit{color}. The applied patch does not increase
the total number of release audit warnings.
{color:red}-1 core tests{color}. The patch failed these unit tests in
hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-core
hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-shuffle:
org.apache.hadoop.mapred.TestShuffleHandler
{color:green}+1 contrib tests{color}. The patch passed contrib unit tests.
Test results:
https://builds.apache.org/job/PreCommit-MAPREDUCE-Build/3364//testReport/
Console output:
https://builds.apache.org/job/PreCommit-MAPREDUCE-Build/3364//console
This message is automatically generated.
> Shuffle does not limit number of outstanding connections
>
>
> Key: MAPREDUCE-5027
> URL: https://issues.apache.org/jira/browse/MAPREDUCE-5027
> Project: Hadoop Map/Reduce
> Issue Type: Bug
>Affects Versions: 2.0.3-alpha, 0.23.5
>Reporter: Jason Lowe
>Assignee: Robert Parker
> Attachments: MAPREDUCE-5027.patch, MAPREDUCE-5027.patch
>
>
> The ShuffleHandler does not have any configurable limits to the number of
> outstanding connections allowed. Therefore a node with many map outputs and
> many reducers in the cluster trying to fetch those outputs can exhaust a
> nodemanager out of file descriptors.
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (MAPREDUCE-5027) Shuffle does not limit number of outstanding connections
[
https://issues.apache.org/jira/browse/MAPREDUCE-5027?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13587422#comment-13587422
]
Hadoop QA commented on MAPREDUCE-5027:
--
{color:red}-1 overall{color}. Here are the results of testing the latest
attachment
http://issues.apache.org/jira/secure/attachment/12571028/MAPREDUCE-5027.patch
against trunk revision .
{color:green}+1 @author{color}. The patch does not contain any @author
tags.
{color:green}+1 tests included{color}. The patch appears to include 1 new
or modified test files.
{color:red}-1 one of tests included doesn't have a timeout.{color}
{color:green}+1 javac{color}. The applied patch does not increase the
total number of javac compiler warnings.
{color:green}+1 javadoc{color}. The javadoc tool did not generate any
warning messages.
{color:green}+1 eclipse:eclipse{color}. The patch built with
eclipse:eclipse.
{color:green}+1 findbugs{color}. The patch does not introduce any new
Findbugs (version 1.3.9) warnings.
{color:green}+1 release audit{color}. The applied patch does not increase
the total number of release audit warnings.
{color:red}-1 core tests{color}. The patch failed these unit tests in
hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-core
hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-shuffle:
org.apache.hadoop.mapred.TestShuffleHandler
{color:green}+1 contrib tests{color}. The patch passed contrib unit tests.
Test results:
https://builds.apache.org/job/PreCommit-MAPREDUCE-Build/3361//testReport/
Console output:
https://builds.apache.org/job/PreCommit-MAPREDUCE-Build/3361//console
This message is automatically generated.
> Shuffle does not limit number of outstanding connections
>
>
> Key: MAPREDUCE-5027
> URL: https://issues.apache.org/jira/browse/MAPREDUCE-5027
> Project: Hadoop Map/Reduce
> Issue Type: Bug
>Affects Versions: 2.0.3-alpha, 0.23.5
>Reporter: Jason Lowe
>Assignee: Robert Parker
> Attachments: MAPREDUCE-5027.patch
>
>
> The ShuffleHandler does not have any configurable limits to the number of
> outstanding connections allowed. Therefore a node with many map outputs and
> many reducers in the cluster trying to fetch those outputs can exhaust a
> nodemanager out of file descriptors.
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (MAPREDUCE-5027) Shuffle does not limit number of outstanding connections
[ https://issues.apache.org/jira/browse/MAPREDUCE-5027?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13586431#comment-13586431 ] Alejandro Abdelnur commented on MAPREDUCE-5027: --- The following may be handy: https://issues.jboss.org/browse/NETTY-311 > Shuffle does not limit number of outstanding connections > > > Key: MAPREDUCE-5027 > URL: https://issues.apache.org/jira/browse/MAPREDUCE-5027 > Project: Hadoop Map/Reduce > Issue Type: Bug >Affects Versions: 2.0.3-alpha, 0.23.5 >Reporter: Jason Lowe > > The ShuffleHandler does not have any configurable limits to the number of > outstanding connections allowed. Therefore a node with many map outputs and > many reducers in the cluster trying to fetch those outputs can exhaust a > nodemanager out of file descriptors. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (MAPREDUCE-5027) Shuffle does not limit number of outstanding connections
[ https://issues.apache.org/jira/browse/MAPREDUCE-5027?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13586267#comment-13586267 ] Jason Lowe commented on MAPREDUCE-5027: --- AFAIK there is no built-in way to have Netty automatically limit the number of active client connections. A quick search on the net indicates one way this is handled is to simply close the extra connections as soon as they are created once we get past a specified number of active connections. > Shuffle does not limit number of outstanding connections > > > Key: MAPREDUCE-5027 > URL: https://issues.apache.org/jira/browse/MAPREDUCE-5027 > Project: Hadoop Map/Reduce > Issue Type: Bug >Affects Versions: 2.0.3-alpha, 0.23.5 >Reporter: Jason Lowe > > The ShuffleHandler does not have any configurable limits to the number of > outstanding connections allowed. Therefore a node with many map outputs and > many reducers in the cluster trying to fetch those outputs can exhaust a > nodemanager out of file descriptors. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators For more information on JIRA, see: http://www.atlassian.com/software/jira
