[jira] [Commented] (MAPREDUCE-4797) LocalContainerAllocator can loop forever trying to contact the RM
[ https://issues.apache.org/jira/browse/MAPREDUCE-4797?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13498010#comment-13498010 ] Hudson commented on MAPREDUCE-4797: --- Integrated in Hadoop-Mapreduce-trunk #1258 (See [https://builds.apache.org/job/Hadoop-Mapreduce-trunk/1258/]) MAPREDUCE-4797. LocalContainerAllocator can loop forever trying to contact the RM (jlowe via bobby) (Revision 1409525) Result = FAILURE bobby : http://svn.apache.org/viewcvs.cgi/?root=Apache-SVN&view=rev&rev=1409525 Files : * /hadoop/common/trunk/hadoop-mapreduce-project/CHANGES.txt * /hadoop/common/trunk/hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-app/src/main/java/org/apache/hadoop/mapreduce/v2/app/local/LocalContainerAllocator.java * /hadoop/common/trunk/hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-app/src/test/java/org/apache/hadoop/mapreduce/v2/app/local * /hadoop/common/trunk/hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-app/src/test/java/org/apache/hadoop/mapreduce/v2/app/local/TestLocalContainerAllocator.java > LocalContainerAllocator can loop forever trying to contact the RM > - > > Key: MAPREDUCE-4797 > URL: https://issues.apache.org/jira/browse/MAPREDUCE-4797 > Project: Hadoop Map/Reduce > Issue Type: Bug > Components: applicationmaster >Affects Versions: 0.23.3, 2.0.1-alpha >Reporter: Jason Lowe >Assignee: Jason Lowe > Fix For: 3.0.0, 2.0.3-alpha, 0.23.5 > > Attachments: MAPREDUCE-4797.patch, MAPREDUCE-4797.patch > > > If LocalContainerAllocator has trouble communicating with the RM it can end > up retrying forever if the nature of the error is not a YarnException. > This can be particulary bad if the connection went down because the cluster > was reset such that the RM and NM have lost track of the process and > therefore nothing else will eventually kill the process. In this scenario, > the looping AM continues to pelt the RM with connection requests every second > using a stale token, and the RM logs the SASL exceptions over and over. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (MAPREDUCE-4797) LocalContainerAllocator can loop forever trying to contact the RM
[ https://issues.apache.org/jira/browse/MAPREDUCE-4797?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13497990#comment-13497990 ] Hudson commented on MAPREDUCE-4797: --- Integrated in Hadoop-Hdfs-0.23-Build #436 (See [https://builds.apache.org/job/Hadoop-Hdfs-0.23-Build/436/]) svn merge -c 1409525 FIXES: MAPREDUCE-4797. LocalContainerAllocator can loop forever trying to contact the RM (jlowe via bobby) (Revision 1409532) Result = SUCCESS bobby : http://svn.apache.org/viewcvs.cgi/?root=Apache-SVN&view=rev&rev=1409532 Files : * /hadoop/common/branches/branch-0.23/hadoop-mapreduce-project/CHANGES.txt * /hadoop/common/branches/branch-0.23/hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-app/src/main/java/org/apache/hadoop/mapreduce/v2/app/local/LocalContainerAllocator.java * /hadoop/common/branches/branch-0.23/hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-app/src/test/java/org/apache/hadoop/mapreduce/v2/app/local * /hadoop/common/branches/branch-0.23/hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-app/src/test/java/org/apache/hadoop/mapreduce/v2/app/local/TestLocalContainerAllocator.java > LocalContainerAllocator can loop forever trying to contact the RM > - > > Key: MAPREDUCE-4797 > URL: https://issues.apache.org/jira/browse/MAPREDUCE-4797 > Project: Hadoop Map/Reduce > Issue Type: Bug > Components: applicationmaster >Affects Versions: 0.23.3, 2.0.1-alpha >Reporter: Jason Lowe >Assignee: Jason Lowe > Fix For: 3.0.0, 2.0.3-alpha, 0.23.5 > > Attachments: MAPREDUCE-4797.patch, MAPREDUCE-4797.patch > > > If LocalContainerAllocator has trouble communicating with the RM it can end > up retrying forever if the nature of the error is not a YarnException. > This can be particulary bad if the connection went down because the cluster > was reset such that the RM and NM have lost track of the process and > therefore nothing else will eventually kill the process. In this scenario, > the looping AM continues to pelt the RM with connection requests every second > using a stale token, and the RM logs the SASL exceptions over and over. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (MAPREDUCE-4797) LocalContainerAllocator can loop forever trying to contact the RM
[ https://issues.apache.org/jira/browse/MAPREDUCE-4797?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13497995#comment-13497995 ] Hudson commented on MAPREDUCE-4797: --- Integrated in Hadoop-Hdfs-trunk #1227 (See [https://builds.apache.org/job/Hadoop-Hdfs-trunk/1227/]) MAPREDUCE-4797. LocalContainerAllocator can loop forever trying to contact the RM (jlowe via bobby) (Revision 1409525) Result = FAILURE bobby : http://svn.apache.org/viewcvs.cgi/?root=Apache-SVN&view=rev&rev=1409525 Files : * /hadoop/common/trunk/hadoop-mapreduce-project/CHANGES.txt * /hadoop/common/trunk/hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-app/src/main/java/org/apache/hadoop/mapreduce/v2/app/local/LocalContainerAllocator.java * /hadoop/common/trunk/hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-app/src/test/java/org/apache/hadoop/mapreduce/v2/app/local * /hadoop/common/trunk/hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-app/src/test/java/org/apache/hadoop/mapreduce/v2/app/local/TestLocalContainerAllocator.java > LocalContainerAllocator can loop forever trying to contact the RM > - > > Key: MAPREDUCE-4797 > URL: https://issues.apache.org/jira/browse/MAPREDUCE-4797 > Project: Hadoop Map/Reduce > Issue Type: Bug > Components: applicationmaster >Affects Versions: 0.23.3, 2.0.1-alpha >Reporter: Jason Lowe >Assignee: Jason Lowe > Fix For: 3.0.0, 2.0.3-alpha, 0.23.5 > > Attachments: MAPREDUCE-4797.patch, MAPREDUCE-4797.patch > > > If LocalContainerAllocator has trouble communicating with the RM it can end > up retrying forever if the nature of the error is not a YarnException. > This can be particulary bad if the connection went down because the cluster > was reset such that the RM and NM have lost track of the process and > therefore nothing else will eventually kill the process. In this scenario, > the looping AM continues to pelt the RM with connection requests every second > using a stale token, and the RM logs the SASL exceptions over and over. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (MAPREDUCE-4797) LocalContainerAllocator can loop forever trying to contact the RM
[ https://issues.apache.org/jira/browse/MAPREDUCE-4797?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13497923#comment-13497923 ] Hudson commented on MAPREDUCE-4797: --- Integrated in Hadoop-Yarn-trunk #37 (See [https://builds.apache.org/job/Hadoop-Yarn-trunk/37/]) MAPREDUCE-4797. LocalContainerAllocator can loop forever trying to contact the RM (jlowe via bobby) (Revision 1409525) Result = SUCCESS bobby : http://svn.apache.org/viewcvs.cgi/?root=Apache-SVN&view=rev&rev=1409525 Files : * /hadoop/common/trunk/hadoop-mapreduce-project/CHANGES.txt * /hadoop/common/trunk/hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-app/src/main/java/org/apache/hadoop/mapreduce/v2/app/local/LocalContainerAllocator.java * /hadoop/common/trunk/hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-app/src/test/java/org/apache/hadoop/mapreduce/v2/app/local * /hadoop/common/trunk/hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-app/src/test/java/org/apache/hadoop/mapreduce/v2/app/local/TestLocalContainerAllocator.java > LocalContainerAllocator can loop forever trying to contact the RM > - > > Key: MAPREDUCE-4797 > URL: https://issues.apache.org/jira/browse/MAPREDUCE-4797 > Project: Hadoop Map/Reduce > Issue Type: Bug > Components: applicationmaster >Affects Versions: 0.23.3, 2.0.1-alpha >Reporter: Jason Lowe >Assignee: Jason Lowe > Fix For: 3.0.0, 2.0.3-alpha, 0.23.5 > > Attachments: MAPREDUCE-4797.patch, MAPREDUCE-4797.patch > > > If LocalContainerAllocator has trouble communicating with the RM it can end > up retrying forever if the nature of the error is not a YarnException. > This can be particulary bad if the connection went down because the cluster > was reset such that the RM and NM have lost track of the process and > therefore nothing else will eventually kill the process. In this scenario, > the looping AM continues to pelt the RM with connection requests every second > using a stale token, and the RM logs the SASL exceptions over and over. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (MAPREDUCE-4797) LocalContainerAllocator can loop forever trying to contact the RM
[ https://issues.apache.org/jira/browse/MAPREDUCE-4797?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13497596#comment-13497596 ] Hudson commented on MAPREDUCE-4797: --- Integrated in Hadoop-trunk-Commit #3019 (See [https://builds.apache.org/job/Hadoop-trunk-Commit/3019/]) MAPREDUCE-4797. LocalContainerAllocator can loop forever trying to contact the RM (jlowe via bobby) (Revision 1409525) Result = SUCCESS bobby : http://svn.apache.org/viewcvs.cgi/?root=Apache-SVN&view=rev&rev=1409525 Files : * /hadoop/common/trunk/hadoop-mapreduce-project/CHANGES.txt * /hadoop/common/trunk/hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-app/src/main/java/org/apache/hadoop/mapreduce/v2/app/local/LocalContainerAllocator.java * /hadoop/common/trunk/hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-app/src/test/java/org/apache/hadoop/mapreduce/v2/app/local * /hadoop/common/trunk/hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-app/src/test/java/org/apache/hadoop/mapreduce/v2/app/local/TestLocalContainerAllocator.java > LocalContainerAllocator can loop forever trying to contact the RM > - > > Key: MAPREDUCE-4797 > URL: https://issues.apache.org/jira/browse/MAPREDUCE-4797 > Project: Hadoop Map/Reduce > Issue Type: Bug > Components: applicationmaster >Affects Versions: 0.23.3, 2.0.1-alpha >Reporter: Jason Lowe >Assignee: Jason Lowe > Fix For: 3.0.0, 2.0.3-alpha, 0.23.5 > > Attachments: MAPREDUCE-4797.patch, MAPREDUCE-4797.patch > > > If LocalContainerAllocator has trouble communicating with the RM it can end > up retrying forever if the nature of the error is not a YarnException. > This can be particulary bad if the connection went down because the cluster > was reset such that the RM and NM have lost track of the process and > therefore nothing else will eventually kill the process. In this scenario, > the looping AM continues to pelt the RM with connection requests every second > using a stale token, and the RM logs the SASL exceptions over and over. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (MAPREDUCE-4797) LocalContainerAllocator can loop forever trying to contact the RM
[ https://issues.apache.org/jira/browse/MAPREDUCE-4797?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13497566#comment-13497566 ] Robert Joseph Evans commented on MAPREDUCE-4797: Looks good +1, I'll check it in. > LocalContainerAllocator can loop forever trying to contact the RM > - > > Key: MAPREDUCE-4797 > URL: https://issues.apache.org/jira/browse/MAPREDUCE-4797 > Project: Hadoop Map/Reduce > Issue Type: Bug > Components: applicationmaster >Affects Versions: 0.23.3, 2.0.1-alpha >Reporter: Jason Lowe >Assignee: Jason Lowe > Attachments: MAPREDUCE-4797.patch, MAPREDUCE-4797.patch > > > If LocalContainerAllocator has trouble communicating with the RM it can end > up retrying forever if the nature of the error is not a YarnException. > This can be particulary bad if the connection went down because the cluster > was reset such that the RM and NM have lost track of the process and > therefore nothing else will eventually kill the process. In this scenario, > the looping AM continues to pelt the RM with connection requests every second > using a stale token, and the RM logs the SASL exceptions over and over. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (MAPREDUCE-4797) LocalContainerAllocator can loop forever trying to contact the RM
[
https://issues.apache.org/jira/browse/MAPREDUCE-4797?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13497532#comment-13497532
]
Hadoop QA commented on MAPREDUCE-4797:
--
{color:green}+1 overall{color}. Here are the results of testing the latest
attachment
http://issues.apache.org/jira/secure/attachment/12553572/MAPREDUCE-4797.patch
against trunk revision .
{color:green}+1 @author{color}. The patch does not contain any @author
tags.
{color:green}+1 tests included{color}. The patch appears to include 1 new
or modified test files.
{color:green}+1 javac{color}. The applied patch does not increase the
total number of javac compiler warnings.
{color:green}+1 javadoc{color}. The javadoc tool did not generate any
warning messages.
{color:green}+1 eclipse:eclipse{color}. The patch built with
eclipse:eclipse.
{color:green}+1 findbugs{color}. The patch does not introduce any new
Findbugs (version 1.3.9) warnings.
{color:green}+1 release audit{color}. The applied patch does not increase
the total number of release audit warnings.
{color:green}+1 core tests{color}. The patch passed unit tests in
hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-app.
{color:green}+1 contrib tests{color}. The patch passed contrib unit tests.
Test results:
https://builds.apache.org/job/PreCommit-MAPREDUCE-Build/3032//testReport/
Console output:
https://builds.apache.org/job/PreCommit-MAPREDUCE-Build/3032//console
This message is automatically generated.
> LocalContainerAllocator can loop forever trying to contact the RM
> -
>
> Key: MAPREDUCE-4797
> URL: https://issues.apache.org/jira/browse/MAPREDUCE-4797
> Project: Hadoop Map/Reduce
> Issue Type: Bug
> Components: applicationmaster
>Affects Versions: 0.23.3, 2.0.1-alpha
>Reporter: Jason Lowe
>Assignee: Jason Lowe
> Attachments: MAPREDUCE-4797.patch, MAPREDUCE-4797.patch
>
>
> If LocalContainerAllocator has trouble communicating with the RM it can end
> up retrying forever if the nature of the error is not a YarnException.
> This can be particulary bad if the connection went down because the cluster
> was reset such that the RM and NM have lost track of the process and
> therefore nothing else will eventually kill the process. In this scenario,
> the looping AM continues to pelt the RM with connection requests every second
> using a stale token, and the RM logs the SASL exceptions over and over.
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (MAPREDUCE-4797) LocalContainerAllocator can loop forever trying to contact the RM
[
https://issues.apache.org/jira/browse/MAPREDUCE-4797?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13497506#comment-13497506
]
Hadoop QA commented on MAPREDUCE-4797:
--
{color:green}+1 overall{color}. Here are the results of testing the latest
attachment
http://issues.apache.org/jira/secure/attachment/12553564/MAPREDUCE-4797.patch
against trunk revision .
{color:green}+1 @author{color}. The patch does not contain any @author
tags.
{color:green}+1 tests included{color}. The patch appears to include 1 new
or modified test files.
{color:green}+1 javac{color}. The applied patch does not increase the
total number of javac compiler warnings.
{color:green}+1 javadoc{color}. The javadoc tool did not generate any
warning messages.
{color:green}+1 eclipse:eclipse{color}. The patch built with
eclipse:eclipse.
{color:green}+1 findbugs{color}. The patch does not introduce any new
Findbugs (version 1.3.9) warnings.
{color:green}+1 release audit{color}. The applied patch does not increase
the total number of release audit warnings.
{color:green}+1 core tests{color}. The patch passed unit tests in
hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-app.
{color:green}+1 contrib tests{color}. The patch passed contrib unit tests.
Test results:
https://builds.apache.org/job/PreCommit-MAPREDUCE-Build/3031//testReport/
Console output:
https://builds.apache.org/job/PreCommit-MAPREDUCE-Build/3031//console
This message is automatically generated.
> LocalContainerAllocator can loop forever trying to contact the RM
> -
>
> Key: MAPREDUCE-4797
> URL: https://issues.apache.org/jira/browse/MAPREDUCE-4797
> Project: Hadoop Map/Reduce
> Issue Type: Bug
> Components: applicationmaster
>Affects Versions: 0.23.3, 2.0.1-alpha
>Reporter: Jason Lowe
>Assignee: Jason Lowe
> Attachments: MAPREDUCE-4797.patch
>
>
> If LocalContainerAllocator has trouble communicating with the RM it can end
> up retrying forever if the nature of the error is not a YarnException.
> This can be particulary bad if the connection went down because the cluster
> was reset such that the RM and NM have lost track of the process and
> therefore nothing else will eventually kill the process. In this scenario,
> the looping AM continues to pelt the RM with connection requests every second
> using a stale token, and the RM logs the SASL exceptions over and over.
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (MAPREDUCE-4797) LocalContainerAllocator can loop forever trying to contact the RM
[ https://issues.apache.org/jira/browse/MAPREDUCE-4797?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13497493#comment-13497493 ] Robert Joseph Evans commented on MAPREDUCE-4797: The code looks good to me. My only comment is with the test. If a different exception is thrown, that exception is eaten and it is likely to be difficult to debug the error. If you could log it some how that would be great. > LocalContainerAllocator can loop forever trying to contact the RM > - > > Key: MAPREDUCE-4797 > URL: https://issues.apache.org/jira/browse/MAPREDUCE-4797 > Project: Hadoop Map/Reduce > Issue Type: Bug > Components: applicationmaster >Affects Versions: 0.23.3, 2.0.1-alpha >Reporter: Jason Lowe >Assignee: Jason Lowe > Attachments: MAPREDUCE-4797.patch > > > If LocalContainerAllocator has trouble communicating with the RM it can end > up retrying forever if the nature of the error is not a YarnException. > This can be particulary bad if the connection went down because the cluster > was reset such that the RM and NM have lost track of the process and > therefore nothing else will eventually kill the process. In this scenario, > the looping AM continues to pelt the RM with connection requests every second > using a stale token, and the RM logs the SASL exceptions over and over. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (MAPREDUCE-4797) LocalContainerAllocator can loop forever trying to contact the RM
[
https://issues.apache.org/jira/browse/MAPREDUCE-4797?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13496769#comment-13496769
]
Jason Lowe commented on MAPREDUCE-4797:
---
The code looks like it will only try to connect so many times before giving up,
but there's a bug in LocalContainerAllocator.heartbeat:
{code:title=LocalContainerAllocator.heartbeat}
AllocateResponse allocateResponse = scheduler.allocate(allocateRequest);
AMResponse response;
try {
response = allocateResponse.getAMResponse();
// Reset retry count if no exception occurred.
retrystartTime = System.currentTimeMillis();
} catch (Exception e) {
{code}
Note that the try block is surrounding the retrieval of the response *after*
the {{allocate}} RPC call, so we're missing where the exception is really being
thrown and not handling it here where it has retry count logic. The exception
then bubbles up to the RMCommunicator allocator thread where if the exception
isn't a {{YarnException}} then it simply loops around to try again, forever.
> LocalContainerAllocator can loop forever trying to contact the RM
> -
>
> Key: MAPREDUCE-4797
> URL: https://issues.apache.org/jira/browse/MAPREDUCE-4797
> Project: Hadoop Map/Reduce
> Issue Type: Bug
> Components: applicationmaster
>Affects Versions: 0.23.3, 2.0.1-alpha
>Reporter: Jason Lowe
>
> If LocalContainerAllocator has trouble communicating with the RM it can end
> up retrying forever if the nature of the error is not a YarnException.
> This can be particulary bad if the connection went down because the cluster
> was reset such that the RM and NM have lost track of the process and
> therefore nothing else will eventually kill the process. In this scenario,
> the looping AM continues to pelt the RM with connection requests every second
> using a stale token, and the RM logs the SASL exceptions over and over.
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators
For more information on JIRA, see: http://www.atlassian.com/software/jira
