[jira] [Commented] (YARN-1386) NodeManager mistakenly loses resources and relocalizes them
[ https://issues.apache.org/jira/browse/YARN-1386?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13821347#comment-13821347 ] Hudson commented on YARN-1386: -- SUCCESS: Integrated in Hadoop-Hdfs-trunk #1581 (See [https://builds.apache.org/job/Hadoop-Hdfs-trunk/1581/]) YARN-1386. NodeManager mistakenly loses resources and relocalizes them (Jason Lowe via jeagles) (jeagles: http://svn.apache.org/viewcvs.cgi/?root=Apache-SVN&view=rev&rev=1541375) * /hadoop/common/trunk/hadoop-yarn-project/CHANGES.txt * /hadoop/common/trunk/hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-nodemanager/src/main/java/org/apache/hadoop/yarn/server/nodemanager/containermanager/localizer/ContainerLocalizer.java * /hadoop/common/trunk/hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-nodemanager/src/test/java/org/apache/hadoop/yarn/server/nodemanager/containermanager/localizer/TestContainerLocalizer.java > NodeManager mistakenly loses resources and relocalizes them > --- > > Key: YARN-1386 > URL: https://issues.apache.org/jira/browse/YARN-1386 > Project: Hadoop YARN > Issue Type: Bug > Components: nodemanager >Affects Versions: 0.23.10, 2.2.0 >Reporter: Jason Lowe >Assignee: Jason Lowe >Priority: Critical > Fix For: 3.0.0, 2.3.0, 0.23.10 > > Attachments: YARN-1386.patch, YARN-1386.patch > > > When a local resource that should already be present is requested again, the > nodemanager checks to see if it still present. However the method it uses to > check for presence is via File.exists() as the user of the nodemanager > process. If the resource was a private resource localized for another user, > it will be localized to a location that is not accessible by the nodemanager > user. Therefore File.exists() returns false, the nodemanager mistakenly > believes the resource is no longer available, and it proceeds to localize it > over and over. -- This message was sent by Atlassian JIRA (v6.1#6144)
[jira] [Commented] (YARN-1386) NodeManager mistakenly loses resources and relocalizes them
[ https://issues.apache.org/jira/browse/YARN-1386?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13821324#comment-13821324 ] Hudson commented on YARN-1386: -- FAILURE: Integrated in Hadoop-Mapreduce-trunk #1607 (See [https://builds.apache.org/job/Hadoop-Mapreduce-trunk/1607/]) YARN-1386. NodeManager mistakenly loses resources and relocalizes them (Jason Lowe via jeagles) (jeagles: http://svn.apache.org/viewcvs.cgi/?root=Apache-SVN&view=rev&rev=1541375) * /hadoop/common/trunk/hadoop-yarn-project/CHANGES.txt * /hadoop/common/trunk/hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-nodemanager/src/main/java/org/apache/hadoop/yarn/server/nodemanager/containermanager/localizer/ContainerLocalizer.java * /hadoop/common/trunk/hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-nodemanager/src/test/java/org/apache/hadoop/yarn/server/nodemanager/containermanager/localizer/TestContainerLocalizer.java > NodeManager mistakenly loses resources and relocalizes them > --- > > Key: YARN-1386 > URL: https://issues.apache.org/jira/browse/YARN-1386 > Project: Hadoop YARN > Issue Type: Bug > Components: nodemanager >Affects Versions: 0.23.10, 2.2.0 >Reporter: Jason Lowe >Assignee: Jason Lowe >Priority: Critical > Fix For: 3.0.0, 2.3.0, 0.23.10 > > Attachments: YARN-1386.patch, YARN-1386.patch > > > When a local resource that should already be present is requested again, the > nodemanager checks to see if it still present. However the method it uses to > check for presence is via File.exists() as the user of the nodemanager > process. If the resource was a private resource localized for another user, > it will be localized to a location that is not accessible by the nodemanager > user. Therefore File.exists() returns false, the nodemanager mistakenly > believes the resource is no longer available, and it proceeds to localize it > over and over. -- This message was sent by Atlassian JIRA (v6.1#6144)
[jira] [Commented] (YARN-1386) NodeManager mistakenly loses resources and relocalizes them
[ https://issues.apache.org/jira/browse/YARN-1386?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13821242#comment-13821242 ] Hudson commented on YARN-1386: -- FAILURE: Integrated in Hadoop-Hdfs-0.23-Build #789 (See [https://builds.apache.org/job/Hadoop-Hdfs-0.23-Build/789/]) YARN-1386. NodeManager mistakenly loses resources and relocalizes them (Jason Lowe via jeagles) (jeagles: http://svn.apache.org/viewcvs.cgi/?root=Apache-SVN&view=rev&rev=1541378) * /hadoop/common/branches/branch-0.23/hadoop-yarn-project/CHANGES.txt * /hadoop/common/branches/branch-0.23/hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-nodemanager/src/main/java/org/apache/hadoop/yarn/server/nodemanager/containermanager/localizer/ContainerLocalizer.java * /hadoop/common/branches/branch-0.23/hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-nodemanager/src/test/java/org/apache/hadoop/yarn/server/nodemanager/containermanager/localizer/TestContainerLocalizer.java > NodeManager mistakenly loses resources and relocalizes them > --- > > Key: YARN-1386 > URL: https://issues.apache.org/jira/browse/YARN-1386 > Project: Hadoop YARN > Issue Type: Bug > Components: nodemanager >Affects Versions: 0.23.10, 2.2.0 >Reporter: Jason Lowe >Assignee: Jason Lowe >Priority: Critical > Fix For: 3.0.0, 2.3.0, 0.23.10 > > Attachments: YARN-1386.patch, YARN-1386.patch > > > When a local resource that should already be present is requested again, the > nodemanager checks to see if it still present. However the method it uses to > check for presence is via File.exists() as the user of the nodemanager > process. If the resource was a private resource localized for another user, > it will be localized to a location that is not accessible by the nodemanager > user. Therefore File.exists() returns false, the nodemanager mistakenly > believes the resource is no longer available, and it proceeds to localize it > over and over. -- This message was sent by Atlassian JIRA (v6.1#6144)
[jira] [Commented] (YARN-1386) NodeManager mistakenly loses resources and relocalizes them
[ https://issues.apache.org/jira/browse/YARN-1386?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13821212#comment-13821212 ] Hudson commented on YARN-1386: -- SUCCESS: Integrated in Hadoop-Yarn-trunk #390 (See [https://builds.apache.org/job/Hadoop-Yarn-trunk/390/]) YARN-1386. NodeManager mistakenly loses resources and relocalizes them (Jason Lowe via jeagles) (jeagles: http://svn.apache.org/viewcvs.cgi/?root=Apache-SVN&view=rev&rev=1541375) * /hadoop/common/trunk/hadoop-yarn-project/CHANGES.txt * /hadoop/common/trunk/hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-nodemanager/src/main/java/org/apache/hadoop/yarn/server/nodemanager/containermanager/localizer/ContainerLocalizer.java * /hadoop/common/trunk/hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-nodemanager/src/test/java/org/apache/hadoop/yarn/server/nodemanager/containermanager/localizer/TestContainerLocalizer.java > NodeManager mistakenly loses resources and relocalizes them > --- > > Key: YARN-1386 > URL: https://issues.apache.org/jira/browse/YARN-1386 > Project: Hadoop YARN > Issue Type: Bug > Components: nodemanager >Affects Versions: 0.23.10, 2.2.0 >Reporter: Jason Lowe >Assignee: Jason Lowe >Priority: Critical > Fix For: 3.0.0, 2.3.0, 0.23.10 > > Attachments: YARN-1386.patch, YARN-1386.patch > > > When a local resource that should already be present is requested again, the > nodemanager checks to see if it still present. However the method it uses to > check for presence is via File.exists() as the user of the nodemanager > process. If the resource was a private resource localized for another user, > it will be localized to a location that is not accessible by the nodemanager > user. Therefore File.exists() returns false, the nodemanager mistakenly > believes the resource is no longer available, and it proceeds to localize it > over and over. -- This message was sent by Atlassian JIRA (v6.1#6144)
[jira] [Commented] (YARN-1386) NodeManager mistakenly loses resources and relocalizes them
[ https://issues.apache.org/jira/browse/YARN-1386?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13820986#comment-13820986 ] Hudson commented on YARN-1386: -- SUCCESS: Integrated in Hadoop-trunk-Commit #4722 (See [https://builds.apache.org/job/Hadoop-trunk-Commit/4722/]) YARN-1386. NodeManager mistakenly loses resources and relocalizes them (Jason Lowe via jeagles) (jeagles: http://svn.apache.org/viewcvs.cgi/?root=Apache-SVN&view=rev&rev=1541375) * /hadoop/common/trunk/hadoop-yarn-project/CHANGES.txt * /hadoop/common/trunk/hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-nodemanager/src/main/java/org/apache/hadoop/yarn/server/nodemanager/containermanager/localizer/ContainerLocalizer.java * /hadoop/common/trunk/hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-nodemanager/src/test/java/org/apache/hadoop/yarn/server/nodemanager/containermanager/localizer/TestContainerLocalizer.java > NodeManager mistakenly loses resources and relocalizes them > --- > > Key: YARN-1386 > URL: https://issues.apache.org/jira/browse/YARN-1386 > Project: Hadoop YARN > Issue Type: Bug > Components: nodemanager >Affects Versions: 0.23.10, 2.2.0 >Reporter: Jason Lowe >Assignee: Jason Lowe >Priority: Critical > Fix For: 3.0.0, 2.3.0, 0.23.10 > > Attachments: YARN-1386.patch, YARN-1386.patch > > > When a local resource that should already be present is requested again, the > nodemanager checks to see if it still present. However the method it uses to > check for presence is via File.exists() as the user of the nodemanager > process. If the resource was a private resource localized for another user, > it will be localized to a location that is not accessible by the nodemanager > user. Therefore File.exists() returns false, the nodemanager mistakenly > believes the resource is no longer available, and it proceeds to localize it > over and over. -- This message was sent by Atlassian JIRA (v6.1#6144)
[jira] [Commented] (YARN-1386) NodeManager mistakenly loses resources and relocalizes them
[ https://issues.apache.org/jira/browse/YARN-1386?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13820682#comment-13820682 ] Jonathan Eagles commented on YARN-1386: --- +1. Great fix, Jason. > NodeManager mistakenly loses resources and relocalizes them > --- > > Key: YARN-1386 > URL: https://issues.apache.org/jira/browse/YARN-1386 > Project: Hadoop YARN > Issue Type: Bug > Components: nodemanager >Affects Versions: 0.23.10, 2.2.0 >Reporter: Jason Lowe >Assignee: Jason Lowe >Priority: Critical > Attachments: YARN-1386.patch, YARN-1386.patch > > > When a local resource that should already be present is requested again, the > nodemanager checks to see if it still present. However the method it uses to > check for presence is via File.exists() as the user of the nodemanager > process. If the resource was a private resource localized for another user, > it will be localized to a location that is not accessible by the nodemanager > user. Therefore File.exists() returns false, the nodemanager mistakenly > believes the resource is no longer available, and it proceeds to localize it > over and over. -- This message was sent by Atlassian JIRA (v6.1#6144)
[jira] [Commented] (YARN-1386) NodeManager mistakenly loses resources and relocalizes them
[
https://issues.apache.org/jira/browse/YARN-1386?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13820485#comment-13820485
]
Hadoop QA commented on YARN-1386:
-
{color:green}+1 overall{color}. Here are the results of testing the latest
attachment
http://issues.apache.org/jira/secure/attachment/12613421/YARN-1386.patch
against trunk revision .
{color:green}+1 @author{color}. The patch does not contain any @author
tags.
{color:green}+1 tests included{color}. The patch appears to include 1 new
or modified test files.
{color:green}+1 javac{color}. The applied patch does not increase the
total number of javac compiler warnings.
{color:green}+1 javadoc{color}. The javadoc tool did not generate any
warning messages.
{color:green}+1 eclipse:eclipse{color}. The patch built with
eclipse:eclipse.
{color:green}+1 findbugs{color}. The patch does not introduce any new
Findbugs (version 1.3.9) warnings.
{color:green}+1 release audit{color}. The applied patch does not increase
the total number of release audit warnings.
{color:green}+1 core tests{color}. The patch passed unit tests in
hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-nodemanager.
{color:green}+1 contrib tests{color}. The patch passed contrib unit tests.
Test results:
https://builds.apache.org/job/PreCommit-YARN-Build/2426//testReport/
Console output: https://builds.apache.org/job/PreCommit-YARN-Build/2426//console
This message is automatically generated.
> NodeManager mistakenly loses resources and relocalizes them
> ---
>
> Key: YARN-1386
> URL: https://issues.apache.org/jira/browse/YARN-1386
> Project: Hadoop YARN
> Issue Type: Bug
> Components: nodemanager
>Affects Versions: 0.23.10, 2.2.0
>Reporter: Jason Lowe
>Assignee: Jason Lowe
>Priority: Critical
> Attachments: YARN-1386.patch, YARN-1386.patch
>
>
> When a local resource that should already be present is requested again, the
> nodemanager checks to see if it still present. However the method it uses to
> check for presence is via File.exists() as the user of the nodemanager
> process. If the resource was a private resource localized for another user,
> it will be localized to a location that is not accessible by the nodemanager
> user. Therefore File.exists() returns false, the nodemanager mistakenly
> believes the resource is no longer available, and it proceeds to localize it
> over and over.
--
This message was sent by Atlassian JIRA
(v6.1#6144)
[jira] [Commented] (YARN-1386) NodeManager mistakenly loses resources and relocalizes them
[ https://issues.apache.org/jira/browse/YARN-1386?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13819557#comment-13819557 ] Jason Lowe commented on YARN-1386: -- Sigh, the hadoop7 (uberjake42) Jenkins machine is apparently full of runaway threads and can't perform a build anymore, so the failure is unrelated to the patch. Will try to find someone who has access to the machine to clean it up and then rekick the build later. > NodeManager mistakenly loses resources and relocalizes them > --- > > Key: YARN-1386 > URL: https://issues.apache.org/jira/browse/YARN-1386 > Project: Hadoop YARN > Issue Type: Bug > Components: nodemanager >Affects Versions: 0.23.10, 2.2.0 >Reporter: Jason Lowe >Assignee: Jason Lowe >Priority: Critical > Attachments: YARN-1386.patch > > > When a local resource that should already be present is requested again, the > nodemanager checks to see if it still present. However the method it uses to > check for presence is via File.exists() as the user of the nodemanager > process. If the resource was a private resource localized for another user, > it will be localized to a location that is not accessible by the nodemanager > user. Therefore File.exists() returns false, the nodemanager mistakenly > believes the resource is no longer available, and it proceeds to localize it > over and over. -- This message was sent by Atlassian JIRA (v6.1#6144)
[jira] [Commented] (YARN-1386) NodeManager mistakenly loses resources and relocalizes them
[
https://issues.apache.org/jira/browse/YARN-1386?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13819519#comment-13819519
]
Hadoop QA commented on YARN-1386:
-
{color:red}-1 overall{color}. Here are the results of testing the latest
attachment
http://issues.apache.org/jira/secure/attachment/12613245/YARN-1386.patch
against trunk revision .
{color:green}+1 @author{color}. The patch does not contain any @author
tags.
{color:green}+1 tests included{color}. The patch appears to include 1 new
or modified test files.
{color:red}-1 javac{color:red}. The patch appears to cause the build to
fail.
Console output: https://builds.apache.org/job/PreCommit-YARN-Build/2418//console
This message is automatically generated.
> NodeManager mistakenly loses resources and relocalizes them
> ---
>
> Key: YARN-1386
> URL: https://issues.apache.org/jira/browse/YARN-1386
> Project: Hadoop YARN
> Issue Type: Bug
> Components: nodemanager
>Affects Versions: 0.23.10, 2.2.0
>Reporter: Jason Lowe
>Assignee: Jason Lowe
>Priority: Critical
> Attachments: YARN-1386.patch
>
>
> When a local resource that should already be present is requested again, the
> nodemanager checks to see if it still present. However the method it uses to
> check for presence is via File.exists() as the user of the nodemanager
> process. If the resource was a private resource localized for another user,
> it will be localized to a location that is not accessible by the nodemanager
> user. Therefore File.exists() returns false, the nodemanager mistakenly
> believes the resource is no longer available, and it proceeds to localize it
> over and over.
--
This message was sent by Atlassian JIRA
(v6.1#6144)
