[jira] [Commented] (YARN-2244) FairScheduler missing handling of containers for unknown application attempts
[ https://issues.apache.org/jira/browse/YARN-2244?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14067476#comment-14067476 ] Hudson commented on YARN-2244: -- FAILURE: Integrated in Hadoop-Yarn-trunk #617 (See [https://builds.apache.org/job/Hadoop-Yarn-trunk/617/]) YARN-2244. FairScheduler missing handling of containers for unknown application attempts. (Anubhav Dhoot via kasha) (kasha: http://svn.apache.org/viewcvs.cgi/?root=Apache-SVNview=revrev=1611840) * /hadoop/common/trunk/hadoop-yarn-project/CHANGES.txt * /hadoop/common/trunk/hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager/src/main/java/org/apache/hadoop/yarn/server/resourcemanager/scheduler/AbstractYarnScheduler.java * /hadoop/common/trunk/hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager/src/main/java/org/apache/hadoop/yarn/server/resourcemanager/scheduler/capacity/CapacityScheduler.java * /hadoop/common/trunk/hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager/src/main/java/org/apache/hadoop/yarn/server/resourcemanager/scheduler/fair/FairScheduler.java * /hadoop/common/trunk/hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager/src/main/java/org/apache/hadoop/yarn/server/resourcemanager/scheduler/fifo/FifoScheduler.java * /hadoop/common/trunk/hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager/src/test/java/org/apache/hadoop/yarn/server/resourcemanager/TestApplicationCleanup.java FairScheduler missing handling of containers for unknown application attempts -- Key: YARN-2244 URL: https://issues.apache.org/jira/browse/YARN-2244 Project: Hadoop YARN Issue Type: Bug Components: fairscheduler Reporter: Anubhav Dhoot Assignee: Anubhav Dhoot Priority: Critical Fix For: 2.6.0 Attachments: YARN-2224.patch, YARN-2244.001.patch, YARN-2244.002.patch, YARN-2244.003.patch, YARN-2244.004.patch, YARN-2244.005.patch, YARN-2244.005.patch We are missing changes in patch MAPREDUCE-3596 in FairScheduler. Among other fixes that were common across schedulers, there were some scheduler specific fixes added to handle containers for unknown application attempts. Without these fair scheduler simply logs that an unknown container was found and continues to let it run. -- This message was sent by Atlassian JIRA (v6.2#6252)
[jira] [Commented] (YARN-2244) FairScheduler missing handling of containers for unknown application attempts
[ https://issues.apache.org/jira/browse/YARN-2244?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14067525#comment-14067525 ] Hudson commented on YARN-2244: -- FAILURE: Integrated in Hadoop-Hdfs-trunk #1809 (See [https://builds.apache.org/job/Hadoop-Hdfs-trunk/1809/]) YARN-2244. FairScheduler missing handling of containers for unknown application attempts. (Anubhav Dhoot via kasha) (kasha: http://svn.apache.org/viewcvs.cgi/?root=Apache-SVNview=revrev=1611840) * /hadoop/common/trunk/hadoop-yarn-project/CHANGES.txt * /hadoop/common/trunk/hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager/src/main/java/org/apache/hadoop/yarn/server/resourcemanager/scheduler/AbstractYarnScheduler.java * /hadoop/common/trunk/hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager/src/main/java/org/apache/hadoop/yarn/server/resourcemanager/scheduler/capacity/CapacityScheduler.java * /hadoop/common/trunk/hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager/src/main/java/org/apache/hadoop/yarn/server/resourcemanager/scheduler/fair/FairScheduler.java * /hadoop/common/trunk/hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager/src/main/java/org/apache/hadoop/yarn/server/resourcemanager/scheduler/fifo/FifoScheduler.java * /hadoop/common/trunk/hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager/src/test/java/org/apache/hadoop/yarn/server/resourcemanager/TestApplicationCleanup.java FairScheduler missing handling of containers for unknown application attempts -- Key: YARN-2244 URL: https://issues.apache.org/jira/browse/YARN-2244 Project: Hadoop YARN Issue Type: Bug Components: fairscheduler Reporter: Anubhav Dhoot Assignee: Anubhav Dhoot Priority: Critical Fix For: 2.6.0 Attachments: YARN-2224.patch, YARN-2244.001.patch, YARN-2244.002.patch, YARN-2244.003.patch, YARN-2244.004.patch, YARN-2244.005.patch, YARN-2244.005.patch We are missing changes in patch MAPREDUCE-3596 in FairScheduler. Among other fixes that were common across schedulers, there were some scheduler specific fixes added to handle containers for unknown application attempts. Without these fair scheduler simply logs that an unknown container was found and continues to let it run. -- This message was sent by Atlassian JIRA (v6.2#6252)
[jira] [Commented] (YARN-2244) FairScheduler missing handling of containers for unknown application attempts
[ https://issues.apache.org/jira/browse/YARN-2244?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14067549#comment-14067549 ] Hudson commented on YARN-2244: -- SUCCESS: Integrated in Hadoop-Mapreduce-trunk #1836 (See [https://builds.apache.org/job/Hadoop-Mapreduce-trunk/1836/]) YARN-2244. FairScheduler missing handling of containers for unknown application attempts. (Anubhav Dhoot via kasha) (kasha: http://svn.apache.org/viewcvs.cgi/?root=Apache-SVNview=revrev=1611840) * /hadoop/common/trunk/hadoop-yarn-project/CHANGES.txt * /hadoop/common/trunk/hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager/src/main/java/org/apache/hadoop/yarn/server/resourcemanager/scheduler/AbstractYarnScheduler.java * /hadoop/common/trunk/hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager/src/main/java/org/apache/hadoop/yarn/server/resourcemanager/scheduler/capacity/CapacityScheduler.java * /hadoop/common/trunk/hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager/src/main/java/org/apache/hadoop/yarn/server/resourcemanager/scheduler/fair/FairScheduler.java * /hadoop/common/trunk/hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager/src/main/java/org/apache/hadoop/yarn/server/resourcemanager/scheduler/fifo/FifoScheduler.java * /hadoop/common/trunk/hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager/src/test/java/org/apache/hadoop/yarn/server/resourcemanager/TestApplicationCleanup.java FairScheduler missing handling of containers for unknown application attempts -- Key: YARN-2244 URL: https://issues.apache.org/jira/browse/YARN-2244 Project: Hadoop YARN Issue Type: Bug Components: fairscheduler Reporter: Anubhav Dhoot Assignee: Anubhav Dhoot Priority: Critical Fix For: 2.6.0 Attachments: YARN-2224.patch, YARN-2244.001.patch, YARN-2244.002.patch, YARN-2244.003.patch, YARN-2244.004.patch, YARN-2244.005.patch, YARN-2244.005.patch We are missing changes in patch MAPREDUCE-3596 in FairScheduler. Among other fixes that were common across schedulers, there were some scheduler specific fixes added to handle containers for unknown application attempts. Without these fair scheduler simply logs that an unknown container was found and continues to let it run. -- This message was sent by Atlassian JIRA (v6.2#6252)
[jira] [Commented] (YARN-2244) FairScheduler missing handling of containers for unknown application attempts
[
https://issues.apache.org/jira/browse/YARN-2244?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14066712#comment-14066712
]
Hadoop QA commented on YARN-2244:
-
{color:red}-1 overall{color}. Here are the results of testing the latest
attachment
http://issues.apache.org/jira/secure/attachment/12656543/YARN-2244.005.patch
against trunk revision .
{color:green}+1 @author{color}. The patch does not contain any @author
tags.
{color:green}+1 tests included{color}. The patch appears to include 1 new
or modified test files.
{color:green}+1 javac{color}. The applied patch does not increase the
total number of javac compiler warnings.
{color:green}+1 javadoc{color}. There were no new javadoc warning messages.
{color:green}+1 eclipse:eclipse{color}. The patch built with
eclipse:eclipse.
{color:green}+1 findbugs{color}. The patch does not introduce any new
Findbugs (version 2.0.3) warnings.
{color:green}+1 release audit{color}. The applied patch does not increase
the total number of release audit warnings.
{color:red}-1 core tests{color}. The patch failed these unit tests in
hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager:
org.apache.hadoop.yarn.server.resourcemanager.webapp.TestRMWebServices
org.apache.hadoop.yarn.server.resourcemanager.webapp.TestRMWebServicesCapacitySched
org.apache.hadoop.yarn.server.resourcemanager.webapp.TestRMWebServicesDelegationTokens
org.apache.hadoop.yarn.server.resourcemanager.webapp.TestRMWebServicesAppsModification
org.apache.hadoop.yarn.server.resourcemanager.webapp.TestRMWebServicesNodes
org.apache.hadoop.yarn.server.resourcemanager.webapp.TestRMWebServicesApps
org.apache.hadoop.yarn.server.resourcemanager.webapp.TestRMWebServicesFairScheduler
{color:green}+1 contrib tests{color}. The patch passed contrib unit tests.
Test results:
https://builds.apache.org/job/PreCommit-YARN-Build/4362//testReport/
Console output: https://builds.apache.org/job/PreCommit-YARN-Build/4362//console
This message is automatically generated.
FairScheduler missing handling of containers for unknown application attempts
--
Key: YARN-2244
URL: https://issues.apache.org/jira/browse/YARN-2244
Project: Hadoop YARN
Issue Type: Bug
Components: fairscheduler
Reporter: Anubhav Dhoot
Assignee: Anubhav Dhoot
Priority: Critical
Attachments: YARN-2224.patch, YARN-2244.001.patch,
YARN-2244.002.patch, YARN-2244.003.patch, YARN-2244.004.patch,
YARN-2244.005.patch
We are missing changes in patch MAPREDUCE-3596 in FairScheduler. Among other
fixes that were common across schedulers, there were some scheduler specific
fixes added to handle containers for unknown application attempts. Without
these fair scheduler simply logs that an unknown container was found and
continues to let it run.
--
This message was sent by Atlassian JIRA
(v6.2#6252)
[jira] [Commented] (YARN-2244) FairScheduler missing handling of containers for unknown application attempts
[ https://issues.apache.org/jira/browse/YARN-2244?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14066719#comment-14066719 ] Karthik Kambatla commented on YARN-2244: [~adhoot] - can you check if the test failures are related? FairScheduler missing handling of containers for unknown application attempts -- Key: YARN-2244 URL: https://issues.apache.org/jira/browse/YARN-2244 Project: Hadoop YARN Issue Type: Bug Components: fairscheduler Reporter: Anubhav Dhoot Assignee: Anubhav Dhoot Priority: Critical Attachments: YARN-2224.patch, YARN-2244.001.patch, YARN-2244.002.patch, YARN-2244.003.patch, YARN-2244.004.patch, YARN-2244.005.patch We are missing changes in patch MAPREDUCE-3596 in FairScheduler. Among other fixes that were common across schedulers, there were some scheduler specific fixes added to handle containers for unknown application attempts. Without these fair scheduler simply logs that an unknown container was found and continues to let it run. -- This message was sent by Atlassian JIRA (v6.2#6252)
[jira] [Commented] (YARN-2244) FairScheduler missing handling of containers for unknown application attempts
[ https://issues.apache.org/jira/browse/YARN-2244?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14066938#comment-14066938 ] Anubhav Dhoot commented on YARN-2244: - Seems unrelated . Most failures were with port binding issues com.sun.jersey.test.framework.spi.container.TestContainerException: java.net.BindException: Address already in use Will trigger a retest FairScheduler missing handling of containers for unknown application attempts -- Key: YARN-2244 URL: https://issues.apache.org/jira/browse/YARN-2244 Project: Hadoop YARN Issue Type: Bug Components: fairscheduler Reporter: Anubhav Dhoot Assignee: Anubhav Dhoot Priority: Critical Attachments: YARN-2224.patch, YARN-2244.001.patch, YARN-2244.002.patch, YARN-2244.003.patch, YARN-2244.004.patch, YARN-2244.005.patch, YARN-2244.005.patch We are missing changes in patch MAPREDUCE-3596 in FairScheduler. Among other fixes that were common across schedulers, there were some scheduler specific fixes added to handle containers for unknown application attempts. Without these fair scheduler simply logs that an unknown container was found and continues to let it run. -- This message was sent by Atlassian JIRA (v6.2#6252)
[jira] [Commented] (YARN-2244) FairScheduler missing handling of containers for unknown application attempts
[
https://issues.apache.org/jira/browse/YARN-2244?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14067050#comment-14067050
]
Hadoop QA commented on YARN-2244:
-
{color:green}+1 overall{color}. Here are the results of testing the latest
attachment
http://issues.apache.org/jira/secure/attachment/12656583/YARN-2244.005.patch
against trunk revision .
{color:green}+1 @author{color}. The patch does not contain any @author
tags.
{color:green}+1 tests included{color}. The patch appears to include 1 new
or modified test files.
{color:green}+1 javac{color}. The applied patch does not increase the
total number of javac compiler warnings.
{color:green}+1 javadoc{color}. There were no new javadoc warning messages.
{color:green}+1 eclipse:eclipse{color}. The patch built with
eclipse:eclipse.
{color:green}+1 findbugs{color}. The patch does not introduce any new
Findbugs (version 2.0.3) warnings.
{color:green}+1 release audit{color}. The applied patch does not increase
the total number of release audit warnings.
{color:green}+1 core tests{color}. The patch passed unit tests in
hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager.
{color:green}+1 contrib tests{color}. The patch passed contrib unit tests.
Test results:
https://builds.apache.org/job/PreCommit-YARN-Build/4365//testReport/
Console output: https://builds.apache.org/job/PreCommit-YARN-Build/4365//console
This message is automatically generated.
FairScheduler missing handling of containers for unknown application attempts
--
Key: YARN-2244
URL: https://issues.apache.org/jira/browse/YARN-2244
Project: Hadoop YARN
Issue Type: Bug
Components: fairscheduler
Reporter: Anubhav Dhoot
Assignee: Anubhav Dhoot
Priority: Critical
Attachments: YARN-2224.patch, YARN-2244.001.patch,
YARN-2244.002.patch, YARN-2244.003.patch, YARN-2244.004.patch,
YARN-2244.005.patch, YARN-2244.005.patch
We are missing changes in patch MAPREDUCE-3596 in FairScheduler. Among other
fixes that were common across schedulers, there were some scheduler specific
fixes added to handle containers for unknown application attempts. Without
these fair scheduler simply logs that an unknown container was found and
continues to let it run.
--
This message was sent by Atlassian JIRA
(v6.2#6252)
[jira] [Commented] (YARN-2244) FairScheduler missing handling of containers for unknown application attempts
[ https://issues.apache.org/jira/browse/YARN-2244?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14067241#comment-14067241 ] Karthik Kambatla commented on YARN-2244: Latest patch looks good to me. +1. FairScheduler missing handling of containers for unknown application attempts -- Key: YARN-2244 URL: https://issues.apache.org/jira/browse/YARN-2244 Project: Hadoop YARN Issue Type: Bug Components: fairscheduler Reporter: Anubhav Dhoot Assignee: Anubhav Dhoot Priority: Critical Attachments: YARN-2224.patch, YARN-2244.001.patch, YARN-2244.002.patch, YARN-2244.003.patch, YARN-2244.004.patch, YARN-2244.005.patch, YARN-2244.005.patch We are missing changes in patch MAPREDUCE-3596 in FairScheduler. Among other fixes that were common across schedulers, there were some scheduler specific fixes added to handle containers for unknown application attempts. Without these fair scheduler simply logs that an unknown container was found and continues to let it run. -- This message was sent by Atlassian JIRA (v6.2#6252)
[jira] [Commented] (YARN-2244) FairScheduler missing handling of containers for unknown application attempts
[ https://issues.apache.org/jira/browse/YARN-2244?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14067242#comment-14067242 ] Karthik Kambatla commented on YARN-2244: Committing this. FairScheduler missing handling of containers for unknown application attempts -- Key: YARN-2244 URL: https://issues.apache.org/jira/browse/YARN-2244 Project: Hadoop YARN Issue Type: Bug Components: fairscheduler Reporter: Anubhav Dhoot Assignee: Anubhav Dhoot Priority: Critical Attachments: YARN-2224.patch, YARN-2244.001.patch, YARN-2244.002.patch, YARN-2244.003.patch, YARN-2244.004.patch, YARN-2244.005.patch, YARN-2244.005.patch We are missing changes in patch MAPREDUCE-3596 in FairScheduler. Among other fixes that were common across schedulers, there were some scheduler specific fixes added to handle containers for unknown application attempts. Without these fair scheduler simply logs that an unknown container was found and continues to let it run. -- This message was sent by Atlassian JIRA (v6.2#6252)
[jira] [Commented] (YARN-2244) FairScheduler missing handling of containers for unknown application attempts
[ https://issues.apache.org/jira/browse/YARN-2244?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14067328#comment-14067328 ] Hudson commented on YARN-2244: -- SUCCESS: Integrated in Hadoop-trunk-Commit #5920 (See [https://builds.apache.org/job/Hadoop-trunk-Commit/5920/]) YARN-2244. FairScheduler missing handling of containers for unknown application attempts. (Anubhav Dhoot via kasha) (kasha: http://svn.apache.org/viewcvs.cgi/?root=Apache-SVNview=revrev=1611840) * /hadoop/common/trunk/hadoop-yarn-project/CHANGES.txt * /hadoop/common/trunk/hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager/src/main/java/org/apache/hadoop/yarn/server/resourcemanager/scheduler/AbstractYarnScheduler.java * /hadoop/common/trunk/hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager/src/main/java/org/apache/hadoop/yarn/server/resourcemanager/scheduler/capacity/CapacityScheduler.java * /hadoop/common/trunk/hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager/src/main/java/org/apache/hadoop/yarn/server/resourcemanager/scheduler/fair/FairScheduler.java * /hadoop/common/trunk/hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager/src/main/java/org/apache/hadoop/yarn/server/resourcemanager/scheduler/fifo/FifoScheduler.java * /hadoop/common/trunk/hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager/src/test/java/org/apache/hadoop/yarn/server/resourcemanager/TestApplicationCleanup.java FairScheduler missing handling of containers for unknown application attempts -- Key: YARN-2244 URL: https://issues.apache.org/jira/browse/YARN-2244 Project: Hadoop YARN Issue Type: Bug Components: fairscheduler Reporter: Anubhav Dhoot Assignee: Anubhav Dhoot Priority: Critical Fix For: 2.6.0 Attachments: YARN-2224.patch, YARN-2244.001.patch, YARN-2244.002.patch, YARN-2244.003.patch, YARN-2244.004.patch, YARN-2244.005.patch, YARN-2244.005.patch We are missing changes in patch MAPREDUCE-3596 in FairScheduler. Among other fixes that were common across schedulers, there were some scheduler specific fixes added to handle containers for unknown application attempts. Without these fair scheduler simply logs that an unknown container was found and continues to let it run. -- This message was sent by Atlassian JIRA (v6.2#6252)
[jira] [Commented] (YARN-2244) FairScheduler missing handling of containers for unknown application attempts
[
https://issues.apache.org/jira/browse/YARN-2244?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14066089#comment-14066089
]
Karthik Kambatla commented on YARN-2244:
Latest patch looks good. A couple of nits:
# The 80 chars limit doesn't apply to imports - can we get them one per line?
{code}
+import org.apache.hadoop.yarn.server.resourcemanager.scheduler.capacity
+.CapacityScheduler;
+import org.apache.hadoop.yarn.server.resourcemanager.scheduler.common.fica
+.FiCaSchedulerApp;
{code}
# Limit the following line to 80 chars.
{code}
+SchedulerApplicationAttempt application =
getCurrentAttemptForContainer(containerId);
{code}
# Unused imports in CapacityScheduler, FairScheduler, FifoScheduler.
# Few comments on the do-while loop in the test:
{code}
int waitCount = 0;
dispatcher.await();
ListContainerId contsToClean;
int cleanedConts = 0;
do {
contsToClean = resp.getContainersToCleanup();
cleanedConts += contsToClean.size();
if (cleanedConts = 1) {
break;
}
Thread.sleep(100);
resp = nm.nodeHeartbeat(true);
dispatcher.await();
}
while(waitCount++ 200);
{code}
## Define waitCount and cleanedCounts on the same line?
## while should be on the same line as the closing brace.
## Remove dispatcher.await() outside the loop and have it as the first
statement in the do-block?
FairScheduler missing handling of containers for unknown application attempts
--
Key: YARN-2244
URL: https://issues.apache.org/jira/browse/YARN-2244
Project: Hadoop YARN
Issue Type: Bug
Components: fairscheduler
Reporter: Anubhav Dhoot
Assignee: Anubhav Dhoot
Priority: Critical
Attachments: YARN-2224.patch, YARN-2244.001.patch,
YARN-2244.002.patch, YARN-2244.003.patch, YARN-2244.004.patch
We are missing changes in patch MAPREDUCE-3596 in FairScheduler. Among other
fixes that were common across schedulers, there were some scheduler specific
fixes added to handle containers for unknown application attempts. Without
these fair scheduler simply logs that an unknown container was found and
continues to let it run.
--
This message was sent by Atlassian JIRA
(v6.2#6252)
[jira] [Commented] (YARN-2244) FairScheduler missing handling of containers for unknown application attempts
[
https://issues.apache.org/jira/browse/YARN-2244?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14064275#comment-14064275
]
Anubhav Dhoot commented on YARN-2244:
-
Fixed other issues except
Can we use AbstractYarnScheduler#killOrphanContainerOnNode() instead
Right now the {{containerLaunchedOnNode}} code is same as Capacity and Fifo
with what it does. If we want to unify the code across the schedulers, i would
prefer leave it like this and then get rid of all three implementations and
have a single {{containerLaunchedOnNode}} in AbstractYarnScheduler instead.
Parametrize the method to also take number of container cleanups to wait for
and use it everywhere.
The only other user in this class has some additional logic for application
cleanup. Unifying across those might make it more complicated than its worth.
FairScheduler missing handling of containers for unknown application attempts
--
Key: YARN-2244
URL: https://issues.apache.org/jira/browse/YARN-2244
Project: Hadoop YARN
Issue Type: Bug
Components: fairscheduler
Reporter: Anubhav Dhoot
Assignee: Anubhav Dhoot
Priority: Critical
Attachments: YARN-2224.patch, YARN-2244.001.patch, YARN-2244.002.patch
We are missing changes in patch MAPREDUCE-3596 in FairScheduler. Among other
fixes that were common across schedulers, there were some scheduler specific
fixes added to handle containers for unknown application attempts. Without
these fair scheduler simply logs that an unknown container was found and
continues to let it run.
--
This message was sent by Atlassian JIRA
(v6.2#6252)
[jira] [Commented] (YARN-2244) FairScheduler missing handling of containers for unknown application attempts
[ https://issues.apache.org/jira/browse/YARN-2244?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14064285#comment-14064285 ] Karthik Kambatla commented on YARN-2244: bq. Right now the containerLaunchedOnNode code is same as Capacity and Fifo with what it does. If we want to unify the code across the schedulers, i would prefer leave it like this and then get rid of all three implementations and have a single containerLaunchedOnNode in AbstractYarnScheduler instead. I am not sure if we should punt on re-using existing code to match other schedulers. I am open to merging containerLaunchedOnNode across schedulers into AbstractYarnScheduler. That is definitely a better solution for this JIRA as well. FairScheduler missing handling of containers for unknown application attempts -- Key: YARN-2244 URL: https://issues.apache.org/jira/browse/YARN-2244 Project: Hadoop YARN Issue Type: Bug Components: fairscheduler Reporter: Anubhav Dhoot Assignee: Anubhav Dhoot Priority: Critical Attachments: YARN-2224.patch, YARN-2244.001.patch, YARN-2244.002.patch We are missing changes in patch MAPREDUCE-3596 in FairScheduler. Among other fixes that were common across schedulers, there were some scheduler specific fixes added to handle containers for unknown application attempts. Without these fair scheduler simply logs that an unknown container was found and continues to let it run. -- This message was sent by Atlassian JIRA (v6.2#6252)
[jira] [Commented] (YARN-2244) FairScheduler missing handling of containers for unknown application attempts
[ https://issues.apache.org/jira/browse/YARN-2244?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14064305#comment-14064305 ] Anubhav Dhoot commented on YARN-2244: - Uploading a new patch with code merged across the schedulers FairScheduler missing handling of containers for unknown application attempts -- Key: YARN-2244 URL: https://issues.apache.org/jira/browse/YARN-2244 Project: Hadoop YARN Issue Type: Bug Components: fairscheduler Reporter: Anubhav Dhoot Assignee: Anubhav Dhoot Priority: Critical Attachments: YARN-2224.patch, YARN-2244.001.patch, YARN-2244.002.patch, YARN-2244.003.patch We are missing changes in patch MAPREDUCE-3596 in FairScheduler. Among other fixes that were common across schedulers, there were some scheduler specific fixes added to handle containers for unknown application attempts. Without these fair scheduler simply logs that an unknown container was found and continues to let it run. -- This message was sent by Atlassian JIRA (v6.2#6252)
[jira] [Commented] (YARN-2244) FairScheduler missing handling of containers for unknown application attempts
[
https://issues.apache.org/jira/browse/YARN-2244?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14064410#comment-14064410
]
Hadoop QA commented on YARN-2244:
-
{color:green}+1 overall{color}. Here are the results of testing the latest
attachment
http://issues.apache.org/jira/secure/attachment/12656175/YARN-2244.004.patch
against trunk revision .
{color:green}+1 @author{color}. The patch does not contain any @author
tags.
{color:green}+1 tests included{color}. The patch appears to include 1 new
or modified test files.
{color:green}+1 javac{color}. The applied patch does not increase the
total number of javac compiler warnings.
{color:green}+1 javadoc{color}. There were no new javadoc warning messages.
{color:green}+1 eclipse:eclipse{color}. The patch built with
eclipse:eclipse.
{color:green}+1 findbugs{color}. The patch does not introduce any new
Findbugs (version 2.0.3) warnings.
{color:green}+1 release audit{color}. The applied patch does not increase
the total number of release audit warnings.
{color:green}+1 core tests{color}. The patch passed unit tests in
hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager.
{color:green}+1 contrib tests{color}. The patch passed contrib unit tests.
Test results:
https://builds.apache.org/job/PreCommit-YARN-Build/4329//testReport/
Console output: https://builds.apache.org/job/PreCommit-YARN-Build/4329//console
This message is automatically generated.
FairScheduler missing handling of containers for unknown application attempts
--
Key: YARN-2244
URL: https://issues.apache.org/jira/browse/YARN-2244
Project: Hadoop YARN
Issue Type: Bug
Components: fairscheduler
Reporter: Anubhav Dhoot
Assignee: Anubhav Dhoot
Priority: Critical
Attachments: YARN-2224.patch, YARN-2244.001.patch,
YARN-2244.002.patch, YARN-2244.003.patch, YARN-2244.004.patch
We are missing changes in patch MAPREDUCE-3596 in FairScheduler. Among other
fixes that were common across schedulers, there were some scheduler specific
fixes added to handle containers for unknown application attempts. Without
these fair scheduler simply logs that an unknown container was found and
continues to let it run.
--
This message was sent by Atlassian JIRA
(v6.2#6252)
[jira] [Commented] (YARN-2244) FairScheduler missing handling of containers for unknown application attempts
[
https://issues.apache.org/jira/browse/YARN-2244?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14061387#comment-14061387
]
Karthik Kambatla commented on YARN-2244:
# Can we use {{AbstractYarnScheduler#killOrphanContainerOnNode()}} instead?
{code}
+ this.rmContext.getDispatcher().getEventHandler()
+ .handle(new RMNodeCleanContainerEvent(node.getNodeID(),
containerId));
{code}
# Thanks for moving the following to a separate method. IMO, we should clean it
up more:
{code}
protected void waitForContainerCleanup(DrainDispatcher dispatcher, MockNM nm,
NodeHeartbeatResponse resp) throws
Exception {
int waitCount;
dispatcher.await();
ListContainerId contsToClean = resp.getContainersToCleanup();
int cleanedConts = contsToClean.size();
waitCount = 0;
while (cleanedConts 1 waitCount++ 200) {
LOG.info(Waiting to get cleanup events.. cleanedConts: + cleanedConts);
Thread.sleep(100);
resp = nm.nodeHeartbeat(true);
dispatcher.await();
contsToClean = resp.getContainersToCleanup();
cleanedConts += contsToClean.size();
}
if (contsToClean.isEmpty()) {
LOG.error(Failed to get any containers to cleanup);
} else {
LOG.info(Got cleanup for + contsToClean.get(0));
}
Assert.assertEquals(1, cleanedConts);
}
{code}
## One line over 80 chars
## {{int waitCount = 0}} can go on oneline
## Fetching containers to clean and other arithmetic before the while loop can
be moved into the while loop. cleanedConts can be initialized to zero. I am
okay with a do-while too.
## Remove the logging - I am not sure why are we logging that information 200
times.
## Parametrize the method to also take number of container cleanups to wait for
and use it everywhere.
FairScheduler missing handling of containers for unknown application attempts
--
Key: YARN-2244
URL: https://issues.apache.org/jira/browse/YARN-2244
Project: Hadoop YARN
Issue Type: Bug
Components: fairscheduler
Reporter: Anubhav Dhoot
Assignee: Anubhav Dhoot
Priority: Critical
Attachments: YARN-2224.patch, YARN-2244.001.patch, YARN-2244.002.patch
We are missing changes in patch MAPREDUCE-3596 in FairScheduler. Among other
fixes that were common across schedulers, there were some scheduler specific
fixes added to handle containers for unknown application attempts. Without
these fair scheduler simply logs that an unknown container was found and
continues to let it run.
--
This message was sent by Atlassian JIRA
(v6.2#6252)
[jira] [Commented] (YARN-2244) FairScheduler missing handling of containers for unknown application attempts
[
https://issues.apache.org/jira/browse/YARN-2244?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14058128#comment-14058128
]
Hadoop QA commented on YARN-2244:
-
{color:green}+1 overall{color}. Here are the results of testing the latest
attachment
http://issues.apache.org/jira/secure/attachment/12655104/YARN-2244.002.patch
against trunk revision .
{color:green}+1 @author{color}. The patch does not contain any @author
tags.
{color:green}+1 tests included{color}. The patch appears to include 1 new
or modified test files.
{color:green}+1 javac{color}. The applied patch does not increase the
total number of javac compiler warnings.
{color:green}+1 javadoc{color}. There were no new javadoc warning messages.
{color:green}+1 eclipse:eclipse{color}. The patch built with
eclipse:eclipse.
{color:green}+1 findbugs{color}. The patch does not introduce any new
Findbugs (version 2.0.3) warnings.
{color:green}+1 release audit{color}. The applied patch does not increase
the total number of release audit warnings.
{color:green}+1 core tests{color}. The patch passed unit tests in
hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager.
{color:green}+1 contrib tests{color}. The patch passed contrib unit tests.
Test results:
https://builds.apache.org/job/PreCommit-YARN-Build/4264//testReport/
Console output: https://builds.apache.org/job/PreCommit-YARN-Build/4264//console
This message is automatically generated.
FairScheduler missing handling of containers for unknown application attempts
--
Key: YARN-2244
URL: https://issues.apache.org/jira/browse/YARN-2244
Project: Hadoop YARN
Issue Type: Bug
Components: fairscheduler
Reporter: Anubhav Dhoot
Assignee: Anubhav Dhoot
Priority: Critical
Attachments: YARN-2224.patch, YARN-2244.001.patch, YARN-2244.002.patch
We are missing changes in patch MAPREDUCE-3596 in FairScheduler. Among other
fixes that were common across schedulers, there were some scheduler specific
fixes added to handle containers for unknown application attempts. Without
these fair scheduler simply logs that an unknown container was found and
continues to let it run.
--
This message was sent by Atlassian JIRA
(v6.2#6252)
[jira] [Commented] (YARN-2244) FairScheduler missing handling of containers for unknown application attempts
[
https://issues.apache.org/jira/browse/YARN-2244?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14056525#comment-14056525
]
Hadoop QA commented on YARN-2244:
-
{color:red}-1 overall{color}. Here are the results of testing the latest
attachment
http://issues.apache.org/jira/secure/attachment/12654841/YARN-2244.001.patch
against trunk revision .
{color:green}+1 @author{color}. The patch does not contain any @author
tags.
{color:green}+1 tests included{color}. The patch appears to include 1 new
or modified test files.
{color:red}-1 javac{color:red}. The patch appears to cause the build to
fail.
Console output: https://builds.apache.org/job/PreCommit-YARN-Build/4240//console
This message is automatically generated.
FairScheduler missing handling of containers for unknown application attempts
--
Key: YARN-2244
URL: https://issues.apache.org/jira/browse/YARN-2244
Project: Hadoop YARN
Issue Type: Bug
Components: fairscheduler
Reporter: Anubhav Dhoot
Assignee: Anubhav Dhoot
Priority: Critical
Attachments: YARN-2224.patch, YARN-2244.001.patch
We are missing changes in patch MAPREDUCE-3596 in FairScheduler. Among other
fixes that were common across schedulers, there were some scheduler specific
fixes added to handle containers for unknown application attempts. Without
these fair scheduler simply logs that an unknown container was found and
continues to let it run.
--
This message was sent by Atlassian JIRA
(v6.2#6252)
