[jira] [Commented] (YARN-2421) CapacityScheduler still allocates containers to an app in the FINISHING state
[ https://issues.apache.org/jira/browse/YARN-2421?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14546254#comment-14546254 ] Jason Lowe commented on YARN-2421: -- +1 latest patch lgtm. Committing this. > CapacityScheduler still allocates containers to an app in the FINISHING state > - > > Key: YARN-2421 > URL: https://issues.apache.org/jira/browse/YARN-2421 > Project: Hadoop YARN > Issue Type: Bug > Components: scheduler >Reporter: Thomas Graves >Assignee: Chang Li > Attachments: YARN-2421.4.patch, YARN-2421.5.patch, YARN-2421.6.patch, > YARN-2421.7.patch, YARN-2421.8.patch, YARN-2421.9.patch, yarn2421.patch, > yarn2421.patch, yarn2421.patch > > > I saw an instance of a bad application master where it unregistered with the > RM but then continued to call into allocate. The RMAppAttempt went to the > FINISHING state, but the capacity scheduler kept allocating it containers. > We should probably have the capacity scheduler check that the application > isn't in one of the terminal states before giving it containers. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (YARN-2421) CapacityScheduler still allocates containers to an app in the FINISHING state
[
https://issues.apache.org/jira/browse/YARN-2421?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14544379#comment-14544379
]
Hadoop QA commented on YARN-2421:
-
\\
\\
| (/) *{color:green}+1 overall{color}* |
\\
\\
|| Vote || Subsystem || Runtime || Comment ||
| {color:blue}0{color} | pre-patch | 14m 41s | Pre-patch trunk compilation is
healthy. |
| {color:green}+1{color} | @author | 0m 0s | The patch does not contain any
@author tags. |
| {color:green}+1{color} | tests included | 0m 0s | The patch appears to
include 1 new or modified test files. |
| {color:green}+1{color} | javac | 7m 35s | There were no new javac warning
messages. |
| {color:green}+1{color} | javadoc | 9m 35s | There were no new javadoc
warning messages. |
| {color:green}+1{color} | release audit | 0m 23s | The applied patch does
not increase the total number of release audit warnings. |
| {color:green}+1{color} | checkstyle | 0m 23s | There were no new checkstyle
issues. |
| {color:green}+1{color} | whitespace | 0m 0s | The patch has no lines that
end in whitespace. |
| {color:green}+1{color} | install | 1m 36s | mvn install still works. |
| {color:green}+1{color} | eclipse:eclipse | 0m 32s | The patch built with
eclipse:eclipse. |
| {color:green}+1{color} | findbugs | 1m 16s | The patch does not introduce
any new Findbugs (version 2.0.3) warnings. |
| {color:green}+1{color} | yarn tests | 49m 56s | Tests passed in
hadoop-yarn-server-resourcemanager. |
| | | 86m 0s | |
\\
\\
|| Subsystem || Report/Notes ||
| Patch URL |
http://issues.apache.org/jira/secure/attachment/12732938/YARN-2421.9.patch |
| Optional Tests | javadoc javac unit findbugs checkstyle |
| git revision | trunk / 15ccd96 |
| hadoop-yarn-server-resourcemanager test log |
https://builds.apache.org/job/PreCommit-YARN-Build/7939/artifact/patchprocess/testrun_hadoop-yarn-server-resourcemanager.txt
|
| Test Results |
https://builds.apache.org/job/PreCommit-YARN-Build/7939/testReport/ |
| Java | 1.7.0_55 |
| uname | Linux asf905.gq1.ygridcore.net 3.13.0-36-lowlatency #63-Ubuntu SMP
PREEMPT Wed Sep 3 21:56:12 UTC 2014 x86_64 x86_64 x86_64 GNU/Linux |
| Console output |
https://builds.apache.org/job/PreCommit-YARN-Build/7939/console |
This message was automatically generated.
> CapacityScheduler still allocates containers to an app in the FINISHING state
> -
>
> Key: YARN-2421
> URL: https://issues.apache.org/jira/browse/YARN-2421
> Project: Hadoop YARN
> Issue Type: Bug
> Components: scheduler
>Reporter: Thomas Graves
>Assignee: Chang Li
> Attachments: YARN-2421.4.patch, YARN-2421.5.patch, YARN-2421.6.patch,
> YARN-2421.7.patch, YARN-2421.8.patch, YARN-2421.9.patch, yarn2421.patch,
> yarn2421.patch, yarn2421.patch
>
>
> I saw an instance of a bad application master where it unregistered with the
> RM but then continued to call into allocate. The RMAppAttempt went to the
> FINISHING state, but the capacity scheduler kept allocating it containers.
> We should probably have the capacity scheduler check that the application
> isn't in one of the terminal states before giving it containers.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Commented] (YARN-2421) CapacityScheduler still allocates containers to an app in the FINISHING state
[ https://issues.apache.org/jira/browse/YARN-2421?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14544390#comment-14544390 ] Chang Li commented on YARN-2421: Thanks [~jlowe] for valuable feedback! have updated my patch according to those suggestions. Could you please help review the latest patch. Thanks! > CapacityScheduler still allocates containers to an app in the FINISHING state > - > > Key: YARN-2421 > URL: https://issues.apache.org/jira/browse/YARN-2421 > Project: Hadoop YARN > Issue Type: Bug > Components: scheduler >Reporter: Thomas Graves >Assignee: Chang Li > Attachments: YARN-2421.4.patch, YARN-2421.5.patch, YARN-2421.6.patch, > YARN-2421.7.patch, YARN-2421.8.patch, YARN-2421.9.patch, yarn2421.patch, > yarn2421.patch, yarn2421.patch > > > I saw an instance of a bad application master where it unregistered with the > RM but then continued to call into allocate. The RMAppAttempt went to the > FINISHING state, but the capacity scheduler kept allocating it containers. > We should probably have the capacity scheduler check that the application > isn't in one of the terminal states before giving it containers. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (YARN-2421) CapacityScheduler still allocates containers to an app in the FINISHING state
[ https://issues.apache.org/jira/browse/YARN-2421?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14543775#comment-14543775 ] Jason Lowe commented on YARN-2421: -- Thanks for updating the patch, Chang! It's looking much better, and all that is left now are mostly nits. The stubbed allocation should be constant like EMPTY_ALLOCATION in AbstractYarnScheduler. There's no need to construct the exact same object each time we need it. For improved readability it'd be nice to have a line between the new unit test and the test before it. The new unit test should be named a bit more specifically than just "bad allocation." Something like testAllocateAfterUnregister since that's literally what the test is doing. Could you elaborate on why the sleep(1000) is needed? In general we want to avoid any unnecessary sleeps, as our unit tests take too long as it is already. Should we be draining a dispatcher instead? > CapacityScheduler still allocates containers to an app in the FINISHING state > - > > Key: YARN-2421 > URL: https://issues.apache.org/jira/browse/YARN-2421 > Project: Hadoop YARN > Issue Type: Bug > Components: scheduler >Reporter: Thomas Graves >Assignee: Chang Li > Attachments: YARN-2421.4.patch, YARN-2421.5.patch, YARN-2421.6.patch, > YARN-2421.7.patch, yarn2421.patch, yarn2421.patch, yarn2421.patch > > > I saw an instance of a bad application master where it unregistered with the > RM but then continued to call into allocate. The RMAppAttempt went to the > FINISHING state, but the capacity scheduler kept allocating it containers. > We should probably have the capacity scheduler check that the application > isn't in one of the terminal states before giving it containers. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (YARN-2421) CapacityScheduler still allocates containers to an app in the FINISHING state
[ https://issues.apache.org/jira/browse/YARN-2421?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14542086#comment-14542086 ] Chang Li commented on YARN-2421: The check style is not related to my change. [~jlowe] could you help review the latest patch. Thanks! > CapacityScheduler still allocates containers to an app in the FINISHING state > - > > Key: YARN-2421 > URL: https://issues.apache.org/jira/browse/YARN-2421 > Project: Hadoop YARN > Issue Type: Bug > Components: scheduler >Reporter: Thomas Graves >Assignee: Chang Li > Attachments: YARN-2421.4.patch, YARN-2421.5.patch, YARN-2421.6.patch, > YARN-2421.7.patch, yarn2421.patch, yarn2421.patch, yarn2421.patch > > > I saw an instance of a bad application master where it unregistered with the > RM but then continued to call into allocate. The RMAppAttempt went to the > FINISHING state, but the capacity scheduler kept allocating it containers. > We should probably have the capacity scheduler check that the application > isn't in one of the terminal states before giving it containers. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (YARN-2421) CapacityScheduler still allocates containers to an app in the FINISHING state
[
https://issues.apache.org/jira/browse/YARN-2421?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14541136#comment-14541136
]
Hadoop QA commented on YARN-2421:
-
\\
\\
| (x) *{color:red}-1 overall{color}* |
\\
\\
|| Vote || Subsystem || Runtime || Comment ||
| {color:blue}0{color} | pre-patch | 15m 3s | Pre-patch trunk compilation is
healthy. |
| {color:green}+1{color} | @author | 0m 0s | The patch does not contain any
@author tags. |
| {color:green}+1{color} | tests included | 0m 0s | The patch appears to
include 1 new or modified test files. |
| {color:green}+1{color} | javac | 7m 47s | There were no new javac warning
messages. |
| {color:green}+1{color} | javadoc | 9m 59s | There were no new javadoc
warning messages. |
| {color:green}+1{color} | release audit | 0m 23s | The applied patch does
not increase the total number of release audit warnings. |
| {color:red}-1{color} | checkstyle | 0m 47s | The applied patch generated 1
new checkstyle issues (total was 30, now 30). |
| {color:green}+1{color} | whitespace | 0m 0s | The patch has no lines that
end in whitespace. |
| {color:green}+1{color} | install | 1m 34s | mvn install still works. |
| {color:green}+1{color} | eclipse:eclipse | 0m 33s | The patch built with
eclipse:eclipse. |
| {color:green}+1{color} | findbugs | 1m 17s | The patch does not introduce
any new Findbugs (version 2.0.3) warnings. |
| {color:green}+1{color} | yarn tests | 52m 27s | Tests passed in
hadoop-yarn-server-resourcemanager. |
| | | 89m 53s | |
\\
\\
|| Subsystem || Report/Notes ||
| Patch URL |
http://issues.apache.org/jira/secure/attachment/12732391/YARN-2421.7.patch |
| Optional Tests | javadoc javac unit findbugs checkstyle |
| git revision | trunk / f24452d |
| checkstyle |
https://builds.apache.org/job/PreCommit-YARN-Build/7906/artifact/patchprocess/diffcheckstylehadoop-yarn-server-resourcemanager.txt
|
| hadoop-yarn-server-resourcemanager test log |
https://builds.apache.org/job/PreCommit-YARN-Build/7906/artifact/patchprocess/testrun_hadoop-yarn-server-resourcemanager.txt
|
| Test Results |
https://builds.apache.org/job/PreCommit-YARN-Build/7906/testReport/ |
| Java | 1.7.0_55 |
| uname | Linux asf903.gq1.ygridcore.net 3.13.0-36-lowlatency #63-Ubuntu SMP
PREEMPT Wed Sep 3 21:56:12 UTC 2014 x86_64 x86_64 x86_64 GNU/Linux |
| Console output |
https://builds.apache.org/job/PreCommit-YARN-Build/7906/console |
This message was automatically generated.
> CapacityScheduler still allocates containers to an app in the FINISHING state
> -
>
> Key: YARN-2421
> URL: https://issues.apache.org/jira/browse/YARN-2421
> Project: Hadoop YARN
> Issue Type: Bug
> Components: scheduler
>Reporter: Thomas Graves
>Assignee: Chang Li
> Attachments: YARN-2421.4.patch, YARN-2421.5.patch, YARN-2421.6.patch,
> YARN-2421.7.patch, yarn2421.patch, yarn2421.patch, yarn2421.patch
>
>
> I saw an instance of a bad application master where it unregistered with the
> RM but then continued to call into allocate. The RMAppAttempt went to the
> FINISHING state, but the capacity scheduler kept allocating it containers.
> We should probably have the capacity scheduler check that the application
> isn't in one of the terminal states before giving it containers.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Commented] (YARN-2421) CapacityScheduler still allocates containers to an app in the FINISHING state
[
https://issues.apache.org/jira/browse/YARN-2421?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14540829#comment-14540829
]
Hadoop QA commented on YARN-2421:
-
\\
\\
| (x) *{color:red}-1 overall{color}* |
\\
\\
|| Vote || Subsystem || Runtime || Comment ||
| {color:blue}0{color} | pre-patch | 14m 38s | Pre-patch trunk compilation is
healthy. |
| {color:green}+1{color} | @author | 0m 0s | The patch does not contain any
@author tags. |
| {color:green}+1{color} | tests included | 0m 0s | The patch appears to
include 1 new or modified test files. |
| {color:green}+1{color} | javac | 7m 35s | There were no new javac warning
messages. |
| {color:green}+1{color} | javadoc | 9m 41s | There were no new javadoc
warning messages. |
| {color:green}+1{color} | release audit | 0m 22s | The applied patch does
not increase the total number of release audit warnings. |
| {color:red}-1{color} | checkstyle | 0m 46s | The applied patch generated 2
new checkstyle issues (total was 30, now 31). |
| {color:green}+1{color} | whitespace | 0m 0s | The patch has no lines that
end in whitespace. |
| {color:green}+1{color} | install | 1m 33s | mvn install still works. |
| {color:green}+1{color} | eclipse:eclipse | 0m 33s | The patch built with
eclipse:eclipse. |
| {color:green}+1{color} | findbugs | 1m 14s | The patch does not introduce
any new Findbugs (version 2.0.3) warnings. |
| {color:green}+1{color} | yarn tests | 52m 11s | Tests passed in
hadoop-yarn-server-resourcemanager. |
| | | 88m 37s | |
\\
\\
|| Subsystem || Report/Notes ||
| Patch URL |
http://issues.apache.org/jira/secure/attachment/12732346/YARN-2421.6.patch |
| Optional Tests | javadoc javac unit findbugs checkstyle |
| git revision | trunk / f24452d |
| checkstyle |
https://builds.apache.org/job/PreCommit-YARN-Build/7900/artifact/patchprocess/diffcheckstylehadoop-yarn-server-resourcemanager.txt
|
| hadoop-yarn-server-resourcemanager test log |
https://builds.apache.org/job/PreCommit-YARN-Build/7900/artifact/patchprocess/testrun_hadoop-yarn-server-resourcemanager.txt
|
| Test Results |
https://builds.apache.org/job/PreCommit-YARN-Build/7900/testReport/ |
| Java | 1.7.0_55 |
| uname | Linux asf909.gq1.ygridcore.net 3.13.0-36-lowlatency #63-Ubuntu SMP
PREEMPT Wed Sep 3 21:56:12 UTC 2014 x86_64 x86_64 x86_64 GNU/Linux |
| Console output |
https://builds.apache.org/job/PreCommit-YARN-Build/7900/console |
This message was automatically generated.
> CapacityScheduler still allocates containers to an app in the FINISHING state
> -
>
> Key: YARN-2421
> URL: https://issues.apache.org/jira/browse/YARN-2421
> Project: Hadoop YARN
> Issue Type: Bug
> Components: scheduler
>Reporter: Thomas Graves
>Assignee: Chang Li
> Attachments: YARN-2421.4.patch, YARN-2421.5.patch, YARN-2421.6.patch,
> yarn2421.patch, yarn2421.patch, yarn2421.patch
>
>
> I saw an instance of a bad application master where it unregistered with the
> RM but then continued to call into allocate. The RMAppAttempt went to the
> FINISHING state, but the capacity scheduler kept allocating it containers.
> We should probably have the capacity scheduler check that the application
> isn't in one of the terminal states before giving it containers.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Commented] (YARN-2421) CapacityScheduler still allocates containers to an app in the FINISHING state
[ https://issues.apache.org/jira/browse/YARN-2421?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14535308#comment-14535308 ] Chang Li commented on YARN-2421: [~cwelch], [~vinodkv], [~jlowe] could you help review this patch. Thanks > CapacityScheduler still allocates containers to an app in the FINISHING state > - > > Key: YARN-2421 > URL: https://issues.apache.org/jira/browse/YARN-2421 > Project: Hadoop YARN > Issue Type: Bug > Components: scheduler >Reporter: Thomas Graves >Assignee: Chang Li > Attachments: YARN-2421.4.patch, YARN-2421.5.patch, yarn2421.patch, > yarn2421.patch, yarn2421.patch > > > I saw an instance of a bad application master where it unregistered with the > RM but then continued to call into allocate. The RMAppAttempt went to the > FINISHING state, but the capacity scheduler kept allocating it containers. > We should probably have the capacity scheduler check that the application > isn't in one of the terminal states before giving it containers. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (YARN-2421) CapacityScheduler still allocates containers to an app in the FINISHING state
[
https://issues.apache.org/jira/browse/YARN-2421?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14535296#comment-14535296
]
Hadoop QA commented on YARN-2421:
-
\\
\\
| (/) *{color:green}+1 overall{color}* |
\\
\\
|| Vote || Subsystem || Runtime || Comment ||
| {color:blue}0{color} | pre-patch | 14m 37s | Pre-patch trunk compilation is
healthy. |
| {color:green}+1{color} | @author | 0m 0s | The patch does not contain any
@author tags. |
| {color:green}+1{color} | tests included | 0m 0s | The patch appears to
include 1 new or modified test files. |
| {color:green}+1{color} | javac | 7m 30s | There were no new javac warning
messages. |
| {color:green}+1{color} | javadoc | 9m 31s | There were no new javadoc
warning messages. |
| {color:green}+1{color} | release audit | 0m 22s | The applied patch does
not increase the total number of release audit warnings. |
| {color:green}+1{color} | checkstyle | 0m 26s | There were no new checkstyle
issues. |
| {color:green}+1{color} | whitespace | 0m 1s | The patch has no lines that
end in whitespace. |
| {color:green}+1{color} | install | 1m 37s | mvn install still works. |
| {color:green}+1{color} | eclipse:eclipse | 0m 33s | The patch built with
eclipse:eclipse. |
| {color:green}+1{color} | findbugs | 1m 14s | The patch does not introduce
any new Findbugs (version 2.0.3) warnings. |
| {color:green}+1{color} | yarn tests | 52m 20s | Tests passed in
hadoop-yarn-server-resourcemanager. |
| | | 88m 15s | |
\\
\\
|| Subsystem || Report/Notes ||
| Patch URL |
http://issues.apache.org/jira/secure/attachment/12731475/YARN-2421.5.patch |
| Optional Tests | javadoc javac unit findbugs checkstyle |
| git revision | trunk / 3cefc02 |
| hadoop-yarn-server-resourcemanager test log |
https://builds.apache.org/job/PreCommit-YARN-Build/7815/artifact/patchprocess/testrun_hadoop-yarn-server-resourcemanager.txt
|
| Test Results |
https://builds.apache.org/job/PreCommit-YARN-Build/7815/testReport/ |
| Java | 1.7.0_55 |
| uname | Linux asf906.gq1.ygridcore.net 3.13.0-36-lowlatency #63-Ubuntu SMP
PREEMPT Wed Sep 3 21:56:12 UTC 2014 x86_64 x86_64 x86_64 GNU/Linux |
| Console output |
https://builds.apache.org/job/PreCommit-YARN-Build/7815/console |
This message was automatically generated.
> CapacityScheduler still allocates containers to an app in the FINISHING state
> -
>
> Key: YARN-2421
> URL: https://issues.apache.org/jira/browse/YARN-2421
> Project: Hadoop YARN
> Issue Type: Bug
> Components: scheduler
>Reporter: Thomas Graves
>Assignee: Chang Li
> Attachments: YARN-2421.4.patch, YARN-2421.5.patch, yarn2421.patch,
> yarn2421.patch, yarn2421.patch
>
>
> I saw an instance of a bad application master where it unregistered with the
> RM but then continued to call into allocate. The RMAppAttempt went to the
> FINISHING state, but the capacity scheduler kept allocating it containers.
> We should probably have the capacity scheduler check that the application
> isn't in one of the terminal states before giving it containers.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Commented] (YARN-2421) CapacityScheduler still allocates containers to an app in the FINISHING state
[
https://issues.apache.org/jira/browse/YARN-2421?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14533727#comment-14533727
]
Hadoop QA commented on YARN-2421:
-
\\
\\
| (x) *{color:red}-1 overall{color}* |
\\
\\
|| Vote || Subsystem || Runtime || Comment ||
| {color:blue}0{color} | pre-patch | 14m 39s | Pre-patch trunk compilation is
healthy. |
| {color:green}+1{color} | @author | 0m 0s | The patch does not contain any
@author tags. |
| {color:green}+1{color} | tests included | 0m 0s | The patch appears to
include 1 new or modified test files. |
| {color:green}+1{color} | javac | 7m 34s | There were no new javac warning
messages. |
| {color:green}+1{color} | javadoc | 9m 33s | There were no new javadoc
warning messages. |
| {color:green}+1{color} | release audit | 0m 22s | The applied patch does
not increase the total number of release audit warnings. |
| {color:red}-1{color} | checkstyle | 0m 51s | The applied patch generated 2
new checkstyle issues (total was 30, now 31). |
| {color:red}-1{color} | whitespace | 0m 0s | The patch has 1 line(s) that
end in whitespace. Use git apply --whitespace=fix. |
| {color:green}+1{color} | install | 1m 34s | mvn install still works. |
| {color:green}+1{color} | eclipse:eclipse | 0m 32s | The patch built with
eclipse:eclipse. |
| {color:green}+1{color} | findbugs | 1m 14s | The patch does not introduce
any new Findbugs (version 2.0.3) warnings. |
| {color:green}+1{color} | yarn tests | 52m 19s | Tests passed in
hadoop-yarn-server-resourcemanager. |
| | | 88m 41s | |
\\
\\
|| Subsystem || Report/Notes ||
| Patch URL |
http://issues.apache.org/jira/secure/attachment/12731296/YARN-2421.4.patch |
| Optional Tests | javadoc javac unit findbugs checkstyle |
| git revision | trunk / 4d9f9e5 |
| checkstyle |
https://builds.apache.org/job/PreCommit-YARN-Build/7785/artifact/patchprocess/diffcheckstylehadoop-yarn-server-resourcemanager.txt
|
| whitespace |
https://builds.apache.org/job/PreCommit-YARN-Build/7785/artifact/patchprocess/whitespace.txt
|
| hadoop-yarn-server-resourcemanager test log |
https://builds.apache.org/job/PreCommit-YARN-Build/7785/artifact/patchprocess/testrun_hadoop-yarn-server-resourcemanager.txt
|
| Test Results |
https://builds.apache.org/job/PreCommit-YARN-Build/7785/testReport/ |
| Java | 1.7.0_55 |
| uname | Linux asf902.gq1.ygridcore.net 3.13.0-36-lowlatency #63-Ubuntu SMP
PREEMPT Wed Sep 3 21:56:12 UTC 2014 x86_64 x86_64 x86_64 GNU/Linux |
| Console output |
https://builds.apache.org/job/PreCommit-YARN-Build/7785/console |
This message was automatically generated.
> CapacityScheduler still allocates containers to an app in the FINISHING state
> -
>
> Key: YARN-2421
> URL: https://issues.apache.org/jira/browse/YARN-2421
> Project: Hadoop YARN
> Issue Type: Bug
> Components: scheduler
>Reporter: Thomas Graves
>Assignee: Chang Li
> Attachments: YARN-2421.4.patch, yarn2421.patch, yarn2421.patch,
> yarn2421.patch
>
>
> I saw an instance of a bad application master where it unregistered with the
> RM but then continued to call into allocate. The RMAppAttempt went to the
> FINISHING state, but the capacity scheduler kept allocating it containers.
> We should probably have the capacity scheduler check that the application
> isn't in one of the terminal states before giving it containers.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Commented] (YARN-2421) CapacityScheduler still allocates containers to an app in the FINISHING state
[ https://issues.apache.org/jira/browse/YARN-2421?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14533524#comment-14533524 ] Chang Li commented on YARN-2421: updated my patch, the current approach ignores allocate requests when app attempt is in final_saving, finishing or isAppFinalStateStored > CapacityScheduler still allocates containers to an app in the FINISHING state > - > > Key: YARN-2421 > URL: https://issues.apache.org/jira/browse/YARN-2421 > Project: Hadoop YARN > Issue Type: Bug > Components: scheduler >Affects Versions: 2.4.1 >Reporter: Thomas Graves >Assignee: Chang Li > Attachments: YARN-2421.4.patch, yarn2421.patch, yarn2421.patch, > yarn2421.patch > > > I saw an instance of a bad application master where it unregistered with the > RM but then continued to call into allocate. The RMAppAttempt went to the > FINISHING state, but the capacity scheduler kept allocating it containers. > We should probably have the capacity scheduler check that the application > isn't in one of the terminal states before giving it containers. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (YARN-2421) CapacityScheduler still allocates containers to an app in the FINISHING state
[ https://issues.apache.org/jira/browse/YARN-2421?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14531549#comment-14531549 ] Chang Li commented on YARN-2421: Thanks for suggestions! I am planning to come back to work on this issue soon. > CapacityScheduler still allocates containers to an app in the FINISHING state > - > > Key: YARN-2421 > URL: https://issues.apache.org/jira/browse/YARN-2421 > Project: Hadoop YARN > Issue Type: Bug > Components: scheduler >Affects Versions: 2.4.1 >Reporter: Thomas Graves >Assignee: Chang Li > Attachments: yarn2421.patch, yarn2421.patch, yarn2421.patch > > > I saw an instance of a bad application master where it unregistered with the > RM but then continued to call into allocate. The RMAppAttempt went to the > FINISHING state, but the capacity scheduler kept allocating it containers. > We should probably have the capacity scheduler check that the application > isn't in one of the terminal states before giving it containers. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (YARN-2421) CapacityScheduler still allocates containers to an app in the FINISHING state
[ https://issues.apache.org/jira/browse/YARN-2421?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14529488#comment-14529488 ] Craig Welch commented on YARN-2421: --- Hi [~lichangleo], thanks for working on this fix. Can you resolve the javac warning and run the TestRMContainerImpl test locally with the patch to verify the patch is not the cause? It seems to be persistently failing. > CapacityScheduler still allocates containers to an app in the FINISHING state > - > > Key: YARN-2421 > URL: https://issues.apache.org/jira/browse/YARN-2421 > Project: Hadoop YARN > Issue Type: Bug > Components: scheduler >Affects Versions: 2.4.1 >Reporter: Thomas Graves >Assignee: Chang Li > Attachments: yarn2421.patch, yarn2421.patch, yarn2421.patch > > > I saw an instance of a bad application master where it unregistered with the > RM but then continued to call into allocate. The RMAppAttempt went to the > FINISHING state, but the capacity scheduler kept allocating it containers. > We should probably have the capacity scheduler check that the application > isn't in one of the terminal states before giving it containers. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (YARN-2421) CapacityScheduler still allocates containers to an app in the FINISHING state
[
https://issues.apache.org/jira/browse/YARN-2421?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14269938#comment-14269938
]
Hadoop QA commented on YARN-2421:
-
{color:red}-1 overall{color}. Here are the results of testing the latest
attachment
http://issues.apache.org/jira/secure/attachment/12663584/yarn2421.patch
against trunk revision 20625c8.
{color:green}+1 @author{color}. The patch does not contain any @author
tags.
{color:green}+1 tests included{color}. The patch appears to include 1 new
or modified test files.
{color:red}-1 javac{color}. The applied patch generated 1215 javac
compiler warnings (more than the trunk's current 1214 warnings).
{color:green}+1 javadoc{color}. There were no new javadoc warning messages.
{color:green}+1 eclipse:eclipse{color}. The patch built with
eclipse:eclipse.
{color:green}+1 findbugs{color}. The patch does not introduce any new
Findbugs (version 2.0.3) warnings.
{color:green}+1 release audit{color}. The applied patch does not increase
the total number of release audit warnings.
{color:red}-1 core tests{color}. The patch failed these unit tests in
hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager:
org.apache.hadoop.yarn.server.resourcemanager.TestMoveApplication
org.apache.hadoop.yarn.server.resourcemanager.rmcontainer.TestRMContainerImpl
org.apache.hadoop.yarn.server.resourcemanager.scheduler.fair.TestFairScheduler
Test results:
https://builds.apache.org/job/PreCommit-YARN-Build/6280//testReport/
Javac warnings:
https://builds.apache.org/job/PreCommit-YARN-Build/6280//artifact/patchprocess/diffJavacWarnings.txt
Console output: https://builds.apache.org/job/PreCommit-YARN-Build/6280//console
This message is automatically generated.
> CapacityScheduler still allocates containers to an app in the FINISHING state
> -
>
> Key: YARN-2421
> URL: https://issues.apache.org/jira/browse/YARN-2421
> Project: Hadoop YARN
> Issue Type: Bug
> Components: scheduler
>Affects Versions: 2.4.1
>Reporter: Thomas Graves
>Assignee: chang li
> Attachments: yarn2421.patch, yarn2421.patch, yarn2421.patch
>
>
> I saw an instance of a bad application master where it unregistered with the
> RM but then continued to call into allocate. The RMAppAttempt went to the
> FINISHING state, but the capacity scheduler kept allocating it containers.
> We should probably have the capacity scheduler check that the application
> isn't in one of the terminal states before giving it containers.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Commented] (YARN-2421) CapacityScheduler still allocates containers to an app in the FINISHING state
[
https://issues.apache.org/jira/browse/YARN-2421?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14106384#comment-14106384
]
Hadoop QA commented on YARN-2421:
-
{color:red}-1 overall{color}. Here are the results of testing the latest
attachment
http://issues.apache.org/jira/secure/attachment/12663498/yarn2421.patch
against trunk revision .
{color:green}+1 @author{color}. The patch does not contain any @author
tags.
{color:green}+1 tests included{color}. The patch appears to include 1 new
or modified test files.
{color:green}+1 javac{color}. The applied patch does not increase the
total number of javac compiler warnings.
{color:green}+1 javadoc{color}. There were no new javadoc warning messages.
{color:green}+1 eclipse:eclipse{color}. The patch built with
eclipse:eclipse.
{color:green}+1 findbugs{color}. The patch does not introduce any new
Findbugs (version 2.0.3) warnings.
{color:red}-1 release audit{color}. The applied patch generated 3
release audit warnings.
{color:green}+1 core tests{color}. The patch passed unit tests in
hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager.
{color:green}+1 contrib tests{color}. The patch passed contrib unit tests.
Test results:
https://builds.apache.org/job/PreCommit-YARN-Build/4693//testReport/
Release audit warnings:
https://builds.apache.org/job/PreCommit-YARN-Build/4693//artifact/trunk/patchprocess/patchReleaseAuditProblems.txt
Console output: https://builds.apache.org/job/PreCommit-YARN-Build/4693//console
This message is automatically generated.
> CapacityScheduler still allocates containers to an app in the FINISHING state
> -
>
> Key: YARN-2421
> URL: https://issues.apache.org/jira/browse/YARN-2421
> Project: Hadoop YARN
> Issue Type: Bug
> Components: scheduler
>Affects Versions: 2.4.1
>Reporter: Thomas Graves
>Assignee: chang li
> Attachments: yarn2421.patch, yarn2421.patch
>
>
> I saw an instance of a bad application master where it unregistered with the
> RM but then continued to call into allocate. The RMAppAttempt went to the
> FINISHING state, but the capacity scheduler kept allocating it containers.
> We should probably have the capacity scheduler check that the application
> isn't in one of the terminal states before giving it containers.
--
This message was sent by Atlassian JIRA
(v6.2#6252)
[jira] [Commented] (YARN-2421) CapacityScheduler still allocates containers to an app in the FINISHING state
[ https://issues.apache.org/jira/browse/YARN-2421?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14105936#comment-14105936 ] chang li commented on YARN-2421: [~airbots], Have removed that, thanks for finding that out :) > CapacityScheduler still allocates containers to an app in the FINISHING state > - > > Key: YARN-2421 > URL: https://issues.apache.org/jira/browse/YARN-2421 > Project: Hadoop YARN > Issue Type: Bug > Components: scheduler >Affects Versions: 2.4.1 >Reporter: Thomas Graves >Assignee: chang li > Attachments: yarn2421.patch, yarn2421.patch > > > I saw an instance of a bad application master where it unregistered with the > RM but then continued to call into allocate. The RMAppAttempt went to the > FINISHING state, but the capacity scheduler kept allocating it containers. > We should probably have the capacity scheduler check that the application > isn't in one of the terminal states before giving it containers. -- This message was sent by Atlassian JIRA (v6.2#6252)
[jira] [Commented] (YARN-2421) CapacityScheduler still allocates containers to an app in the FINISHING state
[
https://issues.apache.org/jira/browse/YARN-2421?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14105927#comment-14105927
]
Chen He commented on YARN-2421:
---
[~lichangleo], is this "+System.out.println("xxx inside test");" your debug
code? If so, remove it from patch. :)
> CapacityScheduler still allocates containers to an app in the FINISHING state
> -
>
> Key: YARN-2421
> URL: https://issues.apache.org/jira/browse/YARN-2421
> Project: Hadoop YARN
> Issue Type: Bug
> Components: scheduler
>Affects Versions: 2.4.1
>Reporter: Thomas Graves
>Assignee: chang li
> Attachments: yarn2421.patch
>
>
> I saw an instance of a bad application master where it unregistered with the
> RM but then continued to call into allocate. The RMAppAttempt went to the
> FINISHING state, but the capacity scheduler kept allocating it containers.
> We should probably have the capacity scheduler check that the application
> isn't in one of the terminal states before giving it containers.
--
This message was sent by Atlassian JIRA
(v6.2#6252)
