[jira] [Commented] (YARN-10854) Support marking inactive node as untracked without configured include path
[
https://issues.apache.org/jira/browse/YARN-10854?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17384386#comment-17384386
]
Kuhu Shukla commented on YARN-10854:
Proposal seems good but since I have been away from the land of YARN for a
while, could [~brahma], [~templedf] or others chime in on the idea as well? I
would love to review the code for this change.
> Support marking inactive node as untracked without configured include path
> --
>
> Key: YARN-10854
> URL: https://issues.apache.org/jira/browse/YARN-10854
> Project: Hadoop YARN
> Issue Type: Improvement
> Components: resourcemanager
>Reporter: Tao Yang
>Assignee: Tao Yang
>Priority: Major
> Attachments: YARN-10854.001.patch
>
>
> Currently inactive nodes which have been decommissioned/shutdown/lost for a
> while(specified expiration time defined via
> {{yarn.resourcemanager.node-removal-untracked.timeout-ms}}, 60 seconds by
> default) and not exist in both include and exclude files can be marked as
> untracked nodes and can be removed from RM state (YARN-4311). It's very
> useful when auto-scaling is enabled in elastic cloud environment, which can
> avoid unlimited increase of inactive nodes (mostly are decommissioned nodes).
> But this only works when the include path is configured, mismatched for most
> of our cloud environments without configured white list of nodes, which can
> lead to easily control for the auto-scaling of nodes without further security
> requirements.
> So I propose to support marking inactive node as untracked without configured
> include path, to be compatible with the former versions, we can add a switch
> config for this.
> Any thoughts/suggestions/feedbacks are welcome!
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
-
To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org
[jira] [Commented] (YARN-10854) Support marking inactive node as untracked without configured include path
[
https://issues.apache.org/jira/browse/YARN-10854?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17384440#comment-17384440
]
Daniel Templeton commented on YARN-10854:
-
In principle the change looks fine to me. I'd like the docs in the XML file to
be a bit more explicit about what this property does. If you know about the
behavior of the include and exclude lists, the explanation is just barely
enough to know what the flag does. It would be better to be more verbose,
provide some context, make it easier to understand without reading source code.
> Support marking inactive node as untracked without configured include path
> --
>
> Key: YARN-10854
> URL: https://issues.apache.org/jira/browse/YARN-10854
> Project: Hadoop YARN
> Issue Type: Improvement
> Components: resourcemanager
>Reporter: Tao Yang
>Assignee: Tao Yang
>Priority: Major
> Attachments: YARN-10854.001.patch
>
>
> Currently inactive nodes which have been decommissioned/shutdown/lost for a
> while(specified expiration time defined via
> {{yarn.resourcemanager.node-removal-untracked.timeout-ms}}, 60 seconds by
> default) and not exist in both include and exclude files can be marked as
> untracked nodes and can be removed from RM state (YARN-4311). It's very
> useful when auto-scaling is enabled in elastic cloud environment, which can
> avoid unlimited increase of inactive nodes (mostly are decommissioned nodes).
> But this only works when the include path is configured, mismatched for most
> of our cloud environments without configured white list of nodes, which can
> lead to easily control for the auto-scaling of nodes without further security
> requirements.
> So I propose to support marking inactive node as untracked without configured
> include path, to be compatible with the former versions, we can add a switch
> config for this.
> Any thoughts/suggestions/feedbacks are welcome!
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
-
To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org
[jira] [Commented] (YARN-10854) Support marking inactive node as untracked without configured include path
[
https://issues.apache.org/jira/browse/YARN-10854?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17387758#comment-17387758
]
Tao Yang commented on YARN-10854:
-
Thanks [~kshukla] and [~templedf] for the feedbacks and review.
Attached v2 patch with further explanation of the new configuration in
yarn-default.xml, [~templedf] could you please help to review that again?
Thanks.
> Support marking inactive node as untracked without configured include path
> --
>
> Key: YARN-10854
> URL: https://issues.apache.org/jira/browse/YARN-10854
> Project: Hadoop YARN
> Issue Type: Improvement
> Components: resourcemanager
>Reporter: Tao Yang
>Assignee: Tao Yang
>Priority: Major
> Attachments: YARN-10854.001.patch, YARN-10854.002.patch
>
>
> Currently inactive nodes which have been decommissioned/shutdown/lost for a
> while(specified expiration time defined via
> {{yarn.resourcemanager.node-removal-untracked.timeout-ms}}, 60 seconds by
> default) and not exist in both include and exclude files can be marked as
> untracked nodes and can be removed from RM state (YARN-4311). It's very
> useful when auto-scaling is enabled in elastic cloud environment, which can
> avoid unlimited increase of inactive nodes (mostly are decommissioned nodes).
> But this only works when the include path is configured, mismatched for most
> of our cloud environments without configured white list of nodes, which can
> lead to easily control for the auto-scaling of nodes without further security
> requirements.
> So I propose to support marking inactive node as untracked without configured
> include path, to be compatible with the former versions, we can add a switch
> config for this.
> Any thoughts/suggestions/feedbacks are welcome!
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
-
To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org
[jira] [Commented] (YARN-10854) Support marking inactive node as untracked without configured include path
[
https://issues.apache.org/jira/browse/YARN-10854?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17387874#comment-17387874
]
Hadoop QA commented on YARN-10854:
--
| (/) *{color:green}+1 overall{color}* |
\\
\\
|| Vote || Subsystem || Runtime || Logfile || Comment ||
| {color:blue}0{color} | {color:blue} reexec {color} | {color:blue} 13m
0s{color} | {color:blue}{color} | {color:blue} Docker mode activated. {color} |
|| || || || {color:brown} Prechecks {color} || ||
| {color:green}+1{color} | {color:green} dupname {color} | {color:green} 0m
0s{color} | {color:green}{color} | {color:green} No case conflicting files
found. {color} |
| {color:green}+1{color} | {color:green} @author {color} | {color:green} 0m
0s{color} | {color:green}{color} | {color:green} The patch does not contain any
@author tags. {color} |
| {color:green}+1{color} | {color:green} {color} | {color:green} 0m 0s{color}
| {color:green}test4tests{color} | {color:green} The patch appears to include 1
new or modified test files. {color} |
|| || || || {color:brown} trunk Compile Tests {color} || ||
| {color:blue}0{color} | {color:blue} mvndep {color} | {color:blue} 2m
10s{color} | {color:blue}{color} | {color:blue} Maven dependency ordering for
branch {color} |
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 19m
39s{color} | {color:green}{color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 9m
17s{color} | {color:green}{color} | {color:green} trunk passed with JDK
Ubuntu-11.0.11+9-Ubuntu-0ubuntu2.20.04 {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 8m
5s{color} | {color:green}{color} | {color:green} trunk passed with JDK Private
Build-1.8.0_292-8u292-b10-0ubuntu1~20.04-b10 {color} |
| {color:green}+1{color} | {color:green} checkstyle {color} | {color:green} 1m
41s{color} | {color:green}{color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} mvnsite {color} | {color:green} 3m
11s{color} | {color:green}{color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} shadedclient {color} | {color:green}
18m 23s{color} | {color:green}{color} | {color:green} branch has no errors when
building and testing our client artifacts. {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 2m
50s{color} | {color:green}{color} | {color:green} trunk passed with JDK
Ubuntu-11.0.11+9-Ubuntu-0ubuntu2.20.04 {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 2m
45s{color} | {color:green}{color} | {color:green} trunk passed with JDK Private
Build-1.8.0_292-8u292-b10-0ubuntu1~20.04-b10 {color} |
| {color:blue}0{color} | {color:blue} spotbugs {color} | {color:blue} 29m
51s{color} | {color:blue}{color} | {color:blue} Both FindBugs and SpotBugs are
enabled, using SpotBugs. {color} |
| {color:green}+1{color} | {color:green} spotbugs {color} | {color:green} 5m
55s{color} | {color:green}{color} | {color:green} trunk passed {color} |
|| || || || {color:brown} Patch Compile Tests {color} || ||
| {color:blue}0{color} | {color:blue} mvndep {color} | {color:blue} 0m
27s{color} | {color:blue}{color} | {color:blue} Maven dependency ordering for
patch {color} |
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 2m
8s{color} | {color:green}{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 10m
13s{color} | {color:green}{color} | {color:green} the patch passed with JDK
Ubuntu-11.0.11+9-Ubuntu-0ubuntu2.20.04 {color} |
| {color:green}+1{color} | {color:green} javac {color} | {color:green} 10m
13s{color} | {color:green}{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 9m
9s{color} | {color:green}{color} | {color:green} the patch passed with JDK
Private Build-1.8.0_292-8u292-b10-0ubuntu1~20.04-b10 {color} |
| {color:green}+1{color} | {color:green} javac {color} | {color:green} 9m
9s{color} | {color:green}{color} | {color:green} the patch passed {color} |
| {color:orange}-0{color} | {color:orange} checkstyle {color} | {color:orange}
1m 30s{color} |
{color:orange}https://ci-hadoop.apache.org/job/PreCommit-YARN-Build/1141/artifact/out/diff-checkstyle-hadoop-yarn-project_hadoop-yarn.txt{color}
| {color:orange} hadoop-yarn-project/hadoop-yarn: The patch generated 1 new +
198 unchanged - 0 fixed = 199 total (was 198) {color} |
| {color:green}+1{color} | {color:green} mvnsite {color} | {color:green} 3m
3s{color} | {color:green}{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} whitespace {color} | {color:green} 0m
0s{color} | {color:green}{color} | {color:green} The patch has no whitespace
issues.
[jira] [Commented] (YARN-10854) Support marking inactive node as untracked without configured include path
[
https://issues.apache.org/jira/browse/YARN-10854?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17388210#comment-17388210
]
Daniel Templeton commented on YARN-10854:
-
The new description looks good. My only other ask would be that you also add a
sentence or two that addresses the bigger picture, e.g. the RM hangs onto
inactive nodes, but when you have a cluster that's being dynamically resized,
the RM needs to be able to let go of nodes that become inactive. Definitely
don't take that verbatim. :)
I'm no longer active enough in YARN to feel comfortable giving you a +1 in any
case, so you'll need to grab another committer for that. Add some motivational
text, and I'll give you my LGTM.
> Support marking inactive node as untracked without configured include path
> --
>
> Key: YARN-10854
> URL: https://issues.apache.org/jira/browse/YARN-10854
> Project: Hadoop YARN
> Issue Type: Improvement
> Components: resourcemanager
>Reporter: Tao Yang
>Assignee: Tao Yang
>Priority: Major
> Attachments: YARN-10854.001.patch, YARN-10854.002.patch
>
>
> Currently inactive nodes which have been decommissioned/shutdown/lost for a
> while(specified expiration time defined via
> {{yarn.resourcemanager.node-removal-untracked.timeout-ms}}, 60 seconds by
> default) and not exist in both include and exclude files can be marked as
> untracked nodes and can be removed from RM state (YARN-4311). It's very
> useful when auto-scaling is enabled in elastic cloud environment, which can
> avoid unlimited increase of inactive nodes (mostly are decommissioned nodes).
> But this only works when the include path is configured, mismatched for most
> of our cloud environments without configured white list of nodes, which can
> lead to easily control for the auto-scaling of nodes without further security
> requirements.
> So I propose to support marking inactive node as untracked without configured
> include path, to be compatible with the former versions, we can add a switch
> config for this.
> Any thoughts/suggestions/feedbacks are welcome!
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
-
To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org
[jira] [Commented] (YARN-10854) Support marking inactive node as untracked without configured include path
[
https://issues.apache.org/jira/browse/YARN-10854?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17389166#comment-17389166
]
Tao Yang commented on YARN-10854:
-
Thanks [~templedf] for the review and feedback.
I would like to clarify that this improvement has nothing to do with the
scheduling process, it just want to mark inactive nodes as untracked after the
timeout specified by yarn.resourcemanager.node-removal-untracked.timeout-ms and
then remove them from nodes list for the YARN cluster without configured
include path, which means RM can periodically clear inactive nodes to avoid
increasing memory to store these data, most desired by elastic cloud
environment with frequent auto-scaling operations.
Attached v3 patch with some background information. [~zhuqi], [~snemeth], could
you please help to take a look at this issue? Thanks.
> Support marking inactive node as untracked without configured include path
> --
>
> Key: YARN-10854
> URL: https://issues.apache.org/jira/browse/YARN-10854
> Project: Hadoop YARN
> Issue Type: Improvement
> Components: resourcemanager
>Reporter: Tao Yang
>Assignee: Tao Yang
>Priority: Major
> Attachments: YARN-10854.001.patch, YARN-10854.002.patch,
> YARN-10854.003.patch
>
>
> Currently inactive nodes which have been decommissioned/shutdown/lost for a
> while(specified expiration time defined via
> {{yarn.resourcemanager.node-removal-untracked.timeout-ms}}, 60 seconds by
> default) and not exist in both include and exclude files can be marked as
> untracked nodes and can be removed from RM state (YARN-4311). It's very
> useful when auto-scaling is enabled in elastic cloud environment, which can
> avoid unlimited increase of inactive nodes (mostly are decommissioned nodes).
> But this only works when the include path is configured, mismatched for most
> of our cloud environments without configured white list of nodes, which can
> lead to easily control for the auto-scaling of nodes without further security
> requirements.
> So I propose to support marking inactive node as untracked without configured
> include path, to be compatible with the former versions, we can add a switch
> config for this.
> Any thoughts/suggestions/feedbacks are welcome!
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
-
To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org
[jira] [Commented] (YARN-10854) Support marking inactive node as untracked without configured include path
[
https://issues.apache.org/jira/browse/YARN-10854?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17389198#comment-17389198
]
Qi Zhu commented on YARN-10854:
---
Thanks [~Tao Yang] for this, the patch LGTM and make sense to me.
The code is just related to the NodesListManager, there is no risk to the
scheduling process. And it's a good point to cloud environment for frequent
auto-scaling operations.
And the test Decommission case is clear, minor things, we'd better fill all the
test case, such as lost and shutdown and graceful decommission etc.
> Support marking inactive node as untracked without configured include path
> --
>
> Key: YARN-10854
> URL: https://issues.apache.org/jira/browse/YARN-10854
> Project: Hadoop YARN
> Issue Type: Improvement
> Components: resourcemanager
>Reporter: Tao Yang
>Assignee: Tao Yang
>Priority: Major
> Attachments: YARN-10854.001.patch, YARN-10854.002.patch,
> YARN-10854.003.patch
>
>
> Currently inactive nodes which have been decommissioned/shutdown/lost for a
> while(specified expiration time defined via
> {{yarn.resourcemanager.node-removal-untracked.timeout-ms}}, 60 seconds by
> default) and not exist in both include and exclude files can be marked as
> untracked nodes and can be removed from RM state (YARN-4311). It's very
> useful when auto-scaling is enabled in elastic cloud environment, which can
> avoid unlimited increase of inactive nodes (mostly are decommissioned nodes).
> But this only works when the include path is configured, mismatched for most
> of our cloud environments without configured white list of nodes, which can
> lead to easily control for the auto-scaling of nodes without further security
> requirements.
> So I propose to support marking inactive node as untracked without configured
> include path, to be compatible with the former versions, we can add a switch
> config for this.
> Any thoughts/suggestions/feedbacks are welcome!
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
-
To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org
[jira] [Commented] (YARN-10854) Support marking inactive node as untracked without configured include path
[
https://issues.apache.org/jira/browse/YARN-10854?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17389249#comment-17389249
]
Hadoop QA commented on YARN-10854:
--
| (x) *{color:red}-1 overall{color}* |
\\
\\
|| Vote || Subsystem || Runtime || Logfile || Comment ||
| {color:blue}0{color} | {color:blue} reexec {color} | {color:blue} 0m
39s{color} | {color:blue}{color} | {color:blue} Docker mode activated. {color} |
|| || || || {color:brown} Prechecks {color} || ||
| {color:green}+1{color} | {color:green} dupname {color} | {color:green} 0m
0s{color} | {color:green}{color} | {color:green} No case conflicting files
found. {color} |
| {color:green}+1{color} | {color:green} @author {color} | {color:green} 0m
0s{color} | {color:green}{color} | {color:green} The patch does not contain any
@author tags. {color} |
| {color:green}+1{color} | {color:green} {color} | {color:green} 0m 0s{color}
| {color:green}test4tests{color} | {color:green} The patch appears to include 1
new or modified test files. {color} |
|| || || || {color:brown} trunk Compile Tests {color} || ||
| {color:blue}0{color} | {color:blue} mvndep {color} | {color:blue} 1m
47s{color} | {color:blue}{color} | {color:blue} Maven dependency ordering for
branch {color} |
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 19m
40s{color} | {color:green}{color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 9m
25s{color} | {color:green}{color} | {color:green} trunk passed with JDK
Ubuntu-11.0.11+9-Ubuntu-0ubuntu2.20.04 {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 8m
5s{color} | {color:green}{color} | {color:green} trunk passed with JDK Private
Build-1.8.0_292-8u292-b10-0ubuntu1~20.04-b10 {color} |
| {color:green}+1{color} | {color:green} checkstyle {color} | {color:green} 1m
40s{color} | {color:green}{color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} mvnsite {color} | {color:green} 3m
11s{color} | {color:green}{color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} shadedclient {color} | {color:green}
18m 31s{color} | {color:green}{color} | {color:green} branch has no errors when
building and testing our client artifacts. {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 2m
47s{color} | {color:green}{color} | {color:green} trunk passed with JDK
Ubuntu-11.0.11+9-Ubuntu-0ubuntu2.20.04 {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 2m
44s{color} | {color:green}{color} | {color:green} trunk passed with JDK Private
Build-1.8.0_292-8u292-b10-0ubuntu1~20.04-b10 {color} |
| {color:blue}0{color} | {color:blue} spotbugs {color} | {color:blue} 30m
58s{color} | {color:blue}{color} | {color:blue} Both FindBugs and SpotBugs are
enabled, using SpotBugs. {color} |
| {color:green}+1{color} | {color:green} spotbugs {color} | {color:green} 7m
1s{color} | {color:green}{color} | {color:green} trunk passed {color} |
|| || || || {color:brown} Patch Compile Tests {color} || ||
| {color:blue}0{color} | {color:blue} mvndep {color} | {color:blue} 0m
28s{color} | {color:blue}{color} | {color:blue} Maven dependency ordering for
patch {color} |
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 2m
25s{color} | {color:green}{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 11m
25s{color} | {color:green}{color} | {color:green} the patch passed with JDK
Ubuntu-11.0.11+9-Ubuntu-0ubuntu2.20.04 {color} |
| {color:green}+1{color} | {color:green} javac {color} | {color:green} 11m
25s{color} | {color:green}{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 11m
4s{color} | {color:green}{color} | {color:green} the patch passed with JDK
Private Build-1.8.0_292-8u292-b10-0ubuntu1~20.04-b10 {color} |
| {color:green}+1{color} | {color:green} javac {color} | {color:green} 11m
4s{color} | {color:green}{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} checkstyle {color} | {color:green} 1m
54s{color} | {color:green}{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} mvnsite {color} | {color:green} 3m
25s{color} | {color:green}{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} whitespace {color} | {color:green} 0m
0s{color} | {color:green}{color} | {color:green} The patch has no whitespace
issues. {color} |
| {color:green}+1{color} | {color:green} xml {color} | {color:green} 0m
2s{color} | {color:green}{color} | {color:green} The patch has no ill-formed
XML file. {color} |
| {color:green}+1{color} | {color:green}
[jira] [Commented] (YARN-10854) Support marking inactive node as untracked without configured include path
[
https://issues.apache.org/jira/browse/YARN-10854?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17390079#comment-17390079
]
Prabhu Joseph commented on YARN-10854:
--
Thanks [~Tao Yang] for the patch. This is very useful to us as well, else we
would have end up in adding lot of code change in managing the includes node
list.
> Support marking inactive node as untracked without configured include path
> --
>
> Key: YARN-10854
> URL: https://issues.apache.org/jira/browse/YARN-10854
> Project: Hadoop YARN
> Issue Type: Improvement
> Components: resourcemanager
>Reporter: Tao Yang
>Assignee: Tao Yang
>Priority: Major
> Attachments: YARN-10854.001.patch, YARN-10854.002.patch,
> YARN-10854.003.patch
>
>
> Currently inactive nodes which have been decommissioned/shutdown/lost for a
> while(specified expiration time defined via
> {{yarn.resourcemanager.node-removal-untracked.timeout-ms}}, 60 seconds by
> default) and not exist in both include and exclude files can be marked as
> untracked nodes and can be removed from RM state (YARN-4311). It's very
> useful when auto-scaling is enabled in elastic cloud environment, which can
> avoid unlimited increase of inactive nodes (mostly are decommissioned nodes).
> But this only works when the include path is configured, mismatched for most
> of our cloud environments without configured white list of nodes, which can
> lead to easily control for the auto-scaling of nodes without further security
> requirements.
> So I propose to support marking inactive node as untracked without configured
> include path, to be compatible with the former versions, we can add a switch
> config for this.
> Any thoughts/suggestions/feedbacks are welcome!
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
-
To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org
[jira] [Commented] (YARN-10854) Support marking inactive node as untracked without configured include path
[
https://issues.apache.org/jira/browse/YARN-10854?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17390425#comment-17390425
]
Tao Yang commented on YARN-10854:
-
Thanks [~zhuqi] and [~prabhujoseph] for the review and feedback.
Attached v4 patch with improvement on UT as Qi suggested. Please help to review
in your free time, Thanks.
> Support marking inactive node as untracked without configured include path
> --
>
> Key: YARN-10854
> URL: https://issues.apache.org/jira/browse/YARN-10854
> Project: Hadoop YARN
> Issue Type: Improvement
> Components: resourcemanager
>Reporter: Tao Yang
>Assignee: Tao Yang
>Priority: Major
> Attachments: YARN-10854.001.patch, YARN-10854.002.patch,
> YARN-10854.003.patch, YARN-10854.004.patch
>
>
> Currently inactive nodes which have been decommissioned/shutdown/lost for a
> while(specified expiration time defined via
> {{yarn.resourcemanager.node-removal-untracked.timeout-ms}}, 60 seconds by
> default) and not exist in both include and exclude files can be marked as
> untracked nodes and can be removed from RM state (YARN-4311). It's very
> useful when auto-scaling is enabled in elastic cloud environment, which can
> avoid unlimited increase of inactive nodes (mostly are decommissioned nodes).
> But this only works when the include path is configured, mismatched for most
> of our cloud environments without configured white list of nodes, which can
> lead to easily control for the auto-scaling of nodes without further security
> requirements.
> So I propose to support marking inactive node as untracked without configured
> include path, to be compatible with the former versions, we can add a switch
> config for this.
> Any thoughts/suggestions/feedbacks are welcome!
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
-
To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org
[jira] [Commented] (YARN-10854) Support marking inactive node as untracked without configured include path
[
https://issues.apache.org/jira/browse/YARN-10854?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17390426#comment-17390426
]
Qi Zhu commented on YARN-10854:
---
Thanks [~Tao Yang] for update, LGTM, i will help commit this patch after the
jenkins passed.
> Support marking inactive node as untracked without configured include path
> --
>
> Key: YARN-10854
> URL: https://issues.apache.org/jira/browse/YARN-10854
> Project: Hadoop YARN
> Issue Type: Improvement
> Components: resourcemanager
>Reporter: Tao Yang
>Assignee: Tao Yang
>Priority: Major
> Attachments: YARN-10854.001.patch, YARN-10854.002.patch,
> YARN-10854.003.patch, YARN-10854.004.patch
>
>
> Currently inactive nodes which have been decommissioned/shutdown/lost for a
> while(specified expiration time defined via
> {{yarn.resourcemanager.node-removal-untracked.timeout-ms}}, 60 seconds by
> default) and not exist in both include and exclude files can be marked as
> untracked nodes and can be removed from RM state (YARN-4311). It's very
> useful when auto-scaling is enabled in elastic cloud environment, which can
> avoid unlimited increase of inactive nodes (mostly are decommissioned nodes).
> But this only works when the include path is configured, mismatched for most
> of our cloud environments without configured white list of nodes, which can
> lead to easily control for the auto-scaling of nodes without further security
> requirements.
> So I propose to support marking inactive node as untracked without configured
> include path, to be compatible with the former versions, we can add a switch
> config for this.
> Any thoughts/suggestions/feedbacks are welcome!
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
-
To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org
[jira] [Commented] (YARN-10854) Support marking inactive node as untracked without configured include path
[
https://issues.apache.org/jira/browse/YARN-10854?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17390513#comment-17390513
]
Hadoop QA commented on YARN-10854:
--
| (x) *{color:red}-1 overall{color}* |
\\
\\
|| Vote || Subsystem || Runtime || Logfile || Comment ||
| {color:blue}0{color} | {color:blue} reexec {color} | {color:blue} 27m
55s{color} | {color:blue}{color} | {color:blue} Docker mode activated. {color} |
|| || || || {color:brown} Prechecks {color} || ||
| {color:green}+1{color} | {color:green} dupname {color} | {color:green} 0m
0s{color} | {color:green}{color} | {color:green} No case conflicting files
found. {color} |
| {color:green}+1{color} | {color:green} @author {color} | {color:green} 0m
0s{color} | {color:green}{color} | {color:green} The patch does not contain any
@author tags. {color} |
| {color:green}+1{color} | {color:green} {color} | {color:green} 0m 0s{color}
| {color:green}test4tests{color} | {color:green} The patch appears to include 1
new or modified test files. {color} |
|| || || || {color:brown} trunk Compile Tests {color} || ||
| {color:blue}0{color} | {color:blue} mvndep {color} | {color:blue} 1m
52s{color} | {color:blue}{color} | {color:blue} Maven dependency ordering for
branch {color} |
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 30m
29s{color} | {color:green}{color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 15m
11s{color} | {color:green}{color} | {color:green} trunk passed with JDK
Ubuntu-11.0.11+9-Ubuntu-0ubuntu2.20.04 {color} |
| {color:red}-1{color} | {color:red} compile {color} | {color:red} 5m
45s{color} |
{color:red}https://ci-hadoop.apache.org/job/PreCommit-YARN-Build/1169/artifact/out/branch-compile-hadoop-yarn-project_hadoop-yarn-jdkPrivateBuild-1.8.0_292-8u292-b10-0ubuntu1~20.04-b10.txt{color}
| {color:red} hadoop-yarn in trunk failed with JDK Private
Build-1.8.0_292-8u292-b10-0ubuntu1~20.04-b10. {color} |
| {color:green}+1{color} | {color:green} checkstyle {color} | {color:green} 1m
59s{color} | {color:green}{color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} mvnsite {color} | {color:green} 3m
26s{color} | {color:green}{color} | {color:green} trunk passed {color} |
| {color:red}-1{color} | {color:red} shadedclient {color} | {color:red} 30m
19s{color} |
{color:red}https://ci-hadoop.apache.org/job/PreCommit-YARN-Build/1169/artifact/out/branch-shadedclient.txt{color}
| {color:red} branch has errors when building and testing our client
artifacts. {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 3m
4s{color} | {color:green}{color} | {color:green} trunk passed with JDK
Ubuntu-11.0.11+9-Ubuntu-0ubuntu2.20.04 {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 2m
44s{color} | {color:green}{color} | {color:green} trunk passed with JDK Private
Build-1.8.0_292-8u292-b10-0ubuntu1~20.04-b10 {color} |
| {color:blue}0{color} | {color:blue} spotbugs {color} | {color:blue} 43m
23s{color} | {color:blue}{color} | {color:blue} Both FindBugs and SpotBugs are
enabled, using SpotBugs. {color} |
| {color:green}+1{color} | {color:green} spotbugs {color} | {color:green} 7m
21s{color} | {color:green}{color} | {color:green} trunk passed {color} |
|| || || || {color:brown} Patch Compile Tests {color} || ||
| {color:blue}0{color} | {color:blue} mvndep {color} | {color:blue} 0m
26s{color} | {color:blue}{color} | {color:blue} Maven dependency ordering for
patch {color} |
| {color:red}-1{color} | {color:red} mvninstall {color} | {color:red} 0m
17s{color} |
{color:red}https://ci-hadoop.apache.org/job/PreCommit-YARN-Build/1169/artifact/out/patch-mvninstall-hadoop-yarn-project_hadoop-yarn_hadoop-yarn-server_hadoop-yarn-server-resourcemanager.txt{color}
| {color:red} hadoop-yarn-server-resourcemanager in the patch failed. {color} |
| {color:red}-1{color} | {color:red} compile {color} | {color:red} 4m
19s{color} |
{color:red}https://ci-hadoop.apache.org/job/PreCommit-YARN-Build/1169/artifact/out/patch-compile-hadoop-yarn-project_hadoop-yarn-jdkUbuntu-11.0.11+9-Ubuntu-0ubuntu2.20.04.txt{color}
| {color:red} hadoop-yarn in the patch failed with JDK
Ubuntu-11.0.11+9-Ubuntu-0ubuntu2.20.04. {color} |
| {color:red}-1{color} | {color:red} javac {color} | {color:red} 4m 19s{color}
|
{color:red}https://ci-hadoop.apache.org/job/PreCommit-YARN-Build/1169/artifact/out/patch-compile-hadoop-yarn-project_hadoop-yarn-jdkUbuntu-11.0.11+9-Ubuntu-0ubuntu2.20.04.txt{color}
| {color:red} hadoop-yarn in the patch failed with JDK
Ubuntu-11.0.11+9-Ubuntu-0ubuntu2.20.04. {color} |
| {color:red}-1{color} | {color:red} compile {color} | {color:red} 3m
14s{color} |
{color:red}https://ci-hadoop.apache.org/job/PreCommit-YARN-Build/1169/artifact/out/patch-compile-hadoop-yarn-project_had
[jira] [Commented] (YARN-10854) Support marking inactive node as untracked without configured include path
[
https://issues.apache.org/jira/browse/YARN-10854?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17390536#comment-17390536
]
Qi Zhu commented on YARN-10854:
---
[~Tao Yang] It seems some jenkins related problems need solved.
> Support marking inactive node as untracked without configured include path
> --
>
> Key: YARN-10854
> URL: https://issues.apache.org/jira/browse/YARN-10854
> Project: Hadoop YARN
> Issue Type: Improvement
> Components: resourcemanager
>Reporter: Tao Yang
>Assignee: Tao Yang
>Priority: Major
> Attachments: YARN-10854.001.patch, YARN-10854.002.patch,
> YARN-10854.003.patch, YARN-10854.004.patch
>
>
> Currently inactive nodes which have been decommissioned/shutdown/lost for a
> while(specified expiration time defined via
> {{yarn.resourcemanager.node-removal-untracked.timeout-ms}}, 60 seconds by
> default) and not exist in both include and exclude files can be marked as
> untracked nodes and can be removed from RM state (YARN-4311). It's very
> useful when auto-scaling is enabled in elastic cloud environment, which can
> avoid unlimited increase of inactive nodes (mostly are decommissioned nodes).
> But this only works when the include path is configured, mismatched for most
> of our cloud environments without configured white list of nodes, which can
> lead to easily control for the auto-scaling of nodes without further security
> requirements.
> So I propose to support marking inactive node as untracked without configured
> include path, to be compatible with the former versions, we can add a switch
> config for this.
> Any thoughts/suggestions/feedbacks are welcome!
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
-
To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org
[jira] [Commented] (YARN-10854) Support marking inactive node as untracked without configured include path
[
https://issues.apache.org/jira/browse/YARN-10854?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17391256#comment-17391256
]
Tao Yang commented on YARN-10854:
-
Thanks [~zhuqi] for the review.
Attached v5 patch to replace illegal import class
'com.google.common.collect.Sets' with 'org.apache.hadoop.util.Sets'.
> Support marking inactive node as untracked without configured include path
> --
>
> Key: YARN-10854
> URL: https://issues.apache.org/jira/browse/YARN-10854
> Project: Hadoop YARN
> Issue Type: Improvement
> Components: resourcemanager
>Reporter: Tao Yang
>Assignee: Tao Yang
>Priority: Major
> Attachments: YARN-10854.001.patch, YARN-10854.002.patch,
> YARN-10854.003.patch, YARN-10854.004.patch, YARN-10854.005.patch
>
>
> Currently inactive nodes which have been decommissioned/shutdown/lost for a
> while(specified expiration time defined via
> {{yarn.resourcemanager.node-removal-untracked.timeout-ms}}, 60 seconds by
> default) and not exist in both include and exclude files can be marked as
> untracked nodes and can be removed from RM state (YARN-4311). It's very
> useful when auto-scaling is enabled in elastic cloud environment, which can
> avoid unlimited increase of inactive nodes (mostly are decommissioned nodes).
> But this only works when the include path is configured, mismatched for most
> of our cloud environments without configured white list of nodes, which can
> lead to easily control for the auto-scaling of nodes without further security
> requirements.
> So I propose to support marking inactive node as untracked without configured
> include path, to be compatible with the former versions, we can add a switch
> config for this.
> Any thoughts/suggestions/feedbacks are welcome!
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
-
To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org
[jira] [Commented] (YARN-10854) Support marking inactive node as untracked without configured include path
[
https://issues.apache.org/jira/browse/YARN-10854?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17391464#comment-17391464
]
Hadoop QA commented on YARN-10854:
--
| (/) *{color:green}+1 overall{color}* |
\\
\\
|| Vote || Subsystem || Runtime || Logfile || Comment ||
| {color:blue}0{color} | {color:blue} reexec {color} | {color:blue} 2m
2s{color} | {color:blue}{color} | {color:blue} Docker mode activated. {color} |
|| || || || {color:brown} Prechecks {color} || ||
| {color:green}+1{color} | {color:green} dupname {color} | {color:green} 0m
0s{color} | {color:green}{color} | {color:green} No case conflicting files
found. {color} |
| {color:green}+1{color} | {color:green} @author {color} | {color:green} 0m
0s{color} | {color:green}{color} | {color:green} The patch does not contain any
@author tags. {color} |
| {color:green}+1{color} | {color:green} {color} | {color:green} 0m 0s{color}
| {color:green}test4tests{color} | {color:green} The patch appears to include 1
new or modified test files. {color} |
|| || || || {color:brown} trunk Compile Tests {color} || ||
| {color:blue}0{color} | {color:blue} mvndep {color} | {color:blue} 11m
50s{color} | {color:blue}{color} | {color:blue} Maven dependency ordering for
branch {color} |
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 26m
19s{color} | {color:green}{color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 13m
0s{color} | {color:green}{color} | {color:green} trunk passed with JDK
Ubuntu-11.0.11+9-Ubuntu-0ubuntu2.20.04 {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 11m
33s{color} | {color:green}{color} | {color:green} trunk passed with JDK Private
Build-1.8.0_292-8u292-b10-0ubuntu1~20.04-b10 {color} |
| {color:green}+1{color} | {color:green} checkstyle {color} | {color:green} 1m
53s{color} | {color:green}{color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} mvnsite {color} | {color:green} 3m
29s{color} | {color:green}{color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} shadedclient {color} | {color:green}
22m 55s{color} | {color:green}{color} | {color:green} branch has no errors when
building and testing our client artifacts. {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 3m
10s{color} | {color:green}{color} | {color:green} trunk passed with JDK
Ubuntu-11.0.11+9-Ubuntu-0ubuntu2.20.04 {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 3m
4s{color} | {color:green}{color} | {color:green} trunk passed with JDK Private
Build-1.8.0_292-8u292-b10-0ubuntu1~20.04-b10 {color} |
| {color:blue}0{color} | {color:blue} spotbugs {color} | {color:blue} 36m
2s{color} | {color:blue}{color} | {color:blue} Both FindBugs and SpotBugs are
enabled, using SpotBugs. {color} |
| {color:green}+1{color} | {color:green} spotbugs {color} | {color:green} 6m
58s{color} | {color:green}{color} | {color:green} trunk passed {color} |
|| || || || {color:brown} Patch Compile Tests {color} || ||
| {color:blue}0{color} | {color:blue} mvndep {color} | {color:blue} 0m
28s{color} | {color:blue}{color} | {color:blue} Maven dependency ordering for
patch {color} |
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 2m
30s{color} | {color:green}{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 11m
38s{color} | {color:green}{color} | {color:green} the patch passed with JDK
Ubuntu-11.0.11+9-Ubuntu-0ubuntu2.20.04 {color} |
| {color:green}+1{color} | {color:green} javac {color} | {color:green} 11m
38s{color} | {color:green}{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 10m
59s{color} | {color:green}{color} | {color:green} the patch passed with JDK
Private Build-1.8.0_292-8u292-b10-0ubuntu1~20.04-b10 {color} |
| {color:green}+1{color} | {color:green} javac {color} | {color:green} 10m
59s{color} | {color:green}{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} checkstyle {color} | {color:green} 1m
46s{color} | {color:green}{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} mvnsite {color} | {color:green} 3m
10s{color} | {color:green}{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} whitespace {color} | {color:green} 0m
0s{color} | {color:green}{color} | {color:green} The patch has no whitespace
issues. {color} |
| {color:green}+1{color} | {color:green} xml {color} | {color:green} 0m
2s{color} | {color:green}{color} | {color:green} The patch has no ill-formed
XML file. {color} |
| {color:green}+1{color} | {color:gree
[jira] [Commented] (YARN-10854) Support marking inactive node as untracked without configured include path
[
https://issues.apache.org/jira/browse/YARN-10854?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17391496#comment-17391496
]
Qi Zhu commented on YARN-10854:
---
Thanks [~Tao Yang] for contribution, and [~templedf] [~prabhujoseph] for review!
Committed to trunk.
> Support marking inactive node as untracked without configured include path
> --
>
> Key: YARN-10854
> URL: https://issues.apache.org/jira/browse/YARN-10854
> Project: Hadoop YARN
> Issue Type: Improvement
> Components: resourcemanager
>Reporter: Tao Yang
>Assignee: Tao Yang
>Priority: Major
> Attachments: YARN-10854.001.patch, YARN-10854.002.patch,
> YARN-10854.003.patch, YARN-10854.004.patch, YARN-10854.005.patch
>
>
> Currently inactive nodes which have been decommissioned/shutdown/lost for a
> while(specified expiration time defined via
> {{yarn.resourcemanager.node-removal-untracked.timeout-ms}}, 60 seconds by
> default) and not exist in both include and exclude files can be marked as
> untracked nodes and can be removed from RM state (YARN-4311). It's very
> useful when auto-scaling is enabled in elastic cloud environment, which can
> avoid unlimited increase of inactive nodes (mostly are decommissioned nodes).
> But this only works when the include path is configured, mismatched for most
> of our cloud environments without configured white list of nodes, which can
> lead to easily control for the auto-scaling of nodes without further security
> requirements.
> So I propose to support marking inactive node as untracked without configured
> include path, to be compatible with the former versions, we can add a switch
> config for this.
> Any thoughts/suggestions/feedbacks are welcome!
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
-
To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org
[jira] [Commented] (YARN-10854) Support marking inactive node as untracked without configured include path
[
https://issues.apache.org/jira/browse/YARN-10854?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17391851#comment-17391851
]
Tao Yang commented on YARN-10854:
-
Thanks [~zhuqi], [~templedf], [~prabhujoseph] and [~kshukla] !
> Support marking inactive node as untracked without configured include path
> --
>
> Key: YARN-10854
> URL: https://issues.apache.org/jira/browse/YARN-10854
> Project: Hadoop YARN
> Issue Type: Improvement
> Components: resourcemanager
>Reporter: Tao Yang
>Assignee: Tao Yang
>Priority: Major
> Fix For: 3.4.0
>
> Attachments: YARN-10854.001.patch, YARN-10854.002.patch,
> YARN-10854.003.patch, YARN-10854.004.patch, YARN-10854.005.patch
>
>
> Currently inactive nodes which have been decommissioned/shutdown/lost for a
> while(specified expiration time defined via
> {{yarn.resourcemanager.node-removal-untracked.timeout-ms}}, 60 seconds by
> default) and not exist in both include and exclude files can be marked as
> untracked nodes and can be removed from RM state (YARN-4311). It's very
> useful when auto-scaling is enabled in elastic cloud environment, which can
> avoid unlimited increase of inactive nodes (mostly are decommissioned nodes).
> But this only works when the include path is configured, mismatched for most
> of our cloud environments without configured white list of nodes, which can
> lead to easily control for the auto-scaling of nodes without further security
> requirements.
> So I propose to support marking inactive node as untracked without configured
> include path, to be compatible with the former versions, we can add a switch
> config for this.
> Any thoughts/suggestions/feedbacks are welcome!
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
-
To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org
