[jira] [Commented] (YARN-10738) When multi thread scheduling with multi node, we should shuffle to prevent hot accessing nodes.
[ https://issues.apache.org/jira/browse/YARN-10738?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17321936#comment-17321936 ] Qi Zhu commented on YARN-10738: --- Thanks a lot [~bibinchundatt] for reply. :D I will move it to MultiNodeLookupPolicy related. > When multi thread scheduling with multi node, we should shuffle to prevent > hot accessing nodes. > --- > > Key: YARN-10738 > URL: https://issues.apache.org/jira/browse/YARN-10738 > Project: Hadoop YARN > Issue Type: Improvement >Reporter: Qi Zhu >Assignee: Qi Zhu >Priority: Major > Labels: pull-request-available > Time Spent: 10m > Remaining Estimate: 0h > > Now the multi threading scheduling with multi node is not reasonable. > In large clusters, it will cause the hot accessing nodes, which will lead the > abnormal boom node. > Solution: > I think we should shuffle the sorted node (such the available resource sort > policy) with an interval. > I will solve the above problem, and avoid the hot accessing node. -- This message was sent by Atlassian Jira (v8.3.4#803005) - To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org
[jira] [Commented] (YARN-10738) When multi thread scheduling with multi node, we should shuffle to prevent hot accessing nodes.
[ https://issues.apache.org/jira/browse/YARN-10738?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17321928#comment-17321928 ] Bibin Chundatt commented on YARN-10738: --- [~zhuqi] I think this should be part of the *MultiNodeLookupPolicy * implementation. > When multi thread scheduling with multi node, we should shuffle to prevent > hot accessing nodes. > --- > > Key: YARN-10738 > URL: https://issues.apache.org/jira/browse/YARN-10738 > Project: Hadoop YARN > Issue Type: Improvement >Reporter: Qi Zhu >Assignee: Qi Zhu >Priority: Major > Labels: pull-request-available > Time Spent: 10m > Remaining Estimate: 0h > > Now the multi threading scheduling with multi node is not reasonable. > In large clusters, it will cause the hot accessing nodes, which will lead the > abnormal boom node. > Solution: > I think we should shuffle the sorted node (such the available resource sort > policy) with an interval. > I will solve the above problem, and avoid the hot accessing node. -- This message was sent by Atlassian Jira (v8.3.4#803005) - To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org
[jira] [Commented] (YARN-10738) When multi thread scheduling with multi node, we should shuffle to prevent hot accessing nodes.
[ https://issues.apache.org/jira/browse/YARN-10738?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17321926#comment-17321926 ] Qi Zhu commented on YARN-10738: --- cc [~ztang] [~snemeth] [~pbacsko] [~gandras] [~ebadger] [~epayne] What's your opinions about this? I think it's very important for large clusters to prevent hot accessing node. Thanks. > When multi thread scheduling with multi node, we should shuffle to prevent > hot accessing nodes. > --- > > Key: YARN-10738 > URL: https://issues.apache.org/jira/browse/YARN-10738 > Project: Hadoop YARN > Issue Type: Improvement >Reporter: Qi Zhu >Assignee: Qi Zhu >Priority: Major > Labels: pull-request-available > Time Spent: 10m > Remaining Estimate: 0h > > Now the multi threading scheduling with multi node is not reasonable. > In large clusters, it will cause the hot accessing nodes, which will lead the > abnormal boom node. > Solution: > I think we should shuffle the sorted node (such the available resource sort > policy) with an interval. > I will solve the above problem, and avoid the hot accessing node. -- This message was sent by Atlassian Jira (v8.3.4#803005) - To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org
