[ https://issues.apache.org/jira/browse/YARN-9278?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16776184#comment-16776184 ]
Yufei Gu commented on YARN-9278: -------------------------------- bq. If our cluster has a lot of long-running jobs, the above method is not helpful. That's unfortunate. Setting a maximum num of nodes to iterate seems a quick-and-dirty way to solve the latency in big clusters. Let's brainstorm the solution. [~Steven Rand] and [~wilfreds], what do you think? > Shuffle nodes when selecting to be preempted nodes > -------------------------------------------------- > > Key: YARN-9278 > URL: https://issues.apache.org/jira/browse/YARN-9278 > Project: Hadoop YARN > Issue Type: Sub-task > Components: fairscheduler > Reporter: Zhaohui Xin > Assignee: Zhaohui Xin > Priority: Major > > We should *shuffle* the nodes to avoid some nodes being preempted frequently. > Also, we should *limit* the num of nodes to make preemption more efficient. > Just like this, > {code:java} > // we should not iterate all nodes, that will be very slow > long maxTryNodeNum = > context.getPreemptionConfig().getToBePreemptedNodeMaxNumOnce(); > if (potentialNodes.size() > maxTryNodeNum){ > Collections.shuffle(potentialNodes); > List<FSSchedulerNode> newPotentialNodes = new ArrayList<FSSchedulerNode>(); > for (int i = 0; i < maxTryNodeNum; i++){ > newPotentialNodes.add(potentialNodes.get(i)); > } > potentialNodes = newPotentialNodes; > {code} > -- This message was sent by Atlassian JIRA (v7.6.3#76005) --------------------------------------------------------------------- To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org