runzhiwang commented on a change in pull request #1371: URL: https://github.com/apache/hadoop-ozone/pull/1371#discussion_r492416925
########## File path: hadoop-hdds/server-scm/src/main/java/org/apache/hadoop/hdds/scm/pipeline/RatisPipelineProvider.java ########## @@ -98,8 +105,65 @@ private boolean exceedPipelineNumberLimit(ReplicationFactor factor) { return false; } + private Map<DatanodeDetails, Integer> getSuggestedLeaderCount( + List<DatanodeDetails> dns) { + Map<DatanodeDetails, Integer> suggestedLeaderCount = new HashMap<>(); + for (DatanodeDetails dn : dns) { + suggestedLeaderCount.put(dn, 0); + + Set<PipelineID> pipelineIDSet = getNodeManager().getPipelines(dn); + for (PipelineID pipelineID : pipelineIDSet) { + try { + Pipeline pipeline = getPipelineStateManager().getPipeline(pipelineID); + if (!pipeline.isClosed() + && dn.getUuid().equals(pipeline.getSuggestedLeaderId())) { Review comment: > Plan weight for each of node as a leader when the cluster has thousands of nodes can be difficult. If each node has similar hardware, i.e. CPU, memory, we just plan weight as now, assign each node with same leader number, it is cheap and reasonable. I think the only case we need to consider is that some nodes' hardware is weaker than other nodes' obviously. I think the weeker datanodes should engage in less pipeline than the stronger datanodes, but ozone does not support this now. If we can support this, the maxum leader number of each datanode should be less or equal to ((1/3) * the pipeline number it engaged in), and we select the datanode as the leader which has lowest value of (leader number / pipeline number it engaged in) in 3 datanodes, this is also cheap. We can change this if there is requirement in the future, but now it is enough to allocate the same leader number in each datanode. ---------------------------------------------------------------- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org --------------------------------------------------------------------- To unsubscribe, e-mail: ozone-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: ozone-issues-h...@hadoop.apache.org