[
https://issues.apache.org/jira/browse/HDDS-4260?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17198594#comment-17198594
]
Xiaoyu Yao commented on HDDS-4260:
----------------------------------
Looks deeper into this. This is more of a rebalancing pipeline issue when the
new nodes are added to the cluster. The Scrubber can be a candidate to remove
the existing pipelines so that the creator can create new ones on the new
nodes.
The existing PipelinePlacementPolicy#chooseDatanodes() already has logic to
consider the existing load on the node(pipeline count). The reason of imbalance
is cause by the time the new nodes are added, each node in the cluster already
reaches it maximum count node allowed. Pipeline creator can't create new
pipeline any more.
> Balance datanode selection for pipeline creation
> ------------------------------------------------
>
> Key: HDDS-4260
> URL: https://issues.apache.org/jira/browse/HDDS-4260
> Project: Hadoop Distributed Data Store
> Issue Type: Improvement
> Affects Versions: 1.0.0
> Reporter: Glen Geng
> Assignee: Xiaoyu Yao
> Priority: Major
>
> Recently [~glengeng] reported an unbalanced node selection when muti-raft is
> enabled in there cluster deployment. After checking the pipeline placement
> policy I found that we chooses node from healthy node one by one if topology
> aware is not configured. Even if topology aware is configured, the anchor
> node and the third nodes are supposed to be on the same rack. Without
> shuffle, it is likely to always get the same set of nodes with the last one
> being unchosen.
> This ticket is opened to add a shuffle of the halthyNodes before
> getResultSet() in
> PipelinePlacementPolicy#chooseDatanodes() to balance the node selection when
> it is invoked multiple times in the case of multi-raft.
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
---------------------------------------------------------------------
To unsubscribe, e-mail: ozone-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: ozone-issues-h...@hadoop.apache.org
