[
https://issues.apache.org/jira/browse/HDDS-1569?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16921587#comment-16921587
]
Siddharth Wagle edited comment on HDDS-1569 at 9/3/19 5:28 PM:
---------------------------------------------------------------
[~timmylicheng] I believe by internal data structures, I meant the
_PipelineStateMap_ but I will take a look at the doc/code and get back if any
other data structures that need an update.
The only way new pipelines are created should be through the Background
pipeline creator job. We should not create any pipelines on client requests, in
fact, we should assume pipelines are available to SCM already and no ad-hoc
pipelines will be created. If a single thread creates pipelines, not need to
blocking queues or synchronization, except the utilization counters used for
selecting pipeline need to be Atomic (current its round-robin allocation).
was (Author: swagle):
[~timmylicheng] I believe by internal data structures, I meant the
_PipelineStateMap_ but I will take a look at the doc/code and get back if any
other data structures that need an update.
The only way new pipelines are created should be through the Background
pipeline creator job. We should not create any pipelines on client requests, in
fact, we should assume pipelines are available to SCM already and no ad-hoc
pipelines will be created. If a single thread creates pipelines, not need to
blocking queues or synchronization, except the utilization counters used for
selecting pipeline need to be Atomic.
> Add ability to SCM for creating multiple pipelines with same datanode
> ---------------------------------------------------------------------
>
> Key: HDDS-1569
> URL: https://issues.apache.org/jira/browse/HDDS-1569
> Project: Hadoop Distributed Data Store
> Issue Type: Sub-task
> Components: SCM
> Reporter: Siddharth Wagle
> Assignee: Li Cheng
> Priority: Major
>
> - Refactor _RatisPipelineProvider.create()_ to be able to create pipelines
> with datanodes that are not a part of sufficient pipelines
> - Define soft and hard upper bounds for pipeline membership
> - Create SCMAllocationManager that can be leveraged to get a candidate set of
> datanodes based on placement policies
> - Add the datanodes to internal datastructures
--
This message was sent by Atlassian Jira
(v8.3.2#803003)
---------------------------------------------------------------------
To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
