[
https://issues.apache.org/jira/browse/HDDS-15729?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Ivan Andika updated HDDS-15729:
-------------------------------
Description:
The SCM placement policy code structure (both Pipeline Placement Policy and the
Container Placement Policy) is poorly abstracted which causes the API to be
very hard to reason about. There are leaky abstractions or logic specific to a
replication config (e.g. Ratis logic in interface that can also be applied to
EC). I found myself getting lost every time I look at the placement policy
logic since it's hard to build a mental model of the whole process. It's time
to take a high level view and identify these abstractions and fix them for
better readability and maintainability, as well as document the exact contracts
and responsibilities of these policies. This will minimize any surprises and
potential bugs due to misunderstandings and reduce cognitive overloads of all
the possible APIs of parent classes.
For example (not exhaustive) of issues:
* PlacementPolicy is supposed to be generic, but we have container placement
specific logic like validateContainerPlacement and
replicasToCopyToFixMisReplication and replicasToRemoveToFixOverreplication
* PipelinePlacementPolicy seems to only used for RATIS pipeline, but we have
EC pipeline, so it gives an impression that PipelinePlacementPolicy can also be
used for EC, which AFAIK is not the case.
** PipelinePlacementPolicy is only used for RATIS, then we should make it
explicit by changing the name to RatisPipelinePlacementPolicy
** PipelinePlacementPolicy also have internal "Ratis" related pipelines logic
like currentRatisThreePipelineCount
* We are still using PlacementPolicy for everything, even if
PipelinePlacementPolicy is more appropriate for pipeline related matters, which
forces PlacementPolicy to implement interfaces that it should not be
implementing
** For example, in RatisPipelineProvider we can change PlacementPolicy to
PipelinePlacementPolicy since we are dealing with pipeline
** We also don't have a separate ContainerPlacementPolicy interface that have
the logic to handle container placement, we use a generic
SCMCommonPlacementPolicy, which is extended by PipelinePlacementPolicy
* Domain concepts like "anchor node" is not properly defined
* Most of the placement policy is still based on the assumption of rack/leaf
hierarchies and clos-style classic topologies (i.e. leaf, spine, etc)
** We need to be able to be more flexible in supporting other schema
(datacenter, region, hall, spine, etc)
** For example, Ceph CRUSH allows administrator to arbitrarily setup the
schema and the CRUSH algorithm will be able to adapt to it (of course there is
risk on this and complexity, so we don't need to go to this far, but it's good
to consider)
** For example, if we only support rack related logic, then we should make it
explicit that placement policy should be used only if the schema is rack/leaf,
and nothing else, since different schema might cause issues (e.g. data loss due
to deletion or replication storms)
** In the future, we should also be able to support more complex network
topologies like Fat-trees, Toruses, Multidimensional meshes which are used for
AI related workloads
To finally test the flexibility of the placement policy improvement, we can
then implement different placement policy (i.e. DC aware placement policy like
HDDS-6818 and HDDS-11988 and/or other fundamentally different placement policy
like CopySet) and these new placement policy should be easier to be implemented
than before.
was:
The SCM placement policy code structure (both Pipeline Placement Policy and the
Container Placement Policy) is poorly abstracted which causes the API to be
very hard to reason about. There are leaky abstractions or logic specific to a
replication config (e.g. Ratis logic in interface that can also be applied to
EC). I found myself getting lost every time I look at the placement policy
logic since it's hard to build a mental model of the whole process. It's time
to take a high level view and identify these abstractions and fix them for
better readability and maintainability, as well as document the exact contracts
and responsibilities of these policies. This will minimize any surprises and
potential bugs due to misunderstandings and reduce cognitive overloads of all
the possible APIs of parent classes.
For example (not exhaustive)
* PlacementPolicy is supposed to be generic, but we have container placement
specific logic like validateContainerPlacement and
replicasToCopyToFixMisReplication and replicasToRemoveToFixOverreplication
* PipelinePlacementPolicy seems to only used for RATIS pipeline, but we have
EC pipeline, so it gives an impression that PipelinePlacementPolicy can also be
used for EC, which AFAIK is not the case.
** PipelinePlacementPolicy is only used for RATIS, then we should make it
explicit by changing the name to RatisPipelinePlacementPolicy
** PipelinePlacementPolicy also have internal "Ratis" related pipelines logic
like currentRatisThreePipelineCount
* We are still using PlacementPolicy for everything, even if
PipelinePlacementPolicy is more appropriate for pipeline related matters, which
forces PlacementPolicy to implement interfaces that it should not be
implementing
** For example, in RatisPipelineProvider we can change PlacementPolicy to
PipelinePlacementPolicy since we are dealing with pipeline
** We also don't have a separate ContainerPlacementPolicy interface that have
the logic to handle container placement, we use a generic
SCMCommonPlacementPolicy, which is extended by PipelinePlacementPolicy
* Domain concepts like "anchor node" is not properly defined
* Most of the placement policy is still based on the assumption of rack/leaf
hierarchies and clos-style classic topologies (i.e. leaf, spine, etc)
** We need to be able to be more flexible in supporting other schema
(datacenter, region, hall, spine, etc)
** For example, Ceph CRUSH allows administrator to arbitrarily setup the
schema and the CRUSH algorithm will be able to adapt to it (of course there is
risk on this and complexity, so we don't need to go to this far, but it's good
to consider)
** For example, if we only support rack related logic, then we should make it
explicit that placement policy should be used only if the schema is rack/leaf,
and nothing else, since different schema might cause issues (e.g. data loss due
to deletion or replication storms)
** In the future, we should also be able to support more complex network
topologies like Fat-trees, Toruses, Multidimensional meshes which are used for
AI related workloads
To finally test the flexibility of the placement policy improvement, we can
then implement different placement policy (i.e. DC aware placement policy like
HDDS-6818 and HDDS-11988 and/or other fundamentally different placement policy
like CopySet) and these new placement policy should be easier to be implemented
than before.
> Ozone Placement Policy API improvements
> ---------------------------------------
>
> Key: HDDS-15729
> URL: https://issues.apache.org/jira/browse/HDDS-15729
> Project: Apache Ozone
> Issue Type: Improvement
> Reporter: Ivan Andika
> Assignee: Ivan Andika
> Priority: Major
>
> The SCM placement policy code structure (both Pipeline Placement Policy and
> the Container Placement Policy) is poorly abstracted which causes the API to
> be very hard to reason about. There are leaky abstractions or logic specific
> to a replication config (e.g. Ratis logic in interface that can also be
> applied to EC). I found myself getting lost every time I look at the
> placement policy logic since it's hard to build a mental model of the whole
> process. It's time to take a high level view and identify these abstractions
> and fix them for better readability and maintainability, as well as document
> the exact contracts and responsibilities of these policies. This will
> minimize any surprises and potential bugs due to misunderstandings and reduce
> cognitive overloads of all the possible APIs of parent classes.
> For example (not exhaustive) of issues:
> * PlacementPolicy is supposed to be generic, but we have container placement
> specific logic like validateContainerPlacement and
> replicasToCopyToFixMisReplication and replicasToRemoveToFixOverreplication
> * PipelinePlacementPolicy seems to only used for RATIS pipeline, but we have
> EC pipeline, so it gives an impression that PipelinePlacementPolicy can also
> be used for EC, which AFAIK is not the case.
> ** PipelinePlacementPolicy is only used for RATIS, then we should make it
> explicit by changing the name to RatisPipelinePlacementPolicy
> ** PipelinePlacementPolicy also have internal "Ratis" related pipelines
> logic like currentRatisThreePipelineCount
> * We are still using PlacementPolicy for everything, even if
> PipelinePlacementPolicy is more appropriate for pipeline related matters,
> which forces PlacementPolicy to implement interfaces that it should not be
> implementing
> ** For example, in RatisPipelineProvider we can change PlacementPolicy to
> PipelinePlacementPolicy since we are dealing with pipeline
> ** We also don't have a separate ContainerPlacementPolicy interface that
> have the logic to handle container placement, we use a generic
> SCMCommonPlacementPolicy, which is extended by PipelinePlacementPolicy
> * Domain concepts like "anchor node" is not properly defined
> * Most of the placement policy is still based on the assumption of rack/leaf
> hierarchies and clos-style classic topologies (i.e. leaf, spine, etc)
> ** We need to be able to be more flexible in supporting other schema
> (datacenter, region, hall, spine, etc)
> ** For example, Ceph CRUSH allows administrator to arbitrarily setup the
> schema and the CRUSH algorithm will be able to adapt to it (of course there
> is risk on this and complexity, so we don't need to go to this far, but it's
> good to consider)
> ** For example, if we only support rack related logic, then we should make
> it explicit that placement policy should be used only if the schema is
> rack/leaf, and nothing else, since different schema might cause issues (e.g.
> data loss due to deletion or replication storms)
> ** In the future, we should also be able to support more complex network
> topologies like Fat-trees, Toruses, Multidimensional meshes which are used
> for AI related workloads
> To finally test the flexibility of the placement policy improvement, we can
> then implement different placement policy (i.e. DC aware placement policy
> like HDDS-6818 and HDDS-11988 and/or other fundamentally different placement
> policy like CopySet) and these new placement policy should be easier to be
> implemented than before.
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
