[
https://issues.apache.org/jira/browse/HDDS-17689?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Tsz-wo Sze updated HDDS-17689:
------------------------------
Description:
When a pipeline is created, do datanodes have the ReplicationType of the
pipeline? If yes, the CHAINED pipelines can create only CHAINED containers.
For stream init, datanode can recognises it by the pipeline ReplicationType.
Is it correct?
----
Claude:
The datanode is sent the pipeline's ReplicationType, but today it throws it
away. Your approach works once the datanode stores the type for each Raft group
and still has it after a restart.
What happens today
- SCM sends CreatePipelineCommandProto \{ pipelineID, type, factor, datanode,
... }, so each datanode does get CHAINED.
- CreatePipelineCommandHandler reads createCommand.getReplicationType() but
only logs it. XceiverServerRatis#addGroup(pipelineId, peers, priorityList) has
no type parameter, and ActivePipelineContext has no type field.
- The datanode that handles the command first creates the group on its peers
through getGroupManagementApi(peer).add(group). When a peer's own command
arrives, server.isExist(...) is already true, so the handler skips that whole
block (CreatePipelineCommandHandler:122). A peer can therefore have a CHAINED
group without ever recording its type, even if we add a type parameter to
addGroup.
- On restart, Ratis reloads groups from its storage. That storage has no type,
and no new command is sent for those groups.
What makes the approach correct
1. Record the type for each group. XceiverServerRatis keeps RaftGroupId →
ReplicationType. CreatePipelineCommandHandler sets it on every command,
including the case where the group already exists.
2. Persist it. Write a small marker (for example a type file) in the group's
state machine storage directory, and reload it with the group on restart. This
adds new on-disk datanode metadata, so I'd like your sign-off before putting it
in the RFC (CLAUDE.md's "ask first" rule). The other option, SCM re-sending the
type, would need a new command or a change to the heartbeat response, which is
a proto change.
3. Look the type up at stream init, not when the state machine is built. The
state machine can exist before the type is known (the peer case above). Stream
requests already carry getGroupId() (ContainerStateMachine:518,555), so
stream() can look up the type when the request arrives. If it is still unknown,
stream init should fail with a retriable error so the client retries or
excludes that datanode. This is a small window, because SCM opens a pipeline as
soon as every datanode reports the group.
What follows from it
- Containers. A CHAINED pipeline creates only CHAINED containers, as you said.
If the group is CHAINED, stream init skips container2BCSIDMap.putIfAbsent. That
one decision keeps CHAINED containers away from the notifyGroupRemove
quasi-close and the UNHEALTHY marking on restart.
- Close. If the pipeline still exists and its group is CHAINED, the datanode
closes the container directly. If the pipeline is gone, the close is already
force=true and closes directly. So the close path doesn't need the container's
own type persisted. This replaces the "force and not in container2BCSIDMap"
heuristic now in §4.2.
- No proto change. The stream header doesn't change; option (b) in open
question 8 isn't needed.
----
For "still has it after a restart", we can close all the CHAINED pipelines if a
datanode restart.
----
Claude:
Yes, that works. Closing every CHAINED pipeline on a restarted datanode means
the type only has to live in memory, so nothing new goes on disk. Two pieces
are missing, though.
1. Today nothing triggers the close.
- When a datanode restarts it registers again. For a node SCM already knows,
SCMNodeManager#register (the else branch, around line 470) only updates the
address, version or ports. It fires no event and doesn't touch pipelines.
- StaleNodeHandler closes a node's pipelines only after the stale interval
(ozone.scm.stale.node.interval), so a fast restart never reaches it.
- So the RFC needs one SCM change: when a known datanode re-registers, close
the CHAINED pipelines that contain it. RATIS pipelines are left alone. Closing
goes through the normal path: the containers get force=true closes and the
datanodes get ClosePipelineCommand.
2. There is a short window right after restart.
- VersionEndpointTask:90 calls ozoneContainer.start(clusterId), which starts
the Ratis and DataStream servers. That happens before registration, and
ClosePipelineCommand only arrives later in a heartbeat response.
- In between, the datanode has reloaded the Raft group, doesn't know it's
CHAINED, and can still accept streams. That can come from a client that got a
block on this pipeline before SCM closed it.
- If the datanode treats such a stream as RATIS, it adds a newly created
container to container2BCSIDMap. When the group is removed, notifyGroupRemove
then quasi-closes that replica, which breaks the "never QUASI_CLOSED" rule.
Ways to close the window:
- (a) Reject unknown-type groups. For any group reloaded at startup whose type
was never recorded, the datanode rejects StreamInitWithPutBlock with a
retriable error. The client excludes the pipeline and gets a new block. The
cost is that RATIS pipelines using
ozone.client.datastream.putblock.without.raft.enabled are also refused after a
restart, until those pipelines are replaced. Plain V2 StreamInit is not
affected.
- (b) Never use the map for this request type. StreamInitWithPutBlock never
adds to container2BCSIDMap when the group's type is unknown, which makes the
window harmless for CHAINED. The cost is that RATIS containers written that way
after a restart are left out of the bcsId restart check. That check means
little for them anyway, because they also have no Raft write entries.
was:
When a pipeline is created, do datanodes have the ReplicationType of the
pipeline? If yes, the CHAINED pipelines can create only CHAINED containers.
For stream init, datanode can recognises it by the pipeline ReplicationType.
Is it correct?
----
Claude:
The datanode is sent the pipeline's ReplicationType, but today it throws it
away. Your approach works once the datanode stores the type for each Raft group
and still has it after a restart.
What happens today
- SCM sends CreatePipelineCommandProto \{ pipelineID, type, factor, datanode,
... }, so each datanode does get CHAINED.
- CreatePipelineCommandHandler reads createCommand.getReplicationType() but
only logs it. XceiverServerRatis#addGroup(pipelineId, peers, priorityList) has
no type parameter, and ActivePipelineContext has no type field.
- The datanode that handles the command first creates the group on its peers
through getGroupManagementApi(peer).add(group). When a peer's own command
arrives, server.isExist(...) is already true, so the handler skips that whole
block (CreatePipelineCommandHandler:122). A peer can therefore have a CHAINED
group without ever recording its type, even if we add a type parameter to
addGroup.
- On restart, Ratis reloads groups from its storage. That storage has no type,
and no new command is sent for those groups.
What makes the approach correct
1. Record the type for each group. XceiverServerRatis keeps RaftGroupId →
ReplicationType. CreatePipelineCommandHandler sets it on every command,
including the case where the group already exists.
2. Persist it. Write a small marker (for example a type file) in the group's
state machine storage directory, and reload it with the group on restart. This
adds new on-disk datanode metadata, so I'd like your sign-off before putting it
in the RFC (CLAUDE.md's "ask first" rule). The other option, SCM re-sending the
type, would need a new command or a change to the heartbeat response, which is
a proto change.
3. Look the type up at stream init, not when the state machine is built. The
state machine can exist before the type is known (the peer case above). Stream
requests already carry getGroupId() (ContainerStateMachine:518,555), so
stream() can look up the type when the request arrives. If it is still unknown,
stream init should fail with a retriable error so the client retries or
excludes that datanode. This is a small window, because SCM opens a pipeline as
soon as every datanode reports the group.
What follows from it
- Containers. A CHAINED pipeline creates only CHAINED containers, as you said.
If the group is CHAINED, stream init skips container2BCSIDMap.putIfAbsent. That
one decision keeps CHAINED containers away from the notifyGroupRemove
quasi-close and the UNHEALTHY marking on restart.
- Close. If the pipeline still exists and its group is CHAINED, the datanode
closes the container directly. If the pipeline is gone, the close is already
force=true and closes directly. So the close path doesn't need the container's
own type persisted. This replaces the "force and not in container2BCSIDMap"
heuristic now in §4.2.
- No proto change. The stream header doesn't change; option (b) in open
question 8 isn't needed.
----
For "still has it after a restart", we can close all the CHAINED pipelines if a
datanode restart.
----
Claude:
Yes, that works. Closing every CHAINED pipeline on a restarted datanode means
the type only has to live in memory, so nothing new goes on disk. Two pieces
are missing, though.
1. Today nothing triggers the close.
- When a datanode restarts it registers again. For a node SCM already knows,
SCMNodeManager#register (the else branch, around line 470) only updates the
address, version or ports. It fires no event and doesn't touch pipelines.
- StaleNodeHandler closes a node's pipelines only after the stale interval
(ozone.scm.stale.node.interval), so a fast restart never reaches it.
- So the RFC needs one SCM change: when a known datanode re-registers, close
the CHAINED pipelines that contain it. RATIS pipelines are left alone. Closing
goes through the normal path: the containers get force=true closes and the
datanodes get ClosePipelineCommand.
2. There is a short window right after restart.
- VersionEndpointTask:90 calls ozoneContainer.start(clusterId), which starts
the Ratis and DataStream servers. That happens before registration, and
ClosePipelineCommand only arrives later in a heartbeat response.
- In between, the datanode has reloaded the Raft group, doesn't know it's
CHAINED, and can still accept streams. That can come from a client that got a
block on this pipeline before SCM closed it.
- If the datanode treats such a stream as RATIS, it adds a newly created
container to container2BCSIDMap. When the group is removed, notifyGroupRemove
then quasi-closes that replica, which breaks the "never QUASI_CLOSED" rule.
Ways to close the window:
- (a) Reject unknown-type groups. For any group reloaded at startup whose type
was never recorded, the datanode rejects StreamInitWithPutBlock with a
retriable error. The client excludes the pipeline and gets a new block. The
cost is that RATIS pipelines using
ozone.client.datastream.putblock.without.raft.enabled are also refused after a
restart, until those pipelines are replaced. Plain V2 StreamInit is not
affected.
- (b) Never use the map for this request type. StreamInitWithPutBlock never
adds to container2BCSIDMap when the group's type is unknown, which makes the
window harmless for CHAINED. The cost is that RATIS containers written that way
after a restart are left out of the bcsId restart check. That check means
little for them anyway, because they also have no Raft write entries.
> Support CHAINED pipeline in datadnodes
> --------------------------------------
>
> Key: HDDS-17689
> URL: https://issues.apache.org/jira/browse/HDDS-17689
> Project: Apache Ozone
> Issue Type: Sub-task
> Components: Ozone Datanode
> Reporter: Tsz-wo Sze
> Priority: Major
>
> When a pipeline is created, do datanodes have the ReplicationType of the
> pipeline? If yes, the CHAINED pipelines can create only CHAINED containers.
> For stream init, datanode can recognises it by the pipeline ReplicationType.
> Is it correct?
> ----
> Claude:
> The datanode is sent the pipeline's ReplicationType, but today it throws it
> away. Your approach works once the datanode stores the type for each Raft
> group and still has it after a restart.
> What happens today
> - SCM sends CreatePipelineCommandProto \{ pipelineID, type, factor, datanode,
> ... }, so each datanode does get CHAINED.
> - CreatePipelineCommandHandler reads createCommand.getReplicationType() but
> only logs it. XceiverServerRatis#addGroup(pipelineId, peers, priorityList)
> has no type parameter, and ActivePipelineContext has no type field.
> - The datanode that handles the command first creates the group on its peers
> through getGroupManagementApi(peer).add(group). When a peer's own command
> arrives, server.isExist(...) is already true, so the handler skips that whole
> block (CreatePipelineCommandHandler:122). A peer can therefore have a CHAINED
> group without ever recording its type, even if we add a type parameter to
> addGroup.
> - On restart, Ratis reloads groups from its storage. That storage has no
> type, and no new command is sent for those groups.
> What makes the approach correct
> 1. Record the type for each group. XceiverServerRatis keeps RaftGroupId →
> ReplicationType. CreatePipelineCommandHandler sets it on every command,
> including the case where the group already exists.
> 2. Persist it. Write a small marker (for example a type file) in the group's
> state machine storage directory, and reload it with the group on restart.
> This adds new on-disk datanode metadata, so I'd like your sign-off before
> putting it in the RFC (CLAUDE.md's "ask first" rule). The other option, SCM
> re-sending the type, would need a new command or a change to the heartbeat
> response, which is a proto change.
> 3. Look the type up at stream init, not when the state machine is built. The
> state machine can exist before the type is known (the peer case above).
> Stream requests already carry getGroupId() (ContainerStateMachine:518,555),
> so stream() can look up the type when the request arrives. If it is still
> unknown, stream init should fail with a retriable error so the client retries
> or excludes that datanode. This is a small window, because SCM opens a
> pipeline as soon as every datanode reports the group.
> What follows from it
> - Containers. A CHAINED pipeline creates only CHAINED containers, as you
> said. If the group is CHAINED, stream init skips
> container2BCSIDMap.putIfAbsent. That one decision keeps CHAINED containers
> away from the notifyGroupRemove quasi-close and the UNHEALTHY marking on
> restart.
> - Close. If the pipeline still exists and its group is CHAINED, the datanode
> closes the container directly. If the pipeline is gone, the close is already
> force=true and closes directly. So the close path doesn't need the
> container's own type persisted. This replaces the "force and not in
> container2BCSIDMap" heuristic now in §4.2.
> - No proto change. The stream header doesn't change; option (b) in open
> question 8 isn't needed.
> ----
> For "still has it after a restart", we can close all the CHAINED pipelines if
> a datanode restart.
> ----
> Claude:
> Yes, that works. Closing every CHAINED pipeline on a restarted datanode means
> the type only has to live in memory, so nothing new goes on disk. Two pieces
> are missing, though.
> 1. Today nothing triggers the close.
> - When a datanode restarts it registers again. For a node SCM already knows,
> SCMNodeManager#register (the else branch, around line 470) only updates the
> address, version or ports. It fires no event and doesn't touch pipelines.
> - StaleNodeHandler closes a node's pipelines only after the stale interval
> (ozone.scm.stale.node.interval), so a fast restart never reaches it.
> - So the RFC needs one SCM change: when a known datanode re-registers, close
> the CHAINED pipelines that contain it. RATIS pipelines are left alone.
> Closing goes through the normal path: the containers get force=true closes
> and the datanodes get ClosePipelineCommand.
> 2. There is a short window right after restart.
> - VersionEndpointTask:90 calls ozoneContainer.start(clusterId), which starts
> the Ratis and DataStream servers. That happens before registration, and
> ClosePipelineCommand only arrives later in a heartbeat response.
> - In between, the datanode has reloaded the Raft group, doesn't know it's
> CHAINED, and can still accept streams. That can come from a client that got a
> block on this pipeline before SCM closed it.
> - If the datanode treats such a stream as RATIS, it adds a newly created
> container to container2BCSIDMap. When the group is removed, notifyGroupRemove
> then quasi-closes that replica, which breaks the "never QUASI_CLOSED" rule.
> Ways to close the window:
> - (a) Reject unknown-type groups. For any group reloaded at startup whose
> type was never recorded, the datanode rejects StreamInitWithPutBlock with a
> retriable error. The client excludes the pipeline and gets a new block. The
> cost is that RATIS pipelines using
> ozone.client.datastream.putblock.without.raft.enabled are also refused after
> a restart, until those pipelines are replaced. Plain V2 StreamInit is not
> affected.
> - (b) Never use the map for this request type. StreamInitWithPutBlock never
> adds to container2BCSIDMap when the group's type is unknown, which makes the
> window harmless for CHAINED. The cost is that RATIS containers written that
> way after a restart are left out of the bcsId restart check. That check means
> little for them anyway, because they also have no Raft write entries.
--
This message was sent by Atlassian Jira
(v10.3.26#10030026)
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
