[
https://issues.apache.org/jira/browse/HDDS-17845?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=18146432#comment-18146432
]
Tsz-wo Sze edited comment on HDDS-17845 at 11/21/26, 5:49 PM:
--------------------------------------------------------------
Note 1:
bq. Does Per-pipeline RocksDB marker need to add a new version in
HDDSLayoutFeature?
Claude: No, this change doesn't need a new HDDSLayoutFeature. A layout feature
has to gate a change when software from before finalization (or after a
downgrade) would misread or lose data the new software wrote. That isn't the
case here.
What the existing features gate:
- HBASE_SUPPORT (8): "an extra table for the last chunk of blocks". It's gated
because with incremental chunk lists, part of a block's metadata lives only in
the new table. An older datanode would ignore that table and see the block as
truncated, which is data loss. So until finalization, BlockManagerImpl rejects
partial chunk lists (:259) and KeyValueHandler rejects FinalizeBlock (:749).
- WITNESSED_CONTAINER_DB_PROTO_VALUE (9): HDDS-13176 changed the values of an
existing table from string to proto, and 2.0.x can't parse the proto values.
Hence the PreviousVersionTables fallback until finalization.
Why PipelineCreateInfoTable is different:
1. No existing data changes. It's a new column family, and no existing table or
value format changes.
2. Older versions open it fine. 2.0.0, 2.1.0 and 2.2.0 all open column families
they don't know about (RocksDatabase.getExtraColumnFamilies), so a downgraded
datanode still opens the witnessed DB. A new datanode creates the empty column
family at startup even before finalization, which is harmless.
3. Nothing is lost if an older version ignores it. Entries are only written for
CHAINED pipelines, and the table only records the type, not data. Committed
blocks stay in the container DB either way. A downgraded datanode would load
such a group as RATIS, but released versions have no CHAINED and no
stream-command PutBlock, so there's nothing CHAINED for them to serve.
4. No CHAINED pipelines exist yet. SCM can't create CHAINED pipelines
(PipelineFactory has no CHAINED provider), so in practice nothing writes to the
table today.
Where a layout feature will likely be needed: step 2. When SCM starts creating
CHAINED pipelines, it must not do so before the cluster is finalized, or a
datanode that's still downgradable could end up serving a CHAINED pipeline.
That's the natural place for a new feature such as CHAINED_REPLICATION(11, …),
checked by SCM before it creates CHAINED pipelines. Separately, SCM should only
pick datanodes that support CHAINED, using a datanode version check like the
HDDSVersion checks the stream client already does.
Once that feature exists, the datanode could also reject addGroup(…, CHAINED,
…) until it's finalized, as a defensive check. That fits better in step 2 than
in this change.
was (Author: szetszwo):
bq. Does Per-pipeline RocksDB marker need to add a new version in
HDDSLayoutFeature?
Claude: No, this change doesn't need a new HDDSLayoutFeature. A layout feature
has to gate a change when software from before finalization (or after a
downgrade) would misread or lose data the new software wrote. That isn't the
case here.
What the existing features gate:
- HBASE_SUPPORT (8): "an extra table for the last chunk of blocks". It's gated
because with incremental chunk lists, part of a block's metadata lives only in
the new table. An older datanode would ignore that table and see the block as
truncated, which is data loss. So until finalization, BlockManagerImpl rejects
partial chunk lists (:259) and KeyValueHandler rejects FinalizeBlock (:749).
- WITNESSED_CONTAINER_DB_PROTO_VALUE (9): HDDS-13176 changed the values of an
existing table from string to proto, and 2.0.x can't parse the proto values.
Hence the PreviousVersionTables fallback until finalization.
Why PipelineCreateInfoTable is different:
1. No existing data changes. It's a new column family, and no existing table or
value format changes.
2. Older versions open it fine. 2.0.0, 2.1.0 and 2.2.0 all open column families
they don't know about (RocksDatabase.getExtraColumnFamilies), so a downgraded
datanode still opens the witnessed DB. A new datanode creates the empty column
family at startup even before finalization, which is harmless.
3. Nothing is lost if an older version ignores it. Entries are only written for
CHAINED pipelines, and the table only records the type, not data. Committed
blocks stay in the container DB either way. A downgraded datanode would load
such a group as RATIS, but released versions have no CHAINED and no
stream-command PutBlock, so there's nothing CHAINED for them to serve.
4. No CHAINED pipelines exist yet. SCM can't create CHAINED pipelines
(PipelineFactory has no CHAINED provider), so in practice nothing writes to the
table today.
Where a layout feature will likely be needed: step 2. When SCM starts creating
CHAINED pipelines, it must not do so before the cluster is finalized, or a
datanode that's still downgradable could end up serving a CHAINED pipeline.
That's the natural place for a new feature such as CHAINED_REPLICATION(11, …),
checked by SCM before it creates CHAINED pipelines. Separately, SCM should only
pick datanodes that support CHAINED, using a datanode version check like the
HDDSVersion checks the stream client already does.
Once that feature exists, the datanode could also reject addGroup(…, CHAINED,
…) until it's finalized, as a defensive check. That fits better in step 2 than
in this change.
> Restore ReplicationType in ContainerStateMachine when a datanode restart
> ------------------------------------------------------------------------
>
> Key: HDDS-17845
> URL: https://issues.apache.org/jira/browse/HDDS-17845
> Project: Apache Ozone
> Issue Type: Sub-task
> Components: Ozone Datanode
> Reporter: Tsz-wo Sze
> Assignee: Tsz-wo Sze
> Priority: Major
> Labels: pull-request-available
>
> (Assisted by Claude)
> When a datanode restart, we need some way to tell the ReplicationType of an
> open container in order to recreate the ContainerStateMachine. Two approaches
> below are considered, SCM reply vs Per-pipeline RocksDB marker:
> || ||SCM reply||Per-pipeline RocksDB marker||
> |On-disk change|none|new column family in the witnessed DB|
> |Protocol change|PipelineReport field and handling|none|
> |Unknown window after restart|~registration + 1 heartbeat, behaving
> permissively|none: known before log replay|
> |Source of truth|SCM, so it can't go stale|the datanode, so a crash between
> steps can leave a stray marker|
> |State machine type|mutable, set once|final, and the subclass split stays
> possible|
> Although SCM reply does not requires on-disk change, there are may edge cases
> listed below. Therefore, we will implement Per-pipeline RocksDB marker
> - SCM doesn't know the pipeline: the existing ClosePipelineCommand path
> removes the group, so the type never matters.
> - Datanode was stale or dead: SCM has already closed the pipeline, which is
> the self-healing path from before.
> - Old SCM, new datanode: the old SCM ignores the flag, so the type stays
> unknown and the datanode keeps allowing both kinds. Safe, just without the
> checks.
> - New SCM, old datanode: an old datanode never sets the flag, so nothing
> changes for it.
> - SCM unreachable or not leader: the type stays unknown until the leader
> answers.
--
This message was sent by Atlassian Jira
(v10.3.26#10030026)
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]