[
https://issues.apache.org/jira/browse/HDDS-17360?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Wei-Chiu Chuang updated HDDS-17360:
-----------------------------------
Description:
h2. Description
An EC write can fail because a datanode has marked the target container as
missing:
{code}
EC stripe write failed: S S F S S
StorageContainerException: ContainerID ... has been lost and and cannot be
recreated on this DataNode
{code}
The EC client excludes the failed pipeline and applicable failed datanodes,
then retries with another block group. However, this failure path does not
initiate closure of the affected container or pipeline in SCM. If SCM still
considers them OPEN, other writers can receive allocations targeting the same
affected container and encounter the same failure.
This was observed while writing keys with RS(3,2). The original cause of the
missing container replicas has not been established.
h2. Expected behavior
A confirmed CONTAINER_MISSING response for an OPEN EC container should lead to
reconciliation in SCM so that the affected container/pipeline stops receiving
new allocations. SCM should initiate the existing closure workflow when
appropriate.
h2. Actual behavior
The client avoids the failed pipeline through its exclusion list, but the
pipeline can remain available to other writers until a separate SCM lifecycle
or health-handling path retires it.
h2. Comparison with Ratis
In the investigated source, Ratis treats CONTAINER_MISSING during state-machine
writes as a fatal error, marking the state machine unhealthy and closing the
local Raft division. Ratis also has server-side paths that report
pipeline-close actions to SCM.
EC writes use the gRPC container service and bypass this Ratis failure
handling. The error is returned to the client without an equivalent
pipeline-close action. This distinction does not apply to every Ratis write
exception: expected container-closure responses are exempted from the fatal
state-machine path.
h2. Relevant code
* ECKeyOutputStream.excludePipelineAndFailedDN()
* WritableECContainerProvider.getContainer()
* HddsDispatcher.dispatch()
* GrpcXceiverService
* ContainerStateMachine.handleCommandResult() and write()
h2. Impact
A container-loss event affecting many OPEN EC containers may cause repeated
write failures across clients, additional block allocations and retries, and
increased write latency or eventual write failure.
h2. Suggested scope
Handle confirmed missing-container failures specifically, rather than closing
EC pipelines for every write exception. Validate the affected
container/pipeline state in SCM and reuse the existing closure workflow.
Validation should cover concurrent writers, repeated failure reports,
allocation/closure races, SCM failover, and readability and reconstruction of
previously acknowledged data.
h2. Environment and reproduction
* Replication: EC RS(3,2)
* Deployed Ozone version: not yet established
* Observed in deployment; controlled reproduction pending
was:
h2. Description
An EC write can fail because a datanode has marked the target container as
missing:
{code}
EC stripe write failed: S S F S S
StorageContainerException: ContainerID ... has been lost and cannot be
recreated on this DataNode
{code}
The EC client excludes the failed pipeline and applicable failed datanodes,
then retries with another block group. However, this failure path does not
initiate closure of the affected container or pipeline in SCM. If SCM still
considers them OPEN, other writers can receive allocations targeting the same
affected container and encounter the same failure.
This was observed while writing keys with RS(3,2). The original cause of the
missing container replicas has not been established.
h2. Expected behavior
A confirmed CONTAINER_MISSING response for an OPEN EC container should lead to
reconciliation in SCM so that the affected container/pipeline stops receiving
new allocations. SCM should initiate the existing closure workflow when
appropriate.
h2. Actual behavior
The client avoids the failed pipeline through its exclusion list, but the
pipeline can remain available to other writers until a separate SCM lifecycle
or health-handling path retires it.
h2. Comparison with Ratis
In the investigated source, Ratis treats CONTAINER_MISSING during state-machine
writes as a fatal error, marking the state machine unhealthy and closing the
local Raft division. Ratis also has server-side paths that report
pipeline-close actions to SCM.
EC writes use the gRPC container service and bypass this Ratis failure
handling. The error is returned to the client without an equivalent
pipeline-close action. This distinction does not apply to every Ratis write
exception: expected container-closure responses are exempted from the fatal
state-machine path.
h2. Relevant code
* ECKeyOutputStream.excludePipelineAndFailedDN()
* WritableECContainerProvider.getContainer()
* HddsDispatcher.dispatch()
* GrpcXceiverService
* ContainerStateMachine.handleCommandResult() and write()
h2. Impact
A container-loss event affecting many OPEN EC containers may cause repeated
write failures across clients, additional block allocations and retries, and
increased write latency or eventual write failure.
h2. Suggested scope
Handle confirmed missing-container failures specifically, rather than closing
EC pipelines for every write exception. Validate the affected
container/pipeline state in SCM and reuse the existing closure workflow.
Validation should cover concurrent writers, repeated failure reports,
allocation/closure races, SCM failover, and readability and reconstruction of
previously acknowledged data.
h2. Environment and reproduction
* Replication: EC RS(3,2)
* Deployed Ozone version: not yet established
* Observed in deployment; controlled reproduction pending
> EC pipelines remain eligible for allocation after writes fail with
> CONTAINER_MISSING
> ------------------------------------------------------------------------------------
>
> Key: HDDS-17360
> URL: https://issues.apache.org/jira/browse/HDDS-17360
> Project: Apache Ozone
> Issue Type: Bug
> Components: EC
> Reporter: Wei-Chiu Chuang
> Assignee: Andrey Yarovoy
> Priority: Major
>
> h2. Description
> An EC write can fail because a datanode has marked the target container as
> missing:
> {code}
> EC stripe write failed: S S F S S
> StorageContainerException: ContainerID ... has been lost and and cannot be
> recreated on this DataNode
> {code}
> The EC client excludes the failed pipeline and applicable failed datanodes,
> then retries with another block group. However, this failure path does not
> initiate closure of the affected container or pipeline in SCM. If SCM still
> considers them OPEN, other writers can receive allocations targeting the same
> affected container and encounter the same failure.
> This was observed while writing keys with RS(3,2). The original cause of the
> missing container replicas has not been established.
> h2. Expected behavior
> A confirmed CONTAINER_MISSING response for an OPEN EC container should lead
> to reconciliation in SCM so that the affected container/pipeline stops
> receiving new allocations. SCM should initiate the existing closure workflow
> when appropriate.
> h2. Actual behavior
> The client avoids the failed pipeline through its exclusion list, but the
> pipeline can remain available to other writers until a separate SCM lifecycle
> or health-handling path retires it.
> h2. Comparison with Ratis
> In the investigated source, Ratis treats CONTAINER_MISSING during
> state-machine writes as a fatal error, marking the state machine unhealthy
> and closing the local Raft division. Ratis also has server-side paths that
> report pipeline-close actions to SCM.
> EC writes use the gRPC container service and bypass this Ratis failure
> handling. The error is returned to the client without an equivalent
> pipeline-close action. This distinction does not apply to every Ratis write
> exception: expected container-closure responses are exempted from the fatal
> state-machine path.
> h2. Relevant code
> * ECKeyOutputStream.excludePipelineAndFailedDN()
> * WritableECContainerProvider.getContainer()
> * HddsDispatcher.dispatch()
> * GrpcXceiverService
> * ContainerStateMachine.handleCommandResult() and write()
> h2. Impact
> A container-loss event affecting many OPEN EC containers may cause repeated
> write failures across clients, additional block allocations and retries, and
> increased write latency or eventual write failure.
> h2. Suggested scope
> Handle confirmed missing-container failures specifically, rather than closing
> EC pipelines for every write exception. Validate the affected
> container/pipeline state in SCM and reuse the existing closure workflow.
> Validation should cover concurrent writers, repeated failure reports,
> allocation/closure races, SCM failover, and readability and reconstruction of
> previously acknowledged data.
> h2. Environment and reproduction
> * Replication: EC RS(3,2)
> * Deployed Ozone version: not yet established
> * Observed in deployment; controlled reproduction pending
--
This message was sent by Atlassian Jira
(v10.3.26#10030026)
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
