Wei-Chiu Chuang created HDDS-17360:
--------------------------------------
Summary: EC pipelines remain eligible for allocation after writes
fail with CONTAINER_MISSING
Key: HDDS-17360
URL: https://issues.apache.org/jira/browse/HDDS-17360
Project: Apache Ozone
Issue Type: Bug
Components: EC
Reporter: Wei-Chiu Chuang
h2. Description
An EC write can fail because a datanode has marked the target container as
missing:
{code}
EC stripe write failed: S S F S S
StorageContainerException: ContainerID ... has been lost and cannot be
recreated on this DataNode
{code}
The EC client excludes the failed pipeline and applicable failed datanodes,
then retries with another block group. However, this failure path does not
initiate closure of the affected container or pipeline in SCM. If SCM still
considers them OPEN, other writers can receive allocations targeting the same
affected container and encounter the same failure.
This was observed while writing keys with RS(3,2). The original cause of the
missing container replicas has not been established.
h2. Expected behavior
A confirmed CONTAINER_MISSING response for an OPEN EC container should lead to
reconciliation in SCM so that the affected container/pipeline stops receiving
new allocations. SCM should initiate the existing closure workflow when
appropriate.
h2. Actual behavior
The client avoids the failed pipeline through its exclusion list, but the
pipeline can remain available to other writers until a separate SCM lifecycle
or health-handling path retires it.
h2. Comparison with Ratis
In the investigated source, Ratis treats CONTAINER_MISSING during state-machine
writes as a fatal error, marking the state machine unhealthy and closing the
local Raft division. Ratis also has server-side paths that report
pipeline-close actions to SCM.
EC writes use the gRPC container service and bypass this Ratis failure
handling. The error is returned to the client without an equivalent
pipeline-close action. This distinction does not apply to every Ratis write
exception: expected container-closure responses are exempted from the fatal
state-machine path.
h2. Relevant code
* ECKeyOutputStream.excludePipelineAndFailedDN()
* WritableECContainerProvider.getContainer()
* HddsDispatcher.dispatch()
* GrpcXceiverService
* ContainerStateMachine.handleCommandResult() and write()
h2. Impact
A container-loss event affecting many OPEN EC containers may cause repeated
write failures across clients, additional block allocations and retries, and
increased write latency or eventual write failure.
h2. Suggested scope
Handle confirmed missing-container failures specifically, rather than closing
EC pipelines for every write exception. Validate the affected
container/pipeline state in SCM and reuse the existing closure workflow.
Validation should cover concurrent writers, repeated failure reports,
allocation/closure races, SCM failover, and readability and reconstruction of
previously acknowledged data.
h2. Environment and reproduction
* Replication: EC RS(3,2)
* Deployed Ozone version: not yet established
* Observed in deployment; controlled reproduction pending
--
This message was sent by Atlassian Jira
(v10.3.26#10030026)
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
