Andrey Yarovoy created HDDS-17730:
-------------------------------------

             Summary: Ratis write that reaches a container replica while a 
forced delete is removing it throws IllegalStateException and closes the 
pipeline
                 Key: HDDS-17730
                 URL: https://issues.apache.org/jira/browse/HDDS-17730
             Project: Apache Ozone
          Issue Type: Bug
          Components: Ozone Datanode
            Reporter: Andrey Yarovoy


h3. Mechanism
{\{HddsDispatcher.dispatchRequest}} looks up the container once, at the start, 
and passes that object to the handler. \{{KeyValueHandler.deleteInternal}} 
deletes the replica under the container write lock, in this order:
# \{{markContainerForDelete}} sets the state to DELETED;
# \{{containerSet.removeContainer}} removes it from the set.

A write that looked up the container before step 2 then works on a container in 
DELETED state:
# \{{KeyValueHandler.checkContainerOpen}} has no case for DELETED and returns 
\{{CONTAINER_INTERNAL_ERROR}}.
# That result is not in \{{canIgnoreException}}, so the dispatcher handles it 
as a failed write and tries to mark the replica unhealthy. Its 
\{{Preconditions.checkState}} then throws \{{IllegalStateException}}: "Expected 
container ... to be in OPEN/CLOSING/RECOVERING state but was DELETED".
# In \{{writeStateMachineData}} the exception marks the state machine 
unhealthy. In \{{applyTransaction}} it fails the apply. Either way the pipeline 
closes.

A write that looks up the container after step 2 does not hit this. It gets 
\{{CONTAINER_NOT_OPEN}} with HDDS-17643, or before that fix recreates the 
container.

h3. Trigger
* SCM force-deletes a replica on a datanode while the replica's pipeline is 
open: the source of a balancer move, or an excess replica.
* A client keeps writing to a block of that container. The write must reach the 
datanode while the delete is between its two steps.

The window is short, but the trigger is the same as in HDDS-17643, which was 
seen thousands of times on a production cluster.

h3. Impact
The pipeline closes for a write that should only have been refused. The data is 
not affected: the replica is deleted as intended.

h3. Reproduction
Confirmed with a unit test in \{{TestHddsDispatcher}}:
# Create a container through a Ratis WriteChunk (write stage, with a 
\{{container2BCSIDMap}}).
# Call \{{markContainerForDelete()}} on it without removing it from the 
container set.
# Dispatch a WriteChunk in the write stage and a PutBlock with an 
applyTransaction context.

Both throw \{{IllegalStateException: Expected container 1 to be in 
OPEN/CLOSING/RECOVERING state but was DELETED}}.

h3. Suggested fix
In \{{KeyValueHandler.checkContainerOpen}}, return \{{CONTAINER_NOT_OPEN}} for 
DELETED, as a refused write to a container that is no longer open. 
\{{ContainerStateMachine}} ignores that result in both stages, and the client 
excludes the container.
* Alternatively, add DELETED to the states that skip marking the replica 
unhealthy, as \{{CLOSED_CONTAINER_IO}} is skipped.
* Test: the steps above, asserting \{{CONTAINER_NOT_OPEN}} and that the 
container is not marked UNHEALTHY.

HDDS-17432 is related: a stream PutBlock racing a close reaches the same 
\{{IllegalStateException}} through \{{CONTAINER_INTERNAL_ERROR}}. Its proposed 
fix re-checks the state under the container lock for a close and does not cover 
DELETED.

Found by code review while validating the HDDS-17643 fix. Checked against HDDS 
issues and apache/ozone pull requests for duplicates.



--
This message was sent by Atlassian Jira
(v10.3.26#10030026)

---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]

Reply via email to