Andrey Yarovoy created HDDS-17730:
-------------------------------------
Summary: Ratis write that reaches a container replica while a
forced delete is removing it throws IllegalStateException and closes the
pipeline
Key: HDDS-17730
URL: https://issues.apache.org/jira/browse/HDDS-17730
Project: Apache Ozone
Issue Type: Bug
Components: Ozone Datanode
Reporter: Andrey Yarovoy
h3. Mechanism
{\{HddsDispatcher.dispatchRequest}} looks up the container once, at the start,
and passes that object to the handler. \{{KeyValueHandler.deleteInternal}}
deletes the replica under the container write lock, in this order:
# \{{markContainerForDelete}} sets the state to DELETED;
# \{{containerSet.removeContainer}} removes it from the set.
A write that looked up the container before step 2 then works on a container in
DELETED state:
# \{{KeyValueHandler.checkContainerOpen}} has no case for DELETED and returns
\{{CONTAINER_INTERNAL_ERROR}}.
# That result is not in \{{canIgnoreException}}, so the dispatcher handles it
as a failed write and tries to mark the replica unhealthy. Its
\{{Preconditions.checkState}} then throws \{{IllegalStateException}}: "Expected
container ... to be in OPEN/CLOSING/RECOVERING state but was DELETED".
# In \{{writeStateMachineData}} the exception marks the state machine
unhealthy. In \{{applyTransaction}} it fails the apply. Either way the pipeline
closes.
A write that looks up the container after step 2 does not hit this. It gets
\{{CONTAINER_NOT_OPEN}} with HDDS-17643, or before that fix recreates the
container.
h3. Trigger
* SCM force-deletes a replica on a datanode while the replica's pipeline is
open: the source of a balancer move, or an excess replica.
* A client keeps writing to a block of that container. The write must reach the
datanode while the delete is between its two steps.
The window is short, but the trigger is the same as in HDDS-17643, which was
seen thousands of times on a production cluster.
h3. Impact
The pipeline closes for a write that should only have been refused. The data is
not affected: the replica is deleted as intended.
h3. Reproduction
Confirmed with a unit test in \{{TestHddsDispatcher}}:
# Create a container through a Ratis WriteChunk (write stage, with a
\{{container2BCSIDMap}}).
# Call \{{markContainerForDelete()}} on it without removing it from the
container set.
# Dispatch a WriteChunk in the write stage and a PutBlock with an
applyTransaction context.
Both throw \{{IllegalStateException: Expected container 1 to be in
OPEN/CLOSING/RECOVERING state but was DELETED}}.
h3. Suggested fix
In \{{KeyValueHandler.checkContainerOpen}}, return \{{CONTAINER_NOT_OPEN}} for
DELETED, as a refused write to a container that is no longer open.
\{{ContainerStateMachine}} ignores that result in both stages, and the client
excludes the container.
* Alternatively, add DELETED to the states that skip marking the replica
unhealthy, as \{{CLOSED_CONTAINER_IO}} is skipped.
* Test: the steps above, asserting \{{CONTAINER_NOT_OPEN}} and that the
container is not marked UNHEALTHY.
HDDS-17432 is related: a stream PutBlock racing a close reaches the same
\{{IllegalStateException}} through \{{CONTAINER_INTERNAL_ERROR}}. Its proposed
fix re-checks the state under the container lock for a close and does not cover
DELETED.
Found by code review while validating the HDDS-17643 fix. Checked against HDDS
issues and apache/ozone pull requests for duplicates.
--
This message was sent by Atlassian Jira
(v10.3.26#10030026)
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]