Andrey Yarovoy created HDDS-17729:
-------------------------------------
Summary: After a datanode restart, a late Ratis write to a
container that SCM deleted closes the pipeline, because the deleted container
is treated as missing
Key: HDDS-17729
URL: https://issues.apache.org/jira/browse/HDDS-17729
Project: Apache Ozone
Issue Type: Bug
Components: Ozone Datanode
Reporter: Andrey Yarovoy
h3. Mechanism
On a running datanode, a Ratis write to a container that was created over the
pipeline and later deleted gets \{{CONTAINER_NOT_OPEN}} after HDDS-17643.
\{{ContainerStateMachine}} ignores that result, and the client retries on a new
block.
After a restart the same write closes the pipeline:
# \{{ContainerStateMachine.buildMissingContainerSet}} loads
\{{container2BCSIDMap}} from the Ratis snapshot. The map also has the
containers deleted since they were created: it is never cleaned up on delete.
# \{{ContainerSet.buildMissingContainerSetAndValidate}} adds every ID of the
map that is not in the container set to the missing container set. It logs
"Adding container ... to missing container set" for each one. Nothing records
that SCM deleted the replica on purpose: the delete also removes the ID from
the container ID table (HDDS-11650).
# For a write to such a container, \{{HddsDispatcher.dispatchRequest}} returns
\{{CONTAINER_MISSING}} before any other check. The dispatcher's
\{{canIgnoreException}} lists \{{CONTAINER_MISSING}}, but
\{{ContainerStateMachine}} does not:
## in \{{handleCommandResult}} (writeStateMachineData) the state machine is
marked unhealthy;
## in \{{applyTransaction}} the apply fails and
\{{handleApplyTransactionFailure}} closes the pipeline.
HDDS-935 added the missing container set for containers lost with a disk.
Failing those writes is intended, but a container deleted by SCM ends up in the
same set.
h3. Trigger
* SCM deletes a replica on a datanode while the replica's pipeline is open: the
source of a balancer move, or an excess replica.
* The datanode restarts, and the pipeline survives it. The snapshot it loads
was taken after the container was created.
* A client that allocated a block in that container before it closed writes to
it.
h3. Impact
* The pipeline closes, and with it every open container on it.
* Every restart logs a "missing container" warning for each container ever
deleted from the pipeline.
Data is not affected: the container is not recreated.
h3. Suggested fix
Either of these:
# Return the same result as without a restart.
\{{buildMissingContainerSetAndValidate}} could skip an ID whose replica was
deleted, if the delete left a record. That record could be a tombstone in the
container ID table, which the Jira HDDS-17643 suggests as the alternative to
its fix. Writes then get \{{CONTAINER_NOT_OPEN}} from the HDDS-17643 check.
# Treat \{{CONTAINER_MISSING}} like \{{CONTAINER_NOT_OPEN}} in the two ignore
lists of \{{ContainerStateMachine}}. The client then excludes the container and
keeps the pipeline.
** For containers lost with a disk, this changes the HDDS-935 behaviour from
closing the pipeline to failing the write. SCM would then need to learn of the
loss from the container report or a close container action, as HDDS-17360 does
for EC.
h3. Reproduction (suggested)
Extend
\{{TestCloseContainerByPipeline#testWriteAfterForceDeleteOnLeaderDoesNotRecreateContainer}}
(HDDS-17643):
# Take a Ratis snapshot on the leader after the container is created.
# Restart the leader after the forced delete.
# Send the WriteChunk. It returns \{{CONTAINER_MISSING}}, and the leader's
state machine is unhealthy.
Found by code review while validating the HDDS-17643 fix. Not yet reproduced on
a cluster. Checked against HDDS issues and apache/ozone pull requests for
duplicates: HDDS-17435 and HDDS-17430 cover log replay after a restart, not
writes that arrive after it.
--
This message was sent by Atlassian Jira
(v10.3.26#10030026)
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
