[
https://issues.apache.org/jira/browse/HDDS-16419?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
ASF GitHub Bot updated HDDS-16419:
----------------------------------
Labels: pull-request-available (was: )
> Datanode failover for GetBlock/ReadChunk is nested, causing up to N×N
> attempts per read
> ---------------------------------------------------------------------------------------
>
> Key: HDDS-16419
> URL: https://issues.apache.org/jira/browse/HDDS-16419
> Project: Apache Ozone
> Issue Type: Bug
> Reporter: Janus Chow
> Assignee: Janus Chow
> Priority: Major
> Labels: pull-request-available
>
> When a datanode is slow or unreachable, a single GetBlock or ReadChunk call
> can
> take N × N × ozone.client.read.timeout before failing, where N is the number
> of
> datanodes in the pipeline.
>
> There are two nested failover loops:
>
> 1. ContainerProtocolCalls#tryEachDatanode (added by HDDS-8024 / HDDS-8090)
> iterates over the datanodes and, for each one, calls
> XceiverClientSpi#sendCommand(request, validators).
>
> 2. XceiverClientGrpc#sendCommandWithRetry, which sendCommand delegates to,
> ignores the datanode chosen by the outer loop and again tries every
> datanode of the pipeline.
>
> So each outer attempt is a full pipeline failover, and every datanode is
> contacted N times. The outer loop sets datanodeUuid (and the replica index for
> EC) for the datanode it picked, but the inner loop may send that request to a
> different datanode.
>
> Together with the read retries in BlockInputStream, the worst case for one
> read becomes:
>
> ozone.client.read.max.retries × N × N × ozone.client.read.timeout
>
> For example, with 2 datanodes, read.timeout=5s and max.retries=3, a read to
> unresponsive datanodes fails only after 60s. The client logs show
> "Failed to execute command ... on the pipeline" every 10s, and
> "will try another datanode" after every second one. With the defaults
> (3 datanodes, 30s timeout, 3 retries) the worst case is 810s.
>
> This is a problem for services such as S3 Gateway. The S3 client usually
> times out and retries long before this, but the gateway thread stays blocked,
> so threads pile up when datanodes are slow.
>
> Neither loop can simply be removed:
> - the outer loop handles errors detected after a successful response
> (e.g. short read, readLen != chunk length) and stops early on
> BLOCK_TOKEN_VERIFICATION_FAILED;
> - the inner loop provides failover for other commands (listBlock,
> readSmallFile, ...) which do not pick a datanode themselves.
>
> Proposal: let the outer loop send each attempt only to the datanode it picked,
> so GetBlock/ReadChunk contact each datanode at most once, in the same order as
> today.
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
