Sergey Soldatov created HDDS-17767:
--------------------------------------
Summary: Streaming reads hold XceiverClientGrpc request permits
for the lifetime of the block stream
Key: HDDS-17767
URL: https://issues.apache.org/jira/browse/HDDS-17767
Project: Apache Ozone
Issue Type: Bug
Components: Ozone Datanode
Affects Versions: 2.2.2
Reporter: Sergey Soldatov
Assignee: Sergey Soldatov
{{XceiverClientGrpc#initStreamRead}} takes a permit from the request semaphore
that also limits unary requests
({{hdds.ratis.raft.client.async.outstanding-requests.max}}). The permit is only
returned by {{completeStreamRead()}} when the block input stream closes, so
each open streaming read holds one for as long as it stays open.
This causes two problems:
* *Unary requests are starved.* Long-lived block streams use up the permits
that short requests to the same pipeline need. With enough streams open, those
requests wait until a stream closes.
* *Readers can hang.* {{semaphore.acquire()}} has no timeout. If every permit
is held by open streams, the next stream open waits with no limit. A reader
that already holds streams on the client waits for a permit only it can
release, so the read never finishes. Two common cases are a positioned read
while a sequential stream is open, and a key input stream that has seeked into
several blocks.
Streaming read concurrency also has no limit of its own. It shares the request
limit, so it can't be tuned separately, even though each open stream can buffer
up to {{ozone.client.stream.read.pre-read-size}} bytes.
Fix: streaming reads get their own permit pool, sized by the new
{{ozone.client.stream.read.max-concurrent-streams}}. When it is unset, it uses
the request limit as before. Waiting for a stream permit can be bounded by
{{ozone.client.stream.read.timeout}}, and a timed-out wait is retried under the
read retry policy.
--
This message was sent by Atlassian Jira
(v10.3.26#10030026)
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]