Sergey Soldatov created HDDS-17767:
--------------------------------------

             Summary: Streaming reads hold XceiverClientGrpc request permits 
for the lifetime of the block stream
                 Key: HDDS-17767
                 URL: https://issues.apache.org/jira/browse/HDDS-17767
             Project: Apache Ozone
          Issue Type: Bug
          Components: Ozone Datanode
    Affects Versions: 2.2.2
            Reporter: Sergey Soldatov
            Assignee: Sergey Soldatov


{{XceiverClientGrpc#initStreamRead}} takes a permit from the request semaphore 
that also limits unary requests 
({{hdds.ratis.raft.client.async.outstanding-requests.max}}). The permit is only 
returned by {{completeStreamRead()}} when the block input stream closes, so 
each open streaming read holds one for as long as it stays open.

This causes two problems:

* *Unary requests are starved.* Long-lived block streams use up the permits 
that short requests to the same pipeline need. With enough streams open, those 
requests wait until a stream closes.
* *Readers can hang.* {{semaphore.acquire()}} has no timeout. If every permit 
is held by open streams, the next stream open waits with no limit. A reader 
that already holds streams on the client waits for a permit only it can 
release, so the read never finishes. Two common cases are a positioned read 
while a sequential stream is open, and a key input stream that has seeked into 
several blocks.

Streaming read concurrency also has no limit of its own. It shares the request 
limit, so it can't be tuned separately, even though each open stream can buffer 
up to {{ozone.client.stream.read.pre-read-size}} bytes.

Fix: streaming reads get their own permit pool, sized by the new 
{{ozone.client.stream.read.max-concurrent-streams}}. When it is unset, it uses 
the request limit as before. Waiting for a stream permit can be bounded by 
{{ozone.client.stream.read.timeout}}, and a timed-out wait is retried under the 
read retry policy.



--
This message was sent by Atlassian Jira
(v10.3.26#10030026)

---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]

Reply via email to