[ 
https://issues.apache.org/jira/browse/HDDS-17767?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Sergey Soldatov updated HDDS-17767:
-----------------------------------
    Status: Patch Available  (was: Open)

> Streaming reads hold XceiverClientGrpc request permits for the lifetime of 
> the block stream
> -------------------------------------------------------------------------------------------
>
>                 Key: HDDS-17767
>                 URL: https://issues.apache.org/jira/browse/HDDS-17767
>             Project: Apache Ozone
>          Issue Type: Bug
>          Components: Ozone Datanode
>    Affects Versions: 2.2.2
>            Reporter: Sergey Soldatov
>            Assignee: Sergey Soldatov
>            Priority: Major
>              Labels: pull-request-available
>
> {{XceiverClientGrpc#initStreamRead}} takes a permit from the request 
> semaphore that also limits unary requests 
> ({{hdds.ratis.raft.client.async.outstanding-requests.max}}). The permit is 
> only returned by {{completeStreamRead()}} when the block input stream closes, 
> so each open streaming read holds one for as long as it stays open.
> This causes two problems:
> * *Unary requests are starved.* Long-lived block streams use up the permits 
> that short requests to the same pipeline need. With enough streams open, 
> those requests wait until a stream closes.
> * *Readers can hang.* {{semaphore.acquire()}} has no timeout. If every permit 
> is held by open streams, the next stream open waits with no limit. A reader 
> that already holds streams on the client waits for a permit only it can 
> release, so the read never finishes. Two common cases are a positioned read 
> while a sequential stream is open, and a key input stream that has seeked 
> into several blocks.
> Streaming read concurrency also has no limit of its own. It shares the 
> request limit, so it can't be tuned separately, even though each open stream 
> can buffer up to {{ozone.client.stream.read.pre-read-size}} bytes.
> Fix: streaming reads get their own permit pool, sized by the new 
> {{ozone.client.stream.read.max-concurrent-streams}}. When it is unset, it 
> uses the request limit as before. Waiting for a stream permit can be bounded 
> by {{ozone.client.stream.read.timeout}}, and a timed-out wait is retried 
> under the read retry policy.



--
This message was sent by Atlassian Jira
(v10.3.26#10030026)

---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]

Reply via email to