[
https://issues.apache.org/jira/browse/HDDS-17872?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Ivan Andika updated HDDS-17872:
-------------------------------
Description:
RATIS-2509 introduced the readOnlyAsync to remove the serde cost by allowing
the server implementation to directly submit reads to the server without
needing to construct RaftClientRequest and ser/deserializing the payload.
However, while developing the ReadIndex batching implementation (RATIS-2524) we
found an issue while benchmarking where the actual reads are serialized instead
of being executed by the caller threads (e.g. the IPC threads).
The main issue is that the current readOnlyAsync couple both the ReadIndex
barrier (i.e. readIndexAsync to send ReadIndex and waitForAdvance to wait for
the current applied index to reach the ReadIndex) and the actual read
implementation (i.e. StateMachine#query).
We use the synchronous "thenCompose" callback to couple the two phases. The
issue is that the read can actually be executed on the thread that calls
`CompletableFuture.complete` (which is usually a lot lower than the IPC
threads). Currently, the completion thread will be called on the gRPC workers
since it's called in StreamObserver.onNext in GrpcService.readIndexAsync. In
the ReadIndex batching the issue is even worse since the ReadIndex future is
batched into a single future of the batch and CompletableFuture#complete for
all these reads is called a single thread. So this can cause multiple reads to
be called one by one which kills performance (we see that linearizable read
with ReadIndex batching performs worse than linearizable read without ReadIndex
batching).
One proposed idea is to have replace the readOnlyAsync with readIndexAsync that
only does the first barrier phase. The actual reads can be done by the server
threads itself. It is then the caller responsibility to call readIndexAsync
before calling the actual read implementation. In Ozone, it will look something
like
{code:java}
division.readIndexAsync(clientId, readRequestType).get();
return stateMachine.query(request).get();{code}
Since the readOnlyAsync is not in any Ratis release, I propose to simply
replace it.
There is a precedent for this, in etcd
[https://github.com/etcd-io/etcd/blob/v3.7.4/server/etcdserver/read/read.go] ,
the etcd server implementation will call LinearizableReadNotify as a way to
trigger and wait for the readindex only, and then the subsequent read is done
in the caller thread itself.
was:
RATIS-2509 introduced the readOnlyAsync to remove the serde cost by allowing
the server implementation to directly submit reads to the server without
needing to construct RaftClientRequest and ser/deserializing the payload.
However, while developing the ReadIndex batching implementation (RATIS-2524) we
found an issue while benchmarking where the actual reads are serialized instead
of being executed by the threads calling the reads (e.g. the IPC threads).
The main issue is that the current readOnlyAsync couple both the ReadIndex
barrier (i.e. readIndexAsync to send ReadIndex and waitForAdvance to wait for
the current applied index to reach the ReadIndex) and the actual read
implementation (i.e. StateMachine#query).
We use the synchronous "thenCompose" callback to couple the two phases. The
issue is that the read can actually be executed on the thread that calls
`CompletableFuture.complete` (which is usually a lot lower than the IPC
threads). Currently, the completion thread will be called on the gRPC workers
since it's called in StreamObserver.onNext in GrpcService.readIndexAsync. In
the ReadIndex batching the issue is even worse since the ReadIndex future is
batched into a single future of the batch and CompletableFuture#complete for
all these reads is called a single thread. So this can cause multiple reads to
be called one by one which kills performance (we see that linearizable read
with ReadIndex batching performs worse than linearizable read without ReadIndex
batching).
One proposed idea is to have replace the readOnlyAsync with readIndexAsync that
only does the first barrier phase. The actual reads can be done by the server
threads itself. It is then the caller responsibility to call readIndexAsync
before calling the actual read implementation. In Ozone, it will look something
like
{code:java}
division.readIndexAsync(clientId, readRequestType).get();
return stateMachine.query(request).get();{code}
Since the readOnlyAsync is not in any Ratis release, I propose to simply
replace it.
There is a precedent for this, in etcd
[https://github.com/etcd-io/etcd/blob/v3.7.4/server/etcdserver/read/read.go] ,
the etcd server implementation will call LinearizableReadNotify as a way to
trigger and wait for the readindex only, and then the subsequent read is done
in the caller thread itself.
> Replace readOnlyAsync API with readIndexAsync
> ---------------------------------------------
>
> Key: HDDS-17872
> URL: https://issues.apache.org/jira/browse/HDDS-17872
> Project: Apache Ozone
> Issue Type: Improvement
> Reporter: Ivan Andika
> Assignee: Ivan Andika
> Priority: Major
>
> RATIS-2509 introduced the readOnlyAsync to remove the serde cost by allowing
> the server implementation to directly submit reads to the server without
> needing to construct RaftClientRequest and ser/deserializing the payload.
> However, while developing the ReadIndex batching implementation (RATIS-2524)
> we found an issue while benchmarking where the actual reads are serialized
> instead of being executed by the caller threads (e.g. the IPC threads).
> The main issue is that the current readOnlyAsync couple both the ReadIndex
> barrier (i.e. readIndexAsync to send ReadIndex and waitForAdvance to wait for
> the current applied index to reach the ReadIndex) and the actual read
> implementation (i.e. StateMachine#query).
> We use the synchronous "thenCompose" callback to couple the two phases. The
> issue is that the read can actually be executed on the thread that calls
> `CompletableFuture.complete` (which is usually a lot lower than the IPC
> threads). Currently, the completion thread will be called on the gRPC workers
> since it's called in StreamObserver.onNext in GrpcService.readIndexAsync. In
> the ReadIndex batching the issue is even worse since the ReadIndex future is
> batched into a single future of the batch and CompletableFuture#complete for
> all these reads is called a single thread. So this can cause multiple reads
> to be called one by one which kills performance (we see that linearizable
> read with ReadIndex batching performs worse than linearizable read without
> ReadIndex batching).
> One proposed idea is to have replace the readOnlyAsync with readIndexAsync
> that only does the first barrier phase. The actual reads can be done by the
> server threads itself. It is then the caller responsibility to call
> readIndexAsync before calling the actual read implementation. In Ozone, it
> will look something like
>
> {code:java}
> division.readIndexAsync(clientId, readRequestType).get();
> return stateMachine.query(request).get();{code}
>
> Since the readOnlyAsync is not in any Ratis release, I propose to simply
> replace it.
> There is a precedent for this, in etcd
> [https://github.com/etcd-io/etcd/blob/v3.7.4/server/etcdserver/read/read.go]
> , the etcd server implementation will call LinearizableReadNotify as a way to
> trigger and wait for the readindex only, and then the subsequent read is done
> in the caller thread itself.
--
This message was sent by Atlassian Jira
(v10.3.26#10030026)
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]