Ivan Andika created HDDS-17872:
----------------------------------
Summary: Replace readOnlyAsync API with readIndexAsync
Key: HDDS-17872
URL: https://issues.apache.org/jira/browse/HDDS-17872
Project: Apache Ozone
Issue Type: Improvement
Reporter: Ivan Andika
Assignee: Ivan Andika
RATIS-2509 introduced the readOnlyAsync to remove the serde cost by allowing
the server implementation to directly submit reads to the server without
needing to construct RaftClientRequest and ser/deserializing the payload.
However, while developing the ReadIndex batching implementation (RATIS-2524) we
found an issue while benchmarking where the actual reads are serialized instead
of being executed by the threads calling the reads (e.g. the IPC threads).
The main issue is that the current readOnlyAsync couple both the ReadIndex
barrier (i.e. readIndexAsync to send ReadIndex and waitForAdvance to wait for
the current applied index to reach the ReadIndex) and the actual read
implementation (i.e. StateMachine#query).
We use the synchronous "thenCompose" callback to couple the two phases. The
issue is that the read can actually be executed on the thread that calls
`CompletableFuture.complete` (which is usually a lot lower than the IPC
threads). Currently, the completion thread will be called on the gRPC workers
since it's called in StreamObserver.onNext in GrpcService.readIndexAsync. In
the ReadIndex batching the issue is even worse since the ReadIndex future is
batched into a single future of the batch and CompletableFuture#complete for
all these reads is called a single thread. So this can cause multiple reads to
be called one by one which kills performance (we see that linearizable read
with ReadIndex batching performs worse than linearizable read without ReadIndex
batching).
One proposed idea is to have replace the readOnlyAsync with readIndexAsync that
only does the first barrier phase. The actual reads can be done by the server
threads itself. It is then the caller responsibility to call readIndexAsync
before calling the actual read implementation. In Ozone, it will look something
like
{code:java}
division.readIndexAsync(clientId, readRequestType).get();
return stateMachine.query(request).get();{code}
Since the readOnlyAsync is not in any Ratis release, I propose to simply
replace it.
There is a precedent for this, in etcd
[https://github.com/etcd-io/etcd/blob/v3.7.4/server/etcdserver/read/read.go] ,
the etcd server implementation will call LinearizableReadNotify as a way to
trigger and wait for the readindex only, and then the subsequent read is done
in the caller thread itself.
--
This message was sent by Atlassian Jira
(v10.3.26#10030026)
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]