[
https://issues.apache.org/jira/browse/RATIS-2509?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Ivan Andika updated RATIS-2509:
-------------------------------
Description:
Recently, we did a benchmark comparing the OM leader-read that does not go
through Ratis and OM leader read that go through Ratis (through
submitClientRequestAsync). We saw that there is up to 25% decrease in read
throughput although the we make the raft.server.read.option to be DEFAULT which
should return immediately (235020 QPS -> 180433 QPS or 24% reduction in
throughput for pure reads with 100 threads).
The overheads seem to be because of request/response proto conversion,
RaftClientRequest construction, future chaining, .get() blocking, Ratis
metrics/reply building, and parsing the Ratis response back into OMResponse.
See [^om-benchmark-leader-read-all-ratis.html] for the flamegraph.
This means that if we submit linearizable read to follower, we incurs these
overhead on top of the linearizable read overhead (e.g. ReadIndex, etc).
We can try to find a way to reduce this overhead. We might need to implement
another read flow without the overhead (unlike writes which requires
AppendEntries request to the followers which require serde, read can be served
locally).
One idea is that if we are submitting to RaftServer, we can use
submitServerRequestAsync which does not require RaftClientRequest and
RaftClientReply serde.
was:
Recently, we did a benchmark comparing the OM leader-read that does not go
through Ratis and OM leader read that go through Ratis (through
submitClientRequestAsync). We saw that there is up to 25% decrease in read
throughput although the we make the raft.server.read.option to be DEFAULT which
should return immediately (235020 QPS -> 180433 QPS or 24% reduction in
throughput for pure reads with 100 threads).
The overheads seem to be because of request/response proto conversion,
RaftClientRequest construction, future chaining, .get() blocking, Ratis
metrics/reply building, and parsing the Ratis response back into OMResponse.
See [^om-benchmark-leader-read-all-ratis.html] for the flamegraph.
This means that if we submit linearizable read to follower, we incurs these
overhead on top of the linearizable read overhead (e.g. ReadIndex, etc).
We can try to find a way to reduce this overhead. We might need to implement
another read flow without the overhead since all reads (unlike writes which
requires AppendEntries request to the followers which require serde, read can
be served locally).
One idea is that if we are submitting to RaftServer, we can use
submitServerRequestAsync which does not require RaftClientRequest and
RaftClientReply serde.
> Optimization to reduce serde cost in Ratis request and response
> ---------------------------------------------------------------
>
> Key: RATIS-2509
> URL: https://issues.apache.org/jira/browse/RATIS-2509
> Project: Ratis
> Issue Type: Improvement
> Reporter: Ivan Andika
> Assignee: Ivan Andika
> Priority: Major
> Attachments: om-benchmark-leader-read-all-ratis.html
>
>
> Recently, we did a benchmark comparing the OM leader-read that does not go
> through Ratis and OM leader read that go through Ratis (through
> submitClientRequestAsync). We saw that there is up to 25% decrease in read
> throughput although the we make the raft.server.read.option to be DEFAULT
> which should return immediately (235020 QPS -> 180433 QPS or 24% reduction in
> throughput for pure reads with 100 threads).
> The overheads seem to be because of request/response proto conversion,
> RaftClientRequest construction, future chaining, .get() blocking, Ratis
> metrics/reply building, and parsing the Ratis response back into OMResponse.
> See [^om-benchmark-leader-read-all-ratis.html] for the flamegraph.
> This means that if we submit linearizable read to follower, we incurs these
> overhead on top of the linearizable read overhead (e.g. ReadIndex, etc).
> We can try to find a way to reduce this overhead. We might need to implement
> another read flow without the overhead (unlike writes which requires
> AppendEntries request to the followers which require serde, read can be
> served locally).
> One idea is that if we are submitting to RaftServer, we can use
> submitServerRequestAsync which does not require RaftClientRequest and
> RaftClientReply serde.
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
