[
https://issues.apache.org/jira/browse/KUDU-1693?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Todd Lipcon updated KUDU-1693:
------------------------------
Component/s: perf
> Flush write operations on per-TS basis and add corresponding limit on the
> buffer space
> --------------------------------------------------------------------------------------
>
> Key: KUDU-1693
> URL: https://issues.apache.org/jira/browse/KUDU-1693
> Project: Kudu
> Issue Type: Improvement
> Components: client, perf
> Affects Versions: 1.0.0
> Reporter: Alexey Serbin
> Priority: Major
>
> Currently, the Kudu C++ client buffers incoming operations regardless of
> their destination tablet server. Accordingly, it's possible to set limit on
> the _total_ buffer space, not per tablet server. This approach works but
> there is room for improvement: there are real-world scenarios where per-TS
> buffering would be more robust. Besides, tablet servers impose limit on the
> RPC operations size.
> Grouping write operations on per-tablet-server basis would be beneficial for
> 'one-out-of-many lagging tablet server' scenario. There, all tablet servers
> for a table perform well except for one which runs slow due to excessive IO,
> network issues, failing disk, etc. The problem is that the lagging server
> hinders the overall performance. This is due to the current approach to the
> buffer turnaround: a buffer is considered 'flushed' and its space is
> reclaimed at once when _all_ operations in the buffer are completed. So, if
> 1000 operations have already been sent but there is 1 operation still in
> progress, the whole buffer space is 'locked' and cannot be used.
> Accordingly, introducing per-tablet-server buffer limit would help to address
> scenarios with concurrent writes into tables with extremely diverse partition
> factors (like 2 and 100). E.g., consider a case when incoming write
> operations for tables with diverse partition factors are intermixed in the
> context of one session. The problem is that setting the total buffer space
> limit high is beneficial for the writes into the table with many partitions
> (assuming those writes are evenly distributed across participating tablets),
> but it may be over the server-side's limit for max transaction size if those
> writes are targeted for the table with a few partitions.
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
