[ https://issues.apache.org/jira/browse/KUDU-1693?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
Todd Lipcon updated KUDU-1693: ------------------------------ Component/s: perf > Flush write operations on per-TS basis and add corresponding limit on the > buffer space > -------------------------------------------------------------------------------------- > > Key: KUDU-1693 > URL: https://issues.apache.org/jira/browse/KUDU-1693 > Project: Kudu > Issue Type: Improvement > Components: client, perf > Affects Versions: 1.0.0 > Reporter: Alexey Serbin > Priority: Major > > Currently, the Kudu C++ client buffers incoming operations regardless of > their destination tablet server. Accordingly, it's possible to set limit on > the _total_ buffer space, not per tablet server. This approach works but > there is room for improvement: there are real-world scenarios where per-TS > buffering would be more robust. Besides, tablet servers impose limit on the > RPC operations size. > Grouping write operations on per-tablet-server basis would be beneficial for > 'one-out-of-many lagging tablet server' scenario. There, all tablet servers > for a table perform well except for one which runs slow due to excessive IO, > network issues, failing disk, etc. The problem is that the lagging server > hinders the overall performance. This is due to the current approach to the > buffer turnaround: a buffer is considered 'flushed' and its space is > reclaimed at once when _all_ operations in the buffer are completed. So, if > 1000 operations have already been sent but there is 1 operation still in > progress, the whole buffer space is 'locked' and cannot be used. > Accordingly, introducing per-tablet-server buffer limit would help to address > scenarios with concurrent writes into tables with extremely diverse partition > factors (like 2 and 100). E.g., consider a case when incoming write > operations for tables with diverse partition factors are intermixed in the > context of one session. The problem is that setting the total buffer space > limit high is beneficial for the writes into the table with many partitions > (assuming those writes are evenly distributed across participating tablets), > but it may be over the server-side's limit for max transaction size if those > writes are targeted for the table with a few partitions. -- This message was sent by Atlassian JIRA (v7.6.3#76005)