[jira] [Updated] (KUDU-3036) RPC size multiplication for DDL operations might hit maximum RPC size limit
[ https://issues.apache.org/jira/browse/KUDU-3036?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Alexey Serbin updated KUDU-3036: Code Review: https://gerrit.cloudera.org/#/c/14999/ > RPC size multiplication for DDL operations might hit maximum RPC size limit > --- > > Key: KUDU-3036 > URL: https://issues.apache.org/jira/browse/KUDU-3036 > Project: Kudu > Issue Type: Bug > Components: master, rpc >Reporter: Alexey Serbin >Priority: Major > Labels: operability, scalability > > When a table uses multi-tier partitioning scheme, with large number of > partitions created, an {{AlterTable}} request that affects many > partitions/tablets turns into a much larger {{UpdateConsensus}} RPC when > leader master pushes the corresponding update on the system tablet to > follower masters. > I did some testing for this use case. With {{AlterTable}} RPC adding new > range partitions, I observed the following: > * With range x 2 hash partitions, with the incoming {{AlterTable}} RPC > request size is 37070 bytes, the size for the corresponding > {{UpdateConsensus}} is 274278 bytes (~ 7x multiplication factor). > * With range x 10 hash partitions, with the incoming {{AlterTable}} RPC > request size is 37070 bytes, the size for the corresponding > {{UpdateConsensus}} when leader master pushes the updates on the system > tablet to followers is 1365438 bytes (~ 36x multiplication factor). > With that, it's easy to hit the limit on the maximum PRC size (controlled via > the {{\-\-rpc_max_message_size}} flag) in case of larger Kudu clusters. If > that happens, Kudu masters start continuous leader re-election cycle since > follower masters don't receive any Raft heartbeats from their leader: the > heartbeats are rejected at the lower RPC layer due to the maximum RPC size > limit. -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Assigned] (KUDU-3036) RPC size multiplication for DDL operations might hit maximum RPC size limit
[ https://issues.apache.org/jira/browse/KUDU-3036?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Alexey Serbin reassigned KUDU-3036: --- Assignee: Alexey Serbin > RPC size multiplication for DDL operations might hit maximum RPC size limit > --- > > Key: KUDU-3036 > URL: https://issues.apache.org/jira/browse/KUDU-3036 > Project: Kudu > Issue Type: Bug > Components: master, rpc >Reporter: Alexey Serbin >Assignee: Alexey Serbin >Priority: Major > Labels: operability, scalability > > When a table uses multi-tier partitioning scheme, with large number of > partitions created, an {{AlterTable}} request that affects many > partitions/tablets turns into a much larger {{UpdateConsensus}} RPC when > leader master pushes the corresponding update on the system tablet to > follower masters. > I did some testing for this use case. With {{AlterTable}} RPC adding new > range partitions, I observed the following: > * With range x 2 hash partitions, with the incoming {{AlterTable}} RPC > request size is 37070 bytes, the size for the corresponding > {{UpdateConsensus}} is 274278 bytes (~ 7x multiplication factor). > * With range x 10 hash partitions, with the incoming {{AlterTable}} RPC > request size is 37070 bytes, the size for the corresponding > {{UpdateConsensus}} when leader master pushes the updates on the system > tablet to followers is 1365438 bytes (~ 36x multiplication factor). > With that, it's easy to hit the limit on the maximum PRC size (controlled via > the {{\-\-rpc_max_message_size}} flag) in case of larger Kudu clusters. If > that happens, Kudu masters start continuous leader re-election cycle since > follower masters don't receive any Raft heartbeats from their leader: the > heartbeats are rejected at the lower RPC layer due to the maximum RPC size > limit. -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Assigned] (KUDU-2975) Spread WAL across multiple data directories
[ https://issues.apache.org/jira/browse/KUDU-2975?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Andrew Wong reassigned KUDU-2975: - Assignee: YangSong > Spread WAL across multiple data directories > --- > > Key: KUDU-2975 > URL: https://issues.apache.org/jira/browse/KUDU-2975 > Project: Kudu > Issue Type: New Feature > Components: fs, tablet, tserver >Reporter: LiFu He >Assignee: YangSong >Priority: Major > Attachments: network.png, tserver-WARNING.png, util.png > > > Recently, we deployed a new kudu cluster and every node has 12 SSD. Then, we > created a big table and loaded data to it through flink. We noticed that the > util of one SSD which is used to store WAL is 100% but others are free. So, > we suggest to spread WAL across multiple data directories. -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Updated] (KUDU-2975) Spread WAL across multiple data directories
[ https://issues.apache.org/jira/browse/KUDU-2975?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Andrew Wong updated KUDU-2975: -- Code Review: https://gerrit.cloudera.org/c/14920/ > Spread WAL across multiple data directories > --- > > Key: KUDU-2975 > URL: https://issues.apache.org/jira/browse/KUDU-2975 > Project: Kudu > Issue Type: New Feature > Components: fs, tablet, tserver >Reporter: LiFu He >Assignee: YangSong >Priority: Major > Attachments: network.png, tserver-WARNING.png, util.png > > > Recently, we deployed a new kudu cluster and every node has 12 SSD. Then, we > created a big table and loaded data to it through flink. We noticed that the > util of one SSD which is used to store WAL is 100% but others are free. So, > we suggest to spread WAL across multiple data directories. -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Created] (KUDU-3038) ClientFailoverTServerTimeoutITest.FailoverOnLeaderTimeout is flaky in TSAN mode
Adar Dembo created KUDU-3038: Summary: ClientFailoverTServerTimeoutITest.FailoverOnLeaderTimeout is flaky in TSAN mode Key: KUDU-3038 URL: https://issues.apache.org/jira/browse/KUDU-3038 Project: Kudu Issue Type: Bug Components: test Affects Versions: 1.12.0 Reporter: Adar Dembo Attachments: client_failover-itest.txt Sometimes the test isn't able to successfully create a table in 30 seconds and crashes. I'm attaching the full log. -- This message was sent by Atlassian Jira (v8.3.4#803005)