Todd Lipcon has submitted this change and it was merged. Change subject: rpc: bump default max message size to 50MB ......................................................................
rpc: bump default max message size to 50MB It seems that the current default of only 8MB can be problematic in a few cases: * If tables contain larger cells, it's not hard to overflow the 8MB for scanner results. * When writing, if users aren't very careful about managing batch sizes, it's also possible to overflow. Of course, it's also possible to overflow 50MB, so in some cases this commit just "delays the inevitable". But, I did a survey of other RPC systems and found that 8MB is far lower than what those expect: - GRPC max frame size is 100MB by default, and allows the user to increase it if necessary[1] - Hadoop RPC defaults to 64MB and is configurable[2]. Responses larger than 1MB will be logged by default. Note that HDFS does not use RPC for data transfer, so >1MB RPCs are unlikely. - HBase RPC sets the maximum request to 256MB and warns for responses larger than 100MB.[3] This commit also changes the remote bootstrap code path to aim for 4MB chunks regardless of the max RPC message size, since it doesn't make sense to do IO in 50MB units. [1] https://github.com/grpc/grpc-java/issues/917 [2] https://github.com/apache/hadoop-common/blob/trunk/hadoop-common-project/hadoop-common/src/main/java/org/apache/hadoop/fs/CommonConfigurationKeys.java#L77 [3] https://github.com/apache/hbase/blob/master/hbase-server/src/main/java/org/apache/hadoop/hbase/ipc/RpcServer.java#L268 Change-Id: I1df399c5dd28ec9539fbd701d26ff2c7467991c8 Reviewed-on: http://gerrit.cloudera.org:8080/2889 Tested-by: Kudu Jenkins Reviewed-by: Adar Dembo <[email protected]> --- M src/kudu/rpc/transfer.cc M src/kudu/tserver/remote_bootstrap_client.cc M src/kudu/tserver/remote_bootstrap_session.cc 3 files changed, 34 insertions(+), 18 deletions(-) Approvals: Adar Dembo: Looks good to me, approved Kudu Jenkins: Verified -- To view, visit http://gerrit.cloudera.org:8080/2889 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-MessageType: merged Gerrit-Change-Id: I1df399c5dd28ec9539fbd701d26ff2c7467991c8 Gerrit-PatchSet: 3 Gerrit-Project: kudu Gerrit-Branch: master Gerrit-Owner: Todd Lipcon <[email protected]> Gerrit-Reviewer: Adar Dembo <[email protected]> Gerrit-Reviewer: Kudu Jenkins Gerrit-Reviewer: Mike Percy <[email protected]> Gerrit-Reviewer: Todd Lipcon <[email protected]>
