Adar Dembo has posted comments on this change. ( http://gerrit.cloudera.org:8080/8090 )
Change subject: KUDU-1125: issue one catalog write per tablet report ...................................................................... Patch Set 8: (1 comment) http://gerrit.cloudera.org:8080/#/c/8090/6//COMMIT_MSG Commit Message: http://gerrit.cloudera.org:8080/#/c/8090/6//COMMIT_MSG@10 PS6, Line 10: When the : master is configured to fsync WAL writes, this can add a lot of load during : election storms and when the master is restarted. > That's fair. If it's going to take a lot of effort I don't think it's worth On our internal flash cluster I ran the following experiment: 1. Reconfigure the master to use log_force_fsync_all=true. 2. Stop the master. 3. Restart a tserver. 4. Start the master. 5. Look at the TSHeartbeat samples in the master's /rpcz page. In this experiment not only does the master receive a full heartbeat from every tserver, but it also has to update a bunch of persistent tablet state for the tablets whose leaders were on the restarted tserver but are now elsewhere. - Without this change, I saw a "fat" heartbeat sample which updated 140 tablets and took 10304 ms. - With this change, an equivalent sample updated 308 tablets and took 157 ms. As for measuring when the number of tablets may run up against the maximum RPC size, I modified the master to LOG during a tablet report the number of tablets and the size of the overall report. I restarted the master and calculated an average across the full tablet reports sent by all of the tservers in the cluster. It worked out to about ~283 bytes per tablet. With our 50MB maximum RPC size, we'll be in danger of blowing that out at around 185,000 tablets. Note that this applies to single-master deployments too as the maximum RPC size applies to the TSHeartbeat itself. -- To view, visit http://gerrit.cloudera.org:8080/8090 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: kudu Gerrit-Branch: master Gerrit-MessageType: comment Gerrit-Change-Id: Ie6f5cf0e4b1cd1160b3b310d89c6dbf3dd62e43b Gerrit-Change-Number: 8090 Gerrit-PatchSet: 8 Gerrit-Owner: Adar Dembo <a...@cloudera.com> Gerrit-Reviewer: Adar Dembo <a...@cloudera.com> Gerrit-Reviewer: Dan Burkert <danburk...@apache.org> Gerrit-Reviewer: Mike Percy <mpe...@apache.org> Gerrit-Reviewer: Todd Lipcon <t...@apache.org> Gerrit-Comment-Date: Sat, 07 Oct 2017 01:23:27 +0000 Gerrit-HasComments: Yes
