Hello David Ribeiro Alves, Kudu Jenkins, I'd like you to reexamine a change. Please visit
http://gerrit.cloudera.org:8080/6802 to look at the new patch set (#2). Change subject: KUDU-1949. Maintenance Manager should trigger flushes earlier ...................................................................... KUDU-1949. Maintenance Manager should trigger flushes earlier This changes the maintenance manager to start triggering flushes at the 60% heap usage threshold, and adjusts the memory limiting to only start rejecting writes at the 80% threshold. This is based on analysis in [1] ("After tweaking memory backpressure rejection") in which we found that earlier flushing could keep the system from making too many rejections. I ran tpch_real_world with SF=300 as described in the doc below before and after the patch. Some key metrics: Before After Mem rejections 1,162,527 62,790 (18.5x fewer) Wall time 3156s 2490s (1.26x faster) Server CPU 37,348s 72,873s (1.95x less efficient) This matches the results from the original experiment. Because the writes come in faster, we don't get enough time to run compactions, and so each write ends up taking more CPU. Some future optimizations are still pending which will reduce the CPU time per bloom lookup. Regardless, this new behavior is configurable such that, if overall CPU efficiency is more important than wall time, we could bump the "memory pressure threshold" up higher than the "memory rejection threshold". [1] https://docs.google.com/document/d/1U1IXS1XD2erZyq8_qG81A1gZaCeHcq2i0unea_eEf5c/edit#heading=h.gw87heq4fq6l Change-Id: I8b7e703c82ac14fbce3a699bdf6a2f0fb4ed93a1 --- M src/kudu/util/maintenance_manager.cc M src/kudu/util/process_memory.cc M src/kudu/util/process_memory.h 3 files changed, 31 insertions(+), 2 deletions(-) git pull ssh://gerrit.cloudera.org:29418/kudu refs/changes/02/6802/2 -- To view, visit http://gerrit.cloudera.org:8080/6802 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-MessageType: newpatchset Gerrit-Change-Id: I8b7e703c82ac14fbce3a699bdf6a2f0fb4ed93a1 Gerrit-PatchSet: 2 Gerrit-Project: kudu Gerrit-Branch: master Gerrit-Owner: Todd Lipcon <t...@apache.org> Gerrit-Reviewer: Adar Dembo <a...@cloudera.com> Gerrit-Reviewer: David Ribeiro Alves <davidral...@gmail.com> Gerrit-Reviewer: Kudu Jenkins Gerrit-Reviewer: Tidy Bot Gerrit-Reviewer: Todd Lipcon <t...@apache.org>