Hello Dan Burkert, Jean-Daniel Cryans, I'd like you to do a code review. Please visit
http://gerrit.cloudera.org:8080/5780 to review the following change. Change subject: WIP: KUDU-1848. in-memory dictionary for binary columns ...................................................................... WIP: KUDU-1848. in-memory dictionary for binary columns Quick prototype to test out this idea. Seems to save about 10% MRS usage for kudu-ts data (which has one compressible string column). Worth trying with some other datasets/workloads, maybe adding some smarts to auto-disable on columns determined to be non-compressible. without patch ------------------ memory consumption: 1.59GB backoffs took a total of 0.000000sec of runtime loaded 10000000 items in 59.522394sec with 6 workers (mean rate 168003.996280/sec) with patch (64 dict) after loading 10M points: ------------------ memory consumption 1.44GB backoffs took a total of 0.000000sec of runtime loaded 10000000 items in 57.729048sec with 6 workers (mean rate 173223.020620/sec) with patch (512 dict) after loading 10M points: ------------------ memory consumption 1.43GB backoffs took a total of 0.000000sec of runtime loaded 10000000 items in 62.279514sec with 6 workers (mean rate 160566.443468/sec) Change-Id: Ic03ef0473383b1dcf22ebefee029ff29d2dae813 --- M src/kudu/tablet/memrowset.cc M src/kudu/tablet/memrowset.h 2 files changed, 73 insertions(+), 10 deletions(-) git pull ssh://gerrit.cloudera.org:29418/kudu refs/changes/80/5780/1 -- To view, visit http://gerrit.cloudera.org:8080/5780 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-MessageType: newchange Gerrit-Change-Id: Ic03ef0473383b1dcf22ebefee029ff29d2dae813 Gerrit-PatchSet: 1 Gerrit-Project: kudu Gerrit-Branch: master Gerrit-Owner: Todd Lipcon <t...@apache.org> Gerrit-Reviewer: Dan Burkert <danburk...@apache.org> Gerrit-Reviewer: Jean-Daniel Cryans <jdcry...@apache.org>