Hello Kudu Jenkins,
I'd like you to reexamine a change. Please visit
http://gerrit.cloudera.org:8080/24731
to look at the new patch set (#2).
Change subject: WIP KUDU-2874 Test ordered scan memory usage with large cells.
......................................................................
WIP KUDU-2874 Test ordered scan memory usage with large cells.
Currently sorted scans has a burned in 1024 size buffer.
With 1 MB of cells this can cause large, untracked memory usage.
This test does the following:
+ Start a tserver with one tablet.
+ Disable compaction for worst case result (compaction falls behind).
+ Write some rows with 1 MB cells into a tablet.
+ Disable flushing then keep writing until memory limit is hit.
+ Start 10 parallel ordered full scans. They are started one-by-one with a
gap between them, so they probably request other parts of the data at
the same time.
+ Compare tmalloc reported memory usage with the trackers.
+ Check the peak memory usage.
One run gave me:
Peak tmalloc memory during test : 25934 MiB
Peak root mem tracker memory during test : 8156 MiB
tmalloc also counts the tests memory usage, but it
should not be more than 1 GiB.
So ~16-17 GiB of untracked memory is unacceptable.
2 things should be addressed to fix the test:
+ Dynamically downsize batches when cells are really large.
+ Properly count memory at scans when copying values from memrowset.
(It is counted properly when loaded from disk rowsets).
Change-Id: I75ceaebac1793c2421302f8b205ba85ad32a141a
---
M src/kudu/tserver/CMakeLists.txt
A src/kudu/tserver/ordered_scan_large_cells-test.cc
2 files changed, 481 insertions(+), 0 deletions(-)
git pull ssh://gerrit.cloudera.org:29418/kudu refs/changes/31/24731/2
--
To view, visit http://gerrit.cloudera.org:8080/24731
To unsubscribe, visit http://gerrit.cloudera.org:8080/settings
Gerrit-Project: kudu
Gerrit-Branch: master
Gerrit-MessageType: newpatchset
Gerrit-Change-Id: I75ceaebac1793c2421302f8b205ba85ad32a141a
Gerrit-Change-Number: 24731
Gerrit-PatchSet: 2
Gerrit-Owner: Zoltan Martonka <[email protected]>
Gerrit-Reviewer: Kudu Jenkins (120)
