[
https://issues.apache.org/jira/browse/KUDU-3180?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17177540#comment-17177540
]
ASF subversion and git services commented on KUDU-3180:
-------------------------------------------------------
Commit 42aff29360bf5be0a141902dac707e865933473f in kudu's branch
refs/heads/master from zhangyifan27
[ https://gitbox.apache.org/repos/asf?p=kudu.git;h=42aff29 ]
KUDU-3180: prioritize larger mem-stores in time-based flusing
Current time-based flush policy will always pick a mem-store
that haven't been flushed in a long time instead of a mem-store
anchoring more memory, this may lead to:
- more memory used by mem-stores.
- more small rowsets on disk so we need to do more compaction.
This patch improve current flush policy by considering both
mem-stores' size and time since last flush. When a mem-store
become large or old enough, it will be more likely to flush,
then we can avoid anchoring large (but below the threshold)
mem-stores or WALs for too long.
Change-Id: I0a826643709a4990e40b0a49f89f4ea34f14163b
Reviewed-on: http://gerrit.cloudera.org:8080/16319
Tested-by: Kudu Jenkins
Reviewed-by: Andrew Wong <aw...@cloudera.com>
> kudu don't always prefer to flush MRS/DMS that anchor more memory
> -----------------------------------------------------------------
>
> Key: KUDU-3180
> URL: https://issues.apache.org/jira/browse/KUDU-3180
> Project: Kudu
> Issue Type: Improvement
> Reporter: YifanZhang
> Priority: Major
> Attachments: image-2020-08-04-20-26-53-749.png,
> image-2020-08-04-20-28-00-665.png
>
>
> Current time-based flush policy always give a flush op a high score if we
> haven't flushed for the tablet in a long time, that may lead to starvation of
> ops that could free more memory.
> We set -flush_threshold_mb=32, -flush_threshold_secs=1800 in a cluster, and
> find that some small MRS/DMS flushes has a higher perf score than big MRS/DMS
> flushes and compactions, which seems not so reasonable.
> !image-2020-08-04-20-26-53-749.png|width=1424,height=317!!image-2020-08-04-20-28-00-665.png|width=1414,height=327!
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
