[
https://issues.apache.org/jira/browse/HBASE-18294?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16304800#comment-16304800
]
Edward Bortnikov commented on HBASE-18294:
------------------------------------------
My 2 cents - after reviewing the code to the best of my ability.
IMO the confusion stems from the unfortunate name - heapSize - the code
historically uses for the overall allocated memory. Such that it is not clear
whether we mean the Java heap or the OS heap. Might be good to replace globally
- maybe in a different jira.
Regarding the per-store flush trigger ... Still not sure what is the reasoning
behind non-uniform handling of on-(Java) heap and off-(Java) heap allocations.
Could someone please re-iterate why just monitor the overall allocated memory
(data + overhead), no matter where, and flush when the threshold is crossed?
Obviously there are all kinds of concerns, but the only experiment on the table
is the one by [~eshcar], which demonstrates that regions flush too early, at
least with on-heap data, due to conservative accounting.
We intend to benchmark the off-heap write path thoroughly, in particular to
evaluate the benefits of the CCM index (HBase-16421). If something unexpected
comes up there, we'll all re-convene and re-discuss. Until then, may I suggest
to keep the things simple and run the accounting context-free, along the lines
with [~eshcar]'s patch.
> Reduce global heap pressure: flush based on heap occupancy
> ----------------------------------------------------------
>
> Key: HBASE-18294
> URL: https://issues.apache.org/jira/browse/HBASE-18294
> Project: HBase
> Issue Type: Improvement
> Affects Versions: 3.0.0
> Reporter: Eshcar Hillel
> Assignee: Eshcar Hillel
> Fix For: 2.0.0-beta-2
>
> Attachments: HBASE-18294.01.patch, HBASE-18294.02.patch,
> HBASE-18294.03.patch, HBASE-18294.04.patch, HBASE-18294.05.patch,
> HBASE-18294.06.patch, HBASE-18294.07.patch, HBASE-18294.07.patch,
> HBASE-18294.08.patch, HBASE-18294.09.patch, HBASE-18294.10.patch,
> HBASE-18294.11.patch, HBASE-18294.11.patch, HBASE-18294.12.patch,
> HBASE-18294.13.patch
>
>
> A region is flushed if its memory component exceed a threshold (default size
> is 128MB).
> A flush policy decides whether to flush a store by comparing the size of the
> store to another threshold (that can be configured with
> hbase.hregion.percolumnfamilyflush.size.lower.bound).
> Currently the implementation (in both cases) compares the data size
> (key-value only) to the threshold where it should compare the heap size
> (which includes index size, and metadata).
--
This message was sent by Atlassian JIRA
(v6.4.14#64029)
