[
https://issues.apache.org/jira/browse/HBASE-13408?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14980339#comment-14980339
]
Anoop Sam John commented on HBASE-13408:
----------------------------------------
The default memstore impl (what we have today) will keep cells in memory and
flush when request comes (as per reach of max size or global heap pressure).
When we say a new memstore impl which will do in memory compaction and keep
more cell in memory for longer time I would expect this to be purely a new
memstore impl alone with out much affecting the other areas.. But on a pass
over the RB, I can see we are touching many other areas adding new APIs around
flush etc... IMHO this should be avoided.. Like it is not good that an
outside module tells Memstore to do in memory flush and in memory compaction
and do disk flush.. It is an internal detail of that memstore impl.
The new memstore impl has to decide till when to keep cells in a CSLM and when
to do an in memory compaction and keep cells in serialized format within
memory.
On general this is the concern.
> HBase In-Memory Memstore Compaction
> -----------------------------------
>
> Key: HBASE-13408
> URL: https://issues.apache.org/jira/browse/HBASE-13408
> Project: HBase
> Issue Type: New Feature
> Reporter: Eshcar Hillel
> Assignee: Eshcar Hillel
> Fix For: 2.0.0
>
> Attachments: HBASE-13408-trunk-v01.patch,
> HBASE-13408-trunk-v02.patch, HBASE-13408-trunk-v03.patch,
> HBASE-13408-trunk-v04.patch, HBASE-13408-trunk-v05.patch,
> HBASE-13408-trunk-v06.patch, HBASE-13408-trunk-v07.patch,
> HBASE-13408-trunk-v08.patch,
> HBaseIn-MemoryMemstoreCompactionDesignDocument-ver02.pdf,
> HBaseIn-MemoryMemstoreCompactionDesignDocument.pdf,
> InMemoryMemstoreCompactionEvaluationResults.pdf,
> InMemoryMemstoreCompactionMasterEvaluationResults.pdf,
> InMemoryMemstoreCompactionScansEvaluationResults.pdf,
> StoreSegmentandStoreSegmentScannerClassHierarchies.pdf
>
>
> A store unit holds a column family in a region, where the memstore is its
> in-memory component. The memstore absorbs all updates to the store; from time
> to time these updates are flushed to a file on disk, where they are
> compacted. Unlike disk components, the memstore is not compacted until it is
> written to the filesystem and optionally to block-cache. This may result in
> underutilization of the memory due to duplicate entries per row, for example,
> when hot data is continuously updated.
> Generally, the faster the data is accumulated in memory, more flushes are
> triggered, the data sinks to disk more frequently, slowing down retrieval of
> data, even if very recent.
> In high-churn workloads, compacting the memstore can help maintain the data
> in memory, and thereby speed up data retrieval.
> We suggest a new compacted memstore with the following principles:
> 1. The data is kept in memory for as long as possible
> 2. Memstore data is either compacted or in process of being compacted
> 3. Allow a panic mode, which may interrupt an in-progress compaction and
> force a flush of part of the memstore.
> We suggest applying this optimization only to in-memory column families.
> A design document is attached.
> This feature was previously discussed in HBASE-5311.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
