[
https://issues.apache.org/jira/browse/HBASE-25302?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17341821#comment-17341821
]
Xiaolin Ha commented on HBASE-25302:
------------------------------------
Hi, [~zhangduo], [~stack] ,the design doc has been updated, please take a look.
Any suggestions are welcome.
> Fast split regions and lightweight compaction with stripe store engine
> ----------------------------------------------------------------------
>
> Key: HBASE-25302
> URL: https://issues.apache.org/jira/browse/HBASE-25302
> Project: HBase
> Issue Type: Improvement
> Reporter: Xiaolin Ha
> Assignee: Xiaolin Ha
> Priority: Major
> Attachments: after-split.png, before-split.png
>
>
> In our company MeiTuan, we have encountered a problem, that is a production
> cluster with about more than 10000 regions set off the split switch by some
> mistakes. And more than 10000 regions are larger than 600GB before we can do
> something to split them. Because online real time R+W QPS of each RS is
> higher than 80000 (8000+ per region), if we just switch on split, the IO
> pressure brought by compactions will influence the availabilities, and all
> large regions need repeatedly compactions until the size of child regions are
> normal.
> Different with the normal circumstances, we used stripe store engine in this
> cluster, after looking through docs of
> https://issues.apache.org/jira/browse/HBASE-7667 , we have implemented a
> fast split region method using the similar ideas of HFileLink and move store
> files between regions when compacting. Actually, this idea has already been
> mentioned in HBASE-7667 , it said that `region splits become marvelously
> simple (if we could move files between regions, no references would be
> needed)`.
> We have implemented the ideas and practiced them on that cluster. The fact
> results show that, a region of about 600G can be splitted to 75G*8 regions
> in about five minutes, with less than 5G total rewrite size(all are L0) in
> the whole process, while normal continuous split needs 600G*3=1800G. By using
> movement of store files, the rewritten size is less than 50G(two stripe size,
> because the rebuild of HFileLinks to stripes may insert some files to L0.).I
> have attached two pictures about a RS before and after splitting all the
> regions using this method.
> This idea can not only be used in stripe store engine, but also default store
> engine, and the store movement methods are very benefit to region MERGE.
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
