[
https://issues.apache.org/jira/browse/HBASE-13959?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14605305#comment-14605305
]
Hari Krishna Dara commented on HBASE-13959:
-------------------------------------------
Yes, I think I understand your proposal better now. Since the max storefile
count should be around the blocking store file count, it makes sense to use it
as the max. However, I came across blogs in which folks used unusually large
blocking store file count (e.g., 200) to optimize their write performance (at
the cost of read performance, of course) which could translate to having too
many threads and causing resource availability and performance issues, so
perhaps we should also enforce an upper limit to the number of logical cores
(as returned by Runtime.getRuntime().availableProcessors()) ?
As a follow up change, we could also create the two references on each
storefile in parallel and shave off another 300 to 400ms in most scenarios. I
can create a separate jira for this and work on it separately once this is
merged.
> Region splitting takes too long because it uses a single thread in most
> common cases
> ------------------------------------------------------------------------------------
>
> Key: HBASE-13959
> URL: https://issues.apache.org/jira/browse/HBASE-13959
> Project: HBase
> Issue Type: Bug
> Components: regionserver
> Affects Versions: 0.98.12
> Reporter: Hari Krishna Dara
> Assignee: Hari Krishna Dara
> Priority: Critical
> Fix For: 0.98.14
>
> Attachments: 13959-suggest.txt, HBASE-13959-2.patch,
> HBASE-13959-3.patch, HBASE-13959-4.patch, HBASE-13959.patch,
> region-split-durations-compared.png
>
>
> When storefiles need to be split as part of a region split, the current logic
> uses a threadpool with the size set to the size of the number of stores.
> Since most common table setup involves only a single column family, this
> translates to having a single store and so the threadpool is run with a
> single thread. However, in a write heavy workload, there could be several
> tens of storefiles in a store at the time of splitting, and with a threadpool
> size of one, these files end up getting split sequentially.
> With a bit of tracing, I noticed that it takes on an average of 350ms to
> create a single reference file, and splitting each storefile involves
> creating two of these, so with a storefile count of 20, it takes about 14s
> just to get through this phase alone (2 reference files for each storefile),
> pushing the total time the region is offline to 18s or more. For environments
> that are setup to fail fast, this makes the client exhaust all retries and
> fail with NotServingRegionException.
> The fix should increase the concurrency of this operation.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
