[ https://issues.apache.org/jira/browse/HBASE-21286?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16646994#comment-16646994 ]
stack commented on HBASE-21286: ------------------------------- Related to https://issues.apache.org/jira/browse/HBASE-20786?focusedCommentId=16603944&page=com.atlassian.jira.plugin.system.issuetabpanels%3Acomment-tabpanel#comment-16603944 ? Could we parallelize for all cases, not just for snapshots? Thanks. > Parallelize computeHDFSBlocksDistribution when getting splits of a > HBaseSnapshot > -------------------------------------------------------------------------------- > > Key: HBASE-21286 > URL: https://issues.apache.org/jira/browse/HBASE-21286 > Project: HBase > Issue Type: Improvement > Components: snapshots > Affects Versions: 1.4.0 > Reporter: Lavinia-Stefania Sirbu > Priority: Minor > Attachments: HBASE-21286.branch-1.4.001.patch, > HBASE-21286.branch-1.4.002.patch > > > Even if this step is called computeHDFSBlocksDistribution, this is executed > no matter the file system of the snapshot. For example, we have observed an > important slowness when we have a snapshot in s3 (~26k regions, 5column > families, 2 files per column family) the getsplits time is ~40min due to the > calls in s3 for listing the files to get the best locations. > Parallelizing this operation can reduce the overall setup time. The thread > pool should be configurable and a good choice could be > "hbase.snapshot.thread.pool.max" that is also used in RestoreSnapshotHelper. -- This message was sent by Atlassian JIRA (v7.6.3#76005)