[
https://issues.apache.org/jira/browse/HBASE-24541?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17152778#comment-17152778
]
Constantin-Catalin Luca commented on HBASE-24541:
-------------------------------------------------
Ok, I will submit a PR for master as well.
> Add support to run LoadIncrementalHFiles in a distributed manner
> ----------------------------------------------------------------
>
> Key: HBASE-24541
> URL: https://issues.apache.org/jira/browse/HBASE-24541
> Project: HBase
> Issue Type: Improvement
> Components: mapreduce, Performance
> Affects Versions: 1.4.0
> Reporter: Constantin-Catalin Luca
> Assignee: Constantin-Catalin Luca
> Priority: Minor
> Attachments: HBASE_24541-1.4.0.patch
>
>
> LoadIncrementalHFiles takes a very long time to complete when running HBase
> on top of S3 and attempting to bulkload 500K-700K files.
> The root cause of this is a combination of the higher latency of S3 (as
> compared to HDFS) as well as the calls made by LoadIncrementalHFiles to the
> underlying filesystem(each file is opened, seeked to the trailer offset at
> the end, and then the trailer is read).
> Increasing the parallelism does not yield any significant improvement. This
> seems to stem from the fact that once the trailer is read the stream is not
> consumed to the end. This causes the underlying HTTP connection to be aborted
> and it cannot be re-used.
>
> The proposed solution would be to also add support to run
> LoadIncrementalHFiles on multiple machines as a map reduce job.
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
