[
https://issues.apache.org/jira/browse/HBASE-28904?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Liangjun He updated HBASE-28904:
--------------------------------
Description:
The Bulkload process consists of two steps:
1. generate hfiles using MR/SPARK and write them to an HDFS cluster.
2. execute 'hbase completebulkload [OPTIONS]
</PATH/TO/HFILEOUTPUTFORMAT-OUTPUT> <TABLENAME> ' or invoke the
BulkLoadHFilesTool API.
[HBASE-1721|https://issues.apache.org/jira/browse/HBASE-15172] implements
tiered storage capabilities for bulkload, but it is only applicable to
scenarios where hfiles generated by MR/SPARK are directly written to the HDFS
cluster used by HBase (tiered storage is configured). However, in most bulkload
scenarios, hfiles generated by MR/SPARK are first written to an offline HDFS
cluster (non-HBase HDFS Cluster, and tiered storage is not configured).
Subsequently, the 'hbase completebulkload' command is used to copy these hfiles
from the offline HDFS cluster to the HDFS cluster used by HBase, and rename
them to the appropriate table/region/columnfamily directory. This scenario is
not supported by
[HBASE-1721|https://issues.apache.org/jira/browse/HBASE-15172], this issue will
support tiered storage for this more general bulkload scenario.
was:
The Bulkload process consists of two steps:
1. generate hfiles using MR/SPARK and write them to an HDFS cluster.
2. execute 'hbase completebulkload [OPTIONS]
</PATH/TO/HFILEOUTPUTFORMAT-OUTPUT> <TABLENAME> ' or invoke the
BulkLoadHFilesTool API.
[HBASE-1721|https://issues.apache.org/jira/browse/HBASE-15172] implements
tiered storage capabilities for bulkload, but it is only applicable to
scenarios where hfiles generated by MR/SPARK are directly written to the HDFS
cluster used by HBase (tiered storage is configured). However, in most bulkload
scenarios, hfiles generated by MR/SPARK are first written to an offline HDFS
cluster (non-HBase HDFS Cluster, and tiered storage is not configured).
Subsequently, the 'hbase completebulkload' command is used to copy these hfiles
from the offline HDFS cluster to the HDFS cluster used by HBase, and rename
them to the appropriate table/partition/columnfamily directory. This scenario
is not supported by
[HBASE-1721|https://issues.apache.org/jira/browse/HBASE-15172], this issue will
support tiered storage for this more general bulkload scenario.
> Supports enabling storage policy in the data copying scenario of bulkload
> -------------------------------------------------------------------------
>
> Key: HBASE-28904
> URL: https://issues.apache.org/jira/browse/HBASE-28904
> Project: HBase
> Issue Type: Improvement
> Reporter: Liangjun He
> Assignee: Liangjun He
> Priority: Major
> Labels: pull-request-available
>
> The Bulkload process consists of two steps:
> 1. generate hfiles using MR/SPARK and write them to an HDFS cluster.
> 2. execute 'hbase completebulkload [OPTIONS]
> </PATH/TO/HFILEOUTPUTFORMAT-OUTPUT> <TABLENAME> ' or invoke the
> BulkLoadHFilesTool API.
> [HBASE-1721|https://issues.apache.org/jira/browse/HBASE-15172] implements
> tiered storage capabilities for bulkload, but it is only applicable to
> scenarios where hfiles generated by MR/SPARK are directly written to the HDFS
> cluster used by HBase (tiered storage is configured). However, in most
> bulkload scenarios, hfiles generated by MR/SPARK are first written to an
> offline HDFS cluster (non-HBase HDFS Cluster, and tiered storage is not
> configured). Subsequently, the 'hbase completebulkload' command is used to
> copy these hfiles from the offline HDFS cluster to the HDFS cluster used by
> HBase, and rename them to the appropriate table/region/columnfamily
> directory. This scenario is not supported by
> [HBASE-1721|https://issues.apache.org/jira/browse/HBASE-15172], this issue
> will support tiered storage for this more general bulkload scenario.
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
