[
https://issues.apache.org/jira/browse/HBASE-28987?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17924635#comment-17924635
]
Andor Molnar commented on HBASE-28987:
--------------------------------------
{quote}Another way is to close the S3 file on demand, maybe every 5 minutes?
Then we need to implement compaction, of course this could be implement outside
the regionserver process but anyway, it will increase the complexity.
{quote}
Compaction will happen when "incremental" endpoint is called from *hbase
backup* command: [~vinayakhegde] can described this in more detail, but the key
is that existing WAL files will be closed and compacted into HFiles and
replication will create and continue with new WAL files.
This is a manual operation, so user has to execute it on a regular basis
similar to normal backup process.
> Developing a Custom ReplicationEndpoint to Support External Storage
> Integration
> -------------------------------------------------------------------------------
>
> Key: HBASE-28987
> URL: https://issues.apache.org/jira/browse/HBASE-28987
> Project: HBase
> Issue Type: Task
> Components: backup&restore
> Affects Versions: 2.6.0, 3.0.0-alpha-4
> Reporter: Vinayak Hegde
> Assignee: Vinayak Hegde
> Priority: Major
>
> *Develop a Custom Replication Endpoint*
> Implement a custom replication endpoint to support the backup of WALs to
> external storage systems, such as HDFS-compliant storages (including HDFS,
> S3, ADLS, and GCS via respective Hadoop connectors).
> *Support for Bulk-loaded Files*
> Add functionality to back up bulk-loaded files in addition to regular WALs.
> *Ensure Process Durability*
> Ensure the backup process is durable, with no WALs being missed, even in the
> event of issues in the cluster.
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
