[
https://issues.apache.org/jira/browse/HBASE-12271?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14172711#comment-14172711
]
Patrick White commented on HBASE-12271:
---------------------------------------
ExportSnapshot will skip files that are the same (either via checksum or via
name+length comparison). We use this to cut down on transfer time.
First we snapshot the table, then export that snapshot to the remote DFS.
Next run, we instead start by copying the previously exported snapshot into the
current snapshot's destination (all on the remote DFS).
We then do the snapshot and export dance again, and it dutifully copies only
the changed files.
Our (fb) HDFS has hard links (last I heard upstream does not) so we can copy
backups around pretty cheaply. In the future we may just throw everything into
one directory and let the manifest be the decider of what files are involved,
but in the mean time it's copy and differential.
> Add counters for files skipped during snapshot export
> -----------------------------------------------------
>
> Key: HBASE-12271
> URL: https://issues.apache.org/jira/browse/HBASE-12271
> Project: HBase
> Issue Type: Improvement
> Components: snapshots
> Affects Versions: 0.98.7
> Reporter: Patrick White
> Assignee: Patrick White
> Priority: Minor
> Fix For: 2.0.0, 0.98.8, 0.99.2
>
> Attachments: 0001-Add-counters-for-skipped-files.patch
>
>
> It's incredibly handy to see the number of files skipped to know/verify delta
> backups are doing what they should, when they should.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
