[
https://issues.apache.org/jira/browse/HBASE-5547?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13251727#comment-13251727
]
[email protected] commented on HBASE-5547:
------------------------------------------------------
-----------------------------------------------------------
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/4633/#review6820
-----------------------------------------------------------
src/main/java/org/apache/hadoop/hbase/client/HBaseAdmin.java
<https://reviews.apache.org/r/4633/#comment15202>
done. Originally modeled after the catalog tracker in naming.
src/main/java/org/apache/hadoop/hbase/client/HBaseAdmin.java
<https://reviews.apache.org/r/4633/#comment15203>
done.
src/main/java/org/apache/hadoop/hbase/client/HBaseAdmin.java
<https://reviews.apache.org/r/4633/#comment15204>
done.
src/main/java/org/apache/hadoop/hbase/client/HBaseAdmin.java
<https://reviews.apache.org/r/4633/#comment15205>
done
src/main/java/org/apache/hadoop/hbase/client/HBaseAdmin.java
<https://reviews.apache.org/r/4633/#comment15206>
done.
src/main/java/org/apache/hadoop/hbase/client/HBaseAdmin.java
<https://reviews.apache.org/r/4633/#comment15207>
Would rather not, since in that case we would keep around a persistent
connection to ZK, which is against what the community has been doing with
cutting the client/zk connection (though admittedly that not directly
applicable here).
src/main/java/org/apache/hadoop/hbase/client/HBaseAdmin.java
<https://reviews.apache.org/r/4633/#comment15208>
changed to confirmedArchiving.
src/main/java/org/apache/hadoop/hbase/client/HBaseAdmin.java
<https://reviews.apache.org/r/4633/#comment15184>
Yeah, it would. But then you should not attempt to expect the archive as
valid and fail your backup process. Was going to leave it to the user to
determine if they wanted to do the cleanup.
However, I would be ok if we added a method that cleaned up the filesystem
for files created in the backup after the archiving started.
Unfortunately, this means add an extra method on the master which I was
trying to avoid, but I guess its not the end of the world. We also have a
two-phase commit issue here as we also need to then write a
'consistent-confirmed' file into the archive when it completes to ensure that
failed clients don't break the archiving.
thoughts?
src/main/java/org/apache/hadoop/hbase/client/HBaseAdmin.java
<https://reviews.apache.org/r/4633/#comment15209>
done.
src/main/java/org/apache/hadoop/hbase/client/HBaseAdmin.java
<https://reviews.apache.org/r/4633/#comment15183>
Method name isn't enough?
src/main/java/org/apache/hadoop/hbase/client/HFileArchiveManager.java
<https://reviews.apache.org/r/4633/#comment15210>
done.
src/main/java/org/apache/hadoop/hbase/client/HFileArchiveManager.java
<https://reviews.apache.org/r/4633/#comment15211>
done.
src/main/java/org/apache/hadoop/hbase/client/HFileArchiveManager.java
<https://reviews.apache.org/r/4633/#comment15212>
done - regionsBeingArchived it is.
src/main/java/org/apache/hadoop/hbase/regionserver/HFileArchiveMonitor.java
<https://reviews.apache.org/r/4633/#comment15213>
done.
src/main/java/org/apache/hadoop/hbase/regionserver/HFileArchiveMonitor.java
<https://reviews.apache.org/r/4633/#comment15185>
Cruft from the earlier version.
src/main/java/org/apache/hadoop/hbase/regionserver/HRegion.java
<https://reviews.apache.org/r/4633/#comment15186>
yeah, guess it should be.
src/main/java/org/apache/hadoop/hbase/util/HFileArchiveUtil.java
<https://reviews.apache.org/r/4633/#comment15214>
done.
src/main/java/org/apache/hadoop/hbase/util/HFileArchiveUtil.java
<https://reviews.apache.org/r/4633/#comment15187>
This is a super rare case where the name of the regiondir to be backed up
is the same as the one in the backup already. Just being extra careful that the
random storefile name doesn't match, though it is _very_ unlikely to happen, it
still could (which could lead to bad circumstances). Currently, the storefile
check doesn't check the archive directory and I think it would be pretty big
overhead to expect it to do that as well (lots of added dependencies, calls,
etc). We can just do the check and renaming here - its not super complicated
and seems to work.
src/main/java/org/apache/hadoop/hbase/zookeeper/HFileArchiveTracker.java
<https://reviews.apache.org/r/4633/#comment15215>
done.
src/main/java/org/apache/hadoop/hbase/zookeeper/RegionServerHFileTracker.java
<https://reviews.apache.org/r/4633/#comment15216>
done.
- Jesse
On 2012-04-07 19:51:11, Jesse Yates wrote:
bq.
bq. -----------------------------------------------------------
bq. This is an automatically generated e-mail. To reply, visit:
bq. https://reviews.apache.org/r/4633/
bq. -----------------------------------------------------------
bq.
bq. (Updated 2012-04-07 19:51:11)
bq.
bq.
bq. Review request for hbase, Michael Stack and Lars Hofhansl.
bq.
bq.
bq. Summary
bq. -------
bq.
bq. Essentially, whenever an hfile would be deleted, it is instead moved to
the archive directory. In this impl, the archive directory is on a per table
basis, but defaults to '.archive'. Removing hfiles occurs in three places -
compaction, merge and catalog janitor. The former and two latter are distinctly
different code paths, but but did pull out some similarities. The latter two
end up calling the same method, so there should be a reasonable amount of
overlap.
bq.
bq. Implementation wise:
bq. Updated the HMasterInterface to pass the calls onto the zookeeper.
bq. Added a zk listener to handle updates from the master to the RS to
backup.
bq. Added a utility for removing files and finding archive directories
bq. Added tests for the regionserver and catalogjanitor approaches.
bq. Added creation of manager in regionserver.
bq.
bq.
bq. This addresses bug HBASE-5547.
bq. https://issues.apache.org/jira/browse/HBASE-5547
bq.
bq.
bq. Diffs
bq. -----
bq.
bq. src/main/java/org/apache/hadoop/hbase/HConstants.java a4b989e
bq. src/main/java/org/apache/hadoop/hbase/client/HBaseAdmin.java 16e4017
bq. src/main/java/org/apache/hadoop/hbase/client/HFileArchiveManager.java
PRE-CREATION
bq. src/main/java/org/apache/hadoop/hbase/master/CatalogJanitor.java 79d5fdd
bq. src/main/java/org/apache/hadoop/hbase/master/HMaster.java fb21bdd
bq.
src/main/java/org/apache/hadoop/hbase/regionserver/HFileArchiveMonitor.java
PRE-CREATION
bq. src/main/java/org/apache/hadoop/hbase/regionserver/HRegion.java c3df319
bq. src/main/java/org/apache/hadoop/hbase/regionserver/HRegionServer.java
8a61f7d
bq.
src/main/java/org/apache/hadoop/hbase/regionserver/RegionServerServices.java
6884d53
bq. src/main/java/org/apache/hadoop/hbase/regionserver/Store.java 509a467
bq. src/main/java/org/apache/hadoop/hbase/util/HFileArchiveUtil.java
PRE-CREATION
bq. src/main/java/org/apache/hadoop/hbase/zookeeper/HFileArchiveTracker.java
PRE-CREATION
bq.
src/main/java/org/apache/hadoop/hbase/zookeeper/RegionServerHFileTracker.java
PRE-CREATION
bq. src/main/java/org/apache/hadoop/hbase/zookeeper/ZooKeeperWatcher.java
4fc105f
bq. src/main/resources/hbase-default.xml 44ee689
bq. src/test/java/org/apache/hadoop/hbase/master/MockRegionServer.java
41616c8
bq. src/test/java/org/apache/hadoop/hbase/master/TestCatalogJanitor.java
b4dcb83
bq.
src/test/java/org/apache/hadoop/hbase/regionserver/TestRegionHFileArchiving.java
PRE-CREATION
bq. src/test/java/org/apache/hadoop/hbase/util/HFileArchiveTestingUtil.java
PRE-CREATION
bq. src/test/java/org/apache/hadoop/hbase/util/MockRegionServerServices.java
7d02759
bq.
bq. Diff: https://reviews.apache.org/r/4633/diff
bq.
bq.
bq. Testing
bq. -------
bq.
bq. Added two tests for the separate cases - archiving via the regionserver
and for the catalog tracker. Former runs in a mini cluster and also touches the
changes to HMasterInterface and zookeeper.
bq.
bq.
bq. Thanks,
bq.
bq. Jesse
bq.
bq.
> Don't delete HFiles when in "backup mode"
> -----------------------------------------
>
> Key: HBASE-5547
> URL: https://issues.apache.org/jira/browse/HBASE-5547
> Project: HBase
> Issue Type: New Feature
> Reporter: Lars Hofhansl
> Assignee: Jesse Yates
>
> This came up in a discussion I had with Stack.
> It would be nice if HBase could be notified that a backup is in progress (via
> a znode for example) and in that case either:
> 1. rename HFiles to be delete to <file>.bck
> 2. rename the HFiles into a special directory
> 3. rename them to a general trash directory (which would not need to be tied
> to backup mode).
> That way it should be able to get a consistent backup based on HFiles (HDFS
> snapshots or hard links would be better options here, but we do not have
> those).
> #1 makes cleanup a bit harder.
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators:
https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira
