[jira] [Commented] (HBASE-7294) Check for snapshot file cleaners on start
[
https://issues.apache.org/jira/browse/HBASE-7294?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13540849#comment-13540849
]
Hadoop QA commented on HBASE-7294:
--
{color:red}-1 overall{color}. Here are the results of testing the latest
attachment
http://issues.apache.org/jira/secure/attachment/12562664/HBASE-7294-v7.patch
against trunk revision .
{color:green}+1 @author{color}. The patch does not contain any @author
tags.
{color:green}+1 tests included{color}. The patch appears to include 15 new
or modified tests.
{color:red}-1 patch{color}. The patch command could not apply the patch.
Console output:
https://builds.apache.org/job/PreCommit-HBASE-Build/3754//console
This message is automatically generated.
Check for snapshot file cleaners on start
-
Key: HBASE-7294
URL: https://issues.apache.org/jira/browse/HBASE-7294
Project: HBase
Issue Type: Sub-task
Components: Client, master, regionserver, snapshots, Zookeeper
Affects Versions: hbase-6055
Reporter: Jesse Yates
Assignee: Matteo Bertozzi
Fix For: hbase-6055, 0.96.0
Attachments: HBASE-7294-v1.patch, HBASE-7294-v2.patch,
HBASE-7294-v3.patch, HBASE-7294-v4.patch, HBASE-7294-v5.patch,
HBASE-7294-v6.patch, HBASE-7294-v7.patch
Snapshots currently use the SnaphotHfileCleaner and SnapshotHLogCleaner to
ensure that any hfiles or hlogs (respectively) that are currently part of a
snapshot are not removed from their respective archive directories (.archive
and .oldlogs).
From Matteo Bertozzi:
{quote}
currently the snapshot cleaner is not in hbase-default.xml
and there's no warning/exception on snapshot/restore operation, if not
enabled.
even if we add the cleaner to the hbase-default.xml how do we ensure that the
user doesn't remove it?
Do we want to hardcode the cleaner at master startup?
Do we want to add a check in snapshot/restore that throws an exception if the
cleaner is not enabled?
{quote}
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HBASE-7294) Check for snapshot file cleaners on start
[
https://issues.apache.org/jira/browse/HBASE-7294?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13540970#comment-13540970
]
Jonathan Hsieh commented on HBASE-7294:
---
+1. Thanks Matteo, Jesse. Committed to offline and online snapshots branches.
Check for snapshot file cleaners on start
-
Key: HBASE-7294
URL: https://issues.apache.org/jira/browse/HBASE-7294
Project: HBase
Issue Type: Sub-task
Components: Client, master, regionserver, snapshots, Zookeeper
Affects Versions: hbase-6055
Reporter: Jesse Yates
Assignee: Matteo Bertozzi
Fix For: hbase-6055, 0.96.0
Attachments: HBASE-7294-v1.patch, HBASE-7294-v2.patch,
HBASE-7294-v3.patch, HBASE-7294-v4.patch, HBASE-7294-v5.patch,
HBASE-7294-v6.patch, HBASE-7294-v7.patch
Snapshots currently use the SnaphotHfileCleaner and SnapshotHLogCleaner to
ensure that any hfiles or hlogs (respectively) that are currently part of a
snapshot are not removed from their respective archive directories (.archive
and .oldlogs).
From Matteo Bertozzi:
{quote}
currently the snapshot cleaner is not in hbase-default.xml
and there's no warning/exception on snapshot/restore operation, if not
enabled.
even if we add the cleaner to the hbase-default.xml how do we ensure that the
user doesn't remove it?
Do we want to hardcode the cleaner at master startup?
Do we want to add a check in snapshot/restore that throws an exception if the
cleaner is not enabled?
{quote}
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HBASE-7294) Check for snapshot file cleaners on start
[
https://issues.apache.org/jira/browse/HBASE-7294?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13540393#comment-13540393
]
Jonathan Hsieh commented on HBASE-7294:
---
Came back with some consistent failures. The first two are likely related to
this patch, the latter was an error present previously and on trunk (so
ignoring it for this patch for now)
{code}
...
17:09:14 Running org.apache.hadoop.hbase.TestInfoServers
17:09:14 Tests run: 3, Failures: 0, Errors: 1, Skipped: 0, Time elapsed:
15.757 sec FAILURE!
...
17:12:05 Running org.apache.hadoop.hbase.master.cleaner.TestSnapshotFromMaster
17:12:05 Tests run: 4, Failures: 1, Errors: 0, Skipped: 0, Time elapsed:
35.534 sec FAILURE!
...
17:19:07 Running org.apache.hadoop.hbase.client.TestHCM
17:19:07 Tests run: 7, Failures: 0, Errors: 1, Skipped: 0, Time elapsed:
76.637 sec FAILURE!
{code}
Check for snapshot file cleaners on start
-
Key: HBASE-7294
URL: https://issues.apache.org/jira/browse/HBASE-7294
Project: HBase
Issue Type: Sub-task
Components: Client, master, regionserver, snapshots, Zookeeper
Affects Versions: hbase-6055
Reporter: Jesse Yates
Assignee: Matteo Bertozzi
Fix For: hbase-6055, 0.96.0
Attachments: HBASE-7294-v1.patch, HBASE-7294-v2.patch,
HBASE-7294-v3.patch, HBASE-7294-v4.patch, HBASE-7294-v5.patch,
HBASE-7294-v6.patch
Snapshots currently use the SnaphotHfileCleaner and SnapshotHLogCleaner to
ensure that any hfiles or hlogs (respectively) that are currently part of a
snapshot are not removed from their respective archive directories (.archive
and .oldlogs).
From Matteo Bertozzi:
{quote}
currently the snapshot cleaner is not in hbase-default.xml
and there's no warning/exception on snapshot/restore operation, if not
enabled.
even if we add the cleaner to the hbase-default.xml how do we ensure that the
user doesn't remove it?
Do we want to hardcode the cleaner at master startup?
Do we want to add a check in snapshot/restore that throws an exception if the
cleaner is not enabled?
{quote}
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HBASE-7294) Check for snapshot file cleaners on start
[
https://issues.apache.org/jira/browse/HBASE-7294?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13540242#comment-13540242
]
Jonathan Hsieh commented on HBASE-7294:
---
looks good. +1. Running through tests and will commit if they pass.
Check for snapshot file cleaners on start
-
Key: HBASE-7294
URL: https://issues.apache.org/jira/browse/HBASE-7294
Project: HBase
Issue Type: Sub-task
Components: Client, master, regionserver, snapshots, Zookeeper
Affects Versions: hbase-6055
Reporter: Jesse Yates
Assignee: Matteo Bertozzi
Fix For: hbase-6055, 0.96.0
Attachments: HBASE-7294-v1.patch, HBASE-7294-v2.patch,
HBASE-7294-v3.patch, HBASE-7294-v4.patch, HBASE-7294-v5.patch,
HBASE-7294-v6.patch
Snapshots currently use the SnaphotHfileCleaner and SnapshotHLogCleaner to
ensure that any hfiles or hlogs (respectively) that are currently part of a
snapshot are not removed from their respective archive directories (.archive
and .oldlogs).
From Matteo Bertozzi:
{quote}
currently the snapshot cleaner is not in hbase-default.xml
and there's no warning/exception on snapshot/restore operation, if not
enabled.
even if we add the cleaner to the hbase-default.xml how do we ensure that the
user doesn't remove it?
Do we want to hardcode the cleaner at master startup?
Do we want to add a check in snapshot/restore that throws an exception if the
cleaner is not enabled?
{quote}
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HBASE-7294) Check for snapshot file cleaners on start
[
https://issues.apache.org/jira/browse/HBASE-7294?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13539614#comment-13539614
]
Jonathan Hsieh commented on HBASE-7294:
---
We really push folks to use hbase.snapshot.enabled
nit:
Should we update this to describe hbase.snapshot.enabled? (seems simpler)
also mention that you have data in the .snapshots dir (might be strange if it
works with the same config, but later does not. at least if we mention the dir
users will have a pointer and an option to remove snapshots.
{code}
+ /**
+ * Throws an exception if snapshot operations (take a snapshot, restore,
clone) are not supported.
+ * Called at the beginning of snapshot() and restoreSnapshot() methods.
+ * @throws UnsupportedOperationException if snapshot are not supported
+ */
+ public void checkSnapshotSupport() throws UnsupportedOperationException {
+if (!this.isSnapshotSupported) {
+ throw new UnsupportedOperationException(
+To use snapshots, the HBase Master must have the proper archive
cleaners enabled. +
+You must add to the hbase-site.xml: +
+' + HFileCleaner.MASTER_HFILE_CLEANER_PLUGINS + ' with ' +
+HFileLinkCleaner.class.getName() + ', ' +
SnapshotHFileCleaner.class.getName() +
+' support. And add ' + HConstants.HBASE_MASTER_LOGCLEANER_PLUGINS +
' with ' +
+SnapshotLogCleaner.class.getName() + ' support.);
+}
+ }
{code}
We should behave the same but emit a warning if folks are using the plugins and
not the hbase.snapshot.enabled property.
We should have a test the verifies snapshots work if configured one way or the
other (and possibly a negative test exercing the case where it does not.
Check for snapshot file cleaners on start
-
Key: HBASE-7294
URL: https://issues.apache.org/jira/browse/HBASE-7294
Project: HBase
Issue Type: Sub-task
Components: Client, master, regionserver, snapshots, Zookeeper
Affects Versions: hbase-6055
Reporter: Jesse Yates
Assignee: Matteo Bertozzi
Fix For: hbase-6055, 0.96.0
Attachments: HBASE-7294-v1.patch, HBASE-7294-v2.patch,
HBASE-7294-v3.patch, HBASE-7294-v4.patch, HBASE-7294-v5.patch
Snapshots currently use the SnaphotHfileCleaner and SnapshotHLogCleaner to
ensure that any hfiles or hlogs (respectively) that are currently part of a
snapshot are not removed from their respective archive directories (.archive
and .oldlogs).
From Matteo Bertozzi:
{quote}
currently the snapshot cleaner is not in hbase-default.xml
and there's no warning/exception on snapshot/restore operation, if not
enabled.
even if we add the cleaner to the hbase-default.xml how do we ensure that the
user doesn't remove it?
Do we want to hardcode the cleaner at master startup?
Do we want to add a check in snapshot/restore that throws an exception if the
cleaner is not enabled?
{quote}
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HBASE-7294) Check for snapshot file cleaners on start
[
https://issues.apache.org/jira/browse/HBASE-7294?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13538118#comment-13538118
]
Jonathan Hsieh commented on HBASE-7294:
---
v4 Patch does not apply snapshot branch ff25969ea0b2ff38cc25512a02569c351e757d66
Looks like this is built upon some of the online snaphots work. What
branch/hash does this go on Matteo?
Check for snapshot file cleaners on start
-
Key: HBASE-7294
URL: https://issues.apache.org/jira/browse/HBASE-7294
Project: HBase
Issue Type: Sub-task
Components: Client, master, regionserver, snapshots, Zookeeper
Affects Versions: hbase-6055
Reporter: Jesse Yates
Assignee: Matteo Bertozzi
Fix For: hbase-6055, 0.96.0
Attachments: HBASE-7294-v1.patch, HBASE-7294-v2.patch,
HBASE-7294-v3.patch, HBASE-7294-v4.patch
Snapshots currently use the SnaphotHfileCleaner and SnapshotHLogCleaner to
ensure that any hfiles or hlogs (respectively) that are currently part of a
snapshot are not removed from their respective archive directories (.archive
and .oldlogs).
From Matteo Bertozzi:
{quote}
currently the snapshot cleaner is not in hbase-default.xml
and there's no warning/exception on snapshot/restore operation, if not
enabled.
even if we add the cleaner to the hbase-default.xml how do we ensure that the
user doesn't remove it?
Do we want to hardcode the cleaner at master startup?
Do we want to add a check in snapshot/restore that throws an exception if the
cleaner is not enabled?
{quote}
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HBASE-7294) Check for snapshot file cleaners on start
[
https://issues.apache.org/jira/browse/HBASE-7294?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13538121#comment-13538121
]
Matteo Bertozzi commented on HBASE-7294:
this was written on top of the online branch. to avoid move around the code
from HMaster to SnapshotManager
Check for snapshot file cleaners on start
-
Key: HBASE-7294
URL: https://issues.apache.org/jira/browse/HBASE-7294
Project: HBase
Issue Type: Sub-task
Components: Client, master, regionserver, snapshots, Zookeeper
Affects Versions: hbase-6055
Reporter: Jesse Yates
Assignee: Matteo Bertozzi
Fix For: hbase-6055, 0.96.0
Attachments: HBASE-7294-v1.patch, HBASE-7294-v2.patch,
HBASE-7294-v3.patch, HBASE-7294-v4.patch
Snapshots currently use the SnaphotHfileCleaner and SnapshotHLogCleaner to
ensure that any hfiles or hlogs (respectively) that are currently part of a
snapshot are not removed from their respective archive directories (.archive
and .oldlogs).
From Matteo Bertozzi:
{quote}
currently the snapshot cleaner is not in hbase-default.xml
and there's no warning/exception on snapshot/restore operation, if not
enabled.
even if we add the cleaner to the hbase-default.xml how do we ensure that the
user doesn't remove it?
Do we want to hardcode the cleaner at master startup?
Do we want to add a check in snapshot/restore that throws an exception if the
cleaner is not enabled?
{quote}
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HBASE-7294) Check for snapshot file cleaners on start
[
https://issues.apache.org/jira/browse/HBASE-7294?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13538233#comment-13538233
]
Jonathan Hsieh commented on HBASE-7294:
---
Ok, this make this hard to commit. I'll add it to my working branch but not
close this until we get the patches it depends upon committed to the online
branch.
[~jesse_yates] -- we were in a similar position we were at with the offline
stuff before -- we can't commit this because it depends on work that hasn't
been committed to a common branch yet, and preventing other folks from working
on this. Can you do a pass and consider +1'ing some things with caveats to
follow-up on? (HBASE-7208, HBASE-7212, HBASE-6864, HBASE-7321).
Check for snapshot file cleaners on start
-
Key: HBASE-7294
URL: https://issues.apache.org/jira/browse/HBASE-7294
Project: HBase
Issue Type: Sub-task
Components: Client, master, regionserver, snapshots, Zookeeper
Affects Versions: hbase-6055
Reporter: Jesse Yates
Assignee: Matteo Bertozzi
Fix For: hbase-6055, 0.96.0
Attachments: HBASE-7294-v1.patch, HBASE-7294-v2.patch,
HBASE-7294-v3.patch, HBASE-7294-v4.patch
Snapshots currently use the SnaphotHfileCleaner and SnapshotHLogCleaner to
ensure that any hfiles or hlogs (respectively) that are currently part of a
snapshot are not removed from their respective archive directories (.archive
and .oldlogs).
From Matteo Bertozzi:
{quote}
currently the snapshot cleaner is not in hbase-default.xml
and there's no warning/exception on snapshot/restore operation, if not
enabled.
even if we add the cleaner to the hbase-default.xml how do we ensure that the
user doesn't remove it?
Do we want to hardcode the cleaner at master startup?
Do we want to add a check in snapshot/restore that throws an exception if the
cleaner is not enabled?
{quote}
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HBASE-7294) Check for snapshot file cleaners on start
[
https://issues.apache.org/jira/browse/HBASE-7294?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13537288#comment-13537288
]
Jesse Yates commented on HBASE-7294:
minor nit: we have a lot of snapshot tests that need the cleaners, maybe
centralize to a single method to add the cleaners in a SnapshotTestingUtilitity
(or something similar)? Could even be a setupCluster(Configuration) method so
we can easily add on other things later (though this may all be excessive :))
+1 if tests are passing.
Check for snapshot file cleaners on start
-
Key: HBASE-7294
URL: https://issues.apache.org/jira/browse/HBASE-7294
Project: HBase
Issue Type: Sub-task
Components: Client, master, regionserver, snapshots, Zookeeper
Affects Versions: hbase-6055
Reporter: Jesse Yates
Assignee: Matteo Bertozzi
Fix For: hbase-6055, 0.96.0
Attachments: HBASE-7294-v1.patch, HBASE-7294-v2.patch,
HBASE-7294-v3.patch, HBASE-7294-v4.patch
Snapshots currently use the SnaphotHfileCleaner and SnapshotHLogCleaner to
ensure that any hfiles or hlogs (respectively) that are currently part of a
snapshot are not removed from their respective archive directories (.archive
and .oldlogs).
From Matteo Bertozzi:
{quote}
currently the snapshot cleaner is not in hbase-default.xml
and there's no warning/exception on snapshot/restore operation, if not
enabled.
even if we add the cleaner to the hbase-default.xml how do we ensure that the
user doesn't remove it?
Do we want to hardcode the cleaner at master startup?
Do we want to add a check in snapshot/restore that throws an exception if the
cleaner is not enabled?
{quote}
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HBASE-7294) Check for snapshot file cleaners on start
[
https://issues.apache.org/jira/browse/HBASE-7294?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13537293#comment-13537293
]
Jonathan Hsieh commented on HBASE-7294:
---
Another consideration is to turn these cleaners on by default, or have a single
is-snapshots-on? config var that does this work. This would be a follow-on
jira, and not necessary here.
Check for snapshot file cleaners on start
-
Key: HBASE-7294
URL: https://issues.apache.org/jira/browse/HBASE-7294
Project: HBase
Issue Type: Sub-task
Components: Client, master, regionserver, snapshots, Zookeeper
Affects Versions: hbase-6055
Reporter: Jesse Yates
Assignee: Matteo Bertozzi
Fix For: hbase-6055, 0.96.0
Attachments: HBASE-7294-v1.patch, HBASE-7294-v2.patch,
HBASE-7294-v3.patch, HBASE-7294-v4.patch
Snapshots currently use the SnaphotHfileCleaner and SnapshotHLogCleaner to
ensure that any hfiles or hlogs (respectively) that are currently part of a
snapshot are not removed from their respective archive directories (.archive
and .oldlogs).
From Matteo Bertozzi:
{quote}
currently the snapshot cleaner is not in hbase-default.xml
and there's no warning/exception on snapshot/restore operation, if not
enabled.
even if we add the cleaner to the hbase-default.xml how do we ensure that the
user doesn't remove it?
Do we want to hardcode the cleaner at master startup?
Do we want to add a check in snapshot/restore that throws an exception if the
cleaner is not enabled?
{quote}
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HBASE-7294) Check for snapshot file cleaners on start
[
https://issues.apache.org/jira/browse/HBASE-7294?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13536026#comment-13536026
]
Hadoop QA commented on HBASE-7294:
--
{color:red}-1 overall{color}. Here are the results of testing the latest
attachment
http://issues.apache.org/jira/secure/attachment/12561699/HBASE-7294-v2.patch
against trunk revision .
{color:green}+1 @author{color}. The patch does not contain any @author
tags.
{color:red}-1 tests included{color}. The patch doesn't appear to include
any new or modified tests.
Please justify why no new tests are needed for this
patch.
Also please list what manual steps were performed to
verify this patch.
{color:red}-1 patch{color}. The patch command could not apply the patch.
Console output:
https://builds.apache.org/job/PreCommit-HBASE-Build/3610//console
This message is automatically generated.
Check for snapshot file cleaners on start
-
Key: HBASE-7294
URL: https://issues.apache.org/jira/browse/HBASE-7294
Project: HBase
Issue Type: Sub-task
Components: Client, master, regionserver, snapshots, Zookeeper
Affects Versions: hbase-6055
Reporter: Jesse Yates
Assignee: Matteo Bertozzi
Fix For: hbase-6055, 0.96.0
Attachments: HBASE-7294-v1.patch, HBASE-7294-v2.patch
Snapshots currently use the SnaphotHfileCleaner and SnapshotHLogCleaner to
ensure that any hfiles or hlogs (respectively) that are currently part of a
snapshot are not removed from their respective archive directories (.archive
and .oldlogs).
From Matteo Bertozzi:
{quote}
currently the snapshot cleaner is not in hbase-default.xml
and there's no warning/exception on snapshot/restore operation, if not
enabled.
even if we add the cleaner to the hbase-default.xml how do we ensure that the
user doesn't remove it?
Do we want to hardcode the cleaner at master startup?
Do we want to add a check in snapshot/restore that throws an exception if the
cleaner is not enabled?
{quote}
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HBASE-7294) Check for snapshot file cleaners on start
[
https://issues.apache.org/jira/browse/HBASE-7294?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13536037#comment-13536037
]
Jonathan Hsieh commented on HBASE-7294:
---
Please fix the To been... comment. It is still grammatically incorrect.
Check for snapshot file cleaners on start
-
Key: HBASE-7294
URL: https://issues.apache.org/jira/browse/HBASE-7294
Project: HBase
Issue Type: Sub-task
Components: Client, master, regionserver, snapshots, Zookeeper
Affects Versions: hbase-6055
Reporter: Jesse Yates
Assignee: Matteo Bertozzi
Fix For: hbase-6055, 0.96.0
Attachments: HBASE-7294-v1.patch, HBASE-7294-v2.patch
Snapshots currently use the SnaphotHfileCleaner and SnapshotHLogCleaner to
ensure that any hfiles or hlogs (respectively) that are currently part of a
snapshot are not removed from their respective archive directories (.archive
and .oldlogs).
From Matteo Bertozzi:
{quote}
currently the snapshot cleaner is not in hbase-default.xml
and there's no warning/exception on snapshot/restore operation, if not
enabled.
even if we add the cleaner to the hbase-default.xml how do we ensure that the
user doesn't remove it?
Do we want to hardcode the cleaner at master startup?
Do we want to add a check in snapshot/restore that throws an exception if the
cleaner is not enabled?
{quote}
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HBASE-7294) Check for snapshot file cleaners on start
[
https://issues.apache.org/jira/browse/HBASE-7294?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13536051#comment-13536051
]
Hadoop QA commented on HBASE-7294:
--
{color:red}-1 overall{color}. Here are the results of testing the latest
attachment
http://issues.apache.org/jira/secure/attachment/12561711/HBASE-7294-v3.patch
against trunk revision .
{color:green}+1 @author{color}. The patch does not contain any @author
tags.
{color:red}-1 tests included{color}. The patch doesn't appear to include
any new or modified tests.
Please justify why no new tests are needed for this
patch.
Also please list what manual steps were performed to
verify this patch.
{color:red}-1 patch{color}. The patch command could not apply the patch.
Console output:
https://builds.apache.org/job/PreCommit-HBASE-Build/3612//console
This message is automatically generated.
Check for snapshot file cleaners on start
-
Key: HBASE-7294
URL: https://issues.apache.org/jira/browse/HBASE-7294
Project: HBase
Issue Type: Sub-task
Components: Client, master, regionserver, snapshots, Zookeeper
Affects Versions: hbase-6055
Reporter: Jesse Yates
Assignee: Matteo Bertozzi
Fix For: hbase-6055, 0.96.0
Attachments: HBASE-7294-v1.patch, HBASE-7294-v2.patch,
HBASE-7294-v3.patch
Snapshots currently use the SnaphotHfileCleaner and SnapshotHLogCleaner to
ensure that any hfiles or hlogs (respectively) that are currently part of a
snapshot are not removed from their respective archive directories (.archive
and .oldlogs).
From Matteo Bertozzi:
{quote}
currently the snapshot cleaner is not in hbase-default.xml
and there's no warning/exception on snapshot/restore operation, if not
enabled.
even if we add the cleaner to the hbase-default.xml how do we ensure that the
user doesn't remove it?
Do we want to hardcode the cleaner at master startup?
Do we want to add a check in snapshot/restore that throws an exception if the
cleaner is not enabled?
{quote}
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HBASE-7294) Check for snapshot file cleaners on start
[
https://issues.apache.org/jira/browse/HBASE-7294?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13536200#comment-13536200
]
Ted Yu commented on HBASE-7294:
---
{code}
+ * Called at startup, to verify if snapshot operation are supported, and to
avoid
{code}
'operation are' - 'operation is'
{code}
+ private void checkSnapshotSupport(final Configuration conf, final
MasterFileSystem mfs)
{code}
Please add @param for the two parameters.
{code}
+// Mark snapshot feature has enabled if cleaners are present
{code}
'feature has' - 'feature as'
{code}
+// otherwise we end up snapshot with data loss.
{code}
'end up snapshot with' - 'end up with snapshot'
{code}
+'hbase.master.hfilecleaner.plugins' with ' +
{code}
Please refer to constants instead of hardcoding. Same with
'hbase.master.logcleaner.plugins'
Check for snapshot file cleaners on start
-
Key: HBASE-7294
URL: https://issues.apache.org/jira/browse/HBASE-7294
Project: HBase
Issue Type: Sub-task
Components: Client, master, regionserver, snapshots, Zookeeper
Affects Versions: hbase-6055
Reporter: Jesse Yates
Assignee: Matteo Bertozzi
Fix For: hbase-6055, 0.96.0
Attachments: HBASE-7294-v1.patch, HBASE-7294-v2.patch,
HBASE-7294-v3.patch
Snapshots currently use the SnaphotHfileCleaner and SnapshotHLogCleaner to
ensure that any hfiles or hlogs (respectively) that are currently part of a
snapshot are not removed from their respective archive directories (.archive
and .oldlogs).
From Matteo Bertozzi:
{quote}
currently the snapshot cleaner is not in hbase-default.xml
and there's no warning/exception on snapshot/restore operation, if not
enabled.
even if we add the cleaner to the hbase-default.xml how do we ensure that the
user doesn't remove it?
Do we want to hardcode the cleaner at master startup?
Do we want to add a check in snapshot/restore that throws an exception if the
cleaner is not enabled?
{quote}
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HBASE-7294) Check for snapshot file cleaners on start
[
https://issues.apache.org/jira/browse/HBASE-7294?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13536240#comment-13536240
]
Jonathan Hsieh commented on HBASE-7294:
---
We need to make sure the snapshots unit tests pass as well -- I believe they
should all fail once this gets in because I don't think all install the
cleaners.
Check for snapshot file cleaners on start
-
Key: HBASE-7294
URL: https://issues.apache.org/jira/browse/HBASE-7294
Project: HBase
Issue Type: Sub-task
Components: Client, master, regionserver, snapshots, Zookeeper
Affects Versions: hbase-6055
Reporter: Jesse Yates
Assignee: Matteo Bertozzi
Fix For: hbase-6055, 0.96.0
Attachments: HBASE-7294-v1.patch, HBASE-7294-v2.patch,
HBASE-7294-v3.patch
Snapshots currently use the SnaphotHfileCleaner and SnapshotHLogCleaner to
ensure that any hfiles or hlogs (respectively) that are currently part of a
snapshot are not removed from their respective archive directories (.archive
and .oldlogs).
From Matteo Bertozzi:
{quote}
currently the snapshot cleaner is not in hbase-default.xml
and there's no warning/exception on snapshot/restore operation, if not
enabled.
even if we add the cleaner to the hbase-default.xml how do we ensure that the
user doesn't remove it?
Do we want to hardcode the cleaner at master startup?
Do we want to add a check in snapshot/restore that throws an exception if the
cleaner is not enabled?
{quote}
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HBASE-7294) Check for snapshot file cleaners on start
[
https://issues.apache.org/jira/browse/HBASE-7294?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13536378#comment-13536378
]
Hadoop QA commented on HBASE-7294:
--
{color:red}-1 overall{color}. Here are the results of testing the latest
attachment
http://issues.apache.org/jira/secure/attachment/12561777/HBASE-7294-v4.patch
against trunk revision .
{color:green}+1 @author{color}. The patch does not contain any @author
tags.
{color:green}+1 tests included{color}. The patch appears to include 21 new
or modified tests.
{color:red}-1 patch{color}. The patch command could not apply the patch.
Console output:
https://builds.apache.org/job/PreCommit-HBASE-Build/3618//console
This message is automatically generated.
Check for snapshot file cleaners on start
-
Key: HBASE-7294
URL: https://issues.apache.org/jira/browse/HBASE-7294
Project: HBase
Issue Type: Sub-task
Components: Client, master, regionserver, snapshots, Zookeeper
Affects Versions: hbase-6055
Reporter: Jesse Yates
Assignee: Matteo Bertozzi
Fix For: hbase-6055, 0.96.0
Attachments: HBASE-7294-v1.patch, HBASE-7294-v2.patch,
HBASE-7294-v3.patch, HBASE-7294-v4.patch
Snapshots currently use the SnaphotHfileCleaner and SnapshotHLogCleaner to
ensure that any hfiles or hlogs (respectively) that are currently part of a
snapshot are not removed from their respective archive directories (.archive
and .oldlogs).
From Matteo Bertozzi:
{quote}
currently the snapshot cleaner is not in hbase-default.xml
and there's no warning/exception on snapshot/restore operation, if not
enabled.
even if we add the cleaner to the hbase-default.xml how do we ensure that the
user doesn't remove it?
Do we want to hardcode the cleaner at master startup?
Do we want to add a check in snapshot/restore that throws an exception if the
cleaner is not enabled?
{quote}
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HBASE-7294) Check for snapshot file cleaners on start
[
https://issues.apache.org/jira/browse/HBASE-7294?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13535437#comment-13535437
]
Jonathan Hsieh commented on HBASE-7294:
---
seems like info level.
{code}
+ LOG.warn(Snapshot feature is not enabled, missing log and hfile
cleaners.);
{code}
slight reword of this
{code}
+To been able to use snapshots HBase Master must have cleaners
enabled. +
{code}
to To use snapshots, the HBase Master must have the proper archive cleaners
enabled.
Does this print out exactly what we want for these *.class? We want it to
print 'org.apache.hadoop.hbaseSnapshotHFileCleaner'. See
http://docs.oracle.com/javase/1.5.0/docs/api/java/lang/Class.html#toString()
about the Class#toString method. I think you many need to make it
SnapshotHFileCleaner.getName() (or maybe it is getCanonicalName()).
{code}
+'hbase.master.hfilecleaner.plugins' with ' +
+HFileLinkCleaner.class + ', ' + SnapshotHFileCleaner.class +
+' support. And add 'hbase.master.logcleaner.plugins' with ' +
{code}
Check for snapshot file cleaners on start
-
Key: HBASE-7294
URL: https://issues.apache.org/jira/browse/HBASE-7294
Project: HBase
Issue Type: Sub-task
Components: Client, master, regionserver, snapshots, Zookeeper
Affects Versions: hbase-6055
Reporter: Jesse Yates
Assignee: Matteo Bertozzi
Fix For: hbase-6055, 0.96.0
Attachments: HBASE-7294-v1.patch
Snapshots currently use the SnaphotHfileCleaner and SnapshotHLogCleaner to
ensure that any hfiles or hlogs (respectively) that are currently part of a
snapshot are not removed from their respective archive directories (.archive
and .oldlogs).
From Matteo Bertozzi:
{quote}
currently the snapshot cleaner is not in hbase-default.xml
and there's no warning/exception on snapshot/restore operation, if not
enabled.
even if we add the cleaner to the hbase-default.xml how do we ensure that the
user doesn't remove it?
Do we want to hardcode the cleaner at master startup?
Do we want to add a check in snapshot/restore that throws an exception if the
cleaner is not enabled?
{quote}
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HBASE-7294) Check for snapshot file cleaners on start
[
https://issues.apache.org/jira/browse/HBASE-7294?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13533815#comment-13533815
]
Hadoop QA commented on HBASE-7294:
--
{color:red}-1 overall{color}. Here are the results of testing the latest
attachment
http://issues.apache.org/jira/secure/attachment/12561265/HBASE-7294-v1.patch
against trunk revision .
{color:green}+1 @author{color}. The patch does not contain any @author
tags.
{color:red}-1 tests included{color}. The patch doesn't appear to include
any new or modified tests.
Please justify why no new tests are needed for this
patch.
Also please list what manual steps were performed to
verify this patch.
{color:red}-1 patch{color}. The patch command could not apply the patch.
Console output:
https://builds.apache.org/job/PreCommit-HBASE-Build/3568//console
This message is automatically generated.
Check for snapshot file cleaners on start
-
Key: HBASE-7294
URL: https://issues.apache.org/jira/browse/HBASE-7294
Project: HBase
Issue Type: Sub-task
Components: Client, master, regionserver, snapshots, Zookeeper
Affects Versions: hbase-6055
Reporter: Jesse Yates
Assignee: Matteo Bertozzi
Fix For: hbase-6055, 0.96.0
Attachments: HBASE-7294-v1.patch
Snapshots currently use the SnaphotHfileCleaner and SnapshotHLogCleaner to
ensure that any hfiles or hlogs (respectively) that are currently part of a
snapshot are not removed from their respective archive directories (.archive
and .oldlogs).
From Matteo Bertozzi:
{quote}
currently the snapshot cleaner is not in hbase-default.xml
and there's no warning/exception on snapshot/restore operation, if not
enabled.
even if we add the cleaner to the hbase-default.xml how do we ensure that the
user doesn't remove it?
Do we want to hardcode the cleaner at master startup?
Do we want to add a check in snapshot/restore that throws an exception if the
cleaner is not enabled?
{quote}
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HBASE-7294) Check for snapshot file cleaners on start
[
https://issues.apache.org/jira/browse/HBASE-7294?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13533376#comment-13533376
]
Hadoop QA commented on HBASE-7294:
--
{color:red}-1 overall{color}. Here are the results of testing the latest
attachment
http://issues.apache.org/jira/secure/attachment/12561184/HBASE-7294-v0.patch
against trunk revision .
{color:green}+1 @author{color}. The patch does not contain any @author
tags.
{color:red}-1 tests included{color}. The patch doesn't appear to include
any new or modified tests.
Please justify why no new tests are needed for this
patch.
Also please list what manual steps were performed to
verify this patch.
{color:red}-1 patch{color}. The patch command could not apply the patch.
Console output:
https://builds.apache.org/job/PreCommit-HBASE-Build/3565//console
This message is automatically generated.
Check for snapshot file cleaners on start
-
Key: HBASE-7294
URL: https://issues.apache.org/jira/browse/HBASE-7294
Project: HBase
Issue Type: Sub-task
Components: Client, master, regionserver, snapshots, Zookeeper
Affects Versions: hbase-6055
Reporter: Jesse Yates
Assignee: Matteo Bertozzi
Fix For: hbase-6055, 0.96.0
Attachments: HBASE-7294-v0.patch
Snapshots currently use the SnaphotHfileCleaner and SnapshotHLogCleaner to
ensure that any hfiles or hlogs (respectively) that are currently part of a
snapshot are not removed from their respective archive directories (.archive
and .oldlogs).
From Matteo Bertozzi:
{quote}
currently the snapshot cleaner is not in hbase-default.xml
and there's no warning/exception on snapshot/restore operation, if not
enabled.
even if we add the cleaner to the hbase-default.xml how do we ensure that the
user doesn't remove it?
Do we want to hardcode the cleaner at master startup?
Do we want to add a check in snapshot/restore that throws an exception if the
cleaner is not enabled?
{quote}
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HBASE-7294) Check for snapshot file cleaners on start
[
https://issues.apache.org/jira/browse/HBASE-7294?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13526484#comment-13526484
]
Jonathan Hsieh commented on HBASE-7294:
---
The first should be the simplest -- no ability to take snapshots if data loss
is possible, no ability to loss data from existing snapshots, and no fancy
policies until we get the basic, default, and definitely correct implementation
in first.
Check for snapshot file cleaners on start
-
Key: HBASE-7294
URL: https://issues.apache.org/jira/browse/HBASE-7294
Project: HBase
Issue Type: Sub-task
Components: Client, master, regionserver, snapshots, Zookeeper
Affects Versions: hbase-6055
Reporter: Jesse Yates
Fix For: hbase-6055, 0.96.0
Snapshots currently use the SnaphotHfileCleaner and SnapshotHLogCleaner to
ensure that any hfiles or hlogs (respectively) that are currently part of a
snapshot are not removed from their respective archive directories (.archive
and .oldlogs).
From Matteo Bertozzi:
{quote}
currently the snapshot cleaner is not in hbase-default.xml
and there's no warning/exception on snapshot/restore operation, if not
enabled.
even if we add the cleaner to the hbase-default.xml how do we ensure that the
user doesn't remove it?
Do we want to hardcode the cleaner at master startup?
Do we want to add a check in snapshot/restore that throws an exception if the
cleaner is not enabled?
{quote}
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HBASE-7294) Check for snapshot file cleaners on start
[
https://issues.apache.org/jira/browse/HBASE-7294?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13526023#comment-13526023
]
Jonathan Hsieh commented on HBASE-7294:
---
I think we should not allow snapshots if the proper cleaner is not installed,
and I think the rs and hm startup code should also check to see if there are
.snapshots present and abort startup if the proper cleaner is not installed
with a log message saying that this must be done.
We *do not* want to even allow an admin accidentally change something that
could result in data loss.
Check for snapshot file cleaners on start
-
Key: HBASE-7294
URL: https://issues.apache.org/jira/browse/HBASE-7294
Project: HBase
Issue Type: Sub-task
Components: Client, master, regionserver, snapshots, Zookeeper
Affects Versions: hbase-6055
Reporter: Jesse Yates
Fix For: hbase-6055, 0.96.0
From Matteo Bertozzi:
{quote}
currently the snapshot cleaner is not in hbase-default.xml
and there's no warning/exception on snapshot/restore operation, if not
enabled.
even if we add the cleaner to the hbase-default.xml how do we ensure that the
user doesn't remove it?
Do we want to hardcode the cleaner at master startup?
Do we want to add a check in snapshot/restore that throws an exception if the
cleaner is not enabled?
{quote}
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HBASE-7294) Check for snapshot file cleaners on start
[
https://issues.apache.org/jira/browse/HBASE-7294?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13526026#comment-13526026
]
Jesse Yates commented on HBASE-7294:
I'd like to see the cleaners added to hbase-default.xml and then have the
snapshot manager check to make sure that the correct cleaners are enabled on
startup (similar to how we cleanup the .snapshot/.tmp directory in HBASE-7240).
If the cleaners aren't enabled, we don't block startup, but instead prevent any
snapshots from being taken.
There is the consideration of 'what if there are existing snapshots'? The last
thing we want is unintentional data loss. Here we have two options: (1) fail
startup with an error message telling the user to remove the snapshots before
preceeding, or (2) add the snapshot cleaner back in.
I think (1) is the better case, but it also requires adding another
configuration option, specifiying the acceptable classes for snapshot cleaning.
A user might want to have their own policies for cleaning up files related to
snaphots (for instance, even if there is a snapshot, we want to remove the
files after 30 days, but otherwise don't remove them). By default, we don't
remove ANY files that are in a snapshot. (there also might be room here to make
a configurable policy for the snapshot file cleaners to take a TTL, but that
defaults to infinity, but that's probably a bit premature).
Check for snapshot file cleaners on start
-
Key: HBASE-7294
URL: https://issues.apache.org/jira/browse/HBASE-7294
Project: HBase
Issue Type: Sub-task
Components: Client, master, regionserver, snapshots, Zookeeper
Affects Versions: hbase-6055
Reporter: Jesse Yates
Fix For: hbase-6055, 0.96.0
From Matteo Bertozzi:
{quote}
currently the snapshot cleaner is not in hbase-default.xml
and there's no warning/exception on snapshot/restore operation, if not
enabled.
even if we add the cleaner to the hbase-default.xml how do we ensure that the
user doesn't remove it?
Do we want to hardcode the cleaner at master startup?
Do we want to add a check in snapshot/restore that throws an exception if the
cleaner is not enabled?
{quote}
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators
For more information on JIRA, see: http://www.atlassian.com/software/jira
