[jira] [Commented] (HBASE-7321) Simple Flush Snapshot
[ https://issues.apache.org/jira/browse/HBASE-7321?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13541485#comment-13541485 ] Jonathan Hsieh commented on HBASE-7321: --- The unstability I mentioned was fixed in v4 of HBSAE-6864 Simple Flush Snapshot - Key: HBASE-7321 URL: https://issues.apache.org/jira/browse/HBASE-7321 Project: HBase Issue Type: Sub-task Reporter: Jonathan Hsieh Assignee: Jonathan Hsieh Attachments: hbase-7321.v2.patch, hbase-7321.v3.patch, hbase-7321.v4.patch, pre-hbase-7321.v2.patch, pre-hbase-7321.v3.patch, pre-hbase-7321.v4.patch This snapshot style just issues a region flush and then snapshots the region. This is a simple implementation that gives the equivalent of copytable consistency. While by most definitions of consistency if a client writes A and then write B to different region servers, only neither, only A, or both A+B writes should be present, this one allows the only B case. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HBASE-7321) Simple Flush Snapshot
[
https://issues.apache.org/jira/browse/HBASE-7321?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13541502#comment-13541502
]
Hadoop QA commented on HBASE-7321:
--
{color:red}-1 overall{color}. Here are the results of testing the latest
attachment
http://issues.apache.org/jira/secure/attachment/12562817/pre-hbase-7321.v4.patch
against trunk revision .
{color:green}+1 @author{color}. The patch does not contain any @author
tags.
{color:green}+1 tests included{color}. The patch appears to include 88 new
or modified tests.
{color:red}-1 patch{color}. The patch command could not apply the patch.
Console output:
https://builds.apache.org/job/PreCommit-HBASE-Build/3786//console
This message is automatically generated.
Simple Flush Snapshot
-
Key: HBASE-7321
URL: https://issues.apache.org/jira/browse/HBASE-7321
Project: HBase
Issue Type: Sub-task
Reporter: Jonathan Hsieh
Assignee: Jonathan Hsieh
Attachments: hbase-7321.v2.patch, hbase-7321.v3.patch,
hbase-7321.v4.patch, pre-hbase-7321.v2.patch, pre-hbase-7321.v3.patch,
pre-hbase-7321.v4.patch
This snapshot style just issues a region flush and then snapshots the
region.
This is a simple implementation that gives the equivalent of copytable
consistency. While by most definitions of consistency if a client writes A
and then write B to different region servers, only neither, only A, or both
A+B writes should be present, this one allows the only B case.
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HBASE-7321) Simple Flush Snapshot
[ https://issues.apache.org/jira/browse/HBASE-7321?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13541006#comment-13541006 ] Jonathan Hsieh commented on HBASE-7321: --- I've addressed ted's comments on the snapshot-work-1229 version. I'll file a follow on for the TODO about refactoring TestFlushSnapshotFromClient. Currently the order of patches has changed, and this is the first online version to make it. The refactor makes sense for when we add a 3rd flavor (offline, flush, + the next one). Simple Flush Snapshot - Key: HBASE-7321 URL: https://issues.apache.org/jira/browse/HBASE-7321 Project: HBase Issue Type: Sub-task Reporter: Jonathan Hsieh Assignee: Jonathan Hsieh Attachments: hbase-7321.v2.patch, pre-hbase-7321.v2.patch This snapshot style just issues a region flush and then snapshots the region. This is a simple implementation that gives the equivalent of copytable consistency. While by most definitions of consistency if a client writes A and then write B to different region servers, only neither, only A, or both A+B writes should be present, this one allows the only B case. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HBASE-7321) Simple Flush Snapshot
[
https://issues.apache.org/jira/browse/HBASE-7321?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13541017#comment-13541017
]
Hadoop QA commented on HBASE-7321:
--
{color:red}-1 overall{color}. Here are the results of testing the latest
attachment
http://issues.apache.org/jira/secure/attachment/12562699/pre-hbase-7321.v3.patch
against trunk revision .
{color:green}+1 @author{color}. The patch does not contain any @author
tags.
{color:green}+1 tests included{color}. The patch appears to include 88 new
or modified tests.
{color:red}-1 patch{color}. The patch command could not apply the patch.
Console output:
https://builds.apache.org/job/PreCommit-HBASE-Build/3760//console
This message is automatically generated.
Simple Flush Snapshot
-
Key: HBASE-7321
URL: https://issues.apache.org/jira/browse/HBASE-7321
Project: HBase
Issue Type: Sub-task
Reporter: Jonathan Hsieh
Assignee: Jonathan Hsieh
Attachments: hbase-7321.v2.patch, hbase-7321.v3.patch,
pre-hbase-7321.v2.patch, pre-hbase-7321.v3.patch
This snapshot style just issues a region flush and then snapshots the
region.
This is a simple implementation that gives the equivalent of copytable
consistency. While by most definitions of consistency if a client writes A
and then write B to different region servers, only neither, only A, or both
A+B writes should be present, this one allows the only B case.
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HBASE-7321) Simple Flush Snapshot
[ https://issues.apache.org/jira/browse/HBASE-7321?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13539721#comment-13539721 ] Jonathan Hsieh commented on HBASE-7321: --- Some of the recent underlaying changes has made the tests in this patch flakey... Simple Flush Snapshot - Key: HBASE-7321 URL: https://issues.apache.org/jira/browse/HBASE-7321 Project: HBase Issue Type: Sub-task Reporter: Jonathan Hsieh Assignee: Jonathan Hsieh Attachments: hbase-7321.v2.patch, pre-hbase-7321.v2.patch This snapshot style just issues a region flush and then snapshots the region. This is a simple implementation that gives the equivalent of copytable consistency. While by most definitions of consistency if a client writes A and then write B to different region servers, only neither, only A, or both A+B writes should be present, this one allows the only B case. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HBASE-7321) Simple Flush Snapshot
[ https://issues.apache.org/jira/browse/HBASE-7321?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13539830#comment-13539830 ] Jonathan Hsieh commented on HBASE-7321: --- Seems the unit tests are still flakey on the snapshot-work-1223 verison. :( Simple Flush Snapshot - Key: HBASE-7321 URL: https://issues.apache.org/jira/browse/HBASE-7321 Project: HBase Issue Type: Sub-task Reporter: Jonathan Hsieh Assignee: Jonathan Hsieh Attachments: hbase-7321.v2.patch, pre-hbase-7321.v2.patch This snapshot style just issues a region flush and then snapshots the region. This is a simple implementation that gives the equivalent of copytable consistency. While by most definitions of consistency if a client writes A and then write B to different region servers, only neither, only A, or both A+B writes should be present, this one allows the only B case. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HBASE-7321) Simple Flush Snapshot
[ https://issues.apache.org/jira/browse/HBASE-7321?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13537301#comment-13537301 ] Jesse Yates commented on HBASE-7321: This strikes me as 'not really a snapshot' in that its very hard to reason about what writes will or won't be included in the snapshot, from a client perspective. To me (and maybe this is a group of 1) snapshot means 'at a point in time', but this is a far more ragged form. Traditionally, I believe snapshots are (the current state of the DB) - (uncommitted transactions), which for our case would be mutations that haven't completed when the snapshot starts. Because we don't coordinate between regionservers, we can't give 'a point in time' as a reference, but rather a just a 'fuzzy' approximation (as you mention in the review, its an optimization on copytable), and therefore I don't feel that 'snapshot' is the best name for this operation as people could easily be confused by this. I'd be ok if you went with 'Fuzzy Snapshot' and a decent description (brings up case where we need to add a section to the refguide about the different types of snapshots with good descriptions) Simple Flush Snapshot - Key: HBASE-7321 URL: https://issues.apache.org/jira/browse/HBASE-7321 Project: HBase Issue Type: Sub-task Reporter: Jonathan Hsieh Assignee: Jonathan Hsieh Attachments: hbase-7321.v2.patch, pre-hbase-7321.v2.patch This snapshot style just issues a region flush and then snapshots the region. This is a simple implementation that gives the equivalent of copytable consistency. While by most definitions of consistency if a client writes A and then write B to different region servers, only neither, only A, or both A+B writes should be present, this one allows the only B case. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HBASE-7321) Simple Flush Snapshot
[ https://issues.apache.org/jira/browse/HBASE-7321?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13537405#comment-13537405 ] Jonathan Hsieh commented on HBASE-7321: --- In my opinion, definition of point-in-time is a bit fuzzy a well. If you are talking about the timestamps snapshot the way I currently understand this there are no gurantees between regionservers so the timestamp snapshot is 'not really a snapshot' either because causality could potentially be violated. (the limitation in the second paragraph of the description above). An improvement on copy table is a good improvement and I'm fine with calling this a fuzzy snapshot or describing it as such in the docs. I'm hesitant to go and change all the names or add add new commands (fuzzy_snapshot command seems like overkill). I've filed docs jiras HBASE-7409, HBASE-7410 to put explanations there and marked them as blockers. Good enough? I don't plan on working through the other flavors until we have this, likely the simplest online snapshot, and all its tooling and plumbing robust and resolved. Simple Flush Snapshot - Key: HBASE-7321 URL: https://issues.apache.org/jira/browse/HBASE-7321 Project: HBase Issue Type: Sub-task Reporter: Jonathan Hsieh Assignee: Jonathan Hsieh Attachments: hbase-7321.v2.patch, pre-hbase-7321.v2.patch This snapshot style just issues a region flush and then snapshots the region. This is a simple implementation that gives the equivalent of copytable consistency. While by most definitions of consistency if a client writes A and then write B to different region servers, only neither, only A, or both A+B writes should be present, this one allows the only B case. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HBASE-7321) Simple Flush Snapshot
[ https://issues.apache.org/jira/browse/HBASE-7321?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13537463#comment-13537463 ] Jesse Yates commented on HBASE-7321: Yeah, that works for me :) Thanks Jon. Simple Flush Snapshot - Key: HBASE-7321 URL: https://issues.apache.org/jira/browse/HBASE-7321 Project: HBase Issue Type: Sub-task Reporter: Jonathan Hsieh Assignee: Jonathan Hsieh Attachments: hbase-7321.v2.patch, pre-hbase-7321.v2.patch This snapshot style just issues a region flush and then snapshots the region. This is a simple implementation that gives the equivalent of copytable consistency. While by most definitions of consistency if a client writes A and then write B to different region servers, only neither, only A, or both A+B writes should be present, this one allows the only B case. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HBASE-7321) Simple Flush Snapshot
[
https://issues.apache.org/jira/browse/HBASE-7321?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13536668#comment-13536668
]
Hadoop QA commented on HBASE-7321:
--
{color:red}-1 overall{color}. Here are the results of testing the latest
attachment
http://issues.apache.org/jira/secure/attachment/12561816/hbase-7321.v2.patch
against trunk revision .
{color:green}+1 @author{color}. The patch does not contain any @author
tags.
{color:green}+1 tests included{color}. The patch appears to include 7 new
or modified tests.
{color:red}-1 patch{color}. The patch command could not apply the patch.
Console output:
https://builds.apache.org/job/PreCommit-HBASE-Build/3625//console
This message is automatically generated.
Simple Flush Snapshot
-
Key: HBASE-7321
URL: https://issues.apache.org/jira/browse/HBASE-7321
Project: HBase
Issue Type: Sub-task
Reporter: Jonathan Hsieh
Assignee: Jonathan Hsieh
Attachments: hbase-7321.v2.patch, pre-hbase-7321.v2.patch
This snapshot style just issues a region flush and then snapshots the
region.
This is a simple implementation that gives the equivalent of copytable
consistency. While by most definitions of consistency if a client writes A
and then write B to different region servers, only neither, only A, or both
A+B writes should be present, this one allows the only B case.
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HBASE-7321) Simple Flush Snapshot
[
https://issues.apache.org/jira/browse/HBASE-7321?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13536717#comment-13536717
]
Hadoop QA commented on HBASE-7321:
--
{color:red}-1 overall{color}. Here are the results of testing the latest
attachment
http://issues.apache.org/jira/secure/attachment/12561816/hbase-7321.v2.patch
against trunk revision .
{color:green}+1 @author{color}. The patch does not contain any @author
tags.
{color:green}+1 tests included{color}. The patch appears to include 7 new
or modified tests.
{color:red}-1 patch{color}. The patch command could not apply the patch.
Console output:
https://builds.apache.org/job/PreCommit-HBASE-Build/3628//console
This message is automatically generated.
Simple Flush Snapshot
-
Key: HBASE-7321
URL: https://issues.apache.org/jira/browse/HBASE-7321
Project: HBase
Issue Type: Sub-task
Reporter: Jonathan Hsieh
Assignee: Jonathan Hsieh
Attachments: hbase-7321.v2.patch, pre-hbase-7321.v2.patch
This snapshot style just issues a region flush and then snapshots the
region.
This is a simple implementation that gives the equivalent of copytable
consistency. While by most definitions of consistency if a client writes A
and then write B to different region servers, only neither, only A, or both
A+B writes should be present, this one allows the only B case.
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HBASE-7321) Simple Flush Snapshot
[
https://issues.apache.org/jira/browse/HBASE-7321?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13536753#comment-13536753
]
Ted Yu commented on HBASE-7321:
---
{code}
+ * (In the future other cancelable HReigon methods could eventulaly add a
{code}
typo: eventulaly
{code}
+ public void addRegionToSnapshot(SnapshotDescription desc,
+ ForeignExceptionCheckable failureMonitor) throws IOException {
{code}
Add javadoc for the parameters.
{code}
+ if (LOG.isDebugEnabled()) LOG.debug(Adding snapshot references for +
storeFiles
+ + hfiles);
{code}
Use curly braces for the above if statement. Here storeFiles.toString() is
called, would that reveal useful information ?
{code}
+// 2.3. create reference to this store file
{code}
I don't see 2.1 or 2.2 :-)
{code}
+ * framework. Its enter stage forces does nothing. Its leave stage then
builds the region
{code}
Can 'forces' be omitted ?
{code}
+public class FlushSnapshotSubprocedure extends Subprocedure {
{code}
Add annotation for audience and stability.
Looking at SnapshotSubprocedurePool.java:
{code}
long keepAlive = conf.getLong(
RegionServerSnapshotManager.SNAPSHOT_TIMEOUT_MILLIS_KEY,
...
executor = new ThreadPoolExecutor(1, threads, keepAlive, TimeUnit.SECONDS,
{code}
keepAlive is in milliseconds, right ? There seems to be inconsistency in units.
{code}
+ class RegionSnapshotTask implements CallableVoid {
{code}
The above class can be private, right ?
{code}
+case LOGROLL:
+ throw new IllegalArgumentException(Unimplememted snapshot type: +
snapshot.getType());
{code}
How about throwing UnsupportedOperationException ?
{code}
+ * Test create/using/deleting snapshots from the client
{code}
create - creating
{code}
+ * TODO This is essentially a clone of TestSnapshotFromClient. This is worth
refactoring this
...
+ * TODO This is essentially a clone of TestRestoreSnapshotFromClient. This is
worth refactoring
{code}
Are you going to create a separate JIRA for the above actions ?
Simple Flush Snapshot
-
Key: HBASE-7321
URL: https://issues.apache.org/jira/browse/HBASE-7321
Project: HBase
Issue Type: Sub-task
Reporter: Jonathan Hsieh
Assignee: Jonathan Hsieh
Attachments: hbase-7321.v2.patch, pre-hbase-7321.v2.patch
This snapshot style just issues a region flush and then snapshots the
region.
This is a simple implementation that gives the equivalent of copytable
consistency. While by most definitions of consistency if a client writes A
and then write B to different region servers, only neither, only A, or both
A+B writes should be present, this one allows the only B case.
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HBASE-7321) Simple Flush Snapshot
[ https://issues.apache.org/jira/browse/HBASE-7321?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13536770#comment-13536770 ] Jonathan Hsieh commented on HBASE-7321: --- Oops, forgot to post link to review board. https://reviews.apache.org/r/8391/ Simple Flush Snapshot - Key: HBASE-7321 URL: https://issues.apache.org/jira/browse/HBASE-7321 Project: HBase Issue Type: Sub-task Reporter: Jonathan Hsieh Assignee: Jonathan Hsieh Attachments: hbase-7321.v2.patch, pre-hbase-7321.v2.patch This snapshot style just issues a region flush and then snapshots the region. This is a simple implementation that gives the equivalent of copytable consistency. While by most definitions of consistency if a client writes A and then write B to different region servers, only neither, only A, or both A+B writes should be present, this one allows the only B case. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HBASE-7321) Simple Flush Snapshot
[ https://issues.apache.org/jira/browse/HBASE-7321?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13528779#comment-13528779 ] Jonathan Hsieh commented on HBASE-7321: --- This version consistently passes the jrub online testing/restoring attached to HBASE-7291. In manual testing on a single node, I've run PerformanceEvaluation with random write, while taking successfully taking multiple snapshots using this online snapshot implementation. Simple Flush Snapshot - Key: HBASE-7321 URL: https://issues.apache.org/jira/browse/HBASE-7321 Project: HBase Issue Type: Sub-task Reporter: Jonathan Hsieh This snapshot style just issues a region flush and then snapshots the region. This is a simple implementation that gives the equivalent of copytable consistency. While by most definitions of consistency if a client writes A and then write B to different region servers, only neither, only A, or both A+B writes should be present, this one allows the only B case. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators For more information on JIRA, see: http://www.atlassian.com/software/jira
