[jira] [Commented] (HBASE-8028) Append, Increment: Adding rollback support
[ https://issues.apache.org/jira/browse/HBASE-8028?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14484743#comment-14484743 ] Nick Dimiduk commented on HBASE-8028: - bq. Ran into this in triage. Seems important. Agreed. Any uptake here? > Append, Increment: Adding rollback support > -- > > Key: HBASE-8028 > URL: https://issues.apache.org/jira/browse/HBASE-8028 > Project: HBase > Issue Type: Bug > Components: regionserver >Affects Versions: 0.94.5 >Reporter: Himanshu Vashishtha >Assignee: Himanshu Vashishtha > Fix For: 1.1.0 > > Attachments: HBase-8028-v1.patch, HBase-8028-v2.patch, > HBase-8028-with-Increments-v1.patch, HBase-8028-with-Increments-v2.patch > > > In case there is an exception while doing the log-sync, the memstore is not > rollbacked, while the mvcc is _always_ forwarded to the writeentry created at > the beginning of the operation. This may lead to scanners seeing results > which are not synched to the fs. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (HBASE-8028) Append, Increment: Adding rollback support
[ https://issues.apache.org/jira/browse/HBASE-8028?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14126683#comment-14126683 ] Hadoop QA commented on HBASE-8028: -- {color:red}-1 overall{color}. Here are the results of testing the latest attachment http://issues.apache.org/jira/secure/attachment/12576251/HBase-8028-with-Increments-v2.patch against trunk revision . ATTACHMENT ID: 12576251 {color:green}+1 @author{color}. The patch does not contain any @author tags. {color:green}+1 tests included{color}. The patch appears to include 5 new or modified tests. {color:red}-1 patch{color}. The patch command could not apply the patch. Console output: https://builds.apache.org/job/PreCommit-HBASE-Build/10787//console This message is automatically generated. > Append, Increment: Adding rollback support > -- > > Key: HBASE-8028 > URL: https://issues.apache.org/jira/browse/HBASE-8028 > Project: HBase > Issue Type: Bug > Components: regionserver >Affects Versions: 0.94.5 >Reporter: Himanshu Vashishtha >Assignee: Himanshu Vashishtha > Fix For: 0.99.1 > > Attachments: HBase-8028-v1.patch, HBase-8028-v2.patch, > HBase-8028-with-Increments-v1.patch, HBase-8028-with-Increments-v2.patch > > > In case there is an exception while doing the log-sync, the memstore is not > rollbacked, while the mvcc is _always_ forwarded to the writeentry created at > the beginning of the operation. This may lead to scanners seeing results > which are not synched to the fs. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (HBASE-8028) Append, Increment: Adding rollback support
[ https://issues.apache.org/jira/browse/HBASE-8028?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14126681#comment-14126681 ] Enis Soztutar commented on HBASE-8028: -- Ran into this in triage. Seems important. > Append, Increment: Adding rollback support > -- > > Key: HBASE-8028 > URL: https://issues.apache.org/jira/browse/HBASE-8028 > Project: HBase > Issue Type: Bug > Components: regionserver >Affects Versions: 0.94.5 >Reporter: Himanshu Vashishtha >Assignee: Himanshu Vashishtha > Fix For: 0.99.1 > > Attachments: HBase-8028-v1.patch, HBase-8028-v2.patch, > HBase-8028-with-Increments-v1.patch, HBase-8028-with-Increments-v2.patch > > > In case there is an exception while doing the log-sync, the memstore is not > rollbacked, while the mvcc is _always_ forwarded to the writeentry created at > the beginning of the operation. This may lead to scanners seeing results > which are not synched to the fs. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (HBASE-8028) Append, Increment: Adding rollback support
[ https://issues.apache.org/jira/browse/HBASE-8028?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13849702#comment-13849702 ] Hadoop QA commented on HBASE-8028: -- {color:red}-1 overall{color}. Here are the results of testing the latest attachment http://issues.apache.org/jira/secure/attachment/12576251/HBase-8028-with-Increments-v2.patch against trunk revision . {color:green}+1 @author{color}. The patch does not contain any @author tags. {color:green}+1 tests included{color}. The patch appears to include 5 new or modified tests. {color:red}-1 patch{color}. The patch command could not apply the patch. Console output: https://builds.apache.org/job/PreCommit-HBASE-Build/8187//console This message is automatically generated. > Append, Increment: Adding rollback support > -- > > Key: HBASE-8028 > URL: https://issues.apache.org/jira/browse/HBASE-8028 > Project: HBase > Issue Type: Bug > Components: regionserver >Affects Versions: 0.94.5 >Reporter: Himanshu Vashishtha >Assignee: Himanshu Vashishtha > Fix For: 0.99.0 > > Attachments: HBase-8028-v1.patch, HBase-8028-v2.patch, > HBase-8028-with-Increments-v1.patch, HBase-8028-with-Increments-v2.patch > > > In case there is an exception while doing the log-sync, the memstore is not > rollbacked, while the mvcc is _always_ forwarded to the writeentry created at > the beginning of the operation. This may lead to scanners seeing results > which are not synched to the fs. -- This message was sent by Atlassian JIRA (v6.1.4#6159)
[jira] [Commented] (HBASE-8028) Append, Increment: Adding rollback support
[ https://issues.apache.org/jira/browse/HBASE-8028?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13752721#comment-13752721 ] stack commented on HBASE-8028: -- Should we just do as [~lhofhansl] suggested up top and make Append/Increment just-like-the-others? > Append, Increment: Adding rollback support > -- > > Key: HBASE-8028 > URL: https://issues.apache.org/jira/browse/HBASE-8028 > Project: HBase > Issue Type: Bug > Components: regionserver >Affects Versions: 0.94.5 >Reporter: Himanshu Vashishtha >Assignee: Himanshu Vashishtha > Fix For: 0.96.0 > > Attachments: HBase-8028-v1.patch, HBase-8028-v2.patch, > HBase-8028-with-Increments-v1.patch, HBase-8028-with-Increments-v2.patch > > > In case there is an exception while doing the log-sync, the memstore is not > rollbacked, while the mvcc is _always_ forwarded to the writeentry created at > the beginning of the operation. This may lead to scanners seeing results > which are not synched to the fs. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HBASE-8028) Append, Increment: Adding rollback support
[ https://issues.apache.org/jira/browse/HBASE-8028?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13620665#comment-13620665 ] Himanshu Vashishtha commented on HBASE-8028: [~jxiang] Very good points. As Ted said, readpoint is not changed. The readpoint which is being used for the rollback is not committed yet, so none of the scanner is reading it at this point. For the snapshots, the patch needs to be altered a bit. Basically, if the kv is present in snapshot then it will not be in memstore kv, and vice versa. So, while deleting the new kv, if it is deleted from snapshot, then the old kv should be only added to the snapshot. Similarly, if the new kv is deleted from memstore's kv, then the old kv should be added to the memstore's kv. I'll upload a revised patch. > Append, Increment: Adding rollback support > -- > > Key: HBASE-8028 > URL: https://issues.apache.org/jira/browse/HBASE-8028 > Project: HBase > Issue Type: Bug > Components: regionserver >Affects Versions: 0.94.5 >Reporter: Himanshu Vashishtha >Assignee: Himanshu Vashishtha > Fix For: 0.95.0 > > Attachments: HBase-8028-v1.patch, HBase-8028-v2.patch, > HBase-8028-with-Increments-v1.patch, HBase-8028-with-Increments-v2.patch > > > In case there is an exception while doing the log-sync, the memstore is not > rollbacked, while the mvcc is _always_ forwarded to the writeentry created at > the beginning of the operation. This may lead to scanners seeing results > which are not synched to the fs. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HBASE-8028) Append, Increment: Adding rollback support
[ https://issues.apache.org/jira/browse/HBASE-8028?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13620409#comment-13620409 ] Ted Yu commented on HBASE-8028: --- bq. Will this change the readpoint? {code} private long internalAdd(final KeyValue toAdd) { long s = heapSizeChange(toAdd, this.kvset.add(toAdd)); timeRangeTracker.includeTimestamp(toAdd); this.size.addAndGet(s); return s; } {code} Looking at the code, I don't think readpoint is affected. > Append, Increment: Adding rollback support > -- > > Key: HBASE-8028 > URL: https://issues.apache.org/jira/browse/HBASE-8028 > Project: HBase > Issue Type: Bug > Components: regionserver >Affects Versions: 0.94.5 >Reporter: Himanshu Vashishtha >Assignee: Himanshu Vashishtha > Fix For: 0.95.0 > > Attachments: HBase-8028-v1.patch, HBase-8028-v2.patch, > HBase-8028-with-Increments-v1.patch, HBase-8028-with-Increments-v2.patch > > > In case there is an exception while doing the log-sync, the memstore is not > rollbacked, while the mvcc is _always_ forwarded to the writeentry created at > the beginning of the operation. This may lead to scanners seeing results > which are not synched to the fs. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HBASE-8028) Append, Increment: Adding rollback support
[ https://issues.apache.org/jira/browse/HBASE-8028?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13619104#comment-13619104 ] Jimmy Xiang commented on HBASE-8028: In MemStore#rollbackUpsert, internalAdd is used to add back the original value. Will this change the readpoint? Should we add it to snapshot as well? > Append, Increment: Adding rollback support > -- > > Key: HBASE-8028 > URL: https://issues.apache.org/jira/browse/HBASE-8028 > Project: HBase > Issue Type: Bug > Components: regionserver >Affects Versions: 0.94.5 >Reporter: Himanshu Vashishtha >Assignee: Himanshu Vashishtha > Fix For: 0.95.0 > > Attachments: HBase-8028-v1.patch, HBase-8028-v2.patch, > HBase-8028-with-Increments-v1.patch, HBase-8028-with-Increments-v2.patch > > > In case there is an exception while doing the log-sync, the memstore is not > rollbacked, while the mvcc is _always_ forwarded to the writeentry created at > the beginning of the operation. This may lead to scanners seeing results > which are not synched to the fs. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HBASE-8028) Append, Increment: Adding rollback support
[ https://issues.apache.org/jira/browse/HBASE-8028?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13618203#comment-13618203 ] Himanshu Vashishtha commented on HBASE-8028: org.apache.hadoop.hbase.mapreduce.TestHFileOutputFormat passes on local. > Append, Increment: Adding rollback support > -- > > Key: HBASE-8028 > URL: https://issues.apache.org/jira/browse/HBASE-8028 > Project: HBase > Issue Type: Bug > Components: regionserver >Affects Versions: 0.94.5 >Reporter: Himanshu Vashishtha >Assignee: Himanshu Vashishtha > Fix For: 0.95.0 > > Attachments: HBase-8028-v1.patch, HBase-8028-v2.patch, > HBase-8028-with-Increments-v1.patch, HBase-8028-with-Increments-v2.patch > > > In case there is an exception while doing the log-sync, the memstore is not > rollbacked, while the mvcc is _always_ forwarded to the writeentry created at > the beginning of the operation. This may lead to scanners seeing results > which are not synched to the fs. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HBASE-8028) Append, Increment: Adding rollback support
[ https://issues.apache.org/jira/browse/HBASE-8028?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13618176#comment-13618176 ] Hadoop QA commented on HBASE-8028: -- {color:red}-1 overall{color}. Here are the results of testing the latest attachment http://issues.apache.org/jira/secure/attachment/12576251/HBase-8028-with-Increments-v2.patch against trunk revision . {color:green}+1 @author{color}. The patch does not contain any @author tags. {color:green}+1 tests included{color}. The patch appears to include 5 new or modified tests. {color:green}+1 hadoop2.0{color}. The patch compiles against the hadoop 2.0 profile. {color:green}+1 javadoc{color}. The javadoc tool did not generate any warning messages. {color:green}+1 javac{color}. The applied patch does not increase the total number of javac compiler warnings. {color:green}+1 findbugs{color}. The patch does not introduce any new Findbugs (version 1.3.9) warnings. {color:green}+1 release audit{color}. The applied patch does not increase the total number of release audit warnings. {color:green}+1 lineLengths{color}. The patch does not introduce lines longer than 100 {color:green}+1 site{color}. The mvn site goal succeeds with this patch. {color:red}-1 core tests{color}. The patch failed these unit tests: org.apache.hadoop.hbase.mapreduce.TestHFileOutputFormat Test results: https://builds.apache.org/job/PreCommit-HBASE-Build/5068//testReport/ Findbugs warnings: https://builds.apache.org/job/PreCommit-HBASE-Build/5068//artifact/trunk/patchprocess/newPatchFindbugsWarningshbase-prefix-tree.html Findbugs warnings: https://builds.apache.org/job/PreCommit-HBASE-Build/5068//artifact/trunk/patchprocess/newPatchFindbugsWarningshbase-client.html Findbugs warnings: https://builds.apache.org/job/PreCommit-HBASE-Build/5068//artifact/trunk/patchprocess/newPatchFindbugsWarningshbase-common.html Findbugs warnings: https://builds.apache.org/job/PreCommit-HBASE-Build/5068//artifact/trunk/patchprocess/newPatchFindbugsWarningshbase-protocol.html Findbugs warnings: https://builds.apache.org/job/PreCommit-HBASE-Build/5068//artifact/trunk/patchprocess/newPatchFindbugsWarningshbase-server.html Findbugs warnings: https://builds.apache.org/job/PreCommit-HBASE-Build/5068//artifact/trunk/patchprocess/newPatchFindbugsWarningshbase-hadoop1-compat.html Findbugs warnings: https://builds.apache.org/job/PreCommit-HBASE-Build/5068//artifact/trunk/patchprocess/newPatchFindbugsWarningshbase-examples.html Findbugs warnings: https://builds.apache.org/job/PreCommit-HBASE-Build/5068//artifact/trunk/patchprocess/newPatchFindbugsWarningshbase-hadoop-compat.html Console output: https://builds.apache.org/job/PreCommit-HBASE-Build/5068//console This message is automatically generated. > Append, Increment: Adding rollback support > -- > > Key: HBASE-8028 > URL: https://issues.apache.org/jira/browse/HBASE-8028 > Project: HBase > Issue Type: Bug > Components: regionserver >Affects Versions: 0.94.5 >Reporter: Himanshu Vashishtha >Assignee: Himanshu Vashishtha > Fix For: 0.95.0 > > Attachments: HBase-8028-v1.patch, HBase-8028-v2.patch, > HBase-8028-with-Increments-v1.patch, HBase-8028-with-Increments-v2.patch > > > In case there is an exception while doing the log-sync, the memstore is not > rollbacked, while the mvcc is _always_ forwarded to the writeentry created at > the beginning of the operation. This may lead to scanners seeing results > which are not synched to the fs. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HBASE-8028) Append, Increment: Adding rollback support
[ https://issues.apache.org/jira/browse/HBASE-8028?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13618165#comment-13618165 ] Himanshu Vashishtha commented on HBASE-8028: Jimmy, Ted: Thanks for the review guys. I added a test class which has methods for testing rollback feature for Append and Increment. For CFs with maxVersion = 1, the current codebase uses upserts. This patch adds a roll back feature for those upsert changes. For CF with maxversion > 1, it uses the existing rollback api. bq. Should rollbackUpsert() return boolean to indicate whether the rollback has succeeded ? It is along the line of the existing rollback api. Incorporate the changes suggested by Ted. > Append, Increment: Adding rollback support > -- > > Key: HBASE-8028 > URL: https://issues.apache.org/jira/browse/HBASE-8028 > Project: HBase > Issue Type: Bug > Components: regionserver >Affects Versions: 0.94.5 >Reporter: Himanshu Vashishtha >Assignee: Himanshu Vashishtha > Fix For: 0.95.0 > > Attachments: HBase-8028-v1.patch, HBase-8028-v2.patch, > HBase-8028-with-Increments-v1.patch, HBase-8028-with-Increments-v2.patch > > > In case there is an exception while doing the log-sync, the memstore is not > rollbacked, while the mvcc is _always_ forwarded to the writeentry created at > the beginning of the operation. This may lead to scanners seeing results > which are not synched to the fs. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HBASE-8028) Append, Increment: Adding rollback support
[ https://issues.apache.org/jira/browse/HBASE-8028?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13617721#comment-13617721 ] Jimmy Xiang commented on HBASE-8028: When maxversion = 1, will the normal rollback work? Do we have to do something different? It will be great if you can add a testcase to demonstrate the problem mentioned in the description, and it's fixed with the patch. > Append, Increment: Adding rollback support > -- > > Key: HBASE-8028 > URL: https://issues.apache.org/jira/browse/HBASE-8028 > Project: HBase > Issue Type: Bug > Components: regionserver >Affects Versions: 0.94.5 >Reporter: Himanshu Vashishtha >Assignee: Himanshu Vashishtha > Fix For: 0.95.0 > > Attachments: HBase-8028-v1.patch, HBase-8028-v2.patch, > HBase-8028-with-Increments-v1.patch > > > In case there is an exception while doing the log-sync, the memstore is not > rollbacked, while the mvcc is _always_ forwarded to the writeentry created at > the beginning of the operation. This may lead to scanners seeing results > which are not synched to the fs. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HBASE-8028) Append, Increment: Adding rollback support
[ https://issues.apache.org/jira/browse/HBASE-8028?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13609482#comment-13609482 ] Ted Yu commented on HBASE-8028: --- {code} +boolean isSyncToWalSucessfull = false; {code} Typo: Sucess. {code} + if(!isSyncToWalSucessfull){ {code} nit: space between if and (, ) and {. {code} + public void rollbackUpsert(List newKVsToRollback, + List oldKVsToReplay, final long mvccStartPoint) { {code} Should rollbackUpsert() return boolean to indicate whether the rollback has succeeded ? {code} + * Rollbacks a upsert transaction in the memstore (used in case of append/increment). {code} 'Rollbacks a upsert' -> 'Rolls back an upsert' {code} + * Iterates over the given set and delete the kvs which matches with the passed kv and + * mvccReadpoint. Set changeSize to true if the memstore size is also needs to change. {code} should read: {code} + * Iterates over the given set and deletes the kvs which match the passed kv and + * mvccReadpoint. Sets changeSize to true if the memstore size also needs to change. {code} > Append, Increment: Adding rollback support > -- > > Key: HBASE-8028 > URL: https://issues.apache.org/jira/browse/HBASE-8028 > Project: HBase > Issue Type: Bug > Components: regionserver >Affects Versions: 0.94.5 >Reporter: Himanshu Vashishtha >Assignee: Himanshu Vashishtha > Fix For: 0.95.0 > > Attachments: HBase-8028-v1.patch, HBase-8028-v2.patch, > HBase-8028-with-Increments-v1.patch > > > In case there is an exception while doing the log-sync, the memstore is not > rollbacked, while the mvcc is _always_ forwarded to the writeentry created at > the beginning of the operation. This may lead to scanners seeing results > which are not synched to the fs. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HBASE-8028) Append, Increment: Adding rollback support
[ https://issues.apache.org/jira/browse/HBASE-8028?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13609258#comment-13609258 ] Himanshu Vashishtha commented on HBASE-8028: Ping... :) > Append, Increment: Adding rollback support > -- > > Key: HBASE-8028 > URL: https://issues.apache.org/jira/browse/HBASE-8028 > Project: HBase > Issue Type: Bug > Components: regionserver >Affects Versions: 0.94.5 >Reporter: Himanshu Vashishtha >Assignee: Himanshu Vashishtha > Fix For: 0.95.0 > > Attachments: HBase-8028-v1.patch, HBase-8028-v2.patch, > HBase-8028-with-Increments-v1.patch > > > In case there is an exception while doing the log-sync, the memstore is not > rollbacked, while the mvcc is _always_ forwarded to the writeentry created at > the beginning of the operation. This may lead to scanners seeing results > which are not synched to the fs. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HBASE-8028) Append, Increment: Adding rollback support
[ https://issues.apache.org/jira/browse/HBASE-8028?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13601290#comment-13601290 ] Himanshu Vashishtha commented on HBASE-8028: Please let me know if you have any other comments/suggestions. > Append, Increment: Adding rollback support > -- > > Key: HBASE-8028 > URL: https://issues.apache.org/jira/browse/HBASE-8028 > Project: HBase > Issue Type: Bug > Components: regionserver >Affects Versions: 0.94.5 >Reporter: Himanshu Vashishtha >Assignee: Himanshu Vashishtha > Fix For: 0.95.0 > > Attachments: HBase-8028-v1.patch, HBase-8028-v2.patch, > HBase-8028-with-Increments-v1.patch > > > In case there is an exception while doing the log-sync, the memstore is not > rollbacked, while the mvcc is _always_ forwarded to the writeentry created at > the beginning of the operation. This may lead to scanners seeing results > which are not synched to the fs. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators For more information on JIRA, see: http://www.atlassian.com/software/jira