[jira] [Commented] (HBASE-5121) MajorCompaction may affect scan's correctness
[ https://issues.apache.org/jira/browse/HBASE-5121?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13179390#comment-13179390 ] Zhihong Yu commented on HBASE-5121: --- Should null assignment for lastTop be after the debug log ? > MajorCompaction may affect scan's correctness > - > > Key: HBASE-5121 > URL: https://issues.apache.org/jira/browse/HBASE-5121 > Project: HBase > Issue Type: Bug > Components: regionserver >Reporter: chunhui shen > Attachments: hbase-5121.patch > > > In our test, there are two families' keyvalue for one row. > But we could find a infrequent problem when doing scan's next if > majorCompaction happens concurrently. > In the client's two continuous doing scan.next(): > 1.First time, scan's next returns the result where family A is null. > 2.Second time, scan's next returns the result where family B is null. > The two next()'s result have the same row. > If there are more families, I think the scenario will be more strange... > We find the reason is that storescanner.peek() is changed after > majorCompaction if there are delete type KeyValue. > This change causes the PriorityQueue of RegionScanner's heap > is not sure to be sorted. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HBASE-5121) MajorCompaction may affect scan's correctness
[ https://issues.apache.org/jira/browse/HBASE-5121?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13179666#comment-13179666 ] Lars Hofhansl commented on HBASE-5121: -- I see the patch is against trunk. Does this happen in trunk only? It might have to do with the various optimization for scanning that the FB folks put in (if I recall FB rarely does major compactions). Maybe Mikhail could have a look at this too. > MajorCompaction may affect scan's correctness > - > > Key: HBASE-5121 > URL: https://issues.apache.org/jira/browse/HBASE-5121 > Project: HBase > Issue Type: Bug > Components: regionserver >Reporter: chunhui shen > Attachments: hbase-5121.patch > > > In our test, there are two families' keyvalue for one row. > But we could find a infrequent problem when doing scan's next if > majorCompaction happens concurrently. > In the client's two continuous doing scan.next(): > 1.First time, scan's next returns the result where family A is null. > 2.Second time, scan's next returns the result where family B is null. > The two next()'s result have the same row. > If there are more families, I think the scenario will be more strange... > We find the reason is that storescanner.peek() is changed after > majorCompaction if there are delete type KeyValue. > This change causes the PriorityQueue of RegionScanner's heap > is not sure to be sorted. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HBASE-5121) MajorCompaction may affect scan's correctness
[ https://issues.apache.org/jira/browse/HBASE-5121?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13179743#comment-13179743 ] Todd Lipcon commented on HBASE-5121: No test for this? Can we get a functional test that can at least be run from the command line to trigger it? Nits: the javadoc on the constructors are unnecessary (they don't give any extra information) Using the exception for control flow here looks dirty IMO. > MajorCompaction may affect scan's correctness > - > > Key: HBASE-5121 > URL: https://issues.apache.org/jira/browse/HBASE-5121 > Project: HBase > Issue Type: Bug > Components: regionserver >Reporter: chunhui shen > Attachments: hbase-5121.patch > > > In our test, there are two families' keyvalue for one row. > But we could find a infrequent problem when doing scan's next if > majorCompaction happens concurrently. > In the client's two continuous doing scan.next(): > 1.First time, scan's next returns the result where family A is null. > 2.Second time, scan's next returns the result where family B is null. > The two next()'s result have the same row. > If there are more families, I think the scenario will be more strange... > We find the reason is that storescanner.peek() is changed after > majorCompaction if there are delete type KeyValue. > This change causes the PriorityQueue of RegionScanner's heap > is not sure to be sorted. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HBASE-5121) MajorCompaction may affect scan's correctness
[ https://issues.apache.org/jira/browse/HBASE-5121?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13180115#comment-13180115 ] chunhui shen commented on HBASE-5121: - @Ted I'm sorry for the mistake when making patch for trunk. > MajorCompaction may affect scan's correctness > - > > Key: HBASE-5121 > URL: https://issues.apache.org/jira/browse/HBASE-5121 > Project: HBase > Issue Type: Bug > Components: regionserver >Reporter: chunhui shen >Assignee: chunhui shen > Attachments: hbase-5121.patch > > > In our test, there are two families' keyvalue for one row. > But we could find a infrequent problem when doing scan's next if > majorCompaction happens concurrently. > In the client's two continuous doing scan.next(): > 1.First time, scan's next returns the result where family A is null. > 2.Second time, scan's next returns the result where family B is null. > The two next()'s result have the same row. > If there are more families, I think the scenario will be more strange... > We find the reason is that storescanner.peek() is changed after > majorCompaction if there are delete type KeyValue. > This change causes the PriorityQueue of RegionScanner's heap > is not sure to be sorted. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HBASE-5121) MajorCompaction may affect scan's correctness
[ https://issues.apache.org/jira/browse/HBASE-5121?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13180116#comment-13180116 ] chunhui shen commented on HBASE-5121: - @Lars Our test environment uses the 0.90 version,but I think it will also happens in 0.92 and trunk too. > MajorCompaction may affect scan's correctness > - > > Key: HBASE-5121 > URL: https://issues.apache.org/jira/browse/HBASE-5121 > Project: HBase > Issue Type: Bug > Components: regionserver >Reporter: chunhui shen >Assignee: chunhui shen > Attachments: hbase-5121.patch > > > In our test, there are two families' keyvalue for one row. > But we could find a infrequent problem when doing scan's next if > majorCompaction happens concurrently. > In the client's two continuous doing scan.next(): > 1.First time, scan's next returns the result where family A is null. > 2.Second time, scan's next returns the result where family B is null. > The two next()'s result have the same row. > If there are more families, I think the scenario will be more strange... > We find the reason is that storescanner.peek() is changed after > majorCompaction if there are delete type KeyValue. > This change causes the PriorityQueue of RegionScanner's heap > is not sure to be sorted. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HBASE-5121) MajorCompaction may affect scan's correctness
[ https://issues.apache.org/jira/browse/HBASE-5121?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13180118#comment-13180118 ] chunhui shen commented on HBASE-5121: - @Todd I will append tests for this patch later. > MajorCompaction may affect scan's correctness > - > > Key: HBASE-5121 > URL: https://issues.apache.org/jira/browse/HBASE-5121 > Project: HBase > Issue Type: Bug > Components: regionserver >Reporter: chunhui shen >Assignee: chunhui shen > Attachments: hbase-5121.patch > > > In our test, there are two families' keyvalue for one row. > But we could find a infrequent problem when doing scan's next if > majorCompaction happens concurrently. > In the client's two continuous doing scan.next(): > 1.First time, scan's next returns the result where family A is null. > 2.Second time, scan's next returns the result where family B is null. > The two next()'s result have the same row. > If there are more families, I think the scenario will be more strange... > We find the reason is that storescanner.peek() is changed after > majorCompaction if there are delete type KeyValue. > This change causes the PriorityQueue of RegionScanner's heap > is not sure to be sorted. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HBASE-5121) MajorCompaction may affect scan's correctness
[ https://issues.apache.org/jira/browse/HBASE-5121?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13180269#comment-13180269 ] chunhui shen commented on HBASE-5121: - I add a unit test case in hbase-5121-testcase.patch for this issue. And the solution is hbase-5121v2.patch > MajorCompaction may affect scan's correctness > - > > Key: HBASE-5121 > URL: https://issues.apache.org/jira/browse/HBASE-5121 > Project: HBase > Issue Type: Bug > Components: regionserver >Reporter: chunhui shen >Assignee: chunhui shen > Attachments: hbase-5121-testcase.patch, hbase-5121.patch, > hbase-5121v2.patch > > > In our test, there are two families' keyvalue for one row. > But we could find a infrequent problem when doing scan's next if > majorCompaction happens concurrently. > In the client's two continuous doing scan.next(): > 1.First time, scan's next returns the result where family A is null. > 2.Second time, scan's next returns the result where family B is null. > The two next()'s result have the same row. > If there are more families, I think the scenario will be more strange... > We find the reason is that storescanner.peek() is changed after > majorCompaction if there are delete type KeyValue. > This change causes the PriorityQueue of RegionScanner's heap > is not sure to be sorted. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HBASE-5121) MajorCompaction may affect scan's correctness
[
https://issues.apache.org/jira/browse/HBASE-5121?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13180279#comment-13180279
]
Zhihong Yu commented on HBASE-5121:
---
>From https://builds.apache.org/job/PreCommit-HBASE-Build/672/console:
{code}
HBASE-5121 patch is being downloaded at Thu Jan 5 10:51:42 UTC 2012 from
http://issues.apache.org/jira/secure/attachment/12509522/hbase-5121-testcase.patch
{code}
In the future, please attach the patch with solution last so that Hadoop QA can
pick up.
> MajorCompaction may affect scan's correctness
> -
>
> Key: HBASE-5121
> URL: https://issues.apache.org/jira/browse/HBASE-5121
> Project: HBase
> Issue Type: Bug
> Components: regionserver
>Reporter: chunhui shen
>Assignee: chunhui shen
> Attachments: hbase-5121-testcase.patch, hbase-5121.patch,
> hbase-5121v2.patch
>
>
> In our test, there are two families' keyvalue for one row.
> But we could find a infrequent problem when doing scan's next if
> majorCompaction happens concurrently.
> In the client's two continuous doing scan.next():
> 1.First time, scan's next returns the result where family A is null.
> 2.Second time, scan's next returns the result where family B is null.
> The two next()'s result have the same row.
> If there are more families, I think the scenario will be more strange...
> We find the reason is that storescanner.peek() is changed after
> majorCompaction if there are delete type KeyValue.
> This change causes the PriorityQueue of RegionScanner's heap
> is not sure to be sorted.
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators:
https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HBASE-5121) MajorCompaction may affect scan's correctness
[ https://issues.apache.org/jira/browse/HBASE-5121?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13180280#comment-13180280 ] Hadoop QA commented on HBASE-5121: -- -1 overall. Here are the results of testing the latest attachment http://issues.apache.org/jira/secure/attachment/12509522/hbase-5121-testcase.patch against trunk revision . +1 @author. The patch does not contain any @author tags. +1 tests included. The patch appears to include 3 new or modified tests. -1 javadoc. The javadoc tool appears to have generated -151 warning messages. +1 javac. The applied patch does not increase the total number of javac compiler warnings. -1 findbugs. The patch appears to introduce 79 new Findbugs (version 1.3.9) warnings. +1 release audit. The applied patch does not increase the total number of release audit warnings. -1 core tests. The patch failed these unit tests: org.apache.hadoop.hbase.regionserver.TestScanner Test results: https://builds.apache.org/job/PreCommit-HBASE-Build/672//testReport/ Findbugs warnings: https://builds.apache.org/job/PreCommit-HBASE-Build/672//artifact/trunk/patchprocess/newPatchFindbugsWarnings.html Console output: https://builds.apache.org/job/PreCommit-HBASE-Build/672//console This message is automatically generated. > MajorCompaction may affect scan's correctness > - > > Key: HBASE-5121 > URL: https://issues.apache.org/jira/browse/HBASE-5121 > Project: HBase > Issue Type: Bug > Components: regionserver >Reporter: chunhui shen >Assignee: chunhui shen > Attachments: hbase-5121-testcase.patch, hbase-5121.patch, > hbase-5121v2.patch > > > In our test, there are two families' keyvalue for one row. > But we could find a infrequent problem when doing scan's next if > majorCompaction happens concurrently. > In the client's two continuous doing scan.next(): > 1.First time, scan's next returns the result where family A is null. > 2.Second time, scan's next returns the result where family B is null. > The two next()'s result have the same row. > If there are more families, I think the scenario will be more strange... > We find the reason is that storescanner.peek() is changed after > majorCompaction if there are delete type KeyValue. > This change causes the PriorityQueue of RegionScanner's heap > is not sure to be sorted. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HBASE-5121) MajorCompaction may affect scan's correctness
[
https://issues.apache.org/jira/browse/HBASE-5121?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13180281#comment-13180281
]
Zhihong Yu commented on HBASE-5121:
---
The above test failure shows the bug, that's great.
About the patch itself, some minor comments:
Please add Apache license to PeekChangedException header.
{code}
++ this.lastTop.toString() + ",and after = "
++ (this.heap.peek() == null ? "null" :
this.heap.peek().toString()));
{code}
The null check on the second line is not needed for string concatenation.
{code}
+ mayContainsMoreRows = currentAsInternal.next(result, limit);
{code}
Since this part of the code is changed, please name the variable
mayContainMoreRows.
Please combine the two patches in one next time they're uploaded.
Great work, Chunhui.
> MajorCompaction may affect scan's correctness
> -
>
> Key: HBASE-5121
> URL: https://issues.apache.org/jira/browse/HBASE-5121
> Project: HBase
> Issue Type: Bug
> Components: regionserver
>Affects Versions: 0.90.4
>Reporter: chunhui shen
>Assignee: chunhui shen
> Fix For: 0.92.0, 0.94.0, 0.90.6
>
> Attachments: hbase-5121-testcase.patch, hbase-5121.patch,
> hbase-5121v2.patch
>
>
> In our test, there are two families' keyvalue for one row.
> But we could find a infrequent problem when doing scan's next if
> majorCompaction happens concurrently.
> In the client's two continuous doing scan.next():
> 1.First time, scan's next returns the result where family A is null.
> 2.Second time, scan's next returns the result where family B is null.
> The two next()'s result have the same row.
> If there are more families, I think the scenario will be more strange...
> We find the reason is that storescanner.peek() is changed after
> majorCompaction if there are delete type KeyValue.
> This change causes the PriorityQueue of RegionScanner's heap
> is not sure to be sorted.
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators:
https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HBASE-5121) MajorCompaction may affect scan's correctness
[
https://issues.apache.org/jira/browse/HBASE-5121?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13180284#comment-13180284
]
Zhihong Yu commented on HBASE-5121:
---
For the new test:
{code}
+} catch (Exception e) {
+ LOG.error("Failed", e);
+ throw e;
{code}
Is the above needed ? The exception, if thrown, would fail the test anyway.
> MajorCompaction may affect scan's correctness
> -
>
> Key: HBASE-5121
> URL: https://issues.apache.org/jira/browse/HBASE-5121
> Project: HBase
> Issue Type: Bug
> Components: regionserver
>Affects Versions: 0.90.4
>Reporter: chunhui shen
>Assignee: chunhui shen
> Fix For: 0.92.0, 0.94.0, 0.90.6
>
> Attachments: hbase-5121-testcase.patch, hbase-5121.patch,
> hbase-5121v2.patch
>
>
> In our test, there are two families' keyvalue for one row.
> But we could find a infrequent problem when doing scan's next if
> majorCompaction happens concurrently.
> In the client's two continuous doing scan.next():
> 1.First time, scan's next returns the result where family A is null.
> 2.Second time, scan's next returns the result where family B is null.
> The two next()'s result have the same row.
> If there are more families, I think the scenario will be more strange...
> We find the reason is that storescanner.peek() is changed after
> majorCompaction if there are delete type KeyValue.
> This change causes the PriorityQueue of RegionScanner's heap
> is not sure to be sorted.
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators:
https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HBASE-5121) MajorCompaction may affect scan's correctness
[ https://issues.apache.org/jira/browse/HBASE-5121?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13180288#comment-13180288 ] chunhui shen commented on HBASE-5121: - @Ted Thanks for the advice. I will submit a new patch later. > MajorCompaction may affect scan's correctness > - > > Key: HBASE-5121 > URL: https://issues.apache.org/jira/browse/HBASE-5121 > Project: HBase > Issue Type: Bug > Components: regionserver >Affects Versions: 0.90.4 >Reporter: chunhui shen >Assignee: chunhui shen > Fix For: 0.92.0, 0.94.0, 0.90.6 > > Attachments: hbase-5121-testcase.patch, hbase-5121.patch, > hbase-5121v2.patch > > > In our test, there are two families' keyvalue for one row. > But we could find a infrequent problem when doing scan's next if > majorCompaction happens concurrently. > In the client's two continuous doing scan.next(): > 1.First time, scan's next returns the result where family A is null. > 2.Second time, scan's next returns the result where family B is null. > The two next()'s result have the same row. > If there are more families, I think the scenario will be more strange... > We find the reason is that storescanner.peek() is changed after > majorCompaction if there are delete type KeyValue. > This change causes the PriorityQueue of RegionScanner's heap > is not sure to be sorted. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HBASE-5121) MajorCompaction may affect scan's correctness
[ https://issues.apache.org/jira/browse/HBASE-5121?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13180631#comment-13180631 ] Lars Hofhansl commented on HBASE-5121: -- bq. We find the reason is that storescanner.peek() is changed after majorCompaction if there are delete type KeyValue. Do we know why/how that happens? Just want to make sure that we're not just pasting over symptoms with this. > MajorCompaction may affect scan's correctness > - > > Key: HBASE-5121 > URL: https://issues.apache.org/jira/browse/HBASE-5121 > Project: HBase > Issue Type: Bug > Components: regionserver >Affects Versions: 0.90.4 >Reporter: chunhui shen >Assignee: chunhui shen > Fix For: 0.92.0, 0.94.0, 0.90.6 > > Attachments: 5121-trunk-combined.txt, hbase-5121-testcase.patch, > hbase-5121.patch, hbase-5121v2.patch > > > In our test, there are two families' keyvalue for one row. > But we could find a infrequent problem when doing scan's next if > majorCompaction happens concurrently. > In the client's two continuous doing scan.next(): > 1.First time, scan's next returns the result where family A is null. > 2.Second time, scan's next returns the result where family B is null. > The two next()'s result have the same row. > If there are more families, I think the scenario will be more strange... > We find the reason is that storescanner.peek() is changed after > majorCompaction if there are delete type KeyValue. > This change causes the PriorityQueue of RegionScanner's heap > is not sure to be sorted. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HBASE-5121) MajorCompaction may affect scan's correctness
[ https://issues.apache.org/jira/browse/HBASE-5121?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13180662#comment-13180662 ] Hadoop QA commented on HBASE-5121: -- -1 overall. Here are the results of testing the latest attachment http://issues.apache.org/jira/secure/attachment/12509573/5121-trunk-combined.txt against trunk revision . +1 @author. The patch does not contain any @author tags. +1 tests included. The patch appears to include 3 new or modified tests. -1 javadoc. The javadoc tool appears to have generated -151 warning messages. +1 javac. The applied patch does not increase the total number of javac compiler warnings. -1 findbugs. The patch appears to introduce 79 new Findbugs (version 1.3.9) warnings. +1 release audit. The applied patch does not increase the total number of release audit warnings. -1 core tests. The patch failed these unit tests: org.apache.hadoop.hbase.replication.TestReplicationPeer org.apache.hadoop.hbase.replication.TestReplication org.apache.hadoop.hbase.mapreduce.TestImportTsv org.apache.hadoop.hbase.mapred.TestTableMapReduce org.apache.hadoop.hbase.mapreduce.TestHFileOutputFormat Test results: https://builds.apache.org/job/PreCommit-HBASE-Build/673//testReport/ Findbugs warnings: https://builds.apache.org/job/PreCommit-HBASE-Build/673//artifact/trunk/patchprocess/newPatchFindbugsWarnings.html Console output: https://builds.apache.org/job/PreCommit-HBASE-Build/673//console This message is automatically generated. > MajorCompaction may affect scan's correctness > - > > Key: HBASE-5121 > URL: https://issues.apache.org/jira/browse/HBASE-5121 > Project: HBase > Issue Type: Bug > Components: regionserver >Affects Versions: 0.90.4 >Reporter: chunhui shen >Assignee: chunhui shen > Fix For: 0.92.0, 0.94.0, 0.90.6 > > Attachments: 5121-trunk-combined.txt, hbase-5121-testcase.patch, > hbase-5121.patch, hbase-5121v2.patch > > > In our test, there are two families' keyvalue for one row. > But we could find a infrequent problem when doing scan's next if > majorCompaction happens concurrently. > In the client's two continuous doing scan.next(): > 1.First time, scan's next returns the result where family A is null. > 2.Second time, scan's next returns the result where family B is null. > The two next()'s result have the same row. > If there are more families, I think the scenario will be more strange... > We find the reason is that storescanner.peek() is changed after > majorCompaction if there are delete type KeyValue. > This change causes the PriorityQueue of RegionScanner's heap > is not sure to be sorted. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HBASE-5121) MajorCompaction may affect scan's correctness
[ https://issues.apache.org/jira/browse/HBASE-5121?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13180887#comment-13180887 ] Zhihong Yu commented on HBASE-5121: --- Amit's review is certainly welcome. We may add more tests involving deletion and major compaction. > MajorCompaction may affect scan's correctness > - > > Key: HBASE-5121 > URL: https://issues.apache.org/jira/browse/HBASE-5121 > Project: HBase > Issue Type: Bug > Components: regionserver >Affects Versions: 0.90.4 >Reporter: chunhui shen >Assignee: chunhui shen > Fix For: 0.92.0, 0.94.0, 0.90.6 > > Attachments: 5121-trunk-combined.txt, hbase-5121-testcase.patch, > hbase-5121.patch, hbase-5121v2.patch > > > In our test, there are two families' keyvalue for one row. > But we could find a infrequent problem when doing scan's next if > majorCompaction happens concurrently. > In the client's two continuous doing scan.next(): > 1.First time, scan's next returns the result where family A is null. > 2.Second time, scan's next returns the result where family B is null. > The two next()'s result have the same row. > If there are more families, I think the scenario will be more strange... > We find the reason is that storescanner.peek() is changed after > majorCompaction if there are delete type KeyValue. > This change causes the PriorityQueue of RegionScanner's heap > is not sure to be sorted. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HBASE-5121) MajorCompaction may affect scan's correctness
[ https://issues.apache.org/jira/browse/HBASE-5121?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13181034#comment-13181034 ] chunhui shen commented on HBASE-5121: - @Lars Its happen need two conditions. 1.storescanner.peek() is delete type keyvalue. 2.do majorcompaction before scan.next(), and this majorcompaction combine the del type KV and put type KV to nothing. > MajorCompaction may affect scan's correctness > - > > Key: HBASE-5121 > URL: https://issues.apache.org/jira/browse/HBASE-5121 > Project: HBase > Issue Type: Bug > Components: regionserver >Affects Versions: 0.90.4 >Reporter: chunhui shen >Assignee: chunhui shen > Fix For: 0.92.0, 0.94.0, 0.90.6 > > Attachments: 5121-trunk-combined.txt, hbase-5121-testcase.patch, > hbase-5121.patch, hbase-5121v2.patch > > > In our test, there are two families' keyvalue for one row. > But we could find a infrequent problem when doing scan's next if > majorCompaction happens concurrently. > In the client's two continuous doing scan.next(): > 1.First time, scan's next returns the result where family A is null. > 2.Second time, scan's next returns the result where family B is null. > The two next()'s result have the same row. > If there are more families, I think the scenario will be more strange... > We find the reason is that storescanner.peek() is changed after > majorCompaction if there are delete type KeyValue. > This change causes the PriorityQueue of RegionScanner's heap > is not sure to be sorted. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HBASE-5121) MajorCompaction may affect scan's correctness
[ https://issues.apache.org/jira/browse/HBASE-5121?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13181120#comment-13181120 ] Zhihong Yu commented on HBASE-5121: --- @Chunhui: Do you mind producing a patch for 0.90 branch ? If you're busy, I can do it. > MajorCompaction may affect scan's correctness > - > > Key: HBASE-5121 > URL: https://issues.apache.org/jira/browse/HBASE-5121 > Project: HBase > Issue Type: Bug > Components: regionserver >Affects Versions: 0.90.4 >Reporter: chunhui shen >Assignee: chunhui shen > Fix For: 0.92.0, 0.94.0, 0.90.6 > > Attachments: 5121-trunk-combined.txt, hbase-5121-testcase.patch, > hbase-5121.patch, hbase-5121v2.patch > > > In our test, there are two families' keyvalue for one row. > But we could find a infrequent problem when doing scan's next if > majorCompaction happens concurrently. > In the client's two continuous doing scan.next(): > 1.First time, scan's next returns the result where family A is null. > 2.Second time, scan's next returns the result where family B is null. > The two next()'s result have the same row. > If there are more families, I think the scenario will be more strange... > We find the reason is that storescanner.peek() is changed after > majorCompaction if there are delete type KeyValue. > This change causes the PriorityQueue of RegionScanner's heap > is not sure to be sorted. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HBASE-5121) MajorCompaction may affect scan's correctness
[ https://issues.apache.org/jira/browse/HBASE-5121?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13181487#comment-13181487 ] Hadoop QA commented on HBASE-5121: -- -1 overall. Here are the results of testing the latest attachment http://issues.apache.org/jira/secure/attachment/12509698/5121.90 against trunk revision . +1 @author. The patch does not contain any @author tags. +1 tests included. The patch appears to include 3 new or modified tests. -1 patch. The patch command could not apply the patch. Console output: https://builds.apache.org/job/PreCommit-HBASE-Build/687//console This message is automatically generated. > MajorCompaction may affect scan's correctness > - > > Key: HBASE-5121 > URL: https://issues.apache.org/jira/browse/HBASE-5121 > Project: HBase > Issue Type: Bug > Components: regionserver >Affects Versions: 0.90.4 >Reporter: chunhui shen >Assignee: chunhui shen > Fix For: 0.92.0, 0.94.0, 0.90.6 > > Attachments: 5121-trunk-combined.txt, 5121.90, > hbase-5121-testcase.patch, hbase-5121.patch, hbase-5121v2.patch > > > In our test, there are two families' keyvalue for one row. > But we could find a infrequent problem when doing scan's next if > majorCompaction happens concurrently. > In the client's two continuous doing scan.next(): > 1.First time, scan's next returns the result where family A is null. > 2.Second time, scan's next returns the result where family B is null. > The two next()'s result have the same row. > If there are more families, I think the scenario will be more strange... > We find the reason is that storescanner.peek() is changed after > majorCompaction if there are delete type KeyValue. > This change causes the PriorityQueue of RegionScanner's heap > is not sure to be sorted. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HBASE-5121) MajorCompaction may affect scan's correctness
[ https://issues.apache.org/jira/browse/HBASE-5121?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13181502#comment-13181502 ] ramkrishna.s.vasudevan commented on HBASE-5121: --- @Ted Thanks for the 0.90 patch. > MajorCompaction may affect scan's correctness > - > > Key: HBASE-5121 > URL: https://issues.apache.org/jira/browse/HBASE-5121 > Project: HBase > Issue Type: Bug > Components: regionserver >Affects Versions: 0.90.4 >Reporter: chunhui shen >Assignee: chunhui shen >Priority: Critical > Fix For: 0.92.0, 0.94.0, 0.90.6 > > Attachments: 5121-trunk-combined.txt, 5121.90, > hbase-5121-testcase.patch, hbase-5121.patch, hbase-5121v2.patch > > > In our test, there are two families' keyvalue for one row. > But we could find a infrequent problem when doing scan's next if > majorCompaction happens concurrently. > In the client's two continuous doing scan.next(): > 1.First time, scan's next returns the result where family A is null. > 2.Second time, scan's next returns the result where family B is null. > The two next()'s result have the same row. > If there are more families, I think the scenario will be more strange... > We find the reason is that storescanner.peek() is changed after > majorCompaction if there are delete type KeyValue. > This change causes the PriorityQueue of RegionScanner's heap > is not sure to be sorted. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HBASE-5121) MajorCompaction may affect scan's correctness
[ https://issues.apache.org/jira/browse/HBASE-5121?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13181735#comment-13181735 ] chunhui shen commented on HBASE-5121: - @Ted when majorcompaction changes storescanner.peek, we need to tell the upper storeheap(regionscanner's heap) to resort this storescanner again. we have tried some other approaches for the above motivation.But we think throwing an exception is much easier and available to transfer the message of peek changed. It only happen in a read handler. So it is healthy and will not affect the kernal thread of regionserver. If there are other approaches, I'd like to discuss about them. Thanks. > MajorCompaction may affect scan's correctness > - > > Key: HBASE-5121 > URL: https://issues.apache.org/jira/browse/HBASE-5121 > Project: HBase > Issue Type: Bug > Components: regionserver >Affects Versions: 0.90.4 >Reporter: chunhui shen >Assignee: chunhui shen >Priority: Critical > Fix For: 0.94.0, 0.92.1, 0.90.6 > > Attachments: 5121-trunk-combined.txt, 5121.90, > hbase-5121-testcase.patch, hbase-5121.patch, hbase-5121v2.patch > > > In our test, there are two families' keyvalue for one row. > But we could find a infrequent problem when doing scan's next if > majorCompaction happens concurrently. > In the client's two continuous doing scan.next(): > 1.First time, scan's next returns the result where family A is null. > 2.Second time, scan's next returns the result where family B is null. > The two next()'s result have the same row. > If there are more families, I think the scenario will be more strange... > We find the reason is that storescanner.peek() is changed after > majorCompaction if there are delete type KeyValue. > This change causes the PriorityQueue of RegionScanner's heap > is not sure to be sorted. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HBASE-5121) MajorCompaction may affect scan's correctness
[
https://issues.apache.org/jira/browse/HBASE-5121?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13181791#comment-13181791
]
Zhihong Yu commented on HBASE-5121:
---
w.r.t. introducing new exception, here is StoreScanner.next():
{code}
* @return true if there are more rows, false if scanner is done
*/
@Override
public synchronized boolean next(List outResult, int limit) throws
IOException {
checkReseek();
{code}
The return value has clear meaning. One approach is to change return type to an
enum.
@Chunhui:
Do you have time to come up with a patch that doesn't rely on new exception.
Instead, it relies on StoreScanner.next() returning an enum (SCAN_DONE,
MORE_ROWS, PEEK_CHANGED).
This way, people can evaluate both approaches.
Of course, Stack and Todd may comment on other areas of the patch.
> MajorCompaction may affect scan's correctness
> -
>
> Key: HBASE-5121
> URL: https://issues.apache.org/jira/browse/HBASE-5121
> Project: HBase
> Issue Type: Bug
> Components: regionserver
>Affects Versions: 0.90.4
>Reporter: chunhui shen
>Assignee: chunhui shen
>Priority: Critical
> Fix For: 0.94.0, 0.92.1, 0.90.6
>
> Attachments: 5121-trunk-combined.txt, 5121.90,
> hbase-5121-testcase.patch, hbase-5121.patch, hbase-5121v2.patch
>
>
> In our test, there are two families' keyvalue for one row.
> But we could find a infrequent problem when doing scan's next if
> majorCompaction happens concurrently.
> In the client's two continuous doing scan.next():
> 1.First time, scan's next returns the result where family A is null.
> 2.Second time, scan's next returns the result where family B is null.
> The two next()'s result have the same row.
> If there are more families, I think the scenario will be more strange...
> We find the reason is that storescanner.peek() is changed after
> majorCompaction if there are delete type KeyValue.
> This change causes the PriorityQueue of RegionScanner's heap
> is not sure to be sorted.
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators:
https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HBASE-5121) MajorCompaction may affect scan's correctness
[ https://issues.apache.org/jira/browse/HBASE-5121?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13182166#comment-13182166 ] chunhui shen commented on HBASE-5121: - @Ted ok, I will make a new patch > MajorCompaction may affect scan's correctness > - > > Key: HBASE-5121 > URL: https://issues.apache.org/jira/browse/HBASE-5121 > Project: HBase > Issue Type: Bug > Components: regionserver >Affects Versions: 0.90.4 >Reporter: chunhui shen >Assignee: chunhui shen >Priority: Critical > Fix For: 0.94.0, 0.92.1, 0.90.6 > > Attachments: 5121-trunk-combined.txt, 5121.90, > hbase-5121-testcase.patch, hbase-5121.patch, hbase-5121v2.patch > > > In our test, there are two families' keyvalue for one row. > But we could find a infrequent problem when doing scan's next if > majorCompaction happens concurrently. > In the client's two continuous doing scan.next(): > 1.First time, scan's next returns the result where family A is null. > 2.Second time, scan's next returns the result where family B is null. > The two next()'s result have the same row. > If there are more families, I think the scenario will be more strange... > We find the reason is that storescanner.peek() is changed after > majorCompaction if there are delete type KeyValue. > This change causes the PriorityQueue of RegionScanner's heap > is not sure to be sorted. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HBASE-5121) MajorCompaction may affect scan's correctness
[ https://issues.apache.org/jira/browse/HBASE-5121?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13182178#comment-13182178 ] chunhui shen commented on HBASE-5121: - @Ted If we change StoreScanner.next() to return an enum , we should change all the implement of InternalScanner.next(), therefore KeyValueHeap.next()'s return should also be changed to an enum. It needs to change the logic who calles KeyValueHeap.next() . I think it changes too much, does it need? > MajorCompaction may affect scan's correctness > - > > Key: HBASE-5121 > URL: https://issues.apache.org/jira/browse/HBASE-5121 > Project: HBase > Issue Type: Bug > Components: regionserver >Affects Versions: 0.90.4 >Reporter: chunhui shen >Assignee: chunhui shen >Priority: Critical > Fix For: 0.94.0, 0.92.1, 0.90.6 > > Attachments: 5121-trunk-combined.txt, 5121.90, > hbase-5121-testcase.patch, hbase-5121.patch, hbase-5121v2.patch > > > In our test, there are two families' keyvalue for one row. > But we could find a infrequent problem when doing scan's next if > majorCompaction happens concurrently. > In the client's two continuous doing scan.next(): > 1.First time, scan's next returns the result where family A is null. > 2.Second time, scan's next returns the result where family B is null. > The two next()'s result have the same row. > If there are more families, I think the scenario will be more strange... > We find the reason is that storescanner.peek() is changed after > majorCompaction if there are delete type KeyValue. > This change causes the PriorityQueue of RegionScanner's heap > is not sure to be sorted. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HBASE-5121) MajorCompaction may affect scan's correctness
[ https://issues.apache.org/jira/browse/HBASE-5121?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13182184#comment-13182184 ] Zhihong Yu commented on HBASE-5121: --- @Chunhui: I agree. That's why I think creating a new exception is acceptable. Maybe Todd or Stack has better idea. > MajorCompaction may affect scan's correctness > - > > Key: HBASE-5121 > URL: https://issues.apache.org/jira/browse/HBASE-5121 > Project: HBase > Issue Type: Bug > Components: regionserver >Affects Versions: 0.90.4 >Reporter: chunhui shen >Assignee: chunhui shen >Priority: Critical > Fix For: 0.94.0, 0.92.1, 0.90.6 > > Attachments: 5121-trunk-combined.txt, 5121.90, > hbase-5121-testcase.patch, hbase-5121.patch, hbase-5121v2.patch > > > In our test, there are two families' keyvalue for one row. > But we could find a infrequent problem when doing scan's next if > majorCompaction happens concurrently. > In the client's two continuous doing scan.next(): > 1.First time, scan's next returns the result where family A is null. > 2.Second time, scan's next returns the result where family B is null. > The two next()'s result have the same row. > If there are more families, I think the scenario will be more strange... > We find the reason is that storescanner.peek() is changed after > majorCompaction if there are delete type KeyValue. > This change causes the PriorityQueue of RegionScanner's heap > is not sure to be sorted. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HBASE-5121) MajorCompaction may affect scan's correctness
[ https://issues.apache.org/jira/browse/HBASE-5121?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13182190#comment-13182190 ] Hadoop QA commented on HBASE-5121: -- -1 overall. Here are the results of testing the latest attachment http://issues.apache.org/jira/secure/attachment/12509823/5121-suggest.txt against trunk revision . +1 @author. The patch does not contain any @author tags. +1 tests included. The patch appears to include 3 new or modified tests. -1 javadoc. The javadoc tool appears to have generated -151 warning messages. +1 javac. The applied patch does not increase the total number of javac compiler warnings. -1 findbugs. The patch appears to introduce 79 new Findbugs (version 1.3.9) warnings. +1 release audit. The applied patch does not increase the total number of release audit warnings. -1 core tests. The patch failed these unit tests: org.apache.hadoop.hbase.mapreduce.TestImportTsv org.apache.hadoop.hbase.mapred.TestTableMapReduce org.apache.hadoop.hbase.mapreduce.TestHFileOutputFormat Test results: https://builds.apache.org/job/PreCommit-HBASE-Build/700//testReport/ Findbugs warnings: https://builds.apache.org/job/PreCommit-HBASE-Build/700//artifact/trunk/patchprocess/newPatchFindbugsWarnings.html Console output: https://builds.apache.org/job/PreCommit-HBASE-Build/700//console This message is automatically generated. > MajorCompaction may affect scan's correctness > - > > Key: HBASE-5121 > URL: https://issues.apache.org/jira/browse/HBASE-5121 > Project: HBase > Issue Type: Bug > Components: regionserver >Affects Versions: 0.90.4 >Reporter: chunhui shen >Assignee: chunhui shen >Priority: Critical > Fix For: 0.94.0, 0.92.1, 0.90.6 > > Attachments: 5121-suggest.txt, 5121-trunk-combined.txt, 5121.90, > hbase-5121-testcase.patch, hbase-5121.patch, hbase-5121v2.patch > > > In our test, there are two families' keyvalue for one row. > But we could find a infrequent problem when doing scan's next if > majorCompaction happens concurrently. > In the client's two continuous doing scan.next(): > 1.First time, scan's next returns the result where family A is null. > 2.Second time, scan's next returns the result where family B is null. > The two next()'s result have the same row. > If there are more families, I think the scenario will be more strange... > We find the reason is that storescanner.peek() is changed after > majorCompaction if there are delete type KeyValue. > This change causes the PriorityQueue of RegionScanner's heap > is not sure to be sorted. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HBASE-5121) MajorCompaction may affect scan's correctness
[ https://issues.apache.org/jira/browse/HBASE-5121?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13182345#comment-13182345 ] chunhui shen commented on HBASE-5121: - @Lars I agree with the suggested patch which is simpler and available. > MajorCompaction may affect scan's correctness > - > > Key: HBASE-5121 > URL: https://issues.apache.org/jira/browse/HBASE-5121 > Project: HBase > Issue Type: Bug > Components: regionserver >Affects Versions: 0.90.4 >Reporter: chunhui shen >Assignee: chunhui shen >Priority: Critical > Fix For: 0.94.0, 0.92.1, 0.90.6 > > Attachments: 5121-suggest.txt, 5121-trunk-combined.txt, 5121.90, > hbase-5121-testcase.patch, hbase-5121.patch, hbase-5121v2.patch > > > In our test, there are two families' keyvalue for one row. > But we could find a infrequent problem when doing scan's next if > majorCompaction happens concurrently. > In the client's two continuous doing scan.next(): > 1.First time, scan's next returns the result where family A is null. > 2.Second time, scan's next returns the result where family B is null. > The two next()'s result have the same row. > If there are more families, I think the scenario will be more strange... > We find the reason is that storescanner.peek() is changed after > majorCompaction if there are delete type KeyValue. > This change causes the PriorityQueue of RegionScanner's heap > is not sure to be sorted. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HBASE-5121) MajorCompaction may affect scan's correctness
[ https://issues.apache.org/jira/browse/HBASE-5121?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13183016#comment-13183016 ] Lars Hofhansl commented on HBASE-5121: -- Any more input on the latest patch? Re: 0.90.6. It seems we should do this only in 0.92 and above as 0.90.x does not provide many of the ACID guarantees anyway (see HBASE-2856 and HBASE-4838). > MajorCompaction may affect scan's correctness > - > > Key: HBASE-5121 > URL: https://issues.apache.org/jira/browse/HBASE-5121 > Project: HBase > Issue Type: Bug > Components: regionserver >Affects Versions: 0.90.4 >Reporter: chunhui shen >Assignee: chunhui shen >Priority: Critical > Fix For: 0.94.0, 0.92.1, 0.90.6 > > Attachments: 5121-suggest.txt, 5121-trunk-combined.txt, 5121.90, > hbase-5121-testcase.patch, hbase-5121.patch, hbase-5121v2.patch > > > In our test, there are two families' keyvalue for one row. > But we could find a infrequent problem when doing scan's next if > majorCompaction happens concurrently. > In the client's two continuous doing scan.next(): > 1.First time, scan's next returns the result where family A is null. > 2.Second time, scan's next returns the result where family B is null. > The two next()'s result have the same row. > If there are more families, I think the scenario will be more strange... > We find the reason is that storescanner.peek() is changed after > majorCompaction if there are delete type KeyValue. > This change causes the PriorityQueue of RegionScanner's heap > is not sure to be sorted. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HBASE-5121) MajorCompaction may affect scan's correctness
[ https://issues.apache.org/jira/browse/HBASE-5121?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13183019#comment-13183019 ] Zhihong Yu commented on HBASE-5121: --- +1 on latest patch going to 0.92 and TRUNK. > MajorCompaction may affect scan's correctness > - > > Key: HBASE-5121 > URL: https://issues.apache.org/jira/browse/HBASE-5121 > Project: HBase > Issue Type: Bug > Components: regionserver >Affects Versions: 0.90.4 >Reporter: chunhui shen >Assignee: chunhui shen >Priority: Critical > Fix For: 0.94.0, 0.92.1, 0.90.6 > > Attachments: 5121-suggest.txt, 5121-trunk-combined.txt, 5121.90, > hbase-5121-testcase.patch, hbase-5121.patch, hbase-5121v2.patch > > > In our test, there are two families' keyvalue for one row. > But we could find a infrequent problem when doing scan's next if > majorCompaction happens concurrently. > In the client's two continuous doing scan.next(): > 1.First time, scan's next returns the result where family A is null. > 2.Second time, scan's next returns the result where family B is null. > The two next()'s result have the same row. > If there are more families, I think the scenario will be more strange... > We find the reason is that storescanner.peek() is changed after > majorCompaction if there are delete type KeyValue. > This change causes the PriorityQueue of RegionScanner's heap > is not sure to be sorted. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HBASE-5121) MajorCompaction may affect scan's correctness
[ https://issues.apache.org/jira/browse/HBASE-5121?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13183101#comment-13183101 ] Lars Hofhansl commented on HBASE-5121: -- OK... will commit tomorrow unless there are any objections. @Ram: You OK with this not being in 0.90.6? > MajorCompaction may affect scan's correctness > - > > Key: HBASE-5121 > URL: https://issues.apache.org/jira/browse/HBASE-5121 > Project: HBase > Issue Type: Bug > Components: regionserver >Affects Versions: 0.90.4 >Reporter: chunhui shen >Assignee: chunhui shen >Priority: Critical > Fix For: 0.94.0, 0.92.1, 0.90.6 > > Attachments: 5121-suggest.txt, 5121-trunk-combined.txt, 5121.90, > hbase-5121-testcase.patch, hbase-5121.patch, hbase-5121v2.patch > > > In our test, there are two families' keyvalue for one row. > But we could find a infrequent problem when doing scan's next if > majorCompaction happens concurrently. > In the client's two continuous doing scan.next(): > 1.First time, scan's next returns the result where family A is null. > 2.Second time, scan's next returns the result where family B is null. > The two next()'s result have the same row. > If there are more families, I think the scenario will be more strange... > We find the reason is that storescanner.peek() is changed after > majorCompaction if there are delete type KeyValue. > This change causes the PriorityQueue of RegionScanner's heap > is not sure to be sorted. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HBASE-5121) MajorCompaction may affect scan's correctness
[ https://issues.apache.org/jira/browse/HBASE-5121?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13183110#comment-13183110 ] ramkrishna.s.vasudevan commented on HBASE-5121: --- @Lars Ok for not going into 0.90.6. Thanks Lars. > MajorCompaction may affect scan's correctness > - > > Key: HBASE-5121 > URL: https://issues.apache.org/jira/browse/HBASE-5121 > Project: HBase > Issue Type: Bug > Components: regionserver >Affects Versions: 0.90.4 >Reporter: chunhui shen >Assignee: chunhui shen >Priority: Critical > Fix For: 0.94.0, 0.92.1, 0.90.6 > > Attachments: 5121-suggest.txt, 5121-trunk-combined.txt, 5121.90, > hbase-5121-testcase.patch, hbase-5121.patch, hbase-5121v2.patch > > > In our test, there are two families' keyvalue for one row. > But we could find a infrequent problem when doing scan's next if > majorCompaction happens concurrently. > In the client's two continuous doing scan.next(): > 1.First time, scan's next returns the result where family A is null. > 2.Second time, scan's next returns the result where family B is null. > The two next()'s result have the same row. > If there are more families, I think the scenario will be more strange... > We find the reason is that storescanner.peek() is changed after > majorCompaction if there are delete type KeyValue. > This change causes the PriorityQueue of RegionScanner's heap > is not sure to be sorted. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HBASE-5121) MajorCompaction may affect scan's correctness
[ https://issues.apache.org/jira/browse/HBASE-5121?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13183803#comment-13183803 ] Hudson commented on HBASE-5121: --- Integrated in HBase-0.92 #239 (See [https://builds.apache.org/job/HBase-0.92/239/]) HBASE-5121 MajorCompaction may affect scan's correctness (chunhui shen and Lars H) larsh : Files : * /hbase/branches/0.92/CHANGES.txt * /hbase/branches/0.92/src/main/java/org/apache/hadoop/hbase/regionserver/KeyValueHeap.java * /hbase/branches/0.92/src/main/java/org/apache/hadoop/hbase/regionserver/StoreScanner.java * /hbase/branches/0.92/src/test/java/org/apache/hadoop/hbase/regionserver/TestScanner.java > MajorCompaction may affect scan's correctness > - > > Key: HBASE-5121 > URL: https://issues.apache.org/jira/browse/HBASE-5121 > Project: HBase > Issue Type: Bug > Components: regionserver >Affects Versions: 0.90.4 >Reporter: chunhui shen >Assignee: chunhui shen >Priority: Critical > Fix For: 0.94.0, 0.92.1 > > Attachments: 5121-0.92.txt, 5121-suggest.txt, > 5121-trunk-combined.txt, 5121.90, hbase-5121-testcase.patch, > hbase-5121.patch, hbase-5121v2.patch > > > In our test, there are two families' keyvalue for one row. > But we could find a infrequent problem when doing scan's next if > majorCompaction happens concurrently. > In the client's two continuous doing scan.next(): > 1.First time, scan's next returns the result where family A is null. > 2.Second time, scan's next returns the result where family B is null. > The two next()'s result have the same row. > If there are more families, I think the scenario will be more strange... > We find the reason is that storescanner.peek() is changed after > majorCompaction if there are delete type KeyValue. > This change causes the PriorityQueue of RegionScanner's heap > is not sure to be sorted. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HBASE-5121) MajorCompaction may affect scan's correctness
[ https://issues.apache.org/jira/browse/HBASE-5121?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13183917#comment-13183917 ] Hudson commented on HBASE-5121: --- Integrated in HBase-0.92-security #71 (See [https://builds.apache.org/job/HBase-0.92-security/71/]) HBASE-5121 MajorCompaction may affect scan's correctness (chunhui shen and Lars H) larsh : Files : * /hbase/branches/0.92/CHANGES.txt * /hbase/branches/0.92/src/main/java/org/apache/hadoop/hbase/regionserver/KeyValueHeap.java * /hbase/branches/0.92/src/main/java/org/apache/hadoop/hbase/regionserver/StoreScanner.java * /hbase/branches/0.92/src/test/java/org/apache/hadoop/hbase/regionserver/TestScanner.java > MajorCompaction may affect scan's correctness > - > > Key: HBASE-5121 > URL: https://issues.apache.org/jira/browse/HBASE-5121 > Project: HBase > Issue Type: Bug > Components: regionserver >Affects Versions: 0.90.4 >Reporter: chunhui shen >Assignee: chunhui shen >Priority: Critical > Fix For: 0.94.0, 0.92.1 > > Attachments: 5121-0.92.txt, 5121-suggest.txt, > 5121-trunk-combined.txt, 5121.90, hbase-5121-testcase.patch, > hbase-5121.patch, hbase-5121v2.patch > > > In our test, there are two families' keyvalue for one row. > But we could find a infrequent problem when doing scan's next if > majorCompaction happens concurrently. > In the client's two continuous doing scan.next(): > 1.First time, scan's next returns the result where family A is null. > 2.Second time, scan's next returns the result where family B is null. > The two next()'s result have the same row. > If there are more families, I think the scenario will be more strange... > We find the reason is that storescanner.peek() is changed after > majorCompaction if there are delete type KeyValue. > This change causes the PriorityQueue of RegionScanner's heap > is not sure to be sorted. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HBASE-5121) MajorCompaction may affect scan's correctness
[ https://issues.apache.org/jira/browse/HBASE-5121?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13183930#comment-13183930 ] Hudson commented on HBASE-5121: --- Integrated in HBase-TRUNK-security #72 (See [https://builds.apache.org/job/HBase-TRUNK-security/72/]) HBASE-5121 MajorCompaction may affect scan's correctness (chunhui shen and Lars H) larsh : Files : * /hbase/trunk/CHANGES.txt * /hbase/trunk/src/main/java/org/apache/hadoop/hbase/regionserver/KeyValueHeap.java * /hbase/trunk/src/main/java/org/apache/hadoop/hbase/regionserver/StoreScanner.java * /hbase/trunk/src/test/java/org/apache/hadoop/hbase/regionserver/TestScanner.java > MajorCompaction may affect scan's correctness > - > > Key: HBASE-5121 > URL: https://issues.apache.org/jira/browse/HBASE-5121 > Project: HBase > Issue Type: Bug > Components: regionserver >Affects Versions: 0.90.4 >Reporter: chunhui shen >Assignee: chunhui shen >Priority: Critical > Fix For: 0.94.0, 0.92.1 > > Attachments: 5121-0.92.txt, 5121-suggest.txt, > 5121-trunk-combined.txt, 5121.90, hbase-5121-testcase.patch, > hbase-5121.patch, hbase-5121v2.patch > > > In our test, there are two families' keyvalue for one row. > But we could find a infrequent problem when doing scan's next if > majorCompaction happens concurrently. > In the client's two continuous doing scan.next(): > 1.First time, scan's next returns the result where family A is null. > 2.Second time, scan's next returns the result where family B is null. > The two next()'s result have the same row. > If there are more families, I think the scenario will be more strange... > We find the reason is that storescanner.peek() is changed after > majorCompaction if there are delete type KeyValue. > This change causes the PriorityQueue of RegionScanner's heap > is not sure to be sorted. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HBASE-5121) MajorCompaction may affect scan's correctness
[ https://issues.apache.org/jira/browse/HBASE-5121?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13183972#comment-13183972 ] Hudson commented on HBASE-5121: --- Integrated in HBase-TRUNK #2617 (See [https://builds.apache.org/job/HBase-TRUNK/2617/]) HBASE-5121 MajorCompaction may affect scan's correctness (chunhui shen and Lars H) larsh : Files : * /hbase/trunk/CHANGES.txt * /hbase/trunk/src/main/java/org/apache/hadoop/hbase/regionserver/KeyValueHeap.java * /hbase/trunk/src/main/java/org/apache/hadoop/hbase/regionserver/StoreScanner.java * /hbase/trunk/src/test/java/org/apache/hadoop/hbase/regionserver/TestScanner.java > MajorCompaction may affect scan's correctness > - > > Key: HBASE-5121 > URL: https://issues.apache.org/jira/browse/HBASE-5121 > Project: HBase > Issue Type: Bug > Components: regionserver >Affects Versions: 0.90.4 >Reporter: chunhui shen >Assignee: chunhui shen >Priority: Critical > Fix For: 0.94.0, 0.92.1 > > Attachments: 5121-0.92.txt, 5121-suggest.txt, > 5121-trunk-combined.txt, 5121.90, hbase-5121-testcase.patch, > hbase-5121.patch, hbase-5121v2.patch > > > In our test, there are two families' keyvalue for one row. > But we could find a infrequent problem when doing scan's next if > majorCompaction happens concurrently. > In the client's two continuous doing scan.next(): > 1.First time, scan's next returns the result where family A is null. > 2.Second time, scan's next returns the result where family B is null. > The two next()'s result have the same row. > If there are more families, I think the scenario will be more strange... > We find the reason is that storescanner.peek() is changed after > majorCompaction if there are delete type KeyValue. > This change causes the PriorityQueue of RegionScanner's heap > is not sure to be sorted. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HBASE-5121) MajorCompaction may affect scan's correctness
[ https://issues.apache.org/jira/browse/HBASE-5121?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13185113#comment-13185113 ] Lars Hofhansl commented on HBASE-5121: -- Thanks for finding and patch chunhui. > MajorCompaction may affect scan's correctness > - > > Key: HBASE-5121 > URL: https://issues.apache.org/jira/browse/HBASE-5121 > Project: HBase > Issue Type: Bug > Components: regionserver >Affects Versions: 0.90.4 >Reporter: chunhui shen >Assignee: chunhui shen >Priority: Critical > Fix For: 0.94.0, 0.92.1 > > Attachments: 5121-0.92.txt, 5121-suggest.txt, > 5121-trunk-combined.txt, 5121.90, hbase-5121-testcase.patch, > hbase-5121.patch, hbase-5121v2.patch > > > In our test, there are two families' keyvalue for one row. > But we could find a infrequent problem when doing scan's next if > majorCompaction happens concurrently. > In the client's two continuous doing scan.next(): > 1.First time, scan's next returns the result where family A is null. > 2.Second time, scan's next returns the result where family B is null. > The two next()'s result have the same row. > If there are more families, I think the scenario will be more strange... > We find the reason is that storescanner.peek() is changed after > majorCompaction if there are delete type KeyValue. > This change causes the PriorityQueue of RegionScanner's heap > is not sure to be sorted. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HBASE-5121) MajorCompaction may affect scan's correctness
[ https://issues.apache.org/jira/browse/HBASE-5121?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13420274#comment-13420274 ] Lars Hofhansl commented on HBASE-5121: -- @chunhiu: Could have a look at the suggested change in HBASE-5659? The observation is that: # after the scanner is reset the top KV will in the vast majority of all case be changed # that is only a problem when the top KV's row has changed, in many case StoreScanner.next indicates more work than necessary to a caller. > MajorCompaction may affect scan's correctness > - > > Key: HBASE-5121 > URL: https://issues.apache.org/jira/browse/HBASE-5121 > Project: HBase > Issue Type: Bug > Components: regionserver >Affects Versions: 0.90.4 >Reporter: chunhui shen >Assignee: chunhui shen >Priority: Critical > Fix For: 0.92.1, 0.94.0 > > Attachments: 5121-0.92.txt, 5121-suggest.txt, > 5121-trunk-combined.txt, 5121.90, hbase-5121-testcase.patch, > hbase-5121.patch, hbase-5121v2.patch > > > In our test, there are two families' keyvalue for one row. > But we could find a infrequent problem when doing scan's next if > majorCompaction happens concurrently. > In the client's two continuous doing scan.next(): > 1.First time, scan's next returns the result where family A is null. > 2.Second time, scan's next returns the result where family B is null. > The two next()'s result have the same row. > If there are more families, I think the scenario will be more strange... > We find the reason is that storescanner.peek() is changed after > majorCompaction if there are delete type KeyValue. > This change causes the PriorityQueue of RegionScanner's heap > is not sure to be sorted. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa For more information on JIRA, see: http://www.atlassian.com/software/jira
