[jira] [Commented] (PHOENIX-3705) SkipScanFilter may repeatedly copy rowKey Columns to startKey
[ https://issues.apache.org/jira/browse/PHOENIX-3705?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15895507#comment-15895507 ] Hudson commented on PHOENIX-3705: - SUCCESS: Integrated in Jenkins build Phoenix-master #1576 (See [https://builds.apache.org/job/Phoenix-master/1576/]) PHOENIX-3705 SkipScanFilter may repeatedly copy rowKey Columns to (chenglei: rev c8612fa1b09f883726102951626798244f73db17) * (edit) phoenix-core/src/test/java/org/apache/phoenix/filter/SkipScanFilterTest.java * (edit) phoenix-core/src/main/java/org/apache/phoenix/filter/SkipScanFilter.java * (edit) phoenix-core/src/main/java/org/apache/phoenix/schema/RowKeySchema.java > SkipScanFilter may repeatedly copy rowKey Columns to startKey > - > > Key: PHOENIX-3705 > URL: https://issues.apache.org/jira/browse/PHOENIX-3705 > Project: Phoenix > Issue Type: Bug >Affects Versions: 4.9.0 >Reporter: chenglei >Assignee: chenglei >Priority: Blocker > Fix For: 4.10.0 > > Attachments: PHOENIX-3705_v2.patch > > > See following simple unit test first,the rowKey is composed of three PInteger > columns,and the slots of SkipScanFilter are: > [ [[1 - 4]], [5, 7], [[9 - 10]] ], > When SkipScanFilter.filterKeyValue method is invoked on a KeyValue whose > rowKey is [2,7,11], obviously SkipScanFilter.filterKeyValue > returns ReturnCode.SEEK_NEXT_USING_HINT and SkipScanFilter.getNextCellHint > returns [3,5,9], but unfortunately, SkipScanFilter.getNextCellHint actually > returns [2,8,5,9] , a very strange value, the unit tests failed. > {code} > @Test > public void testNavigate() { > RowKeySchemaBuilder builder = new RowKeySchemaBuilder(3); > for(int i=0;i<3;i++) { > builder.addField( > new PDatum() { > @Override > public boolean isNullable() { > return false; > } > @Override > public PDataType getDataType() { > return PInteger.INSTANCE; > } > @Override > public Integer getMaxLength() { > return PInteger.INSTANCE.getMaxLength(null); > } > @Override > public Integer getScale() { > return PInteger.INSTANCE.getScale(null); > } > @Override > public SortOrder getSortOrder() { > return SortOrder.getDefault(); > } > }, false, SortOrder.getDefault()); > } > > ListrowKeyColumnRangesList=Arrays.asList( > Arrays.asList( > > PInteger.INSTANCE.getKeyRange(PInteger.INSTANCE.toBytes(1), true, > PInteger.INSTANCE.toBytes(4), true)), > Arrays.asList( > KeyRange.getKeyRange(PInteger.INSTANCE.toBytes(5)), > KeyRange.getKeyRange(PInteger.INSTANCE.toBytes(7))), > Arrays.asList( > > PInteger.INSTANCE.getKeyRange(PInteger.INSTANCE.toBytes(9), true, > PInteger.INSTANCE.toBytes(10), true)) > ); > > SkipScanFilter skipScanFilter=new > SkipScanFilter(rowKeyColumnRangesList, builder.build()); > > System.out.println(skipScanFilter); > > byte[] rowKey=ByteUtil.concat( > PInteger.INSTANCE.toBytes(2), > PInteger.INSTANCE.toBytes(7), > PInteger.INSTANCE.toBytes(11)); > KeyValue keyValue=KeyValue.createFirstOnRow(rowKey); > ReturnCode returnCode=skipScanFilter.filterKeyValue(keyValue); > assertTrue(returnCode == ReturnCode.SEEK_NEXT_USING_HINT); > Cell nextCellHint=skipScanFilter.getNextCellHint(keyValue); > > assertTrue(Bytes.toStringBinary(CellUtil.cloneRow(nextCellHint)).equals( > "\\x80\\x00\\x00\\x03\\x80\\x00\\x00\\x05\\x80\\x00\\x00\\x09")); > } > {code} > Let us see what's wrong, first column of rowKey [2,7,11 ] is 2, which is in > SkipScanFilter's first slot range [1-4], so position[0] is 0 and we go to the > second column 7, which match the second range [7] of SkipScanFilter's second > slot [5, 7],so position[1] is 1 and we go to the third column 11, which is > bigger than the third slot range [9 - 10],so position[2] is 0 and the > {{SkipScanFilter.ptr}} which points to current column still stay on the third > column. Now we begin to backtrack to second column, because the second range > [7] of SkipScanFilter's second slot is singleKey and
[jira] [Commented] (PHOENIX-3705) SkipScanFilter may repeatedly copy rowKey Columns to startKey
[ https://issues.apache.org/jira/browse/PHOENIX-3705?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15895471#comment-15895471 ] chenglei commented on PHOENIX-3705: --- Thank you for the review, [~jamestaylor]. Pushed to the master, 4.x-HBase-0.98, and 4.x-HBase-1.1 branches. > SkipScanFilter may repeatedly copy rowKey Columns to startKey > - > > Key: PHOENIX-3705 > URL: https://issues.apache.org/jira/browse/PHOENIX-3705 > Project: Phoenix > Issue Type: Bug >Affects Versions: 4.9.0 >Reporter: chenglei >Priority: Blocker > Fix For: 4.10.0 > > Attachments: PHOENIX-3705_v2.patch > > > See following simple unit test first,the rowKey is composed of three PInteger > columns,and the slots of SkipScanFilter are: > [ [[1 - 4]], [5, 7], [[9 - 10]] ], > When SkipScanFilter.filterKeyValue method is invoked on a KeyValue whose > rowKey is [2,7,11], obviously SkipScanFilter.filterKeyValue > returns ReturnCode.SEEK_NEXT_USING_HINT and SkipScanFilter.getNextCellHint > returns [3,5,9], but unfortunately, SkipScanFilter.getNextCellHint actually > returns [2,8,5,9] , a very strange value, the unit tests failed. > {code} > @Test > public void testNavigate() { > RowKeySchemaBuilder builder = new RowKeySchemaBuilder(3); > for(int i=0;i<3;i++) { > builder.addField( > new PDatum() { > @Override > public boolean isNullable() { > return false; > } > @Override > public PDataType getDataType() { > return PInteger.INSTANCE; > } > @Override > public Integer getMaxLength() { > return PInteger.INSTANCE.getMaxLength(null); > } > @Override > public Integer getScale() { > return PInteger.INSTANCE.getScale(null); > } > @Override > public SortOrder getSortOrder() { > return SortOrder.getDefault(); > } > }, false, SortOrder.getDefault()); > } > > ListrowKeyColumnRangesList=Arrays.asList( > Arrays.asList( > > PInteger.INSTANCE.getKeyRange(PInteger.INSTANCE.toBytes(1), true, > PInteger.INSTANCE.toBytes(4), true)), > Arrays.asList( > KeyRange.getKeyRange(PInteger.INSTANCE.toBytes(5)), > KeyRange.getKeyRange(PInteger.INSTANCE.toBytes(7))), > Arrays.asList( > > PInteger.INSTANCE.getKeyRange(PInteger.INSTANCE.toBytes(9), true, > PInteger.INSTANCE.toBytes(10), true)) > ); > > SkipScanFilter skipScanFilter=new > SkipScanFilter(rowKeyColumnRangesList, builder.build()); > > System.out.println(skipScanFilter); > > byte[] rowKey=ByteUtil.concat( > PInteger.INSTANCE.toBytes(2), > PInteger.INSTANCE.toBytes(7), > PInteger.INSTANCE.toBytes(11)); > KeyValue keyValue=KeyValue.createFirstOnRow(rowKey); > ReturnCode returnCode=skipScanFilter.filterKeyValue(keyValue); > assertTrue(returnCode == ReturnCode.SEEK_NEXT_USING_HINT); > Cell nextCellHint=skipScanFilter.getNextCellHint(keyValue); > > assertTrue(Bytes.toStringBinary(CellUtil.cloneRow(nextCellHint)).equals( > "\\x80\\x00\\x00\\x03\\x80\\x00\\x00\\x05\\x80\\x00\\x00\\x09")); > } > {code} > Let us see what's wrong, first column of rowKey [2,7,11 ] is 2, which is in > SkipScanFilter's first slot range [1-4], so position[0] is 0 and we go to the > second column 7, which match the second range [7] of SkipScanFilter's second > slot [5, 7],so position[1] is 1 and we go to the third column 11, which is > bigger than the third slot range [9 - 10],so position[2] is 0 and the > {{SkipScanFilter.ptr}} which points to current column still stay on the third > column. Now we begin to backtrack to second column, because the second range > [7] of SkipScanFilter's second slot is singleKey and there is no more > range,so position[1] is 0 and we continue to backtrack to first column, > because the first slot range [1-4] is not singleKey so we stop backtracking > at first column. > Now the problem comes, in following line 448 of {{SkipScanFilter.navigate}} > method,{{SkipScanFilter.setStartKey}} method is invoked,first copy rowKey > columns before {{SkipScanFilter.ptr}} to
[jira] [Commented] (PHOENIX-3705) SkipScanFilter may repeatedly copy rowKey Columns to startKey
[ https://issues.apache.org/jira/browse/PHOENIX-3705?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15894937#comment-15894937 ] James Taylor commented on PHOENIX-3705: --- +1 on patch. Nice work, [~comnetwork]. Please commit to master and 4.x branches. > SkipScanFilter may repeatedly copy rowKey Columns to startKey > - > > Key: PHOENIX-3705 > URL: https://issues.apache.org/jira/browse/PHOENIX-3705 > Project: Phoenix > Issue Type: Bug >Affects Versions: 4.9.0 >Reporter: chenglei >Priority: Blocker > Fix For: 4.10.0 > > Attachments: PHOENIX-3705_v2.patch > > > See following simple unit test first,the rowKey is composed of three PInteger > columns,and the slots of SkipScanFilter are: > [ [[1 - 4]], [5, 7], [[9 - 10]] ], > When SkipScanFilter.filterKeyValue method is invoked on a KeyValue whose > rowKey is [2,7,11], obviously SkipScanFilter.filterKeyValue > returns ReturnCode.SEEK_NEXT_USING_HINT and SkipScanFilter.getNextCellHint > returns [3,5,9], but unfortunately, SkipScanFilter.getNextCellHint actually > returns [2,8,5,9] , a very strange value, the unit tests failed. > {code} > @Test > public void testNavigate() { > RowKeySchemaBuilder builder = new RowKeySchemaBuilder(3); > for(int i=0;i<3;i++) { > builder.addField( > new PDatum() { > @Override > public boolean isNullable() { > return false; > } > @Override > public PDataType getDataType() { > return PInteger.INSTANCE; > } > @Override > public Integer getMaxLength() { > return PInteger.INSTANCE.getMaxLength(null); > } > @Override > public Integer getScale() { > return PInteger.INSTANCE.getScale(null); > } > @Override > public SortOrder getSortOrder() { > return SortOrder.getDefault(); > } > }, false, SortOrder.getDefault()); > } > > ListrowKeyColumnRangesList=Arrays.asList( > Arrays.asList( > > PInteger.INSTANCE.getKeyRange(PInteger.INSTANCE.toBytes(1), true, > PInteger.INSTANCE.toBytes(4), true)), > Arrays.asList( > KeyRange.getKeyRange(PInteger.INSTANCE.toBytes(5)), > KeyRange.getKeyRange(PInteger.INSTANCE.toBytes(7))), > Arrays.asList( > > PInteger.INSTANCE.getKeyRange(PInteger.INSTANCE.toBytes(9), true, > PInteger.INSTANCE.toBytes(10), true)) > ); > > SkipScanFilter skipScanFilter=new > SkipScanFilter(rowKeyColumnRangesList, builder.build()); > > System.out.println(skipScanFilter); > > byte[] rowKey=ByteUtil.concat( > PInteger.INSTANCE.toBytes(2), > PInteger.INSTANCE.toBytes(7), > PInteger.INSTANCE.toBytes(11)); > KeyValue keyValue=KeyValue.createFirstOnRow(rowKey); > ReturnCode returnCode=skipScanFilter.filterKeyValue(keyValue); > assertTrue(returnCode == ReturnCode.SEEK_NEXT_USING_HINT); > Cell nextCellHint=skipScanFilter.getNextCellHint(keyValue); > > assertTrue(Bytes.toStringBinary(CellUtil.cloneRow(nextCellHint)).equals( > "\\x80\\x00\\x00\\x03\\x80\\x00\\x00\\x05\\x80\\x00\\x00\\x09")); > } > {code} > Let us see what's wrong, first column of rowKey [2,7,11 ] is 2, which is in > SkipScanFilter's first slot range [1-4], so position[0] is 0 and we go to the > second column 7, which match the second range [7] of SkipScanFilter's second > slot [5, 7],so position[1] is 1 and we go to the third column 11, which is > bigger than the third slot range [9 - 10],so position[2] is 0 and the > {{SkipScanFilter.ptr}} which points to current column still stay on the third > column. Now we begin to backtrack to second column, because the second range > [7] of SkipScanFilter's second slot is singleKey and there is no more > range,so position[1] is 0 and we continue to backtrack to first column, > because the first slot range [1-4] is not singleKey so we stop backtracking > at first column. > Now the problem comes, in following line 448 of {{SkipScanFilter.navigate}} > method,{{SkipScanFilter.setStartKey}} method is invoked,first copy rowKey > columns before {{SkipScanFilter.ptr}} to {{SkipScanFilter.startKey}}, because >
[jira] [Commented] (PHOENIX-3705) SkipScanFilter may repeatedly copy rowKey Columns to startKey
[ https://issues.apache.org/jira/browse/PHOENIX-3705?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15894274#comment-15894274 ] Hadoop QA commented on PHOENIX-3705: {color:red}-1 overall{color}. Here are the results of testing the latest attachment http://issues.apache.org/jira/secure/attachment/12855828/PHOENIX-3705_v2.patch against master branch at commit cf65fb27edf6291500e3f7e7549c4b83240f. ATTACHMENT ID: 12855828 {color:green}+1 @author{color}. The patch does not contain any @author tags. {color:green}+1 tests included{color}. The patch appears to include 3 new or modified tests. {color:green}+1 javac{color}. The applied patch does not increase the total number of javac compiler warnings. {color:red}-1 javadoc{color}. The javadoc tool appears to have generated 45 warning messages. {color:red}-1 release audit{color}. The applied patch generated 1 release audit warnings (more than the master's current 0 warnings). {color:red}-1 lineLengths{color}. The patch introduces the following lines longer than 100: +//for PHOENIX-3705, now ptr is still point to slot i, we must make ptr point to slot j+1, +//because following setStartKey method will copy rowKey columns before ptr to startKey and +//then copy the lower bound of slots from j+1, according to position array, so if we do not +//make ptr point to the first rowKey column of slot j,why we need slotSpan[j] because for Row Value Constructor(RVC), +//slot j may span multiple rowKey columns, so the length of ptr must consider the slotSpan[j]. +schema.iterator(startKey, minOffset, maxOffset, ptr, ScanUtil.getRowKeyPosition(slotSpan, j)+1,slotSpan[j]); +public Boolean iterator(byte[] src, int srcOffset, int srcLength, ImmutableBytesWritable ptr, int position,int extraColumnSpan) { +public Boolean iterator(byte[] src, int srcOffset, int srcLength, ImmutableBytesWritable ptr, int position) { +public SkipScanFilterTest(Listcnf, int[] widths, int[] slotSpans,List expectations) { + PInteger.INSTANCE.getKeyRange(PInteger.INSTANCE.toBytes(1), true, PInteger.INSTANCE.toBytes(4), true) {color:red}-1 core tests{color}. The patch failed these unit tests: ./phoenix-core/target/failsafe-reports/TEST-org.apache.phoenix.end2end.index.MutableIndexFailureIT Test results: https://builds.apache.org/job/PreCommit-PHOENIX-Build/791//testReport/ Release audit warnings: https://builds.apache.org/job/PreCommit-PHOENIX-Build/791//artifact/patchprocess/patchReleaseAuditWarnings.txt Javadoc warnings: https://builds.apache.org/job/PreCommit-PHOENIX-Build/791//artifact/patchprocess/patchJavadocWarnings.txt Console output: https://builds.apache.org/job/PreCommit-PHOENIX-Build/791//console This message is automatically generated. > SkipScanFilter may repeatedly copy rowKey Columns to startKey > - > > Key: PHOENIX-3705 > URL: https://issues.apache.org/jira/browse/PHOENIX-3705 > Project: Phoenix > Issue Type: Bug >Affects Versions: 4.9.0 >Reporter: chenglei >Priority: Blocker > Fix For: 4.10.0 > > Attachments: PHOENIX-3705_v2.patch > > > See following simple unit test first,the rowKey is composed of three PInteger > columns,and the slots of SkipScanFilter are: > [ [[1 - 4]], [5, 7], [[9 - 10]] ], > When SkipScanFilter.filterKeyValue method is invoked on a KeyValue whose > rowKey is [2,7,11], obviously SkipScanFilter.filterKeyValue > returns ReturnCode.SEEK_NEXT_USING_HINT and SkipScanFilter.getNextCellHint > returns [3,5,9], but unfortunately, SkipScanFilter.getNextCellHint actually > returns [2,8,5,9] , a very strange value, the unit tests failed. > {code} > @Test > public void testNavigate() { > RowKeySchemaBuilder builder = new RowKeySchemaBuilder(3); > for(int i=0;i<3;i++) { > builder.addField( > new PDatum() { > @Override > public boolean isNullable() { > return false; > } > @Override > public PDataType getDataType() { > return PInteger.INSTANCE; > } > @Override > public Integer getMaxLength() { > return PInteger.INSTANCE.getMaxLength(null); > } > @Override > public Integer getScale() { > return PInteger.INSTANCE.getScale(null); > } >
[jira] [Commented] (PHOENIX-3705) SkipScanFilter may repeatedly copy rowKey Columns to startKey
[ https://issues.apache.org/jira/browse/PHOENIX-3705?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15894167#comment-15894167 ] chenglei commented on PHOENIX-3705: --- I uploaded my new patch, with additional changes for RVC and more tests for RVC, please help me have a review. > SkipScanFilter may repeatedly copy rowKey Columns to startKey > - > > Key: PHOENIX-3705 > URL: https://issues.apache.org/jira/browse/PHOENIX-3705 > Project: Phoenix > Issue Type: Bug >Affects Versions: 4.9.0 >Reporter: chenglei >Priority: Blocker > Fix For: 4.10.0 > > Attachments: PHOENIX-3705_v2.patch > > > See following simple unit test first,the rowKey is composed of three PInteger > columns,and the slots of SkipScanFilter are: > [ [[1 - 4]], [5, 7], [[9 - 10]] ], > When SkipScanFilter.filterKeyValue method is invoked on a KeyValue whose > rowKey is [2,7,11], obviously SkipScanFilter.filterKeyValue > returns ReturnCode.SEEK_NEXT_USING_HINT and SkipScanFilter.getNextCellHint > returns [3,5,9], but unfortunately, SkipScanFilter.getNextCellHint actually > returns [2,8,5,9] , a very strange value, the unit tests failed. > {code} > @Test > public void testNavigate() { > RowKeySchemaBuilder builder = new RowKeySchemaBuilder(3); > for(int i=0;i<3;i++) { > builder.addField( > new PDatum() { > @Override > public boolean isNullable() { > return false; > } > @Override > public PDataType getDataType() { > return PInteger.INSTANCE; > } > @Override > public Integer getMaxLength() { > return PInteger.INSTANCE.getMaxLength(null); > } > @Override > public Integer getScale() { > return PInteger.INSTANCE.getScale(null); > } > @Override > public SortOrder getSortOrder() { > return SortOrder.getDefault(); > } > }, false, SortOrder.getDefault()); > } > > ListrowKeyColumnRangesList=Arrays.asList( > Arrays.asList( > > PInteger.INSTANCE.getKeyRange(PInteger.INSTANCE.toBytes(1), true, > PInteger.INSTANCE.toBytes(4), true)), > Arrays.asList( > KeyRange.getKeyRange(PInteger.INSTANCE.toBytes(5)), > KeyRange.getKeyRange(PInteger.INSTANCE.toBytes(7))), > Arrays.asList( > > PInteger.INSTANCE.getKeyRange(PInteger.INSTANCE.toBytes(9), true, > PInteger.INSTANCE.toBytes(10), true)) > ); > > SkipScanFilter skipScanFilter=new > SkipScanFilter(rowKeyColumnRangesList, builder.build()); > > System.out.println(skipScanFilter); > > byte[] rowKey=ByteUtil.concat( > PInteger.INSTANCE.toBytes(2), > PInteger.INSTANCE.toBytes(7), > PInteger.INSTANCE.toBytes(11)); > KeyValue keyValue=KeyValue.createFirstOnRow(rowKey); > ReturnCode returnCode=skipScanFilter.filterKeyValue(keyValue); > assertTrue(returnCode == ReturnCode.SEEK_NEXT_USING_HINT); > Cell nextCellHint=skipScanFilter.getNextCellHint(keyValue); > > assertTrue(Bytes.toStringBinary(CellUtil.cloneRow(nextCellHint)).equals( > "\\x80\\x00\\x00\\x03\\x80\\x00\\x00\\x05\\x80\\x00\\x00\\x09")); > } > {code} > Let us see what's wrong, first column of rowKey [2,7,11 ] is 2, which is in > SkipScanFilter's first slot range [1-4], so position[0] is 0 and we go to the > second column 7, which match the second range [7] of SkipScanFilter's second > slot [5, 7],so position[1] is 1 and we go to the third column 11, which is > bigger than the third slot range [9 - 10],so position[2] is 0 and the > {{SkipScanFilter.ptr}} which points to current column still stay on the third > column. Now we begin to backtrack to second column, because the second range > [7] of SkipScanFilter's second slot is singleKey and there is no more > range,so position[1] is 0 and we continue to backtrack to first column, > because the first slot range [1-4] is not singleKey so we stop backtracking > at first column. > Now the problem comes, in following line 448 of {{SkipScanFilter.navigate}} > method,{{SkipScanFilter.setStartKey}} method is invoked,first copy rowKey > columns before {{SkipScanFilter.ptr}} to
[jira] [Commented] (PHOENIX-3705) SkipScanFilter may repeatedly copy rowKey Columns to startKey
[ https://issues.apache.org/jira/browse/PHOENIX-3705?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15892629#comment-15892629 ] chenglei commented on PHOENIX-3705: --- I write following unit test for Row Value Constructor(RVC), ,the rowKey is composed of four PInteger columns,and the slots of SkipScanFilter are: [ [[(1,2) - (3,4)]], [5, 7], [[9 - 10]] ], the first slot is RVC,when SkipScanFilter.filterKeyValue method is invoked on a KeyValue whose rowKey is [2,3,7,11], SkipScanFilter.filterKeyValue should return ReturnCode.SEEK_NEXT_USING_HINT and SkipScanFilter.getNextCellHint should return [2,4,5,9]. {code} @Test public void testRVCSkipScanFilter() throws Exception { RowKeySchemaBuilder builder = new RowKeySchemaBuilder(4); for(int i=0;i<4;i++) { builder.addField( new PDatum() { @Override public boolean isNullable() { return false; } @Override public PDataType getDataType() { return PInteger.INSTANCE; } @Override public Integer getMaxLength() { return PInteger.INSTANCE.getMaxLength(null); } @Override public Integer getScale() { return PInteger.INSTANCE.getScale(null); } @Override public SortOrder getSortOrder() { return SortOrder.getDefault(); } }, false, SortOrder.getDefault()); } ListrowKeyColumnRangesList=Arrays.asList( Arrays.asList( RowKeyRange.createRowKeyRange( ByteUtil.concat(PInteger.INSTANCE.toBytes(1),PInteger.INSTANCE.toBytes(2)), true, ByteUtil.concat(PInteger.INSTANCE.toBytes(3),PInteger.INSTANCE.toBytes(4)), true)), Arrays.asList( RowKeyRange.createRowKeyRange(PInteger.INSTANCE.toBytes(5)), RowKeyRange.createRowKeyRange(PInteger.INSTANCE.toBytes(7))), Arrays.asList( PInteger.INSTANCE.getKeyRange(PInteger.INSTANCE.toBytes(9), true, PInteger.INSTANCE.toBytes(10), true)) ); SkipScanFilter skipScanFilter=new SkipScanFilter(rowKeyColumnRangesList, new int[]{1,0,0},builder.build()); byte[] rowKey=ByteUtil.concat( PInteger.INSTANCE.toBytes(2), PInteger.INSTANCE.toBytes(3), PInteger.INSTANCE.toBytes(7), PInteger.INSTANCE.toBytes(11)); KeyValue keyValue=KeyValue.createFirstOnRow(rowKey); ReturnCode returnCode=skipScanFilter.filterKeyValue(keyValue); assertTrue(returnCode == ReturnCode.SEEK_NEXT_USING_HINT); Cell nextCellHint=skipScanFilter.getNextCellHint(keyValue); assertTrue(Arrays.equals(CellUtil.cloneRow(nextCellHint), ByteUtil.concat( PInteger.INSTANCE.toBytes(2), PInteger.INSTANCE.toBytes(4), PInteger.INSTANCE.toBytes(5), PInteger.INSTANCE.toBytes(9)) )); } {code} Take the above unit test for example, we can not
[jira] [Commented] (PHOENIX-3705) SkipScanFilter may repeatedly copy rowKey Columns to startKey
[ https://issues.apache.org/jira/browse/PHOENIX-3705?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15891527#comment-15891527 ] James Taylor commented on PHOENIX-3705: --- Thanks, [~comnetwork]. Let me know what you find out about when a row value constructor is used. I'm not sure yet what the right thing to do is there. Really appreciate you digging into this - you're doing a great job! > SkipScanFilter may repeatedly copy rowKey Columns to startKey > - > > Key: PHOENIX-3705 > URL: https://issues.apache.org/jira/browse/PHOENIX-3705 > Project: Phoenix > Issue Type: Bug >Affects Versions: 4.9.0 >Reporter: chenglei >Priority: Blocker > Fix For: 4.10.0 > > Attachments: PHOENIX-3705_v1.patch > > > See following simple unit test first,the rowKey is composed of three PInteger > columns,and the slots of SkipScanFilter are: > [ [[1 - 4]], [5, 7], [[9 - 10]] ], > When SkipScanFilter.filterKeyValue method is invoked on a KeyValue whose > rowKey is [2,7,11], obviously SkipScanFilter.filterKeyValue > returns ReturnCode.SEEK_NEXT_USING_HINT and SkipScanFilter.getNextCellHint > returns [3,5,9], but unfortunately, SkipScanFilter.getNextCellHint actually > returns [2,8,5,9] , a very strange value, the unit tests failed. > {code} > @Test > public void testNavigate() { > RowKeySchemaBuilder builder = new RowKeySchemaBuilder(3); > for(int i=0;i<3;i++) { > builder.addField( > new PDatum() { > @Override > public boolean isNullable() { > return false; > } > @Override > public PDataType getDataType() { > return PInteger.INSTANCE; > } > @Override > public Integer getMaxLength() { > return PInteger.INSTANCE.getMaxLength(null); > } > @Override > public Integer getScale() { > return PInteger.INSTANCE.getScale(null); > } > @Override > public SortOrder getSortOrder() { > return SortOrder.getDefault(); > } > }, false, SortOrder.getDefault()); > } > > ListrowKeyColumnRangesList=Arrays.asList( > Arrays.asList( > > PInteger.INSTANCE.getKeyRange(PInteger.INSTANCE.toBytes(1), true, > PInteger.INSTANCE.toBytes(4), true)), > Arrays.asList( > KeyRange.getKeyRange(PInteger.INSTANCE.toBytes(5)), > KeyRange.getKeyRange(PInteger.INSTANCE.toBytes(7))), > Arrays.asList( > > PInteger.INSTANCE.getKeyRange(PInteger.INSTANCE.toBytes(9), true, > PInteger.INSTANCE.toBytes(10), true)) > ); > > SkipScanFilter skipScanFilter=new > SkipScanFilter(rowKeyColumnRangesList, builder.build()); > > System.out.println(skipScanFilter); > > byte[] rowKey=ByteUtil.concat( > PInteger.INSTANCE.toBytes(2), > PInteger.INSTANCE.toBytes(7), > PInteger.INSTANCE.toBytes(11)); > KeyValue keyValue=KeyValue.createFirstOnRow(rowKey); > ReturnCode returnCode=skipScanFilter.filterKeyValue(keyValue); > assertTrue(returnCode == ReturnCode.SEEK_NEXT_USING_HINT); > Cell nextCellHint=skipScanFilter.getNextCellHint(keyValue); > > assertTrue(Bytes.toStringBinary(CellUtil.cloneRow(nextCellHint)).equals( > "\\x80\\x00\\x00\\x03\\x80\\x00\\x00\\x05\\x80\\x00\\x00\\x09")); > } > {code} > Let us see what's wrong, first column of rowKey [2,7,11 ] is 2, which is in > SkipScanFilter's first slot range [1-4], so position[0] is 0 and we go to the > second column 7, which match the second range [7] of SkipScanFilter's second > slot [5, 7],so position[1] is 1 and we go to the third column 11, which is > bigger than the third slot range [9 - 10],so position[2] is 0 and the > {{SkipScanFilter.ptr}} which points to current column still stay on the third > column. Now we begin to backtrack to second column, because the second range > [7] of SkipScanFilter's second slot is singleKey and there is no more > range,so position[1] is 0 and we continue to backtrack to first column, > because the first slot range [1-4] is not singleKey so we stop backtracking > at first column. > Now the problem comes, in following line 448 of {{SkipScanFilter.navigate}} >
[jira] [Commented] (PHOENIX-3705) SkipScanFilter may repeatedly copy rowKey Columns to startKey
[ https://issues.apache.org/jira/browse/PHOENIX-3705?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15891484#comment-15891484 ] chenglei commented on PHOENIX-3705: --- Thanks for review, [~jamestaylor],I will add more tests and modify following your suggestion. > SkipScanFilter may repeatedly copy rowKey Columns to startKey > - > > Key: PHOENIX-3705 > URL: https://issues.apache.org/jira/browse/PHOENIX-3705 > Project: Phoenix > Issue Type: Bug >Affects Versions: 4.9.0 >Reporter: chenglei >Priority: Blocker > Fix For: 4.10.0 > > Attachments: PHOENIX-3705_v1.patch > > > See following simple unit test first,the rowKey is composed of three PInteger > columns,and the slots of SkipScanFilter are: > [ [[1 - 4]], [5, 7], [[9 - 10]] ], > When SkipScanFilter.filterKeyValue method is invoked on a KeyValue whose > rowKey is [2,7,11], obviously SkipScanFilter.filterKeyValue > returns ReturnCode.SEEK_NEXT_USING_HINT and SkipScanFilter.getNextCellHint > returns [3,5,9], but unfortunately, SkipScanFilter.getNextCellHint actually > returns [2,8,5,9] , a very strange value, the unit tests failed. > {code} > @Test > public void testNavigate() { > RowKeySchemaBuilder builder = new RowKeySchemaBuilder(3); > for(int i=0;i<3;i++) { > builder.addField( > new PDatum() { > @Override > public boolean isNullable() { > return false; > } > @Override > public PDataType getDataType() { > return PInteger.INSTANCE; > } > @Override > public Integer getMaxLength() { > return PInteger.INSTANCE.getMaxLength(null); > } > @Override > public Integer getScale() { > return PInteger.INSTANCE.getScale(null); > } > @Override > public SortOrder getSortOrder() { > return SortOrder.getDefault(); > } > }, false, SortOrder.getDefault()); > } > > ListrowKeyColumnRangesList=Arrays.asList( > Arrays.asList( > > PInteger.INSTANCE.getKeyRange(PInteger.INSTANCE.toBytes(1), true, > PInteger.INSTANCE.toBytes(4), true)), > Arrays.asList( > KeyRange.getKeyRange(PInteger.INSTANCE.toBytes(5)), > KeyRange.getKeyRange(PInteger.INSTANCE.toBytes(7))), > Arrays.asList( > > PInteger.INSTANCE.getKeyRange(PInteger.INSTANCE.toBytes(9), true, > PInteger.INSTANCE.toBytes(10), true)) > ); > > SkipScanFilter skipScanFilter=new > SkipScanFilter(rowKeyColumnRangesList, builder.build()); > > System.out.println(skipScanFilter); > > byte[] rowKey=ByteUtil.concat( > PInteger.INSTANCE.toBytes(2), > PInteger.INSTANCE.toBytes(7), > PInteger.INSTANCE.toBytes(11)); > KeyValue keyValue=KeyValue.createFirstOnRow(rowKey); > ReturnCode returnCode=skipScanFilter.filterKeyValue(keyValue); > assertTrue(returnCode == ReturnCode.SEEK_NEXT_USING_HINT); > Cell nextCellHint=skipScanFilter.getNextCellHint(keyValue); > > assertTrue(Bytes.toStringBinary(CellUtil.cloneRow(nextCellHint)).equals( > "\\x80\\x00\\x00\\x03\\x80\\x00\\x00\\x05\\x80\\x00\\x00\\x09")); > } > {code} > Let us see what's wrong, first column of rowKey [2,7,11 ] is 2, which is in > SkipScanFilter's first slot range [1-4], so position[0] is 0 and we go to the > second column 7, which match the second range [7] of SkipScanFilter's second > slot [5, 7],so position[1] is 1 and we go to the third column 11, which is > bigger than the third slot range [9 - 10],so position[2] is 0 and the > {{SkipScanFilter.ptr}} which points to current column still stay on the third > column. Now we begin to backtrack to second column, because the second range > [7] of SkipScanFilter's second slot is singleKey and there is no more > range,so position[1] is 0 and we continue to backtrack to first column, > because the first slot range [1-4] is not singleKey so we stop backtracking > at first column. > Now the problem comes, in following line 448 of {{SkipScanFilter.navigate}} > method,{{SkipScanFilter.setStartKey}} method is invoked,first copy rowKey > columns before {{SkipScanFilter.ptr}} to {{SkipScanFilter.startKey}}, because
[jira] [Commented] (PHOENIX-3705) SkipScanFilter may repeatedly copy rowKey Columns to startKey
[ https://issues.apache.org/jira/browse/PHOENIX-3705?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15890854#comment-15890854 ] James Taylor commented on PHOENIX-3705: --- Very nice, [~comnetwork]. It seems like this bug could affect the final query result, perhaps skipping rows it shouldn't depending on the data, no? Anytime the seek next hint is wrong, I think this could be the case. Small question on whether perhaps one more change is required. I noticed you used {{ScanUtil.getRowKeyPosition(slotSpan, j + 1)}} below in the call to reposition, but further down (in existing code), we use {{ScanUtil.getRowKeyPosition(slotSpan, j)+1}}. Should the latter be changed as you've coded? Would be good to have a test case that uses Row Value Constructor (i.e. has a slot span) to confirm: {code} schema.reposition( ptr, ScanUtil.getRowKeyPosition(slotSpan, i), ScanUtil.getRowKeyPosition(slotSpan, j + 1), minOffset, maxOffset, slotSpan[j + 1]); int currentLength = setStartKey(ptr, minOffset, j+1, nSlots, false); // From here on, we use startKey as our buffer (resetting minOffset and maxOffset) // We've copied the part of the current key above that we need into startKey // Reinitialize the iterator to be positioned at previous slot position minOffset = 0; maxOffset = startKeyLength; schema.iterator(startKey, minOffset, maxOffset, ptr, ScanUtil.getRowKeyPosition(slotSpan, j)+1); {code} > SkipScanFilter may repeatedly copy rowKey Columns to startKey > - > > Key: PHOENIX-3705 > URL: https://issues.apache.org/jira/browse/PHOENIX-3705 > Project: Phoenix > Issue Type: Bug >Affects Versions: 4.9.0 >Reporter: chenglei > Attachments: PHOENIX-3705_v1.patch > > > See following simple unit test first,the rowKey is composed of three PInteger > columns,and the slots of SkipScanFilter are: > [ [[1 - 4]], [5, 7], [[9 - 10]] ], > When SkipScanFilter.filterKeyValue method is invoked on a KeyValue whose > rowKey is [2,7,11], obviously SkipScanFilter.filterKeyValue > returns ReturnCode.SEEK_NEXT_USING_HINT and SkipScanFilter.getNextCellHint > returns [3,5,9], but unfortunately, SkipScanFilter.getNextCellHint actually > returns [2,8,5,9] , a very strange value, the unit tests failed. > {code} > @Test > public void testNavigate() { > RowKeySchemaBuilder builder = new RowKeySchemaBuilder(3); > for(int i=0;i<3;i++) { > builder.addField( > new PDatum() { > @Override > public boolean isNullable() { > return false; > } > @Override > public PDataType getDataType() { > return PInteger.INSTANCE; > } > @Override > public Integer getMaxLength() { > return PInteger.INSTANCE.getMaxLength(null); > } > @Override > public Integer getScale() { > return PInteger.INSTANCE.getScale(null); > } > @Override > public SortOrder getSortOrder() { > return SortOrder.getDefault(); > } > }, false, SortOrder.getDefault()); > } > > ListrowKeyColumnRangesList=Arrays.asList( > Arrays.asList( > > PInteger.INSTANCE.getKeyRange(PInteger.INSTANCE.toBytes(1), true, > PInteger.INSTANCE.toBytes(4), true)), > Arrays.asList( > KeyRange.getKeyRange(PInteger.INSTANCE.toBytes(5)), > KeyRange.getKeyRange(PInteger.INSTANCE.toBytes(7))), > Arrays.asList( > > PInteger.INSTANCE.getKeyRange(PInteger.INSTANCE.toBytes(9), true, > PInteger.INSTANCE.toBytes(10), true)) > ); > > SkipScanFilter skipScanFilter=new > SkipScanFilter(rowKeyColumnRangesList, builder.build()); > > System.out.println(skipScanFilter); > > byte[] rowKey=ByteUtil.concat( > PInteger.INSTANCE.toBytes(2), > PInteger.INSTANCE.toBytes(7), > PInteger.INSTANCE.toBytes(11)); > KeyValue keyValue=KeyValue.createFirstOnRow(rowKey); >
[jira] [Commented] (PHOENIX-3705) SkipScanFilter may repeatedly copy rowKey Columns to startKey
[ https://issues.apache.org/jira/browse/PHOENIX-3705?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15889941#comment-15889941 ] chenglei commented on PHOENIX-3705: --- I uploaded my first patch, please help me have a review. > SkipScanFilter may repeatedly copy rowKey Columns to startKey > - > > Key: PHOENIX-3705 > URL: https://issues.apache.org/jira/browse/PHOENIX-3705 > Project: Phoenix > Issue Type: Bug >Affects Versions: 4.9.0 >Reporter: chenglei > Attachments: PHOENIX-3705_v1.patch > > > See following simple unit test first,the rowKey is composed of three PInteger > columns,and the slots of SkipScanFilter are: > [ [[1 - 4]], [5, 7], [[9 - 10]] ], > When SkipScanFilter.filterKeyValue method is invoked on a KeyValue whose > rowKey is [2,7,11], obviously SkipScanFilter.filterKeyValue > returns ReturnCode.SEEK_NEXT_USING_HINT and SkipScanFilter.getNextCellHint > returns [3,5,9], but unfortunately, SkipScanFilter.getNextCellHint actually > returns [2,8,5,9] , a very strange value, the unit tests failed. > {code} > @Test > public void testNavigate() { > RowKeySchemaBuilder builder = new RowKeySchemaBuilder(3); > for(int i=0;i<3;i++) { > builder.addField( > new PDatum() { > @Override > public boolean isNullable() { > return false; > } > @Override > public PDataType getDataType() { > return PInteger.INSTANCE; > } > @Override > public Integer getMaxLength() { > return PInteger.INSTANCE.getMaxLength(null); > } > @Override > public Integer getScale() { > return PInteger.INSTANCE.getScale(null); > } > @Override > public SortOrder getSortOrder() { > return SortOrder.getDefault(); > } > }, false, SortOrder.getDefault()); > } > > ListrowKeyColumnRangesList=Arrays.asList( > Arrays.asList( > > PInteger.INSTANCE.getKeyRange(PInteger.INSTANCE.toBytes(1), true, > PInteger.INSTANCE.toBytes(4), true)), > Arrays.asList( > KeyRange.getKeyRange(PInteger.INSTANCE.toBytes(5)), > KeyRange.getKeyRange(PInteger.INSTANCE.toBytes(7))), > Arrays.asList( > > PInteger.INSTANCE.getKeyRange(PInteger.INSTANCE.toBytes(9), true, > PInteger.INSTANCE.toBytes(10), true)) > ); > > SkipScanFilter skipScanFilter=new > SkipScanFilter(rowKeyColumnRangesList, builder.build()); > > System.out.println(skipScanFilter); > > byte[] rowKey=ByteUtil.concat( > PInteger.INSTANCE.toBytes(2), > PInteger.INSTANCE.toBytes(7), > PInteger.INSTANCE.toBytes(11)); > KeyValue keyValue=KeyValue.createFirstOnRow(rowKey); > ReturnCode returnCode=skipScanFilter.filterKeyValue(keyValue); > assertTrue(returnCode == ReturnCode.SEEK_NEXT_USING_HINT); > Cell nextCellHint=skipScanFilter.getNextCellHint(keyValue); > > assertTrue(Bytes.toStringBinary(CellUtil.cloneRow(nextCellHint)).equals( > "\\x80\\x00\\x00\\x03\\x80\\x00\\x00\\x05\\x80\\x00\\x00\\x09")); > } > {code} > Let us see what's wrong, first column of rowKey [2,7,11 ] is 2, which is in > SkipScanFilter's first slot range [1-4], so position[0] is 0 and we go to the > second column 7, which match the second range [7] of SkipScanFilter's second > slot [5, 7],so position[1] is 1 and we go to the third column 11, which is > bigger than the third slot range [9 - 10],so position[2] is 0 and we begin to > backtrack to second column, because the second range [7] of SkipScanFilter's > second slot is singleKey and there is no more range,so position[1] is 0 and > we continue to backtrack to first column, because the first slot range [1-4] > is not singleKey so we stop backtracking at first column. > Now the problem comes, in following line 448 of SkipScanFilter.navigate > method,SkipScanFilter.setStartKey method is invoked,first copy rowKey columns > before ptr to SkipScanFilter.startKey, because now ptr still point to the > third column, so copy the first and second columns to > SkipScanFilter.startKey,SkipScanFilter.startKey is [2,7] after this step , > then setStartKey method copy the lower bound SkipScanFilter.slots