[jira] [Commented] (PHOENIX-3392) SortMergeJoinIT#testSubJoin[0] is failing with encodecolumns2 branch
[ https://issues.apache.org/jira/browse/PHOENIX-3392?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16034016#comment-16034016 ] Samarth Jain commented on PHOENIX-3392: --- Thanks for reporting this issue, [~allen_xu]. I have filed PHOENIX-3904. > SortMergeJoinIT#testSubJoin[0] is failing with encodecolumns2 branch > > > Key: PHOENIX-3392 > URL: https://issues.apache.org/jira/browse/PHOENIX-3392 > Project: Phoenix > Issue Type: Bug >Reporter: James Taylor >Assignee: Samarth Jain > Fix For: 4.10.0 > > > The SortMergeJoinIT#testSubJoin[0] is failing with encodecolumns2 branch. To > repro, checkout the encodecolumns2 branch and run the SortMergeJoinIT tests. > Here's the stack trace is here: > https://builds.apache.org/job/Phoenix-encode-columns/16/testReport/org.apache.phoenix.end2end/SortMergeJoinIT/testSubJoin_0_/ > The basic idea of column encoding over mutable tables is that we take control > of the column qualifier name, storing in PColumn the mapping of real column > name to column qualifier name. We use serialized integers as the column > qualifier names so that we can do positional lookups into the List we > get back from HBase APIs. There are a few "reserved" column qualifiers for > things like our empty key value, etc. -- This message was sent by Atlassian JIRA (v6.3.15#6346)
[jira] [Commented] (PHOENIX-3392) SortMergeJoinIT#testSubJoin[0] is failing with encodecolumns2 branch
[ https://issues.apache.org/jira/browse/PHOENIX-3392?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16032594#comment-16032594 ] Allen Xu commented on PHOENIX-3392: --- I have encountered this "qualifier out of range problem" with query like select * from myns.test where "cf"."col1" like '%name2%' limit 10 offset 10 Please check stackoverflow: https://stackoverflow.com/questions/44214659/qualifier-out-of-valid-range-error-in-apache-phoenix-select-with-where-and-of Thanks. > SortMergeJoinIT#testSubJoin[0] is failing with encodecolumns2 branch > > > Key: PHOENIX-3392 > URL: https://issues.apache.org/jira/browse/PHOENIX-3392 > Project: Phoenix > Issue Type: Bug >Reporter: James Taylor > > The SortMergeJoinIT#testSubJoin[0] is failing with encodecolumns2 branch. To > repro, checkout the encodecolumns2 branch and run the SortMergeJoinIT tests. > Here's the stack trace is here: > https://builds.apache.org/job/Phoenix-encode-columns/16/testReport/org.apache.phoenix.end2end/SortMergeJoinIT/testSubJoin_0_/ > The basic idea of column encoding over mutable tables is that we take control > of the column qualifier name, storing in PColumn the mapping of real column > name to column qualifier name. We use serialized integers as the column > qualifier names so that we can do positional lookups into the List we > get back from HBase APIs. There are a few "reserved" column qualifiers for > things like our empty key value, etc. -- This message was sent by Atlassian JIRA (v6.3.15#6346)
[jira] [Commented] (PHOENIX-3392) SortMergeJoinIT#testSubJoin[0] is failing with encodecolumns2 branch
[ https://issues.apache.org/jira/browse/PHOENIX-3392?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15596815#comment-15596815 ] Samarth Jain commented on PHOENIX-3392: --- Sorry, didn't update the issue but I did talk to Maryann offline to let her know that I have figured out the issue. > SortMergeJoinIT#testSubJoin[0] is failing with encodecolumns2 branch > > > Key: PHOENIX-3392 > URL: https://issues.apache.org/jira/browse/PHOENIX-3392 > Project: Phoenix > Issue Type: Bug >Reporter: James Taylor > > The SortMergeJoinIT#testSubJoin[0] is failing with encodecolumns2 branch. To > repro, checkout the encodecolumns2 branch and run the SortMergeJoinIT tests. > Here's the stack trace is here: > https://builds.apache.org/job/Phoenix-encode-columns/16/testReport/org.apache.phoenix.end2end/SortMergeJoinIT/testSubJoin_0_/ > The basic idea of column encoding over mutable tables is that we take control > of the column qualifier name, storing in PColumn the mapping of real column > name to column qualifier name. We use serialized integers as the column > qualifier names so that we can do positional lookups into the List we > get back from HBase APIs. There are a few "reserved" column qualifiers for > things like our empty key value, etc. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (PHOENIX-3392) SortMergeJoinIT#testSubJoin[0] is failing with encodecolumns2 branch
[ https://issues.apache.org/jira/browse/PHOENIX-3392?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15596804#comment-15596804 ] James Taylor commented on PHOENIX-3392: --- [~maryannxue] - looks like [~samarthjain] figured out the issue. Hopefully you haven't spent time on it already. > SortMergeJoinIT#testSubJoin[0] is failing with encodecolumns2 branch > > > Key: PHOENIX-3392 > URL: https://issues.apache.org/jira/browse/PHOENIX-3392 > Project: Phoenix > Issue Type: Bug >Reporter: James Taylor > > The SortMergeJoinIT#testSubJoin[0] is failing with encodecolumns2 branch. To > repro, checkout the encodecolumns2 branch and run the SortMergeJoinIT tests. > Here's the stack trace is here: > https://builds.apache.org/job/Phoenix-encode-columns/16/testReport/org.apache.phoenix.end2end/SortMergeJoinIT/testSubJoin_0_/ > The basic idea of column encoding over mutable tables is that we take control > of the column qualifier name, storing in PColumn the mapping of real column > name to column qualifier name. We use serialized integers as the column > qualifier names so that we can do positional lookups into the List we > get back from HBase APIs. There are a few "reserved" column qualifiers for > things like our empty key value, etc. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (PHOENIX-3392) SortMergeJoinIT#testSubJoin[0] is failing with encodecolumns2 branch
[
https://issues.apache.org/jira/browse/PHOENIX-3392?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15592844#comment-15592844
]
Samarth Jain commented on PHOENIX-3392:
---
Thanks for taking a look, [~maryannxue]. The general idea behind the encoding
scheme, to save space and to optimize key value look up, is to use number based
column qualifiers. When constructing the scan object, we figure out what the
range of column qualifiers that we need to project in the scan. This is done in
the following method in BaseResultIterators
{code}
private static Pair getMinMaxQualifiers(Scan scan,
StatementContext context)
{code}
This range is then set in the BaseResultIterators.initializeScan method like
this:
{code}
if (setMinMaxQualifiersOnScan(table)) {
Pair minMaxQualifiers =
getMinMaxQualifiers(scan, context);
if (minMaxQualifiers != null) {
scan.setAttribute(BaseScannerRegionObserver.MIN_QUALIFIER,
PInteger.INSTANCE.toBytes(minMaxQualifiers.getFirst()));
scan.setAttribute(BaseScannerRegionObserver.MAX_QUALIFIER,
PInteger.INSTANCE.toBytes(minMaxQualifiers.getSecond()));
}
}
{code}
On the server side, we use a custom list implementation called
EncodedColumnQualifierCellsList. Using this list, we can do an O(1) look up for
a particular key value by using the number based column qualifier as an index
in the list. The range that we set on the scan object on the client side is
then used to appropriately size the EncodedColumnQualifierCellsList.
In this particular test case, it looks like that the way we are determining the
range of column qualifiers is incorrect. The server side scanner is returning
key value with qualifier 12 even though we expect the largest qualifier number
to be returned to be 11. One thing to note here is that we reserve the range
(0, 10) for phoenix internal column qualifiers like the empty column's
qualifier, result column qualifier, etc. So in this case our client side code
determined that there was only one column qualifier to be projected whose
qualifier name was 11.
{code}
Caused by: java.lang.IndexOutOfBoundsException: Qualifier 12 is out of the
valid range. Reserved: (0, 10). Table column qualifier range: (11, 11)
{code}
> SortMergeJoinIT#testSubJoin[0] is failing with encodecolumns2 branch
>
>
> Key: PHOENIX-3392
> URL: https://issues.apache.org/jira/browse/PHOENIX-3392
> Project: Phoenix
> Issue Type: Bug
>Reporter: James Taylor
>
> The SortMergeJoinIT#testSubJoin[0] is failing with encodecolumns2 branch. To
> repro, checkout the encodecolumns2 branch and run the SortMergeJoinIT tests.
> Here's the stack trace is here:
> https://builds.apache.org/job/Phoenix-encode-columns/16/testReport/org.apache.phoenix.end2end/SortMergeJoinIT/testSubJoin_0_/
> The basic idea of column encoding over mutable tables is that we take control
> of the column qualifier name, storing in PColumn the mapping of real column
> name to column qualifier name. We use serialized integers as the column
> qualifier names so that we can do positional lookups into the List we
> get back from HBase APIs. There are a few "reserved" column qualifiers for
> things like our empty key value, etc.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Commented] (PHOENIX-3392) SortMergeJoinIT#testSubJoin[0] is failing with encodecolumns2 branch
[ https://issues.apache.org/jira/browse/PHOENIX-3392?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15592779#comment-15592779 ] Maryann Xue commented on PHOENIX-3392: -- [~samarthjain], would you mind giving me a quick overview, like any special handling done for joins, and what would be the most likely cause of this exception? > SortMergeJoinIT#testSubJoin[0] is failing with encodecolumns2 branch > > > Key: PHOENIX-3392 > URL: https://issues.apache.org/jira/browse/PHOENIX-3392 > Project: Phoenix > Issue Type: Bug >Reporter: James Taylor > > The SortMergeJoinIT#testSubJoin[0] is failing with encodecolumns2 branch. To > repro, checkout the encodecolumns2 branch and run the SortMergeJoinIT tests. > Here's the stack trace is here: > https://builds.apache.org/job/Phoenix-encode-columns/16/testReport/org.apache.phoenix.end2end/SortMergeJoinIT/testSubJoin_0_/ > The basic idea of column encoding over mutable tables is that we take control > of the column qualifier name, storing in PColumn the mapping of real column > name to column qualifier name. We use serialized integers as the column > qualifier names so that we can do positional lookups into the List we > get back from HBase APIs. There are a few "reserved" column qualifiers for > things like our empty key value, etc. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (PHOENIX-3392) SortMergeJoinIT#testSubJoin[0] is failing with encodecolumns2 branch
[ https://issues.apache.org/jira/browse/PHOENIX-3392?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15592769#comment-15592769 ] Maryann Xue commented on PHOENIX-3392: -- Sure, I will. > SortMergeJoinIT#testSubJoin[0] is failing with encodecolumns2 branch > > > Key: PHOENIX-3392 > URL: https://issues.apache.org/jira/browse/PHOENIX-3392 > Project: Phoenix > Issue Type: Bug >Reporter: James Taylor > > The SortMergeJoinIT#testSubJoin[0] is failing with encodecolumns2 branch. To > repro, checkout the encodecolumns2 branch and run the SortMergeJoinIT tests. > Here's the stack trace is here: > https://builds.apache.org/job/Phoenix-encode-columns/16/testReport/org.apache.phoenix.end2end/SortMergeJoinIT/testSubJoin_0_/ > The basic idea of column encoding over mutable tables is that we take control > of the column qualifier name, storing in PColumn the mapping of real column > name to column qualifier name. We use serialized integers as the column > qualifier names so that we can do positional lookups into the List we > get back from HBase APIs. There are a few "reserved" column qualifiers for > things like our empty key value, etc. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (PHOENIX-3392) SortMergeJoinIT#testSubJoin[0] is failing with encodecolumns2 branch
[ https://issues.apache.org/jira/browse/PHOENIX-3392?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15592629#comment-15592629 ] James Taylor commented on PHOENIX-3392: --- Do you think you'd have any spare cycles to take a look, [~maryannxue]? This is the last failing test on the encodecolumns2 branch. [~samarthjain] will be able to answer any detailed questions you may have. > SortMergeJoinIT#testSubJoin[0] is failing with encodecolumns2 branch > > > Key: PHOENIX-3392 > URL: https://issues.apache.org/jira/browse/PHOENIX-3392 > Project: Phoenix > Issue Type: Bug >Reporter: James Taylor > > The SortMergeJoinIT#testSubJoin[0] is failing with encodecolumns2 branch. To > repro, checkout the encodecolumns2 branch and run the SortMergeJoinIT tests. > Here's the stack trace is here: > https://builds.apache.org/job/Phoenix-encode-columns/16/testReport/org.apache.phoenix.end2end/SortMergeJoinIT/testSubJoin_0_/ > The basic idea of column encoding over mutable tables is that we take control > of the column qualifier name, storing in PColumn the mapping of real column > name to column qualifier name. We use serialized integers as the column > qualifier names so that we can do positional lookups into the List we > get back from HBase APIs. There are a few "reserved" column qualifiers for > things like our empty key value, etc. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
