Re: Error while attempting join query

[Deepak Gopalakrishnan]

Adding Dev group

On Wed, Apr 6, 2016 at 10:35 AM, Deepak Gopalakrishnan 
wrote:

> Hello All,
>
> I did move on to 4.7 and my HBase version is 0.98+. Below is the script I
> execute in order to get it installed before HBase is started.
>
>
>
> After doing this on all nodes ( including master ), when I try a
> ./sqline.py localhost, it just freezes and doesnt proceed. Can someone
> please help ?
>
>
> #!/bin/sh
> mkdir /home/hadoop/phoenix/
> wget
> http://mirror.fibergrid.in/apache/phoenix/phoenix-4.7.0-HBase-0.98/bin/phoenix-4.7.0-HBase-0.98-bin.tar.gz
> -P /home/hadoop/phoenix/
> cd /home/hadoop/phoenix/
> tar -xvf phoenix-4.7.0-HBase-0.98-bin.tar.gz
> cp -r phoenix-4.7.0-HBase-0.98-bin/phoenix-*.jar /home/hadoop/hbase/lib/
>
> On Tue, Apr 5, 2016 at 10:39 PM, Deepak Gopalakrishnan 
> wrote:
>
>> Hello James,
>>
>> Sure I understand. But any idea why JOIN fails on the server ? Is it not
>> supported on 2.1.2 ?
>>
>> Thanks
>> Deepak
>>
>> On Tue, Apr 5, 2016 at 9:48 PM, James Taylor 
>> wrote:
>>
>>> Support for the 2.x line was dropped over a year ago. I'd recommend
>>> encouraging the EMR folks to upgrade to the latest version of Phoenix and
>>> HBase.
>>>
>>>
>>> On Tuesday, April 5, 2016, Deepak Gopalakrishnan 
>>> wrote:
>>>
 Hello,

 I'm trying to join two tables that I created using Phoenix. My tables
 are A and B. I've setup phoenix on EMR using
 https://phoenix.apache.org/phoenix_on_emr.html. I've used the
 phoenix-2.1.2 version and I can do basic queries like SELECT. But when I
 try a join, I get an error. It looks like the SQL query is not even being
 parsed. I'm not sure what I'm doing here. I'm using the sqline client on
 the server itself.

 select A.name,B.dname from A as a inner join B as b on a.name=b.name;

 Error: ERROR 602 (42P00): Syntax error. Missing "EOF" at line 1, column
 35. (state=42P00,code=602)
 Please let me know if you guys can help ?

 --
 Regards,
 *Deepak Gopalakrishnan*
 *Mobile*:+918891509774
 *Skype* : deepakgk87
 http://myexps.blogspot.com


>>
>>
>> --
>> Regards,
>> *Deepak Gopalakrishnan*
>> *Mobile*:+918891509774
>> *Skype* : deepakgk87
>> http://myexps.blogspot.com
>>
>>
>
>
> --
> Regards,
> *Deepak Gopalakrishnan*
> *Mobile*:+918891509774
> *Skype* : deepakgk87
> http://myexps.blogspot.com
>
>


-- 
Regards,
*Deepak Gopalakrishnan*
*Mobile*:+918891509774
*Skype* : deepakgk87
http://myexps.blogspot.com

[jira] [Commented] (PHOENIX-2722) support mysql "limit,offset" clauses

[ASF GitHub Bot (JIRA)]

[ 
https://issues.apache.org/jira/browse/PHOENIX-2722?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15227641#comment-15227641
 ] 

ASF GitHub Bot commented on PHOENIX-2722:
-

Github user JamesRTaylor commented on a diff in the pull request:

https://github.com/apache/phoenix/pull/154#discussion_r58646326
  
--- Diff: 
phoenix-core/src/main/java/org/apache/phoenix/compile/SubselectRewriter.java ---
@@ -188,6 +190,13 @@ private SelectStatement flatten(SelectStatement 
select, SelectStatement subselec
 }
 }
 
+OffsetNode offset = select.getOffset();
+if (offsetRewrite != null || (limitRewrite != null & offset != 
null)) {
--- End diff --

You're right, @ankitsinghal - won't have an impact.


> support mysql "limit,offset" clauses 
> -
>
> Key: PHOENIX-2722
> URL: https://issues.apache.org/jira/browse/PHOENIX-2722
> Project: Phoenix
>  Issue Type: New Feature
>Reporter: Ankit Singhal
>Assignee: Ankit Singhal
>Priority: Minor
> Attachments: PHOENIX-2722.patch, PHOENIX-2722_formatted.patch
>
>
> For serial query(query with “serial" hint or  “limit" without "order by”), we 
> can limit each scan(using page filter) to “limit+offset” instead of limit 
> earlier.
> And then, for all queries, we can forward the relevant client iterators to 
> the offset provided and then return the result.
> syntax
> {code}
> [ LIMIT { count } ]
> [ OFFSET start [ ROW | ROWS ] ]
> [ FETCH { FIRST | NEXT } [ count ] { ROW | ROWS } ONLY ]
> {code}
> Some new keywords(OFFSET,FETCH,ROW, ROWS,ONLY) are getting introduced so 
> users might need to see that they are not using them as column name or 
> something.
> WDYT, [~jamestaylor]



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[GitHub] phoenix pull request: PHOENIX-2722 support mysql limit,offset clau...

[JamesRTaylor]

Github user JamesRTaylor commented on a diff in the pull request:

https://github.com/apache/phoenix/pull/154#discussion_r58646326
  
--- Diff: 
phoenix-core/src/main/java/org/apache/phoenix/compile/SubselectRewriter.java ---
@@ -188,6 +190,13 @@ private SelectStatement flatten(SelectStatement 
select, SelectStatement subselec
 }
 }
 
+OffsetNode offset = select.getOffset();
+if (offsetRewrite != null || (limitRewrite != null & offset != 
null)) {
--- End diff --

You're right, @ankitsinghal - won't have an impact.


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

[jira] [Commented] (PHOENIX-2628) Ensure split when iterating through results handled correctly

[ASF GitHub Bot (JIRA)]

[ 
https://issues.apache.org/jira/browse/PHOENIX-2628?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15227561#comment-15227561
 ] 

ASF GitHub Bot commented on PHOENIX-2628:
-

Github user chrajeshbabu commented on a diff in the pull request:

https://github.com/apache/phoenix/pull/156#discussion_r58642550
  
--- Diff: 
phoenix-core/src/main/java/org/apache/phoenix/iterate/BaseResultIterators.java 
---
@@ -556,35 +564,55 @@ private static String toString(List gps) {
 } else {
 endKey = regionBoundaries.get(regionIndex);
 }
-HRegionLocation regionLocation = 
regionLocations.get(regionIndex);
-if (isLocalIndex) {
-HRegionInfo regionInfo = 
regionLocation.getRegionInfo();
-endRegionKey = regionInfo.getEndKey();
-keyOffset = 
ScanUtil.getRowKeyOffset(regionInfo.getStartKey(), endRegionKey);
-}
-try {
-while (guideIndex < gpsSize && 
(currentGuidePost.compareTo(endKey) <= 0 || endKey.length == 0)) {
-Scan newScan = scanRanges.intersectScan(scan, 
currentKeyBytes, currentGuidePostBytes, keyOffset,
-false);
-estimatedRows += 
gps.getRowCounts().get(guideIndex);
-estimatedSize += 
gps.getByteCounts().get(guideIndex);
-scans = addNewScan(parallelScans, scans, newScan, 
currentGuidePostBytes, false, regionLocation);
-currentKeyBytes = currentGuidePost.copyBytes();
-currentGuidePost = PrefixByteCodec.decode(decoder, 
input);
-currentGuidePostBytes = 
currentGuidePost.copyBytes();
-guideIndex++;
-}
-} catch (EOFException e) {}
-Scan newScan = scanRanges.intersectScan(scan, 
currentKeyBytes, endKey, keyOffset, true);
-if (isLocalIndex) {
-if (newScan != null) {
-newScan.setAttribute(EXPECTED_UPPER_REGION_KEY, 
endRegionKey);
-} else if (!scans.isEmpty()) {
-
scans.get(scans.size()-1).setAttribute(EXPECTED_UPPER_REGION_KEY, endRegionKey);
-}
-}
-scans = addNewScan(parallelScans, scans, newScan, endKey, 
true, regionLocation);
-currentKeyBytes = endKey;
+if (Bytes.compareTo(scan.getStartRow(), 
context.getScan().getStartRow()) != 0
+ || Bytes.compareTo(scan.getStopRow(), 
context.getScan().getStopRow()) != 0) {
--- End diff --

getParalleScans is the entry point. Let me move it out.


> Ensure split when iterating through results handled correctly
> -
>
> Key: PHOENIX-2628
> URL: https://issues.apache.org/jira/browse/PHOENIX-2628
> Project: Phoenix
>  Issue Type: Bug
>Reporter: James Taylor
>Assignee: Rajeshbabu Chintaguntla
> Fix For: 4.8.0
>
> Attachments: PHOENIX-2628-wip.patch, PHOENIX-2628.patch, 
> PHOENIX-2628_v7.patch, PHOENIX-2628_v8.patch
>
>
> We should start with a test case to ensure this works correctly, both for 
> scans and aggregates.



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[GitHub] phoenix pull request: PHOENIX-2628 Ensure split when iterating thr...

[chrajeshbabu]

Github user chrajeshbabu commented on a diff in the pull request:

https://github.com/apache/phoenix/pull/156#discussion_r58642550
  
--- Diff: 
phoenix-core/src/main/java/org/apache/phoenix/iterate/BaseResultIterators.java 
---
@@ -556,35 +564,55 @@ private static String toString(List gps) {
 } else {
 endKey = regionBoundaries.get(regionIndex);
 }
-HRegionLocation regionLocation = 
regionLocations.get(regionIndex);
-if (isLocalIndex) {
-HRegionInfo regionInfo = 
regionLocation.getRegionInfo();
-endRegionKey = regionInfo.getEndKey();
-keyOffset = 
ScanUtil.getRowKeyOffset(regionInfo.getStartKey(), endRegionKey);
-}
-try {
-while (guideIndex < gpsSize && 
(currentGuidePost.compareTo(endKey) <= 0 || endKey.length == 0)) {
-Scan newScan = scanRanges.intersectScan(scan, 
currentKeyBytes, currentGuidePostBytes, keyOffset,
-false);
-estimatedRows += 
gps.getRowCounts().get(guideIndex);
-estimatedSize += 
gps.getByteCounts().get(guideIndex);
-scans = addNewScan(parallelScans, scans, newScan, 
currentGuidePostBytes, false, regionLocation);
-currentKeyBytes = currentGuidePost.copyBytes();
-currentGuidePost = PrefixByteCodec.decode(decoder, 
input);
-currentGuidePostBytes = 
currentGuidePost.copyBytes();
-guideIndex++;
-}
-} catch (EOFException e) {}
-Scan newScan = scanRanges.intersectScan(scan, 
currentKeyBytes, endKey, keyOffset, true);
-if (isLocalIndex) {
-if (newScan != null) {
-newScan.setAttribute(EXPECTED_UPPER_REGION_KEY, 
endRegionKey);
-} else if (!scans.isEmpty()) {
-
scans.get(scans.size()-1).setAttribute(EXPECTED_UPPER_REGION_KEY, endRegionKey);
-}
-}
-scans = addNewScan(parallelScans, scans, newScan, endKey, 
true, regionLocation);
-currentKeyBytes = endKey;
+if (Bytes.compareTo(scan.getStartRow(), 
context.getScan().getStartRow()) != 0
+ || Bytes.compareTo(scan.getStopRow(), 
context.getScan().getStopRow()) != 0) {
--- End diff --

getParalleScans is the entry point. Let me move it out.


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

[jira] [Commented] (PHOENIX-2628) Ensure split when iterating through results handled correctly

[ASF GitHub Bot (JIRA)]

[ 
https://issues.apache.org/jira/browse/PHOENIX-2628?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15227554#comment-15227554
 ] 

ASF GitHub Bot commented on PHOENIX-2628:
-

Github user chrajeshbabu commented on a diff in the pull request:

https://github.com/apache/phoenix/pull/156#discussion_r58642188
  
--- Diff: 
phoenix-core/src/main/java/org/apache/phoenix/iterate/TableResultIterator.java 
---
@@ -107,8 +127,37 @@ public synchronized void close() throws SQLException {
 @Override
 public synchronized Tuple next() throws SQLException {
 initScanner();
-Tuple t = scanIterator.next();
-return t;
+try {
+lastTuple = scanIterator.next();
+if (lastTuple != null) {
+ImmutableBytesWritable ptr = new ImmutableBytesWritable();
+lastTuple.getKey(ptr);
+}
+} catch (SQLException e) {
+try {
+throw ServerUtil.parseServerException(e);
+} catch(StaleRegionBoundaryCacheException e1) {
+if(scan.getAttribute(NON_AGGREGATE_QUERY)!=null) {
+Scan newScan = ScanUtil.newScan(scan);
+if(lastTuple != null) {
+lastTuple.getKey(ptr);
+byte[] startRowSuffix = 
ByteUtil.copyKeyBytesIfNecessary(ptr);
+if(ScanUtil.isLocalIndex(newScan)) {
+newScan.setAttribute(SCAN_START_ROW_SUFFIX, 
ByteUtil.nextKey(startRowSuffix));
+} else {
+
newScan.setStartRow(ByteUtil.nextKey(startRowSuffix));
+}
+}
+
plan.getContext().getConnection().getQueryServices().clearTableRegionCache(htable.getTableName());
+this.scanIterator =
+
plan.iterator(DefaultParallelScanGrouper.getInstance(), newScan);
--- End diff --

Agree with you James. I think I can raise another issue for this and work 
on it. Wdyt?


> Ensure split when iterating through results handled correctly
> -
>
> Key: PHOENIX-2628
> URL: https://issues.apache.org/jira/browse/PHOENIX-2628
> Project: Phoenix
>  Issue Type: Bug
>Reporter: James Taylor
>Assignee: Rajeshbabu Chintaguntla
> Fix For: 4.8.0
>
> Attachments: PHOENIX-2628-wip.patch, PHOENIX-2628.patch, 
> PHOENIX-2628_v7.patch, PHOENIX-2628_v8.patch
>
>
> We should start with a test case to ensure this works correctly, both for 
> scans and aggregates.



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[GitHub] phoenix pull request: PHOENIX-2628 Ensure split when iterating thr...

[chrajeshbabu]

Github user chrajeshbabu commented on a diff in the pull request:

https://github.com/apache/phoenix/pull/156#discussion_r58642188
  
--- Diff: 
phoenix-core/src/main/java/org/apache/phoenix/iterate/TableResultIterator.java 
---
@@ -107,8 +127,37 @@ public synchronized void close() throws SQLException {
 @Override
 public synchronized Tuple next() throws SQLException {
 initScanner();
-Tuple t = scanIterator.next();
-return t;
+try {
+lastTuple = scanIterator.next();
+if (lastTuple != null) {
+ImmutableBytesWritable ptr = new ImmutableBytesWritable();
+lastTuple.getKey(ptr);
+}
+} catch (SQLException e) {
+try {
+throw ServerUtil.parseServerException(e);
+} catch(StaleRegionBoundaryCacheException e1) {
+if(scan.getAttribute(NON_AGGREGATE_QUERY)!=null) {
+Scan newScan = ScanUtil.newScan(scan);
+if(lastTuple != null) {
+lastTuple.getKey(ptr);
+byte[] startRowSuffix = 
ByteUtil.copyKeyBytesIfNecessary(ptr);
+if(ScanUtil.isLocalIndex(newScan)) {
+newScan.setAttribute(SCAN_START_ROW_SUFFIX, 
ByteUtil.nextKey(startRowSuffix));
+} else {
+
newScan.setStartRow(ByteUtil.nextKey(startRowSuffix));
+}
+}
+
plan.getContext().getConnection().getQueryServices().clearTableRegionCache(htable.getTableName());
+this.scanIterator =
+
plan.iterator(DefaultParallelScanGrouper.getInstance(), newScan);
--- End diff --

Agree with you James. I think I can raise another issue for this and work 
on it. Wdyt?


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

[jira] [Commented] (PHOENIX-2628) Ensure split when iterating through results handled correctly

[ASF GitHub Bot (JIRA)]

[ 
https://issues.apache.org/jira/browse/PHOENIX-2628?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15227543#comment-15227543
 ] 

ASF GitHub Bot commented on PHOENIX-2628:
-

Github user JamesRTaylor commented on a diff in the pull request:

https://github.com/apache/phoenix/pull/156#discussion_r58641527
  
--- Diff: 
phoenix-core/src/main/java/org/apache/phoenix/iterate/BaseResultIterators.java 
---
@@ -556,35 +564,55 @@ private static String toString(List gps) {
 } else {
 endKey = regionBoundaries.get(regionIndex);
 }
-HRegionLocation regionLocation = 
regionLocations.get(regionIndex);
-if (isLocalIndex) {
-HRegionInfo regionInfo = 
regionLocation.getRegionInfo();
-endRegionKey = regionInfo.getEndKey();
-keyOffset = 
ScanUtil.getRowKeyOffset(regionInfo.getStartKey(), endRegionKey);
-}
-try {
-while (guideIndex < gpsSize && 
(currentGuidePost.compareTo(endKey) <= 0 || endKey.length == 0)) {
-Scan newScan = scanRanges.intersectScan(scan, 
currentKeyBytes, currentGuidePostBytes, keyOffset,
-false);
-estimatedRows += 
gps.getRowCounts().get(guideIndex);
-estimatedSize += 
gps.getByteCounts().get(guideIndex);
-scans = addNewScan(parallelScans, scans, newScan, 
currentGuidePostBytes, false, regionLocation);
-currentKeyBytes = currentGuidePost.copyBytes();
-currentGuidePost = PrefixByteCodec.decode(decoder, 
input);
-currentGuidePostBytes = 
currentGuidePost.copyBytes();
-guideIndex++;
-}
-} catch (EOFException e) {}
-Scan newScan = scanRanges.intersectScan(scan, 
currentKeyBytes, endKey, keyOffset, true);
-if (isLocalIndex) {
-if (newScan != null) {
-newScan.setAttribute(EXPECTED_UPPER_REGION_KEY, 
endRegionKey);
-} else if (!scans.isEmpty()) {
-
scans.get(scans.size()-1).setAttribute(EXPECTED_UPPER_REGION_KEY, endRegionKey);
-}
-}
-scans = addNewScan(parallelScans, scans, newScan, endKey, 
true, regionLocation);
-currentKeyBytes = endKey;
+if (Bytes.compareTo(scan.getStartRow(), 
context.getScan().getStartRow()) != 0
+ || Bytes.compareTo(scan.getStopRow(), 
context.getScan().getStopRow()) != 0) {
--- End diff --

Seems like you've forked the bulk of the code completely. The only think 
you'd duplicate would be the loop and a bit of setup code up top (that could be 
factored out). What's the entry point for this? Is the problem that the entry 
point is the same?


> Ensure split when iterating through results handled correctly
> -
>
> Key: PHOENIX-2628
> URL: https://issues.apache.org/jira/browse/PHOENIX-2628
> Project: Phoenix
>  Issue Type: Bug
>Reporter: James Taylor
>Assignee: Rajeshbabu Chintaguntla
> Fix For: 4.8.0
>
> Attachments: PHOENIX-2628-wip.patch, PHOENIX-2628.patch, 
> PHOENIX-2628_v7.patch, PHOENIX-2628_v8.patch
>
>
> We should start with a test case to ensure this works correctly, both for 
> scans and aggregates.



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[GitHub] phoenix pull request: PHOENIX-2628 Ensure split when iterating thr...

[JamesRTaylor]

Github user JamesRTaylor commented on a diff in the pull request:

https://github.com/apache/phoenix/pull/156#discussion_r58641527
  
--- Diff: 
phoenix-core/src/main/java/org/apache/phoenix/iterate/BaseResultIterators.java 
---
@@ -556,35 +564,55 @@ private static String toString(List gps) {
 } else {
 endKey = regionBoundaries.get(regionIndex);
 }
-HRegionLocation regionLocation = 
regionLocations.get(regionIndex);
-if (isLocalIndex) {
-HRegionInfo regionInfo = 
regionLocation.getRegionInfo();
-endRegionKey = regionInfo.getEndKey();
-keyOffset = 
ScanUtil.getRowKeyOffset(regionInfo.getStartKey(), endRegionKey);
-}
-try {
-while (guideIndex < gpsSize && 
(currentGuidePost.compareTo(endKey) <= 0 || endKey.length == 0)) {
-Scan newScan = scanRanges.intersectScan(scan, 
currentKeyBytes, currentGuidePostBytes, keyOffset,
-false);
-estimatedRows += 
gps.getRowCounts().get(guideIndex);
-estimatedSize += 
gps.getByteCounts().get(guideIndex);
-scans = addNewScan(parallelScans, scans, newScan, 
currentGuidePostBytes, false, regionLocation);
-currentKeyBytes = currentGuidePost.copyBytes();
-currentGuidePost = PrefixByteCodec.decode(decoder, 
input);
-currentGuidePostBytes = 
currentGuidePost.copyBytes();
-guideIndex++;
-}
-} catch (EOFException e) {}
-Scan newScan = scanRanges.intersectScan(scan, 
currentKeyBytes, endKey, keyOffset, true);
-if (isLocalIndex) {
-if (newScan != null) {
-newScan.setAttribute(EXPECTED_UPPER_REGION_KEY, 
endRegionKey);
-} else if (!scans.isEmpty()) {
-
scans.get(scans.size()-1).setAttribute(EXPECTED_UPPER_REGION_KEY, endRegionKey);
-}
-}
-scans = addNewScan(parallelScans, scans, newScan, endKey, 
true, regionLocation);
-currentKeyBytes = endKey;
+if (Bytes.compareTo(scan.getStartRow(), 
context.getScan().getStartRow()) != 0
+ || Bytes.compareTo(scan.getStopRow(), 
context.getScan().getStopRow()) != 0) {
--- End diff --

Seems like you've forked the bulk of the code completely. The only think 
you'd duplicate would be the loop and a bit of setup code up top (that could be 
factored out). What's the entry point for this? Is the problem that the entry 
point is the same?


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

[jira] [Commented] (PHOENIX-2628) Ensure split when iterating through results handled correctly

[ASF GitHub Bot (JIRA)]

[ 
https://issues.apache.org/jira/browse/PHOENIX-2628?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15227541#comment-15227541
 ] 

ASF GitHub Bot commented on PHOENIX-2628:
-

Github user JamesRTaylor commented on a diff in the pull request:

https://github.com/apache/phoenix/pull/156#discussion_r58641320
  
--- Diff: 
phoenix-core/src/main/java/org/apache/phoenix/iterate/BaseResultIterators.java 
---
@@ -556,35 +564,55 @@ private static String toString(List gps) {
 } else {
 endKey = regionBoundaries.get(regionIndex);
 }
-HRegionLocation regionLocation = 
regionLocations.get(regionIndex);
-if (isLocalIndex) {
-HRegionInfo regionInfo = 
regionLocation.getRegionInfo();
-endRegionKey = regionInfo.getEndKey();
-keyOffset = 
ScanUtil.getRowKeyOffset(regionInfo.getStartKey(), endRegionKey);
-}
-try {
-while (guideIndex < gpsSize && 
(currentGuidePost.compareTo(endKey) <= 0 || endKey.length == 0)) {
-Scan newScan = scanRanges.intersectScan(scan, 
currentKeyBytes, currentGuidePostBytes, keyOffset,
-false);
-estimatedRows += 
gps.getRowCounts().get(guideIndex);
-estimatedSize += 
gps.getByteCounts().get(guideIndex);
-scans = addNewScan(parallelScans, scans, newScan, 
currentGuidePostBytes, false, regionLocation);
-currentKeyBytes = currentGuidePost.copyBytes();
-currentGuidePost = PrefixByteCodec.decode(decoder, 
input);
-currentGuidePostBytes = 
currentGuidePost.copyBytes();
-guideIndex++;
-}
-} catch (EOFException e) {}
-Scan newScan = scanRanges.intersectScan(scan, 
currentKeyBytes, endKey, keyOffset, true);
-if (isLocalIndex) {
-if (newScan != null) {
-newScan.setAttribute(EXPECTED_UPPER_REGION_KEY, 
endRegionKey);
-} else if (!scans.isEmpty()) {
-
scans.get(scans.size()-1).setAttribute(EXPECTED_UPPER_REGION_KEY, endRegionKey);
-}
-}
-scans = addNewScan(parallelScans, scans, newScan, endKey, 
true, regionLocation);
-currentKeyBytes = endKey;
+if (Bytes.compareTo(scan.getStartRow(), 
context.getScan().getStartRow()) != 0
+ || Bytes.compareTo(scan.getStopRow(), 
context.getScan().getStopRow()) != 0) {
+Scan newScan = ScanUtil.newScan(scan);
+if(ScanUtil.isLocalIndex(scan)) {
+newScan.setStartRow(regionInfo.getStartKey());
+newScan.setAttribute(SCAN_ACTUAL_START_ROW, 
regionInfo.getStartKey());
+newScan.setStopRow(regionInfo.getEndKey());
+newScan.setAttribute(EXPECTED_UPPER_REGION_KEY, 
regionInfo.getEndKey());
+} else {
+if(Bytes.compareTo(scan.getStartRow(), 
regionInfo.getStartKey())<=0) {
+newScan.setAttribute(SCAN_ACTUAL_START_ROW, 
regionInfo.getStartKey());
+newScan.setStartRow(regionInfo.getStartKey());
+}
+if(scan.getStopRow().length == 0 || 
(regionInfo.getEndKey().length != 0 && Bytes.compareTo(scan.getStopRow(), 
regionInfo.getEndKey())>0)) {
+newScan.setStopRow(regionInfo.getEndKey());
+}
+}   
+scans = addNewScan(parallelScans, scans, newScan, 
endKey, true, regionLocation);
+ } else {
--- End diff --

Can we remove the if (isLocalIndex) checks below since the above if 
statement would have been entered instead?


> Ensure split when iterating through results handled correctly
> -
>
> Key: PHOENIX-2628
> URL: https://issues.apache.org/jira/browse/PHOENIX-2628
> Project: Phoenix
>  Issue Type: Bug
>Reporter: James Taylor
>Assignee: Rajeshbabu Chintaguntla
> Fix For: 4.8.0
>
> Attachments: PHOENIX-2628-wip.patch, PHOENIX-2628.patch, 
> PHOENIX-2628_v7.patch, PHOENIX-2628_v8.patch
>
>
> We should start with a test case to ensure this works correctly, both for 
> scans and aggregates.



--
This 

[GitHub] phoenix pull request: PHOENIX-2628 Ensure split when iterating thr...

[JamesRTaylor]

Github user JamesRTaylor commented on a diff in the pull request:

https://github.com/apache/phoenix/pull/156#discussion_r58641320
  
--- Diff: 
phoenix-core/src/main/java/org/apache/phoenix/iterate/BaseResultIterators.java 
---
@@ -556,35 +564,55 @@ private static String toString(List gps) {
 } else {
 endKey = regionBoundaries.get(regionIndex);
 }
-HRegionLocation regionLocation = 
regionLocations.get(regionIndex);
-if (isLocalIndex) {
-HRegionInfo regionInfo = 
regionLocation.getRegionInfo();
-endRegionKey = regionInfo.getEndKey();
-keyOffset = 
ScanUtil.getRowKeyOffset(regionInfo.getStartKey(), endRegionKey);
-}
-try {
-while (guideIndex < gpsSize && 
(currentGuidePost.compareTo(endKey) <= 0 || endKey.length == 0)) {
-Scan newScan = scanRanges.intersectScan(scan, 
currentKeyBytes, currentGuidePostBytes, keyOffset,
-false);
-estimatedRows += 
gps.getRowCounts().get(guideIndex);
-estimatedSize += 
gps.getByteCounts().get(guideIndex);
-scans = addNewScan(parallelScans, scans, newScan, 
currentGuidePostBytes, false, regionLocation);
-currentKeyBytes = currentGuidePost.copyBytes();
-currentGuidePost = PrefixByteCodec.decode(decoder, 
input);
-currentGuidePostBytes = 
currentGuidePost.copyBytes();
-guideIndex++;
-}
-} catch (EOFException e) {}
-Scan newScan = scanRanges.intersectScan(scan, 
currentKeyBytes, endKey, keyOffset, true);
-if (isLocalIndex) {
-if (newScan != null) {
-newScan.setAttribute(EXPECTED_UPPER_REGION_KEY, 
endRegionKey);
-} else if (!scans.isEmpty()) {
-
scans.get(scans.size()-1).setAttribute(EXPECTED_UPPER_REGION_KEY, endRegionKey);
-}
-}
-scans = addNewScan(parallelScans, scans, newScan, endKey, 
true, regionLocation);
-currentKeyBytes = endKey;
+if (Bytes.compareTo(scan.getStartRow(), 
context.getScan().getStartRow()) != 0
+ || Bytes.compareTo(scan.getStopRow(), 
context.getScan().getStopRow()) != 0) {
+Scan newScan = ScanUtil.newScan(scan);
+if(ScanUtil.isLocalIndex(scan)) {
+newScan.setStartRow(regionInfo.getStartKey());
+newScan.setAttribute(SCAN_ACTUAL_START_ROW, 
regionInfo.getStartKey());
+newScan.setStopRow(regionInfo.getEndKey());
+newScan.setAttribute(EXPECTED_UPPER_REGION_KEY, 
regionInfo.getEndKey());
+} else {
+if(Bytes.compareTo(scan.getStartRow(), 
regionInfo.getStartKey())<=0) {
+newScan.setAttribute(SCAN_ACTUAL_START_ROW, 
regionInfo.getStartKey());
+newScan.setStartRow(regionInfo.getStartKey());
+}
+if(scan.getStopRow().length == 0 || 
(regionInfo.getEndKey().length != 0 && Bytes.compareTo(scan.getStopRow(), 
regionInfo.getEndKey())>0)) {
+newScan.setStopRow(regionInfo.getEndKey());
+}
+}   
+scans = addNewScan(parallelScans, scans, newScan, 
endKey, true, regionLocation);
+ } else {
--- End diff --

Can we remove the if (isLocalIndex) checks below since the above if 
statement would have been entered instead?


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

[jira] [Commented] (PHOENIX-2628) Ensure split when iterating through results handled correctly

[ASF GitHub Bot (JIRA)]

[ 
https://issues.apache.org/jira/browse/PHOENIX-2628?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15227540#comment-15227540
 ] 

ASF GitHub Bot commented on PHOENIX-2628:
-

Github user JamesRTaylor commented on a diff in the pull request:

https://github.com/apache/phoenix/pull/156#discussion_r58641238
  
--- Diff: 
phoenix-core/src/main/java/org/apache/phoenix/iterate/BaseResultIterators.java 
---
@@ -556,35 +564,55 @@ private static String toString(List gps) {
 } else {
 endKey = regionBoundaries.get(regionIndex);
 }
-HRegionLocation regionLocation = 
regionLocations.get(regionIndex);
-if (isLocalIndex) {
-HRegionInfo regionInfo = 
regionLocation.getRegionInfo();
-endRegionKey = regionInfo.getEndKey();
-keyOffset = 
ScanUtil.getRowKeyOffset(regionInfo.getStartKey(), endRegionKey);
-}
-try {
-while (guideIndex < gpsSize && 
(currentGuidePost.compareTo(endKey) <= 0 || endKey.length == 0)) {
-Scan newScan = scanRanges.intersectScan(scan, 
currentKeyBytes, currentGuidePostBytes, keyOffset,
-false);
-estimatedRows += 
gps.getRowCounts().get(guideIndex);
-estimatedSize += 
gps.getByteCounts().get(guideIndex);
-scans = addNewScan(parallelScans, scans, newScan, 
currentGuidePostBytes, false, regionLocation);
-currentKeyBytes = currentGuidePost.copyBytes();
-currentGuidePost = PrefixByteCodec.decode(decoder, 
input);
-currentGuidePostBytes = 
currentGuidePost.copyBytes();
-guideIndex++;
-}
-} catch (EOFException e) {}
-Scan newScan = scanRanges.intersectScan(scan, 
currentKeyBytes, endKey, keyOffset, true);
-if (isLocalIndex) {
-if (newScan != null) {
-newScan.setAttribute(EXPECTED_UPPER_REGION_KEY, 
endRegionKey);
-} else if (!scans.isEmpty()) {
-
scans.get(scans.size()-1).setAttribute(EXPECTED_UPPER_REGION_KEY, endRegionKey);
-}
-}
-scans = addNewScan(parallelScans, scans, newScan, endKey, 
true, regionLocation);
-currentKeyBytes = endKey;
+if (Bytes.compareTo(scan.getStartRow(), 
context.getScan().getStartRow()) != 0
+ || Bytes.compareTo(scan.getStopRow(), 
context.getScan().getStopRow()) != 0) {
+Scan newScan = ScanUtil.newScan(scan);
+if(ScanUtil.isLocalIndex(scan)) {
+newScan.setStartRow(regionInfo.getStartKey());
+newScan.setAttribute(SCAN_ACTUAL_START_ROW, 
regionInfo.getStartKey());
--- End diff --

If we don't need EXPECTED_UPPER_REGION_KEY, let's remove it.


> Ensure split when iterating through results handled correctly
> -
>
> Key: PHOENIX-2628
> URL: https://issues.apache.org/jira/browse/PHOENIX-2628
> Project: Phoenix
>  Issue Type: Bug
>Reporter: James Taylor
>Assignee: Rajeshbabu Chintaguntla
> Fix For: 4.8.0
>
> Attachments: PHOENIX-2628-wip.patch, PHOENIX-2628.patch, 
> PHOENIX-2628_v7.patch, PHOENIX-2628_v8.patch
>
>
> We should start with a test case to ensure this works correctly, both for 
> scans and aggregates.



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[GitHub] phoenix pull request: PHOENIX-2628 Ensure split when iterating thr...

[JamesRTaylor]

Github user JamesRTaylor commented on a diff in the pull request:

https://github.com/apache/phoenix/pull/156#discussion_r58641238
  
--- Diff: 
phoenix-core/src/main/java/org/apache/phoenix/iterate/BaseResultIterators.java 
---
@@ -556,35 +564,55 @@ private static String toString(List gps) {
 } else {
 endKey = regionBoundaries.get(regionIndex);
 }
-HRegionLocation regionLocation = 
regionLocations.get(regionIndex);
-if (isLocalIndex) {
-HRegionInfo regionInfo = 
regionLocation.getRegionInfo();
-endRegionKey = regionInfo.getEndKey();
-keyOffset = 
ScanUtil.getRowKeyOffset(regionInfo.getStartKey(), endRegionKey);
-}
-try {
-while (guideIndex < gpsSize && 
(currentGuidePost.compareTo(endKey) <= 0 || endKey.length == 0)) {
-Scan newScan = scanRanges.intersectScan(scan, 
currentKeyBytes, currentGuidePostBytes, keyOffset,
-false);
-estimatedRows += 
gps.getRowCounts().get(guideIndex);
-estimatedSize += 
gps.getByteCounts().get(guideIndex);
-scans = addNewScan(parallelScans, scans, newScan, 
currentGuidePostBytes, false, regionLocation);
-currentKeyBytes = currentGuidePost.copyBytes();
-currentGuidePost = PrefixByteCodec.decode(decoder, 
input);
-currentGuidePostBytes = 
currentGuidePost.copyBytes();
-guideIndex++;
-}
-} catch (EOFException e) {}
-Scan newScan = scanRanges.intersectScan(scan, 
currentKeyBytes, endKey, keyOffset, true);
-if (isLocalIndex) {
-if (newScan != null) {
-newScan.setAttribute(EXPECTED_UPPER_REGION_KEY, 
endRegionKey);
-} else if (!scans.isEmpty()) {
-
scans.get(scans.size()-1).setAttribute(EXPECTED_UPPER_REGION_KEY, endRegionKey);
-}
-}
-scans = addNewScan(parallelScans, scans, newScan, endKey, 
true, regionLocation);
-currentKeyBytes = endKey;
+if (Bytes.compareTo(scan.getStartRow(), 
context.getScan().getStartRow()) != 0
+ || Bytes.compareTo(scan.getStopRow(), 
context.getScan().getStopRow()) != 0) {
+Scan newScan = ScanUtil.newScan(scan);
+if(ScanUtil.isLocalIndex(scan)) {
+newScan.setStartRow(regionInfo.getStartKey());
+newScan.setAttribute(SCAN_ACTUAL_START_ROW, 
regionInfo.getStartKey());
--- End diff --

If we don't need EXPECTED_UPPER_REGION_KEY, let's remove it.


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

[jira] [Commented] (PHOENIX-2628) Ensure split when iterating through results handled correctly

[ASF GitHub Bot (JIRA)]

[ 
https://issues.apache.org/jira/browse/PHOENIX-2628?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15227535#comment-15227535
 ] 

ASF GitHub Bot commented on PHOENIX-2628:
-

Github user chrajeshbabu commented on a diff in the pull request:

https://github.com/apache/phoenix/pull/156#discussion_r58641031
  
--- Diff: 
phoenix-core/src/main/java/org/apache/phoenix/iterate/BaseResultIterators.java 
---
@@ -326,11 +325,12 @@ private static void 
optimizeProjection(StatementContext context, Scan scan, PTab
 }
 }
 
-public BaseResultIterators(QueryPlan plan, Integer perScanLimit, 
ParallelScanGrouper scanGrouper) throws SQLException {
+public BaseResultIterators(QueryPlan plan, Integer perScanLimit, 
ParallelScanGrouper scanGrouper, Scan scan) throws SQLException {
 super(plan.getContext(), plan.getTableRef(), plan.getGroupBy(), 
plan.getOrderBy(), plan.getStatement().getHint(), plan.getLimit());
 this.plan = plan;
 this.scanGrouper = scanGrouper;
 StatementContext context = plan.getContext();
+this.scan = scan == null ? context.getScan() : scan;
--- End diff --

This null check is not required will remove it.


> Ensure split when iterating through results handled correctly
> -
>
> Key: PHOENIX-2628
> URL: https://issues.apache.org/jira/browse/PHOENIX-2628
> Project: Phoenix
>  Issue Type: Bug
>Reporter: James Taylor
>Assignee: Rajeshbabu Chintaguntla
> Fix For: 4.8.0
>
> Attachments: PHOENIX-2628-wip.patch, PHOENIX-2628.patch, 
> PHOENIX-2628_v7.patch, PHOENIX-2628_v8.patch
>
>
> We should start with a test case to ensure this works correctly, both for 
> scans and aggregates.



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Commented] (PHOENIX-2628) Ensure split when iterating through results handled correctly

[ASF GitHub Bot (JIRA)]

[ 
https://issues.apache.org/jira/browse/PHOENIX-2628?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15227534#comment-15227534
 ] 

ASF GitHub Bot commented on PHOENIX-2628:
-

Github user chrajeshbabu commented on a diff in the pull request:

https://github.com/apache/phoenix/pull/156#discussion_r58641009
  
--- Diff: 
phoenix-core/src/main/java/org/apache/phoenix/iterate/BaseResultIterators.java 
---
@@ -465,7 +465,14 @@ private static String toString(List gps) {
 }
 
 private List getParallelScans() throws SQLException {
-return getParallelScans(EMPTY_BYTE_ARRAY, EMPTY_BYTE_ARRAY);
+if (scan == null
+|| (ScanUtil.isLocalIndex(scan)
+&& 
Bytes.compareTo(context.getScan().getStartRow(), scan.getStartRow()) == 0 && 
Bytes
+.compareTo(context.getScan().getStopRow(), 
scan.getStopRow()) == 0)) {
--- End diff --

Sure will move to ScanUtil


> Ensure split when iterating through results handled correctly
> -
>
> Key: PHOENIX-2628
> URL: https://issues.apache.org/jira/browse/PHOENIX-2628
> Project: Phoenix
>  Issue Type: Bug
>Reporter: James Taylor
>Assignee: Rajeshbabu Chintaguntla
> Fix For: 4.8.0
>
> Attachments: PHOENIX-2628-wip.patch, PHOENIX-2628.patch, 
> PHOENIX-2628_v7.patch, PHOENIX-2628_v8.patch
>
>
> We should start with a test case to ensure this works correctly, both for 
> scans and aggregates.



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[GitHub] phoenix pull request: PHOENIX-2628 Ensure split when iterating thr...

[chrajeshbabu]

Github user chrajeshbabu commented on a diff in the pull request:

https://github.com/apache/phoenix/pull/156#discussion_r58641031
  
--- Diff: 
phoenix-core/src/main/java/org/apache/phoenix/iterate/BaseResultIterators.java 
---
@@ -326,11 +325,12 @@ private static void 
optimizeProjection(StatementContext context, Scan scan, PTab
 }
 }
 
-public BaseResultIterators(QueryPlan plan, Integer perScanLimit, 
ParallelScanGrouper scanGrouper) throws SQLException {
+public BaseResultIterators(QueryPlan plan, Integer perScanLimit, 
ParallelScanGrouper scanGrouper, Scan scan) throws SQLException {
 super(plan.getContext(), plan.getTableRef(), plan.getGroupBy(), 
plan.getOrderBy(), plan.getStatement().getHint(), plan.getLimit());
 this.plan = plan;
 this.scanGrouper = scanGrouper;
 StatementContext context = plan.getContext();
+this.scan = scan == null ? context.getScan() : scan;
--- End diff --

This null check is not required will remove it.


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

[jira] [Commented] (PHOENIX-2628) Ensure split when iterating through results handled correctly

[ASF GitHub Bot (JIRA)]

[ 
https://issues.apache.org/jira/browse/PHOENIX-2628?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15227533#comment-15227533
 ] 

ASF GitHub Bot commented on PHOENIX-2628:
-

Github user chrajeshbabu commented on a diff in the pull request:

https://github.com/apache/phoenix/pull/156#discussion_r58640990
  
--- Diff: 
phoenix-core/src/main/java/org/apache/phoenix/iterate/BaseResultIterators.java 
---
@@ -465,7 +465,14 @@ private static String toString(List gps) {
 }
 
 private List getParallelScans() throws SQLException {
-return getParallelScans(EMPTY_BYTE_ARRAY, EMPTY_BYTE_ARRAY);
+if (scan == null
+|| (ScanUtil.isLocalIndex(scan)
+&& 
Bytes.compareTo(context.getScan().getStartRow(), scan.getStartRow()) == 0 && 
Bytes
+.compareTo(context.getScan().getStopRow(), 
scan.getStopRow()) == 0)) {
+return getParallelScans(EMPTY_BYTE_ARRAY, EMPTY_BYTE_ARRAY);
--- End diff --

This check detects whether scan boundaries  are equal to context scan 
boundaries or not. If they are same we are going by getting all parallel scans 
for the table. Will document it.


> Ensure split when iterating through results handled correctly
> -
>
> Key: PHOENIX-2628
> URL: https://issues.apache.org/jira/browse/PHOENIX-2628
> Project: Phoenix
>  Issue Type: Bug
>Reporter: James Taylor
>Assignee: Rajeshbabu Chintaguntla
> Fix For: 4.8.0
>
> Attachments: PHOENIX-2628-wip.patch, PHOENIX-2628.patch, 
> PHOENIX-2628_v7.patch, PHOENIX-2628_v8.patch
>
>
> We should start with a test case to ensure this works correctly, both for 
> scans and aggregates.



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[GitHub] phoenix pull request: PHOENIX-2628 Ensure split when iterating thr...

[chrajeshbabu]

Github user chrajeshbabu commented on a diff in the pull request:

https://github.com/apache/phoenix/pull/156#discussion_r58640990
  
--- Diff: 
phoenix-core/src/main/java/org/apache/phoenix/iterate/BaseResultIterators.java 
---
@@ -465,7 +465,14 @@ private static String toString(List gps) {
 }
 
 private List getParallelScans() throws SQLException {
-return getParallelScans(EMPTY_BYTE_ARRAY, EMPTY_BYTE_ARRAY);
+if (scan == null
+|| (ScanUtil.isLocalIndex(scan)
+&& 
Bytes.compareTo(context.getScan().getStartRow(), scan.getStartRow()) == 0 && 
Bytes
+.compareTo(context.getScan().getStopRow(), 
scan.getStopRow()) == 0)) {
+return getParallelScans(EMPTY_BYTE_ARRAY, EMPTY_BYTE_ARRAY);
--- End diff --

This check detects whether scan boundaries  are equal to context scan 
boundaries or not. If they are same we are going by getting all parallel scans 
for the table. Will document it.


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

[GitHub] phoenix pull request: PHOENIX-2628 Ensure split when iterating thr...

[chrajeshbabu]

Github user chrajeshbabu commented on a diff in the pull request:

https://github.com/apache/phoenix/pull/156#discussion_r58641009
  
--- Diff: 
phoenix-core/src/main/java/org/apache/phoenix/iterate/BaseResultIterators.java 
---
@@ -465,7 +465,14 @@ private static String toString(List gps) {
 }
 
 private List getParallelScans() throws SQLException {
-return getParallelScans(EMPTY_BYTE_ARRAY, EMPTY_BYTE_ARRAY);
+if (scan == null
+|| (ScanUtil.isLocalIndex(scan)
+&& 
Bytes.compareTo(context.getScan().getStartRow(), scan.getStartRow()) == 0 && 
Bytes
+.compareTo(context.getScan().getStopRow(), 
scan.getStopRow()) == 0)) {
--- End diff --

Sure will move to ScanUtil


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

[jira] [Commented] (PHOENIX-2628) Ensure split when iterating through results handled correctly

[ASF GitHub Bot (JIRA)]

[ 
https://issues.apache.org/jira/browse/PHOENIX-2628?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15227527#comment-15227527
 ] 

ASF GitHub Bot commented on PHOENIX-2628:
-

Github user JamesRTaylor commented on a diff in the pull request:

https://github.com/apache/phoenix/pull/156#discussion_r58640786
  
--- Diff: 
phoenix-core/src/main/java/org/apache/phoenix/iterate/TableResultIterator.java 
---
@@ -107,8 +127,37 @@ public synchronized void close() throws SQLException {
 @Override
 public synchronized Tuple next() throws SQLException {
 initScanner();
-Tuple t = scanIterator.next();
-return t;
+try {
+lastTuple = scanIterator.next();
+if (lastTuple != null) {
+ImmutableBytesWritable ptr = new ImmutableBytesWritable();
+lastTuple.getKey(ptr);
+}
+} catch (SQLException e) {
+try {
+throw ServerUtil.parseServerException(e);
+} catch(StaleRegionBoundaryCacheException e1) {
+if(scan.getAttribute(NON_AGGREGATE_QUERY)!=null) {
+Scan newScan = ScanUtil.newScan(scan);
+if(lastTuple != null) {
+lastTuple.getKey(ptr);
+byte[] startRowSuffix = 
ByteUtil.copyKeyBytesIfNecessary(ptr);
+if(ScanUtil.isLocalIndex(newScan)) {
+newScan.setAttribute(SCAN_START_ROW_SUFFIX, 
ByteUtil.nextKey(startRowSuffix));
+} else {
+
newScan.setStartRow(ByteUtil.nextKey(startRowSuffix));
--- End diff --

See other comment below, but how about we serialize the row key we found 
which is out-of-region in the StaleRegionBoundaryCacheException instead of 
mucking with the startRowSuffix?


> Ensure split when iterating through results handled correctly
> -
>
> Key: PHOENIX-2628
> URL: https://issues.apache.org/jira/browse/PHOENIX-2628
> Project: Phoenix
>  Issue Type: Bug
>Reporter: James Taylor
>Assignee: Rajeshbabu Chintaguntla
> Fix For: 4.8.0
>
> Attachments: PHOENIX-2628-wip.patch, PHOENIX-2628.patch, 
> PHOENIX-2628_v7.patch, PHOENIX-2628_v8.patch
>
>
> We should start with a test case to ensure this works correctly, both for 
> scans and aggregates.



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[GitHub] phoenix pull request: PHOENIX-2628 Ensure split when iterating thr...

[chrajeshbabu]

Github user chrajeshbabu commented on a diff in the pull request:

https://github.com/apache/phoenix/pull/156#discussion_r58640818
  
--- Diff: 
phoenix-core/src/main/java/org/apache/phoenix/iterate/BaseResultIterators.java 
---
@@ -556,35 +564,55 @@ private static String toString(List gps) {
 } else {
 endKey = regionBoundaries.get(regionIndex);
 }
-HRegionLocation regionLocation = 
regionLocations.get(regionIndex);
-if (isLocalIndex) {
-HRegionInfo regionInfo = 
regionLocation.getRegionInfo();
-endRegionKey = regionInfo.getEndKey();
-keyOffset = 
ScanUtil.getRowKeyOffset(regionInfo.getStartKey(), endRegionKey);
-}
-try {
-while (guideIndex < gpsSize && 
(currentGuidePost.compareTo(endKey) <= 0 || endKey.length == 0)) {
-Scan newScan = scanRanges.intersectScan(scan, 
currentKeyBytes, currentGuidePostBytes, keyOffset,
-false);
-estimatedRows += 
gps.getRowCounts().get(guideIndex);
-estimatedSize += 
gps.getByteCounts().get(guideIndex);
-scans = addNewScan(parallelScans, scans, newScan, 
currentGuidePostBytes, false, regionLocation);
-currentKeyBytes = currentGuidePost.copyBytes();
-currentGuidePost = PrefixByteCodec.decode(decoder, 
input);
-currentGuidePostBytes = 
currentGuidePost.copyBytes();
-guideIndex++;
-}
-} catch (EOFException e) {}
-Scan newScan = scanRanges.intersectScan(scan, 
currentKeyBytes, endKey, keyOffset, true);
-if (isLocalIndex) {
-if (newScan != null) {
-newScan.setAttribute(EXPECTED_UPPER_REGION_KEY, 
endRegionKey);
-} else if (!scans.isEmpty()) {
-
scans.get(scans.size()-1).setAttribute(EXPECTED_UPPER_REGION_KEY, endRegionKey);
-}
-}
-scans = addNewScan(parallelScans, scans, newScan, endKey, 
true, regionLocation);
-currentKeyBytes = endKey;
+if (Bytes.compareTo(scan.getStartRow(), 
context.getScan().getStartRow()) != 0
+ || Bytes.compareTo(scan.getStopRow(), 
context.getScan().getStopRow()) != 0) {
--- End diff --

If I move creating parallel scans for this special case I need to duplicate 
lot of code. That's why I have added special case as part of creating existing 
parallel scans only.


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

[jira] [Commented] (PHOENIX-2628) Ensure split when iterating through results handled correctly

[ASF GitHub Bot (JIRA)]

[ 
https://issues.apache.org/jira/browse/PHOENIX-2628?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15227528#comment-15227528
 ] 

ASF GitHub Bot commented on PHOENIX-2628:
-

Github user chrajeshbabu commented on a diff in the pull request:

https://github.com/apache/phoenix/pull/156#discussion_r58640818
  
--- Diff: 
phoenix-core/src/main/java/org/apache/phoenix/iterate/BaseResultIterators.java 
---
@@ -556,35 +564,55 @@ private static String toString(List gps) {
 } else {
 endKey = regionBoundaries.get(regionIndex);
 }
-HRegionLocation regionLocation = 
regionLocations.get(regionIndex);
-if (isLocalIndex) {
-HRegionInfo regionInfo = 
regionLocation.getRegionInfo();
-endRegionKey = regionInfo.getEndKey();
-keyOffset = 
ScanUtil.getRowKeyOffset(regionInfo.getStartKey(), endRegionKey);
-}
-try {
-while (guideIndex < gpsSize && 
(currentGuidePost.compareTo(endKey) <= 0 || endKey.length == 0)) {
-Scan newScan = scanRanges.intersectScan(scan, 
currentKeyBytes, currentGuidePostBytes, keyOffset,
-false);
-estimatedRows += 
gps.getRowCounts().get(guideIndex);
-estimatedSize += 
gps.getByteCounts().get(guideIndex);
-scans = addNewScan(parallelScans, scans, newScan, 
currentGuidePostBytes, false, regionLocation);
-currentKeyBytes = currentGuidePost.copyBytes();
-currentGuidePost = PrefixByteCodec.decode(decoder, 
input);
-currentGuidePostBytes = 
currentGuidePost.copyBytes();
-guideIndex++;
-}
-} catch (EOFException e) {}
-Scan newScan = scanRanges.intersectScan(scan, 
currentKeyBytes, endKey, keyOffset, true);
-if (isLocalIndex) {
-if (newScan != null) {
-newScan.setAttribute(EXPECTED_UPPER_REGION_KEY, 
endRegionKey);
-} else if (!scans.isEmpty()) {
-
scans.get(scans.size()-1).setAttribute(EXPECTED_UPPER_REGION_KEY, endRegionKey);
-}
-}
-scans = addNewScan(parallelScans, scans, newScan, endKey, 
true, regionLocation);
-currentKeyBytes = endKey;
+if (Bytes.compareTo(scan.getStartRow(), 
context.getScan().getStartRow()) != 0
+ || Bytes.compareTo(scan.getStopRow(), 
context.getScan().getStopRow()) != 0) {
--- End diff --

If I move creating parallel scans for this special case I need to duplicate 
lot of code. That's why I have added special case as part of creating existing 
parallel scans only.


> Ensure split when iterating through results handled correctly
> -
>
> Key: PHOENIX-2628
> URL: https://issues.apache.org/jira/browse/PHOENIX-2628
> Project: Phoenix
>  Issue Type: Bug
>Reporter: James Taylor
>Assignee: Rajeshbabu Chintaguntla
> Fix For: 4.8.0
>
> Attachments: PHOENIX-2628-wip.patch, PHOENIX-2628.patch, 
> PHOENIX-2628_v7.patch, PHOENIX-2628_v8.patch
>
>
> We should start with a test case to ensure this works correctly, both for 
> scans and aggregates.



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[GitHub] phoenix pull request: PHOENIX-2628 Ensure split when iterating thr...

[JamesRTaylor]

Github user JamesRTaylor commented on a diff in the pull request:

https://github.com/apache/phoenix/pull/156#discussion_r58640786
  
--- Diff: 
phoenix-core/src/main/java/org/apache/phoenix/iterate/TableResultIterator.java 
---
@@ -107,8 +127,37 @@ public synchronized void close() throws SQLException {
 @Override
 public synchronized Tuple next() throws SQLException {
 initScanner();
-Tuple t = scanIterator.next();
-return t;
+try {
+lastTuple = scanIterator.next();
+if (lastTuple != null) {
+ImmutableBytesWritable ptr = new ImmutableBytesWritable();
+lastTuple.getKey(ptr);
+}
+} catch (SQLException e) {
+try {
+throw ServerUtil.parseServerException(e);
+} catch(StaleRegionBoundaryCacheException e1) {
+if(scan.getAttribute(NON_AGGREGATE_QUERY)!=null) {
+Scan newScan = ScanUtil.newScan(scan);
+if(lastTuple != null) {
+lastTuple.getKey(ptr);
+byte[] startRowSuffix = 
ByteUtil.copyKeyBytesIfNecessary(ptr);
+if(ScanUtil.isLocalIndex(newScan)) {
+newScan.setAttribute(SCAN_START_ROW_SUFFIX, 
ByteUtil.nextKey(startRowSuffix));
+} else {
+
newScan.setStartRow(ByteUtil.nextKey(startRowSuffix));
--- End diff --

See other comment below, but how about we serialize the row key we found 
which is out-of-region in the StaleRegionBoundaryCacheException instead of 
mucking with the startRowSuffix?


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

[jira] [Commented] (PHOENIX-2628) Ensure split when iterating through results handled correctly

[ASF GitHub Bot (JIRA)]

[ 
https://issues.apache.org/jira/browse/PHOENIX-2628?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15227523#comment-15227523
 ] 

ASF GitHub Bot commented on PHOENIX-2628:
-

Github user JamesRTaylor commented on a diff in the pull request:

https://github.com/apache/phoenix/pull/156#discussion_r58640556
  
--- Diff: 
phoenix-core/src/main/java/org/apache/phoenix/iterate/TableResultIterator.java 
---
@@ -107,8 +127,37 @@ public synchronized void close() throws SQLException {
 @Override
 public synchronized Tuple next() throws SQLException {
 initScanner();
-Tuple t = scanIterator.next();
-return t;
+try {
+lastTuple = scanIterator.next();
+if (lastTuple != null) {
+ImmutableBytesWritable ptr = new ImmutableBytesWritable();
+lastTuple.getKey(ptr);
+}
+} catch (SQLException e) {
+try {
+throw ServerUtil.parseServerException(e);
+} catch(StaleRegionBoundaryCacheException e1) {
+if(scan.getAttribute(NON_AGGREGATE_QUERY)!=null) {
+Scan newScan = ScanUtil.newScan(scan);
+if(lastTuple != null) {
+lastTuple.getKey(ptr);
+byte[] startRowSuffix = 
ByteUtil.copyKeyBytesIfNecessary(ptr);
+if(ScanUtil.isLocalIndex(newScan)) {
+newScan.setAttribute(SCAN_START_ROW_SUFFIX, 
ByteUtil.nextKey(startRowSuffix));
+} else {
+
newScan.setStartRow(ByteUtil.nextKey(startRowSuffix));
+}
+}
+
plan.getContext().getConnection().getQueryServices().clearTableRegionCache(htable.getTableName());
+this.scanIterator =
+
plan.iterator(DefaultParallelScanGrouper.getInstance(), newScan);
--- End diff --

Yes, exactly. Except for the ordered aggregation case, which needs to be 
handled as the non aggregate is handled (i.e. we'd still throw the stale region 
boundary exception, but the client would need to react to it as you've done for 
the non aggregate case. We need to know which exact row key we're on though 
(which the client won't know). Seems like the more feasible approach would be 
to include the row key in the exception and deserialize it on the client side.


> Ensure split when iterating through results handled correctly
> -
>
> Key: PHOENIX-2628
> URL: https://issues.apache.org/jira/browse/PHOENIX-2628
> Project: Phoenix
>  Issue Type: Bug
>Reporter: James Taylor
>Assignee: Rajeshbabu Chintaguntla
> Fix For: 4.8.0
>
> Attachments: PHOENIX-2628-wip.patch, PHOENIX-2628.patch, 
> PHOENIX-2628_v7.patch, PHOENIX-2628_v8.patch
>
>
> We should start with a test case to ensure this works correctly, both for 
> scans and aggregates.



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[GitHub] phoenix pull request: PHOENIX-2628 Ensure split when iterating thr...

[JamesRTaylor]

Github user JamesRTaylor commented on a diff in the pull request:

https://github.com/apache/phoenix/pull/156#discussion_r58640556
  
--- Diff: 
phoenix-core/src/main/java/org/apache/phoenix/iterate/TableResultIterator.java 
---
@@ -107,8 +127,37 @@ public synchronized void close() throws SQLException {
 @Override
 public synchronized Tuple next() throws SQLException {
 initScanner();
-Tuple t = scanIterator.next();
-return t;
+try {
+lastTuple = scanIterator.next();
+if (lastTuple != null) {
+ImmutableBytesWritable ptr = new ImmutableBytesWritable();
+lastTuple.getKey(ptr);
+}
+} catch (SQLException e) {
+try {
+throw ServerUtil.parseServerException(e);
+} catch(StaleRegionBoundaryCacheException e1) {
+if(scan.getAttribute(NON_AGGREGATE_QUERY)!=null) {
+Scan newScan = ScanUtil.newScan(scan);
+if(lastTuple != null) {
+lastTuple.getKey(ptr);
+byte[] startRowSuffix = 
ByteUtil.copyKeyBytesIfNecessary(ptr);
+if(ScanUtil.isLocalIndex(newScan)) {
+newScan.setAttribute(SCAN_START_ROW_SUFFIX, 
ByteUtil.nextKey(startRowSuffix));
+} else {
+
newScan.setStartRow(ByteUtil.nextKey(startRowSuffix));
+}
+}
+
plan.getContext().getConnection().getQueryServices().clearTableRegionCache(htable.getTableName());
+this.scanIterator =
+
plan.iterator(DefaultParallelScanGrouper.getInstance(), newScan);
--- End diff --

Yes, exactly. Except for the ordered aggregation case, which needs to be 
handled as the non aggregate is handled (i.e. we'd still throw the stale region 
boundary exception, but the client would need to react to it as you've done for 
the non aggregate case. We need to know which exact row key we're on though 
(which the client won't know). Seems like the more feasible approach would be 
to include the row key in the exception and deserialize it on the client side.


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

[jira] [Commented] (PHOENIX-2628) Ensure split when iterating through results handled correctly

[ASF GitHub Bot (JIRA)]

[ 
https://issues.apache.org/jira/browse/PHOENIX-2628?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15227518#comment-15227518
 ] 

ASF GitHub Bot commented on PHOENIX-2628:
-

Github user chrajeshbabu commented on a diff in the pull request:

https://github.com/apache/phoenix/pull/156#discussion_r58640455
  
--- Diff: 
phoenix-core/src/main/java/org/apache/phoenix/iterate/BaseResultIterators.java 
---
@@ -556,35 +564,55 @@ private static String toString(List gps) {
 } else {
 endKey = regionBoundaries.get(regionIndex);
 }
-HRegionLocation regionLocation = 
regionLocations.get(regionIndex);
-if (isLocalIndex) {
-HRegionInfo regionInfo = 
regionLocation.getRegionInfo();
-endRegionKey = regionInfo.getEndKey();
-keyOffset = 
ScanUtil.getRowKeyOffset(regionInfo.getStartKey(), endRegionKey);
-}
-try {
-while (guideIndex < gpsSize && 
(currentGuidePost.compareTo(endKey) <= 0 || endKey.length == 0)) {
-Scan newScan = scanRanges.intersectScan(scan, 
currentKeyBytes, currentGuidePostBytes, keyOffset,
-false);
-estimatedRows += 
gps.getRowCounts().get(guideIndex);
-estimatedSize += 
gps.getByteCounts().get(guideIndex);
-scans = addNewScan(parallelScans, scans, newScan, 
currentGuidePostBytes, false, regionLocation);
-currentKeyBytes = currentGuidePost.copyBytes();
-currentGuidePost = PrefixByteCodec.decode(decoder, 
input);
-currentGuidePostBytes = 
currentGuidePost.copyBytes();
-guideIndex++;
-}
-} catch (EOFException e) {}
-Scan newScan = scanRanges.intersectScan(scan, 
currentKeyBytes, endKey, keyOffset, true);
-if (isLocalIndex) {
-if (newScan != null) {
-newScan.setAttribute(EXPECTED_UPPER_REGION_KEY, 
endRegionKey);
-} else if (!scans.isEmpty()) {
-
scans.get(scans.size()-1).setAttribute(EXPECTED_UPPER_REGION_KEY, endRegionKey);
-}
-}
-scans = addNewScan(parallelScans, scans, newScan, endKey, 
true, regionLocation);
-currentKeyBytes = endKey;
+if (Bytes.compareTo(scan.getStartRow(), 
context.getScan().getStartRow()) != 0
+ || Bytes.compareTo(scan.getStopRow(), 
context.getScan().getStopRow()) != 0) {
+Scan newScan = ScanUtil.newScan(scan);
+if(ScanUtil.isLocalIndex(scan)) {
+newScan.setStartRow(regionInfo.getStartKey());
+newScan.setAttribute(SCAN_ACTUAL_START_ROW, 
regionInfo.getStartKey());
--- End diff --

First one is actual start key of the scan second one is end key of region. 
After this patch we don't need EXPECTED_UPPER_REGION_KEY but kept it for 
compatibility.


> Ensure split when iterating through results handled correctly
> -
>
> Key: PHOENIX-2628
> URL: https://issues.apache.org/jira/browse/PHOENIX-2628
> Project: Phoenix
>  Issue Type: Bug
>Reporter: James Taylor
>Assignee: Rajeshbabu Chintaguntla
> Fix For: 4.8.0
>
> Attachments: PHOENIX-2628-wip.patch, PHOENIX-2628.patch, 
> PHOENIX-2628_v7.patch, PHOENIX-2628_v8.patch
>
>
> We should start with a test case to ensure this works correctly, both for 
> scans and aggregates.



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[GitHub] phoenix pull request: PHOENIX-2628 Ensure split when iterating thr...

[chrajeshbabu]

Github user chrajeshbabu commented on a diff in the pull request:

https://github.com/apache/phoenix/pull/156#discussion_r58640455
  
--- Diff: 
phoenix-core/src/main/java/org/apache/phoenix/iterate/BaseResultIterators.java 
---
@@ -556,35 +564,55 @@ private static String toString(List gps) {
 } else {
 endKey = regionBoundaries.get(regionIndex);
 }
-HRegionLocation regionLocation = 
regionLocations.get(regionIndex);
-if (isLocalIndex) {
-HRegionInfo regionInfo = 
regionLocation.getRegionInfo();
-endRegionKey = regionInfo.getEndKey();
-keyOffset = 
ScanUtil.getRowKeyOffset(regionInfo.getStartKey(), endRegionKey);
-}
-try {
-while (guideIndex < gpsSize && 
(currentGuidePost.compareTo(endKey) <= 0 || endKey.length == 0)) {
-Scan newScan = scanRanges.intersectScan(scan, 
currentKeyBytes, currentGuidePostBytes, keyOffset,
-false);
-estimatedRows += 
gps.getRowCounts().get(guideIndex);
-estimatedSize += 
gps.getByteCounts().get(guideIndex);
-scans = addNewScan(parallelScans, scans, newScan, 
currentGuidePostBytes, false, regionLocation);
-currentKeyBytes = currentGuidePost.copyBytes();
-currentGuidePost = PrefixByteCodec.decode(decoder, 
input);
-currentGuidePostBytes = 
currentGuidePost.copyBytes();
-guideIndex++;
-}
-} catch (EOFException e) {}
-Scan newScan = scanRanges.intersectScan(scan, 
currentKeyBytes, endKey, keyOffset, true);
-if (isLocalIndex) {
-if (newScan != null) {
-newScan.setAttribute(EXPECTED_UPPER_REGION_KEY, 
endRegionKey);
-} else if (!scans.isEmpty()) {
-
scans.get(scans.size()-1).setAttribute(EXPECTED_UPPER_REGION_KEY, endRegionKey);
-}
-}
-scans = addNewScan(parallelScans, scans, newScan, endKey, 
true, regionLocation);
-currentKeyBytes = endKey;
+if (Bytes.compareTo(scan.getStartRow(), 
context.getScan().getStartRow()) != 0
+ || Bytes.compareTo(scan.getStopRow(), 
context.getScan().getStopRow()) != 0) {
+Scan newScan = ScanUtil.newScan(scan);
+if(ScanUtil.isLocalIndex(scan)) {
+newScan.setStartRow(regionInfo.getStartKey());
+newScan.setAttribute(SCAN_ACTUAL_START_ROW, 
regionInfo.getStartKey());
--- End diff --

First one is actual start key of the scan second one is end key of region. 
After this patch we don't need EXPECTED_UPPER_REGION_KEY but kept it for 
compatibility.


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

[jira] [Commented] (PHOENIX-2628) Ensure split when iterating through results handled correctly

[ASF GitHub Bot (JIRA)]

[ 
https://issues.apache.org/jira/browse/PHOENIX-2628?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15227514#comment-15227514
 ] 

ASF GitHub Bot commented on PHOENIX-2628:
-

Github user chrajeshbabu commented on a diff in the pull request:

https://github.com/apache/phoenix/pull/156#discussion_r58640276
  
--- Diff: 
phoenix-core/src/main/java/org/apache/phoenix/iterate/TableResultIterator.java 
---
@@ -121,8 +170,21 @@ public synchronized void initScanner() throws 
SQLException {
 this.scanIterator =
 new 
ScanningResultIterator(htable.getScanner(scan), scanMetrics);
 } catch (IOException e) {
-Closeables.closeQuietly(htable);
-throw ServerUtil.parseServerException(e);
+if(handleSplitRegionBoundaryFailureDuringInitialization) {
--- End diff --

This is required for ChunkedResultIterator if we are going to deprecate or 
not use it we don't need these changes. Other than that we might need to deal 
with it in PhoenixRecordReader(currently we skipping range check but for local 
indexes it's compulsory to handle otherwise we might miss one of the daughter 
records)


> Ensure split when iterating through results handled correctly
> -
>
> Key: PHOENIX-2628
> URL: https://issues.apache.org/jira/browse/PHOENIX-2628
> Project: Phoenix
>  Issue Type: Bug
>Reporter: James Taylor
>Assignee: Rajeshbabu Chintaguntla
> Fix For: 4.8.0
>
> Attachments: PHOENIX-2628-wip.patch, PHOENIX-2628.patch, 
> PHOENIX-2628_v7.patch, PHOENIX-2628_v8.patch
>
>
> We should start with a test case to ensure this works correctly, both for 
> scans and aggregates.



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[GitHub] phoenix pull request: PHOENIX-2628 Ensure split when iterating thr...

[chrajeshbabu]

Github user chrajeshbabu commented on a diff in the pull request:

https://github.com/apache/phoenix/pull/156#discussion_r58640276
  
--- Diff: 
phoenix-core/src/main/java/org/apache/phoenix/iterate/TableResultIterator.java 
---
@@ -121,8 +170,21 @@ public synchronized void initScanner() throws 
SQLException {
 this.scanIterator =
 new 
ScanningResultIterator(htable.getScanner(scan), scanMetrics);
 } catch (IOException e) {
-Closeables.closeQuietly(htable);
-throw ServerUtil.parseServerException(e);
+if(handleSplitRegionBoundaryFailureDuringInitialization) {
--- End diff --

This is required for ChunkedResultIterator if we are going to deprecate or 
not use it we don't need these changes. Other than that we might need to deal 
with it in PhoenixRecordReader(currently we skipping range check but for local 
indexes it's compulsory to handle otherwise we might miss one of the daughter 
records)


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

[GitHub] phoenix pull request: PHOENIX-2628 Ensure split when iterating thr...

[chrajeshbabu]

Github user chrajeshbabu commented on a diff in the pull request:

https://github.com/apache/phoenix/pull/156#discussion_r58639974
  
--- Diff: 
phoenix-core/src/main/java/org/apache/phoenix/iterate/TableResultIterator.java 
---
@@ -107,8 +127,37 @@ public synchronized void close() throws SQLException {
 @Override
 public synchronized Tuple next() throws SQLException {
 initScanner();
-Tuple t = scanIterator.next();
-return t;
+try {
+lastTuple = scanIterator.next();
+if (lastTuple != null) {
+ImmutableBytesWritable ptr = new ImmutableBytesWritable();
+lastTuple.getKey(ptr);
+}
+} catch (SQLException e) {
+try {
+throw ServerUtil.parseServerException(e);
+} catch(StaleRegionBoundaryCacheException e1) {
+if(scan.getAttribute(NON_AGGREGATE_QUERY)!=null) {
--- End diff --

Will move the check to ScanUtil.


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

[GitHub] phoenix pull request: PHOENIX-2628 Ensure split when iterating thr...

[chrajeshbabu]

Github user chrajeshbabu commented on a diff in the pull request:

https://github.com/apache/phoenix/pull/156#discussion_r58639989
  
--- Diff: 
phoenix-core/src/main/java/org/apache/phoenix/iterate/TableResultIterator.java 
---
@@ -107,8 +127,37 @@ public synchronized void close() throws SQLException {
 @Override
 public synchronized Tuple next() throws SQLException {
 initScanner();
-Tuple t = scanIterator.next();
-return t;
+try {
+lastTuple = scanIterator.next();
+if (lastTuple != null) {
+ImmutableBytesWritable ptr = new ImmutableBytesWritable();
+lastTuple.getKey(ptr);
+}
+} catch (SQLException e) {
+try {
+throw ServerUtil.parseServerException(e);
+} catch(StaleRegionBoundaryCacheException e1) {
--- End diff --

I will add.


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

[jira] [Commented] (PHOENIX-2628) Ensure split when iterating through results handled correctly

[ASF GitHub Bot (JIRA)]

[ 
https://issues.apache.org/jira/browse/PHOENIX-2628?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15227508#comment-15227508
 ] 

ASF GitHub Bot commented on PHOENIX-2628:
-

Github user chrajeshbabu commented on a diff in the pull request:

https://github.com/apache/phoenix/pull/156#discussion_r58639989
  
--- Diff: 
phoenix-core/src/main/java/org/apache/phoenix/iterate/TableResultIterator.java 
---
@@ -107,8 +127,37 @@ public synchronized void close() throws SQLException {
 @Override
 public synchronized Tuple next() throws SQLException {
 initScanner();
-Tuple t = scanIterator.next();
-return t;
+try {
+lastTuple = scanIterator.next();
+if (lastTuple != null) {
+ImmutableBytesWritable ptr = new ImmutableBytesWritable();
+lastTuple.getKey(ptr);
+}
+} catch (SQLException e) {
+try {
+throw ServerUtil.parseServerException(e);
+} catch(StaleRegionBoundaryCacheException e1) {
--- End diff --

I will add.


> Ensure split when iterating through results handled correctly
> -
>
> Key: PHOENIX-2628
> URL: https://issues.apache.org/jira/browse/PHOENIX-2628
> Project: Phoenix
>  Issue Type: Bug
>Reporter: James Taylor
>Assignee: Rajeshbabu Chintaguntla
> Fix For: 4.8.0
>
> Attachments: PHOENIX-2628-wip.patch, PHOENIX-2628.patch, 
> PHOENIX-2628_v7.patch, PHOENIX-2628_v8.patch
>
>
> We should start with a test case to ensure this works correctly, both for 
> scans and aggregates.



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Commented] (PHOENIX-2628) Ensure split when iterating through results handled correctly

[ASF GitHub Bot (JIRA)]

[ 
https://issues.apache.org/jira/browse/PHOENIX-2628?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15227506#comment-15227506
 ] 

ASF GitHub Bot commented on PHOENIX-2628:
-

Github user chrajeshbabu commented on a diff in the pull request:

https://github.com/apache/phoenix/pull/156#discussion_r58639956
  
--- Diff: 
phoenix-core/src/main/java/org/apache/phoenix/iterate/TableResultIterator.java 
---
@@ -107,8 +127,37 @@ public synchronized void close() throws SQLException {
 @Override
 public synchronized Tuple next() throws SQLException {
 initScanner();
-Tuple t = scanIterator.next();
-return t;
+try {
+lastTuple = scanIterator.next();
+if (lastTuple != null) {
+ImmutableBytesWritable ptr = new ImmutableBytesWritable();
+lastTuple.getKey(ptr);
+}
+} catch (SQLException e) {
+try {
+throw ServerUtil.parseServerException(e);
+} catch(StaleRegionBoundaryCacheException e1) {
+if(scan.getAttribute(NON_AGGREGATE_QUERY)!=null) {
+Scan newScan = ScanUtil.newScan(scan);
+if(lastTuple != null) {
+lastTuple.getKey(ptr);
+byte[] startRowSuffix = 
ByteUtil.copyKeyBytesIfNecessary(ptr);
+if(ScanUtil.isLocalIndex(newScan)) {
+newScan.setAttribute(SCAN_START_ROW_SUFFIX, 
ByteUtil.nextKey(startRowSuffix));
+} else {
+
newScan.setStartRow(ByteUtil.nextKey(startRowSuffix));
--- End diff --

Will raise improvement action for this James.


> Ensure split when iterating through results handled correctly
> -
>
> Key: PHOENIX-2628
> URL: https://issues.apache.org/jira/browse/PHOENIX-2628
> Project: Phoenix
>  Issue Type: Bug
>Reporter: James Taylor
>Assignee: Rajeshbabu Chintaguntla
> Fix For: 4.8.0
>
> Attachments: PHOENIX-2628-wip.patch, PHOENIX-2628.patch, 
> PHOENIX-2628_v7.patch, PHOENIX-2628_v8.patch
>
>
> We should start with a test case to ensure this works correctly, both for 
> scans and aggregates.



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Commented] (PHOENIX-2628) Ensure split when iterating through results handled correctly

[ASF GitHub Bot (JIRA)]

[ 
https://issues.apache.org/jira/browse/PHOENIX-2628?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15227507#comment-15227507
 ] 

ASF GitHub Bot commented on PHOENIX-2628:
-

Github user chrajeshbabu commented on a diff in the pull request:

https://github.com/apache/phoenix/pull/156#discussion_r58639974
  
--- Diff: 
phoenix-core/src/main/java/org/apache/phoenix/iterate/TableResultIterator.java 
---
@@ -107,8 +127,37 @@ public synchronized void close() throws SQLException {
 @Override
 public synchronized Tuple next() throws SQLException {
 initScanner();
-Tuple t = scanIterator.next();
-return t;
+try {
+lastTuple = scanIterator.next();
+if (lastTuple != null) {
+ImmutableBytesWritable ptr = new ImmutableBytesWritable();
+lastTuple.getKey(ptr);
+}
+} catch (SQLException e) {
+try {
+throw ServerUtil.parseServerException(e);
+} catch(StaleRegionBoundaryCacheException e1) {
+if(scan.getAttribute(NON_AGGREGATE_QUERY)!=null) {
--- End diff --

Will move the check to ScanUtil.


> Ensure split when iterating through results handled correctly
> -
>
> Key: PHOENIX-2628
> URL: https://issues.apache.org/jira/browse/PHOENIX-2628
> Project: Phoenix
>  Issue Type: Bug
>Reporter: James Taylor
>Assignee: Rajeshbabu Chintaguntla
> Fix For: 4.8.0
>
> Attachments: PHOENIX-2628-wip.patch, PHOENIX-2628.patch, 
> PHOENIX-2628_v7.patch, PHOENIX-2628_v8.patch
>
>
> We should start with a test case to ensure this works correctly, both for 
> scans and aggregates.



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[GitHub] phoenix pull request: PHOENIX-2628 Ensure split when iterating thr...

[chrajeshbabu]

Github user chrajeshbabu commented on a diff in the pull request:

https://github.com/apache/phoenix/pull/156#discussion_r58639956
  
--- Diff: 
phoenix-core/src/main/java/org/apache/phoenix/iterate/TableResultIterator.java 
---
@@ -107,8 +127,37 @@ public synchronized void close() throws SQLException {
 @Override
 public synchronized Tuple next() throws SQLException {
 initScanner();
-Tuple t = scanIterator.next();
-return t;
+try {
+lastTuple = scanIterator.next();
+if (lastTuple != null) {
+ImmutableBytesWritable ptr = new ImmutableBytesWritable();
+lastTuple.getKey(ptr);
+}
+} catch (SQLException e) {
+try {
+throw ServerUtil.parseServerException(e);
+} catch(StaleRegionBoundaryCacheException e1) {
+if(scan.getAttribute(NON_AGGREGATE_QUERY)!=null) {
+Scan newScan = ScanUtil.newScan(scan);
+if(lastTuple != null) {
+lastTuple.getKey(ptr);
+byte[] startRowSuffix = 
ByteUtil.copyKeyBytesIfNecessary(ptr);
+if(ScanUtil.isLocalIndex(newScan)) {
+newScan.setAttribute(SCAN_START_ROW_SUFFIX, 
ByteUtil.nextKey(startRowSuffix));
+} else {
+
newScan.setStartRow(ByteUtil.nextKey(startRowSuffix));
--- End diff --

Will raise improvement action for this James.


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

[jira] [Commented] (PHOENIX-2628) Ensure split when iterating through results handled correctly

[ASF GitHub Bot (JIRA)]

[ 
https://issues.apache.org/jira/browse/PHOENIX-2628?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15227501#comment-15227501
 ] 

ASF GitHub Bot commented on PHOENIX-2628:
-

Github user chrajeshbabu commented on a diff in the pull request:

https://github.com/apache/phoenix/pull/156#discussion_r58639845
  
--- Diff: 
phoenix-core/src/main/java/org/apache/phoenix/iterate/TableResultIterator.java 
---
@@ -80,13 +94,19 @@
 };
 
 
-public TableResultIterator(MutationState mutationState, TableRef 
tableRef, Scan scan, CombinableMetric scanMetrics, long renewLeaseThreshold) 
throws SQLException {
+public TableResultIterator(MutationState mutationState, Scan scan, 
CombinableMetric scanMetrics, long renewLeaseThreshold, QueryPlan plan) throws 
SQLException {
+this(mutationState, scan, scanMetrics, renewLeaseThreshold, plan, 
false);
+}
+
+public TableResultIterator(MutationState mutationState, Scan scan, 
CombinableMetric scanMetrics, long renewLeaseThreshold, QueryPlan plan, boolean 
handleSplitRegionBoundaryFailureDuringInitialization) throws SQLException {
 this.scan = scan;
 this.scanMetrics = scanMetrics;
-PTable table = tableRef.getTable();
+PTable table = plan.getTableRef().getTable();
 htable = mutationState.getHTable(table);
 this.scanIterator = UNINITIALIZED_SCANNER;
 this.renewLeaseThreshold = renewLeaseThreshold;
+this.plan = plan;
+this.handleSplitRegionBoundaryFailureDuringInitialization = 
handleSplitRegionBoundaryFailureDuringInitialization;
--- End diff --

handleSplitRegionBoundaryFailureDuringInitialization will be true for 
scanning intermediate chunks in ChunkedResultIterator. When we create scanner 
for new chunk we might see stale region boundaries in that case also we need to 
forcibly recreate the iterator with new boundaries. In normal case if we get 
StaleRegionBoundary exception while creating scanner BaseResultIterator handles 
it. 


> Ensure split when iterating through results handled correctly
> -
>
> Key: PHOENIX-2628
> URL: https://issues.apache.org/jira/browse/PHOENIX-2628
> Project: Phoenix
>  Issue Type: Bug
>Reporter: James Taylor
>Assignee: Rajeshbabu Chintaguntla
> Fix For: 4.8.0
>
> Attachments: PHOENIX-2628-wip.patch, PHOENIX-2628.patch, 
> PHOENIX-2628_v7.patch, PHOENIX-2628_v8.patch
>
>
> We should start with a test case to ensure this works correctly, both for 
> scans and aggregates.



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[GitHub] phoenix pull request: PHOENIX-2628 Ensure split when iterating thr...

[chrajeshbabu]

Github user chrajeshbabu commented on a diff in the pull request:

https://github.com/apache/phoenix/pull/156#discussion_r58639845
  
--- Diff: 
phoenix-core/src/main/java/org/apache/phoenix/iterate/TableResultIterator.java 
---
@@ -80,13 +94,19 @@
 };
 
 
-public TableResultIterator(MutationState mutationState, TableRef 
tableRef, Scan scan, CombinableMetric scanMetrics, long renewLeaseThreshold) 
throws SQLException {
+public TableResultIterator(MutationState mutationState, Scan scan, 
CombinableMetric scanMetrics, long renewLeaseThreshold, QueryPlan plan) throws 
SQLException {
+this(mutationState, scan, scanMetrics, renewLeaseThreshold, plan, 
false);
+}
+
+public TableResultIterator(MutationState mutationState, Scan scan, 
CombinableMetric scanMetrics, long renewLeaseThreshold, QueryPlan plan, boolean 
handleSplitRegionBoundaryFailureDuringInitialization) throws SQLException {
 this.scan = scan;
 this.scanMetrics = scanMetrics;
-PTable table = tableRef.getTable();
+PTable table = plan.getTableRef().getTable();
 htable = mutationState.getHTable(table);
 this.scanIterator = UNINITIALIZED_SCANNER;
 this.renewLeaseThreshold = renewLeaseThreshold;
+this.plan = plan;
+this.handleSplitRegionBoundaryFailureDuringInitialization = 
handleSplitRegionBoundaryFailureDuringInitialization;
--- End diff --

handleSplitRegionBoundaryFailureDuringInitialization will be true for 
scanning intermediate chunks in ChunkedResultIterator. When we create scanner 
for new chunk we might see stale region boundaries in that case also we need to 
forcibly recreate the iterator with new boundaries. In normal case if we get 
StaleRegionBoundary exception while creating scanner BaseResultIterator handles 
it. 


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

[jira] [Commented] (PHOENIX-2628) Ensure split when iterating through results handled correctly

[ASF GitHub Bot (JIRA)]

[ 
https://issues.apache.org/jira/browse/PHOENIX-2628?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15227496#comment-15227496
 ] 

ASF GitHub Bot commented on PHOENIX-2628:
-

Github user chrajeshbabu commented on a diff in the pull request:

https://github.com/apache/phoenix/pull/156#discussion_r58639358
  
--- Diff: 
phoenix-core/src/main/java/org/apache/phoenix/iterate/TableResultIterator.java 
---
@@ -107,8 +127,37 @@ public synchronized void close() throws SQLException {
 @Override
 public synchronized Tuple next() throws SQLException {
 initScanner();
-Tuple t = scanIterator.next();
-return t;
+try {
+lastTuple = scanIterator.next();
+if (lastTuple != null) {
+ImmutableBytesWritable ptr = new ImmutableBytesWritable();
+lastTuple.getKey(ptr);
+}
+} catch (SQLException e) {
+try {
+throw ServerUtil.parseServerException(e);
+} catch(StaleRegionBoundaryCacheException e1) {
+if(scan.getAttribute(NON_AGGREGATE_QUERY)!=null) {
+Scan newScan = ScanUtil.newScan(scan);
+if(lastTuple != null) {
+lastTuple.getKey(ptr);
+byte[] startRowSuffix = 
ByteUtil.copyKeyBytesIfNecessary(ptr);
+if(ScanUtil.isLocalIndex(newScan)) {
+newScan.setAttribute(SCAN_START_ROW_SUFFIX, 
ByteUtil.nextKey(startRowSuffix));
+} else {
+
newScan.setStartRow(ByteUtil.nextKey(startRowSuffix));
+}
+}
+
plan.getContext().getConnection().getQueryServices().clearTableRegionCache(htable.getTableName());
+this.scanIterator =
+
plan.iterator(DefaultParallelScanGrouper.getInstance(), newScan);
--- End diff --

You mean we can check the in region boundary again in post scanner open and 
if it's out of range throw stale region boundary exception which will be 
handled by BaseResultIterators?


> Ensure split when iterating through results handled correctly
> -
>
> Key: PHOENIX-2628
> URL: https://issues.apache.org/jira/browse/PHOENIX-2628
> Project: Phoenix
>  Issue Type: Bug
>Reporter: James Taylor
>Assignee: Rajeshbabu Chintaguntla
> Fix For: 4.8.0
>
> Attachments: PHOENIX-2628-wip.patch, PHOENIX-2628.patch, 
> PHOENIX-2628_v7.patch, PHOENIX-2628_v8.patch
>
>
> We should start with a test case to ensure this works correctly, both for 
> scans and aggregates.



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[GitHub] phoenix pull request: PHOENIX-2628 Ensure split when iterating thr...

[chrajeshbabu]

Github user chrajeshbabu commented on a diff in the pull request:

https://github.com/apache/phoenix/pull/156#discussion_r58639358
  
--- Diff: 
phoenix-core/src/main/java/org/apache/phoenix/iterate/TableResultIterator.java 
---
@@ -107,8 +127,37 @@ public synchronized void close() throws SQLException {
 @Override
 public synchronized Tuple next() throws SQLException {
 initScanner();
-Tuple t = scanIterator.next();
-return t;
+try {
+lastTuple = scanIterator.next();
+if (lastTuple != null) {
+ImmutableBytesWritable ptr = new ImmutableBytesWritable();
+lastTuple.getKey(ptr);
+}
+} catch (SQLException e) {
+try {
+throw ServerUtil.parseServerException(e);
+} catch(StaleRegionBoundaryCacheException e1) {
+if(scan.getAttribute(NON_AGGREGATE_QUERY)!=null) {
+Scan newScan = ScanUtil.newScan(scan);
+if(lastTuple != null) {
+lastTuple.getKey(ptr);
+byte[] startRowSuffix = 
ByteUtil.copyKeyBytesIfNecessary(ptr);
+if(ScanUtil.isLocalIndex(newScan)) {
+newScan.setAttribute(SCAN_START_ROW_SUFFIX, 
ByteUtil.nextKey(startRowSuffix));
+} else {
+
newScan.setStartRow(ByteUtil.nextKey(startRowSuffix));
+}
+}
+
plan.getContext().getConnection().getQueryServices().clearTableRegionCache(htable.getTableName());
+this.scanIterator =
+
plan.iterator(DefaultParallelScanGrouper.getInstance(), newScan);
--- End diff --

You mean we can check the in region boundary again in post scanner open and 
if it's out of range throw stale region boundary exception which will be 
handled by BaseResultIterators?


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

[jira] [Updated] (PHOENIX-1614) ALTER TABLE ADD IF NOT EXISTS doesn't work as expected

[James Taylor (JIRA)]

 [ 
https://issues.apache.org/jira/browse/PHOENIX-1614?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

James Taylor updated PHOENIX-1614:
--
Labels: argus  (was: )

> ALTER TABLE ADD IF NOT EXISTS doesn't work as expected
> --
>
> Key: PHOENIX-1614
> URL: https://issues.apache.org/jira/browse/PHOENIX-1614
> Project: Phoenix
>  Issue Type: Bug
>Reporter: Gabriel Reid
>Assignee: Thomas D'Silva
>  Labels: argus
> Fix For: 4.8.0
>
>
> On an existing table table, executing "ALTER TABLE ADD IF NOT EXISTS
> thisColAlreadyExists varchar, thisColDoesNotExist varchar", then
> nothing will be changed in the table because thisColAlreadyExists
> already exists.
> Omitting the already-existing column from the statement, all new columns
> do get created.
> The general expectation would be that when you use ADD IF NOT EXISTS, all
> non-existent columns will be added, and all existing columns in the
> statement will just be ignored. There is already an integration test
> (AlterTableIT#testAddVarCols) that actually demonstrates the current
> behavior, although this is probably not correct.
> As pointed out in the related mailing list thread [1], ALTER TABLE DROP 
> COLUMN likely suffers from the same issue.
> 1. http://s.apache.org/LMT 



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Updated] (PHOENIX-1614) ALTER TABLE ADD IF NOT EXISTS doesn't work as expected

[James Taylor (JIRA)]

 [ 
https://issues.apache.org/jira/browse/PHOENIX-1614?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

James Taylor updated PHOENIX-1614:
--
Fix Version/s: 4.8.0

> ALTER TABLE ADD IF NOT EXISTS doesn't work as expected
> --
>
> Key: PHOENIX-1614
> URL: https://issues.apache.org/jira/browse/PHOENIX-1614
> Project: Phoenix
>  Issue Type: Bug
>Reporter: Gabriel Reid
>Assignee: Thomas D'Silva
>  Labels: argus
> Fix For: 4.8.0
>
>
> On an existing table table, executing "ALTER TABLE ADD IF NOT EXISTS
> thisColAlreadyExists varchar, thisColDoesNotExist varchar", then
> nothing will be changed in the table because thisColAlreadyExists
> already exists.
> Omitting the already-existing column from the statement, all new columns
> do get created.
> The general expectation would be that when you use ADD IF NOT EXISTS, all
> non-existent columns will be added, and all existing columns in the
> statement will just be ignored. There is already an integration test
> (AlterTableIT#testAddVarCols) that actually demonstrates the current
> behavior, although this is probably not correct.
> As pointed out in the related mailing list thread [1], ALTER TABLE DROP 
> COLUMN likely suffers from the same issue.
> 1. http://s.apache.org/LMT 



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Updated] (PHOENIX-1614) ALTER TABLE ADD IF NOT EXISTS doesn't work as expected

[James Taylor (JIRA)]

 [ 
https://issues.apache.org/jira/browse/PHOENIX-1614?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

James Taylor updated PHOENIX-1614:
--
Assignee: Thomas D'Silva

> ALTER TABLE ADD IF NOT EXISTS doesn't work as expected
> --
>
> Key: PHOENIX-1614
> URL: https://issues.apache.org/jira/browse/PHOENIX-1614
> Project: Phoenix
>  Issue Type: Bug
>Reporter: Gabriel Reid
>Assignee: Thomas D'Silva
>  Labels: argus
> Fix For: 4.8.0
>
>
> On an existing table table, executing "ALTER TABLE ADD IF NOT EXISTS
> thisColAlreadyExists varchar, thisColDoesNotExist varchar", then
> nothing will be changed in the table because thisColAlreadyExists
> already exists.
> Omitting the already-existing column from the statement, all new columns
> do get created.
> The general expectation would be that when you use ADD IF NOT EXISTS, all
> non-existent columns will be added, and all existing columns in the
> statement will just be ignored. There is already an integration test
> (AlterTableIT#testAddVarCols) that actually demonstrates the current
> behavior, although this is probably not correct.
> As pointed out in the related mailing list thread [1], ALTER TABLE DROP 
> COLUMN likely suffers from the same issue.
> 1. http://s.apache.org/LMT 



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Commented] (PHOENIX-2628) Ensure split when iterating through results handled correctly

[ASF GitHub Bot (JIRA)]

[ 
https://issues.apache.org/jira/browse/PHOENIX-2628?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15227489#comment-15227489
 ] 

ASF GitHub Bot commented on PHOENIX-2628:
-

Github user chrajeshbabu commented on a diff in the pull request:

https://github.com/apache/phoenix/pull/156#discussion_r58638946
  
--- Diff: 
phoenix-core/src/main/java/org/apache/phoenix/compile/ScanRanges.java ---
@@ -385,9 +387,25 @@ public Scan intersectScan(Scan scan, final byte[] 
originalStartKey, final byte[]
 if (scanStopKey.length > 0 && Bytes.compareTo(scanStartKey, 
scanStopKey) >= 0) { 
 return null; 
 }
-newScan.setAttribute(SCAN_ACTUAL_START_ROW, scanStartKey);
-newScan.setStartRow(scanStartKey);
-newScan.setStopRow(scanStopKey);
+if(ScanUtil.isLocalIndex(scan)) {
--- End diff --

I will try moving outside of this method.


> Ensure split when iterating through results handled correctly
> -
>
> Key: PHOENIX-2628
> URL: https://issues.apache.org/jira/browse/PHOENIX-2628
> Project: Phoenix
>  Issue Type: Bug
>Reporter: James Taylor
>Assignee: Rajeshbabu Chintaguntla
> Fix For: 4.8.0
>
> Attachments: PHOENIX-2628-wip.patch, PHOENIX-2628.patch, 
> PHOENIX-2628_v7.patch, PHOENIX-2628_v8.patch
>
>
> We should start with a test case to ensure this works correctly, both for 
> scans and aggregates.



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[GitHub] phoenix pull request: PHOENIX-2628 Ensure split when iterating thr...

[chrajeshbabu]

Github user chrajeshbabu commented on a diff in the pull request:

https://github.com/apache/phoenix/pull/156#discussion_r58638911
  
--- Diff: 
phoenix-core/src/main/java/org/apache/phoenix/coprocessor/GroupedAggregateRegionObserver.java
 ---
@@ -402,8 +405,8 @@ private RegionScanner 
scanUnordered(ObserverContext

[GitHub] phoenix pull request: PHOENIX-2628 Ensure split when iterating thr...

[chrajeshbabu]

Github user chrajeshbabu commented on a diff in the pull request:

https://github.com/apache/phoenix/pull/156#discussion_r58638946
  
--- Diff: 
phoenix-core/src/main/java/org/apache/phoenix/compile/ScanRanges.java ---
@@ -385,9 +387,25 @@ public Scan intersectScan(Scan scan, final byte[] 
originalStartKey, final byte[]
 if (scanStopKey.length > 0 && Bytes.compareTo(scanStartKey, 
scanStopKey) >= 0) { 
 return null; 
 }
-newScan.setAttribute(SCAN_ACTUAL_START_ROW, scanStartKey);
-newScan.setStartRow(scanStartKey);
-newScan.setStopRow(scanStopKey);
+if(ScanUtil.isLocalIndex(scan)) {
--- End diff --

I will try moving outside of this method.


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

[jira] [Updated] (PHOENIX-2791) Support append only schema declaration

[James Taylor (JIRA)]

 [ 
https://issues.apache.org/jira/browse/PHOENIX-2791?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

James Taylor updated PHOENIX-2791:
--
Labels: argus  (was: )

> Support append only schema declaration
> --
>
> Key: PHOENIX-2791
> URL: https://issues.apache.org/jira/browse/PHOENIX-2791
> Project: Phoenix
>  Issue Type: Sub-task
>Reporter: James Taylor
>Assignee: Thomas D'Silva
>  Labels: argus
> Fix For: 4.8.0
>
>
> If we know in advance that columns will only be added to but never removed 
> from a schema, we can prevent the RPC from the client to the server when the 
> client already has all columns declared in the CREATE TABLE/VIEW IF NOT 
> EXISTS. To enable this, we can add an APPEND_ONLY_SCHEMA boolean flag to 
> SYSTEM.CATALOG. Or another potential name would be IMMUTABLE_SCHEMA to match 
> IMMUTABLE_ROWS?



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Commented] (PHOENIX-2628) Ensure split when iterating through results handled correctly

[ASF GitHub Bot (JIRA)]

[ 
https://issues.apache.org/jira/browse/PHOENIX-2628?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15227488#comment-15227488
 ] 

ASF GitHub Bot commented on PHOENIX-2628:
-

Github user chrajeshbabu commented on a diff in the pull request:

https://github.com/apache/phoenix/pull/156#discussion_r58638911
  
--- Diff: 
phoenix-core/src/main/java/org/apache/phoenix/coprocessor/GroupedAggregateRegionObserver.java
 ---
@@ -402,8 +405,8 @@ private RegionScanner 
scanUnordered(ObserverContext Ensure split when iterating through results handled correctly
> -
>
> Key: PHOENIX-2628
> URL: https://issues.apache.org/jira/browse/PHOENIX-2628
> Project: Phoenix
>  Issue Type: Bug
>Reporter: James Taylor
>Assignee: Rajeshbabu Chintaguntla
> Fix For: 4.8.0
>
> Attachments: PHOENIX-2628-wip.patch, PHOENIX-2628.patch, 
> PHOENIX-2628_v7.patch, PHOENIX-2628_v8.patch
>
>
> We should start with a test case to ensure this works correctly, both for 
> scans and aggregates.



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Updated] (PHOENIX-2795) Support auto partition for views

[James Taylor (JIRA)]

 [ 
https://issues.apache.org/jira/browse/PHOENIX-2795?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

James Taylor updated PHOENIX-2795:
--
Labels: argus  (was: )

> Support auto partition for views
> 
>
> Key: PHOENIX-2795
> URL: https://issues.apache.org/jira/browse/PHOENIX-2795
> Project: Phoenix
>  Issue Type: Sub-task
>Reporter: James Taylor
>Assignee: Thomas D'Silva
>  Labels: argus
> Fix For: 4.8.0
>
>
> When a view or base table is created, we should have an string 
> AUTO_PARTITION_SEQ parameter on CREATE TABLE which uses a sequence based on 
> the argument on the server side to generate a WHERE clause with the first PK 
> column and the unique identifier from the sequence.
> For example:
> {code}
> CREATE SEQUENCE metric_id_seq;
> CREATE TABLE metric_table (metric_id INTEGER, val DOUBLE) 
> AUTO_PARTITION_SEQ=metric_id_seq;
> CREATE VIEW my_view1 AS SELECT * FROM base_table;
> {code}
> would tack on a WHERE clause base on the next value in a sequence, logically 
> like this:
> {code}
> WHERE partition_id =  NEXT VALUE FROM metric_id_seq
> {code}
> It's important that the sequence be generated *after* the check for the 
> existence of the view so that we don't burn sequence values needlessly if the 
> view already exists.



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Updated] (PHOENIX-2795) Support auto partition for views

[James Taylor (JIRA)]

 [ 
https://issues.apache.org/jira/browse/PHOENIX-2795?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

James Taylor updated PHOENIX-2795:
--
Fix Version/s: 4.8.0

> Support auto partition for views
> 
>
> Key: PHOENIX-2795
> URL: https://issues.apache.org/jira/browse/PHOENIX-2795
> Project: Phoenix
>  Issue Type: Sub-task
>Reporter: James Taylor
>Assignee: Thomas D'Silva
>  Labels: argus
> Fix For: 4.8.0
>
>
> When a view or base table is created, we should have an string 
> AUTO_PARTITION_SEQ parameter on CREATE TABLE which uses a sequence based on 
> the argument on the server side to generate a WHERE clause with the first PK 
> column and the unique identifier from the sequence.
> For example:
> {code}
> CREATE SEQUENCE metric_id_seq;
> CREATE TABLE metric_table (metric_id INTEGER, val DOUBLE) 
> AUTO_PARTITION_SEQ=metric_id_seq;
> CREATE VIEW my_view1 AS SELECT * FROM base_table;
> {code}
> would tack on a WHERE clause base on the next value in a sequence, logically 
> like this:
> {code}
> WHERE partition_id =  NEXT VALUE FROM metric_id_seq
> {code}
> It's important that the sequence be generated *after* the check for the 
> existence of the view so that we don't burn sequence values needlessly if the 
> view already exists.



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Commented] (PHOENIX-2628) Ensure split when iterating through results handled correctly

[ASF GitHub Bot (JIRA)]

[ 
https://issues.apache.org/jira/browse/PHOENIX-2628?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15227486#comment-15227486
 ] 

ASF GitHub Bot commented on PHOENIX-2628:
-

Github user JamesRTaylor commented on a diff in the pull request:

https://github.com/apache/phoenix/pull/156#discussion_r58638661
  
--- Diff: 
phoenix-core/src/main/java/org/apache/phoenix/iterate/TableResultIterator.java 
---
@@ -107,8 +127,37 @@ public synchronized void close() throws SQLException {
 @Override
 public synchronized Tuple next() throws SQLException {
 initScanner();
-Tuple t = scanIterator.next();
-return t;
+try {
+lastTuple = scanIterator.next();
+if (lastTuple != null) {
+ImmutableBytesWritable ptr = new ImmutableBytesWritable();
+lastTuple.getKey(ptr);
+}
+} catch (SQLException e) {
+try {
+throw ServerUtil.parseServerException(e);
+} catch(StaleRegionBoundaryCacheException e1) {
+if(scan.getAttribute(NON_AGGREGATE_QUERY)!=null) {
+Scan newScan = ScanUtil.newScan(scan);
+if(lastTuple != null) {
+lastTuple.getKey(ptr);
+byte[] startRowSuffix = 
ByteUtil.copyKeyBytesIfNecessary(ptr);
+if(ScanUtil.isLocalIndex(newScan)) {
+newScan.setAttribute(SCAN_START_ROW_SUFFIX, 
ByteUtil.nextKey(startRowSuffix));
+} else {
+
newScan.setStartRow(ByteUtil.nextKey(startRowSuffix));
+}
+}
+
plan.getContext().getConnection().getQueryServices().clearTableRegionCache(htable.getTableName());
+this.scanIterator =
+
plan.iterator(DefaultParallelScanGrouper.getInstance(), newScan);
--- End diff --

Ah, I misread that. The case of an ordered aggregate falls into the same 
category, however, as we're aggregating in-place. I don't think we need any 
additional code in the unordered aggregate code path in 
GroupedAggregateRegionObserver (other than the duplicate in-region check I 
mentioned above).



> Ensure split when iterating through results handled correctly
> -
>
> Key: PHOENIX-2628
> URL: https://issues.apache.org/jira/browse/PHOENIX-2628
> Project: Phoenix
>  Issue Type: Bug
>Reporter: James Taylor
>Assignee: Rajeshbabu Chintaguntla
> Fix For: 4.8.0
>
> Attachments: PHOENIX-2628-wip.patch, PHOENIX-2628.patch, 
> PHOENIX-2628_v7.patch, PHOENIX-2628_v8.patch
>
>
> We should start with a test case to ensure this works correctly, both for 
> scans and aggregates.



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[GitHub] phoenix pull request: PHOENIX-2628 Ensure split when iterating thr...

[JamesRTaylor]

Github user JamesRTaylor commented on a diff in the pull request:

https://github.com/apache/phoenix/pull/156#discussion_r58638661
  
--- Diff: 
phoenix-core/src/main/java/org/apache/phoenix/iterate/TableResultIterator.java 
---
@@ -107,8 +127,37 @@ public synchronized void close() throws SQLException {
 @Override
 public synchronized Tuple next() throws SQLException {
 initScanner();
-Tuple t = scanIterator.next();
-return t;
+try {
+lastTuple = scanIterator.next();
+if (lastTuple != null) {
+ImmutableBytesWritable ptr = new ImmutableBytesWritable();
+lastTuple.getKey(ptr);
+}
+} catch (SQLException e) {
+try {
+throw ServerUtil.parseServerException(e);
+} catch(StaleRegionBoundaryCacheException e1) {
+if(scan.getAttribute(NON_AGGREGATE_QUERY)!=null) {
+Scan newScan = ScanUtil.newScan(scan);
+if(lastTuple != null) {
+lastTuple.getKey(ptr);
+byte[] startRowSuffix = 
ByteUtil.copyKeyBytesIfNecessary(ptr);
+if(ScanUtil.isLocalIndex(newScan)) {
+newScan.setAttribute(SCAN_START_ROW_SUFFIX, 
ByteUtil.nextKey(startRowSuffix));
+} else {
+
newScan.setStartRow(ByteUtil.nextKey(startRowSuffix));
+}
+}
+
plan.getContext().getConnection().getQueryServices().clearTableRegionCache(htable.getTableName());
+this.scanIterator =
+
plan.iterator(DefaultParallelScanGrouper.getInstance(), newScan);
--- End diff --

Ah, I misread that. The case of an ordered aggregate falls into the same 
category, however, as we're aggregating in-place. I don't think we need any 
additional code in the unordered aggregate code path in 
GroupedAggregateRegionObserver (other than the duplicate in-region check I 
mentioned above).



---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

[jira] [Commented] (PHOENIX-2628) Ensure split when iterating through results handled correctly

[ASF GitHub Bot (JIRA)]

[ 
https://issues.apache.org/jira/browse/PHOENIX-2628?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15227483#comment-15227483
 ] 

ASF GitHub Bot commented on PHOENIX-2628:
-

Github user chrajeshbabu commented on a diff in the pull request:

https://github.com/apache/phoenix/pull/156#discussion_r58638568
  
--- Diff: 
phoenix-core/src/main/java/org/apache/phoenix/coprocessor/GroupedAggregateRegionObserver.java
 ---
@@ -423,7 +426,14 @@ private RegionScanner 
scanUnordered(ObserverContext Ensure split when iterating through results handled correctly
> -
>
> Key: PHOENIX-2628
> URL: https://issues.apache.org/jira/browse/PHOENIX-2628
> Project: Phoenix
>  Issue Type: Bug
>Reporter: James Taylor
>Assignee: Rajeshbabu Chintaguntla
> Fix For: 4.8.0
>
> Attachments: PHOENIX-2628-wip.patch, PHOENIX-2628.patch, 
> PHOENIX-2628_v7.patch, PHOENIX-2628_v8.patch
>
>
> We should start with a test case to ensure this works correctly, both for 
> scans and aggregates.



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[GitHub] phoenix pull request: PHOENIX-2628 Ensure split when iterating thr...

[chrajeshbabu]

Github user chrajeshbabu commented on a diff in the pull request:

https://github.com/apache/phoenix/pull/156#discussion_r58638568
  
--- Diff: 
phoenix-core/src/main/java/org/apache/phoenix/coprocessor/GroupedAggregateRegionObserver.java
 ---
@@ -423,7 +426,14 @@ private RegionScanner 
scanUnordered(ObserverContext

[jira] [Commented] (PHOENIX-2628) Ensure split when iterating through results handled correctly

[ASF GitHub Bot (JIRA)]

[ 
https://issues.apache.org/jira/browse/PHOENIX-2628?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15227463#comment-15227463
 ] 

ASF GitHub Bot commented on PHOENIX-2628:
-

Github user chrajeshbabu commented on a diff in the pull request:

https://github.com/apache/phoenix/pull/156#discussion_r58637933
  
--- Diff: 
phoenix-core/src/main/java/org/apache/phoenix/iterate/TableResultIterator.java 
---
@@ -107,8 +127,37 @@ public synchronized void close() throws SQLException {
 @Override
 public synchronized Tuple next() throws SQLException {
 initScanner();
-Tuple t = scanIterator.next();
-return t;
+try {
+lastTuple = scanIterator.next();
+if (lastTuple != null) {
+ImmutableBytesWritable ptr = new ImmutableBytesWritable();
+lastTuple.getKey(ptr);
+}
+} catch (SQLException e) {
+try {
+throw ServerUtil.parseServerException(e);
+} catch(StaleRegionBoundaryCacheException e1) {
+if(scan.getAttribute(NON_AGGREGATE_QUERY)!=null) {
+Scan newScan = ScanUtil.newScan(scan);
+if(lastTuple != null) {
+lastTuple.getKey(ptr);
+byte[] startRowSuffix = 
ByteUtil.copyKeyBytesIfNecessary(ptr);
+if(ScanUtil.isLocalIndex(newScan)) {
+newScan.setAttribute(SCAN_START_ROW_SUFFIX, 
ByteUtil.nextKey(startRowSuffix));
+} else {
+
newScan.setStartRow(ByteUtil.nextKey(startRowSuffix));
+}
+}
+
plan.getContext().getConnection().getQueryServices().clearTableRegionCache(htable.getTableName());
+this.scanIterator =
+
plan.iterator(DefaultParallelScanGrouper.getInstance(), newScan);
--- End diff --

Yes aggregate queries are already handled properly. This code is only for 
handling splits when we are in the middle of non aggregate queries. If there 
are splits in the starting of the query then we are throwing out the stale 
region exception to BaseResultIterators which handles creating the proper 
parallel scans.
+if(scan.getAttribute(NON_AGGREGATE_QUERY)!=null) {



> Ensure split when iterating through results handled correctly
> -
>
> Key: PHOENIX-2628
> URL: https://issues.apache.org/jira/browse/PHOENIX-2628
> Project: Phoenix
>  Issue Type: Bug
>Reporter: James Taylor
>Assignee: Rajeshbabu Chintaguntla
> Fix For: 4.8.0
>
> Attachments: PHOENIX-2628-wip.patch, PHOENIX-2628.patch, 
> PHOENIX-2628_v7.patch, PHOENIX-2628_v8.patch
>
>
> We should start with a test case to ensure this works correctly, both for 
> scans and aggregates.



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[GitHub] phoenix pull request: PHOENIX-2628 Ensure split when iterating thr...

[chrajeshbabu]

Github user chrajeshbabu commented on a diff in the pull request:

https://github.com/apache/phoenix/pull/156#discussion_r58637933
  
--- Diff: 
phoenix-core/src/main/java/org/apache/phoenix/iterate/TableResultIterator.java 
---
@@ -107,8 +127,37 @@ public synchronized void close() throws SQLException {
 @Override
 public synchronized Tuple next() throws SQLException {
 initScanner();
-Tuple t = scanIterator.next();
-return t;
+try {
+lastTuple = scanIterator.next();
+if (lastTuple != null) {
+ImmutableBytesWritable ptr = new ImmutableBytesWritable();
+lastTuple.getKey(ptr);
+}
+} catch (SQLException e) {
+try {
+throw ServerUtil.parseServerException(e);
+} catch(StaleRegionBoundaryCacheException e1) {
+if(scan.getAttribute(NON_AGGREGATE_QUERY)!=null) {
+Scan newScan = ScanUtil.newScan(scan);
+if(lastTuple != null) {
+lastTuple.getKey(ptr);
+byte[] startRowSuffix = 
ByteUtil.copyKeyBytesIfNecessary(ptr);
+if(ScanUtil.isLocalIndex(newScan)) {
+newScan.setAttribute(SCAN_START_ROW_SUFFIX, 
ByteUtil.nextKey(startRowSuffix));
+} else {
+
newScan.setStartRow(ByteUtil.nextKey(startRowSuffix));
+}
+}
+
plan.getContext().getConnection().getQueryServices().clearTableRegionCache(htable.getTableName());
+this.scanIterator =
+
plan.iterator(DefaultParallelScanGrouper.getInstance(), newScan);
--- End diff --

Yes aggregate queries are already handled properly. This code is only for 
handling splits when we are in the middle of non aggregate queries. If there 
are splits in the starting of the query then we are throwing out the stale 
region exception to BaseResultIterators which handles creating the proper 
parallel scans.
+if(scan.getAttribute(NON_AGGREGATE_QUERY)!=null) {



---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

[jira] [Commented] (PHOENIX-2628) Ensure split when iterating through results handled correctly

[ASF GitHub Bot (JIRA)]

[ 
https://issues.apache.org/jira/browse/PHOENIX-2628?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15227454#comment-15227454
 ] 

ASF GitHub Bot commented on PHOENIX-2628:
-

Github user JamesRTaylor commented on a diff in the pull request:

https://github.com/apache/phoenix/pull/156#discussion_r58637402
  
--- Diff: 
phoenix-core/src/main/java/org/apache/phoenix/iterate/TableResultIterator.java 
---
@@ -107,8 +127,37 @@ public synchronized void close() throws SQLException {
 @Override
 public synchronized Tuple next() throws SQLException {
 initScanner();
-Tuple t = scanIterator.next();
-return t;
+try {
+lastTuple = scanIterator.next();
+if (lastTuple != null) {
+ImmutableBytesWritable ptr = new ImmutableBytesWritable();
+lastTuple.getKey(ptr);
+}
+} catch (SQLException e) {
+try {
+throw ServerUtil.parseServerException(e);
+} catch(StaleRegionBoundaryCacheException e1) {
+if(scan.getAttribute(NON_AGGREGATE_QUERY)!=null) {
+Scan newScan = ScanUtil.newScan(scan);
+if(lastTuple != null) {
+lastTuple.getKey(ptr);
+byte[] startRowSuffix = 
ByteUtil.copyKeyBytesIfNecessary(ptr);
+if(ScanUtil.isLocalIndex(newScan)) {
+newScan.setAttribute(SCAN_START_ROW_SUFFIX, 
ByteUtil.nextKey(startRowSuffix));
+} else {
+
newScan.setStartRow(ByteUtil.nextKey(startRowSuffix));
+}
+}
+
plan.getContext().getConnection().getQueryServices().clearTableRegionCache(htable.getTableName());
+this.scanIterator =
+
plan.iterator(DefaultParallelScanGrouper.getInstance(), newScan);
--- End diff --

So we're re-running the scan, only if it's an aggregate scan? Does this 
apply only for an "ordered" group by (which is grouping by the leading part of 
the pk and doing so in-place)? In the unordered case, each scan is done during 
the post open and the results are cached on the RS, so a split won't impact the 
results. There may be a race condition between the pre open (where we check if 
we're in the region) and the post open (where we start the region operation), 
but I think that could be solved by doing the in-region check again). In these 
cases, we already handle this through the exception catching code in 
BaseResultIterators.


> Ensure split when iterating through results handled correctly
> -
>
> Key: PHOENIX-2628
> URL: https://issues.apache.org/jira/browse/PHOENIX-2628
> Project: Phoenix
>  Issue Type: Bug
>Reporter: James Taylor
>Assignee: Rajeshbabu Chintaguntla
> Fix For: 4.8.0
>
> Attachments: PHOENIX-2628-wip.patch, PHOENIX-2628.patch, 
> PHOENIX-2628_v7.patch, PHOENIX-2628_v8.patch
>
>
> We should start with a test case to ensure this works correctly, both for 
> scans and aggregates.



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[GitHub] phoenix pull request: PHOENIX-2628 Ensure split when iterating thr...

[JamesRTaylor]

Github user JamesRTaylor commented on a diff in the pull request:

https://github.com/apache/phoenix/pull/156#discussion_r58637402
  
--- Diff: 
phoenix-core/src/main/java/org/apache/phoenix/iterate/TableResultIterator.java 
---
@@ -107,8 +127,37 @@ public synchronized void close() throws SQLException {
 @Override
 public synchronized Tuple next() throws SQLException {
 initScanner();
-Tuple t = scanIterator.next();
-return t;
+try {
+lastTuple = scanIterator.next();
+if (lastTuple != null) {
+ImmutableBytesWritable ptr = new ImmutableBytesWritable();
+lastTuple.getKey(ptr);
+}
+} catch (SQLException e) {
+try {
+throw ServerUtil.parseServerException(e);
+} catch(StaleRegionBoundaryCacheException e1) {
+if(scan.getAttribute(NON_AGGREGATE_QUERY)!=null) {
+Scan newScan = ScanUtil.newScan(scan);
+if(lastTuple != null) {
+lastTuple.getKey(ptr);
+byte[] startRowSuffix = 
ByteUtil.copyKeyBytesIfNecessary(ptr);
+if(ScanUtil.isLocalIndex(newScan)) {
+newScan.setAttribute(SCAN_START_ROW_SUFFIX, 
ByteUtil.nextKey(startRowSuffix));
+} else {
+
newScan.setStartRow(ByteUtil.nextKey(startRowSuffix));
+}
+}
+
plan.getContext().getConnection().getQueryServices().clearTableRegionCache(htable.getTableName());
+this.scanIterator =
+
plan.iterator(DefaultParallelScanGrouper.getInstance(), newScan);
--- End diff --

So we're re-running the scan, only if it's an aggregate scan? Does this 
apply only for an "ordered" group by (which is grouping by the leading part of 
the pk and doing so in-place)? In the unordered case, each scan is done during 
the post open and the results are cached on the RS, so a split won't impact the 
results. There may be a race condition between the pre open (where we check if 
we're in the region) and the post open (where we start the region operation), 
but I think that could be solved by doing the in-region check again). In these 
cases, we already handle this through the exception catching code in 
BaseResultIterators.


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

[jira] [Commented] (PHOENIX-2628) Ensure split when iterating through results handled correctly

[ASF GitHub Bot (JIRA)]

[ 
https://issues.apache.org/jira/browse/PHOENIX-2628?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15227436#comment-15227436
 ] 

ASF GitHub Bot commented on PHOENIX-2628:
-

Github user JamesRTaylor commented on a diff in the pull request:

https://github.com/apache/phoenix/pull/156#discussion_r58636239
  
--- Diff: 
phoenix-core/src/main/java/org/apache/phoenix/compile/ScanRanges.java ---
@@ -385,9 +387,25 @@ public Scan intersectScan(Scan scan, final byte[] 
originalStartKey, final byte[]
 if (scanStopKey.length > 0 && Bytes.compareTo(scanStartKey, 
scanStopKey) >= 0) { 
 return null; 
 }
-newScan.setAttribute(SCAN_ACTUAL_START_ROW, scanStartKey);
-newScan.setStartRow(scanStartKey);
-newScan.setStopRow(scanStopKey);
+if(ScanUtil.isLocalIndex(scan)) {
--- End diff --

How about moving the setting of these special attributes outside of this 
method?


> Ensure split when iterating through results handled correctly
> -
>
> Key: PHOENIX-2628
> URL: https://issues.apache.org/jira/browse/PHOENIX-2628
> Project: Phoenix
>  Issue Type: Bug
>Reporter: James Taylor
>Assignee: Rajeshbabu Chintaguntla
> Fix For: 4.8.0
>
> Attachments: PHOENIX-2628-wip.patch, PHOENIX-2628.patch, 
> PHOENIX-2628_v7.patch, PHOENIX-2628_v8.patch
>
>
> We should start with a test case to ensure this works correctly, both for 
> scans and aggregates.



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[GitHub] phoenix pull request: PHOENIX-2628 Ensure split when iterating thr...

[JamesRTaylor]

Github user JamesRTaylor commented on a diff in the pull request:

https://github.com/apache/phoenix/pull/156#discussion_r58636239
  
--- Diff: 
phoenix-core/src/main/java/org/apache/phoenix/compile/ScanRanges.java ---
@@ -385,9 +387,25 @@ public Scan intersectScan(Scan scan, final byte[] 
originalStartKey, final byte[]
 if (scanStopKey.length > 0 && Bytes.compareTo(scanStartKey, 
scanStopKey) >= 0) { 
 return null; 
 }
-newScan.setAttribute(SCAN_ACTUAL_START_ROW, scanStartKey);
-newScan.setStartRow(scanStartKey);
-newScan.setStopRow(scanStopKey);
+if(ScanUtil.isLocalIndex(scan)) {
--- End diff --

How about moving the setting of these special attributes outside of this 
method?


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

[jira] [Commented] (PHOENIX-2628) Ensure split when iterating through results handled correctly

[ASF GitHub Bot (JIRA)]

[ 
https://issues.apache.org/jira/browse/PHOENIX-2628?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15227368#comment-15227368
 ] 

ASF GitHub Bot commented on PHOENIX-2628:
-

Github user chrajeshbabu commented on a diff in the pull request:

https://github.com/apache/phoenix/pull/156#discussion_r58632305
  
--- Diff: 
phoenix-core/src/main/java/org/apache/phoenix/compile/ScanRanges.java ---
@@ -385,9 +387,25 @@ public Scan intersectScan(Scan scan, final byte[] 
originalStartKey, final byte[]
 if (scanStopKey.length > 0 && Bytes.compareTo(scanStartKey, 
scanStopKey) >= 0) { 
 return null; 
 }
-newScan.setAttribute(SCAN_ACTUAL_START_ROW, scanStartKey);
-newScan.setStartRow(scanStartKey);
-newScan.setStopRow(scanStopKey);
+if(ScanUtil.isLocalIndex(scan)) {
--- End diff --

@JamesRTaylor  we cannot use keyoffset > 0 check always for considering 
local indexes scan because in special case like if a table has only one region 
then region start key length and end key length is zero so keyoffset also 
becoming zero. In that case we need to check local index scan or not to set the 
attributes properly. Wdyt? 


> Ensure split when iterating through results handled correctly
> -
>
> Key: PHOENIX-2628
> URL: https://issues.apache.org/jira/browse/PHOENIX-2628
> Project: Phoenix
>  Issue Type: Bug
>Reporter: James Taylor
>Assignee: Rajeshbabu Chintaguntla
> Fix For: 4.8.0
>
> Attachments: PHOENIX-2628-wip.patch, PHOENIX-2628.patch, 
> PHOENIX-2628_v7.patch, PHOENIX-2628_v8.patch
>
>
> We should start with a test case to ensure this works correctly, both for 
> scans and aggregates.



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[GitHub] phoenix pull request: PHOENIX-2628 Ensure split when iterating thr...

[chrajeshbabu]

Github user chrajeshbabu commented on a diff in the pull request:

https://github.com/apache/phoenix/pull/156#discussion_r58632305
  
--- Diff: 
phoenix-core/src/main/java/org/apache/phoenix/compile/ScanRanges.java ---
@@ -385,9 +387,25 @@ public Scan intersectScan(Scan scan, final byte[] 
originalStartKey, final byte[]
 if (scanStopKey.length > 0 && Bytes.compareTo(scanStartKey, 
scanStopKey) >= 0) { 
 return null; 
 }
-newScan.setAttribute(SCAN_ACTUAL_START_ROW, scanStartKey);
-newScan.setStartRow(scanStartKey);
-newScan.setStopRow(scanStopKey);
+if(ScanUtil.isLocalIndex(scan)) {
--- End diff --

@JamesRTaylor  we cannot use keyoffset > 0 check always for considering 
local indexes scan because in special case like if a table has only one region 
then region start key length and end key length is zero so keyoffset also 
becoming zero. In that case we need to check local index scan or not to set the 
attributes properly. Wdyt? 


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

[jira] [Commented] (PHOENIX-2817) Phoenix-Spark plugin doesn't work in secured env

[Sergey Soldatov (JIRA)]

[ 
https://issues.apache.org/jira/browse/PHOENIX-2817?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15227151#comment-15227151
 ] 

Sergey Soldatov commented on PHOENIX-2817:
--

[~ankit.singhal] I think we can stay with the code suggested by Josh, since it 
will looks easy to read and doesn't require to understand internal structure of 
{{PhoenixEmbeddedDriver.ConnectionInfo}}. As for the principal/keytab, we 
definitely don't need it there.  All M/R jobs as well as Spark apps are using 
HBase delegation tokens, but not principal/keytab pair.  

> Phoenix-Spark plugin doesn't work in secured env
> 
>
> Key: PHOENIX-2817
> URL: https://issues.apache.org/jira/browse/PHOENIX-2817
> Project: Phoenix
>  Issue Type: Bug
>Affects Versions: 4.4.0, 4.7.0
>Reporter: Sergey Soldatov
>Assignee: Sergey Soldatov
> Attachments: PHOENIX-2817-1.patch, PHOENIX-2817-2.patch, 
> PHOENIX-2817-3.patch
>
>
> When phoenix spark plugin is used with secured setup any attempt to perform 
> operation with PhoenixRDD cause an exception : 
> {noformat}
> Caused by: java.io.IOException: Login failure for 2181 from keytab /hbase: 
> javax.security.auth.login.LoginException: Unable to obtain password from user
>   at 
> org.apache.hadoop.security.UserGroupInformation.loginUserFromKeytab(UserGroupInformation.java:962)
>   at org.apache.hadoop.security.SecurityUtil.login(SecurityUtil.java:275)
>   at 
> org.apache.hadoop.hbase.security.User$SecureHadoopUser.login(User.java:386)
>   at org.apache.hadoop.hbase.security.User.login(User.java:253)
>   at 
> org.apache.phoenix.query.ConnectionQueryServicesImpl.openConnection(ConnectionQueryServicesImpl.java:282)
>   ... 107 more
> Caused by: javax.security.auth.login.LoginException: Unable to obtain 
> password from user
>   at 
> com.sun.security.auth.module.Krb5LoginModule.promptForPass(Krb5LoginModule.java:897)
>   at 
> com.sun.security.auth.module.Krb5LoginModule.attemptAuthentication(Krb5LoginModule.java:760)
>   at 
> com.sun.security.auth.module.Krb5LoginModule.login(Krb5LoginModule.java:617)
>   at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
>   at 
> sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
>   at 
> sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
>   at java.lang.reflect.Method.invoke(Method.java:497)
>   at javax.security.auth.login.LoginContext.invoke(LoginContext.java:755)
>   at 
> javax.security.auth.login.LoginContext.access$000(LoginContext.java:195)
>   at javax.security.auth.login.LoginContext$4.run(LoginContext.java:682)
>   at javax.security.auth.login.LoginContext$4.run(LoginContext.java:680)
>   at java.security.AccessController.doPrivileged(Native Method)
>   at 
> javax.security.auth.login.LoginContext.invokePriv(LoginContext.java:680)
>   at javax.security.auth.login.LoginContext.login(LoginContext.java:587)
>   at 
> org.apache.hadoop.security.UserGroupInformation.loginUserFromKeytab(UserGroupInformation.java:953)
>   ... 111 more
> {noformat}
> The reason is the how zkUrl is handled in PhoenixRDD: 
> {noformat}
> config.set(HConstants.ZOOKEEPER_QUORUM, url )
> {noformat}
> At the same time the {{ConnectionUtil.getInputConnection}} expects to see all 
> parameters (quorum address, port, znodeParent) in different Configuration 
> properties. As the result it gets default values for port and znodeParent and 
> adds it to the provided url, so the {{PhoenixEmbededDriver.create}} receives 
> something like that:
> {noformat}
> jdbc:phoenix:quorum:2181:/hbase-secure:2181:/hbase
> {noformat}
> and consider 2 fields as kerberos principal and keytab.  



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Commented] (PHOENIX-2822) Tests that extend BaseHBaseManagedTimeIT are very slow

[ASF GitHub Bot (JIRA)]

[ 
https://issues.apache.org/jira/browse/PHOENIX-2822?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15227000#comment-15227000
 ] 

ASF GitHub Bot commented on PHOENIX-2822:
-

GitHub user churrodog opened a pull request:

https://github.com/apache/phoenix/pull/158

PHOENIX-2822 - Tests that extend BaseHBaseManagedTimeIT are very slow



You can merge this pull request into a Git repository by running:

$ git pull https://github.com/churrodog/phoenix PHOENIX-2822

Alternatively you can review and apply these changes as the patch at:

https://github.com/apache/phoenix/pull/158.patch

To close this pull request, make a commit to your master/trunk branch
with (at least) the following in the commit message:

This closes #158


commit 8325d00cd01b2a7d71b52b7c6d0ff5f6147ba47a
Author: rahul gidwani 
Date:   2016-04-05T19:38:21Z

PHOENIX-2822 - Tests that extend BaseHBaseManagedTimeIT are very slow

Conflicts:
phoenix-core/src/test/java/org/apache/phoenix/query/BaseTest.java




> Tests that extend BaseHBaseManagedTimeIT are very slow
> --
>
> Key: PHOENIX-2822
> URL: https://issues.apache.org/jira/browse/PHOENIX-2822
> Project: Phoenix
>  Issue Type: Improvement
>Affects Versions: 4.8.0
>Reporter: churro morales
>Assignee: churro morales
>  Labels: HBASEDEPENDENCIES
>
> Since I am trying to refactor out all the hbase private dependencies, I have 
> to constantly run tests to make sure I didn't break anything.  The tests that 
> extend BaseHBaseManagedTimeIT are very slow as they have to delete all 
> non-system tables after every test case.  This takes around 5-10 seconds to 
> accomplish.  This adds significant time to the test suite. 
> I created a new class named: BaseHBaseManagedTimeTableReuseIT and it creates 
> a random table name such that we dont have collisions for tests.  It also 
> doesn't do any cleanup after each test case or class because these table 
> names should be unique.  I moved about 30-35 tests out from 
> BaseHBaseManagedTimeIT to BaseHBaseManagedTimeTableReuseIT and it 
> significantly improved the overall time it takes to run tests.



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[GitHub] phoenix pull request: PHOENIX-2822 - Tests that extend BaseHBaseMa...

[churrodog]

GitHub user churrodog opened a pull request:

https://github.com/apache/phoenix/pull/158

PHOENIX-2822 - Tests that extend BaseHBaseManagedTimeIT are very slow



You can merge this pull request into a Git repository by running:

$ git pull https://github.com/churrodog/phoenix PHOENIX-2822

Alternatively you can review and apply these changes as the patch at:

https://github.com/apache/phoenix/pull/158.patch

To close this pull request, make a commit to your master/trunk branch
with (at least) the following in the commit message:

This closes #158


commit 8325d00cd01b2a7d71b52b7c6d0ff5f6147ba47a
Author: rahul gidwani 
Date:   2016-04-05T19:38:21Z

PHOENIX-2822 - Tests that extend BaseHBaseManagedTimeIT are very slow

Conflicts:
phoenix-core/src/test/java/org/apache/phoenix/query/BaseTest.java




---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

[jira] [Created] (PHOENIX-2822) Tests that extend BaseHBaseManagedTimeIT are very slow

[churro morales (JIRA)]

churro morales created PHOENIX-2822:
---

 Summary: Tests that extend BaseHBaseManagedTimeIT are very slow
 Key: PHOENIX-2822
 URL: https://issues.apache.org/jira/browse/PHOENIX-2822
 Project: Phoenix
  Issue Type: Improvement
Affects Versions: 4.8.0
Reporter: churro morales
Assignee: churro morales


Since I am trying to refactor out all the hbase private dependencies, I have to 
constantly run tests to make sure I didn't break anything.  The tests that 
extend BaseHBaseManagedTimeIT are very slow as they have to delete all 
non-system tables after every test case.  This takes around 5-10 seconds to 
accomplish.  This adds significant time to the test suite. 

I created a new class named: BaseHBaseManagedTimeTableReuseIT and it creates a 
random table name such that we dont have collisions for tests.  It also doesn't 
do any cleanup after each test case or class because these table names should 
be unique.  I moved about 30-35 tests out from BaseHBaseManagedTimeIT to 
BaseHBaseManagedTimeTableReuseIT and it significantly improved the overall time 
it takes to run tests.



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Resolved] (PHOENIX-2819) Remove problematic method ValueSchemaBuilder.addField(Field)

[Maryann Xue (JIRA)]

 [ 
https://issues.apache.org/jira/browse/PHOENIX-2819?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Maryann Xue resolved PHOENIX-2819.
--
Resolution: Fixed

> Remove problematic method ValueSchemaBuilder.addField(Field)
> 
>
> Key: PHOENIX-2819
> URL: https://issues.apache.org/jira/browse/PHOENIX-2819
> Project: Phoenix
>  Issue Type: Bug
>Affects Versions: 4.7.0
>Reporter: Maryann Xue
>Assignee: Maryann Xue
>  Labels: secondary_index
> Fix For: 4.8.0
>
> Attachments: PHOENIX-2819-2.patch, PHOENIX-2819.patch
>
>
> We should discard the method "ValueSchemaBuilder.addField(Field)" and use the 
> other method signature instead, since this method might add more fields than 
> wanted.
> {code}
> public ValueSchemaBuilder addField(Field field) {
> fields.add(field);
> return this;
> }
> {code}
> If {{field.size}} is larger than 1, the above statement {{fields.add(field)}} 
> will add more than one field into the list while the caller is expecting to 
> add only ONE.
> The only caller of this method is IndexMaintainer.generateIndexRowKeySchema() 
> and it might generate a wrong row-key-schema as a result.



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Commented] (PHOENIX-1311) HBase namespaces surfaced in phoenix

[ASF GitHub Bot (JIRA)]

[ 
https://issues.apache.org/jira/browse/PHOENIX-1311?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15226939#comment-15226939
 ] 

ASF GitHub Bot commented on PHOENIX-1311:
-

Github user samarthjain commented on a diff in the pull request:

https://github.com/apache/phoenix/pull/153#discussion_r58597202
  
--- Diff: 
phoenix-core/src/main/java/org/apache/phoenix/util/UpgradeUtil.java ---
@@ -1279,4 +1304,129 @@ public static boolean truncateStats(HTableInterface 
metaTable, HTableInterface s
 }
 return false;
 }
+
+public static void mapTableToNamespace(HBaseAdmin admin, 
HTableInterface metatable, String srcTableName,
+String destTableName, ReadOnlyProps props, Long ts, String 
phoenixTableName, PTableType pTableType)
+throws SnapshotCreationException, 
IllegalArgumentException, IOException, InterruptedException,
+SQLException {
+srcTableName = SchemaUtil.normalizeIdentifier(srcTableName);
+if (!SchemaUtil.isNamespaceMappingEnabled(
+SchemaUtil.isSystemTable(srcTableName.getBytes()) ? 
PTableType.SYSTEM : null,
+props)) { throw new 
IllegalArgumentException(SchemaUtil.isSystemTable(srcTableName.getBytes())
+? "For system table " + 
QueryServices.IS_SYSTEM_TABLE_MAPPED_TO_NAMESPACE
++ " also needs to be enabled along with " 
+ QueryServices.IS_NAMESPACE_MAPPING_ENABLED
+: QueryServices.IS_NAMESPACE_MAPPING_ENABLED + " 
is not enabled"); }
+
+if (PTableType.TABLE.equals(pTableType) || 
PTableType.INDEX.equals(pTableType)) {
+admin.snapshot(srcTableName, srcTableName);
+admin.cloneSnapshot(srcTableName.getBytes(), 
destTableName.getBytes());
+admin.disableTable(srcTableName);
--- End diff --

Does disabling and deleting the srcTable here also delete the snapshot you 
created above. If not, would it make sense to delete the snapshot? This is also 
making me think of failure scenarios. What if any of the steps here fails? Do 
we restore the table using the snapshot?


> HBase namespaces surfaced in phoenix
> 
>
> Key: PHOENIX-1311
> URL: https://issues.apache.org/jira/browse/PHOENIX-1311
> Project: Phoenix
>  Issue Type: New Feature
>Reporter: nicolas maillard
>Assignee: Ankit Singhal
>Priority: Minor
> Fix For: 4.8.0
>
> Attachments: PHOENIX-1311.docx, PHOENIX-1311_v1.patch, 
> PHOENIX-1311_v2.patch, PHOENIX-1311_wip.patch, PHOENIX-1311_wip_2.patch
>
>
> Hbase (HBASE-8015) has the concept of namespaces in the form of 
> myNamespace:MyTable it would be great if Phoenix leveraged this feature to 
> give a database like feature on top of the table.
> Maybe to stay close to Hbase it could also be a create DB:Table...
> or DB.Table which is a more standard annotation?



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[GitHub] phoenix pull request: PHOENIX-1311 HBase namespaces surfaced in ph...

[samarthjain]

Github user samarthjain commented on a diff in the pull request:

https://github.com/apache/phoenix/pull/153#discussion_r58597202
  
--- Diff: 
phoenix-core/src/main/java/org/apache/phoenix/util/UpgradeUtil.java ---
@@ -1279,4 +1304,129 @@ public static boolean truncateStats(HTableInterface 
metaTable, HTableInterface s
 }
 return false;
 }
+
+public static void mapTableToNamespace(HBaseAdmin admin, 
HTableInterface metatable, String srcTableName,
+String destTableName, ReadOnlyProps props, Long ts, String 
phoenixTableName, PTableType pTableType)
+throws SnapshotCreationException, 
IllegalArgumentException, IOException, InterruptedException,
+SQLException {
+srcTableName = SchemaUtil.normalizeIdentifier(srcTableName);
+if (!SchemaUtil.isNamespaceMappingEnabled(
+SchemaUtil.isSystemTable(srcTableName.getBytes()) ? 
PTableType.SYSTEM : null,
+props)) { throw new 
IllegalArgumentException(SchemaUtil.isSystemTable(srcTableName.getBytes())
+? "For system table " + 
QueryServices.IS_SYSTEM_TABLE_MAPPED_TO_NAMESPACE
++ " also needs to be enabled along with " 
+ QueryServices.IS_NAMESPACE_MAPPING_ENABLED
+: QueryServices.IS_NAMESPACE_MAPPING_ENABLED + " 
is not enabled"); }
+
+if (PTableType.TABLE.equals(pTableType) || 
PTableType.INDEX.equals(pTableType)) {
+admin.snapshot(srcTableName, srcTableName);
+admin.cloneSnapshot(srcTableName.getBytes(), 
destTableName.getBytes());
+admin.disableTable(srcTableName);
--- End diff --

Does disabling and deleting the srcTable here also delete the snapshot you 
created above. If not, would it make sense to delete the snapshot? This is also 
making me think of failure scenarios. What if any of the steps here fails? Do 
we restore the table using the snapshot?


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

[jira] [Commented] (PHOENIX-1311) HBase namespaces surfaced in phoenix

[ASF GitHub Bot (JIRA)]

[ 
https://issues.apache.org/jira/browse/PHOENIX-1311?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15226916#comment-15226916
 ] 

ASF GitHub Bot commented on PHOENIX-1311:
-

Github user samarthjain commented on a diff in the pull request:

https://github.com/apache/phoenix/pull/153#discussion_r58594903
  
--- Diff: 
phoenix-core/src/main/java/org/apache/phoenix/util/UpgradeUtil.java ---
@@ -1279,4 +1304,129 @@ public static boolean truncateStats(HTableInterface 
metaTable, HTableInterface s
 }
 return false;
 }
+
+public static void mapTableToNamespace(HBaseAdmin admin, 
HTableInterface metatable, String srcTableName,
--- End diff --

Is it possible to add any tests around the upgrade code here? 


> HBase namespaces surfaced in phoenix
> 
>
> Key: PHOENIX-1311
> URL: https://issues.apache.org/jira/browse/PHOENIX-1311
> Project: Phoenix
>  Issue Type: New Feature
>Reporter: nicolas maillard
>Assignee: Ankit Singhal
>Priority: Minor
> Fix For: 4.8.0
>
> Attachments: PHOENIX-1311.docx, PHOENIX-1311_v1.patch, 
> PHOENIX-1311_v2.patch, PHOENIX-1311_wip.patch, PHOENIX-1311_wip_2.patch
>
>
> Hbase (HBASE-8015) has the concept of namespaces in the form of 
> myNamespace:MyTable it would be great if Phoenix leveraged this feature to 
> give a database like feature on top of the table.
> Maybe to stay close to Hbase it could also be a create DB:Table...
> or DB.Table which is a more standard annotation?



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[GitHub] phoenix pull request: PHOENIX-1311 HBase namespaces surfaced in ph...

[samarthjain]

Github user samarthjain commented on a diff in the pull request:

https://github.com/apache/phoenix/pull/153#discussion_r58594903
  
--- Diff: 
phoenix-core/src/main/java/org/apache/phoenix/util/UpgradeUtil.java ---
@@ -1279,4 +1304,129 @@ public static boolean truncateStats(HTableInterface 
metaTable, HTableInterface s
 }
 return false;
 }
+
+public static void mapTableToNamespace(HBaseAdmin admin, 
HTableInterface metatable, String srcTableName,
--- End diff --

Is it possible to add any tests around the upgrade code here? 


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

[jira] [Commented] (PHOENIX-1311) HBase namespaces surfaced in phoenix

[ASF GitHub Bot (JIRA)]

[ 
https://issues.apache.org/jira/browse/PHOENIX-1311?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15226914#comment-15226914
 ] 

ASF GitHub Bot commented on PHOENIX-1311:
-

Github user samarthjain commented on a diff in the pull request:

https://github.com/apache/phoenix/pull/153#discussion_r58594707
  
--- Diff: 
phoenix-core/src/main/java/org/apache/phoenix/util/UpgradeUtil.java ---
@@ -1279,4 +1304,129 @@ public static boolean truncateStats(HTableInterface 
metaTable, HTableInterface s
 }
 return false;
 }
+
+public static void mapTableToNamespace(HBaseAdmin admin, 
HTableInterface metatable, String srcTableName,
+String destTableName, ReadOnlyProps props, Long ts, String 
phoenixTableName, PTableType pTableType)
+throws SnapshotCreationException, 
IllegalArgumentException, IOException, InterruptedException,
+SQLException {
+srcTableName = SchemaUtil.normalizeIdentifier(srcTableName);
+if (!SchemaUtil.isNamespaceMappingEnabled(
+SchemaUtil.isSystemTable(srcTableName.getBytes()) ? 
PTableType.SYSTEM : null,
+props)) { throw new 
IllegalArgumentException(SchemaUtil.isSystemTable(srcTableName.getBytes())
+? "For system table " + 
QueryServices.IS_SYSTEM_TABLE_MAPPED_TO_NAMESPACE
++ " also needs to be enabled along with " 
+ QueryServices.IS_NAMESPACE_MAPPING_ENABLED
+: QueryServices.IS_NAMESPACE_MAPPING_ENABLED + " 
is not enabled"); }
+
+if (PTableType.TABLE.equals(pTableType) || 
PTableType.INDEX.equals(pTableType)) {
+admin.snapshot(srcTableName, srcTableName);
+admin.cloneSnapshot(srcTableName.getBytes(), 
destTableName.getBytes());
+admin.disableTable(srcTableName);
+admin.deleteTable(srcTableName);
+}
+if (phoenixTableName == null) {
+phoenixTableName = srcTableName;
+}
+Put put = new Put(SchemaUtil.getTableKey(null, 
SchemaUtil.getSchemaNameFromFullName(phoenixTableName),
+SchemaUtil.getTableNameFromFullName(phoenixTableName)), 
ts);
+put.addColumn(QueryConstants.DEFAULT_COLUMN_FAMILY_BYTES, 
PhoenixDatabaseMetaData.IS_NAMESPACE_MAPPED_BYTES,
+PBoolean.INSTANCE.toBytes(Boolean.TRUE));
+metatable.put(put);
+}
+
+public static void mapTableToNamespace(HBaseAdmin admin, 
HTableInterface metatable, String tableName,
+ReadOnlyProps props, Long ts) throws 
SnapshotCreationException, IllegalArgumentException, IOException,
+InterruptedException, SQLException {
+String destTablename = SchemaUtil
+
.normalizeIdentifier(SchemaUtil.getPhysicalTableName(tableName, 
props).getNameAsString());
+mapTableToNamespace(admin, metatable, tableName, destTablename, 
props, ts, null, PTableType.TABLE);
+}
+
+public static void upgradeTable(PhoenixConnection conn, String 
srcTable) throws SQLException,
+SnapshotCreationException, IllegalArgumentException, 
IOException, InterruptedException {
+ReadOnlyProps readOnlyProps = conn.getQueryServices().getProps();
+if (conn.getClientInfo(PhoenixRuntime.TENANT_ID_ATTRIB) != null) { 
throw new SQLException(
+"May not specify the TENANT_ID_ATTRIB property when 
upgrading"); }
+try (HBaseAdmin admin = conn.getQueryServices().getAdmin();
+HTableInterface metatable = conn.getQueryServices()
+.getTable(SchemaUtil
+
.getPhysicalName(PhoenixDatabaseMetaData.SYSTEM_CATALOG_NAME_BYTES, 
readOnlyProps)
+.getName());) {
+String tableName = SchemaUtil.normalizeIdentifier(srcTable);
+String schemaName = 
SchemaUtil.getSchemaNameFromFullName(tableName);
+
+// Upgrade is not required if schemaName is not present.
+if (schemaName.equals("")) { throw new 
IllegalArgumentException("Table doesn't have schema name"); }
+
+// Confirm table is not already upgraded
+PTable table = PhoenixRuntime.getTable(conn, tableName);
+if (table.isNamespaceMapped()) { throw new 
IllegalArgumentException("Table is already upgraded"); }
+conn.createStatement().execute("CREATE SCHEMA IF NOT EXISTS " 
+ schemaName);
+String newPhysicalTablename = SchemaUtil
+
.normalizeIdentifier(SchemaUtil.getPhysicalTableName(table.getPhysicalName().getString(),
 readOnlyProps).getNameAsString());
+

[GitHub] phoenix pull request: PHOENIX-1311 HBase namespaces surfaced in ph...

[samarthjain]

Github user samarthjain commented on a diff in the pull request:

https://github.com/apache/phoenix/pull/153#discussion_r58594707
  
--- Diff: 
phoenix-core/src/main/java/org/apache/phoenix/util/UpgradeUtil.java ---
@@ -1279,4 +1304,129 @@ public static boolean truncateStats(HTableInterface 
metaTable, HTableInterface s
 }
 return false;
 }
+
+public static void mapTableToNamespace(HBaseAdmin admin, 
HTableInterface metatable, String srcTableName,
+String destTableName, ReadOnlyProps props, Long ts, String 
phoenixTableName, PTableType pTableType)
+throws SnapshotCreationException, 
IllegalArgumentException, IOException, InterruptedException,
+SQLException {
+srcTableName = SchemaUtil.normalizeIdentifier(srcTableName);
+if (!SchemaUtil.isNamespaceMappingEnabled(
+SchemaUtil.isSystemTable(srcTableName.getBytes()) ? 
PTableType.SYSTEM : null,
+props)) { throw new 
IllegalArgumentException(SchemaUtil.isSystemTable(srcTableName.getBytes())
+? "For system table " + 
QueryServices.IS_SYSTEM_TABLE_MAPPED_TO_NAMESPACE
++ " also needs to be enabled along with " 
+ QueryServices.IS_NAMESPACE_MAPPING_ENABLED
+: QueryServices.IS_NAMESPACE_MAPPING_ENABLED + " 
is not enabled"); }
+
+if (PTableType.TABLE.equals(pTableType) || 
PTableType.INDEX.equals(pTableType)) {
+admin.snapshot(srcTableName, srcTableName);
+admin.cloneSnapshot(srcTableName.getBytes(), 
destTableName.getBytes());
+admin.disableTable(srcTableName);
+admin.deleteTable(srcTableName);
+}
+if (phoenixTableName == null) {
+phoenixTableName = srcTableName;
+}
+Put put = new Put(SchemaUtil.getTableKey(null, 
SchemaUtil.getSchemaNameFromFullName(phoenixTableName),
+SchemaUtil.getTableNameFromFullName(phoenixTableName)), 
ts);
+put.addColumn(QueryConstants.DEFAULT_COLUMN_FAMILY_BYTES, 
PhoenixDatabaseMetaData.IS_NAMESPACE_MAPPED_BYTES,
+PBoolean.INSTANCE.toBytes(Boolean.TRUE));
+metatable.put(put);
+}
+
+public static void mapTableToNamespace(HBaseAdmin admin, 
HTableInterface metatable, String tableName,
+ReadOnlyProps props, Long ts) throws 
SnapshotCreationException, IllegalArgumentException, IOException,
+InterruptedException, SQLException {
+String destTablename = SchemaUtil
+
.normalizeIdentifier(SchemaUtil.getPhysicalTableName(tableName, 
props).getNameAsString());
+mapTableToNamespace(admin, metatable, tableName, destTablename, 
props, ts, null, PTableType.TABLE);
+}
+
+public static void upgradeTable(PhoenixConnection conn, String 
srcTable) throws SQLException,
+SnapshotCreationException, IllegalArgumentException, 
IOException, InterruptedException {
+ReadOnlyProps readOnlyProps = conn.getQueryServices().getProps();
+if (conn.getClientInfo(PhoenixRuntime.TENANT_ID_ATTRIB) != null) { 
throw new SQLException(
+"May not specify the TENANT_ID_ATTRIB property when 
upgrading"); }
+try (HBaseAdmin admin = conn.getQueryServices().getAdmin();
+HTableInterface metatable = conn.getQueryServices()
+.getTable(SchemaUtil
+
.getPhysicalName(PhoenixDatabaseMetaData.SYSTEM_CATALOG_NAME_BYTES, 
readOnlyProps)
+.getName());) {
+String tableName = SchemaUtil.normalizeIdentifier(srcTable);
+String schemaName = 
SchemaUtil.getSchemaNameFromFullName(tableName);
+
+// Upgrade is not required if schemaName is not present.
+if (schemaName.equals("")) { throw new 
IllegalArgumentException("Table doesn't have schema name"); }
+
+// Confirm table is not already upgraded
+PTable table = PhoenixRuntime.getTable(conn, tableName);
+if (table.isNamespaceMapped()) { throw new 
IllegalArgumentException("Table is already upgraded"); }
+conn.createStatement().execute("CREATE SCHEMA IF NOT EXISTS " 
+ schemaName);
+String newPhysicalTablename = SchemaUtil
+
.normalizeIdentifier(SchemaUtil.getPhysicalTableName(table.getPhysicalName().getString(),
 readOnlyProps).getNameAsString());
+
+// Upgrade the data or main table
+UpgradeUtil.mapTableToNamespace(admin, metatable, tableName, 
newPhysicalTablename, readOnlyProps,
--- End diff --

minor nit: no need to use UpgradeUtil. here


---
If 

[jira] [Commented] (PHOENIX-2819) Remove problematic method ValueSchemaBuilder.addField(Field)

[Hudson (JIRA)]

[ 
https://issues.apache.org/jira/browse/PHOENIX-2819?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15226906#comment-15226906
 ] 

Hudson commented on PHOENIX-2819:
-

SUCCESS: Integrated in Phoenix-master #1183 (See 
[https://builds.apache.org/job/Phoenix-master/1183/])
PHOENIX-2819 Remove problematic method (maryannxue: rev 
3420a973c3fbe5ee4483216e54596869665c7cc9)
* phoenix-core/src/it/java/org/apache/phoenix/end2end/index/ViewIndexIT.java
* phoenix-core/src/main/java/org/apache/phoenix/index/IndexMaintainer.java
* phoenix-core/src/main/java/org/apache/phoenix/schema/ValueSchema.java


> Remove problematic method ValueSchemaBuilder.addField(Field)
> 
>
> Key: PHOENIX-2819
> URL: https://issues.apache.org/jira/browse/PHOENIX-2819
> Project: Phoenix
>  Issue Type: Bug
>Affects Versions: 4.7.0
>Reporter: Maryann Xue
>Assignee: Maryann Xue
>  Labels: secondary_index
> Fix For: 4.8.0
>
> Attachments: PHOENIX-2819-2.patch, PHOENIX-2819.patch
>
>
> We should discard the method "ValueSchemaBuilder.addField(Field)" and use the 
> other method signature instead, since this method might add more fields than 
> wanted.
> {code}
> public ValueSchemaBuilder addField(Field field) {
> fields.add(field);
> return this;
> }
> {code}
> If {{field.size}} is larger than 1, the above statement {{fields.add(field)}} 
> will add more than one field into the list while the caller is expecting to 
> add only ONE.
> The only caller of this method is IndexMaintainer.generateIndexRowKeySchema() 
> and it might generate a wrong row-key-schema as a result.



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Commented] (PHOENIX-2722) support mysql "limit,offset" clauses

[ASF GitHub Bot (JIRA)]

[ 
https://issues.apache.org/jira/browse/PHOENIX-2722?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15226863#comment-15226863
 ] 

ASF GitHub Bot commented on PHOENIX-2722:
-

Github user ankitsinghal commented on a diff in the pull request:

https://github.com/apache/phoenix/pull/154#discussion_r58591817
  
--- Diff: 
phoenix-core/src/main/java/org/apache/phoenix/compile/SubselectRewriter.java ---
@@ -188,6 +190,13 @@ private SelectStatement flatten(SelectStatement 
select, SelectStatement subselec
 }
 }
 
+OffsetNode offset = select.getOffset();
+if (offsetRewrite != null || (limitRewrite != null & offset != 
null)) {
--- End diff --

@JamesRTaylor , will it really affect the boolean operands.


> support mysql "limit,offset" clauses 
> -
>
> Key: PHOENIX-2722
> URL: https://issues.apache.org/jira/browse/PHOENIX-2722
> Project: Phoenix
>  Issue Type: New Feature
>Reporter: Ankit Singhal
>Assignee: Ankit Singhal
>Priority: Minor
> Attachments: PHOENIX-2722.patch, PHOENIX-2722_formatted.patch
>
>
> For serial query(query with “serial" hint or  “limit" without "order by”), we 
> can limit each scan(using page filter) to “limit+offset” instead of limit 
> earlier.
> And then, for all queries, we can forward the relevant client iterators to 
> the offset provided and then return the result.
> syntax
> {code}
> [ LIMIT { count } ]
> [ OFFSET start [ ROW | ROWS ] ]
> [ FETCH { FIRST | NEXT } [ count ] { ROW | ROWS } ONLY ]
> {code}
> Some new keywords(OFFSET,FETCH,ROW, ROWS,ONLY) are getting introduced so 
> users might need to see that they are not using them as column name or 
> something.
> WDYT, [~jamestaylor]



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Commented] (PHOENIX-2722) support mysql "limit,offset" clauses

[ASF GitHub Bot (JIRA)]

[ 
https://issues.apache.org/jira/browse/PHOENIX-2722?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15226824#comment-15226824
 ] 

ASF GitHub Bot commented on PHOENIX-2722:
-

Github user ankitsinghal commented on a diff in the pull request:

https://github.com/apache/phoenix/pull/154#discussion_r58588402
  
--- Diff: 
phoenix-core/src/main/java/org/apache/phoenix/iterate/OffsetScanGrouper.java ---
@@ -0,0 +1,44 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements.  See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership.  The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License.  You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+package org.apache.phoenix.iterate;
+
+import java.util.List;
+
+import org.apache.hadoop.hbase.client.Scan;
+import org.apache.phoenix.compile.QueryPlan;
+
+/**
+ * Default implementation that creates a scan group if a plan is row key 
ordered (which requires a merge sort),
+ * or if a scan crosses a region boundary and the table is salted or a 
local index.   
+ */
+public class OffsetScanGrouper implements ParallelScanGrouper {
--- End diff --

It is needed to avoid scans in parallel even in serialIterators. should I 
use a anonymous class while creating SerialIterators to do offset on server?


> support mysql "limit,offset" clauses 
> -
>
> Key: PHOENIX-2722
> URL: https://issues.apache.org/jira/browse/PHOENIX-2722
> Project: Phoenix
>  Issue Type: New Feature
>Reporter: Ankit Singhal
>Assignee: Ankit Singhal
>Priority: Minor
> Attachments: PHOENIX-2722.patch, PHOENIX-2722_formatted.patch
>
>
> For serial query(query with “serial" hint or  “limit" without "order by”), we 
> can limit each scan(using page filter) to “limit+offset” instead of limit 
> earlier.
> And then, for all queries, we can forward the relevant client iterators to 
> the offset provided and then return the result.
> syntax
> {code}
> [ LIMIT { count } ]
> [ OFFSET start [ ROW | ROWS ] ]
> [ FETCH { FIRST | NEXT } [ count ] { ROW | ROWS } ONLY ]
> {code}
> Some new keywords(OFFSET,FETCH,ROW, ROWS,ONLY) are getting introduced so 
> users might need to see that they are not using them as column name or 
> something.
> WDYT, [~jamestaylor]



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[GitHub] phoenix pull request: PHOENIX-2722 support mysql limit,offset clau...

[ankitsinghal]

Github user ankitsinghal commented on a diff in the pull request:

https://github.com/apache/phoenix/pull/154#discussion_r58588402
  
--- Diff: 
phoenix-core/src/main/java/org/apache/phoenix/iterate/OffsetScanGrouper.java ---
@@ -0,0 +1,44 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements.  See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership.  The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License.  You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+package org.apache.phoenix.iterate;
+
+import java.util.List;
+
+import org.apache.hadoop.hbase.client.Scan;
+import org.apache.phoenix.compile.QueryPlan;
+
+/**
+ * Default implementation that creates a scan group if a plan is row key 
ordered (which requires a merge sort),
+ * or if a scan crosses a region boundary and the table is salted or a 
local index.   
+ */
+public class OffsetScanGrouper implements ParallelScanGrouper {
--- End diff --

It is needed to avoid scans in parallel even in serialIterators. should I 
use a anonymous class while creating SerialIterators to do offset on server?


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

[jira] [Updated] (PHOENIX-2819) Remove problematic method ValueSchemaBuilder.addField(Field)

[Maryann Xue (JIRA)]

 [ 
https://issues.apache.org/jira/browse/PHOENIX-2819?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Maryann Xue updated PHOENIX-2819:
-
Attachment: PHOENIX-2819-2.patch

Thank you very much for reminding me, [~jamestaylor]! You are right, they are 
required. But the original logic was wrong by appending "," regardless of if 
there are preceding DDL options.

> Remove problematic method ValueSchemaBuilder.addField(Field)
> 
>
> Key: PHOENIX-2819
> URL: https://issues.apache.org/jira/browse/PHOENIX-2819
> Project: Phoenix
>  Issue Type: Bug
>Affects Versions: 4.7.0
>Reporter: Maryann Xue
>Assignee: Maryann Xue
>  Labels: secondary_index
> Fix For: 4.8.0
>
> Attachments: PHOENIX-2819-2.patch, PHOENIX-2819.patch
>
>
> We should discard the method "ValueSchemaBuilder.addField(Field)" and use the 
> other method signature instead, since this method might add more fields than 
> wanted.
> {code}
> public ValueSchemaBuilder addField(Field field) {
> fields.add(field);
> return this;
> }
> {code}
> If {{field.size}} is larger than 1, the above statement {{fields.add(field)}} 
> will add more than one field into the list while the caller is expecting to 
> add only ONE.
> The only caller of this method is IndexMaintainer.generateIndexRowKeySchema() 
> and it might generate a wrong row-key-schema as a result.



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[GitHub] phoenix pull request: merging latest avatica

[codymarcel]

Github user codymarcel commented on the pull request:

https://github.com/apache/phoenix/pull/157#issuecomment-205858884
  
General Comments.

1. Please add some unit tests. Especially around the instanceResolver and 
RulesApplier. We user 
https://github.com/apache/phoenix/blob/master/phoenix-pherf/src/test/resources/scenario/test_scenario.xml
 as both test base and doc for supported features.
2. Formatting needs some work across the board. Make sure you have the 
formatter for the project applied and your ide is set to convert tabs to spaces.
3. Remove all code that's in comments. 



---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

[jira] [Commented] (PHOENIX-2722) support mysql "limit,offset" clauses

[ASF GitHub Bot (JIRA)]

[ 
https://issues.apache.org/jira/browse/PHOENIX-2722?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15226454#comment-15226454
 ] 

ASF GitHub Bot commented on PHOENIX-2722:
-

Github user JamesRTaylor commented on a diff in the pull request:

https://github.com/apache/phoenix/pull/154#discussion_r58558564
  
--- Diff: 
phoenix-core/src/main/java/org/apache/phoenix/compile/SubselectRewriter.java ---
@@ -188,6 +190,13 @@ private SelectStatement flatten(SelectStatement 
select, SelectStatement subselec
 }
 }
 
+OffsetNode offset = select.getOffset();
+if (offsetRewrite != null || (limitRewrite != null & offset != 
null)) {
--- End diff --

Based on this not being caught by one of your new unit tests, I'd say that 
there's likely a hole in the test coverage. Would you mind adding a unit test 
that would have caught this?


> support mysql "limit,offset" clauses 
> -
>
> Key: PHOENIX-2722
> URL: https://issues.apache.org/jira/browse/PHOENIX-2722
> Project: Phoenix
>  Issue Type: New Feature
>Reporter: Ankit Singhal
>Assignee: Ankit Singhal
>Priority: Minor
> Attachments: PHOENIX-2722.patch, PHOENIX-2722_formatted.patch
>
>
> For serial query(query with “serial" hint or  “limit" without "order by”), we 
> can limit each scan(using page filter) to “limit+offset” instead of limit 
> earlier.
> And then, for all queries, we can forward the relevant client iterators to 
> the offset provided and then return the result.
> syntax
> {code}
> [ LIMIT { count } ]
> [ OFFSET start [ ROW | ROWS ] ]
> [ FETCH { FIRST | NEXT } [ count ] { ROW | ROWS } ONLY ]
> {code}
> Some new keywords(OFFSET,FETCH,ROW, ROWS,ONLY) are getting introduced so 
> users might need to see that they are not using them as column name or 
> something.
> WDYT, [~jamestaylor]



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[GitHub] phoenix pull request: PHOENIX-2722 support mysql limit,offset clau...

[JamesRTaylor]

Github user JamesRTaylor commented on a diff in the pull request:

https://github.com/apache/phoenix/pull/154#discussion_r58558564
  
--- Diff: 
phoenix-core/src/main/java/org/apache/phoenix/compile/SubselectRewriter.java ---
@@ -188,6 +190,13 @@ private SelectStatement flatten(SelectStatement 
select, SelectStatement subselec
 }
 }
 
+OffsetNode offset = select.getOffset();
+if (offsetRewrite != null || (limitRewrite != null & offset != 
null)) {
--- End diff --

Based on this not being caught by one of your new unit tests, I'd say that 
there's likely a hole in the test coverage. Would you mind adding a unit test 
that would have caught this?


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

[GitHub] phoenix pull request: merging latest avatica

[codymarcel]

Github user codymarcel commented on a diff in the pull request:

https://github.com/apache/phoenix/pull/157#discussion_r58557911
  
--- Diff: 
phoenix-pherf/src/main/java/org/apache/phoenix/pherf/workload/WriteWorkload.java
 ---
@@ -253,13 +255,22 @@ private void waitForBatches(DataLoadTimeSummary 
dataLoadTimeSummary, Scenario sc
 
 last = start = System.currentTimeMillis();
 String sql = buildSql(columns, tableName);
+//System.out.print("going to create statement");
--- End diff --

remove commented out code.


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

[GitHub] phoenix pull request: merging latest avatica

[JamesRTaylor]

Github user JamesRTaylor commented on the pull request:

https://github.com/apache/phoenix/pull/157#issuecomment-205851053
  
@gkanade - please file a JIRA and update you git commit message to include 
the JIRA (i.e. PHOENIX- Update Pherf to support ).


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

[jira] [Commented] (PHOENIX-2722) support mysql "limit,offset" clauses

[ASF GitHub Bot (JIRA)]

[ 
https://issues.apache.org/jira/browse/PHOENIX-2722?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15226421#comment-15226421
 ] 

ASF GitHub Bot commented on PHOENIX-2722:
-

Github user JamesRTaylor commented on the pull request:

https://github.com/apache/phoenix/pull/154#issuecomment-205850495
  
A few minor items, but otherwise it looks good. Nice work, @ankitsinghal! 
If you could file a JIRA for the corresponding website/doc changes and submit a 
patch for that when we're closer to the 4.8 release, that'd be much appreciated!


> support mysql "limit,offset" clauses 
> -
>
> Key: PHOENIX-2722
> URL: https://issues.apache.org/jira/browse/PHOENIX-2722
> Project: Phoenix
>  Issue Type: New Feature
>Reporter: Ankit Singhal
>Assignee: Ankit Singhal
>Priority: Minor
> Attachments: PHOENIX-2722.patch, PHOENIX-2722_formatted.patch
>
>
> For serial query(query with “serial" hint or  “limit" without "order by”), we 
> can limit each scan(using page filter) to “limit+offset” instead of limit 
> earlier.
> And then, for all queries, we can forward the relevant client iterators to 
> the offset provided and then return the result.
> syntax
> {code}
> [ LIMIT { count } ]
> [ OFFSET start [ ROW | ROWS ] ]
> [ FETCH { FIRST | NEXT } [ count ] { ROW | ROWS } ONLY ]
> {code}
> Some new keywords(OFFSET,FETCH,ROW, ROWS,ONLY) are getting introduced so 
> users might need to see that they are not using them as column name or 
> something.
> WDYT, [~jamestaylor]



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[GitHub] phoenix pull request: PHOENIX-2722 support mysql limit,offset clau...

[JamesRTaylor]

Github user JamesRTaylor commented on the pull request:

https://github.com/apache/phoenix/pull/154#issuecomment-205850495
  
A few minor items, but otherwise it looks good. Nice work, @ankitsinghal! 
If you could file a JIRA for the corresponding website/doc changes and submit a 
patch for that when we're closer to the 4.8 release, that'd be much appreciated!


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

[jira] [Commented] (PHOENIX-2722) support mysql "limit,offset" clauses

[ASF GitHub Bot (JIRA)]

[ 
https://issues.apache.org/jira/browse/PHOENIX-2722?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15226418#comment-15226418
 ] 

ASF GitHub Bot commented on PHOENIX-2722:
-

Github user JamesRTaylor commented on a diff in the pull request:

https://github.com/apache/phoenix/pull/154#discussion_r58555090
  
--- Diff: 
phoenix-core/src/main/java/org/apache/phoenix/query/QueryConstants.java ---
@@ -138,6 +138,10 @@
 public final static String SYSTEM_SCHEMA_NAME = "SYSTEM";
 public final static byte[] SYSTEM_SCHEMA_NAME_BYTES = 
Bytes.toBytes(SYSTEM_SCHEMA_NAME);
 public final static String PHOENIX_METADATA = "table";
+public final static String SCAN_OFFSET = "OFFSET";
--- End diff --

Please change this to "_RowOffset" (to make is a little less general) and 
move to BaseScannerRegionObserver (this is where we keep all attributes we set 
on the Scan so we hopefully notice if we have a duplicate).


> support mysql "limit,offset" clauses 
> -
>
> Key: PHOENIX-2722
> URL: https://issues.apache.org/jira/browse/PHOENIX-2722
> Project: Phoenix
>  Issue Type: New Feature
>Reporter: Ankit Singhal
>Assignee: Ankit Singhal
>Priority: Minor
> Attachments: PHOENIX-2722.patch, PHOENIX-2722_formatted.patch
>
>
> For serial query(query with “serial" hint or  “limit" without "order by”), we 
> can limit each scan(using page filter) to “limit+offset” instead of limit 
> earlier.
> And then, for all queries, we can forward the relevant client iterators to 
> the offset provided and then return the result.
> syntax
> {code}
> [ LIMIT { count } ]
> [ OFFSET start [ ROW | ROWS ] ]
> [ FETCH { FIRST | NEXT } [ count ] { ROW | ROWS } ONLY ]
> {code}
> Some new keywords(OFFSET,FETCH,ROW, ROWS,ONLY) are getting introduced so 
> users might need to see that they are not using them as column name or 
> something.
> WDYT, [~jamestaylor]



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[GitHub] phoenix pull request: PHOENIX-2722 support mysql limit,offset clau...

[JamesRTaylor]

Github user JamesRTaylor commented on a diff in the pull request:

https://github.com/apache/phoenix/pull/154#discussion_r58555090
  
--- Diff: 
phoenix-core/src/main/java/org/apache/phoenix/query/QueryConstants.java ---
@@ -138,6 +138,10 @@
 public final static String SYSTEM_SCHEMA_NAME = "SYSTEM";
 public final static byte[] SYSTEM_SCHEMA_NAME_BYTES = 
Bytes.toBytes(SYSTEM_SCHEMA_NAME);
 public final static String PHOENIX_METADATA = "table";
+public final static String SCAN_OFFSET = "OFFSET";
--- End diff --

Please change this to "_RowOffset" (to make is a little less general) and 
move to BaseScannerRegionObserver (this is where we keep all attributes we set 
on the Scan so we hopefully notice if we have a duplicate).


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

[GitHub] phoenix pull request: merging latest avatica

[codymarcel]

Github user codymarcel commented on a diff in the pull request:

https://github.com/apache/phoenix/pull/157#discussion_r58555009
  
--- Diff: 
phoenix-pherf/src/main/java/org/apache/phoenix/pherf/configuration/QuerySet.java
 ---
@@ -31,7 +31,24 @@
private long numberOfExecutions = 
PherfConstants.DEFAULT_NUMBER_OF_EXECUTIONS;
private long executionDurationInMs = 
PherfConstants.DEFAULT_THREAD_DURATION_IN_MS;
private ExecutionType executionType = ExecutionType.SERIAL;
-
+   private boolean randomPointRead = false;
+   private String primaryKey = PherfConstants.DEFAULT_PRIMARY_KEY;
+   
+   @XmlAttribute
+   public String getPrimaryKey() {
+   return primaryKey;
--- End diff --

nit: more formatting. Make sure you import the code stye preferences into 
your project.


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

[GitHub] phoenix pull request: merging latest avatica

[codymarcel]

Github user codymarcel commented on a diff in the pull request:

https://github.com/apache/phoenix/pull/157#discussion_r58554617
  
--- Diff: 
phoenix-pherf/src/main/java/org/apache/phoenix/pherf/configuration/DataTypeMapping.java
 ---
@@ -25,7 +25,8 @@
 CHAR("CHAR", Types.CHAR),
 DECIMAL("DECIMAL", Types.DECIMAL),
 INTEGER("INTEGER", Types.INTEGER),
-DATE("DATE", Types.DATE);
+DATE("DATE", Types.DATE),
+   YCSBKEY("YCSBKEY", Types.VARCHAR);
--- End diff --

nit: formatting


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

[GitHub] phoenix pull request: merging latest avatica

[codymarcel]

Github user codymarcel commented on a diff in the pull request:

https://github.com/apache/phoenix/pull/157#discussion_r58554472
  
--- Diff: phoenix-pherf/src/main/java/org/apache/phoenix/pherf/Pherf.java 
---
@@ -254,6 +254,7 @@ public void run() throws Exception {
 // Schema and Data Load
 if (preLoadData) {
 logger.info("\nStarting Data Load...");
+System.out.print("Starting Data Load ...");
--- End diff --

Should be able to use the logger config for stdout instead.


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

[GitHub] phoenix pull request: merging latest avatica

[codymarcel]

Github user codymarcel commented on the pull request:

https://github.com/apache/phoenix/pull/157#issuecomment-205847652
  
Yea, I'll take a look. What is the associated Jira?

On Mon, Apr 4, 2016 at 8:02 PM, James Taylor 
wrote:

> @codymarcel  - would it be possible for
> you to review this pull as Mujtaba is on PTO?
>
> —
> You are receiving this because you were mentioned.
> Reply to this email directly or view it on GitHub
> 
>



---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

[jira] [Commented] (PHOENIX-2722) support mysql "limit,offset" clauses

[ASF GitHub Bot (JIRA)]

[ 
https://issues.apache.org/jira/browse/PHOENIX-2722?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15226411#comment-15226411
 ] 

ASF GitHub Bot commented on PHOENIX-2722:
-

Github user JamesRTaylor commented on a diff in the pull request:

https://github.com/apache/phoenix/pull/154#discussion_r58553769
  
--- Diff: 
phoenix-core/src/main/java/org/apache/phoenix/iterate/OrderedResultIterator.java
 ---
@@ -130,9 +135,9 @@ public OrderedResultIterator(ResultIterator delegate, 
List or
 SizedUtil.OBJECT_SIZE + estimatedRowSize;
 
 // Make sure we don't overflow Long, though this is really 
unlikely to happen.
-assert(limit == null || Long.MAX_VALUE / estimatedEntrySize >= 
limit);
+assert (limit == null || Long.MAX_VALUE / estimatedEntrySize >= 
limit + this.offset);
--- End diff --

Based on the above diff, you added a space after "assert". Please remove so 
our formatting is consistent.


> support mysql "limit,offset" clauses 
> -
>
> Key: PHOENIX-2722
> URL: https://issues.apache.org/jira/browse/PHOENIX-2722
> Project: Phoenix
>  Issue Type: New Feature
>Reporter: Ankit Singhal
>Assignee: Ankit Singhal
>Priority: Minor
> Attachments: PHOENIX-2722.patch, PHOENIX-2722_formatted.patch
>
>
> For serial query(query with “serial" hint or  “limit" without "order by”), we 
> can limit each scan(using page filter) to “limit+offset” instead of limit 
> earlier.
> And then, for all queries, we can forward the relevant client iterators to 
> the offset provided and then return the result.
> syntax
> {code}
> [ LIMIT { count } ]
> [ OFFSET start [ ROW | ROWS ] ]
> [ FETCH { FIRST | NEXT } [ count ] { ROW | ROWS } ONLY ]
> {code}
> Some new keywords(OFFSET,FETCH,ROW, ROWS,ONLY) are getting introduced so 
> users might need to see that they are not using them as column name or 
> something.
> WDYT, [~jamestaylor]



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[GitHub] phoenix pull request: PHOENIX-2722 support mysql limit,offset clau...

[JamesRTaylor]

Github user JamesRTaylor commented on a diff in the pull request:

https://github.com/apache/phoenix/pull/154#discussion_r58553769
  
--- Diff: 
phoenix-core/src/main/java/org/apache/phoenix/iterate/OrderedResultIterator.java
 ---
@@ -130,9 +135,9 @@ public OrderedResultIterator(ResultIterator delegate, 
List or
 SizedUtil.OBJECT_SIZE + estimatedRowSize;
 
 // Make sure we don't overflow Long, though this is really 
unlikely to happen.
-assert(limit == null || Long.MAX_VALUE / estimatedEntrySize >= 
limit);
+assert (limit == null || Long.MAX_VALUE / estimatedEntrySize >= 
limit + this.offset);
--- End diff --

Based on the above diff, you added a space after "assert". Please remove so 
our formatting is consistent.


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

[jira] [Commented] (PHOENIX-2722) support mysql "limit,offset" clauses

[ASF GitHub Bot (JIRA)]

[ 
https://issues.apache.org/jira/browse/PHOENIX-2722?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15226407#comment-15226407
 ] 

ASF GitHub Bot commented on PHOENIX-2722:
-

Github user JamesRTaylor commented on a diff in the pull request:

https://github.com/apache/phoenix/pull/154#discussion_r58553617
  
--- Diff: 
phoenix-core/src/main/java/org/apache/phoenix/iterate/OffsetScanGrouper.java ---
@@ -0,0 +1,44 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements.  See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership.  The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License.  You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+package org.apache.phoenix.iterate;
+
+import java.util.List;
+
+import org.apache.hadoop.hbase.client.Scan;
+import org.apache.phoenix.compile.QueryPlan;
+
+/**
+ * Default implementation that creates a scan group if a plan is row key 
ordered (which requires a merge sort),
+ * or if a scan crosses a region boundary and the table is salted or a 
local index.   
+ */
+public class OffsetScanGrouper implements ParallelScanGrouper {
--- End diff --

Why is this class needed as it doesn't appear to do anything.


> support mysql "limit,offset" clauses 
> -
>
> Key: PHOENIX-2722
> URL: https://issues.apache.org/jira/browse/PHOENIX-2722
> Project: Phoenix
>  Issue Type: New Feature
>Reporter: Ankit Singhal
>Assignee: Ankit Singhal
>Priority: Minor
> Attachments: PHOENIX-2722.patch, PHOENIX-2722_formatted.patch
>
>
> For serial query(query with “serial" hint or  “limit" without "order by”), we 
> can limit each scan(using page filter) to “limit+offset” instead of limit 
> earlier.
> And then, for all queries, we can forward the relevant client iterators to 
> the offset provided and then return the result.
> syntax
> {code}
> [ LIMIT { count } ]
> [ OFFSET start [ ROW | ROWS ] ]
> [ FETCH { FIRST | NEXT } [ count ] { ROW | ROWS } ONLY ]
> {code}
> Some new keywords(OFFSET,FETCH,ROW, ROWS,ONLY) are getting introduced so 
> users might need to see that they are not using them as column name or 
> something.
> WDYT, [~jamestaylor]



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[GitHub] phoenix pull request: PHOENIX-2722 support mysql limit,offset clau...

[JamesRTaylor]

Github user JamesRTaylor commented on a diff in the pull request:

https://github.com/apache/phoenix/pull/154#discussion_r58553617
  
--- Diff: 
phoenix-core/src/main/java/org/apache/phoenix/iterate/OffsetScanGrouper.java ---
@@ -0,0 +1,44 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements.  See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership.  The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License.  You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+package org.apache.phoenix.iterate;
+
+import java.util.List;
+
+import org.apache.hadoop.hbase.client.Scan;
+import org.apache.phoenix.compile.QueryPlan;
+
+/**
+ * Default implementation that creates a scan group if a plan is row key 
ordered (which requires a merge sort),
+ * or if a scan crosses a region boundary and the table is salted or a 
local index.   
+ */
+public class OffsetScanGrouper implements ParallelScanGrouper {
--- End diff --

Why is this class needed as it doesn't appear to do anything.


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

[jira] [Commented] (PHOENIX-2722) support mysql "limit,offset" clauses

[ASF GitHub Bot (JIRA)]

[ 
https://issues.apache.org/jira/browse/PHOENIX-2722?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15226104#comment-15226104
 ] 

ASF GitHub Bot commented on PHOENIX-2722:
-

Github user ankitsinghal commented on a diff in the pull request:

https://github.com/apache/phoenix/pull/154#discussion_r58523285
  
--- Diff: 
phoenix-core/src/test/java/org/apache/phoenix/execute/CorrelatePlanTest.java ---
@@ -172,17 +172,38 @@ public void testCorrelatePlanWithSingleValueOnly() 
throws SQLException {
 {2, "2", "2", 20},
 {5, "5", "5", 100},
 };
-testCorrelatePlan(LEFT_RELATION, rightRelation, 1, 0, 
JoinType.Inner, expected);
+testCorrelatePlan(LEFT_RELATION, rightRelation, 1, 0, 
JoinType.Inner, expected);
 }
 
-private void testCorrelatePlan(Object[][] leftRelation, Object[][] 
rightRelation, int leftCorrelColumn, int rightCorrelColumn, JoinType type, 
Object[][] expectedResult) throws SQLException {
+@Test
--- End diff --

Added in a last commit.


> support mysql "limit,offset" clauses 
> -
>
> Key: PHOENIX-2722
> URL: https://issues.apache.org/jira/browse/PHOENIX-2722
> Project: Phoenix
>  Issue Type: New Feature
>Reporter: Ankit Singhal
>Assignee: Ankit Singhal
>Priority: Minor
> Attachments: PHOENIX-2722.patch, PHOENIX-2722_formatted.patch
>
>
> For serial query(query with “serial" hint or  “limit" without "order by”), we 
> can limit each scan(using page filter) to “limit+offset” instead of limit 
> earlier.
> And then, for all queries, we can forward the relevant client iterators to 
> the offset provided and then return the result.
> syntax
> {code}
> [ LIMIT { count } ]
> [ OFFSET start [ ROW | ROWS ] ]
> [ FETCH { FIRST | NEXT } [ count ] { ROW | ROWS } ONLY ]
> {code}
> Some new keywords(OFFSET,FETCH,ROW, ROWS,ONLY) are getting introduced so 
> users might need to see that they are not using them as column name or 
> something.
> WDYT, [~jamestaylor]



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[GitHub] phoenix pull request: PHOENIX-2722 support mysql limit,offset clau...

[ankitsinghal]

Github user ankitsinghal commented on a diff in the pull request:

https://github.com/apache/phoenix/pull/154#discussion_r58523285
  
--- Diff: 
phoenix-core/src/test/java/org/apache/phoenix/execute/CorrelatePlanTest.java ---
@@ -172,17 +172,38 @@ public void testCorrelatePlanWithSingleValueOnly() 
throws SQLException {
 {2, "2", "2", 20},
 {5, "5", "5", 100},
 };
-testCorrelatePlan(LEFT_RELATION, rightRelation, 1, 0, 
JoinType.Inner, expected);
+testCorrelatePlan(LEFT_RELATION, rightRelation, 1, 0, 
JoinType.Inner, expected);
 }
 
-private void testCorrelatePlan(Object[][] leftRelation, Object[][] 
rightRelation, int leftCorrelColumn, int rightCorrelColumn, JoinType type, 
Object[][] expectedResult) throws SQLException {
+@Test
--- End diff --

Added in a last commit.


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

[jira] [Commented] (PHOENIX-2722) support mysql "limit,offset" clauses

[ASF GitHub Bot (JIRA)]

[ 
https://issues.apache.org/jira/browse/PHOENIX-2722?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15226103#comment-15226103
 ] 

ASF GitHub Bot commented on PHOENIX-2722:
-

Github user ankitsinghal commented on a diff in the pull request:

https://github.com/apache/phoenix/pull/154#discussion_r58523267
  
--- Diff: 
phoenix-core/src/main/java/org/apache/phoenix/iterate/OrderedResultIterator.java
 ---
@@ -130,9 +135,9 @@ public OrderedResultIterator(ResultIterator delegate, 
List or
 SizedUtil.OBJECT_SIZE + estimatedRowSize;
 
 // Make sure we don't overflow Long, though this is really 
unlikely to happen.
-assert(limit == null || Long.MAX_VALUE / estimatedEntrySize >= 
limit);
+assert (limit == null || Long.MAX_VALUE / estimatedEntrySize >= 
limit + this.offset);
--- End diff --

I checked and there was no space before assert.


> support mysql "limit,offset" clauses 
> -
>
> Key: PHOENIX-2722
> URL: https://issues.apache.org/jira/browse/PHOENIX-2722
> Project: Phoenix
>  Issue Type: New Feature
>Reporter: Ankit Singhal
>Assignee: Ankit Singhal
>Priority: Minor
> Attachments: PHOENIX-2722.patch, PHOENIX-2722_formatted.patch
>
>
> For serial query(query with “serial" hint or  “limit" without "order by”), we 
> can limit each scan(using page filter) to “limit+offset” instead of limit 
> earlier.
> And then, for all queries, we can forward the relevant client iterators to 
> the offset provided and then return the result.
> syntax
> {code}
> [ LIMIT { count } ]
> [ OFFSET start [ ROW | ROWS ] ]
> [ FETCH { FIRST | NEXT } [ count ] { ROW | ROWS } ONLY ]
> {code}
> Some new keywords(OFFSET,FETCH,ROW, ROWS,ONLY) are getting introduced so 
> users might need to see that they are not using them as column name or 
> something.
> WDYT, [~jamestaylor]



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[GitHub] phoenix pull request: PHOENIX-2722 support mysql limit,offset clau...

[ankitsinghal]

Github user ankitsinghal commented on a diff in the pull request:

https://github.com/apache/phoenix/pull/154#discussion_r58523267
  
--- Diff: 
phoenix-core/src/main/java/org/apache/phoenix/iterate/OrderedResultIterator.java
 ---
@@ -130,9 +135,9 @@ public OrderedResultIterator(ResultIterator delegate, 
List or
 SizedUtil.OBJECT_SIZE + estimatedRowSize;
 
 // Make sure we don't overflow Long, though this is really 
unlikely to happen.
-assert(limit == null || Long.MAX_VALUE / estimatedEntrySize >= 
limit);
+assert (limit == null || Long.MAX_VALUE / estimatedEntrySize >= 
limit + this.offset);
--- End diff --

I checked and there was no space before assert.


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

[jira] [Commented] (PHOENIX-2817) Phoenix-Spark plugin doesn't work in secured env

[Ankit Singhal (JIRA)]

[ 
https://issues.apache.org/jira/browse/PHOENIX-2817?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15226031#comment-15226031
 ] 

Ankit Singhal commented on PHOENIX-2817:


bq. We can, but the match/cases will be still there
do we really need match case and concat after {code} new 
PhoenixEmbeddedDriver.ConnectionInfo(zkQuorum, port, 
znodeParent).toString(){code}

bq. As for kerberos stuff I'm not sure. principal/keytab pair is specific for 
the particular machine/session. It's fine when we use it in jdbc driver because 
we know that all communication with HBase will be done on this machine, but for 
MR/Yarn jobs that would not work

I think MR framework serializes the client configuration and made it available 
in all M/R context. And bulkload tool(FormatToBytesWritableMapper#setup) in 
phoenix use it to open connection. But anyways ,your patch can still work if 
hbase-site.xml is in class path of client machine as it will be included in 
configuration before sending to other nodes.






> Phoenix-Spark plugin doesn't work in secured env
> 
>
> Key: PHOENIX-2817
> URL: https://issues.apache.org/jira/browse/PHOENIX-2817
> Project: Phoenix
>  Issue Type: Bug
>Affects Versions: 4.4.0, 4.7.0
>Reporter: Sergey Soldatov
>Assignee: Sergey Soldatov
> Attachments: PHOENIX-2817-1.patch, PHOENIX-2817-2.patch, 
> PHOENIX-2817-3.patch
>
>
> When phoenix spark plugin is used with secured setup any attempt to perform 
> operation with PhoenixRDD cause an exception : 
> {noformat}
> Caused by: java.io.IOException: Login failure for 2181 from keytab /hbase: 
> javax.security.auth.login.LoginException: Unable to obtain password from user
>   at 
> org.apache.hadoop.security.UserGroupInformation.loginUserFromKeytab(UserGroupInformation.java:962)
>   at org.apache.hadoop.security.SecurityUtil.login(SecurityUtil.java:275)
>   at 
> org.apache.hadoop.hbase.security.User$SecureHadoopUser.login(User.java:386)
>   at org.apache.hadoop.hbase.security.User.login(User.java:253)
>   at 
> org.apache.phoenix.query.ConnectionQueryServicesImpl.openConnection(ConnectionQueryServicesImpl.java:282)
>   ... 107 more
> Caused by: javax.security.auth.login.LoginException: Unable to obtain 
> password from user
>   at 
> com.sun.security.auth.module.Krb5LoginModule.promptForPass(Krb5LoginModule.java:897)
>   at 
> com.sun.security.auth.module.Krb5LoginModule.attemptAuthentication(Krb5LoginModule.java:760)
>   at 
> com.sun.security.auth.module.Krb5LoginModule.login(Krb5LoginModule.java:617)
>   at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
>   at 
> sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
>   at 
> sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
>   at java.lang.reflect.Method.invoke(Method.java:497)
>   at javax.security.auth.login.LoginContext.invoke(LoginContext.java:755)
>   at 
> javax.security.auth.login.LoginContext.access$000(LoginContext.java:195)
>   at javax.security.auth.login.LoginContext$4.run(LoginContext.java:682)
>   at javax.security.auth.login.LoginContext$4.run(LoginContext.java:680)
>   at java.security.AccessController.doPrivileged(Native Method)
>   at 
> javax.security.auth.login.LoginContext.invokePriv(LoginContext.java:680)
>   at javax.security.auth.login.LoginContext.login(LoginContext.java:587)
>   at 
> org.apache.hadoop.security.UserGroupInformation.loginUserFromKeytab(UserGroupInformation.java:953)
>   ... 111 more
> {noformat}
> The reason is the how zkUrl is handled in PhoenixRDD: 
> {noformat}
> config.set(HConstants.ZOOKEEPER_QUORUM, url )
> {noformat}
> At the same time the {{ConnectionUtil.getInputConnection}} expects to see all 
> parameters (quorum address, port, znodeParent) in different Configuration 
> properties. As the result it gets default values for port and znodeParent and 
> adds it to the provided url, so the {{PhoenixEmbededDriver.create}} receives 
> something like that:
> {noformat}
> jdbc:phoenix:quorum:2181:/hbase-secure:2181:/hbase
> {noformat}
> and consider 2 fields as kerberos principal and keytab.  



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Commented] (PHOENIX-2722) support mysql "limit,offset" clauses

[ASF GitHub Bot (JIRA)]

[ 
https://issues.apache.org/jira/browse/PHOENIX-2722?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15226010#comment-15226010
 ] 

ASF GitHub Bot commented on PHOENIX-2722:
-

Github user ankitsinghal commented on a diff in the pull request:

https://github.com/apache/phoenix/pull/154#discussion_r58512767
  
--- Diff: 
phoenix-core/src/main/java/org/apache/phoenix/iterate/TableSerialIterators.java 
---
@@ -0,0 +1,129 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements.  See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership.  The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License.  You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+package org.apache.phoenix.iterate;
+
+import static org.apache.phoenix.monitoring.MetricType.SCAN_BYTES;
+
+import java.sql.SQLException;
+import java.util.Collections;
+import java.util.List;
+import java.util.Queue;
+import java.util.concurrent.ExecutorService;
+import java.util.concurrent.Future;
+
+import org.apache.hadoop.hbase.client.Scan;
+import org.apache.hadoop.hbase.util.Bytes;
+import org.apache.hadoop.hbase.util.Pair;
+import org.apache.phoenix.compile.QueryPlan;
+import org.apache.phoenix.jdbc.PhoenixConnection;
+import org.apache.phoenix.job.JobManager.JobCallable;
+import org.apache.phoenix.monitoring.TaskExecutionMetricsHolder;
+import org.apache.phoenix.query.QueryConstants;
+import org.apache.phoenix.trace.util.Tracing;
+import org.apache.phoenix.util.ScanUtil;
+
+import com.google.common.base.Preconditions;
+import com.google.common.collect.Lists;
+
+
+/**
+ *
+ * Class that parallelizes the scan over a table using the ExecutorService 
provided.  Each region of the table will be scanned in parallel with
+ * the results accessible through {@link #getIterators()}
+ *
+ * 
+ * @since 0.1
+ */
+public class TableSerialIterators extends BaseResultIterators {
--- End diff --

Yes, I am able re-used it now.. earlier I was facing some issues..


> support mysql "limit,offset" clauses 
> -
>
> Key: PHOENIX-2722
> URL: https://issues.apache.org/jira/browse/PHOENIX-2722
> Project: Phoenix
>  Issue Type: New Feature
>Reporter: Ankit Singhal
>Assignee: Ankit Singhal
>Priority: Minor
> Attachments: PHOENIX-2722.patch, PHOENIX-2722_formatted.patch
>
>
> For serial query(query with “serial" hint or  “limit" without "order by”), we 
> can limit each scan(using page filter) to “limit+offset” instead of limit 
> earlier.
> And then, for all queries, we can forward the relevant client iterators to 
> the offset provided and then return the result.
> syntax
> {code}
> [ LIMIT { count } ]
> [ OFFSET start [ ROW | ROWS ] ]
> [ FETCH { FIRST | NEXT } [ count ] { ROW | ROWS } ONLY ]
> {code}
> Some new keywords(OFFSET,FETCH,ROW, ROWS,ONLY) are getting introduced so 
> users might need to see that they are not using them as column name or 
> something.
> WDYT, [~jamestaylor]



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)