[jira] [Commented] (PHOENIX-4292) Filters on Tables and Views with composite PK of VARCHAR fields with sort direction DESC do not work

[Hadoop QA (JIRA)]

[ 
https://issues.apache.org/jira/browse/PHOENIX-4292?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16210564#comment-16210564
 ] 

Hadoop QA commented on PHOENIX-4292:


{color:red}-1 overall{color}.  Here are the results of testing the latest 
attachment 
  http://issues.apache.org/jira/secure/attachment/12892967/PHOENIX-4292.patch
  against master branch at commit c594038a1bee96b92edd9c788969e3e2dbd6db5a.
  ATTACHMENT ID: 12892967

{color:green}+1 @author{color}.  The patch does not contain any @author 
tags.

{color:red}-1 tests included{color}.  The patch doesn't appear to include 
any new or modified tests.
Please justify why no new tests are needed for this 
patch.
Also please list what manual steps were performed to 
verify this patch.

{color:green}+1 javac{color}.  The applied patch does not increase the 
total number of javac compiler warnings.

{color:green}+1 release audit{color}.  The applied patch does not increase 
the total number of release audit warnings.

{color:red}-1 lineLengths{color}.  The patch introduces the following lines 
longer than 100:
+   + " (TENANT_ID CHAR(15) NOT NULL, 
KEY_PREFIX CHAR(3) NOT NULL, CREATED_DATE DATE, CREATED_BY CHAR(15), 
SYSTEM_MODSTAMP DATE CONSTRAINT PK PRIMARY KEY (TENANT_ID, KEY_PREFIX)) 
VERSIONS=1, MULTI_TENANT=true, IMMUTABLE_ROWS=TRUE, REPLICATION_SCOPE=1");
+   + " (pk1 VARCHAR(10) 
NOT NULL, pk2 VARCHAR(10) NOT NULL, col1 DATE, col3 DECIMAL CONSTRAINT PK 
PRIMARY KEY (pk1 DESC, pk2 DESC)) AS SELECT * FROM "
+   + " (pk1, pk2, col1, col3) 
VALUES ('testa', 'testb', TO_DATE('2017-10-16 22:00:00', '-MM-dd 
HH:mm:ss'), 10)");
+   + " (pk1, pk2, col1, col3) 
VALUES ('testa', 'testc', TO_DATE('2017-10-16 22:00:00', '-MM-dd 
HH:mm:ss'), 10)");
+   + " (pk1, pk2, col1, col3) 
VALUES ('testa', 'testd', TO_DATE('2017-10-16 22:00:00', '-MM-dd 
HH:mm:ss'), 10)");
+   + " (pk1, pk2, col1, col3) 
VALUES ('testa', 'teste', TO_DATE('2017-10-16 22:00:00', '-MM-dd 
HH:mm:ss'), 10)");
+   + " (pk1, pk2, col1, col3) 
VALUES ('testb', 'testa', TO_DATE('2017-10-16 22:00:00', '-MM-dd 
HH:mm:ss'), 10)");

 {color:red}-1 core tests{color}.  The patch failed these unit tests:
 
./phoenix-core/target/failsafe-reports/TEST-org.apache.phoenix.end2end.SetPropertyOnNonEncodedTableIT

Test results: 
https://builds.apache.org/job/PreCommit-PHOENIX-Build/1556//testReport/
Console output: 
https://builds.apache.org/job/PreCommit-PHOENIX-Build/1556//console

This message is automatically generated.

> Filters on Tables and Views with composite PK of VARCHAR fields with sort 
> direction DESC do not work
> 
>
> Key: PHOENIX-4292
> URL: https://issues.apache.org/jira/browse/PHOENIX-4292
> Project: Phoenix
>  Issue Type: Bug
>Affects Versions: 4.10.0
>Reporter: Jan Fernando
>Assignee: Thomas D'Silva
> Fix For: 4.13.0, 4.12.1
>
> Attachments: PHOENIX-4292.patch
>
>
> We noticed that in certain instances on tables and views that were defined 
> with a Composite PK and where the elements of the PK were all DESC that 
> queries exhibited strange behavior and did not return results when expected. 
> A simple query on the first element of the PK returned 0 results e.g SELECT * 
> FROM MY_TABLE WHERE PK1 = 'myvaluethatexists' would return 0 results.
> After some investigation it appears that querying tables and views with a 
> Composite PK that :
> a) have multiple VARCHAR columns in the PK
> b) the sort direction of all the VARCHAR columns is defined as DESC 
>  does not work correctly and the filters are not honored and SQL appears 
> broken to the end user.
> Detailed repro steps:
> ---
> -- 1. Create Global Base Table
> CREATE TABLE IF NOT EXISTS TEST.BASETABLE (
> TENANT_ID CHAR(15) NOT NULL, 
> KEY_PREFIX CHAR(3) NOT NULL, 
> CREATED_DATE DATE,
> CREATED_BY CHAR(15),
> SYSTEM_MODSTAMP DATE
> CONSTRAINT PK PRIMARY KEY (
> TENANT_ID, 
> KEY_PREFIX 
> )
> ) VERSIONS=1, MULTI_TENANT=true, IMMUTABLE_ROWS=TRUE, REPLICATION_SCOPE=1
> -- 2. Create various TENANT SPECIFIC VIEWS i.e. use a tenant specific 
> connection
> CREATE VIEW IF NOT EXISTS TEST."abc" (pk1 VARCHAR(10) NOT NULL, pk2 
> VARCHAR(10) NOT NULL, col1 DATE,  col3 DECIMAL CONSTRAINT PK PRIMARY KEY (pk1 
> DESC, pk2 DESC)) AS SELECT * FROM TEST.BASETABLE WHERE KEY_PREFIX = 'abc';
> CREATE VIEW IF NOT 

[jira] [Commented] (PHOENIX-4292) Filters on Tables and Views with composite PK of VARCHAR fields with sort direction DESC do not work

[James Taylor (JIRA)]

[ 
https://issues.apache.org/jira/browse/PHOENIX-4292?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16210494#comment-16210494
 ] 

James Taylor commented on PHOENIX-4292:
---

+1. Nice work, [~tdsilva].

> Filters on Tables and Views with composite PK of VARCHAR fields with sort 
> direction DESC do not work
> 
>
> Key: PHOENIX-4292
> URL: https://issues.apache.org/jira/browse/PHOENIX-4292
> Project: Phoenix
>  Issue Type: Bug
>Affects Versions: 4.10.0
>Reporter: Jan Fernando
>Assignee: Thomas D'Silva
> Fix For: 4.13.0, 4.12.1
>
> Attachments: PHOENIX-4292.patch
>
>
> We noticed that in certain instances on tables and views that were defined 
> with a Composite PK and where the elements of the PK were all DESC that 
> queries exhibited strange behavior and did not return results when expected. 
> A simple query on the first element of the PK returned 0 results e.g SELECT * 
> FROM MY_TABLE WHERE PK1 = 'myvaluethatexists' would return 0 results.
> After some investigation it appears that querying tables and views with a 
> Composite PK that :
> a) have multiple VARCHAR columns in the PK
> b) the sort direction of all the VARCHAR columns is defined as DESC 
>  does not work correctly and the filters are not honored and SQL appears 
> broken to the end user.
> Detailed repro steps:
> ---
> -- 1. Create Global Base Table
> CREATE TABLE IF NOT EXISTS TEST.BASETABLE (
> TENANT_ID CHAR(15) NOT NULL, 
> KEY_PREFIX CHAR(3) NOT NULL, 
> CREATED_DATE DATE,
> CREATED_BY CHAR(15),
> SYSTEM_MODSTAMP DATE
> CONSTRAINT PK PRIMARY KEY (
> TENANT_ID, 
> KEY_PREFIX 
> )
> ) VERSIONS=1, MULTI_TENANT=true, IMMUTABLE_ROWS=TRUE, REPLICATION_SCOPE=1
> -- 2. Create various TENANT SPECIFIC VIEWS i.e. use a tenant specific 
> connection
> CREATE VIEW IF NOT EXISTS TEST."abc" (pk1 VARCHAR(10) NOT NULL, pk2 
> VARCHAR(10) NOT NULL, col1 DATE,  col3 DECIMAL CONSTRAINT PK PRIMARY KEY (pk1 
> DESC, pk2 DESC)) AS SELECT * FROM TEST.BASETABLE WHERE KEY_PREFIX = 'abc';
> CREATE VIEW IF NOT EXISTS TEST."ab2" (pk1 VARCHAR(10) NOT NULL, pk2 
> VARCHAR(10) NOT NULL, col1 DATE,  col3 DECIMAL CONSTRAINT PK PRIMARY KEY (pk1 
> DESC, pk2 ASC)) AS SELECT * FROM TEST.BASETABLE WHERE KEY_PREFIX = 'ab2';
> CREATE VIEW IF NOT EXISTS TEST."ab3" (pk1 DATE(10) NOT NULL, pk2 DATE(10) NOT 
> NULL, col1 VARCHAR(10),  col3 DECIMAL CONSTRAINT PK PRIMARY KEY (pk1 DESC, 
> pk2 DESC)) AS SELECT * FROM TEST.BASETABLE WHERE KEY_PREFIX = 'ab3';
> CREATE VIEW IF NOT EXISTS TEST."ab4" (pk1 DATE(10) NOT NULL, pk2 DECIMAL NOT 
> NULL, pk3 VARCHAR(10) NOT NULL,  col3 DECIMAL CONSTRAINT PK PRIMARY KEY (pk1 
> DESC, pk2 DESC, pk3 DESC)) AS SELECT * FROM TEST.BASETABLE WHERE KEY_PREFIX = 
> 'ab4';
> -- 3. Test cases that exhibit this issues
> -- SIMULATE EQUALITY QUERY PROBLEMS: View with composite PK with multiple PK 
> values of VARCHAR values DESC
> upsert into TEST."abc" (pk1, pk2, col1, col3) VALUES ('testa', 'testb', 
> TO_DATE('2017-10-16 22:00:00', '-MM-dd HH:mm:ss'), 10); 
> upsert into TEST."abc" (pk1, pk2, col1, col3) VALUES ('testa', 'testc', 
> TO_DATE('2017-10-16 22:00:00', '-MM-dd HH:mm:ss'), 10); 
> upsert into TEST."abc" (pk1, pk2, col1, col3) VALUES ('testa', 'testd', 
> TO_DATE('2017-10-16 22:00:00', '-MM-dd HH:mm:ss'), 10); 
> upsert into TEST."abc" (pk1, pk2, col1, col3) VALUES ('testa', 'teste', 
> TO_DATE('2017-10-16 22:00:00', '-MM-dd HH:mm:ss'), 10); 
> upsert into TEST."abc" (pk1, pk2, col1, col3) VALUES ('testb', 'testa', 
> TO_DATE('2017-10-16 22:00:00', '-MM-dd HH:mm:ss'), 10); 
> SELECT * FROM TEST."abc" WHERE pk1 = 'testa'; -- INCORRECT RESULT: This query 
> returns no records, expected to return 4
> SELECT * FROM TEST."abc"; -- Returns 5 rows as expected
> SELECT * FROM TEST."abc" WHERE pk1 >= 'testa'; -- INCORRECT RESULT: This 
> query returns 1 record, expected to return 5
> SELECT * FROM TEST."abc" WHERE pk1 <= 'testa';  -- Returns 4 rows as expected
> SELECT * FROM TEST."abc" WHERE pk1 > 'testa'; -- Returns 1 row as expected
> SELECT * FROM TEST."abc" WHERE pk1 < 'testa'; -- INCORRECT RESULT: This query 
> returns 1 record, expected to return 0
> -- The following are cases where everything works as expected and which don't 
> have composite VARCHAR PKs
> -- DEMONOSTRATE NO QUERY PROBLEMS WHEN HAVE VIEW WITH SINGLE DESC TEXT PK: 
> View with composite PK with single pk value DESC
> upsert into TEST."ab2" (pk1, pk2, col1, col3) VALUES ('testa', 'testb', 
> TO_DATE('2017-10-16 22:00:00', '-MM-dd HH:mm:ss'), 10); 
> upsert into TEST."ab2" (pk1, pk2, col1, col3) VALUES ('testa', 'testc', 
> TO_DATE('2017-10-16 22:00:00', '-MM-dd HH:mm:ss'), 10); 
> upsert into TEST."ab2" (pk1, pk2, col1, 

[jira] [Commented] (PHOENIX-4280) Delete doesn't work when immutable indexes are in building state

[Hadoop QA (JIRA)]

[ 
https://issues.apache.org/jira/browse/PHOENIX-4280?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16210483#comment-16210483
 ] 

Hadoop QA commented on PHOENIX-4280:


{color:red}-1 overall{color}.  Here are the results of testing the latest 
attachment 
  http://issues.apache.org/jira/secure/attachment/12892953/PHOENIX-4280_v6.patch
  against master branch at commit c594038a1bee96b92edd9c788969e3e2dbd6db5a.
  ATTACHMENT ID: 12892953

{color:green}+1 @author{color}.  The patch does not contain any @author 
tags.

{color:red}-1 tests included{color}.  The patch doesn't appear to include 
any new or modified tests.
Please justify why no new tests are needed for this 
patch.
Also please list what manual steps were performed to 
verify this patch.

{color:green}+1 javac{color}.  The applied patch does not increase the 
total number of javac compiler warnings.

{color:green}+1 release audit{color}.  The applied patch does not increase 
the total number of release audit warnings.

{color:red}-1 lineLengths{color}.  The patch introduces the following lines 
longer than 100:
+rs = conn.createStatement().executeQuery("SELECT /*+ 
NO_INDEX*/ COUNT(*) FROM " + fullTableName);
+
TestUtil.dumpTable(conn.unwrap(PhoenixConnection.class).getQueryServices().getTable(Bytes.toBytes(fullTableName)));
+rs = conn.createStatement().executeQuery("SELECT /*+ NO_INDEX*/ 
COUNT(*) FROM " + fullTableName);
+rs = conn.createStatement().executeQuery("SELECT /*+ NO_INDEX*/ 
COUNT(*) FROM " + fullTableName);
+String dml = "DELETE from " + fullTableName + " WHERE 
varchar_col1='varchar_a' AND varchar_pk='varchar1'";
+
TestUtil.dumpTable(conn.unwrap(PhoenixConnection.class).getQueryServices().getTable(Bytes.toBytes(fullTableName)));
+rs = conn.createStatement().executeQuery("SELECT /*+ NO_INDEX*/ 
COUNT(*) FROM " + fullTableName);
+IndexMaintainer maintainer = 
indexTableRefs.get(i).getTable().getIndexMaintainer(table, connection);
+indexPtr.set(maintainer.buildRowKey(null, indexPtr, 
null, null, HConstants.LATEST_TIMESTAMP));
+if (index.getIndexState() != PIndexState.DISABLE && 
index.getIndexType() == IndexType.GLOBAL) {

 {color:red}-1 core tests{color}.  The patch failed these unit tests:
 
./phoenix-core/target/failsafe-reports/TEST-org.apache.phoenix.end2end.index.DropColumnIT
./phoenix-core/target/failsafe-reports/TEST-org.apache.phoenix.end2end.index.LocalImmutableNonTxIndexIT
./phoenix-core/target/failsafe-reports/TEST-org.apache.phoenix.end2end.index.IndexMaintenanceIT

Test results: 
https://builds.apache.org/job/PreCommit-PHOENIX-Build/1555//testReport/
Console output: 
https://builds.apache.org/job/PreCommit-PHOENIX-Build/1555//console

This message is automatically generated.

> Delete doesn't work when immutable indexes are in building state
> 
>
> Key: PHOENIX-4280
> URL: https://issues.apache.org/jira/browse/PHOENIX-4280
> Project: Phoenix
>  Issue Type: Bug
>Reporter: James Taylor
>Assignee: James Taylor
> Fix For: 4.12.1
>
> Attachments: PHOENIX-4280.patch, PHOENIX-4280_v2.patch, 
> PHOENIX-4280_v3.patch, PHOENIX-4280_v4.patch, PHOENIX-4280_v5.patch, 
> PHOENIX-4280_v6.patch, PHOENIX-4280_wip.patch
>
>
> We don't generate the Delete mutations correctly in this case as we assume 
> we're scanning over the index table.



--
This message was sent by Atlassian JIRA
(v6.4.14#64029)

[jira] [Updated] (PHOENIX-4292) Filters on Tables and Views with composite PK of VARCHAR fields with sort direction DESC do not work

[Thomas D'Silva (JIRA)]

 [ 
https://issues.apache.org/jira/browse/PHOENIX-4292?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Thomas D'Silva updated PHOENIX-4292:

Attachment: PHOENIX-4292.patch

[~jamestaylor]

I thinks its a different issue. The test passes if I change 
{{ValueSchema.Field}} constructor to set the sort order correctly instead of 
always using SortOrder.ASC. I have attached a patch.

> Filters on Tables and Views with composite PK of VARCHAR fields with sort 
> direction DESC do not work
> 
>
> Key: PHOENIX-4292
> URL: https://issues.apache.org/jira/browse/PHOENIX-4292
> Project: Phoenix
>  Issue Type: Bug
>Affects Versions: 4.10.0
>Reporter: Jan Fernando
>Assignee: Thomas D'Silva
> Fix For: 4.13.0, 4.12.1
>
> Attachments: PHOENIX-4292.patch
>
>
> We noticed that in certain instances on tables and views that were defined 
> with a Composite PK and where the elements of the PK were all DESC that 
> queries exhibited strange behavior and did not return results when expected. 
> A simple query on the first element of the PK returned 0 results e.g SELECT * 
> FROM MY_TABLE WHERE PK1 = 'myvaluethatexists' would return 0 results.
> After some investigation it appears that querying tables and views with a 
> Composite PK that :
> a) have multiple VARCHAR columns in the PK
> b) the sort direction of all the VARCHAR columns is defined as DESC 
>  does not work correctly and the filters are not honored and SQL appears 
> broken to the end user.
> Detailed repro steps:
> ---
> -- 1. Create Global Base Table
> CREATE TABLE IF NOT EXISTS TEST.BASETABLE (
> TENANT_ID CHAR(15) NOT NULL, 
> KEY_PREFIX CHAR(3) NOT NULL, 
> CREATED_DATE DATE,
> CREATED_BY CHAR(15),
> SYSTEM_MODSTAMP DATE
> CONSTRAINT PK PRIMARY KEY (
> TENANT_ID, 
> KEY_PREFIX 
> )
> ) VERSIONS=1, MULTI_TENANT=true, IMMUTABLE_ROWS=TRUE, REPLICATION_SCOPE=1
> -- 2. Create various TENANT SPECIFIC VIEWS i.e. use a tenant specific 
> connection
> CREATE VIEW IF NOT EXISTS TEST."abc" (pk1 VARCHAR(10) NOT NULL, pk2 
> VARCHAR(10) NOT NULL, col1 DATE,  col3 DECIMAL CONSTRAINT PK PRIMARY KEY (pk1 
> DESC, pk2 DESC)) AS SELECT * FROM TEST.BASETABLE WHERE KEY_PREFIX = 'abc';
> CREATE VIEW IF NOT EXISTS TEST."ab2" (pk1 VARCHAR(10) NOT NULL, pk2 
> VARCHAR(10) NOT NULL, col1 DATE,  col3 DECIMAL CONSTRAINT PK PRIMARY KEY (pk1 
> DESC, pk2 ASC)) AS SELECT * FROM TEST.BASETABLE WHERE KEY_PREFIX = 'ab2';
> CREATE VIEW IF NOT EXISTS TEST."ab3" (pk1 DATE(10) NOT NULL, pk2 DATE(10) NOT 
> NULL, col1 VARCHAR(10),  col3 DECIMAL CONSTRAINT PK PRIMARY KEY (pk1 DESC, 
> pk2 DESC)) AS SELECT * FROM TEST.BASETABLE WHERE KEY_PREFIX = 'ab3';
> CREATE VIEW IF NOT EXISTS TEST."ab4" (pk1 DATE(10) NOT NULL, pk2 DECIMAL NOT 
> NULL, pk3 VARCHAR(10) NOT NULL,  col3 DECIMAL CONSTRAINT PK PRIMARY KEY (pk1 
> DESC, pk2 DESC, pk3 DESC)) AS SELECT * FROM TEST.BASETABLE WHERE KEY_PREFIX = 
> 'ab4';
> -- 3. Test cases that exhibit this issues
> -- SIMULATE EQUALITY QUERY PROBLEMS: View with composite PK with multiple PK 
> values of VARCHAR values DESC
> upsert into TEST."abc" (pk1, pk2, col1, col3) VALUES ('testa', 'testb', 
> TO_DATE('2017-10-16 22:00:00', '-MM-dd HH:mm:ss'), 10); 
> upsert into TEST."abc" (pk1, pk2, col1, col3) VALUES ('testa', 'testc', 
> TO_DATE('2017-10-16 22:00:00', '-MM-dd HH:mm:ss'), 10); 
> upsert into TEST."abc" (pk1, pk2, col1, col3) VALUES ('testa', 'testd', 
> TO_DATE('2017-10-16 22:00:00', '-MM-dd HH:mm:ss'), 10); 
> upsert into TEST."abc" (pk1, pk2, col1, col3) VALUES ('testa', 'teste', 
> TO_DATE('2017-10-16 22:00:00', '-MM-dd HH:mm:ss'), 10); 
> upsert into TEST."abc" (pk1, pk2, col1, col3) VALUES ('testb', 'testa', 
> TO_DATE('2017-10-16 22:00:00', '-MM-dd HH:mm:ss'), 10); 
> SELECT * FROM TEST."abc" WHERE pk1 = 'testa'; -- INCORRECT RESULT: This query 
> returns no records, expected to return 4
> SELECT * FROM TEST."abc"; -- Returns 5 rows as expected
> SELECT * FROM TEST."abc" WHERE pk1 >= 'testa'; -- INCORRECT RESULT: This 
> query returns 1 record, expected to return 5
> SELECT * FROM TEST."abc" WHERE pk1 <= 'testa';  -- Returns 4 rows as expected
> SELECT * FROM TEST."abc" WHERE pk1 > 'testa'; -- Returns 1 row as expected
> SELECT * FROM TEST."abc" WHERE pk1 < 'testa'; -- INCORRECT RESULT: This query 
> returns 1 record, expected to return 0
> -- The following are cases where everything works as expected and which don't 
> have composite VARCHAR PKs
> -- DEMONOSTRATE NO QUERY PROBLEMS WHEN HAVE VIEW WITH SINGLE DESC TEXT PK: 
> View with composite PK with single pk value DESC
> upsert into TEST."ab2" (pk1, pk2, col1, col3) VALUES ('testa', 'testb', 
> TO_DATE('2017-10-16 22:00:00', '-MM-dd HH:mm:ss'), 10); 
> 

[jira] [Commented] (PHOENIX-4292) Filters on Tables and Views with composite PK of VARCHAR fields with sort direction DESC do not work

[James Taylor (JIRA)]

[ 
https://issues.apache.org/jira/browse/PHOENIX-4292?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16210440#comment-16210440
 ] 

James Taylor commented on PHOENIX-4292:
---

Do you think this is a duplicate of PHOENIX-3383? I have a WIP patch for that, 
but never got the chance to complete it.

> Filters on Tables and Views with composite PK of VARCHAR fields with sort 
> direction DESC do not work
> 
>
> Key: PHOENIX-4292
> URL: https://issues.apache.org/jira/browse/PHOENIX-4292
> Project: Phoenix
>  Issue Type: Bug
>Affects Versions: 4.10.0
>Reporter: Jan Fernando
>Assignee: Thomas D'Silva
> Fix For: 4.13.0, 4.12.1
>
>
> We noticed that in certain instances on tables and views that were defined 
> with a Composite PK and where the elements of the PK were all DESC that 
> queries exhibited strange behavior and did not return results when expected. 
> A simple query on the first element of the PK returned 0 results e.g SELECT * 
> FROM MY_TABLE WHERE PK1 = 'myvaluethatexists' would return 0 results.
> After some investigation it appears that querying tables and views with a 
> Composite PK that :
> a) have multiple VARCHAR columns in the PK
> b) the sort direction of all the VARCHAR columns is defined as DESC 
>  does not work correctly and the filters are not honored and SQL appears 
> broken to the end user.
> Detailed repro steps:
> ---
> -- 1. Create Global Base Table
> CREATE TABLE IF NOT EXISTS TEST.BASETABLE (
> TENANT_ID CHAR(15) NOT NULL, 
> KEY_PREFIX CHAR(3) NOT NULL, 
> CREATED_DATE DATE,
> CREATED_BY CHAR(15),
> SYSTEM_MODSTAMP DATE
> CONSTRAINT PK PRIMARY KEY (
> TENANT_ID, 
> KEY_PREFIX 
> )
> ) VERSIONS=1, MULTI_TENANT=true, IMMUTABLE_ROWS=TRUE, REPLICATION_SCOPE=1
> -- 2. Create various TENANT SPECIFIC VIEWS i.e. use a tenant specific 
> connection
> CREATE VIEW IF NOT EXISTS TEST."abc" (pk1 VARCHAR(10) NOT NULL, pk2 
> VARCHAR(10) NOT NULL, col1 DATE,  col3 DECIMAL CONSTRAINT PK PRIMARY KEY (pk1 
> DESC, pk2 DESC)) AS SELECT * FROM TEST.BASETABLE WHERE KEY_PREFIX = 'abc';
> CREATE VIEW IF NOT EXISTS TEST."ab2" (pk1 VARCHAR(10) NOT NULL, pk2 
> VARCHAR(10) NOT NULL, col1 DATE,  col3 DECIMAL CONSTRAINT PK PRIMARY KEY (pk1 
> DESC, pk2 ASC)) AS SELECT * FROM TEST.BASETABLE WHERE KEY_PREFIX = 'ab2';
> CREATE VIEW IF NOT EXISTS TEST."ab3" (pk1 DATE(10) NOT NULL, pk2 DATE(10) NOT 
> NULL, col1 VARCHAR(10),  col3 DECIMAL CONSTRAINT PK PRIMARY KEY (pk1 DESC, 
> pk2 DESC)) AS SELECT * FROM TEST.BASETABLE WHERE KEY_PREFIX = 'ab3';
> CREATE VIEW IF NOT EXISTS TEST."ab4" (pk1 DATE(10) NOT NULL, pk2 DECIMAL NOT 
> NULL, pk3 VARCHAR(10) NOT NULL,  col3 DECIMAL CONSTRAINT PK PRIMARY KEY (pk1 
> DESC, pk2 DESC, pk3 DESC)) AS SELECT * FROM TEST.BASETABLE WHERE KEY_PREFIX = 
> 'ab4';
> -- 3. Test cases that exhibit this issues
> -- SIMULATE EQUALITY QUERY PROBLEMS: View with composite PK with multiple PK 
> values of VARCHAR values DESC
> upsert into TEST."abc" (pk1, pk2, col1, col3) VALUES ('testa', 'testb', 
> TO_DATE('2017-10-16 22:00:00', '-MM-dd HH:mm:ss'), 10); 
> upsert into TEST."abc" (pk1, pk2, col1, col3) VALUES ('testa', 'testc', 
> TO_DATE('2017-10-16 22:00:00', '-MM-dd HH:mm:ss'), 10); 
> upsert into TEST."abc" (pk1, pk2, col1, col3) VALUES ('testa', 'testd', 
> TO_DATE('2017-10-16 22:00:00', '-MM-dd HH:mm:ss'), 10); 
> upsert into TEST."abc" (pk1, pk2, col1, col3) VALUES ('testa', 'teste', 
> TO_DATE('2017-10-16 22:00:00', '-MM-dd HH:mm:ss'), 10); 
> upsert into TEST."abc" (pk1, pk2, col1, col3) VALUES ('testb', 'testa', 
> TO_DATE('2017-10-16 22:00:00', '-MM-dd HH:mm:ss'), 10); 
> SELECT * FROM TEST."abc" WHERE pk1 = 'testa'; -- INCORRECT RESULT: This query 
> returns no records, expected to return 4
> SELECT * FROM TEST."abc"; -- Returns 5 rows as expected
> SELECT * FROM TEST."abc" WHERE pk1 >= 'testa'; -- INCORRECT RESULT: This 
> query returns 1 record, expected to return 5
> SELECT * FROM TEST."abc" WHERE pk1 <= 'testa';  -- Returns 4 rows as expected
> SELECT * FROM TEST."abc" WHERE pk1 > 'testa'; -- Returns 1 row as expected
> SELECT * FROM TEST."abc" WHERE pk1 < 'testa'; -- INCORRECT RESULT: This query 
> returns 1 record, expected to return 0
> -- The following are cases where everything works as expected and which don't 
> have composite VARCHAR PKs
> -- DEMONOSTRATE NO QUERY PROBLEMS WHEN HAVE VIEW WITH SINGLE DESC TEXT PK: 
> View with composite PK with single pk value DESC
> upsert into TEST."ab2" (pk1, pk2, col1, col3) VALUES ('testa', 'testb', 
> TO_DATE('2017-10-16 22:00:00', '-MM-dd HH:mm:ss'), 10); 
> upsert into TEST."ab2" (pk1, pk2, col1, col3) VALUES ('testa', 'testc', 
> TO_DATE('2017-10-16 22:00:00', '-MM-dd HH:mm:ss'), 

[jira] [Updated] (PHOENIX-4292) Filters on Tables and Views with composite PK of VARCHAR fields with sort direction DESC do not work

[Thomas D'Silva (JIRA)]

 [ 
https://issues.apache.org/jira/browse/PHOENIX-4292?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Thomas D'Silva updated PHOENIX-4292:

Summary: Filters on Tables and Views with composite PK of VARCHAR fields 
with sort direction DESC do not work  (was: SOQL Filters on Tables and Views 
with composite PK of VARCHAR fields with sort direction DESC do not work)

> Filters on Tables and Views with composite PK of VARCHAR fields with sort 
> direction DESC do not work
> 
>
> Key: PHOENIX-4292
> URL: https://issues.apache.org/jira/browse/PHOENIX-4292
> Project: Phoenix
>  Issue Type: Bug
>Affects Versions: 4.10.0
>Reporter: Jan Fernando
>Assignee: Thomas D'Silva
> Fix For: 4.13.0, 4.12.1
>
>
> We noticed that in certain instances on tables and views that were defined 
> with a Composite PK and where the elements of the PK were all DESC that 
> queries exhibited strange behavior and did not return results when expected. 
> A simple query on the first element of the PK returned 0 results e.g SELECT * 
> FROM MY_TABLE WHERE PK1 = 'myvaluethatexists' would return 0 results.
> After some investigation it appears that querying tables and views with a 
> Composite PK that :
> a) have multiple VARCHAR columns in the PK
> b) the sort direction of all the VARCHAR columns is defined as DESC 
>  does not work correctly and the filters are not honored and SQL appears 
> broken to the end user.
> Detailed repro steps:
> ---
> -- 1. Create Global Base Table
> CREATE TABLE IF NOT EXISTS TEST.BASETABLE (
> TENANT_ID CHAR(15) NOT NULL, 
> KEY_PREFIX CHAR(3) NOT NULL, 
> CREATED_DATE DATE,
> CREATED_BY CHAR(15),
> SYSTEM_MODSTAMP DATE
> CONSTRAINT PK PRIMARY KEY (
> TENANT_ID, 
> KEY_PREFIX 
> )
> ) VERSIONS=1, MULTI_TENANT=true, IMMUTABLE_ROWS=TRUE, REPLICATION_SCOPE=1
> -- 2. Create various TENANT SPECIFIC VIEWS i.e. use a tenant specific 
> connection
> CREATE VIEW IF NOT EXISTS TEST."abc" (pk1 VARCHAR(10) NOT NULL, pk2 
> VARCHAR(10) NOT NULL, col1 DATE,  col3 DECIMAL CONSTRAINT PK PRIMARY KEY (pk1 
> DESC, pk2 DESC)) AS SELECT * FROM TEST.BASETABLE WHERE KEY_PREFIX = 'abc';
> CREATE VIEW IF NOT EXISTS TEST."ab2" (pk1 VARCHAR(10) NOT NULL, pk2 
> VARCHAR(10) NOT NULL, col1 DATE,  col3 DECIMAL CONSTRAINT PK PRIMARY KEY (pk1 
> DESC, pk2 ASC)) AS SELECT * FROM TEST.BASETABLE WHERE KEY_PREFIX = 'ab2';
> CREATE VIEW IF NOT EXISTS TEST."ab3" (pk1 DATE(10) NOT NULL, pk2 DATE(10) NOT 
> NULL, col1 VARCHAR(10),  col3 DECIMAL CONSTRAINT PK PRIMARY KEY (pk1 DESC, 
> pk2 DESC)) AS SELECT * FROM TEST.BASETABLE WHERE KEY_PREFIX = 'ab3';
> CREATE VIEW IF NOT EXISTS TEST."ab4" (pk1 DATE(10) NOT NULL, pk2 DECIMAL NOT 
> NULL, pk3 VARCHAR(10) NOT NULL,  col3 DECIMAL CONSTRAINT PK PRIMARY KEY (pk1 
> DESC, pk2 DESC, pk3 DESC)) AS SELECT * FROM TEST.BASETABLE WHERE KEY_PREFIX = 
> 'ab4';
> -- 3. Test cases that exhibit this issues
> -- SIMULATE EQUALITY QUERY PROBLEMS: View with composite PK with multiple PK 
> values of VARCHAR values DESC
> upsert into TEST."abc" (pk1, pk2, col1, col3) VALUES ('testa', 'testb', 
> TO_DATE('2017-10-16 22:00:00', '-MM-dd HH:mm:ss'), 10); 
> upsert into TEST."abc" (pk1, pk2, col1, col3) VALUES ('testa', 'testc', 
> TO_DATE('2017-10-16 22:00:00', '-MM-dd HH:mm:ss'), 10); 
> upsert into TEST."abc" (pk1, pk2, col1, col3) VALUES ('testa', 'testd', 
> TO_DATE('2017-10-16 22:00:00', '-MM-dd HH:mm:ss'), 10); 
> upsert into TEST."abc" (pk1, pk2, col1, col3) VALUES ('testa', 'teste', 
> TO_DATE('2017-10-16 22:00:00', '-MM-dd HH:mm:ss'), 10); 
> upsert into TEST."abc" (pk1, pk2, col1, col3) VALUES ('testb', 'testa', 
> TO_DATE('2017-10-16 22:00:00', '-MM-dd HH:mm:ss'), 10); 
> SELECT * FROM TEST."abc" WHERE pk1 = 'testa'; -- INCORRECT RESULT: This query 
> returns no records, expected to return 4
> SELECT * FROM TEST."abc"; -- Returns 5 rows as expected
> SELECT * FROM TEST."abc" WHERE pk1 >= 'testa'; -- INCORRECT RESULT: This 
> query returns 1 record, expected to return 5
> SELECT * FROM TEST."abc" WHERE pk1 <= 'testa';  -- Returns 4 rows as expected
> SELECT * FROM TEST."abc" WHERE pk1 > 'testa'; -- Returns 1 row as expected
> SELECT * FROM TEST."abc" WHERE pk1 < 'testa'; -- INCORRECT RESULT: This query 
> returns 1 record, expected to return 0
> -- The following are cases where everything works as expected and which don't 
> have composite VARCHAR PKs
> -- DEMONOSTRATE NO QUERY PROBLEMS WHEN HAVE VIEW WITH SINGLE DESC TEXT PK: 
> View with composite PK with single pk value DESC
> upsert into TEST."ab2" (pk1, pk2, col1, col3) VALUES ('testa', 'testb', 
> TO_DATE('2017-10-16 22:00:00', '-MM-dd HH:mm:ss'), 10); 
> upsert into TEST."ab2" (pk1, pk2, col1, col3) VALUES 

[jira] [Updated] (PHOENIX-4292) Filters on Tables and Views with composite PK of VARCHAR fields with sort direction DESC do not work

[Thomas D'Silva (JIRA)]

 [ 
https://issues.apache.org/jira/browse/PHOENIX-4292?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Thomas D'Silva updated PHOENIX-4292:

Fix Version/s: 4.13.0

> Filters on Tables and Views with composite PK of VARCHAR fields with sort 
> direction DESC do not work
> 
>
> Key: PHOENIX-4292
> URL: https://issues.apache.org/jira/browse/PHOENIX-4292
> Project: Phoenix
>  Issue Type: Bug
>Affects Versions: 4.10.0
>Reporter: Jan Fernando
>Assignee: Thomas D'Silva
> Fix For: 4.13.0, 4.12.1
>
>
> We noticed that in certain instances on tables and views that were defined 
> with a Composite PK and where the elements of the PK were all DESC that 
> queries exhibited strange behavior and did not return results when expected. 
> A simple query on the first element of the PK returned 0 results e.g SELECT * 
> FROM MY_TABLE WHERE PK1 = 'myvaluethatexists' would return 0 results.
> After some investigation it appears that querying tables and views with a 
> Composite PK that :
> a) have multiple VARCHAR columns in the PK
> b) the sort direction of all the VARCHAR columns is defined as DESC 
>  does not work correctly and the filters are not honored and SQL appears 
> broken to the end user.
> Detailed repro steps:
> ---
> -- 1. Create Global Base Table
> CREATE TABLE IF NOT EXISTS TEST.BASETABLE (
> TENANT_ID CHAR(15) NOT NULL, 
> KEY_PREFIX CHAR(3) NOT NULL, 
> CREATED_DATE DATE,
> CREATED_BY CHAR(15),
> SYSTEM_MODSTAMP DATE
> CONSTRAINT PK PRIMARY KEY (
> TENANT_ID, 
> KEY_PREFIX 
> )
> ) VERSIONS=1, MULTI_TENANT=true, IMMUTABLE_ROWS=TRUE, REPLICATION_SCOPE=1
> -- 2. Create various TENANT SPECIFIC VIEWS i.e. use a tenant specific 
> connection
> CREATE VIEW IF NOT EXISTS TEST."abc" (pk1 VARCHAR(10) NOT NULL, pk2 
> VARCHAR(10) NOT NULL, col1 DATE,  col3 DECIMAL CONSTRAINT PK PRIMARY KEY (pk1 
> DESC, pk2 DESC)) AS SELECT * FROM TEST.BASETABLE WHERE KEY_PREFIX = 'abc';
> CREATE VIEW IF NOT EXISTS TEST."ab2" (pk1 VARCHAR(10) NOT NULL, pk2 
> VARCHAR(10) NOT NULL, col1 DATE,  col3 DECIMAL CONSTRAINT PK PRIMARY KEY (pk1 
> DESC, pk2 ASC)) AS SELECT * FROM TEST.BASETABLE WHERE KEY_PREFIX = 'ab2';
> CREATE VIEW IF NOT EXISTS TEST."ab3" (pk1 DATE(10) NOT NULL, pk2 DATE(10) NOT 
> NULL, col1 VARCHAR(10),  col3 DECIMAL CONSTRAINT PK PRIMARY KEY (pk1 DESC, 
> pk2 DESC)) AS SELECT * FROM TEST.BASETABLE WHERE KEY_PREFIX = 'ab3';
> CREATE VIEW IF NOT EXISTS TEST."ab4" (pk1 DATE(10) NOT NULL, pk2 DECIMAL NOT 
> NULL, pk3 VARCHAR(10) NOT NULL,  col3 DECIMAL CONSTRAINT PK PRIMARY KEY (pk1 
> DESC, pk2 DESC, pk3 DESC)) AS SELECT * FROM TEST.BASETABLE WHERE KEY_PREFIX = 
> 'ab4';
> -- 3. Test cases that exhibit this issues
> -- SIMULATE EQUALITY QUERY PROBLEMS: View with composite PK with multiple PK 
> values of VARCHAR values DESC
> upsert into TEST."abc" (pk1, pk2, col1, col3) VALUES ('testa', 'testb', 
> TO_DATE('2017-10-16 22:00:00', '-MM-dd HH:mm:ss'), 10); 
> upsert into TEST."abc" (pk1, pk2, col1, col3) VALUES ('testa', 'testc', 
> TO_DATE('2017-10-16 22:00:00', '-MM-dd HH:mm:ss'), 10); 
> upsert into TEST."abc" (pk1, pk2, col1, col3) VALUES ('testa', 'testd', 
> TO_DATE('2017-10-16 22:00:00', '-MM-dd HH:mm:ss'), 10); 
> upsert into TEST."abc" (pk1, pk2, col1, col3) VALUES ('testa', 'teste', 
> TO_DATE('2017-10-16 22:00:00', '-MM-dd HH:mm:ss'), 10); 
> upsert into TEST."abc" (pk1, pk2, col1, col3) VALUES ('testb', 'testa', 
> TO_DATE('2017-10-16 22:00:00', '-MM-dd HH:mm:ss'), 10); 
> SELECT * FROM TEST."abc" WHERE pk1 = 'testa'; -- INCORRECT RESULT: This query 
> returns no records, expected to return 4
> SELECT * FROM TEST."abc"; -- Returns 5 rows as expected
> SELECT * FROM TEST."abc" WHERE pk1 >= 'testa'; -- INCORRECT RESULT: This 
> query returns 1 record, expected to return 5
> SELECT * FROM TEST."abc" WHERE pk1 <= 'testa';  -- Returns 4 rows as expected
> SELECT * FROM TEST."abc" WHERE pk1 > 'testa'; -- Returns 1 row as expected
> SELECT * FROM TEST."abc" WHERE pk1 < 'testa'; -- INCORRECT RESULT: This query 
> returns 1 record, expected to return 0
> -- The following are cases where everything works as expected and which don't 
> have composite VARCHAR PKs
> -- DEMONOSTRATE NO QUERY PROBLEMS WHEN HAVE VIEW WITH SINGLE DESC TEXT PK: 
> View with composite PK with single pk value DESC
> upsert into TEST."ab2" (pk1, pk2, col1, col3) VALUES ('testa', 'testb', 
> TO_DATE('2017-10-16 22:00:00', '-MM-dd HH:mm:ss'), 10); 
> upsert into TEST."ab2" (pk1, pk2, col1, col3) VALUES ('testa', 'testc', 
> TO_DATE('2017-10-16 22:00:00', '-MM-dd HH:mm:ss'), 10); 
> upsert into TEST."ab2" (pk1, pk2, col1, col3) VALUES ('testa', 'testd', 
> TO_DATE('2017-10-16 22:00:00', '-MM-dd 

[jira] [Updated] (PHOENIX-4280) Delete doesn't work when immutable indexes are in building state

[James Taylor (JIRA)]

 [ 
https://issues.apache.org/jira/browse/PHOENIX-4280?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

James Taylor updated PHOENIX-4280:
--
Attachment: PHOENIX-4280_v6.patch

> Delete doesn't work when immutable indexes are in building state
> 
>
> Key: PHOENIX-4280
> URL: https://issues.apache.org/jira/browse/PHOENIX-4280
> Project: Phoenix
>  Issue Type: Bug
>Reporter: James Taylor
>Assignee: James Taylor
> Fix For: 4.12.1
>
> Attachments: PHOENIX-4280.patch, PHOENIX-4280_v2.patch, 
> PHOENIX-4280_v3.patch, PHOENIX-4280_v4.patch, PHOENIX-4280_v5.patch, 
> PHOENIX-4280_v6.patch, PHOENIX-4280_wip.patch
>
>
> We don't generate the Delete mutations correctly in this case as we assume 
> we're scanning over the index table.



--
This message was sent by Atlassian JIRA
(v6.4.14#64029)

[jira] [Updated] (PHOENIX-4280) Delete doesn't work when immutable indexes are in building state

[James Taylor (JIRA)]

 [ 
https://issues.apache.org/jira/browse/PHOENIX-4280?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

James Taylor updated PHOENIX-4280:
--
Attachment: PHOENIX-4280_v5.patch

> Delete doesn't work when immutable indexes are in building state
> 
>
> Key: PHOENIX-4280
> URL: https://issues.apache.org/jira/browse/PHOENIX-4280
> Project: Phoenix
>  Issue Type: Bug
>Reporter: James Taylor
>Assignee: James Taylor
> Fix For: 4.12.1
>
> Attachments: PHOENIX-4280.patch, PHOENIX-4280_v2.patch, 
> PHOENIX-4280_v3.patch, PHOENIX-4280_v4.patch, PHOENIX-4280_v5.patch, 
> PHOENIX-4280_wip.patch
>
>
> We don't generate the Delete mutations correctly in this case as we assume 
> we're scanning over the index table.



--
This message was sent by Atlassian JIRA
(v6.4.14#64029)

[jira] [Commented] (PHOENIX-4242) Fix Indexer post-compact hook logging of NPE and TableNotFound

[Hadoop QA (JIRA)]

[ 
https://issues.apache.org/jira/browse/PHOENIX-4242?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16210378#comment-16210378
 ] 

Hadoop QA commented on PHOENIX-4242:


{color:red}-1 overall{color}.  Here are the results of testing the latest 
attachment 
  
http://issues.apache.org/jira/secure/attachment/12892916/PHOENIX-4242.v4.master.patch
  against master branch at commit c594038a1bee96b92edd9c788969e3e2dbd6db5a.
  ATTACHMENT ID: 12892916

{color:green}+1 @author{color}.  The patch does not contain any @author 
tags.

{color:red}-1 tests included{color}.  The patch doesn't appear to include 
any new or modified tests.
Please justify why no new tests are needed for this 
patch.
Also please list what manual steps were performed to 
verify this patch.

{color:green}+1 javac{color}.  The applied patch does not increase the 
total number of javac compiler warnings.

{color:green}+1 release audit{color}.  The applied patch does not increase 
the total number of release audit warnings.

{color:red}-1 lineLengths{color}.  The patch introduces the following lines 
longer than 100:
+public void postCompact(final 
ObserverContext c, final Store store,
+String fullTableName = 
c.getEnvironment().getRegion().getRegionInfo().getTable().getNameAsString();
+"Major compaction running while index on table is 
disabled.  Clearing index disable timestamp: "
+if (result.getMutationCode() == 
MutationCode.TABLE_ALREADY_EXISTS && result.getTable() == null) {

{color:green}+1 core tests{color}.  The patch passed unit tests in .

Test results: 
https://builds.apache.org/job/PreCommit-PHOENIX-Build/1553//testReport/
Console output: 
https://builds.apache.org/job/PreCommit-PHOENIX-Build/1553//console

This message is automatically generated.

> Fix Indexer post-compact hook logging of NPE and TableNotFound
> --
>
> Key: PHOENIX-4242
> URL: https://issues.apache.org/jira/browse/PHOENIX-4242
> Project: Phoenix
>  Issue Type: Bug
>Affects Versions: 4.12.0
>Reporter: Vincent Poon
>Assignee: Vincent Poon
> Attachments: PHOENIX-4242.v2.master.patch, 
> PHOENIX-4242.v3.master.patch, PHOENIX-4242.v4.master.patch, 
> PHOENIX-4747.v1.master.patch
>
>
> The post-compact hook in the Indexer seems to log extraneous log messages 
> indicating NPE or TableNotFound.  The TableNotFound exceptions seem to 
> indicate actual table names prefixed with MERGE or RESTORE, and sometimes 
> suffixed with a digit, so perhaps these are views or something similar.
> Examples:
> 2017-09-28 13:35:03,118 WARN  [ctions-1506410238599] index.Indexer - Unable 
> to permanently disable indexes being partially rebuild for SYSTEM.SEQUENCE
> java.lang.NullPointerException
> 2017-09-28 10:20:56,406 WARN  [ctions-1506410238415] index.Indexer - Unable 
> to permanently disable indexes being partially rebuild for 
> MERGE_PLATFORM_ENTITY.PLATFORM_IMMUTABLE_ENTITY_DATA2
> org.apache.phoenix.schema.TableNotFoundException: ERROR 1012 (42M03): Table 
> undefined. tableName=MERGE_PLATFORM_ENTITY.PLATFORM_IMMUTABLE_ENTITY_DATA2



--
This message was sent by Atlassian JIRA
(v6.4.14#64029)

[jira] [Commented] (PHOENIX-4298) refactoring to avoid using deprecated API for Put/Delete

[Hadoop QA (JIRA)]

[ 
https://issues.apache.org/jira/browse/PHOENIX-4298?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16210376#comment-16210376
 ] 

Hadoop QA commented on PHOENIX-4298:


{color:red}-1 overall{color}.  Here are the results of testing the latest 
attachment 
  http://issues.apache.org/jira/secure/attachment/12892917/PHOENIX-4298.patch
  against master branch at commit c594038a1bee96b92edd9c788969e3e2dbd6db5a.
  ATTACHMENT ID: 12892917

{color:green}+1 @author{color}.  The patch does not contain any @author 
tags.

{color:green}+1 tests included{color}.  The patch appears to include 37 new 
or modified tests.

{color:green}+1 javac{color}.  The applied patch does not increase the 
total number of javac compiler warnings.

{color:green}+1 release audit{color}.  The applied patch does not increase 
the total number of release audit warnings.

{color:red}-1 lineLengths{color}.  The patch introduces the following lines 
longer than 100:
+put.addColumn(QueryConstants.DEFAULT_COLUMN_FAMILY_BYTES, 
dv, Bytes.toBytes("default"));
+put.addColumn(QueryConstants.DEFAULT_COLUMN_FAMILY_BYTES, 
first, Bytes.toBytes("first"));
+put.addColumn(A_CF, QueryConstants.DEFAULT_COLUMN_FAMILY_BYTES, 
ByteUtil.EMPTY_BYTE_ARRAY);
+put.addColumn(A_CF, ByteUtil.concat(MAX_CLICK_COUNT_DYNCOL_PREFIX, 
USER_ID2_BYTES), PInteger.INSTANCE.toBytes(ENTRY1_CLICK_COUNT));
+put.addColumn(B_CF, ByteUtil.concat(LAST_LOGIN_TIME_DYNCOL_PREFIX, 
USER_ID1_BYTES), PTime.INSTANCE.toBytes(ENTRY1_USER_ID1_LOGIN_TIME));
+put.addColumn(B_CF, ByteUtil.concat(LAST_LOGIN_TIME_DYNCOL_PREFIX, 
USER_ID2_BYTES), PTime.INSTANCE.toBytes(ENTRY1_USER_ID2_LOGIN_TIME));
+put.addColumn(A_CF, QueryConstants.DEFAULT_COLUMN_FAMILY_BYTES, 
ByteUtil.EMPTY_BYTE_ARRAY);
+put.addColumn(A_CF, ByteUtil.concat(MAX_CLICK_COUNT_DYNCOL_PREFIX, 
USER_ID3_BYTES), PInteger.INSTANCE.toBytes(ENTRY2_CLICK_COUNT));
+put.addColumn(B_CF, ByteUtil.concat(LAST_LOGIN_TIME_DYNCOL_PREFIX, 
USER_ID2_BYTES), PTime.INSTANCE.toBytes(ENTRY2_USER_ID2_LOGIN_TIME));
+put.addColumn(B_CF, ByteUtil.concat(LAST_LOGIN_TIME_DYNCOL_PREFIX, 
USER_ID3_BYTES), PTime.INSTANCE.toBytes(ENTRY2_USER_ID3_LOGIN_TIME));

{color:green}+1 core tests{color}.  The patch passed unit tests in .

Test results: 
https://builds.apache.org/job/PreCommit-PHOENIX-Build/1552//testReport/
Console output: 
https://builds.apache.org/job/PreCommit-PHOENIX-Build/1552//console

This message is automatically generated.

> refactoring to avoid using deprecated API for Put/Delete
> 
>
> Key: PHOENIX-4298
> URL: https://issues.apache.org/jira/browse/PHOENIX-4298
> Project: Phoenix
>  Issue Type: Sub-task
>Reporter: Sergey Soldatov
>Assignee: Sergey Soldatov
>  Labels: HBase-2.0
> Fix For: 4.13.0
>
> Attachments: PHOENIX-4298.patch
>
>




--
This message was sent by Atlassian JIRA
(v6.4.14#64029)

[jira] [Commented] (PHOENIX-4280) Delete doesn't work when immutable indexes are in building state

[Hadoop QA (JIRA)]

[ 
https://issues.apache.org/jira/browse/PHOENIX-4280?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16210369#comment-16210369
 ] 

Hadoop QA commented on PHOENIX-4280:


{color:red}-1 overall{color}.  Here are the results of testing the latest 
attachment 
  http://issues.apache.org/jira/secure/attachment/12892932/PHOENIX-4280_v4.patch
  against master branch at commit c594038a1bee96b92edd9c788969e3e2dbd6db5a.
  ATTACHMENT ID: 12892932

{color:green}+1 @author{color}.  The patch does not contain any @author 
tags.

{color:red}-1 tests included{color}.  The patch doesn't appear to include 
any new or modified tests.
Please justify why no new tests are needed for this 
patch.
Also please list what manual steps were performed to 
verify this patch.

{color:green}+1 javac{color}.  The applied patch does not increase the 
total number of javac compiler warnings.

{color:green}+1 release audit{color}.  The applied patch does not increase 
the total number of release audit warnings.

{color:red}-1 lineLengths{color}.  The patch introduces the following lines 
longer than 100:
+rs = conn.createStatement().executeQuery("SELECT /*+ 
NO_INDEX*/ COUNT(*) FROM " + fullTableName);
+
TestUtil.dumpTable(conn.unwrap(PhoenixConnection.class).getQueryServices().getTable(Bytes.toBytes(fullTableName)));
+rs = conn.createStatement().executeQuery("SELECT /*+ NO_INDEX*/ 
COUNT(*) FROM " + fullTableName);
+rs = conn.createStatement().executeQuery("SELECT /*+ NO_INDEX*/ 
COUNT(*) FROM " + fullTableName);
+String dml = "DELETE from " + fullTableName + " WHERE 
varchar_col1='varchar_a' AND varchar_pk='varchar1'";
+
TestUtil.dumpTable(conn.unwrap(PhoenixConnection.class).getQueryServices().getTable(Bytes.toBytes(fullTableName)));
+rs = conn.createStatement().executeQuery("SELECT /*+ NO_INDEX*/ 
COUNT(*) FROM " + fullTableName);
+IndexMaintainer maintainer = 
indexTableRefs.get(i).getTable().getIndexMaintainer(table, connection);
+indexPtr.set(maintainer.buildRowKey(null, indexPtr, 
null, null, HConstants.LATEST_TIMESTAMP));
+if (index.getIndexState() != PIndexState.DISABLE && 
index.getIndexType() == IndexType.GLOBAL) {

 {color:red}-1 core tests{color}.  The patch failed these unit tests:
   
org.apache.phoenix.hbase.index.covered.LocalTableStateTest

Test results: 
https://builds.apache.org/job/PreCommit-PHOENIX-Build/1554//testReport/
Console output: 
https://builds.apache.org/job/PreCommit-PHOENIX-Build/1554//console

This message is automatically generated.

> Delete doesn't work when immutable indexes are in building state
> 
>
> Key: PHOENIX-4280
> URL: https://issues.apache.org/jira/browse/PHOENIX-4280
> Project: Phoenix
>  Issue Type: Bug
>Reporter: James Taylor
>Assignee: James Taylor
> Fix For: 4.12.1
>
> Attachments: PHOENIX-4280.patch, PHOENIX-4280_v2.patch, 
> PHOENIX-4280_v3.patch, PHOENIX-4280_v4.patch, PHOENIX-4280_wip.patch
>
>
> We don't generate the Delete mutations correctly in this case as we assume 
> we're scanning over the index table.



--
This message was sent by Atlassian JIRA
(v6.4.14#64029)

[jira] [Updated] (PHOENIX-4280) Delete doesn't work when immutable indexes are in building state

[James Taylor (JIRA)]

 [ 
https://issues.apache.org/jira/browse/PHOENIX-4280?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

James Taylor updated PHOENIX-4280:
--
Attachment: PHOENIX-4280_v4.patch

Attaching new version for test run. Will still clean up a little bit, but 
functionally this is complete.

> Delete doesn't work when immutable indexes are in building state
> 
>
> Key: PHOENIX-4280
> URL: https://issues.apache.org/jira/browse/PHOENIX-4280
> Project: Phoenix
>  Issue Type: Bug
>Reporter: James Taylor
>Assignee: James Taylor
> Fix For: 4.12.1
>
> Attachments: PHOENIX-4280.patch, PHOENIX-4280_v2.patch, 
> PHOENIX-4280_v3.patch, PHOENIX-4280_v4.patch, PHOENIX-4280_wip.patch
>
>
> We don't generate the Delete mutations correctly in this case as we assume 
> we're scanning over the index table.



--
This message was sent by Atlassian JIRA
(v6.4.14#64029)

[jira] [Commented] (PHOENIX-4300) Do not use preIncrementAfterRowLock for UPSERT VALUES ON DUPLICATE KEY statement

[James Taylor (JIRA)]

[ 
https://issues.apache.org/jira/browse/PHOENIX-4300?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16210259#comment-16210259
 ] 

James Taylor commented on PHOENIX-4300:
---

No, no dependencies on HBase 2.0 features.

> Do not use preIncrementAfterRowLock for UPSERT VALUES ON DUPLICATE KEY 
> statement
> 
>
> Key: PHOENIX-4300
> URL: https://issues.apache.org/jira/browse/PHOENIX-4300
> Project: Phoenix
>  Issue Type: Bug
>Reporter: James Taylor
>  Labels: HBase-2.0
>
> Instead of using Increment on the client and preIncrementAfterRowLock on the 
> server (in Indexer) for UPSERT VALUES ... ON DUPLICATE KEY command, we can do 
> the following:
> - Use regular Put with the same attributes carrying the ON DUPLICATE KEY info
> - Lock rows with ON DUPLICATE KEY info with our Phoenix-based (borrowed from 
> HBase) locking
> - Once all rows are locked, do an mvcc.await(). This and the previous step 
> will put the rows into the same state they were in during the 
> preIncrementAfterRowLock hook (but actually be more efficient as we'll only 
> do a single mvcc.await() instead of one per row).
> - Execute the {{this.builder.executeAtomicOp(inc)}} call as is done in the 
> preIncrementAfterRowLock using the Mutation instead of the Increment
> - Add a postBatchMutate() hook and unlock the rows we locked above. Unlike 
> with secondary index implementation, we don't need to hold the locks any 
> longer



--
This message was sent by Atlassian JIRA
(v6.4.14#64029)

[jira] [Commented] (PHOENIX-4300) Do not use preIncrementAfterRowLock for UPSERT VALUES ON DUPLICATE KEY statement

[Geoffrey Jacoby (JIRA)]

[ 
https://issues.apache.org/jira/browse/PHOENIX-4300?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16210234#comment-16210234
 ] 

Geoffrey Jacoby commented on PHOENIX-4300:
--

[~jamestaylor] - would this depend on any new HBase 2.0 features or could this 
also go into the 1.x branch(es)? 

> Do not use preIncrementAfterRowLock for UPSERT VALUES ON DUPLICATE KEY 
> statement
> 
>
> Key: PHOENIX-4300
> URL: https://issues.apache.org/jira/browse/PHOENIX-4300
> Project: Phoenix
>  Issue Type: Bug
>Reporter: James Taylor
>  Labels: HBase-2.0
>
> Instead of using Increment on the client and preIncrementAfterRowLock on the 
> server (in Indexer) for UPSERT VALUES ... ON DUPLICATE KEY command, we can do 
> the following:
> - Use regular Put with the same attributes carrying the ON DUPLICATE KEY info
> - Lock rows with ON DUPLICATE KEY info with our Phoenix-based (borrowed from 
> HBase) locking
> - Once all rows are locked, do an mvcc.await(). This and the previous step 
> will put the rows into the same state they were in during the 
> preIncrementAfterRowLock hook (but actually be more efficient as we'll only 
> do a single mvcc.await() instead of one per row).
> - Execute the {{this.builder.executeAtomicOp(inc)}} call as is done in the 
> preIncrementAfterRowLock using the Mutation instead of the Increment
> - Add a postBatchMutate() hook and unlock the rows we locked above. Unlike 
> with secondary index implementation, we don't need to hold the locks any 
> longer



--
This message was sent by Atlassian JIRA
(v6.4.14#64029)

[jira] [Updated] (PHOENIX-4100) Simplify mutable secondary index implementation

[James Taylor (JIRA)]

 [ 
https://issues.apache.org/jira/browse/PHOENIX-4100?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

James Taylor updated PHOENIX-4100:
--
Description: 
There's a lot of code for mutable secondary indexes, a lot of which is 
commented out (because it's not working under load). With all the latest 
patches, in particular PHOENIX-4089, we don't need most of it. Instead, we can 
do a simple scan to find the current value of the data row. We won't have 
mutations at the same time due to our locking and due to us timestamping the 
rows.

This will get rid of many private HBase classes used by Phoenix, such as 
IndexMemStore, KeyValueStore, etc. and thus is likely a prerequisite to run on 
top of HBase 2.0.

  was:There's a lot of code for mutable secondary indexes, a lot of which is 
commented out (because it's not working under load). With all the latest 
patches, in particular PHOENIX-4089, we don't need most of it. Instead, we can 
do a simple scan to find the current value of the data row. We won't have 
mutations at the same time due to our locking and due to us timestamping the 
rows.


> Simplify mutable secondary index implementation
> ---
>
> Key: PHOENIX-4100
> URL: https://issues.apache.org/jira/browse/PHOENIX-4100
> Project: Phoenix
>  Issue Type: Improvement
>Reporter: James Taylor
>  Labels: HBase-2.0
>
> There's a lot of code for mutable secondary indexes, a lot of which is 
> commented out (because it's not working under load). With all the latest 
> patches, in particular PHOENIX-4089, we don't need most of it. Instead, we 
> can do a simple scan to find the current value of the data row. We won't have 
> mutations at the same time due to our locking and due to us timestamping the 
> rows.
> This will get rid of many private HBase classes used by Phoenix, such as 
> IndexMemStore, KeyValueStore, etc. and thus is likely a prerequisite to run 
> on top of HBase 2.0.



--
This message was sent by Atlassian JIRA
(v6.4.14#64029)

[jira] [Updated] (PHOENIX-4100) Simplify mutable secondary index implementation

[James Taylor (JIRA)]

 [ 
https://issues.apache.org/jira/browse/PHOENIX-4100?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

James Taylor updated PHOENIX-4100:
--
Labels: HBase-2.0  (was: )

> Simplify mutable secondary index implementation
> ---
>
> Key: PHOENIX-4100
> URL: https://issues.apache.org/jira/browse/PHOENIX-4100
> Project: Phoenix
>  Issue Type: Improvement
>Reporter: James Taylor
>  Labels: HBase-2.0
>
> There's a lot of code for mutable secondary indexes, a lot of which is 
> commented out (because it's not working under load). With all the latest 
> patches, in particular PHOENIX-4089, we don't need most of it. Instead, we 
> can do a simple scan to find the current value of the data row. We won't have 
> mutations at the same time due to our locking and due to us timestamping the 
> rows.



--
This message was sent by Atlassian JIRA
(v6.4.14#64029)

Re: [DISCUSS] Road to HBase 2.0.0

[James Taylor]

FYI, I filed JIRAs for the Phoenix changes we talked about in the
spreadsheet and labeled them all with HBase-2.0 [1].

[1]
https://issues.apache.org/jira/issues/?jql=project%20%3D%20PHOENIX%20AND%20resolution%20%3D%20Unresolved%20AND%20labels%3D%22HBase-2.0%22

On Wed, Oct 18, 2017 at 3:38 PM, Stack  wrote:

> On Wed, Oct 18, 2017 at 12:13 PM, Josh Elser  wrote:
>
> >
> > On 10/18/17 2:51 PM, Andrew Purtell wrote:
> >
> >> Hopefully, Phoenix will not find itself in a position where
> >>>
> >> business-critical things it's doing are being removed from API (and
> >> Phoenix
> >> has no recourse to hack what it needs back in place).
> >>
> >> I feel fairly confident this is going to happen given the radical
> excision
> >> of methods from the interfaces. What I hope is we can get a round of
> >> evaluation by the Phoenix folks of the fallout in between our
> introduction
> >> of these changes in an alpha before we cement the changes in a GA
> release.
> >>
> >
> > /me nods
> >
> > I know Ankit and Rajeshbabu have already started looking at this
> > internally (I'll take the blame there -- I didn't expect 4.12 to go out
> the
> > door so quickly, I had recommended they just do it locally for now).
> >
> > I've asked them to pull the work they've done in porting out to Apache.
> > I'm expecting that we'll see them push up a branch with what they have
> > tomorrow :)
> >
> > Getting us to a place where we have a branch of Phoenix with the
> necessary
> > HBase 2.0 fixes in place is the path forward I see. I'll help out as much
> > as I can here too.
> >
>
>
> Post-alpha-4, can we soon-after organize a few hours where a bunch of us
> work on bringing Phoenix over together all sitting in the one hangout? As
> per Andrew, a bunch of stuff is being purged (access to Private classes
> mostly, etc.) but there is usually an alternate route. A lone Phoenixer
> trying to figure it may get lost. If those of us who have been working on
> the CP hackup were to-hand, we could show(doc)-the-way or if none,
> plumb-the-way and get the new routes in before 2.0.0RC0.
>
> Thanks,
> S
>

[jira] [Updated] (PHOENIX-4297) Remove deprecated API umbrella

[James Taylor (JIRA)]

 [ 
https://issues.apache.org/jira/browse/PHOENIX-4297?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

James Taylor updated PHOENIX-4297:
--
Labels: HBase-2.0  (was: )

> Remove deprecated API umbrella
> --
>
> Key: PHOENIX-4297
> URL: https://issues.apache.org/jira/browse/PHOENIX-4297
> Project: Phoenix
>  Issue Type: Improvement
>Affects Versions: 4.12.0
>Reporter: Sergey Soldatov
>  Labels: HBase-2.0
> Fix For: 4.13.0
>
>
> For further support of HBase 2.0 we need to remove all deprecated API that 
> was used because of 0.98 support. This is umbrella ticket because some of the 
> changes may not  be trivial.



--
This message was sent by Atlassian JIRA
(v6.4.14#64029)

[jira] [Updated] (PHOENIX-4298) refactoring to avoid using deprecated API for Put/Delete

[James Taylor (JIRA)]

 [ 
https://issues.apache.org/jira/browse/PHOENIX-4298?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

James Taylor updated PHOENIX-4298:
--
Labels: HBase-2.0  (was: )

> refactoring to avoid using deprecated API for Put/Delete
> 
>
> Key: PHOENIX-4298
> URL: https://issues.apache.org/jira/browse/PHOENIX-4298
> Project: Phoenix
>  Issue Type: Sub-task
>Reporter: Sergey Soldatov
>Assignee: Sergey Soldatov
>  Labels: HBase-2.0
> Fix For: 4.13.0
>
> Attachments: PHOENIX-4298.patch
>
>




--
This message was sent by Atlassian JIRA
(v6.4.14#64029)

[jira] [Created] (PHOENIX-4301) Use operation.setPriority() instead of having our own PhoenixRpcSchedulerFactory

[James Taylor (JIRA)]

James Taylor created PHOENIX-4301:
-

 Summary: Use operation.setPriority() instead of having our own 
PhoenixRpcSchedulerFactory
 Key: PHOENIX-4301
 URL: https://issues.apache.org/jira/browse/PHOENIX-4301
 Project: Phoenix
  Issue Type: Bug
Reporter: James Taylor


Now that it's possible to set the priority of an operation, we can get rid of 
the usage of PhoenixRpcSchedulerFactory and just set the priority ourselves 
since the RS->RS calls are in a known set of limited locations:
- ParallelWriterIndexCommitter {{table.batch()}} call
- TrackingParallelWriterIndexCommitter {{table.batch()}} call
- Invocation of MetaDataEndPointImpl calls such as the one that updates the 
index status in PhoenixIndexFailurePolicy
- Get issued while running stats to get guidepost width in 
DefaultStatisticsCollector

The last two aren't as important as the first two. If feasible, we can make all 
interactions with SYSTEM.CATALOG and MetaDataEndPointImpl higher priority.



--
This message was sent by Atlassian JIRA
(v6.4.14#64029)

Re: [DISCUSS] Road to HBase 2.0.0

[Stack]

On Wed, Oct 18, 2017 at 12:13 PM, Josh Elser  wrote:

>
> On 10/18/17 2:51 PM, Andrew Purtell wrote:
>
>> Hopefully, Phoenix will not find itself in a position where
>>>
>> business-critical things it's doing are being removed from API (and
>> Phoenix
>> has no recourse to hack what it needs back in place).
>>
>> I feel fairly confident this is going to happen given the radical excision
>> of methods from the interfaces. What I hope is we can get a round of
>> evaluation by the Phoenix folks of the fallout in between our introduction
>> of these changes in an alpha before we cement the changes in a GA release.
>>
>
> /me nods
>
> I know Ankit and Rajeshbabu have already started looking at this
> internally (I'll take the blame there -- I didn't expect 4.12 to go out the
> door so quickly, I had recommended they just do it locally for now).
>
> I've asked them to pull the work they've done in porting out to Apache.
> I'm expecting that we'll see them push up a branch with what they have
> tomorrow :)
>
> Getting us to a place where we have a branch of Phoenix with the necessary
> HBase 2.0 fixes in place is the path forward I see. I'll help out as much
> as I can here too.
>


Post-alpha-4, can we soon-after organize a few hours where a bunch of us
work on bringing Phoenix over together all sitting in the one hangout? As
per Andrew, a bunch of stuff is being purged (access to Private classes
mostly, etc.) but there is usually an alternate route. A lone Phoenixer
trying to figure it may get lost. If those of us who have been working on
the CP hackup were to-hand, we could show(doc)-the-way or if none,
plumb-the-way and get the new routes in before 2.0.0RC0.

Thanks,
S

[jira] [Updated] (PHOENIX-4300) Do not use preIncrementAfterRowLock for UPSERT VALUES ON DUPLICATE KEY statement

[James Taylor (JIRA)]

 [ 
https://issues.apache.org/jira/browse/PHOENIX-4300?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

James Taylor updated PHOENIX-4300:
--
Description: 
Instead of using Increment on the client and preIncrementAfterRowLock on the 
server (in Indexer) for UPSERT VALUES ... ON DUPLICATE KEY command, we can do 
the following:
- Use regular Put with the same attributes carrying the ON DUPLICATE KEY info
- Lock rows with ON DUPLICATE KEY info with our Phoenix-based (borrowed from 
HBase) locking
- Once all rows are locked, do an mvcc.await(). This and the previous step will 
put the rows into the same state they were in during the 
preIncrementAfterRowLock hook (but actually be more efficient as we'll only do 
a single mvcc.await() instead of one per row).
- Execute the {{this.builder.executeAtomicOp(inc)}} call as is done in the 
preIncrementAfterRowLock using the Mutation instead of the Increment
- Add a postBatchMutate() hook and unlock the rows we locked above. Unlike with 
secondary index implementation, we don't need to hold the locks any longer

  was:Instead of using Increment on the client and 


> Do not use preIncrementAfterRowLock for UPSERT VALUES ON DUPLICATE KEY 
> statement
> 
>
> Key: PHOENIX-4300
> URL: https://issues.apache.org/jira/browse/PHOENIX-4300
> Project: Phoenix
>  Issue Type: Bug
>Reporter: James Taylor
>  Labels: HBase-2.0
>
> Instead of using Increment on the client and preIncrementAfterRowLock on the 
> server (in Indexer) for UPSERT VALUES ... ON DUPLICATE KEY command, we can do 
> the following:
> - Use regular Put with the same attributes carrying the ON DUPLICATE KEY info
> - Lock rows with ON DUPLICATE KEY info with our Phoenix-based (borrowed from 
> HBase) locking
> - Once all rows are locked, do an mvcc.await(). This and the previous step 
> will put the rows into the same state they were in during the 
> preIncrementAfterRowLock hook (but actually be more efficient as we'll only 
> do a single mvcc.await() instead of one per row).
> - Execute the {{this.builder.executeAtomicOp(inc)}} call as is done in the 
> preIncrementAfterRowLock using the Mutation instead of the Increment
> - Add a postBatchMutate() hook and unlock the rows we locked above. Unlike 
> with secondary index implementation, we don't need to hold the locks any 
> longer



--
This message was sent by Atlassian JIRA
(v6.4.14#64029)

[jira] [Updated] (PHOENIX-4300) Do not use preIncrementAfterRowLock for UPSERT VALUES ON DUPLICATE KEY statement

[James Taylor (JIRA)]

 [ 
https://issues.apache.org/jira/browse/PHOENIX-4300?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

James Taylor updated PHOENIX-4300:
--
Description: Instead of using Increment on the client and 

> Do not use preIncrementAfterRowLock for UPSERT VALUES ON DUPLICATE KEY 
> statement
> 
>
> Key: PHOENIX-4300
> URL: https://issues.apache.org/jira/browse/PHOENIX-4300
> Project: Phoenix
>  Issue Type: Bug
>Reporter: James Taylor
>  Labels: HBase-2.0
>
> Instead of using Increment on the client and 



--
This message was sent by Atlassian JIRA
(v6.4.14#64029)

[jira] [Created] (PHOENIX-4300) Do not use preIncrementAfterRowLock for UPSERT VALUES ON DUPLICATE KEY statement

[James Taylor (JIRA)]

James Taylor created PHOENIX-4300:
-

 Summary: Do not use preIncrementAfterRowLock for UPSERT VALUES ON 
DUPLICATE KEY statement
 Key: PHOENIX-4300
 URL: https://issues.apache.org/jira/browse/PHOENIX-4300
 Project: Phoenix
  Issue Type: Bug
Reporter: James Taylor






--
This message was sent by Atlassian JIRA
(v6.4.14#64029)

[jira] [Updated] (PHOENIX-4242) Fix Indexer post-compact hook logging of NPE and TableNotFound

[Vincent Poon (JIRA)]

 [ 
https://issues.apache.org/jira/browse/PHOENIX-4242?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Vincent Poon updated PHOENIX-4242:
--
Attachment: PHOENIX-4242.v4.master.patch

[~giacomotaylor] version that just handles NPE and TableNotFound.  Removes the 
postCompact in Indexer, and has a small fix to MetaDataClient for system tables.

> Fix Indexer post-compact hook logging of NPE and TableNotFound
> --
>
> Key: PHOENIX-4242
> URL: https://issues.apache.org/jira/browse/PHOENIX-4242
> Project: Phoenix
>  Issue Type: Bug
>Affects Versions: 4.12.0
>Reporter: Vincent Poon
>Assignee: Vincent Poon
> Attachments: PHOENIX-4242.v2.master.patch, 
> PHOENIX-4242.v3.master.patch, PHOENIX-4242.v4.master.patch, 
> PHOENIX-4747.v1.master.patch
>
>
> The post-compact hook in the Indexer seems to log extraneous log messages 
> indicating NPE or TableNotFound.  The TableNotFound exceptions seem to 
> indicate actual table names prefixed with MERGE or RESTORE, and sometimes 
> suffixed with a digit, so perhaps these are views or something similar.
> Examples:
> 2017-09-28 13:35:03,118 WARN  [ctions-1506410238599] index.Indexer - Unable 
> to permanently disable indexes being partially rebuild for SYSTEM.SEQUENCE
> java.lang.NullPointerException
> 2017-09-28 10:20:56,406 WARN  [ctions-1506410238415] index.Indexer - Unable 
> to permanently disable indexes being partially rebuild for 
> MERGE_PLATFORM_ENTITY.PLATFORM_IMMUTABLE_ENTITY_DATA2
> org.apache.phoenix.schema.TableNotFoundException: ERROR 1012 (42M03): Table 
> undefined. tableName=MERGE_PLATFORM_ENTITY.PLATFORM_IMMUTABLE_ENTITY_DATA2



--
This message was sent by Atlassian JIRA
(v6.4.14#64029)

[jira] [Updated] (PHOENIX-4298) refactoring to avoid using deprecated API for Put/Delete

[Sergey Soldatov (JIRA)]

 [ 
https://issues.apache.org/jira/browse/PHOENIX-4298?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Sergey Soldatov updated PHOENIX-4298:
-
Attachment: PHOENIX-4298.patch

Changes:
Put.add => Put.addColumn
Delete.deleteColumn => Delete.addColumn

> refactoring to avoid using deprecated API for Put/Delete
> 
>
> Key: PHOENIX-4298
> URL: https://issues.apache.org/jira/browse/PHOENIX-4298
> Project: Phoenix
>  Issue Type: Sub-task
>Reporter: Sergey Soldatov
>Assignee: Sergey Soldatov
> Fix For: 4.13.0
>
> Attachments: PHOENIX-4298.patch
>
>




--
This message was sent by Atlassian JIRA
(v6.4.14#64029)

[jira] [Created] (PHOENIX-4299) Local index - Data vs index row count does not match

[Mujtaba Chohan (JIRA)]

Mujtaba Chohan created PHOENIX-4299:
---

 Summary: Local index - Data vs index row count does not match
 Key: PHOENIX-4299
 URL: https://issues.apache.org/jira/browse/PHOENIX-4299
 Project: Phoenix
  Issue Type: Bug
Affects Versions: 4.12.0
 Environment: HBase 1.3.1, Phoenix 4.12.0
Reporter: Mujtaba Chohan


* In my test row count of local index is ~3K less than the total row count of 
approx 5M when comparing it using {{count( *)}} aggregation with {{NO_INDEX}} 
hint. 
* For data shape, at-least 50% of the upserts leads to multiple version with 
batch size of 250 using 4 writer threads. There are no deletes.
* This mutable table contains 18 columns with 5 of them being keys
* There are a total of 5 covered local indexes on the table
* Table get automatically split in 8 regions

I'll update here with more info. about any split/compaction activity that 
happened around the timestamp for the missing rows.



--
This message was sent by Atlassian JIRA
(v6.4.14#64029)

[jira] [Created] (PHOENIX-4298) refactoring to avoid using deprecated API for Put/Delete

[Sergey Soldatov (JIRA)]

Sergey Soldatov created PHOENIX-4298:


 Summary: refactoring to avoid using deprecated API for Put/Delete
 Key: PHOENIX-4298
 URL: https://issues.apache.org/jira/browse/PHOENIX-4298
 Project: Phoenix
  Issue Type: Sub-task
Reporter: Sergey Soldatov
Assignee: Sergey Soldatov






--
This message was sent by Atlassian JIRA
(v6.4.14#64029)

[jira] [Updated] (PHOENIX-4297) Remove deprecated API umbrella

[Sergey Soldatov (JIRA)]

 [ 
https://issues.apache.org/jira/browse/PHOENIX-4297?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Sergey Soldatov updated PHOENIX-4297:
-
Issue Type: Improvement  (was: Bug)

> Remove deprecated API umbrella
> --
>
> Key: PHOENIX-4297
> URL: https://issues.apache.org/jira/browse/PHOENIX-4297
> Project: Phoenix
>  Issue Type: Improvement
>Affects Versions: 4.12.0
>Reporter: Sergey Soldatov
> Fix For: 4.13.0
>
>
> For further support of HBase 2.0 we need to remove all deprecated API that 
> was used because of 0.98 support. This is umbrella ticket because some of the 
> changes may not  be trivial.



--
This message was sent by Atlassian JIRA
(v6.4.14#64029)

[jira] [Created] (PHOENIX-4297) Remove deprecated API umbrella

[Sergey Soldatov (JIRA)]

Sergey Soldatov created PHOENIX-4297:


 Summary: Remove deprecated API umbrella
 Key: PHOENIX-4297
 URL: https://issues.apache.org/jira/browse/PHOENIX-4297
 Project: Phoenix
  Issue Type: Bug
Affects Versions: 4.12.0
Reporter: Sergey Soldatov
 Fix For: 4.13.0


For further support of HBase 2.0 we need to remove all deprecated API that was 
used because of 0.98 support. This is umbrella ticket because some of the 
changes may not  be trivial.



--
This message was sent by Atlassian JIRA
(v6.4.14#64029)

Re: Date issue when copying data from Phoenix to Spark

[Sergey Soldatov]

Just a guess. Date is storying as the regular timestamp in GMT. So Phoenix
shows you the GMT time while Spark is convert GMT time to you local time
zone.

Thanks,
Sergey

On Tue, Oct 17, 2017 at 3:54 PM, Pavani Addanki 
wrote:

> I am trying to insert phoenix data into spark, found that the date is
> changing after inserting into spark. For ex: Date in Phoenix is showing as
> 2017-10-10 but when same is seen in spark it is showing as 2017-10-19.
>
> Did anyone else face the same issue.
>
>
> Thanks and Regards,
>
> Pavani A
>

[jira] [Commented] (PHOENIX-4280) Delete doesn't work when immutable indexes are in building state

[Hadoop QA (JIRA)]

[ 
https://issues.apache.org/jira/browse/PHOENIX-4280?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16209866#comment-16209866
 ] 

Hadoop QA commented on PHOENIX-4280:


{color:red}-1 overall{color}.  Here are the results of testing the latest 
attachment 
  http://issues.apache.org/jira/secure/attachment/12892821/PHOENIX-4280_v3.patch
  against master branch at commit c594038a1bee96b92edd9c788969e3e2dbd6db5a.
  ATTACHMENT ID: 12892821

{color:green}+1 @author{color}.  The patch does not contain any @author 
tags.

{color:red}-1 tests included{color}.  The patch doesn't appear to include 
any new or modified tests.
Please justify why no new tests are needed for this 
patch.
Also please list what manual steps were performed to 
verify this patch.

{color:green}+1 javac{color}.  The applied patch does not increase the 
total number of javac compiler warnings.

{color:green}+1 release audit{color}.  The applied patch does not increase 
the total number of release audit warnings.

{color:red}-1 lineLengths{color}.  The patch introduces the following lines 
longer than 100:
+rs = conn.createStatement().executeQuery("SELECT /*+ 
NO_INDEX*/ COUNT(*) FROM " + fullTableName);
+
TestUtil.dumpTable(conn.unwrap(PhoenixConnection.class).getQueryServices().getTable(Bytes.toBytes(fullTableName)));
+rs = conn.createStatement().executeQuery("SELECT /*+ NO_INDEX*/ 
COUNT(*) FROM " + fullTableName);
+IndexMaintainer maintainer = 
indexTableRefs.get(i).getTable().getIndexMaintainer(table, connection);
+indexPtr.set(maintainer.buildRowKey(null, indexPtr, 
null, null, HConstants.LATEST_TIMESTAMP));
+if (index.getIndexState() != PIndexState.DISABLE && 
index.getIndexType() == IndexType.GLOBAL) {
+CoveredDeleteScanner scanner = 
this.scannerBuilder.buildIndexedColumnScanner(indexedColumns, tracker, ts, 
returnNullScannerIfRowNotFound,
+// by definition, there won't be any. If we have indexed non row key 
columns, then we need to
+  public CoveredDeleteScanner buildIndexedColumnScanner(Collection indexedColumns, ColumnTracker tracker, long ts, boolean 
returnNullIfRowNotFound, boolean returnNullScanner) {
+Pair statePair = 
state.getIndexUpdateState(cols, metaData.getReplayWrite() != null, 
!metaData.isImmutableRows(), context);

 {color:red}-1 core tests{color}.  The patch failed these unit tests:
   
org.apache.phoenix.hbase.index.covered.LocalTableStateTest

Test results: 
https://builds.apache.org/job/PreCommit-PHOENIX-Build/1551//testReport/
Console output: 
https://builds.apache.org/job/PreCommit-PHOENIX-Build/1551//console

This message is automatically generated.

> Delete doesn't work when immutable indexes are in building state
> 
>
> Key: PHOENIX-4280
> URL: https://issues.apache.org/jira/browse/PHOENIX-4280
> Project: Phoenix
>  Issue Type: Bug
>Reporter: James Taylor
>Assignee: James Taylor
> Fix For: 4.12.1
>
> Attachments: PHOENIX-4280.patch, PHOENIX-4280_v2.patch, 
> PHOENIX-4280_v3.patch, PHOENIX-4280_wip.patch
>
>
> We don't generate the Delete mutations correctly in this case as we assume 
> we're scanning over the index table.



--
This message was sent by Atlassian JIRA
(v6.4.14#64029)

[jira] [Commented] (PHOENIX-4287) Incorrect aggregate query results when stats are disable for parallelization

[Samarth Jain (JIRA)]

[ 
https://issues.apache.org/jira/browse/PHOENIX-4287?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16209858#comment-16209858
 ] 

Samarth Jain commented on PHOENIX-4287:
---

Looks like it is limited to local indexes. Will keep looking.

> Incorrect aggregate query results when stats are disable for parallelization
> 
>
> Key: PHOENIX-4287
> URL: https://issues.apache.org/jira/browse/PHOENIX-4287
> Project: Phoenix
>  Issue Type: Bug
>Affects Versions: 4.12.0
> Environment: HBase 1.3.1
>Reporter: Mujtaba Chohan
>Assignee: Samarth Jain
> Fix For: 4.12.1
>
>
> With {{phoenix.use.stats.parallelization}} set to {{false}}, aggregate query 
> returns incorrect results when stats are available.
> With local index and stats disabled for parallelization:
> {noformat}
> explain select count(*) from TABLE_T;
> +---+-++---+
> | PLAN
>   | EST_BYTES_READ  | EST_ROWS_READ  |  EST_INFO |
> +---+-++---+
> | CLIENT 0-CHUNK 332170 ROWS 625043899 BYTES PARALLEL 0-WAY RANGE SCAN OVER 
> TABLE_T [1]  | 625043899   | 332170 | 150792825 |
> | SERVER FILTER BY FIRST KEY ONLY 
>   | 625043899   | 332170 | 150792825 |
> | SERVER AGGREGATE INTO SINGLE ROW
>   | 625043899   | 332170 | 150792825 |
> +---+-++---+
> select count(*) from TABLE_T;
> +---+
> | COUNT(1)  |
> +---+
> | 0 |
> +---+
> {noformat}
> Using data table
> {noformat}
> explain select /*+NO_INDEX*/ count(*) from TABLE_T;
> +--+-+++
> |   PLAN  
>  | EST_BYTES_READ  | EST_ROWS_READ  |  EST_INFO_TS   |
> +--+-+++
> | CLIENT 2-CHUNK 332151 ROWS 438492470 BYTES PARALLEL 1-WAY FULL SCAN OVER 
> TABLE_T  | 438492470   | 332151 | 1507928257617  |
> | SERVER FILTER BY FIRST KEY ONLY 
>  | 438492470   | 332151 | 1507928257617  |
> | SERVER AGGREGATE INTO SINGLE ROW
>  | 438492470   | 332151 | 1507928257617  |
> +--+-+++
> select /*+NO_INDEX*/ count(*) from TABLE_T;
> +---+
> | COUNT(1)  |
> +---+
> | 14|
> +---+
> {noformat}
> Without stats available, results are correct:
> {noformat}
> explain select /*+NO_INDEX*/ count(*) from TABLE_T;
> +--+-++--+
> | PLAN | 
> EST_BYTES_READ  | EST_ROWS_READ  | EST_INFO_TS  |
> +--+-++--+
> | CLIENT 2-CHUNK PARALLEL 1-WAY FULL SCAN OVER TABLE_T  | null| 
> null   | null |
> | SERVER FILTER BY FIRST KEY ONLY  | null 
>| null   | null |
> | SERVER AGGREGATE INTO SINGLE ROW | null 
>| null   | null |
> +--+-++--+
> select /*+NO_INDEX*/ count(*) from TABLE_T;
> +---+
> | COUNT(1)  |
> +---+
> | 27|
> +---+
> {noformat}



--
This message was sent by Atlassian JIRA
(v6.4.14#64029)

Re: [DISCUSS] Road to HBase 2.0.0

[Josh Elser]

On 10/18/17 2:51 PM, Andrew Purtell wrote:

Hopefully, Phoenix will not find itself in a position where

business-critical things it's doing are being removed from API (and Phoenix
has no recourse to hack what it needs back in place).

I feel fairly confident this is going to happen given the radical excision
of methods from the interfaces. What I hope is we can get a round of
evaluation by the Phoenix folks of the fallout in between our introduction
of these changes in an alpha before we cement the changes in a GA release.


/me nods

I know Ankit and Rajeshbabu have already started looking at this 
internally (I'll take the blame there -- I didn't expect 4.12 to go out 
the door so quickly, I had recommended they just do it locally for now).


I've asked them to pull the work they've done in porting out to Apache. 
I'm expecting that we'll see them push up a branch with what they have 
tomorrow :)


Getting us to a place where we have a branch of Phoenix with the 
necessary HBase 2.0 fixes in place is the path forward I see. I'll help 
out as much as I can here too.

[jira] [Commented] (PHOENIX-4294) Allow scalar function to declare that it's not thread safe

[Hudson (JIRA)]

[ 
https://issues.apache.org/jira/browse/PHOENIX-4294?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16209815#comment-16209815
 ] 

Hudson commented on PHOENIX-4294:
-

SUCCESS: Integrated in Jenkins build Phoenix-master #1842 (See 
[https://builds.apache.org/job/Phoenix-master/1842/])
PHOENIX-4294 Allow scalar function to declare that it's not thread safe 
(jtaylor: rev ccc44386e05fd7191cd498c8e35b4bb96df2eef7)
* (edit) 
phoenix-core/src/main/java/org/apache/phoenix/expression/function/ScalarFunction.java
* (edit) 
phoenix-core/src/main/java/org/apache/phoenix/expression/visitor/CloneExpressionVisitor.java


> Allow scalar function to declare that it's not thread safe
> --
>
> Key: PHOENIX-4294
> URL: https://issues.apache.org/jira/browse/PHOENIX-4294
> Project: Phoenix
>  Issue Type: Improvement
>Reporter: James Taylor
>Assignee: James Taylor
> Fix For: 4.13.0, 4.12.1
>
> Attachments: PHOENIX-4294.patch
>
>
> We have our CloneExpressionVisitor which determines when an expression 
> subtree needs to be cloned. For ease of use and clarity, we should have a new 
> method on ScalarFunction that may be overridden to determine when the 
> function is cloned versus not cloned.



--
This message was sent by Atlassian JIRA
(v6.4.14#64029)

[jira] [Commented] (PHOENIX-4295) Fix argument order for StatsCollectorIT derived classes

[Hudson (JIRA)]

[ 
https://issues.apache.org/jira/browse/PHOENIX-4295?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16209816#comment-16209816
 ] 

Hudson commented on PHOENIX-4295:
-

SUCCESS: Integrated in Jenkins build Phoenix-master #1842 (See 
[https://builds.apache.org/job/Phoenix-master/1842/])
PHOENIX-4295 Fix argument order for StatsCollectorIT derived classes (jtaylor: 
rev c594038a1bee96b92edd9c788969e3e2dbd6db5a)
* (edit) 
phoenix-core/src/it/java/org/apache/phoenix/end2end/ColumnEncodedImmutableTxStatsCollectorIT.java
* (edit) 
phoenix-core/src/it/java/org/apache/phoenix/end2end/NonColumnEncodedImmutableNonTxStatsCollectorIT.java
* (edit) 
phoenix-core/src/it/java/org/apache/phoenix/end2end/ColumnEncodedMutableTxStatsCollectorIT.java
* (edit) 
phoenix-core/src/it/java/org/apache/phoenix/end2end/ColumnEncodedMutableNonTxStatsCollectorIT.java
* (edit) 
phoenix-core/src/it/java/org/apache/phoenix/end2end/NonColumnEncodedImmutableTxStatsCollectorIT.java
* (edit) 
phoenix-core/src/it/java/org/apache/phoenix/end2end/ColumnEncodedImmutableNonTxStatsCollectorIT.java
* (edit) 
phoenix-core/src/it/java/org/apache/phoenix/end2end/SysTableNamespaceMappedStatsCollectorIT.java


> Fix argument order for StatsCollectorIT derived classes
> ---
>
> Key: PHOENIX-4295
> URL: https://issues.apache.org/jira/browse/PHOENIX-4295
> Project: Phoenix
>  Issue Type: Test
>Reporter: James Taylor
>Assignee: James Taylor
> Fix For: 4.13.0, 4.12.1
>
> Attachments: PHOENIX-4295.patch
>
>




--
This message was sent by Atlassian JIRA
(v6.4.14#64029)

Re: [DISCUSS] Road to HBase 2.0.0

[Andrew Purtell]

> Hopefully, Phoenix will not find itself in a position where
business-critical things it's doing are being removed from API (and Phoenix
has no recourse to hack what it needs back in place).

I feel fairly confident this is going to happen given the radical excision
of methods from the interfaces. What I hope is we can get a round of
evaluation by the Phoenix folks of the fallout in between our introduction
of these changes in an alpha before we cement the changes in a GA release.


On Wed, Oct 18, 2017 at 11:50 AM, Josh Elser  wrote:

> Yes, the earlier the better, but this is also a bit chicken-and-egg. It's
> tough down in Phoenix to make sure it still works if the API we're trying
> to write against is still changing.
>
> I know that Anoop from the HBase side has been peering into the Phoenix
> side -- I've been operating under the assumption that things will be OK.
> Hopefully, Phoenix will not find itself in a position where
> business-critical things it's doing are being removed from API (and Phoenix
> has no recourse to hack what it needs back in place).
>
> With an HBase hat on, I think Phoenix is a good candidate (among others
> like YARN) for validating the alpha-4 release, signaling whether we're
> actually ready to say HBase 2 is ready for a beta-1 (as opposed to an
> alpha-5 to fix some more).
>
> Happy to entertain a broader discussion if you think there's merit on
> dev@hbase too, S.
>
>
> On 10/17/17 5:35 PM, Stack wrote:
>
>> Some notes on this effort:
>>
>> + Perhaps this discussion should be cc'd to dev@hbase? There is a good
>> bit
>> of overlap between dev-set here and dev-set on hbase and some
>> interest/concern around Phoenix on HBase2.
>> + Here are some notes I took after looking at Phoenix last week that I ran
>> by James off-list (I filled in some of his feedback) [1]. The notes try to
>> list what Phoenix uses from HBase core. I was looking to Coprocessor usage
>> mostly. I noticed that CP-usage is mostly inside the bounds of appropriate
>> use. It is elsewhere that P is in violation.
>> + After my splunking and going by feedback from James, do we have to carry
>> all of Phoenix over to hbase2? Can we drop some stuff that is not used
>> anymore or can we work out an ordained API P could use instead of do its
>> own workaround?
>>
>> It is late in the day to be having this conversation -- we are about to
>> release alpha4 which is supposed to be locked-down internal and external
>> API -- but better late than never.
>>
>> Thanks,
>> St.Ack
>> 1.  https://docs.google.com/document/d/10cabwp_
>> aR3OmpHVoeh544YLC3KwqMD9KiTIrHZAmfec/edit#heading=h.7swwa1jl6wiw
>>
>> On Wed, Oct 11, 2017 at 3:00 PM, Josh Elser  wrote:
>>
>> Since 4.12.0 is out and we have the concurrent discussions about the 0.98,
>>> 1.1, and 1.2 HBase branches, do folks have a vision of how we get to
>>> HBase
>>> 2.0.0?
>>>
>>> The lack of chatter is pretty obvious that the Calcite work (the previous
>>> impetus for Phoenix 5) has slowed. Once we get to an HBase 2.0.0-alpha4,
>>> coprocessor API should stabilize and give us a point against which we can
>>> start Phoenix work.
>>>
>>> Should a release of Phoenix that supports HBase 2.0 be worthy of the
>>> Phoenix 5.0 label, or should we stick to the 4.x numbering? Given the
>>> breaking changes going into HBase 2.0 and James' previous -1 to
>>> shim-layers
>>> for 0.98, do we see the same for an HBase 2.0 branch or is HBase 1.x/2.x
>>> a
>>> different beast? I can see pros/cons for both sides.
>>>
>>> - Josh
>>>
>>>
>>


-- 
Best regards,
Andrew

Words like orphans lost among the crosstalk, meaning torn from truth's
decrepit hands
   - A23, Crosstalk

Re: [DISCUSS] Road to HBase 2.0.0

[Josh Elser]

Yes, the earlier the better, but this is also a bit chicken-and-egg. 
It's tough down in Phoenix to make sure it still works if the API we're 
trying to write against is still changing.


I know that Anoop from the HBase side has been peering into the Phoenix 
side -- I've been operating under the assumption that things will be OK. 
Hopefully, Phoenix will not find itself in a position where 
business-critical things it's doing are being removed from API (and 
Phoenix has no recourse to hack what it needs back in place).


With an HBase hat on, I think Phoenix is a good candidate (among others 
like YARN) for validating the alpha-4 release, signaling whether we're 
actually ready to say HBase 2 is ready for a beta-1 (as opposed to an 
alpha-5 to fix some more).


Happy to entertain a broader discussion if you think there's merit on 
dev@hbase too, S.


On 10/17/17 5:35 PM, Stack wrote:

Some notes on this effort:

+ Perhaps this discussion should be cc'd to dev@hbase? There is a good bit
of overlap between dev-set here and dev-set on hbase and some
interest/concern around Phoenix on HBase2.
+ Here are some notes I took after looking at Phoenix last week that I ran
by James off-list (I filled in some of his feedback) [1]. The notes try to
list what Phoenix uses from HBase core. I was looking to Coprocessor usage
mostly. I noticed that CP-usage is mostly inside the bounds of appropriate
use. It is elsewhere that P is in violation.
+ After my splunking and going by feedback from James, do we have to carry
all of Phoenix over to hbase2? Can we drop some stuff that is not used
anymore or can we work out an ordained API P could use instead of do its
own workaround?

It is late in the day to be having this conversation -- we are about to
release alpha4 which is supposed to be locked-down internal and external
API -- but better late than never.

Thanks,
St.Ack
1.  https://docs.google.com/document/d/10cabwp_
aR3OmpHVoeh544YLC3KwqMD9KiTIrHZAmfec/edit#heading=h.7swwa1jl6wiw

On Wed, Oct 11, 2017 at 3:00 PM, Josh Elser  wrote:


Since 4.12.0 is out and we have the concurrent discussions about the 0.98,
1.1, and 1.2 HBase branches, do folks have a vision of how we get to HBase
2.0.0?

The lack of chatter is pretty obvious that the Calcite work (the previous
impetus for Phoenix 5) has slowed. Once we get to an HBase 2.0.0-alpha4,
coprocessor API should stabilize and give us a point against which we can
start Phoenix work.

Should a release of Phoenix that supports HBase 2.0 be worthy of the
Phoenix 5.0 label, or should we stick to the 4.x numbering? Given the
breaking changes going into HBase 2.0 and James' previous -1 to shim-layers
for 0.98, do we see the same for an HBase 2.0 branch or is HBase 1.x/2.x a
different beast? I can see pros/cons for both sides.

- Josh

[jira] [Updated] (PHOENIX-4280) Delete doesn't work when immutable indexes are in building state

[James Taylor (JIRA)]

 [ 
https://issues.apache.org/jira/browse/PHOENIX-4280?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

James Taylor updated PHOENIX-4280:
--
Attachment: PHOENIX-4280_v3.patch

Still need to clean this up a bit, but there are a number of issues with 
deletes over local indexes. The client should not be sending index mutations 
for local indexes. Also, we should not be executing a scan to lookup the prior 
row unless it's a delete 

> Delete doesn't work when immutable indexes are in building state
> 
>
> Key: PHOENIX-4280
> URL: https://issues.apache.org/jira/browse/PHOENIX-4280
> Project: Phoenix
>  Issue Type: Bug
>Reporter: James Taylor
>Assignee: James Taylor
> Fix For: 4.12.1
>
> Attachments: PHOENIX-4280.patch, PHOENIX-4280_v2.patch, 
> PHOENIX-4280_v3.patch, PHOENIX-4280_wip.patch
>
>
> We don't generate the Delete mutations correctly in this case as we assume 
> we're scanning over the index table.



--
This message was sent by Atlassian JIRA
(v6.4.14#64029)

[jira] [Commented] (PHOENIX-4292) SOQL Filters on Tables and Views with composite PK of VARCHAR fields with sort direction DESC do not work

[Thomas D'Silva (JIRA)]

[ 
https://issues.apache.org/jira/browse/PHOENIX-4292?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16209696#comment-16209696
 ] 

Thomas D'Silva commented on PHOENIX-4292:
-

Sure.

> SOQL Filters on Tables and Views with composite PK of VARCHAR fields with 
> sort direction DESC do not work
> -
>
> Key: PHOENIX-4292
> URL: https://issues.apache.org/jira/browse/PHOENIX-4292
> Project: Phoenix
>  Issue Type: Bug
>Affects Versions: 4.10.0
>Reporter: Jan Fernando
>Assignee: Thomas D'Silva
> Fix For: 4.12.1
>
>
> We noticed that in certain instances on tables and views that were defined 
> with a Composite PK and where the elements of the PK were all DESC that 
> queries exhibited strange behavior and did not return results when expected. 
> A simple query on the first element of the PK returned 0 results e.g SELECT * 
> FROM MY_TABLE WHERE PK1 = 'myvaluethatexists' would return 0 results.
> After some investigation it appears that querying tables and views with a 
> Composite PK that :
> a) have multiple VARCHAR columns in the PK
> b) the sort direction of all the VARCHAR columns is defined as DESC 
>  does not work correctly and the filters are not honored and SQL appears 
> broken to the end user.
> Detailed repro steps:
> ---
> -- 1. Create Global Base Table
> CREATE TABLE IF NOT EXISTS TEST.BASETABLE (
> TENANT_ID CHAR(15) NOT NULL, 
> KEY_PREFIX CHAR(3) NOT NULL, 
> CREATED_DATE DATE,
> CREATED_BY CHAR(15),
> SYSTEM_MODSTAMP DATE
> CONSTRAINT PK PRIMARY KEY (
> TENANT_ID, 
> KEY_PREFIX 
> )
> ) VERSIONS=1, MULTI_TENANT=true, IMMUTABLE_ROWS=TRUE, REPLICATION_SCOPE=1
> -- 2. Create various TENANT SPECIFIC VIEWS i.e. use a tenant specific 
> connection
> CREATE VIEW IF NOT EXISTS TEST."abc" (pk1 VARCHAR(10) NOT NULL, pk2 
> VARCHAR(10) NOT NULL, col1 DATE,  col3 DECIMAL CONSTRAINT PK PRIMARY KEY (pk1 
> DESC, pk2 DESC)) AS SELECT * FROM TEST.BASETABLE WHERE KEY_PREFIX = 'abc';
> CREATE VIEW IF NOT EXISTS TEST."ab2" (pk1 VARCHAR(10) NOT NULL, pk2 
> VARCHAR(10) NOT NULL, col1 DATE,  col3 DECIMAL CONSTRAINT PK PRIMARY KEY (pk1 
> DESC, pk2 ASC)) AS SELECT * FROM TEST.BASETABLE WHERE KEY_PREFIX = 'ab2';
> CREATE VIEW IF NOT EXISTS TEST."ab3" (pk1 DATE(10) NOT NULL, pk2 DATE(10) NOT 
> NULL, col1 VARCHAR(10),  col3 DECIMAL CONSTRAINT PK PRIMARY KEY (pk1 DESC, 
> pk2 DESC)) AS SELECT * FROM TEST.BASETABLE WHERE KEY_PREFIX = 'ab3';
> CREATE VIEW IF NOT EXISTS TEST."ab4" (pk1 DATE(10) NOT NULL, pk2 DECIMAL NOT 
> NULL, pk3 VARCHAR(10) NOT NULL,  col3 DECIMAL CONSTRAINT PK PRIMARY KEY (pk1 
> DESC, pk2 DESC, pk3 DESC)) AS SELECT * FROM TEST.BASETABLE WHERE KEY_PREFIX = 
> 'ab4';
> -- 3. Test cases that exhibit this issues
> -- SIMULATE EQUALITY QUERY PROBLEMS: View with composite PK with multiple PK 
> values of VARCHAR values DESC
> upsert into TEST."abc" (pk1, pk2, col1, col3) VALUES ('testa', 'testb', 
> TO_DATE('2017-10-16 22:00:00', '-MM-dd HH:mm:ss'), 10); 
> upsert into TEST."abc" (pk1, pk2, col1, col3) VALUES ('testa', 'testc', 
> TO_DATE('2017-10-16 22:00:00', '-MM-dd HH:mm:ss'), 10); 
> upsert into TEST."abc" (pk1, pk2, col1, col3) VALUES ('testa', 'testd', 
> TO_DATE('2017-10-16 22:00:00', '-MM-dd HH:mm:ss'), 10); 
> upsert into TEST."abc" (pk1, pk2, col1, col3) VALUES ('testa', 'teste', 
> TO_DATE('2017-10-16 22:00:00', '-MM-dd HH:mm:ss'), 10); 
> upsert into TEST."abc" (pk1, pk2, col1, col3) VALUES ('testb', 'testa', 
> TO_DATE('2017-10-16 22:00:00', '-MM-dd HH:mm:ss'), 10); 
> SELECT * FROM TEST."abc" WHERE pk1 = 'testa'; -- INCORRECT RESULT: This query 
> returns no records, expected to return 4
> SELECT * FROM TEST."abc"; -- Returns 5 rows as expected
> SELECT * FROM TEST."abc" WHERE pk1 >= 'testa'; -- INCORRECT RESULT: This 
> query returns 1 record, expected to return 5
> SELECT * FROM TEST."abc" WHERE pk1 <= 'testa';  -- Returns 4 rows as expected
> SELECT * FROM TEST."abc" WHERE pk1 > 'testa'; -- Returns 1 row as expected
> SELECT * FROM TEST."abc" WHERE pk1 < 'testa'; -- INCORRECT RESULT: This query 
> returns 1 record, expected to return 0
> -- The following are cases where everything works as expected and which don't 
> have composite VARCHAR PKs
> -- DEMONOSTRATE NO QUERY PROBLEMS WHEN HAVE VIEW WITH SINGLE DESC TEXT PK: 
> View with composite PK with single pk value DESC
> upsert into TEST."ab2" (pk1, pk2, col1, col3) VALUES ('testa', 'testb', 
> TO_DATE('2017-10-16 22:00:00', '-MM-dd HH:mm:ss'), 10); 
> upsert into TEST."ab2" (pk1, pk2, col1, col3) VALUES ('testa', 'testc', 
> TO_DATE('2017-10-16 22:00:00', '-MM-dd HH:mm:ss'), 10); 
> upsert into TEST."ab2" (pk1, pk2, col1, col3) VALUES ('testa', 'testd', 
> TO_DATE('2017-10-16 

[jira] [Commented] (PHOENIX-4237) Allow sorting on (Java) collation keys for non-English locales

[ASF GitHub Bot (JIRA)]

[ 
https://issues.apache.org/jira/browse/PHOENIX-4237?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16209676#comment-16209676
 ] 

ASF GitHub Bot commented on PHOENIX-4237:
-

Github user JamesRTaylor commented on the issue:

https://github.com/apache/phoenix/pull/275
  
Looking very good. Couple minor nits and the testing needs to be rounded 
out just a bit.


> Allow sorting on (Java) collation keys for non-English locales
> --
>
> Key: PHOENIX-4237
> URL: https://issues.apache.org/jira/browse/PHOENIX-4237
> Project: Phoenix
>  Issue Type: Improvement
>Reporter: Shehzaad Nakhoda
> Fix For: 4.12.0
>
>
> Strings stored via Phoenix can be composed from a subset of the entire set of 
> Unicode characters. The natural sort order for strings for different 
> languages often differs from the order dictated by the binary representation 
> of the characters of these strings. Java provides the idea of a Collator 
> which given an input string and a (language) locale can generate a Collation 
> Key which can then be used to compare strings in that natural order.
> Salesforce has recently open-sourced grammaticus. IBM has open-sourced ICU4J 
> some time ago. These technologies can be combined to provide a robust new 
> Phoenix function that can be used in an ORDER BY clause to sort strings 
> according to the user's locale.



--
This message was sent by Atlassian JIRA
(v6.4.14#64029)

[GitHub] phoenix issue #275: PHOENIX-4237: Add function to calculate Java collation k...

[JamesRTaylor]

Github user JamesRTaylor commented on the issue:

https://github.com/apache/phoenix/pull/275
  
Looking very good. Couple minor nits and the testing needs to be rounded 
out just a bit.


---

[jira] [Commented] (PHOENIX-4237) Allow sorting on (Java) collation keys for non-English locales

[ASF GitHub Bot (JIRA)]

[ 
https://issues.apache.org/jira/browse/PHOENIX-4237?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16209673#comment-16209673
 ] 

ASF GitHub Bot commented on PHOENIX-4237:
-

Github user JamesRTaylor commented on a diff in the pull request:

https://github.com/apache/phoenix/pull/275#discussion_r145475423
  
--- Diff: 
phoenix-core/src/test/java/org/apache/phoenix/expression/function/CollationKeyFunctionTest.java
 ---
@@ -0,0 +1,143 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements.  See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership.  The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License.  You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+package org.apache.phoenix.expression.function;
+
+import static org.junit.Assert.assertEquals;
+import static org.junit.Assert.assertArrayEquals;
+import static org.junit.Assert.fail;
+
+import java.util.List;
+
+import org.apache.commons.codec.binary.Hex;
+import org.apache.hadoop.hbase.io.ImmutableBytesWritable;
+import org.apache.phoenix.expression.function.CollationKeyFunction;
+import org.apache.phoenix.schema.SortOrder;
+import org.apache.phoenix.schema.types.PBoolean;
+import org.apache.phoenix.schema.types.PInteger;
+import org.apache.phoenix.schema.types.PVarchar;
+
+import org.apache.phoenix.expression.Expression;
+import org.apache.phoenix.expression.LiteralExpression;
+
+import org.junit.Test;
+
+import com.google.common.collect.Lists;
+
+/**
+ * "Unit" tests for CollationKeyFunction
+ * 
+ * @author snakhoda-sfdc
+ *
+ */
+public class CollationKeyFunctionTest {
+
+   @Test
+   public void testChineseCollationKeyBytes() throws Exception {
+   
+   // Chinese (China)
+   test("\u963f", "zh", "02eb0001");
+   test("\u55c4", "zh", "14ad0001");
+   test("\u963e", "zh", "8000963f00010001");
+   test("\u554a", "zh", "02ea0001");
+   test("\u4ec8", "zh", "80004ec900010001");
+   test("\u3d9a", "zh", "80003d9b00010001");
+   test("\u9f51", "zh", "19050001");
+   
+   // Chinese (Taiwan)
+   test("\u963f", "zh_TW", "063d0001");
+   test("\u55c4", "zh_TW", "241e0001");
+   test("\u963e", "zh_TW", "8000963f00010001");
+   test("\u554a", "zh_TW", "09c90001");
+   test("\u4ec8", "zh_TW", "181b0001");
+   test("\u3d9a", "zh_TW", "80003d9b00010001");
+   test("\u9f51", "zh_TW", "80009f5200010001");
+   
+   // Chinese (Taiwan, Stroke)
+   test("\u963f", "zh_TW_STROKE", "5450010500");
+   test("\u55c4", "zh_TW_STROKE", "7334010500");
+   test("\u963e", "zh_TW_STROKE", "544f010500");
+   test("\u554a", "zh_TW_STROKE", "62de010500");
+   test("\u4ec8", "zh_TW_STROKE", "46be010500");
+   test("\u3d9a", "zh_TW_STROKE", "a50392010500");
+   test("\u9f51", "zh_TW_STROKE", "8915010500");
+   
+   // Chinese (China, Stroke)
+   test("\u963f", "zh__STROKE", "28010500");
+   test("\u55c4", "zh__STROKE", "2a010500");
+   test("\u963e", "zh__STROKE", "7575010500");
+   test("\u554a", "zh__STROKE", "2b010500");
+   test("\u4ec8", "zh__STROKE", "51a1010500");
+   test("\u3d9a", "zh__STROKE", "a50392010500");
+   test("\u9f51", "zh__STROKE", "6935010500");
+   
+   // Chinese (China, Pinyin)
+   test("\u963f", "zh__PINYIN", "28010500");
+   test("\u55c4", "zh__PINYIN", "2a010500");
+   test("\u963e", "zh__PINYIN", "7575010500");
+   test("\u554a", "zh__PINYIN", "2b010500");
+   test("\u4ec8", "zh__PINYIN", "51a1010500");
+   test("\u3d9a", "zh__PINYIN", "a50392010500");
+   test("\u9f51", "zh__PINYIN", "6935010500");
+   
+   }
+
+   private 

[GitHub] phoenix pull request #275: PHOENIX-4237: Add function to calculate Java coll...

[JamesRTaylor]

Github user JamesRTaylor commented on a diff in the pull request:

https://github.com/apache/phoenix/pull/275#discussion_r145475530
  
--- Diff: 
phoenix-core/src/test/java/org/apache/phoenix/expression/function/CollationKeyFunctionTest.java
 ---
@@ -0,0 +1,143 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements.  See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership.  The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License.  You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+package org.apache.phoenix.expression.function;
+
+import static org.junit.Assert.assertEquals;
+import static org.junit.Assert.assertArrayEquals;
+import static org.junit.Assert.fail;
+
+import java.util.List;
+
+import org.apache.commons.codec.binary.Hex;
+import org.apache.hadoop.hbase.io.ImmutableBytesWritable;
+import org.apache.phoenix.expression.function.CollationKeyFunction;
+import org.apache.phoenix.schema.SortOrder;
+import org.apache.phoenix.schema.types.PBoolean;
+import org.apache.phoenix.schema.types.PInteger;
+import org.apache.phoenix.schema.types.PVarchar;
+
+import org.apache.phoenix.expression.Expression;
+import org.apache.phoenix.expression.LiteralExpression;
+
+import org.junit.Test;
+
+import com.google.common.collect.Lists;
+
+/**
+ * "Unit" tests for CollationKeyFunction
+ * 
+ * @author snakhoda-sfdc
+ *
+ */
+public class CollationKeyFunctionTest {
+
+   @Test
+   public void testChineseCollationKeyBytes() throws Exception {
+   
+   // Chinese (China)
+   test("\u963f", "zh", "02eb0001");
+   test("\u55c4", "zh", "14ad0001");
+   test("\u963e", "zh", "8000963f00010001");
+   test("\u554a", "zh", "02ea0001");
+   test("\u4ec8", "zh", "80004ec900010001");
+   test("\u3d9a", "zh", "80003d9b00010001");
+   test("\u9f51", "zh", "19050001");
+   
+   // Chinese (Taiwan)
+   test("\u963f", "zh_TW", "063d0001");
+   test("\u55c4", "zh_TW", "241e0001");
+   test("\u963e", "zh_TW", "8000963f00010001");
+   test("\u554a", "zh_TW", "09c90001");
+   test("\u4ec8", "zh_TW", "181b0001");
+   test("\u3d9a", "zh_TW", "80003d9b00010001");
+   test("\u9f51", "zh_TW", "80009f5200010001");
+   
+   // Chinese (Taiwan, Stroke)
+   test("\u963f", "zh_TW_STROKE", "5450010500");
+   test("\u55c4", "zh_TW_STROKE", "7334010500");
+   test("\u963e", "zh_TW_STROKE", "544f010500");
+   test("\u554a", "zh_TW_STROKE", "62de010500");
+   test("\u4ec8", "zh_TW_STROKE", "46be010500");
+   test("\u3d9a", "zh_TW_STROKE", "a50392010500");
+   test("\u9f51", "zh_TW_STROKE", "8915010500");
+   
+   // Chinese (China, Stroke)
+   test("\u963f", "zh__STROKE", "28010500");
+   test("\u55c4", "zh__STROKE", "2a010500");
+   test("\u963e", "zh__STROKE", "7575010500");
+   test("\u554a", "zh__STROKE", "2b010500");
+   test("\u4ec8", "zh__STROKE", "51a1010500");
+   test("\u3d9a", "zh__STROKE", "a50392010500");
+   test("\u9f51", "zh__STROKE", "6935010500");
+   
+   // Chinese (China, Pinyin)
+   test("\u963f", "zh__PINYIN", "28010500");
+   test("\u55c4", "zh__PINYIN", "2a010500");
+   test("\u963e", "zh__PINYIN", "7575010500");
+   test("\u554a", "zh__PINYIN", "2b010500");
+   test("\u4ec8", "zh__PINYIN", "51a1010500");
+   test("\u3d9a", "zh__PINYIN", "a50392010500");
+   test("\u9f51", "zh__PINYIN", "6935010500");
+   
+   }
+
+   private static void test(String inputStr, String localeIsoCode, String 
expectedCollationKeyBytesHex)
+   throws Exception {
+   boolean ret1 = testExpression(inputStr, localeIsoCode, 
SortOrder.ASC, 

[GitHub] phoenix pull request #275: PHOENIX-4237: Add function to calculate Java coll...

[JamesRTaylor]

Github user JamesRTaylor commented on a diff in the pull request:

https://github.com/apache/phoenix/pull/275#discussion_r145475423
  
--- Diff: 
phoenix-core/src/test/java/org/apache/phoenix/expression/function/CollationKeyFunctionTest.java
 ---
@@ -0,0 +1,143 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements.  See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership.  The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License.  You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+package org.apache.phoenix.expression.function;
+
+import static org.junit.Assert.assertEquals;
+import static org.junit.Assert.assertArrayEquals;
+import static org.junit.Assert.fail;
+
+import java.util.List;
+
+import org.apache.commons.codec.binary.Hex;
+import org.apache.hadoop.hbase.io.ImmutableBytesWritable;
+import org.apache.phoenix.expression.function.CollationKeyFunction;
+import org.apache.phoenix.schema.SortOrder;
+import org.apache.phoenix.schema.types.PBoolean;
+import org.apache.phoenix.schema.types.PInteger;
+import org.apache.phoenix.schema.types.PVarchar;
+
+import org.apache.phoenix.expression.Expression;
+import org.apache.phoenix.expression.LiteralExpression;
+
+import org.junit.Test;
+
+import com.google.common.collect.Lists;
+
+/**
+ * "Unit" tests for CollationKeyFunction
+ * 
+ * @author snakhoda-sfdc
+ *
+ */
+public class CollationKeyFunctionTest {
+
+   @Test
+   public void testChineseCollationKeyBytes() throws Exception {
+   
+   // Chinese (China)
+   test("\u963f", "zh", "02eb0001");
+   test("\u55c4", "zh", "14ad0001");
+   test("\u963e", "zh", "8000963f00010001");
+   test("\u554a", "zh", "02ea0001");
+   test("\u4ec8", "zh", "80004ec900010001");
+   test("\u3d9a", "zh", "80003d9b00010001");
+   test("\u9f51", "zh", "19050001");
+   
+   // Chinese (Taiwan)
+   test("\u963f", "zh_TW", "063d0001");
+   test("\u55c4", "zh_TW", "241e0001");
+   test("\u963e", "zh_TW", "8000963f00010001");
+   test("\u554a", "zh_TW", "09c90001");
+   test("\u4ec8", "zh_TW", "181b0001");
+   test("\u3d9a", "zh_TW", "80003d9b00010001");
+   test("\u9f51", "zh_TW", "80009f5200010001");
+   
+   // Chinese (Taiwan, Stroke)
+   test("\u963f", "zh_TW_STROKE", "5450010500");
+   test("\u55c4", "zh_TW_STROKE", "7334010500");
+   test("\u963e", "zh_TW_STROKE", "544f010500");
+   test("\u554a", "zh_TW_STROKE", "62de010500");
+   test("\u4ec8", "zh_TW_STROKE", "46be010500");
+   test("\u3d9a", "zh_TW_STROKE", "a50392010500");
+   test("\u9f51", "zh_TW_STROKE", "8915010500");
+   
+   // Chinese (China, Stroke)
+   test("\u963f", "zh__STROKE", "28010500");
+   test("\u55c4", "zh__STROKE", "2a010500");
+   test("\u963e", "zh__STROKE", "7575010500");
+   test("\u554a", "zh__STROKE", "2b010500");
+   test("\u4ec8", "zh__STROKE", "51a1010500");
+   test("\u3d9a", "zh__STROKE", "a50392010500");
+   test("\u9f51", "zh__STROKE", "6935010500");
+   
+   // Chinese (China, Pinyin)
+   test("\u963f", "zh__PINYIN", "28010500");
+   test("\u55c4", "zh__PINYIN", "2a010500");
+   test("\u963e", "zh__PINYIN", "7575010500");
+   test("\u554a", "zh__PINYIN", "2b010500");
+   test("\u4ec8", "zh__PINYIN", "51a1010500");
+   test("\u3d9a", "zh__PINYIN", "a50392010500");
+   test("\u9f51", "zh__PINYIN", "6935010500");
+   
+   }
+
+   private static void test(String inputStr, String localeIsoCode, String 
expectedCollationKeyBytesHex)
+   throws Exception {
+   boolean ret1 = testExpression(inputStr, localeIsoCode, 
SortOrder.ASC, 

[jira] [Commented] (PHOENIX-4237) Allow sorting on (Java) collation keys for non-English locales

[ASF GitHub Bot (JIRA)]

[ 
https://issues.apache.org/jira/browse/PHOENIX-4237?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16209670#comment-16209670
 ] 

ASF GitHub Bot commented on PHOENIX-4237:
-

Github user JamesRTaylor commented on a diff in the pull request:

https://github.com/apache/phoenix/pull/275#discussion_r145474350
  
--- Diff: 
phoenix-core/src/main/java/org/apache/phoenix/expression/function/CollationKeyFunction.java
 ---
@@ -0,0 +1,230 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements.  See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership.  The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License.  You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+package org.apache.phoenix.expression.function;
+
+import java.io.DataInput;
+import java.io.IOException;
+import java.sql.SQLException;
+import java.text.Collator;
+import java.util.List;
+import java.util.Locale;
+
+import org.apache.commons.lang.BooleanUtils;
+import org.apache.commons.logging.Log;
+import org.apache.commons.logging.LogFactory;
+import org.apache.hadoop.hbase.io.ImmutableBytesWritable;
+import org.apache.phoenix.expression.Expression;
+import org.apache.phoenix.expression.LiteralExpression;
+import org.apache.phoenix.parse.FunctionParseNode;
+import org.apache.phoenix.schema.tuple.Tuple;
+import org.apache.phoenix.schema.types.PBoolean;
+import org.apache.phoenix.schema.types.PDataType;
+import org.apache.phoenix.schema.types.PInteger;
+import org.apache.phoenix.schema.types.PVarbinary;
+import org.apache.phoenix.schema.types.PVarchar;
+import org.apache.phoenix.util.VarBinaryFormatter;
+
+import com.force.db.i18n.LinguisticSort;
+import com.force.i18n.LocaleUtils;
+
+/**
+ * A Phoenix Function that calculates a collation key for an input
+ * string based on a caller-provided locale and collator strength and
+ * decomposition settings.
+ * 
+ * The locale should be specified as xx_yy_variant where xx is the ISO
+ * 639-1 2-letter language code, yy is the the ISO 3166 2-letter
+ * country code. Both countryCode and variant are optional. For
+ * example, zh_TW_STROKE, zh_TW and zh are all valid locale
+ * representations. Note the language code, country code and variant
+ * are used as arguments to the constructor of java.util.Locale.
+ *
+ * This function uses the open-source grammaticus and i18n-util
+ * packages to obtain the collators it needs from the provided locale.
+ *
+ * The LinguisticSort implementation in i18n-util encapsulates
+ * sort-related functionality for a substantive list of locales. For
+ * each locale, it provides a collator and an Oracle-specific database
+ * function that can be used to sort strings according to the natural
+ * language rules of that locale.
+ *
+ * This function uses the collator returned by
+ * LinguisticSort.getCollator to produce a collation key for its input
+ * string. A user can expect that the sorting semantics of this
+ * function for a given locale is equivalent to the sorting behaviour
+ * of an Oracle query that is constructed using the Oracle functions
+ * returned by LinguisticSort for that locale.
+ *
+ * The optional third argument to the function is a boolean that
+ * specifies whether to use the upper-case collator (case-insensitive)
+ * returned by LinguisticSort.getUpperCaseCollator.
+ *
+ * The optional fourth and fifth arguments are used to set
+ * respectively the strength and composition of the collator returned
+ * by LinguisticSort using the setStrength and setDecomposition
+ * methods of java.text.Collator.
+ * 
+ * @author snakhoda-sfdc
+ *
+ */
+@FunctionParseNode.BuiltInFunction(name = CollationKeyFunction.NAME, args 
= {
+   // input string
+   @FunctionParseNode.Argument(allowedTypes = { PVarchar.class }),
+   // ISO Code for Locale
+   @FunctionParseNode.Argument(allowedTypes = { PVarchar.class }, 
isConstant = true),
+   // whether to use special upper case collator
+   

[GitHub] phoenix pull request #275: PHOENIX-4237: Add function to calculate Java coll...

[JamesRTaylor]

Github user JamesRTaylor commented on a diff in the pull request:

https://github.com/apache/phoenix/pull/275#discussion_r145474350
  
--- Diff: 
phoenix-core/src/main/java/org/apache/phoenix/expression/function/CollationKeyFunction.java
 ---
@@ -0,0 +1,230 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements.  See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership.  The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License.  You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+package org.apache.phoenix.expression.function;
+
+import java.io.DataInput;
+import java.io.IOException;
+import java.sql.SQLException;
+import java.text.Collator;
+import java.util.List;
+import java.util.Locale;
+
+import org.apache.commons.lang.BooleanUtils;
+import org.apache.commons.logging.Log;
+import org.apache.commons.logging.LogFactory;
+import org.apache.hadoop.hbase.io.ImmutableBytesWritable;
+import org.apache.phoenix.expression.Expression;
+import org.apache.phoenix.expression.LiteralExpression;
+import org.apache.phoenix.parse.FunctionParseNode;
+import org.apache.phoenix.schema.tuple.Tuple;
+import org.apache.phoenix.schema.types.PBoolean;
+import org.apache.phoenix.schema.types.PDataType;
+import org.apache.phoenix.schema.types.PInteger;
+import org.apache.phoenix.schema.types.PVarbinary;
+import org.apache.phoenix.schema.types.PVarchar;
+import org.apache.phoenix.util.VarBinaryFormatter;
+
+import com.force.db.i18n.LinguisticSort;
+import com.force.i18n.LocaleUtils;
+
+/**
+ * A Phoenix Function that calculates a collation key for an input
+ * string based on a caller-provided locale and collator strength and
+ * decomposition settings.
+ * 
+ * The locale should be specified as xx_yy_variant where xx is the ISO
+ * 639-1 2-letter language code, yy is the the ISO 3166 2-letter
+ * country code. Both countryCode and variant are optional. For
+ * example, zh_TW_STROKE, zh_TW and zh are all valid locale
+ * representations. Note the language code, country code and variant
+ * are used as arguments to the constructor of java.util.Locale.
+ *
+ * This function uses the open-source grammaticus and i18n-util
+ * packages to obtain the collators it needs from the provided locale.
+ *
+ * The LinguisticSort implementation in i18n-util encapsulates
+ * sort-related functionality for a substantive list of locales. For
+ * each locale, it provides a collator and an Oracle-specific database
+ * function that can be used to sort strings according to the natural
+ * language rules of that locale.
+ *
+ * This function uses the collator returned by
+ * LinguisticSort.getCollator to produce a collation key for its input
+ * string. A user can expect that the sorting semantics of this
+ * function for a given locale is equivalent to the sorting behaviour
+ * of an Oracle query that is constructed using the Oracle functions
+ * returned by LinguisticSort for that locale.
+ *
+ * The optional third argument to the function is a boolean that
+ * specifies whether to use the upper-case collator (case-insensitive)
+ * returned by LinguisticSort.getUpperCaseCollator.
+ *
+ * The optional fourth and fifth arguments are used to set
+ * respectively the strength and composition of the collator returned
+ * by LinguisticSort using the setStrength and setDecomposition
+ * methods of java.text.Collator.
+ * 
+ * @author snakhoda-sfdc
+ *
+ */
+@FunctionParseNode.BuiltInFunction(name = CollationKeyFunction.NAME, args 
= {
+   // input string
+   @FunctionParseNode.Argument(allowedTypes = { PVarchar.class }),
+   // ISO Code for Locale
+   @FunctionParseNode.Argument(allowedTypes = { PVarchar.class }, 
isConstant = true),
+   // whether to use special upper case collator
+   @FunctionParseNode.Argument(allowedTypes = { PBoolean.class }, 
defaultValue = "false", isConstant = true),
+   // collator strength
+   @FunctionParseNode.Argument(allowedTypes = { PInteger.class }, 
defaultValue = "null", isConstant = true),

[GitHub] phoenix pull request #275: PHOENIX-4237: Add function to calculate Java coll...

[JamesRTaylor]

Github user JamesRTaylor commented on a diff in the pull request:

https://github.com/apache/phoenix/pull/275#discussion_r145473757
  
--- Diff: 
phoenix-core/src/main/java/org/apache/phoenix/expression/function/CollationKeyFunction.java
 ---
@@ -0,0 +1,230 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements.  See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership.  The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License.  You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+package org.apache.phoenix.expression.function;
+
+import java.io.DataInput;
+import java.io.IOException;
+import java.sql.SQLException;
+import java.text.Collator;
+import java.util.List;
+import java.util.Locale;
+
+import org.apache.commons.lang.BooleanUtils;
+import org.apache.commons.logging.Log;
+import org.apache.commons.logging.LogFactory;
+import org.apache.hadoop.hbase.io.ImmutableBytesWritable;
+import org.apache.phoenix.expression.Expression;
+import org.apache.phoenix.expression.LiteralExpression;
+import org.apache.phoenix.parse.FunctionParseNode;
+import org.apache.phoenix.schema.tuple.Tuple;
+import org.apache.phoenix.schema.types.PBoolean;
+import org.apache.phoenix.schema.types.PDataType;
+import org.apache.phoenix.schema.types.PInteger;
+import org.apache.phoenix.schema.types.PVarbinary;
+import org.apache.phoenix.schema.types.PVarchar;
+import org.apache.phoenix.util.VarBinaryFormatter;
+
+import com.force.db.i18n.LinguisticSort;
+import com.force.i18n.LocaleUtils;
+
+/**
+ * A Phoenix Function that calculates a collation key for an input
+ * string based on a caller-provided locale and collator strength and
+ * decomposition settings.
+ * 
+ * The locale should be specified as xx_yy_variant where xx is the ISO
+ * 639-1 2-letter language code, yy is the the ISO 3166 2-letter
+ * country code. Both countryCode and variant are optional. For
+ * example, zh_TW_STROKE, zh_TW and zh are all valid locale
+ * representations. Note the language code, country code and variant
+ * are used as arguments to the constructor of java.util.Locale.
+ *
+ * This function uses the open-source grammaticus and i18n-util
+ * packages to obtain the collators it needs from the provided locale.
+ *
+ * The LinguisticSort implementation in i18n-util encapsulates
+ * sort-related functionality for a substantive list of locales. For
+ * each locale, it provides a collator and an Oracle-specific database
+ * function that can be used to sort strings according to the natural
+ * language rules of that locale.
+ *
+ * This function uses the collator returned by
+ * LinguisticSort.getCollator to produce a collation key for its input
+ * string. A user can expect that the sorting semantics of this
+ * function for a given locale is equivalent to the sorting behaviour
+ * of an Oracle query that is constructed using the Oracle functions
+ * returned by LinguisticSort for that locale.
+ *
+ * The optional third argument to the function is a boolean that
+ * specifies whether to use the upper-case collator (case-insensitive)
+ * returned by LinguisticSort.getUpperCaseCollator.
+ *
+ * The optional fourth and fifth arguments are used to set
+ * respectively the strength and composition of the collator returned
+ * by LinguisticSort using the setStrength and setDecomposition
+ * methods of java.text.Collator.
+ * 
+ * @author snakhoda-sfdc
+ *
+ */
+@FunctionParseNode.BuiltInFunction(name = CollationKeyFunction.NAME, args 
= {
+   // input string
+   @FunctionParseNode.Argument(allowedTypes = { PVarchar.class }),
+   // ISO Code for Locale
+   @FunctionParseNode.Argument(allowedTypes = { PVarchar.class }, 
isConstant = true),
+   // whether to use special upper case collator
+   @FunctionParseNode.Argument(allowedTypes = { PBoolean.class }, 
defaultValue = "false", isConstant = true),
+   // collator strength
+   @FunctionParseNode.Argument(allowedTypes = { PInteger.class }, 
defaultValue = "null", isConstant = true),

[jira] [Commented] (PHOENIX-4237) Allow sorting on (Java) collation keys for non-English locales

[ASF GitHub Bot (JIRA)]

[ 
https://issues.apache.org/jira/browse/PHOENIX-4237?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16209669#comment-16209669
 ] 

ASF GitHub Bot commented on PHOENIX-4237:
-

Github user JamesRTaylor commented on a diff in the pull request:

https://github.com/apache/phoenix/pull/275#discussion_r145473757
  
--- Diff: 
phoenix-core/src/main/java/org/apache/phoenix/expression/function/CollationKeyFunction.java
 ---
@@ -0,0 +1,230 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements.  See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership.  The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License.  You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+package org.apache.phoenix.expression.function;
+
+import java.io.DataInput;
+import java.io.IOException;
+import java.sql.SQLException;
+import java.text.Collator;
+import java.util.List;
+import java.util.Locale;
+
+import org.apache.commons.lang.BooleanUtils;
+import org.apache.commons.logging.Log;
+import org.apache.commons.logging.LogFactory;
+import org.apache.hadoop.hbase.io.ImmutableBytesWritable;
+import org.apache.phoenix.expression.Expression;
+import org.apache.phoenix.expression.LiteralExpression;
+import org.apache.phoenix.parse.FunctionParseNode;
+import org.apache.phoenix.schema.tuple.Tuple;
+import org.apache.phoenix.schema.types.PBoolean;
+import org.apache.phoenix.schema.types.PDataType;
+import org.apache.phoenix.schema.types.PInteger;
+import org.apache.phoenix.schema.types.PVarbinary;
+import org.apache.phoenix.schema.types.PVarchar;
+import org.apache.phoenix.util.VarBinaryFormatter;
+
+import com.force.db.i18n.LinguisticSort;
+import com.force.i18n.LocaleUtils;
+
+/**
+ * A Phoenix Function that calculates a collation key for an input
+ * string based on a caller-provided locale and collator strength and
+ * decomposition settings.
+ * 
+ * The locale should be specified as xx_yy_variant where xx is the ISO
+ * 639-1 2-letter language code, yy is the the ISO 3166 2-letter
+ * country code. Both countryCode and variant are optional. For
+ * example, zh_TW_STROKE, zh_TW and zh are all valid locale
+ * representations. Note the language code, country code and variant
+ * are used as arguments to the constructor of java.util.Locale.
+ *
+ * This function uses the open-source grammaticus and i18n-util
+ * packages to obtain the collators it needs from the provided locale.
+ *
+ * The LinguisticSort implementation in i18n-util encapsulates
+ * sort-related functionality for a substantive list of locales. For
+ * each locale, it provides a collator and an Oracle-specific database
+ * function that can be used to sort strings according to the natural
+ * language rules of that locale.
+ *
+ * This function uses the collator returned by
+ * LinguisticSort.getCollator to produce a collation key for its input
+ * string. A user can expect that the sorting semantics of this
+ * function for a given locale is equivalent to the sorting behaviour
+ * of an Oracle query that is constructed using the Oracle functions
+ * returned by LinguisticSort for that locale.
+ *
+ * The optional third argument to the function is a boolean that
+ * specifies whether to use the upper-case collator (case-insensitive)
+ * returned by LinguisticSort.getUpperCaseCollator.
+ *
+ * The optional fourth and fifth arguments are used to set
+ * respectively the strength and composition of the collator returned
+ * by LinguisticSort using the setStrength and setDecomposition
+ * methods of java.text.Collator.
+ * 
+ * @author snakhoda-sfdc
+ *
+ */
+@FunctionParseNode.BuiltInFunction(name = CollationKeyFunction.NAME, args 
= {
+   // input string
+   @FunctionParseNode.Argument(allowedTypes = { PVarchar.class }),
+   // ISO Code for Locale
+   @FunctionParseNode.Argument(allowedTypes = { PVarchar.class }, 
isConstant = true),
+   // whether to use special upper case collator
+   

[jira] [Commented] (PHOENIX-4237) Allow sorting on (Java) collation keys for non-English locales

[ASF GitHub Bot (JIRA)]

[ 
https://issues.apache.org/jira/browse/PHOENIX-4237?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16209667#comment-16209667
 ] 

ASF GitHub Bot commented on PHOENIX-4237:
-

Github user JamesRTaylor commented on a diff in the pull request:

https://github.com/apache/phoenix/pull/275#discussion_r145473581
  
--- Diff: 
phoenix-core/src/main/java/org/apache/phoenix/expression/function/CollationKeyFunction.java
 ---
@@ -0,0 +1,230 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements.  See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership.  The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License.  You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+package org.apache.phoenix.expression.function;
+
+import java.io.DataInput;
+import java.io.IOException;
+import java.sql.SQLException;
+import java.text.Collator;
+import java.util.List;
+import java.util.Locale;
+
+import org.apache.commons.lang.BooleanUtils;
+import org.apache.commons.logging.Log;
+import org.apache.commons.logging.LogFactory;
+import org.apache.hadoop.hbase.io.ImmutableBytesWritable;
+import org.apache.phoenix.expression.Expression;
+import org.apache.phoenix.expression.LiteralExpression;
+import org.apache.phoenix.parse.FunctionParseNode;
+import org.apache.phoenix.schema.tuple.Tuple;
+import org.apache.phoenix.schema.types.PBoolean;
+import org.apache.phoenix.schema.types.PDataType;
+import org.apache.phoenix.schema.types.PInteger;
+import org.apache.phoenix.schema.types.PVarbinary;
+import org.apache.phoenix.schema.types.PVarchar;
+import org.apache.phoenix.util.VarBinaryFormatter;
+
+import com.force.db.i18n.LinguisticSort;
+import com.force.i18n.LocaleUtils;
+
+/**
+ * A Phoenix Function that calculates a collation key for an input
+ * string based on a caller-provided locale and collator strength and
+ * decomposition settings.
+ * 
+ * The locale should be specified as xx_yy_variant where xx is the ISO
+ * 639-1 2-letter language code, yy is the the ISO 3166 2-letter
+ * country code. Both countryCode and variant are optional. For
+ * example, zh_TW_STROKE, zh_TW and zh are all valid locale
+ * representations. Note the language code, country code and variant
+ * are used as arguments to the constructor of java.util.Locale.
+ *
+ * This function uses the open-source grammaticus and i18n-util
+ * packages to obtain the collators it needs from the provided locale.
+ *
+ * The LinguisticSort implementation in i18n-util encapsulates
+ * sort-related functionality for a substantive list of locales. For
+ * each locale, it provides a collator and an Oracle-specific database
+ * function that can be used to sort strings according to the natural
+ * language rules of that locale.
+ *
+ * This function uses the collator returned by
+ * LinguisticSort.getCollator to produce a collation key for its input
+ * string. A user can expect that the sorting semantics of this
+ * function for a given locale is equivalent to the sorting behaviour
+ * of an Oracle query that is constructed using the Oracle functions
+ * returned by LinguisticSort for that locale.
+ *
+ * The optional third argument to the function is a boolean that
+ * specifies whether to use the upper-case collator (case-insensitive)
+ * returned by LinguisticSort.getUpperCaseCollator.
+ *
+ * The optional fourth and fifth arguments are used to set
+ * respectively the strength and composition of the collator returned
+ * by LinguisticSort using the setStrength and setDecomposition
+ * methods of java.text.Collator.
+ * 
+ * @author snakhoda-sfdc
+ *
+ */
+@FunctionParseNode.BuiltInFunction(name = CollationKeyFunction.NAME, args 
= {
+   // input string
+   @FunctionParseNode.Argument(allowedTypes = { PVarchar.class }),
+   // ISO Code for Locale
+   @FunctionParseNode.Argument(allowedTypes = { PVarchar.class }, 
isConstant = true),
+   // whether to use special upper case collator
+   

[GitHub] phoenix pull request #275: PHOENIX-4237: Add function to calculate Java coll...

[JamesRTaylor]

Github user JamesRTaylor commented on a diff in the pull request:

https://github.com/apache/phoenix/pull/275#discussion_r145473581
  
--- Diff: 
phoenix-core/src/main/java/org/apache/phoenix/expression/function/CollationKeyFunction.java
 ---
@@ -0,0 +1,230 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements.  See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership.  The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License.  You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+package org.apache.phoenix.expression.function;
+
+import java.io.DataInput;
+import java.io.IOException;
+import java.sql.SQLException;
+import java.text.Collator;
+import java.util.List;
+import java.util.Locale;
+
+import org.apache.commons.lang.BooleanUtils;
+import org.apache.commons.logging.Log;
+import org.apache.commons.logging.LogFactory;
+import org.apache.hadoop.hbase.io.ImmutableBytesWritable;
+import org.apache.phoenix.expression.Expression;
+import org.apache.phoenix.expression.LiteralExpression;
+import org.apache.phoenix.parse.FunctionParseNode;
+import org.apache.phoenix.schema.tuple.Tuple;
+import org.apache.phoenix.schema.types.PBoolean;
+import org.apache.phoenix.schema.types.PDataType;
+import org.apache.phoenix.schema.types.PInteger;
+import org.apache.phoenix.schema.types.PVarbinary;
+import org.apache.phoenix.schema.types.PVarchar;
+import org.apache.phoenix.util.VarBinaryFormatter;
+
+import com.force.db.i18n.LinguisticSort;
+import com.force.i18n.LocaleUtils;
+
+/**
+ * A Phoenix Function that calculates a collation key for an input
+ * string based on a caller-provided locale and collator strength and
+ * decomposition settings.
+ * 
+ * The locale should be specified as xx_yy_variant where xx is the ISO
+ * 639-1 2-letter language code, yy is the the ISO 3166 2-letter
+ * country code. Both countryCode and variant are optional. For
+ * example, zh_TW_STROKE, zh_TW and zh are all valid locale
+ * representations. Note the language code, country code and variant
+ * are used as arguments to the constructor of java.util.Locale.
+ *
+ * This function uses the open-source grammaticus and i18n-util
+ * packages to obtain the collators it needs from the provided locale.
+ *
+ * The LinguisticSort implementation in i18n-util encapsulates
+ * sort-related functionality for a substantive list of locales. For
+ * each locale, it provides a collator and an Oracle-specific database
+ * function that can be used to sort strings according to the natural
+ * language rules of that locale.
+ *
+ * This function uses the collator returned by
+ * LinguisticSort.getCollator to produce a collation key for its input
+ * string. A user can expect that the sorting semantics of this
+ * function for a given locale is equivalent to the sorting behaviour
+ * of an Oracle query that is constructed using the Oracle functions
+ * returned by LinguisticSort for that locale.
+ *
+ * The optional third argument to the function is a boolean that
+ * specifies whether to use the upper-case collator (case-insensitive)
+ * returned by LinguisticSort.getUpperCaseCollator.
+ *
+ * The optional fourth and fifth arguments are used to set
+ * respectively the strength and composition of the collator returned
+ * by LinguisticSort using the setStrength and setDecomposition
+ * methods of java.text.Collator.
+ * 
+ * @author snakhoda-sfdc
+ *
+ */
+@FunctionParseNode.BuiltInFunction(name = CollationKeyFunction.NAME, args 
= {
+   // input string
+   @FunctionParseNode.Argument(allowedTypes = { PVarchar.class }),
+   // ISO Code for Locale
+   @FunctionParseNode.Argument(allowedTypes = { PVarchar.class }, 
isConstant = true),
+   // whether to use special upper case collator
+   @FunctionParseNode.Argument(allowedTypes = { PBoolean.class }, 
defaultValue = "false", isConstant = true),
+   // collator strength
+   @FunctionParseNode.Argument(allowedTypes = { PInteger.class }, 
defaultValue = "null", isConstant = true),

[jira] [Resolved] (PHOENIX-4294) Allow scalar function to declare that it's not thread safe

[James Taylor (JIRA)]

 [ 
https://issues.apache.org/jira/browse/PHOENIX-4294?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

James Taylor resolved PHOENIX-4294.
---
   Resolution: Fixed
Fix Version/s: 4.13.0

> Allow scalar function to declare that it's not thread safe
> --
>
> Key: PHOENIX-4294
> URL: https://issues.apache.org/jira/browse/PHOENIX-4294
> Project: Phoenix
>  Issue Type: Improvement
>Reporter: James Taylor
>Assignee: James Taylor
> Fix For: 4.13.0, 4.12.1
>
> Attachments: PHOENIX-4294.patch
>
>
> We have our CloneExpressionVisitor which determines when an expression 
> subtree needs to be cloned. For ease of use and clarity, we should have a new 
> method on ScalarFunction that may be overridden to determine when the 
> function is cloned versus not cloned.



--
This message was sent by Atlassian JIRA
(v6.4.14#64029)

[jira] [Resolved] (PHOENIX-4295) Fix argument order for StatsCollectorIT derived classes

[James Taylor (JIRA)]

 [ 
https://issues.apache.org/jira/browse/PHOENIX-4295?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

James Taylor resolved PHOENIX-4295.
---
   Resolution: Fixed
Fix Version/s: 4.13.0

> Fix argument order for StatsCollectorIT derived classes
> ---
>
> Key: PHOENIX-4295
> URL: https://issues.apache.org/jira/browse/PHOENIX-4295
> Project: Phoenix
>  Issue Type: Test
>Reporter: James Taylor
>Assignee: James Taylor
> Fix For: 4.13.0, 4.12.1
>
> Attachments: PHOENIX-4295.patch
>
>




--
This message was sent by Atlassian JIRA
(v6.4.14#64029)

[jira] [Assigned] (PHOENIX-4167) Phoenix SELECT query returns duplicate data in the same varchar/char column if a trim() is applied on the column AND a distinct arbitrary column is generated in the qu

[Dumindu Buddhika (JIRA)]

 [ 
https://issues.apache.org/jira/browse/PHOENIX-4167?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Dumindu Buddhika reassigned PHOENIX-4167:
-

Assignee: Dumindu Buddhika

> Phoenix SELECT query returns duplicate data in the same varchar/char column 
> if a trim() is applied on the column AND a distinct arbitrary column is 
> generated in the query
> --
>
> Key: PHOENIX-4167
> URL: https://issues.apache.org/jira/browse/PHOENIX-4167
> Project: Phoenix
>  Issue Type: Bug
>Affects Versions: 4.7.0
>Reporter: Pulkit Bhardwaj
>Assignee: Dumindu Buddhika
>Priority: Minor
>
> 1. Created a simple table in phoenix
> {code:sql}
> create table test_select(nam VARCHAR(20), address VARCHAR(20), id BIGINT 
> constraint my_pk primary key (id));
> {code}
> 2. Insert a sample row
> {code:sql}
> upsert into test_select (nam, address,id) values('user','place',1);
> {code}
> 3. Confirm that the row is present
> {code:sql}
> 0: jdbc:phoenix:> select * from test_select;
> +-+--+-+
> |   NAM   | ADDRESS  | ID  |
> +-+--+-+
> | user  | place   | 1   |
> +-+--+-+
> {code}
> 4. Now run the following query
> {code:sql}
> 0: jdbc:phoenix:> select distinct 'arbitrary' as "test_column", trim(nam), 
> trim(nam) from test_select;
> This would generate the following output
> +--+++
> | test_column  |   TRIM(NAM)|   TRIM(NAM)|
> +--+++
> | arbitrary  | useruser  | useruser  |
> +--+++
> {code}
> As we can see the output for the trim(name) which should have been 'user' is 
> actually printed as 'useruser'
> The concatenation to the string is actually the number of times the column is 
> printed.
> The following
> {code:sql}
> 0: jdbc:phoenix:> select distinct 'arbitrary' as "test_column", trim(nam), 
> trim(nam), trim(nam) from test_select;
> {code}
> Would generate
> {code:sql}
> +--+---+---+---+
> | test_column  |   TRIM(NAM)   |   TRIM(NAM)   |   
> TRIM(NAM)   |
> +--+---+---+---+
> | arbitrary  | useruseruser  | useruseruser  | useruseruser  |
> +--+---+---+---+
> {code}
> A couple of things to notice
> 1. If I remove the —— distinct 'harshit' as "test_column" ——  The issue is 
> not seen
> {code:sql}
> 0: jdbc:phoenix:> select trim(nam), trim(nam), trim(nam) from test_select;
> ++++
> | TRIM(NAM)  | TRIM(NAM)  | TRIM(NAM)  |
> ++++
> | user | user | user |
> ++++
> {code}
> 2. If I remove the trim() again the issue is not seen
> {code:sql}
> 0: jdbc:phoenix:> select distinct 'arbitrary' as "test_column" ,nam, nam from 
> test_select;
> +--+-+-+
> | test_column  |   NAM   |   NAM   |
> +--+-+-+
> | arbitrary| user  | user  |
> +--+-+-+
> {code}



--
This message was sent by Atlassian JIRA
(v6.4.14#64029)

[jira] [Commented] (PHOENIX-4296) Dead loop in HBase reverse scan when amount of scan data is greater than SCAN_RESULT_CHUNK_SIZE

[Chen Feng (JIRA)]

[ 
https://issues.apache.org/jira/browse/PHOENIX-4296?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16209010#comment-16209010
 ] 

Chen Feng commented on PHOENIX-4296:


For example, if there are 3001 rows, whose ids are  [1, 2, 3, ..., 3001].
In the first scan with scan.startRow=1, stopRow=3001, since it is a reverse 
scan, the lastKey will be 3001, 3000, 2999, ... 3.
At the end of the first scan, lastKey will be 2. 

In the next scan, the code
scan.setStartRow(ByteUtil.copyKeyBytesIfNecessary(lastKey)); will set 
the scan with scan.startRow=2, stopRow=3001.

Therefore, the outer scan never ends with internal scans with repeated rows 
from 3 to 3001.


> Dead loop in HBase reverse scan when amount of scan data is greater than 
> SCAN_RESULT_CHUNK_SIZE
> ---
>
> Key: PHOENIX-4296
> URL: https://issues.apache.org/jira/browse/PHOENIX-4296
> Project: Phoenix
>  Issue Type: Bug
>Affects Versions: 4.6.0
>Reporter: rukawakang
>
> This problem seems to only occur with reverse scan not forward scan. When 
> amount of scan data is greater than SCAN_RESULT_CHUNK_SIZE(default 2999), 
> Class ChunkedResultIteratorFactory will multiple calls function 
> getResultIterator. But in function getResultIterator it always readjusts 
> startRow, in fact, if in reverse scan we should readjust stopRow. For example 
> {code:java}
> if (ScanUtil.isReversed(scan)) {
> scan.setStopRow(ByteUtil.copyKeyBytesIfNecessary(lastKey));
> } else {
> scan.setStartRow(ByteUtil.copyKeyBytesIfNecessary(lastKey));
> }
> {code}



--
This message was sent by Atlassian JIRA
(v6.4.14#64029)

[jira] [Created] (PHOENIX-4296) Dead loop in HBase reverse scan when amount of scan data is greater than SCAN_RESULT_CHUNK_SIZE

[rukawakang (JIRA)]

rukawakang created PHOENIX-4296:
---

 Summary: Dead loop in HBase reverse scan when amount of scan data 
is greater than SCAN_RESULT_CHUNK_SIZE
 Key: PHOENIX-4296
 URL: https://issues.apache.org/jira/browse/PHOENIX-4296
 Project: Phoenix
  Issue Type: Bug
Affects Versions: 4.6.0
Reporter: rukawakang


This problem seems to only occur with reverse scan not forward scan. When 
amount of scan data is greater than SCAN_RESULT_CHUNK_SIZE(default 2999), Class 
ChunkedResultIteratorFactory will multiple calls function getResultIterator. 
But in function getResultIterator it always readjusts startRow, in fact, if in 
reverse scan we should readjust stopRow. For example 
{code:java}
if (ScanUtil.isReversed(scan)) {
scan.setStopRow(ByteUtil.copyKeyBytesIfNecessary(lastKey));
} else {
scan.setStartRow(ByteUtil.copyKeyBytesIfNecessary(lastKey));
}
{code}




--
This message was sent by Atlassian JIRA
(v6.4.14#64029)

[jira] [Commented] (PHOENIX-4280) Delete doesn't work when immutable indexes are in building state

[James Taylor (JIRA)]

[ 
https://issues.apache.org/jira/browse/PHOENIX-4280?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16208915#comment-16208915
 ] 

James Taylor commented on PHOENIX-4280:
---

You're right, [~tdsilva]. Let me look at this closer. We should never do a scan 
for prior row values for immutable tables (except for on a rollback). Something 
is wrong with the rowkey sent over for the Put that represents the delete.

> Delete doesn't work when immutable indexes are in building state
> 
>
> Key: PHOENIX-4280
> URL: https://issues.apache.org/jira/browse/PHOENIX-4280
> Project: Phoenix
>  Issue Type: Bug
>Reporter: James Taylor
>Assignee: James Taylor
> Fix For: 4.12.1
>
> Attachments: PHOENIX-4280.patch, PHOENIX-4280_v2.patch, 
> PHOENIX-4280_wip.patch
>
>
> We don't generate the Delete mutations correctly in this case as we assume 
> we're scanning over the index table.



--
This message was sent by Atlassian JIRA
(v6.4.14#64029)