Apache-Phoenix | 4.x-HBase-1.2 | Build Successful

[Apache Jenkins Server]

4.x-HBase-1.2 branch build status Successful

Source repository https://git-wip-us.apache.org/repos/asf?p=phoenix.git;a=shortlog;h=refs/heads/4.x-HBase-1.2

Compiled Artifacts https://builds.apache.org/job/Phoenix-4.x-HBase-1.2/lastSuccessfulBuild/artifact/

Test Report https://builds.apache.org/job/Phoenix-4.x-HBase-1.2/lastCompletedBuild/testReport/

Changes
[jtaylor] PHOENIX-4278 Implement pure client side transactional index maintenance



Build times for last couple of runsLatest build time is the right most | Legend blue: normal, red: test failure, gray: timeout

Build failed in Jenkins: Phoenix-4.x-HBase-1.1 #660

[Apache Jenkins Server]

See 


Changes:

[jtaylor] PHOENIX-4278 Implement pure client side transactional index 
maintenance

--
[...truncated 104.89 KB...]
[INFO] Tests run: 6, Failures: 0, Errors: 0, Skipped: 0, Time elapsed: 67.314 s 
- in org.apache.phoenix.end2end.index.GlobalIndexOptimizationIT
[INFO] Running org.apache.phoenix.end2end.index.IndexMaintenanceIT
[INFO] Running org.apache.phoenix.end2end.index.GlobalMutableTxIndexIT
[INFO] Tests run: 16, Failures: 0, Errors: 0, Skipped: 0, Time elapsed: 414.519 
s - in org.apache.phoenix.end2end.index.DropColumnIT
[INFO] Tests run: 14, Failures: 0, Errors: 0, Skipped: 0, Time elapsed: 151.2 s 
- in org.apache.phoenix.end2end.index.IndexMaintenanceIT
[INFO] Tests run: 40, Failures: 0, Errors: 0, Skipped: 0, Time elapsed: 355.294 
s - in org.apache.phoenix.end2end.index.GlobalImmutableNonTxIndexIT
[INFO] Running org.apache.phoenix.end2end.index.IndexMetadataIT
[INFO] Running org.apache.phoenix.end2end.index.IndexUsageIT
[INFO] Running org.apache.phoenix.end2end.index.IndexWithTableSchemaChangeIT
[INFO] Tests run: 40, Failures: 0, Errors: 0, Skipped: 0, Time elapsed: 378.008 
s - in org.apache.phoenix.end2end.index.GlobalImmutableTxIndexIT
[INFO] Running org.apache.phoenix.end2end.index.LocalImmutableNonTxIndexIT
[INFO] Tests run: 40, Failures: 0, Errors: 0, Skipped: 0, Time elapsed: 350.633 
s - in org.apache.phoenix.end2end.index.GlobalMutableNonTxIndexIT
[INFO] Tests run: 14, Failures: 0, Errors: 0, Skipped: 0, Time elapsed: 140.528 
s - in org.apache.phoenix.end2end.index.IndexMetadataIT
[INFO] Running org.apache.phoenix.end2end.index.LocalMutableNonTxIndexIT
[INFO] Running org.apache.phoenix.end2end.index.LocalImmutableTxIndexIT
[INFO] Tests run: 40, Failures: 0, Errors: 0, Skipped: 0, Time elapsed: 377.119 
s - in org.apache.phoenix.end2end.index.GlobalMutableTxIndexIT
[INFO] Running org.apache.phoenix.end2end.index.LocalMutableTxIndexIT
[INFO] Tests run: 18, Failures: 0, Errors: 0, Skipped: 0, Time elapsed: 251.442 
s - in org.apache.phoenix.end2end.index.IndexWithTableSchemaChangeIT
[INFO] Running org.apache.phoenix.end2end.index.MutableIndexIT
[INFO] Tests run: 37, Failures: 0, Errors: 0, Skipped: 0, Time elapsed: 383.047 
s - in org.apache.phoenix.end2end.index.IndexUsageIT
[INFO] Running org.apache.phoenix.end2end.index.SaltedIndexIT
[INFO] Tests run: 3, Failures: 0, Errors: 0, Skipped: 0, Time elapsed: 25.172 s 
- in org.apache.phoenix.end2end.index.SaltedIndexIT
[INFO] Running org.apache.phoenix.end2end.index.ViewIndexIT
[INFO] Tests run: 40, Failures: 0, Errors: 0, Skipped: 0, Time elapsed: 462.311 
s - in org.apache.phoenix.end2end.index.LocalImmutableNonTxIndexIT
[WARNING] Tests run: 12, Failures: 0, Errors: 0, Skipped: 2, Time elapsed: 
104.632 s - in org.apache.phoenix.end2end.index.ViewIndexIT
[INFO] Running org.apache.phoenix.end2end.index.txn.MutableRollbackIT
[INFO] Running org.apache.phoenix.end2end.index.txn.RollbackIT
[INFO] Tests run: 40, Failures: 0, Errors: 0, Skipped: 0, Time elapsed: 466.277 
s - in org.apache.phoenix.end2end.index.LocalMutableNonTxIndexIT
[INFO] Running org.apache.phoenix.end2end.join.HashJoinCacheIT
[INFO] Tests run: 40, Failures: 0, Errors: 0, Skipped: 0, Time elapsed: 486.718 
s - in org.apache.phoenix.end2end.index.LocalImmutableTxIndexIT
[INFO] Tests run: 1, Failures: 0, Errors: 0, Skipped: 0, Time elapsed: 14.378 s 
- in org.apache.phoenix.end2end.join.HashJoinCacheIT
[INFO] Running org.apache.phoenix.end2end.join.HashJoinLocalIndexIT
[INFO] Tests run: 8, Failures: 0, Errors: 0, Skipped: 0, Time elapsed: 84.019 s 
- in org.apache.phoenix.end2end.index.txn.RollbackIT
[INFO] Running org.apache.phoenix.end2end.join.HashJoinMoreIT
[INFO] Running org.apache.phoenix.end2end.join.HashJoinGlobalIndexIT
[INFO] Tests run: 6, Failures: 0, Errors: 0, Skipped: 0, Time elapsed: 113.315 
s - in org.apache.phoenix.end2end.index.txn.MutableRollbackIT
[INFO] Running org.apache.phoenix.end2end.join.HashJoinNoIndexIT
[INFO] Tests run: 40, Failures: 0, Errors: 0, Skipped: 0, Time elapsed: 499.067 
s - in org.apache.phoenix.end2end.index.LocalMutableTxIndexIT
[INFO] Running org.apache.phoenix.end2end.join.SortMergeJoinGlobalIndexIT
[INFO] Tests run: 8, Failures: 0, Errors: 0, Skipped: 0, Time elapsed: 104.654 
s - in org.apache.phoenix.end2end.join.HashJoinMoreIT
[INFO] Running org.apache.phoenix.end2end.join.SortMergeJoinLocalIndexIT
[INFO] Tests run: 64, Failures: 0, Errors: 0, Skipped: 0, Time elapsed: 742.601 
s - in org.apache.phoenix.end2end.index.MutableIndexIT
[INFO] Tests run: 33, Failures: 0, Errors: 0, Skipped: 0, Time elapsed: 361.36 
s - in org.apache.phoenix.end2end.join.HashJoinNoIndexIT
[INFO] Running org.apache.phoenix.end2end.join.SortMergeJoinNoIndexIT
[INFO] Running org.apache.phoenix.end2end.join.SubqueryIT
[INFO] Tests run: 33, Failures: 0, Errors: 0, Skipped: 0, Time elapsed: 613.936 
s - in 

Apache-Phoenix | 4.x-HBase-1.2 | Build Successful

[Apache Jenkins Server]

4.x-HBase-1.2 branch build status Successful

Source repository https://git-wip-us.apache.org/repos/asf?p=phoenix.git;a=shortlog;h=refs/heads/4.x-HBase-1.2

Compiled Artifacts https://builds.apache.org/job/Phoenix-4.x-HBase-1.2/lastSuccessfulBuild/artifact/

Test Report https://builds.apache.org/job/Phoenix-4.x-HBase-1.2/lastCompletedBuild/testReport/

Changes
[maryannxue] PHOENIX-1556 Base hash versus sort merge join decision on cost



Build times for last couple of runsLatest build time is the right most | Legend blue: normal, red: test failure, gray: timeout

Build failed in Jenkins: Phoenix-4.x-HBase-1.1 #659

[Apache Jenkins Server]

See 


Changes:

[mujtaba] PHOENIX-4549 Pherf - Column override and sequenced index creation

--
[...truncated 325.57 KB...]
96296 [RpcServer.reader=5,bindAddress=asf934.gq1.ygridcore.net,port=41909] INFO 
 SecurityLogger.org.apache.hadoop.hbase.Server  - Connection from 67.195.81.168 
port: 55548 with version info: version: "1.1.9" url: 
"git://diocles.local/Volumes/hbase-1.1.9/hbase" revision: 
"0d1feabed5295495ed2257d31fab9e6553e8a9d7" user: "ndimiduk" date: "Mon Feb 20 
22:35:28 PST 2017" src_checksum: "b68339108ddccd1dfc44a76646588a58"
- Can create schema RDD and execute query
99080 [RpcServer.reader=8,bindAddress=asf934.gq1.ygridcore.net,port=45441] INFO 
 SecurityLogger.org.apache.hadoop.hbase.Server  - Connection from 67.195.81.168 
port: 51994 with version info: version: "1.1.9" url: 
"git://diocles.local/Volumes/hbase-1.1.9/hbase" revision: 
"0d1feabed5295495ed2257d31fab9e6553e8a9d7" user: "ndimiduk" date: "Mon Feb 20 
22:35:28 PST 2017" src_checksum: "b68339108ddccd1dfc44a76646588a58"
99091 [RpcServer.reader=6,bindAddress=asf934.gq1.ygridcore.net,port=41909] INFO 
 SecurityLogger.org.apache.hadoop.hbase.Server  - Connection from 67.195.81.168 
port: 4 with version info: version: "1.1.9" url: 
"git://diocles.local/Volumes/hbase-1.1.9/hbase" revision: 
"0d1feabed5295495ed2257d31fab9e6553e8a9d7" user: "ndimiduk" date: "Mon Feb 20 
22:35:28 PST 2017" src_checksum: "b68339108ddccd1dfc44a76646588a58"
- Can create schema RDD and execute query on case sensitive table (no config)
99623 [RpcServer.reader=9,bindAddress=asf934.gq1.ygridcore.net,port=45441] INFO 
 SecurityLogger.org.apache.hadoop.hbase.Server  - Connection from 67.195.81.168 
port: 52000 with version info: version: "1.1.9" url: 
"git://diocles.local/Volumes/hbase-1.1.9/hbase" revision: 
"0d1feabed5295495ed2257d31fab9e6553e8a9d7" user: "ndimiduk" date: "Mon Feb 20 
22:35:28 PST 2017" src_checksum: "b68339108ddccd1dfc44a76646588a58"
99639 [RpcServer.reader=7,bindAddress=asf934.gq1.ygridcore.net,port=41909] INFO 
 SecurityLogger.org.apache.hadoop.hbase.Server  - Connection from 67.195.81.168 
port: 55560 with version info: version: "1.1.9" url: 
"git://diocles.local/Volumes/hbase-1.1.9/hbase" revision: 
"0d1feabed5295495ed2257d31fab9e6553e8a9d7" user: "ndimiduk" date: "Mon Feb 20 
22:35:28 PST 2017" src_checksum: "b68339108ddccd1dfc44a76646588a58"
99732 [RpcServer.reader=0,bindAddress=asf934.gq1.ygridcore.net,port=45441] INFO 
 SecurityLogger.org.apache.hadoop.hbase.Server  - Connection from 67.195.81.168 
port: 52006 with version info: version: "1.1.9" url: 
"git://diocles.local/Volumes/hbase-1.1.9/hbase" revision: 
"0d1feabed5295495ed2257d31fab9e6553e8a9d7" user: "ndimiduk" date: "Mon Feb 20 
22:35:28 PST 2017" src_checksum: "b68339108ddccd1dfc44a76646588a58"
99751 [RpcServer.reader=8,bindAddress=asf934.gq1.ygridcore.net,port=41909] INFO 
 SecurityLogger.org.apache.hadoop.hbase.Server  - Connection from 67.195.81.168 
port: 55566 with version info: version: "1.1.9" url: 
"git://diocles.local/Volumes/hbase-1.1.9/hbase" revision: 
"0d1feabed5295495ed2257d31fab9e6553e8a9d7" user: "ndimiduk" date: "Mon Feb 20 
22:35:28 PST 2017" src_checksum: "b68339108ddccd1dfc44a76646588a58"
- Can create schema RDD and execute constrained query
101237 [RpcServer.reader=1,bindAddress=asf934.gq1.ygridcore.net,port=45441] 
INFO  SecurityLogger.org.apache.hadoop.hbase.Server  - Connection from 
67.195.81.168 port: 52014 with version info: version: "1.1.9" url: 
"git://diocles.local/Volumes/hbase-1.1.9/hbase" revision: 
"0d1feabed5295495ed2257d31fab9e6553e8a9d7" user: "ndimiduk" date: "Mon Feb 20 
22:35:28 PST 2017" src_checksum: "b68339108ddccd1dfc44a76646588a58"
101248 [RpcServer.reader=9,bindAddress=asf934.gq1.ygridcore.net,port=41909] 
INFO  SecurityLogger.org.apache.hadoop.hbase.Server  - Connection from 
67.195.81.168 port: 55574 with version info: version: "1.1.9" url: 
"git://diocles.local/Volumes/hbase-1.1.9/hbase" revision: 
"0d1feabed5295495ed2257d31fab9e6553e8a9d7" user: "ndimiduk" date: "Mon Feb 20 
22:35:28 PST 2017" src_checksum: "b68339108ddccd1dfc44a76646588a58"
- Can create schema RDD with predicate that will never match
101569 [RpcServer.reader=2,bindAddress=asf934.gq1.ygridcore.net,port=45441] 
INFO  SecurityLogger.org.apache.hadoop.hbase.Server  - Connection from 
67.195.81.168 port: 52020 with version info: version: "1.1.9" url: 
"git://diocles.local/Volumes/hbase-1.1.9/hbase" revision: 
"0d1feabed5295495ed2257d31fab9e6553e8a9d7" user: "ndimiduk" date: "Mon Feb 20 
22:35:28 PST 2017" src_checksum: "b68339108ddccd1dfc44a76646588a58"
101584 [RpcServer.reader=0,bindAddress=asf934.gq1.ygridcore.net,port=41909] 
INFO  SecurityLogger.org.apache.hadoop.hbase.Server  - Connection from 
67.195.81.168 port: 55580 with version info: version: "1.1.9" url: 
"git://diocles.local/Volumes/hbase-1.1.9/hbase" revision: 

Build failed in Jenkins: Phoenix | Master #1934

[Apache Jenkins Server]

See 


Changes:

[maryannxue] PHOENIX-1556 Base hash versus sort merge join decision on cost

--
[...truncated 1.18 MB...]
[INFO] Running org.apache.phoenix.end2end.join.SortMergeJoinNoIndexIT
[INFO] Running org.apache.phoenix.end2end.join.SubqueryIT
[INFO] Tests run: 64, Failures: 0, Errors: 0, Skipped: 0, Time elapsed: 752.877 
s - in org.apache.phoenix.end2end.index.MutableIndexIT
[INFO] Tests run: 33, Failures: 0, Errors: 0, Skipped: 0, Time elapsed: 362.917 
s - in org.apache.phoenix.end2end.join.HashJoinNoIndexIT
[INFO] Running org.apache.phoenix.end2end.join.SubqueryUsingSortMergeJoinIT
[INFO] Running org.apache.phoenix.end2end.salted.SaltedTableIT
[INFO] Tests run: 8, Failures: 0, Errors: 0, Skipped: 0, Time elapsed: 26.799 s 
- in org.apache.phoenix.end2end.salted.SaltedTableIT
[INFO] Running org.apache.phoenix.end2end.salted.SaltedTableUpsertSelectIT
[INFO] Tests run: 34, Failures: 0, Errors: 0, Skipped: 0, Time elapsed: 377.108 
s - in org.apache.phoenix.end2end.join.SortMergeJoinNoIndexIT
[INFO] Tests run: 33, Failures: 0, Errors: 0, Skipped: 0, Time elapsed: 624.917 
s - in org.apache.phoenix.end2end.join.HashJoinGlobalIndexIT
[INFO] Tests run: 6, Failures: 0, Errors: 0, Skipped: 0, Time elapsed: 49.888 s 
- in org.apache.phoenix.end2end.salted.SaltedTableUpsertSelectIT
[INFO] Running org.apache.phoenix.iterate.RoundRobinResultIteratorIT
[INFO] Tests run: 24, Failures: 0, Errors: 0, Skipped: 0, Time elapsed: 401.72 
s - in org.apache.phoenix.end2end.join.SubqueryIT
[INFO] Running org.apache.phoenix.iterate.PhoenixQueryTimeoutIT
[INFO] Running org.apache.phoenix.end2end.salted.SaltedTableVarLengthRowKeyIT
[INFO] Tests run: 1, Failures: 0, Errors: 0, Skipped: 0, Time elapsed: 4.409 s 
- in org.apache.phoenix.end2end.salted.SaltedTableVarLengthRowKeyIT
[INFO] Running org.apache.phoenix.rpc.UpdateCacheIT
[INFO] Tests run: 2, Failures: 0, Errors: 0, Skipped: 0, Time elapsed: 9.137 s 
- in org.apache.phoenix.iterate.PhoenixQueryTimeoutIT
[INFO] Running org.apache.phoenix.trace.PhoenixTableMetricsWriterIT
[INFO] Tests run: 2, Failures: 0, Errors: 0, Skipped: 0, Time elapsed: 14.485 s 
- in org.apache.phoenix.trace.PhoenixTableMetricsWriterIT
[INFO] Running org.apache.phoenix.trace.PhoenixTracingEndToEndIT
[INFO] Running org.apache.phoenix.replication.SystemCatalogWALEntryFilterIT
[INFO] Tests run: 7, Failures: 0, Errors: 0, Skipped: 0, Time elapsed: 42.919 s 
- in org.apache.phoenix.rpc.UpdateCacheIT
[INFO] Running org.apache.phoenix.tx.FlappingTransactionIT
[INFO] Tests run: 2, Failures: 0, Errors: 0, Skipped: 0, Time elapsed: 0.28 s - 
in org.apache.phoenix.replication.SystemCatalogWALEntryFilterIT
[INFO] Running org.apache.phoenix.tx.ParameterizedTransactionIT
[INFO] Tests run: 9, Failures: 0, Errors: 0, Skipped: 0, Time elapsed: 75.685 s 
- in org.apache.phoenix.iterate.RoundRobinResultIteratorIT
[INFO] Tests run: 4, Failures: 0, Errors: 0, Skipped: 0, Time elapsed: 18.176 s 
- in org.apache.phoenix.tx.FlappingTransactionIT
[INFO] Running org.apache.phoenix.tx.TxCheckpointIT
[INFO] Running org.apache.phoenix.tx.TransactionIT
[INFO] Tests run: 8, Failures: 0, Errors: 0, Skipped: 0, Time elapsed: 104.294 
s - in org.apache.phoenix.trace.PhoenixTracingEndToEndIT
[INFO] Running org.apache.phoenix.util.IndexScrutinyIT
[INFO] Tests run: 3, Failures: 0, Errors: 0, Skipped: 0, Time elapsed: 30.33 s 
- in org.apache.phoenix.util.IndexScrutinyIT
[INFO] Tests run: 34, Failures: 0, Errors: 0, Skipped: 0, Time elapsed: 643.447 
s - in org.apache.phoenix.end2end.join.SortMergeJoinGlobalIndexIT
[INFO] Tests run: 34, Failures: 0, Errors: 0, Skipped: 0, Time elapsed: 824.309 
s - in org.apache.phoenix.end2end.join.HashJoinLocalIndexIT
[INFO] Tests run: 15, Failures: 0, Errors: 0, Skipped: 0, Time elapsed: 278.839 
s - in org.apache.phoenix.end2end.join.SubqueryUsingSortMergeJoinIT
[INFO] Tests run: 8, Failures: 0, Errors: 0, Skipped: 0, Time elapsed: 93.689 s 
- in org.apache.phoenix.tx.TransactionIT
[INFO] Tests run: 34, Failures: 0, Errors: 0, Skipped: 0, Time elapsed: 810.135 
s - in org.apache.phoenix.end2end.join.SortMergeJoinLocalIndexIT
[WARNING] Tests run: 52, Failures: 0, Errors: 0, Skipped: 4, Time elapsed: 
360.9 s - in org.apache.phoenix.tx.ParameterizedTransactionIT
[INFO] Tests run: 40, Failures: 0, Errors: 0, Skipped: 0, Time elapsed: 391.803 
s - in org.apache.phoenix.tx.TxCheckpointIT
[INFO] 
[INFO] Results:
[INFO] 
[ERROR] Failures: 
[ERROR]   ConcurrentMutationsIT.testConcurrentDeletesAndUpsertValues:214 
Expected to find PK in data table: (0,0)
[ERROR]   DefaultColumnValueIT.testDefaultIndexed:978
[ERROR]   RowValueConstructorIT.testRVCLastPkIsTable1stPkIndex:1584
[ERROR]   
IndexMetadataIT.testMutableTableOnlyHasPrimaryKeyIndex:623->helpTestTableOnlyHasPrimaryKeyIndex:662
[ERROR] Errors: 
[ERROR]   

[2/2] phoenix git commit: PHOENIX-4278 Implement pure client side transactional index maintenance (Ohad Shacham)

[jamestaylor]

PHOENIX-4278 Implement pure client side transactional index maintenance (Ohad 
Shacham)


Project: http://git-wip-us.apache.org/repos/asf/phoenix/repo
Commit: http://git-wip-us.apache.org/repos/asf/phoenix/commit/47dc9359
Tree: http://git-wip-us.apache.org/repos/asf/phoenix/tree/47dc9359
Diff: http://git-wip-us.apache.org/repos/asf/phoenix/diff/47dc9359

Branch: refs/heads/4.x-HBase-1.1
Commit: 47dc935945cda24e0a507a032c3b7c2fe5b2cd43
Parents: aef828d
Author: James Taylor 
Authored: Mon Feb 12 12:27:10 2018 -0800
Committer: James Taylor 
Committed: Mon Feb 12 15:30:00 2018 -0800

--
 .../phoenix/end2end/index/BaseIndexIT.java  |   2 +-
 .../phoenix/end2end/index/ImmutableIndexIT.java |   4 +-
 .../org/apache/phoenix/cache/HashCache.java |   1 +
 .../phoenix/cache/IndexMetaDataCache.java   |   8 +
 .../apache/phoenix/cache/ServerCacheClient.java |   7 +-
 .../org/apache/phoenix/cache/TenantCache.java   |   2 +-
 .../apache/phoenix/cache/TenantCacheImpl.java   |   4 +-
 .../apache/phoenix/compile/DeleteCompiler.java  |   3 +
 .../apache/phoenix/compile/UpsertCompiler.java  |   2 +
 .../coprocessor/MetaDataRegionObserver.java |   4 +-
 .../coprocessor/ServerCachingEndpointImpl.java  |   4 +-
 .../coprocessor/ServerCachingProtocol.java  |   2 +-
 .../UngroupedAggregateRegionObserver.java   |  21 +-
 .../generated/ServerCachingProtos.java  | 117 -
 .../apache/phoenix/execute/MutationState.java   |  30 +-
 .../PhoenixTxnIndexMutationGenerator.java   | 519 +++
 .../hbase/index/covered/LocalTableState.java|   6 -
 .../phoenix/hbase/index/covered/TableState.java |   7 -
 .../index/IndexMetaDataCacheFactory.java|   7 +-
 .../apache/phoenix/index/PhoenixIndexCodec.java |   3 +-
 .../phoenix/index/PhoenixIndexMetaData.java |  12 +
 .../index/PhoenixTransactionalIndexer.java  |   9 +-
 .../phoenix/jdbc/PhoenixDatabaseMetaData.java   |   1 +
 .../apache/phoenix/join/HashCacheFactory.java   |  34 +-
 .../index/PhoenixIndexPartialBuildMapper.java   |   5 +
 .../query/ConnectionQueryServicesImpl.java  |   8 -
 .../org/apache/phoenix/schema/PTableImpl.java   |  25 +-
 .../apache/phoenix/cache/TenantCacheTest.java   |  14 +-
 .../src/main/ServerCachingService.proto |   1 +
 29 files changed, 759 insertions(+), 103 deletions(-)
--


http://git-wip-us.apache.org/repos/asf/phoenix/blob/47dc9359/phoenix-core/src/it/java/org/apache/phoenix/end2end/index/BaseIndexIT.java
--
diff --git 
a/phoenix-core/src/it/java/org/apache/phoenix/end2end/index/BaseIndexIT.java 
b/phoenix-core/src/it/java/org/apache/phoenix/end2end/index/BaseIndexIT.java
index b92da4a..1483c58 100644
--- a/phoenix-core/src/it/java/org/apache/phoenix/end2end/index/BaseIndexIT.java
+++ b/phoenix-core/src/it/java/org/apache/phoenix/end2end/index/BaseIndexIT.java
@@ -246,7 +246,7 @@ public abstract class BaseIndexIT extends 
ParallelStatsDisabledIT {
 PTable table = PhoenixRuntime.getTable(conn, 
Bytes.toString(tableName));
 assertTrue(table.getType() == PTableType.TABLE); // should be 
data table
 boolean hasIndexData = iterator.hasNext();
-assertFalse(hasIndexData); // should have no index data
+assertFalse(hasIndexData && !transactional); // should have no 
index data
 }
 }
 }

http://git-wip-us.apache.org/repos/asf/phoenix/blob/47dc9359/phoenix-core/src/it/java/org/apache/phoenix/end2end/index/ImmutableIndexIT.java
--
diff --git 
a/phoenix-core/src/it/java/org/apache/phoenix/end2end/index/ImmutableIndexIT.java
 
b/phoenix-core/src/it/java/org/apache/phoenix/end2end/index/ImmutableIndexIT.java
index e0398c7..d520824 100644
--- 
a/phoenix-core/src/it/java/org/apache/phoenix/end2end/index/ImmutableIndexIT.java
+++ 
b/phoenix-core/src/it/java/org/apache/phoenix/end2end/index/ImmutableIndexIT.java
@@ -75,6 +75,7 @@ import com.google.common.collect.Maps;
 public class ImmutableIndexIT extends BaseUniqueNamesOwnClusterIT {
 
 private final boolean localIndex;
+private final boolean transactional;
 private final String tableDDLOptions;
 
 private volatile boolean stopThreads = false;
@@ -86,6 +87,7 @@ public class ImmutableIndexIT extends 
BaseUniqueNamesOwnClusterIT {
 public ImmutableIndexIT(boolean localIndex, boolean transactional, boolean 
columnEncoded) {
 StringBuilder optionBuilder = new StringBuilder("IMMUTABLE_ROWS=true");
 this.localIndex = localIndex;
+this.transactional = transactional;
 if (!columnEncoded) {
 if (optionBuilder.length()!=0)
 optionBuilder.append(",");
@@ -250,7 

[1/2] phoenix git commit: PHOENIX-4278 Implement pure client side transactional index maintenance (Ohad Shacham)

[jamestaylor]

Repository: phoenix
Updated Branches:
  refs/heads/4.x-HBase-1.1 aef828d00 -> 47dc93594


http://git-wip-us.apache.org/repos/asf/phoenix/blob/47dc9359/phoenix-core/src/main/java/org/apache/phoenix/mapreduce/index/PhoenixIndexPartialBuildMapper.java
--
diff --git 
a/phoenix-core/src/main/java/org/apache/phoenix/mapreduce/index/PhoenixIndexPartialBuildMapper.java
 
b/phoenix-core/src/main/java/org/apache/phoenix/mapreduce/index/PhoenixIndexPartialBuildMapper.java
index 5b85da5..7c154f0 100644
--- 
a/phoenix-core/src/main/java/org/apache/phoenix/mapreduce/index/PhoenixIndexPartialBuildMapper.java
+++ 
b/phoenix-core/src/main/java/org/apache/phoenix/mapreduce/index/PhoenixIndexPartialBuildMapper.java
@@ -33,9 +33,11 @@ import org.apache.hadoop.hbase.client.Put;
 import org.apache.hadoop.hbase.client.Result;
 import org.apache.hadoop.hbase.io.ImmutableBytesWritable;
 import org.apache.hadoop.hbase.mapreduce.TableMapper;
+import org.apache.hadoop.hbase.util.Bytes;
 import org.apache.hadoop.io.IntWritable;
 import org.apache.phoenix.cache.ServerCacheClient;
 import org.apache.phoenix.coprocessor.BaseScannerRegionObserver;
+import org.apache.phoenix.coprocessor.MetaDataProtocol;
 import org.apache.phoenix.hbase.index.util.ImmutableBytesPtr;
 import org.apache.phoenix.index.PhoenixIndexCodec;
 import org.apache.phoenix.jdbc.PhoenixConnection;
@@ -105,6 +107,7 @@ public class PhoenixIndexPartialBuildMapper extends 
TableMapperhttp://git-wip-us.apache.org/repos/asf/phoenix/blob/47dc9359/phoenix-core/src/main/java/org/apache/phoenix/query/ConnectionQueryServicesImpl.java
--
diff --git 
a/phoenix-core/src/main/java/org/apache/phoenix/query/ConnectionQueryServicesImpl.java
 
b/phoenix-core/src/main/java/org/apache/phoenix/query/ConnectionQueryServicesImpl.java
index bba4a87..517a275 100644
--- 
a/phoenix-core/src/main/java/org/apache/phoenix/query/ConnectionQueryServicesImpl.java
+++ 
b/phoenix-core/src/main/java/org/apache/phoenix/query/ConnectionQueryServicesImpl.java
@@ -175,7 +175,6 @@ import org.apache.phoenix.hbase.index.util.KeyValueBuilder;
 import org.apache.phoenix.hbase.index.util.VersionUtil;
 import org.apache.phoenix.index.PhoenixIndexBuilder;
 import org.apache.phoenix.index.PhoenixIndexCodec;
-import org.apache.phoenix.index.PhoenixTransactionalIndexer;
 import org.apache.phoenix.iterate.TableResultIterator;
 import org.apache.phoenix.iterate.TableResultIterator.RenewLeaseStatus;
 import org.apache.phoenix.jdbc.PhoenixConnection;
@@ -848,19 +847,12 @@ public class ConnectionQueryServicesImpl extends 
DelegateQueryServices implement
 && !SchemaUtil.isMetaTable(tableName)
 && !SchemaUtil.isStatsTable(tableName)) {
 if (isTransactional) {
-if 
(!descriptor.hasCoprocessor(PhoenixTransactionalIndexer.class.getName())) {
-
descriptor.addCoprocessor(PhoenixTransactionalIndexer.class.getName(), null, 
priority, null);
-}
 // For alter table, remove non transactional index 
coprocessor
 if (descriptor.hasCoprocessor(Indexer.class.getName())) {
 descriptor.removeCoprocessor(Indexer.class.getName());
 }
 } else {
 if (!descriptor.hasCoprocessor(Indexer.class.getName())) {
-// If exception on alter table to transition back to 
non transactional
-if 
(descriptor.hasCoprocessor(PhoenixTransactionalIndexer.class.getName())) {
-
descriptor.removeCoprocessor(PhoenixTransactionalIndexer.class.getName());
-}
 Map opts = 
Maps.newHashMapWithExpectedSize(1);
 opts.put(NonTxIndexBuilder.CODEC_CLASS_NAME_KEY, 
PhoenixIndexCodec.class.getName());
 Indexer.enableIndexing(descriptor, 
PhoenixIndexBuilder.class, opts, priority);

http://git-wip-us.apache.org/repos/asf/phoenix/blob/47dc9359/phoenix-core/src/main/java/org/apache/phoenix/schema/PTableImpl.java
--
diff --git 
a/phoenix-core/src/main/java/org/apache/phoenix/schema/PTableImpl.java 
b/phoenix-core/src/main/java/org/apache/phoenix/schema/PTableImpl.java
index 9525127..fb30bc7 100644
--- a/phoenix-core/src/main/java/org/apache/phoenix/schema/PTableImpl.java
+++ b/phoenix-core/src/main/java/org/apache/phoenix/schema/PTableImpl.java
@@ -1038,32 +1038,21 @@ public class PTableImpl implements PTable {
 @Override
 public void delete() {
 newMutations();
-// we're using the Tephra column family delete marker here to 
prevent the translation 
-// of deletes to puts by the Tephra's TransactionProcessor
-  

[1/2] phoenix git commit: PHOENIX-4278 Implement pure client side transactional index maintenance (Ohad Shacham)

[jamestaylor]

Repository: phoenix
Updated Branches:
  refs/heads/4.x-cdh5.11.2 06ecae7a0 -> e1566570a


http://git-wip-us.apache.org/repos/asf/phoenix/blob/e1566570/phoenix-core/src/main/java/org/apache/phoenix/mapreduce/index/PhoenixIndexPartialBuildMapper.java
--
diff --git 
a/phoenix-core/src/main/java/org/apache/phoenix/mapreduce/index/PhoenixIndexPartialBuildMapper.java
 
b/phoenix-core/src/main/java/org/apache/phoenix/mapreduce/index/PhoenixIndexPartialBuildMapper.java
index 5b85da5..7c154f0 100644
--- 
a/phoenix-core/src/main/java/org/apache/phoenix/mapreduce/index/PhoenixIndexPartialBuildMapper.java
+++ 
b/phoenix-core/src/main/java/org/apache/phoenix/mapreduce/index/PhoenixIndexPartialBuildMapper.java
@@ -33,9 +33,11 @@ import org.apache.hadoop.hbase.client.Put;
 import org.apache.hadoop.hbase.client.Result;
 import org.apache.hadoop.hbase.io.ImmutableBytesWritable;
 import org.apache.hadoop.hbase.mapreduce.TableMapper;
+import org.apache.hadoop.hbase.util.Bytes;
 import org.apache.hadoop.io.IntWritable;
 import org.apache.phoenix.cache.ServerCacheClient;
 import org.apache.phoenix.coprocessor.BaseScannerRegionObserver;
+import org.apache.phoenix.coprocessor.MetaDataProtocol;
 import org.apache.phoenix.hbase.index.util.ImmutableBytesPtr;
 import org.apache.phoenix.index.PhoenixIndexCodec;
 import org.apache.phoenix.jdbc.PhoenixConnection;
@@ -105,6 +107,7 @@ public class PhoenixIndexPartialBuildMapper extends 
TableMapperhttp://git-wip-us.apache.org/repos/asf/phoenix/blob/e1566570/phoenix-core/src/main/java/org/apache/phoenix/query/ConnectionQueryServicesImpl.java
--
diff --git 
a/phoenix-core/src/main/java/org/apache/phoenix/query/ConnectionQueryServicesImpl.java
 
b/phoenix-core/src/main/java/org/apache/phoenix/query/ConnectionQueryServicesImpl.java
index 5b7735e..6926c4e 100644
--- 
a/phoenix-core/src/main/java/org/apache/phoenix/query/ConnectionQueryServicesImpl.java
+++ 
b/phoenix-core/src/main/java/org/apache/phoenix/query/ConnectionQueryServicesImpl.java
@@ -176,7 +176,6 @@ import org.apache.phoenix.hbase.index.util.KeyValueBuilder;
 import org.apache.phoenix.hbase.index.util.VersionUtil;
 import org.apache.phoenix.index.PhoenixIndexBuilder;
 import org.apache.phoenix.index.PhoenixIndexCodec;
-import org.apache.phoenix.index.PhoenixTransactionalIndexer;
 import org.apache.phoenix.iterate.TableResultIterator;
 import org.apache.phoenix.iterate.TableResultIterator.RenewLeaseStatus;
 import org.apache.phoenix.jdbc.PhoenixConnection;
@@ -850,19 +849,12 @@ public class ConnectionQueryServicesImpl extends 
DelegateQueryServices implement
 && !SchemaUtil.isMetaTable(tableName)
 && !SchemaUtil.isStatsTable(tableName)) {
 if (isTransactional) {
-if 
(!descriptor.hasCoprocessor(PhoenixTransactionalIndexer.class.getName())) {
-
descriptor.addCoprocessor(PhoenixTransactionalIndexer.class.getName(), null, 
priority, null);
-}
 // For alter table, remove non transactional index 
coprocessor
 if (descriptor.hasCoprocessor(Indexer.class.getName())) {
 descriptor.removeCoprocessor(Indexer.class.getName());
 }
 } else {
 if (!descriptor.hasCoprocessor(Indexer.class.getName())) {
-// If exception on alter table to transition back to 
non transactional
-if 
(descriptor.hasCoprocessor(PhoenixTransactionalIndexer.class.getName())) {
-
descriptor.removeCoprocessor(PhoenixTransactionalIndexer.class.getName());
-}
 Map opts = 
Maps.newHashMapWithExpectedSize(1);
 opts.put(NonTxIndexBuilder.CODEC_CLASS_NAME_KEY, 
PhoenixIndexCodec.class.getName());
 Indexer.enableIndexing(descriptor, 
PhoenixIndexBuilder.class, opts, priority);

http://git-wip-us.apache.org/repos/asf/phoenix/blob/e1566570/phoenix-core/src/main/java/org/apache/phoenix/schema/PTableImpl.java
--
diff --git 
a/phoenix-core/src/main/java/org/apache/phoenix/schema/PTableImpl.java 
b/phoenix-core/src/main/java/org/apache/phoenix/schema/PTableImpl.java
index 9525127..fb30bc7 100644
--- a/phoenix-core/src/main/java/org/apache/phoenix/schema/PTableImpl.java
+++ b/phoenix-core/src/main/java/org/apache/phoenix/schema/PTableImpl.java
@@ -1038,32 +1038,21 @@ public class PTableImpl implements PTable {
 @Override
 public void delete() {
 newMutations();
-// we're using the Tephra column family delete marker here to 
prevent the translation 
-// of deletes to puts by the Tephra's TransactionProcessor
-  

[2/2] phoenix git commit: PHOENIX-4278 Implement pure client side transactional index maintenance (Ohad Shacham)

[jamestaylor]

PHOENIX-4278 Implement pure client side transactional index maintenance (Ohad 
Shacham)


Project: http://git-wip-us.apache.org/repos/asf/phoenix/repo
Commit: http://git-wip-us.apache.org/repos/asf/phoenix/commit/e1566570
Tree: http://git-wip-us.apache.org/repos/asf/phoenix/tree/e1566570
Diff: http://git-wip-us.apache.org/repos/asf/phoenix/diff/e1566570

Branch: refs/heads/4.x-cdh5.11.2
Commit: e1566570ad81680e115b18b0124e083ca3ab9e40
Parents: 06ecae7
Author: James Taylor 
Authored: Mon Feb 12 12:27:10 2018 -0800
Committer: James Taylor 
Committed: Mon Feb 12 15:26:25 2018 -0800

--
 .../phoenix/end2end/index/BaseIndexIT.java  |   2 +-
 .../phoenix/end2end/index/ImmutableIndexIT.java |   4 +-
 .../org/apache/phoenix/cache/HashCache.java |   1 +
 .../phoenix/cache/IndexMetaDataCache.java   |   8 +
 .../apache/phoenix/cache/ServerCacheClient.java |   7 +-
 .../org/apache/phoenix/cache/TenantCache.java   |   2 +-
 .../apache/phoenix/cache/TenantCacheImpl.java   |   4 +-
 .../apache/phoenix/compile/DeleteCompiler.java  |   3 +
 .../apache/phoenix/compile/UpsertCompiler.java  |   2 +
 .../coprocessor/MetaDataRegionObserver.java |   2 +
 .../coprocessor/ServerCachingEndpointImpl.java  |   4 +-
 .../coprocessor/ServerCachingProtocol.java  |   2 +-
 .../UngroupedAggregateRegionObserver.java   |  21 +-
 .../generated/ServerCachingProtos.java  | 117 -
 .../apache/phoenix/execute/MutationState.java   |  30 +-
 .../PhoenixTxnIndexMutationGenerator.java   | 519 +++
 .../hbase/index/covered/LocalTableState.java|   6 -
 .../phoenix/hbase/index/covered/TableState.java |   7 -
 .../index/IndexMetaDataCacheFactory.java|   7 +-
 .../apache/phoenix/index/PhoenixIndexCodec.java |   3 +-
 .../phoenix/index/PhoenixIndexMetaData.java |  12 +
 .../index/PhoenixTransactionalIndexer.java  |   9 +-
 .../phoenix/jdbc/PhoenixDatabaseMetaData.java   |   1 +
 .../apache/phoenix/join/HashCacheFactory.java   |  34 +-
 .../index/PhoenixIndexPartialBuildMapper.java   |   5 +
 .../query/ConnectionQueryServicesImpl.java  |   8 -
 .../org/apache/phoenix/schema/PTableImpl.java   |  25 +-
 .../apache/phoenix/cache/TenantCacheTest.java   |  14 +-
 .../src/main/ServerCachingService.proto |   1 +
 29 files changed, 758 insertions(+), 102 deletions(-)
--


http://git-wip-us.apache.org/repos/asf/phoenix/blob/e1566570/phoenix-core/src/it/java/org/apache/phoenix/end2end/index/BaseIndexIT.java
--
diff --git 
a/phoenix-core/src/it/java/org/apache/phoenix/end2end/index/BaseIndexIT.java 
b/phoenix-core/src/it/java/org/apache/phoenix/end2end/index/BaseIndexIT.java
index b92da4a..1483c58 100644
--- a/phoenix-core/src/it/java/org/apache/phoenix/end2end/index/BaseIndexIT.java
+++ b/phoenix-core/src/it/java/org/apache/phoenix/end2end/index/BaseIndexIT.java
@@ -246,7 +246,7 @@ public abstract class BaseIndexIT extends 
ParallelStatsDisabledIT {
 PTable table = PhoenixRuntime.getTable(conn, 
Bytes.toString(tableName));
 assertTrue(table.getType() == PTableType.TABLE); // should be 
data table
 boolean hasIndexData = iterator.hasNext();
-assertFalse(hasIndexData); // should have no index data
+assertFalse(hasIndexData && !transactional); // should have no 
index data
 }
 }
 }

http://git-wip-us.apache.org/repos/asf/phoenix/blob/e1566570/phoenix-core/src/it/java/org/apache/phoenix/end2end/index/ImmutableIndexIT.java
--
diff --git 
a/phoenix-core/src/it/java/org/apache/phoenix/end2end/index/ImmutableIndexIT.java
 
b/phoenix-core/src/it/java/org/apache/phoenix/end2end/index/ImmutableIndexIT.java
index e0398c7..d520824 100644
--- 
a/phoenix-core/src/it/java/org/apache/phoenix/end2end/index/ImmutableIndexIT.java
+++ 
b/phoenix-core/src/it/java/org/apache/phoenix/end2end/index/ImmutableIndexIT.java
@@ -75,6 +75,7 @@ import com.google.common.collect.Maps;
 public class ImmutableIndexIT extends BaseUniqueNamesOwnClusterIT {
 
 private final boolean localIndex;
+private final boolean transactional;
 private final String tableDDLOptions;
 
 private volatile boolean stopThreads = false;
@@ -86,6 +87,7 @@ public class ImmutableIndexIT extends 
BaseUniqueNamesOwnClusterIT {
 public ImmutableIndexIT(boolean localIndex, boolean transactional, boolean 
columnEncoded) {
 StringBuilder optionBuilder = new StringBuilder("IMMUTABLE_ROWS=true");
 this.localIndex = localIndex;
+this.transactional = transactional;
 if (!columnEncoded) {
 if (optionBuilder.length()!=0)
 optionBuilder.append(",");
@@ -250,7 

[1/2] phoenix git commit: PHOENIX-4278 Implement pure client side transactional index maintenance (Ohad Shacham)

[jamestaylor]

Repository: phoenix
Updated Branches:
  refs/heads/4.x-HBase-1.2 c73025115 -> 39964669d


http://git-wip-us.apache.org/repos/asf/phoenix/blob/39964669/phoenix-core/src/main/java/org/apache/phoenix/mapreduce/index/PhoenixIndexPartialBuildMapper.java
--
diff --git 
a/phoenix-core/src/main/java/org/apache/phoenix/mapreduce/index/PhoenixIndexPartialBuildMapper.java
 
b/phoenix-core/src/main/java/org/apache/phoenix/mapreduce/index/PhoenixIndexPartialBuildMapper.java
index 5b85da5..7c154f0 100644
--- 
a/phoenix-core/src/main/java/org/apache/phoenix/mapreduce/index/PhoenixIndexPartialBuildMapper.java
+++ 
b/phoenix-core/src/main/java/org/apache/phoenix/mapreduce/index/PhoenixIndexPartialBuildMapper.java
@@ -33,9 +33,11 @@ import org.apache.hadoop.hbase.client.Put;
 import org.apache.hadoop.hbase.client.Result;
 import org.apache.hadoop.hbase.io.ImmutableBytesWritable;
 import org.apache.hadoop.hbase.mapreduce.TableMapper;
+import org.apache.hadoop.hbase.util.Bytes;
 import org.apache.hadoop.io.IntWritable;
 import org.apache.phoenix.cache.ServerCacheClient;
 import org.apache.phoenix.coprocessor.BaseScannerRegionObserver;
+import org.apache.phoenix.coprocessor.MetaDataProtocol;
 import org.apache.phoenix.hbase.index.util.ImmutableBytesPtr;
 import org.apache.phoenix.index.PhoenixIndexCodec;
 import org.apache.phoenix.jdbc.PhoenixConnection;
@@ -105,6 +107,7 @@ public class PhoenixIndexPartialBuildMapper extends 
TableMapperhttp://git-wip-us.apache.org/repos/asf/phoenix/blob/39964669/phoenix-core/src/main/java/org/apache/phoenix/query/ConnectionQueryServicesImpl.java
--
diff --git 
a/phoenix-core/src/main/java/org/apache/phoenix/query/ConnectionQueryServicesImpl.java
 
b/phoenix-core/src/main/java/org/apache/phoenix/query/ConnectionQueryServicesImpl.java
index 5b7735e..6926c4e 100644
--- 
a/phoenix-core/src/main/java/org/apache/phoenix/query/ConnectionQueryServicesImpl.java
+++ 
b/phoenix-core/src/main/java/org/apache/phoenix/query/ConnectionQueryServicesImpl.java
@@ -176,7 +176,6 @@ import org.apache.phoenix.hbase.index.util.KeyValueBuilder;
 import org.apache.phoenix.hbase.index.util.VersionUtil;
 import org.apache.phoenix.index.PhoenixIndexBuilder;
 import org.apache.phoenix.index.PhoenixIndexCodec;
-import org.apache.phoenix.index.PhoenixTransactionalIndexer;
 import org.apache.phoenix.iterate.TableResultIterator;
 import org.apache.phoenix.iterate.TableResultIterator.RenewLeaseStatus;
 import org.apache.phoenix.jdbc.PhoenixConnection;
@@ -850,19 +849,12 @@ public class ConnectionQueryServicesImpl extends 
DelegateQueryServices implement
 && !SchemaUtil.isMetaTable(tableName)
 && !SchemaUtil.isStatsTable(tableName)) {
 if (isTransactional) {
-if 
(!descriptor.hasCoprocessor(PhoenixTransactionalIndexer.class.getName())) {
-
descriptor.addCoprocessor(PhoenixTransactionalIndexer.class.getName(), null, 
priority, null);
-}
 // For alter table, remove non transactional index 
coprocessor
 if (descriptor.hasCoprocessor(Indexer.class.getName())) {
 descriptor.removeCoprocessor(Indexer.class.getName());
 }
 } else {
 if (!descriptor.hasCoprocessor(Indexer.class.getName())) {
-// If exception on alter table to transition back to 
non transactional
-if 
(descriptor.hasCoprocessor(PhoenixTransactionalIndexer.class.getName())) {
-
descriptor.removeCoprocessor(PhoenixTransactionalIndexer.class.getName());
-}
 Map opts = 
Maps.newHashMapWithExpectedSize(1);
 opts.put(NonTxIndexBuilder.CODEC_CLASS_NAME_KEY, 
PhoenixIndexCodec.class.getName());
 Indexer.enableIndexing(descriptor, 
PhoenixIndexBuilder.class, opts, priority);

http://git-wip-us.apache.org/repos/asf/phoenix/blob/39964669/phoenix-core/src/main/java/org/apache/phoenix/schema/PTableImpl.java
--
diff --git 
a/phoenix-core/src/main/java/org/apache/phoenix/schema/PTableImpl.java 
b/phoenix-core/src/main/java/org/apache/phoenix/schema/PTableImpl.java
index 9525127..fb30bc7 100644
--- a/phoenix-core/src/main/java/org/apache/phoenix/schema/PTableImpl.java
+++ b/phoenix-core/src/main/java/org/apache/phoenix/schema/PTableImpl.java
@@ -1038,32 +1038,21 @@ public class PTableImpl implements PTable {
 @Override
 public void delete() {
 newMutations();
-// we're using the Tephra column family delete marker here to 
prevent the translation 
-// of deletes to puts by the Tephra's TransactionProcessor
-  

[1/2] phoenix git commit: PHOENIX-4278 Implement pure client side transactional index maintenance (Ohad Shacham)

[jamestaylor]

Repository: phoenix
Updated Branches:
  refs/heads/master 9461d0d6a -> 0d0c0feae


http://git-wip-us.apache.org/repos/asf/phoenix/blob/0d0c0fea/phoenix-core/src/main/java/org/apache/phoenix/mapreduce/index/PhoenixIndexPartialBuildMapper.java
--
diff --git 
a/phoenix-core/src/main/java/org/apache/phoenix/mapreduce/index/PhoenixIndexPartialBuildMapper.java
 
b/phoenix-core/src/main/java/org/apache/phoenix/mapreduce/index/PhoenixIndexPartialBuildMapper.java
index 5b85da5..7c154f0 100644
--- 
a/phoenix-core/src/main/java/org/apache/phoenix/mapreduce/index/PhoenixIndexPartialBuildMapper.java
+++ 
b/phoenix-core/src/main/java/org/apache/phoenix/mapreduce/index/PhoenixIndexPartialBuildMapper.java
@@ -33,9 +33,11 @@ import org.apache.hadoop.hbase.client.Put;
 import org.apache.hadoop.hbase.client.Result;
 import org.apache.hadoop.hbase.io.ImmutableBytesWritable;
 import org.apache.hadoop.hbase.mapreduce.TableMapper;
+import org.apache.hadoop.hbase.util.Bytes;
 import org.apache.hadoop.io.IntWritable;
 import org.apache.phoenix.cache.ServerCacheClient;
 import org.apache.phoenix.coprocessor.BaseScannerRegionObserver;
+import org.apache.phoenix.coprocessor.MetaDataProtocol;
 import org.apache.phoenix.hbase.index.util.ImmutableBytesPtr;
 import org.apache.phoenix.index.PhoenixIndexCodec;
 import org.apache.phoenix.jdbc.PhoenixConnection;
@@ -105,6 +107,7 @@ public class PhoenixIndexPartialBuildMapper extends 
TableMapperhttp://git-wip-us.apache.org/repos/asf/phoenix/blob/0d0c0fea/phoenix-core/src/main/java/org/apache/phoenix/query/ConnectionQueryServicesImpl.java
--
diff --git 
a/phoenix-core/src/main/java/org/apache/phoenix/query/ConnectionQueryServicesImpl.java
 
b/phoenix-core/src/main/java/org/apache/phoenix/query/ConnectionQueryServicesImpl.java
index 5b7735e..6926c4e 100644
--- 
a/phoenix-core/src/main/java/org/apache/phoenix/query/ConnectionQueryServicesImpl.java
+++ 
b/phoenix-core/src/main/java/org/apache/phoenix/query/ConnectionQueryServicesImpl.java
@@ -176,7 +176,6 @@ import org.apache.phoenix.hbase.index.util.KeyValueBuilder;
 import org.apache.phoenix.hbase.index.util.VersionUtil;
 import org.apache.phoenix.index.PhoenixIndexBuilder;
 import org.apache.phoenix.index.PhoenixIndexCodec;
-import org.apache.phoenix.index.PhoenixTransactionalIndexer;
 import org.apache.phoenix.iterate.TableResultIterator;
 import org.apache.phoenix.iterate.TableResultIterator.RenewLeaseStatus;
 import org.apache.phoenix.jdbc.PhoenixConnection;
@@ -850,19 +849,12 @@ public class ConnectionQueryServicesImpl extends 
DelegateQueryServices implement
 && !SchemaUtil.isMetaTable(tableName)
 && !SchemaUtil.isStatsTable(tableName)) {
 if (isTransactional) {
-if 
(!descriptor.hasCoprocessor(PhoenixTransactionalIndexer.class.getName())) {
-
descriptor.addCoprocessor(PhoenixTransactionalIndexer.class.getName(), null, 
priority, null);
-}
 // For alter table, remove non transactional index 
coprocessor
 if (descriptor.hasCoprocessor(Indexer.class.getName())) {
 descriptor.removeCoprocessor(Indexer.class.getName());
 }
 } else {
 if (!descriptor.hasCoprocessor(Indexer.class.getName())) {
-// If exception on alter table to transition back to 
non transactional
-if 
(descriptor.hasCoprocessor(PhoenixTransactionalIndexer.class.getName())) {
-
descriptor.removeCoprocessor(PhoenixTransactionalIndexer.class.getName());
-}
 Map opts = 
Maps.newHashMapWithExpectedSize(1);
 opts.put(NonTxIndexBuilder.CODEC_CLASS_NAME_KEY, 
PhoenixIndexCodec.class.getName());
 Indexer.enableIndexing(descriptor, 
PhoenixIndexBuilder.class, opts, priority);

http://git-wip-us.apache.org/repos/asf/phoenix/blob/0d0c0fea/phoenix-core/src/main/java/org/apache/phoenix/schema/PTableImpl.java
--
diff --git 
a/phoenix-core/src/main/java/org/apache/phoenix/schema/PTableImpl.java 
b/phoenix-core/src/main/java/org/apache/phoenix/schema/PTableImpl.java
index 9525127..fb30bc7 100644
--- a/phoenix-core/src/main/java/org/apache/phoenix/schema/PTableImpl.java
+++ b/phoenix-core/src/main/java/org/apache/phoenix/schema/PTableImpl.java
@@ -1038,32 +1038,21 @@ public class PTableImpl implements PTable {
 @Override
 public void delete() {
 newMutations();
-// we're using the Tephra column family delete marker here to 
prevent the translation 
-// of deletes to puts by the Tephra's TransactionProcessor
-

[2/2] phoenix git commit: PHOENIX-4278 Implement pure client side transactional index maintenance (Ohad Shacham)

[jamestaylor]

PHOENIX-4278 Implement pure client side transactional index maintenance (Ohad 
Shacham)


Project: http://git-wip-us.apache.org/repos/asf/phoenix/repo
Commit: http://git-wip-us.apache.org/repos/asf/phoenix/commit/0d0c0fea
Tree: http://git-wip-us.apache.org/repos/asf/phoenix/tree/0d0c0fea
Diff: http://git-wip-us.apache.org/repos/asf/phoenix/diff/0d0c0fea

Branch: refs/heads/master
Commit: 0d0c0feaeb22080b161f26790ac526f6e57f1777
Parents: 9461d0d
Author: James Taylor 
Authored: Mon Feb 12 12:27:10 2018 -0800
Committer: James Taylor 
Committed: Mon Feb 12 15:23:17 2018 -0800

--
 .../phoenix/end2end/index/BaseIndexIT.java  |   2 +-
 .../phoenix/end2end/index/ImmutableIndexIT.java |   4 +-
 .../org/apache/phoenix/cache/HashCache.java |   1 +
 .../phoenix/cache/IndexMetaDataCache.java   |   8 +
 .../apache/phoenix/cache/ServerCacheClient.java |   7 +-
 .../org/apache/phoenix/cache/TenantCache.java   |   2 +-
 .../apache/phoenix/cache/TenantCacheImpl.java   |   4 +-
 .../apache/phoenix/compile/DeleteCompiler.java  |   3 +
 .../apache/phoenix/compile/UpsertCompiler.java  |   2 +
 .../coprocessor/MetaDataRegionObserver.java |   2 +
 .../coprocessor/ServerCachingEndpointImpl.java  |   4 +-
 .../coprocessor/ServerCachingProtocol.java  |   2 +-
 .../UngroupedAggregateRegionObserver.java   |  21 +-
 .../generated/ServerCachingProtos.java  | 117 -
 .../apache/phoenix/execute/MutationState.java   |  30 +-
 .../PhoenixTxnIndexMutationGenerator.java   | 519 +++
 .../hbase/index/covered/LocalTableState.java|   6 -
 .../phoenix/hbase/index/covered/TableState.java |   7 -
 .../index/IndexMetaDataCacheFactory.java|   7 +-
 .../apache/phoenix/index/PhoenixIndexCodec.java |   3 +-
 .../phoenix/index/PhoenixIndexMetaData.java |  12 +
 .../index/PhoenixTransactionalIndexer.java  |   9 +-
 .../phoenix/jdbc/PhoenixDatabaseMetaData.java   |   1 +
 .../apache/phoenix/join/HashCacheFactory.java   |  34 +-
 .../index/PhoenixIndexPartialBuildMapper.java   |   5 +
 .../query/ConnectionQueryServicesImpl.java  |   8 -
 .../org/apache/phoenix/schema/PTableImpl.java   |  25 +-
 .../apache/phoenix/cache/TenantCacheTest.java   |  14 +-
 .../src/main/ServerCachingService.proto |   1 +
 29 files changed, 758 insertions(+), 102 deletions(-)
--


http://git-wip-us.apache.org/repos/asf/phoenix/blob/0d0c0fea/phoenix-core/src/it/java/org/apache/phoenix/end2end/index/BaseIndexIT.java
--
diff --git 
a/phoenix-core/src/it/java/org/apache/phoenix/end2end/index/BaseIndexIT.java 
b/phoenix-core/src/it/java/org/apache/phoenix/end2end/index/BaseIndexIT.java
index b92da4a..1483c58 100644
--- a/phoenix-core/src/it/java/org/apache/phoenix/end2end/index/BaseIndexIT.java
+++ b/phoenix-core/src/it/java/org/apache/phoenix/end2end/index/BaseIndexIT.java
@@ -246,7 +246,7 @@ public abstract class BaseIndexIT extends 
ParallelStatsDisabledIT {
 PTable table = PhoenixRuntime.getTable(conn, 
Bytes.toString(tableName));
 assertTrue(table.getType() == PTableType.TABLE); // should be 
data table
 boolean hasIndexData = iterator.hasNext();
-assertFalse(hasIndexData); // should have no index data
+assertFalse(hasIndexData && !transactional); // should have no 
index data
 }
 }
 }

http://git-wip-us.apache.org/repos/asf/phoenix/blob/0d0c0fea/phoenix-core/src/it/java/org/apache/phoenix/end2end/index/ImmutableIndexIT.java
--
diff --git 
a/phoenix-core/src/it/java/org/apache/phoenix/end2end/index/ImmutableIndexIT.java
 
b/phoenix-core/src/it/java/org/apache/phoenix/end2end/index/ImmutableIndexIT.java
index e0398c7..d520824 100644
--- 
a/phoenix-core/src/it/java/org/apache/phoenix/end2end/index/ImmutableIndexIT.java
+++ 
b/phoenix-core/src/it/java/org/apache/phoenix/end2end/index/ImmutableIndexIT.java
@@ -75,6 +75,7 @@ import com.google.common.collect.Maps;
 public class ImmutableIndexIT extends BaseUniqueNamesOwnClusterIT {
 
 private final boolean localIndex;
+private final boolean transactional;
 private final String tableDDLOptions;
 
 private volatile boolean stopThreads = false;
@@ -86,6 +87,7 @@ public class ImmutableIndexIT extends 
BaseUniqueNamesOwnClusterIT {
 public ImmutableIndexIT(boolean localIndex, boolean transactional, boolean 
columnEncoded) {
 StringBuilder optionBuilder = new StringBuilder("IMMUTABLE_ROWS=true");
 this.localIndex = localIndex;
+this.transactional = transactional;
 if (!columnEncoded) {
 if (optionBuilder.length()!=0)
 optionBuilder.append(",");
@@ -250,7 +252,7 

[2/2] phoenix git commit: PHOENIX-4278 Implement pure client side transactional index maintenance (Ohad Shacham)

[jamestaylor]

PHOENIX-4278 Implement pure client side transactional index maintenance (Ohad 
Shacham)


Project: http://git-wip-us.apache.org/repos/asf/phoenix/repo
Commit: http://git-wip-us.apache.org/repos/asf/phoenix/commit/c40a18c9
Tree: http://git-wip-us.apache.org/repos/asf/phoenix/tree/c40a18c9
Diff: http://git-wip-us.apache.org/repos/asf/phoenix/diff/c40a18c9

Branch: refs/heads/4.x-HBase-1.3
Commit: c40a18c9370a8edff0b3ad22722eeb6b4a69b593
Parents: ab90709
Author: James Taylor 
Authored: Mon Feb 12 12:27:10 2018 -0800
Committer: James Taylor 
Committed: Mon Feb 12 15:19:22 2018 -0800

--
 .../phoenix/end2end/index/BaseIndexIT.java  |   2 +-
 .../phoenix/end2end/index/ImmutableIndexIT.java |   4 +-
 .../org/apache/phoenix/cache/HashCache.java |   1 +
 .../phoenix/cache/IndexMetaDataCache.java   |   8 +
 .../apache/phoenix/cache/ServerCacheClient.java |   7 +-
 .../org/apache/phoenix/cache/TenantCache.java   |   2 +-
 .../apache/phoenix/cache/TenantCacheImpl.java   |   4 +-
 .../apache/phoenix/compile/DeleteCompiler.java  |   3 +
 .../apache/phoenix/compile/UpsertCompiler.java  |   2 +
 .../coprocessor/MetaDataRegionObserver.java |   2 +
 .../coprocessor/ServerCachingEndpointImpl.java  |   4 +-
 .../coprocessor/ServerCachingProtocol.java  |   2 +-
 .../UngroupedAggregateRegionObserver.java   |  21 +-
 .../generated/ServerCachingProtos.java  | 117 -
 .../apache/phoenix/execute/MutationState.java   |  30 +-
 .../PhoenixTxnIndexMutationGenerator.java   | 519 +++
 .../hbase/index/covered/LocalTableState.java|   6 -
 .../phoenix/hbase/index/covered/TableState.java |   7 -
 .../index/IndexMetaDataCacheFactory.java|   7 +-
 .../apache/phoenix/index/PhoenixIndexCodec.java |   3 +-
 .../phoenix/index/PhoenixIndexMetaData.java |  12 +
 .../index/PhoenixTransactionalIndexer.java  |   9 +-
 .../phoenix/jdbc/PhoenixDatabaseMetaData.java   |   1 +
 .../apache/phoenix/join/HashCacheFactory.java   |  34 +-
 .../index/PhoenixIndexPartialBuildMapper.java   |   5 +
 .../query/ConnectionQueryServicesImpl.java  |   8 -
 .../org/apache/phoenix/schema/PTableImpl.java   |  25 +-
 .../apache/phoenix/cache/TenantCacheTest.java   |  14 +-
 .../src/main/ServerCachingService.proto |   1 +
 29 files changed, 758 insertions(+), 102 deletions(-)
--


http://git-wip-us.apache.org/repos/asf/phoenix/blob/c40a18c9/phoenix-core/src/it/java/org/apache/phoenix/end2end/index/BaseIndexIT.java
--
diff --git 
a/phoenix-core/src/it/java/org/apache/phoenix/end2end/index/BaseIndexIT.java 
b/phoenix-core/src/it/java/org/apache/phoenix/end2end/index/BaseIndexIT.java
index b92da4a..1483c58 100644
--- a/phoenix-core/src/it/java/org/apache/phoenix/end2end/index/BaseIndexIT.java
+++ b/phoenix-core/src/it/java/org/apache/phoenix/end2end/index/BaseIndexIT.java
@@ -246,7 +246,7 @@ public abstract class BaseIndexIT extends 
ParallelStatsDisabledIT {
 PTable table = PhoenixRuntime.getTable(conn, 
Bytes.toString(tableName));
 assertTrue(table.getType() == PTableType.TABLE); // should be 
data table
 boolean hasIndexData = iterator.hasNext();
-assertFalse(hasIndexData); // should have no index data
+assertFalse(hasIndexData && !transactional); // should have no 
index data
 }
 }
 }

http://git-wip-us.apache.org/repos/asf/phoenix/blob/c40a18c9/phoenix-core/src/it/java/org/apache/phoenix/end2end/index/ImmutableIndexIT.java
--
diff --git 
a/phoenix-core/src/it/java/org/apache/phoenix/end2end/index/ImmutableIndexIT.java
 
b/phoenix-core/src/it/java/org/apache/phoenix/end2end/index/ImmutableIndexIT.java
index e0398c7..d520824 100644
--- 
a/phoenix-core/src/it/java/org/apache/phoenix/end2end/index/ImmutableIndexIT.java
+++ 
b/phoenix-core/src/it/java/org/apache/phoenix/end2end/index/ImmutableIndexIT.java
@@ -75,6 +75,7 @@ import com.google.common.collect.Maps;
 public class ImmutableIndexIT extends BaseUniqueNamesOwnClusterIT {
 
 private final boolean localIndex;
+private final boolean transactional;
 private final String tableDDLOptions;
 
 private volatile boolean stopThreads = false;
@@ -86,6 +87,7 @@ public class ImmutableIndexIT extends 
BaseUniqueNamesOwnClusterIT {
 public ImmutableIndexIT(boolean localIndex, boolean transactional, boolean 
columnEncoded) {
 StringBuilder optionBuilder = new StringBuilder("IMMUTABLE_ROWS=true");
 this.localIndex = localIndex;
+this.transactional = transactional;
 if (!columnEncoded) {
 if (optionBuilder.length()!=0)
 optionBuilder.append(",");
@@ -250,7 

[1/2] phoenix git commit: PHOENIX-4278 Implement pure client side transactional index maintenance (Ohad Shacham)

[jamestaylor]

Repository: phoenix
Updated Branches:
  refs/heads/4.x-HBase-1.3 ab9070910 -> c40a18c93


http://git-wip-us.apache.org/repos/asf/phoenix/blob/c40a18c9/phoenix-core/src/main/java/org/apache/phoenix/mapreduce/index/PhoenixIndexPartialBuildMapper.java
--
diff --git 
a/phoenix-core/src/main/java/org/apache/phoenix/mapreduce/index/PhoenixIndexPartialBuildMapper.java
 
b/phoenix-core/src/main/java/org/apache/phoenix/mapreduce/index/PhoenixIndexPartialBuildMapper.java
index 5b85da5..7c154f0 100644
--- 
a/phoenix-core/src/main/java/org/apache/phoenix/mapreduce/index/PhoenixIndexPartialBuildMapper.java
+++ 
b/phoenix-core/src/main/java/org/apache/phoenix/mapreduce/index/PhoenixIndexPartialBuildMapper.java
@@ -33,9 +33,11 @@ import org.apache.hadoop.hbase.client.Put;
 import org.apache.hadoop.hbase.client.Result;
 import org.apache.hadoop.hbase.io.ImmutableBytesWritable;
 import org.apache.hadoop.hbase.mapreduce.TableMapper;
+import org.apache.hadoop.hbase.util.Bytes;
 import org.apache.hadoop.io.IntWritable;
 import org.apache.phoenix.cache.ServerCacheClient;
 import org.apache.phoenix.coprocessor.BaseScannerRegionObserver;
+import org.apache.phoenix.coprocessor.MetaDataProtocol;
 import org.apache.phoenix.hbase.index.util.ImmutableBytesPtr;
 import org.apache.phoenix.index.PhoenixIndexCodec;
 import org.apache.phoenix.jdbc.PhoenixConnection;
@@ -105,6 +107,7 @@ public class PhoenixIndexPartialBuildMapper extends 
TableMapperhttp://git-wip-us.apache.org/repos/asf/phoenix/blob/c40a18c9/phoenix-core/src/main/java/org/apache/phoenix/query/ConnectionQueryServicesImpl.java
--
diff --git 
a/phoenix-core/src/main/java/org/apache/phoenix/query/ConnectionQueryServicesImpl.java
 
b/phoenix-core/src/main/java/org/apache/phoenix/query/ConnectionQueryServicesImpl.java
index 5b7735e..6926c4e 100644
--- 
a/phoenix-core/src/main/java/org/apache/phoenix/query/ConnectionQueryServicesImpl.java
+++ 
b/phoenix-core/src/main/java/org/apache/phoenix/query/ConnectionQueryServicesImpl.java
@@ -176,7 +176,6 @@ import org.apache.phoenix.hbase.index.util.KeyValueBuilder;
 import org.apache.phoenix.hbase.index.util.VersionUtil;
 import org.apache.phoenix.index.PhoenixIndexBuilder;
 import org.apache.phoenix.index.PhoenixIndexCodec;
-import org.apache.phoenix.index.PhoenixTransactionalIndexer;
 import org.apache.phoenix.iterate.TableResultIterator;
 import org.apache.phoenix.iterate.TableResultIterator.RenewLeaseStatus;
 import org.apache.phoenix.jdbc.PhoenixConnection;
@@ -850,19 +849,12 @@ public class ConnectionQueryServicesImpl extends 
DelegateQueryServices implement
 && !SchemaUtil.isMetaTable(tableName)
 && !SchemaUtil.isStatsTable(tableName)) {
 if (isTransactional) {
-if 
(!descriptor.hasCoprocessor(PhoenixTransactionalIndexer.class.getName())) {
-
descriptor.addCoprocessor(PhoenixTransactionalIndexer.class.getName(), null, 
priority, null);
-}
 // For alter table, remove non transactional index 
coprocessor
 if (descriptor.hasCoprocessor(Indexer.class.getName())) {
 descriptor.removeCoprocessor(Indexer.class.getName());
 }
 } else {
 if (!descriptor.hasCoprocessor(Indexer.class.getName())) {
-// If exception on alter table to transition back to 
non transactional
-if 
(descriptor.hasCoprocessor(PhoenixTransactionalIndexer.class.getName())) {
-
descriptor.removeCoprocessor(PhoenixTransactionalIndexer.class.getName());
-}
 Map opts = 
Maps.newHashMapWithExpectedSize(1);
 opts.put(NonTxIndexBuilder.CODEC_CLASS_NAME_KEY, 
PhoenixIndexCodec.class.getName());
 Indexer.enableIndexing(descriptor, 
PhoenixIndexBuilder.class, opts, priority);

http://git-wip-us.apache.org/repos/asf/phoenix/blob/c40a18c9/phoenix-core/src/main/java/org/apache/phoenix/schema/PTableImpl.java
--
diff --git 
a/phoenix-core/src/main/java/org/apache/phoenix/schema/PTableImpl.java 
b/phoenix-core/src/main/java/org/apache/phoenix/schema/PTableImpl.java
index 9525127..fb30bc7 100644
--- a/phoenix-core/src/main/java/org/apache/phoenix/schema/PTableImpl.java
+++ b/phoenix-core/src/main/java/org/apache/phoenix/schema/PTableImpl.java
@@ -1038,32 +1038,21 @@ public class PTableImpl implements PTable {
 @Override
 public void delete() {
 newMutations();
-// we're using the Tephra column family delete marker here to 
prevent the translation 
-// of deletes to puts by the Tephra's TransactionProcessor
-  

[1/2] phoenix git commit: PHOENIX-1556 Base hash versus sort merge join decision on cost

[maryannxue]

Repository: phoenix
Updated Branches:
  refs/heads/4.x-HBase-1.2 ff4ad203b -> c73025115


http://git-wip-us.apache.org/repos/asf/phoenix/blob/c7302511/phoenix-core/src/main/java/org/apache/phoenix/execute/ClientAggregatePlan.java
--
diff --git 
a/phoenix-core/src/main/java/org/apache/phoenix/execute/ClientAggregatePlan.java
 
b/phoenix-core/src/main/java/org/apache/phoenix/execute/ClientAggregatePlan.java
index a15ab35..21cbc2d 100644
--- 
a/phoenix-core/src/main/java/org/apache/phoenix/execute/ClientAggregatePlan.java
+++ 
b/phoenix-core/src/main/java/org/apache/phoenix/execute/ClientAggregatePlan.java
@@ -35,6 +35,10 @@ import org.apache.phoenix.compile.QueryPlan;
 import org.apache.phoenix.compile.RowProjector;
 import org.apache.phoenix.compile.StatementContext;
 import org.apache.phoenix.coprocessor.BaseScannerRegionObserver;
+import org.apache.phoenix.execute.visitor.AvgRowWidthVisitor;
+import org.apache.phoenix.execute.visitor.ByteCountVisitor;
+import org.apache.phoenix.execute.visitor.QueryPlanVisitor;
+import org.apache.phoenix.execute.visitor.RowCountVisitor;
 import org.apache.phoenix.expression.Expression;
 import org.apache.phoenix.expression.OrderByExpression;
 import org.apache.phoenix.expression.aggregator.Aggregators;
@@ -90,25 +94,30 @@ public class ClientAggregatePlan extends 
ClientProcessingPlan {
 
 @Override
 public Cost getCost() {
-Long byteCount = null;
-try {
-byteCount = getEstimatedBytesToScan();
-} catch (SQLException e) {
-// ignored.
-}
-
-if (byteCount == null) {
+Double outputBytes = this.accept(new ByteCountVisitor());
+Double inputRows = this.getDelegate().accept(new RowCountVisitor());
+Double rowWidth = this.accept(new AvgRowWidthVisitor());
+if (inputRows == null || outputBytes == null || rowWidth == null) {
 return Cost.UNKNOWN;
 }
+double inputBytes = inputRows * rowWidth;
+double rowsBeforeHaving = RowCountVisitor.aggregate(
+RowCountVisitor.filter(
+inputRows.doubleValue(),
+RowCountVisitor.stripSkipScanFilter(
+context.getScan().getFilter())),
+groupBy);
+double rowsAfterHaving = RowCountVisitor.filter(rowsBeforeHaving, 
having);
+double bytesBeforeHaving = rowWidth * rowsBeforeHaving;
+double bytesAfterHaving = rowWidth * rowsAfterHaving;
 
 int parallelLevel = CostUtil.estimateParallelLevel(
 false, context.getConnection().getQueryServices());
-Cost cost = CostUtil.estimateAggregateCost(byteCount,
-groupBy, clientAggregators.getEstimatedByteSize(), 
parallelLevel);
+Cost cost = CostUtil.estimateAggregateCost(
+inputBytes, bytesBeforeHaving, groupBy, parallelLevel);
 if (!orderBy.getOrderByExpressions().isEmpty()) {
-double outputBytes = CostUtil.estimateAggregateOutputBytes(
-byteCount, groupBy, 
clientAggregators.getEstimatedByteSize());
-Cost orderByCost = CostUtil.estimateOrderByCost(outputBytes, 
parallelLevel);
+Cost orderByCost = CostUtil.estimateOrderByCost(
+bytesAfterHaving, outputBytes, parallelLevel);
 cost = cost.plus(orderByCost);
 }
 return super.getCost().plus(cost);
@@ -210,7 +219,16 @@ public class ClientAggregatePlan extends 
ClientProcessingPlan {
 public GroupBy getGroupBy() {
 return groupBy;
 }
-
+
+@Override
+public  T accept(QueryPlanVisitor visitor) {
+return visitor.visit(this);
+}
+
+public Expression getHaving() {
+return having;
+}
+
 private static class ClientGroupedAggregatingResultIterator extends 
BaseGroupedAggregatingResultIterator {
 private final List groupByExpressions;
 

http://git-wip-us.apache.org/repos/asf/phoenix/blob/c7302511/phoenix-core/src/main/java/org/apache/phoenix/execute/ClientProcessingPlan.java
--
diff --git 
a/phoenix-core/src/main/java/org/apache/phoenix/execute/ClientProcessingPlan.java
 
b/phoenix-core/src/main/java/org/apache/phoenix/execute/ClientProcessingPlan.java
index ac43919..75ba8f2 100644
--- 
a/phoenix-core/src/main/java/org/apache/phoenix/execute/ClientProcessingPlan.java
+++ 
b/phoenix-core/src/main/java/org/apache/phoenix/execute/ClientProcessingPlan.java
@@ -85,4 +85,8 @@ public abstract class ClientProcessingPlan extends 
DelegateQueryPlan {
 public FilterableStatement getStatement() {
 return statement;
 }
+
+public Expression getWhere() {
+return where;
+}
 }

http://git-wip-us.apache.org/repos/asf/phoenix/blob/c7302511/phoenix-core/src/main/java/org/apache/phoenix/execute/ClientScanPlan.java

[2/2] phoenix git commit: PHOENIX-1556 Base hash versus sort merge join decision on cost

[maryannxue]

PHOENIX-1556 Base hash versus sort merge join decision on cost


Project: http://git-wip-us.apache.org/repos/asf/phoenix/repo
Commit: http://git-wip-us.apache.org/repos/asf/phoenix/commit/c7302511
Tree: http://git-wip-us.apache.org/repos/asf/phoenix/tree/c7302511
Diff: http://git-wip-us.apache.org/repos/asf/phoenix/diff/c7302511

Branch: refs/heads/4.x-HBase-1.2
Commit: c73025115fd007240b705b5ce25fbcb37b999ec9
Parents: ff4ad20
Author: maryannxue 
Authored: Mon Feb 12 14:23:52 2018 -0800
Committer: maryannxue 
Committed: Mon Feb 12 14:23:52 2018 -0800

--
 .../phoenix/end2end/CostBasedDecisionIT.java| 420 -
 .../apache/phoenix/compile/JoinCompiler.java|  43 ++
 .../phoenix/compile/ListJarsQueryPlan.java  |   8 +-
 .../apache/phoenix/compile/QueryCompiler.java   | 449 ++-
 .../org/apache/phoenix/compile/QueryPlan.java   |   2 +
 .../apache/phoenix/compile/TraceQueryPlan.java  |   6 +
 .../apache/phoenix/execute/AggregatePlan.java   |  41 +-
 .../phoenix/execute/ClientAggregatePlan.java|  46 +-
 .../phoenix/execute/ClientProcessingPlan.java   |   4 +
 .../apache/phoenix/execute/ClientScanPlan.java  |  22 +-
 .../apache/phoenix/execute/CorrelatePlan.java   |  26 +-
 .../apache/phoenix/execute/CursorFetchPlan.java |   6 +
 .../apache/phoenix/execute/HashJoinPlan.java| 128 --
 .../execute/LiteralResultIterationPlan.java |   6 +
 .../org/apache/phoenix/execute/ScanPlan.java|  14 +-
 .../phoenix/execute/SortMergeJoinPlan.java  |  20 +-
 .../phoenix/execute/TupleProjectionPlan.java|   6 +
 .../org/apache/phoenix/execute/UnionPlan.java   |  12 +-
 .../apache/phoenix/execute/UnnestArrayPlan.java |   6 +
 .../execute/visitor/AvgRowWidthVisitor.java | 205 +
 .../execute/visitor/ByteCountVisitor.java   | 125 ++
 .../execute/visitor/QueryPlanVisitor.java   |  46 ++
 .../execute/visitor/RowCountVisitor.java| 335 ++
 .../apache/phoenix/jdbc/PhoenixStatement.java   |   6 +
 .../java/org/apache/phoenix/util/CostUtil.java  |  61 +--
 .../query/ParallelIteratorsSplitTest.java   |   6 +
 26 files changed, 1612 insertions(+), 437 deletions(-)
--


http://git-wip-us.apache.org/repos/asf/phoenix/blob/c7302511/phoenix-core/src/it/java/org/apache/phoenix/end2end/CostBasedDecisionIT.java
--
diff --git 
a/phoenix-core/src/it/java/org/apache/phoenix/end2end/CostBasedDecisionIT.java 
b/phoenix-core/src/it/java/org/apache/phoenix/end2end/CostBasedDecisionIT.java
index a3584ce..493855a 100644
--- 
a/phoenix-core/src/it/java/org/apache/phoenix/end2end/CostBasedDecisionIT.java
+++ 
b/phoenix-core/src/it/java/org/apache/phoenix/end2end/CostBasedDecisionIT.java
@@ -32,12 +32,16 @@ import org.apache.phoenix.query.QueryServices;
 import org.apache.phoenix.util.PropertiesUtil;
 import org.apache.phoenix.util.QueryUtil;
 import org.apache.phoenix.util.ReadOnlyProps;
+
 import org.junit.BeforeClass;
 import org.junit.Test;
 
 import com.google.common.collect.Maps;
 
 public class CostBasedDecisionIT extends BaseUniqueNamesOwnClusterIT {
+private final String testTable500;
+private final String testTable990;
+private final String testTable1000;
 
 @BeforeClass
 public static void doSetup() throws Exception {
@@ -46,9 +50,16 @@ public class CostBasedDecisionIT extends 
BaseUniqueNamesOwnClusterIT {
 props.put(QueryServices.STATS_UPDATE_FREQ_MS_ATTRIB, Long.toString(5));
 props.put(QueryServices.USE_STATS_FOR_PARALLELIZATION, 
Boolean.toString(true));
 props.put(QueryServices.COST_BASED_OPTIMIZER_ENABLED, 
Boolean.toString(true));
+props.put(QueryServices.MAX_SERVER_CACHE_SIZE_ATTRIB, 
Long.toString(15));
 setUpTestDriver(new ReadOnlyProps(props.entrySet().iterator()));
 }
 
+public CostBasedDecisionIT() throws Exception {
+testTable500 = initTestTableValues(500);
+testTable990 = initTestTableValues(990);
+testTable1000 = initTestTableValues(1000);
+}
+
 @Test
 public void testCostOverridesStaticPlanOrdering1() throws Exception {
 Properties props = PropertiesUtil.deepCopy(TEST_PROPERTIES);
@@ -64,10 +75,7 @@ public class CostBasedDecisionIT extends 
BaseUniqueNamesOwnClusterIT {
 
 String query = "SELECT rowkey, c1, c2 FROM " + tableName + " where 
c1 LIKE 'X0%' ORDER BY rowkey";
 // Use the data table plan that opts out order-by when stats are 
not available.
-ResultSet rs = conn.createStatement().executeQuery("explain " + 
query);
-String plan = QueryUtil.getExplainPlan(rs);
-assertTrue("Expected 'FULL SCAN' in the plan:\n" + plan + ".",
-plan.contains("FULL SCAN"));
+

[1/2] phoenix git commit: PHOENIX-1556 Base hash versus sort merge join decision on cost

[maryannxue]

Repository: phoenix
Updated Branches:
  refs/heads/4.x-HBase-1.3 74b2dcc53 -> ab9070910


http://git-wip-us.apache.org/repos/asf/phoenix/blob/ab907091/phoenix-core/src/main/java/org/apache/phoenix/execute/ClientAggregatePlan.java
--
diff --git 
a/phoenix-core/src/main/java/org/apache/phoenix/execute/ClientAggregatePlan.java
 
b/phoenix-core/src/main/java/org/apache/phoenix/execute/ClientAggregatePlan.java
index a15ab35..21cbc2d 100644
--- 
a/phoenix-core/src/main/java/org/apache/phoenix/execute/ClientAggregatePlan.java
+++ 
b/phoenix-core/src/main/java/org/apache/phoenix/execute/ClientAggregatePlan.java
@@ -35,6 +35,10 @@ import org.apache.phoenix.compile.QueryPlan;
 import org.apache.phoenix.compile.RowProjector;
 import org.apache.phoenix.compile.StatementContext;
 import org.apache.phoenix.coprocessor.BaseScannerRegionObserver;
+import org.apache.phoenix.execute.visitor.AvgRowWidthVisitor;
+import org.apache.phoenix.execute.visitor.ByteCountVisitor;
+import org.apache.phoenix.execute.visitor.QueryPlanVisitor;
+import org.apache.phoenix.execute.visitor.RowCountVisitor;
 import org.apache.phoenix.expression.Expression;
 import org.apache.phoenix.expression.OrderByExpression;
 import org.apache.phoenix.expression.aggregator.Aggregators;
@@ -90,25 +94,30 @@ public class ClientAggregatePlan extends 
ClientProcessingPlan {
 
 @Override
 public Cost getCost() {
-Long byteCount = null;
-try {
-byteCount = getEstimatedBytesToScan();
-} catch (SQLException e) {
-// ignored.
-}
-
-if (byteCount == null) {
+Double outputBytes = this.accept(new ByteCountVisitor());
+Double inputRows = this.getDelegate().accept(new RowCountVisitor());
+Double rowWidth = this.accept(new AvgRowWidthVisitor());
+if (inputRows == null || outputBytes == null || rowWidth == null) {
 return Cost.UNKNOWN;
 }
+double inputBytes = inputRows * rowWidth;
+double rowsBeforeHaving = RowCountVisitor.aggregate(
+RowCountVisitor.filter(
+inputRows.doubleValue(),
+RowCountVisitor.stripSkipScanFilter(
+context.getScan().getFilter())),
+groupBy);
+double rowsAfterHaving = RowCountVisitor.filter(rowsBeforeHaving, 
having);
+double bytesBeforeHaving = rowWidth * rowsBeforeHaving;
+double bytesAfterHaving = rowWidth * rowsAfterHaving;
 
 int parallelLevel = CostUtil.estimateParallelLevel(
 false, context.getConnection().getQueryServices());
-Cost cost = CostUtil.estimateAggregateCost(byteCount,
-groupBy, clientAggregators.getEstimatedByteSize(), 
parallelLevel);
+Cost cost = CostUtil.estimateAggregateCost(
+inputBytes, bytesBeforeHaving, groupBy, parallelLevel);
 if (!orderBy.getOrderByExpressions().isEmpty()) {
-double outputBytes = CostUtil.estimateAggregateOutputBytes(
-byteCount, groupBy, 
clientAggregators.getEstimatedByteSize());
-Cost orderByCost = CostUtil.estimateOrderByCost(outputBytes, 
parallelLevel);
+Cost orderByCost = CostUtil.estimateOrderByCost(
+bytesAfterHaving, outputBytes, parallelLevel);
 cost = cost.plus(orderByCost);
 }
 return super.getCost().plus(cost);
@@ -210,7 +219,16 @@ public class ClientAggregatePlan extends 
ClientProcessingPlan {
 public GroupBy getGroupBy() {
 return groupBy;
 }
-
+
+@Override
+public  T accept(QueryPlanVisitor visitor) {
+return visitor.visit(this);
+}
+
+public Expression getHaving() {
+return having;
+}
+
 private static class ClientGroupedAggregatingResultIterator extends 
BaseGroupedAggregatingResultIterator {
 private final List groupByExpressions;
 

http://git-wip-us.apache.org/repos/asf/phoenix/blob/ab907091/phoenix-core/src/main/java/org/apache/phoenix/execute/ClientProcessingPlan.java
--
diff --git 
a/phoenix-core/src/main/java/org/apache/phoenix/execute/ClientProcessingPlan.java
 
b/phoenix-core/src/main/java/org/apache/phoenix/execute/ClientProcessingPlan.java
index ac43919..75ba8f2 100644
--- 
a/phoenix-core/src/main/java/org/apache/phoenix/execute/ClientProcessingPlan.java
+++ 
b/phoenix-core/src/main/java/org/apache/phoenix/execute/ClientProcessingPlan.java
@@ -85,4 +85,8 @@ public abstract class ClientProcessingPlan extends 
DelegateQueryPlan {
 public FilterableStatement getStatement() {
 return statement;
 }
+
+public Expression getWhere() {
+return where;
+}
 }

http://git-wip-us.apache.org/repos/asf/phoenix/blob/ab907091/phoenix-core/src/main/java/org/apache/phoenix/execute/ClientScanPlan.java

[2/2] phoenix git commit: PHOENIX-1556 Base hash versus sort merge join decision on cost

[maryannxue]

PHOENIX-1556 Base hash versus sort merge join decision on cost


Project: http://git-wip-us.apache.org/repos/asf/phoenix/repo
Commit: http://git-wip-us.apache.org/repos/asf/phoenix/commit/ab907091
Tree: http://git-wip-us.apache.org/repos/asf/phoenix/tree/ab907091
Diff: http://git-wip-us.apache.org/repos/asf/phoenix/diff/ab907091

Branch: refs/heads/4.x-HBase-1.3
Commit: ab90709104f775cf8fbf7e23079e50bbd8f570e9
Parents: 74b2dcc
Author: maryannxue 
Authored: Mon Feb 12 14:22:54 2018 -0800
Committer: maryannxue 
Committed: Mon Feb 12 14:22:54 2018 -0800

--
 .../phoenix/end2end/CostBasedDecisionIT.java| 420 -
 .../apache/phoenix/compile/JoinCompiler.java|  43 ++
 .../phoenix/compile/ListJarsQueryPlan.java  |   8 +-
 .../apache/phoenix/compile/QueryCompiler.java   | 449 ++-
 .../org/apache/phoenix/compile/QueryPlan.java   |   2 +
 .../apache/phoenix/compile/TraceQueryPlan.java  |   6 +
 .../apache/phoenix/execute/AggregatePlan.java   |  41 +-
 .../phoenix/execute/ClientAggregatePlan.java|  46 +-
 .../phoenix/execute/ClientProcessingPlan.java   |   4 +
 .../apache/phoenix/execute/ClientScanPlan.java  |  22 +-
 .../apache/phoenix/execute/CorrelatePlan.java   |  26 +-
 .../apache/phoenix/execute/CursorFetchPlan.java |   6 +
 .../apache/phoenix/execute/HashJoinPlan.java| 128 --
 .../execute/LiteralResultIterationPlan.java |   6 +
 .../org/apache/phoenix/execute/ScanPlan.java|  14 +-
 .../phoenix/execute/SortMergeJoinPlan.java  |  20 +-
 .../phoenix/execute/TupleProjectionPlan.java|   6 +
 .../org/apache/phoenix/execute/UnionPlan.java   |  12 +-
 .../apache/phoenix/execute/UnnestArrayPlan.java |   6 +
 .../execute/visitor/AvgRowWidthVisitor.java | 205 +
 .../execute/visitor/ByteCountVisitor.java   | 125 ++
 .../execute/visitor/QueryPlanVisitor.java   |  46 ++
 .../execute/visitor/RowCountVisitor.java| 335 ++
 .../apache/phoenix/jdbc/PhoenixStatement.java   |   6 +
 .../java/org/apache/phoenix/util/CostUtil.java  |  61 +--
 .../query/ParallelIteratorsSplitTest.java   |   6 +
 26 files changed, 1612 insertions(+), 437 deletions(-)
--


http://git-wip-us.apache.org/repos/asf/phoenix/blob/ab907091/phoenix-core/src/it/java/org/apache/phoenix/end2end/CostBasedDecisionIT.java
--
diff --git 
a/phoenix-core/src/it/java/org/apache/phoenix/end2end/CostBasedDecisionIT.java 
b/phoenix-core/src/it/java/org/apache/phoenix/end2end/CostBasedDecisionIT.java
index a3584ce..493855a 100644
--- 
a/phoenix-core/src/it/java/org/apache/phoenix/end2end/CostBasedDecisionIT.java
+++ 
b/phoenix-core/src/it/java/org/apache/phoenix/end2end/CostBasedDecisionIT.java
@@ -32,12 +32,16 @@ import org.apache.phoenix.query.QueryServices;
 import org.apache.phoenix.util.PropertiesUtil;
 import org.apache.phoenix.util.QueryUtil;
 import org.apache.phoenix.util.ReadOnlyProps;
+
 import org.junit.BeforeClass;
 import org.junit.Test;
 
 import com.google.common.collect.Maps;
 
 public class CostBasedDecisionIT extends BaseUniqueNamesOwnClusterIT {
+private final String testTable500;
+private final String testTable990;
+private final String testTable1000;
 
 @BeforeClass
 public static void doSetup() throws Exception {
@@ -46,9 +50,16 @@ public class CostBasedDecisionIT extends 
BaseUniqueNamesOwnClusterIT {
 props.put(QueryServices.STATS_UPDATE_FREQ_MS_ATTRIB, Long.toString(5));
 props.put(QueryServices.USE_STATS_FOR_PARALLELIZATION, 
Boolean.toString(true));
 props.put(QueryServices.COST_BASED_OPTIMIZER_ENABLED, 
Boolean.toString(true));
+props.put(QueryServices.MAX_SERVER_CACHE_SIZE_ATTRIB, 
Long.toString(15));
 setUpTestDriver(new ReadOnlyProps(props.entrySet().iterator()));
 }
 
+public CostBasedDecisionIT() throws Exception {
+testTable500 = initTestTableValues(500);
+testTable990 = initTestTableValues(990);
+testTable1000 = initTestTableValues(1000);
+}
+
 @Test
 public void testCostOverridesStaticPlanOrdering1() throws Exception {
 Properties props = PropertiesUtil.deepCopy(TEST_PROPERTIES);
@@ -64,10 +75,7 @@ public class CostBasedDecisionIT extends 
BaseUniqueNamesOwnClusterIT {
 
 String query = "SELECT rowkey, c1, c2 FROM " + tableName + " where 
c1 LIKE 'X0%' ORDER BY rowkey";
 // Use the data table plan that opts out order-by when stats are 
not available.
-ResultSet rs = conn.createStatement().executeQuery("explain " + 
query);
-String plan = QueryUtil.getExplainPlan(rs);
-assertTrue("Expected 'FULL SCAN' in the plan:\n" + plan + ".",
-plan.contains("FULL SCAN"));
+

[1/2] phoenix git commit: PHOENIX-1556 Base hash versus sort merge join decision on cost

[maryannxue]

Repository: phoenix
Updated Branches:
  refs/heads/master a6bf7350d -> 9461d0d6a


http://git-wip-us.apache.org/repos/asf/phoenix/blob/9461d0d6/phoenix-core/src/main/java/org/apache/phoenix/execute/ClientAggregatePlan.java
--
diff --git 
a/phoenix-core/src/main/java/org/apache/phoenix/execute/ClientAggregatePlan.java
 
b/phoenix-core/src/main/java/org/apache/phoenix/execute/ClientAggregatePlan.java
index a15ab35..21cbc2d 100644
--- 
a/phoenix-core/src/main/java/org/apache/phoenix/execute/ClientAggregatePlan.java
+++ 
b/phoenix-core/src/main/java/org/apache/phoenix/execute/ClientAggregatePlan.java
@@ -35,6 +35,10 @@ import org.apache.phoenix.compile.QueryPlan;
 import org.apache.phoenix.compile.RowProjector;
 import org.apache.phoenix.compile.StatementContext;
 import org.apache.phoenix.coprocessor.BaseScannerRegionObserver;
+import org.apache.phoenix.execute.visitor.AvgRowWidthVisitor;
+import org.apache.phoenix.execute.visitor.ByteCountVisitor;
+import org.apache.phoenix.execute.visitor.QueryPlanVisitor;
+import org.apache.phoenix.execute.visitor.RowCountVisitor;
 import org.apache.phoenix.expression.Expression;
 import org.apache.phoenix.expression.OrderByExpression;
 import org.apache.phoenix.expression.aggregator.Aggregators;
@@ -90,25 +94,30 @@ public class ClientAggregatePlan extends 
ClientProcessingPlan {
 
 @Override
 public Cost getCost() {
-Long byteCount = null;
-try {
-byteCount = getEstimatedBytesToScan();
-} catch (SQLException e) {
-// ignored.
-}
-
-if (byteCount == null) {
+Double outputBytes = this.accept(new ByteCountVisitor());
+Double inputRows = this.getDelegate().accept(new RowCountVisitor());
+Double rowWidth = this.accept(new AvgRowWidthVisitor());
+if (inputRows == null || outputBytes == null || rowWidth == null) {
 return Cost.UNKNOWN;
 }
+double inputBytes = inputRows * rowWidth;
+double rowsBeforeHaving = RowCountVisitor.aggregate(
+RowCountVisitor.filter(
+inputRows.doubleValue(),
+RowCountVisitor.stripSkipScanFilter(
+context.getScan().getFilter())),
+groupBy);
+double rowsAfterHaving = RowCountVisitor.filter(rowsBeforeHaving, 
having);
+double bytesBeforeHaving = rowWidth * rowsBeforeHaving;
+double bytesAfterHaving = rowWidth * rowsAfterHaving;
 
 int parallelLevel = CostUtil.estimateParallelLevel(
 false, context.getConnection().getQueryServices());
-Cost cost = CostUtil.estimateAggregateCost(byteCount,
-groupBy, clientAggregators.getEstimatedByteSize(), 
parallelLevel);
+Cost cost = CostUtil.estimateAggregateCost(
+inputBytes, bytesBeforeHaving, groupBy, parallelLevel);
 if (!orderBy.getOrderByExpressions().isEmpty()) {
-double outputBytes = CostUtil.estimateAggregateOutputBytes(
-byteCount, groupBy, 
clientAggregators.getEstimatedByteSize());
-Cost orderByCost = CostUtil.estimateOrderByCost(outputBytes, 
parallelLevel);
+Cost orderByCost = CostUtil.estimateOrderByCost(
+bytesAfterHaving, outputBytes, parallelLevel);
 cost = cost.plus(orderByCost);
 }
 return super.getCost().plus(cost);
@@ -210,7 +219,16 @@ public class ClientAggregatePlan extends 
ClientProcessingPlan {
 public GroupBy getGroupBy() {
 return groupBy;
 }
-
+
+@Override
+public  T accept(QueryPlanVisitor visitor) {
+return visitor.visit(this);
+}
+
+public Expression getHaving() {
+return having;
+}
+
 private static class ClientGroupedAggregatingResultIterator extends 
BaseGroupedAggregatingResultIterator {
 private final List groupByExpressions;
 

http://git-wip-us.apache.org/repos/asf/phoenix/blob/9461d0d6/phoenix-core/src/main/java/org/apache/phoenix/execute/ClientProcessingPlan.java
--
diff --git 
a/phoenix-core/src/main/java/org/apache/phoenix/execute/ClientProcessingPlan.java
 
b/phoenix-core/src/main/java/org/apache/phoenix/execute/ClientProcessingPlan.java
index ac43919..75ba8f2 100644
--- 
a/phoenix-core/src/main/java/org/apache/phoenix/execute/ClientProcessingPlan.java
+++ 
b/phoenix-core/src/main/java/org/apache/phoenix/execute/ClientProcessingPlan.java
@@ -85,4 +85,8 @@ public abstract class ClientProcessingPlan extends 
DelegateQueryPlan {
 public FilterableStatement getStatement() {
 return statement;
 }
+
+public Expression getWhere() {
+return where;
+}
 }

http://git-wip-us.apache.org/repos/asf/phoenix/blob/9461d0d6/phoenix-core/src/main/java/org/apache/phoenix/execute/ClientScanPlan.java

[2/2] phoenix git commit: PHOENIX-1556 Base hash versus sort merge join decision on cost

[maryannxue]

PHOENIX-1556 Base hash versus sort merge join decision on cost


Project: http://git-wip-us.apache.org/repos/asf/phoenix/repo
Commit: http://git-wip-us.apache.org/repos/asf/phoenix/commit/9461d0d6
Tree: http://git-wip-us.apache.org/repos/asf/phoenix/tree/9461d0d6
Diff: http://git-wip-us.apache.org/repos/asf/phoenix/diff/9461d0d6

Branch: refs/heads/master
Commit: 9461d0d6a299bb3cbcf53905d7e9b73895a99299
Parents: a6bf735
Author: maryannxue 
Authored: Mon Feb 12 14:07:30 2018 -0800
Committer: maryannxue 
Committed: Mon Feb 12 14:07:30 2018 -0800

--
 .../phoenix/end2end/CostBasedDecisionIT.java| 420 -
 .../apache/phoenix/compile/JoinCompiler.java|  43 ++
 .../phoenix/compile/ListJarsQueryPlan.java  |   8 +-
 .../apache/phoenix/compile/QueryCompiler.java   | 449 ++-
 .../org/apache/phoenix/compile/QueryPlan.java   |   2 +
 .../apache/phoenix/compile/TraceQueryPlan.java  |   6 +
 .../apache/phoenix/execute/AggregatePlan.java   |  41 +-
 .../phoenix/execute/ClientAggregatePlan.java|  46 +-
 .../phoenix/execute/ClientProcessingPlan.java   |   4 +
 .../apache/phoenix/execute/ClientScanPlan.java  |  22 +-
 .../apache/phoenix/execute/CorrelatePlan.java   |  26 +-
 .../apache/phoenix/execute/CursorFetchPlan.java |   6 +
 .../apache/phoenix/execute/HashJoinPlan.java| 128 --
 .../execute/LiteralResultIterationPlan.java |   6 +
 .../org/apache/phoenix/execute/ScanPlan.java|  14 +-
 .../phoenix/execute/SortMergeJoinPlan.java  |  20 +-
 .../phoenix/execute/TupleProjectionPlan.java|   6 +
 .../org/apache/phoenix/execute/UnionPlan.java   |  12 +-
 .../apache/phoenix/execute/UnnestArrayPlan.java |   6 +
 .../execute/visitor/AvgRowWidthVisitor.java | 205 +
 .../execute/visitor/ByteCountVisitor.java   | 125 ++
 .../execute/visitor/QueryPlanVisitor.java   |  46 ++
 .../execute/visitor/RowCountVisitor.java| 335 ++
 .../apache/phoenix/jdbc/PhoenixStatement.java   |   6 +
 .../java/org/apache/phoenix/util/CostUtil.java  |  61 +--
 .../query/ParallelIteratorsSplitTest.java   |   6 +
 26 files changed, 1612 insertions(+), 437 deletions(-)
--


http://git-wip-us.apache.org/repos/asf/phoenix/blob/9461d0d6/phoenix-core/src/it/java/org/apache/phoenix/end2end/CostBasedDecisionIT.java
--
diff --git 
a/phoenix-core/src/it/java/org/apache/phoenix/end2end/CostBasedDecisionIT.java 
b/phoenix-core/src/it/java/org/apache/phoenix/end2end/CostBasedDecisionIT.java
index a3584ce..493855a 100644
--- 
a/phoenix-core/src/it/java/org/apache/phoenix/end2end/CostBasedDecisionIT.java
+++ 
b/phoenix-core/src/it/java/org/apache/phoenix/end2end/CostBasedDecisionIT.java
@@ -32,12 +32,16 @@ import org.apache.phoenix.query.QueryServices;
 import org.apache.phoenix.util.PropertiesUtil;
 import org.apache.phoenix.util.QueryUtil;
 import org.apache.phoenix.util.ReadOnlyProps;
+
 import org.junit.BeforeClass;
 import org.junit.Test;
 
 import com.google.common.collect.Maps;
 
 public class CostBasedDecisionIT extends BaseUniqueNamesOwnClusterIT {
+private final String testTable500;
+private final String testTable990;
+private final String testTable1000;
 
 @BeforeClass
 public static void doSetup() throws Exception {
@@ -46,9 +50,16 @@ public class CostBasedDecisionIT extends 
BaseUniqueNamesOwnClusterIT {
 props.put(QueryServices.STATS_UPDATE_FREQ_MS_ATTRIB, Long.toString(5));
 props.put(QueryServices.USE_STATS_FOR_PARALLELIZATION, 
Boolean.toString(true));
 props.put(QueryServices.COST_BASED_OPTIMIZER_ENABLED, 
Boolean.toString(true));
+props.put(QueryServices.MAX_SERVER_CACHE_SIZE_ATTRIB, 
Long.toString(15));
 setUpTestDriver(new ReadOnlyProps(props.entrySet().iterator()));
 }
 
+public CostBasedDecisionIT() throws Exception {
+testTable500 = initTestTableValues(500);
+testTable990 = initTestTableValues(990);
+testTable1000 = initTestTableValues(1000);
+}
+
 @Test
 public void testCostOverridesStaticPlanOrdering1() throws Exception {
 Properties props = PropertiesUtil.deepCopy(TEST_PROPERTIES);
@@ -64,10 +75,7 @@ public class CostBasedDecisionIT extends 
BaseUniqueNamesOwnClusterIT {
 
 String query = "SELECT rowkey, c1, c2 FROM " + tableName + " where 
c1 LIKE 'X0%' ORDER BY rowkey";
 // Use the data table plan that opts out order-by when stats are 
not available.
-ResultSet rs = conn.createStatement().executeQuery("explain " + 
query);
-String plan = QueryUtil.getExplainPlan(rs);
-assertTrue("Expected 'FULL SCAN' in the plan:\n" + plan + ".",
-plan.contains("FULL SCAN"));
+

Build failed in Jenkins: Phoenix-4.x-HBase-1.3 #35

[Apache Jenkins Server]

See 

--
Started by an SCM change
[EnvInject] - Loading node environment variables.
Building remotely on H22 (ubuntu xenial) in workspace 

java.io.IOException: Failed to mkdirs: 

at hudson.FilePath.mkdirs(FilePath.java:1170)
at hudson.model.AbstractProject.checkout(AbstractProject.java:1200)
at 
hudson.model.AbstractBuild$AbstractBuildExecution.defaultCheckout(AbstractBuild.java:574)
at jenkins.scm.SCMCheckoutStrategy.checkout(SCMCheckoutStrategy.java:86)
at 
hudson.model.AbstractBuild$AbstractBuildExecution.run(AbstractBuild.java:499)
at hudson.model.Run.execute(Run.java:1724)
at hudson.model.FreeStyleBuild.run(FreeStyleBuild.java:43)
at hudson.model.ResourceController.execute(ResourceController.java:97)
at hudson.model.Executor.run(Executor.java:421)
Retrying after 10 seconds
java.io.IOException: Failed to mkdirs: 

at hudson.FilePath.mkdirs(FilePath.java:1170)
at hudson.model.AbstractProject.checkout(AbstractProject.java:1200)
at 
hudson.model.AbstractBuild$AbstractBuildExecution.defaultCheckout(AbstractBuild.java:574)
at jenkins.scm.SCMCheckoutStrategy.checkout(SCMCheckoutStrategy.java:86)
at 
hudson.model.AbstractBuild$AbstractBuildExecution.run(AbstractBuild.java:499)
at hudson.model.Run.execute(Run.java:1724)
at hudson.model.FreeStyleBuild.run(FreeStyleBuild.java:43)
at hudson.model.ResourceController.execute(ResourceController.java:97)
at hudson.model.Executor.run(Executor.java:421)
Retrying after 10 seconds
java.io.IOException: Failed to mkdirs: 

at hudson.FilePath.mkdirs(FilePath.java:1170)
at hudson.model.AbstractProject.checkout(AbstractProject.java:1200)
at 
hudson.model.AbstractBuild$AbstractBuildExecution.defaultCheckout(AbstractBuild.java:574)
at jenkins.scm.SCMCheckoutStrategy.checkout(SCMCheckoutStrategy.java:86)
at 
hudson.model.AbstractBuild$AbstractBuildExecution.run(AbstractBuild.java:499)
at hudson.model.Run.execute(Run.java:1724)
at hudson.model.FreeStyleBuild.run(FreeStyleBuild.java:43)
at hudson.model.ResourceController.execute(ResourceController.java:97)
at hudson.model.Executor.run(Executor.java:421)
Archiving artifacts
ERROR: Build step failed with exception
Also:   hudson.remoting.Channel$CallSiteStackTrace: Remote call to H22
at 
hudson.remoting.Channel.attachCallSiteStackTrace(Channel.java:1693)
at hudson.remoting.UserResponse.retrieve(UserRequest.java:310)
at hudson.remoting.Channel.call(Channel.java:908)
at hudson.FilePath.act(FilePath.java:986)
at hudson.FilePath.act(FilePath.java:975)
at 
hudson.tasks.ArtifactArchiver.perform(ArtifactArchiver.java:243)
at 
hudson.tasks.BuildStepCompatibilityLayer.perform(BuildStepCompatibilityLayer.java:81)
at 
hudson.tasks.BuildStepMonitor$1.perform(BuildStepMonitor.java:20)
at 
hudson.model.AbstractBuild$AbstractBuildExecution.perform(AbstractBuild.java:744)
at 
hudson.model.AbstractBuild$AbstractBuildExecution.performAllBuildSteps(AbstractBuild.java:690)
at hudson.model.Build$BuildExecution.post2(Build.java:186)
at 
hudson.model.AbstractBuild$AbstractBuildExecution.post(AbstractBuild.java:635)
at hudson.model.Run.execute(Run.java:1749)
at hudson.model.FreeStyleBuild.run(FreeStyleBuild.java:43)
at 
hudson.model.ResourceController.execute(ResourceController.java:97)
at hudson.model.Executor.run(Executor.java:421)
 does not exist.
at 
org.apache.tools.ant.types.AbstractFileSet.getDirectoryScanner(AbstractFileSet.java:483)
at 
org.apache.tools.ant.types.AbstractFileSet.getDirectoryScanner(AbstractFileSet.java:460)
at 
hudson.tasks.ArtifactArchiver$ListFiles.invoke(ArtifactArchiver.java:298)
at 
hudson.tasks.ArtifactArchiver$ListFiles.invoke(ArtifactArchiver.java:278)
at hudson.FilePath$FileCallableWrapper.call(FilePath.java:2760)
at hudson.remoting.UserRequest.perform(UserRequest.java:207)
at hudson.remoting.UserRequest.perform(UserRequest.java:53)
at hudson.remoting.Request$2.run(Request.java:358)
at 
hudson.remoting.InterceptingExecutorService$1.call(InterceptingExecutorService.java:72)
at java.util.concurrent.FutureTask.run(FutureTask.java:266)
at 

phoenix git commit: PHOENIX-4549 Pherf - Column override and sequenced index creation support

[mujtaba]

Repository: phoenix
Updated Branches:
  refs/heads/4.x-HBase-1.3 3d67e8d5c -> 74b2dcc53


PHOENIX-4549 Pherf - Column override and sequenced index creation support


Project: http://git-wip-us.apache.org/repos/asf/phoenix/repo
Commit: http://git-wip-us.apache.org/repos/asf/phoenix/commit/74b2dcc5
Tree: http://git-wip-us.apache.org/repos/asf/phoenix/tree/74b2dcc5
Diff: http://git-wip-us.apache.org/repos/asf/phoenix/diff/74b2dcc5

Branch: refs/heads/4.x-HBase-1.3
Commit: 74b2dcc53e1f26e3c09ca5b8444530fb3f21b8f1
Parents: 3d67e8d
Author: Mujtaba 
Authored: Mon Feb 12 13:32:23 2018 -0800
Committer: Mujtaba 
Committed: Mon Feb 12 13:32:23 2018 -0800

--
 .../phoenix/pherf/configuration/Column.java |  23 +--
 .../pherf/configuration/DataTypeMapping.java|   6 +-
 .../phoenix/pherf/configuration/Query.java  |  25 
 .../phoenix/pherf/configuration/Scenario.java   |   3 +-
 .../phoenix/pherf/result/QueryResult.java   |  10 +-
 .../phoenix/pherf/result/ResultManager.java |  18 ++-
 .../apache/phoenix/pherf/result/ResultUtil.java |   9 +-
 .../phoenix/pherf/rules/RulesApplier.java   | 142 +++
 .../phoenix/pherf/schema/SchemaReader.java  |   2 +-
 .../apache/phoenix/pherf/util/PhoenixUtil.java  |  44 +-
 .../pherf/workload/MultiThreadedRunner.java |  35 -
 .../phoenix/pherf/workload/QueryExecutor.java   |  36 ++---
 .../phoenix/pherf/workload/WriteWorkload.java   |  54 ++-
 .../scenario/prod_test_unsalted_scenario.xml|  14 +-
 .../org/apache/phoenix/pherf/ColumnTest.java|   3 +
 .../phoenix/pherf/ConfigurationParserTest.java  |   4 +-
 .../apache/phoenix/pherf/RuleGeneratorTest.java | 107 +-
 .../test/resources/datamodel/test_schema.sql|   4 +
 .../test/resources/scenario/test_scenario.xml   |  48 ++-
 19 files changed, 484 insertions(+), 103 deletions(-)
--


http://git-wip-us.apache.org/repos/asf/phoenix/blob/74b2dcc5/phoenix-pherf/src/main/java/org/apache/phoenix/pherf/configuration/Column.java
--
diff --git 
a/phoenix-pherf/src/main/java/org/apache/phoenix/pherf/configuration/Column.java
 
b/phoenix-pherf/src/main/java/org/apache/phoenix/pherf/configuration/Column.java
index 7c9e180..0d64a39 100644
--- 
a/phoenix-pherf/src/main/java/org/apache/phoenix/pherf/configuration/Column.java
+++ 
b/phoenix-pherf/src/main/java/org/apache/phoenix/pherf/configuration/Column.java
@@ -28,7 +28,8 @@ public class Column {
private String name;
 private String prefix;
 private DataSequence dataSequence;
-private int length, minValue, maxValue, precision;
+private int length, precision;
+private long minValue, maxValue;
 private int nullChance;
 private boolean userDefined;
 private List dataValues;
@@ -40,8 +41,8 @@ public class Column {
 // Initialize int to negative value so we can distinguish 0 in 
mutations
 // Object fields can be detected with null
 this.length = Integer.MIN_VALUE;
-this.minValue = Integer.MIN_VALUE;
-this.maxValue = Integer.MIN_VALUE;
+this.minValue = Long.MIN_VALUE;
+this.maxValue = Long.MIN_VALUE;
 this.precision = Integer.MIN_VALUE;
 this.nullChance = Integer.MIN_VALUE;
 this.userDefined = false;
@@ -84,6 +85,10 @@ public class Column {
public int getLength() {
return length;
}
+   
+   public int getLengthExcludingPrefix() {
+   return (this.getPrefix() == null) ? this.length : this.length - 
this.getPrefix().length();
+   }
 
public void setLength(int length) {
this.length = length;
@@ -97,19 +102,19 @@ public class Column {
this.type = type;
}
 
-public int getMinValue() {
+public long getMinValue() {
 return minValue;
 }
 
-public void setMinValue(int minValue) {
+public void setMinValue(long minValue) {
 this.minValue = minValue;
 }
 
-public int getMaxValue() {
+public long getMaxValue() {
 return maxValue;
 }
 
-public void setMaxValue(int maxValue) {
+public void setMaxValue(long maxValue) {
 this.maxValue = maxValue;
 }
 
@@ -134,11 +139,11 @@ public class Column {
  *   obj contains only the fields you want to mutate this 
object into.
  */
 public void mutate(Column column) {
-if (column.getMinValue() != Integer.MIN_VALUE) {
+if (column.getMinValue() != Long.MIN_VALUE) {
 setMinValue(column.getMinValue());
 }
 
-if (column.getMaxValue() != Integer.MIN_VALUE) {
+if (column.getMaxValue() != Long.MIN_VALUE) {
 setMaxValue(column.getMaxValue());
 }
 

phoenix git commit: PHOENIX-4549 Pherf - Column override and sequenced index creation support

[mujtaba]

Repository: phoenix
Updated Branches:
  refs/heads/4.x-HBase-1.1 66fc54c68 -> aef828d00


PHOENIX-4549 Pherf - Column override and sequenced index creation support


Project: http://git-wip-us.apache.org/repos/asf/phoenix/repo
Commit: http://git-wip-us.apache.org/repos/asf/phoenix/commit/aef828d0
Tree: http://git-wip-us.apache.org/repos/asf/phoenix/tree/aef828d0
Diff: http://git-wip-us.apache.org/repos/asf/phoenix/diff/aef828d0

Branch: refs/heads/4.x-HBase-1.1
Commit: aef828d00080f714a228233d72bd8e1919ea0467
Parents: 66fc54c
Author: Mujtaba 
Authored: Mon Feb 12 13:30:12 2018 -0800
Committer: Mujtaba 
Committed: Mon Feb 12 13:30:12 2018 -0800

--
 .../phoenix/pherf/configuration/Column.java |  23 +--
 .../pherf/configuration/DataTypeMapping.java|   6 +-
 .../phoenix/pherf/configuration/Query.java  |  25 
 .../phoenix/pherf/configuration/Scenario.java   |   3 +-
 .../phoenix/pherf/result/QueryResult.java   |  10 +-
 .../phoenix/pherf/result/ResultManager.java |  18 ++-
 .../apache/phoenix/pherf/result/ResultUtil.java |   9 +-
 .../phoenix/pherf/rules/RulesApplier.java   | 142 +++
 .../phoenix/pherf/schema/SchemaReader.java  |   2 +-
 .../apache/phoenix/pherf/util/PhoenixUtil.java  |  44 +-
 .../pherf/workload/MultiThreadedRunner.java |  35 -
 .../phoenix/pherf/workload/QueryExecutor.java   |  36 ++---
 .../phoenix/pherf/workload/WriteWorkload.java   |  54 ++-
 .../scenario/prod_test_unsalted_scenario.xml|  14 +-
 .../org/apache/phoenix/pherf/ColumnTest.java|   3 +
 .../phoenix/pherf/ConfigurationParserTest.java  |   4 +-
 .../apache/phoenix/pherf/RuleGeneratorTest.java | 107 +-
 .../test/resources/datamodel/test_schema.sql|   4 +
 .../test/resources/scenario/test_scenario.xml   |  48 ++-
 19 files changed, 484 insertions(+), 103 deletions(-)
--


http://git-wip-us.apache.org/repos/asf/phoenix/blob/aef828d0/phoenix-pherf/src/main/java/org/apache/phoenix/pherf/configuration/Column.java
--
diff --git 
a/phoenix-pherf/src/main/java/org/apache/phoenix/pherf/configuration/Column.java
 
b/phoenix-pherf/src/main/java/org/apache/phoenix/pherf/configuration/Column.java
index 7c9e180..0d64a39 100644
--- 
a/phoenix-pherf/src/main/java/org/apache/phoenix/pherf/configuration/Column.java
+++ 
b/phoenix-pherf/src/main/java/org/apache/phoenix/pherf/configuration/Column.java
@@ -28,7 +28,8 @@ public class Column {
private String name;
 private String prefix;
 private DataSequence dataSequence;
-private int length, minValue, maxValue, precision;
+private int length, precision;
+private long minValue, maxValue;
 private int nullChance;
 private boolean userDefined;
 private List dataValues;
@@ -40,8 +41,8 @@ public class Column {
 // Initialize int to negative value so we can distinguish 0 in 
mutations
 // Object fields can be detected with null
 this.length = Integer.MIN_VALUE;
-this.minValue = Integer.MIN_VALUE;
-this.maxValue = Integer.MIN_VALUE;
+this.minValue = Long.MIN_VALUE;
+this.maxValue = Long.MIN_VALUE;
 this.precision = Integer.MIN_VALUE;
 this.nullChance = Integer.MIN_VALUE;
 this.userDefined = false;
@@ -84,6 +85,10 @@ public class Column {
public int getLength() {
return length;
}
+   
+   public int getLengthExcludingPrefix() {
+   return (this.getPrefix() == null) ? this.length : this.length - 
this.getPrefix().length();
+   }
 
public void setLength(int length) {
this.length = length;
@@ -97,19 +102,19 @@ public class Column {
this.type = type;
}
 
-public int getMinValue() {
+public long getMinValue() {
 return minValue;
 }
 
-public void setMinValue(int minValue) {
+public void setMinValue(long minValue) {
 this.minValue = minValue;
 }
 
-public int getMaxValue() {
+public long getMaxValue() {
 return maxValue;
 }
 
-public void setMaxValue(int maxValue) {
+public void setMaxValue(long maxValue) {
 this.maxValue = maxValue;
 }
 
@@ -134,11 +139,11 @@ public class Column {
  *   obj contains only the fields you want to mutate this 
object into.
  */
 public void mutate(Column column) {
-if (column.getMinValue() != Integer.MIN_VALUE) {
+if (column.getMinValue() != Long.MIN_VALUE) {
 setMinValue(column.getMinValue());
 }
 
-if (column.getMaxValue() != Integer.MIN_VALUE) {
+if (column.getMaxValue() != Long.MIN_VALUE) {
 setMaxValue(column.getMaxValue());
 }
 

Apache-Phoenix | 4.x-HBase-1.1 | Build Successful

[Apache Jenkins Server]

4.x-HBase-1.1 branch build status Successful

Source repository https://git-wip-us.apache.org/repos/asf?p=phoenix.git;a=shortlog;h=refs/heads/4.x-HBase-1.1

Compiled Artifacts https://builds.apache.org/job/Phoenix-4.x-HBase-1.1/lastSuccessfulBuild/artifact/

Test Report https://builds.apache.org/job/Phoenix-4.x-HBase-1.1/lastCompletedBuild/testReport/

Changes
[rajeshbabu] PHOENIX-4591 Possible IndexOutOfBoundsException with delete query on



Build times for last couple of runsLatest build time is the right most | Legend blue: normal, red: test failure, gray: timeout

Apache-Phoenix | 4.x-HBase-1.3 | Build Successful

[Apache Jenkins Server]

4.x-HBase-1.3 branch build status Successful

Source repository https://git-wip-us.apache.org/repos/asf?p=phoenix.git;a=shortlog;h=refs/heads/4.x-HBase-1.3

Compiled Artifacts https://builds.apache.org/job/Phoenix-4.x-HBase-1.3/lastSuccessfulBuild/artifact/

Test Report https://builds.apache.org/job/Phoenix-4.x-HBase-1.3/lastCompletedBuild/testReport/

Changes
[rajeshbabu] PHOENIX-4591 Possible IndexOutOfBoundsException with delete query on



Build times for last couple of runsLatest build time is the right most | Legend blue: normal, red: test failure, gray: timeout

Build failed in Jenkins: Phoenix | Master #1933

[Apache Jenkins Server]

See 


Changes:

[rajeshbabu] PHOENIX-4591 Possible IndexOutOfBoundsException with delete query 
on

--
[...truncated 1.20 MB...]
[INFO] Running org.apache.phoenix.end2end.join.SubqueryIT
[INFO] Running org.apache.phoenix.end2end.join.SortMergeJoinNoIndexIT
[INFO] Tests run: 33, Failures: 0, Errors: 0, Skipped: 0, Time elapsed: 359.777 
s - in org.apache.phoenix.end2end.join.HashJoinNoIndexIT
[INFO] Tests run: 64, Failures: 0, Errors: 0, Skipped: 0, Time elapsed: 750.126 
s - in org.apache.phoenix.end2end.index.MutableIndexIT
[INFO] Running org.apache.phoenix.end2end.join.SubqueryUsingSortMergeJoinIT
[INFO] Tests run: 34, Failures: 0, Errors: 0, Skipped: 0, Time elapsed: 373.734 
s - in org.apache.phoenix.end2end.join.SortMergeJoinNoIndexIT
[INFO] Tests run: 24, Failures: 0, Errors: 0, Skipped: 0, Time elapsed: 388.69 
s - in org.apache.phoenix.end2end.join.SubqueryIT
[INFO] Running org.apache.phoenix.end2end.salted.SaltedTableIT
[INFO] Running org.apache.phoenix.end2end.salted.SaltedTableUpsertSelectIT
[INFO] Tests run: 8, Failures: 0, Errors: 0, Skipped: 0, Time elapsed: 27.924 s 
- in org.apache.phoenix.end2end.salted.SaltedTableIT
[INFO] Running org.apache.phoenix.iterate.PhoenixQueryTimeoutIT
[INFO] Running org.apache.phoenix.end2end.salted.SaltedTableVarLengthRowKeyIT
[INFO] Tests run: 2, Failures: 0, Errors: 0, Skipped: 0, Time elapsed: 9.291 s 
- in org.apache.phoenix.iterate.PhoenixQueryTimeoutIT
[INFO] Running org.apache.phoenix.iterate.RoundRobinResultIteratorIT
[INFO] Tests run: 1, Failures: 0, Errors: 0, Skipped: 0, Time elapsed: 4.519 s 
- in org.apache.phoenix.end2end.salted.SaltedTableVarLengthRowKeyIT
[INFO] Running org.apache.phoenix.replication.SystemCatalogWALEntryFilterIT
[INFO] Tests run: 2, Failures: 0, Errors: 0, Skipped: 0, Time elapsed: 1.02 s - 
in org.apache.phoenix.replication.SystemCatalogWALEntryFilterIT
[INFO] Running org.apache.phoenix.rpc.UpdateCacheIT
[INFO] Tests run: 33, Failures: 0, Errors: 0, Skipped: 0, Time elapsed: 621.514 
s - in org.apache.phoenix.end2end.join.HashJoinGlobalIndexIT
[INFO] Tests run: 6, Failures: 0, Errors: 0, Skipped: 0, Time elapsed: 55.79 s 
- in org.apache.phoenix.end2end.salted.SaltedTableUpsertSelectIT
[INFO] Running org.apache.phoenix.trace.PhoenixTracingEndToEndIT
[INFO] Tests run: 7, Failures: 0, Errors: 0, Skipped: 0, Time elapsed: 42.704 s 
- in org.apache.phoenix.rpc.UpdateCacheIT
[INFO] Running org.apache.phoenix.tx.FlappingTransactionIT
[INFO] Running org.apache.phoenix.trace.PhoenixTableMetricsWriterIT
[INFO] Tests run: 4, Failures: 0, Errors: 0, Skipped: 0, Time elapsed: 17.927 s 
- in org.apache.phoenix.tx.FlappingTransactionIT
[INFO] Running org.apache.phoenix.tx.ParameterizedTransactionIT
[INFO] Tests run: 9, Failures: 0, Errors: 0, Skipped: 0, Time elapsed: 76.45 s 
- in org.apache.phoenix.iterate.RoundRobinResultIteratorIT
[INFO] Running org.apache.phoenix.tx.TransactionIT
[INFO] Tests run: 2, Failures: 0, Errors: 0, Skipped: 0, Time elapsed: 14.417 s 
- in org.apache.phoenix.trace.PhoenixTableMetricsWriterIT
[INFO] Running org.apache.phoenix.tx.TxCheckpointIT
[INFO] Tests run: 8, Failures: 0, Errors: 0, Skipped: 0, Time elapsed: 104.219 
s - in org.apache.phoenix.trace.PhoenixTracingEndToEndIT
[INFO] Tests run: 34, Failures: 0, Errors: 0, Skipped: 0, Time elapsed: 647.295 
s - in org.apache.phoenix.end2end.join.SortMergeJoinGlobalIndexIT
[INFO] Tests run: 8, Failures: 0, Errors: 0, Skipped: 0, Time elapsed: 103.404 
s - in org.apache.phoenix.tx.TransactionIT
[INFO] Running org.apache.phoenix.util.IndexScrutinyIT
[INFO] Tests run: 15, Failures: 0, Errors: 0, Skipped: 0, Time elapsed: 288.446 
s - in org.apache.phoenix.end2end.join.SubqueryUsingSortMergeJoinIT
[INFO] Tests run: 3, Failures: 0, Errors: 0, Skipped: 0, Time elapsed: 29.827 s 
- in org.apache.phoenix.util.IndexScrutinyIT
[INFO] Tests run: 34, Failures: 0, Errors: 0, Skipped: 0, Time elapsed: 824.318 
s - in org.apache.phoenix.end2end.join.HashJoinLocalIndexIT
[INFO] Tests run: 34, Failures: 0, Errors: 0, Skipped: 0, Time elapsed: 830.32 
s - in org.apache.phoenix.end2end.join.SortMergeJoinLocalIndexIT
[WARNING] Tests run: 52, Failures: 0, Errors: 0, Skipped: 4, Time elapsed: 
374.011 s - in org.apache.phoenix.tx.ParameterizedTransactionIT
[INFO] Tests run: 40, Failures: 0, Errors: 0, Skipped: 0, Time elapsed: 461.117 
s - in org.apache.phoenix.tx.TxCheckpointIT
[INFO] 
[INFO] Results:
[INFO] 
[ERROR] Failures: 
[ERROR]   ConcurrentMutationsIT.testConcurrentDeletesAndUpsertValues:214 
Expected to find PK in data table: (0,0)
[ERROR]   DefaultColumnValueIT.testDefaultIndexed:978
[ERROR]   RowValueConstructorIT.testRVCLastPkIsTable1stPkIndex:1584
[ERROR]   
IndexMetadataIT.testMutableTableOnlyHasPrimaryKeyIndex:623->helpTestTableOnlyHasPrimaryKeyIndex:662
[ERROR] Errors: 
[ERROR]   

Build failed in Jenkins: Phoenix | 4.x-HBase-0.98 #1814

[Apache Jenkins Server]

See 


Changes:

[rajeshbabu] PHOENIX-4591 Possible IndexOutOfBoundsException with delete query 
on

--
[...truncated 110.06 KB...]
[ERROR] 
testNonTxToTxTable[TransactionIT_mutable=true,columnEncoded=true](org.apache.phoenix.tx.ParameterizedTransactionIT)
  Time elapsed: 5.032 s  <<< ERROR!
org.apache.phoenix.execute.CommitException: 
org.apache.hadoop.hbase.client.RetriesExhaustedWithDetailsException: Failed 1 
action: org.apache.hadoop.hbase.DoNotRetryIOException: Failed 1 action: 
RegionOpeningException: 1 time, servers with issues: 
asf935.gq1.ygridcore.net,39908,1518427200174, 
at 
org.apache.phoenix.util.ServerUtil.createIOException(ServerUtil.java:80)
at 
org.apache.phoenix.util.ServerUtil.throwIOException(ServerUtil.java:62)
at 
org.apache.phoenix.index.PhoenixTransactionalIndexer.postBatchMutateIndispensably(PhoenixTransactionalIndexer.java:240)
at 
org.apache.hadoop.hbase.regionserver.RegionCoprocessorHost$37.call(RegionCoprocessorHost.java:1040)
at 
org.apache.hadoop.hbase.regionserver.RegionCoprocessorHost$RegionOperation.call(RegionCoprocessorHost.java:1656)
at 
org.apache.hadoop.hbase.regionserver.RegionCoprocessorHost.execOperation(RegionCoprocessorHost.java:1733)
at 
org.apache.hadoop.hbase.regionserver.RegionCoprocessorHost.execOperation(RegionCoprocessorHost.java:1688)
at 
org.apache.hadoop.hbase.regionserver.RegionCoprocessorHost.postBatchMutateIndispensably(RegionCoprocessorHost.java:1036)
at 
org.apache.hadoop.hbase.regionserver.HRegion.doMiniBatchMutation(HRegion.java:2767)
at 
org.apache.hadoop.hbase.regionserver.HRegion.batchMutate(HRegion.java:2359)
at 
org.apache.hadoop.hbase.regionserver.HRegion.batchMutate(HRegion.java:2314)
at 
org.apache.hadoop.hbase.regionserver.HRegion.batchMutate(HRegion.java:2318)
at 
org.apache.hadoop.hbase.regionserver.HRegionServer.doBatchOp(HRegionServer.java:4678)
at 
org.apache.hadoop.hbase.regionserver.HRegionServer.doNonAtomicRegionMutation(HRegionServer.java:3835)
at 
org.apache.hadoop.hbase.regionserver.HRegionServer.multi(HRegionServer.java:3680)
at 
org.apache.hadoop.hbase.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:32500)
at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:2195)
at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:104)
at 
org.apache.hadoop.hbase.ipc.RpcExecutor.consumerLoop(RpcExecutor.java:133)
at org.apache.hadoop.hbase.ipc.RpcExecutor$1.run(RpcExecutor.java:108)
at java.lang.Thread.run(Thread.java:745)
Caused by: org.apache.hadoop.hbase.client.RetriesExhaustedWithDetailsException: 
Failed 1 action: RegionOpeningException: 1 time, servers with issues: 
asf935.gq1.ygridcore.net,39908,1518427200174, 
at 
org.apache.hadoop.hbase.client.AsyncProcess$BatchErrors.makeException(AsyncProcess.java:211)
at 
org.apache.hadoop.hbase.client.AsyncProcess$BatchErrors.access$500(AsyncProcess.java:195)
at 
org.apache.hadoop.hbase.client.AsyncProcess.getErrors(AsyncProcess.java:1082)
at 
org.apache.hadoop.hbase.client.HConnectionManager$HConnectionImplementation.processBatchCallback(HConnectionManager.java:2479)
at org.apache.hadoop.hbase.client.HTable.batchCallback(HTable.java:898)
at org.apache.hadoop.hbase.client.HTable.batchCallback(HTable.java:913)
at org.apache.hadoop.hbase.client.HTable.batch(HTable.java:888)
at 
org.apache.phoenix.hbase.index.write.ParallelWriterIndexCommitter$1.call(ParallelWriterIndexCommitter.java:170)
at 
org.apache.phoenix.hbase.index.write.ParallelWriterIndexCommitter$1.call(ParallelWriterIndexCommitter.java:133)
at java.util.concurrent.FutureTask.run(FutureTask.java:262)
at 
java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1145)
at 
java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615)
... 1 more
: 1 time, servers with issues: asf935.gq1.ygridcore.net,39908,1518427200174, 
at 
org.apache.phoenix.tx.ParameterizedTransactionIT.testNonTxToTxTable(ParameterizedTransactionIT.java:288)
Caused by: org.apache.hadoop.hbase.client.RetriesExhaustedWithDetailsException: 
Failed 1 action: org.apache.hadoop.hbase.DoNotRetryIOException: Failed 1 
action: RegionOpeningException: 1 time, servers with issues: 
asf935.gq1.ygridcore.net,39908,1518427200174, 
at 
org.apache.phoenix.util.ServerUtil.createIOException(ServerUtil.java:80)
at 
org.apache.phoenix.util.ServerUtil.throwIOException(ServerUtil.java:62)
at 
org.apache.phoenix.index.PhoenixTransactionalIndexer.postBatchMutateIndispensably(PhoenixTransactionalIndexer.java:240)
at 

Build failed in Jenkins: Phoenix Compile Compatibility with HBase #546

[Apache Jenkins Server]

See 


--
[...truncated 39.70 KB...]
[ERROR] 
:[364,5]
 method does not override or implement a method from a supertype
[ERROR] 
:[370,5]
 method does not override or implement a method from a supertype
[ERROR] 
:[376,5]
 method does not override or implement a method from a supertype
[ERROR] 
:[382,5]
 method does not override or implement a method from a supertype
[ERROR] Failed to execute goal 
org.apache.maven.plugins:maven-compiler-plugin:3.0:compile (default-compile) on 
project phoenix-core: Compilation failure: Compilation failure: 
[ERROR] 
:[34,39]
 cannot find symbol
[ERROR]   symbol:   class MetricRegistry
[ERROR]   location: package org.apache.hadoop.hbase.metrics
[ERROR] 
:[144,16]
 cannot find symbol
[ERROR]   symbol:   class MetricRegistry
[ERROR]   location: class 
org.apache.phoenix.coprocessor.PhoenixMetaDataCoprocessorHost.PhoenixMetaDataControllerEnvironment
[ERROR] 
:[24,35]
 cannot find symbol
[ERROR]   symbol:   class DelegatingHBaseRpcController
[ERROR]   location: package org.apache.hadoop.hbase.ipc
[ERROR] 
:[25,35]
 cannot find symbol
[ERROR]   symbol:   class HBaseRpcController
[ERROR]   location: package org.apache.hadoop.hbase.ipc
[ERROR] 
:[37,37]
 cannot find symbol
[ERROR]   symbol: class DelegatingHBaseRpcController
[ERROR] 
:[56,38]
 cannot find symbol
[ERROR]   symbol:   class HBaseRpcController
[ERROR]   location: class 
org.apache.hadoop.hbase.ipc.controller.MetadataRpcController
[ERROR] 
:[26,35]
 cannot find symbol
[ERROR]   symbol:   class HBaseRpcController
[ERROR]   location: package org.apache.hadoop.hbase.ipc
[ERROR] 
:[40,12]
 cannot find symbol
[ERROR]   symbol:   class HBaseRpcController
[ERROR]   location: class 
org.apache.hadoop.hbase.ipc.controller.InterRegionServerMetadataRpcControllerFactory
[ERROR] 
:[46,12]
 cannot find symbol
[ERROR]   symbol:   class HBaseRpcController
[ERROR]   location: class 
org.apache.hadoop.hbase.ipc.controller.InterRegionServerMetadataRpcControllerFactory
[ERROR] 
:[52,12]
 cannot find symbol
[ERROR]   symbol:   class HBaseRpcController
[ERROR]   location: class 
org.apache.hadoop.hbase.ipc.controller.InterRegionServerMetadataRpcControllerFactory
[ERROR] 
:[57,46]
 cannot 

phoenix git commit: PHOENIX-4591 Possible IndexOutOfBoundsException with delete query on bigger table(Rajeshbabu)

[rajeshbabu]

Repository: phoenix
Updated Branches:
  refs/heads/5.x-HBase-2.0 8c1746c21 -> 7cb65d155


PHOENIX-4591 Possible IndexOutOfBoundsException with delete query on bigger 
table(Rajeshbabu)


Project: http://git-wip-us.apache.org/repos/asf/phoenix/repo
Commit: http://git-wip-us.apache.org/repos/asf/phoenix/commit/7cb65d15
Tree: http://git-wip-us.apache.org/repos/asf/phoenix/tree/7cb65d15
Diff: http://git-wip-us.apache.org/repos/asf/phoenix/diff/7cb65d15

Branch: refs/heads/5.x-HBase-2.0
Commit: 7cb65d155b716aa1a4bb6179fe440b1e5f6d79e9
Parents: 8c1746c
Author: Rajeshbabu Chintaguntla 
Authored: Mon Feb 12 14:22:38 2018 +0530
Committer: Rajeshbabu Chintaguntla 
Committed: Mon Feb 12 14:22:38 2018 +0530

--
 .../org/apache/phoenix/end2end/DeleteIT.java| 36 
 .../apache/phoenix/compile/DeleteCompiler.java  |  4 ++-
 2 files changed, 39 insertions(+), 1 deletion(-)
--


http://git-wip-us.apache.org/repos/asf/phoenix/blob/7cb65d15/phoenix-core/src/it/java/org/apache/phoenix/end2end/DeleteIT.java
--
diff --git a/phoenix-core/src/it/java/org/apache/phoenix/end2end/DeleteIT.java 
b/phoenix-core/src/it/java/org/apache/phoenix/end2end/DeleteIT.java
index 498aeff..5e65927 100644
--- a/phoenix-core/src/it/java/org/apache/phoenix/end2end/DeleteIT.java
+++ b/phoenix-core/src/it/java/org/apache/phoenix/end2end/DeleteIT.java
@@ -17,6 +17,7 @@
  */
 package org.apache.phoenix.end2end;
 
+import static org.apache.phoenix.util.TestUtil.TEST_PROPERTIES;
 import static org.junit.Assert.assertEquals;
 import static org.junit.Assert.assertNotEquals;
 import static org.junit.Assert.assertTrue;
@@ -33,7 +34,10 @@ import java.sql.Types;
 import java.util.Arrays;
 import java.util.Collections;
 import java.util.List;
+import java.util.Properties;
 
+import org.apache.phoenix.query.QueryServices;
+import org.apache.phoenix.util.PropertiesUtil;
 import org.apache.phoenix.util.QueryUtil;
 import org.junit.Test;
 
@@ -799,6 +803,38 @@ public class DeleteIT extends ParallelStatsDisabledIT {
 }
 }
 }
+
+@Test
+public void testDeleteShouldNotFailWhenTheRowsMoreThanMaxMutationSize() 
throws Exception {
+String tableName = generateUniqueName();
+String indexName1 = generateUniqueName();
+String ddl =
+"CREATE TABLE IF NOT EXISTS "
++ tableName
++ " (pk1 DECIMAL NOT NULL, v1 VARCHAR, v2 VARCHAR 
CONSTRAINT PK PRIMARY KEY (pk1))"
++ " IMMUTABLE_ROWS=true";
+String idx1 = "CREATE INDEX " + indexName1 + " ON " + tableName + 
"(v1)";
+Properties props = PropertiesUtil.deepCopy(TEST_PROPERTIES);
+
props.setProperty(QueryServices.MAX_MUTATION_SIZE_ATTRIB,Integer.toString(10));
+try (Connection conn = DriverManager.getConnection(getUrl(), props)) {
+conn.createStatement().execute(ddl);
+conn.createStatement().execute(idx1);
+Statement stmt = conn.createStatement();
+for(int i = 0; i < 20; i++) {
+stmt.executeUpdate("UPSERT INTO " + tableName + " VALUES 
("+i+",'value"+i+"', 'value2')");
+if (i % 10 == 0) {
+conn.commit();
+}
+}
+conn.commit();
+conn.setAutoCommit(true);
+try {
+conn.createStatement().execute("DELETE FROM " + tableName);
+} catch (Exception e) {
+fail("Should not throw any exception");
+}
+}
+}
 }
 
 

http://git-wip-us.apache.org/repos/asf/phoenix/blob/7cb65d15/phoenix-core/src/main/java/org/apache/phoenix/compile/DeleteCompiler.java
--
diff --git 
a/phoenix-core/src/main/java/org/apache/phoenix/compile/DeleteCompiler.java 
b/phoenix-core/src/main/java/org/apache/phoenix/compile/DeleteCompiler.java
index b77fcbe..53fc398 100644
--- a/phoenix-core/src/main/java/org/apache/phoenix/compile/DeleteCompiler.java
+++ b/phoenix-core/src/main/java/org/apache/phoenix/compile/DeleteCompiler.java
@@ -239,7 +239,9 @@ public class DeleteCompiler {
 connection.getMutationState().send();
 mutations.clear();
 if (indexMutations != null) {
-indexMutations.clear();
+for (Map 
multiRowMutationState : indexMutations) {
+multiRowMutationState.clear();
+}
 }
 }
 }

phoenix git commit: PHOENIX-4591 Possible IndexOutOfBoundsException with delete query on bigger table(Rajeshbabu)

[rajeshbabu]

Repository: phoenix
Updated Branches:
  refs/heads/4.x-HBase-1.1 587f78de2 -> 66fc54c68


PHOENIX-4591 Possible IndexOutOfBoundsException with delete query on bigger 
table(Rajeshbabu)


Project: http://git-wip-us.apache.org/repos/asf/phoenix/repo
Commit: http://git-wip-us.apache.org/repos/asf/phoenix/commit/66fc54c6
Tree: http://git-wip-us.apache.org/repos/asf/phoenix/tree/66fc54c6
Diff: http://git-wip-us.apache.org/repos/asf/phoenix/diff/66fc54c6

Branch: refs/heads/4.x-HBase-1.1
Commit: 66fc54c689ce9c809d55cef91bc0b316f5cb5ca0
Parents: 587f78d
Author: Rajeshbabu Chintaguntla 
Authored: Mon Feb 12 14:00:20 2018 +0530
Committer: Rajeshbabu Chintaguntla 
Committed: Mon Feb 12 14:00:20 2018 +0530

--
 .../org/apache/phoenix/end2end/DeleteIT.java| 37 
 .../apache/phoenix/compile/DeleteCompiler.java  |  4 ++-
 2 files changed, 40 insertions(+), 1 deletion(-)
--


http://git-wip-us.apache.org/repos/asf/phoenix/blob/66fc54c6/phoenix-core/src/it/java/org/apache/phoenix/end2end/DeleteIT.java
--
diff --git a/phoenix-core/src/it/java/org/apache/phoenix/end2end/DeleteIT.java 
b/phoenix-core/src/it/java/org/apache/phoenix/end2end/DeleteIT.java
index 6e4cb2a..c31fdd9 100644
--- a/phoenix-core/src/it/java/org/apache/phoenix/end2end/DeleteIT.java
+++ b/phoenix-core/src/it/java/org/apache/phoenix/end2end/DeleteIT.java
@@ -17,9 +17,11 @@
  */
 package org.apache.phoenix.end2end;
 
+import static org.apache.phoenix.util.TestUtil.TEST_PROPERTIES;
 import static org.junit.Assert.assertEquals;
 import static org.junit.Assert.assertNotEquals;
 import static org.junit.Assert.assertTrue;
+import static org.junit.Assert.fail;
 
 import java.sql.Connection;
 import java.sql.Date;
@@ -32,7 +34,10 @@ import java.sql.Types;
 import java.util.Arrays;
 import java.util.Collections;
 import java.util.List;
+import java.util.Properties;
 
+import org.apache.phoenix.query.QueryServices;
+import org.apache.phoenix.util.PropertiesUtil;
 import org.apache.phoenix.util.QueryUtil;
 import org.junit.Test;
 
@@ -769,6 +774,38 @@ public class DeleteIT extends ParallelStatsDisabledIT {
 }
 
 }
+
+@Test
+public void testDeleteShouldNotFailWhenTheRowsMoreThanMaxMutationSize() 
throws Exception {
+String tableName = generateUniqueName();
+String indexName1 = generateUniqueName();
+String ddl =
+"CREATE TABLE IF NOT EXISTS "
++ tableName
++ " (pk1 DECIMAL NOT NULL, v1 VARCHAR, v2 VARCHAR 
CONSTRAINT PK PRIMARY KEY (pk1))"
++ " IMMUTABLE_ROWS=true";
+String idx1 = "CREATE INDEX " + indexName1 + " ON " + tableName + 
"(v1)";
+Properties props = PropertiesUtil.deepCopy(TEST_PROPERTIES);
+
props.setProperty(QueryServices.MAX_MUTATION_SIZE_ATTRIB,Integer.toString(10));
+try (Connection conn = DriverManager.getConnection(getUrl(), props)) {
+conn.createStatement().execute(ddl);
+conn.createStatement().execute(idx1);
+Statement stmt = conn.createStatement();
+for(int i = 0; i < 20; i++) {
+stmt.executeUpdate("UPSERT INTO " + tableName + " VALUES 
("+i+",'value"+i+"', 'value2')");
+if (i % 10 == 0) {
+conn.commit();
+}
+}
+conn.commit();
+conn.setAutoCommit(true);
+try {
+conn.createStatement().execute("DELETE FROM " + tableName);
+} catch (Exception e) {
+fail("Should not throw any exception");
+}
+}
+}
 }
 
 

http://git-wip-us.apache.org/repos/asf/phoenix/blob/66fc54c6/phoenix-core/src/main/java/org/apache/phoenix/compile/DeleteCompiler.java
--
diff --git 
a/phoenix-core/src/main/java/org/apache/phoenix/compile/DeleteCompiler.java 
b/phoenix-core/src/main/java/org/apache/phoenix/compile/DeleteCompiler.java
index bc96627..46f4542 100644
--- a/phoenix-core/src/main/java/org/apache/phoenix/compile/DeleteCompiler.java
+++ b/phoenix-core/src/main/java/org/apache/phoenix/compile/DeleteCompiler.java
@@ -255,7 +255,9 @@ public class DeleteCompiler {
 connection.getMutationState().send();
 mutations.clear();
 if (otherMutations != null) {
-otherMutations.clear();
+for(MultiRowMutationState multiRowMutationState: 
otherMutations) {
+multiRowMutationState.clear();
+}
 }
 }
 }

phoenix git commit: PHOENIX-4591 Possible IndexOutOfBoundsException with delete query on bigger table(Rajeshbabu)

[rajeshbabu]

Repository: phoenix
Updated Branches:
  refs/heads/4.x-HBase-1.2 e4566fcd7 -> ff4ad203b


PHOENIX-4591 Possible IndexOutOfBoundsException with delete query on bigger 
table(Rajeshbabu)


Project: http://git-wip-us.apache.org/repos/asf/phoenix/repo
Commit: http://git-wip-us.apache.org/repos/asf/phoenix/commit/ff4ad203
Tree: http://git-wip-us.apache.org/repos/asf/phoenix/tree/ff4ad203
Diff: http://git-wip-us.apache.org/repos/asf/phoenix/diff/ff4ad203

Branch: refs/heads/4.x-HBase-1.2
Commit: ff4ad203b5ac932506d06aa6496a7858031003c7
Parents: e4566fc
Author: Rajeshbabu Chintaguntla 
Authored: Mon Feb 12 13:46:42 2018 +0530
Committer: Rajeshbabu Chintaguntla 
Committed: Mon Feb 12 13:46:42 2018 +0530

--
 .../org/apache/phoenix/end2end/DeleteIT.java| 36 
 .../apache/phoenix/compile/DeleteCompiler.java  |  4 ++-
 2 files changed, 39 insertions(+), 1 deletion(-)
--


http://git-wip-us.apache.org/repos/asf/phoenix/blob/ff4ad203/phoenix-core/src/it/java/org/apache/phoenix/end2end/DeleteIT.java
--
diff --git a/phoenix-core/src/it/java/org/apache/phoenix/end2end/DeleteIT.java 
b/phoenix-core/src/it/java/org/apache/phoenix/end2end/DeleteIT.java
index 498aeff..5e65927 100644
--- a/phoenix-core/src/it/java/org/apache/phoenix/end2end/DeleteIT.java
+++ b/phoenix-core/src/it/java/org/apache/phoenix/end2end/DeleteIT.java
@@ -17,6 +17,7 @@
  */
 package org.apache.phoenix.end2end;
 
+import static org.apache.phoenix.util.TestUtil.TEST_PROPERTIES;
 import static org.junit.Assert.assertEquals;
 import static org.junit.Assert.assertNotEquals;
 import static org.junit.Assert.assertTrue;
@@ -33,7 +34,10 @@ import java.sql.Types;
 import java.util.Arrays;
 import java.util.Collections;
 import java.util.List;
+import java.util.Properties;
 
+import org.apache.phoenix.query.QueryServices;
+import org.apache.phoenix.util.PropertiesUtil;
 import org.apache.phoenix.util.QueryUtil;
 import org.junit.Test;
 
@@ -799,6 +803,38 @@ public class DeleteIT extends ParallelStatsDisabledIT {
 }
 }
 }
+
+@Test
+public void testDeleteShouldNotFailWhenTheRowsMoreThanMaxMutationSize() 
throws Exception {
+String tableName = generateUniqueName();
+String indexName1 = generateUniqueName();
+String ddl =
+"CREATE TABLE IF NOT EXISTS "
++ tableName
++ " (pk1 DECIMAL NOT NULL, v1 VARCHAR, v2 VARCHAR 
CONSTRAINT PK PRIMARY KEY (pk1))"
++ " IMMUTABLE_ROWS=true";
+String idx1 = "CREATE INDEX " + indexName1 + " ON " + tableName + 
"(v1)";
+Properties props = PropertiesUtil.deepCopy(TEST_PROPERTIES);
+
props.setProperty(QueryServices.MAX_MUTATION_SIZE_ATTRIB,Integer.toString(10));
+try (Connection conn = DriverManager.getConnection(getUrl(), props)) {
+conn.createStatement().execute(ddl);
+conn.createStatement().execute(idx1);
+Statement stmt = conn.createStatement();
+for(int i = 0; i < 20; i++) {
+stmt.executeUpdate("UPSERT INTO " + tableName + " VALUES 
("+i+",'value"+i+"', 'value2')");
+if (i % 10 == 0) {
+conn.commit();
+}
+}
+conn.commit();
+conn.setAutoCommit(true);
+try {
+conn.createStatement().execute("DELETE FROM " + tableName);
+} catch (Exception e) {
+fail("Should not throw any exception");
+}
+}
+}
 }
 
 

http://git-wip-us.apache.org/repos/asf/phoenix/blob/ff4ad203/phoenix-core/src/main/java/org/apache/phoenix/compile/DeleteCompiler.java
--
diff --git 
a/phoenix-core/src/main/java/org/apache/phoenix/compile/DeleteCompiler.java 
b/phoenix-core/src/main/java/org/apache/phoenix/compile/DeleteCompiler.java
index 54e63d2..6e500c0 100644
--- a/phoenix-core/src/main/java/org/apache/phoenix/compile/DeleteCompiler.java
+++ b/phoenix-core/src/main/java/org/apache/phoenix/compile/DeleteCompiler.java
@@ -256,7 +256,9 @@ public class DeleteCompiler {
 connection.getMutationState().send();
 mutations.clear();
 if (otherMutations != null) {
-otherMutations.clear();
+for(MultiRowMutationState multiRowMutationState: 
otherMutations) {
+multiRowMutationState.clear();
+}
 }
 }
 }

phoenix git commit: PHOENIX-4591 Possible IndexOutOfBoundsException with delete query on bigger table(Rajeshbabu)

[rajeshbabu]

Repository: phoenix
Updated Branches:
  refs/heads/4.x-HBase-0.98 1d34ad38b -> c9e23eac5


PHOENIX-4591 Possible IndexOutOfBoundsException with delete query on bigger 
table(Rajeshbabu)


Project: http://git-wip-us.apache.org/repos/asf/phoenix/repo
Commit: http://git-wip-us.apache.org/repos/asf/phoenix/commit/c9e23eac
Tree: http://git-wip-us.apache.org/repos/asf/phoenix/tree/c9e23eac
Diff: http://git-wip-us.apache.org/repos/asf/phoenix/diff/c9e23eac

Branch: refs/heads/4.x-HBase-0.98
Commit: c9e23eac59bba3f3dc1df9079a31a31b16afd39c
Parents: 1d34ad3
Author: Rajeshbabu Chintaguntla 
Authored: Mon Feb 12 13:44:49 2018 +0530
Committer: Rajeshbabu Chintaguntla 
Committed: Mon Feb 12 13:44:49 2018 +0530

--
 .../org/apache/phoenix/end2end/DeleteIT.java| 36 
 .../apache/phoenix/compile/DeleteCompiler.java  |  4 ++-
 2 files changed, 39 insertions(+), 1 deletion(-)
--


http://git-wip-us.apache.org/repos/asf/phoenix/blob/c9e23eac/phoenix-core/src/it/java/org/apache/phoenix/end2end/DeleteIT.java
--
diff --git a/phoenix-core/src/it/java/org/apache/phoenix/end2end/DeleteIT.java 
b/phoenix-core/src/it/java/org/apache/phoenix/end2end/DeleteIT.java
index 498aeff..5e65927 100644
--- a/phoenix-core/src/it/java/org/apache/phoenix/end2end/DeleteIT.java
+++ b/phoenix-core/src/it/java/org/apache/phoenix/end2end/DeleteIT.java
@@ -17,6 +17,7 @@
  */
 package org.apache.phoenix.end2end;
 
+import static org.apache.phoenix.util.TestUtil.TEST_PROPERTIES;
 import static org.junit.Assert.assertEquals;
 import static org.junit.Assert.assertNotEquals;
 import static org.junit.Assert.assertTrue;
@@ -33,7 +34,10 @@ import java.sql.Types;
 import java.util.Arrays;
 import java.util.Collections;
 import java.util.List;
+import java.util.Properties;
 
+import org.apache.phoenix.query.QueryServices;
+import org.apache.phoenix.util.PropertiesUtil;
 import org.apache.phoenix.util.QueryUtil;
 import org.junit.Test;
 
@@ -799,6 +803,38 @@ public class DeleteIT extends ParallelStatsDisabledIT {
 }
 }
 }
+
+@Test
+public void testDeleteShouldNotFailWhenTheRowsMoreThanMaxMutationSize() 
throws Exception {
+String tableName = generateUniqueName();
+String indexName1 = generateUniqueName();
+String ddl =
+"CREATE TABLE IF NOT EXISTS "
++ tableName
++ " (pk1 DECIMAL NOT NULL, v1 VARCHAR, v2 VARCHAR 
CONSTRAINT PK PRIMARY KEY (pk1))"
++ " IMMUTABLE_ROWS=true";
+String idx1 = "CREATE INDEX " + indexName1 + " ON " + tableName + 
"(v1)";
+Properties props = PropertiesUtil.deepCopy(TEST_PROPERTIES);
+
props.setProperty(QueryServices.MAX_MUTATION_SIZE_ATTRIB,Integer.toString(10));
+try (Connection conn = DriverManager.getConnection(getUrl(), props)) {
+conn.createStatement().execute(ddl);
+conn.createStatement().execute(idx1);
+Statement stmt = conn.createStatement();
+for(int i = 0; i < 20; i++) {
+stmt.executeUpdate("UPSERT INTO " + tableName + " VALUES 
("+i+",'value"+i+"', 'value2')");
+if (i % 10 == 0) {
+conn.commit();
+}
+}
+conn.commit();
+conn.setAutoCommit(true);
+try {
+conn.createStatement().execute("DELETE FROM " + tableName);
+} catch (Exception e) {
+fail("Should not throw any exception");
+}
+}
+}
 }
 
 

http://git-wip-us.apache.org/repos/asf/phoenix/blob/c9e23eac/phoenix-core/src/main/java/org/apache/phoenix/compile/DeleteCompiler.java
--
diff --git 
a/phoenix-core/src/main/java/org/apache/phoenix/compile/DeleteCompiler.java 
b/phoenix-core/src/main/java/org/apache/phoenix/compile/DeleteCompiler.java
index 54e63d2..6e500c0 100644
--- a/phoenix-core/src/main/java/org/apache/phoenix/compile/DeleteCompiler.java
+++ b/phoenix-core/src/main/java/org/apache/phoenix/compile/DeleteCompiler.java
@@ -256,7 +256,9 @@ public class DeleteCompiler {
 connection.getMutationState().send();
 mutations.clear();
 if (otherMutations != null) {
-otherMutations.clear();
+for(MultiRowMutationState multiRowMutationState: 
otherMutations) {
+multiRowMutationState.clear();
+}
 }
 }
 }

phoenix git commit: PHOENIX-4591 Possible IndexOutOfBoundsException with delete query on bigger table(Rajeshbabu)

[rajeshbabu]

Repository: phoenix
Updated Branches:
  refs/heads/4.x-HBase-1.3 d15efd1bf -> 3d67e8d5c


PHOENIX-4591 Possible IndexOutOfBoundsException with delete query on bigger 
table(Rajeshbabu)


Project: http://git-wip-us.apache.org/repos/asf/phoenix/repo
Commit: http://git-wip-us.apache.org/repos/asf/phoenix/commit/3d67e8d5
Tree: http://git-wip-us.apache.org/repos/asf/phoenix/tree/3d67e8d5
Diff: http://git-wip-us.apache.org/repos/asf/phoenix/diff/3d67e8d5

Branch: refs/heads/4.x-HBase-1.3
Commit: 3d67e8d5c614627dab7826c1273b0e440e49f73f
Parents: d15efd1
Author: Rajeshbabu Chintaguntla 
Authored: Mon Feb 12 13:37:21 2018 +0530
Committer: Rajeshbabu Chintaguntla 
Committed: Mon Feb 12 13:37:21 2018 +0530

--
 .../org/apache/phoenix/end2end/DeleteIT.java| 36 
 .../apache/phoenix/compile/DeleteCompiler.java  |  4 ++-
 2 files changed, 39 insertions(+), 1 deletion(-)
--


http://git-wip-us.apache.org/repos/asf/phoenix/blob/3d67e8d5/phoenix-core/src/it/java/org/apache/phoenix/end2end/DeleteIT.java
--
diff --git a/phoenix-core/src/it/java/org/apache/phoenix/end2end/DeleteIT.java 
b/phoenix-core/src/it/java/org/apache/phoenix/end2end/DeleteIT.java
index 498aeff..5e65927 100644
--- a/phoenix-core/src/it/java/org/apache/phoenix/end2end/DeleteIT.java
+++ b/phoenix-core/src/it/java/org/apache/phoenix/end2end/DeleteIT.java
@@ -17,6 +17,7 @@
  */
 package org.apache.phoenix.end2end;
 
+import static org.apache.phoenix.util.TestUtil.TEST_PROPERTIES;
 import static org.junit.Assert.assertEquals;
 import static org.junit.Assert.assertNotEquals;
 import static org.junit.Assert.assertTrue;
@@ -33,7 +34,10 @@ import java.sql.Types;
 import java.util.Arrays;
 import java.util.Collections;
 import java.util.List;
+import java.util.Properties;
 
+import org.apache.phoenix.query.QueryServices;
+import org.apache.phoenix.util.PropertiesUtil;
 import org.apache.phoenix.util.QueryUtil;
 import org.junit.Test;
 
@@ -799,6 +803,38 @@ public class DeleteIT extends ParallelStatsDisabledIT {
 }
 }
 }
+
+@Test
+public void testDeleteShouldNotFailWhenTheRowsMoreThanMaxMutationSize() 
throws Exception {
+String tableName = generateUniqueName();
+String indexName1 = generateUniqueName();
+String ddl =
+"CREATE TABLE IF NOT EXISTS "
++ tableName
++ " (pk1 DECIMAL NOT NULL, v1 VARCHAR, v2 VARCHAR 
CONSTRAINT PK PRIMARY KEY (pk1))"
++ " IMMUTABLE_ROWS=true";
+String idx1 = "CREATE INDEX " + indexName1 + " ON " + tableName + 
"(v1)";
+Properties props = PropertiesUtil.deepCopy(TEST_PROPERTIES);
+
props.setProperty(QueryServices.MAX_MUTATION_SIZE_ATTRIB,Integer.toString(10));
+try (Connection conn = DriverManager.getConnection(getUrl(), props)) {
+conn.createStatement().execute(ddl);
+conn.createStatement().execute(idx1);
+Statement stmt = conn.createStatement();
+for(int i = 0; i < 20; i++) {
+stmt.executeUpdate("UPSERT INTO " + tableName + " VALUES 
("+i+",'value"+i+"', 'value2')");
+if (i % 10 == 0) {
+conn.commit();
+}
+}
+conn.commit();
+conn.setAutoCommit(true);
+try {
+conn.createStatement().execute("DELETE FROM " + tableName);
+} catch (Exception e) {
+fail("Should not throw any exception");
+}
+}
+}
 }
 
 

http://git-wip-us.apache.org/repos/asf/phoenix/blob/3d67e8d5/phoenix-core/src/main/java/org/apache/phoenix/compile/DeleteCompiler.java
--
diff --git 
a/phoenix-core/src/main/java/org/apache/phoenix/compile/DeleteCompiler.java 
b/phoenix-core/src/main/java/org/apache/phoenix/compile/DeleteCompiler.java
index 54e63d2..6e500c0 100644
--- a/phoenix-core/src/main/java/org/apache/phoenix/compile/DeleteCompiler.java
+++ b/phoenix-core/src/main/java/org/apache/phoenix/compile/DeleteCompiler.java
@@ -256,7 +256,9 @@ public class DeleteCompiler {
 connection.getMutationState().send();
 mutations.clear();
 if (otherMutations != null) {
-otherMutations.clear();
+for(MultiRowMutationState multiRowMutationState: 
otherMutations) {
+multiRowMutationState.clear();
+}
 }
 }
 }

phoenix git commit: PHOENIX-4591 Possible IndexOutOfBoundsException with delete query on bigger table(Rajeshbabu)

[rajeshbabu]

Repository: phoenix
Updated Branches:
  refs/heads/master f5512105c -> a6bf7350d


PHOENIX-4591 Possible IndexOutOfBoundsException with delete query on bigger 
table(Rajeshbabu)


Project: http://git-wip-us.apache.org/repos/asf/phoenix/repo
Commit: http://git-wip-us.apache.org/repos/asf/phoenix/commit/a6bf7350
Tree: http://git-wip-us.apache.org/repos/asf/phoenix/tree/a6bf7350
Diff: http://git-wip-us.apache.org/repos/asf/phoenix/diff/a6bf7350

Branch: refs/heads/master
Commit: a6bf7350d311e42402d330b07cb412b422eeaae9
Parents: f551210
Author: Rajeshbabu Chintaguntla 
Authored: Mon Feb 12 13:36:18 2018 +0530
Committer: Rajeshbabu Chintaguntla 
Committed: Mon Feb 12 13:36:18 2018 +0530

--
 .../org/apache/phoenix/end2end/DeleteIT.java| 36 
 .../apache/phoenix/compile/DeleteCompiler.java  |  4 ++-
 2 files changed, 39 insertions(+), 1 deletion(-)
--


http://git-wip-us.apache.org/repos/asf/phoenix/blob/a6bf7350/phoenix-core/src/it/java/org/apache/phoenix/end2end/DeleteIT.java
--
diff --git a/phoenix-core/src/it/java/org/apache/phoenix/end2end/DeleteIT.java 
b/phoenix-core/src/it/java/org/apache/phoenix/end2end/DeleteIT.java
index 498aeff..5e65927 100644
--- a/phoenix-core/src/it/java/org/apache/phoenix/end2end/DeleteIT.java
+++ b/phoenix-core/src/it/java/org/apache/phoenix/end2end/DeleteIT.java
@@ -17,6 +17,7 @@
  */
 package org.apache.phoenix.end2end;
 
+import static org.apache.phoenix.util.TestUtil.TEST_PROPERTIES;
 import static org.junit.Assert.assertEquals;
 import static org.junit.Assert.assertNotEquals;
 import static org.junit.Assert.assertTrue;
@@ -33,7 +34,10 @@ import java.sql.Types;
 import java.util.Arrays;
 import java.util.Collections;
 import java.util.List;
+import java.util.Properties;
 
+import org.apache.phoenix.query.QueryServices;
+import org.apache.phoenix.util.PropertiesUtil;
 import org.apache.phoenix.util.QueryUtil;
 import org.junit.Test;
 
@@ -799,6 +803,38 @@ public class DeleteIT extends ParallelStatsDisabledIT {
 }
 }
 }
+
+@Test
+public void testDeleteShouldNotFailWhenTheRowsMoreThanMaxMutationSize() 
throws Exception {
+String tableName = generateUniqueName();
+String indexName1 = generateUniqueName();
+String ddl =
+"CREATE TABLE IF NOT EXISTS "
++ tableName
++ " (pk1 DECIMAL NOT NULL, v1 VARCHAR, v2 VARCHAR 
CONSTRAINT PK PRIMARY KEY (pk1))"
++ " IMMUTABLE_ROWS=true";
+String idx1 = "CREATE INDEX " + indexName1 + " ON " + tableName + 
"(v1)";
+Properties props = PropertiesUtil.deepCopy(TEST_PROPERTIES);
+
props.setProperty(QueryServices.MAX_MUTATION_SIZE_ATTRIB,Integer.toString(10));
+try (Connection conn = DriverManager.getConnection(getUrl(), props)) {
+conn.createStatement().execute(ddl);
+conn.createStatement().execute(idx1);
+Statement stmt = conn.createStatement();
+for(int i = 0; i < 20; i++) {
+stmt.executeUpdate("UPSERT INTO " + tableName + " VALUES 
("+i+",'value"+i+"', 'value2')");
+if (i % 10 == 0) {
+conn.commit();
+}
+}
+conn.commit();
+conn.setAutoCommit(true);
+try {
+conn.createStatement().execute("DELETE FROM " + tableName);
+} catch (Exception e) {
+fail("Should not throw any exception");
+}
+}
+}
 }
 
 

http://git-wip-us.apache.org/repos/asf/phoenix/blob/a6bf7350/phoenix-core/src/main/java/org/apache/phoenix/compile/DeleteCompiler.java
--
diff --git 
a/phoenix-core/src/main/java/org/apache/phoenix/compile/DeleteCompiler.java 
b/phoenix-core/src/main/java/org/apache/phoenix/compile/DeleteCompiler.java
index 54e63d2..6e500c0 100644
--- a/phoenix-core/src/main/java/org/apache/phoenix/compile/DeleteCompiler.java
+++ b/phoenix-core/src/main/java/org/apache/phoenix/compile/DeleteCompiler.java
@@ -256,7 +256,9 @@ public class DeleteCompiler {
 connection.getMutationState().send();
 mutations.clear();
 if (otherMutations != null) {
-otherMutations.clear();
+for(MultiRowMutationState multiRowMutationState: 
otherMutations) {
+multiRowMutationState.clear();
+}
 }
 }
 }