Adar Dembo has submitted this change and it was merged. Change subject: KUDU-1065: [java client] Flexible Partition Pruning ......................................................................
KUDU-1065: [java client] Flexible Partition Pruning This commit introduces an internal utility ByteVec class which is a mashup of the C++ std::string / Rust Vec<u8> types. KeyEncoder has been transitioned to use this type instead of ByteArrayOutputStream. The partition pruning algorithm incrementally builds up partition keys from predicates, and requires cloning the keys as they are being built in order to multiply over hash partition buckets. ByteArrayOutputStream doesn't have a clone method. ByteArrayOutputStream is also synchronized internally, which is dumb. Thus begat ByteVec. This version of partition pruning only looks at predicates when determining which partitions to prune. Constraints in the primary key bounds are not considered, unless the table is range partitioned over the primary key columns and not hash partitioned (simple partitioning). This limits the pruned partitions in some pretty rare cases, but the workaround of explicitly setting the predicate is not too onerous. Finally, this commit changes the default test flags to remove mini cluster verbose logging, since it is extremely noisy. Change-Id: Ib27b54841d87cf854175ab8cdfa8798b337d71f9 Reviewed-on: http://gerrit.cloudera.org:8080/4299 Tested-by: Kudu Jenkins Reviewed-by: Adar Dembo <a...@cloudera.com> --- M java/kudu-client/src/main/java/org/apache/kudu/client/AbstractKuduScannerBuilder.java M java/kudu-client/src/main/java/org/apache/kudu/client/AsyncKuduClient.java M java/kudu-client/src/main/java/org/apache/kudu/client/AsyncKuduScanner.java M java/kudu-client/src/main/java/org/apache/kudu/client/KeyEncoder.java M java/kudu-client/src/main/java/org/apache/kudu/client/KuduPredicate.java M java/kudu-client/src/main/java/org/apache/kudu/client/KuduScanToken.java M java/kudu-client/src/main/java/org/apache/kudu/client/KuduScanner.java M java/kudu-client/src/main/java/org/apache/kudu/client/KuduTable.java M java/kudu-client/src/main/java/org/apache/kudu/client/PartialRow.java A java/kudu-client/src/main/java/org/apache/kudu/client/PartitionPruner.java M java/kudu-client/src/main/java/org/apache/kudu/client/PartitionSchema.java A java/kudu-client/src/main/java/org/apache/kudu/util/ByteVec.java M java/kudu-client/src/test/java/org/apache/kudu/client/TestKeyEncoding.java A java/kudu-client/src/test/java/org/apache/kudu/client/TestPartitionPruner.java A java/kudu-client/src/test/java/org/apache/kudu/util/TestByteVec.java M java/kudu-client/src/test/resources/flags M src/kudu/common/partition_pruner-test.cc 17 files changed, 1,722 insertions(+), 198 deletions(-) Approvals: Adar Dembo: Looks good to me, approved Kudu Jenkins: Verified -- To view, visit http://gerrit.cloudera.org:8080/4299 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-MessageType: merged Gerrit-Change-Id: Ib27b54841d87cf854175ab8cdfa8798b337d71f9 Gerrit-PatchSet: 4 Gerrit-Project: kudu Gerrit-Branch: master Gerrit-Owner: Dan Burkert <d...@cloudera.com> Gerrit-Reviewer: Adar Dembo <a...@cloudera.com> Gerrit-Reviewer: Dan Burkert <d...@cloudera.com> Gerrit-Reviewer: Kudu Jenkins Gerrit-Reviewer: Todd Lipcon <t...@apache.org>
