-----------------------------------------------------------
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/65130/
-----------------------------------------------------------
(Updated Feb. 1, 2018, 7:24 p.m.)
Review request for hive, Eugene Koifman, Gopal V, and Jason Dere.
Changes
-------
Added two new fields in Table metadata.
1. bucketingVersion : Default is 2. 1 for older tables which use JAVA hash. 2
for new tables which will use murmur hash. The plumbing is done. The code to
actually change the hashing logic is yet to be done.
2. expertMode : Default false. If a user loads data into bucketed table which
does not launch a Tez job(pending work), this is set to true. helps in
debugging issues for wrong results in queries with bucketed tables.
Load data on bucketed tables can only take names like 000000_0, 000001_0 etc
for the file names. It will reject the load otherwise.
Fixed CustomPartitionVertex code for SMB and Bucket map joins. The logic to
iterate and assign bucket id is replaced by examining the file name and
assigning bucket id.
For SMB, small table must have less than or equal number of buckets as big
table.
Bugs: HIVE-18350
https://issues.apache.org/jira/browse/HIVE-18350
Repository: hive-git
Description
-------
Made changes for both bucketed and non-bucketed tables.
Added a positive test for non-bucketed table which renames the loaded file.
Added couple of negative tests for bucketed table which reject a load with
inconsistent file name.
Diffs (updated)
-----
hcatalog/core/src/test/java/org/apache/hive/hcatalog/common/TestHCatUtil.java
91aa4fa269
itests/hcatalog-unit/src/test/java/org/apache/hive/hcatalog/listener/TestDbNotificationListener.java
9614114083
itests/hive-unit/src/test/java/org/apache/hadoop/hive/metastore/TestReplChangeManager.java
6ade76d0c2
ql/src/java/org/apache/hadoop/hive/ql/exec/tez/CustomPartitionVertex.java
26afe90faa
ql/src/java/org/apache/hadoop/hive/ql/exec/tez/CustomVertexConfiguration.java
ef5e7edcd6
ql/src/java/org/apache/hadoop/hive/ql/exec/tez/DagUtils.java 9885038588
ql/src/java/org/apache/hadoop/hive/ql/metadata/Table.java 9b0ffe0e91
ql/src/java/org/apache/hadoop/hive/ql/optimizer/ConvertJoinMapJoin.java
dc698c8de8
ql/src/java/org/apache/hadoop/hive/ql/optimizer/metainfo/annotation/OpTraitsRulesProcFactory.java
69d9f3125a
ql/src/java/org/apache/hadoop/hive/ql/optimizer/spark/SparkMapJoinOptimizer.java
bacc44482a
ql/src/java/org/apache/hadoop/hive/ql/parse/LoadSemanticAnalyzer.java
5868d4dd56
ql/src/java/org/apache/hadoop/hive/ql/plan/OpTraits.java 9621c3be53
ql/src/test/queries/clientpositive/auto_sortmerge_join_2.q e5fdcb57e4
ql/src/test/queries/clientpositive/auto_sortmerge_join_4.q abf09e5534
ql/src/test/queries/clientpositive/auto_sortmerge_join_5.q b85c4a7aa3
ql/src/test/queries/clientpositive/auto_sortmerge_join_7.q bd780861e3
ql/src/test/results/clientpositive/auto_sortmerge_join_2.q.out 5cfc35aa73
ql/src/test/results/clientpositive/auto_sortmerge_join_4.q.out 0d586fd26b
ql/src/test/results/clientpositive/auto_sortmerge_join_5.q.out 45704d1253
ql/src/test/results/clientpositive/auto_sortmerge_join_7.q.out 1959075912
ql/src/test/results/clientpositive/llap/auto_sortmerge_join_2.q.out
054b0d00be
ql/src/test/results/clientpositive/llap/auto_sortmerge_join_4.q.out
95d329862c
ql/src/test/results/clientpositive/llap/auto_sortmerge_join_5.q.out
e711715aa5
ql/src/test/results/clientpositive/llap/auto_sortmerge_join_7.q.out
53c685cb11
ql/src/test/results/clientpositive/spark/auto_sortmerge_join_2.q.out
8cfa113794
ql/src/test/results/clientpositive/spark/auto_sortmerge_join_4.q.out
fce5e0cfc4
ql/src/test/results/clientpositive/spark/auto_sortmerge_join_5.q.out
8250eca099
ql/src/test/results/clientpositive/spark/auto_sortmerge_join_7.q.out
eb813c1734
standalone-metastore/src/gen/thrift/gen-cpp/hive_metastore_types.h df646a7d17
standalone-metastore/src/gen/thrift/gen-cpp/hive_metastore_types.cpp
27f8c0f2fc
standalone-metastore/src/gen/thrift/gen-javabean/org/apache/hadoop/hive/metastore/api/Table.java
f317b0393f
standalone-metastore/src/gen/thrift/gen-php/metastore/Types.php 6878ee1be7
standalone-metastore/src/gen/thrift/gen-py/hive_metastore/ttypes.py
25e9a889b2
standalone-metastore/src/gen/thrift/gen-rb/hive_metastore_types.rb 3a11a0582a
standalone-metastore/src/main/java/org/apache/hadoop/hive/metastore/ObjectStore.java
b3d99a1da5
standalone-metastore/src/main/java/org/apache/hadoop/hive/metastore/client/builder/TableBuilder.java
69acf3cfff
standalone-metastore/src/main/java/org/apache/hadoop/hive/metastore/model/MTable.java
6c40ae8753
standalone-metastore/src/main/thrift/hive_metastore.thrift 93f3e53de2
standalone-metastore/src/test/java/org/apache/hadoop/hive/metastore/TestHiveMetaStorePartitionSpecs.java
57e5a4126e
standalone-metastore/src/test/java/org/apache/hadoop/hive/metastore/TestObjectStore.java
372dee6369
standalone-metastore/src/test/java/org/apache/hadoop/hive/metastore/TestOldSchema.java
6a44833a67
standalone-metastore/src/test/java/org/apache/hadoop/hive/metastore/cache/TestCachedStore.java
b9a8f61c69
Diff: https://reviews.apache.org/r/65130/diff/5/
Changes: https://reviews.apache.org/r/65130/diff/4-5/
Testing
-------
Thanks,
Deepak Jaiswal
