Impala Public Jenkins has submitted this change and it was merged. ( http://gerrit.cloudera.org:8080/19383 )
Change subject: IMPALA-11809: Support non unique primary key for Kudu ...................................................................... IMPALA-11809: Support non unique primary key for Kudu Kudu engine recently enables the auto-incrementing column feature (KUDU-1945). The feature works by appending a system generated auto-incrementing column to the primary key columns to guarantee the uniqueness on primary key when the primary key columns can be non unique. The non unique primary key columns and the auto-incrementing column form the effective unique composite primary key. This auto-incrementing column is named as 'auto_incrementing_id' with big int type. The assignment to it during insertion is automatic so insertion statements should not specify values for auto-incrementing column. In current Kudu implementation, there is no central key provider for auto-incrementing columns. It uses a per tablet-server global counter to assign values for auto-incrementing columns. So the values of auto-incrementing columns are not unique in a Kudu table, but unique within a continuous region of the table served by a tablet-server. This patch also upgraded Kudu version to 345fd44ca3 to pick up Kudu changes needed for supporting non-unique primary key. It added syntactic support for creating Kudu table with non unique primary key. When creating a Kudu table, specifying PRIMARY KEY is optional. If there is no primary key attribute specified, the partition key columns will be promoted as non unique primary key if those columns are the beginning columns of the table. New column "key_unique" is added to the output of 'describe' table command for Kudu table. Examples of CREATE TABLE statement with non unique primary key: CREATE TABLE tbl (i INT NON UNIQUE PRIMARY KEY, s STRING) PARTITION BY HASH (i) PARTITIONS 3 STORED as KUDU; CREATE TABLE tbl (i INT, s STRING, NON UNIQUE PRIMARY KEY(i)) PARTITION BY HASH (i) PARTITIONS 3 STORED as KUDU; CREATE TABLE tbl NON UNIQUE PRIMARY KEY(id) PARTITION BY HASH (id) PARTITIONS 3 STORED as KUDU AS SELECT id, string_col FROM functional.alltypes WHERE id = 10; CREATE TABLE tbl NON UNIQUE PRIMARY KEY(id) PARTITION BY RANGE (id) (PARTITION VALUES <= 1000, PARTITION 1000 < VALUES <= 2000, PARTITION 2000 < VALUES <= 3000, PARTITION 3000 < VALUES) STORED as KUDU AS SELECT id, int_col FROM functional.alltypestiny ORDER BY id ASC LIMIT 4000; CREATE TABLE tbl (id INT, name STRING, NON UNIQUE PRIMARY KEY(id)) STORED as KUDU; CREATE TABLE tbl (a INT, b STRING, c FLOAT) PARTITION BY HASH (a, b) PARTITIONS 3 STORED as KUDU; SELECT statement does not show the system generated auto-incrementing column unless the column is explicitly specified in the select list. Auto-incrementing column cannot be added, removed or renamed with ALTER TABLE statements. UPSERT operation is not supported now for Kudu tables with auto incrementing column due to limitation in Kudu engine. Testing: - Ran manual test in impala-shell with queries to create Kudu tables with non unique primary key, and tested insert/update/delete operations for these tables with non unique primary key. - Added front end tests, and end to end unit tests for Kudu tables with non unique primary key. - Passed exhaustive test. Change-Id: I4d7882bf3d01a3492cc9827c072d1f3200d9eebd Reviewed-on: http://gerrit.cloudera.org:8080/19383 Reviewed-by: Riza Suminto <riza.sumi...@cloudera.com> Reviewed-by: Wenzhe Zhou <wz...@cloudera.com> Tested-by: Impala Public Jenkins <impala-public-jenk...@cloudera.com> --- M bin/impala-config.sh M common/thrift/CatalogObjects.thrift M common/thrift/JniCatalog.thrift M fe/src/main/cup/sql-parser.cup M fe/src/main/java/org/apache/impala/analysis/AlterTableAddColsStmt.java M fe/src/main/java/org/apache/impala/analysis/AlterTableAlterColStmt.java M fe/src/main/java/org/apache/impala/analysis/ColumnDef.java M fe/src/main/java/org/apache/impala/analysis/CreateTableAsSelectStmt.java M fe/src/main/java/org/apache/impala/analysis/CreateTableLikeFileStmt.java M fe/src/main/java/org/apache/impala/analysis/CreateTableStmt.java M fe/src/main/java/org/apache/impala/analysis/InsertStmt.java M fe/src/main/java/org/apache/impala/analysis/ModifyStmt.java M fe/src/main/java/org/apache/impala/analysis/SelectStmt.java M fe/src/main/java/org/apache/impala/analysis/TableDef.java M fe/src/main/java/org/apache/impala/analysis/ToSqlUtils.java M fe/src/main/java/org/apache/impala/catalog/Db.java M fe/src/main/java/org/apache/impala/catalog/FeDb.java M fe/src/main/java/org/apache/impala/catalog/FeKuduTable.java M fe/src/main/java/org/apache/impala/catalog/KuduColumn.java M fe/src/main/java/org/apache/impala/catalog/KuduTable.java M fe/src/main/java/org/apache/impala/catalog/local/LocalDb.java M fe/src/main/java/org/apache/impala/catalog/local/LocalKuduTable.java M fe/src/main/java/org/apache/impala/service/DescribeResultFactory.java M fe/src/main/java/org/apache/impala/service/Frontend.java M fe/src/main/java/org/apache/impala/service/KuduCatalogOpExecutor.java M fe/src/main/java/org/apache/impala/util/KuduUtil.java M fe/src/main/jflex/sql-scanner.flex M fe/src/test/java/org/apache/impala/analysis/AnalyzeDDLTest.java M fe/src/test/java/org/apache/impala/analysis/AnalyzeKuduDDLTest.java M fe/src/test/java/org/apache/impala/analysis/ParserTest.java M testdata/workloads/functional-query/queries/QueryTest/kudu-scan-node.test M testdata/workloads/functional-query/queries/QueryTest/kudu_alter.test M testdata/workloads/functional-query/queries/QueryTest/kudu_create.test M testdata/workloads/functional-query/queries/QueryTest/kudu_delete.test M testdata/workloads/functional-query/queries/QueryTest/kudu_describe.test M testdata/workloads/functional-query/queries/QueryTest/kudu_hms_alter.test M testdata/workloads/functional-query/queries/QueryTest/kudu_insert.test M testdata/workloads/functional-query/queries/QueryTest/kudu_partition_ddl.test M testdata/workloads/functional-query/queries/QueryTest/kudu_stats.test M testdata/workloads/functional-query/queries/QueryTest/kudu_update.test M testdata/workloads/functional-query/queries/QueryTest/kudu_upsert.test M tests/custom_cluster/test_kudu.py M tests/metadata/test_ddl_base.py M tests/query_test/test_kudu.py 44 files changed, 1,353 insertions(+), 207 deletions(-) Approvals: Riza Suminto: Looks good to me, but someone else must approve Wenzhe Zhou: Looks good to me, approved Impala Public Jenkins: Verified -- To view, visit http://gerrit.cloudera.org:8080/19383 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-MessageType: merged Gerrit-Change-Id: I4d7882bf3d01a3492cc9827c072d1f3200d9eebd Gerrit-Change-Number: 19383 Gerrit-PatchSet: 19 Gerrit-Owner: Wenzhe Zhou <wz...@cloudera.com> Gerrit-Reviewer: Abhishek Chennaka <achenn...@cloudera.com> Gerrit-Reviewer: Alexey Serbin <ale...@apache.org> Gerrit-Reviewer: Impala Public Jenkins <impala-public-jenk...@cloudera.com> Gerrit-Reviewer: Kurt Deschler <kdesc...@cloudera.com> Gerrit-Reviewer: Marton Greber <greber...@gmail.com> Gerrit-Reviewer: Qifan Chen <qfc...@hotmail.com> Gerrit-Reviewer: Riza Suminto <riza.sumi...@cloudera.com> Gerrit-Reviewer: Wenzhe Zhou <wz...@cloudera.com>
