[Impala-ASF-CR] WIP IMPALA-11809: Support non unique primary key for Kudu
Qifan Chen has posted comments on this change. (
http://gerrit.cloudera.org:8080/19383 )
Change subject: WIP IMPALA-11809: Support non unique primary key for Kudu
..
Patch Set 10:
(13 comments)
Added some comments on testing and the commit message.
http://gerrit.cloudera.org:8080/#/c/19383/10//COMMIT_MSG
Commit Message:
http://gerrit.cloudera.org:8080/#/c/19383/10//COMMIT_MSG@30
PS10, Line 30: Kudu assign values for auto-incrementing column automatically
: when inserting rows so insertion statements don't need to specify
: values for auto-incrementing column.
Kudu recently enables the auto-incrementing column feature. The feature works
by appending a system generated column to the primary key columns to guarantee
the uniqueness on primary key, when the primary key columns can be non-unique.
This auto column is of type big int and the assignment to it during insertion
is automatic.
http://gerrit.cloudera.org:8080/#/c/19383/10/fe/src/test/java/org/apache/impala/analysis/AnalyzeDDLTest.java
File fe/src/test/java/org/apache/impala/analysis/AnalyzeDDLTest.java:
http://gerrit.cloudera.org:8080/#/c/19383/10/fe/src/test/java/org/apache/impala/analysis/AnalyzeDDLTest.java@2365
PS10, Line 2365: partition by hash
may add some more test cases with the following.
1. No partition by clause
2. Non unique primary key includes all selected columns
It seems even when primary key list is empty, the auto incrementing column can
still function as the primary key. Do we allow such a case?
http://gerrit.cloudera.org:8080/#/c/19383/10/fe/src/test/java/org/apache/impala/analysis/AnalyzeKuduDDLTest.java
File fe/src/test/java/org/apache/impala/analysis/AnalyzeKuduDDLTest.java:
http://gerrit.cloudera.org:8080/#/c/19383/10/fe/src/test/java/org/apache/impala/analysis/AnalyzeKuduDDLTest.java@88
PS10, Line 88: AnalyzesOk("create table tab (x int non unique primary key)
partition by hash(x) " +
: "partitions 8 stored as kudu", isExternalPurgeTbl);
: AnalyzesOk("create table tab (x int, non unique primary
key(x)) " +
: "partition by hash(x) partitions 8 stored as kudu",
isExternalPurgeTbl);
: AnalyzesOk("create table tab (x int, y int, non unique
primary key (x, y)) " +
: "partition by hash(x, y) partitions 8 stored as kudu",
isExternalPurgeTbl);
: AnalyzesOk("create table tab (x int, y int, non unique
primary key (x)) " +
: "partition by hash(x) partitions 8 stored as kudu",
isExternalPurgeTbl);
: AnalyzesOk("create table tab (x int, y int, non unique
primary key(x, y)) " +
: "partition by hash(y) partitions 8 stored as kudu",
isExternalPurgeTbl);
: AnalyzesOk("create table tab (x timestamp, y timestamp, non
unique primary key(x)) "+
: "partition by hash(x) partitions 8 stored as kudu",
isExternalPurgeTbl);
: // Promote all partition columns as non unique primary key
columns if primary keys
: // are not declared, but partition columns must be the first
columns in the table.
: AnalyzesOk("create table tab (x int, y int) partition by
hash(x) partitions 8 " +
: "stored as kudu", isExternalPurgeTbl);
: AnalyzesOk("create table tab (x int, y int) partition by
hash(x, y) partitions 8 " +
: "stored as kudu", isExternalPurgeTbl);
: AnalysisError("create table tab (x int, y int) partition by
hash(y) partitions 8 " +
: "stored as kudu", "Specify primary key or non unique
primary key for the Kudu " +
: "table, or create partitions with the beginning columns
of the table.",
: isExternalPurgeTbl);
In some of the new tests, may try partition by range and no partition to cover
more cases. See https://kudu.apache.org/docs/kudu_impala_integration.html.
http://gerrit.cloudera.org:8080/#/c/19383/10/testdata/workloads/functional-query/queries/QueryTest/kudu-scan-node.test
File testdata/workloads/functional-query/queries/QueryTest/kudu-scan-node.test:
http://gerrit.cloudera.org:8080/#/c/19383/10/testdata/workloads/functional-query/queries/QueryTest/kudu-scan-node.test@204
PS10, Line 204: order by id
Wonder if we can order DESC by auto_incrementing_id which can be useful to get
the latest rows.
http://gerrit.cloudera.org:8080/#/c/19383/10/testdata/workloads/functional-query/queries/QueryTest/kudu_alter.test
File testdata/workloads/functional-query/queries/QueryTest/kudu_alter.test:
http://gerrit.cloudera.org:8080/#/c/19383/10/testdata/workloads/functional-query/queries/QueryTest/kudu_alter.test@709
PS10, Line 709: auto-incrementing column auto_incrementing_id
nit. The column name appeared twice.
http://gerrit.cloudera.org:
[Impala-ASF-CR] WIP IMPALA-11809: Support non unique primary key for Kudu
Impala Public Jenkins has posted comments on this change. ( http://gerrit.cloudera.org:8080/19383 ) Change subject: WIP IMPALA-11809: Support non unique primary key for Kudu .. Patch Set 10: Build Failed https://jenkins.impala.io/job/gerrit-code-review-checks/12176/ : Initial code review checks failed. See linked job for details on the failure. -- To view, visit http://gerrit.cloudera.org:8080/19383 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-MessageType: comment Gerrit-Change-Id: I4d7882bf3d01a3492cc9827c072d1f3200d9eebd Gerrit-Change-Number: 19383 Gerrit-PatchSet: 10 Gerrit-Owner: Wenzhe Zhou Gerrit-Reviewer: Abhishek Chennaka Gerrit-Reviewer: Alexey Serbin Gerrit-Reviewer: Impala Public Jenkins Gerrit-Reviewer: Kurt Deschler Gerrit-Reviewer: Marton Greber Gerrit-Reviewer: Qifan Chen Gerrit-Reviewer: Wenzhe Zhou Gerrit-Comment-Date: Tue, 17 Jan 2023 06:34:16 + Gerrit-HasComments: No
[Impala-ASF-CR] WIP IMPALA-11809: Support non unique primary key for Kudu
Wenzhe Zhou has uploaded a new patch set (#10). ( http://gerrit.cloudera.org:8080/19383 ) Change subject: WIP IMPALA-11809: Support non unique primary key for Kudu .. WIP IMPALA-11809: Support non unique primary key for Kudu Kudu engine recently adds support for non unique primary key. Kudu server automatically adds an auto-incrementing column named 'auto_increment_id' in a Kudu table which has non unique primary key. The non unique primary key and 'auto_increment_id' form the effective unique composite primary key. This patch integrated new version of Kudu which support non unique primary key, added syntactic support for creating table with non unique primary key. Example: CREATE TABLE tbl (i INT NON UNIQUE PRIMARY KEY, s STRING) PARTITION BY HASH (i) partitions 3 STORED as KUDU; CREATE TABLE tbl (i INT, s STRING, NON UNIQUE PRIMARY KEY(i)) PARTITION BY HASH (i) partitions 3 STORED as KUDU; CREATE TABLE tbl NON UNIQUE PRIMARY KEY(id) PARTITION BY HASH (id) partitions 3 STORED as KUDU AS SELECT id, string_col FROM functional.alltypes WHERE id = 10; Kudu assign values for auto-incrementing column automatically when inserting rows so insertion statements don't need to specify values for auto-incrementing column. SELECT statement does not show auto-incrementing column unless the column is explicitly specified in select list. UPSERT operation is not supported now for Kudu table with auto incrementing column due to limitation in Kudu engine. When creating a Kudu table, specifying PRIMARY KEY is optional. If there is no primary key attribute specified, the partition key columes will be promoted as non unique primary key if those columns are the beginning columns of the table. New column "key_unique" is added to the output of 'describe' table command for Kudu table. Testing: - Integrated new version of Kudu built on local machine, ran manual test in impala-shell with queries to create tables with non unique primary key, and tested insert/update/delete operations for Kudu tables with non unique primary key. - Added front end and end to end unit tests. Passed query_test/test_kudu.py and custom_cluster/test_kudu.py on local environment with new version of Kudu built on local machine. - TODO build toolchian with new version of Kudu, including the commits for KUDU-1945. Run core test. Change-Id: I4d7882bf3d01a3492cc9827c072d1f3200d9eebd --- M common/thrift/CatalogObjects.thrift M common/thrift/JniCatalog.thrift M fe/src/main/cup/sql-parser.cup M fe/src/main/java/org/apache/impala/analysis/AlterTableAddColsStmt.java M fe/src/main/java/org/apache/impala/analysis/ColumnDef.java M fe/src/main/java/org/apache/impala/analysis/CreateTableAsSelectStmt.java M fe/src/main/java/org/apache/impala/analysis/CreateTableLikeFileStmt.java M fe/src/main/java/org/apache/impala/analysis/CreateTableStmt.java M fe/src/main/java/org/apache/impala/analysis/InsertStmt.java M fe/src/main/java/org/apache/impala/analysis/SelectStmt.java M fe/src/main/java/org/apache/impala/analysis/TableDef.java M fe/src/main/java/org/apache/impala/analysis/ToSqlUtils.java M fe/src/main/java/org/apache/impala/catalog/Db.java M fe/src/main/java/org/apache/impala/catalog/FeDb.java M fe/src/main/java/org/apache/impala/catalog/FeKuduTable.java M fe/src/main/java/org/apache/impala/catalog/KuduColumn.java M fe/src/main/java/org/apache/impala/catalog/KuduTable.java M fe/src/main/java/org/apache/impala/catalog/local/LocalDb.java M fe/src/main/java/org/apache/impala/catalog/local/LocalKuduTable.java M fe/src/main/java/org/apache/impala/service/DescribeResultFactory.java M fe/src/main/java/org/apache/impala/service/Frontend.java M fe/src/main/java/org/apache/impala/service/KuduCatalogOpExecutor.java M fe/src/main/java/org/apache/impala/util/KuduUtil.java M fe/src/main/jflex/sql-scanner.flex M fe/src/test/java/org/apache/impala/analysis/AnalyzeDDLTest.java M fe/src/test/java/org/apache/impala/analysis/AnalyzeKuduDDLTest.java M fe/src/test/java/org/apache/impala/analysis/ParserTest.java M testdata/workloads/functional-query/queries/QueryTest/kudu-scan-node.test M testdata/workloads/functional-query/queries/QueryTest/kudu_alter.test M testdata/workloads/functional-query/queries/QueryTest/kudu_create.test M testdata/workloads/functional-query/queries/QueryTest/kudu_delete.test M testdata/workloads/functional-query/queries/QueryTest/kudu_describe.test M testdata/workloads/functional-query/queries/QueryTest/kudu_hms_alter.test M testdata/workloads/functional-query/queries/QueryTest/kudu_insert.test M testdata/workloads/functional-query/queries/QueryTest/kudu_partition_ddl.test M testdata/workloads/functional-query/queries/QueryTest/kudu_stats.test M testdata/workloads/functional-query/queries/QueryTest/kudu_update.test M testdata/workloads/functional-query/queries/QueryTest/kudu_upsert.test M tests/custom_cluster/test_kudu.py M tests/query_test/test_kudu.py 4
[Impala-ASF-CR] WIP IMPALA-11809: Support non unique primary key for Kudu
Wenzhe Zhou has posted comments on this change. (
http://gerrit.cloudera.org:8080/19383 )
Change subject: WIP IMPALA-11809: Support non unique primary key for Kudu
..
Patch Set 9:
(16 comments)
Thanks Qifan for your review.
http://gerrit.cloudera.org:8080/#/c/19383/9//COMMIT_MSG
Commit Message:
http://gerrit.cloudera.org:8080/#/c/19383/9//COMMIT_MSG@9
PS9, Line 9: Kudu engine adds support for non unique primary key. It adds
> nit recently
Done
http://gerrit.cloudera.org:8080/#/c/19383/9/common/thrift/CatalogObjects.thrift
File common/thrift/CatalogObjects.thrift:
http://gerrit.cloudera.org:8080/#/c/19383/9/common/thrift/CatalogObjects.thrift@580
PS9, Line 580: add
> nit adds
Done
http://gerrit.cloudera.org:8080/#/c/19383/9/common/thrift/CatalogObjects.thrift@581
PS9, Line 581: .
> nit. ", in this case, this field is set to false"
Done
http://gerrit.cloudera.org:8080/#/c/19383/9/fe/src/main/java/org/apache/impala/analysis/AlterTableAddColsStmt.java
File fe/src/main/java/org/apache/impala/analysis/AlterTableAddColsStmt.java:
http://gerrit.cloudera.org:8080/#/c/19383/9/fe/src/main/java/org/apache/impala/analysis/AlterTableAddColsStmt.java@99
PS9, Line 99: c.isPrimaryKeyUnique() ? "" : "non unique ", c.toString()
> This statement can be integrated into ColumnDef.toString().
use KuduUtil.getPrimaryKeyString()
http://gerrit.cloudera.org:8080/#/c/19383/9/fe/src/main/java/org/apache/impala/analysis/CreateTableAsSelectStmt.java
File fe/src/main/java/org/apache/impala/analysis/CreateTableAsSelectStmt.java:
http://gerrit.cloudera.org:8080/#/c/19383/9/fe/src/main/java/org/apache/impala/analysis/CreateTableAsSelectStmt.java@227
PS9, Line 227: createStmt_.isPrimaryKeyUnique(),
createStmt_.getPrimaryKeyColumnDefs(),
> nit. It may be more logic to switch the two arguments, as the new argument
Done
http://gerrit.cloudera.org:8080/#/c/19383/9/fe/src/main/java/org/apache/impala/analysis/TableDef.java
File fe/src/main/java/org/apache/impala/analysis/TableDef.java:
http://gerrit.cloudera.org:8080/#/c/19383/9/fe/src/main/java/org/apache/impala/analysis/TableDef.java@87
PS9, Line 87: An auto-incrementing column will be added
: // automatically by Kudu engine as key column if primary key is
not unique.
> nit. If not, an auto-incrementing column will be added
Done
http://gerrit.cloudera.org:8080/#/c/19383/9/fe/src/main/java/org/apache/impala/analysis/TableDef.java@537
PS9, Line 537: only the columns, on which the
: // partitions are being created,
> nit. the partition columns have to be first in the primary key columns.
Done
http://gerrit.cloudera.org:8080/#/c/19383/9/fe/src/main/java/org/apache/impala/analysis/TableDef.java@566
PS9, Line 566: throw new AnalysisException(String.format("Multiple
%sprimary keys specified. " +
: "Composite %sprimary keys can be specified using the "
+
: "%sPRIMARY KEY (col1, col2, ...) syntax at the end of
the column definition.",
: isPrimaryKeyUnique() ? "" : "non unique ",
: isPrimaryKeyUnique() ? "" : "non unique ",
: isPrimaryKeyUnique() ? "" : "NON UNIQUE "));
> nit. This looks similar to the lines starting at 520.
use KuduUtil.getPrimaryKeyString()
http://gerrit.cloudera.org:8080/#/c/19383/9/fe/src/main/java/org/apache/impala/catalog/Db.java
File fe/src/main/java/org/apache/impala/catalog/Db.java:
http://gerrit.cloudera.org:8080/#/c/19383/9/fe/src/main/java/org/apache/impala/catalog/Db.java@242
PS9, Line 242: boolean isPrimaryKeyUnique, List primaryKeyColumnDefs,
> As noted in another comment, these two arguments probably should be swapped
Done
http://gerrit.cloudera.org:8080/#/c/19383/9/fe/src/main/java/org/apache/impala/catalog/KuduColumn.java
File fe/src/main/java/org/apache/impala/catalog/KuduColumn.java:
http://gerrit.cloudera.org:8080/#/c/19383/9/fe/src/main/java/org/apache/impala/catalog/KuduColumn.java@134
PS9, Line 134: position
> nit. 'position'.
Done
http://gerrit.cloudera.org:8080/#/c/19383/9/fe/src/main/java/org/apache/impala/catalog/KuduColumn.java@135
PS9, Line 135: if
> nit. when
Done
http://gerrit.cloudera.org:8080/#/c/19383/9/fe/src/main/java/org/apache/impala/catalog/KuduColumn.java@144
PS9, Line 144: false
> One may call this function even when column.isIs_primary_key_unique() is tr
This is a static function and it is only called when the primary key of the
table is not unique.
http://gerrit.cloudera.org:8080/#/c/19383/9/fe/src/main/java/org/apache/impala/service/KuduCatalogOpExecutor.java
File fe/src/main/java/org/apache/impala/service/KuduCatalogOpExecutor.java:
http://gerrit.cloudera.org:8080/#/c/19383/9/fe/src/main/java/org/apache/impala/service/KuduCatalogOpExecutor.java@133
PS9, Line 133: if (isKey && !isKeyUnique) {
: csb.nonUniqueKey(true);
: } else {
[Impala-ASF-CR] WIP IMPALA-11809: Support non unique primary key for Kudu
Qifan Chen has posted comments on this change. (
http://gerrit.cloudera.org:8080/19383 )
Change subject: WIP IMPALA-11809: Support non unique primary key for Kudu
..
Patch Set 9:
(16 comments)
Looks good!
Will review the test part of the patch next.
http://gerrit.cloudera.org:8080/#/c/19383/9//COMMIT_MSG
Commit Message:
http://gerrit.cloudera.org:8080/#/c/19383/9//COMMIT_MSG@9
PS9, Line 9: Kudu engine adds support for non unique primary key. It adds
nit recently
http://gerrit.cloudera.org:8080/#/c/19383/9/common/thrift/CatalogObjects.thrift
File common/thrift/CatalogObjects.thrift:
http://gerrit.cloudera.org:8080/#/c/19383/9/common/thrift/CatalogObjects.thrift@580
PS9, Line 580: add
nit adds
http://gerrit.cloudera.org:8080/#/c/19383/9/common/thrift/CatalogObjects.thrift@581
PS9, Line 581: .
nit. ", in this case, this field is set to false"
http://gerrit.cloudera.org:8080/#/c/19383/9/fe/src/main/java/org/apache/impala/analysis/AlterTableAddColsStmt.java
File fe/src/main/java/org/apache/impala/analysis/AlterTableAddColsStmt.java:
http://gerrit.cloudera.org:8080/#/c/19383/9/fe/src/main/java/org/apache/impala/analysis/AlterTableAddColsStmt.java@99
PS9, Line 99: c.isPrimaryKeyUnique() ? "" : "non unique ", c.toString()
This statement can be integrated into ColumnDef.toString().
http://gerrit.cloudera.org:8080/#/c/19383/9/fe/src/main/java/org/apache/impala/analysis/CreateTableAsSelectStmt.java
File fe/src/main/java/org/apache/impala/analysis/CreateTableAsSelectStmt.java:
http://gerrit.cloudera.org:8080/#/c/19383/9/fe/src/main/java/org/apache/impala/analysis/CreateTableAsSelectStmt.java@227
PS9, Line 227: createStmt_.isPrimaryKeyUnique(),
createStmt_.getPrimaryKeyColumnDefs(),
nit. It may be more logic to switch the two arguments, as the new argument
further refines the primary key columns.
http://gerrit.cloudera.org:8080/#/c/19383/9/fe/src/main/java/org/apache/impala/analysis/TableDef.java
File fe/src/main/java/org/apache/impala/analysis/TableDef.java:
http://gerrit.cloudera.org:8080/#/c/19383/9/fe/src/main/java/org/apache/impala/analysis/TableDef.java@87
PS9, Line 87: An auto-incrementing column will be added
: // automatically by Kudu engine as key column if primary key is
not unique.
nit. If not, an auto-incrementing column will be added
automatically by Kudu engine. This extra key column helps produce a unique
composite primary key (primary keys + auto-incrementing construct).
http://gerrit.cloudera.org:8080/#/c/19383/9/fe/src/main/java/org/apache/impala/analysis/TableDef.java@537
PS9, Line 537: only the columns, on which the
: // partitions are being created,
nit. the partition columns have to be first in the primary key columns.
http://gerrit.cloudera.org:8080/#/c/19383/9/fe/src/main/java/org/apache/impala/analysis/TableDef.java@566
PS9, Line 566: throw new AnalysisException(String.format("Multiple
%sprimary keys specified. " +
: "Composite %sprimary keys can be specified using the "
+
: "%sPRIMARY KEY (col1, col2, ...) syntax at the end of
the column definition.",
: isPrimaryKeyUnique() ? "" : "non unique ",
: isPrimaryKeyUnique() ? "" : "non unique ",
: isPrimaryKeyUnique() ? "" : "NON UNIQUE "));
nit. This looks similar to the lines starting at 520.
http://gerrit.cloudera.org:8080/#/c/19383/9/fe/src/main/java/org/apache/impala/catalog/Db.java
File fe/src/main/java/org/apache/impala/catalog/Db.java:
http://gerrit.cloudera.org:8080/#/c/19383/9/fe/src/main/java/org/apache/impala/catalog/Db.java@242
PS9, Line 242: boolean isPrimaryKeyUnique, List primaryKeyColumnDefs,
As noted in another comment, these two arguments probably should be swapped.
http://gerrit.cloudera.org:8080/#/c/19383/9/fe/src/main/java/org/apache/impala/catalog/KuduColumn.java
File fe/src/main/java/org/apache/impala/catalog/KuduColumn.java:
http://gerrit.cloudera.org:8080/#/c/19383/9/fe/src/main/java/org/apache/impala/catalog/KuduColumn.java@134
PS9, Line 134: position
nit. 'position'.
http://gerrit.cloudera.org:8080/#/c/19383/9/fe/src/main/java/org/apache/impala/catalog/KuduColumn.java@135
PS9, Line 135: if
nit. when
http://gerrit.cloudera.org:8080/#/c/19383/9/fe/src/main/java/org/apache/impala/catalog/KuduColumn.java@144
PS9, Line 144: false
One may call this function even when column.isIs_primary_key_unique() is true.
Suggest to replace false with column.isIs_primary_key_unique().
http://gerrit.cloudera.org:8080/#/c/19383/9/fe/src/main/java/org/apache/impala/service/KuduCatalogOpExecutor.java
File fe/src/main/java/org/apache/impala/service/KuduCatalogOpExecutor.java:
http://gerrit.cloudera.org:8080/#/c/19383/9/fe/src/main/java/org/apache/impala/service/KuduCatalogOpExecutor.java@133
PS9, Line 133: if (isKey && !isKeyUnique) {
: csb.nonUniqueKey(t
[Impala-ASF-CR] WIP IMPALA-11809: Support non unique primary key for Kudu
Impala Public Jenkins has posted comments on this change. ( http://gerrit.cloudera.org:8080/19383 ) Change subject: WIP IMPALA-11809: Support non unique primary key for Kudu .. Patch Set 9: Build Failed https://jenkins.impala.io/job/gerrit-code-review-checks/12145/ : Initial code review checks failed. See linked job for details on the failure. -- To view, visit http://gerrit.cloudera.org:8080/19383 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-MessageType: comment Gerrit-Change-Id: I4d7882bf3d01a3492cc9827c072d1f3200d9eebd Gerrit-Change-Number: 19383 Gerrit-PatchSet: 9 Gerrit-Owner: Wenzhe Zhou Gerrit-Reviewer: Abhishek Chennaka Gerrit-Reviewer: Impala Public Jenkins Gerrit-Reviewer: Kurt Deschler Gerrit-Reviewer: Marton Greber Gerrit-Reviewer: Wenzhe Zhou Gerrit-Comment-Date: Wed, 11 Jan 2023 07:22:26 + Gerrit-HasComments: No
[Impala-ASF-CR] WIP IMPALA-11809: Support non unique primary key for Kudu
Wenzhe Zhou has uploaded a new patch set (#9). ( http://gerrit.cloudera.org:8080/19383 ) Change subject: WIP IMPALA-11809: Support non unique primary key for Kudu .. WIP IMPALA-11809: Support non unique primary key for Kudu Kudu engine adds support for non unique primary key. It adds a column 'auto_increment_id' automatically in a table which has non unique primary key. The non unique primary key and 'auto_increment_id' form unique composite primary key. This patch integrated new version of Kudu which support non unique primary key, added syntactic support for creating table with non unique primary key. Example: CREATE TABLE tbl (i INT NON UNIQUE PRIMARY KEY, s STRING) PARTITION BY HASH (i) partitions 3 STORED as KUDU; CREATE TABLE tbl (i INT, s STRING, NON UNIQUE PRIMARY KEY(i)) PARTITION BY HASH (i) partitions 3 STORED as KUDU; CREATE TABLE tbl NON UNIQUE PRIMARY KEY(id) PARTITION BY HASH (id) partitions 3 STORED as KUDU AS SELECT id, string_col FROM functional.alltypes WHERE id = 10; Kudu assign values for column 'auto_increment_id' automatically when inserting rows so insertion statements don't need to specify values for column 'auto_increment_id'. SELECT statement does not show 'auto_increment_id' column unless the column is explicitly specified in select list. UPSERT operation is not supported now for Kudu table with auto incrementing column due to limitation in Kudu engine. When creating a Kudu table, specifying PRIMARY KEY is optional. If there is no primary key attribute specified, the partition key columes will be promoted as non unique primary key if those columns are the beginning columns of the table. New column "key_unique" is added to the output of 'describe' table command for Kudu table. Testing: - Integrated new version of Kudu built on local machine, ran manual test in impala-shell with queries to create tables with non unique primary key, and tested insert/update/delete operations for Kudu tables with non unique primary key. - Added front end and end to end unit tests. Passed query_test/test_kudu.py and custom_cluster/test_kudu.py on local environment with new version of Kudu built on local machine. - TODO build toolchian with new version of Kudu, including the commits for KUDU-1945. Run core test. Change-Id: I4d7882bf3d01a3492cc9827c072d1f3200d9eebd --- M common/thrift/CatalogObjects.thrift M common/thrift/JniCatalog.thrift M fe/src/main/cup/sql-parser.cup M fe/src/main/java/org/apache/impala/analysis/AlterTableAddColsStmt.java M fe/src/main/java/org/apache/impala/analysis/ColumnDef.java M fe/src/main/java/org/apache/impala/analysis/CreateTableAsSelectStmt.java M fe/src/main/java/org/apache/impala/analysis/CreateTableLikeFileStmt.java M fe/src/main/java/org/apache/impala/analysis/CreateTableStmt.java M fe/src/main/java/org/apache/impala/analysis/InsertStmt.java M fe/src/main/java/org/apache/impala/analysis/SelectStmt.java M fe/src/main/java/org/apache/impala/analysis/TableDef.java M fe/src/main/java/org/apache/impala/analysis/ToSqlUtils.java M fe/src/main/java/org/apache/impala/catalog/Db.java M fe/src/main/java/org/apache/impala/catalog/FeDb.java M fe/src/main/java/org/apache/impala/catalog/FeKuduTable.java M fe/src/main/java/org/apache/impala/catalog/KuduColumn.java M fe/src/main/java/org/apache/impala/catalog/KuduTable.java M fe/src/main/java/org/apache/impala/catalog/local/LocalDb.java M fe/src/main/java/org/apache/impala/catalog/local/LocalKuduTable.java M fe/src/main/java/org/apache/impala/service/DescribeResultFactory.java M fe/src/main/java/org/apache/impala/service/Frontend.java M fe/src/main/java/org/apache/impala/service/KuduCatalogOpExecutor.java M fe/src/main/java/org/apache/impala/util/KuduUtil.java M fe/src/main/jflex/sql-scanner.flex M fe/src/test/java/org/apache/impala/analysis/AnalyzeDDLTest.java M fe/src/test/java/org/apache/impala/analysis/AnalyzeKuduDDLTest.java M fe/src/test/java/org/apache/impala/analysis/ParserTest.java M testdata/workloads/functional-query/queries/QueryTest/kudu-scan-node.test M testdata/workloads/functional-query/queries/QueryTest/kudu_alter.test M testdata/workloads/functional-query/queries/QueryTest/kudu_create.test M testdata/workloads/functional-query/queries/QueryTest/kudu_delete.test M testdata/workloads/functional-query/queries/QueryTest/kudu_describe.test M testdata/workloads/functional-query/queries/QueryTest/kudu_hms_alter.test M testdata/workloads/functional-query/queries/QueryTest/kudu_insert.test M testdata/workloads/functional-query/queries/QueryTest/kudu_partition_ddl.test M testdata/workloads/functional-query/queries/QueryTest/kudu_stats.test M testdata/workloads/functional-query/queries/QueryTest/kudu_update.test M testdata/workloads/functional-query/queries/QueryTest/kudu_upsert.test M tests/custom_cluster/test_kudu.py M tests/query_test/test_kudu.py 40 files changed, 1,105 insertions(+), 191 deletions(-)
[Impala-ASF-CR] WIP IMPALA-11809: Support non unique primary key for Kudu
Abhishek Chennaka has posted comments on this change. ( http://gerrit.cloudera.org:8080/19383 ) Change subject: WIP IMPALA-11809: Support non unique primary key for Kudu .. Patch Set 8: (4 comments) http://gerrit.cloudera.org:8080/#/c/19383/8/fe/src/test/java/org/apache/impala/analysis/AnalyzeKuduDDLTest.java File fe/src/test/java/org/apache/impala/analysis/AnalyzeKuduDDLTest.java: http://gerrit.cloudera.org:8080/#/c/19383/8/fe/src/test/java/org/apache/impala/analysis/AnalyzeKuduDDLTest.java@107 PS8, Line 107: A primary key is required for a Kudu table Is this the right Error message for this case? Additionally, if a user wants to create multiple partition levels, the columns on which the partitions are being created have to be specified first, right? In that case does the order matter as long as all the columns are the beginning of the table? http://gerrit.cloudera.org:8080/#/c/19383/8/fe/src/test/java/org/apache/impala/analysis/AnalyzeKuduDDLTest.java@715 PS8, Line 715: add Do we need an equivalent test for dropping of the columns? http://gerrit.cloudera.org:8080/#/c/19383/8/fe/src/test/java/org/apache/impala/analysis/ParserTest.java File fe/src/test/java/org/apache/impala/analysis/ParserTest.java: http://gerrit.cloudera.org:8080/#/c/19383/8/fe/src/test/java/org/apache/impala/analysis/ParserTest.java@2798 PS8, Line 2798: i INT PRIMARY KEY, NON UNIQUE PRIMARY KEY(i) This looks a bit confusing for the end user where is declared as both PRIMARY KEY and NON UNIQUE PRIMARY KEY. Do we want to allow this? Additionally what if we do something like below? What would the result be? i INT NON UNIQUE PRIMARY KEY, PRIMARY KEY(i) http://gerrit.cloudera.org:8080/#/c/19383/8/testdata/workloads/functional-query/queries/QueryTest/kudu_create.test File testdata/workloads/functional-query/queries/QueryTest/kudu_create.test: http://gerrit.cloudera.org:8080/#/c/19383/8/testdata/workloads/functional-query/queries/QueryTest/kudu_create.test@535 PS8, Line 535: A primary key is required for a Kudu table As pointed out before, maybe a more helpful message might help here? -- To view, visit http://gerrit.cloudera.org:8080/19383 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-MessageType: comment Gerrit-Change-Id: I4d7882bf3d01a3492cc9827c072d1f3200d9eebd Gerrit-Change-Number: 19383 Gerrit-PatchSet: 8 Gerrit-Owner: Wenzhe Zhou Gerrit-Reviewer: Abhishek Chennaka Gerrit-Reviewer: Impala Public Jenkins Gerrit-Reviewer: Kurt Deschler Gerrit-Reviewer: Marton Greber Gerrit-Reviewer: Wenzhe Zhou Gerrit-Comment-Date: Tue, 10 Jan 2023 19:25:02 + Gerrit-HasComments: Yes
[Impala-ASF-CR] WIP IMPALA-11809: Support non unique primary key for Kudu
Impala Public Jenkins has posted comments on this change. ( http://gerrit.cloudera.org:8080/19383 ) Change subject: WIP IMPALA-11809: Support non unique primary key for Kudu .. Patch Set 8: Build Failed https://jenkins.impala.io/job/gerrit-code-review-checks/12132/ : Initial code review checks failed. See linked job for details on the failure. -- To view, visit http://gerrit.cloudera.org:8080/19383 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-MessageType: comment Gerrit-Change-Id: I4d7882bf3d01a3492cc9827c072d1f3200d9eebd Gerrit-Change-Number: 19383 Gerrit-PatchSet: 8 Gerrit-Owner: Wenzhe Zhou Gerrit-Reviewer: Abhishek Chennaka Gerrit-Reviewer: Impala Public Jenkins Gerrit-Reviewer: Kurt Deschler Gerrit-Reviewer: Marton Greber Gerrit-Reviewer: Wenzhe Zhou Gerrit-Comment-Date: Tue, 10 Jan 2023 00:29:07 + Gerrit-HasComments: No
[Impala-ASF-CR] WIP IMPALA-11809: Support non unique primary key for Kudu
Wenzhe Zhou has uploaded a new patch set (#8). ( http://gerrit.cloudera.org:8080/19383 ) Change subject: WIP IMPALA-11809: Support non unique primary key for Kudu .. WIP IMPALA-11809: Support non unique primary key for Kudu Kudu engine adds support for non unique primary key. It adds a column 'auto_increment_id' automatically in a table which has non unique primary key. The non unique primary key and 'auto_increment_id' form unique composite primary key. This patch integrated new version of Kudu which support non unique primary key, added syntactic support for creating table with non unique primary key. Example: CREATE TABLE tbl (i INT NON UNIQUE PRIMARY KEY, s STRING) PARTITION BY HASH (i) partitions 3 STORED as KUDU; CREATE TABLE tbl (i INT, s STRING, NON UNIQUE PRIMARY KEY(i)) PARTITION BY HASH (i) partitions 3 STORED as KUDU; CREATE TABLE tbl NON UNIQUE PRIMARY KEY(id) PARTITION BY HASH (id) partitions 3 STORED as KUDU AS SELECT id, string_col FROM functional.alltypes WHERE id = 10; Kudu assign values for column 'auto_increment_id' automatically when inserting rows so insertion statements don't need to specify values for column 'auto_increment_id'. SELECT statement does not show 'auto_increment_id' column unless the column is explicitly specified in select list. UPSERT operation is not supported now for Kudu table with auto incrementing column due to limitation in Kudu engine. Testing: - Integrated new version of Kudu built on local machine, ran manual test in impala-shell with queries to create tables with non unique primary key, and tested insert/update/delete operations for Kudu tables with non unique primary key. - Added front end and end to end unit tests. Passed query_test/test_kudu.py and custom_cluster/test_kudu.py on local environment with new version of Kudu built on local machine. - TODO build toolchian with new version of Kudu, including the commits for KUDU-1945. Run core test. Change-Id: I4d7882bf3d01a3492cc9827c072d1f3200d9eebd --- M common/thrift/CatalogObjects.thrift M common/thrift/JniCatalog.thrift M fe/src/main/cup/sql-parser.cup M fe/src/main/java/org/apache/impala/analysis/AlterTableAddColsStmt.java M fe/src/main/java/org/apache/impala/analysis/ColumnDef.java M fe/src/main/java/org/apache/impala/analysis/CreateTableAsSelectStmt.java M fe/src/main/java/org/apache/impala/analysis/CreateTableLikeFileStmt.java M fe/src/main/java/org/apache/impala/analysis/CreateTableStmt.java M fe/src/main/java/org/apache/impala/analysis/InsertStmt.java M fe/src/main/java/org/apache/impala/analysis/SelectStmt.java M fe/src/main/java/org/apache/impala/analysis/TableDef.java M fe/src/main/java/org/apache/impala/analysis/ToSqlUtils.java M fe/src/main/java/org/apache/impala/catalog/Db.java M fe/src/main/java/org/apache/impala/catalog/FeDb.java M fe/src/main/java/org/apache/impala/catalog/FeKuduTable.java M fe/src/main/java/org/apache/impala/catalog/KuduColumn.java M fe/src/main/java/org/apache/impala/catalog/KuduTable.java M fe/src/main/java/org/apache/impala/catalog/local/LocalDb.java M fe/src/main/java/org/apache/impala/catalog/local/LocalKuduTable.java M fe/src/main/java/org/apache/impala/service/DescribeResultFactory.java M fe/src/main/java/org/apache/impala/service/Frontend.java M fe/src/main/java/org/apache/impala/service/KuduCatalogOpExecutor.java M fe/src/main/java/org/apache/impala/util/KuduUtil.java M fe/src/main/jflex/sql-scanner.flex M fe/src/test/java/org/apache/impala/analysis/AnalyzeDDLTest.java M fe/src/test/java/org/apache/impala/analysis/AnalyzeKuduDDLTest.java M fe/src/test/java/org/apache/impala/analysis/ParserTest.java M testdata/workloads/functional-query/queries/QueryTest/kudu-scan-node.test M testdata/workloads/functional-query/queries/QueryTest/kudu_alter.test M testdata/workloads/functional-query/queries/QueryTest/kudu_create.test M testdata/workloads/functional-query/queries/QueryTest/kudu_delete.test M testdata/workloads/functional-query/queries/QueryTest/kudu_describe.test M testdata/workloads/functional-query/queries/QueryTest/kudu_hms_alter.test M testdata/workloads/functional-query/queries/QueryTest/kudu_insert.test M testdata/workloads/functional-query/queries/QueryTest/kudu_partition_ddl.test M testdata/workloads/functional-query/queries/QueryTest/kudu_stats.test M testdata/workloads/functional-query/queries/QueryTest/kudu_update.test M testdata/workloads/functional-query/queries/QueryTest/kudu_upsert.test M tests/custom_cluster/test_kudu.py M tests/query_test/test_kudu.py 40 files changed, 1,101 insertions(+), 190 deletions(-) git pull ssh://gerrit.cloudera.org:29418/Impala-ASF refs/changes/83/19383/8 -- To view, visit http://gerrit.cloudera.org:8080/19383 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-MessageType: newpatchset Gerrit-Change-Id: I4d7882bf3d01a3492cc9827c072d1f32
[Impala-ASF-CR] WIP IMPALA-11809: Support non unique primary key for Kudu
Impala Public Jenkins has posted comments on this change. ( http://gerrit.cloudera.org:8080/19383 ) Change subject: WIP IMPALA-11809: Support non unique primary key for Kudu .. Patch Set 7: Build Failed https://jenkins.impala.io/job/gerrit-code-review-checks/12130/ : Initial code review checks failed. See linked job for details on the failure. -- To view, visit http://gerrit.cloudera.org:8080/19383 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-MessageType: comment Gerrit-Change-Id: I4d7882bf3d01a3492cc9827c072d1f3200d9eebd Gerrit-Change-Number: 19383 Gerrit-PatchSet: 7 Gerrit-Owner: Wenzhe Zhou Gerrit-Reviewer: Abhishek Chennaka Gerrit-Reviewer: Impala Public Jenkins Gerrit-Reviewer: Kurt Deschler Gerrit-Reviewer: Marton Greber Gerrit-Comment-Date: Tue, 10 Jan 2023 00:12:17 + Gerrit-HasComments: No
[Impala-ASF-CR] WIP IMPALA-11809: Support non unique primary key for Kudu
Wenzhe Zhou has uploaded a new patch set (#7). ( http://gerrit.cloudera.org:8080/19383 ) Change subject: WIP IMPALA-11809: Support non unique primary key for Kudu .. WIP IMPALA-11809: Support non unique primary key for Kudu Kudu engine adds support for non unique primary key. It adds a column 'auto_increment_id' automatically in a table which has non unique primary key. The non unique primary key and 'auto_increment_id' form unique composite primary key. This patch integrated new version of Kudu which support non unique primary key, added syntactic support for creating table with non unique primary key. Example: CREATE TABLE tbl (i INT NON UNIQUE PRIMARY KEY, s STRING) PARTITION BY HASH (i) partitions 3 STORED as KUDU; CREATE TABLE tbl (i INT, s STRING, NON UNIQUE PRIMARY KEY(i)) PARTITION BY HASH (i) partitions 3 STORED as KUDU; CREATE TABLE tbl NON UNIQUE PRIMARY KEY(id) PARTITION BY HASH (id) partitions 3 STORED as KUDU AS SELECT id, string_col FROM functional.alltypes WHERE id = 10; Kudu assign values for column 'auto_increment_id' automatically when inserting rows so insertion statements don't need to specify values for column 'auto_increment_id'. SELECT statement does not show 'auto_increment_id' column unless the column is explicitly specified in select list. UPSERT operation is not supported now for Kudu table with auto incrementing column due to limitation in Kudu engine. Testing: - Integrated new version of Kudu built on local machine, ran manual test in impala-shell with queries to create tables with non unique primary key, and tested insert/update/delete operations for Kudu tables with non unique primary key. - Added front end and end to end unit tests. Passed query_test/test_kudu.py and custom_cluster/test_kudu.py on local environment with new version of Kudu built on local machine. - TODO build toolchian with new version of Kudu, including the commits for KUDU-1945. Run core test. Change-Id: I4d7882bf3d01a3492cc9827c072d1f3200d9eebd --- M common/thrift/CatalogObjects.thrift M common/thrift/JniCatalog.thrift M fe/src/main/cup/sql-parser.cup M fe/src/main/java/org/apache/impala/analysis/AlterTableAddColsStmt.java M fe/src/main/java/org/apache/impala/analysis/ColumnDef.java M fe/src/main/java/org/apache/impala/analysis/CreateTableAsSelectStmt.java M fe/src/main/java/org/apache/impala/analysis/CreateTableLikeFileStmt.java M fe/src/main/java/org/apache/impala/analysis/CreateTableStmt.java M fe/src/main/java/org/apache/impala/analysis/InsertStmt.java M fe/src/main/java/org/apache/impala/analysis/SelectStmt.java M fe/src/main/java/org/apache/impala/analysis/TableDef.java M fe/src/main/java/org/apache/impala/analysis/ToSqlUtils.java M fe/src/main/java/org/apache/impala/catalog/Db.java M fe/src/main/java/org/apache/impala/catalog/FeDb.java M fe/src/main/java/org/apache/impala/catalog/FeKuduTable.java M fe/src/main/java/org/apache/impala/catalog/KuduColumn.java M fe/src/main/java/org/apache/impala/catalog/KuduTable.java M fe/src/main/java/org/apache/impala/catalog/local/LocalDb.java M fe/src/main/java/org/apache/impala/catalog/local/LocalKuduTable.java M fe/src/main/java/org/apache/impala/service/DescribeResultFactory.java M fe/src/main/java/org/apache/impala/service/Frontend.java M fe/src/main/java/org/apache/impala/service/KuduCatalogOpExecutor.java M fe/src/main/java/org/apache/impala/util/KuduUtil.java M fe/src/main/jflex/sql-scanner.flex M fe/src/test/java/org/apache/impala/analysis/AnalyzeDDLTest.java M fe/src/test/java/org/apache/impala/analysis/AnalyzeKuduDDLTest.java M fe/src/test/java/org/apache/impala/analysis/ParserTest.java M testdata/workloads/functional-query/queries/QueryTest/kudu-scan-node.test M testdata/workloads/functional-query/queries/QueryTest/kudu_alter.test M testdata/workloads/functional-query/queries/QueryTest/kudu_create.test M testdata/workloads/functional-query/queries/QueryTest/kudu_delete.test M testdata/workloads/functional-query/queries/QueryTest/kudu_describe.test M testdata/workloads/functional-query/queries/QueryTest/kudu_hms_alter.test M testdata/workloads/functional-query/queries/QueryTest/kudu_insert.test M testdata/workloads/functional-query/queries/QueryTest/kudu_partition_ddl.test M testdata/workloads/functional-query/queries/QueryTest/kudu_stats.test M testdata/workloads/functional-query/queries/QueryTest/kudu_update.test M testdata/workloads/functional-query/queries/QueryTest/kudu_upsert.test M tests/custom_cluster/test_kudu.py M tests/query_test/test_kudu.py 40 files changed, 1,100 insertions(+), 189 deletions(-) git pull ssh://gerrit.cloudera.org:29418/Impala-ASF refs/changes/83/19383/7 -- To view, visit http://gerrit.cloudera.org:8080/19383 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-MessageType: newpatchset Gerrit-Change-Id: I4d7882bf3d01a3492cc9827c072d1f32
[Impala-ASF-CR] WIP IMPALA-11809: Support non unique primary key for Kudu
Impala Public Jenkins has posted comments on this change. ( http://gerrit.cloudera.org:8080/19383 ) Change subject: WIP IMPALA-11809: Support non unique primary key for Kudu .. Patch Set 7: (1 comment) http://gerrit.cloudera.org:8080/#/c/19383/7/tests/query_test/test_kudu.py File tests/query_test/test_kudu.py: http://gerrit.cloudera.org:8080/#/c/19383/7/tests/query_test/test_kudu.py@667 PS7, Line 667: \ flake8: E502 the backslash is redundant between brackets -- To view, visit http://gerrit.cloudera.org:8080/19383 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-MessageType: comment Gerrit-Change-Id: I4d7882bf3d01a3492cc9827c072d1f3200d9eebd Gerrit-Change-Number: 19383 Gerrit-PatchSet: 7 Gerrit-Owner: Wenzhe Zhou Gerrit-Reviewer: Abhishek Chennaka Gerrit-Reviewer: Impala Public Jenkins Gerrit-Reviewer: Kurt Deschler Gerrit-Reviewer: Marton Greber Gerrit-Comment-Date: Tue, 10 Jan 2023 00:01:44 + Gerrit-HasComments: Yes
[Impala-ASF-CR] WIP IMPALA-11809: Support non unique primary key for Kudu
Impala Public Jenkins has posted comments on this change. ( http://gerrit.cloudera.org:8080/19383 ) Change subject: WIP IMPALA-11809: Support non unique primary key for Kudu .. Patch Set 6: Build Failed https://jenkins.impala.io/job/gerrit-code-review-checks/12125/ : Initial code review checks failed. See linked job for details on the failure. -- To view, visit http://gerrit.cloudera.org:8080/19383 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-MessageType: comment Gerrit-Change-Id: I4d7882bf3d01a3492cc9827c072d1f3200d9eebd Gerrit-Change-Number: 19383 Gerrit-PatchSet: 6 Gerrit-Owner: Wenzhe Zhou Gerrit-Reviewer: Abhishek Chennaka Gerrit-Reviewer: Impala Public Jenkins Gerrit-Reviewer: Kurt Deschler Gerrit-Reviewer: Marton Greber Gerrit-Comment-Date: Sat, 07 Jan 2023 02:20:11 + Gerrit-HasComments: No
[Impala-ASF-CR] WIP IMPALA-11809: Support non unique primary key for Kudu
Wenzhe Zhou has uploaded a new patch set (#6). ( http://gerrit.cloudera.org:8080/19383 ) Change subject: WIP IMPALA-11809: Support non unique primary key for Kudu .. WIP IMPALA-11809: Support non unique primary key for Kudu Kudu engine adds support for non unique primary key. It adds a column 'auto_increment_id' automatically in a table which has non unique primary key. The non unique primary key and 'auto_increment_id' form unique composite primary key. This patch integrated new version of Kudu which support non unique primary key, added syntactic support for creating table with non unique primary key. Example: CREATE TABLE tbl (i INT NON UNIQUE PRIMARY KEY, s STRING) PARTITION BY HASH (i) partitions 3 STORED as KUDU; CREATE TABLE tbl (i INT, s STRING, NON UNIQUE PRIMARY KEY(i)) PARTITION BY HASH (i) partitions 3 STORED as KUDU; CREATE TABLE tbl NON UNIQUE PRIMARY KEY(id) PARTITION BY HASH (id) partitions 3 STORED as KUDU AS SELECT id, string_col FROM functional.alltypes WHERE id = 10; Kudu assign values for column 'auto_increment_id' automatically when inserting rows so insertion statements don't need to specify values for column 'auto_increment_id'. SELECT statement does not show 'auto_increment_id' column unless the column is explicitly specified in select list. UPSERT operation is not supported now for Kudu table with auto incrementing column due to limitation in Kudu engine. Testing: - Integrated new version of Kudu built on local machine, ran manual test in impala-shell with queries to create tables with non unique primary key, and tested insert/update/delete operations for Kudu tables with non unique primary key. - TODO build toolchian with new version of Kudu, including the commits for KUDU-1945. Run core test. Change-Id: I4d7882bf3d01a3492cc9827c072d1f3200d9eebd --- M common/thrift/CatalogObjects.thrift M common/thrift/JniCatalog.thrift M fe/src/main/cup/sql-parser.cup M fe/src/main/java/org/apache/impala/analysis/AlterTableAddColsStmt.java M fe/src/main/java/org/apache/impala/analysis/ColumnDef.java M fe/src/main/java/org/apache/impala/analysis/CreateTableAsSelectStmt.java M fe/src/main/java/org/apache/impala/analysis/CreateTableLikeFileStmt.java M fe/src/main/java/org/apache/impala/analysis/CreateTableStmt.java M fe/src/main/java/org/apache/impala/analysis/InsertStmt.java M fe/src/main/java/org/apache/impala/analysis/SelectStmt.java M fe/src/main/java/org/apache/impala/analysis/TableDef.java M fe/src/main/java/org/apache/impala/analysis/ToSqlUtils.java M fe/src/main/java/org/apache/impala/catalog/Db.java M fe/src/main/java/org/apache/impala/catalog/FeDb.java M fe/src/main/java/org/apache/impala/catalog/FeKuduTable.java M fe/src/main/java/org/apache/impala/catalog/KuduColumn.java M fe/src/main/java/org/apache/impala/catalog/KuduTable.java M fe/src/main/java/org/apache/impala/catalog/local/LocalDb.java M fe/src/main/java/org/apache/impala/catalog/local/LocalKuduTable.java M fe/src/main/java/org/apache/impala/service/DescribeResultFactory.java M fe/src/main/java/org/apache/impala/service/Frontend.java M fe/src/main/java/org/apache/impala/service/KuduCatalogOpExecutor.java M fe/src/main/java/org/apache/impala/util/KuduUtil.java M fe/src/main/jflex/sql-scanner.flex M fe/src/test/java/org/apache/impala/analysis/AnalyzeDDLTest.java M fe/src/test/java/org/apache/impala/analysis/AnalyzeKuduDDLTest.java M fe/src/test/java/org/apache/impala/analysis/ParserTest.java M testdata/workloads/functional-query/queries/QueryTest/kudu-scan-node.test M testdata/workloads/functional-query/queries/QueryTest/kudu_alter.test M testdata/workloads/functional-query/queries/QueryTest/kudu_create.test M testdata/workloads/functional-query/queries/QueryTest/kudu_insert.test M testdata/workloads/functional-query/queries/QueryTest/kudu_update.test M testdata/workloads/functional-query/queries/QueryTest/kudu_upsert.test 33 files changed, 747 insertions(+), 103 deletions(-) git pull ssh://gerrit.cloudera.org:29418/Impala-ASF refs/changes/83/19383/6 -- To view, visit http://gerrit.cloudera.org:8080/19383 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-MessageType: newpatchset Gerrit-Change-Id: I4d7882bf3d01a3492cc9827c072d1f3200d9eebd Gerrit-Change-Number: 19383 Gerrit-PatchSet: 6 Gerrit-Owner: Wenzhe Zhou Gerrit-Reviewer: Abhishek Chennaka Gerrit-Reviewer: Impala Public Jenkins Gerrit-Reviewer: Kurt Deschler Gerrit-Reviewer: Marton Greber
[Impala-ASF-CR] WIP IMPALA-11809: Support non unique primary key for Kudu
Impala Public Jenkins has posted comments on this change. ( http://gerrit.cloudera.org:8080/19383 ) Change subject: WIP IMPALA-11809: Support non unique primary key for Kudu .. Patch Set 5: Build Failed https://jenkins.impala.io/job/gerrit-code-review-checks/12116/ : Initial code review checks failed. See linked job for details on the failure. -- To view, visit http://gerrit.cloudera.org:8080/19383 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-MessageType: comment Gerrit-Change-Id: I4d7882bf3d01a3492cc9827c072d1f3200d9eebd Gerrit-Change-Number: 19383 Gerrit-PatchSet: 5 Gerrit-Owner: Wenzhe Zhou Gerrit-Reviewer: Abhishek Chennaka Gerrit-Reviewer: Impala Public Jenkins Gerrit-Reviewer: Kurt Deschler Gerrit-Reviewer: Marton Greber Gerrit-Comment-Date: Fri, 06 Jan 2023 01:41:34 + Gerrit-HasComments: No
[Impala-ASF-CR] WIP IMPALA-11809: Support non unique primary key for Kudu
Wenzhe Zhou has uploaded a new patch set (#5). ( http://gerrit.cloudera.org:8080/19383 ) Change subject: WIP IMPALA-11809: Support non unique primary key for Kudu .. WIP IMPALA-11809: Support non unique primary key for Kudu Kudu engine adds support for non unique primary key. It adds a column 'auto_increment_id' automatically in a table which has non unique primary key. The non unique primary key and 'auto_increment_id' form unique composite primary key. This patch integrated new version of Kudu which support non unique primary key, added syntactic support for creating table with non unique primary key. Example: CREATE TABLE tbl (i INT NON UNIQUE PRIMARY KEY, s STRING) PARTITION BY HASH (i) partitions 3 STORED as KUDU; CREATE TABLE tbl (i INT, s STRING, NON UNIQUE PRIMARY KEY(i)) PARTITION BY HASH (i) partitions 3 STORED as KUDU; CREATE TABLE tbl NON UNIQUE PRIMARY KEY(id) PARTITION BY HASH (id) partitions 3 STORED as KUDU AS SELECT id, string_col FROM functional.alltypes WHERE id = 10; Kudu assign values for column 'auto_increment_id' automatically when inserting rows so insertion statements don't need to specify values for column 'auto_increment_id'. SELECT statement does not show 'auto_increment_id' column unless the column is explicitly specified in select list. UPSERT operation is not supported now for Kudu table with auto incrementing column due to limitation in Kudu engine. Testing: - Integrated new version of Kudu built on local machine, ran manual test in impala-shell with queries to create tables with non unique primary key, and tested insert/update/delete operations for Kudu tables with non unique primary key. - TODO build toolchian with new version of Kudu, including the commits for KUDU-1945. Run core test. Change-Id: I4d7882bf3d01a3492cc9827c072d1f3200d9eebd --- M common/thrift/CatalogObjects.thrift M common/thrift/JniCatalog.thrift M fe/src/main/cup/sql-parser.cup M fe/src/main/java/org/apache/impala/analysis/AlterTableAddColsStmt.java M fe/src/main/java/org/apache/impala/analysis/ColumnDef.java M fe/src/main/java/org/apache/impala/analysis/CreateTableAsSelectStmt.java M fe/src/main/java/org/apache/impala/analysis/CreateTableLikeFileStmt.java M fe/src/main/java/org/apache/impala/analysis/CreateTableStmt.java M fe/src/main/java/org/apache/impala/analysis/InsertStmt.java M fe/src/main/java/org/apache/impala/analysis/SelectStmt.java M fe/src/main/java/org/apache/impala/analysis/TableDef.java M fe/src/main/java/org/apache/impala/analysis/ToSqlUtils.java M fe/src/main/java/org/apache/impala/catalog/Db.java M fe/src/main/java/org/apache/impala/catalog/FeDb.java M fe/src/main/java/org/apache/impala/catalog/FeKuduTable.java M fe/src/main/java/org/apache/impala/catalog/KuduColumn.java M fe/src/main/java/org/apache/impala/catalog/KuduTable.java M fe/src/main/java/org/apache/impala/catalog/local/LocalDb.java M fe/src/main/java/org/apache/impala/catalog/local/LocalKuduTable.java M fe/src/main/java/org/apache/impala/service/KuduCatalogOpExecutor.java M fe/src/main/java/org/apache/impala/util/KuduUtil.java M fe/src/main/jflex/sql-scanner.flex M fe/src/test/java/org/apache/impala/analysis/AnalyzeDDLTest.java M fe/src/test/java/org/apache/impala/analysis/AnalyzeKuduDDLTest.java M fe/src/test/java/org/apache/impala/analysis/ParserTest.java M testdata/workloads/functional-query/queries/QueryTest/kudu-scan-node.test M testdata/workloads/functional-query/queries/QueryTest/kudu_alter.test M testdata/workloads/functional-query/queries/QueryTest/kudu_create.test M testdata/workloads/functional-query/queries/QueryTest/kudu_insert.test M testdata/workloads/functional-query/queries/QueryTest/kudu_update.test M testdata/workloads/functional-query/queries/QueryTest/kudu_upsert.test 31 files changed, 735 insertions(+), 101 deletions(-) git pull ssh://gerrit.cloudera.org:29418/Impala-ASF refs/changes/83/19383/5 -- To view, visit http://gerrit.cloudera.org:8080/19383 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-MessageType: newpatchset Gerrit-Change-Id: I4d7882bf3d01a3492cc9827c072d1f3200d9eebd Gerrit-Change-Number: 19383 Gerrit-PatchSet: 5 Gerrit-Owner: Wenzhe Zhou Gerrit-Reviewer: Abhishek Chennaka Gerrit-Reviewer: Impala Public Jenkins Gerrit-Reviewer: Kurt Deschler Gerrit-Reviewer: Marton Greber
[Impala-ASF-CR] WIP IMPALA-11809: Support non unique primary key for Kudu
Impala Public Jenkins has posted comments on this change. ( http://gerrit.cloudera.org:8080/19383 ) Change subject: WIP IMPALA-11809: Support non unique primary key for Kudu .. Patch Set 4: Build Failed https://jenkins.impala.io/job/gerrit-code-review-checks/12057/ : Initial code review checks failed. See linked job for details on the failure. -- To view, visit http://gerrit.cloudera.org:8080/19383 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-MessageType: comment Gerrit-Change-Id: I4d7882bf3d01a3492cc9827c072d1f3200d9eebd Gerrit-Change-Number: 19383 Gerrit-PatchSet: 4 Gerrit-Owner: Wenzhe Zhou Gerrit-Reviewer: Abhishek Chennaka Gerrit-Reviewer: Impala Public Jenkins Gerrit-Reviewer: Kurt Deschler Gerrit-Reviewer: Marton Greber Gerrit-Comment-Date: Wed, 21 Dec 2022 22:12:28 + Gerrit-HasComments: No
[Impala-ASF-CR] WIP IMPALA-11809: Support non unique primary key for Kudu
Wenzhe Zhou has uploaded a new patch set (#4). ( http://gerrit.cloudera.org:8080/19383 ) Change subject: WIP IMPALA-11809: Support non unique primary key for Kudu .. WIP IMPALA-11809: Support non unique primary key for Kudu Kudu engine adds support for non unique primary key. It adds a column 'auto_increment_id' automatically in a table which has non unique primary key. The non unique primary key and 'auto_increment_id' form unique composite primary key. This patch integrated new version of Kudu which support non unique primary key, added syntactic support for creating table with non unique primary key. Example: CREATE TABLE tbl (i INT NON UNIQUE PRIMARY KEY, s STRING) PARTITION BY HASH (i) partitions 3 STORED as KUDU; CREATE TABLE tbl (i INT, s STRING, NON UNIQUE PRIMARY KEY(i)) PARTITION BY HASH (i) partitions 3 STORED as KUDU; CREATE TABLE tbl NON UNIQUE PRIMARY KEY(id) PARTITION BY HASH (id) partitions 3 STORED as KUDU AS SELECT id, string_col FROM functional.alltypes WHERE id = 10; Kudu assign values for column 'auto_increment_id' automatically when inserting rows so insertion statements don't need to specify values for column 'auto_increment_id'. SELECT statement does not show 'auto_increment_id' column unless the column is explicitly specified in select list. UPSERT operation is not supported now for Kudu table with auto incrementing column due to limitation in Kudu engine. Testing: - Integrated new version of Kudu built on local machine, ran manual test in impala-shell with queries to create tables with non unique primary key, and tested insert/update/delete operations for Kudu tables with non unique primary key. - TODO build toolchian with new version of Kudu, including the commits for KUDU-1945. Run core test. Change-Id: I4d7882bf3d01a3492cc9827c072d1f3200d9eebd --- M common/thrift/CatalogObjects.thrift M common/thrift/JniCatalog.thrift M fe/src/main/cup/sql-parser.cup M fe/src/main/java/org/apache/impala/analysis/AlterTableAddColsStmt.java M fe/src/main/java/org/apache/impala/analysis/ColumnDef.java M fe/src/main/java/org/apache/impala/analysis/CreateTableAsSelectStmt.java M fe/src/main/java/org/apache/impala/analysis/CreateTableLikeFileStmt.java M fe/src/main/java/org/apache/impala/analysis/CreateTableStmt.java M fe/src/main/java/org/apache/impala/analysis/InsertStmt.java M fe/src/main/java/org/apache/impala/analysis/ModifyStmt.java M fe/src/main/java/org/apache/impala/analysis/SelectStmt.java M fe/src/main/java/org/apache/impala/analysis/TableDef.java M fe/src/main/java/org/apache/impala/analysis/ToSqlUtils.java M fe/src/main/java/org/apache/impala/catalog/Db.java M fe/src/main/java/org/apache/impala/catalog/FeDb.java M fe/src/main/java/org/apache/impala/catalog/FeKuduTable.java M fe/src/main/java/org/apache/impala/catalog/KuduColumn.java M fe/src/main/java/org/apache/impala/catalog/KuduTable.java M fe/src/main/java/org/apache/impala/catalog/local/LocalDb.java M fe/src/main/java/org/apache/impala/catalog/local/LocalKuduTable.java M fe/src/main/java/org/apache/impala/service/Frontend.java M fe/src/main/java/org/apache/impala/service/KuduCatalogOpExecutor.java M fe/src/main/java/org/apache/impala/util/KuduUtil.java M fe/src/main/jflex/sql-scanner.flex M fe/src/test/java/org/apache/impala/analysis/AnalyzeDDLTest.java M fe/src/test/java/org/apache/impala/analysis/AnalyzeKuduDDLTest.java M fe/src/test/java/org/apache/impala/analysis/ParserTest.java M testdata/workloads/functional-query/queries/QueryTest/kudu-scan-node.test M testdata/workloads/functional-query/queries/QueryTest/kudu_alter.test M testdata/workloads/functional-query/queries/QueryTest/kudu_create.test M testdata/workloads/functional-query/queries/QueryTest/kudu_insert.test M testdata/workloads/functional-query/queries/QueryTest/kudu_update.test M testdata/workloads/functional-query/queries/QueryTest/kudu_upsert.test 33 files changed, 733 insertions(+), 97 deletions(-) git pull ssh://gerrit.cloudera.org:29418/Impala-ASF refs/changes/83/19383/4 -- To view, visit http://gerrit.cloudera.org:8080/19383 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-MessageType: newpatchset Gerrit-Change-Id: I4d7882bf3d01a3492cc9827c072d1f3200d9eebd Gerrit-Change-Number: 19383 Gerrit-PatchSet: 4 Gerrit-Owner: Wenzhe Zhou Gerrit-Reviewer: Abhishek Chennaka Gerrit-Reviewer: Impala Public Jenkins Gerrit-Reviewer: Kurt Deschler Gerrit-Reviewer: Marton Greber
[Impala-ASF-CR] WIP IMPALA-11809: Support non unique primary key for Kudu
Impala Public Jenkins has posted comments on this change. ( http://gerrit.cloudera.org:8080/19383 ) Change subject: WIP IMPALA-11809: Support non unique primary key for Kudu .. Patch Set 3: Build Failed https://jenkins.impala.io/job/gerrit-code-review-checks/12055/ : Initial code review checks failed. See linked job for details on the failure. -- To view, visit http://gerrit.cloudera.org:8080/19383 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-MessageType: comment Gerrit-Change-Id: I4d7882bf3d01a3492cc9827c072d1f3200d9eebd Gerrit-Change-Number: 19383 Gerrit-PatchSet: 3 Gerrit-Owner: Wenzhe Zhou Gerrit-Reviewer: Abhishek Chennaka Gerrit-Reviewer: Impala Public Jenkins Gerrit-Reviewer: Kurt Deschler Gerrit-Reviewer: Marton Greber Gerrit-Comment-Date: Wed, 21 Dec 2022 16:50:23 + Gerrit-HasComments: No
[Impala-ASF-CR] WIP IMPALA-11809: Support non unique primary key for Kudu
Wenzhe Zhou has uploaded a new patch set (#3). ( http://gerrit.cloudera.org:8080/19383 ) Change subject: WIP IMPALA-11809: Support non unique primary key for Kudu .. WIP IMPALA-11809: Support non unique primary key for Kudu Kudu engine adds support for non unique primary key. It adds a column 'auto_increment_id' automatically in a table which has non unique primary key. The non unique primary key and 'auto_increment_id' form unique composite primary key. This patch integrated new version of Kudu which support non unique primary key, added syntactic support for creating table with non unique primary key. Example: CREATE TABLE tbl (i INT NON UNIQUE PRIMARY KEY, s STRING) PARTITION BY HASH (i) partitions 3 STORED as KUDU; CREATE TABLE tbl (i INT, s STRING, NON UNIQUE PRIMARY KEY(i)) PARTITION BY HASH (i) partitions 3 STORED as KUDU; CREATE TABLE tbl NON UNIQUE PRIMARY KEY(id) PARTITION BY HASH (id) partitions 3 STORED as KUDU AS SELECT id, string_col FROM functional.alltypes WHERE id = 10; Kudu assign values for column 'auto_increment_id' automatically when inserting rows so insertion statements don't need to specify values for column 'auto_increment_id'. Select statement does not show 'auto_increment_id' column unless the column is explicitly specified in select list. Testing: - Integrated new version of Kudu built on local machine, ran manual test in impala-shell with queries to create tables with non unique primary key, and tested insert/update/delete operations for Kudu tables with non unique primary key. - TODO build toolchian with new version of Kudu, including the commits for KUDU-1945. Run core test. Change-Id: I4d7882bf3d01a3492cc9827c072d1f3200d9eebd --- M common/thrift/CatalogObjects.thrift M common/thrift/JniCatalog.thrift M fe/src/main/cup/sql-parser.cup M fe/src/main/java/org/apache/impala/analysis/AlterTableAddColsStmt.java M fe/src/main/java/org/apache/impala/analysis/ColumnDef.java M fe/src/main/java/org/apache/impala/analysis/CreateTableAsSelectStmt.java M fe/src/main/java/org/apache/impala/analysis/CreateTableLikeFileStmt.java M fe/src/main/java/org/apache/impala/analysis/CreateTableStmt.java M fe/src/main/java/org/apache/impala/analysis/InsertStmt.java M fe/src/main/java/org/apache/impala/analysis/ModifyStmt.java M fe/src/main/java/org/apache/impala/analysis/SelectStmt.java M fe/src/main/java/org/apache/impala/analysis/TableDef.java M fe/src/main/java/org/apache/impala/analysis/ToSqlUtils.java M fe/src/main/java/org/apache/impala/catalog/Db.java M fe/src/main/java/org/apache/impala/catalog/FeDb.java M fe/src/main/java/org/apache/impala/catalog/FeKuduTable.java M fe/src/main/java/org/apache/impala/catalog/KuduColumn.java M fe/src/main/java/org/apache/impala/catalog/KuduTable.java M fe/src/main/java/org/apache/impala/catalog/local/LocalDb.java M fe/src/main/java/org/apache/impala/catalog/local/LocalKuduTable.java M fe/src/main/java/org/apache/impala/service/Frontend.java M fe/src/main/java/org/apache/impala/service/KuduCatalogOpExecutor.java M fe/src/main/java/org/apache/impala/util/KuduUtil.java M fe/src/main/jflex/sql-scanner.flex M fe/src/test/java/org/apache/impala/analysis/AnalyzeDDLTest.java M fe/src/test/java/org/apache/impala/analysis/AnalyzeKuduDDLTest.java M fe/src/test/java/org/apache/impala/analysis/ParserTest.java M testdata/workloads/functional-query/queries/QueryTest/kudu-scan-node.test M testdata/workloads/functional-query/queries/QueryTest/kudu_alter.test M testdata/workloads/functional-query/queries/QueryTest/kudu_create.test M testdata/workloads/functional-query/queries/QueryTest/kudu_insert.test M testdata/workloads/functional-query/queries/QueryTest/kudu_update.test M testdata/workloads/functional-query/queries/QueryTest/kudu_upsert.test 33 files changed, 733 insertions(+), 97 deletions(-) git pull ssh://gerrit.cloudera.org:29418/Impala-ASF refs/changes/83/19383/3 -- To view, visit http://gerrit.cloudera.org:8080/19383 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-MessageType: newpatchset Gerrit-Change-Id: I4d7882bf3d01a3492cc9827c072d1f3200d9eebd Gerrit-Change-Number: 19383 Gerrit-PatchSet: 3 Gerrit-Owner: Wenzhe Zhou Gerrit-Reviewer: Abhishek Chennaka Gerrit-Reviewer: Impala Public Jenkins Gerrit-Reviewer: Kurt Deschler Gerrit-Reviewer: Marton Greber
[Impala-ASF-CR] WIP IMPALA-11809: Support non unique primary key for Kudu
Impala Public Jenkins has posted comments on this change. ( http://gerrit.cloudera.org:8080/19383 ) Change subject: WIP IMPALA-11809: Support non unique primary key for Kudu .. Patch Set 2: Build Failed https://jenkins.impala.io/job/gerrit-code-review-checks/12050/ : Initial code review checks failed. See linked job for details on the failure. -- To view, visit http://gerrit.cloudera.org:8080/19383 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-MessageType: comment Gerrit-Change-Id: I4d7882bf3d01a3492cc9827c072d1f3200d9eebd Gerrit-Change-Number: 19383 Gerrit-PatchSet: 2 Gerrit-Owner: Wenzhe Zhou Gerrit-Reviewer: Abhishek Chennaka Gerrit-Reviewer: Impala Public Jenkins Gerrit-Reviewer: Kurt Deschler Gerrit-Reviewer: Marton Greber Gerrit-Comment-Date: Tue, 20 Dec 2022 22:32:11 + Gerrit-HasComments: No
[Impala-ASF-CR] WIP IMPALA-11809: Support non unique primary key for Kudu
Wenzhe Zhou has uploaded a new patch set (#2). ( http://gerrit.cloudera.org:8080/19383 ) Change subject: WIP IMPALA-11809: Support non unique primary key for Kudu .. WIP IMPALA-11809: Support non unique primary key for Kudu Kudu engine adds support for non unique primary key. It adds a column 'auto_increment_id' automatically in a table which has non unique primary key. The non unique primary key and 'auto_increment_id' form unique composite primary key. This patch integrated new version of Kudu which support non unique primary key, added syntactic support for creating table with non unique primary key. Example: CREATE TABLE tbl (i INT NON UNIQUE PRIMARY KEY, s STRING) PARTITION BY HASH (i) partitions 3 STORED as KUDU; CREATE TABLE tbl (i INT, s STRING, NON UNIQUE PRIMARY KEY(i)) PARTITION BY HASH (i) partitions 3 STORED as KUDU; CREATE TABLE tbl NON UNIQUE PRIMARY KEY(id) PARTITION BY HASH (id) partitions 3 STORED as KUDU AS SELECT id, string_col FROM functional.alltypes WHERE id = 10; Kudu assign values for column 'auto_increment_id' automatically when inserting rows so insertion statements don't need to specify values for column 'auto_increment_id'. Select statement does not show 'auto_increment_id' column unless the column is explicitly specified in select list. Testing: - Integrated new version of Kudu built on local machine, ran manual test in impala-shell with queries to create tables with non unique primary key, and tested insert/update/delete operations for Kudu tables with non unique primary key. - TODO build toolchian with new version of Kudu, including the commits for KUDU-1945. Run core test. Change-Id: I4d7882bf3d01a3492cc9827c072d1f3200d9eebd --- M be/src/exec/kudu/kudu-util.cc M common/thrift/CatalogObjects.thrift M common/thrift/JniCatalog.thrift M fe/src/main/cup/sql-parser.cup M fe/src/main/java/org/apache/impala/analysis/AlterTableAddColsStmt.java M fe/src/main/java/org/apache/impala/analysis/ColumnDef.java M fe/src/main/java/org/apache/impala/analysis/CreateTableAsSelectStmt.java M fe/src/main/java/org/apache/impala/analysis/CreateTableLikeFileStmt.java M fe/src/main/java/org/apache/impala/analysis/CreateTableStmt.java M fe/src/main/java/org/apache/impala/analysis/InsertStmt.java M fe/src/main/java/org/apache/impala/analysis/ModifyStmt.java M fe/src/main/java/org/apache/impala/analysis/SelectStmt.java M fe/src/main/java/org/apache/impala/analysis/TableDef.java M fe/src/main/java/org/apache/impala/analysis/ToSqlUtils.java M fe/src/main/java/org/apache/impala/catalog/Db.java M fe/src/main/java/org/apache/impala/catalog/FeDb.java M fe/src/main/java/org/apache/impala/catalog/FeKuduTable.java M fe/src/main/java/org/apache/impala/catalog/KuduColumn.java M fe/src/main/java/org/apache/impala/catalog/KuduTable.java M fe/src/main/java/org/apache/impala/catalog/local/LocalDb.java M fe/src/main/java/org/apache/impala/catalog/local/LocalKuduTable.java M fe/src/main/java/org/apache/impala/service/Frontend.java M fe/src/main/java/org/apache/impala/service/KuduCatalogOpExecutor.java M fe/src/main/java/org/apache/impala/util/KuduUtil.java M fe/src/main/jflex/sql-scanner.flex M fe/src/test/java/org/apache/impala/analysis/AnalyzeDDLTest.java M fe/src/test/java/org/apache/impala/analysis/AnalyzeKuduDDLTest.java M fe/src/test/java/org/apache/impala/analysis/ParserTest.java M testdata/workloads/functional-query/queries/QueryTest/kudu-scan-node.test M testdata/workloads/functional-query/queries/QueryTest/kudu_alter.test M testdata/workloads/functional-query/queries/QueryTest/kudu_create.test M testdata/workloads/functional-query/queries/QueryTest/kudu_insert.test M testdata/workloads/functional-query/queries/QueryTest/kudu_update.test M testdata/workloads/functional-query/queries/QueryTest/kudu_upsert.test 34 files changed, 735 insertions(+), 97 deletions(-) git pull ssh://gerrit.cloudera.org:29418/Impala-ASF refs/changes/83/19383/2 -- To view, visit http://gerrit.cloudera.org:8080/19383 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-MessageType: newpatchset Gerrit-Change-Id: I4d7882bf3d01a3492cc9827c072d1f3200d9eebd Gerrit-Change-Number: 19383 Gerrit-PatchSet: 2 Gerrit-Owner: Wenzhe Zhou Gerrit-Reviewer: Abhishek Chennaka Gerrit-Reviewer: Impala Public Jenkins Gerrit-Reviewer: Kurt Deschler Gerrit-Reviewer: Marton Greber
[Impala-ASF-CR] WIP IMPALA-11809: Support non unique primary key for Kudu
Impala Public Jenkins has posted comments on this change. ( http://gerrit.cloudera.org:8080/19383 ) Change subject: WIP IMPALA-11809: Support non unique primary key for Kudu .. Patch Set 1: Build Failed https://jenkins.impala.io/job/gerrit-code-review-checks/12049/ : Initial code review checks failed. See linked job for details on the failure. -- To view, visit http://gerrit.cloudera.org:8080/19383 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-MessageType: comment Gerrit-Change-Id: I4d7882bf3d01a3492cc9827c072d1f3200d9eebd Gerrit-Change-Number: 19383 Gerrit-PatchSet: 1 Gerrit-Owner: Wenzhe Zhou Gerrit-Reviewer: Abhishek Chennaka Gerrit-Reviewer: Impala Public Jenkins Gerrit-Reviewer: Kurt Deschler Gerrit-Reviewer: Marton Greber Gerrit-Comment-Date: Tue, 20 Dec 2022 21:52:53 + Gerrit-HasComments: No
[Impala-ASF-CR] WIP IMPALA-11809: Support non unique primary key for Kudu
Wenzhe Zhou has uploaded this change for review. ( http://gerrit.cloudera.org:8080/19383 Change subject: WIP IMPALA-11809: Support non unique primary key for Kudu .. WIP IMPALA-11809: Support non unique primary key for Kudu Kudu engine adds support for non unique primary key. It adds a column 'auto_increment_id' automatically in a table which has non unique primary key. The non unique primary key and 'auto_increment_id' form unique composite primary key. This patch integrated new version of Kudu which support non unique primary key, added syntactic support for creating table with non unique primary key. Example: CREATE TABLE tbl (i INT NON UNIQUE PRIMARY KEY, s STRING) PARTITION BY HASH (i) partitions 3 STORED as KUDU; CREATE TABLE tbl (i INT, s STRING, NON UNIQUE PRIMARY KEY(i)) PARTITION BY HASH (i) partitions 3 STORED as KUDU; CREATE TABLE tbl NON UNIQUE PRIMARY KEY(id) PARTITION BY HASH (id) partitions 3 STORED as KUDU AS SELECT id, string_col FROM functional.alltypes WHERE id = 10; Kudu assign values for column 'auto_increment_id' automatically when inserting rows so insertion statements don't need to specify values for column 'auto_increment_id'. Select statement does not show 'auto_increment_id' column unless the column is explicitly specified in select list. Testing: - Integrat new version of Kudu with the commits for KUDU-1945 Change-Id: I4d7882bf3d01a3492cc9827c072d1f3200d9eebd --- M be/src/exec/kudu/kudu-util.cc M common/thrift/CatalogObjects.thrift M common/thrift/JniCatalog.thrift M fe/src/main/cup/sql-parser.cup M fe/src/main/java/org/apache/impala/analysis/AlterTableAddColsStmt.java M fe/src/main/java/org/apache/impala/analysis/ColumnDef.java M fe/src/main/java/org/apache/impala/analysis/CreateTableAsSelectStmt.java M fe/src/main/java/org/apache/impala/analysis/CreateTableLikeFileStmt.java M fe/src/main/java/org/apache/impala/analysis/CreateTableStmt.java M fe/src/main/java/org/apache/impala/analysis/InsertStmt.java M fe/src/main/java/org/apache/impala/analysis/ModifyStmt.java M fe/src/main/java/org/apache/impala/analysis/SelectStmt.java M fe/src/main/java/org/apache/impala/analysis/TableDef.java M fe/src/main/java/org/apache/impala/analysis/ToSqlUtils.java M fe/src/main/java/org/apache/impala/catalog/Db.java M fe/src/main/java/org/apache/impala/catalog/FeDb.java M fe/src/main/java/org/apache/impala/catalog/FeKuduTable.java M fe/src/main/java/org/apache/impala/catalog/KuduColumn.java M fe/src/main/java/org/apache/impala/catalog/KuduTable.java M fe/src/main/java/org/apache/impala/catalog/local/LocalDb.java M fe/src/main/java/org/apache/impala/catalog/local/LocalKuduTable.java M fe/src/main/java/org/apache/impala/service/Frontend.java M fe/src/main/java/org/apache/impala/service/KuduCatalogOpExecutor.java M fe/src/main/java/org/apache/impala/util/KuduUtil.java M fe/src/main/jflex/sql-scanner.flex M fe/src/test/java/org/apache/impala/analysis/AnalyzeDDLTest.java M fe/src/test/java/org/apache/impala/analysis/AnalyzeKuduDDLTest.java M fe/src/test/java/org/apache/impala/analysis/ParserTest.java M testdata/workloads/functional-query/queries/QueryTest/kudu-scan-node.test M testdata/workloads/functional-query/queries/QueryTest/kudu_alter.test M testdata/workloads/functional-query/queries/QueryTest/kudu_create.test M testdata/workloads/functional-query/queries/QueryTest/kudu_insert.test M testdata/workloads/functional-query/queries/QueryTest/kudu_update.test M testdata/workloads/functional-query/queries/QueryTest/kudu_upsert.test 34 files changed, 742 insertions(+), 97 deletions(-) git pull ssh://gerrit.cloudera.org:29418/Impala-ASF refs/changes/83/19383/1 -- To view, visit http://gerrit.cloudera.org:8080/19383 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-MessageType: newchange Gerrit-Change-Id: I4d7882bf3d01a3492cc9827c072d1f3200d9eebd Gerrit-Change-Number: 19383 Gerrit-PatchSet: 1 Gerrit-Owner: Wenzhe Zhou Gerrit-Reviewer: Abhishek Chennaka Gerrit-Reviewer: Marton Greber
[Impala-ASF-CR] WIP IMPALA-11809: Support non unique primary key for Kudu
Impala Public Jenkins has posted comments on this change. (
http://gerrit.cloudera.org:8080/19383 )
Change subject: WIP IMPALA-11809: Support non unique primary key for Kudu
..
Patch Set 1:
(1 comment)
http://gerrit.cloudera.org:8080/#/c/19383/1/fe/src/main/java/org/apache/impala/analysis/TableDef.java
File fe/src/main/java/org/apache/impala/analysis/TableDef.java:
http://gerrit.cloudera.org:8080/#/c/19383/1/fe/src/main/java/org/apache/impala/analysis/TableDef.java@513
PS1, Line 513: throw new AnalysisException("Non unique primary key is
only supported for Kudu.");
line too long (92 > 90)
--
To view, visit http://gerrit.cloudera.org:8080/19383
To unsubscribe, visit http://gerrit.cloudera.org:8080/settings
Gerrit-Project: Impala-ASF
Gerrit-Branch: master
Gerrit-MessageType: comment
Gerrit-Change-Id: I4d7882bf3d01a3492cc9827c072d1f3200d9eebd
Gerrit-Change-Number: 19383
Gerrit-PatchSet: 1
Gerrit-Owner: Wenzhe Zhou
Gerrit-Reviewer: Abhishek Chennaka
Gerrit-Reviewer: Impala Public Jenkins
Gerrit-Reviewer: Marton Greber
Gerrit-Comment-Date: Tue, 20 Dec 2022 21:44:15 +
Gerrit-HasComments: Yes
