Wenzhe Zhou has uploaded a new patch set (#9). ( http://gerrit.cloudera.org:8080/17553 )
Change subject: IMPALA-10557: Support Kudu's multi-row transaction ...................................................................... IMPALA-10557: Support Kudu's multi-row transaction Kudu added multi-row transaction so Impala could run query that inserts multiple rows into Kudu's table in the context of a single transaction. Kudu provides new Java/C++ client APIs to open/commit/rollback transaction, create session with transaction, serialize/deserialize metadata of transaction object. Kudu transaction object has built-in heartbeater. This patch added Impala support to use Kudu's multiple-row transaction. - Added a new query option to enable Kudu's transaction. - When the query option is set, a new Kudu transaction should be started for "insert" statements by Impala's frontend of coordinator. - The Kudu transaction objects are kept in KuduTransactionManager until the transactions are going to be aborted or committed. - Frontend serialize the transaction metadata into a transaction token and pass to executors. - Executors deserialize the transaction token and ingest via that transaction handle. For Kudu session in the context of a transaction, return the first error if there are any pending errors for the Kudu session so that the Kudu transaction will be aborted. - Coordinator commits the transaction after updating catalog if everything goes well. Otherwise, aborts the transaction. Also changed code to store KuduClient as shared pointer since KuduClient has to be passed as a shared pointer when KuduTransaction::Deserialize() is called. Testing: - Added new e-to-e tests for Kudu transaction. - Passed core test. Change-Id: I876ada48991afdff5d61b5d6a0417571aba7cb34 --- M be/src/exec/kudu-scan-node-base.h M be/src/exec/kudu-table-sink.cc M be/src/exec/kudu-table-sink.h M be/src/exprs/kudu-partition-expr.cc M be/src/exprs/kudu-partition-expr.h M be/src/runtime/exec-env.cc M be/src/runtime/exec-env.h M be/src/service/client-request-state.cc M be/src/service/client-request-state.h M be/src/service/frontend.cc M be/src/service/frontend.h M be/src/service/query-options.cc M be/src/service/query-options.h M common/thrift/DataSinks.thrift M common/thrift/ImpalaService.thrift M common/thrift/Query.thrift M fe/src/main/java/org/apache/impala/analysis/InsertStmt.java A fe/src/main/java/org/apache/impala/common/KuduTransactionManager.java M fe/src/main/java/org/apache/impala/common/TransactionException.java M fe/src/main/java/org/apache/impala/planner/KuduTableSink.java M fe/src/main/java/org/apache/impala/planner/TableSink.java M fe/src/main/java/org/apache/impala/service/Frontend.java M fe/src/main/java/org/apache/impala/service/JniFrontend.java M testdata/cluster/node_templates/common/etc/kudu/master.conf.tmpl M testdata/cluster/node_templates/common/etc/kudu/tserver.conf.tmpl M tests/custom_cluster/test_kudu.py 26 files changed, 537 insertions(+), 63 deletions(-) git pull ssh://gerrit.cloudera.org:29418/Impala-ASF refs/changes/53/17553/9 -- To view, visit http://gerrit.cloudera.org:8080/17553 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-MessageType: newpatchset Gerrit-Change-Id: I876ada48991afdff5d61b5d6a0417571aba7cb34 Gerrit-Change-Number: 17553 Gerrit-PatchSet: 9 Gerrit-Owner: Wenzhe Zhou <wz...@cloudera.com> Gerrit-Reviewer: Alexey Serbin <aser...@cloudera.com> Gerrit-Reviewer: Andrew Wong <aw...@cloudera.com> Gerrit-Reviewer: Bikramjeet Vig <bikramjeet....@cloudera.com> Gerrit-Reviewer: Impala Public Jenkins <impala-public-jenk...@cloudera.com> Gerrit-Reviewer: Qifan Chen <qc...@cloudera.com> Gerrit-Reviewer: Wenzhe Zhou <wz...@cloudera.com>
