Wenzhe Zhou has uploaded a new patch set (#5). ( http://gerrit.cloudera.org:8080/17553 )
Change subject: WIP IMPALA-10557: Support Kudu's multi-row transaction ...................................................................... WIP IMPALA-10557: Support Kudu's multi-row transaction Kudu added multi-row transaction so Impala could run query that inserts multiple rows into Kudu's table in the context of a single transaction. Kudu provides new Java/C++ client APIs to open/commit/rollback transaction, create session with transaction, serialize/deserialize metadata of transaction object. Kudu transaction object has built-in heartbeater. This patch added Impala support to use Kudu's multiple-row transaction. - Added a new query option to enable Kudu's transaction. - When the query option is set, a new Kudu transaction should be started for "insert" statements by Impala's frontend of coordinator. - The Kudu transaction objects are kept in KuduTransactionManager until the transactions are going to be aborted or committed. - Frontend serialize the transaction metadata into a binary and pass to executors. - Executors deserialize the binary and ingest via that transaction handle. - Coordinator commits the transaction after updating catalog if everything goes well. Otherwise, aborts the transaction. Also changed code to store KuduClient as shared pointer since KuduClient has to be passed as a shared pointer when KuduTransaction::Deserialize() is called. Testing: - TODO add e-to-e tests. Change-Id: I876ada48991afdff5d61b5d6a0417571aba7cb34 --- M be/src/exec/kudu-scan-node-base.h M be/src/exec/kudu-table-sink.cc M be/src/exec/kudu-table-sink.h M be/src/exprs/kudu-partition-expr.cc M be/src/exprs/kudu-partition-expr.h M be/src/runtime/exec-env.cc M be/src/runtime/exec-env.h M be/src/service/client-request-state.cc M be/src/service/client-request-state.h M be/src/service/frontend.cc M be/src/service/frontend.h M be/src/service/query-options.cc M be/src/service/query-options.h M common/thrift/DataSinks.thrift M common/thrift/ImpalaService.thrift M common/thrift/Query.thrift M fe/src/main/java/org/apache/impala/analysis/InsertStmt.java A fe/src/main/java/org/apache/impala/common/KuduTransactionManager.java M fe/src/main/java/org/apache/impala/planner/KuduTableSink.java M fe/src/main/java/org/apache/impala/planner/TableSink.java M fe/src/main/java/org/apache/impala/service/Frontend.java M fe/src/main/java/org/apache/impala/service/JniFrontend.java M testdata/cluster/node_templates/common/etc/kudu/master.conf.tmpl M testdata/cluster/node_templates/common/etc/kudu/tserver.conf.tmpl 24 files changed, 371 insertions(+), 52 deletions(-) git pull ssh://gerrit.cloudera.org:29418/Impala-ASF refs/changes/53/17553/5 -- To view, visit http://gerrit.cloudera.org:8080/17553 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-MessageType: newpatchset Gerrit-Change-Id: I876ada48991afdff5d61b5d6a0417571aba7cb34 Gerrit-Change-Number: 17553 Gerrit-PatchSet: 5 Gerrit-Owner: Wenzhe Zhou <wz...@cloudera.com> Gerrit-Reviewer: Alexey Serbin <aser...@cloudera.com> Gerrit-Reviewer: Andrew Wong <aw...@cloudera.com> Gerrit-Reviewer: Bikramjeet Vig <bikramjeet....@cloudera.com> Gerrit-Reviewer: Impala Public Jenkins <impala-public-jenk...@cloudera.com> Gerrit-Reviewer: Qifan Chen <qc...@cloudera.com> Gerrit-Reviewer: Wenzhe Zhou <wz...@cloudera.com>