Wenzhe Zhou has uploaded a new patch set (#2). ( http://gerrit.cloudera.org:8080/17553 )
Change subject: WIP IMPALA-10557: Support Kudu's multi-row transaction ...................................................................... WIP IMPALA-10557: Support Kudu's multi-row transaction Kudu added multi-row transaction so Impala could run query that inserts multiple rows into Kudu's table in the context of a single transaction. Kudu provides new Java/C++ client APIs to open/commit/rollback transaction, create session with transaction, serialize/deserialize metadata of transaction object. Kudu transaction object has built-in heartbeater. This patch added Impala support to use Kudu's multiple-row transaction. - Added a new query option to enable Kudu's transaction. - When the query option is set, a new Kudu transaction should be started for "insert" statements by Impala's frontend of coordinator. - The Kudu transaction objects are kept in KuduTransactionManager until the transactions are going to be aborted or committed. - Frontend serialize the transaction metadata into a binary and pass to executors. - Executors deserialize the binary and ingest via that transaction handle. - Coordinator commits the transaction after updating catalog if everything goes well. Otherwise, aborts the transaction. Also changed code to store KuduClient as shared pointer by apllying a Thomas's old patch since we have to pass KuduClient as shared pointer for Kudu API KuduTransaction::Deserialize(). Testing: - TODO add e-to-e tests. Change-Id: I876ada48991afdff5d61b5d6a0417571aba7cb34 --- M be/src/exec/kudu-scan-node-base.h M be/src/exec/kudu-table-sink.cc M be/src/exec/kudu-table-sink.h M be/src/exprs/kudu-partition-expr.cc M be/src/exprs/kudu-partition-expr.h M be/src/runtime/exec-env.cc M be/src/runtime/exec-env.h M be/src/service/client-request-state.cc M be/src/service/client-request-state.h M be/src/service/frontend.cc M be/src/service/frontend.h M be/src/service/query-options.cc M be/src/service/query-options.h M common/thrift/DataSinks.thrift M common/thrift/ImpalaService.thrift M common/thrift/Query.thrift M fe/src/main/java/org/apache/impala/analysis/InsertStmt.java A fe/src/main/java/org/apache/impala/common/KuduTransactionManager.java M fe/src/main/java/org/apache/impala/planner/KuduTableSink.java M fe/src/main/java/org/apache/impala/planner/TableSink.java M fe/src/main/java/org/apache/impala/service/Frontend.java M fe/src/main/java/org/apache/impala/service/JniFrontend.java M testdata/cluster/node_templates/common/etc/kudu/master.conf.tmpl 23 files changed, 364 insertions(+), 47 deletions(-) git pull ssh://gerrit.cloudera.org:29418/Impala-ASF refs/changes/53/17553/2 -- To view, visit http://gerrit.cloudera.org:8080/17553 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-MessageType: newpatchset Gerrit-Change-Id: I876ada48991afdff5d61b5d6a0417571aba7cb34 Gerrit-Change-Number: 17553 Gerrit-PatchSet: 2 Gerrit-Owner: Wenzhe Zhou <wz...@cloudera.com> Gerrit-Reviewer: Andrew Wong <aw...@cloudera.com> Gerrit-Reviewer: Impala Public Jenkins <impala-public-jenk...@cloudera.com> Gerrit-Reviewer: Qifan Chen <qc...@cloudera.com> Gerrit-Reviewer: Wenzhe Zhou <wz...@cloudera.com>
