Wenzhe Zhou has uploaded a new patch set (#12). (
http://gerrit.cloudera.org:8080/17553 )
Change subject: IMPALA-10557: Support Kudu's multi-row transaction
..
IMPALA-10557: Support Kudu's multi-row transaction
Kudu added multi-row transaction so Impala could run query that inserts
multiple rows into Kudu's table in the context of a single transaction.
Kudu provides new Java/C++ client APIs to open/commit/rollback
transaction, create session with transaction, serialize/deserialize
metadata of transaction object. Kudu transaction object has built-in
heartbeater.
This patch added Impala support to use Kudu's multiple-row transaction.
- Added a new query option to enable Kudu's transaction.
- When the query option is set, a new Kudu transaction should be
started for "insert" and "CTAS" statements by Impala's frontend of
coordinator.
- The Kudu transaction objects are kept in KuduTransactionManager until
the transactions are going to be aborted or committed.
- Frontend serialize the transaction metadata into a transaction token
and pass to executors.
- Executors deserialize the transaction token and ingest via that
transaction handle. For Kudu session in the context of a transaction,
return the first error if there are any pending errors for the Kudu
session so that the Kudu transaction will be aborted.
- Coordinator commits the transaction after updating catalog if
everything goes well. Otherwise, aborts the transaction.
Also changed code to store KuduClient as shared pointer since KuduClient
has to be passed as a shared pointer when KuduTransaction::Deserialize()
is called.
Testing:
- Added new e-to-e tests for Kudu transaction.
- Passed core test.
Change-Id: I876ada48991afdff5d61b5d6a0417571aba7cb34
---
M be/src/exec/kudu-scan-node-base.h
M be/src/exec/kudu-table-sink.cc
M be/src/exec/kudu-table-sink.h
M be/src/exprs/kudu-partition-expr.cc
M be/src/exprs/kudu-partition-expr.h
M be/src/runtime/exec-env.cc
M be/src/runtime/exec-env.h
M be/src/service/client-request-state.cc
M be/src/service/client-request-state.h
M be/src/service/frontend.cc
M be/src/service/frontend.h
M be/src/service/query-options.cc
M be/src/service/query-options.h
M common/thrift/DataSinks.thrift
M common/thrift/ImpalaService.thrift
M common/thrift/Query.thrift
M fe/src/main/java/org/apache/impala/analysis/InsertStmt.java
A fe/src/main/java/org/apache/impala/common/KuduTransactionManager.java
M fe/src/main/java/org/apache/impala/common/TransactionException.java
M fe/src/main/java/org/apache/impala/planner/KuduTableSink.java
M fe/src/main/java/org/apache/impala/planner/TableSink.java
M fe/src/main/java/org/apache/impala/service/Frontend.java
M fe/src/main/java/org/apache/impala/service/JniFrontend.java
M testdata/cluster/node_templates/common/etc/kudu/master.conf.tmpl
M testdata/cluster/node_templates/common/etc/kudu/tserver.conf.tmpl
M tests/custom_cluster/test_kudu.py
26 files changed, 616 insertions(+), 63 deletions(-)
git pull ssh://gerrit.cloudera.org:29418/Impala-ASF refs/changes/53/17553/12
--
To view, visit http://gerrit.cloudera.org:8080/17553
To unsubscribe, visit http://gerrit.cloudera.org:8080/settings
Gerrit-Project: Impala-ASF
Gerrit-Branch: master
Gerrit-MessageType: newpatchset
Gerrit-Change-Id: I876ada48991afdff5d61b5d6a0417571aba7cb34
Gerrit-Change-Number: 17553
Gerrit-PatchSet: 12
Gerrit-Owner: Wenzhe Zhou
Gerrit-Reviewer: Alexey Serbin
Gerrit-Reviewer: Andrew Wong
Gerrit-Reviewer: Bikramjeet Vig
Gerrit-Reviewer: Impala Public Jenkins
Gerrit-Reviewer: Qifan Chen
Gerrit-Reviewer: Wenzhe Zhou
