Hello Dan Burkert, Todd Lipcon,
I'd like you to do a code review. Please visit
http://gerrit.cloudera.org:8080/7853
to review the following change.
Change subject: WIP: use C++ ExternalMiniCluster for Java and Python tests
......................................................................
WIP: use C++ ExternalMiniCluster for Java and Python tests
Maintaining Kudu clients across various languages has been an ongoing
maintenance burden. Even when the client is just a thin wrapper around
another client (e.g. Kudu Python bindings), a great deal of work goes into
client testability. In practice, this has meant a bespoke mini cluster
implementation for each language. On the surface this doesn't seem that bad;
we just need to spawn some masters and tservers, right? Well, the work
quickly adds up:
o While the C++ mini cluster is heavily used and has seen many improvements,
the Java mini cluster has not received the same kind of love, and is less
robust as a result. KUDU-1976 is a great example of this deficiency.
o With the inclusion of authz came the addition of a "mini KDC", a special
daemon for Kerberized mini clusters. It was originally implemented in C++
and ported to Java, but has yet to be ported to the Python client; this is
one of the obstacles towards porting full authz support to Python.
o Dan has been prototyping Hive Metastore and Sentry integration for Kudu,
the testing of which will require "mini HMS" and possibly "mini Sentry"
testing implementations in C++, Java, and eventually, Python.
In sum, good support for non-C++ mini clusters is an ongoing commitment and
requires a great deal of work. This work hasn't always been forthcoming, and
the non-C++ clusters are deficient as a result. But it doesn't have to be
this way! Here's a thought: what if we reused the C++ mini cluster for tests
written in these other languages? We could write a "proxy" application whose
job it is to manage the C++ mini cluster and expose a rudimentary API that's
easily programmable from Java and Python.
This patch attempts to do just that. It adds a "run_cluster" mode to the
Kudu CLI. When invoked, it spawns an ExternalMiniCluster and provides a
simple, machine-readable shell over stdin/stdout. The shell responds to
commands by manipulating the cluster and its daemons, and kills them when
the shell client disconnects. As a proof of concept, the patch also replaces
the bespoke Java mini cluster with callouts to the new shell.
I should add that I like the idea of shipping "run_cluster" into production
as part of the CLI, as it helps realize the vision of a single Kudu artifact
that can provide Kudu testability for any integrating product.
WIP because, well, it should be pretty obvious. I was able to get through a
full run of "mvn verify" locally, so I have confidence that this can work.
But I'd like to solicit feedback on the general approach before spending
more time applying spit and polish.
Change-Id: I0e693921ef780dc4a06e536c6b7408f7f0b252f6
---
M java/kudu-client/src/test/java/org/apache/kudu/client/BaseKuduTest.java
D java/kudu-client/src/test/java/org/apache/kudu/client/MiniKdc.java
M java/kudu-client/src/test/java/org/apache/kudu/client/MiniKuduCluster.java
M
java/kudu-client/src/test/java/org/apache/kudu/client/TestClientFailoverSupport.java
D java/kudu-client/src/test/java/org/apache/kudu/client/TestMiniKdc.java
M java/kudu-client/src/test/java/org/apache/kudu/client/TestMiniKuduCluster.java
M
java/kudu-client/src/test/java/org/apache/kudu/client/TestMultipleLeaderFailover.java
M java/kudu-client/src/test/java/org/apache/kudu/client/TestUtils.java
D java/kudu-client/src/test/resources/flags
M src/kudu/security/test/mini_kdc.cc
M src/kudu/tools/CMakeLists.txt
M src/kudu/tools/tool_action.h
A src/kudu/tools/tool_action_test.cc
M src/kudu/tools/tool_main.cc
14 files changed, 532 insertions(+), 1,210 deletions(-)
git pull ssh://gerrit.cloudera.org:29418/kudu refs/changes/53/7853/1
--
To view, visit http://gerrit.cloudera.org:8080/7853
To unsubscribe, visit http://gerrit.cloudera.org:8080/settings
Gerrit-MessageType: newchange
Gerrit-Change-Id: I0e693921ef780dc4a06e536c6b7408f7f0b252f6
Gerrit-PatchSet: 1
Gerrit-Project: kudu
Gerrit-Branch: master
Gerrit-Owner: Adar Dembo <a...@cloudera.com>
Gerrit-Reviewer: Dan Burkert <danburk...@apache.org>
Gerrit-Reviewer: Todd Lipcon <t...@apache.org>
