Hello Tidy Bot, Alexey Serbin, Dan Burkert, Kudu Jenkins, Todd Lipcon, I'd like you to reexamine a change. Please visit
http://gerrit.cloudera.org:8080/7853 to look at the new patch set (#3). Change subject: WIP: use C++ ExternalMiniCluster for Java and Python tests ...................................................................... WIP: use C++ ExternalMiniCluster for Java and Python tests Maintaining Kudu clients across various languages has been an ongoing maintenance burden. Even when the client is just a thin wrapper around another client (e.g. Kudu Python bindings), a great deal of work goes into client testability. In practice, this has meant a bespoke mini cluster implementation for each language. On the surface this doesn't seem that bad; we just need to spawn some masters and tservers, right? Well, the work quickly adds up: o While the C++ mini cluster is heavily used and has seen many improvements, the Java mini cluster has not received the same kind of love, and is less robust as a result. KUDU-1976 is a great example of this deficiency. o With the inclusion of authz came the addition of a "mini KDC", a special daemon for Kerberized mini clusters. It was originally implemented in C++ and ported to Java, but has yet to be ported to the Python client; this is one of the obstacles towards porting full authz support to Python. o Dan has been prototyping Hive Metastore and Sentry integration for Kudu, the testing of which will require "mini HMS" and possibly "mini Sentry" testing implementations in C++, Java, and eventually, Python. In sum, good support for non-C++ mini clusters is an ongoing commitment and requires a great deal of work. This work hasn't always been forthcoming, and the non-C++ clusters are deficient as a result. But it doesn't have to be this way! Here's a thought: what if we reused the C++ mini cluster for tests written in these other languages? We could write a "proxy" application whose job it is to manage the C++ mini cluster and expose a rudimentary API that's easily programmable from Java and Python. This patch attempts to do just that. It adds a "shell" mode to the Kudu CLI which provides a rudimentary control shell that can be used to spin up an ExternalMiniCluster. The shell is controlled via a wire protocol over stdin/stdout. The first cut of the protocol was sh-like, with simple word-based commands. It was then revised into a PB-based protocol (with optional JSON encoding) based on feedback. As a proof of concept, the patch also replaces the bespoke Java mini cluster with callouts to the new shell. I should add that I like the idea of shipping "shell" into production as part of the CLI, as it helps realize the vision of a single Kudu artifact that can provide Kudu testability for any integrating product. WIP because, well, it should be pretty obvious. I was able to get through a full run of "mvn verify" locally, so I have confidence that this can work. But I'd like to solicit feedback on the general approach before spending more time applying spit and polish. Change-Id: I0e693921ef780dc4a06e536c6b7408f7f0b252f6 --- M java/kudu-client/src/test/java/org/apache/kudu/client/BaseKuduTest.java D java/kudu-client/src/test/java/org/apache/kudu/client/MiniKdc.java M java/kudu-client/src/test/java/org/apache/kudu/client/MiniKuduCluster.java M java/kudu-client/src/test/java/org/apache/kudu/client/TestClientFailoverSupport.java D java/kudu-client/src/test/java/org/apache/kudu/client/TestMiniKdc.java M java/kudu-client/src/test/java/org/apache/kudu/client/TestMiniKuduCluster.java M java/kudu-client/src/test/java/org/apache/kudu/client/TestMultipleLeaderFailover.java M java/kudu-client/src/test/java/org/apache/kudu/client/TestUtils.java D java/kudu-client/src/test/resources/flags M java/kudu-spark/src/test/scala/org/apache/kudu/spark/kudu/TestContext.scala M src/kudu/integration-tests/external_mini_cluster.h M src/kudu/security/test/mini_kdc.cc M src/kudu/tools/CMakeLists.txt A src/kudu/tools/tool.proto M src/kudu/tools/tool_action.h A src/kudu/tools/tool_action_test.cc M src/kudu/tools/tool_main.cc 17 files changed, 919 insertions(+), 1,256 deletions(-) git pull ssh://gerrit.cloudera.org:29418/kudu refs/changes/53/7853/3 -- To view, visit http://gerrit.cloudera.org:8080/7853 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: kudu Gerrit-Branch: master Gerrit-MessageType: newpatchset Gerrit-Change-Id: I0e693921ef780dc4a06e536c6b7408f7f0b252f6 Gerrit-Change-Number: 7853 Gerrit-PatchSet: 3 Gerrit-Owner: Adar Dembo <a...@cloudera.com> Gerrit-Reviewer: Adar Dembo <a...@cloudera.com> Gerrit-Reviewer: Alexey Serbin <aser...@cloudera.com> Gerrit-Reviewer: Dan Burkert <danburk...@apache.org> Gerrit-Reviewer: Kudu Jenkins Gerrit-Reviewer: Tidy Bot Gerrit-Reviewer: Todd Lipcon <t...@apache.org>