Hello Tidy Bot, Alexey Serbin, Dan Burkert, Kudu Jenkins, Todd Lipcon, I'd like you to reexamine a change. Please visit
http://gerrit.cloudera.org:8080/7853 to look at the new patch set (#5). Change subject: tool: add cluster shell action ...................................................................... tool: add cluster shell action Maintaining Kudu clients across various languages has been an ongoing maintenance burden. Even when the client is just a thin wrapper around another client (e.g. Kudu Python bindings), a great deal of work goes into client testability. In practice, this has meant a bespoke mini cluster implementation for each language. On the surface this doesn't seem that bad; we just need to spawn some masters and tservers, right? Well, the work quickly adds up: o While the C++ mini cluster is heavily used and has seen many improvements, the Java mini cluster has not received the same kind of love, and is less robust as a result. KUDU-1976 is a great example of this deficiency. o With the inclusion of authn came the addition of a "mini KDC", a special daemon for Kerberized mini clusters. It was originally implemented in C++ and ported to Java, but has yet to be ported to the Python client; this is one of the obstacles towards porting full authn support to Python. o Dan has been prototyping Hive Metastore and Sentry integration for Kudu, the testing of which will require "mini HMS" and possibly "mini Sentry" testing implementations in C++, Java, and eventually, Python. In sum, good support for non-C++ mini clusters is an ongoing commitment and requires a great deal of work. This work hasn't always been forthcoming, and the non-C++ clusters are deficient as a result. But it doesn't have to be this way! Here's a thought: what if we reused the C++ mini cluster for tests written in these other languages? We could write a "proxy" application whose job it is to manage the C++ mini cluster and expose a rudimentary API that's easily programmable from Java and Python. This patch attempts to do just that. It adds a "shell" action to the Kudu CLI which provides a rudimentary control shell that can be used to spin up an ExternalMiniCluster. The shell is controlled via a wire protocol over stdin/stdout. The protocol is protobuf-based with optional JSON encoding. I should add that I like the idea of shipping "shell" into production as part of the CLI, as it helps realize the vision of a single Kudu artifact that can provide Kudu testability for any integrating product. Change-Id: I0e693921ef780dc4a06e536c6b7408f7f0b252f6 --- M src/kudu/mini-cluster/external_mini_cluster.h M src/kudu/tools/CMakeLists.txt M src/kudu/tools/kudu-tool-test.cc A src/kudu/tools/tool.proto M src/kudu/tools/tool_action.h M src/kudu/tools/tool_action_common.cc M src/kudu/tools/tool_action_common.h A src/kudu/tools/tool_action_test.cc M src/kudu/tools/tool_main.cc 9 files changed, 1,104 insertions(+), 9 deletions(-) git pull ssh://gerrit.cloudera.org:29418/kudu refs/changes/53/7853/5 -- To view, visit http://gerrit.cloudera.org:8080/7853 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: kudu Gerrit-Branch: master Gerrit-MessageType: newpatchset Gerrit-Change-Id: I0e693921ef780dc4a06e536c6b7408f7f0b252f6 Gerrit-Change-Number: 7853 Gerrit-PatchSet: 5 Gerrit-Owner: Adar Dembo <a...@cloudera.com> Gerrit-Reviewer: Adar Dembo <a...@cloudera.com> Gerrit-Reviewer: Alexey Serbin <aser...@cloudera.com> Gerrit-Reviewer: Dan Burkert <danburk...@apache.org> Gerrit-Reviewer: Kudu Jenkins Gerrit-Reviewer: Tidy Bot Gerrit-Reviewer: Todd Lipcon <t...@apache.org>