I wanted to provide a detailed update on the public testing utilities work that we have been doing since this email thread started to hopefully continue the discussion and motivate wrapping things up.
Recently I restructured the binary location logic to make it a bit more friendly to external projects. We also adjusted our testing base class to be a rule so it could be composed into other tests. Then shortly after branching Kudu 1.8 we broke out the testing utilities into their own jar. The idea, is that Kudu 1.9 would contain some public testing artifacts that other projects could use. Some of those relevant commits are: - [test] Clean up MiniKuduCluster and BaseKuduTest <https://github.com/apache/kudu/commit/fd1ffd0fb65e138f1f015a55aa96ae870c1d51cd> - [test] Adjust Kudu binary locator logic. <https://github.com/apache/kudu/commit/34e88d3dafc421ccaabae5767aad2fd9fa015d39> - [test] Move BaseKuduTest to a Junit Rule <https://github.com/apache/kudu/commit/dc8ae79961f71b8bdc344781fc89d38d94152fc4> - KUDU-2411: (Part 1) Break out existing test utilities into a seperate module <https://github.com/apache/kudu/commit/15f1416f67dcb714842d02647a1f2e06e675660d> - [java] Allow command line override of kuduBinDir <https://github.com/apache/kudu/commit/785490ce509e68029f8062882bb7021895e3b446> I tested Mike's patch (here <https://gerrit.cloudera.org/#/c/11377/>) and the generated artifact using -DkuduBinDir and things look like they will work. It seams like we need to find a way to package this stuff up, deploy it, and then write a utility class to locate it on the classpath and call it. Recently we have been looking at the way protoc is built and published <https://github.com/protocolbuffers/protobuf/tree/master/protoc-artifact> for inspiration and to see how they handle OSX. Tim's project <https://github.com/timrobertson100/kudu-test-server> is a good example of finding it on the classpath and executing it. Combining those 2 things we should have something useable. Any feedback, tips, suggestion or collaboration is more than welcome. Thanks, Grant On Sat, Aug 18, 2018 at 5:40 AM Tim Robertson <timrobertson...@gmail.com> wrote: > I've updated the GH to reflect the idea outlined above Mike > https://github.com/timrobertson100/kudu-test-server > > - code is fairly hacky > - deletes extracted binaries (FYI: deleteOnExit() will not delete dirs > unless empty) > - uses maven classifier (only linux exists) > > Can you PTAL Mike when you get a chance? - no rush at all > > > > On Thu, Aug 9, 2018 at 6:07 PM Tim <timrobertson...@gmail.com> wrote: > > > Thanks Mike - no apologies needed at all > > > > I’ll aim to reshape the GH repo I did to illustrate what we outlined, > with > > an example of use. > > > > Outstanding is some OS X binaries if anyone has some time? > > > > Have a good trip. > > > > Tim, > > Sent from my iPhone > > > > > On 9 Aug 2018, at 14:05, Mike Percy <mpe...@apache.org> wrote: > > > > > > Hi Tim, > > > Sorry for the delay in responding, I've been trying to get back to > this. > > > You make some great points. If we can forego Maven/Gradle plugins, we > can > > > do this with a lot less work. Initially, I was concerned about > unpacking > > > the bits potentially more than strictly necessary, but it seems likely > > that > > > the CPU / IO required to do the unpacking would probably not be > > noticeable > > > in the grand scheme of running tests against a Kudu MiniCluster, > > especially > > > if it's only done once per test file (using @BeforeClass or similar). > > Also, > > > as long as there is some way to clean up the files that were unpacked > and > > > avoid potentially filling up a temp dir then nobody should have a > problem > > > with this approach... perhaps we can register a JVM shutdown hook and > > also > > > provide some kind of teardown() method so people can ensure the files > get > > > cleaned up as appropriate (I saw the TODO in your test code for this; > > > protoc uses File.deleteOnExit()). > > > > > > Regarding creating the binary artifacts, I don't think it's too big of > a > > > burden to ask the release manager to upload either a RHEL 6 or macOS > > binary > > > test artifact based on the output of a script, and ask the PMC for help > > to > > > get a binary for the other platform. > > > > > > I'm out of town for the next couple of weeks but once I get back I'll > see > > > if I can push this forward a bit more. > > > > > > Thanks! > > > Mike > > > > > > > > > On Thu, Aug 2, 2018 at 10:57 PM Tim Robertson < > timrobertson...@gmail.com > > > > > > wrote: > > > > > >> Thanks to you for making this test possible Mike. > > >> > > >> I was approaching this emulating protoc where they put the binaries as > > >> artifacts for Maven [1]. > > >> A slight difference is that protoc is a single file while your tarball > > has > > >> several. I notice that the protoc build plugin for maven also copies > out > > >> from the jar to a local filesystem [2] so I just copied that approach. > > >> > > >> What I could imagine is we have a jar with the binaries and a single > > class > > >> (say EmbeddedKudu) that copies the binaries into a temporary directory > > (as > > >> my hack in GH). > > >> > > >> A user would then do the following: > > >> > > >> <!-- finds the environment --> > > >> <build> > > >> <extensions> > > >> <extension> > > >> <groupId>kr.motd.maven</groupId> > > >> <artifactId>os-maven-plugin</artifactId> > > >> <version>1.6.0</version> > > >> </extension> > > >> </extensions> > > >> </build> > > >> .... > > >> <!-- Adds mini cluster --> > > >> <dependency> > > >> <groupId>org.apache.kudu</groupId> > > >> <artifactId>kudu-client</artifactId> > > >> <version>1.7.0</version> > > >> <classifier>tests</classifier> > > >> </dependency> > > >> <!-- Adds an embedded Kudu --> > > >> <dependency> > > >> <groupId>org.apache.kudu</groupId> > > >> <artifactId>kudu-test-server</artifactId> > > >> <version>1.7.0</version> > > >> <classifier>${os.detected.classifier}</classifier> > > >> </dependency> > > >> > > >> > > >> The detect classifier stuff is copied from protoc, but perhaps we'd > just > > >> state that available options are (?) linux / OSX and ignore > > autodetection. > > >> > > >> And in code users would do something like: > > >> > > >> EmbeddedKudu.prepare(); // copies kudu from the jar and sets the > > >> system variable (as per my example) > > >> MiniKuduCluster miniCluster = > > >> new > > >> > > > MiniKuduCluster.MiniKuduClusterBuilder().numMasters(1).numTservers(1).build(); > > >> > > >> > > >> What this would mean: > > >> > > >> - no change to the existing Kudu code packkaging > > >> - leverage caching of the binaries as any Java artifact > > >> - download would be at build time not test runtime - only copying out > > from > > >> the jar each time > > >> - simple to include in most build environments (not tied to maven as a > > >> plugin) > > >> > > >> I can't comment if it makes sense to do this WRT the binaries though > > and it > > >> would mean building and releasing binaries into a jar on each Kudu > > release > > >> (as protoc does). > > >> > > >> WDYT? > > >> > > >> Thanks, > > >> Tim > > >> > > >> [1] http://repo1.maven.org/maven2/com/google/protobuf/protoc/3.6.1/ > > >> [2] > > >> > > >> > > > https://github.com/os72/protoc-jar-maven-plugin/blob/master/src/main/java/com/github/os72/protocjar/maven/ProtocJarMojo.java#L664 > > >> > > >>> On Thu, Aug 2, 2018 at 10:28 PM, Mike Percy <mpe...@apache.org> > wrote: > > >>> > > >>> Ha, neat, thanks for posting this, Tim. It's a nice proof of concept. > > >>> > > >>> I was imagining that we would try to implement the downloading part > as > > a > > >>> Maven plugin, but maybe it could work to try to download the > artifacts > > at > > >>> runtime with a JUnit test. Do you think we could cache the artifacts > > >>> somewhere, maybe in the Maven repo somehow, so we don't have to > > download > > >>> the artifact every time we want to use it? I was hoping to simply be > > able > > >>> to ship a tarball or a jar of the binaries separately from the test > > >>> framework code. > > >>> > > >>> Mike > > >>> > > >>> On Thu, Aug 2, 2018 at 8:42 AM Tim Robertson < > > timrobertson...@gmail.com> > > >>> wrote: > > >>> > > >>>> Hi folks > > >>>> > > >>>> I've not had too much time, but I threw this together using Mike's > > >>>> binaries: > > >>>> https://github.com/timrobertson100/kudu-test-server > > >>>> > > >>>> I think this shows that running a mini cluster is possible using the > > >>>> binaries Mike prepared when they are included in a jar (on CentOS > 7.4 > > >> at > > >>>> least). > > >>>> > > >>>> Please don't flame me for the code - it was a rush job - but perhaps > > >> you > > >>>> could verify it works for you Mike? > > >>>> > > >>>> > > >>>> > > >>>> > > >>>> > > >>>> > > >>>> On Thu, Jul 19, 2018 at 12:35 AM, Mike Percy <mpe...@apache.org> > > >> wrote: > > >>>> > > >>>>> On Wed, Jul 18, 2018 at 1:24 PM Tim Robertson < > > >>> timrobertson...@gmail.com > > >>>>> > > >>>>> wrote: > > >>>>> > > >>>>>>> I'm pretty busy this week and would be happy to get some help on > > >>> this > > >>>>> if > > >>>>>> anybody has cycles to spare. > > >>>>>> > > >>>>>> Thanks Mike - I'll look into 3) and if we get it working I'm happy > > >> to > > >>>>> offer > > >>>>>> a PR for 4). > > >>>>>> This is an evening project for me so I might be a little slow too > - > > >>> if > > >>>>>> someone else is keen and has time please feel free to jump in. > > >>>>>> > > >>>>> > > >>>>> Sounds great! I think Grant started working on #4 but I don't know > > >> how > > >>>> far > > >>>>> he got. > > >>>>> > > >>>>> Mike > > >>>>> > > >>>> > > >>> > > >> > > > -- Grant Henke Software Engineer | Cloudera gr...@cloudera.com | twitter.com/gchenke | linkedin.com/in/granthenke
