Thanks Mike - no apologies needed at all
I’ll aim to reshape the GH repo I did to illustrate what we outlined, with an
example of use.
Outstanding is some OS X binaries if anyone has some time?
Have a good trip.
Tim,
Sent from my iPhone
> On 9 Aug 2018, at 14:05, Mike Percy <mpe...@apache.org> wrote:
>
> Hi Tim,
> Sorry for the delay in responding, I've been trying to get back to this.
> You make some great points. If we can forego Maven/Gradle plugins, we can
> do this with a lot less work. Initially, I was concerned about unpacking
> the bits potentially more than strictly necessary, but it seems likely that
> the CPU / IO required to do the unpacking would probably not be noticeable
> in the grand scheme of running tests against a Kudu MiniCluster, especially
> if it's only done once per test file (using @BeforeClass or similar). Also,
> as long as there is some way to clean up the files that were unpacked and
> avoid potentially filling up a temp dir then nobody should have a problem
> with this approach... perhaps we can register a JVM shutdown hook and also
> provide some kind of teardown() method so people can ensure the files get
> cleaned up as appropriate (I saw the TODO in your test code for this;
> protoc uses File.deleteOnExit()).
>
> Regarding creating the binary artifacts, I don't think it's too big of a
> burden to ask the release manager to upload either a RHEL 6 or macOS binary
> test artifact based on the output of a script, and ask the PMC for help to
> get a binary for the other platform.
>
> I'm out of town for the next couple of weeks but once I get back I'll see
> if I can push this forward a bit more.
>
> Thanks!
> Mike
>
>
> On Thu, Aug 2, 2018 at 10:57 PM Tim Robertson <timrobertson...@gmail.com>
> wrote:
>
>> Thanks to you for making this test possible Mike.
>>
>> I was approaching this emulating protoc where they put the binaries as
>> artifacts for Maven [1].
>> A slight difference is that protoc is a single file while your tarball has
>> several. I notice that the protoc build plugin for maven also copies out
>> from the jar to a local filesystem [2] so I just copied that approach.
>>
>> What I could imagine is we have a jar with the binaries and a single class
>> (say EmbeddedKudu) that copies the binaries into a temporary directory (as
>> my hack in GH).
>>
>> A user would then do the following:
>>
>> <!-- finds the environment -->
>> <build>
>> <extensions>
>> <extension>
>> <groupId>kr.motd.maven</groupId>
>> <artifactId>os-maven-plugin</artifactId>
>> <version>1.6.0</version>
>> </extension>
>> </extensions>
>> </build>
>> ....
>> <!-- Adds mini cluster -->
>> <dependency>
>> <groupId>org.apache.kudu</groupId>
>> <artifactId>kudu-client</artifactId>
>> <version>1.7.0</version>
>> <classifier>tests</classifier>
>> </dependency>
>> <!-- Adds an embedded Kudu -->
>> <dependency>
>> <groupId>org.apache.kudu</groupId>
>> <artifactId>kudu-test-server</artifactId>
>> <version>1.7.0</version>
>> <classifier>${os.detected.classifier}</classifier>
>> </dependency>
>>
>>
>> The detect classifier stuff is copied from protoc, but perhaps we'd just
>> state that available options are (?) linux / OSX and ignore autodetection.
>>
>> And in code users would do something like:
>>
>> EmbeddedKudu.prepare(); // copies kudu from the jar and sets the
>> system variable (as per my example)
>> MiniKuduCluster miniCluster =
>> new
>> MiniKuduCluster.MiniKuduClusterBuilder().numMasters(1).numTservers(1).build();
>>
>>
>> What this would mean:
>>
>> - no change to the existing Kudu code packkaging
>> - leverage caching of the binaries as any Java artifact
>> - download would be at build time not test runtime - only copying out from
>> the jar each time
>> - simple to include in most build environments (not tied to maven as a
>> plugin)
>>
>> I can't comment if it makes sense to do this WRT the binaries though and it
>> would mean building and releasing binaries into a jar on each Kudu release
>> (as protoc does).
>>
>> WDYT?
>>
>> Thanks,
>> Tim
>>
>> [1] http://repo1.maven.org/maven2/com/google/protobuf/protoc/3.6.1/
>> [2]
>>
>> https://github.com/os72/protoc-jar-maven-plugin/blob/master/src/main/java/com/github/os72/protocjar/maven/ProtocJarMojo.java#L664
>>
>>> On Thu, Aug 2, 2018 at 10:28 PM, Mike Percy <mpe...@apache.org> wrote:
>>>
>>> Ha, neat, thanks for posting this, Tim. It's a nice proof of concept.
>>>
>>> I was imagining that we would try to implement the downloading part as a
>>> Maven plugin, but maybe it could work to try to download the artifacts at
>>> runtime with a JUnit test. Do you think we could cache the artifacts
>>> somewhere, maybe in the Maven repo somehow, so we don't have to download
>>> the artifact every time we want to use it? I was hoping to simply be able
>>> to ship a tarball or a jar of the binaries separately from the test
>>> framework code.
>>>
>>> Mike
>>>
>>> On Thu, Aug 2, 2018 at 8:42 AM Tim Robertson <timrobertson...@gmail.com>
>>> wrote:
>>>
>>>> Hi folks
>>>>
>>>> I've not had too much time, but I threw this together using Mike's
>>>> binaries:
>>>> https://github.com/timrobertson100/kudu-test-server
>>>>
>>>> I think this shows that running a mini cluster is possible using the
>>>> binaries Mike prepared when they are included in a jar (on CentOS 7.4
>> at
>>>> least).
>>>>
>>>> Please don't flame me for the code - it was a rush job - but perhaps
>> you
>>>> could verify it works for you Mike?
>>>>
>>>>
>>>>
>>>>
>>>>
>>>>
>>>> On Thu, Jul 19, 2018 at 12:35 AM, Mike Percy <mpe...@apache.org>
>> wrote:
>>>>
>>>>> On Wed, Jul 18, 2018 at 1:24 PM Tim Robertson <
>>> timrobertson...@gmail.com
>>>>>
>>>>> wrote:
>>>>>
>>>>>>> I'm pretty busy this week and would be happy to get some help on
>>> this
>>>>> if
>>>>>> anybody has cycles to spare.
>>>>>>
>>>>>> Thanks Mike - I'll look into 3) and if we get it working I'm happy
>> to
>>>>> offer
>>>>>> a PR for 4).
>>>>>> This is an evening project for me so I might be a little slow too -
>>> if
>>>>>> someone else is keen and has time please feel free to jump in.
>>>>>>
>>>>>
>>>>> Sounds great! I think Grant started working on #4 but I don't know
>> how
>>>> far
>>>>> he got.
>>>>>
>>>>> Mike
>>>>>
>>>>
>>>
>>
