On Fri, Jun 29, 2018 at 12:31 PM, Mike Percy <mpe...@apache.org> wrote:
> This is something I've been thinking about and toying with and I'd like to > see if we can't get binaries available via Maven for at least one platform > (say, RHEL 7). Similar to how protobuf does it. > What about depending on a docker container than runs the kudu minicluster in "host" networking mode? eg https://github.com/MartinWeindel/kudu-docker is one possibility > How many platforms would need to be supported for it to be viable for Beam? > > Thanks, > Mike > > On Fri, Jun 29, 2018 at 10:01 AM Tim <timrobertson...@gmail.com> wrote: > > > Thanks Attila > > > > That’s great feedback and helpful for me to reference as guidance. > > > > By “Kudu installation” I was referring to the possibility that an install > > might set config etc, beyond just having the binary. I got it running on > > CentOS similar to how you outline now. > > > > I too believe mocking makes most sense, especially as we have the IT > > running as well, but was asked to explore this further. It’s useful to > know > > you’d agree. > > > > Thanks > > > > Tim > > > > > On 29 Jun 2018, at 17:33, Attila Bukor <abu...@cloudera.com> wrote: > > > > > > Hi Tim, > > > > > > I’m not sure what you mean by relying on actual installations. If you > > have the kudu, kudu-master and kudu-tserver binaries at the same location > > and they can be executed, MiniKuduCluster can be used (“binDir” property > > should be set to the directory containing the Kudu binaries). You should > > also look into BaseKuduTest as that will set up the MiniKuduCluster for > you > > and you don’t have to do it manually. > > > > > > Extracting the Kudu binaries from an rpm should probably work, but that > > binds you to CDH as currently Cloudera is the only one that ships Kudu > > binaries and MacOS builds are not available anywhere afaik. Also, 1.4.0 > is > > around a year old, you might want to use this repository instead (from > CDH > > 5.13 Kudu is part of the CDH): > > http://archive.cloudera.com/cdh5/redhat/7/x86_64/cdh/5/ > RPMS/x86_64/kudu-1.7.0+cdh5.15.0+0-1.cdh5.15.0.p0.52.el7.x86_64.rpm > > > > > > As a general suggestion, I would recommend mocking Kudu for unit tests > > (that’s what a unit test is for after all) and create separate > integration > > tests that actually use Kudu that can be skipped where Kudu is not > > available. Of course the CI should be set up to be able to provide all > > necessary integrations for the tests, but a developer wouldn’t have to > set > > up Kudu, or use Docker to run the tests if their change doesn’t affect > the > > Kudu integration. > > > > > > Attila > > > > > >> On 2018. Jun 29., at 16:42, Tim Robertson <timrobertson...@gmail.com> > > wrote: > > >> > > >> Hi folks, > > >> > > >> I've written Java KuduIO for Apache Beam with integration tests making > > use > > >> of Kudu in Docker. It is yet to be committed on Apache Beam. > > >> > > >> Rather than mocking Kudu client for unit tests I'd like to explore use > > of > > >> the MiniKuduCluster which "Depends on precompiled kudu, kudu-master, > and > > >> kudu-tserver binaries". > > >> > > >> I'd need unit tests to run on the main linux distros and OS X. > > >> > > >> For the linux distros, would an approach where I extract the binaries > > from > > >> the packages [1] work please? Or does the MiniKuduCluster rely on > actual > > >> installations? I am pretty weak on C builds and linked libraries etc > > (Java > > >> guy, sorry). > > >> > > >> For CentOS I'm exploring this for example: > > >> rpm2cpio ./kudu-1.4.0+cdh5.12.2+0-1.cdh5.12.2.p0.8.el7.x86_64.rpm | > > cpio > > >> -idmv > > >> > > >> I haven't explored OS X options yet. > > >> > > >> Any advice here would greatly be appreciated to save me going down a > > dead > > >> end. > > >> > > >> Many thanks, > > >> Tim > > >> > > >> > > >> [1] http://kudu.apache.org/docs/installation.html#install_packages > > > > > > > > > -- Todd Lipcon Software Engineer, Cloudera
