This is something I've been thinking about and toying with and I'd like to see if we can't get binaries available via Maven for at least one platform (say, RHEL 7). Similar to how protobuf does it.
How many platforms would need to be supported for it to be viable for Beam? Thanks, Mike On Fri, Jun 29, 2018 at 10:01 AM Tim <timrobertson...@gmail.com> wrote: > Thanks Attila > > That’s great feedback and helpful for me to reference as guidance. > > By “Kudu installation” I was referring to the possibility that an install > might set config etc, beyond just having the binary. I got it running on > CentOS similar to how you outline now. > > I too believe mocking makes most sense, especially as we have the IT > running as well, but was asked to explore this further. It’s useful to know > you’d agree. > > Thanks > > Tim > > > On 29 Jun 2018, at 17:33, Attila Bukor <abu...@cloudera.com> wrote: > > > > Hi Tim, > > > > I’m not sure what you mean by relying on actual installations. If you > have the kudu, kudu-master and kudu-tserver binaries at the same location > and they can be executed, MiniKuduCluster can be used (“binDir” property > should be set to the directory containing the Kudu binaries). You should > also look into BaseKuduTest as that will set up the MiniKuduCluster for you > and you don’t have to do it manually. > > > > Extracting the Kudu binaries from an rpm should probably work, but that > binds you to CDH as currently Cloudera is the only one that ships Kudu > binaries and MacOS builds are not available anywhere afaik. Also, 1.4.0 is > around a year old, you might want to use this repository instead (from CDH > 5.13 Kudu is part of the CDH): > http://archive.cloudera.com/cdh5/redhat/7/x86_64/cdh/5/RPMS/x86_64/kudu-1.7.0+cdh5.15.0+0-1.cdh5.15.0.p0.52.el7.x86_64.rpm > > > > As a general suggestion, I would recommend mocking Kudu for unit tests > (that’s what a unit test is for after all) and create separate integration > tests that actually use Kudu that can be skipped where Kudu is not > available. Of course the CI should be set up to be able to provide all > necessary integrations for the tests, but a developer wouldn’t have to set > up Kudu, or use Docker to run the tests if their change doesn’t affect the > Kudu integration. > > > > Attila > > > >> On 2018. Jun 29., at 16:42, Tim Robertson <timrobertson...@gmail.com> > wrote: > >> > >> Hi folks, > >> > >> I've written Java KuduIO for Apache Beam with integration tests making > use > >> of Kudu in Docker. It is yet to be committed on Apache Beam. > >> > >> Rather than mocking Kudu client for unit tests I'd like to explore use > of > >> the MiniKuduCluster which "Depends on precompiled kudu, kudu-master, and > >> kudu-tserver binaries". > >> > >> I'd need unit tests to run on the main linux distros and OS X. > >> > >> For the linux distros, would an approach where I extract the binaries > from > >> the packages [1] work please? Or does the MiniKuduCluster rely on actual > >> installations? I am pretty weak on C builds and linked libraries etc > (Java > >> guy, sorry). > >> > >> For CentOS I'm exploring this for example: > >> rpm2cpio ./kudu-1.4.0+cdh5.12.2+0-1.cdh5.12.2.p0.8.el7.x86_64.rpm | > cpio > >> -idmv > >> > >> I haven't explored OS X options yet. > >> > >> Any advice here would greatly be appreciated to save me going down a > dead > >> end. > >> > >> Many thanks, > >> Tim > >> > >> > >> [1] http://kudu.apache.org/docs/installation.html#install_packages > > > > >
