Thanks for bringing this to the table, Patrick! A couple of comments: - adding to what Olaf said earlier: there are two ways of donating a codebase to ASF. One is to go through the Incubator, learn all the ropes and become a TLP. Another one, is by joining an existing TLP as a subproject. There's a number of steps being involved wrt IP clearance, license checks, SGA, CCLA, yada-yada. I am not going into more details like this, because you're not considering this path anyway ;) - from a quick glance at the project, looks like it fits the general bucket of data fabric platforms. Something like Apache Ignite comes to mind, while Aluxio, as a simple caching solution, only has partial target functionality. While there's no limitation on having multiple components with overlapping functionality in a given Bigtop stack (after all, we have HDFS and QFS), it's an aspect worthy of some consideration. - and to the RJ's point: with source code licensing question out of the way, it'd desirable to make sure that Crail doesn't re-distribute anything under conflicting licences. The ones in question are of GPL family and some more as you can see in [1]. While this isn't much of a concern for Bigtop, you might benefit from doing this for the project itself.
Regards,
Cos
On Wed, Jul 05, 2017 at 10:25AM, RJ Nowling wrote:
> I believe we already package several other open source projects that are
> not Apache projects. That should not be a hurdle. We do have restrictions
> on licenses, but it seems that Crail is licensed under the Apache License:
>
> https://github.com/zrlio/crail/blob/master/LICENSE
>
> I do not believe we have a formal process for accepting / declining. The
> key considerations, as Olaf mentioned, are licensing and willingness to
> maintain the packages. We do have a policy that any packages which break
> and do not have maintainers (or the maintainers do not respond) will be
> removed in the following release.
>
> I would suggest creating some JIRA tickets around the integration of Crail
> and your plans for each step. e.g., creating packages (RPMs, DEBs),
> deploying via our Puppet scripts, and some support in our integration
> testing frameworks. This would allow other community members to comment
> and provide feedback on the technical aspects.
>
>
> On Tue, Jul 4, 2017 at 6:18 AM, patrick stuedi <pstu...@gmail.com> wrote:
>
> > Olaf,
> >
> > many thanks for the quick reply!
> >
> > let me try to clarify -- the wording "donating the Crail code" was
> > misleading...what we propose is actually to contribute the packaging,
> > integration, automation and deployment of Crail to Bigtop, and of course
> > the continuous maintenance of it. Being an optional addon to the regular
> > BigTop stack would be absolutely fine. Question, do you guys have a
> > democratic process where you accept/decline the integration of new projects
> > like Crail? What next steps do you recommend?
> >
> > For us, integrating Crail into Bigtop would be interesting as it makes the
> > deployment and management of Crail easier. Currently deploying and
> > configuring Crail on a system/cluster is a rather complex and task.
> >
> > Regarding the RDMA integration, yes Crail at its core is based on native
> > verbs, but it is written in Java and uses DiSNI (
> > https://github.com/zrlio/disni) which essentially is a one-to-one mapping
> > of RDMA verbs into Java implemented on top of native verbs. But I do want
> > to clarify that Crail is not just about RDMA, it's a fast storage platform
> > designed for user-level APIs (RDMA, DPDK, SPDK, etc.) and high speed
> > networking and storage hardware (100GbE, NVMe, NVMe-oF, etc.).
> >
> > Thank you for mentioning Roman in the context of Apache incubator, we'll
> > reach out to him. Applying for Apache incubator is one of our absolute
> > goals. What is unclear to me is whether an integration of Crail into BigTop
> > is gated by Crail being Apache incubator already, or whether those efforts
> > can go on in parallel, can you say something about this?
> >
> >
> > On Mon, Jul 3, 2017 at 10:22 PM, Olaf Flebbe <o...@oflebbe.de> wrote:
> > >
> > > Well, let me speak up as an individual:
> > >
> > > I thank you for making this substantial project available under Apache
> > License!
> > >
> > > -- In my former job I was part of an HPC team and have some cursory
> > ideas of RDMA mostly with MPI or for cluster filesystems. At first glance
> > it looks to me like the first fully open sourced RDMA support for Big Data.
> > I am assuming the library is based on the standard RDMA "Verbs" API and not
> > on proprietary extensions of the IBM JDK like efforts before. --
> > >
> > > The big question is: what do you expect from Apache Bigtop / Apache
> > Software Foundation?
> > >
> > > If you like to contribute all your code to the Apache Foundation in order
> > to have a platform for you and other contributors to work on the code, then
> > I would recommend you to check out Apache Incubator. Roman can give you
> > more information, since he is PMC of Apache Incubator as well.
> > >
> > > Contributing all the things to Bigtop: I would say: Thank you, we are
> > honoured, but it does not fit. We are an integration and automation
> > project.
> > >
> > > If you like you to contribute packaging, integration, automation of
> > configuration and deployment of your code (All these seems to be missing,
> > at first glance) to Bigtop, feel free: We love to see innovative usecases
> > built with our stack. We surely can only accept this contribution as an
> > optional addon to our regular stack. And we will ask your group to maintain
> > it in future (see MAINTAINERS at our toplevel directory), since the code
> > and needed (virtual/real) infrastructure is not middle of the road. There
> > are a couple of contributions to Bigtop where the developers became
> > committers and are now actively working within Bigtop.
> > >
> > > Be please aware that this is my personal opinion, there may be other
> > views.
> > >
> > > Olaf
> > >
> > >
> >
signature.asc
Description: Digital signature
