Thanks for bringing this to the table, Patrick! A couple of comments: - adding to what Olaf said earlier: there are two ways of donating a codebase to ASF. One is to go through the Incubator, learn all the ropes and become a TLP. Another one, is by joining an existing TLP as a subproject. There's a number of steps being involved wrt IP clearance, license checks, SGA, CCLA, yada-yada. I am not going into more details like this, because you're not considering this path anyway ;) - from a quick glance at the project, looks like it fits the general bucket of data fabric platforms. Something like Apache Ignite comes to mind, while Aluxio, as a simple caching solution, only has partial target functionality. While there's no limitation on having multiple components with overlapping functionality in a given Bigtop stack (after all, we have HDFS and QFS), it's an aspect worthy of some consideration. - and to the RJ's point: with source code licensing question out of the way, it'd desirable to make sure that Crail doesn't re-distribute anything under conflicting licences. The ones in question are of GPL family and some more as you can see in [1]. While this isn't much of a concern for Bigtop, you might benefit from doing this for the project itself.
Regards, Cos On Wed, Jul 05, 2017 at 10:25AM, RJ Nowling wrote: > I believe we already package several other open source projects that are > not Apache projects. That should not be a hurdle. We do have restrictions > on licenses, but it seems that Crail is licensed under the Apache License: > > https://github.com/zrlio/crail/blob/master/LICENSE > > I do not believe we have a formal process for accepting / declining. The > key considerations, as Olaf mentioned, are licensing and willingness to > maintain the packages. We do have a policy that any packages which break > and do not have maintainers (or the maintainers do not respond) will be > removed in the following release. > > I would suggest creating some JIRA tickets around the integration of Crail > and your plans for each step. e.g., creating packages (RPMs, DEBs), > deploying via our Puppet scripts, and some support in our integration > testing frameworks. This would allow other community members to comment > and provide feedback on the technical aspects. > > > On Tue, Jul 4, 2017 at 6:18 AM, patrick stuedi <pstu...@gmail.com> wrote: > > > Olaf, > > > > many thanks for the quick reply! > > > > let me try to clarify -- the wording "donating the Crail code" was > > misleading...what we propose is actually to contribute the packaging, > > integration, automation and deployment of Crail to Bigtop, and of course > > the continuous maintenance of it. Being an optional addon to the regular > > BigTop stack would be absolutely fine. Question, do you guys have a > > democratic process where you accept/decline the integration of new projects > > like Crail? What next steps do you recommend? > > > > For us, integrating Crail into Bigtop would be interesting as it makes the > > deployment and management of Crail easier. Currently deploying and > > configuring Crail on a system/cluster is a rather complex and task. > > > > Regarding the RDMA integration, yes Crail at its core is based on native > > verbs, but it is written in Java and uses DiSNI ( > > https://github.com/zrlio/disni) which essentially is a one-to-one mapping > > of RDMA verbs into Java implemented on top of native verbs. But I do want > > to clarify that Crail is not just about RDMA, it's a fast storage platform > > designed for user-level APIs (RDMA, DPDK, SPDK, etc.) and high speed > > networking and storage hardware (100GbE, NVMe, NVMe-oF, etc.). > > > > Thank you for mentioning Roman in the context of Apache incubator, we'll > > reach out to him. Applying for Apache incubator is one of our absolute > > goals. What is unclear to me is whether an integration of Crail into BigTop > > is gated by Crail being Apache incubator already, or whether those efforts > > can go on in parallel, can you say something about this? > > > > > > On Mon, Jul 3, 2017 at 10:22 PM, Olaf Flebbe <o...@oflebbe.de> wrote: > > > > > > Well, let me speak up as an individual: > > > > > > I thank you for making this substantial project available under Apache > > License! > > > > > > -- In my former job I was part of an HPC team and have some cursory > > ideas of RDMA mostly with MPI or for cluster filesystems. At first glance > > it looks to me like the first fully open sourced RDMA support for Big Data. > > I am assuming the library is based on the standard RDMA "Verbs" API and not > > on proprietary extensions of the IBM JDK like efforts before. -- > > > > > > The big question is: what do you expect from Apache Bigtop / Apache > > Software Foundation? > > > > > > If you like to contribute all your code to the Apache Foundation in order > > to have a platform for you and other contributors to work on the code, then > > I would recommend you to check out Apache Incubator. Roman can give you > > more information, since he is PMC of Apache Incubator as well. > > > > > > Contributing all the things to Bigtop: I would say: Thank you, we are > > honoured, but it does not fit. We are an integration and automation > > project. > > > > > > If you like you to contribute packaging, integration, automation of > > configuration and deployment of your code (All these seems to be missing, > > at first glance) to Bigtop, feel free: We love to see innovative usecases > > built with our stack. We surely can only accept this contribution as an > > optional addon to our regular stack. And we will ask your group to maintain > > it in future (see MAINTAINERS at our toplevel directory), since the code > > and needed (virtual/real) infrastructure is not middle of the road. There > > are a couple of contributions to Bigtop where the developers became > > committers and are now actively working within Bigtop. > > > > > > Be please aware that this is my personal opinion, there may be other > > views. > > > > > > Olaf > > > > > > > >
signature.asc
Description: Digital signature