This is a good example [1] of what we were referring to. Thanks, Cos
[1] https://issues.apache.org/jira/browse/BIGTOP-2283 -- Take care, Konstantin (Cos) Boudnik 2CAC 8312 4870 D885 8616 6115 220F 6980 1F27 E622 Disclaimer: Opinions expressed in this email are those of the author, and do not necessarily represent the views of any company the author might be affiliated with at the moment of writing. On Fri, Jul 7, 2017 at 1:28 PM, patrick stuedi <pstu...@gmail.com> wrote: > Cos, RJ, thanks for the response/advice..what I hear is that opening a JIRA > ticket on the crail integration would be a good starting point..we'll do so > and take it from it there.. > > On Jul 5, 2017 8:06 PM, "Konstantin Boudnik" <c...@apache.org> wrote: > > Thanks for bringing this to the table, Patrick! > > A couple of comments: > - adding to what Olaf said earlier: there are two ways of donating a > codebase > to ASF. One is to go through the Incubator, learn all the ropes and > become > a TLP. Another one, is by joining an existing TLP as a subproject. > There's > a number of steps being involved wrt IP clearance, license checks, SGA, > CCLA, yada-yada. I am not going into more details like this, because > you're > not considering this path anyway ;) > - from a quick glance at the project, looks like it fits the general bucket > of data fabric platforms. Something like Apache Ignite comes to mind, > while > Aluxio, as a simple caching solution, only has partial target > functionality. While there's no limitation on having multiple components > with overlapping functionality in a given Bigtop stack (after all, we > have > HDFS and QFS), it's an aspect worthy of some consideration. > - and to the RJ's point: with source code licensing question out of the > way, > it'd desirable to make sure that Crail doesn't re-distribute anything > under > conflicting licences. The ones in question are of GPL family and some > more > as you can see in [1]. While this isn't much of a concern for Bigtop, you > might benefit from doing this for the project itself. > > Regards, > Cos > > On Wed, Jul 05, 2017 at 10:25AM, RJ Nowling wrote: >> I believe we already package several other open source projects that are >> not Apache projects. That should not be a hurdle. We do have > restrictions >> on licenses, but it seems that Crail is licensed under the Apache License: >> >> https://github.com/zrlio/crail/blob/master/LICENSE >> >> I do not believe we have a formal process for accepting / declining. The >> key considerations, as Olaf mentioned, are licensing and willingness to >> maintain the packages. We do have a policy that any packages which break >> and do not have maintainers (or the maintainers do not respond) will be >> removed in the following release. >> >> I would suggest creating some JIRA tickets around the integration of Crail >> and your plans for each step. e.g., creating packages (RPMs, DEBs), >> deploying via our Puppet scripts, and some support in our integration >> testing frameworks. This would allow other community members to comment >> and provide feedback on the technical aspects. >> >> >> On Tue, Jul 4, 2017 at 6:18 AM, patrick stuedi <pstu...@gmail.com> wrote: >> >> > Olaf, >> > >> > many thanks for the quick reply! >> > >> > let me try to clarify -- the wording "donating the Crail code" was >> > misleading...what we propose is actually to contribute the packaging, >> > integration, automation and deployment of Crail to Bigtop, and of course >> > the continuous maintenance of it. Being an optional addon to the regular >> > BigTop stack would be absolutely fine. Question, do you guys have a >> > democratic process where you accept/decline the integration of new > projects >> > like Crail? What next steps do you recommend? >> > >> > For us, integrating Crail into Bigtop would be interesting as it makes > the >> > deployment and management of Crail easier. Currently deploying and >> > configuring Crail on a system/cluster is a rather complex and task. >> > >> > Regarding the RDMA integration, yes Crail at its core is based on native >> > verbs, but it is written in Java and uses DiSNI ( >> > https://github.com/zrlio/disni) which essentially is a one-to-one > mapping >> > of RDMA verbs into Java implemented on top of native verbs. But I do > want >> > to clarify that Crail is not just about RDMA, it's a fast storage > platform >> > designed for user-level APIs (RDMA, DPDK, SPDK, etc.) and high speed >> > networking and storage hardware (100GbE, NVMe, NVMe-oF, etc.). >> > >> > Thank you for mentioning Roman in the context of Apache incubator, we'll >> > reach out to him. Applying for Apache incubator is one of our absolute >> > goals. What is unclear to me is whether an integration of Crail into > BigTop >> > is gated by Crail being Apache incubator already, or whether those > efforts >> > can go on in parallel, can you say something about this? >> > >> > >> > On Mon, Jul 3, 2017 at 10:22 PM, Olaf Flebbe <o...@oflebbe.de> wrote: >> > > >> > > Well, let me speak up as an individual: >> > > >> > > I thank you for making this substantial project available under Apache >> > License! >> > > >> > > -- In my former job I was part of an HPC team and have some cursory >> > ideas of RDMA mostly with MPI or for cluster filesystems. At first > glance >> > it looks to me like the first fully open sourced RDMA support for Big > Data. >> > I am assuming the library is based on the standard RDMA "Verbs" API and > not >> > on proprietary extensions of the IBM JDK like efforts before. -- >> > > >> > > The big question is: what do you expect from Apache Bigtop / Apache >> > Software Foundation? >> > > >> > > If you like to contribute all your code to the Apache Foundation in > order >> > to have a platform for you and other contributors to work on the code, > then >> > I would recommend you to check out Apache Incubator. Roman can give you >> > more information, since he is PMC of Apache Incubator as well. >> > > >> > > Contributing all the things to Bigtop: I would say: Thank you, we are >> > honoured, but it does not fit. We are an integration and automation >> > project. >> > > >> > > If you like you to contribute packaging, integration, automation of >> > configuration and deployment of your code (All these seems to be > missing, >> > at first glance) to Bigtop, feel free: We love to see innovative > usecases >> > built with our stack. We surely can only accept this contribution as an >> > optional addon to our regular stack. And we will ask your group to > maintain >> > it in future (see MAINTAINERS at our toplevel directory), since the code >> > and needed (virtual/real) infrastructure is not middle of the road. > There >> > are a couple of contributions to Bigtop where the developers became >> > committers and are now actively working within Bigtop. >> > > >> > > Be please aware that this is my personal opinion, there may be other >> > views. >> > > >> > > Olaf >> > > >> > > >> >
