Re: Crail as part of BigTop
Hello Everyone, We have submitted the grant application. Thanks a lot Roman for pointing it out. The deadline extension was helpful for us to get our application together. PS~ I had to re-submit the grant application because I did not receive any confirmation email even after 24 hours. Regards, -- Animesh On Tue, Jul 11, 2017 at 2:26 PM, Animesh Trivedi wrote: > Hi Roman, > > Many thanks for pointing out the grant program. We are certainly > interested, and barring any unforeseen circumstances, we hope to submit an > application by the coming Friday ! > > Thanks, > -- > Animesh > > On Mon, Jul 10, 2017 at 10:10 PM, Roman Shaposhnik > wrote: > >> Piling on top of others: I think this also will make an excellent >> proposal. Any chance you can consider this for: >>http://www.prnewswire.com/news-releases/odpi-launches-apach >> e-bigtop-grant-fund-program-300472753.html >> >> ? >> >> Thanks, >> Roman. >> >> On Mon, Jul 10, 2017 at 4:15 PM, Animesh Trivedi >> wrote: >> > Hello all, >> > >> > many thanks for positive responses. We have created a JIRA for tracking >> the >> > progress of Crail and BigTop integration here: >> > >> > https://issues.apache.org/jira/browse/BIGTOP-2837 >> > >> > Any feedback/suggestions/recommendations are appreciated. >> > >> > Thanks, >> > -- >> > Animesh >> > >> > On Mon, Jul 10, 2017 at 2:16 PM, MrAsanjar . >> wrote: >> > >> >> Thanks Cos, >> >> >> >> >> >> On Fri, Jul 7, 2017 at 3:39 PM, Konstantin Boudnik >> wrote: >> >> >> >> > This is a good example [1] of what we were referring to. >> >> > >> >> > Thanks, >> >> > Cos >> >> > >> >> > [1] https://issues.apache.org/jira/browse/BIGTOP-2283 >> >> > -- >> >> > Take care, >> >> > Konstantin (Cos) Boudnik >> >> > 2CAC 8312 4870 D885 8616 6115 220F 6980 1F27 E622 >> >> > >> >> > Disclaimer: Opinions expressed in this email are those of the author, >> >> > and do not necessarily represent the views of any company the author >> >> > might be affiliated with at the moment of writing. >> >> > >> >> > >> >> > On Fri, Jul 7, 2017 at 1:28 PM, patrick stuedi >> >> wrote: >> >> > > Cos, RJ, thanks for the response/advice..what I hear is that >> opening a >> >> > JIRA >> >> > > ticket on the crail integration would be a good starting >> point..we'll >> >> do >> >> > so >> >> > > and take it from it there.. >> >> > > >> >> > > On Jul 5, 2017 8:06 PM, "Konstantin Boudnik" >> wrote: >> >> > > >> >> > > Thanks for bringing this to the table, Patrick! >> >> > > >> >> > > A couple of comments: >> >> > > - adding to what Olaf said earlier: there are two ways of >> donating a >> >> > > codebase >> >> > >to ASF. One is to go through the Incubator, learn all the ropes >> and >> >> > > become >> >> > >a TLP. Another one, is by joining an existing TLP as a >> subproject. >> >> > > There's >> >> > >a number of steps being involved wrt IP clearance, license >> checks, >> >> > SGA, >> >> > >CCLA, yada-yada. I am not going into more details like this, >> because >> >> > > you're >> >> > >not considering this path anyway ;) >> >> > > - from a quick glance at the project, looks like it fits the >> general >> >> > bucket >> >> > >of data fabric platforms. Something like Apache Ignite comes to >> >> mind, >> >> > > while >> >> > >Aluxio, as a simple caching solution, only has partial target >> >> > >functionality. While there's no limitation on having multiple >> >> > components >> >> > >with overlapping functionality in a given Bigtop stack (after >> all, >> >> we >> >> > > have >> >> > >HDFS and QFS), it's an aspect worthy of some consideration. >> >> > > - and to the RJ's point: with source code licensing question out >> of >> >> the >> >> > > way, >> >> > >it'd desirable to make sure that Crail doesn't re-distribute >> >> anything >> >> > > under >> >> > >conflicting licences. The ones in question are of GPL family and >> >> some >> >> > > more >> >> > >as you can see in [1]. While this isn't much of a concern for >> >> Bigtop, >> >> > you >> >> > >might benefit from doing this for the project itself. >> >> > > >> >> > > Regards, >> >> > > Cos >> >> > > >> >> > > On Wed, Jul 05, 2017 at 10:25AM, RJ Nowling wrote: >> >> > >> I believe we already package several other open source projects >> that >> >> are >> >> > >> not Apache projects. That should not be a hurdle. We do have >> >> > > restrictions >> >> > >> on licenses, but it seems that Crail is licensed under the Apache >> >> > License: >> >> > >> >> >> > >> https://github.com/zrlio/crail/blob/master/LICENSE >> >> > >> >> >> > >> I do not believe we have a formal process for accepting / >> declining. >> >> > The >> >> > >> key considerations, as Olaf mentioned, are licensing and >> willingness >> >> to >> >> > >> maintain the packages. We do have a policy that any packages >> which >> >> > break >> >> > >> and do not have maintainers (or the maintainers do not respond) >> will >> >> be >> >> > >> removed in the following r
Re: Crail as part of BigTop
Hi Roman, Many thanks for pointing out the grant program. We are certainly interested, and barring any unforeseen circumstances, we hope to submit an application by the coming Friday ! Thanks, -- Animesh On Mon, Jul 10, 2017 at 10:10 PM, Roman Shaposhnik wrote: > Piling on top of others: I think this also will make an excellent > proposal. Any chance you can consider this for: >http://www.prnewswire.com/news-releases/odpi-launches- > apache-bigtop-grant-fund-program-300472753.html > > ? > > Thanks, > Roman. > > On Mon, Jul 10, 2017 at 4:15 PM, Animesh Trivedi > wrote: > > Hello all, > > > > many thanks for positive responses. We have created a JIRA for tracking > the > > progress of Crail and BigTop integration here: > > > > https://issues.apache.org/jira/browse/BIGTOP-2837 > > > > Any feedback/suggestions/recommendations are appreciated. > > > > Thanks, > > -- > > Animesh > > > > On Mon, Jul 10, 2017 at 2:16 PM, MrAsanjar . wrote: > > > >> Thanks Cos, > >> > >> > >> On Fri, Jul 7, 2017 at 3:39 PM, Konstantin Boudnik > wrote: > >> > >> > This is a good example [1] of what we were referring to. > >> > > >> > Thanks, > >> > Cos > >> > > >> > [1] https://issues.apache.org/jira/browse/BIGTOP-2283 > >> > -- > >> > Take care, > >> > Konstantin (Cos) Boudnik > >> > 2CAC 8312 4870 D885 8616 6115 220F 6980 1F27 E622 > >> > > >> > Disclaimer: Opinions expressed in this email are those of the author, > >> > and do not necessarily represent the views of any company the author > >> > might be affiliated with at the moment of writing. > >> > > >> > > >> > On Fri, Jul 7, 2017 at 1:28 PM, patrick stuedi > >> wrote: > >> > > Cos, RJ, thanks for the response/advice..what I hear is that > opening a > >> > JIRA > >> > > ticket on the crail integration would be a good starting > point..we'll > >> do > >> > so > >> > > and take it from it there.. > >> > > > >> > > On Jul 5, 2017 8:06 PM, "Konstantin Boudnik" > wrote: > >> > > > >> > > Thanks for bringing this to the table, Patrick! > >> > > > >> > > A couple of comments: > >> > > - adding to what Olaf said earlier: there are two ways of donating > a > >> > > codebase > >> > >to ASF. One is to go through the Incubator, learn all the ropes > and > >> > > become > >> > >a TLP. Another one, is by joining an existing TLP as a > subproject. > >> > > There's > >> > >a number of steps being involved wrt IP clearance, license > checks, > >> > SGA, > >> > >CCLA, yada-yada. I am not going into more details like this, > because > >> > > you're > >> > >not considering this path anyway ;) > >> > > - from a quick glance at the project, looks like it fits the > general > >> > bucket > >> > >of data fabric platforms. Something like Apache Ignite comes to > >> mind, > >> > > while > >> > >Aluxio, as a simple caching solution, only has partial target > >> > >functionality. While there's no limitation on having multiple > >> > components > >> > >with overlapping functionality in a given Bigtop stack (after > all, > >> we > >> > > have > >> > >HDFS and QFS), it's an aspect worthy of some consideration. > >> > > - and to the RJ's point: with source code licensing question out of > >> the > >> > > way, > >> > >it'd desirable to make sure that Crail doesn't re-distribute > >> anything > >> > > under > >> > >conflicting licences. The ones in question are of GPL family and > >> some > >> > > more > >> > >as you can see in [1]. While this isn't much of a concern for > >> Bigtop, > >> > you > >> > >might benefit from doing this for the project itself. > >> > > > >> > > Regards, > >> > > Cos > >> > > > >> > > On Wed, Jul 05, 2017 at 10:25AM, RJ Nowling wrote: > >> > >> I believe we already package several other open source projects > that > >> are > >> > >> not Apache projects. That should not be a hurdle. We do have > >> > > restrictions > >> > >> on licenses, but it seems that Crail is licensed under the Apache > >> > License: > >> > >> > >> > >> https://github.com/zrlio/crail/blob/master/LICENSE > >> > >> > >> > >> I do not believe we have a formal process for accepting / > declining. > >> > The > >> > >> key considerations, as Olaf mentioned, are licensing and > willingness > >> to > >> > >> maintain the packages. We do have a policy that any packages which > >> > break > >> > >> and do not have maintainers (or the maintainers do not respond) > will > >> be > >> > >> removed in the following release. > >> > >> > >> > >> I would suggest creating some JIRA tickets around the integration > of > >> > Crail > >> > >> and your plans for each step. e.g., creating packages (RPMs, > DEBs), > >> > >> deploying via our Puppet scripts, and some support in our > integration > >> > >> testing frameworks. This would allow other community members to > >> comment > >> > >> and provide feedback on the technical aspects. > >> > >> > >> > >> > >> > >> On Tue, Jul 4, 2017 at 6:18 AM, patrick stuedi > >> > wrote: > >> > >> > >> > >> > Olaf
Re: Crail as part of BigTop
Piling on top of others: I think this also will make an excellent proposal. Any chance you can consider this for: http://www.prnewswire.com/news-releases/odpi-launches-apache-bigtop-grant-fund-program-300472753.html ? Thanks, Roman. On Mon, Jul 10, 2017 at 4:15 PM, Animesh Trivedi wrote: > Hello all, > > many thanks for positive responses. We have created a JIRA for tracking the > progress of Crail and BigTop integration here: > > https://issues.apache.org/jira/browse/BIGTOP-2837 > > Any feedback/suggestions/recommendations are appreciated. > > Thanks, > -- > Animesh > > On Mon, Jul 10, 2017 at 2:16 PM, MrAsanjar . wrote: > >> Thanks Cos, >> >> >> On Fri, Jul 7, 2017 at 3:39 PM, Konstantin Boudnik wrote: >> >> > This is a good example [1] of what we were referring to. >> > >> > Thanks, >> > Cos >> > >> > [1] https://issues.apache.org/jira/browse/BIGTOP-2283 >> > -- >> > Take care, >> > Konstantin (Cos) Boudnik >> > 2CAC 8312 4870 D885 8616 6115 220F 6980 1F27 E622 >> > >> > Disclaimer: Opinions expressed in this email are those of the author, >> > and do not necessarily represent the views of any company the author >> > might be affiliated with at the moment of writing. >> > >> > >> > On Fri, Jul 7, 2017 at 1:28 PM, patrick stuedi >> wrote: >> > > Cos, RJ, thanks for the response/advice..what I hear is that opening a >> > JIRA >> > > ticket on the crail integration would be a good starting point..we'll >> do >> > so >> > > and take it from it there.. >> > > >> > > On Jul 5, 2017 8:06 PM, "Konstantin Boudnik" wrote: >> > > >> > > Thanks for bringing this to the table, Patrick! >> > > >> > > A couple of comments: >> > > - adding to what Olaf said earlier: there are two ways of donating a >> > > codebase >> > >to ASF. One is to go through the Incubator, learn all the ropes and >> > > become >> > >a TLP. Another one, is by joining an existing TLP as a subproject. >> > > There's >> > >a number of steps being involved wrt IP clearance, license checks, >> > SGA, >> > >CCLA, yada-yada. I am not going into more details like this, because >> > > you're >> > >not considering this path anyway ;) >> > > - from a quick glance at the project, looks like it fits the general >> > bucket >> > >of data fabric platforms. Something like Apache Ignite comes to >> mind, >> > > while >> > >Aluxio, as a simple caching solution, only has partial target >> > >functionality. While there's no limitation on having multiple >> > components >> > >with overlapping functionality in a given Bigtop stack (after all, >> we >> > > have >> > >HDFS and QFS), it's an aspect worthy of some consideration. >> > > - and to the RJ's point: with source code licensing question out of >> the >> > > way, >> > >it'd desirable to make sure that Crail doesn't re-distribute >> anything >> > > under >> > >conflicting licences. The ones in question are of GPL family and >> some >> > > more >> > >as you can see in [1]. While this isn't much of a concern for >> Bigtop, >> > you >> > >might benefit from doing this for the project itself. >> > > >> > > Regards, >> > > Cos >> > > >> > > On Wed, Jul 05, 2017 at 10:25AM, RJ Nowling wrote: >> > >> I believe we already package several other open source projects that >> are >> > >> not Apache projects. That should not be a hurdle. We do have >> > > restrictions >> > >> on licenses, but it seems that Crail is licensed under the Apache >> > License: >> > >> >> > >> https://github.com/zrlio/crail/blob/master/LICENSE >> > >> >> > >> I do not believe we have a formal process for accepting / declining. >> > The >> > >> key considerations, as Olaf mentioned, are licensing and willingness >> to >> > >> maintain the packages. We do have a policy that any packages which >> > break >> > >> and do not have maintainers (or the maintainers do not respond) will >> be >> > >> removed in the following release. >> > >> >> > >> I would suggest creating some JIRA tickets around the integration of >> > Crail >> > >> and your plans for each step. e.g., creating packages (RPMs, DEBs), >> > >> deploying via our Puppet scripts, and some support in our integration >> > >> testing frameworks. This would allow other community members to >> comment >> > >> and provide feedback on the technical aspects. >> > >> >> > >> >> > >> On Tue, Jul 4, 2017 at 6:18 AM, patrick stuedi >> > wrote: >> > >> >> > >> > Olaf, >> > >> > >> > >> > many thanks for the quick reply! >> > >> > >> > >> > let me try to clarify -- the wording "donating the Crail code" was >> > >> > misleading...what we propose is actually to contribute the >> packaging, >> > >> > integration, automation and deployment of Crail to Bigtop, and of >> > course >> > >> > the continuous maintenance of it. Being an optional addon to the >> > regular >> > >> > BigTop stack would be absolutely fine. Question, do you guys have a >> > >> > democratic process where you accept/decline the integration of new >> > > projects
Re: Crail as part of BigTop
Hello all, many thanks for positive responses. We have created a JIRA for tracking the progress of Crail and BigTop integration here: https://issues.apache.org/jira/browse/BIGTOP-2837 Any feedback/suggestions/recommendations are appreciated. Thanks, -- Animesh On Mon, Jul 10, 2017 at 2:16 PM, MrAsanjar . wrote: > Thanks Cos, > > > On Fri, Jul 7, 2017 at 3:39 PM, Konstantin Boudnik wrote: > > > This is a good example [1] of what we were referring to. > > > > Thanks, > > Cos > > > > [1] https://issues.apache.org/jira/browse/BIGTOP-2283 > > -- > > Take care, > > Konstantin (Cos) Boudnik > > 2CAC 8312 4870 D885 8616 6115 220F 6980 1F27 E622 > > > > Disclaimer: Opinions expressed in this email are those of the author, > > and do not necessarily represent the views of any company the author > > might be affiliated with at the moment of writing. > > > > > > On Fri, Jul 7, 2017 at 1:28 PM, patrick stuedi > wrote: > > > Cos, RJ, thanks for the response/advice..what I hear is that opening a > > JIRA > > > ticket on the crail integration would be a good starting point..we'll > do > > so > > > and take it from it there.. > > > > > > On Jul 5, 2017 8:06 PM, "Konstantin Boudnik" wrote: > > > > > > Thanks for bringing this to the table, Patrick! > > > > > > A couple of comments: > > > - adding to what Olaf said earlier: there are two ways of donating a > > > codebase > > >to ASF. One is to go through the Incubator, learn all the ropes and > > > become > > >a TLP. Another one, is by joining an existing TLP as a subproject. > > > There's > > >a number of steps being involved wrt IP clearance, license checks, > > SGA, > > >CCLA, yada-yada. I am not going into more details like this, because > > > you're > > >not considering this path anyway ;) > > > - from a quick glance at the project, looks like it fits the general > > bucket > > >of data fabric platforms. Something like Apache Ignite comes to > mind, > > > while > > >Aluxio, as a simple caching solution, only has partial target > > >functionality. While there's no limitation on having multiple > > components > > >with overlapping functionality in a given Bigtop stack (after all, > we > > > have > > >HDFS and QFS), it's an aspect worthy of some consideration. > > > - and to the RJ's point: with source code licensing question out of > the > > > way, > > >it'd desirable to make sure that Crail doesn't re-distribute > anything > > > under > > >conflicting licences. The ones in question are of GPL family and > some > > > more > > >as you can see in [1]. While this isn't much of a concern for > Bigtop, > > you > > >might benefit from doing this for the project itself. > > > > > > Regards, > > > Cos > > > > > > On Wed, Jul 05, 2017 at 10:25AM, RJ Nowling wrote: > > >> I believe we already package several other open source projects that > are > > >> not Apache projects. That should not be a hurdle. We do have > > > restrictions > > >> on licenses, but it seems that Crail is licensed under the Apache > > License: > > >> > > >> https://github.com/zrlio/crail/blob/master/LICENSE > > >> > > >> I do not believe we have a formal process for accepting / declining. > > The > > >> key considerations, as Olaf mentioned, are licensing and willingness > to > > >> maintain the packages. We do have a policy that any packages which > > break > > >> and do not have maintainers (or the maintainers do not respond) will > be > > >> removed in the following release. > > >> > > >> I would suggest creating some JIRA tickets around the integration of > > Crail > > >> and your plans for each step. e.g., creating packages (RPMs, DEBs), > > >> deploying via our Puppet scripts, and some support in our integration > > >> testing frameworks. This would allow other community members to > comment > > >> and provide feedback on the technical aspects. > > >> > > >> > > >> On Tue, Jul 4, 2017 at 6:18 AM, patrick stuedi > > wrote: > > >> > > >> > Olaf, > > >> > > > >> > many thanks for the quick reply! > > >> > > > >> > let me try to clarify -- the wording "donating the Crail code" was > > >> > misleading...what we propose is actually to contribute the > packaging, > > >> > integration, automation and deployment of Crail to Bigtop, and of > > course > > >> > the continuous maintenance of it. Being an optional addon to the > > regular > > >> > BigTop stack would be absolutely fine. Question, do you guys have a > > >> > democratic process where you accept/decline the integration of new > > > projects > > >> > like Crail? What next steps do you recommend? > > >> > > > >> > For us, integrating Crail into Bigtop would be interesting as it > makes > > > the > > >> > deployment and management of Crail easier. Currently deploying and > > >> > configuring Crail on a system/cluster is a rather complex and task. > > >> > > > >> > Regarding the RDMA integration, yes Crail at its core is based on > > native > > >> > verbs, but it is written in J
Re: Crail as part of BigTop
Thanks Cos, On Fri, Jul 7, 2017 at 3:39 PM, Konstantin Boudnik wrote: > This is a good example [1] of what we were referring to. > > Thanks, > Cos > > [1] https://issues.apache.org/jira/browse/BIGTOP-2283 > -- > Take care, > Konstantin (Cos) Boudnik > 2CAC 8312 4870 D885 8616 6115 220F 6980 1F27 E622 > > Disclaimer: Opinions expressed in this email are those of the author, > and do not necessarily represent the views of any company the author > might be affiliated with at the moment of writing. > > > On Fri, Jul 7, 2017 at 1:28 PM, patrick stuedi wrote: > > Cos, RJ, thanks for the response/advice..what I hear is that opening a > JIRA > > ticket on the crail integration would be a good starting point..we'll do > so > > and take it from it there.. > > > > On Jul 5, 2017 8:06 PM, "Konstantin Boudnik" wrote: > > > > Thanks for bringing this to the table, Patrick! > > > > A couple of comments: > > - adding to what Olaf said earlier: there are two ways of donating a > > codebase > >to ASF. One is to go through the Incubator, learn all the ropes and > > become > >a TLP. Another one, is by joining an existing TLP as a subproject. > > There's > >a number of steps being involved wrt IP clearance, license checks, > SGA, > >CCLA, yada-yada. I am not going into more details like this, because > > you're > >not considering this path anyway ;) > > - from a quick glance at the project, looks like it fits the general > bucket > >of data fabric platforms. Something like Apache Ignite comes to mind, > > while > >Aluxio, as a simple caching solution, only has partial target > >functionality. While there's no limitation on having multiple > components > >with overlapping functionality in a given Bigtop stack (after all, we > > have > >HDFS and QFS), it's an aspect worthy of some consideration. > > - and to the RJ's point: with source code licensing question out of the > > way, > >it'd desirable to make sure that Crail doesn't re-distribute anything > > under > >conflicting licences. The ones in question are of GPL family and some > > more > >as you can see in [1]. While this isn't much of a concern for Bigtop, > you > >might benefit from doing this for the project itself. > > > > Regards, > > Cos > > > > On Wed, Jul 05, 2017 at 10:25AM, RJ Nowling wrote: > >> I believe we already package several other open source projects that are > >> not Apache projects. That should not be a hurdle. We do have > > restrictions > >> on licenses, but it seems that Crail is licensed under the Apache > License: > >> > >> https://github.com/zrlio/crail/blob/master/LICENSE > >> > >> I do not believe we have a formal process for accepting / declining. > The > >> key considerations, as Olaf mentioned, are licensing and willingness to > >> maintain the packages. We do have a policy that any packages which > break > >> and do not have maintainers (or the maintainers do not respond) will be > >> removed in the following release. > >> > >> I would suggest creating some JIRA tickets around the integration of > Crail > >> and your plans for each step. e.g., creating packages (RPMs, DEBs), > >> deploying via our Puppet scripts, and some support in our integration > >> testing frameworks. This would allow other community members to comment > >> and provide feedback on the technical aspects. > >> > >> > >> On Tue, Jul 4, 2017 at 6:18 AM, patrick stuedi > wrote: > >> > >> > Olaf, > >> > > >> > many thanks for the quick reply! > >> > > >> > let me try to clarify -- the wording "donating the Crail code" was > >> > misleading...what we propose is actually to contribute the packaging, > >> > integration, automation and deployment of Crail to Bigtop, and of > course > >> > the continuous maintenance of it. Being an optional addon to the > regular > >> > BigTop stack would be absolutely fine. Question, do you guys have a > >> > democratic process where you accept/decline the integration of new > > projects > >> > like Crail? What next steps do you recommend? > >> > > >> > For us, integrating Crail into Bigtop would be interesting as it makes > > the > >> > deployment and management of Crail easier. Currently deploying and > >> > configuring Crail on a system/cluster is a rather complex and task. > >> > > >> > Regarding the RDMA integration, yes Crail at its core is based on > native > >> > verbs, but it is written in Java and uses DiSNI ( > >> > https://github.com/zrlio/disni) which essentially is a one-to-one > > mapping > >> > of RDMA verbs into Java implemented on top of native verbs. But I do > > want > >> > to clarify that Crail is not just about RDMA, it's a fast storage > > platform > >> > designed for user-level APIs (RDMA, DPDK, SPDK, etc.) and high speed > >> > networking and storage hardware (100GbE, NVMe, NVMe-oF, etc.). > >> > > >> > Thank you for mentioning Roman in the context of Apache incubator, > we'll > >> > reach out to him. Applying for Apache incubator is one of our
Re: Crail as part of BigTop
This is a good example [1] of what we were referring to. Thanks, Cos [1] https://issues.apache.org/jira/browse/BIGTOP-2283 -- Take care, Konstantin (Cos) Boudnik 2CAC 8312 4870 D885 8616 6115 220F 6980 1F27 E622 Disclaimer: Opinions expressed in this email are those of the author, and do not necessarily represent the views of any company the author might be affiliated with at the moment of writing. On Fri, Jul 7, 2017 at 1:28 PM, patrick stuedi wrote: > Cos, RJ, thanks for the response/advice..what I hear is that opening a JIRA > ticket on the crail integration would be a good starting point..we'll do so > and take it from it there.. > > On Jul 5, 2017 8:06 PM, "Konstantin Boudnik" wrote: > > Thanks for bringing this to the table, Patrick! > > A couple of comments: > - adding to what Olaf said earlier: there are two ways of donating a > codebase >to ASF. One is to go through the Incubator, learn all the ropes and > become >a TLP. Another one, is by joining an existing TLP as a subproject. > There's >a number of steps being involved wrt IP clearance, license checks, SGA, >CCLA, yada-yada. I am not going into more details like this, because > you're >not considering this path anyway ;) > - from a quick glance at the project, looks like it fits the general bucket >of data fabric platforms. Something like Apache Ignite comes to mind, > while >Aluxio, as a simple caching solution, only has partial target >functionality. While there's no limitation on having multiple components >with overlapping functionality in a given Bigtop stack (after all, we > have >HDFS and QFS), it's an aspect worthy of some consideration. > - and to the RJ's point: with source code licensing question out of the > way, >it'd desirable to make sure that Crail doesn't re-distribute anything > under >conflicting licences. The ones in question are of GPL family and some > more >as you can see in [1]. While this isn't much of a concern for Bigtop, you >might benefit from doing this for the project itself. > > Regards, > Cos > > On Wed, Jul 05, 2017 at 10:25AM, RJ Nowling wrote: >> I believe we already package several other open source projects that are >> not Apache projects. That should not be a hurdle. We do have > restrictions >> on licenses, but it seems that Crail is licensed under the Apache License: >> >> https://github.com/zrlio/crail/blob/master/LICENSE >> >> I do not believe we have a formal process for accepting / declining. The >> key considerations, as Olaf mentioned, are licensing and willingness to >> maintain the packages. We do have a policy that any packages which break >> and do not have maintainers (or the maintainers do not respond) will be >> removed in the following release. >> >> I would suggest creating some JIRA tickets around the integration of Crail >> and your plans for each step. e.g., creating packages (RPMs, DEBs), >> deploying via our Puppet scripts, and some support in our integration >> testing frameworks. This would allow other community members to comment >> and provide feedback on the technical aspects. >> >> >> On Tue, Jul 4, 2017 at 6:18 AM, patrick stuedi wrote: >> >> > Olaf, >> > >> > many thanks for the quick reply! >> > >> > let me try to clarify -- the wording "donating the Crail code" was >> > misleading...what we propose is actually to contribute the packaging, >> > integration, automation and deployment of Crail to Bigtop, and of course >> > the continuous maintenance of it. Being an optional addon to the regular >> > BigTop stack would be absolutely fine. Question, do you guys have a >> > democratic process where you accept/decline the integration of new > projects >> > like Crail? What next steps do you recommend? >> > >> > For us, integrating Crail into Bigtop would be interesting as it makes > the >> > deployment and management of Crail easier. Currently deploying and >> > configuring Crail on a system/cluster is a rather complex and task. >> > >> > Regarding the RDMA integration, yes Crail at its core is based on native >> > verbs, but it is written in Java and uses DiSNI ( >> > https://github.com/zrlio/disni) which essentially is a one-to-one > mapping >> > of RDMA verbs into Java implemented on top of native verbs. But I do > want >> > to clarify that Crail is not just about RDMA, it's a fast storage > platform >> > designed for user-level APIs (RDMA, DPDK, SPDK, etc.) and high speed >> > networking and storage hardware (100GbE, NVMe, NVMe-oF, etc.). >> > >> > Thank you for mentioning Roman in the context of Apache incubator, we'll >> > reach out to him. Applying for Apache incubator is one of our absolute >> > goals. What is unclear to me is whether an integration of Crail into > BigTop >> > is gated by Crail being Apache incubator already, or whether those > efforts >> > can go on in parallel, can you say something about this? >> > >> > >> > On Mon, Jul 3, 2017 at 10:22 PM, Olaf Flebbe wrote: >> > > >> > >
Re: Crail as part of BigTop
Cos, RJ, thanks for the response/advice..what I hear is that opening a JIRA ticket on the crail integration would be a good starting point..we'll do so and take it from it there.. On Jul 5, 2017 8:06 PM, "Konstantin Boudnik" wrote: Thanks for bringing this to the table, Patrick! A couple of comments: - adding to what Olaf said earlier: there are two ways of donating a codebase to ASF. One is to go through the Incubator, learn all the ropes and become a TLP. Another one, is by joining an existing TLP as a subproject. There's a number of steps being involved wrt IP clearance, license checks, SGA, CCLA, yada-yada. I am not going into more details like this, because you're not considering this path anyway ;) - from a quick glance at the project, looks like it fits the general bucket of data fabric platforms. Something like Apache Ignite comes to mind, while Aluxio, as a simple caching solution, only has partial target functionality. While there's no limitation on having multiple components with overlapping functionality in a given Bigtop stack (after all, we have HDFS and QFS), it's an aspect worthy of some consideration. - and to the RJ's point: with source code licensing question out of the way, it'd desirable to make sure that Crail doesn't re-distribute anything under conflicting licences. The ones in question are of GPL family and some more as you can see in [1]. While this isn't much of a concern for Bigtop, you might benefit from doing this for the project itself. Regards, Cos On Wed, Jul 05, 2017 at 10:25AM, RJ Nowling wrote: > I believe we already package several other open source projects that are > not Apache projects. That should not be a hurdle. We do have restrictions > on licenses, but it seems that Crail is licensed under the Apache License: > > https://github.com/zrlio/crail/blob/master/LICENSE > > I do not believe we have a formal process for accepting / declining. The > key considerations, as Olaf mentioned, are licensing and willingness to > maintain the packages. We do have a policy that any packages which break > and do not have maintainers (or the maintainers do not respond) will be > removed in the following release. > > I would suggest creating some JIRA tickets around the integration of Crail > and your plans for each step. e.g., creating packages (RPMs, DEBs), > deploying via our Puppet scripts, and some support in our integration > testing frameworks. This would allow other community members to comment > and provide feedback on the technical aspects. > > > On Tue, Jul 4, 2017 at 6:18 AM, patrick stuedi wrote: > > > Olaf, > > > > many thanks for the quick reply! > > > > let me try to clarify -- the wording "donating the Crail code" was > > misleading...what we propose is actually to contribute the packaging, > > integration, automation and deployment of Crail to Bigtop, and of course > > the continuous maintenance of it. Being an optional addon to the regular > > BigTop stack would be absolutely fine. Question, do you guys have a > > democratic process where you accept/decline the integration of new projects > > like Crail? What next steps do you recommend? > > > > For us, integrating Crail into Bigtop would be interesting as it makes the > > deployment and management of Crail easier. Currently deploying and > > configuring Crail on a system/cluster is a rather complex and task. > > > > Regarding the RDMA integration, yes Crail at its core is based on native > > verbs, but it is written in Java and uses DiSNI ( > > https://github.com/zrlio/disni) which essentially is a one-to-one mapping > > of RDMA verbs into Java implemented on top of native verbs. But I do want > > to clarify that Crail is not just about RDMA, it's a fast storage platform > > designed for user-level APIs (RDMA, DPDK, SPDK, etc.) and high speed > > networking and storage hardware (100GbE, NVMe, NVMe-oF, etc.). > > > > Thank you for mentioning Roman in the context of Apache incubator, we'll > > reach out to him. Applying for Apache incubator is one of our absolute > > goals. What is unclear to me is whether an integration of Crail into BigTop > > is gated by Crail being Apache incubator already, or whether those efforts > > can go on in parallel, can you say something about this? > > > > > > On Mon, Jul 3, 2017 at 10:22 PM, Olaf Flebbe wrote: > > > > > > Well, let me speak up as an individual: > > > > > > I thank you for making this substantial project available under Apache > > License! > > > > > > -- In my former job I was part of an HPC team and have some cursory > > ideas of RDMA mostly with MPI or for cluster filesystems. At first glance > > it looks to me like the first fully open sourced RDMA support for Big Data. > > I am assuming the library is based on the standard RDMA "Verbs" API and not > > on proprietary extensions of the IBM JDK like efforts before. -- > > > > > > The big question is: what do you expect from Apache Bigtop / Apache > > Softw
Re: Crail as part of BigTop
Thanks for bringing this to the table, Patrick! A couple of comments: - adding to what Olaf said earlier: there are two ways of donating a codebase to ASF. One is to go through the Incubator, learn all the ropes and become a TLP. Another one, is by joining an existing TLP as a subproject. There's a number of steps being involved wrt IP clearance, license checks, SGA, CCLA, yada-yada. I am not going into more details like this, because you're not considering this path anyway ;) - from a quick glance at the project, looks like it fits the general bucket of data fabric platforms. Something like Apache Ignite comes to mind, while Aluxio, as a simple caching solution, only has partial target functionality. While there's no limitation on having multiple components with overlapping functionality in a given Bigtop stack (after all, we have HDFS and QFS), it's an aspect worthy of some consideration. - and to the RJ's point: with source code licensing question out of the way, it'd desirable to make sure that Crail doesn't re-distribute anything under conflicting licences. The ones in question are of GPL family and some more as you can see in [1]. While this isn't much of a concern for Bigtop, you might benefit from doing this for the project itself. Regards, Cos On Wed, Jul 05, 2017 at 10:25AM, RJ Nowling wrote: > I believe we already package several other open source projects that are > not Apache projects. That should not be a hurdle. We do have restrictions > on licenses, but it seems that Crail is licensed under the Apache License: > > https://github.com/zrlio/crail/blob/master/LICENSE > > I do not believe we have a formal process for accepting / declining. The > key considerations, as Olaf mentioned, are licensing and willingness to > maintain the packages. We do have a policy that any packages which break > and do not have maintainers (or the maintainers do not respond) will be > removed in the following release. > > I would suggest creating some JIRA tickets around the integration of Crail > and your plans for each step. e.g., creating packages (RPMs, DEBs), > deploying via our Puppet scripts, and some support in our integration > testing frameworks. This would allow other community members to comment > and provide feedback on the technical aspects. > > > On Tue, Jul 4, 2017 at 6:18 AM, patrick stuedi wrote: > > > Olaf, > > > > many thanks for the quick reply! > > > > let me try to clarify -- the wording "donating the Crail code" was > > misleading...what we propose is actually to contribute the packaging, > > integration, automation and deployment of Crail to Bigtop, and of course > > the continuous maintenance of it. Being an optional addon to the regular > > BigTop stack would be absolutely fine. Question, do you guys have a > > democratic process where you accept/decline the integration of new projects > > like Crail? What next steps do you recommend? > > > > For us, integrating Crail into Bigtop would be interesting as it makes the > > deployment and management of Crail easier. Currently deploying and > > configuring Crail on a system/cluster is a rather complex and task. > > > > Regarding the RDMA integration, yes Crail at its core is based on native > > verbs, but it is written in Java and uses DiSNI ( > > https://github.com/zrlio/disni) which essentially is a one-to-one mapping > > of RDMA verbs into Java implemented on top of native verbs. But I do want > > to clarify that Crail is not just about RDMA, it's a fast storage platform > > designed for user-level APIs (RDMA, DPDK, SPDK, etc.) and high speed > > networking and storage hardware (100GbE, NVMe, NVMe-oF, etc.). > > > > Thank you for mentioning Roman in the context of Apache incubator, we'll > > reach out to him. Applying for Apache incubator is one of our absolute > > goals. What is unclear to me is whether an integration of Crail into BigTop > > is gated by Crail being Apache incubator already, or whether those efforts > > can go on in parallel, can you say something about this? > > > > > > On Mon, Jul 3, 2017 at 10:22 PM, Olaf Flebbe wrote: > > > > > > Well, let me speak up as an individual: > > > > > > I thank you for making this substantial project available under Apache > > License! > > > > > > -- In my former job I was part of an HPC team and have some cursory > > ideas of RDMA mostly with MPI or for cluster filesystems. At first glance > > it looks to me like the first fully open sourced RDMA support for Big Data. > > I am assuming the library is based on the standard RDMA "Verbs" API and not > > on proprietary extensions of the IBM JDK like efforts before. -- > > > > > > The big question is: what do you expect from Apache Bigtop / Apache > > Software Foundation? > > > > > > If you like to contribute all your code to the Apache Foundation in order > > to have a platform for you and other contributors to work on the code, then > > I would recommend you to check out Apac
Re: Crail as part of BigTop
I believe we already package several other open source projects that are not Apache projects. That should not be a hurdle. We do have restrictions on licenses, but it seems that Crail is licensed under the Apache License: https://github.com/zrlio/crail/blob/master/LICENSE I do not believe we have a formal process for accepting / declining. The key considerations, as Olaf mentioned, are licensing and willingness to maintain the packages. We do have a policy that any packages which break and do not have maintainers (or the maintainers do not respond) will be removed in the following release. I would suggest creating some JIRA tickets around the integration of Crail and your plans for each step. e.g., creating packages (RPMs, DEBs), deploying via our Puppet scripts, and some support in our integration testing frameworks. This would allow other community members to comment and provide feedback on the technical aspects. On Tue, Jul 4, 2017 at 6:18 AM, patrick stuedi wrote: > Olaf, > > many thanks for the quick reply! > > let me try to clarify -- the wording "donating the Crail code" was > misleading...what we propose is actually to contribute the packaging, > integration, automation and deployment of Crail to Bigtop, and of course > the continuous maintenance of it. Being an optional addon to the regular > BigTop stack would be absolutely fine. Question, do you guys have a > democratic process where you accept/decline the integration of new projects > like Crail? What next steps do you recommend? > > For us, integrating Crail into Bigtop would be interesting as it makes the > deployment and management of Crail easier. Currently deploying and > configuring Crail on a system/cluster is a rather complex and task. > > Regarding the RDMA integration, yes Crail at its core is based on native > verbs, but it is written in Java and uses DiSNI ( > https://github.com/zrlio/disni) which essentially is a one-to-one mapping > of RDMA verbs into Java implemented on top of native verbs. But I do want > to clarify that Crail is not just about RDMA, it's a fast storage platform > designed for user-level APIs (RDMA, DPDK, SPDK, etc.) and high speed > networking and storage hardware (100GbE, NVMe, NVMe-oF, etc.). > > Thank you for mentioning Roman in the context of Apache incubator, we'll > reach out to him. Applying for Apache incubator is one of our absolute > goals. What is unclear to me is whether an integration of Crail into BigTop > is gated by Crail being Apache incubator already, or whether those efforts > can go on in parallel, can you say something about this? > > > On Mon, Jul 3, 2017 at 10:22 PM, Olaf Flebbe wrote: > > > > Well, let me speak up as an individual: > > > > I thank you for making this substantial project available under Apache > License! > > > > -- In my former job I was part of an HPC team and have some cursory > ideas of RDMA mostly with MPI or for cluster filesystems. At first glance > it looks to me like the first fully open sourced RDMA support for Big Data. > I am assuming the library is based on the standard RDMA "Verbs" API and not > on proprietary extensions of the IBM JDK like efforts before. -- > > > > The big question is: what do you expect from Apache Bigtop / Apache > Software Foundation? > > > > If you like to contribute all your code to the Apache Foundation in order > to have a platform for you and other contributors to work on the code, then > I would recommend you to check out Apache Incubator. Roman can give you > more information, since he is PMC of Apache Incubator as well. > > > > Contributing all the things to Bigtop: I would say: Thank you, we are > honoured, but it does not fit. We are an integration and automation > project. > > > > If you like you to contribute packaging, integration, automation of > configuration and deployment of your code (All these seems to be missing, > at first glance) to Bigtop, feel free: We love to see innovative usecases > built with our stack. We surely can only accept this contribution as an > optional addon to our regular stack. And we will ask your group to maintain > it in future (see MAINTAINERS at our toplevel directory), since the code > and needed (virtual/real) infrastructure is not middle of the road. There > are a couple of contributions to Bigtop where the developers became > committers and are now actively working within Bigtop. > > > > Be please aware that this is my personal opinion, there may be other > views. > > > > Olaf > > > > >
Re: Crail as part of BigTop
Olaf, many thanks for the quick reply! let me try to clarify -- the wording "donating the Crail code" was misleading...what we propose is actually to contribute the packaging, integration, automation and deployment of Crail to Bigtop, and of course the continuous maintenance of it. Being an optional addon to the regular BigTop stack would be absolutely fine. Question, do you guys have a democratic process where you accept/decline the integration of new projects like Crail? What next steps do you recommend? For us, integrating Crail into Bigtop would be interesting as it makes the deployment and management of Crail easier. Currently deploying and configuring Crail on a system/cluster is a rather complex and task. Regarding the RDMA integration, yes Crail at its core is based on native verbs, but it is written in Java and uses DiSNI ( https://github.com/zrlio/disni) which essentially is a one-to-one mapping of RDMA verbs into Java implemented on top of native verbs. But I do want to clarify that Crail is not just about RDMA, it's a fast storage platform designed for user-level APIs (RDMA, DPDK, SPDK, etc.) and high speed networking and storage hardware (100GbE, NVMe, NVMe-oF, etc.). Thank you for mentioning Roman in the context of Apache incubator, we'll reach out to him. Applying for Apache incubator is one of our absolute goals. What is unclear to me is whether an integration of Crail into BigTop is gated by Crail being Apache incubator already, or whether those efforts can go on in parallel, can you say something about this? On Mon, Jul 3, 2017 at 10:22 PM, Olaf Flebbe wrote: > > Well, let me speak up as an individual: > > I thank you for making this substantial project available under Apache License! > > -- In my former job I was part of an HPC team and have some cursory ideas of RDMA mostly with MPI or for cluster filesystems. At first glance it looks to me like the first fully open sourced RDMA support for Big Data. I am assuming the library is based on the standard RDMA "Verbs" API and not on proprietary extensions of the IBM JDK like efforts before. -- > > The big question is: what do you expect from Apache Bigtop / Apache Software Foundation? > > If you like to contribute all your code to the Apache Foundation in order to have a platform for you and other contributors to work on the code, then I would recommend you to check out Apache Incubator. Roman can give you more information, since he is PMC of Apache Incubator as well. > > Contributing all the things to Bigtop: I would say: Thank you, we are honoured, but it does not fit. We are an integration and automation project. > > If you like you to contribute packaging, integration, automation of configuration and deployment of your code (All these seems to be missing, at first glance) to Bigtop, feel free: We love to see innovative usecases built with our stack. We surely can only accept this contribution as an optional addon to our regular stack. And we will ask your group to maintain it in future (see MAINTAINERS at our toplevel directory), since the code and needed (virtual/real) infrastructure is not middle of the road. There are a couple of contributions to Bigtop where the developers became committers and are now actively working within Bigtop. > > Be please aware that this is my personal opinion, there may be other views. > > Olaf > >
Re: Crail as part of BigTop
Well, let me speak up as an individual: I thank you for making this substantial project available under Apache License! -- In my former job I was part of an HPC team and have some cursory ideas of RDMA mostly with MPI or for cluster filesystems. At first glance it looks to me like the first fully open sourced RDMA support for Big Data. I am assuming the library is based on the standard RDMA "Verbs" API and not on proprietary extensions of the IBM JDK like efforts before. -- The big question is: what do you expect from Apache Bigtop / Apache Software Foundation? If you like to contribute all your code to the Apache Foundation in order to have a platform for you and other contributors to work on the code, then I would recommend you to check out Apache Incubator. Roman can give you more information, since he is PMC of Apache Incubator as well. Contributing all the things to Bigtop: I would say: Thank you, we are honoured, but it does not fit. We are an integration and automation project. If you like you to contribute packaging, integration, automation of configuration and deployment of your code (All these seems to be missing, at first glance) to Bigtop, feel free: We love to see innovative usecases built with our stack. We surely can only accept this contribution as an optional addon to our regular stack. And we will ask your group to maintain it in future (see MAINTAINERS at our toplevel directory), since the code and needed (virtual/real) infrastructure is not middle of the road. There are a couple of contributions to Bigtop where the developers became committers and are now actively working within Bigtop. Be please aware that this is my personal opinion, there may be other views. Olaf signature.asc Description: Message signed with OpenPGP
