Re: [VOTE] Accept Airflow into the Incubator
+1 non-binding :) On Thu, Mar 24, 2016 at 8:00 PM, Siddharth Anand wrote: > Following the discussion earlier: > https://s.apache.org/AirflowDiscussion > > I would like to call a VOTE for accepting Airflow as a new incubator > project. > > The proposal is available at: > https://wiki.apache.org/incubator/AirflowProposal > > The proposal is also included at the bottom of this email. > > Vote is open until at least Tues, 29 March 2016, 23:59:00 PDT > [ ] +1 accept Airflow into the Apache Incubator > [ ] ±0 > [ ] -1 because... > > +1 (non-binding) > > Thanks, > -s (Sid) > > > == Abstract == > > Airflow is a workflow automation and scheduling system that can be > used to author and manage data pipelines. > > == Proposal == > > Airflow provides a system for authoring and managing workflows a.k.a. > data pipelines a.k.a. DAGs (Directed Acyclic Graphs). The developer > authors DAGs in Python using an Airflow-provided framework. He/She > then executes the DAG using Airflow’s scheduler or registers the DAG > for event-based execution. A web-based UI provides the developer with > a range of options for managing and viewing his/her data pipelines. > Background > > Airflow was developed at Airbnb to enable easier authorship and > management of DAGs than were possible with existing solutions such as > Oozie and Azkaban. For starters, both Oozie and Azkaban rely on one or > more XML or property files to be bundled together to define a > workflow. This separation of code and config can present a challenge > to understanding the DAG - in Azkaban, a DAG’s structure is reflected > by its file system tree and one can find himself/herself traversing > the file system when inspecting or changing the structure of the DAG. > Airflow workflows, on the other hand, are simply and elegantly defined > in Python code, often a single file. Airflow merges the powerful > Web-based management aspects of projects like Azkaban and Oozie with > the simplicity and elegance of defining workflows in Python. Airflow, > less than a year old in terms of its Open Source launch, is currently > used in production environments in more than 30 companies and boasts > an active contributor list of more than 100 developers, the vast > majority of which (>95%) are outside of Airbnb. > > We would like to share it with the ASF and begin developing a > community of developers and users within Apache. > > == Rationale == > > Many organizations (>30) already benefit from running Airflow to > manage data pipelines. Our 100+ contributors continue to provide > integrations with 3rd party systems through the implementation of new > hooks and operators, both of which are used in defining the tasks that > compose workflows. > > == Current Status == > > === Meritocracy === > > Our intent with this incubator proposal is to start building a diverse > developer community around Airflow following the Apache meritocracy > model. Since Airflow was open-sourced in mid-2015, we have had fast > adoption and contributions by multiple organizations the world over. > We plan to continue to support new contributors and we will work to > actively promote those who contribute significantly to the project to > committers. > > === Community === > > Airflow is currently being used in over 30 companies. We hope to > extend our contributor base significantly and invite all those who are > interested in building large-scale distributed systems to participate. > > === Core Developers === > > Airflow is currently being developed by four engineers: Maxime > Beauchemin, Siddharth Anand, Bolke de Bruin, and Chris Riccomini. > Chris is a member of the Apache Samza PMC and a contributor to various > Apache projects, including Apache Kafka and Apache YARN. Maxime, > Siddharth, and Bolke have contributed to Airflow. > > === Alignment === > The ASF is the natural choice to host the Airflow project as its goal > of encouraging community-driven open-source projects fits with our > vision for Airflow. > > == Known Risks == > > === Orphaned Products === > > The core developers plan to work part time on the project. There is > very little risk of Airflow being abandoned as all of our companies > rely on it. > > === Inexperience with Open Source === > > All of the core developers have experience with open source > development. Chris is a member of the Apache Samza PMC and a > contributor to various Apache projects, including Apache Kafka and > Apache YARN. Bolke is contributor on multiple open source projects and > a few Apache projects as well, including Apache Hive, Apache Hadoop, > and Apache Ranger. > > === Homogeneous Developers === > > The current core developers are all from different companies. Our > community of 100 contributors hail from over 30 different companies > from across the world. > > === Reliance on Salaried Developers === > > Currently, the only developer paid to work on this project is Maxime. > > === Relationships with Other Apache Products === > > Airflow is deeply integrate
Re: [VOTE] Accept Airflow into the Incubator
+1 (binding) -jakob > On Mar 24, 2016, at 8:20 PM, Chris Riccomini wrote: > > +1 non-binding :) > > On Thu, Mar 24, 2016 at 8:00 PM, Siddharth Anand > wrote: > >> Following the discussion earlier: >>https://s.apache.org/AirflowDiscussion >> >> I would like to call a VOTE for accepting Airflow as a new incubator >> project. >> >> The proposal is available at: >> https://wiki.apache.org/incubator/AirflowProposal >> >> The proposal is also included at the bottom of this email. >> >> Vote is open until at least Tues, 29 March 2016, 23:59:00 PDT >> [ ] +1 accept Airflow into the Apache Incubator >> [ ] ±0 >> [ ] -1 because... >> >> +1 (non-binding) >> >> Thanks, >> -s (Sid) >> >> >> == Abstract == >> >> Airflow is a workflow automation and scheduling system that can be >> used to author and manage data pipelines. >> >> == Proposal == >> >> Airflow provides a system for authoring and managing workflows a.k.a. >> data pipelines a.k.a. DAGs (Directed Acyclic Graphs). The developer >> authors DAGs in Python using an Airflow-provided framework. He/She >> then executes the DAG using Airflow’s scheduler or registers the DAG >> for event-based execution. A web-based UI provides the developer with >> a range of options for managing and viewing his/her data pipelines. >> Background >> >> Airflow was developed at Airbnb to enable easier authorship and >> management of DAGs than were possible with existing solutions such as >> Oozie and Azkaban. For starters, both Oozie and Azkaban rely on one or >> more XML or property files to be bundled together to define a >> workflow. This separation of code and config can present a challenge >> to understanding the DAG - in Azkaban, a DAG’s structure is reflected >> by its file system tree and one can find himself/herself traversing >> the file system when inspecting or changing the structure of the DAG. >> Airflow workflows, on the other hand, are simply and elegantly defined >> in Python code, often a single file. Airflow merges the powerful >> Web-based management aspects of projects like Azkaban and Oozie with >> the simplicity and elegance of defining workflows in Python. Airflow, >> less than a year old in terms of its Open Source launch, is currently >> used in production environments in more than 30 companies and boasts >> an active contributor list of more than 100 developers, the vast >> majority of which (>95%) are outside of Airbnb. >> >> We would like to share it with the ASF and begin developing a >> community of developers and users within Apache. >> >> == Rationale == >> >> Many organizations (>30) already benefit from running Airflow to >> manage data pipelines. Our 100+ contributors continue to provide >> integrations with 3rd party systems through the implementation of new >> hooks and operators, both of which are used in defining the tasks that >> compose workflows. >> >> == Current Status == >> >> === Meritocracy === >> >> Our intent with this incubator proposal is to start building a diverse >> developer community around Airflow following the Apache meritocracy >> model. Since Airflow was open-sourced in mid-2015, we have had fast >> adoption and contributions by multiple organizations the world over. >> We plan to continue to support new contributors and we will work to >> actively promote those who contribute significantly to the project to >> committers. >> >> === Community === >> >> Airflow is currently being used in over 30 companies. We hope to >> extend our contributor base significantly and invite all those who are >> interested in building large-scale distributed systems to participate. >> >> === Core Developers === >> >> Airflow is currently being developed by four engineers: Maxime >> Beauchemin, Siddharth Anand, Bolke de Bruin, and Chris Riccomini. >> Chris is a member of the Apache Samza PMC and a contributor to various >> Apache projects, including Apache Kafka and Apache YARN. Maxime, >> Siddharth, and Bolke have contributed to Airflow. >> >> === Alignment === >> The ASF is the natural choice to host the Airflow project as its goal >> of encouraging community-driven open-source projects fits with our >> vision for Airflow. >> >> == Known Risks == >> >> === Orphaned Products === >> >> The core developers plan to work part time on the project. There is >> very little risk of Airflow being abandoned as all of our companies >> rely on it. >> >> === Inexperience with Open Source === >> >> All of the core developers have experience with open source >> development. Chris is a member of the Apache Samza PMC and a >> contributor to various Apache projects, including Apache Kafka and >> Apache YARN. Bolke is contributor on multiple open source projects and >> a few Apache projects as well, including Apache Hive, Apache Hadoop, >> and Apache Ranger. >> >> === Homogeneous Developers === >> >> The current core developers are all from different companies. Our >> community of 100 contributors hail from over 30 different
Re: [VOTE] Accept Airflow into the Incubator
+ 1 (binding) All the best, Venkatesh On Thu, Mar 24, 2016 at 10:06 PM Jakob Homan wrote: > +1 (binding) > > -jakob > > > On Mar 24, 2016, at 8:20 PM, Chris Riccomini > wrote: > > > > +1 non-binding :) > > > > On Thu, Mar 24, 2016 at 8:00 PM, Siddharth Anand > > > wrote: > > > >> Following the discussion earlier: > >>https://s.apache.org/AirflowDiscussion > >> > >> I would like to call a VOTE for accepting Airflow as a new incubator > >> project. > >> > >> The proposal is available at: > >> https://wiki.apache.org/incubator/AirflowProposal > >> > >> The proposal is also included at the bottom of this email. > >> > >> Vote is open until at least Tues, 29 March 2016, 23:59:00 PDT > >> [ ] +1 accept Airflow into the Apache Incubator > >> [ ] ±0 > >> [ ] -1 because... > >> > >> +1 (non-binding) > >> > >> Thanks, > >> -s (Sid) > >> > >> > >> == Abstract == > >> > >> Airflow is a workflow automation and scheduling system that can be > >> used to author and manage data pipelines. > >> > >> == Proposal == > >> > >> Airflow provides a system for authoring and managing workflows a.k.a. > >> data pipelines a.k.a. DAGs (Directed Acyclic Graphs). The developer > >> authors DAGs in Python using an Airflow-provided framework. He/She > >> then executes the DAG using Airflow’s scheduler or registers the DAG > >> for event-based execution. A web-based UI provides the developer with > >> a range of options for managing and viewing his/her data pipelines. > >> Background > >> > >> Airflow was developed at Airbnb to enable easier authorship and > >> management of DAGs than were possible with existing solutions such as > >> Oozie and Azkaban. For starters, both Oozie and Azkaban rely on one or > >> more XML or property files to be bundled together to define a > >> workflow. This separation of code and config can present a challenge > >> to understanding the DAG - in Azkaban, a DAG’s structure is reflected > >> by its file system tree and one can find himself/herself traversing > >> the file system when inspecting or changing the structure of the DAG. > >> Airflow workflows, on the other hand, are simply and elegantly defined > >> in Python code, often a single file. Airflow merges the powerful > >> Web-based management aspects of projects like Azkaban and Oozie with > >> the simplicity and elegance of defining workflows in Python. Airflow, > >> less than a year old in terms of its Open Source launch, is currently > >> used in production environments in more than 30 companies and boasts > >> an active contributor list of more than 100 developers, the vast > >> majority of which (>95%) are outside of Airbnb. > >> > >> We would like to share it with the ASF and begin developing a > >> community of developers and users within Apache. > >> > >> == Rationale == > >> > >> Many organizations (>30) already benefit from running Airflow to > >> manage data pipelines. Our 100+ contributors continue to provide > >> integrations with 3rd party systems through the implementation of new > >> hooks and operators, both of which are used in defining the tasks that > >> compose workflows. > >> > >> == Current Status == > >> > >> === Meritocracy === > >> > >> Our intent with this incubator proposal is to start building a diverse > >> developer community around Airflow following the Apache meritocracy > >> model. Since Airflow was open-sourced in mid-2015, we have had fast > >> adoption and contributions by multiple organizations the world over. > >> We plan to continue to support new contributors and we will work to > >> actively promote those who contribute significantly to the project to > >> committers. > >> > >> === Community === > >> > >> Airflow is currently being used in over 30 companies. We hope to > >> extend our contributor base significantly and invite all those who are > >> interested in building large-scale distributed systems to participate. > >> > >> === Core Developers === > >> > >> Airflow is currently being developed by four engineers: Maxime > >> Beauchemin, Siddharth Anand, Bolke de Bruin, and Chris Riccomini. > >> Chris is a member of the Apache Samza PMC and a contributor to various > >> Apache projects, including Apache Kafka and Apache YARN. Maxime, > >> Siddharth, and Bolke have contributed to Airflow. > >> > >> === Alignment === > >> The ASF is the natural choice to host the Airflow project as its goal > >> of encouraging community-driven open-source projects fits with our > >> vision for Airflow. > >> > >> == Known Risks == > >> > >> === Orphaned Products === > >> > >> The core developers plan to work part time on the project. There is > >> very little risk of Airflow being abandoned as all of our companies > >> rely on it. > >> > >> === Inexperience with Open Source === > >> > >> All of the core developers have experience with open source > >> development. Chris is a member of the Apache Samza PMC and a > >> contributor to various Apache projects, including Apache Kafka and > >> Apache YARN. Bolke is contribut
Re: [VOTE] Accept Airflow into the Incubator
+1 (binding) — HItesh On Mar 24, 2016, at 8:00 PM, Siddharth Anand wrote: > Following the discussion earlier: >https://s.apache.org/AirflowDiscussion > > I would like to call a VOTE for accepting Airflow as a new incubator project. > > The proposal is available at: > https://wiki.apache.org/incubator/AirflowProposal > > The proposal is also included at the bottom of this email. > > Vote is open until at least Tues, 29 March 2016, 23:59:00 PDT > [ ] +1 accept Airflow into the Apache Incubator > [ ] ±0 > [ ] -1 because... > > +1 (non-binding) > > Thanks, > -s (Sid) > > > == Abstract == > > Airflow is a workflow automation and scheduling system that can be > used to author and manage data pipelines. > > == Proposal == > > Airflow provides a system for authoring and managing workflows a.k.a. > data pipelines a.k.a. DAGs (Directed Acyclic Graphs). The developer > authors DAGs in Python using an Airflow-provided framework. He/She > then executes the DAG using Airflow’s scheduler or registers the DAG > for event-based execution. A web-based UI provides the developer with > a range of options for managing and viewing his/her data pipelines. > Background > > Airflow was developed at Airbnb to enable easier authorship and > management of DAGs than were possible with existing solutions such as > Oozie and Azkaban. For starters, both Oozie and Azkaban rely on one or > more XML or property files to be bundled together to define a > workflow. This separation of code and config can present a challenge > to understanding the DAG - in Azkaban, a DAG’s structure is reflected > by its file system tree and one can find himself/herself traversing > the file system when inspecting or changing the structure of the DAG. > Airflow workflows, on the other hand, are simply and elegantly defined > in Python code, often a single file. Airflow merges the powerful > Web-based management aspects of projects like Azkaban and Oozie with > the simplicity and elegance of defining workflows in Python. Airflow, > less than a year old in terms of its Open Source launch, is currently > used in production environments in more than 30 companies and boasts > an active contributor list of more than 100 developers, the vast > majority of which (>95%) are outside of Airbnb. > > We would like to share it with the ASF and begin developing a > community of developers and users within Apache. > > == Rationale == > > Many organizations (>30) already benefit from running Airflow to > manage data pipelines. Our 100+ contributors continue to provide > integrations with 3rd party systems through the implementation of new > hooks and operators, both of which are used in defining the tasks that > compose workflows. > > == Current Status == > > === Meritocracy === > > Our intent with this incubator proposal is to start building a diverse > developer community around Airflow following the Apache meritocracy > model. Since Airflow was open-sourced in mid-2015, we have had fast > adoption and contributions by multiple organizations the world over. > We plan to continue to support new contributors and we will work to > actively promote those who contribute significantly to the project to > committers. > > === Community === > > Airflow is currently being used in over 30 companies. We hope to > extend our contributor base significantly and invite all those who are > interested in building large-scale distributed systems to participate. > > === Core Developers === > > Airflow is currently being developed by four engineers: Maxime > Beauchemin, Siddharth Anand, Bolke de Bruin, and Chris Riccomini. > Chris is a member of the Apache Samza PMC and a contributor to various > Apache projects, including Apache Kafka and Apache YARN. Maxime, > Siddharth, and Bolke have contributed to Airflow. > > === Alignment === > The ASF is the natural choice to host the Airflow project as its goal > of encouraging community-driven open-source projects fits with our > vision for Airflow. > > == Known Risks == > > === Orphaned Products === > > The core developers plan to work part time on the project. There is > very little risk of Airflow being abandoned as all of our companies > rely on it. > > === Inexperience with Open Source === > > All of the core developers have experience with open source > development. Chris is a member of the Apache Samza PMC and a > contributor to various Apache projects, including Apache Kafka and > Apache YARN. Bolke is contributor on multiple open source projects and > a few Apache projects as well, including Apache Hive, Apache Hadoop, > and Apache Ranger. > > === Homogeneous Developers === > > The current core developers are all from different companies. Our > community of 100 contributors hail from over 30 different companies > from across the world. > > === Reliance on Salaried Developers === > > Currently, the only developer paid to work on this project is Maxime. > > === Relationships with Other Apache Products ===
Re: [VOTE] Accept Airflow into the Incubator
+1 (non-binding) Max On Thu, Mar 24, 2016 at 11:07 PM, Hitesh Shah wrote: > +1 (binding) > > — HItesh > > On Mar 24, 2016, at 8:00 PM, Siddharth Anand > wrote: > > > Following the discussion earlier: > >https://s.apache.org/AirflowDiscussion > > > > I would like to call a VOTE for accepting Airflow as a new incubator > project. > > > > The proposal is available at: > https://wiki.apache.org/incubator/AirflowProposal > > > > The proposal is also included at the bottom of this email. > > > > Vote is open until at least Tues, 29 March 2016, 23:59:00 PDT > > [ ] +1 accept Airflow into the Apache Incubator > > [ ] ±0 > > [ ] -1 because... > > > > +1 (non-binding) > > > > Thanks, > > -s (Sid) > > > > > > == Abstract == > > > > Airflow is a workflow automation and scheduling system that can be > > used to author and manage data pipelines. > > > > == Proposal == > > > > Airflow provides a system for authoring and managing workflows a.k.a. > > data pipelines a.k.a. DAGs (Directed Acyclic Graphs). The developer > > authors DAGs in Python using an Airflow-provided framework. He/She > > then executes the DAG using Airflow’s scheduler or registers the DAG > > for event-based execution. A web-based UI provides the developer with > > a range of options for managing and viewing his/her data pipelines. > > Background > > > > Airflow was developed at Airbnb to enable easier authorship and > > management of DAGs than were possible with existing solutions such as > > Oozie and Azkaban. For starters, both Oozie and Azkaban rely on one or > > more XML or property files to be bundled together to define a > > workflow. This separation of code and config can present a challenge > > to understanding the DAG - in Azkaban, a DAG’s structure is reflected > > by its file system tree and one can find himself/herself traversing > > the file system when inspecting or changing the structure of the DAG. > > Airflow workflows, on the other hand, are simply and elegantly defined > > in Python code, often a single file. Airflow merges the powerful > > Web-based management aspects of projects like Azkaban and Oozie with > > the simplicity and elegance of defining workflows in Python. Airflow, > > less than a year old in terms of its Open Source launch, is currently > > used in production environments in more than 30 companies and boasts > > an active contributor list of more than 100 developers, the vast > > majority of which (>95%) are outside of Airbnb. > > > > We would like to share it with the ASF and begin developing a > > community of developers and users within Apache. > > > > == Rationale == > > > > Many organizations (>30) already benefit from running Airflow to > > manage data pipelines. Our 100+ contributors continue to provide > > integrations with 3rd party systems through the implementation of new > > hooks and operators, both of which are used in defining the tasks that > > compose workflows. > > > > == Current Status == > > > > === Meritocracy === > > > > Our intent with this incubator proposal is to start building a diverse > > developer community around Airflow following the Apache meritocracy > > model. Since Airflow was open-sourced in mid-2015, we have had fast > > adoption and contributions by multiple organizations the world over. > > We plan to continue to support new contributors and we will work to > > actively promote those who contribute significantly to the project to > > committers. > > > > === Community === > > > > Airflow is currently being used in over 30 companies. We hope to > > extend our contributor base significantly and invite all those who are > > interested in building large-scale distributed systems to participate. > > > > === Core Developers === > > > > Airflow is currently being developed by four engineers: Maxime > > Beauchemin, Siddharth Anand, Bolke de Bruin, and Chris Riccomini. > > Chris is a member of the Apache Samza PMC and a contributor to various > > Apache projects, including Apache Kafka and Apache YARN. Maxime, > > Siddharth, and Bolke have contributed to Airflow. > > > > === Alignment === > > The ASF is the natural choice to host the Airflow project as its goal > > of encouraging community-driven open-source projects fits with our > > vision for Airflow. > > > > == Known Risks == > > > > === Orphaned Products === > > > > The core developers plan to work part time on the project. There is > > very little risk of Airflow being abandoned as all of our companies > > rely on it. > > > > === Inexperience with Open Source === > > > > All of the core developers have experience with open source > > development. Chris is a member of the Apache Samza PMC and a > > contributor to various Apache projects, including Apache Kafka and > > Apache YARN. Bolke is contributor on multiple open source projects and > > a few Apache projects as well, including Apache Hive, Apache Hadoop, > > and Apache Ranger. > > > > === Homogeneous Developers === > > > > The current core developers are all from diffe
Re: [VOTE] Accept Airflow into the Incubator
+1 (binding) On Fri, Mar 25, 2016 at 7:32 AM, Maxime Beauchemin < maximebeauche...@gmail.com> wrote: > +1 (non-binding) > > Max > > On Thu, Mar 24, 2016 at 11:07 PM, Hitesh Shah wrote: > > > +1 (binding) > > > > — HItesh > > > > On Mar 24, 2016, at 8:00 PM, Siddharth Anand > > wrote: > > > > > Following the discussion earlier: > > >https://s.apache.org/AirflowDiscussion > > > > > > I would like to call a VOTE for accepting Airflow as a new incubator > > project. > > > > > > The proposal is available at: > > https://wiki.apache.org/incubator/AirflowProposal > > > > > > The proposal is also included at the bottom of this email. > > > > > > Vote is open until at least Tues, 29 March 2016, 23:59:00 PDT > > > [ ] +1 accept Airflow into the Apache Incubator > > > [ ] ±0 > > > [ ] -1 because... > > > > > > +1 (non-binding) > > > > > > Thanks, > > > -s (Sid) > > > > > > > > > == Abstract == > > > > > > Airflow is a workflow automation and scheduling system that can be > > > used to author and manage data pipelines. > > > > > > == Proposal == > > > > > > Airflow provides a system for authoring and managing workflows a.k.a. > > > data pipelines a.k.a. DAGs (Directed Acyclic Graphs). The developer > > > authors DAGs in Python using an Airflow-provided framework. He/She > > > then executes the DAG using Airflow’s scheduler or registers the DAG > > > for event-based execution. A web-based UI provides the developer with > > > a range of options for managing and viewing his/her data pipelines. > > > Background > > > > > > Airflow was developed at Airbnb to enable easier authorship and > > > management of DAGs than were possible with existing solutions such as > > > Oozie and Azkaban. For starters, both Oozie and Azkaban rely on one or > > > more XML or property files to be bundled together to define a > > > workflow. This separation of code and config can present a challenge > > > to understanding the DAG - in Azkaban, a DAG’s structure is reflected > > > by its file system tree and one can find himself/herself traversing > > > the file system when inspecting or changing the structure of the DAG. > > > Airflow workflows, on the other hand, are simply and elegantly defined > > > in Python code, often a single file. Airflow merges the powerful > > > Web-based management aspects of projects like Azkaban and Oozie with > > > the simplicity and elegance of defining workflows in Python. Airflow, > > > less than a year old in terms of its Open Source launch, is currently > > > used in production environments in more than 30 companies and boasts > > > an active contributor list of more than 100 developers, the vast > > > majority of which (>95%) are outside of Airbnb. > > > > > > We would like to share it with the ASF and begin developing a > > > community of developers and users within Apache. > > > > > > == Rationale == > > > > > > Many organizations (>30) already benefit from running Airflow to > > > manage data pipelines. Our 100+ contributors continue to provide > > > integrations with 3rd party systems through the implementation of new > > > hooks and operators, both of which are used in defining the tasks that > > > compose workflows. > > > > > > == Current Status == > > > > > > === Meritocracy === > > > > > > Our intent with this incubator proposal is to start building a diverse > > > developer community around Airflow following the Apache meritocracy > > > model. Since Airflow was open-sourced in mid-2015, we have had fast > > > adoption and contributions by multiple organizations the world over. > > > We plan to continue to support new contributors and we will work to > > > actively promote those who contribute significantly to the project to > > > committers. > > > > > > === Community === > > > > > > Airflow is currently being used in over 30 companies. We hope to > > > extend our contributor base significantly and invite all those who are > > > interested in building large-scale distributed systems to participate. > > > > > > === Core Developers === > > > > > > Airflow is currently being developed by four engineers: Maxime > > > Beauchemin, Siddharth Anand, Bolke de Bruin, and Chris Riccomini. > > > Chris is a member of the Apache Samza PMC and a contributor to various > > > Apache projects, including Apache Kafka and Apache YARN. Maxime, > > > Siddharth, and Bolke have contributed to Airflow. > > > > > > === Alignment === > > > The ASF is the natural choice to host the Airflow project as its goal > > > of encouraging community-driven open-source projects fits with our > > > vision for Airflow. > > > > > > == Known Risks == > > > > > > === Orphaned Products === > > > > > > The core developers plan to work part time on the project. There is > > > very little risk of Airflow being abandoned as all of our companies > > > rely on it. > > > > > > === Inexperience with Open Source === > > > > > > All of the core developers have experience with open source > > > development. Chris is a member of the Apache Samza PMC
Re: [VOTE] Accept Airflow into the Incubator
+ 1 (binding). Suresh > On Mar 24, 2016, at 11:00 PM, Siddharth Anand > wrote: > > Following the discussion earlier: >https://s.apache.org/AirflowDiscussion > > I would like to call a VOTE for accepting Airflow as a new incubator project. > > The proposal is available at: > https://wiki.apache.org/incubator/AirflowProposal > > The proposal is also included at the bottom of this email. > > Vote is open until at least Tues, 29 March 2016, 23:59:00 PDT > [ ] +1 accept Airflow into the Apache Incubator > [ ] ±0 > [ ] -1 because... > > +1 (non-binding) > > Thanks, > -s (Sid) > > > == Abstract == > > Airflow is a workflow automation and scheduling system that can be > used to author and manage data pipelines. > > == Proposal == > > Airflow provides a system for authoring and managing workflows a.k.a. > data pipelines a.k.a. DAGs (Directed Acyclic Graphs). The developer > authors DAGs in Python using an Airflow-provided framework. He/She > then executes the DAG using Airflow’s scheduler or registers the DAG > for event-based execution. A web-based UI provides the developer with > a range of options for managing and viewing his/her data pipelines. > Background > > Airflow was developed at Airbnb to enable easier authorship and > management of DAGs than were possible with existing solutions such as > Oozie and Azkaban. For starters, both Oozie and Azkaban rely on one or > more XML or property files to be bundled together to define a > workflow. This separation of code and config can present a challenge > to understanding the DAG - in Azkaban, a DAG’s structure is reflected > by its file system tree and one can find himself/herself traversing > the file system when inspecting or changing the structure of the DAG. > Airflow workflows, on the other hand, are simply and elegantly defined > in Python code, often a single file. Airflow merges the powerful > Web-based management aspects of projects like Azkaban and Oozie with > the simplicity and elegance of defining workflows in Python. Airflow, > less than a year old in terms of its Open Source launch, is currently > used in production environments in more than 30 companies and boasts > an active contributor list of more than 100 developers, the vast > majority of which (>95%) are outside of Airbnb. > > We would like to share it with the ASF and begin developing a > community of developers and users within Apache. > > == Rationale == > > Many organizations (>30) already benefit from running Airflow to > manage data pipelines. Our 100+ contributors continue to provide > integrations with 3rd party systems through the implementation of new > hooks and operators, both of which are used in defining the tasks that > compose workflows. > > == Current Status == > > === Meritocracy === > > Our intent with this incubator proposal is to start building a diverse > developer community around Airflow following the Apache meritocracy > model. Since Airflow was open-sourced in mid-2015, we have had fast > adoption and contributions by multiple organizations the world over. > We plan to continue to support new contributors and we will work to > actively promote those who contribute significantly to the project to > committers. > > === Community === > > Airflow is currently being used in over 30 companies. We hope to > extend our contributor base significantly and invite all those who are > interested in building large-scale distributed systems to participate. > > === Core Developers === > > Airflow is currently being developed by four engineers: Maxime > Beauchemin, Siddharth Anand, Bolke de Bruin, and Chris Riccomini. > Chris is a member of the Apache Samza PMC and a contributor to various > Apache projects, including Apache Kafka and Apache YARN. Maxime, > Siddharth, and Bolke have contributed to Airflow. > > === Alignment === > The ASF is the natural choice to host the Airflow project as its goal > of encouraging community-driven open-source projects fits with our > vision for Airflow. > > == Known Risks == > > === Orphaned Products === > > The core developers plan to work part time on the project. There is > very little risk of Airflow being abandoned as all of our companies > rely on it. > > === Inexperience with Open Source === > > All of the core developers have experience with open source > development. Chris is a member of the Apache Samza PMC and a > contributor to various Apache projects, including Apache Kafka and > Apache YARN. Bolke is contributor on multiple open source projects and > a few Apache projects as well, including Apache Hive, Apache Hadoop, > and Apache Ranger. > > === Homogeneous Developers === > > The current core developers are all from different companies. Our > community of 100 contributors hail from over 30 different companies > from across the world. > > === Reliance on Salaried Developers === > > Currently, the only developer paid to work on this project is Maxime. > > === Relationships with Other Apache Prod
Re: [VOTE] Accept Airflow into the Incubator
+1 (non-binding) Jeremiah > On Mar 25, 2016, at 8:26 AM, Suresh Marru wrote: > > + 1 (binding). > > Suresh > >> On Mar 24, 2016, at 11:00 PM, Siddharth Anand >> wrote: >> >> Following the discussion earlier: >> https://s.apache.org/AirflowDiscussion >> >> I would like to call a VOTE for accepting Airflow as a new incubator project. >> >> The proposal is available at: >> https://wiki.apache.org/incubator/AirflowProposal >> >> The proposal is also included at the bottom of this email. >> >> Vote is open until at least Tues, 29 March 2016, 23:59:00 PDT >> [ ] +1 accept Airflow into the Apache Incubator >> [ ] ±0 >> [ ] -1 because... >> >> +1 (non-binding) >> >> Thanks, >> -s (Sid) >> >> >> == Abstract == >> >> Airflow is a workflow automation and scheduling system that can be >> used to author and manage data pipelines. >> >> == Proposal == >> >> Airflow provides a system for authoring and managing workflows a.k.a. >> data pipelines a.k.a. DAGs (Directed Acyclic Graphs). The developer >> authors DAGs in Python using an Airflow-provided framework. He/She >> then executes the DAG using Airflow’s scheduler or registers the DAG >> for event-based execution. A web-based UI provides the developer with >> a range of options for managing and viewing his/her data pipelines. >> Background >> >> Airflow was developed at Airbnb to enable easier authorship and >> management of DAGs than were possible with existing solutions such as >> Oozie and Azkaban. For starters, both Oozie and Azkaban rely on one or >> more XML or property files to be bundled together to define a >> workflow. This separation of code and config can present a challenge >> to understanding the DAG - in Azkaban, a DAG’s structure is reflected >> by its file system tree and one can find himself/herself traversing >> the file system when inspecting or changing the structure of the DAG. >> Airflow workflows, on the other hand, are simply and elegantly defined >> in Python code, often a single file. Airflow merges the powerful >> Web-based management aspects of projects like Azkaban and Oozie with >> the simplicity and elegance of defining workflows in Python. Airflow, >> less than a year old in terms of its Open Source launch, is currently >> used in production environments in more than 30 companies and boasts >> an active contributor list of more than 100 developers, the vast >> majority of which (>95%) are outside of Airbnb. >> >> We would like to share it with the ASF and begin developing a >> community of developers and users within Apache. >> >> == Rationale == >> >> Many organizations (>30) already benefit from running Airflow to >> manage data pipelines. Our 100+ contributors continue to provide >> integrations with 3rd party systems through the implementation of new >> hooks and operators, both of which are used in defining the tasks that >> compose workflows. >> >> == Current Status == >> >> === Meritocracy === >> >> Our intent with this incubator proposal is to start building a diverse >> developer community around Airflow following the Apache meritocracy >> model. Since Airflow was open-sourced in mid-2015, we have had fast >> adoption and contributions by multiple organizations the world over. >> We plan to continue to support new contributors and we will work to >> actively promote those who contribute significantly to the project to >> committers. >> >> === Community === >> >> Airflow is currently being used in over 30 companies. We hope to >> extend our contributor base significantly and invite all those who are >> interested in building large-scale distributed systems to participate. >> >> === Core Developers === >> >> Airflow is currently being developed by four engineers: Maxime >> Beauchemin, Siddharth Anand, Bolke de Bruin, and Chris Riccomini. >> Chris is a member of the Apache Samza PMC and a contributor to various >> Apache projects, including Apache Kafka and Apache YARN. Maxime, >> Siddharth, and Bolke have contributed to Airflow. >> >> === Alignment === >> The ASF is the natural choice to host the Airflow project as its goal >> of encouraging community-driven open-source projects fits with our >> vision for Airflow. >> >> == Known Risks == >> >> === Orphaned Products === >> >> The core developers plan to work part time on the project. There is >> very little risk of Airflow being abandoned as all of our companies >> rely on it. >> >> === Inexperience with Open Source === >> >> All of the core developers have experience with open source >> development. Chris is a member of the Apache Samza PMC and a >> contributor to various Apache projects, including Apache Kafka and >> Apache YARN. Bolke is contributor on multiple open source projects and >> a few Apache projects as well, including Apache Hive, Apache Hadoop, >> and Apache Ranger. >> >> === Homogeneous Developers === >> >> The current core developers are all from different companies. Our >> community of 100 contributors hail from over 30
Re: [VOTE] Accept Airflow into the Incubator
+1 (binding) On Fri, Mar 25, 2016 at 5:57 AM, Jeremiah Lowin wrote: > +1 (non-binding) > > Jeremiah > > > > On Mar 25, 2016, at 8:26 AM, Suresh Marru wrote: > > > > + 1 (binding). > > > > Suresh > > > >> On Mar 24, 2016, at 11:00 PM, Siddharth Anand > wrote: > >> > >> Following the discussion earlier: > >> https://s.apache.org/AirflowDiscussion > >> > >> I would like to call a VOTE for accepting Airflow as a new incubator > project. > >> > >> The proposal is available at: > https://wiki.apache.org/incubator/AirflowProposal > >> > >> The proposal is also included at the bottom of this email. > >> > >> Vote is open until at least Tues, 29 March 2016, 23:59:00 PDT > >> [ ] +1 accept Airflow into the Apache Incubator > >> [ ] ±0 > >> [ ] -1 because... > >> > >> +1 (non-binding) > >> > >> Thanks, > >> -s (Sid) > >> > >> > >> == Abstract == > >> > >> Airflow is a workflow automation and scheduling system that can be > >> used to author and manage data pipelines. > >> > >> == Proposal == > >> > >> Airflow provides a system for authoring and managing workflows a.k.a. > >> data pipelines a.k.a. DAGs (Directed Acyclic Graphs). The developer > >> authors DAGs in Python using an Airflow-provided framework. He/She > >> then executes the DAG using Airflow’s scheduler or registers the DAG > >> for event-based execution. A web-based UI provides the developer with > >> a range of options for managing and viewing his/her data pipelines. > >> Background > >> > >> Airflow was developed at Airbnb to enable easier authorship and > >> management of DAGs than were possible with existing solutions such as > >> Oozie and Azkaban. For starters, both Oozie and Azkaban rely on one or > >> more XML or property files to be bundled together to define a > >> workflow. This separation of code and config can present a challenge > >> to understanding the DAG - in Azkaban, a DAG’s structure is reflected > >> by its file system tree and one can find himself/herself traversing > >> the file system when inspecting or changing the structure of the DAG. > >> Airflow workflows, on the other hand, are simply and elegantly defined > >> in Python code, often a single file. Airflow merges the powerful > >> Web-based management aspects of projects like Azkaban and Oozie with > >> the simplicity and elegance of defining workflows in Python. Airflow, > >> less than a year old in terms of its Open Source launch, is currently > >> used in production environments in more than 30 companies and boasts > >> an active contributor list of more than 100 developers, the vast > >> majority of which (>95%) are outside of Airbnb. > >> > >> We would like to share it with the ASF and begin developing a > >> community of developers and users within Apache. > >> > >> == Rationale == > >> > >> Many organizations (>30) already benefit from running Airflow to > >> manage data pipelines. Our 100+ contributors continue to provide > >> integrations with 3rd party systems through the implementation of new > >> hooks and operators, both of which are used in defining the tasks that > >> compose workflows. > >> > >> == Current Status == > >> > >> === Meritocracy === > >> > >> Our intent with this incubator proposal is to start building a diverse > >> developer community around Airflow following the Apache meritocracy > >> model. Since Airflow was open-sourced in mid-2015, we have had fast > >> adoption and contributions by multiple organizations the world over. > >> We plan to continue to support new contributors and we will work to > >> actively promote those who contribute significantly to the project to > >> committers. > >> > >> === Community === > >> > >> Airflow is currently being used in over 30 companies. We hope to > >> extend our contributor base significantly and invite all those who are > >> interested in building large-scale distributed systems to participate. > >> > >> === Core Developers === > >> > >> Airflow is currently being developed by four engineers: Maxime > >> Beauchemin, Siddharth Anand, Bolke de Bruin, and Chris Riccomini. > >> Chris is a member of the Apache Samza PMC and a contributor to various > >> Apache projects, including Apache Kafka and Apache YARN. Maxime, > >> Siddharth, and Bolke have contributed to Airflow. > >> > >> === Alignment === > >> The ASF is the natural choice to host the Airflow project as its goal > >> of encouraging community-driven open-source projects fits with our > >> vision for Airflow. > >> > >> == Known Risks == > >> > >> === Orphaned Products === > >> > >> The core developers plan to work part time on the project. There is > >> very little risk of Airflow being abandoned as all of our companies > >> rely on it. > >> > >> === Inexperience with Open Source === > >> > >> All of the core developers have experience with open source > >> development. Chris is a member of the Apache Samza PMC and a > >> contributor to various Apache projects, including Apache Kafka and > >> Apache YARN. Bolke is contributor on multiple open
Re: [VOTE] Accept Airflow into the Incubator
+1 (binding) --Chris Nauroth On 3/24/16, 8:00 PM, "Siddharth Anand" wrote: >Following the discussion earlier: >https://s.apache.org/AirflowDiscussion > >I would like to call a VOTE for accepting Airflow as a new incubator >project. > >The proposal is available at: >https://wiki.apache.org/incubator/AirflowProposal > >The proposal is also included at the bottom of this email. > >Vote is open until at least Tues, 29 March 2016, 23:59:00 PDT >[ ] +1 accept Airflow into the Apache Incubator >[ ] ±0 >[ ] -1 because... > >+1 (non-binding) > >Thanks, >-s (Sid) > > >== Abstract == > >Airflow is a workflow automation and scheduling system that can be >used to author and manage data pipelines. > >== Proposal == > >Airflow provides a system for authoring and managing workflows a.k.a. >data pipelines a.k.a. DAGs (Directed Acyclic Graphs). The developer >authors DAGs in Python using an Airflow-provided framework. He/She >then executes the DAG using Airflow¹s scheduler or registers the DAG >for event-based execution. A web-based UI provides the developer with >a range of options for managing and viewing his/her data pipelines. >Background > >Airflow was developed at Airbnb to enable easier authorship and >management of DAGs than were possible with existing solutions such as >Oozie and Azkaban. For starters, both Oozie and Azkaban rely on one or >more XML or property files to be bundled together to define a >workflow. This separation of code and config can present a challenge >to understanding the DAG - in Azkaban, a DAG¹s structure is reflected >by its file system tree and one can find himself/herself traversing >the file system when inspecting or changing the structure of the DAG. >Airflow workflows, on the other hand, are simply and elegantly defined >in Python code, often a single file. Airflow merges the powerful >Web-based management aspects of projects like Azkaban and Oozie with >the simplicity and elegance of defining workflows in Python. Airflow, >less than a year old in terms of its Open Source launch, is currently >used in production environments in more than 30 companies and boasts >an active contributor list of more than 100 developers, the vast >majority of which (>95%) are outside of Airbnb. > >We would like to share it with the ASF and begin developing a >community of developers and users within Apache. > >== Rationale == > >Many organizations (>30) already benefit from running Airflow to >manage data pipelines. Our 100+ contributors continue to provide >integrations with 3rd party systems through the implementation of new >hooks and operators, both of which are used in defining the tasks that >compose workflows. > >== Current Status == > >=== Meritocracy === > >Our intent with this incubator proposal is to start building a diverse >developer community around Airflow following the Apache meritocracy >model. Since Airflow was open-sourced in mid-2015, we have had fast >adoption and contributions by multiple organizations the world over. >We plan to continue to support new contributors and we will work to >actively promote those who contribute significantly to the project to >committers. > >=== Community === > >Airflow is currently being used in over 30 companies. We hope to >extend our contributor base significantly and invite all those who are >interested in building large-scale distributed systems to participate. > >=== Core Developers === > >Airflow is currently being developed by four engineers: Maxime >Beauchemin, Siddharth Anand, Bolke de Bruin, and Chris Riccomini. >Chris is a member of the Apache Samza PMC and a contributor to various >Apache projects, including Apache Kafka and Apache YARN. Maxime, >Siddharth, and Bolke have contributed to Airflow. > >=== Alignment === >The ASF is the natural choice to host the Airflow project as its goal >of encouraging community-driven open-source projects fits with our >vision for Airflow. > >== Known Risks == > >=== Orphaned Products === > >The core developers plan to work part time on the project. There is >very little risk of Airflow being abandoned as all of our companies >rely on it. > >=== Inexperience with Open Source === > >All of the core developers have experience with open source >development. Chris is a member of the Apache Samza PMC and a >contributor to various Apache projects, including Apache Kafka and >Apache YARN. Bolke is contributor on multiple open source projects and >a few Apache projects as well, including Apache Hive, Apache Hadoop, >and Apache Ranger. > >=== Homogeneous Developers === > >The current core developers are all from different companies. Our >community of 100 contributors hail from over 30 different companies >from across the world. > >=== Reliance on Salaried Developers === > >Currently, the only developer paid to work on this project is Maxime. > >=== Relationships with Other Apache Products === > >Airflow is deeply integrated with Apache products. It currently >provides hooks and operators to enable workflows
Re: [VOTE] Accept Airflow into the Incubator
+1 (non-binding) Regards, Arthur On Mar 25, 2016 9:27 AM, "Chris Nauroth" wrote: > +1 (binding) > > --Chris Nauroth > > > > > On 3/24/16, 8:00 PM, "Siddharth Anand" wrote: > > >Following the discussion earlier: > >https://s.apache.org/AirflowDiscussion > > > >I would like to call a VOTE for accepting Airflow as a new incubator > >project. > > > >The proposal is available at: > >https://wiki.apache.org/incubator/AirflowProposal > > > >The proposal is also included at the bottom of this email. > > > >Vote is open until at least Tues, 29 March 2016, 23:59:00 PDT > >[ ] +1 accept Airflow into the Apache Incubator > >[ ] ±0 > >[ ] -1 because... > > > >+1 (non-binding) > > > >Thanks, > >-s (Sid) > > > > > >== Abstract == > > > >Airflow is a workflow automation and scheduling system that can be > >used to author and manage data pipelines. > > > >== Proposal == > > > >Airflow provides a system for authoring and managing workflows a.k.a. > >data pipelines a.k.a. DAGs (Directed Acyclic Graphs). The developer > >authors DAGs in Python using an Airflow-provided framework. He/She > >then executes the DAG using Airflow¹s scheduler or registers the DAG > >for event-based execution. A web-based UI provides the developer with > >a range of options for managing and viewing his/her data pipelines. > >Background > > > >Airflow was developed at Airbnb to enable easier authorship and > >management of DAGs than were possible with existing solutions such as > >Oozie and Azkaban. For starters, both Oozie and Azkaban rely on one or > >more XML or property files to be bundled together to define a > >workflow. This separation of code and config can present a challenge > >to understanding the DAG - in Azkaban, a DAG¹s structure is reflected > >by its file system tree and one can find himself/herself traversing > >the file system when inspecting or changing the structure of the DAG. > >Airflow workflows, on the other hand, are simply and elegantly defined > >in Python code, often a single file. Airflow merges the powerful > >Web-based management aspects of projects like Azkaban and Oozie with > >the simplicity and elegance of defining workflows in Python. Airflow, > >less than a year old in terms of its Open Source launch, is currently > >used in production environments in more than 30 companies and boasts > >an active contributor list of more than 100 developers, the vast > >majority of which (>95%) are outside of Airbnb. > > > >We would like to share it with the ASF and begin developing a > >community of developers and users within Apache. > > > >== Rationale == > > > >Many organizations (>30) already benefit from running Airflow to > >manage data pipelines. Our 100+ contributors continue to provide > >integrations with 3rd party systems through the implementation of new > >hooks and operators, both of which are used in defining the tasks that > >compose workflows. > > > >== Current Status == > > > >=== Meritocracy === > > > >Our intent with this incubator proposal is to start building a diverse > >developer community around Airflow following the Apache meritocracy > >model. Since Airflow was open-sourced in mid-2015, we have had fast > >adoption and contributions by multiple organizations the world over. > >We plan to continue to support new contributors and we will work to > >actively promote those who contribute significantly to the project to > >committers. > > > >=== Community === > > > >Airflow is currently being used in over 30 companies. We hope to > >extend our contributor base significantly and invite all those who are > >interested in building large-scale distributed systems to participate. > > > >=== Core Developers === > > > >Airflow is currently being developed by four engineers: Maxime > >Beauchemin, Siddharth Anand, Bolke de Bruin, and Chris Riccomini. > >Chris is a member of the Apache Samza PMC and a contributor to various > >Apache projects, including Apache Kafka and Apache YARN. Maxime, > >Siddharth, and Bolke have contributed to Airflow. > > > >=== Alignment === > >The ASF is the natural choice to host the Airflow project as its goal > >of encouraging community-driven open-source projects fits with our > >vision for Airflow. > > > >== Known Risks == > > > >=== Orphaned Products === > > > >The core developers plan to work part time on the project. There is > >very little risk of Airflow being abandoned as all of our companies > >rely on it. > > > >=== Inexperience with Open Source === > > > >All of the core developers have experience with open source > >development. Chris is a member of the Apache Samza PMC and a > >contributor to various Apache projects, including Apache Kafka and > >Apache YARN. Bolke is contributor on multiple open source projects and > >a few Apache projects as well, including Apache Hive, Apache Hadoop, > >and Apache Ranger. > > > >=== Homogeneous Developers === > > > >The current core developers are all from different companies. Our > >community of 100 contributors hail from over 30
Re: [VOTE] Accept Airflow into the Incubator
+1 (non-binding) Tim On Fri, Mar 25, 2016 at 9:37 AM, Arthur Wiedmer wrote: > +1 (non-binding) > > Regards, > Arthur > On Mar 25, 2016 9:27 AM, "Chris Nauroth" wrote: > >> +1 (binding) >> >> --Chris Nauroth >> >> >> >> >> On 3/24/16, 8:00 PM, "Siddharth Anand" wrote: >> >> >Following the discussion earlier: >> >https://s.apache.org/AirflowDiscussion >> > >> >I would like to call a VOTE for accepting Airflow as a new incubator >> >project. >> > >> >The proposal is available at: >> >https://wiki.apache.org/incubator/AirflowProposal >> > >> >The proposal is also included at the bottom of this email. >> > >> >Vote is open until at least Tues, 29 March 2016, 23:59:00 PDT >> >[ ] +1 accept Airflow into the Apache Incubator >> >[ ] ±0 >> >[ ] -1 because... >> > >> >+1 (non-binding) >> > >> >Thanks, >> >-s (Sid) >> > >> > >> >== Abstract == >> > >> >Airflow is a workflow automation and scheduling system that can be >> >used to author and manage data pipelines. >> > >> >== Proposal == >> > >> >Airflow provides a system for authoring and managing workflows a.k.a. >> >data pipelines a.k.a. DAGs (Directed Acyclic Graphs). The developer >> >authors DAGs in Python using an Airflow-provided framework. He/She >> >then executes the DAG using Airflow¹s scheduler or registers the DAG >> >for event-based execution. A web-based UI provides the developer with >> >a range of options for managing and viewing his/her data pipelines. >> >Background >> > >> >Airflow was developed at Airbnb to enable easier authorship and >> >management of DAGs than were possible with existing solutions such as >> >Oozie and Azkaban. For starters, both Oozie and Azkaban rely on one or >> >more XML or property files to be bundled together to define a >> >workflow. This separation of code and config can present a challenge >> >to understanding the DAG - in Azkaban, a DAG¹s structure is reflected >> >by its file system tree and one can find himself/herself traversing >> >the file system when inspecting or changing the structure of the DAG. >> >Airflow workflows, on the other hand, are simply and elegantly defined >> >in Python code, often a single file. Airflow merges the powerful >> >Web-based management aspects of projects like Azkaban and Oozie with >> >the simplicity and elegance of defining workflows in Python. Airflow, >> >less than a year old in terms of its Open Source launch, is currently >> >used in production environments in more than 30 companies and boasts >> >an active contributor list of more than 100 developers, the vast >> >majority of which (>95%) are outside of Airbnb. >> > >> >We would like to share it with the ASF and begin developing a >> >community of developers and users within Apache. >> > >> >== Rationale == >> > >> >Many organizations (>30) already benefit from running Airflow to >> >manage data pipelines. Our 100+ contributors continue to provide >> >integrations with 3rd party systems through the implementation of new >> >hooks and operators, both of which are used in defining the tasks that >> >compose workflows. >> > >> >== Current Status == >> > >> >=== Meritocracy === >> > >> >Our intent with this incubator proposal is to start building a diverse >> >developer community around Airflow following the Apache meritocracy >> >model. Since Airflow was open-sourced in mid-2015, we have had fast >> >adoption and contributions by multiple organizations the world over. >> >We plan to continue to support new contributors and we will work to >> >actively promote those who contribute significantly to the project to >> >committers. >> > >> >=== Community === >> > >> >Airflow is currently being used in over 30 companies. We hope to >> >extend our contributor base significantly and invite all those who are >> >interested in building large-scale distributed systems to participate. >> > >> >=== Core Developers === >> > >> >Airflow is currently being developed by four engineers: Maxime >> >Beauchemin, Siddharth Anand, Bolke de Bruin, and Chris Riccomini. >> >Chris is a member of the Apache Samza PMC and a contributor to various >> >Apache projects, including Apache Kafka and Apache YARN. Maxime, >> >Siddharth, and Bolke have contributed to Airflow. >> > >> >=== Alignment === >> >The ASF is the natural choice to host the Airflow project as its goal >> >of encouraging community-driven open-source projects fits with our >> >vision for Airflow. >> > >> >== Known Risks == >> > >> >=== Orphaned Products === >> > >> >The core developers plan to work part time on the project. There is >> >very little risk of Airflow being abandoned as all of our companies >> >rely on it. >> > >> >=== Inexperience with Open Source === >> > >> >All of the core developers have experience with open source >> >development. Chris is a member of the Apache Samza PMC and a >> >contributor to various Apache projects, including Apache Kafka and >> >Apache YARN. Bolke is contributor on multiple open source projects and >> >a few Apache projects as well, including Ap
Re: [VOTE] Accept Airflow into the Incubator
+1 (non-binding) On Fri, Mar 25, 2016 at 1:05 PM Timothy Chen wrote: > +1 (non-binding) > > Tim > > On Fri, Mar 25, 2016 at 9:37 AM, Arthur Wiedmer > wrote: > > +1 (non-binding) > > > > Regards, > > Arthur > > On Mar 25, 2016 9:27 AM, "Chris Nauroth" > wrote: > > > >> +1 (binding) > >> > >> --Chris Nauroth > >> > >> > >> > >> > >> On 3/24/16, 8:00 PM, "Siddharth Anand" > wrote: > >> > >> >Following the discussion earlier: > >> >https://s.apache.org/AirflowDiscussion > >> > > >> >I would like to call a VOTE for accepting Airflow as a new incubator > >> >project. > >> > > >> >The proposal is available at: > >> >https://wiki.apache.org/incubator/AirflowProposal > >> > > >> >The proposal is also included at the bottom of this email. > >> > > >> >Vote is open until at least Tues, 29 March 2016, 23:59:00 PDT > >> >[ ] +1 accept Airflow into the Apache Incubator > >> >[ ] ±0 > >> >[ ] -1 because... > >> > > >> >+1 (non-binding) > >> > > >> >Thanks, > >> >-s (Sid) > >> > > >> > > >> >== Abstract == > >> > > >> >Airflow is a workflow automation and scheduling system that can be > >> >used to author and manage data pipelines. > >> > > >> >== Proposal == > >> > > >> >Airflow provides a system for authoring and managing workflows a.k.a. > >> >data pipelines a.k.a. DAGs (Directed Acyclic Graphs). The developer > >> >authors DAGs in Python using an Airflow-provided framework. He/She > >> >then executes the DAG using Airflow¹s scheduler or registers the DAG > >> >for event-based execution. A web-based UI provides the developer with > >> >a range of options for managing and viewing his/her data pipelines. > >> >Background > >> > > >> >Airflow was developed at Airbnb to enable easier authorship and > >> >management of DAGs than were possible with existing solutions such as > >> >Oozie and Azkaban. For starters, both Oozie and Azkaban rely on one or > >> >more XML or property files to be bundled together to define a > >> >workflow. This separation of code and config can present a challenge > >> >to understanding the DAG - in Azkaban, a DAG¹s structure is reflected > >> >by its file system tree and one can find himself/herself traversing > >> >the file system when inspecting or changing the structure of the DAG. > >> >Airflow workflows, on the other hand, are simply and elegantly defined > >> >in Python code, often a single file. Airflow merges the powerful > >> >Web-based management aspects of projects like Azkaban and Oozie with > >> >the simplicity and elegance of defining workflows in Python. Airflow, > >> >less than a year old in terms of its Open Source launch, is currently > >> >used in production environments in more than 30 companies and boasts > >> >an active contributor list of more than 100 developers, the vast > >> >majority of which (>95%) are outside of Airbnb. > >> > > >> >We would like to share it with the ASF and begin developing a > >> >community of developers and users within Apache. > >> > > >> >== Rationale == > >> > > >> >Many organizations (>30) already benefit from running Airflow to > >> >manage data pipelines. Our 100+ contributors continue to provide > >> >integrations with 3rd party systems through the implementation of new > >> >hooks and operators, both of which are used in defining the tasks that > >> >compose workflows. > >> > > >> >== Current Status == > >> > > >> >=== Meritocracy === > >> > > >> >Our intent with this incubator proposal is to start building a diverse > >> >developer community around Airflow following the Apache meritocracy > >> >model. Since Airflow was open-sourced in mid-2015, we have had fast > >> >adoption and contributions by multiple organizations the world over. > >> >We plan to continue to support new contributors and we will work to > >> >actively promote those who contribute significantly to the project to > >> >committers. > >> > > >> >=== Community === > >> > > >> >Airflow is currently being used in over 30 companies. We hope to > >> >extend our contributor base significantly and invite all those who are > >> >interested in building large-scale distributed systems to participate. > >> > > >> >=== Core Developers === > >> > > >> >Airflow is currently being developed by four engineers: Maxime > >> >Beauchemin, Siddharth Anand, Bolke de Bruin, and Chris Riccomini. > >> >Chris is a member of the Apache Samza PMC and a contributor to various > >> >Apache projects, including Apache Kafka and Apache YARN. Maxime, > >> >Siddharth, and Bolke have contributed to Airflow. > >> > > >> >=== Alignment === > >> >The ASF is the natural choice to host the Airflow project as its goal > >> >of encouraging community-driven open-source projects fits with our > >> >vision for Airflow. > >> > > >> >== Known Risks == > >> > > >> >=== Orphaned Products === > >> > > >> >The core developers plan to work part time on the project. There is > >> >very little risk of Airflow being abandoned as all of our companies > >> >rely on it. > >> > > >> >=== Inexperience with Open Source === >
Re: [VOTE] Accept Airflow into the Incubator
+1 (non-binding) On Fri, Mar 25, 2016 at 1:41 PM moon soo Lee wrote: > +1 (non-binding) > > On Fri, Mar 25, 2016 at 1:05 PM Timothy Chen wrote: > > > +1 (non-binding) > > > > Tim > > > > On Fri, Mar 25, 2016 at 9:37 AM, Arthur Wiedmer > > wrote: > > > +1 (non-binding) > > > > > > Regards, > > > Arthur > > > On Mar 25, 2016 9:27 AM, "Chris Nauroth" > > wrote: > > > > > >> +1 (binding) > > >> > > >> --Chris Nauroth > > >> > > >> > > >> > > >> > > >> On 3/24/16, 8:00 PM, "Siddharth Anand" > > wrote: > > >> > > >> >Following the discussion earlier: > > >> >https://s.apache.org/AirflowDiscussion > > >> > > > >> >I would like to call a VOTE for accepting Airflow as a new incubator > > >> >project. > > >> > > > >> >The proposal is available at: > > >> >https://wiki.apache.org/incubator/AirflowProposal > > >> > > > >> >The proposal is also included at the bottom of this email. > > >> > > > >> >Vote is open until at least Tues, 29 March 2016, 23:59:00 PDT > > >> >[ ] +1 accept Airflow into the Apache Incubator > > >> >[ ] ±0 > > >> >[ ] -1 because... > > >> > > > >> >+1 (non-binding) > > >> > > > >> >Thanks, > > >> >-s (Sid) > > >> > > > >> > > > >> >== Abstract == > > >> > > > >> >Airflow is a workflow automation and scheduling system that can be > > >> >used to author and manage data pipelines. > > >> > > > >> >== Proposal == > > >> > > > >> >Airflow provides a system for authoring and managing workflows a.k.a. > > >> >data pipelines a.k.a. DAGs (Directed Acyclic Graphs). The developer > > >> >authors DAGs in Python using an Airflow-provided framework. He/She > > >> >then executes the DAG using Airflow¹s scheduler or registers the DAG > > >> >for event-based execution. A web-based UI provides the developer with > > >> >a range of options for managing and viewing his/her data pipelines. > > >> >Background > > >> > > > >> >Airflow was developed at Airbnb to enable easier authorship and > > >> >management of DAGs than were possible with existing solutions such as > > >> >Oozie and Azkaban. For starters, both Oozie and Azkaban rely on one > or > > >> >more XML or property files to be bundled together to define a > > >> >workflow. This separation of code and config can present a challenge > > >> >to understanding the DAG - in Azkaban, a DAG¹s structure is reflected > > >> >by its file system tree and one can find himself/herself traversing > > >> >the file system when inspecting or changing the structure of the DAG. > > >> >Airflow workflows, on the other hand, are simply and elegantly > defined > > >> >in Python code, often a single file. Airflow merges the powerful > > >> >Web-based management aspects of projects like Azkaban and Oozie with > > >> >the simplicity and elegance of defining workflows in Python. Airflow, > > >> >less than a year old in terms of its Open Source launch, is currently > > >> >used in production environments in more than 30 companies and boasts > > >> >an active contributor list of more than 100 developers, the vast > > >> >majority of which (>95%) are outside of Airbnb. > > >> > > > >> >We would like to share it with the ASF and begin developing a > > >> >community of developers and users within Apache. > > >> > > > >> >== Rationale == > > >> > > > >> >Many organizations (>30) already benefit from running Airflow to > > >> >manage data pipelines. Our 100+ contributors continue to provide > > >> >integrations with 3rd party systems through the implementation of new > > >> >hooks and operators, both of which are used in defining the tasks > that > > >> >compose workflows. > > >> > > > >> >== Current Status == > > >> > > > >> >=== Meritocracy === > > >> > > > >> >Our intent with this incubator proposal is to start building a > diverse > > >> >developer community around Airflow following the Apache meritocracy > > >> >model. Since Airflow was open-sourced in mid-2015, we have had fast > > >> >adoption and contributions by multiple organizations the world over. > > >> >We plan to continue to support new contributors and we will work to > > >> >actively promote those who contribute significantly to the project to > > >> >committers. > > >> > > > >> >=== Community === > > >> > > > >> >Airflow is currently being used in over 30 companies. We hope to > > >> >extend our contributor base significantly and invite all those who > are > > >> >interested in building large-scale distributed systems to > participate. > > >> > > > >> >=== Core Developers === > > >> > > > >> >Airflow is currently being developed by four engineers: Maxime > > >> >Beauchemin, Siddharth Anand, Bolke de Bruin, and Chris Riccomini. > > >> >Chris is a member of the Apache Samza PMC and a contributor to > various > > >> >Apache projects, including Apache Kafka and Apache YARN. Maxime, > > >> >Siddharth, and Bolke have contributed to Airflow. > > >> > > > >> >=== Alignment === > > >> >The ASF is the natural choice to host the Airflow project as its goal > > >> >of encouraging community-driven open-source projects fits with
Re: [VOTE] Accept Airflow into the Incubator
+1 (non-binding) On 25 Mar 2016 03:00, "Siddharth Anand" wrote: > Following the discussion earlier: > https://s.apache.org/AirflowDiscussion > > I would like to call a VOTE for accepting Airflow as a new incubator > project. > > The proposal is available at: > https://wiki.apache.org/incubator/AirflowProposal > > The proposal is also included at the bottom of this email. > > Vote is open until at least Tues, 29 March 2016, 23:59:00 PDT > [ ] +1 accept Airflow into the Apache Incubator > [ ] ±0 > [ ] -1 because... > > +1 (non-binding) > > Thanks, > -s (Sid) > > > == Abstract == > > Airflow is a workflow automation and scheduling system that can be > used to author and manage data pipelines. > > == Proposal == > > Airflow provides a system for authoring and managing workflows a.k.a. > data pipelines a.k.a. DAGs (Directed Acyclic Graphs). The developer > authors DAGs in Python using an Airflow-provided framework. He/She > then executes the DAG using Airflow’s scheduler or registers the DAG > for event-based execution. A web-based UI provides the developer with > a range of options for managing and viewing his/her data pipelines. > Background > > Airflow was developed at Airbnb to enable easier authorship and > management of DAGs than were possible with existing solutions such as > Oozie and Azkaban. For starters, both Oozie and Azkaban rely on one or > more XML or property files to be bundled together to define a > workflow. This separation of code and config can present a challenge > to understanding the DAG - in Azkaban, a DAG’s structure is reflected > by its file system tree and one can find himself/herself traversing > the file system when inspecting or changing the structure of the DAG. > Airflow workflows, on the other hand, are simply and elegantly defined > in Python code, often a single file. Airflow merges the powerful > Web-based management aspects of projects like Azkaban and Oozie with > the simplicity and elegance of defining workflows in Python. Airflow, > less than a year old in terms of its Open Source launch, is currently > used in production environments in more than 30 companies and boasts > an active contributor list of more than 100 developers, the vast > majority of which (>95%) are outside of Airbnb. > > We would like to share it with the ASF and begin developing a > community of developers and users within Apache. > > == Rationale == > > Many organizations (>30) already benefit from running Airflow to > manage data pipelines. Our 100+ contributors continue to provide > integrations with 3rd party systems through the implementation of new > hooks and operators, both of which are used in defining the tasks that > compose workflows. > > == Current Status == > > === Meritocracy === > > Our intent with this incubator proposal is to start building a diverse > developer community around Airflow following the Apache meritocracy > model. Since Airflow was open-sourced in mid-2015, we have had fast > adoption and contributions by multiple organizations the world over. > We plan to continue to support new contributors and we will work to > actively promote those who contribute significantly to the project to > committers. > > === Community === > > Airflow is currently being used in over 30 companies. We hope to > extend our contributor base significantly and invite all those who are > interested in building large-scale distributed systems to participate. > > === Core Developers === > > Airflow is currently being developed by four engineers: Maxime > Beauchemin, Siddharth Anand, Bolke de Bruin, and Chris Riccomini. > Chris is a member of the Apache Samza PMC and a contributor to various > Apache projects, including Apache Kafka and Apache YARN. Maxime, > Siddharth, and Bolke have contributed to Airflow. > > === Alignment === > The ASF is the natural choice to host the Airflow project as its goal > of encouraging community-driven open-source projects fits with our > vision for Airflow. > > == Known Risks == > > === Orphaned Products === > > The core developers plan to work part time on the project. There is > very little risk of Airflow being abandoned as all of our companies > rely on it. > > === Inexperience with Open Source === > > All of the core developers have experience with open source > development. Chris is a member of the Apache Samza PMC and a > contributor to various Apache projects, including Apache Kafka and > Apache YARN. Bolke is contributor on multiple open source projects and > a few Apache projects as well, including Apache Hive, Apache Hadoop, > and Apache Ranger. > > === Homogeneous Developers === > > The current core developers are all from different companies. Our > community of 100 contributors hail from over 30 different companies > from across the world. > > === Reliance on Salaried Developers === > > Currently, the only developer paid to work on this project is Maxime. > > === Relationships with Other Apache Products === > > Airflow is deeply integrated with Apac
AirPal to Apache? Re: [VOTE] Accept Airflow into the Incubator
Any plans from AirBnB to bring AirPal to Apache? On 3/24/16, 8:00 PM, "Siddharth Anand" wrote: >Following the discussion earlier: >https://s.apache.org/AirflowDiscussion > >I would like to call a VOTE for accepting Airflow as a new incubator >project. > >The proposal is available at: >https://wiki.apache.org/incubator/AirflowProposal > >The proposal is also included at the bottom of this email. > >Vote is open until at least Tues, 29 March 2016, 23:59:00 PDT >[ ] +1 accept Airflow into the Apache Incubator >[ ] ±0 >[ ] -1 because... > >+1 (non-binding) > >Thanks, >-s (Sid) > > >== Abstract == > >Airflow is a workflow automation and scheduling system that can be >used to author and manage data pipelines. > >== Proposal == > >Airflow provides a system for authoring and managing workflows a.k.a. >data pipelines a.k.a. DAGs (Directed Acyclic Graphs). The developer >authors DAGs in Python using an Airflow-provided framework. He/She >then executes the DAG using Airflow¹s scheduler or registers the DAG >for event-based execution. A web-based UI provides the developer with >a range of options for managing and viewing his/her data pipelines. >Background > >Airflow was developed at Airbnb to enable easier authorship and >management of DAGs than were possible with existing solutions such as >Oozie and Azkaban. For starters, both Oozie and Azkaban rely on one or >more XML or property files to be bundled together to define a >workflow. This separation of code and config can present a challenge >to understanding the DAG - in Azkaban, a DAG¹s structure is reflected >by its file system tree and one can find himself/herself traversing >the file system when inspecting or changing the structure of the DAG. >Airflow workflows, on the other hand, are simply and elegantly defined >in Python code, often a single file. Airflow merges the powerful >Web-based management aspects of projects like Azkaban and Oozie with >the simplicity and elegance of defining workflows in Python. Airflow, >less than a year old in terms of its Open Source launch, is currently >used in production environments in more than 30 companies and boasts >an active contributor list of more than 100 developers, the vast >majority of which (>95%) are outside of Airbnb. > >We would like to share it with the ASF and begin developing a >community of developers and users within Apache. > >== Rationale == > >Many organizations (>30) already benefit from running Airflow to >manage data pipelines. Our 100+ contributors continue to provide >integrations with 3rd party systems through the implementation of new >hooks and operators, both of which are used in defining the tasks that >compose workflows. > >== Current Status == > >=== Meritocracy === > >Our intent with this incubator proposal is to start building a diverse >developer community around Airflow following the Apache meritocracy >model. Since Airflow was open-sourced in mid-2015, we have had fast >adoption and contributions by multiple organizations the world over. >We plan to continue to support new contributors and we will work to >actively promote those who contribute significantly to the project to >committers. > >=== Community === > >Airflow is currently being used in over 30 companies. We hope to >extend our contributor base significantly and invite all those who are >interested in building large-scale distributed systems to participate. > >=== Core Developers === > >Airflow is currently being developed by four engineers: Maxime >Beauchemin, Siddharth Anand, Bolke de Bruin, and Chris Riccomini. >Chris is a member of the Apache Samza PMC and a contributor to various >Apache projects, including Apache Kafka and Apache YARN. Maxime, >Siddharth, and Bolke have contributed to Airflow. > >=== Alignment === >The ASF is the natural choice to host the Airflow project as its goal >of encouraging community-driven open-source projects fits with our >vision for Airflow. > >== Known Risks == > >=== Orphaned Products === > >The core developers plan to work part time on the project. There is >very little risk of Airflow being abandoned as all of our companies >rely on it. > >=== Inexperience with Open Source === > >All of the core developers have experience with open source >development. Chris is a member of the Apache Samza PMC and a >contributor to various Apache projects, including Apache Kafka and >Apache YARN. Bolke is contributor on multiple open source projects and >a few Apache projects as well, including Apache Hive, Apache Hadoop, >and Apache Ranger. > >=== Homogeneous Developers === > >The current core developers are all from different companies. Our >community of 100 contributors hail from over 30 different companies >from across the world. > >=== Reliance on Salaried Developers === > >Currently, the only developer paid to work on this project is Maxime. > >=== Relationships with Other Apache Products === > >Airflow is deeply integrated with Apache products. It currently >provides hooks and operators to
Re: AirPal to Apache? Re: [VOTE] Accept Airflow into the Incubator
You would need to ask AirBnB. John On Fri, Mar 25, 2016 at 9:20 AM Adunuthula, Seshu wrote: > Any plans from AirBnB to bring AirPal to Apache? > > > > On 3/24/16, 8:00 PM, "Siddharth Anand" wrote: > > >Following the discussion earlier: > >https://s.apache.org/AirflowDiscussion > > > >I would like to call a VOTE for accepting Airflow as a new incubator > >project. > > > >The proposal is available at: > >https://wiki.apache.org/incubator/AirflowProposal > > > >The proposal is also included at the bottom of this email. > > > >Vote is open until at least Tues, 29 March 2016, 23:59:00 PDT > >[ ] +1 accept Airflow into the Apache Incubator > >[ ] ±0 > >[ ] -1 because... > > > >+1 (non-binding) > > > >Thanks, > >-s (Sid) > > > > > >== Abstract == > > > >Airflow is a workflow automation and scheduling system that can be > >used to author and manage data pipelines. > > > >== Proposal == > > > >Airflow provides a system for authoring and managing workflows a.k.a. > >data pipelines a.k.a. DAGs (Directed Acyclic Graphs). The developer > >authors DAGs in Python using an Airflow-provided framework. He/She > >then executes the DAG using Airflow¹s scheduler or registers the DAG > >for event-based execution. A web-based UI provides the developer with > >a range of options for managing and viewing his/her data pipelines. > >Background > > > >Airflow was developed at Airbnb to enable easier authorship and > >management of DAGs than were possible with existing solutions such as > >Oozie and Azkaban. For starters, both Oozie and Azkaban rely on one or > >more XML or property files to be bundled together to define a > >workflow. This separation of code and config can present a challenge > >to understanding the DAG - in Azkaban, a DAG¹s structure is reflected > >by its file system tree and one can find himself/herself traversing > >the file system when inspecting or changing the structure of the DAG. > >Airflow workflows, on the other hand, are simply and elegantly defined > >in Python code, often a single file. Airflow merges the powerful > >Web-based management aspects of projects like Azkaban and Oozie with > >the simplicity and elegance of defining workflows in Python. Airflow, > >less than a year old in terms of its Open Source launch, is currently > >used in production environments in more than 30 companies and boasts > >an active contributor list of more than 100 developers, the vast > >majority of which (>95%) are outside of Airbnb. > > > >We would like to share it with the ASF and begin developing a > >community of developers and users within Apache. > > > >== Rationale == > > > >Many organizations (>30) already benefit from running Airflow to > >manage data pipelines. Our 100+ contributors continue to provide > >integrations with 3rd party systems through the implementation of new > >hooks and operators, both of which are used in defining the tasks that > >compose workflows. > > > >== Current Status == > > > >=== Meritocracy === > > > >Our intent with this incubator proposal is to start building a diverse > >developer community around Airflow following the Apache meritocracy > >model. Since Airflow was open-sourced in mid-2015, we have had fast > >adoption and contributions by multiple organizations the world over. > >We plan to continue to support new contributors and we will work to > >actively promote those who contribute significantly to the project to > >committers. > > > >=== Community === > > > >Airflow is currently being used in over 30 companies. We hope to > >extend our contributor base significantly and invite all those who are > >interested in building large-scale distributed systems to participate. > > > >=== Core Developers === > > > >Airflow is currently being developed by four engineers: Maxime > >Beauchemin, Siddharth Anand, Bolke de Bruin, and Chris Riccomini. > >Chris is a member of the Apache Samza PMC and a contributor to various > >Apache projects, including Apache Kafka and Apache YARN. Maxime, > >Siddharth, and Bolke have contributed to Airflow. > > > >=== Alignment === > >The ASF is the natural choice to host the Airflow project as its goal > >of encouraging community-driven open-source projects fits with our > >vision for Airflow. > > > >== Known Risks == > > > >=== Orphaned Products === > > > >The core developers plan to work part time on the project. There is > >very little risk of Airflow being abandoned as all of our companies > >rely on it. > > > >=== Inexperience with Open Source === > > > >All of the core developers have experience with open source > >development. Chris is a member of the Apache Samza PMC and a > >contributor to various Apache projects, including Apache Kafka and > >Apache YARN. Bolke is contributor on multiple open source projects and > >a few Apache projects as well, including Apache Hive, Apache Hadoop, > >and Apache Ranger. > > > >=== Homogeneous Developers === > > > >The current core developers are all from different companies. Our > >community of 100 cont
Re: AirPal to Apache? Re: [VOTE] Accept Airflow into the Incubator
I am hoping there are a few AirBnB guys on this DL, though its dependency on Presto could prohibit it becoming an Apache project… On 3/25/16, 7:38 AM, "John D. Ament" wrote: >You would need to ask AirBnB. > >John > >On Fri, Mar 25, 2016 at 9:20 AM Adunuthula, Seshu >wrote: > >> Any plans from AirBnB to bring AirPal to Apache? >> >> >> >> On 3/24/16, 8:00 PM, "Siddharth Anand" wrote: >> >> >Following the discussion earlier: >> >https://s.apache.org/AirflowDiscussion >> > >> >I would like to call a VOTE for accepting Airflow as a new incubator >> >project. >> > >> >The proposal is available at: >> >https://wiki.apache.org/incubator/AirflowProposal >> > >> >The proposal is also included at the bottom of this email. >> > >> >Vote is open until at least Tues, 29 March 2016, 23:59:00 PDT >> >[ ] +1 accept Airflow into the Apache Incubator >> >[ ] ±0 >> >[ ] -1 because... >> > >> >+1 (non-binding) >> > >> >Thanks, >> >-s (Sid) >> > >> > >> >== Abstract == >> > >> >Airflow is a workflow automation and scheduling system that can be >> >used to author and manage data pipelines. >> > >> >== Proposal == >> > >> >Airflow provides a system for authoring and managing workflows a.k.a. >> >data pipelines a.k.a. DAGs (Directed Acyclic Graphs). The developer >> >authors DAGs in Python using an Airflow-provided framework. He/She >> >then executes the DAG using Airflow¹s scheduler or registers the DAG >> >for event-based execution. A web-based UI provides the developer with >> >a range of options for managing and viewing his/her data pipelines. >> >Background >> > >> >Airflow was developed at Airbnb to enable easier authorship and >> >management of DAGs than were possible with existing solutions such as >> >Oozie and Azkaban. For starters, both Oozie and Azkaban rely on one or >> >more XML or property files to be bundled together to define a >> >workflow. This separation of code and config can present a challenge >> >to understanding the DAG - in Azkaban, a DAG¹s structure is reflected >> >by its file system tree and one can find himself/herself traversing >> >the file system when inspecting or changing the structure of the DAG. >> >Airflow workflows, on the other hand, are simply and elegantly defined >> >in Python code, often a single file. Airflow merges the powerful >> >Web-based management aspects of projects like Azkaban and Oozie with >> >the simplicity and elegance of defining workflows in Python. Airflow, >> >less than a year old in terms of its Open Source launch, is currently >> >used in production environments in more than 30 companies and boasts >> >an active contributor list of more than 100 developers, the vast >> >majority of which (>95%) are outside of Airbnb. >> > >> >We would like to share it with the ASF and begin developing a >> >community of developers and users within Apache. >> > >> >== Rationale == >> > >> >Many organizations (>30) already benefit from running Airflow to >> >manage data pipelines. Our 100+ contributors continue to provide >> >integrations with 3rd party systems through the implementation of new >> >hooks and operators, both of which are used in defining the tasks that >> >compose workflows. >> > >> >== Current Status == >> > >> >=== Meritocracy === >> > >> >Our intent with this incubator proposal is to start building a diverse >> >developer community around Airflow following the Apache meritocracy >> >model. Since Airflow was open-sourced in mid-2015, we have had fast >> >adoption and contributions by multiple organizations the world over. >> >We plan to continue to support new contributors and we will work to >> >actively promote those who contribute significantly to the project to >> >committers. >> > >> >=== Community === >> > >> >Airflow is currently being used in over 30 companies. We hope to >> >extend our contributor base significantly and invite all those who are >> >interested in building large-scale distributed systems to participate. >> > >> >=== Core Developers === >> > >> >Airflow is currently being developed by four engineers: Maxime >> >Beauchemin, Siddharth Anand, Bolke de Bruin, and Chris Riccomini. >> >Chris is a member of the Apache Samza PMC and a contributor to various >> >Apache projects, including Apache Kafka and Apache YARN. Maxime, >> >Siddharth, and Bolke have contributed to Airflow. >> > >> >=== Alignment === >> >The ASF is the natural choice to host the Airflow project as its goal >> >of encouraging community-driven open-source projects fits with our >> >vision for Airflow. >> > >> >== Known Risks == >> > >> >=== Orphaned Products === >> > >> >The core developers plan to work part time on the project. There is >> >very little risk of Airflow being abandoned as all of our companies >> >rely on it. >> > >> >=== Inexperience with Open Source === >> > >> >All of the core developers have experience with open source >> >development. Chris is a member of the Apache Samza PMC and a >> >contributor to various Apache projects, including Apache Kafka and >>
