Some thoughts on adding an Airflow Ecosystem section to the website, or something similarly named.
I like the idea of highlighting how Airflow is being used and tend to appreciate a small paragraph about HOW it is used by Company XYZ rather than just a list of companies. I also think linking to companies that provide professional support is helpful. This should be done with care as other open source projects tend to lean heavily towards a single company or two which can split the community. Cassandra is one example that comes to mind. This particular area may need to be handled differently than the others. Highlighting helpful tools and resources seems like a no brainer to me. The way the Groovy Language (https://groovy-lang.org/learn.html) project does it seems to be a good starting point. They list everything in "Ecosystem" and include other projects that support or work well with Groovy. Something similar would help highlight the growing ecosystem around Airflow. On Fri, Jan 17, 2020 at 9:19 AM Jarek Potiuk <[email protected]> wrote: > TL;DR; I have a proposal to add "Powered By" section (or similar) to the > official Airflow Website > > This is the result of a discussion we had with other PMCs regarding whether > we should have links to external services/tools etc. using Apache Airflow. > We think devlist is the right place to bring it and all the members of our > community should have a chance to share their > opinions/concerns/expectations. > > I believe we should have a page where we describe some other sources of > information and some other notable uses of Airflow that are outside of the > "official" source for Apache Airflow. Those are outside of the community > control - but nevertheless, super useful for the users. > > The "working title" for this page could be "Others about Airflow", > "Powered by Airflow" - I am sure we could come with a good name for it in > the community. > > I think we could make such a page with a very clear disclaimer that those > are useful sources, but outside of the community control and we should be > free to remove any of such pages from the list in case we see that there > are some violations with the Apache Way. > > I thought about some general areas on such page (and we might decide to > include only some of those): > > 1) Other sources of information about Airflow > - here https://github.com/jghoman/awesome-apache-airflow is definitely > something that should be on the list - it's volunteering effort of Jakob > but highly regarded among the community, maybe other sources could be added > here. > > 2) Who's Using Airflow: > - here we could have a list of companies that are using Airflow. We already > have it in the README.md but I think it really belongs to the website > rather than the README.md (but love to hear what others think about it). > > 3) Airflow-as-a-service > - here we could add Composer, Astronomer and possibly other > "airflow-as-a-service" companies > > 4) Integrating with Airflow: > - here we could add all the workflow solutions that somehow integrate with > Airflow and use it under the hood or export DAGs to Airflow : > - our "Ooozie-2-Airflow" library > - the CWL -> Airflow (https://github.com/Barski-lab/cwl-airflow) those > CWL developers who wanted to contribute the code to Airflow, > - Databand.ai -> they build specialized ML python DSL to generate > Airflow DAGs and use Airflow as execution engine > > Those are just examples, but I think they can be very useful to be > published for the users. However it should be CRYSTAL CLEAR - those are not > products/companies Apache Airflow Community has an impact on and control > over. > > Let me know what you think, I look forward to hear what others think about > it. > > J. > > -- > > Jarek Potiuk > Polidea <https://www.polidea.com/> | Principal Software Engineer > > M: +48 660 796 129 <+48660796129> > [image: Polidea] <https://www.polidea.com/> > -- Nathan Maynes <http://bit.ly/115hXAt> @nathanmaynes
