Hi everyone, As I've submitted BIGTOP-4360 [1], I'd like to propose incorporating Apache Airflow [2] into our stack, because we don't have any job scheduler/workflow orchestrator since we've dropped Oozie from 3.3.0.
Rationales in my mind are as follows: * As Astronomer and major cloud vendors provide the managed service of Airflow [3][4][5][6], it is the de-facto standard of open-source workflow orchestrator. * Airflow community is very active [7], so we don't need to be worried about its stagnation for several years at least. * I'm also an Airflow committer, so I can be responsible for maintaining it as a Bigtop component. Any thoughts or comments? [1]: https://issues.apache.org/jira/browse/BIGTOP-4360 [2]: https://airflow.apache.org/ [3]: https://www.astronomer.io/ [4]: https://aws.amazon.com/managed-workflows-for-apache-airflow/ [5]: https://learn.microsoft.com/en-us/fabric/data-factory/create-apache-airflow-jobs [6]: https://cloud.google.com/composer [7]: https://github.com/apache/airflow/graphs/contributors Kengo Seki <[email protected]>
