Thank you for the comments, everyone. > Airflow move very fast.
If you're concerning its EOL, the EOL date of the current 2.x line is not determined as of now [1]. Also, current target version (2.10.4) supports up to Python 3.12.x [2], which EOL is in 2028 [3]. So I think that's not a problem. > In my experience better way to start with airflow it’s docker compose, and > after POC move to k8s. That may be true for the k8s users, but not all users run their services on k8s. In addition, adding Airflow to the Bigtop doesn't mean enforcing to use deb/rpm-packaged Airflow. Rather, it provides a new option to users. Even if Airflow is incorporated into Bigtop, users still can deploy their cluster using our deb/rpm packages except for Airflow, and setup only Airflow using docker compose or official Helm chart, instead of our package. That's totally up to users. [1]: https://airflow.apache.org/docs/apache-airflow/stable/installation/supported-versions.html [2]: https://github.com/apache/airflow/blob/2.10.4/pyproject.toml#L42 [3]: https://devguide.python.org/versions/ Kengo Seki <[email protected]> On Fri, Feb 21, 2025 at 6:46 PM Masahiro Tanaka <[email protected]> wrote: > > Hi Antony, > > Do you mean we should use dockerized Airflow instead of installing RPM/DEB > packages? > > Basically I'm +1 on adding Airflow for the non-containerized users. > > Regards, > > > 2025年2月21日(金) 17:46 Антон Александров <[email protected]>: > > > Hi, i think it’s bad idea. > > Airflow move very fast. > > In my experience better way to start with airflow it’s docker compose, and > > after POC move to k8s. > > > > best regards > > Antony Aleksandrov > > > > > 21 февр. 2025 г., в 10:53, Jialiang Cai <[email protected]> > > написал(а): > > > > > > +1 add Airflow > > > > > >> On Feb 21, 2025, at 15:51, Kengo Seki <[email protected]> wrote: > > >> > > >> Hi everyone, > > >> > > >> As I've submitted BIGTOP-4360 [1], I'd like to propose incorporating > > >> Apache Airflow [2] into our stack, > > >> because we don't have any job scheduler/workflow orchestrator since > > >> we've dropped Oozie from 3.3.0. > > >> > > >> Rationales in my mind are as follows: > > >> > > >> * As Astronomer and major cloud vendors provide the managed service of > > >> Airflow [3][4][5][6], > > >> it is the de-facto standard of open-source workflow orchestrator. > > >> > > >> * Airflow community is very active [7], so we don't need to be worried > > >> about its stagnation > > >> for several years at least. > > >> > > >> * I'm also an Airflow committer, so I can be responsible for > > >> maintaining it as a Bigtop component. > > >> > > >> Any thoughts or comments? > > >> > > >> [1]: https://issues.apache.org/jira/browse/BIGTOP-4360 > > >> [2]: https://airflow.apache.org/ > > >> [3]: https://www.astronomer.io/ > > >> [4]: https://aws.amazon.com/managed-workflows-for-apache-airflow/ > > >> [5]: > > https://learn.microsoft.com/en-us/fabric/data-factory/create-apache-airflow-jobs > > >> [6]: https://cloud.google.com/composer > > >> [7]: https://github.com/apache/airflow/graphs/contributors > > >> > > >> Kengo Seki <[email protected]> > > > > > > >
