One additional comment: I don't insist on including this change into the upcoming 3.4.0 release, it's totally OK to push it back to 3.5.0.
Kengo Seki <[email protected]> On Sat, Feb 22, 2025 at 12:27 AM Kengo Seki <[email protected]> wrote: > > Thank you for the comments, everyone. > > > Airflow move very fast. > > If you're concerning its EOL, the EOL date of the current 2.x line is > not determined as of now [1]. > Also, current target version (2.10.4) supports up to Python 3.12.x > [2], which EOL is in 2028 [3]. > So I think that's not a problem. > > > In my experience better way to start with airflow it’s docker compose, and > > after POC move to k8s. > > That may be true for the k8s users, but not all users run their services on > k8s. > In addition, adding Airflow to the Bigtop doesn't mean enforcing to > use deb/rpm-packaged Airflow. > Rather, it provides a new option to users. Even if Airflow is > incorporated into Bigtop, > users still can deploy their cluster using our deb/rpm packages except > for Airflow, > and setup only Airflow using docker compose or official Helm chart, > instead of our package. > That's totally up to users. > > [1]: > https://airflow.apache.org/docs/apache-airflow/stable/installation/supported-versions.html > [2]: https://github.com/apache/airflow/blob/2.10.4/pyproject.toml#L42 > [3]: https://devguide.python.org/versions/ > > Kengo Seki <[email protected]> > > On Fri, Feb 21, 2025 at 6:46 PM Masahiro Tanaka <[email protected]> wrote: > > > > Hi Antony, > > > > Do you mean we should use dockerized Airflow instead of installing RPM/DEB > > packages? > > > > Basically I'm +1 on adding Airflow for the non-containerized users. > > > > Regards, > > > > > > 2025年2月21日(金) 17:46 Антон Александров <[email protected]>: > > > > > Hi, i think it’s bad idea. > > > Airflow move very fast. > > > In my experience better way to start with airflow it’s docker compose, and > > > after POC move to k8s. > > > > > > best regards > > > Antony Aleksandrov > > > > > > > 21 февр. 2025 г., в 10:53, Jialiang Cai <[email protected]> > > > написал(а): > > > > > > > > +1 add Airflow > > > > > > > >> On Feb 21, 2025, at 15:51, Kengo Seki <[email protected]> wrote: > > > >> > > > >> Hi everyone, > > > >> > > > >> As I've submitted BIGTOP-4360 [1], I'd like to propose incorporating > > > >> Apache Airflow [2] into our stack, > > > >> because we don't have any job scheduler/workflow orchestrator since > > > >> we've dropped Oozie from 3.3.0. > > > >> > > > >> Rationales in my mind are as follows: > > > >> > > > >> * As Astronomer and major cloud vendors provide the managed service of > > > >> Airflow [3][4][5][6], > > > >> it is the de-facto standard of open-source workflow orchestrator. > > > >> > > > >> * Airflow community is very active [7], so we don't need to be worried > > > >> about its stagnation > > > >> for several years at least. > > > >> > > > >> * I'm also an Airflow committer, so I can be responsible for > > > >> maintaining it as a Bigtop component. > > > >> > > > >> Any thoughts or comments? > > > >> > > > >> [1]: https://issues.apache.org/jira/browse/BIGTOP-4360 > > > >> [2]: https://airflow.apache.org/ > > > >> [3]: https://www.astronomer.io/ > > > >> [4]: https://aws.amazon.com/managed-workflows-for-apache-airflow/ > > > >> [5]: > > > https://learn.microsoft.com/en-us/fabric/data-factory/create-apache-airflow-jobs > > > >> [6]: https://cloud.google.com/composer > > > >> [7]: https://github.com/apache/airflow/graphs/contributors > > > >> > > > >> Kengo Seki <[email protected]> > > > > > > > > > >
