Support for adding Airflow components: I believe that Airflow Docker installation is one use case scenario, but there is also a need for RPM installation. Therefore, we support RPM installation - those who want to use Docker can continue with Docker installation, while those who prefer RPM can use RPM installation.
> On Feb 21, 2025, at 23:27, Kengo Seki <[email protected]> wrote: > > Thank you for the comments, everyone. > >> Airflow move very fast. > > If you're concerning its EOL, the EOL date of the current 2.x line is > not determined as of now [1]. > Also, current target version (2.10.4) supports up to Python 3.12.x > [2], which EOL is in 2028 [3]. > So I think that's not a problem. > >> In my experience better way to start with airflow it’s docker compose, and >> after POC move to k8s. > > That may be true for the k8s users, but not all users run their services on > k8s. > In addition, adding Airflow to the Bigtop doesn't mean enforcing to > use deb/rpm-packaged Airflow. > Rather, it provides a new option to users. Even if Airflow is > incorporated into Bigtop, > users still can deploy their cluster using our deb/rpm packages except > for Airflow, > and setup only Airflow using docker compose or official Helm chart, > instead of our package. > That's totally up to users. > > [1]: > https://airflow.apache.org/docs/apache-airflow/stable/installation/supported-versions.html > [2]: https://github.com/apache/airflow/blob/2.10.4/pyproject.toml#L42 > [3]: https://devguide.python.org/versions/ > > Kengo Seki <[email protected]> > > On Fri, Feb 21, 2025 at 6:46 PM Masahiro Tanaka <[email protected]> wrote: >> >> Hi Antony, >> >> Do you mean we should use dockerized Airflow instead of installing RPM/DEB >> packages? >> >> Basically I'm +1 on adding Airflow for the non-containerized users. >> >> Regards, >> >> >> 2025年2月21日(金) 17:46 Антон Александров <[email protected]>: >> >>> Hi, i think it’s bad idea. >>> Airflow move very fast. >>> In my experience better way to start with airflow it’s docker compose, and >>> after POC move to k8s. >>> >>> best regards >>> Antony Aleksandrov >>> >>>> 21 февр. 2025 г., в 10:53, Jialiang Cai <[email protected]> >>> написал(а): >>>> >>>> +1 add Airflow >>>> >>>>> On Feb 21, 2025, at 15:51, Kengo Seki <[email protected]> wrote: >>>>> >>>>> Hi everyone, >>>>> >>>>> As I've submitted BIGTOP-4360 [1], I'd like to propose incorporating >>>>> Apache Airflow [2] into our stack, >>>>> because we don't have any job scheduler/workflow orchestrator since >>>>> we've dropped Oozie from 3.3.0. >>>>> >>>>> Rationales in my mind are as follows: >>>>> >>>>> * As Astronomer and major cloud vendors provide the managed service of >>>>> Airflow [3][4][5][6], >>>>> it is the de-facto standard of open-source workflow orchestrator. >>>>> >>>>> * Airflow community is very active [7], so we don't need to be worried >>>>> about its stagnation >>>>> for several years at least. >>>>> >>>>> * I'm also an Airflow committer, so I can be responsible for >>>>> maintaining it as a Bigtop component. >>>>> >>>>> Any thoughts or comments? >>>>> >>>>> [1]: https://issues.apache.org/jira/browse/BIGTOP-4360 >>>>> [2]: https://airflow.apache.org/ >>>>> [3]: https://www.astronomer.io/ >>>>> [4]: https://aws.amazon.com/managed-workflows-for-apache-airflow/ >>>>> [5]: >>> https://learn.microsoft.com/en-us/fabric/data-factory/create-apache-airflow-jobs >>>>> [6]: https://cloud.google.com/composer >>>>> [7]: https://github.com/apache/airflow/graphs/contributors >>>>> >>>>> Kengo Seki <[email protected]> >>>> >>> >>>
