While I am usually negative about adding new products due to lack of developer
resources,
Kengo should be eligible to keep maintaining it as a long standing committer.
Since branch-3.4 is already cut for preparing 3.4.0 release, the target version
should be 3.5.0.
Thanks,
Masatake Iwasaki
On 2025/02/22 0:27, Kengo Seki wrote:
Thank you for the comments, everyone.
Airflow move very fast.
If you're concerning its EOL, the EOL date of the current 2.x line is
not determined as of now [1].
Also, current target version (2.10.4) supports up to Python 3.12.x
[2], which EOL is in 2028 [3].
So I think that's not a problem.
In my experience better way to start with airflow it’s docker compose, and
after POC move to k8s.
That may be true for the k8s users, but not all users run their services on k8s.
In addition, adding Airflow to the Bigtop doesn't mean enforcing to
use deb/rpm-packaged Airflow.
Rather, it provides a new option to users. Even if Airflow is
incorporated into Bigtop,
users still can deploy their cluster using our deb/rpm packages except
for Airflow,
and setup only Airflow using docker compose or official Helm chart,
instead of our package.
That's totally up to users.
[1]:
https://airflow.apache.org/docs/apache-airflow/stable/installation/supported-versions.html
[2]: https://github.com/apache/airflow/blob/2.10.4/pyproject.toml#L42
[3]: https://devguide.python.org/versions/
Kengo Seki <[email protected]>
On Fri, Feb 21, 2025 at 6:46 PM Masahiro Tanaka <[email protected]> wrote:
Hi Antony,
Do you mean we should use dockerized Airflow instead of installing RPM/DEB
packages?
Basically I'm +1 on adding Airflow for the non-containerized users.
Regards,
2025年2月21日(金) 17:46 Антон Александров <[email protected]>:
Hi, i think it’s bad idea.
Airflow move very fast.
In my experience better way to start with airflow it’s docker compose, and
after POC move to k8s.
best regards
Antony Aleksandrov
21 февр. 2025 г., в 10:53, Jialiang Cai <[email protected]>
написал(а):
+1 add Airflow
On Feb 21, 2025, at 15:51, Kengo Seki <[email protected]> wrote:
Hi everyone,
As I've submitted BIGTOP-4360 [1], I'd like to propose incorporating
Apache Airflow [2] into our stack,
because we don't have any job scheduler/workflow orchestrator since
we've dropped Oozie from 3.3.0.
Rationales in my mind are as follows:
* As Astronomer and major cloud vendors provide the managed service of
Airflow [3][4][5][6],
it is the de-facto standard of open-source workflow orchestrator.
* Airflow community is very active [7], so we don't need to be worried
about its stagnation
for several years at least.
* I'm also an Airflow committer, so I can be responsible for
maintaining it as a Bigtop component.
Any thoughts or comments?
[1]: https://issues.apache.org/jira/browse/BIGTOP-4360
[2]: https://airflow.apache.org/
[3]: https://www.astronomer.io/
[4]: https://aws.amazon.com/managed-workflows-for-apache-airflow/
[5]:
https://learn.microsoft.com/en-us/fabric/data-factory/create-apache-airflow-jobs
[6]: https://cloud.google.com/composer
[7]: https://github.com/apache/airflow/graphs/contributors
Kengo Seki <[email protected]>
