GitHub user ginolegigot closed a discussion: Asking for insights on the best way to set up a resilient airflow celery cluster
Hello! Our current airflow celery cluster architecture is composed of: - two "admin" nodes, each having a webserver, a scheduler and a dag-processsor processes, so if an admin node falls the other one is still reachable and we still have a scheduler, a webserver and a dag-processor running. - two "worker" nodes, each having two processes: airflow celery worker and airflow celery flower. In the same spirit, if a worker node falls the other one is still up in addition of distributing the workload on both worker nodes when the two nodes are up. With airflow 3, regarding the airflow celery worker configuration, i looked for a way to set the apiserver base_url with both admin nodes but i believe it will only take one url not two. In my architecture, how shall i set api server config in workers nodes to tell the celery worker node "hey if the api server first node does not answer, reach the second one" except by setting a proxy like haproxy in front of both admin nodes ? is there a more "airflow" solution ? GitHub link: https://github.com/apache/airflow/discussions/60890 ---- This is an automatically sent email for [email protected]. To unsubscribe, please send an email to: [email protected]
