HadarSha commented on issue #26834: URL: https://github.com/apache/airflow/issues/26834#issuecomment-1304803317
@potiuk Unfortunately we have since rolled back to 2.3.4, particularly due to this very issue, so actual logs may be hard to come by. I will share our deployment and steps that led to the issue, however: * We are using Airflow on Kubernetes (AWS EKS) with Airflow's Official Helm Chart * 4 Airflow Schedulers running at all times (for HA and performance reasons - maybe it's an issue with the `airflow db upgrade` command in this context?) * Metadata DB is PostgreSQL on AWS RDS (i.e. not part of K8s cluster), accessed via a `PgBouncer` pod in the cluster (comes as part of the helm chart I believe) * The Executor is CeleryKubernetesExecutor, with RabbitMQ as the broker/backend * Some workers are always up (Celery) and some are autoscaled with HPA * We have some ~200 DAGs currently, delivered to the pods via `git-sync` containers on each of the Airflow services The steps we performed were as follows: 1. Verify no DAGs are running 2. Update Airflow package from 2.3.4 to 2.4.0 (we also updated provider packages, I believe) from one of the pods (e.g. Airflow Webserver), using pip 3. Run `airflow db upgrade` so as to upgrade the metadata DB tables, schemas etc. before actually updating the rest of the Airflow components (schedulers, workers, etc.) to 2.4.0 4. Command would run for about a minute and eventually fail with the exact same error as the issue opener attached. 5. I think that `airflow dags reserialize` would give a similar error, which forced us to downgrade back to 2.3.4 and ditch the 2.4.0 version for the time being. I apologize for not being able to provide more detailed logs/explanations, perhaps if we tried again at some point (maybe with the current version or thereabout) we could reproduce it - though in its essence it's very similar, if not identical, to what the author of this issue had already attached. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected]
