ruarfff commented on issue #36920: URL: https://github.com/apache/airflow/issues/36920#issuecomment-1953856212
Just to add some information on our experience of this issue. We deploy Airflow with [Astronomer](https://www.astronomer.io/). We started seeing something like this issue in version 9.4.0 which includes Ariflow 2.7.2. When I say 'this issue' I mean the following: - DAGs start going into a skipped state and are not run. - Initially this is sporadic and for a while it's possible to trigger DAGs manually. - Eventually things degrade to a point where all DAGs skip and you can no longer get them to run by manually triggering them. - We see logs like the following: `{scheduler_job_runner.py:1444} INFO - DAG our_dag scheduling was skipped, probably because the DAG record was locked`. - We see some other suspicious logs but searching for those individually seems to indicate they are not part of the issue. - We are running everything in Kubernetes, version 1.26.6. - We have a single scheduler instance running in the namespace. These are the provider versions we have installed and we're using Python 3.11: ``` apache-airflow-providers-cncf-kubernetes==7.14.0 apache-airflow-providers-databricks==4.6.0 apache-airflow-providers-microsoft-azure==8.0.0 apache-airflow-providers-microsoft-mssql==3.5.0 apache-airflow-providers-mongo==3.3.0 apache-airflow-providers-postgres==5.7.1 apache-airflow-providers-snowflake==5.3.0 ``` We went over our DAGs in detail to see if we had introduced something terrible and we tried turning some DAGs off having the scheduler only running a couple of DAGs. We tried upgrading to Airflow 2.8.0. That initially seemed to fix the issue but we eventually learned that any time we restart the scheduler, the issue resolves temporarily but always comes back. We also tried upgrading to 2.8.1. We completely removed the Airflow deployment, recreating it from scratch, but the issue came back. Finally, we downgraded to Airflow 2.7.0 and this seems to have solved the issue. We still have no idea what the actual cause of the issue was, only that it does not happen in 2.7.0 but does in all versions from 2.7.2 and above. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected]
