ruarfff commented on issue #36920:
URL: https://github.com/apache/airflow/issues/36920#issuecomment-1953856212

   Just to add some information on our experience of this issue. 
   
   We deploy Airflow with [Astronomer](https://www.astronomer.io/). We started 
seeing something like this issue in version 9.4.0 which includes Ariflow 2.7.2.
   
   When I say 'this issue' I mean the following:
   
   - DAGs start going into a skipped state and are not run.
   - Initially this is sporadic and for a while it's possible to trigger DAGs 
manually.
   - Eventually things degrade to a point where all DAGs skip and you can no 
longer get them to run by manually triggering them.
   - We see logs like the following: `{scheduler_job_runner.py:1444} INFO - DAG 
our_dag scheduling was skipped, probably because the DAG record was locked`.
   - We see some other suspicious logs but searching for those individually 
seems to indicate they are not part of the issue. 
   - We are running everything in Kubernetes, version 1.26.6.
   - We have a single scheduler instance running in the namespace.
   
   These are the provider versions we have installed and we're using Python 
3.11:
   ```
   apache-airflow-providers-cncf-kubernetes==7.14.0
   apache-airflow-providers-databricks==4.6.0
   apache-airflow-providers-microsoft-azure==8.0.0
   apache-airflow-providers-microsoft-mssql==3.5.0
   apache-airflow-providers-mongo==3.3.0
   apache-airflow-providers-postgres==5.7.1
   apache-airflow-providers-snowflake==5.3.0
   ```
   
   We went over our DAGs in detail to see if we had introduced something 
terrible and we tried turning some DAGs off having the scheduler only running a 
couple of DAGs.
   
   We tried upgrading to Airflow 2.8.0. That initially seemed to fix the issue 
but we eventually learned that any time we restart the scheduler, the issue 
resolves temporarily but always comes back. We also tried upgrading to 2.8.1.
   
   We completely removed the Airflow deployment, recreating it from scratch, 
but the issue came back. 
   
   Finally, we downgraded to Airflow 2.7.0 and this seems to have solved the 
issue. We still have no idea what the actual cause of the issue was, only that 
it does not happen in 2.7.0 but does in all versions from 2.7.2 and above.


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: [email protected]

For queries about this service, please contact Infrastructure at:
[email protected]

Reply via email to