vchiapaikeo commented on issue #36384: URL: https://github.com/apache/airflow/issues/36384#issuecomment-1868524593
Has anyone noticed unusual behavior with the DagFileProcessor lately? I've observed some unusual behavior where a relatively simple dag is continually getting removed and added back. I went ahead and reverted [my change](https://github.com/apache/airflow/pull/36226) locally to see if I could still repro this bug and I could so do not think it is related. However, it is hard to test my change because this simple dag seems to be continually getting removed and re-added. Logs below - <img width="1000" alt="image" src="https://github.com/apache/airflow/assets/9200263/435e8caf-562a-417f-b1de-aab3635731f3"> Scheduler Logs ``` 12023-12-24T13:51:44.460+0000] {manager. py:523} INFO - DAG my_dag is missing and will be deactivated. [2023-12-2413:51:44.462+0000] {manager.py:535} INFO - Deactivated 1 DAGs which are no longer present in file. [2023-12-2413:51:44.463+00001 {manager. py:539} INFO - Deleted DAG my_dag in serialized_dag table [2023-12-2413:52:48.417+0000] {scheduler_job_runner.py: 696} INFO - Received executor event with state success for task instance TaskInstanceKey (dag_id='my_dag', ta sk_id='task', run_id='scheduled__2021-01-01T02:05:00+00:00', try_number=1, map_ind ex=-1) 12023-12-2413:52:48.419+0000] {scheduler_job_runner.py:733} INFO - TaskInstance Finished: dag_id=my_dag, task_id=task, run_id=scheduled__2021-01-01T02:05:00+00:00, map_index=-1, run_start_date=2023-12-24 13:50:47. 082990+00:00, run_end_date=2023-12-24 13:52:47.473307+00:00, run_duration=120.390317, state=success, executor_stat e=success, try_number=1, max_tries=2, job_id=2080, pool=default_pool, queue=defaul t, priority_weight=1, operator=BashOperator, queued_dttm=2023-12-24 13:50:46.878463+00:00, queued_by_job_id=43, pid=4500 [2023-12-2413:53: 06.037+0000] {processor-py: 277} WARNING - Killing DAGFileProcess orProcess (PID=4696) ``` Dag File Processor Logs ``` [2023-12-24T13:47:13.820+0000] {manager.py:879} INFO - ================================================================================ DAG File Processing Stats File Path PID Runtime # DAGs # Errors Last Runtime Last Run ------------------------------------------- ----- --------- -------- ---------- -------------- ------------------- /files/dags/my_dag.py 1549 40.92s 0 1 100.14s 2023-12-24T13:46:02 /files/dags/example_dynamic_task_mapping.py 1 0 0.63s 2023-12-24T13:46:48 ================================================================================ [2023-12-24T13:47:44.032+0000] {manager.py:879} INFO - ================================================================================ DAG File Processing Stats File Path PID Runtime # DAGs # Errors Last Runtime Last Run ------------------------------------------- ----- --------- -------- ---------- -------------- ------------------- /files/dags/my_dag.py 1549 71.13s 0 1 100.14s 2023-12-24T13:46:02 /files/dags/example_dynamic_task_mapping.py 1 0 0.59s 2023-12-24T13:47:19 ================================================================================ [2023-12-24T13:48:13.391+0000] {manager.py:1201} ERROR - Processor for /files/dags/my_dag.py with PID 1549 started at 2023-12-24T13:46:32.902112+00:00 has timed out, killing it. [2023-12-24T13:48:14.430+0000] {manager.py:879} INFO - ================================================================================ DAG File Processing Stats File Path PID Runtime # DAGs # Errors Last Runtime Last Run ------------------------------------------- ----- --------- -------- ---------- -------------- ------------------- /files/dags/my_dag.py 0 1 100.49s 2023-12-24T13:48:13 /files/dags/example_dynamic_task_mapping.py 1 0 0.47s 2023-12-24T13:47:50 ================================================================================ ``` my_dag.py dag file ```py import datetime from airflow import DAG from airflow.operators.bash import BashOperator # from airflow.providers.smtp.notifications.smtp import SmtpNotifier from datetime import timedelta EMAIL = EMAIL with DAG( dag_id="my_dag", start_date=datetime.datetime(2021, 1, 1), schedule="*/5 * * * *", max_active_tasks=1, max_active_runs=1, # sla_miss_callback=SmtpNotifier(from_email=EMAIL, to=EMAIL), ): BashOperator( task_id="task", bash_command="sleep 120", retries=2, sla=timedelta(seconds=30), # on_success_callback=SmtpNotifier(from_email=EMAIL, to=EMAIL), # on_failure_callback=SmtpNotifier(from_email=EMAIL, to=EMAIL), # on_retry_callback=SmtpNotifier(from_email=EMAIL, to=EMAIL), ) ``` -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected]
