liangriyu opened a new issue, #35144:
URL: https://github.com/apache/airflow/issues/35144

   ### Apache Airflow version
   
   Other Airflow 2 version (please specify below)
   
   ### What happened
   
   2023-10-24 00:10:20,620 ERROR - Exception when executing 
SchedulerJob._run_scheduler_loop
   Traceback (most recent call last):
     File 
"/usr/local/miniconda3/envs/airflow/lib/python3.9/site-packages/airflow/jobs/scheduler_job_runner.py",
 line 835, in _execute
       self._run_scheduler_loop()
     File 
"/usr/local/miniconda3/envs/airflow/lib/python3.9/site-packages/airflow/jobs/scheduler_job_runner.py",
 line 969, in _run_scheduler_loop
       num_queued_tis = self._do_scheduling(session)
     File 
"/usr/local/miniconda3/envs/airflow/lib/python3.9/site-packages/airflow/jobs/scheduler_job_runner.py",
 line 1051, in _do_scheduling
       callback_tuples = self._schedule_all_dag_runs(guard, dag_runs, session)
     File 
"/usr/local/miniconda3/envs/airflow/lib/python3.9/site-packages/airflow/utils/retries.py",
 line 78, in wrapped_function
       for attempt in run_with_db_retries(max_retries=retries, logger=logger, 
**retry_kwargs):
     File 
"/usr/local/miniconda3/envs/airflow/lib/python3.9/site-packages/tenacity/__init__.py",
 line 347, in __iter__
       do = self.iter(retry_state=retry_state)
     File 
"/usr/local/miniconda3/envs/airflow/lib/python3.9/site-packages/tenacity/__init__.py",
 line 314, in iter
       return fut.result()
     File 
"/usr/local/miniconda3/envs/airflow/lib/python3.9/concurrent/futures/_base.py", 
line 439, in result
       return self.__get_result()
     File 
"/usr/local/miniconda3/envs/airflow/lib/python3.9/concurrent/futures/_base.py", 
line 391, in __get_result
       raise self._exception
     File 
"/usr/local/miniconda3/envs/airflow/lib/python3.9/site-packages/airflow/utils/retries.py",
 line 87, in wrapped_function
       return func(*args, **kwargs)
     File 
"/usr/local/miniconda3/envs/airflow/lib/python3.9/site-packages/airflow/jobs/scheduler_job_runner.py",
 line 1355, in _schedule_all_dag_runs
       for dag_run in dag_runs:
     File 
"/usr/local/miniconda3/envs/airflow/lib/python3.9/site-packages/sqlalchemy/orm/query.py",
 line 2901, in __iter__
       result = self._iter()
     File 
"/usr/local/miniconda3/envs/airflow/lib/python3.9/site-packages/sqlalchemy/orm/query.py",
 line 2916, in _iter
       result = self.session.execute(
     File 
"/usr/local/miniconda3/envs/airflow/lib/python3.9/site-packages/sqlalchemy/orm/session.py",
 line 1716, in execute
       conn = self._connection_for_bind(bind)
     File 
"/usr/local/miniconda3/envs/airflow/lib/python3.9/site-packages/sqlalchemy/orm/session.py",
 line 1555, in _connection_for_bind
       return self._transaction._connection_for_bind(
     File 
"/usr/local/miniconda3/envs/airflow/lib/python3.9/site-packages/sqlalchemy/orm/session.py",
 line 724, in _connection_for_bind
       self._assert_active()
     File 
"/usr/local/miniconda3/envs/airflow/lib/python3.9/site-packages/sqlalchemy/orm/session.py",
 line 604, in _assert_active
       raise sa_exc.PendingRollbackError(
   sqlalchemy.exc.PendingRollbackError: This Session's transaction has been 
rolled back due to a previous exception during flush. To begin a new 
transaction with this Session, first issue Session.rollback(). Original 
exception was: (mysql.connector.errors.InternalError) 1213 (40001): Deadlock 
found when trying to get lock; try restarting transaction
   [SQL: UPDATE dag_run SET 
last_scheduling_decision=%(last_scheduling_decision)s, 
updated_at=%(updated_at)s WHERE dag_run.id = %(dag_run_id)s]
   [parameters: ({'last_scheduling_decision': datetime.datetime(2023, 10, 23, 
16, 10, 20, 153406), 'updated_at': datetime.datetime(2023, 10, 23, 16, 10, 20, 
389092), 'dag_run_id': 154}, {'last_scheduling_decision': 
datetime.datetime(2023, 10, 23, 16, 10, 19, 835645), 'updated_at': 
datetime.datetime(2023, 10, 23, 16, 10, 20, 389109), 'dag_run_id': 157}, 
{'last_scheduling_decision': datetime.datetime(2023, 10, 23, 16, 10, 20, 
81685), 'updated_at': datetime.datetime(2023, 10, 23, 16, 10, 20, 389115), 
'dag_run_id': 158}, {'last_scheduling_decision': datetime.datetime(2023, 10, 
23, 16, 10, 20, 116207), 'updated_at': datetime.datetime(2023, 10, 23, 16, 10, 
20, 389119), 'dag_run_id': 159})]
   (Background on this error at: https://sqlalche.me/e/14/2j85) (Background on 
this error at: https://sqlalche.me/e/14/7s2a)
   2023-10-24 00:10:21,697 INFO - Sending Signals.SIGTERM to group 31151. PIDs 
of all processes in the group: [32813, 32842, 31151]
   2023-10-24 00:10:21,698 INFO - Sending the signal Signals.SIGTERM to group 
31151
   2023-10-24 00:10:21,991 INFO - Process psutil.Process(pid=32813, 
status='terminated', started='00:10:20') (32813) terminated with exit code None
   2023-10-24 00:10:22,084 INFO - Process psutil.Process(pid=32842, 
status='terminated', started='00:10:20') (32842) terminated with exit code None
   2023-10-24 00:10:22,136 INFO - Process psutil.Process(pid=31151, 
status='terminated', exitcode=0, started='00:09:41') (31151) terminated with 
exit code 0
   2023-10-24 00:10:22,138 INFO - Exited execute loop
   
   ### What you think should happen instead
   
   _No response_
   
   ### How to reproduce
   
   When multiple dag tasks fail simultaneously
   
   ### Operating System
   
   centos7
   
   ### Versions of Apache Airflow Providers
   
   apache-airflow                      2.6.3
   apache-airflow-providers-celery     3.2.1
   apache-airflow-providers-common-sql 1.5.2
   apache-airflow-providers-datadog    3.3.1
   apache-airflow-providers-ftp        3.4.2
   apache-airflow-providers-http       4.4.2
   apache-airflow-providers-imap       3.2.2
   apache-airflow-providers-mysql      5.1.1
   apache-airflow-providers-redis      3.2.1
   apache-airflow-providers-sqlite     3.4.2
   
   ### Deployment
   
   Virtualenv installation
   
   ### Deployment details
   
   _No response_
   
   ### Anything else
   
   _No response_
   
   ### Are you willing to submit PR?
   
   - [ ] Yes I am willing to submit a PR!
   
   ### Code of Conduct
   
   - [X] I agree to follow this project's [Code of 
Conduct](https://github.com/apache/airflow/blob/main/CODE_OF_CONDUCT.md)
   


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: [email protected]

For queries about this service, please contact Infrastructure at:
[email protected]

Reply via email to