shivanshs9 commented on issue #11899:
URL: https://github.com/apache/airflow/issues/11899#issuecomment-717560284
@potiuk this was observed in RDS, so difficult to provide all the DB logs.
I'll spin up a local MySQL pod and try and replicate the issue with the similar
load.
For now, I managed to get the latest deadlock (ran `SHOW ENGINE INNODB
STATUS;`):
```
------------------------
LATEST DETECTED DEADLOCK
------------------------
2020-10-27 21:40:59 0x2ad0ba744700
*** (1) TRANSACTION:
TRANSACTION 4308769990, ACTIVE 0 sec fetching rows
mysql tables in use 1, locked 1
LOCK WAIT 4 lock struct(s), heap size 1136, 4 row lock(s)
MySQL thread id 19920923, OS thread handle 47071625484032, query id
[truncated].compute-1.amazonaws.com [truncated] root Sending data
SELECT dag.dag_id AS dag_dag_id, dag.root_dag_id AS dag_root_dag_id,
dag.is_paused AS dag_is_paused, dag.is_subdag AS dag_is_subdag, dag.is_active
AS dag_is_active, dag.last_scheduler_run AS dag_last_scheduler_run,
dag.last_pickled AS dag_last_pickled, dag.last_expired AS dag_last_expired,
dag.scheduler_lock AS dag_scheduler_lock, dag.pickle_id AS dag_pickle_id,
dag.fileloc AS dag_fileloc, dag.owners AS dag_owners, dag.description AS
dag_description, dag.default_view AS dag_default_view, dag.schedule_interval AS
dag_schedule_interval, dag.concurrency AS dag_concurrency,
dag.has_task_concurrency_limits AS dag_has_task_concurrency_limits,
dag.next_dagrun AS dag_next_dagrun, dag.next_dagrun_create_after AS
dag_next_dagrun_create_after
FROM dag
WHERE dag.is_paused IS false AND dag.is_active IS true AND
dag.next_dagrun_create_after <= now() ORDER BY dag.next_dagr
*** (1) WAITING FOR THIS LOCK TO BE GRANTED:
RECORD LOCKS space id 5563 page no 3 n bits 168 index PRIMARY of table
`chronos_airflow`.`dag` trx id 4308769990 lock_mode X locks rec but not gap
waiting
Record lock, heap no 97 PHYSICAL RECORD: n_fields 21; compact format; info
bits 0
0: len 18; hex 6572676f5f6a6f625f636f6c6c6563746f72; asc
ergo_job_collector;;
1: len 6; hex 000100d29cc5; asc ;;
2: len 7; hex 420006001209ca; asc B ;;
3: len 1; hex 80; asc ;;
4: len 1; hex 80; asc ;;
5: len 1; hex 81; asc ;;
6: SQL NULL;
7: SQL NULL;
8: SQL NULL;
9: SQL NULL;
10: SQL NULL;
11: len 29; hex 2f6f70742f616972666c6f772f646167732f6461675f6572676f2e7079;
asc /opt/airflow/dags/dag_ergo.py;;
12: len 7; hex 616972666c6f77; asc airflow;;
13: SQL NULL;
14: len 4; hex 74726565; asc tree;;
15: len 30; hex
7b2274797065223a202274696d6564656c7461222c20226174747273223a; asc {"type":
"timedelta", "attrs":; (total 77 bytes);
16: SQL NULL;
17: len 7; hex 5f9893e102c57c; asc _ |;;
18: len 7; hex 5f9893eb02c57c; asc _ |;;
19: len 4; hex 80000010; asc ;;
20: len 1; hex 80; asc ;;
*** (2) TRANSACTION:
TRANSACTION 4308769989, ACTIVE 0 sec updating or deleting
mysql tables in use 2, locked 2
7 lock struct(s), heap size 1136, 5 row lock(s), undo log entries 1
MySQL thread id 19927787, OS thread handle 47075969746688, query id
[truncated].compute-1.amazonaws.com [truncated] root Updating
UPDATE dag SET next_dagrun='2020-10-27 21:40:49.181628',
next_dagrun_create_after='2020-10-27 21:40:59.181628' WHERE dag.dag_id =
'ergo_job_collector'
*** (2) HOLDS THE LOCK(S):
RECORD LOCKS space id 5563 page no 3 n bits 168 index PRIMARY of table
`chronos_airflow`.`dag` trx id 4308769989 lock_mode X locks rec but not gap
Record lock, heap no 95 PHYSICAL RECORD: n_fields 21; compact format; info
bits 0
0: len 16; hex 6572676f5f7461736b5f717565756572; asc ergo_task_queuer;;
1: len 6; hex 000000000000; asc ;;
2: len 7; hex 80000000000000; asc ;;
3: len 1; hex 80; asc ;;
4: len 1; hex 80; asc ;;
5: len 1; hex 81; asc ;;
6: SQL NULL;
7: SQL NULL;
8: SQL NULL;
9: SQL NULL;
10: SQL NULL;
11: len 29; hex 2f6f70742f616972666c6f772f646167732f6461675f6572676f2e7079;
asc /opt/airflow/dags/dag_ergo.py;;
12: len 7; hex 616972666c6f77; asc airflow;;
13: SQL NULL;
14: len 4; hex 74726565; asc tree;;
15: len 30; hex
7b2274797065223a202274696d6564656c7461222c20226174747273223a; asc {"type":
"timedelta", "attrs":; (total 77 bytes);
16: SQL NULL;
17: len 7; hex 5f9893e60e6fa0; asc _ o ;;
18: SQL NULL;
19: len 4; hex 80000010; asc ;;
20: len 1; hex 80; asc ;;
Record lock, heap no 97 PHYSICAL RECORD: n_fields 21; compact format; info
bits 0
0: len 18; hex 6572676f5f6a6f625f636f6c6c6563746f72; asc
ergo_job_collector;;
1: len 6; hex 000100d29cc5; asc ;;
2: len 7; hex 420006001209ca; asc B ;;
3: len 1; hex 80; asc ;;
4: len 1; hex 80; asc ;;
5: len 1; hex 81; asc ;;
6: SQL NULL;
7: SQL NULL;
8: SQL NULL;
9: SQL NULL;
10: SQL NULL;
11: len 29; hex 2f6f70742f616972666c6f772f646167732f6461675f6572676f2e7079;
asc /opt/airflow/dags/dag_ergo.py;;
12: len 7; hex 616972666c6f77; asc airflow;;
13: SQL NULL;
14: len 4; hex 74726565; asc tree;;
15: len 30; hex
7b2274797065223a202274696d6564656c7461222c20226174747273223a; asc {"type":
"timedelta", "attrs":; (total 77 bytes);
16: SQL NULL;
17: len 7; hex 5f9893e102c57c; asc _ |;;
18: len 7; hex 5f9893eb02c57c; asc _ |;;
19: len 4; hex 80000010; asc ;;
20: len 1; hex 80; asc ;;
*** (2) WAITING FOR THIS LOCK TO BE GRANTED:
RECORD LOCKS space id 5563 page no 5 n bits 168 index
idx_next_dagrun_create_after of table `chronos_airflow`.`dag` trx id 4308769989
lock_mode X locks rec but not gap waiting
Record lock, heap no 101 PHYSICAL RECORD: n_fields 2; compact format; info
bits 0
0: len 7; hex 5f9893ea058c2f; asc _ /;;
1: len 18; hex 6572676f5f6a6f625f636f6c6c6563746f72; asc
ergo_job_collector;;
*** WE ROLL BACK TRANSACTION (1)
```
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
[email protected]
