DominikBalga commented on issue #65708: URL: https://github.com/apache/airflow/issues/65708#issuecomment-4304290390
> We noticed the problem where heartbeat DB queries queue up. This makes workers timeout on the heartbeat request and retry. I can imagine that the failed heartbeat can put task instance into a terminal state (**but not sure though**). > > ``` > usename state xact_duration query_duration query > db_user idle in transaction 00:00:22.17721 00:00:00.06658 SELECT task_instance.state AS task_instance_state > FROM task_instance > WHERE task_instance.id = '0<zzzyyy>54'::uuid::UUID > db_user active 00:00:22.123495 00:00:22.120761 UPDATE task_instance SET last_heartbeat_at='2026-04-23T09:03:18.258390+00:00'::timestamptz, updated_at='2026-04-23T09:03:18.261693+00:00'::timestamptz WHERE task_instance.id > db_user active 00:00:22.058237 00:00:22.054038 UPDATE task_instance SET last_heartbeat_at='2026-04-23T09:03:18.320462+00:00'::timestamptz, updated_at='2026-04-23T09:03:18.324080+00:00'::timestamptz WHERE task_instance.id > db_user active 00:00:21.901896 00:00:21.89822 UPDATE task_instance SET last_heartbeat_at='2026-04-23T09:03:18.479643+00:00'::timestamptz, updated_at='2026-04-23T09:03:18.483318+00:00'::timestamptz WHERE task_instance.id > db_user active 00:00:21.537818 00:00:21.534485 UPDATE task_instance SET last_heartbeat_at='2026-04-23T09:03:18.844164+00:00'::timestamptz, updated_at='2026-04-23T09:03:18.847447+00:00'::timestamptz WHERE task_instance.id > ... > ``` after further investigation it seems that the issue we are experiencing is the combination of the idle transactions (i do see idle transactions up to 50 secs durararion as well) and issue described here https://github.com/apache/airflow/pull/63355 -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected]
