codecae commented on issue #64877: URL: https://github.com/apache/airflow/issues/64877#issuecomment-4345007109
attempted to switch to gunicorn for more resiliency and get the following in the logs when things fall apart: 2026-04-29T14:47:47.020086Z [info [] request finished [http.access] client_addr=10.195.69.34:29323 duration_us=2132954 loc=http_access_log.py:98 method=PUT path=/execution/task-instances/019dd8a6-963c-7740-bf17-d76befb1369c/heartbeat query= status_code=204 2026-04-29T14:49:02.747620Z [critical [] WORKER TIMEOUT (pid:211) [gunicorn.error] loc=glogging.py:268 2026-04-29T14:49:03.841194Z [error [] Worker (pid:211) was sent SIGKILL! Perhaps out of memory? [gunicorn.error] loc=glogging.py:271 2026-04-29T14:49:03.849707Z [info [] Control socket listening at /home/airflow/.gunicorn/gunicorn.ctl [gunicorn.error] loc=glogging.py:277 The pods are not out of memory. they have plenty of available compute. When this occurs, a domino effect begins. the workers can't communicate so they eventually die off as well killing all active tasks that may have been running at the time. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected]
