basantjamwal commented on issue #76573: URL: https://github.com/apache/airflow/issues/76573#issuecomment-6679208867
Hi @opensky2023, the heartbeat kill looks fixed on `main` by #73244 (in the 3.4.0 release candidates, not in 3.2.x). It raised `[workers] max_failed_heartbeats` from `3` to `30`. With the default `min_heartbeat_interval` of 5s, that is about 150s of API downtime tolerated instead of about 15s, which fits your 1.5 to 2 minute API restart. **Workaround on 3.2.2:** ```ini [workers] max_failed_heartbeats = 30 ``` Keep `5 * max_failed_heartbeats` below `[scheduler] task_instance_heartbeat_timeout` (default 300s). The Celery 24h switch looks related to #54479 (closed as fixed). If it still happens on a newer version, please share real worker and scheduler logs from around the 24h mark. The api-server pool exhaustion is a separate problem and would be better as its own issue. Could you retest with the workaround? -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected]
