GitHub user eajimmy added a comment to the discussion: dagProcessor liveness
probe always fails
Same problem here on Airflow 3.2.1 (KubernetesExecutor + CeleryExecutor).
I confirmed it with this query, running it repeatedly:
```sql
SELECT id, hostname, state,
latest_heartbeat,
EXTRACT(EPOCH FROM (now() - latest_heartbeat)) AS age_seconds
FROM job
WHERE job_type = 'DagProcessorJob'
ORDER BY latest_heartbeat DESC
LIMIT 30;
```
age_seconds oscillates continuously between ~30 and ~60s and never drops close
to 0.
Setting AIRFLOW__DAG_PROCESSOR__HEALTH_CHECK_THRESHOLD=120 fixes it, as
mentioned above.
I also tried increasing parsing_processes and min_file_process_interval, but
these had no effect on age_seconds.
It seems that the staleness is intrinsic to the heartbeat mechanism, not to
parsing load.
I even confirmed this with the processor fully idle
(min_file_process_interval=300, no files to parse) and the age still stayed in
the 30–60s range.
GitHub link:
https://github.com/apache/airflow/discussions/69667#discussioncomment-17588039
----
This is an automatically sent email for [email protected].
To unsubscribe, please send an email to: [email protected]
