GitHub user apoorva-01 closed the discussion with a comment: How does airflow find its compontents?
Your hunch is right. The metadata DB is the coordination point, there's no direct network discovery between components. Every long-running Airflow process (scheduler, triggerer, dag processor) writes a row into the `job` table (`airflow.jobs.job.Job`) when it starts up, with its hostname and a heartbeat timestamp, and then keeps bumping that timestamp on an interval. That's the whole "registry." Nobody tells the API server where the triggerer is. The API server just reads the `job` table and looks at `latest_triggerer_heartbeat`: if the newest one is within the threshold (30s by default) it shows healthy, if it's stale it shows down. Same mechanism for the scheduler. So they're not really talking to each other at all. They each connect to the same Postgres, write their own heartbeat, and read everyone else's rows. The DB is the message bus, which is exactly the entries you found. (Airflow 3 detail: workers no longer hit the DB directly, they go through the Task Execution API on the API server. But the scheduler/triggerer heartbeating into the `job` table is unchanged.) GitHub link: https://github.com/apache/airflow/discussions/68715#discussioncomment-17526346 ---- This is an automatically sent email for [email protected]. To unsubscribe, please send an email to: [email protected]
