GitHub user apoorva-01 added a comment to the discussion: Is HA mode for airflow dag-processor supported?
Short version: there's no active/standby HA for the dag-processor, and running two of them against the same DAGs isn't a redundancy setup, it's just duplicated parsing work. Two processors watching the same bundle both parse everything, and historically they'd even fight over deactivating each other's DAGs. The "make it horizontally scalable" ask is tracked in [#32966](https://github.com/apache/airflow/issues/32966), still open, and the PR that tried it got closed without merging. So no, not on the roadmap in the shipped-and-supported sense today. What you *can* do in Airflow 3 is shard by bundle. Split your DAGs into multiple bundles and run one processor per set: ``` airflow dag-processor -B bundle-a -B bundle-b airflow dag-processor -B bundle-c ``` `-B/--bundle-name` can be passed more than once, so each processor owns a disjoint slice. That scales parsing horizontally, it just doesn't give you a hot standby for any single bundle. For actual availability the practical answer is the boring one: run it under something that restarts it. On K8s a single-replica Deployment (or the official Helm chart's `dagProcessor`) will reschedule the pod if it dies. You lose parsing for the few seconds it takes to come back, but nothing gets scheduled off stale DAGs in the meantime, so it's usually fine. Just don't set that replica count to 2 on the same bundle expecting failover, you'll get double parsing, not redundancy. GitHub link: https://github.com/apache/airflow/discussions/69364#discussioncomment-17527333 ---- This is an automatically sent email for [email protected]. To unsubscribe, please send an email to: [email protected]
