GitHub user pedro-cf edited a discussion: Best practices for restarting Airflow
while there are DAG Runs in progress
I'm looking for the correct/recommended way to handle Airflow restarts when
there are DAG runs in progress. Currently, I'm doing this:
1. Save current DAG pause states
2. Pause all DAGs
3. Change running DAG runs to queued state
4. Restart Airflow
5. Restore original DAG pause states
My current approach using CLI:
```bash
# Save states and pause all
airflow dags list --output json | jq -r '.[] | "\(.dag_id)|\(.paused)"' >
/tmp/dag_states.txt
airflow dags pause --dag-id all
# Change state of 'running' Dag Runs to 'failed' and then 'queued'
airflow dags list --output json | jq -r '.[] | .dag_id' | while read dag_id; do
airflow dags show "$dag_id" --output json | \
jq -r '.[] | select(.state=="running") | "\(.dag_id)|\(.run_id)"' | \
while IFS="|" read -r dag run_id; do
airflow dags backfill --reset-dagruns \
--start-date $(date -d "1 day ago" +%Y-%m-%d) \
--end-date $(date +%Y-%m-%d) \
"$dag"
done
done
# Restart happens here
# Restore states
while IFS="|" read -r dag_id paused; do
if [ "$paused" = "false" ]; then
airflow dags unpause --dag_id "$dag_id"
fi
done < /tmp/dag_states.txt
```
## Questions:
1. Is this the recommended approach?
2. Any potential issues with this approach?
3. Are there built-in Airflow features I should be using instead?
## Environment:
- Airflow version: 2.11.0
- Executor: CeleryExecutor
Thanks in advance for any guidance!
GitHub link: https://github.com/apache/airflow/discussions/50965
----
This is an automatically sent email for [email protected].
To unsubscribe, please send an email to: [email protected]
