GitHub user enchant3dmango edited a comment on the discussion: Airflow ci/cd 
pipeline to rebuild airflow

If you want to pause DAGs in your CI/CD pipeline, you can use this 
[CLI](https://airflow.apache.org/docs/apache-airflow/stable/cli-and-env-variables-ref.html#pause).

It will be something like the following. You can use `docker compose up --build 
-d` instead of separating the build and restart command.
```yaml
rebuild_project:
  stage: rebuild
  script:
    - ssh ${DEPLOY_USER}@${DEPLOY_SERVER} << 'EOF'
      cd ${DEPLOY_PATH}

      # Pause all DAGs
      docker compose exec airflow-webserver airflow dags pause <regex-for-your 
dags> --treat-dag-id-as-regex

      # Wait until all tasks are finished
      while docker compose exec airflow-webserver airflow tasks list_instances 
| grep -q running; do
        echo "Waiting for running tasks to finish..."
        sleep 10
      done

      # Stop Airflow cleanly
      docker compose down

      # Rebuild and restart
      docker compose up --build -d

      # Unpause previously paused DAGs
      docker compose exec airflow-webserver airflow dags unpause 
<regex-for-your dags> --treat-dag-id-as-regex
    EOF
  rules:
    - if: '$CI_COMMIT_BRANCH == "main"'
      changes:
        - requirements.txt
        - start.sh
        - docker-compose.yaml
        - Dockerfile
        - plugins/**/*
        - config/**/*
```

GitHub link: 
https://github.com/apache/airflow/discussions/47149#discussioncomment-12340565

----
This is an automatically sent email for [email protected].
To unsubscribe, please send an email to: [email protected]

Reply via email to