cjj1120 commented on issue #42542:
URL: https://github.com/apache/airflow/issues/42542#issuecomment-2425479275

   # How to reproduce 
   1. Deploy Airflow with below helm chart. 
   2. Remove DAG file and push the changes. 
   3. Verify that latest changes is synced.
   
   The DAG is not deactivated, it has to be paused/ deleted manually from the 
UI. 
   
   # Operating System 
   Mac OS
   
   # Version of Apache Airflow Providers 
   apache-airflow==2.9.3
   apache-airflow-providers-postgres==5.11.3
   apache-airflow-providers-cncf-kubernetes==8.3.4
   
   # Deployment 
    [community helm chart](https://github.com/airflow-helm/charts/tree/main). 
   Testing is done locally with cluster set up with `kind`. 
   
   # Deployment details 
   
   Base Docker image: `apache/airflow:slim-2.9.3-python3.11`
   helm chart value file: 
   ```yaml
   ########################################
   ## CONFIG | Airflow Configs
   ########################################
   airflow:
     ## if we use legacy 1.10 airflow commands
     legacyCommands: false
   
     ## configs for the airflow container image
     ## [FAQ] 
https://github.com/airflow-helm/charts/blob/main/charts/airflow/docs/faq/configuration/airflow-version.md
     image:
       # repository: localhost:5000/airflow-custom
       # tag: latest
       repository: airflow-custom
       tag: latest
   
     ## the airflow executor type to use
     executor: KubernetesExecutor
   
     ## the fernet encryption key (sets `AIRFLOW__CORE__FERNET_KEY`)
     ## [FAQ] 
https://github.com/airflow-helm/charts/blob/main/charts/airflow/docs/faq/security/set-fernet-key.md
     ## [WARNING] change from default value to ensure security
     fernetKey: ""
   
     ## the secret_key for flask (sets `AIRFLOW__WEBSERVER__SECRET_KEY`)
     ## [FAQ] 
https://github.com/airflow-helm/charts/blob/main/charts/airflow/docs/faq/security/set-webserver-secret-key.md
     ## [WARNING] change from default value to ensure security
     webserverSecretKey: "THIS IS UNSAFE!"
   
     ## environment variables for airflow configs
     ## [FAQ] 
https://github.com/airflow-helm/charts/blob/main/charts/airflow/docs/faq/configuration/airflow-configs.md
     config:
       AIRFLOW__WEBSERVER__EXPOSE_CONFIG: "False"
       AIRFLOW__CORE__LOAD_EXAMPLES: "False"
       AIRFLOW__CORE__DEFAULT_TIMEZONE: "utc"    
       # AIRFLOW__KUBERNETES_EXECUTOR__DELETE_WORKER_PODS_ON_FAILURE: "False"
       AIRFLOW__SCHEDULER__CATCHUP_BY_DEFAULT: "False"
       AIRFLOW__SCHEDULER__DAG_DIR_LIST_INTERVAL: 120
       AIRFLOW__SCHEDULER__PARSING_CLEANUP_INTERVAL: 6
       AIRFLOW__SCHEDULER__STALE_DAG_THRESHOLD: 6
       AIRFLOW__SCHEDULER__TASK_QUEUED_TIMEOUT: 300
       AIRFLOW_VAR_ENVIRONMENT: "DEV"    
       AIRFLOW__LOGGING__LOGGING_LEVEL: "INFO"
       # AIRFLOW__LOGGING__DAG_PROCESSOR_MANAGER_LOG_STDOUT: "True"
       # AIRFLOW__LOGGING__DAG_PROCESSOR_LOG_TARGET: "stdout" # default fo 
files in scheduler pod. 
       AIRFLOW__WEBSERVER__BASE_URL: http://localhost:8080
       AIRFLOW__WEBSERVER__DEFAULT_WRAP: True
       AIRFLOW__WEBSERVER__ENABLE_PROXY_FIX: True
       AIRFLOW__METRICS__STATSD_ON: True
       AIRFLOW__METRICS__STATSD_HOST: statsd-exporter
       AIRFLOW__METRICS__STATSD_PORT: 9125
       AIRFLOW__METRICS__STATSD_PREFIX: airflow
   
     ## a list of users to create
     ## [FAQ] 
https://github.com/airflow-helm/charts/blob/main/charts/airflow/docs/faq/security/airflow-users.md
     users:
       - username: admin
         password: admin
         role: Admin
         email: [email protected]
         firstName: admin
         lastName: admin
   
     ## a list airflow connections to create
     ## [FAQ] 
https://github.com/airflow-helm/charts/blob/main/charts/airflow/docs/faq/dags/airflow-connections.md
     connections: []
   
     ## a list airflow variables to create
     ## [FAQ] 
https://github.com/airflow-helm/charts/blob/main/charts/airflow/docs/faq/dags/airflow-variables.md
     variables: []
   
     ## a list airflow pools to create
     ## [FAQ] 
https://github.com/airflow-helm/charts/blob/main/charts/airflow/docs/faq/dags/airflow-pools.md
     pools: []
   
     ## extra pip packages to install in airflow Pods
     ## [FAQ] 
https://github.com/airflow-helm/charts/blob/main/charts/airflow/docs/faq/configuration/extra-python-packages.md
     ## [WARNING] this feature is not recommended for production use, see docs
     extraPipPackages: []
   
     ## extra environment variables for the airflow Pods
     ## [FAQ] 
https://github.com/airflow-helm/charts/blob/main/charts/airflow/docs/faq/kubernetes/mount-environment-variables.md
     extraEnv: []
   
     ## extra VolumeMounts for the airflow Pods
     ## [FAQ] 
https://github.com/airflow-helm/charts/blob/main/charts/airflow/docs/faq/kubernetes/mount-persistent-volumes.md
     ## [FAQ] 
https://github.com/airflow-helm/charts/blob/main/charts/airflow/docs/faq/kubernetes/mount-files.md
     extraVolumeMounts: []
       # - name: airflow-logs-volume
       #   mountPath: /airflow-logs
   
     ## extra Volumes for the airflow Pods
     ## [FAQ] 
https://github.com/airflow-helm/charts/blob/main/charts/airflow/docs/faq/kubernetes/mount-persistent-volumes.md
     ## [FAQ] 
https://github.com/airflow-helm/charts/blob/main/charts/airflow/docs/faq/kubernetes/mount-files.md
     extraVolumes: []
       # - name: airflow-logs-volume
       #   persistentVolumeClaim:
       #     claimName: airflow-logs-claim
   
     ## configs generating the `pod_template.yaml` file for 
`AIRFLOW__KUBERNETES__POD_TEMPLATE_FILE`
     ## [NOTE] the `dags.gitSync` values will create a git-sync init-container 
in the pod
     ## [NOTE] the `airflow.extraPipPackages` will NOT be installed
     kubernetesPodTemplate:
   
       ## the full content of the pod-template file (as a string)
       ## [NOTE] all other `kubernetesPodTemplate.*` are disabled when this is 
set
       stringOverride: ""
   
       ## resource requests/limits for the Pod template "base" container
       ## [SPEC] 
https://kubernetes.io/docs/reference/generated/kubernetes-api/v1.29/#resourcerequirements-v1-core
       resources: {}
   
       ## extra pip packages to install in the Pod template
       ## [FAQ] 
https://github.com/airflow-helm/charts/blob/main/charts/airflow/docs/faq/configuration/extra-python-packages.md
       ## [WARNING] this feature is not recommended for production use, see docs
       extraPipPackages: []
   
       ## extra VolumeMounts for the Pod template
       ## [FAQ] 
https://github.com/airflow-helm/charts/blob/main/charts/airflow/docs/faq/kubernetes/mount-persistent-volumes.md
       ## [FAQ] 
https://github.com/airflow-helm/charts/blob/main/charts/airflow/docs/faq/kubernetes/mount-files.md
       extraVolumeMounts: []
   
       ## extra Volumes for the Pod template
       ## [FAQ] 
https://github.com/airflow-helm/charts/blob/main/charts/airflow/docs/faq/kubernetes/mount-persistent-volumes.md
       ## [FAQ] 
https://github.com/airflow-helm/charts/blob/main/charts/airflow/docs/faq/kubernetes/mount-files.md
       extraVolumes: []
   
   ###################################
   ## COMPONENT | Airflow Scheduler
   ###################################
   scheduler:
     ## the number of scheduler Pods to run
     replicas: 1
   
     ## resource requests/limits for the scheduler Pods
     ## [SPEC] 
https://kubernetes.io/docs/reference/generated/kubernetes-api/v1.29/#resourcerequirements-v1-core
     resources: {}
   
     ## configs for the log-cleanup sidecar of the scheduler
     ## [FAQ] 
https://github.com/airflow-helm/charts/blob/main/charts/airflow/docs/faq/monitoring/log-cleanup.md
     logCleanup:
       enabled: false
       retentionMinutes: 21600
   
     ## configs for the scheduler Pods' liveness probe
     ## [FAQ] 
https://github.com/airflow-helm/charts/blob/main/charts/airflow/docs/faq/monitoring/scheduler-liveness-probe.md
     livenessProbe:
       enabled: true
   
       ## configs for an additional check that ensures tasks are being created 
by the scheduler
       ## [FAQ] 
https://github.com/airflow-helm/charts/blob/main/charts/airflow/docs/faq/monitoring/scheduler-liveness-probe.md
       taskCreationCheck:
         enabled: false
         thresholdSeconds: 300
         schedulerAgeBeforeCheck: 180
   
   ###################################
   ## COMPONENT | Airflow Webserver
   ###################################
   web:
     ## the number of web Pods to run
     replicas: 1
   
     ## resource requests/limits for the web Pods
     ## [SPEC] 
https://kubernetes.io/docs/reference/generated/kubernetes-api/v1.29/#resourcerequirements-v1-core
     resources: {}
   
     ## configs for the Service of the web Pods
     service:
       type: ClusterIP
       externalPort: 8080
   
   
     # DEFAULT 
     ## configs generating the `webserver_config.py` file
     ## [FAQ] 
https://github.com/airflow-helm/charts/blob/main/charts/airflow/docs/faq/configuration/airflow-configs.md#webserver_configpy
     # webserverConfig:
     #   ## the full content of the `webserver_config.py` file (as a string)
     #   stringOverride: |
     #     from airflow import configuration as conf
     #     from flask_appbuilder.security.manager import AUTH_DB
         
     #     # the SQLAlchemy connection string
     #     SQLALCHEMY_DATABASE_URI = conf.get("core", "SQL_ALCHEMY_CONN")
         
     #     # use embedded DB for auth
     #     AUTH_TYPE = AUTH_DB
   
     # OVERWRITE   
     webserverConfig:
       ## if the `webserver_config.py` file is mounted
       ## - set to false if you wish to mount your own `webserver_config.py` 
file
       ##
       enabled: true
   
       ## the full content of the `webserver_config.py` file (as a string)
       ## - docs for Flask-AppBuilder security configs:
       ##   https://flask-appbuilder.readthedocs.io/en/latest/security.html
       ##
       ## ____ EXAMPLE _______________
       ##   stringOverride: |
       ##     from airflow import configuration as conf
       ##     from flask_appbuilder.security.manager import AUTH_DB
       ##
       ##     # the SQLAlchemy connection string
       ##     SQLALCHEMY_DATABASE_URI = conf.get('core', 'SQL_ALCHEMY_CONN')
       ##
       ##     # use embedded DB for auth
       ##     AUTH_TYPE = AUTH_DB
       ##
       stringOverride: ""
   
       ## the name of a Secret containing a `webserver_config.py` key
       ##
       existingSecret: ""
   
     ## the number of web Pods to run
     ## - if you set this >1 we recommend defining a `web.podDisruptionBudget`
     ##
   
   ###################################
   ## COMPONENT | Airflow Workers
   ###################################
   workers:
     ## if the airflow workers StatefulSet should be deployed
     enabled: false
   
   ###################################
   ## COMPONENT | Triggerer
   ###################################
   triggerer:
     ## if the airflow triggerer should be deployed
     enabled: true
   
     ## the number of triggerer Pods to run
     replicas: 1
   
     ## resource requests/limits for the triggerer Pods
     ## [SPEC] 
https://kubernetes.io/docs/reference/generated/kubernetes-api/v1.29/#resourcerequirements-v1-core
     resources: {}
   
     ## maximum number of triggers each triggerer will run at once (sets 
`AIRFLOW__TRIGGERER__DEFAULT_CAPACITY`)
     capacity: 1000
   
   ###################################
   ## COMPONENT | Flower
   ###################################
   flower:
     ## if the airflow flower UI should be deployed
     enabled: false
   
   ###################################
   ## CONFIG | Airflow Logs
   ###################################
   logs:
     ## the airflow logs folder
     path: /opt/airflow/logs
   
     ## configs for the logs PVC
     ## [FAQ] 
https://github.com/airflow-helm/charts/blob/main/charts/airflow/docs/faq/monitoring/log-persistence.md
     persistence:
       enabled: false
       # ## the name of your existing PersistentVolumeClaim
       # existingClaim: airflow-logs-claim
       # ## WARNING: as multiple pods will write logs, this MUST be 
ReadWriteMany
       # accessMode: ReadWriteMany
   
   ###################################
   ## CONFIG | Airflow DAGs
   ###################################
   dags:
     ## the airflow dags folder
     path: /opt/airflow/dags
   
     ## configs for the dags PVC
     ## [FAQ] 
https://github.com/airflow-helm/charts/blob/main/charts/airflow/docs/faq/dags/load-dag-definitions.md
     persistence:
       enabled: false
   
     ## configs for the git-sync sidecar
     ## [FAQ] 
https://github.com/airflow-helm/charts/blob/main/charts/airflow/docs/faq/dags/load-dag-definitions.md
     gitSync:
       enabled: true
       image:
         repository: registry.k8s.io/git-sync/git-sync
         tag: v3.6.9
         pullPolicy: IfNotPresent
         uid: 65533
         gid: 65533
       # SSH
       repo: "#REMOVED"
       repoSubPath: "dags"
       sshSecret: "#REMOVED"
       sshSecretKey: #REMOVED    
       branch: "main"
       revision: "HEAD"
       # depth: 1    
       maxFailures: 5
       syncWait: 15
       # "known_hosts" verification can be disabled by setting to ... 
       sshKnownHosts: ""         
       # # HTTP 
       # repo: https://github.com/cjj1120/local_k8s_airflow_dags.git
       # ## the name of a pre-created Secret with git http credentials
       # ##
       # httpSecret: "#REMOVED"
   
       # ## the key in `dags.gitSync.httpSecret` with your git username
       # ##
       # httpSecretUsernameKey: #REMOVED
   
       # ## the key in `dags.gitSync.httpSecret` with your git password/token
       # ##
       # httpSecretPasswordKey: #REMOVED
   
   ###################################
   ## CONFIG | Kubernetes Ingress
   ###################################
   ingress:
     ## if we should deploy Ingress resources
     ## [FAQ] 
https://github.com/airflow-helm/charts/blob/main/charts/airflow/docs/faq/kubernetes/ingress.md
     enabled: false
   
   ###################################
   ## CONFIG | Kubernetes ServiceAccount
   ###################################
   serviceAccount:
   
     ## if a Kubernetes ServiceAccount is created
     create: true
   
     ## the name of the ServiceAccount
     name: ""
   
     ## annotations for the ServiceAccount
     annotations: {}
   
   ###################################
   ## CONFIG | Kubernetes Extra Manifests
   ###################################
   
   ## a list of extra Kubernetes manifests that will be deployed alongside the 
chart
   ## [FAQ] 
https://github.com/airflow-helm/charts/blob/main/charts/airflow/docs/faq/kubernetes/extra-manifests.md
   extraManifests: []
   # extraManifests:
   #   - |
   #     apiVersion: apps/v1
   #     kind: Deployment
   #     metadata:
   #       name: {{ include "airflow.fullname" . }}-busybox
   #       labels:
   #         app: {{ include "airflow.labels.app" . }}
   #         component: busybox
   #         chart: {{ include "airflow.labels.chart" . }}
   #         release: {{ .Release.Name }}
   #         heritage: {{ .Release.Service }}
   #     spec:
   #       replicas: 1
   #       selector:
   #         matchLabels:
   #           app: {{ include "airflow.labels.app" . }}
   #           component: busybox
   #           release: {{ .Release.Name }}
   #       template:
   #         metadata:
   #           labels:
   #             app: {{ include "airflow.labels.app" . }}
   #             component: busybox
   #             release: {{ .Release.Name }}
   #         spec:
   #           containers:
   #             - name: busybox
   #               image: busybox:1.35
   #               command:
   #                 - "/bin/sh"
   #                 - "-c"
   #               args:
   #                 - |
   #                   ## to break the infinite loop when we receive SIGTERM
   #                   trap "exit 0" SIGTERM;
   #                   ## keep the container running (so people can `kubectl 
exec -it` into it)
   #                   while true; do
   #                     echo "I am alive...";
   #                     sleep 30;
   #                   done
   
   ###################################
   ## DATABASE | PgBouncer
   ###################################
   pgbouncer:
     ## if the pgbouncer Deployment is created
     ## [FAQ] 
https://github.com/airflow-helm/charts/blob/main/charts/airflow/docs/faq/database/pgbouncer.md
     enabled: false
   
     ## resource requests/limits for the pgbouncer Pods
     ## [SPEC] 
https://kubernetes.io/docs/reference/generated/kubernetes-api/v1.29/#resourcerequirements-v1-core
     resources: {}
   
     ## sets pgbouncer config: `auth_type`
     authType: md5
   
   ###################################
   ## DATABASE | Embedded Postgres
   ###################################
   postgresql:
     ## if the `stable/postgresql` chart is used
     ## [FAQ] 
https://github.com/airflow-helm/charts/blob/main/charts/airflow/docs/faq/database/embedded-database.md
     ## [WARNING] the embedded Postgres is NOT SUITABLE for production 
deployments of Airflow
     ## [WARNING] consider using an external database with `externalDatabase.*`
     enabled: false
     ## configs for the PVC of postgresql
     persistence:
       enabled: true
         
       storageClass: ""
       size: 8Gi
   
   ###################################
   ## DATABASE | External Database
   ###################################
   externalDatabase:
     type: postgres
     host: #REMOVED
     port: 5432
     database: airflow
     userSecret: #REMOVED
     userSecretKey: #REMOVED
     passwordSecret: #REMOVED
     passwordSecretKey: #REMOVED
   
   
   ###################################
   ## DATABASE | Embedded Redis
   ###################################
   redis:
     ## if the `stable/redis` chart is used
     enabled: false
   
   ###################################
   ## DATABASE | External Redis
   ###################################
   externalRedis:
     ## the host of the external redis
     ## [FAQ] 
https://github.com/airflow-helm/charts/blob/main/charts/airflow/docs/faq/database/external-redis.md
     host: localhost`
   
   ```
   


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: [email protected]

For queries about this service, please contact Infrastructure at:
[email protected]

Reply via email to