shivaam commented on issue #62244: URL: https://github.com/apache/airflow/issues/62244#issuecomment-3988264254
Reporting issue with dag bundles: @vincbeck also posted earlier here: https://github.com/apache/airflow/issues/62246#issuecomment-3986372844 When using dag_bundle_config_list with team_name in multi-team mode, the dag-processor's sync_bundles_to_db method logs "Added new DAG bundle" for team-scoped bundles but the records are not persisted to the dag_bundle table. Non-team bundles (like shared_dags and example_dags) persist correctly. The dag-processor then continuously logs "Bundle model not found" for the team bundles, and their DAGs are never processed. ``` Startup — all 4 bundles load and get "added": 02:23:06.165 [info] DAG bundles loaded: team_alpha_dags, team_beta_dags, shared_dags, example_dags 02:23:06.182 [info] Added new DAG bundle team_alpha_dags to the database 02:23:06.188 [info] Added new DAG bundle team_beta_dags to the database 02:23:06.192 [info] Added new DAG bundle shared_dags to the database 02:23:06.193 [info] Added new DAG bundle example_dags to the database Less than 1 second later — team bundles can't be found in the DB: 02:23:07.121 [warning] Bundle model not found for team_alpha_dags 02:23:07.997 [warning] Bundle model not found for team_beta_dags Meanwhile the non-team bundles work fine: 02:23:08.497 [info] Refreshing bundle shared_dags 02:23:08.522 [info] Searching for files in shared_dags at /tmp/airflow/dag_bundles/shared_dags 02:23:08.523 [info] Found 1 files for bundle shared_dags 02:23:08.539 [info] Refreshing bundle example_dags 02:23:08.558 [info] Found 61 files for bundle example_dags Then it repeats every ~5 seconds, never resolving — all the way to the end of the log at 02:32:02: 02:23:11.300 [warning] Bundle model not found for team_alpha_dags 02:23:11.306 [warning] Bundle model not found for team_beta_dags 02:23:16.552 [warning] Bundle model not found for team_alpha_dags 02:23:16.554 [warning] Bundle model not found for team_beta_dags ... ``` <img width="824" height="526" alt="Image" src="https://github.com/user-attachments/assets/b4d2cf59-41c0-4ea3-9be9-6a032342002d"; /> Config ``` executor = LocalExecutor;team_alpha=airflow.providers.amazon.aws.executors.batch.batch_executor.AwsBatchExecutor;team_beta=airflow.providers.amazon.aws.executors.batch.batch_executor.AwsBatchExecutor [dag_processor] dag_bundle_config_list = [{"name": "team_alpha_dags", "classpath": "airflow.providers.amazon.aws.bundles.s3.S3DagBundle", "kwargs": {"bucket_name": "airflow-ecs-dags-x-us-west-2", "prefix": "team_alpha"}, "team_name": "team_alpha"}, {"name": "team_beta_dags", "classpath": "airflow.providers.amazon.aws.bundles.s3.S3DagBundle", "kwargs": {"bucket_name": "airflow-ecs-dags-x-us-west-2", "prefix": "team_beta"}, "team_name": "team_beta"}, {"name": "shared_dags", "classpath": "airflow.providers.amazon.aws.bundles.s3.S3DagBundle", "kwargs": {"bucket_name": "airflow-ecs-dags-x-us-west-2", "prefix": "shared"}}] [team_alpha=aws_batch_executor] job_name = alpha-airflow-worker job_queue = alpha-batch-queue job_definition = alpha-batch-job-def region_name = us-west-2 [team_beta=aws_batch_executor] job_name = beta-airflow-worker job_queue = beta-batch-queue job_definition = beta-batch-job-def region_name = us-west-2 -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected]
