rjtshrm commented on issue #52501: URL: https://github.com/apache/airflow/issues/52501#issuecomment-3089760190
One more thing I found out that, for every workload execute_workload.py, always loads the `load_logging_config` from `logging_config.py`. As you could see the logs, I set a trace in `load_logging_config` method and its the first thing initialized. I guess that's how airflow initializes it's settings. https://github.com/apache/airflow/blob/5245f15b0649d9973e299930309faa26a7e2d592/airflow-core/src/airflow/__init__.py#L75-L79 https://github.com/apache/airflow/blob/5245f15b0649d9973e299930309faa26a7e2d592/airflow-core/src/airflow/settings.py#L605-L616 ```python 1 /home/airflow/.local/lib/python3.11/site-packages/airflow/configuration.py:858 DeprecationWarning: The secret_key option in [webserver] has been moved to the secret_key option in [api] - the old setting has been used, but please update your config. 2 Function: load_logging_config, File: /home/airflow/.local/lib/python3.11/site-packages/airflow/logging_config.py, Line: 49 3 Function: configure_logging, File: /home/airflow/.local/lib/python3.11/site-packages/airflow/logging_config.py, Line: 91 4 Function: initialize, File: /home/airflow/.local/lib/python3.11/site-packages/airflow/settings.py, Line: 610 5 Function: <module>, File: /home/airflow/.local/lib/python3.11/site-packages/airflow/__init__.py, Line: 79 6 Function: _call_with_frames_removed, File: <frozen importlib._bootstrap>, Line: 241 7 Function: exec_module, File: <frozen importlib._bootstrap_external>, Line: 940 8 Function: _load_unlocked, File: <frozen importlib._bootstrap>, Line: 690 9 Function: _find_and_load_unlocked, File: <frozen importlib._bootstrap>, Line: 1147 10 Function: _find_and_load, File: <frozen importlib._bootstrap>, Line: 1176 11 Function: _call_with_frames_removed, File: <frozen importlib._bootstrap>, Line: 241 12 Function: _find_and_load_unlocked, File: <frozen importlib._bootstrap>, Line: 1126 13 Function: _find_and_load, File: <frozen importlib._bootstrap>, Line: 1176 14 Function: _call_with_frames_removed, File: <frozen importlib._bootstrap>, Line: 241 15 Function: _find_and_load_unlocked, File: <frozen importlib._bootstrap>, Line: 1126 16 Function: _find_and_load, File: <frozen importlib._bootstrap>, Line: 1176 17 Function: _get_module_details, File: <frozen runpy>, Line: 112 18 Function: _run_module_as_main, File: <frozen runpy>, Line: 189 19 [2025-07-18T13:06:38.955+0000] {base_aws.py:632} WARNING - Unable to find AWS Connection ID 'MyCloudwatchConn', switching to empty. 20 [2025-07-18T13:06:38.956+0000] {base_aws.py:201} INFO - No connection ID provided. Fallback on boto3 credential strategy (region_name='eu-central-1'). See: https://boto3.amazonaws.com/v1/documentation/api/latest/guide/configuration.html ``` So I guess, that's the issue here. We are defining logging config two times, one by default with Airflow initialization and other time in execute_workload.py which should be only case if it's not initialized before. And the second time, it's also not picking the airflow configuration but setting default loggers and handlers and clearing existing handlers. I managed to get it working, you can see my changes https://github.com/apache/airflow/pull/53499/files. I applied this patch in my airflow image. And logs seems to be working fine. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected]
