psyphonix opened a new issue, #65729:
URL: https://github.com/apache/airflow/issues/65729
### Under which category would you file this issue?
Providers
### Apache Airflow version
3.2.1
### What happened and how to reproduce it?
We have picked up what seems to be a bug affecting the Airflow `triggerer`
component when using remote logging to Azure blob storage.
For tasks executed on `workers` these `attempt=1.log` files are succesfully
shipped (uploaded) to remote blob storage without issue.
However, for tasks executed in Deferred Operators via the `triggerer` these
form a unique log file such as `attempt=1.log.trigger.4117334.log` are not
being sent to the remote storage account, with the Triggerer pod throwing the
following error:
```
Traceback (most recent call last):
File
"/home/airflow/.local/lib/python3.13/site-packages/airflow/providers/microsoft/azure/log/wasb_task_handler.py",
line 170, in write
self.hook.load_string(log, self.wasb_container, remote_log_location,
overwrite=True)
~~~~~~~~~~~~~~~~~~~~~^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
File
"/home/airflow/.local/lib/python3.13/site-packages/airflow/providers/microsoft/azure/hooks/wasb.py",
line 386, in load_string
self.upload(
~~~~~~~~~~~^
container_name=container_name,
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
...<3 lines>...
**kwargs,
^^^^^^^^^
)
^
File
"/home/airflow/.local/lib/python3.13/site-packages/airflow/providers/microsoft/azure/hooks/wasb.py",
line 443, in upload
blob_client = self._get_blob_client(container_name, blob_name)
File
"/home/airflow/.local/lib/python3.13/site-packages/airflow/providers/microsoft/azure/hooks/wasb.py",
line 245, in _get_blob_client
return
self.blob_service_client.get_blob_client(container=container_name,
blob=blob_name)
^^^^^^^^^^^^^^^^^^^^^^^^
File
"/home/airflow/.local/lib/python3.13/site-packages/airflow/providers/microsoft/azure/hooks/wasb.py",
line 160, in blob_service_client
self._blob_service_client = self.get_conn()
~~~~~~~~~~~~~^^
File
"/home/airflow/.local/lib/python3.13/site-packages/airflow/providers/microsoft/azure/hooks/wasb.py",
line 170, in get_conn
conn = self.get_connection(self.conn_id)
File
"/home/airflow/.local/lib/python3.13/site-packages/airflow/sdk/bases/hook.py",
line 61, in get_connection
conn = Connection.get(conn_id)
File
"/home/airflow/.local/lib/python3.13/site-packages/airflow/sdk/definitions/connection.py",
line 250, in get
return _get_connection(conn_id)
File
"/home/airflow/.local/lib/python3.13/site-packages/airflow/sdk/execution_time/context.py",
line 177, in _get_connection
raise AirflowNotFoundException(f"The conn_id `{conn_id}` isn't defined")
airflow.sdk.exceptions.AirflowNotFoundException: The conn_id
`logs_to_azure_blob` isn't defined
```
**What we tried:**
- Confirmed that the `apache-airflow-providers-microsoft-azure=13.1.1` is
installed on the Triggerer component.
- when running `airflow connections get logs_to_azure_blob -o yaml` within
the Triggerer, it returns the connection details successfully.
- we attempted to explicitly add the following ENV VAR to the Triggerer
setting it to the `get_uri` encoded connection, but this too had no effect.
Pseudo example:
`AIRFLOW_CONN_LOGS_TO_AZURE_BLOB='wasb://my_storage_account:my_secret_key@'`
**Snippet of relevant Airflow configuration:**
```
Apache Airflow
version | 3.2.1
remote_base_log_folder | wasb
...
Providers info
apache-airflow-providers-microsoft-azure | 13.1.1
...
```
**airflow.cfg**
```
[logging]
remote_logging = True
worker_log_server_port = 8793
trigger_log_server_port = 8794
remote_base_log_folder = wasb
remote_log_conn_id = logs_to_azure_blob
task_log_reader = task
[azure_remote_logging]
remote_wasb_log_container = airflow-logs
[triggerer]
capacity = 1000
job_heartbeat_sec = 5
triggerer_health_check_threshold = 30
max_trigger_to_select_per_loop = 50
queues_enabled = False
```
### What you think should happen instead?
`attempt=1.log.trigger.4117334.log` type files generated by Triggers should
be able to upload to remote storage accounts based on remote_logging
configuration, without custom overhead.
As these logs are not currently being shipped to remote storage, they get
removed from the Airflow UI when inspecting task instance logs making debugging
and monitoring of deferred tasks / Triggers a challenge.
Any advise / support would be greatly appreciated.
Should you need any further information, please let us know.
### Operating System
Debian GNU/Linux 12 (bookworm)
### Deployment
Other 3rd-party Helm chart
### Apache Airflow Provider(s)
microsoft-azure
### Versions of Apache Airflow Providers
apache-airflow-providers-microsoft-azure | 13.1.1
### Official Helm Chart version
1.20.0 (latest released)
### Kubernetes Version
v1.32.4
### Helm Chart configuration
_No response_
### Docker Image customizations
_No response_
### Anything else?
_No response_
### Are you willing to submit PR?
- [x] Yes I am willing to submit a PR!
### Code of Conduct
- [x] I agree to follow this project's [Code of
Conduct](https://github.com/apache/airflow/blob/main/CODE_OF_CONDUCT.md)
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: [email protected]
For queries about this service, please contact Infrastructure at:
[email protected]
