NicolasBearingPoint opened a new issue, #50491:
URL: https://github.com/apache/airflow/issues/50491
### Apache Airflow version
2.10.5
### If "Other Airflow 2 version" selected, which one?
_No response_
### What happened?
I'm encountering an issue when deserializing XComs using a custom XCom
backend that loads data from S3 using `S3Hook(aws_conn_id='aws')`. `aws` is the
connection Id defined with the aws credentials. The same code is working for
Airflow 2.10.4 but not anymore from 2.10.5 and further.
Despite having a valid AWS connection configured in Airflow (visible and
**working well** in **ALL** other parts of the DAG), I get the following error
during deserialization steps:
```botocore.exceptions.NoCredentialsError: Unable to locate credentials````
With the function:
```
hook = S3Hook(aws_conn_id='aws')
head_object = hook.head_object(key=key,
bucket_name=CustomXComBackendPandas.get_bucket_name())
```
This error is not visible in the Airflow UI but I found it in the scheduler
container's logs. The branching task is set as success but the following
branched task is queued 'forever'.
After switching to manual session initialization using `boto3.Session(...)`,
I still have the same error...
### What you think should happen instead?
The 'aws' connection is not resolved correctly anymore by the scheduler,
even though it is defined in Airflow.
### How to reproduce
```
class CustomXComBackendPandas(BaseXCom):
XCOM_PREFIX = "xcom_s3://"
S3_PATH_PREFIX = "tmp"
_BUCKET_NAME = ""
DATAFRAME_EXTENSION = ".parquet"
@staticmethod
def deserialize_value(result: BaseXCom) -> Any:
hook = S3Hook(aws_conn_id='aws')
head_object = hook.head_object(key='key', bucket_name='bucket')
```
### Operating System
macOS 15.4.1 (Sonoma)
### Versions of Apache Airflow Providers
_No response_
### Deployment
Astronomer
### Deployment details
_No response_
### Anything else?
_No response_
### Are you willing to submit PR?
- [ ] Yes I am willing to submit a PR!
### Code of Conduct
- [x] I agree to follow this project's [Code of
Conduct](https://github.com/apache/airflow/blob/main/CODE_OF_CONDUCT.md)
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: [email protected]
For queries about this service, please contact Infrastructure at:
[email protected]
