gopidesupavan commented on issue #52501:
URL: https://github.com/apache/airflow/issues/52501#issuecomment-3071031843
> I think I am having a bit of the same issue with remote logging to Minio
S3. The tasks are completing successfully but aren't saving the logs to S3. We
are running Airflow in k8s using the KubernetesExecutor. I have an aws
connection named Minio for our on prem S3 instance. I have the connection using
the extra fields json section since we have to point to Minio endpoint and not
AWS. This same setup works fine in 2.9.3 instance of Airflow but doesn't in
3.0.2. What is weird is that if I create a task that uses S3Hook it has no
issue finding the Minio connection when looking at the logs on the k8s pods.
The pod also then gives the same error about not the connection id.
>
> ```
> {"timestamp":"2025-07-14T19:56:48.737458","level":"info","event":"Filling
up the DagBag from
/usr/local/airflow/dags/test_dag.py","logger":"airflow.models.dagbag.DagBag"}
> {"timestamp":"2025-07-14T19:56:49.741806Z","level":"info","event":"hello
world!","chan":"stdout","logger":"supervisor"}
>
{"timestamp":"2025-07-14T19:56:49.741424","level":"info","event":"Connection
Retrieved 'Minio'","logger":"airflow.hooks.base"}
> {"timestamp":"2025-07-14T19:56:49.741805","level":"info","event":"AWS
Connection (conn_id='Minio', conn_type='aws') credentials retrieved from login
and
password.","logger":"airflow.providers.amazon.aws.utils.connection_wrapper.AwsConnectionWrapper"}
>
{"timestamp":"2025-07-14T19:56:50.676600Z","level":"info","event":"['test/test.csv']","chan":"stdout","logger":"supervisor"}
> {"timestamp":"2025-07-14T19:56:50.678866","level":"info","event":"Done.
Returned value was:
None","logger":"airflow.task.operators.airflow.providers.standard.operators.python.PythonOperator"}
> {"event":"Unable to find AWS Connection ID 'Minio', switching to
empty.","level":"warning","logger":"airflow.task.hooks.airflow.providers.amazon.aws.hooks.s3.S3Hook","timestamp":"2025-07-14T19:56:50.733131Z"}
> {"event":"No connection ID provided. Fallback on boto3 credential strategy
(region_name=None). See:
https://boto3.amazonaws.com/v1/documentation/api/latest/guide/configuration.html","level":"info","logger":"airflow.providers.amazon.aws.hooks.base_aws.BaseSessionFactory","timestamp":"2025-07-14T19:56:50.733774Z"}
> {"event":"Could not verify previous log to
append","level":"error","logger":"airflow.providers.amazon.aws.log.s3_task_handler.S3RemoteLogIO","timestamp":"2025-07-14T19:56:53.629692Z","exception":[{"exc_type":"NoCredentialsError","exc_value":"Unable
to locate
credentials","exc_notes":[],"syntax_error":null,"is_cause":false,"frames":[{"filename":"/usr/local/lib/python3.12/site-packages/airflow/providers/amazon/aws/log/s3_task_handler.py","lineno":120,"name":"write"},{"filename":"/usr/local/lib/python3.12/site-packages/airflow/providers/amazon/aws/log/s3_task_handler.py","lineno":81,"name":"s3_log_exists"},{"filename":"/usr/local/lib/python3.12/site-packages/airflow/providers/amazon/aws/hooks/s3.py","lineno":153,"name":"wrapper"},{"filename":"/usr/local/lib/python3.12/site-packages/airflow/providers/amazon/aws/hooks/s3.py","lineno":126,"name":"wrapper"},{"filename":"/usr/local/lib/python3.12/site-packages/airflow/providers/amazon/aws/hooks/s3.py","lineno":953,"name":"check_for_key"
},{"filename":"/usr/local/lib/python3.12/site-packages/airflow/providers/amazon/aws/hooks/s3.py","lineno":153,"name":"wrapper"},{"filename":"/usr/local/lib/python3.12/site-packages/airflow/providers/amazon/aws/hooks/s3.py","lineno":126,"name":"wrapper"},{"filename":"/usr/local/lib/python3.12/site-packages/airflow/providers/amazon/aws/hooks/s3.py","lineno":934,"name":"head_object"},{"filename":"/usr/local/lib/python3.12/site-packages/botocore/client.py","lineno":569,"name":"_api_call"},{"filename":"/usr/local/lib/python3.12/site-packages/botocore/client.py","lineno":1005,"name":"_make_api_call"},{"filename":"/usr/local/lib/python3.12/site-packages/botocore/client.py","lineno":1029,"name":"_make_request"},{"filename":"/usr/local/lib/python3.12/site-packages/botocore/endpoint.py","lineno":119,"name":"make_request"},{"filename":"/usr/local/lib/python3.12/site-packages/botocore/endpoint.py","lineno":196,"name":"_send_request"},{"filename":"/usr/local/lib/python3.12/site-packages/botocore
/endpoint.py","lineno":132,"name":"create_request"},{"filename":"/usr/local/lib/python3.12/site-packages/botocore/hooks.py","lineno":412,"name":"emit"},{"filename":"/usr/local/lib/python3.12/site-packages/botocore/hooks.py","lineno":256,"name":"emit"},{"filename":"/usr/local/lib/python3.12/site-packages/botocore/hooks.py","lineno":239,"name":"_emit"},{"filename":"/usr/local/lib/python3.12/site-packages/botocore/signers.py","lineno":106,"name":"handler"},{"filename":"/usr/local/lib/python3.12/site-packages/botocore/signers.py","lineno":198,"name":"sign"},{"filename":"/usr/local/lib/python3.12/site-packages/botocore/auth.py","lineno":424,"name":"add_auth"}],"is_group":false,"exceptions":[]}]}
> {"timestamp":"2025-07-14T19:56:53.630904Z","level":"info","event":"Task
finished","exit_code":0,"duration":6.419133752002381,"final_state":"success","logger":"supervisor"}
> ```
Airflow/AWS provider dont support explicitly minio.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: [email protected]
For queries about this service, please contact Infrastructure at:
[email protected]
