derbeneviv opened a new issue, #33894:
URL: https://github.com/apache/airflow/issues/33894
### Apache Airflow version
Other Airflow 2 version (please specify below)
### What happened
# Versions:
* Apache airflow 2.5.3 (can't update to later right now)
* apache-airflow-providers-amazon 8.6.0
I want to connect logs to minio s3 bucket.
for this i create connection:
```
{
"conn_type": "aws",
"description": "connection for log storage",
"login": "Access Key",
"password": "Secret Key",
"extra": {
"endpoint_url": "https://my_minio";,
"verify": "False"
}
}
```
minio is accessible by HTTPS endpoint.
airflow.cfg:
```
base_log_folder = /var/airflow/logs
remote_logging = True
remote_base_log_folder = s3://airflow-prod-logs/logs
remote_log_conn_id = minio_logs
encrypt_s3_logs = False
```
and if i try to get DAG logs, i'm getting the following:
```
Aug 29 18:46:22 host airflow[19441]: [2023-08-29 18:46:22,237] {app.py:1741}
ERROR - Exception on /get_logs_with_metadata [GET]
Aug 29 18:46:22 host airflow[19441]: Traceback (most recent call last):
Aug 29 18:46:22 host airflow[19441]: File
"/usr/local/lib/python3.10/site-packages/urllib3/util/ssl_.py", line 405, in
ssl_wrap_socket
Aug 29 18:46:22 host airflow[19441]: context.load_verify_locations(ca_certs,
ca_cert_dir, ca_cert_data)
Aug 29 18:46:22 host airflow[19441]: FileNotFoundError: [Errno 2] No such
file or directory
Aug 29 18:46:22 host airflow[19441]: During handling of the above exception,
another exception occurred:
Aug 29 18:46:22 host airflow[19441]: Traceback (most recent call last):
Aug 29 18:46:22 host airflow[19441]: File
"/usr/local/lib/python3.10/site-packages/botocore/httpsession.py", line 465, in
send
Aug 29 18:46:22 host airflow[19441]: urllib_response = conn.urlopen(
Aug 29 18:46:22 host airflow[19441]: File
"/usr/local/lib/python3.10/site-packages/urllib3/connectionpool.py", line 787,
in urlopen
Aug 29 18:46:22 host airflow[19441]: retries = retries.increment(
Aug 29 18:46:22 host airflow[19441]: File
"/usr/local/lib/python3.10/site-packages/urllib3/util/retry.py", line 525, in
increment
Aug 29 18:46:22 host airflow[19441]: raise six.reraise(type(error), error,
_stacktrace)
Aug 29 18:46:22 host airflow[19441]: File
"/usr/local/lib/python3.10/site-packages/urllib3/packages/six.py", line 770, in
reraise
Aug 29 18:46:22 host airflow[19441]: raise value
Aug 29 18:46:22 host airflow[19441]: File
"/usr/local/lib/python3.10/site-packages/urllib3/connectionpool.py", line 703,
in urlopen
Aug 29 18:46:22 host airflow[19441]: httplib_response = self._make_request(
Aug 29 18:46:22 host airflow[19441]: File
"/usr/local/lib/python3.10/site-packages/urllib3/connectionpool.py", line 386,
in _make_request
Aug 29 18:46:22 host airflow[19441]: self._validate_conn(conn)
Aug 29 18:46:22 host airflow[19441]: File
"/usr/local/lib/python3.10/site-packages/urllib3/connectionpool.py", line 1042,
in _validate_conn
Aug 29 18:46:22 host airflow[19441]: conn.connect()
Aug 29 18:46:22 host airflow[19441]: File
"/usr/local/lib/python3.10/site-packages/urllib3/connection.py", line 414, in
connect
Aug 29 18:46:22 host airflow[19441]: self.sock = ssl_wrap_socket(
Aug 29 18:46:22 host airflow[19441]: File
"/usr/local/lib/python3.10/site-packages/urllib3/util/ssl_.py", line 407, in
ssl_wrap_socket
Aug 29 18:46:22 host airflow[19441]: raise SSLError(e)
Aug 29 18:46:22 host airflow[19441]: urllib3.exceptions.SSLError: [Errno 2]
No such file or directory
Aug 29 18:46:22 host airflow[19441]: During handling of the above exception,
another exception occurred:
Aug 29 18:46:22 host airflow[19441]: Traceback (most recent call last):
Aug 29 18:46:22 host airflow[19441]: File
"/usr/local/lib/python3.10/site-packages/flask/app.py", line 2525, in wsgi_app
Aug 29 18:46:22 host airflow[19441]: response = self.full_dispatch_request()
Aug 29 18:46:22 host airflow[19441]: File
"/usr/local/lib/python3.10/site-packages/flask/app.py", line 1822, in
full_dispatch_request
Aug 29 18:46:22 host airflow[19441]: rv = self.handle_user_exception(e)
Aug 29 18:46:22 host airflow[19441]: File
"/usr/local/lib/python3.10/site-packages/flask/app.py", line 1820, in
full_dispatch_request
Aug 29 18:46:22 host airflow[19441]: rv = self.dispatch_request()
Aug 29 18:46:22 host airflow[19441]: File
"/usr/local/lib/python3.10/site-packages/flask/app.py", line 1796, in
dispatch_request
Aug 29 18:46:22 host airflow[19441]: return
self.ensure_sync(self.view_functions[rule.endpoint])(**view_args)
Aug 29 18:46:22 host airflow[19441]: File
"/usr/local/lib/python3.10/site-packages/airflow/www/auth.py", line 47, in
decorated
Aug 29 18:46:22 host airflow[19441]: return func(*args, **kwargs)
Aug 29 18:46:22 host airflow[19441]: File
"/usr/local/lib/python3.10/site-packages/airflow/www/decorators.py", line 125,
in wrapper
Aug 29 18:46:22 host airflow[19441]: return f(*args, **kwargs)
Aug 29 18:46:22 host airflow[19441]: File
"/usr/local/lib/python3.10/site-packages/airflow/utils/session.py", line 75, in
wrapper
Aug 29 18:46:22 host airflow[19441]: return func(*args, session=session,
**kwargs)
Aug 29 18:46:22 host airflow[19441]: File
"/usr/local/lib/python3.10/site-packages/airflow/www/views.py", line 1480, in
get_logs_with_metadata
Aug 29 18:46:22 host airflow[19441]: logs, metadata =
task_log_reader.read_log_chunks(ti, try_number, metadata)
Aug 29 18:46:22 host airflow[19441]: File
"/usr/local/lib/python3.10/site-packages/airflow/utils/log/log_reader.py", line
58, in read_log_chunks
Aug 29 18:46:22 host airflow[19441]: logs, metadatas =
self.log_handler.read(ti, try_number, metadata=metadata)
Aug 29 18:46:22 host airflow[19441]: File
"/usr/local/lib/python3.10/site-packages/airflow/utils/log/file_task_handler.py",
line 306, in read
Aug 29 18:46:22 host airflow[19441]: log, out_metadata =
self._read(task_instance, try_number_element, metadata)
Aug 29 18:46:22 host airflow[19441]: File
"/usr/local/lib/python3.10/site-packages/airflow/providers/amazon/aws/log/s3_task_handler.py",
line 151, in _read
Aug 29 18:46:22 host airflow[19441]: messages, logs =
self._read_remote_logs(ti, try_number, metadata)
Aug 29 18:46:22 host airflow[19441]: File
"/usr/local/lib/python3.10/site-packages/airflow/providers/amazon/aws/log/s3_task_handler.py",
line 123, in _read_remote_logs
Aug 29 18:46:22 host airflow[19441]: keys =
self.hook.list_keys(bucket_name=bucket, prefix=prefix)
Aug 29 18:46:22 host airflow[19441]: File
"/usr/local/lib/python3.10/site-packages/airflow/providers/amazon/aws/hooks/s3.py",
line 92, in wrapper
Aug 29 18:46:22 host airflow[19441]: return func(*bound_args.args,
**bound_args.kwargs)
Aug 29 18:46:22 host airflow[19441]: File
"/usr/local/lib/python3.10/site-packages/airflow/providers/amazon/aws/hooks/s3.py",
line 822, in list_keys
Aug 29 18:46:22 host airflow[19441]: for page in response:
Aug 29 18:46:22 host airflow[19441]: File
"/usr/local/lib/python3.10/site-packages/botocore/paginate.py", line 269, in
__iter__
Aug 29 18:46:22 host airflow[19441]: response =
self._make_request(current_kwargs)
Aug 29 18:46:22 host airflow[19441]: File
"/usr/local/lib/python3.10/site-packages/botocore/paginate.py", line 357, in
_make_request
Aug 29 18:46:22 host airflow[19441]: return self._method(**current_kwargs)
Aug 29 18:46:22 host airflow[19441]: File
"/usr/local/lib/python3.10/site-packages/botocore/client.py", line 535, in
_api_call
Aug 29 18:46:22 host airflow[19441]: return
self._make_api_call(operation_name, kwargs)
Aug 29 18:46:22 host airflow[19441]: File
"/usr/local/lib/python3.10/site-packages/botocore/client.py", line 963, in
_make_api_call
Aug 29 18:46:22 host airflow[19441]: http, parsed_response =
self._make_request(
Aug 29 18:46:22 host airflow[19441]: File
"/usr/local/lib/python3.10/site-packages/botocore/client.py", line 986, in
_make_request
Aug 29 18:46:22 host airflow[19441]: return
self._endpoint.make_request(operation_model, request_dict)
Aug 29 18:46:22 host airflow[19441]: File
"/usr/local/lib/python3.10/site-packages/botocore/endpoint.py", line 119, in
make_request
Aug 29 18:46:22 host airflow[19441]: return self._send_request(request_dict,
operation_model)
Aug 29 18:46:22 host airflow[19441]: File
"/usr/local/lib/python3.10/site-packages/botocore/endpoint.py", line 202, in
_send_request
Aug 29 18:46:22 host airflow[19441]: while self._needs_retry(
Aug 29 18:46:22 host airflow[19441]: File
"/usr/local/lib/python3.10/site-packages/botocore/endpoint.py", line 354, in
_needs_retry
Aug 29 18:46:22 host airflow[19441]: responses = self._event_emitter.emit(
Aug 29 18:46:22 host airflow[19441]: File
"/usr/local/lib/python3.10/site-packages/botocore/hooks.py", line 412, in emit
Aug 29 18:46:22 host airflow[19441]: return
self._emitter.emit(aliased_event_name, **kwargs)
Aug 29 18:46:22 host airflow[19441]: File
"/usr/local/lib/python3.10/site-packages/botocore/hooks.py", line 256, in emit
Aug 29 18:46:22 host airflow[19441]: return self._emit(event_name, kwargs)
Aug 29 18:46:22 host airflow[19441]: File
"/usr/local/lib/python3.10/site-packages/botocore/hooks.py", line 239, in _emit
Aug 29 18:46:22 host airflow[19441]: response = handler(**kwargs)
Aug 29 18:46:22 host airflow[19441]: File
"/usr/local/lib/python3.10/site-packages/botocore/retryhandler.py", line 207,
in __call__
Aug 29 18:46:22 host airflow[19441]: if self._checker(**checker_kwargs):
Aug 29 18:46:22 host airflow[19441]: File
"/usr/local/lib/python3.10/site-packages/botocore/retryhandler.py", line 284,
in __call__
Aug 29 18:46:22 host airflow[19441]: should_retry = self._should_retry(
Aug 29 18:46:22 host airflow[19441]: File
"/usr/local/lib/python3.10/site-packages/botocore/retryhandler.py", line 320,
in _should_retry
Aug 29 18:46:22 host airflow[19441]: return self._checker(attempt_number,
response, caught_exception)
Aug 29 18:46:22 host airflow[19441]: File
"/usr/local/lib/python3.10/site-packages/botocore/retryhandler.py", line 363,
in __call__
Aug 29 18:46:22 host airflow[19441]: checker_response = checker(
Aug 29 18:46:22 host airflow[19441]: File
"/usr/local/lib/python3.10/site-packages/botocore/retryhandler.py", line 247,
in __call__
Aug 29 18:46:22 host airflow[19441]: return self._check_caught_exception(
Aug 29 18:46:22 host airflow[19441]: File
"/usr/local/lib/python3.10/site-packages/botocore/retryhandler.py", line 416,
in _check_caught_exception
Aug 29 18:46:22 host airflow[19441]: raise caught_exception
Aug 29 18:46:22 host airflow[19441]: File
"/usr/local/lib/python3.10/site-packages/botocore/endpoint.py", line 281, in
_do_get_response
Aug 29 18:46:22 host airflow[19441]: http_response = self._send(request)
Aug 29 18:46:22 host airflow[19441]: File
"/usr/local/lib/python3.10/site-packages/botocore/endpoint.py", line 377, in
_send
Aug 29 18:46:22 host airflow[19441]: return self.http_session.send(request)
Aug 29 18:46:22 host airflow[19441]: File
"/usr/local/lib/python3.10/site-packages/botocore/httpsession.py", line 492, in
send
Aug 29 18:46:22 host airflow[19441]: raise
SSLError(endpoint_url=request.url, error=e)
Aug 29 18:46:22 host airflow[19441]: botocore.exceptions.SSLError: SSL
validation failed for
https://my_minio/airflow-prod-logs?list-type=2&prefix=logs%2Fdag_id%3Dexample_kubernetes_executor%2Frun_id%3Dmanual__2023-08-29T15%3A24%3A30.546563%2B00%3A00%2Ftask_id%3Dcompute_task%2Fattempt%3D1.log&delimiter=&start-after=&encoding-type=url
[Errno 2] No such file or directory
```
after some digging i found that my *ca_certs* variable is set to "False" at
urllib3/util/ssl_.py:402
Setting "verify": "<path_to_ca_file> " solves the issue, the same as
removing verify at all and adding SSL_CERT_FILE env variable
### What you think should happen instead
airflow S3 hook should respect boolean variant of verify
https://github.com/apache/airflow/blob/main/airflow/providers/amazon/aws/hooks/s3.py#L315
as stated in botocore
https://github.com/boto/boto3/blob/develop/boto3/session.py#L348
### How to reproduce
* add aws connection with custom endpoint_url and verify: False
```
{
"conn_type": "aws",
"description": "connection for log storage",
"login": "Access Key",
"password": "Secret Key",
"extra": {
"endpoint_url": "https://my_minio";,
"verify": "False"
}
}
```
* try to read DAG logs
### Operating System
centos 7
### Versions of Apache Airflow Providers
_No response_
### Deployment
Virtualenv installation
### Deployment details
_No response_
### Anything else
_No response_
### Are you willing to submit PR?
- [ ] Yes I am willing to submit a PR!
### Code of Conduct
- [X] I agree to follow this project's [Code of
Conduct](https://github.com/apache/airflow/blob/main/CODE_OF_CONDUCT.md)
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscr...@airflow.apache.org.apache.org
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
