collinpowerkariman opened a new issue, #45475:
URL: https://github.com/apache/airflow/issues/45475
### Apache Airflow Provider(s)
celery
### Versions of Apache Airflow Providers
apache-airflow==2.10.4
apache-airflow-providers-amazon==9.1.0
apache-airflow-providers-celery==3.8.5
apache-airflow-providers-cncf-kubernetes==10.0.1
apache-airflow-providers-common-compat==1.2.2
apache-airflow-providers-common-io==1.4.2
apache-airflow-providers-common-sql==1.20.0
apache-airflow-providers-docker==3.14.1
apache-airflow-providers-elasticsearch==5.5.3
apache-airflow-providers-fab==1.5.1
apache-airflow-providers-ftp==3.11.1
apache-airflow-providers-google==11.0.0
apache-airflow-providers-grpc==3.6.0
apache-airflow-providers-hashicorp==3.8.0
apache-airflow-providers-http==4.13.3
apache-airflow-providers-imap==3.7.0
apache-airflow-providers-microsoft-azure==11.1.0
apache-airflow-providers-mysql==5.7.4
apache-airflow-providers-odbc==4.8.1
apache-airflow-providers-openlineage==1.14.0
apache-airflow-providers-postgres==5.14.0
apache-airflow-providers-redis==3.8.0
apache-airflow-providers-sendgrid==3.6.0
apache-airflow-providers-sftp==4.11.1
apache-airflow-providers-slack==8.9.2
apache-airflow-providers-smtp==1.8.1
apache-airflow-providers-snowflake==5.8.1
apache-airflow-providers-sqlite==3.9.1
apache-airflow-providers-ssh==3.14.0
google-cloud-orchestration-airflow==1.15.1
### Apache Airflow version
2.10.4
### Operating System
Debian GNU/Linux 12 (bookworm)
### Deployment
Official Apache Airflow Helm Chart
### Deployment details
environment variables:
* AIRFLOW__CELERY__POOL=gevent
* _AIRFLOW_PATCH_GEVENT=1
### What happened
When using gevent pool on celery and enabling gevent monkey patch using
envvar `_AIRFLOW_PATCH_GEVENT=1` cause task to failed, with the following
traceback.
```
Traceback (most recent call last):
File
"/home/airflow/.local/lib/python3.12/site-packages/airflow/models/taskinstance.py",
line 767, in _execute_task
result = _execute_callable(context=context, **execute_callable_kwargs)
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
File
"/home/airflow/.local/lib/python3.12/site-packages/airflow/models/taskinstance.py",
line 733, in _execute_callable
return ExecutionCallableRunner(
^^^^^^^^^^^^^^^^^^^^^^^^
File
"/home/airflow/.local/lib/python3.12/site-packages/airflow/utils/operator_helpers.py",
line 252, in run
return self.func(*args, **kwargs)
^^^^^^^^^^^^^^^^^^^^^^^^^^
File
"/home/airflow/.local/lib/python3.12/site-packages/airflow/models/baseoperator.py",
line 413, in wrapper
cls._sentinel.callers[sentinel_key] = sentinel
^^^^^^^^^^^^^^^^^^^^^
File "src/gevent/local.py", line 410, in
gevent._gevent_clocal.local.__getattribute__
AttributeError: 'gevent._gevent_clocal.local' object has no attribute
'callers'
```
### What you think should happen instead
_No response_
### How to reproduce
1. Use celery worker with following envvar `AIRFLOW__CELERY__POOL=gevent`
and `_AIRFLOW_PATCH_GEVENT=1`.
2. Create new DAG with simple bash operator that sleep for 5 seconds.
3. Unpause the new DAG.
### Anything else
If `_AIRFLOW_PATCH_GEVENT` is not set, the task finish successfully but the
subsequent task run will hang in queued state and when checking worker log show
`RecursionError: maximum recursion depth exceeded`.
```
Traceback (most recent call last):
File
"/home/airflow/.local/lib/python3.12/site-packages/airflow/providers/amazon/aws/log/s3_task_handler.py",
line 167, in s3_write
if append and self.s3_log_exists(remote_log_location):
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
File
"/home/airflow/.local/lib/python3.12/site-packages/airflow/providers/amazon/aws/log/s3_task_handler.py",
line 134, in s3_log_exists
return self.hook.check_for_key(remote_log_location)
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
File
"/home/airflow/.local/lib/python3.12/site-packages/airflow/providers/amazon/aws/hooks/s3.py",
line 152, in wrapper
return func(*bound_args.args, **bound_args.kwargs)
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
File
"/home/airflow/.local/lib/python3.12/site-packages/airflow/providers/amazon/aws/hooks/s3.py",
line 125, in wrapper
return func(*bound_args.args, **bound_args.kwargs)
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
File
"/home/airflow/.local/lib/python3.12/site-packages/airflow/providers/amazon/aws/hooks/s3.py",
line 958, in check_for_key
obj = self.head_object(key, bucket_name)
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
File
"/home/airflow/.local/lib/python3.12/site-packages/airflow/providers/amazon/aws/hooks/s3.py",
line 152, in wrapper
return func(*bound_args.args, **bound_args.kwargs)
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
File
"/home/airflow/.local/lib/python3.12/site-packages/airflow/providers/amazon/aws/hooks/s3.py",
line 125, in wrapper
return func(*bound_args.args, **bound_args.kwargs)
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
File
"/home/airflow/.local/lib/python3.12/site-packages/airflow/providers/amazon/aws/hooks/s3.py",
line 938, in head_object
return self.get_conn().head_object(Bucket=bucket_name, Key=key)
^^^^^^^^^^^^^^^
File
"/home/airflow/.local/lib/python3.12/site-packages/airflow/providers/amazon/aws/hooks/base_aws.py",
line 783, in get_conn
return self.conn
^^^^^^^^^
File "/usr/local/lib/python3.12/functools.py", line 995, in __get__
val = self.func(instance)
^^^^^^^^^^^^^^^^^^^
File
"/home/airflow/.local/lib/python3.12/site-packages/airflow/providers/amazon/aws/hooks/base_aws.py",
line 740, in conn
return self.get_client_type(region_name=self.region_name)
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
File
"/home/airflow/.local/lib/python3.12/site-packages/airflow/providers/amazon/aws/hooks/base_aws.py",
line 710, in get_client_type
return session.client(
^^^^^^^^^^^^^^^
File "/home/airflow/.local/lib/python3.12/site-packages/boto3/session.py",
line 299, in client
return self._session.create_client(
^^^^^^^^^^^^^^^^^^^^^^^^^^^^
File
"/home/airflow/.local/lib/python3.12/site-packages/botocore/session.py", line
951, in create_client
credentials = self.get_credentials()
^^^^^^^^^^^^^^^^^^^^^^
File
"/home/airflow/.local/lib/python3.12/site-packages/botocore/session.py", line
507, in get_credentials
self._credentials = self._components.get_component(
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
File
"/home/airflow/.local/lib/python3.12/site-packages/botocore/session.py", line
1108, in get_component
self._components[name] = factory()
^^^^^^^^^
File
"/home/airflow/.local/lib/python3.12/site-packages/botocore/session.py", line
186, in _create_credential_resolver
return botocore.credentials.create_credential_resolver(
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
File
"/home/airflow/.local/lib/python3.12/site-packages/botocore/credentials.py",
line 92, in create_credential_resolver
container_provider = ContainerProvider()
^^^^^^^^^^^^^^^^^^^
File
"/home/airflow/.local/lib/python3.12/site-packages/botocore/credentials.py",
line 1893, in __init__
fetcher = ContainerMetadataFetcher()
^^^^^^^^^^^^^^^^^^^^^^^^^^
File
"/home/airflow/.local/lib/python3.12/site-packages/botocore/utils.py", line
2910, in __init__
session = botocore.httpsession.URLLib3Session(
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
File
"/home/airflow/.local/lib/python3.12/site-packages/botocore/httpsession.py",
line 323, in __init__
self._manager = PoolManager(**self._get_pool_manager_kwargs())
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
File
"/home/airflow/.local/lib/python3.12/site-packages/botocore/httpsession.py",
line 341, in _get_pool_manager_kwargs
'ssl_context': self._get_ssl_context(),
^^^^^^^^^^^^^^^^^^^^^^^
File
"/home/airflow/.local/lib/python3.12/site-packages/botocore/httpsession.py",
line 350, in _get_ssl_context
return create_urllib3_context()
^^^^^^^^^^^^^^^^^^^^^^^^
File
"/home/airflow/.local/lib/python3.12/site-packages/botocore/httpsession.py",
line 139, in create_urllib3_context
context.options |= options
^^^^^^^^^^^^^^^
File "/usr/local/lib/python3.12/ssl.py", line 561, in options
super(SSLContext, SSLContext).options.__set__(self, value)
File "/usr/local/lib/python3.12/ssl.py", line 561, in options
super(SSLContext, SSLContext).options.__set__(self, value)
File "/usr/local/lib/python3.12/ssl.py", line 561, in options
super(SSLContext, SSLContext).options.__set__(self, value)
[Previous line repeated 929 more times]
RecursionError: maximum recursion depth exceeded
```
I think this happen because the monkey patch in celery_command.py does not
happen early enough as is clearly shown by warning of the 2nd line of this logs.
```
/home/airflow/.local/lib/python3.12/site-packages/airflow/metrics/statsd_logger.py:184
RemovedInAirflow3Warning: The basic metric validator will be deprecated in the
future in favor of pattern-matching. You can try this now by setting config
option metrics_use_pattern_match to True.
/home/airflow/.local/lib/python3.12/site-packages/celery/__init__.py:113
MonkeyPatchWarning: Monkey-patching ssl after ssl has already been imported may
lead to errors, including RecursionError on Python 3.6. It may also silently
lead to incorrect behaviour on Python 3.7. Please monkey-patch earlier. See
https://github.com/gevent/gevent/issues/1016. Modules that had direct imports
(NOT patched): ['urllib3.util.ssl_
(/home/airflow/.local/lib/python3.12/site-packages/urllib3/util/ssl_.py)',
'aiohttp.client_exceptions
(/home/airflow/.local/lib/python3.12/site-packages/aiohttp/client_exceptions.py)',
'urllib3.contrib.pyopenssl
(/home/airflow/.local/lib/python3.12/site-packages/urllib3/contrib/pyopenssl.py)',
'aiohttp.connector
(/home/airflow/.local/lib/python3.12/site-packages/aiohttp/connector.py)',
'aiohttp.client_reqrep
(/home/airflow/.local/lib/python3.12/site-packages/aiohttp/client_reqrep.py)',
'urllib3.util
(/home/airflow/.local/lib/python3.12/site-packages/urllib3/util/__init_
_.py)', 'jwt.jwks_client
(/home/airflow/.local/lib/python3.12/site-packages/jwt/jwks_client.py)',
'botocore.httpsession
(/home/airflow/.local/lib/python3.12/site-packages/botocore/httpsession.py)'].
[2025-01-07 11:15:35 +0000] [15] [INFO] Starting gunicorn 23.0.0
[2025-01-07 11:15:35 +0000] [15] [INFO] Listening at: http://[::]:8793 (15)
[2025-01-07 11:15:35 +0000] [15] [INFO] Using worker: sync
[2025-01-07 11:15:35 +0000] [16] [INFO] Booting worker with pid: 16
[2025-01-07 11:15:35 +0000] [17] [INFO] Booting worker with pid: 17
-------------- celery@airflow-ptu-worker-alt-pool-5b78c98f5f-6zxg2 v5.4.0
(opalescent)
--- ***** -----
-- ******* ---- Linux-6.1.112-x86_64-with-glibc2.36 2025-01-07 11:15:36
- *** --- * ---
- ** ---------- [config]
- ** ---------- .> app:
airflow.providers.celery.executors.celery_executor:0x7f8457cd8b90
- ** ---------- .> transport: ***
- ** ---------- .> results: ***
- *** --- * --- .> concurrency: 24 (gevent)
-- ******* ---- .> task events: OFF (enable -E to monitor tasks in this
worker)
--- ***** -----
-------------- [queues]
.> alt_pool exchange=alt_pool(direct) key=alt_pool
[tasks]
. airflow.providers.celery.executors.celery_executor_utils.execute_command
```
### Are you willing to submit PR?
- [X] Yes I am willing to submit a PR!
### Code of Conduct
- [X] I agree to follow this project's [Code of
Conduct](https://github.com/apache/airflow/blob/main/CODE_OF_CONDUCT.md)
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: [email protected]
For queries about this service, please contact Infrastructure at:
[email protected]