anmolxlight commented on code in PR #67127:
URL: https://github.com/apache/airflow/pull/67127#discussion_r3263291970
##########
providers/celery/src/airflow/providers/celery/executors/celery_executor_utils.py:
##########
@@ -160,6 +160,21 @@ def create_celery_app(team_conf: ExecutorConf |
AirflowConfigParser) -> Celery:
return celery_app
+@lru_cache(maxsize=8)
Review Comment:
Changed. Replaced the arbitrary `maxsize=8` with `@cache`, so configured
teams are cached without eviction in the publishing process.
##########
providers/celery/src/airflow/providers/celery/executors/celery_executor_utils.py:
##########
@@ -160,6 +160,21 @@ def create_celery_app(team_conf: ExecutorConf |
AirflowConfigParser) -> Celery:
return celery_app
+@lru_cache(maxsize=8)
+def _get_celery_app_for_workload(team_name: str | None) -> Celery:
+ """Return a subprocess-local Celery app cached by team name for task
publishing."""
Review Comment:
Fixed. Updated the docstring to describe both scheduler-inline and
publisher-subprocess execution paths, and made the cache process-local rather
than subprocess-specific.
##########
providers/celery/src/airflow/providers/celery/executors/celery_executor_utils.py:
##########
@@ -160,6 +160,21 @@ def create_celery_app(team_conf: ExecutorConf |
AirflowConfigParser) -> Celery:
return celery_app
+@lru_cache(maxsize=8)
+def _get_celery_app_for_workload(team_name: str | None) -> Celery:
+ """Return a subprocess-local Celery app cached by team name for task
publishing."""
+ if TYPE_CHECKING:
+ _conf: ExecutorConf | AirflowConfigParser
Review Comment:
Fixed. Removed the leftover `TYPE_CHECKING` annotation block from
`_get_celery_app_for_workload`.
##########
providers/celery/tests/unit/celery/executors/test_celery_executor.py:
##########
@@ -77,6 +77,13 @@
pytestmark = pytest.mark.db_test
[email protected](autouse=True)
+def clear_cached_workload_celery_apps():
+ celery_executor_utils._get_celery_app_for_workload.cache_clear()
+ yield
+ celery_executor_utils._get_celery_app_for_workload.cache_clear()
Review Comment:
Fixed. Added direct cache tests for `team_name=None`, same-team reuse, and
distinct team names.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: [email protected]
For queries about this service, please contact Infrastructure at:
[email protected]
