HsiuChuanHsu commented on issue #49001: URL: https://github.com/apache/airflow/issues/49001#issuecomment-3034581729
Hi @noamst-monday, Based on your description, I have reproduced the issue where Celery worker pods scale up unnecessarily when using multiple executors (`CeleryExecutor,KubernetesExecutor`), due to the KEDA query relying on the `queue` field instead of the `executor` field. --- ### Verify Unnecessary Scaling: - Notice that `airflow-dev-worker-0` (Celery worker pods) scale up (e.g., from 0/2 to 2/2) even though the tasks are configured with `executor='KubernetesExecutor'` and `queue='default'`. - The KEDA query interprets tasks with `queue='default'` as eligible for Celery worker scaling, despite their executor setting, leading to unnecessary pod creation and termination cycles.   --- ### Next Steps Apply the corrected KEDA query from the PR (using `executor != 'KubernetesExecutor'`) and repeat the test to verify that Celery workers no longer scale up for `KubernetesExecutor` tasks. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected]
