dstandish commented on a change in pull request #13247:
URL: https://github.com/apache/airflow/pull/13247#discussion_r567264843
##########
File path: airflow/cli/cli_parser.py
##########
@@ -58,9 +58,15 @@ class DefaultHelpParser(argparse.ArgumentParser):
def _check_value(self, action, value):
"""Override _check_value and check conditionally added command"""
executor = conf.get('core', 'EXECUTOR')
- if value == 'celery' and executor not in (CELERY_EXECUTOR,
CELERY_KUBERNETES_EXECUTOR):
- message = f'celery subcommand works only with CeleryExecutor, your
current executor: {executor}'
- raise ArgumentError(action, message)
+ if value == 'celery':
+ if executor != CELERY_EXECUTOR:
Review comment:
@kaxil the problem is that when executor==CELERY_KUBERNETES_EXECUTOR,
`airflow celery worker` does not work
you get some fork pool error
running `airflow celery worker` with `CELERY_EXECUTOR` instead resolves the
issue
celery workers don't need to know that _the scheduler_ is using CKE
i know this is a bit hacky. our code should handle having all components
set to use the same executor. however, that simply doesn't work right now, and
this change (forcing the C workers to think CeleryExecutor is used) makes CKE
work again immediately.
perhaps it makes sense to push out this hacky fix and then later look into
why we get fork pool issues. but that's a question for you guys
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
[email protected]
