[
https://issues.apache.org/jira/browse/AIRFLOW-2119?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16369766#comment-16369766
]
Yuliya Volkova edited comment on AIRFLOW-2119 at 2/20/18 6:52 AM:
------------------------------------------------------------------
[~paymahn], standard python import didn't support possible to load modules
whitespace inside names and to do this must be construction with __import__("")
but it's anyway terrible, I don't think what need to support space inside file
names? why you need it?
was (Author: xnuinside):
[~paymahn], standard python import didn't support possible to load modules
whitespace inside names and to do this must be construction with {{
__import__("") but it's anyway terrible, I don't think what need to support
space inside file names? why you need it? }}
> Celery worker fails when dag has space in filename
> --------------------------------------------------
>
> Key: AIRFLOW-2119
> URL: https://issues.apache.org/jira/browse/AIRFLOW-2119
> Project: Apache Airflow
> Issue Type: Bug
> Components: celery, worker
> Affects Versions: 1.9.0
> Reporter: Paymahn Moghadasian
> Priority: Minor
>
> A dag whose filename has a space will cause celery workers to fail as follows:
>
> {noformat}
> [2018-02-16 22:58:55,976] {driver.py:120} INFO - Generating grammar tables
> from /usr/lib/python3.5/lib2to3/Grammar.txt
> [2018-02-16 22:58:56,021] {driver.py:120} INFO - Generating grammar tables
> from /usr/lib/python3.5/lib2to3/PatternGrammar.txt
> [2018-02-16 22:58:56,322] {configuration.py:206} WARNING - section/key
> [celery/celery_ssl_active] not found in config
> [2018-02-16 22:58:56,322] {default_celery.py:41} WARNING - Celery Executor
> will run without SSL
> [2018-02-16 22:58:56,323] {__init__.py:45} INFO - Using executor
> CeleryExecutor
> Starting flask
> [2018-02-16 22:58:56,403] {_internal.py:88} INFO - * Running on
> http://0.0.0.0:8793/ (Press CTRL+C to quit)
> [2018-02-16 22:59:25,181] {celery_executor.py:50} INFO - Executing command in
> Celery: airflow run broken_hello_world dummy_task 2018-02-15T12:00:00 --local
> -sd /home/paymahn/scheduler/airflow-home/dags/hello world.py
> [2018-02-16 22:59:25,569] {driver.py:120} INFO - Generating grammar tables
> from /usr/lib/python3.5/lib2to3/Grammar.txt
> [2018-02-16 22:59:25,610] {driver.py:120} INFO - Generating grammar tables
> from /usr/lib/python3.5/lib2to3/PatternGrammar.txt
> [2018-02-16 22:59:25,885] {configuration.py:206} WARNING - section/key
> [celery/celery_ssl_active] not found in config
> [2018-02-16 22:59:25,885] {default_celery.py:41} WARNING - Celery Executor
> will run without SSL
> [2018-02-16 22:59:25,886] {__init__.py:45} INFO - Using executor
> CeleryExecutor
> usage: airflow [-h]
>
> {flower,kerberos,upgradedb,worker,render,serve_logs,backfill,task_state,dag_state,test,connections,pause,unpause,list_tasks,scheduler,run,list_dags,webserver,trigger_dag,version,pool,resetdb,clear,variables,initdb,task_failed_deps}
> ...
> airflow: error: unrecognized arguments: world.py
> [2018-02-16 22:59:26,055] {celery_executor.py:54} ERROR - Command 'airflow
> run broken_hello_world dummy_task 2018-02-15T12:00:00 --local -sd
> /home/paymahn/scheduler/airflow-home/dags/hello world.py' returned non-zero
> exit status 2
> [2018-02-16 22:59:26,117: ERROR/ForkPoolWorker-16] Task
> airflow.executors.celery_executor.execute_command[114e9ff2-380e-4ba6-84e4-c913ea85189c]
> raised unexpected: AirflowException('Celery command failed',)
> Traceback (most recent call last):
> File
> "/home/paymahn/scheduler/venv/lib/python3.5/site-packages/airflow/executors/celery_executor.py",
> line 52, in execute_command
> subprocess.check_call(command, shell=True)
> File "/usr/lib/python3.5/subprocess.py", line 581, in check_call
> raise CalledProcessError(retcode, cmd)
> subprocess.CalledProcessError: Command 'airflow run broken_hello_world
> dummy_task 2018-02-15T12:00:00 --local -sd
> /home/paymahn/scheduler/airflow-home/dags/hello world.py' returned non-zero
> exit status 2
> During handling of the above exception, another exception occurred:
> Traceback (most recent call last):
> File
> "/home/paymahn/scheduler/venv/lib/python3.5/site-packages/celery/app/trace.py",
> line 374, in trace_task
> R = retval = fun(*args, **kwargs)
> File
> "/home/paymahn/scheduler/venv/lib/python3.5/site-packages/celery/app/trace.py",
> line 629, in __protected_call__
> return self.run(*args, **kwargs)
> File
> "/home/paymahn/scheduler/venv/lib/python3.5/site-packages/airflow/executors/celery_executor.py",
> line 55, in execute_command
> raise AirflowException('Celery command failed')
> airflow.exceptions.AirflowException: Celery command failed
> [2018-02-16 22:59:27,420] {celery_executor.py:50} INFO - Executing command in
> Celery: airflow run broken_hello_world dummy_task 2018-02-16T22:59:26.096432
> --local -sd /home/paymahn/scheduler/airflow-home/dags/hello world.py
> [2018-02-16 22:59:27,840] {driver.py:120} INFO - Generating grammar tables
> from /usr/lib/python3.5/lib2to3/Grammar.txt
> [2018-02-16 22:59:27,877] {driver.py:120} INFO - Generating grammar tables
> from /usr/lib/python3.5/lib2to3/PatternGrammar.txt
> [2018-02-16 22:59:28,148] {configuration.py:206} WARNING - section/key
> [celery/celery_ssl_active] not found in config
> [2018-02-16 22:59:28,148] {default_celery.py:41} WARNING - Celery Executor
> will run without SSL
> [2018-02-16 22:59:28,149] {__init__.py:45} INFO - Using executor
> CeleryExecutor
> usage: airflow [-h]
>
> {resetdb,version,dag_state,trigger_dag,connections,task_state,variables,upgradedb,webserver,kerberos,pool,serve_logs,run,list_dags,scheduler,render,flower,task_failed_deps,worker,unpause,backfill,test,initdb,list_tasks,pause,clear}
> ...
> airflow: error: unrecognized arguments: world.py
> [2018-02-16 22:59:28,316] {celery_executor.py:54} ERROR - Command 'airflow
> run broken_hello_world dummy_task 2018-02-16T22:59:26.096432 --local -sd
> /home/paymahn/scheduler/airflow-home/dags/hello world.py' returned non-zero
> exit status 2
> [2018-02-16 22:59:28,370: ERROR/ForkPoolWorker-8] Task
> airflow.executors.celery_executor.execute_command[5bd0d338-1df5-4ced-8a29-f87d68c643bd]
> raised unexpected: AirflowException('Celery command failed',)
> Traceback (most recent call last):
> File
> "/home/paymahn/scheduler/venv/lib/python3.5/site-packages/airflow/executors/celery_executor.py",
> line 52, in execute_command
> subprocess.check_call(command, shell=True)
> File "/usr/lib/python3.5/subprocess.py", line 581, in check_call
> raise CalledProcessError(retcode, cmd)
> subprocess.CalledProcessError: Command 'airflow run broken_hello_world
> dummy_task 2018-02-16T22:59:26.096432 --local -sd
> /home/paymahn/scheduler/airflow-home/dags/hello world.py' returned non-zero
> exit status 2
> During handling of the above exception, another exception occurred:
> Traceback (most recent call last):
> File
> "/home/paymahn/scheduler/venv/lib/python3.5/site-packages/celery/app/trace.py",
> line 374, in trace_task
> R = retval = fun(*args, **kwargs)
> File
> "/home/paymahn/scheduler/venv/lib/python3.5/site-packages/celery/app/trace.py",
> line 629, in __protected_call__
> return self.run(*args, **kwargs)
> File
> "/home/paymahn/scheduler/venv/lib/python3.5/site-packages/airflow/executors/celery_executor.py",
> line 55, in execute_command
> raise AirflowException('Celery command failed')
> airflow.exceptions.AirflowException: Celery command failed
> {noformat}
> I suspect the problem is that there's an unescaped space in the name of the
> file that gets passed to one of the subprocess calls.
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
