[ https://issues.apache.org/jira/browse/AIRFLOW-3415?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
Yuri Bendana updated AIRFLOW-3415: ---------------------------------- Description: When triggering dagruns in a loop, the imported references become null on the second iteration. Here is an example [ gist | [https://gist.github.com/ybendana/3bc0791fe00b099be04aca47a8d524c9] ]. For the purposes here, you can ignore the DagRunSensor task. On the first iteration the 'sleeper' dag gets triggered but on the second iteration I see a {noformat} TypeError: 'NoneType' object is not callable{noformat} To workaround this, I have to copy the import (in this case trigger_dag) inside the loop. Here is the stacktrace: {code:java} [2018-11-28 18:23:42,492] {{models.py:1789}} INFO - All retries failed; marking task as FAILED [2018-11-28 18:23:42,517] {{base_task_runner.py:101}} INFO - Job 1354170: Subtask t1 Traceback (most recent call last): [2018-11-28 18:23:42,518] {{base_task_runner.py:101}} INFO - Job 1354170: Subtask t1 File "/opt/venv/sgmo/bin/airflow", line 32, in <module> [2018-11-28 18:23:42,518] {{base_task_runner.py:101}} INFO - Job 1354170: Subtask t1 args.func(args) [2018-11-28 18:23:42,518] {{base_task_runner.py:101}} INFO - Job 1354170: Subtask t1 File "/opt/venv/sgmo/lib/python2.7/site-packages/airflow/utils/cli.py", line 74, in wrapper [2018-11-28 18:23:42,518] {{base_task_runner.py:101}} INFO - Job 1354170: Subtask t1 return f(*args, **kwargs) [2018-11-28 18:23:42,518] {{base_task_runner.py:101}} INFO - Job 1354170: Subtask t1 File "/opt/venv/sgmo/lib/python2.7/site-packages/airflow/bin/cli.py", line 490, in run [2018-11-28 18:23:42,519] {{base_task_runner.py:101}} INFO - Job 1354170: Subtask t1 _run(args, dag, ti) [2018-11-28 18:23:42,519] {{base_task_runner.py:101}} INFO - Job 1354170: Subtask t1 File "/opt/venv/sgmo/lib/python2.7/site-packages/airflow/bin/cli.py", line 406, in _run [2018-11-28 18:23:42,519] {{base_task_runner.py:101}} INFO - Job 1354170: Subtask t1 pool=args.pool, [2018-11-28 18:23:42,519] {{base_task_runner.py:101}} INFO - Job 1354170: Subtask t1 File "/opt/venv/sgmo/lib/python2.7/site-packages/airflow/utils/db.py", line 74, in wrapper [2018-11-28 18:23:42,519] {{base_task_runner.py:101}} INFO - Job 1354170: Subtask t1 return func(*args, **kwargs) [2018-11-28 18:23:42,520] {{base_task_runner.py:101}} INFO - Job 1354170: Subtask t1 File "/opt/venv/sgmo/lib/python2.7/site-packages/airflow/models.py", line 1659, in _run_raw_task [2018-11-28 18:23:42,520] {{base_task_runner.py:101}} INFO - Job 1354170: Subtask t1 result = task_copy.execute(context=context) [2018-11-28 18:23:42,520] {{base_task_runner.py:101}} INFO - Job 1354170: Subtask t1 File "/opt/venv/sgmo/lib/python2.7/site-packages/airflow/operators/python_operator.py", line 95, in execute [2018-11-28 18:23:42,520] {{base_task_runner.py:101}} INFO - Job 1354170: Subtask t1 return_value = self.execute_callable() [2018-11-28 18:23:42,520] {{base_task_runner.py:101}} INFO - Job 1354170: Subtask t1 File "/opt/venv/sgmo/lib/python2.7/site-packages/airflow/operators/python_operator.py", line 100, in execute_callable [2018-11-28 18:23:42,521] {{base_task_runner.py:101}} INFO - Job 1354170: Subtask t1 return self.python_callable(*self.op_args, **self.op_kwargs) [2018-11-28 18:23:42,521] {{base_task_runner.py:101}} INFO - Job 1354170: Subtask t1 File "/opt/airflow/dags/sleepersensor.py", line 11, in triggersleeper [2018-11-28 18:23:42,521] {{base_task_runner.py:101}} INFO - Job 1354170: Subtask t1 tdr = trigger_dag('sleeper') [2018-11-28 18:23:42,521] {{base_task_runner.py:101}} INFO - Job 1354170: Subtask t1 TypeError: 'NoneType' object is not callable {code} was: When triggering dagruns in a loop, the imported references become null on the second iteration. Here is an example [ gist | [https://gist.github.com/ybendana/3bc0791fe00b099be04aca47a8d524c9] ]. For the purposes here, you can ignore the DagRunSensor task. On the first iteration the 'sleeper' dag gets triggered but on the second iteration I see a {noformat} TypeError: 'NoneType' object is not callable{noformat} To workaround this, I have to copy the import (in this case trigger_dag) inside the loop. > Imports become null when triggering dagruns in a loop > ----------------------------------------------------- > > Key: AIRFLOW-3415 > URL: https://issues.apache.org/jira/browse/AIRFLOW-3415 > Project: Apache Airflow > Issue Type: Bug > Components: DagRun > Affects Versions: 1.10.1 > Environment: CentOS 7 > Reporter: Yuri Bendana > Priority: Minor > Attachments: triggerdag_stacktrace.txt > > > When triggering dagruns in a loop, the imported references become null on the > second iteration. Here is an example [ gist | > [https://gist.github.com/ybendana/3bc0791fe00b099be04aca47a8d524c9] ]. For > the purposes here, you can ignore the DagRunSensor task. On the first > iteration the 'sleeper' dag gets triggered but on the second iteration I see a > {noformat} > TypeError: 'NoneType' object is not callable{noformat} > To workaround this, I have to copy the import (in this case trigger_dag) > inside the loop. > Here is the stacktrace: > {code:java} > [2018-11-28 18:23:42,492] {{models.py:1789}} INFO - All retries failed; > marking task as FAILED > [2018-11-28 18:23:42,517] {{base_task_runner.py:101}} INFO - Job 1354170: > Subtask t1 Traceback (most recent call last): > [2018-11-28 18:23:42,518] {{base_task_runner.py:101}} INFO - Job 1354170: > Subtask t1 File "/opt/venv/sgmo/bin/airflow", line 32, in <module> > [2018-11-28 18:23:42,518] {{base_task_runner.py:101}} INFO - Job 1354170: > Subtask t1 args.func(args) > [2018-11-28 18:23:42,518] {{base_task_runner.py:101}} INFO - Job 1354170: > Subtask t1 File > "/opt/venv/sgmo/lib/python2.7/site-packages/airflow/utils/cli.py", line 74, > in wrapper > [2018-11-28 18:23:42,518] {{base_task_runner.py:101}} INFO - Job 1354170: > Subtask t1 return f(*args, **kwargs) > [2018-11-28 18:23:42,518] {{base_task_runner.py:101}} INFO - Job 1354170: > Subtask t1 File > "/opt/venv/sgmo/lib/python2.7/site-packages/airflow/bin/cli.py", line 490, in > run > [2018-11-28 18:23:42,519] {{base_task_runner.py:101}} INFO - Job 1354170: > Subtask t1 _run(args, dag, ti) > [2018-11-28 18:23:42,519] {{base_task_runner.py:101}} INFO - Job 1354170: > Subtask t1 File > "/opt/venv/sgmo/lib/python2.7/site-packages/airflow/bin/cli.py", line 406, in > _run > [2018-11-28 18:23:42,519] {{base_task_runner.py:101}} INFO - Job 1354170: > Subtask t1 pool=args.pool, > [2018-11-28 18:23:42,519] {{base_task_runner.py:101}} INFO - Job 1354170: > Subtask t1 File > "/opt/venv/sgmo/lib/python2.7/site-packages/airflow/utils/db.py", line 74, in > wrapper > [2018-11-28 18:23:42,519] {{base_task_runner.py:101}} INFO - Job 1354170: > Subtask t1 return func(*args, **kwargs) > [2018-11-28 18:23:42,520] {{base_task_runner.py:101}} INFO - Job 1354170: > Subtask t1 File > "/opt/venv/sgmo/lib/python2.7/site-packages/airflow/models.py", line 1659, in > _run_raw_task > [2018-11-28 18:23:42,520] {{base_task_runner.py:101}} INFO - Job 1354170: > Subtask t1 result = task_copy.execute(context=context) > [2018-11-28 18:23:42,520] {{base_task_runner.py:101}} INFO - Job 1354170: > Subtask t1 File > "/opt/venv/sgmo/lib/python2.7/site-packages/airflow/operators/python_operator.py", > line 95, in execute > [2018-11-28 18:23:42,520] {{base_task_runner.py:101}} INFO - Job 1354170: > Subtask t1 return_value = self.execute_callable() > [2018-11-28 18:23:42,520] {{base_task_runner.py:101}} INFO - Job 1354170: > Subtask t1 File > "/opt/venv/sgmo/lib/python2.7/site-packages/airflow/operators/python_operator.py", > line 100, in execute_callable > [2018-11-28 18:23:42,521] {{base_task_runner.py:101}} INFO - Job 1354170: > Subtask t1 return self.python_callable(*self.op_args, **self.op_kwargs) > [2018-11-28 18:23:42,521] {{base_task_runner.py:101}} INFO - Job 1354170: > Subtask t1 File "/opt/airflow/dags/sleepersensor.py", line 11, in > triggersleeper > [2018-11-28 18:23:42,521] {{base_task_runner.py:101}} INFO - Job 1354170: > Subtask t1 tdr = trigger_dag('sleeper') > [2018-11-28 18:23:42,521] {{base_task_runner.py:101}} INFO - Job 1354170: > Subtask t1 TypeError: 'NoneType' object is not callable > {code} > -- This message was sent by Atlassian JIRA (v7.6.3#76005)