GitHub user dpopowich-aca edited a discussion: Using a partial to save
boilerplate when using @task.external_python generates a NameError
I'm aware that you cannot reference top-level module code when using
`@task.external_python`, but I'm surprised by the following...
To avoid boilerplate and ensure consistency across our codebase, I've placed
the following in `mylib`, our custom python library installed into the airflow
venv:
```python
# mylib
import functools
from airflow.sdk import task
PATH_TO_PYTHON = '/path/to/our/python/on/worker'
mytask = functools.partial(task.external_python, python=PATH_TO_PYTHON)
```
Then in a dag source file:
```python
# a test dag
from airflow.sdk import dag
from mylib import mytask
@dag(...)
def mydag():
@mytask()
def some_task():
...
some_task()
mydag()
```
When I trigger this dag I get a `NameError: name 'mytask' is not defined`.
Unfortunately, I can't see the temporary file on the worker because it gets
cleaned up after executing (anyway of saving that for debugging?), but
anecdotal evidence suggests that airflow includes the decorator in the
generated source and bombs because `mytask` is unknown.
Is my only option explicitly using `@task.external_python(...)`.
GitHub link: https://github.com/apache/airflow/discussions/61409
----
This is an automatically sent email for [email protected].
To unsubscribe, please send an email to: [email protected]
