GitHub user yudin-s added a comment to the discussion: Serialization issues
with external_python decorator
After your update, I would look less at the body of the failing
`external_python` task and more at what is being passed into it.
For `@task.external_python`, Airflow still serializes the callable arguments
before it starts the external interpreter. In the current operator code, the
args/kwargs are written with the selected serializer as roughly:
```python
{"args": self.op_args, "kwargs": resolved_kwargs}
```
The docs also call out two separate constraints:
- the function body is cut out into a temporary file, so imports must be inside
the function and globals should not be relied on;
- `op_args`, `op_kwargs`, and return values still need to be serializable
between the Airflow process and the external Python process.
That matches the clue you added: the function named in the `PicklingError` is
not part of the task that fails. It can still be inside an upstream task's
return value, a config object, a connection wrapper, a partially-applied
argument, or something captured in an expanded/partial mapped argument. Airflow
may be failing while serializing the arguments for the external task, not while
running the function body.
I would narrow it this way:
1. Do not pass Airflow `Connection` objects, hooks, callables, modules, or rich
config objects through XCom into an external task. Pass only primitives:
strings, numbers, booleans, lists, and dicts of those.
2. For connections, pass a `conn_id` or a plain dict of exactly the fields you
need:
```python
@task
def get_config():
return {
"source_conn_id": "my_source",
"target_conn_id": "my_target",
}
```
Then resolve/import inside the external task:
```python
@task.external_python(python="/path/to/venv/bin/python")
def load_data(config, rows):
from airflow.hooks.base import BaseHook
source = BaseHook.get_connection(config["source_conn_id"])
# or import your own DB client and use primitive host/user/schema fields
```
If the external environment does not have the same Airflow version installed,
avoid `BaseHook` inside it and pass primitive connection fields instead.
3. Check `.partial(...)` and `.expand(...)` inputs specifically. A
non-serializable object in a partial kwarg can fail later in a downstream
mapped external task, which makes the error look unrelated.
4. The `Can't pickle local object my_dag.<locals>.get_config` error usually
means a function object itself is being passed or captured somewhere. Make sure
you are passing `get_config()` / `XComArg` results, not the `get_config`
function object, and that any returned config dict does not contain callables.
5. If you truly need to pass a more complex but still normal Python object, try
`serializer="cloudpickle"` and install `cloudpickle` where Airflow expects it.
But I would not use that to pass live Airflow objects; it is better to make the
task boundary plain-data-only.
So the practical rule is: keep all imports inside the external function, but
also make every upstream XCom, mapped argument, and partial kwarg crossing into
`external_python` plain data. The error naming `get_connection` / `get_config`
is probably pointing at one of those values being serialized, not necessarily
code executed by the failing task.
If this points you to the bad mapped/partial value, please mark it as answered
so other `external_python` serialization cases can find the argument-boundary
diagnosis.
GitHub link:
https://github.com/apache/airflow/discussions/71281#discussioncomment-17960128
----
This is an automatically sent email for [email protected].
To unsubscribe, please send an email to: [email protected]
