GitHub user latent-9 added a comment to the discussion: Serialization issues with external_python decorator
`@task.external_python` runs your callable in a separate interpreter, so everything it touches has to survive being serialized and re-imported there. The two errors point at two separate causes. `can't pickle local object` comes from defining helpers or closures inside the DAG or task function. Locals are pickled by value and nested functions cannot be, so move `get_config` and any helpers into a module that is importable inside the external environment and import them normally. The connection error comes from touching Airflow objects inside the external task. `get_connection` and `Connection` belong to the Airflow runtime, which the external interpreter does not share, so the identity check fails. The fix is to not use Airflow connections inside an `external_python` task. Fetch the connection in a normal `@task`, pull out the plain values you need, and pass those primitives in. Keep external_python inputs and outputs to picklable data only. If you need the venv isolation together with Airflow context, `@task.virtualenv` shares more of the runtime, or a cloudpickle based serializer is the usual alternative. GitHub link: https://github.com/apache/airflow/discussions/71281#discussioncomment-17961100 ---- This is an automatically sent email for [email protected]. To unsubscribe, please send an email to: [email protected]
