GitHub user pykenny edited a comment on the discussion: Separate Python virtual environments for each DAG(not Task)
I read the [slides about remote worker](https://airflowsummit.org/sessions/2024/running-airflow-tasks-anywhere-in-any-language/) (or edge worker, to be more specifically) introduced on last year's Airflow Summit and feel that it could possibly be the solution when the feature becomes fully available. An executor still need to run in an environment where the same Python+Airflow version as the core service does, but at runner level it won't matter as long as it supports the recognized task execution protocol (using compatible Task SDK to communicate with executor). Finally DAG authors can write task flow in this way to specify the environment(s): ```python @dag(...): def my_dag: # ... task_venv = task.external("environment_001") upstream = task_venv.upstream_job() task_venv.downstream_job(data=upstream) # ... ``` Unfortunately at the time of writing, Airflow officials haven't release any SDK that supports language outside of Python (Golang, JavaScript, TypeScript, etc.) yet, and current Python Task SDK has to take care of [backward compatibility of existing codebase](https://airflow.apache.org/blog/airflow-three-point-oh-is-here/#:~:text=The%20Airflow%203%20release%20includes%20the%20Python%20TaskSDK%20which%20enables%20backward%20compatibility%20for%20existing%20DAGs.%20TaskSDKs%20for%20additional%20languages%2C%20starting%20with%20Golang%20will%20be%20released%20over%20the%20next%20few%20months.), meaning it's [heavily tied to the core system itself](https://github.com/apache/airflow/blob/35186bccf3da10d5d01b744966be8156721be5c7/task-sdk/pyproject.toml#L49) (runner using Python SDK should run in an environment with Airflow installed), which doesn't offer the same flexibility as demonstrated in the summit. GitHub link: https://github.com/apache/airflow/discussions/40651#discussioncomment-13072576 ---- This is an automatically sent email for [email protected]. To unsubscribe, please send an email to: [email protected]
