uranusjr opened a new issue, #75273:
URL: https://github.com/apache/airflow/issues/75273
The YAML Dag importer supports using `run:` in a task to specify the task
body is supplied by a Task SDK handler. This automatically works for foreign
language SDKs (currently Go, Java, TypeScript) through AIP-108, but there is
currently no way to implement a `run:` task's body in Python.
This proposes creating an interface in Python equivalent to AIP-108 task
handlers. This would allow Dag-authroing interfaces in other languages (if they
implement and equivalent to Python’s `@task.stub`) to specify a task’s body
should be found in Python via a `@task_handler` decorator in the Task SDK. The
YAML Dag authoring interface is the only current user through `run:`.
(Technicality note: You *could* in theory create a `@task.stub` in Python,
and tell the coordinator to also find the body in a Python `@task_handler`, but
nobody would do this in practice. Using `@task` makes infinitely more sense.)
```python
from airflow.sdk import task_handler
@task_handler(dag_id="my_yaml_dag", task_id="transform")
def transform(rows: list[dict], factor: int) -> int:
return len(rows) * factor
```
```yaml
# ...
tasks:
# ...
- id: transform
queue: python-workers
run:
rows: {$x: extract}
factor: 2
```
In scope:
- A `@task_handler` decorator exposed in `airflow.sdk`. The registry to store
declared handlers.
- A configurable `PythonCoordinator` that launches the activity subprocess
and
resolves/describes the handler for a given task.
- Execution-time dispatch from the stub-task branch in the task runner.
- Best-effort parse-time validation of `run:` tasks against available
handlers.
- Docs for both `@task_handler` (in Task SDK) and how to use it in YAML (in
core).
Out of scope / future (not tracked by this issue):
- Foreign-SDK `parse_task_handler` (compiled Go/Java/TS handler description).
- ADR-0013 persisted argument-binding storage.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: [email protected]
For queries about this service, please contact Infrastructure at:
[email protected]
