uranusjr commented on code in PR #71757:
URL: https://github.com/apache/airflow/pull/71757#discussion_r4357268398
##########
contributing-docs/30_new_language_sdk.rst:
##########
@@ -454,75 +445,154 @@ authored in the target language. An SDK declares each
one independently.
``retry-policy`` (MAY)
The SDK lets task code inspect a failure and override whether the task
retries or fails,
- optionally with a custom retry delay. This is distinct from reporting
``up_for_retry``:
- an SDK can support ordinary retries without exposing a task-facing
retry-policy API.
+ optionally with a custom retry delay, the equivalent of Python's
``RetryPolicy``. This is
+ distinct from reporting ``up_for_retry``: an SDK can support ordinary
retries without
+ exposing a task-facing retry-policy API.
``task-state-store`` (MAY)
The task can read and write the per-task state store.
``asset-state-store`` (MAY)
- The task can read and write the per-asset state store.
+ The task can read and write the state store keyed by an asset name or URI.
It says
+ nothing about whether the SDK can *declare* an asset, which is
``asset-definition``.
``asset-event-emit`` (MAY)
The task can emit asset events — outlet events, including alias events —
as it runs.
``asset-event-read`` (MAY)
The task can read prior asset events.
-Native-Dag authoring
-~~~~~~~~~~~~~~~~~~~~~~
+``object-store`` (MAY)
+ The SDK exposes an object-storage API (an ``ObjectStoragePath``
equivalent) usable from
+ task code.
+
+``listeners`` (MAY)
+ A task feeds Airflow's listener API, so a listener plugin sees it as it
does a Python
+ task. OpenLineage is the main consumer: without this, a task of this SDK
produces no
+ lineage facets of its own.
-Native-Dag authoring lets an entire Dag — not just individual task bodies — be
written in
+Dag definition
+~~~~~~~~~~~~~~
+
+A Dag definition lets an entire Dag — not just individual task bodies — be
written in
the target language, without a Python Dag file. It is gated by a single
umbrella
capability:
-``native-dag-authoring`` (SHOULD)
+``dag-definition-authoring`` (SHOULD)
The SDK can define a whole Dag in the target language. The project intends
every
Language SDK to reach this bar, but an SDK is not required to ship it
before it is
ready; an SDK that only executes ``@task.stub`` tasks from a Python Dag is
still useful
and remains conformant to the MUST tier.
-The remaining native-Dag capabilities are **conditional**: their tier applies
only when
-``native-dag-authoring`` is supported (written ``†``). For an SDK that does
not (yet) author
-native Dags they are *not applicable* (``n/a``) rather than unsupported.
+The remaining Dag-definition capabilities are **conditional**: their tier
applies only when
+``dag-definition-authoring`` is supported (written ``†``). For an SDK that
does not (yet) author Dags
+in its own language they are *not applicable* (``n/a``) rather than
unsupported.
+
+``dag-args`` (MUST †)
+ Dag-level arguments (schedule, tags, default queue, and so on) can be set
on a Dag definition.
``task-args`` (MUST †)
- Task-level arguments can be bound to a native task.
+ Task-level arguments (retries, pool, queue, and so on) can be set on a
task of a Dag definition.
``dag-params`` (MUST †)
- Dag-level params can be declared on a native Dag.
+ Dag-level params can be declared on a Dag definition.
+
+``explicit-dependencies`` (MUST †)
+ Order-only dependencies can be drawn between tasks, the equivalent of
Python's ``>>`` and
+ ``<<``.
+
+``edge-labels`` (MAY †)
+ A dependency can carry a label, the equivalent of Python's
+ ``loaded >> Label("when empty") >> notify``, and the Dag graph shows it.
``taskflow-dependencies`` (MUST †)
Task dependencies can be expressed in a TaskFlow style, including passing
a task's
return value to a downstream task.
-``branching`` (SHOULD †)
- The native DSL offers a branch construct that skips the non-selected
branches.
-
-``dag-test`` (SHOULD †)
- A native Dag can be exercised locally with ``airflow dags test``.
-
``task-group`` (MAY †)
Tasks can be organised into task groups.
-``dynamic-task-mapping`` (MAY †)
- Tasks can be expanded at runtime via dynamic task mapping.
+``task-loops`` (MAY †)
+ A task group can be repeated at runtime, the equivalent of Python's
``@task_group``
+ with ``.loop()``, bounded by ``max_iterations`` and an optional ``until``
condition.
+
+``if-else-branching`` (SHOULD †)
+ A task can decide between two downstream paths and skip the one not taken.
+
+``switch-case-branching`` (SHOULD †)
+ A task can pick one of several downstream paths and skip the others.
+
+``trigger-dag-run`` (MAY †)
+ A Dag definition can declare a task that triggers a run of another Dag,
like
+ ``TriggerDagRunOperator``.
+
+``provider-operators`` (MAY †)
+ A Dag definition can declare a task that runs a Python provider operator,
which executes
+ in the Python runtime. The SDK surface is a declarative binding over the
operator's
+ parameters rather than a reimplementation, the same shape the YAML Dag
importer's
+ ``uses`` takes, so the bindings are meant to be generated from the Python
operators as
+ the source of truth. A built-in task that mirrors an operator's name while
running in the
+ SDK's own runtime, such as ``trigger-dag-run``, does not count.
-``asset-inlets-outlets`` (MAY †)
- Native tasks can declare asset inlets and outlets.
+``dynamic-dag-generation`` (MAY †)
+ One artifact can generate several Dags from code, for example in a loop.
+
+``human-in-the-loop`` (MAY †)
+ A Dag definition can declare a task that waits for human input.
+
+``dag-test`` (SHOULD †)
+ A Dag definition can be exercised locally with ``airflow dags test``.
+
+``asset-definition`` (MAY †)
+ Assets can be declared in the target language: the equivalents of Python's
``Asset``,
+ ``AssetAlias``, ``AssetAll`` / ``AssetAny``, and the partition mappers a
partitioned
+ asset timetable consumes.
``asset-scheduling`` (MAY †)
- A native Dag can be scheduled on asset events.
+ A Dag definition can be scheduled on asset events, including the
``AssetWatcher``
+ equivalent that turns an external trigger into those events.
-``object-store`` (MAY †)
- The SDK exposes an object-storage API (an ``ObjectStoragePath``
equivalent) usable from
- native Dag code.
+Python Dag with stub TaskHandler
+~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
+
+These capabilities cover tasks declared in a Python Dag with the
``@task.stub`` decorator and
+implemented in the target language.
+
+``mixed-lang-stub-target`` (MUST)
+ The SDK can execute a task declared in a Python Dag with the ``@task.stub``
+ decorator. This is the primary execution path for every Language SDK.
+
+``taskflow-binding`` (MUST)
+ Arguments bound to a ``@task.stub`` target in the Python TaskFlow Dag —
literal values
+ and upstream XCom results alike — are delivered to the task function's
parameters at
+ runtime.
+
+Shared capabilities
+~~~~~~~~~~~~~~~~~~~
+
+These capabilities exist in both authoring modes, and an SDK may support them
in one mode but not
+the other, so each one is declared once per mode. In a Dag definition the tier
applies only when
+``dag-definition-authoring`` is supported.
+
+``mapped-tasks`` (MAY)
+ A task can be expanded at runtime via dynamic task mapping, and each
mapped instance
+ receives its own arguments.
+
+``asset-inlets-outlets`` (MAY)
+ A task's asset inlets and outlets are honored, so a successful task emits
events for its
+ outlets.
+
+``callbacks`` (MAY)
+ Task and Dag callbacks (``on_failure_callback`` and the rest) run. The two
authoring modes
+ ask different things of the SDK: in a Dag definition the SDK declares the
callback as a
+ function in its own language, while for a stub task the callback is
declared in the Python
+ Dag and the SDK's execution path has to let the Python runtime run it.
Review Comment:
This only strictly refer to native non-Python Dags. Dag callbacks in a
Python stub dag can run fine, for obvious reasons. THis should probably be
clarified. It may also be a good idea to separate Dag and task level callbacks
into two capabilities since they run in different contexts and need to be
implemented separately anyway.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: [email protected]
For queries about this service, please contact Infrastructure at:
[email protected]
