This is an automated email from the ASF dual-hosted git repository.

uranusjr pushed a commit to branch main
in repository https://gitbox.apache.org/repos/asf/airflow.git


The following commit(s) were added to refs/heads/main by this push:
     new da9fa5750b1 Amend ADR-0016 for SDK spelling and kwargs (#77239)
da9fa5750b1 is described below

commit da9fa5750b1e88531aa66c1a4ea581a14636bce5
Author: Tzu-ping Chung <[email protected]>
AuthorDate: Thu Nov 19 02:00:18 2026 +0800

    Amend ADR-0016 for SDK spelling and kwargs (#77239)
    
    SDKs were about to spell mapping their own way, and #77081's rule that
    a handler's default does not count left a Dag definition with no way
    to use expand_kwargs() over an upstream's dicts: its handler is the
    signature, yet its types are gone by the time the Dag is written.
    
    Record that each SDK keeps Python's spelling where its binding allows,
    narrow the default rule to a stub paired with a task handler, and add
    a kwargs entry whose dict the runtime spreads, so a Dag definition's
    own task keeps its handler's defaults while the server still picks the
    element. Also define what a handler's own getXCom read of a mapped
    producer returns.
    
    
    Claude-Session: https://claude.ai/code/session_01GmjkXyAqtBrP7Qi97jgw4C
    
    Co-authored-by: Claude Opus 5.5 <[email protected]>
---
 .../lang-sdk/0016-mapped-tasks-across-lang-sdks.md | 91 ++++++++++++++++++----
 1 file changed, 77 insertions(+), 14 deletions(-)

diff --git a/airflow-core/adr/lang-sdk/0016-mapped-tasks-across-lang-sdks.md 
b/airflow-core/adr/lang-sdk/0016-mapped-tasks-across-lang-sdks.md
index 55683b1c421..b44d71cca40 100644
--- a/airflow-core/adr/lang-sdk/0016-mapped-tasks-across-lang-sdks.md
+++ b/airflow-core/adr/lang-sdk/0016-mapped-tasks-across-lang-sdks.md
@@ -90,8 +90,15 @@ The coordinator only starts the SDK runtime subprocess; the 
supervisor handles e
 
 ## Authoring interface
 
+- Each SDK keeps Python's spelling as far as its argument binding allows. 
Where an SDK cannot, the
+  substitute is the same in every SDK that needs it, never one per language.
 - `expand` takes the arguments that vary per instance; `partial` takes the 
arguments that stay the same at every instance.
-- A mapped task returns the same reference type as any other task, so an edge 
or a later `expand` treats it identically.
+  An SDK that binds arguments by position cannot split them by keyword, so it 
marks each expanded
+  argument in place instead, under one marker name shared by every such SDK, 
and every unmarked
+  argument is a `partial` one.
+- A mapped task's reference goes wherever a task reference goes on an edge, 
and into a later `expand`.
+  Where the type system can tell it from an unmapped one, it has its own type, 
so passing it to an
+  unmapped task type-checks as a reduce: only where the parameter is a 
sequence of its element type.
 - `zip`, `concat` and `map` are methods on a reference, not free functions, so 
they chain in declaration order.
 - A reduce consumer receives a sequence of the element type, not the element 
type.
 - Expanding over a list of `T` gives each instance a `T`; expanding over a 
dict gives each instance a `[key, value]` pair, as a Python task gets `(key, 
value)`.
@@ -106,9 +113,9 @@ async function report(args: { totals: XComSequence<number> 
}): Promise<void>;
 ```
 
 ```ts
-const rows = dag.task("extract", extract);
-const totals = dag.task("transform", transform).partial({ region: "eu" 
}).expand({ row: rows });
-dag.task("report", report, { totals });
+const rows = dag.task(extract)();
+const totals = dag.task(transform).partial({ region: "eu" }).expand({ row: 
rows });
+dag.task(report)({ totals });
 ```
 
 `transform` runs once per element of `extract`'s output and receives one 
`row`; `report` runs once
@@ -136,7 +143,7 @@ run.
     "arg": {"task_id": "extract", "key": "return_value"},
     "callables": ["(row) => row.trim()"]}}}}}
 
-// .expand({ row: rows.zip(lookup, { fillvalue: "" }) })
+// .expand({ row: rows.zip(lookup, { fillValue: "" }) })
 "expand_input": {"type": "dict-of-lists", "value": {"__type": "dict", "__var": 
{
   "row": {"__type": "xcomref", "__var": {
     "type": "zip",
@@ -169,6 +176,7 @@ per map index.
 | `xcom` | an upstream task's output | `task_id` |
 | `sequence` | a mapped upstream's whole output, read as a lazy sequence | 
`task_id` |
 | `param` | the server derives it per map index | `default`, `group_arg` |
+| `kwargs` | a Dag definition's own task only: this index's `expand_kwargs()` 
dict, which the runtime spreads | none |
 
 ```jsonc
 // @task.stub def transform(region: str, row: str, scale: int = 2) -> int: ...
@@ -181,7 +189,7 @@ per map index.
 ]
 ```
 
-Every kind carries `name` and `value_schema`: the parameter's name, and the 
JSON Schema of its
+Every kind but `kwargs` carries `name` and `value_schema`: the parameter's 
name, and the JSON Schema of its
 annotation for a runtime that decodes into machine types. Declaration order is 
load-bearing — an SDK
 that binds by position, as Go does, has nothing else to go on.
 
@@ -190,8 +198,24 @@ the call does not set it through `partial()` or 
`expand()`. `"default": null` is
 or its equivalent in the runtime's language, not a missing default: a 
parameter that defaults to
 `None` is common in Python. So a writer emits `default` whenever the server 
may read it, `null`
 included, a reader tells the two cases apart by the key and not by its value, 
and anything that
-reads and rewrites an entry keeps the key's presence. A default declared only 
in the task handler,
-in the runtime's language, does not count.
+reads and rewrites an entry keeps the key's presence. For a stub paired with a 
task handler, only
+the stub's default counts. One declared only in the handler does not, even 
where the two are written
+in the same language, because the stub is the signature the call is checked 
against. A Dag
+definition's own task has no stub, so its handler is the signature and its 
defaults apply, which the
+`kwargs` entry below relies on.
+
+`expand_kwargs()` is the one form that needs every parameter's real name when 
the Dag is written,
+because each instance's dict is matched against `name` by key. A stub always 
has them. A Dag
+definition's own task may not: a TypeScript handler's argument type is erased, 
so a call whose dicts
+come from an upstream names no keys. Such a task writes a single `kwargs` 
entry in place of a `param`
+per expanded parameter. The server still picks this index's dict, and the 
runtime spreads it into the
+handler's arguments beside the `partial()` ones. A key the dict leaves out 
takes the handler's own
+default; a key `partial()` also sets fails the instance, as Python's `mapping 
already partial` does;
+and a key the handler does not take is passed on with the rest, since the 
runtime has no parameter
+list to check it against. A runtime that binds by position cannot spread a 
dict either, so an SDK
+that writes synthetic names, as Go writes `arg0`, `arg1`, … since a Go 
function keeps no parameter
+names at run time, offers `expandKwargs` only for a handler whose names it can 
read, such as the
+fields of a struct argument.
 
 A `param` entry is the signature, not the call. The mapped values stay in the 
fields Python already
 writes: `op_kwargs_expand_input`, a decorated task's name for `expand_input`, 
and
@@ -199,14 +223,17 @@ writes: `op_kwargs_expand_input`, a decorated task's name 
for `expand_input`, an
 `{group_id, key}`, naming an expand argument of an enclosing mapped task group 
— then from those
 fields by `name`, then from `default` if the entry has one; a parameter still 
unbound fails the
 task. Only `expand_kwargs()` can leave a parameter unbound until run time, so 
only it needs that
-fallback.
+fallback. A `kwargs` entry resolves from the same expand input to this index's 
dict, sent with
+`spread` set.
 
 Parse time emits `param` just twice: for a mapped stub's parameters, and for a 
group member's
-argument that names its group. A member's other arguments are ordinary 
`literal`, `xcom` or `sequence`
+argument that names its group. A Dag definition's own task called through 
`expand_kwargs()` writes
+`param` for its `partial()` arguments and one `kwargs` entry, last, for the 
rest. A member's other arguments are ordinary `literal`, `xcom` or `sequence`
 entries, and its unbound parameters take their defaults at parse time, because 
that
 fallback does not vary by index.
 
-Two kinds appear on one side only. `param` never reaches the runtime, the 
server having resolved it.
+Three kinds appear on one side only. `param` and `kwargs` never reach the 
runtime, the server having
+resolved them, a `kwargs` entry into a binding with `spread`.
 `tuple`, the `zip()` element below, is never in the Dag, since only the server 
knows which rows meet
 at this index. Nor is the kind in the Dag always the kind on the wire: an 
argument naming a task in
 the same mapped group is `sequence`, that producer being expanded itself, and 
the server narrows it
@@ -248,7 +275,9 @@ ArgPart = Annotated[XComRef | LiteralValue, 
Field(discriminator="kind")]
 
 
 class _ArgBindingBase(BaseModel):
-    name: str
+    name: str | None = None
+    """The parameter. Absent only with ``spread``, which fills several."""
+
     value_schema: ArgValueSchema | None = None
 
     apply_map: bool = False
@@ -260,6 +289,10 @@ class _ArgBindingBase(BaseModel):
     fallback: ArgPart | None = None
     """The value when ``kwarg`` is missing: the ``partial()`` value or the 
signature default."""
 
+    spread: bool = False
+    """``expand_kwargs()`` on a Dag definition's own task only: the value, 
after any ``map()``
+    callables, is a dict the runtime spreads into the handler's arguments."""
+
 
 class XComArgBinding(_ArgBindingBase, XComRef):
     """An argument read from an upstream task."""
@@ -410,6 +443,7 @@ What the server sends for each authoring form, and what the 
runtime receives aft
 | `.expand({ a: …, b: … })` | one binding per argument, each with its own 
coordinate | one `literal` per argument |
 | `.partial({ region: … })` | `literal`, or `xcom` with no index | the same 
value at every index |
 | `.expandKwargs(rows)` | per parameter: `xcom` with `element_index=i`, 
`kwarg` and `fallback` | `literal`: that key of element `i`, else the fallback |
+| `.expandKwargs(rows)`, from a `kwargs` entry | one `xcom` with 
`element_index=i` and `spread` | `literal`: element `i`, spread into the 
arguments |
 | `.zip(a, b)` | `tuple`, one part per zipped argument | `literal`: the parts 
as one array |
 | `.concat(a, b)` | `xcom` for whichever argument owns this index | `literal`: 
that element |
 | `.map(f)` | the wrapped form's binding, with `apply_map` | see below |
@@ -419,6 +453,7 @@ What the server sends for each authoring form, and what the 
runtime receives aft
 An element of a dict is its `(key, value)` pair in the stored value's order, 
the pair a Python task reading the same XCom gets.
 A runtime receives it as a two-item array, such as `["uk", 3]`.
 With `.expandKwargs(…)`, a parameter missing from this index's dict takes its 
`partial()` value or its default, and fails the task with neither, as a missing 
argument does in Python.
+With `spread`, a key missing from the dict is left to the handler's own 
default.
 
 A Dag definition and a Python Dag with stub TaskHandler differ only in who 
applies `map()` callables.
 
@@ -449,7 +484,7 @@ Chained `map()` calls are one wrapper with several 
callables, applied in order.
 
 ```ts
 const clean = (row: string) => row.trim();
-dag.task("transform", transform).partial({ region: "eu" }).expand({ row: 
rows.map(clean) });
+dag.task(transform).partial({ region: "eu" }).expand({ row: rows.map(clean) });
 ```
 
 ```
@@ -524,6 +559,25 @@ Any other error fails the task, as does an 
`APPLY_MAP_ERROR` response to `ApplyM
 6. The server decides which producer rows are live and orders them iteration 
first, then map index. The runtime sends only the offset it wants.
 7. With a configured `[core] xcom_backend`, the supervisor's Python sidecar 
decodes each value on the way back.
 
+### A handler's own read of a mapped producer
+
+A handler can also read an upstream's XCom itself, through its SDK client's 
`getXCom`. A map index
+means the same there in every SDK. The server decides whether the producer is 
mapped from its
+serialized task, as it does for bindings, not from its rows, so a producer 
that expanded to nothing
+still reads as mapped:
+
+| Producer | Map index | The read returns |
+|---|---|---|
+| Not mapped | none | its value |
+| Not mapped | any | an error |
+| Mapped | none | every instance's value, as a list in map-index order; `[]` 
when it expanded to nothing |
+| Mapped | `0 <= i <` instance count | instance `i`'s value |
+| Mapped | any other, negative included | an error |
+
+An error is not a missing value: a read in range with nothing pushed still 
returns null.
+Unlike a reduce binding, the whole-output read is materialized, because the 
handler asked for the value.
+It is an Execution API mode of its own; Python's `XCom.get_one`, where no map 
index still means the `-1` row, is unchanged.
+
 ### Python sidecar subprocess for the XCom backend
 
 The supervisor runs `[core] xcom_backend` for a Lang-SDK task, so a runtime 
only sends and receives plain values.
@@ -540,7 +594,7 @@ What every SDK implements, in order. Nothing here is 
language-specific.
 
 1. Read `ti.map_index` and `ti_context.expanded_ti_count` from 
`StartupDetails`. A `map_index` of `-1` means the task is not mapped.
 2. Take the delivered binding for this map index: a `literal`, or a `sequence` 
for a reduce. A kind the SDK does not implement fails the task before the 
handler runs.
-3. For a binding with `apply_map`, get its `map()` callables run: a Dag 
definition applies its own, and a stub TaskHandler sends `ApplyMap`. Then take 
`kwarg`, or `fallback` when the key is missing. Run the callables once per 
instance, even when several parameters share them. A skip from them skips the 
instance.
+3. For a binding with `apply_map`, get its `map()` callables run: a Dag 
definition applies its own, and a stub TaskHandler sends `ApplyMap`. Then take 
`kwarg`, or `fallback` when the key is missing; with `spread`, merge the 
value's keys into the handler's arguments, failing the instance on one a 
`partial()` binding also sets. Run the callables once per instance, even when 
several parameters share them. A skip from them skips the instance.
 4. Optionally send `SetRenderedMapIndex` to label the instance.
 5. Run the handler with that one element.
 6. Push the return value. The supervisor adds the map length; the SDK sends 
none.
@@ -577,6 +631,15 @@ out a `None` default, where a typed model's natural 
spelling cannot tell an abse
 The flag says nothing the key's presence does not, every SDK's JSON library 
can write a present
 `null` with one idiom, and a second field admits entries where the two 
disagree.
 
+**A Dag definition lists the keys of an `expand_kwargs()` input, with Dag-side 
defaults.** It
+would repeat the handler's signature in the Dag for the one form that needs 
it, where a Dag
+definition's handler already is the signature. A type checker can hold the 
list to the handler's
+type, but not check that it is complete.
+
+**Handler defaults for a stub paired with a task handler.** The call is 
checked against the stub, so
+a default only the handler declares would let one Dag bind differently 
depending on which handler
+serves the task.
+
 **The server reads the XCom to pick an element.** Under an offloading XCom 
backend the row holds only a storage path,
 and every `ti_run` of a mapped Lang-SDK task would read XCom values. The 
supervisor already reads through the backend.
 

Reply via email to