kaxil commented on code in PR #77080: URL: https://github.com/apache/airflow/pull/77080#discussion_r4608485726
########## airflow-core/adr/yaml-dags/0001-task-mapping-authoring-interface.md: ########## @@ -0,0 +1,211 @@ +<!-- + Licensed to the Apache Software Foundation (ASF) under one + or more contributor license agreements. See the NOTICE file + distributed with this work for additional information + regarding copyright ownership. The ASF licenses this file + to you under the Apache License, Version 2.0 (the + "License"); you may not use this file except in compliance + with the License. You may obtain a copy of the License at + + http://www.apache.org/licenses/LICENSE-2.0 + + Unless required by applicable law or agreed to in writing, + software distributed under the License is distributed on an + "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY + KIND, either express or implied. See the License for the + specific language governing permissions and limitations + under the License. + --> + +# ADR-0001: Task Mapping — YAML Authoring Interface + +## Status + +Proposed. + +## Context + +What a mapped task is, and the rules every authoring surface follows, are in [Language SDK ADR-0016](../lang-sdk/0016-mapped-tasks-across-lang-sdks.md). This ADR records their spelling in YAML. `map` is deferred since it requires a way to reference a callable (see Decision 4 below). + +YAML has no method calls to chain. ADR-0016 makes `zip` and `concat` methods on a reference, so they chain in declaration order. YAML nests them instead. Since YAML already marks a value in place (`{$x: ...}` for an upstream output), an expanded argument is marked the same way: + +```yaml +tasks: + # Returns e.g. [1, 2, 3] + - id: list_files + run: {} + + # Equivalent to process.partial(bucket="raw").expand(path=list_files) + - id: process + run: + bucket: raw + path: {$expand: {$x: list_files}} + + # Reducing; equivalent to summarize(sizes=process) + - id: summarize + run: + sizes: {$x: process} +``` + +## Decision + +1. **An expanded argument is marked in place with `$expand`. Every unmarked argument is a `partial()`.** + - For example, `path: {$expand: <source>}` under `with:` or `run:` is Python's `.expand(path=<source>)`. + - Several `$expand` arguments give a cross product, as several `expand()` keywords do. It is not allowed to nest `$expand` inside a construct. + +2. **Expansion is possible on a literal list, a `$x`, or one of `$zip` and `$concat`.** + - A list gives one instance per item. Its items are literal values; a marker inside it is an error. Any other literal is an error. + - `{$x: task}` expands over the task's return value. When `task` is itself mapped, the instances line up one-to-one by map index. `$x` with a `key` is rejected, as in Python. Review Comment: This isn't quite what Python does. A mapped producer instance that returns `None` writes no XCom row, so the consumer gets one instance per surviving result, and instance `i` reads the `i`-th row of the sequence rather than producer map index `i`. Can this say "one instance per non-`None` result, in map-index order"? The `map_index=i` row in ADR-0016's read-path table makes the same assumption. ########## airflow-core/adr/yaml-dags/0001-task-mapping-authoring-interface.md: ########## @@ -0,0 +1,211 @@ +<!-- + Licensed to the Apache Software Foundation (ASF) under one + or more contributor license agreements. See the NOTICE file + distributed with this work for additional information + regarding copyright ownership. The ASF licenses this file + to you under the Apache License, Version 2.0 (the + "License"); you may not use this file except in compliance + with the License. You may obtain a copy of the License at + + http://www.apache.org/licenses/LICENSE-2.0 + + Unless required by applicable law or agreed to in writing, + software distributed under the License is distributed on an + "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY + KIND, either express or implied. See the License for the + specific language governing permissions and limitations + under the License. + --> + +# ADR-0001: Task Mapping — YAML Authoring Interface + +## Status + +Proposed. + +## Context + +What a mapped task is, and the rules every authoring surface follows, are in [Language SDK ADR-0016](../lang-sdk/0016-mapped-tasks-across-lang-sdks.md). This ADR records their spelling in YAML. `map` is deferred since it requires a way to reference a callable (see Decision 4 below). + +YAML has no method calls to chain. ADR-0016 makes `zip` and `concat` methods on a reference, so they chain in declaration order. YAML nests them instead. Since YAML already marks a value in place (`{$x: ...}` for an upstream output), an expanded argument is marked the same way: + +```yaml +tasks: + # Returns e.g. [1, 2, 3] + - id: list_files + run: {} + + # Equivalent to process.partial(bucket="raw").expand(path=list_files) + - id: process + run: + bucket: raw + path: {$expand: {$x: list_files}} + + # Reducing; equivalent to summarize(sizes=process) + - id: summarize + run: + sizes: {$x: process} +``` + +## Decision + +1. **An expanded argument is marked in place with `$expand`. Every unmarked argument is a `partial()`.** + - For example, `path: {$expand: <source>}` under `with:` or `run:` is Python's `.expand(path=<source>)`. + - Several `$expand` arguments give a cross product, as several `expand()` keywords do. It is not allowed to nest `$expand` inside a construct. + +2. **Expansion is possible on a literal list, a `$x`, or one of `$zip` and `$concat`.** + - A list gives one instance per item. Its items are literal values; a marker inside it is an error. Any other literal is an error. + - `{$x: task}` expands over the task's return value. When `task` is itself mapped, the instances line up one-to-one by map index. `$x` with a `key` is rejected, as in Python. + +3. **`$zip` and `$concat` spell Python's `zip()` and `concat()`.** + - `{$zip: [a, b, ...]}`, optionally with `default: v` beside the `$zip` key. Without `default`, the zip is as long as its shortest source; with it, as long as its longest, and `v` stands in for the missing items of the shorter ones. + - A marker's operand is the value of its `$` key, and its options are plain keys beside it. A key that is not one of that marker's options is an error. + - `{$concat: [a, b, ...]}`. + - What each takes as an item, and where either may appear, follows Language SDK ADR-0016 (Decision 4 and its authoring rules). + +4. **`map()` is deferred until YAML can reference a callable.** + - YAML cannot define one, and no mechanism yet references a callable in a task handler's language, so `$map` is rejected for now. + - When one exists, the intended spelling is `{$map: v, apply: f}`: the source is the operand, as for `$zip` and `$concat`, and `apply` is one callable. Nesting `$map` such as `{$map: {$map: v, apply: f1}, apply: f2}` is allowed. + +5. **`$expand` as a key of `with:` or `run:` is Python's `expand_kwargs()`.** + + ```yaml + id: foo + run: + path: {$: expand} + bucket: raw + size: {$: expand} + $expand: {$x: rows} + ``` + + is roughly equivalent to + + ```python + @task.stub + def foo(path, bucket, size): ... + + + foo.partial(bucket="raw").expand_kwargs(rows) + ``` + + - The value of a `$expand` indicates the source, as in Decision 2, except that a literal list's items must be mappings or `$x`. Each element is one instance's keyword arguments. Review Comment: Since this reuses Decision 2's sources, `$zip` is allowed here, but each zipped element is a tuple and `expand_kwargs()` fails on it (`expects a list[dict], not list[tuple]`). Should the key form reject `$zip` until `$map` exists to turn the tuples into mappings? ########## airflow-core/adr/yaml-dags/0001-task-mapping-authoring-interface.md: ########## @@ -0,0 +1,211 @@ +<!-- + Licensed to the Apache Software Foundation (ASF) under one + or more contributor license agreements. See the NOTICE file + distributed with this work for additional information + regarding copyright ownership. The ASF licenses this file + to you under the Apache License, Version 2.0 (the + "License"); you may not use this file except in compliance + with the License. You may obtain a copy of the License at + + http://www.apache.org/licenses/LICENSE-2.0 + + Unless required by applicable law or agreed to in writing, + software distributed under the License is distributed on an + "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY + KIND, either express or implied. See the License for the + specific language governing permissions and limitations + under the License. + --> + +# ADR-0001: Task Mapping — YAML Authoring Interface + +## Status + +Proposed. + +## Context + +What a mapped task is, and the rules every authoring surface follows, are in [Language SDK ADR-0016](../lang-sdk/0016-mapped-tasks-across-lang-sdks.md). This ADR records their spelling in YAML. `map` is deferred since it requires a way to reference a callable (see Decision 4 below). + +YAML has no method calls to chain. ADR-0016 makes `zip` and `concat` methods on a reference, so they chain in declaration order. YAML nests them instead. Since YAML already marks a value in place (`{$x: ...}` for an upstream output), an expanded argument is marked the same way: + +```yaml +tasks: + # Returns e.g. [1, 2, 3] + - id: list_files + run: {} + + # Equivalent to process.partial(bucket="raw").expand(path=list_files) + - id: process + run: + bucket: raw + path: {$expand: {$x: list_files}} + + # Reducing; equivalent to summarize(sizes=process) + - id: summarize + run: + sizes: {$x: process} +``` + +## Decision + +1. **An expanded argument is marked in place with `$expand`. Every unmarked argument is a `partial()`.** + - For example, `path: {$expand: <source>}` under `with:` or `run:` is Python's `.expand(path=<source>)`. + - Several `$expand` arguments give a cross product, as several `expand()` keywords do. It is not allowed to nest `$expand` inside a construct. + +2. **Expansion is possible on a literal list, a `$x`, or one of `$zip` and `$concat`.** + - A list gives one instance per item. Its items are literal values; a marker inside it is an error. Any other literal is an error. + - `{$x: task}` expands over the task's return value. When `task` is itself mapped, the instances line up one-to-one by map index. `$x` with a `key` is rejected, as in Python. + +3. **`$zip` and `$concat` spell Python's `zip()` and `concat()`.** + - `{$zip: [a, b, ...]}`, optionally with `default: v` beside the `$zip` key. Without `default`, the zip is as long as its shortest source; with it, as long as its longest, and `v` stands in for the missing items of the shorter ones. + - A marker's operand is the value of its `$` key, and its options are plain keys beside it. A key that is not one of that marker's options is an error. + - `{$concat: [a, b, ...]}`. + - What each takes as an item, and where either may appear, follows Language SDK ADR-0016 (Decision 4 and its authoring rules). + +4. **`map()` is deferred until YAML can reference a callable.** + - YAML cannot define one, and no mechanism yet references a callable in a task handler's language, so `$map` is rejected for now. + - When one exists, the intended spelling is `{$map: v, apply: f}`: the source is the operand, as for `$zip` and `$concat`, and `apply` is one callable. Nesting `$map` such as `{$map: {$map: v, apply: f1}, apply: f2}` is allowed. + +5. **`$expand` as a key of `with:` or `run:` is Python's `expand_kwargs()`.** + + ```yaml + id: foo + run: + path: {$: expand} + bucket: raw + size: {$: expand} + $expand: {$x: rows} + ``` + + is roughly equivalent to + + ```python + @task.stub + def foo(path, bucket, size): ... + + + foo.partial(bucket="raw").expand_kwargs(rows) + ``` + + - The value of a `$expand` indicates the source, as in Decision 2, except that a literal list's items must be mappings or `$x`. Each element is one instance's keyword arguments. + - On a `uses:` task, the element's keys are the operator's arguments. + - On a `run:` task, the `run:` keys are the handler's parameters, in order. `{$: expand}` (Decision 6) marks the ones set by `$expand` elements, and there must be at least one. `{$: expand, default: v}` gives the argument a default, used when an element leaves it out; without one, such an element fails the instance. + - A task may use either this form, or argument-level `$expand`s, never both. + - Any other key starting with `$` in `with:` or `run:` is an error. + +6. **`{$: ...}` as a keyword marker.** + - The key `$` marks the value as a keyword naming the marker, for markers that need no argument of their own. + - `{$: expand}` stands for "this argument comes from each element" and is valid only as an argument's whole value in a `run:` task that has the `$expand` key after merging; a template may carry it for the tasks that extend it (Decision 10). It has one option `default` (see Decision 5). + +7. **A reduce is implicit.** + - `{$x: task}` of a mapped task outside `$expand` gives the consumer every instance's result, as ADR-0016 Decision 8's lazy sequence. + +8. **On a `uses:` operator, any constructor argument can be expanded**. + - This applies whether or not the value is templated. An expanded value is never rendered. + +9. **The instance label follows the body.** + - A `uses:` task keeps `map_index_template`. A `run:` task rejects it; its handler labels itself (ADR-0016 Decision 7). + +10. **A template can declare a signature, but cannot map.** + - A template may contain `{$: expand}`, so tasks can share one handler signature: its arguments, their order, and which ones each element sets. + - `$expand`, as a key or as an argument's value, is allowed only on a concrete task. Whether a task is mapped is therefore always visible on the task itself. + - A task that inherits `{$: expand}` but has no `$expand` key is an error naming the template; one that overrides every inherited `{$: expand}` with a value is an ordinary unmapped task. + +## More examples + +The less common forms, each with its Python equivalent. + +```yaml +tasks: + # a cross product: four instances + - id: process_grid + run: + path: {$expand: [a.csv, b.csv]} + bucket: {$expand: [raw, eu]} + + # one instance per process instance + - id: audit + run: + size: {$expand: {$x: process}} + + # one [key, value] pair per instance + - id: tag + run: + entry: {$expand: [[env, prod], [team, data]]} + + # .expand(p=list_files.output.zip(process.output, fillvalue=None)) + - id: pair + run: + p: {$expand: {$zip: [{$x: list_files}, {$x: process}], default: null}} + + # ProcessOperator.partial(bucket="raw").expand_kwargs(rows.output) + - id: process_rows + uses: pkg.ProcessOperator + with: + bucket: raw + $expand: {$x: rows} # each row's keys are operator arguments + + # BashOperator.partial(env=...).expand(bash_command=commands.output) + - id: run_commands + uses: airflow.providers.standard.operators.bash.BashOperator + map_index_template: "{{ task.bash_command }}" + with: + env: {MODE: batch} + bash_command: {$expand: {$x: commands}} +``` + +A template can declare a signature for several tasks. Each task supplies its own `$expand`, and an overridden argument keeps the position the template gave it. + +```yaml +templates: + process_sig: + run: + path: {$: expand} + bucket: raw + size: {$: expand, default: 0} + +tasks: + - id: process_a + extends: [process_sig] + run: + $expand: {$x: rows_a} + + - id: process_b + extends: [process_sig] + run: + bucket: eu # replaces the partial in place + $expand: {$x: rows_b} + + - id: process_one # the same signature, not mapped + extends: [process_sig] + run: + path: only.csv + size: 10 +``` + +## Consequences + +- **A mapped task reads like an unmapped one with marked values**, and templates and `extends` need no new merge rule. +- **A signature can be shared through a template, but a source cannot.** A task repeats its `$expand` line. Allowing `$expand` in templates later only adds syntax. +- **A row key that names no argument fails the instance on both bodies**: the operator's constructor rejects it on `uses:`, the supervisor on `run:`. Review Comment: How does the supervisor reject an unknown row key on `run:`? Under ADR-0016 each parameter's binding takes its own `kwarg` from the row and falls back from there, so nothing looks at the row's other keys, and a typo like `sise` just runs with the default. Same for a row key that names a partial argument like `bucket`: Python's `expand_kwargs()` rejects that through `prevent_duplicates`, so it fails on `uses:`, but nothing rejects it on `run:`. Should this ADR (or 0016) add a whole-row check so both bodies fail the instance? ########## airflow-core/adr/yaml-dags/0001-task-mapping-authoring-interface.md: ########## @@ -0,0 +1,211 @@ +<!-- + Licensed to the Apache Software Foundation (ASF) under one + or more contributor license agreements. See the NOTICE file + distributed with this work for additional information + regarding copyright ownership. The ASF licenses this file + to you under the Apache License, Version 2.0 (the + "License"); you may not use this file except in compliance + with the License. You may obtain a copy of the License at + + http://www.apache.org/licenses/LICENSE-2.0 + + Unless required by applicable law or agreed to in writing, + software distributed under the License is distributed on an + "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY + KIND, either express or implied. See the License for the + specific language governing permissions and limitations + under the License. + --> + +# ADR-0001: Task Mapping — YAML Authoring Interface + +## Status + +Proposed. + +## Context + +What a mapped task is, and the rules every authoring surface follows, are in [Language SDK ADR-0016](../lang-sdk/0016-mapped-tasks-across-lang-sdks.md). This ADR records their spelling in YAML. `map` is deferred since it requires a way to reference a callable (see Decision 4 below). + +YAML has no method calls to chain. ADR-0016 makes `zip` and `concat` methods on a reference, so they chain in declaration order. YAML nests them instead. Since YAML already marks a value in place (`{$x: ...}` for an upstream output), an expanded argument is marked the same way: + +```yaml +tasks: + # Returns e.g. [1, 2, 3] + - id: list_files + run: {} + + # Equivalent to process.partial(bucket="raw").expand(path=list_files) + - id: process + run: + bucket: raw + path: {$expand: {$x: list_files}} + + # Reducing; equivalent to summarize(sizes=process) + - id: summarize + run: + sizes: {$x: process} +``` + +## Decision + +1. **An expanded argument is marked in place with `$expand`. Every unmarked argument is a `partial()`.** + - For example, `path: {$expand: <source>}` under `with:` or `run:` is Python's `.expand(path=<source>)`. + - Several `$expand` arguments give a cross product, as several `expand()` keywords do. It is not allowed to nest `$expand` inside a construct. + +2. **Expansion is possible on a literal list, a `$x`, or one of `$zip` and `$concat`.** + - A list gives one instance per item. Its items are literal values; a marker inside it is an error. Any other literal is an error. + - `{$x: task}` expands over the task's return value. When `task` is itself mapped, the instances line up one-to-one by map index. `$x` with a `key` is rejected, as in Python. + +3. **`$zip` and `$concat` spell Python's `zip()` and `concat()`.** + - `{$zip: [a, b, ...]}`, optionally with `default: v` beside the `$zip` key. Without `default`, the zip is as long as its shortest source; with it, as long as its longest, and `v` stands in for the missing items of the shorter ones. + - A marker's operand is the value of its `$` key, and its options are plain keys beside it. A key that is not one of that marker's options is an error. Review Comment: `$x` already puts its option inside the operand (`{$x: {task: extract, key: rows}}`), so this reads as a rule it breaks. Is it meant only for the new markers? ########## airflow-core/adr/yaml-dags/0001-task-mapping-authoring-interface.md: ########## @@ -0,0 +1,211 @@ +<!-- + Licensed to the Apache Software Foundation (ASF) under one + or more contributor license agreements. See the NOTICE file + distributed with this work for additional information + regarding copyright ownership. The ASF licenses this file + to you under the Apache License, Version 2.0 (the + "License"); you may not use this file except in compliance + with the License. You may obtain a copy of the License at + + http://www.apache.org/licenses/LICENSE-2.0 + + Unless required by applicable law or agreed to in writing, + software distributed under the License is distributed on an + "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY + KIND, either express or implied. See the License for the + specific language governing permissions and limitations + under the License. + --> + +# ADR-0001: Task Mapping — YAML Authoring Interface + +## Status + +Proposed. + +## Context + +What a mapped task is, and the rules every authoring surface follows, are in [Language SDK ADR-0016](../lang-sdk/0016-mapped-tasks-across-lang-sdks.md). This ADR records their spelling in YAML. `map` is deferred since it requires a way to reference a callable (see Decision 4 below). + +YAML has no method calls to chain. ADR-0016 makes `zip` and `concat` methods on a reference, so they chain in declaration order. YAML nests them instead. Since YAML already marks a value in place (`{$x: ...}` for an upstream output), an expanded argument is marked the same way: + +```yaml +tasks: + # Returns e.g. [1, 2, 3] + - id: list_files + run: {} + + # Equivalent to process.partial(bucket="raw").expand(path=list_files) + - id: process + run: + bucket: raw + path: {$expand: {$x: list_files}} + + # Reducing; equivalent to summarize(sizes=process) + - id: summarize + run: + sizes: {$x: process} +``` + +## Decision + +1. **An expanded argument is marked in place with `$expand`. Every unmarked argument is a `partial()`.** + - For example, `path: {$expand: <source>}` under `with:` or `run:` is Python's `.expand(path=<source>)`. + - Several `$expand` arguments give a cross product, as several `expand()` keywords do. It is not allowed to nest `$expand` inside a construct. + +2. **Expansion is possible on a literal list, a `$x`, or one of `$zip` and `$concat`.** + - A list gives one instance per item. Its items are literal values; a marker inside it is an error. Any other literal is an error. + - `{$x: task}` expands over the task's return value. When `task` is itself mapped, the instances line up one-to-one by map index. `$x` with a `key` is rejected, as in Python. + +3. **`$zip` and `$concat` spell Python's `zip()` and `concat()`.** + - `{$zip: [a, b, ...]}`, optionally with `default: v` beside the `$zip` key. Without `default`, the zip is as long as its shortest source; with it, as long as its longest, and `v` stands in for the missing items of the shorter ones. + - A marker's operand is the value of its `$` key, and its options are plain keys beside it. A key that is not one of that marker's options is an error. + - `{$concat: [a, b, ...]}`. + - What each takes as an item, and where either may appear, follows Language SDK ADR-0016 (Decision 4 and its authoring rules). + +4. **`map()` is deferred until YAML can reference a callable.** + - YAML cannot define one, and no mechanism yet references a callable in a task handler's language, so `$map` is rejected for now. + - When one exists, the intended spelling is `{$map: v, apply: f}`: the source is the operand, as for `$zip` and `$concat`, and `apply` is one callable. Nesting `$map` such as `{$map: {$map: v, apply: f1}, apply: f2}` is allowed. + +5. **`$expand` as a key of `with:` or `run:` is Python's `expand_kwargs()`.** + + ```yaml + id: foo + run: + path: {$: expand} + bucket: raw + size: {$: expand} + $expand: {$x: rows} + ``` + + is roughly equivalent to + + ```python + @task.stub + def foo(path, bucket, size): ... + + + foo.partial(bucket="raw").expand_kwargs(rows) + ``` + + - The value of a `$expand` indicates the source, as in Decision 2, except that a literal list's items must be mappings or `$x`. Each element is one instance's keyword arguments. + - On a `uses:` task, the element's keys are the operator's arguments. + - On a `run:` task, the `run:` keys are the handler's parameters, in order. `{$: expand}` (Decision 6) marks the ones set by `$expand` elements, and there must be at least one. `{$: expand, default: v}` gives the argument a default, used when an element leaves it out; without one, such an element fails the instance. + - A task may use either this form, or argument-level `$expand`s, never both. + - Any other key starting with `$` in `with:` or `run:` is an error. + +6. **`{$: ...}` as a keyword marker.** + - The key `$` marks the value as a keyword naming the marker, for markers that need no argument of their own. Review Comment: Are other `{$: ...}` keywords planned? With `expand` as the only one, a dedicated marker name would avoid the `{$: expand}` / `{$expand: ...}` near-miss that line 193 has to special-case. ########## airflow-core/adr/yaml-dags/0001-task-mapping-authoring-interface.md: ########## @@ -0,0 +1,211 @@ +<!-- + Licensed to the Apache Software Foundation (ASF) under one + or more contributor license agreements. See the NOTICE file + distributed with this work for additional information + regarding copyright ownership. The ASF licenses this file + to you under the Apache License, Version 2.0 (the + "License"); you may not use this file except in compliance + with the License. You may obtain a copy of the License at + + http://www.apache.org/licenses/LICENSE-2.0 + + Unless required by applicable law or agreed to in writing, + software distributed under the License is distributed on an + "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY + KIND, either express or implied. See the License for the + specific language governing permissions and limitations + under the License. + --> + +# ADR-0001: Task Mapping — YAML Authoring Interface + +## Status + +Proposed. + +## Context + +What a mapped task is, and the rules every authoring surface follows, are in [Language SDK ADR-0016](../lang-sdk/0016-mapped-tasks-across-lang-sdks.md). This ADR records their spelling in YAML. `map` is deferred since it requires a way to reference a callable (see Decision 4 below). + +YAML has no method calls to chain. ADR-0016 makes `zip` and `concat` methods on a reference, so they chain in declaration order. YAML nests them instead. Since YAML already marks a value in place (`{$x: ...}` for an upstream output), an expanded argument is marked the same way: + +```yaml +tasks: + # Returns e.g. [1, 2, 3] + - id: list_files + run: {} + + # Equivalent to process.partial(bucket="raw").expand(path=list_files) + - id: process + run: + bucket: raw + path: {$expand: {$x: list_files}} + + # Reducing; equivalent to summarize(sizes=process) + - id: summarize + run: + sizes: {$x: process} +``` + +## Decision + +1. **An expanded argument is marked in place with `$expand`. Every unmarked argument is a `partial()`.** + - For example, `path: {$expand: <source>}` under `with:` or `run:` is Python's `.expand(path=<source>)`. + - Several `$expand` arguments give a cross product, as several `expand()` keywords do. It is not allowed to nest `$expand` inside a construct. + +2. **Expansion is possible on a literal list, a `$x`, or one of `$zip` and `$concat`.** + - A list gives one instance per item. Its items are literal values; a marker inside it is an error. Any other literal is an error. + - `{$x: task}` expands over the task's return value. When `task` is itself mapped, the instances line up one-to-one by map index. `$x` with a `key` is rejected, as in Python. + +3. **`$zip` and `$concat` spell Python's `zip()` and `concat()`.** + - `{$zip: [a, b, ...]}`, optionally with `default: v` beside the `$zip` key. Without `default`, the zip is as long as its shortest source; with it, as long as its longest, and `v` stands in for the missing items of the shorter ones. + - A marker's operand is the value of its `$` key, and its options are plain keys beside it. A key that is not one of that marker's options is an error. + - `{$concat: [a, b, ...]}`. + - What each takes as an item, and where either may appear, follows Language SDK ADR-0016 (Decision 4 and its authoring rules). + +4. **`map()` is deferred until YAML can reference a callable.** + - YAML cannot define one, and no mechanism yet references a callable in a task handler's language, so `$map` is rejected for now. + - When one exists, the intended spelling is `{$map: v, apply: f}`: the source is the operand, as for `$zip` and `$concat`, and `apply` is one callable. Nesting `$map` such as `{$map: {$map: v, apply: f1}, apply: f2}` is allowed. + +5. **`$expand` as a key of `with:` or `run:` is Python's `expand_kwargs()`.** + + ```yaml + id: foo + run: + path: {$: expand} + bucket: raw + size: {$: expand} + $expand: {$x: rows} + ``` + + is roughly equivalent to + + ```python + @task.stub + def foo(path, bucket, size): ... + + + foo.partial(bucket="raw").expand_kwargs(rows) + ``` + + - The value of a `$expand` indicates the source, as in Decision 2, except that a literal list's items must be mappings or `$x`. Each element is one instance's keyword arguments. + - On a `uses:` task, the element's keys are the operator's arguments. + - On a `run:` task, the `run:` keys are the handler's parameters, in order. `{$: expand}` (Decision 6) marks the ones set by `$expand` elements, and there must be at least one. `{$: expand, default: v}` gives the argument a default, used when an element leaves it out; without one, such an element fails the instance. + - A task may use either this form, or argument-level `$expand`s, never both. + - Any other key starting with `$` in `with:` or `run:` is an error. + +6. **`{$: ...}` as a keyword marker.** + - The key `$` marks the value as a keyword naming the marker, for markers that need no argument of their own. + - `{$: expand}` stands for "this argument comes from each element" and is valid only as an argument's whole value in a `run:` task that has the `$expand` key after merging; a template may carry it for the tasks that extend it (Decision 10). It has one option `default` (see Decision 5). + +7. **A reduce is implicit.** + - `{$x: task}` of a mapped task outside `$expand` gives the consumer every instance's result, as ADR-0016 Decision 8's lazy sequence. + +8. **On a `uses:` operator, any constructor argument can be expanded**. Review Comment: Python's `expand()` also takes `BaseOperator` arguments (`retries`, `pool`, `executor_config`), but the importer only allows those at task level, where markers are rejected. Is expanding those out of scope? If so, "any argument under `with:`" would say what's actually supported. ########## airflow-core/adr/yaml-dags/0001-task-mapping-authoring-interface.md: ########## @@ -0,0 +1,211 @@ +<!-- + Licensed to the Apache Software Foundation (ASF) under one + or more contributor license agreements. See the NOTICE file + distributed with this work for additional information + regarding copyright ownership. The ASF licenses this file + to you under the Apache License, Version 2.0 (the + "License"); you may not use this file except in compliance + with the License. You may obtain a copy of the License at + + http://www.apache.org/licenses/LICENSE-2.0 + + Unless required by applicable law or agreed to in writing, + software distributed under the License is distributed on an + "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY + KIND, either express or implied. See the License for the + specific language governing permissions and limitations + under the License. + --> + +# ADR-0001: Task Mapping — YAML Authoring Interface + +## Status + +Proposed. + +## Context + +What a mapped task is, and the rules every authoring surface follows, are in [Language SDK ADR-0016](../lang-sdk/0016-mapped-tasks-across-lang-sdks.md). This ADR records their spelling in YAML. `map` is deferred since it requires a way to reference a callable (see Decision 4 below). + +YAML has no method calls to chain. ADR-0016 makes `zip` and `concat` methods on a reference, so they chain in declaration order. YAML nests them instead. Since YAML already marks a value in place (`{$x: ...}` for an upstream output), an expanded argument is marked the same way: + +```yaml +tasks: + # Returns e.g. [1, 2, 3] + - id: list_files + run: {} + + # Equivalent to process.partial(bucket="raw").expand(path=list_files) + - id: process + run: + bucket: raw + path: {$expand: {$x: list_files}} + + # Reducing; equivalent to summarize(sizes=process) + - id: summarize + run: + sizes: {$x: process} +``` + +## Decision + +1. **An expanded argument is marked in place with `$expand`. Every unmarked argument is a `partial()`.** + - For example, `path: {$expand: <source>}` under `with:` or `run:` is Python's `.expand(path=<source>)`. + - Several `$expand` arguments give a cross product, as several `expand()` keywords do. It is not allowed to nest `$expand` inside a construct. + +2. **Expansion is possible on a literal list, a `$x`, or one of `$zip` and `$concat`.** + - A list gives one instance per item. Its items are literal values; a marker inside it is an error. Any other literal is an error. Review Comment: Python and ADR-0016 both expand a dict into `(key, value)` pairs, and the `tag` example below works around this with a list of pairs. Is a literal mapping rejected because it would be ambiguous with marker mappings like `{$x: ...}`? Worth a line under Rejected Alternatives either way. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected]
