uranusjr commented on code in PR #77080:
URL: https://github.com/apache/airflow/pull/77080#discussion_r4625643110


##########
airflow-core/adr/yaml-dags/0001-task-mapping-authoring-interface.md:
##########
@@ -0,0 +1,211 @@
+<!--
+ Licensed to the Apache Software Foundation (ASF) under one
+ or more contributor license agreements.  See the NOTICE file
+ distributed with this work for additional information
+ regarding copyright ownership.  The ASF licenses this file
+ to you under the Apache License, Version 2.0 (the
+ "License"); you may not use this file except in compliance
+ with the License.  You may obtain a copy of the License at
+
+   http://www.apache.org/licenses/LICENSE-2.0
+
+ Unless required by applicable law or agreed to in writing,
+ software distributed under the License is distributed on an
+ "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+ KIND, either express or implied.  See the License for the
+ specific language governing permissions and limitations
+ under the License.
+ -->
+
+# ADR-0001: Task Mapping — YAML Authoring Interface
+
+## Status
+
+Proposed.
+
+## Context
+
+What a mapped task is, and the rules every authoring surface follows, are in 
[Language SDK ADR-0016](../lang-sdk/0016-mapped-tasks-across-lang-sdks.md). 
This ADR records their spelling in YAML. `map` is deferred since it requires a 
way to reference a callable (see Decision 4 below).
+
+YAML has no method calls to chain. ADR-0016 makes `zip` and `concat` methods 
on a reference, so they chain in declaration order. YAML nests them instead. 
Since YAML already marks a value in place (`{$x: ...}` for an upstream output), 
an expanded argument is marked the same way:
+
+```yaml
+tasks:
+  # Returns e.g. [1, 2, 3]
+  - id: list_files
+    run: {}
+
+  # Equivalent to process.partial(bucket="raw").expand(path=list_files)
+  - id: process
+    run:
+      bucket: raw
+      path: {$expand: {$x: list_files}}
+
+  # Reducing; equivalent to summarize(sizes=process)
+  - id: summarize
+    run:
+      sizes: {$x: process}
+```
+
+## Decision
+
+1. **An expanded argument is marked in place with `$expand`. Every unmarked 
argument is a `partial()`.**
+   - For example, `path: {$expand: <source>}` under `with:` or `run:` is 
Python's `.expand(path=<source>)`.
+   - Several `$expand` arguments give a cross product, as several `expand()` 
keywords do. It is not allowed to nest `$expand` inside a construct.
+
+2. **Expansion is possible on a literal list, a `$x`, or one of `$zip` and 
`$concat`.**
+   - A list gives one instance per item. Its items are literal values; a 
marker inside it is an error. Any other literal is an error.
+   - `{$x: task}` expands over the task's return value. When `task` is itself 
mapped, the instances line up one-to-one by map index. `$x` with a `key` is 
rejected, as in Python.
+
+3. **`$zip` and `$concat` spell Python's `zip()` and `concat()`.**
+   - `{$zip: [a, b, ...]}`, optionally with `default: v` beside the `$zip` 
key. Without `default`, the zip is as long as its shortest source; with it, as 
long as its longest, and `v` stands in for the missing items of the shorter 
ones.
+   - A marker's operand is the value of its `$` key, and its options are plain 
keys beside it. A key that is not one of that marker's options is an error.
+   - `{$concat: [a, b, ...]}`.
+   - What each takes as an item, and where either may appear, follows Language 
SDK ADR-0016 (Decision 4 and its authoring rules).
+
+4. **`map()` is deferred until YAML can reference a callable.**
+   - YAML cannot define one, and no mechanism yet references a callable in a 
task handler's language, so `$map` is rejected for now.
+   - When one exists, the intended spelling is `{$map: v, apply: f}`: the 
source is the operand, as for `$zip` and `$concat`, and `apply` is one 
callable. Nesting `$map` such as `{$map: {$map: v, apply: f1}, apply: f2}` is 
allowed.
+
+5. **`$expand` as a key of `with:` or `run:` is Python's `expand_kwargs()`.**
+
+     ```yaml
+     id: foo
+     run:
+       path: {$: expand}
+       bucket: raw
+       size: {$: expand}
+       $expand: {$x: rows}
+     ```
+
+     is roughly equivalent to
+
+     ```python
+     @task.stub
+     def foo(path, bucket, size): ...
+
+
+     foo.partial(bucket="raw").expand_kwargs(rows)
+     ```
+
+   - The value of a `$expand` indicates the source, as in Decision 2, except 
that a literal list's items must be mappings or `$x`. Each element is one 
instance's keyword arguments.
+   - On a `uses:` task, the element's keys are the operator's arguments.
+   - On a `run:` task, the `run:` keys are the handler's parameters, in order. 
`{$: expand}` (Decision 6) marks the ones set by `$expand` elements, and there 
must be at least one. `{$: expand, default: v}` gives the argument a default, 
used when an element leaves it out; without one, such an element fails the 
instance.
+   - A task may use either this form, or argument-level `$expand`s, never both.
+   - Any other key starting with `$` in `with:` or `run:` is an error.
+
+6. **`{$: ...}` as a keyword marker.**
+   - The key `$` marks the value as a keyword naming the marker, for markers 
that need no argument of their own.
+   - `{$: expand}` stands for "this argument comes from each element" and is 
valid only as an argument's whole value in a `run:` task that has the `$expand` 
key after merging; a template may carry it for the tasks that extend it 
(Decision 10). It has one option `default` (see Decision 5).
+
+7. **A reduce is implicit.**
+   - `{$x: task}` of a mapped task outside `$expand` gives the consumer every 
instance's result, as ADR-0016 Decision 8's lazy sequence.
+
+8. **On a `uses:` operator, any constructor argument can be expanded**.

Review Comment:
   I reworded this to explicitly say only fields in `with:` can be expanded. 
Other arguments (BaseOperator constructor arguments) mostly aren’t supported in 
the first place (also rejected by Python). Only three can _technically_ work 
with `expand`: `do_xcom_push`, `multiple_outputs`, 
`render_template_as_native_obj`. The first two only make sense for TaskFlow 
tasks, not operators. The last is the only one that can be marginally useful, 
but it’s still not particularly practical to want to run an operator _both_ 
with it turned on and off. So all top-level fields will simply be excluded from 
being expand-able in YAML.



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: [email protected]

For queries about this service, please contact Infrastructure at:
[email protected]

Reply via email to