Miretpl commented on code in PR #71808:
URL: https://github.com/apache/airflow/pull/71808#discussion_r3856035128
##########
providers/standard/docs/operators/bash.rst:
##########
@@ -226,6 +226,58 @@ Here's how you can use the result_processor with the
BashOperator:
)
+Multiple XCom outputs
+---------------------
+
+Pair ``output_processor`` with ``multiple_outputs=True`` to push more than one
XCom from a single task. When
+the processed output is a dictionary, each key is pushed as its own XCom,
which lets downstream tasks pull
+individual values by name instead of pulling the whole dictionary and indexing
into it.
+
+.. tab-set::
+
+ .. tab-item:: @task.bash
+ :sync: taskflow
+
+ .. exampleinclude::
/../src/airflow/providers/standard/example_dags/example_bash_decorator.py
+ :language: python
+ :dedent: 4
+ :start-after: [START howto_decorator_bash_multiple_outputs]
+ :end-before: [END howto_decorator_bash_multiple_outputs]
+
+ .. tab-item:: BashOperator
+ :sync: operator
+
+ .. exampleinclude::
/../src/airflow/providers/standard/example_dags/example_bash_operator.py
+ :language: python
+ :dedent: 4
+ :start-after: [START howto_operator_bash_multiple_outputs]
+ :end-before: [END howto_operator_bash_multiple_outputs]
+
+The producing task above pushes an XCom for ``dag_folder`` and one for
``file_count``. The full dictionary is
+*also* pushed as the task's return value, so ``{{
ti.xcom_pull(task_ids="describe_dag_folder") }}`` still
+resolves to ``{"dag_folder": ..., "file_count": ...}``.
+
+.. important::
+
+ Only the **last line** written by the command is captured, so the
dictionary must be the final thing the
+ command emits. A few consequences worth designing around:
Review Comment:
It that true? I checked the code and it looks to me that `output_processor`
gets whole command output and the `output_processor` outcome is returned from
the task (which is input for pushing xcom).
##########
providers/standard/docs/operators/bash.rst:
##########
@@ -226,6 +226,58 @@ Here's how you can use the result_processor with the
BashOperator:
)
+Multiple XCom outputs
Review Comment:
As far as I checked, we are missing similar doc for the other operators. The
doc regarding `multiple_outputs` is in the core itself, but maybe, as a
follow-up, it would be good to also add the respective doc here. WDYT?
##########
providers/standard/docs/operators/bash.rst:
##########
@@ -226,6 +226,58 @@ Here's how you can use the result_processor with the
BashOperator:
)
+Multiple XCom outputs
+---------------------
+
+Pair ``output_processor`` with ``multiple_outputs=True`` to push more than one
XCom from a single task. When
+the processed output is a dictionary, each key is pushed as its own XCom,
which lets downstream tasks pull
+individual values by name instead of pulling the whole dictionary and indexing
into it.
+
+.. tab-set::
+
+ .. tab-item:: @task.bash
+ :sync: taskflow
+
+ .. exampleinclude::
/../src/airflow/providers/standard/example_dags/example_bash_decorator.py
+ :language: python
+ :dedent: 4
+ :start-after: [START howto_decorator_bash_multiple_outputs]
+ :end-before: [END howto_decorator_bash_multiple_outputs]
+
+ .. tab-item:: BashOperator
+ :sync: operator
+
+ .. exampleinclude::
/../src/airflow/providers/standard/example_dags/example_bash_operator.py
+ :language: python
+ :dedent: 4
+ :start-after: [START howto_operator_bash_multiple_outputs]
+ :end-before: [END howto_operator_bash_multiple_outputs]
+
+The producing task above pushes an XCom for ``dag_folder`` and one for
``file_count``. The full dictionary is
+*also* pushed as the task's return value, so ``{{
ti.xcom_pull(task_ids="describe_dag_folder") }}`` still
+resolves to ``{"dag_folder": ..., "file_count": ...}``.
+
+.. important::
+
+ Only the **last line** written by the command is captured, so the
dictionary must be the final thing the
+ command emits. A few consequences worth designing around:
+
+ * A trailing ``echo`` with no arguments emits an empty line, which becomes
the captured output instead of
+ your dictionary.
+ * ``stderr`` is merged into ``stdout``, so a subcommand that writes to
``stderr`` last will overwrite the
+ captured value. Redirect noisy subcommands (for example ``2>/dev/null``)
to avoid this.
+ * The dictionary must fit on a single line. Use ``jq -c`` rather than
pretty-printed output, and prefer
+ ``printf`` over a multi-line ``printf`` format string.
Review Comment:
Connected to the above comment.
##########
providers/standard/docs/operators/bash.rst:
##########
@@ -226,6 +226,58 @@ Here's how you can use the result_processor with the
BashOperator:
)
+Multiple XCom outputs
+---------------------
+
+Pair ``output_processor`` with ``multiple_outputs=True`` to push more than one
XCom from a single task. When
+the processed output is a dictionary, each key is pushed as its own XCom,
which lets downstream tasks pull
+individual values by name instead of pulling the whole dictionary and indexing
into it.
+
+.. tab-set::
+
+ .. tab-item:: @task.bash
+ :sync: taskflow
+
+ .. exampleinclude::
/../src/airflow/providers/standard/example_dags/example_bash_decorator.py
+ :language: python
+ :dedent: 4
+ :start-after: [START howto_decorator_bash_multiple_outputs]
+ :end-before: [END howto_decorator_bash_multiple_outputs]
+
+ .. tab-item:: BashOperator
+ :sync: operator
+
+ .. exampleinclude::
/../src/airflow/providers/standard/example_dags/example_bash_operator.py
+ :language: python
+ :dedent: 4
+ :start-after: [START howto_operator_bash_multiple_outputs]
+ :end-before: [END howto_operator_bash_multiple_outputs]
+
+The producing task above pushes an XCom for ``dag_folder`` and one for
``file_count``. The full dictionary is
+*also* pushed as the task's return value, so ``{{
ti.xcom_pull(task_ids="describe_dag_folder") }}`` still
+resolves to ``{"dag_folder": ..., "file_count": ...}``.
+
+.. important::
+
+ Only the **last line** written by the command is captured, so the
dictionary must be the final thing the
+ command emits. A few consequences worth designing around:
+
+ * A trailing ``echo`` with no arguments emits an empty line, which becomes
the captured output instead of
+ your dictionary.
+ * ``stderr`` is merged into ``stdout``, so a subcommand that writes to
``stderr`` last will overwrite the
+ captured value. Redirect noisy subcommands (for example ``2>/dev/null``)
to avoid this.
+ * The dictionary must fit on a single line. Use ``jq -c`` rather than
pretty-printed output, and prefer
+ ``printf`` over a multi-line ``printf`` format string.
+ * Every line the command writes is sent to the task log, including the
line holding your values. Avoid
+ emitting secrets this way.
Review Comment:
It is rather general advice rather than scoped only for `Multiple XCom
outputs`. Could we move it to more suitable place?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: [email protected]
For queries about this service, please contact Infrastructure at:
[email protected]
