kacpermuda commented on code in PR #74752: URL: https://github.com/apache/airflow/pull/74752#discussion_r4325815647
########## providers/openlineage/tests/system/openlineage/example_openlineage_task_loop_dag.py: ########## @@ -0,0 +1,151 @@ +# Licensed to the Apache Software Foundation (ASF) under one +# or more contributor license agreements. See the NOTICE file +# distributed with this work for additional information +# regarding copyright ownership. The ASF licenses this file +# to you under the Apache License, Version 2.0 (the +# "License"); you may not use this file except in compliance +# with the License. You may obtain a copy of the License at +# +# http://www.apache.org/licenses/LICENSE-2.0 +# +# Unless required by applicable law or agreed to in writing, +# software distributed under the License is distributed on an +# "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY +# KIND, either express or implied. See the License for the +# specific language governing permissions and limitations +# under the License. +""" +DAG with a plain task and task loops, some of their tasks retrying and one stopped early by `until` +(requires Airflow 3.4+). + +It checks, for a plain task, tasks, a mapped task and a mapped task group in a loop, and loop gates: + - taskInstance's `try_number`, `map_index`, `loop_index` and `loop_max_iterations` of every event: the + iteration is the zero-based `loop_index`, `map_index` the mapped position, and a plain task has no + `loop_index` + - taskInstance's `loop_group_id` of every event: the loop group of a task in a loop, absent for a plain task + - taskInstance's `max_tries` of the tasks other than the gates, the `retries` of each task + - taskInstance's `id` of every event, and `region_id` and `region_index` of the tasks in a loop: the + `region_index` is the iteration for a task that isn't mapped and the map index for a mapped one, and a + plain task has neither + - the gates' `loop_decision`: absent on START; for the loops without `until`, `stop` true only at the last + iteration and no `reason`; for the loop with `until` (whose gate is named after it), the `reason` it + returns and `stop` true at the iteration it ends the loop, before `max_iterations`; absent for the plain + task + - a mapped task group inside a loop: its task's `map_index` and `region_index` are the mapped position, + its `loop_index` the iteration, and its `loop_group_id` the loop's, unlike its own `task_group` + - the loop groups' `max_iterations`, `has_until`, `gate_task_id` and `terminal_task_id`, in the task's + `task_group` and in the `taskGroups` of the Dag's job facet + - a retried task emits START and FAIL for each failed try, then START and COMPLETE for the last one + - every execution (iteration, map index, try) has its own `runId`, shared by its START and COMPLETE/FAIL + (the `runId` check of OpenLineageTestOperator is enabled by default) +""" + +from __future__ import annotations + +from datetime import datetime, timedelta + +from airflow import DAG +from airflow.sdk import LoopDecision, task, task_group + +from system.openlineage.constants import DEFAULT_DAGRUN_TIMEOUT +from system.openlineage.expected_events import get_expected_event_file_path +from system.openlineage.operator import OpenLineageTestOperator + +DAG_ID = "openlineage_task_loop_dag" + + +def sort_by_execution(event: dict) -> tuple[int, int, int]: + # Dag run events have no airflow facet + task_instance = event["run"]["facets"].get("airflow", {}).get("taskInstance", {}) + return ( + task_instance.get("loop_index", -1), + task_instance.get("map_index") if task_instance.get("map_index") is not None else -1, + task_instance.get("try_number", -1), + ) + + +with DAG( + dagrun_timeout=DEFAULT_DAGRUN_TIMEOUT, + dag_id=DAG_ID, + start_date=datetime(2021, 1, 1), + schedule=None, + catchup=False, + default_args={"retries": 0}, +) as dag: + + @task + def plain_task(): + return None + + @task_group + def accumulate(): + @task(retries=2, retry_delay=timedelta(seconds=1)) + def increment(*, loop, ti): + # The second iteration fails on its first try + if loop.index == 1 and ti.try_number < 2: Review Comment: Yeah, that's a good call, was confused in the UI. Will adjust the test (here or in the followup PR if there are no other nits in this PR) -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected]
