dheerajturaga commented on code in PR #62279: URL: https://github.com/apache/airflow/pull/62279#discussion_r2836395329
########## airflow-core/docs/core-concepts/dag-versioning.rst: ########## @@ -0,0 +1,340 @@ + .. Licensed to the Apache Software Foundation (ASF) under one + or more contributor license agreements. See the NOTICE file + distributed with this work for additional information + regarding copyright ownership. The ASF licenses this file + to you under the Apache License, Version 2.0 (the + "License"); you may not use this file except in compliance + with the License. You may obtain a copy of the License at + + .. http://www.apache.org/licenses/LICENSE-2.0 + + .. Unless required by applicable law or agreed to in writing, + software distributed under the License is distributed on an + "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY + KIND, either express or implied. See the License for the + specific language governing permissions and limitations + under the License. + +.. _concepts-dag-versioning: + +Dag Versioning +============== + +.. versionadded:: 3.0 + +Airflow automatically tracks changes to your Dag definitions over time through Dag versioning. Each time +you modify a Dag's structure and the scheduler parses the updated file, Airflow creates a new Dag version +that captures the serialized Dag structure and code at that point in time. + +This means that when you view a historical Dag run in the UI, you see the exact Dag structure and code as +it existed when that run executed --- not the current version of the Dag. This is essential for debugging, +auditing, and understanding the behavior of past runs, especially in production environments where Dags +evolve frequently. + +.. note:: + + Dag versioning is primarily a **UI and history feature**. When Airflow executes a task, it always uses + the **latest** Dag code and structure, not the version from the original run. The one exception is when + using a versioned Dag bundle (such as ``GitDagBundle``), which can check out and run code from a + specific commit. See :ref:`Bundle Version Tracking <bundle-version-tracking>` below for details. + +Each Dag version is identified by a version number that auto-increments per Dag (e.g., ``my_dag-1``, +``my_dag-2``). Versions are immutable once created; modifying your Dag always produces a new version rather +than updating an existing one. + +How Dag Versions Are Created +---------------------------- + +A new Dag version is created when the scheduler parses a Dag file and detects that the Dag's structure has +changed since the last version. Structural changes include: + +- Tasks added or removed +- Task dependencies changed +- Task parameters modified (e.g., ``bash_command``, ``trigger_rule``) +- Dag-level parameters modified (e.g., ``schedule``, ``default_args``) + +Each Dag version stores: + +- The **serialized Dag structure** (tasks, dependencies, and configuration) +- The **Dag code** (the Python source of the Dag file) +- The **bundle version** (e.g., a git commit hash, if using a versioned Dag bundle) + +.. note:: + + Cosmetic changes to your Dag file that do not affect the serialized structure (such as adding comments + or reformatting code) may not trigger a new version, since Airflow compares the serialized representation + rather than the raw source text. + +**Example:** Consider a simple Dag with two tasks: + +.. code-block:: python + + from airflow.sdk import DAG + from airflow.providers.standard.operators.bash import BashOperator + + with DAG("etl_pipeline", schedule="@daily"): + extract = BashOperator(task_id="extract", bash_command="echo extract") Review Comment: same here. `@task.bash` is better ########## airflow-core/docs/core-concepts/dag-versioning.rst: ########## @@ -0,0 +1,340 @@ + .. Licensed to the Apache Software Foundation (ASF) under one + or more contributor license agreements. See the NOTICE file + distributed with this work for additional information + regarding copyright ownership. The ASF licenses this file + to you under the Apache License, Version 2.0 (the + "License"); you may not use this file except in compliance + with the License. You may obtain a copy of the License at + + .. http://www.apache.org/licenses/LICENSE-2.0 + + .. Unless required by applicable law or agreed to in writing, + software distributed under the License is distributed on an + "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY + KIND, either express or implied. See the License for the + specific language governing permissions and limitations + under the License. + +.. _concepts-dag-versioning: + +Dag Versioning +============== + +.. versionadded:: 3.0 + +Airflow automatically tracks changes to your Dag definitions over time through Dag versioning. Each time +you modify a Dag's structure and the scheduler parses the updated file, Airflow creates a new Dag version +that captures the serialized Dag structure and code at that point in time. + +This means that when you view a historical Dag run in the UI, you see the exact Dag structure and code as +it existed when that run executed --- not the current version of the Dag. This is essential for debugging, +auditing, and understanding the behavior of past runs, especially in production environments where Dags +evolve frequently. + +.. note:: + + Dag versioning is primarily a **UI and history feature**. When Airflow executes a task, it always uses + the **latest** Dag code and structure, not the version from the original run. The one exception is when + using a versioned Dag bundle (such as ``GitDagBundle``), which can check out and run code from a + specific commit. See :ref:`Bundle Version Tracking <bundle-version-tracking>` below for details. + +Each Dag version is identified by a version number that auto-increments per Dag (e.g., ``my_dag-1``, +``my_dag-2``). Versions are immutable once created; modifying your Dag always produces a new version rather +than updating an existing one. + +How Dag Versions Are Created +---------------------------- + +A new Dag version is created when the scheduler parses a Dag file and detects that the Dag's structure has +changed since the last version. Structural changes include: + +- Tasks added or removed +- Task dependencies changed +- Task parameters modified (e.g., ``bash_command``, ``trigger_rule``) +- Dag-level parameters modified (e.g., ``schedule``, ``default_args``) + +Each Dag version stores: + +- The **serialized Dag structure** (tasks, dependencies, and configuration) +- The **Dag code** (the Python source of the Dag file) +- The **bundle version** (e.g., a git commit hash, if using a versioned Dag bundle) + +.. note:: + + Cosmetic changes to your Dag file that do not affect the serialized structure (such as adding comments + or reformatting code) may not trigger a new version, since Airflow compares the serialized representation + rather than the raw source text. + +**Example:** Consider a simple Dag with two tasks: + +.. code-block:: python + + from airflow.sdk import DAG + from airflow.providers.standard.operators.bash import BashOperator + + with DAG("etl_pipeline", schedule="@daily"): Review Comment: probably a good idea to use `@dag` decorator for the example instead -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected]
