This is an automated email from the ASF dual-hosted git repository.

shahar1 pushed a commit to branch main
in repository https://gitbox.apache.org/repos/asf/airflow.git


The following commit(s) were added to refs/heads/main by this push:
     new f01e432bedb Databricks: add Delta table version producer for 
externally written tables (#76156)
f01e432bedb is described below

commit f01e432bedb43f04556824a5893d6aff38231ea2
Author: Yossi Eliaz <[email protected]>
AuthorDate: Wed Nov 18 21:32:30 2026 +0200

    Databricks: add Delta table version producer for externally written tables 
(#76156)
    
    * Add DatabricksDeltaTableVersionProducerOperator for externally written 
Delta tables
    
    Publish an asset event for each Delta table version committed outside of
    Airflow. The latest event the producer published is its checkpoint, so the
    event and the checkpoint advance in one metadata-database write when the 
task
    succeeds. Delivery is at least once and each event carries a dedup_id.
    
    * Fix mypy errors in Delta table producer tests
    
    * Bound the Delta producer checkpoint lookup and require the table asset in 
outlets
    
    Airflow 3.2 and 3.3 can order and limit inlet events but cannot filter them
    by extra, so the checkpoint lookup read every event of the asset on each 
run.
    It now reads the newest events 100 at a time and stops at the first event
    with the producer's checkpoint key. The extra filter is still used where the
    accessor supports it.
    
    Explicit outlets that leave out unity_table.to_asset() never record a
    checkpoint, so every run would publish a new enrollment event. That is now a
    ValueError. A single inlet or outlet is accepted as well as a list.
    
    Signed-off-by: zozo123 <[email protected]>
    
    * Document the checkpoint lookup per Airflow version and use the compat DAG 
import
    
    Signed-off-by: zozo123 <[email protected]>
    
    * Fix Delta table producer tests on Airflow 3.0 and 3.1
    
    Asset is not a usable plain dict key on those versions (unhashable Asset
    on 3.0; hashing fails on mutable extra on 3.1). Key the test event
    accessors by URI the same way runtime accessors resolve assets.
    
    Co-authored-by: Yossi Eliaz <[email protected]>
    
    * Report Delta producer operation from the newest data commit
    
    When maintenance commits follow a data write, consumers reading the event
    operation should see the write rather than OPTIMIZE or VACUUM, while still
    using the head version for VERSION AS OF.
    
    Co-authored-by: Yossi Eliaz <[email protected]>
    
    * Drop databricks_ prefix from new Delta table producer modules
    
    Newer provider modules omit the provider name from the filename; keep that
    convention for the files this PR adds.
    
    Signed-off-by: Yossi Eliaz <[email protected]>
    Co-authored-by: Yossi Eliaz <[email protected]>
    
    * Address review feedback on Delta table producer tests
    
    Tighten the maintenance-window operation assertion so newest-first
    scanning is required, harden the hook mock with autospec, and warn when
    asset event paging cannot advance past a shared timestamp.
    
    Signed-off-by: Yossi Eliaz <[email protected]>
    Co-authored-by: Yossi Eliaz <[email protected]>
    
    * Keep Delta producer checkpoint key on the table asset only
    
    Additional outlets still need version and change metadata for consumers, 
but the
    checkpoint marker must stay on the observed table asset so other events do 
not
    look like producer checkpoints.
    
    Signed-off-by: Yossi Eliaz <[email protected]>
    Co-authored-by: Yossi Eliaz <[email protected]>
    
    * Collect Delta producer tests on Airflow 2.11
    
    Import AssetEvent from the tests_common compat shim so pytest can
    collect test_delta_table.py against Airflow 2.11.1.
    
    Signed-off-by: Yossi Eliaz <[email protected]>
    
    ---------
    
    Signed-off-by: zozo123 <[email protected]>
    Signed-off-by: Yossi Eliaz <[email protected]>
    Co-authored-by: Yossi Eliaz <[email protected]>
---
 .../docs/operators/delta_table_producer.rst        | 138 +++++
 providers/databricks/provider.yaml                 |   2 +
 .../src/airflow/providers/databricks/exceptions.py |   4 +
 .../providers/databricks/get_provider_info.py      |   2 +
 .../providers/databricks/operators/delta_table.py  | 452 +++++++++++++++
 .../example_databricks_delta_producer.py           |  81 +++
 .../unit/databricks/operators/test_delta_table.py  | 635 +++++++++++++++++++++
 7 files changed, 1314 insertions(+)

diff --git a/providers/databricks/docs/operators/delta_table_producer.rst 
b/providers/databricks/docs/operators/delta_table_producer.rst
new file mode 100644
index 00000000000..8da36533cc8
--- /dev/null
+++ b/providers/databricks/docs/operators/delta_table_producer.rst
@@ -0,0 +1,138 @@
+ .. Licensed to the Apache Software Foundation (ASF) under one
+    or more contributor license agreements.  See the NOTICE file
+    distributed with this work for additional information
+    regarding copyright ownership.  The ASF licenses this file
+    to you under the Apache License, Version 2.0 (the
+    "License"); you may not use this file except in compliance
+    with the License.  You may obtain a copy of the License at
+
+ ..   http://www.apache.org/licenses/LICENSE-2.0
+
+ .. Unless required by applicable law or agreed to in writing,
+    software distributed under the License is distributed on an
+    "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+    KIND, either express or implied.  See the License for the
+    specific language governing permissions and limitations
+    under the License.
+
+.. _howto/operator:DatabricksDeltaTableVersionProducerOperator:
+
+
+DatabricksDeltaTableVersionProducerOperator
+===========================================
+
+Use 
:class:`~airflow.providers.databricks.operators.delta_table.DatabricksDeltaTableVersionProducerOperator`
+when a Unity Catalog Delta table is written by jobs or systems outside 
Airflow, and Dags should run
+when it changes. Put the operator in a Dag that runs on a schedule. Each run 
reads the newest
+Delta version and publishes an asset event for ``unity_table.to_asset()`` when 
that version is newer
+than the last one it published. Otherwise the task is skipped and no event is 
published.
+
+Downstream Dags schedule on the same asset that
+:class:`~airflow.providers.databricks.operators.databricks_sql.DatabricksCopyIntoAssetOperator`
 uses
+for the table, so there is a single ``databricks://`` URI per table.
+
+To wait inside an already scheduled Dag run instead, use a sensor. This 
operator is for producing
+events continuously.
+
+Example
+-------
+
+.. exampleinclude:: 
/../../databricks/tests/system/databricks/example_databricks_delta_producer.py
+    :language: python
+    :start-after: [START 
howto_operator_databricks_delta_table_version_producer]
+    :end-before: [END howto_operator_databricks_delta_table_version_producer]
+
+What an event means
+-------------------
+
+An event means that the producer observed a newer committed Delta version. Its 
``extra`` contains:
+
+* ``version`` and ``observed_version``: the newest Delta version at the time 
of the check.
+* ``previous_version``: the version of the previous event, or 
``baseline_version``. ``None`` for enrollment.
+* ``table_identity``: the ``host``, ``catalog``, ``schema`` and ``table`` of 
the table.
+* ``operation``: the newest non-ignored Delta history operation in
+  ``(previous_version, version]``. When the producer publishes without a full 
check of that
+  window (incomplete history or more than ``max_history_rows`` commits), this 
can be an ignored
+  operation such as ``OPTIMIZE``.
+* ``timestamp``: the Delta history timestamp of ``version``.
+* ``change``: ``enrollment``, ``new_version`` or ``recreated``.
+* ``delta_table_id``: the Delta table id from ``DESCRIBE DETAIL``.
+* ``dedup_id``: the asset URI, table id and version. Use it to drop duplicate 
events.
+* ``databricks_delta_checkpoint_key``: the ``checkpoint_key`` of the producer. 
Present only on
+  the table asset's event, not on other outlets.
+
+Commits are coalesced. If the table moved from version 5 to version 9 between 
two runs, the producer
+publishes one event for version 9. It does not publish events for versions 6, 
7 and 8.
+
+An event for version N does not make downstream queries read version N. The 
table can change again
+before the consumer runs. Consumers that need a reproducible read query 
``VERSION AS OF`` the
+``version`` in the event, as the example does. Other tasks can publish events 
for the same asset,
+for example ``COPY INTO`` with outlets, and those events may not have a 
``version``. Consumers
+that read ``version`` check ``change`` or ``databricks_delta_checkpoint_key`` 
first.
+
+Checkpoints
+-----------
+
+The producer does not store state of its own. Its checkpoint is the newest 
event it published for the
+asset with the same ``checkpoint_key``. The operator adds the asset to its 
``inlets`` to read these
+events. Events from other tasks for the same asset, such as ``COPY INTO`` or 
sensor outlet events, do
+not change the checkpoint.
+
+``checkpoint_key`` defaults to ``"<dag_id>.<task_id>"``. Renaming the Dag or 
task, or changing the
+key, starts again with enrollment.
+
+How the checkpoint is found depends on the Airflow version. Versions that can 
filter asset events by
+``extra`` ask the API server for the newest event with the producer's 
``checkpoint_key``. Airflow 3.2
+and 3.3 read the asset's events newest first, 100 per request, and stop at the 
first match. Airflow 3.0
+and 3.1 cannot order or limit asset events, so each run reads all events on 
the asset.
+
+On the first run without a checkpoint, the producer publishes an 
``enrollment`` event for the current
+version, so that later runs have a checkpoint. To avoid triggering consumers 
on enrollment, pass
+``baseline_version``. It is used as the checkpoint until the first event is 
published.
+
+Delivery guarantees
+-------------------
+
+Airflow records the outlet event when the task succeeds, in the same database 
transaction that marks
+the task successful. Publishing the event and advancing the checkpoint are 
therefore one write:
+
+* If the task fails or the worker dies before the task succeeds, no event is 
published and the
+  checkpoint does not move. The next run or retry publishes the newest version 
at that time.
+* After the task succeeds, the event and the checkpoint exist together.
+
+Delivery is at least once, not exactly once. The same version can be published 
twice when two runs of
+the producer read the checkpoint at the same time, or when a successful run is 
cleared and runs again.
+The operator sets ``max_active_tis_per_dag=1`` by default, and the producer 
Dag should set
+``max_active_runs=1``. Consumers that must not process a version twice 
deduplicate on ``dedup_id``.
+
+Maintenance commits
+-------------------
+
+``OPTIMIZE``, ``VACUUM START`` and ``VACUUM END`` commits do not change the 
data, so they do not
+produce events by default. Pass ``ignored_operations`` to change the list, or 
an empty list to publish
+for every commit.
+
+If the newest commit is an ignored operation, the producer reads the history 
back to the checkpoint
+to check that every commit after it is ignored. When a data commit is followed 
by ignored
+operations, the event still carries the head ``version`` (for ``VERSION AS 
OF``), while
+``operation`` reports the newest non-ignored commit in that window. It 
publishes when it cannot
+check the window fully, because the history was removed by Delta log retention 
or because more
+than ``max_history_rows`` commits were made since the checkpoint. The task is 
not skipped when it
+is not sure that the data did not change.
+
+Recreated and renamed tables
+----------------------------
+
+The checkpoint stores the Delta table id. If the table is dropped and created 
again, or another table
+is renamed to the observed name, the id changes and the version can go back. 
With
+``on_recreation="publish"`` (the default), the producer publishes a 
``recreated`` event and uses the new
+table from then on. With ``on_recreation="fail"``, the task fails and the 
checkpoint is kept.
+
+If the table is missing, cannot be accessed, or is not a Delta table, the task 
fails and publishes
+nothing. Task retries apply as usual.
+
+Cost
+----
+
+Each run starts or uses a SQL warehouse to run ``DESCRIBE DETAIL`` and 
``DESCRIBE HISTORY``. Choose the
+schedule of the producer Dag with this cost in mind.
diff --git a/providers/databricks/provider.yaml 
b/providers/databricks/provider.yaml
index 894c986dd59..93260bb5d38 100644
--- a/providers/databricks/provider.yaml
+++ b/providers/databricks/provider.yaml
@@ -134,6 +134,7 @@ integrations:
       - /docs/apache-airflow-providers-databricks/operators/sql_statements.rst
       - /docs/apache-airflow-providers-databricks/operators/warehouse.rst
       - /docs/apache-airflow-providers-databricks/operators/copy_into.rst
+      - 
/docs/apache-airflow-providers-databricks/operators/delta_table_producer.rst
     tags: [service]
   - integration-name: Databricks Repos
     external-doc-url: https://docs.databricks.com/repos/index.html
@@ -159,6 +160,7 @@ operators:
   - integration-name: Databricks SQL
     python-modules:
       - airflow.providers.databricks.operators.databricks_sql
+      - airflow.providers.databricks.operators.delta_table
       - airflow.providers.databricks.operators.warehouse
   - integration-name: Databricks Repos
     python-modules:
diff --git 
a/providers/databricks/src/airflow/providers/databricks/exceptions.py 
b/providers/databricks/src/airflow/providers/databricks/exceptions.py
index 88e44011a0b..1a63de52605 100644
--- a/providers/databricks/src/airflow/providers/databricks/exceptions.py
+++ b/providers/databricks/src/airflow/providers/databricks/exceptions.py
@@ -59,6 +59,10 @@ class DatabricksApiError(AirflowException):
         self.error_code = error_code
 
 
+class DatabricksDeltaTableError(AirflowException):
+    """Raised when a Delta table cannot be read or its version cannot be 
determined."""
+
+
 class DatabricksUnityMCPError(DatabricksApiError):
     """Raised when a call to a Unity Gateway MCP Service fails."""
 
diff --git 
a/providers/databricks/src/airflow/providers/databricks/get_provider_info.py 
b/providers/databricks/src/airflow/providers/databricks/get_provider_info.py
index 76af2eebfc1..ce56c2cdcfb 100644
--- a/providers/databricks/src/airflow/providers/databricks/get_provider_info.py
+++ b/providers/databricks/src/airflow/providers/databricks/get_provider_info.py
@@ -49,6 +49,7 @@ def get_provider_info():
                     
"/docs/apache-airflow-providers-databricks/operators/sql_statements.rst",
                     
"/docs/apache-airflow-providers-databricks/operators/warehouse.rst",
                     
"/docs/apache-airflow-providers-databricks/operators/copy_into.rst",
+                    
"/docs/apache-airflow-providers-databricks/operators/delta_table_producer.rst",
                 ],
                 "tags": ["service"],
             },
@@ -86,6 +87,7 @@ def get_provider_info():
                 "integration-name": "Databricks SQL",
                 "python-modules": [
                     "airflow.providers.databricks.operators.databricks_sql",
+                    "airflow.providers.databricks.operators.delta_table",
                     "airflow.providers.databricks.operators.warehouse",
                 ],
             },
diff --git 
a/providers/databricks/src/airflow/providers/databricks/operators/delta_table.py
 
b/providers/databricks/src/airflow/providers/databricks/operators/delta_table.py
new file mode 100644
index 00000000000..db36ea88ea6
--- /dev/null
+++ 
b/providers/databricks/src/airflow/providers/databricks/operators/delta_table.py
@@ -0,0 +1,452 @@
+#
+# Licensed to the Apache Software Foundation (ASF) under one
+# or more contributor license agreements.  See the NOTICE file
+# distributed with this work for additional information
+# regarding copyright ownership.  The ASF licenses this file
+# to you under the Apache License, Version 2.0 (the
+# "License"); you may not use this file except in compliance
+# with the License.  You may obtain a copy of the License at
+#
+#   http://www.apache.org/licenses/LICENSE-2.0
+#
+# Unless required by applicable law or agreed to in writing,
+# software distributed under the License is distributed on an
+# "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+# KIND, either express or implied.  See the License for the
+# specific language governing permissions and limitations
+# under the License.
+"""Produce asset events for Delta table versions committed outside of 
Airflow."""
+
+from __future__ import annotations
+
+from collections.abc import Collection, Iterable, Iterator, Sequence
+from dataclasses import dataclass
+from datetime import date
+from functools import cached_property
+from typing import TYPE_CHECKING, Any, Literal
+
+from airflow.providers.common.compat.sdk import AirflowSkipException, 
BaseOperator
+from airflow.providers.common.sql.hooks.handlers import fetch_all_handler
+from airflow.providers.databricks.exceptions import DatabricksDeltaTableError
+from airflow.providers.databricks.hooks.databricks_sql import DatabricksSqlHook
+from airflow.providers.databricks.utils.query_tags import build_query_tags
+
+if TYPE_CHECKING:
+    from airflow.providers.common.compat.assets import Asset
+    from airflow.providers.common.compat.sdk import Context
+    from airflow.providers.databricks.assets.databricks import 
UnityTableIdentity
+
+CHECKPOINT_KEY_EXTRA = "databricks_delta_checkpoint_key"
+"""Asset event ``extra`` key that marks an event as a checkpoint written by 
this producer."""
+
+DEFAULT_IGNORED_OPERATIONS = ("OPTIMIZE", "VACUUM START", "VACUUM END")
+
+_EVENT_PAGE_SIZE = 100
+
+ChangeKind = Literal["enrollment", "new_version", "recreated"]
+
+
+@dataclass(frozen=True)
+class DeltaCheckpoint:
+    """Last Delta version this producer published for a table, read back from 
an asset event."""
+
+    version: int
+    table_id: str | None
+
+
+@dataclass(frozen=True)
+class _HistoryEntry:
+    version: int
+    operation: str | None
+    timestamp: str | None
+
+
+def _quote_identifier(part: str) -> str:
+    return "`" + part.replace("`", "``") + "`"
+
+
+def _as_list(value: Any) -> list[Any]:
+    if value is None:
+        return []
+    if isinstance(value, (list, tuple)):
+        return list(value)
+    return [value]
+
+
+def _row_to_dict(row: Any) -> dict[str, Any]:
+    if hasattr(row, "_asdict"):
+        return row._asdict()
+    if hasattr(row, "asDict"):
+        return row.asDict()
+    return dict(row)
+
+
+class DatabricksDeltaTableVersionProducerOperator(BaseOperator):
+    """
+    Publish an asset event when a Unity Catalog Delta table gets a newer 
committed version.
+
+    Use this operator in a recurring producer Dag to turn Delta commits made 
by other jobs or
+    systems into Airflow asset events. Downstream Dags schedule on 
``unity_table.to_asset()``.
+
+    Each run reads the newest Delta version, compares it with the checkpoint, 
and either publishes
+    one asset event for the newest version or skips. Several commits between 
two runs are coalesced
+    into one event for the newest version. Intermediate commits are not 
delivered one by one.
+
+    The checkpoint is the newest asset event that this producer published for 
the asset, found by
+    ``checkpoint_key`` in the event ``extra``. Airflow records the event when 
the task succeeds, so
+    publishing the event and advancing the checkpoint are the same write. 
Delivery is at least
+    once: concurrent runs or a cleared run can publish the same version again. 
Consumers that must
+    not process a version twice deduplicate on ``extra["dedup_id"]``.
+
+    An event for version N does not pin downstream reads to N. A consumer that 
needs a
+    reproducible read queries ``VERSION AS OF`` ``extra["version"]``.
+
+    .. seealso::
+        For more information on how to use this operator, take a look at the 
guide:
+        :ref:`howto/operator:DatabricksDeltaTableVersionProducerOperator`
+
+    :param unity_table: Static identity of the observed table. It is added to 
``inlets`` (to read
+        the checkpoint) and, when ``outlets`` is omitted, ``outlets`` is 
``[unity_table.to_asset()]``.
+        Explicit ``outlets`` must include ``unity_table.to_asset()``, because 
the checkpoint is the
+        event published on that asset.
+    :param checkpoint_key: Scope of the checkpoint. Producers with different 
keys keep separate
+        checkpoints for the same asset. Defaults to ``"<dag_id>.<task_id>"``. 
Changing it starts a
+        new enrollment.
+    :param baseline_version: Version treated as already delivered when no 
checkpoint exists. When
+        ``None`` (the default), the first run publishes an ``enrollment`` 
event for the current version.
+    :param ignored_operations: Delta history operations that do not count as a 
change. Defaults to
+        ``OPTIMIZE``, ``VACUUM START`` and ``VACUUM END``. Pass an empty 
collection to publish for
+        every commit.
+    :param max_history_rows: Largest ``DESCRIBE HISTORY`` window read to prove 
that all commits after
+        the checkpoint are ignored operations. When the window or the retained 
history does not
+        reach the checkpoint, the operator publishes rather than risk missing 
a data change.
+    :param on_recreation: What to do when the table id differs from the 
checkpoint, or the version
+        moved backwards, which means the table was dropped and recreated or 
replaced by a rename.
+        ``"publish"`` publishes a ``recreated`` event and checkpoints the new 
table. ``"fail"``
+        raises and leaves the checkpoint untouched.
+    :param databricks_conn_id: Reference to :ref:`Databricks connection 
id<howto/connection:databricks>`.
+    :param http_path: HTTP path of a Databricks SQL warehouse or all-purpose 
cluster.
+    :param sql_warehouse_name: Name of a Databricks SQL warehouse, used when 
``http_path`` is not set.
+    :param session_configuration: Spark session parameters.
+    :param http_headers: (k, v) pairs set as HTTP headers on every request.
+    :param client_parameters: Additional Databricks SQL connector parameters.
+    :param query_tags: Query tags attached to the Databricks SQL queries. 
(templated)
+    :param include_airflow_query_tags: If True, add Airflow Dag/task/run 
metadata as query tags.
+    """
+
+    template_fields: Sequence[str] = ("databricks_conn_id", "http_headers", 
"query_tags")
+
+    def __init__(
+        self,
+        *,
+        unity_table: UnityTableIdentity,
+        checkpoint_key: str | None = None,
+        baseline_version: int | None = None,
+        ignored_operations: Collection[str] | None = None,
+        max_history_rows: int = 1000,
+        on_recreation: Literal["publish", "fail"] = "publish",
+        databricks_conn_id: str = DatabricksSqlHook.default_conn_name,
+        http_path: str | None = None,
+        sql_warehouse_name: str | None = None,
+        session_configuration: dict[str, str] | None = None,
+        http_headers: list[tuple[str, str]] | None = None,
+        client_parameters: dict[str, Any] | None = None,
+        query_tags: dict[str, str | None] | None = None,
+        include_airflow_query_tags: bool = True,
+        **kwargs,
+    ) -> None:
+        if baseline_version is not None and (
+            isinstance(baseline_version, bool)
+            or not isinstance(baseline_version, int)
+            or baseline_version < 0
+        ):
+            raise ValueError(f"baseline_version must be a non-negative int, 
got {baseline_version!r}.")
+        if max_history_rows < 1:
+            raise ValueError(f"max_history_rows must be at least 1, got 
{max_history_rows!r}.")
+        if on_recreation not in ("publish", "fail"):
+            raise ValueError(f"on_recreation must be 'publish' or 'fail', got 
{on_recreation!r}.")
+        if checkpoint_key is not None and not checkpoint_key:
+            raise ValueError("checkpoint_key must not be empty.")
+
+        asset = unity_table.to_asset()
+        outlets = _as_list(kwargs.get("outlets", [asset]))
+        if not any(getattr(outlet, "uri", None) == asset.uri for outlet in 
outlets):
+            # The checkpoint is the event published on this asset. Without it 
every run re-enrolls.
+            raise ValueError(f"outlets must include unity_table.to_asset() 
({asset.uri!r}).")
+        kwargs["outlets"] = outlets
+        inlets = _as_list(kwargs.pop("inlets", None))
+        if not any(getattr(inlet, "uri", None) == asset.uri for inlet in 
inlets):
+            inlets.append(asset)
+        kwargs["inlets"] = inlets
+        # Concurrent runs of one producer read the same checkpoint and publish 
duplicates.
+        kwargs.setdefault("max_active_tis_per_dag", 1)
+        super().__init__(**kwargs)
+
+        self.unity_table = unity_table
+        self._checkpoint_key = checkpoint_key
+        self.baseline_version = baseline_version
+        self.ignored_operations = tuple(
+            DEFAULT_IGNORED_OPERATIONS if ignored_operations is None else 
ignored_operations
+        )
+        self._ignored = frozenset(op.casefold() for op in 
self.ignored_operations)
+        self.max_history_rows = max_history_rows
+        self.on_recreation = on_recreation
+        self.databricks_conn_id = databricks_conn_id
+        self.http_path = http_path
+        self.sql_warehouse_name = sql_warehouse_name
+        self.session_configuration = session_configuration
+        self.http_headers = http_headers
+        self.client_parameters = client_parameters or {}
+        self.query_tags = query_tags or {}
+        self.include_airflow_query_tags = include_airflow_query_tags
+
+    @property
+    def checkpoint_key(self) -> str:
+        return self._checkpoint_key or f"{self.dag_id}.{self.task_id}"
+
+    @cached_property
+    def asset(self) -> Asset:
+        return self.unity_table.to_asset()
+
+    @cached_property
+    def table_name(self) -> str:
+        t = self.unity_table
+        return ".".join(_quote_identifier(p) for p in (t.catalog, t.schema, 
t.table))
+
+    @cached_property
+    def hook(self) -> DatabricksSqlHook:
+        return DatabricksSqlHook(
+            self.databricks_conn_id,
+            http_path=self.http_path,
+            sql_endpoint_name=self.sql_warehouse_name,
+            session_configuration=self.session_configuration,
+            http_headers=self.http_headers,
+            caller="DatabricksDeltaTableVersionProducerOperator",
+            **self.client_parameters,
+        )
+
+    def execute(self, context: Context) -> dict[str, Any]:
+        hook = self.hook
+        if (hook.host or "").lower() != self.unity_table.host:
+            raise ValueError(
+                f"Databricks connection host {hook.host!r} does not match "
+                f"unity_table host {self.unity_table.host!r}."
+            )
+        hook.query_tags = build_query_tags(context, self.query_tags, 
self.include_airflow_query_tags)
+
+        checkpoint = self.read_checkpoint(context)
+        table_id = self._describe_table_id()
+        latest = self._read_history(limit=1)
+        if not latest:
+            raise DatabricksDeltaTableError(f"DESCRIBE HISTORY returned no 
rows for {self.table_name}.")
+        head = latest[0]
+
+        change: ChangeKind
+        previous: int | None
+        if checkpoint is None:
+            if self.baseline_version is None:
+                change, previous = "enrollment", None
+            elif head.version < self.baseline_version:
+                raise DatabricksDeltaTableError(
+                    f"Delta table {self.table_name} is at version 
{head.version}, "
+                    f"older than baseline_version {self.baseline_version}."
+                )
+            else:
+                previous = self.baseline_version
+                change = "new_version"
+        elif (
+            checkpoint.table_id is not None and table_id is not None and 
checkpoint.table_id != table_id
+        ) or head.version < checkpoint.version:
+            if self.on_recreation == "fail":
+                raise DatabricksDeltaTableError(
+                    f"Delta table {self.table_name} changed identity since the 
checkpoint "
+                    f"(table id {checkpoint.table_id!r} version 
{checkpoint.version} -> "
+                    f"table id {table_id!r} version {head.version}). Was it 
dropped and recreated?"
+                )
+            change, previous = "recreated", checkpoint.version
+        else:
+            change, previous = "new_version", checkpoint.version
+
+        operation_entry = head
+        if change == "new_version" and previous is not None:
+            if head.version == previous:
+                raise AirflowSkipException(
+                    f"Delta table {self.table_name} is still at version 
{previous}; no event published."
+                )
+            relevant = self._find_newest_relevant_commit(head, previous)
+            if relevant is None:
+                raise AirflowSkipException(
+                    f"Delta table {self.table_name} only has 
{sorted(self.ignored_operations)} commits "
+                    f"after version {previous}; no event published."
+                )
+            # ``version`` stays at head for VERSION AS OF; ``operation`` names 
the data change.
+            operation_entry = relevant
+
+        extra = self._build_extra(
+            head=head, operation_entry=operation_entry, table_id=table_id, 
previous=previous, change=change
+        )
+        for outlet in self.outlets:
+            if getattr(outlet, "uri", None) == self.asset.uri:
+                context["outlet_events"][outlet].extra = extra
+            else:
+                # Checkpoint markers are only for the table asset event used 
as the producer checkpoint.
+                context["outlet_events"][outlet].extra = {
+                    key: value for key, value in extra.items() if key != 
CHECKPOINT_KEY_EXTRA
+                }
+        self.log.info(
+            "Publishing %s event for %s at Delta version %s (previous %s).",
+            change,
+            self.asset.uri,
+            head.version,
+            previous,
+        )
+        return extra
+
+    def read_checkpoint(self, context: Context) -> DeltaCheckpoint | None:
+        """Return the newest checkpoint for ``checkpoint_key`` from the 
asset's events, if any."""
+        accessor = context["inlet_events"][self.asset]
+        events: Iterable[Any]
+        if all(hasattr(accessor, name) for name in ("extra", "ascending", 
"limit")):
+            # Airflow versions that filter asset events by ``extra`` on the 
API server.
+            events = accessor.extra(CHECKPOINT_KEY_EXTRA, 
self.checkpoint_key).ascending(False).limit(1)
+        elif all(hasattr(accessor, name) for name in ("ascending", "limit", 
"before")):
+            # Airflow 3.2 and 3.3 can order and limit events but not filter 
them by ``extra``.
+            events = self._newest_events_paged(accessor)
+        else:
+            events = reversed(accessor)
+        for event in events:
+            extra = event.extra or {}
+            if extra.get(CHECKPOINT_KEY_EXTRA) != self.checkpoint_key:
+                continue
+            version = extra.get("version")
+            if isinstance(version, bool) or not isinstance(version, int):
+                raise DatabricksDeltaTableError(
+                    f"Checkpoint event for {self.asset.uri} with key 
{self.checkpoint_key!r} "
+                    f"has no integer version: {extra!r}."
+                )
+            table_id = extra.get("delta_table_id")
+            return DeltaCheckpoint(version=version, table_id=str(table_id) if 
table_id is not None else None)
+        return None
+
+    def _newest_events_paged(self, accessor: Any) -> Iterator[Any]:
+        """Yield the asset's events newest first, fetching 
``_EVENT_PAGE_SIZE`` events per request."""
+        seen: set[Any] = set()
+        page = accessor.ascending(False).limit(_EVENT_PAGE_SIZE)
+        while True:
+            events = list(page)
+            for event in events:
+                event_id = getattr(event, "id", None)
+                if event_id is not None and event_id in seen:
+                    continue
+                seen.add(event_id)
+                yield event
+            if len(events) < _EVENT_PAGE_SIZE:
+                return
+            oldest = events[-1].timestamp
+            if oldest == events[0].timestamp:
+                # A full page with one timestamp. ``before`` is inclusive, so 
stop instead of looping.
+                self.log.warning(
+                    "Cannot advance asset event paging for %s: a full page of 
%s events shares "
+                    "timestamp %s. Older events are not scanned; this run may 
emit an enrollment event.",
+                    self.asset.uri,
+                    _EVENT_PAGE_SIZE,
+                    oldest,
+                )
+                return
+            page = accessor.before(oldest)
+
+    def _describe_table_id(self) -> str | None:
+        rows = self.hook.run(f"DESCRIBE DETAIL {self.table_name}", 
handler=fetch_all_handler)
+        if not rows:
+            raise DatabricksDeltaTableError(f"DESCRIBE DETAIL returned no rows 
for {self.table_name}.")
+        detail = _row_to_dict(rows[0])
+        table_format = detail.get("format")
+        if table_format is not None and str(table_format).lower() != "delta":
+            raise DatabricksDeltaTableError(
+                f"{self.table_name} is a {table_format!r} table; only Delta 
tables have versions."
+            )
+        table_id = detail.get("id")
+        return str(table_id) if table_id is not None else None
+
+    def _read_history(self, *, limit: int) -> list[_HistoryEntry]:
+        rows = self.hook.run(
+            f"DESCRIBE HISTORY {self.table_name} LIMIT {int(limit)}", 
handler=fetch_all_handler
+        )
+        entries = []
+        for row in rows or []:
+            data = _row_to_dict(row)
+            timestamp = data.get("timestamp")
+            entries.append(
+                _HistoryEntry(
+                    version=int(data["version"]),
+                    operation=data.get("operation"),
+                    timestamp=timestamp.isoformat()
+                    if isinstance(timestamp, date)
+                    else (str(timestamp) if timestamp is not None else None),
+                )
+            )
+        return sorted(entries, key=lambda e: e.version, reverse=True)
+
+    def _find_newest_relevant_commit(self, head: _HistoryEntry, previous: int) 
-> _HistoryEntry | None:
+        """
+        Newest non-ignored commit in ``(previous, head.version]``.
+
+        Returns ``None`` when every retained commit in the window is an 
ignored operation. When the
+        window cannot be fully checked (too large or incomplete history), 
returns ``head`` so the
+        caller publishes without proof — ``operation`` may then be an ignored 
operation.
+        """
+        if not self._ignored:
+            return head
+        if not self._is_ignored(head):
+            return head
+        needed = head.version - previous
+        if needed > self.max_history_rows:
+            self.log.warning(
+                "%s commits since version %s exceed max_history_rows=%s; 
publishing without proof "
+                "that they are all ignored operations.",
+                needed,
+                previous,
+                self.max_history_rows,
+            )
+            return head
+        # History is newest-first, so the first non-ignored entry is the 
newest relevant commit.
+        window = [e for e in self._read_history(limit=needed) if e.version > 
previous]
+        if len(window) < needed:
+            self.log.warning(
+                "Delta history after version %s is incomplete (%s of %s 
commits retained); "
+                "publishing without proof that they are all ignored 
operations.",
+                previous,
+                len(window),
+                needed,
+            )
+            return head
+        for entry in window:
+            if not self._is_ignored(entry):
+                return entry
+        return None
+
+    def _is_ignored(self, entry: _HistoryEntry) -> bool:
+        return entry.operation is not None and entry.operation.casefold() in 
self._ignored
+
+    def _build_extra(
+        self,
+        *,
+        head: _HistoryEntry,
+        operation_entry: _HistoryEntry,
+        table_id: str | None,
+        previous: int | None,
+        change: ChangeKind,
+    ) -> dict[str, Any]:
+        t = self.unity_table
+        return {
+            "version": head.version,
+            "observed_version": head.version,
+            "previous_version": previous,
+            "table_identity": {"host": t.host, "catalog": t.catalog, "schema": 
t.schema, "table": t.table},
+            "operation": operation_entry.operation,
+            "timestamp": head.timestamp,
+            "change": change,
+            "delta_table_id": table_id,
+            "dedup_id": f"{self.asset.uri}@{table_id or '-'}:{head.version}",
+            CHECKPOINT_KEY_EXTRA: self.checkpoint_key,
+        }
diff --git 
a/providers/databricks/tests/system/databricks/example_databricks_delta_producer.py
 
b/providers/databricks/tests/system/databricks/example_databricks_delta_producer.py
new file mode 100644
index 00000000000..0c3d7efce2d
--- /dev/null
+++ 
b/providers/databricks/tests/system/databricks/example_databricks_delta_producer.py
@@ -0,0 +1,81 @@
+#
+# Licensed to the Apache Software Foundation (ASF) under one
+# or more contributor license agreements.  See the NOTICE file
+# distributed with this work for additional information
+# regarding copyright ownership.  The ASF licenses this file
+# to you under the Apache License, Version 2.0 (the
+# "License"); you may not use this file except in compliance
+# with the License.  You may obtain a copy of the License at
+#
+#   http://www.apache.org/licenses/LICENSE-2.0
+#
+# Unless required by applicable law or agreed to in writing,
+# software distributed under the License is distributed on an
+# "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+# KIND, either express or implied.  See the License for the
+# specific language governing permissions and limitations
+# under the License.
+"""
+Example Dags that turn Delta table commits made outside Airflow into asset 
events.
+
+The producer Dag polls a Unity Catalog table every 10 minutes and publishes an 
asset event when the
+table has a newer Delta version. The consumer Dag is scheduled on that asset 
and reads the exact
+version that the event reports.
+"""
+
+from __future__ import annotations
+
+import os
+from datetime import datetime
+
+from airflow.providers.common.compat.sdk import DAG
+from airflow.providers.databricks.assets.databricks import UnityTableIdentity
+from airflow.providers.databricks.operators.databricks_sql import 
DatabricksSqlOperator
+from airflow.providers.databricks.operators.delta_table import (
+    DatabricksDeltaTableVersionProducerOperator,
+)
+
+ENV_ID = os.environ.get("SYSTEM_TESTS_ENV_ID")
+DAG_ID = "example_databricks_delta_producer"
+DATABRICKS_HOST = os.environ.get("DATABRICKS_HOST", 
"my-workspace.cloud.databricks.com")
+SQL_WAREHOUSE_NAME = os.environ.get("DATABRICKS_SQL_WAREHOUSE_NAME", "My 
Warehouse")
+
+# [START howto_operator_databricks_delta_table_version_producer]
+orders = UnityTableIdentity(host=DATABRICKS_HOST, catalog="main", 
schema="sales", table="orders")
+
+with DAG(
+    dag_id=DAG_ID,
+    schedule="*/10 * * * *",
+    start_date=datetime(2021, 1, 1),
+    catchup=False,
+    max_active_runs=1,
+    tags=["example"],
+) as dag:
+    DatabricksDeltaTableVersionProducerOperator(
+        task_id="publish_orders_version",
+        unity_table=orders,
+        sql_warehouse_name=SQL_WAREHOUSE_NAME,
+    )
+
+with DAG(
+    dag_id=f"{DAG_ID}_consumer",
+    schedule=[orders.to_asset()],
+    start_date=datetime(2021, 1, 1),
+    catchup=False,
+    tags=["example"],
+) as consumer_dag:
+    DatabricksSqlOperator(
+        task_id="read_published_version",
+        sql_endpoint_name=SQL_WAREHOUSE_NAME,
+        sql=(
+            "SELECT count(*) FROM main.sales.orders VERSION AS OF "
+            "{{ (triggering_asset_events.values() | first | 
last).extra['version'] }}"
+        ),
+    )
+# [END howto_operator_databricks_delta_table_version_producer]
+
+
+from tests_common.test_utils.system_tests import get_test_run  # noqa: E402
+
+# Needed to run the example DAG with pytest (see: 
contributing-docs/testing/system_tests.rst)
+test_run = get_test_run(dag)
diff --git 
a/providers/databricks/tests/unit/databricks/operators/test_delta_table.py 
b/providers/databricks/tests/unit/databricks/operators/test_delta_table.py
new file mode 100644
index 00000000000..21813fcedc3
--- /dev/null
+++ b/providers/databricks/tests/unit/databricks/operators/test_delta_table.py
@@ -0,0 +1,635 @@
+#
+# Licensed to the Apache Software Foundation (ASF) under one
+# or more contributor license agreements.  See the NOTICE file
+# distributed with this work for additional information
+# regarding copyright ownership.  The ASF licenses this file
+# to you under the Apache License, Version 2.0 (the
+# "License"); you may not use this file except in compliance
+# with the License.  You may obtain a copy of the License at
+#
+#   http://www.apache.org/licenses/LICENSE-2.0
+#
+# Unless required by applicable law or agreed to in writing,
+# software distributed under the License is distributed on an
+# "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+# KIND, either express or implied.  See the License for the
+# specific language governing permissions and limitations
+# under the License.
+from __future__ import annotations
+
+import re
+from collections.abc import Sequence
+from datetime import UTC, datetime, timedelta
+from types import SimpleNamespace
+from typing import Any
+from unittest import mock
+
+import pytest
+from sqlalchemy import select
+
+from airflow.providers.common.compat.assets import Asset
+from airflow.providers.common.compat.sdk import DAG, AirflowSkipException
+from airflow.providers.databricks.assets.databricks import UnityTableIdentity
+from airflow.providers.databricks.exceptions import DatabricksDeltaTableError
+from airflow.providers.databricks.hooks.databricks_sql import 
DatabricksSqlExecutionError
+from airflow.providers.databricks.operators.delta_table import (
+    CHECKPOINT_KEY_EXTRA,
+    DatabricksDeltaTableVersionProducerOperator,
+    DeltaCheckpoint,
+)
+from airflow.utils.session import create_session
+
+from tests_common.test_utils.compat import AssetEvent
+from tests_common.test_utils.dag import sync_dag_to_db
+from tests_common.test_utils.db import clear_db_assets, clear_db_dags, 
clear_db_runs
+from tests_common.test_utils.version_compat import AIRFLOW_V_3_2_PLUS
+
+HOST = "xx.cloud.databricks.com"
+TABLE = UnityTableIdentity(host=HOST, catalog="Main", schema="Sales", 
table="Orders")
+URI = f"databricks://{HOST}/main/sales/orders"
+QUOTED = "`main`.`sales`.`orders`"
+TABLE_ID = "11111111-1111-1111-1111-111111111111"
+OTHER_TABLE_ID = "22222222-2222-2222-2222-222222222222"
+KEY = "producer.watch_orders"
+HOOK_PATH = 
"airflow.providers.databricks.operators.delta_table.DatabricksSqlHook"
+
+
+class FakeDeltaTable:
+    """In-memory stand-in for ``DESCRIBE DETAIL`` / ``DESCRIBE HISTORY`` on 
one table."""
+
+    def __init__(
+        self,
+        commits: list[tuple[int, str]],
+        *,
+        table_id: str = TABLE_ID,
+        table_format: str = "delta",
+        retained_from: int = 0,
+    ):
+        self.commits = commits
+        self.table_id = table_id
+        self.table_format = table_format
+        self.retained_from = retained_from
+        self.statements: list[str] = []
+
+    def run(self, sql: str, handler=None):
+        self.statements.append(sql)
+        if sql.startswith("DESCRIBE DETAIL"):
+            return [SimpleNamespace(_asdict=lambda: {"format": 
self.table_format, "id": self.table_id})]
+        match = re.fullmatch(r"DESCRIBE HISTORY (.+) LIMIT (\d+)", sql)
+        assert match, sql
+        limit = int(match.group(2))
+        retained = [c for c in self.commits if c[0] >= self.retained_from]
+        newest_first = sorted(retained, reverse=True)[:limit]
+        return [
+            SimpleNamespace(
+                _asdict=lambda v=v, op=op: {
+                    "version": v,
+                    "operation": op,
+                    "timestamp": datetime(2026, 10, 1, 12, v % 60, tzinfo=UTC),
+                }
+            )
+            for v, op in newest_first
+        ]
+
+
+def make_event(extra: dict[str, Any]) -> SimpleNamespace:
+    return SimpleNamespace(extra=extra)
+
+
+def checkpoint_event(version: int, *, key: str = KEY, table_id: str = 
TABLE_ID) -> SimpleNamespace:
+    return make_event({"version": version, "delta_table_id": table_id, 
CHECKPOINT_KEY_EXTRA: key})
+
+
+class FilteringAccessor(Sequence):
+    """Accessor shaped like the Airflow 3 one that filters events on the API 
server."""
+
+    def __init__(self, events):
+        self._events = list(events)
+        self.calls: list[tuple] = []
+
+    def extra(self, key, value):
+        self.calls.append(("extra", key, value))
+        self._events = [e for e in self._events if (e.extra or {}).get(key) == 
value]
+        return self
+
+    def ascending(self, ascending=True):
+        self.calls.append(("ascending", ascending))
+        if not ascending:
+            self._events = list(reversed(self._events))
+        return self
+
+    def limit(self, limit):
+        self.calls.append(("limit", limit))
+        self._events = self._events[:limit]
+        return self
+
+    def __getitem__(self, i):
+        return self._events[i]
+
+    def __len__(self):
+        return len(self._events)
+
+
+class PagingAccessor(Sequence):
+    """Accessor shaped like the Airflow 3.2/3.3 one: ordering, limit and 
``before``, no ``extra`` filter."""
+
+    def __init__(self, events):
+        self._all = list(events)
+        self._ascending = True
+        self._limit = None
+        self._before = None
+        self.requests = 0
+
+        self._cache = None
+
+    def ascending(self, ascending=True):
+        self._ascending, self._cache = ascending, None
+        return self
+
+    def limit(self, limit):
+        self._limit, self._cache = limit, None
+        return self
+
+    def before(self, before):
+        self._before, self._cache = before, None
+        return self
+
+    @property
+    def _events(self):
+        # Like the real accessor, one request per filter change, cached until 
the filter changes again.
+        if self._cache is None:
+            self.requests += 1
+            events = [e for e in self._all if self._before is None or 
e.timestamp <= self._before]
+            events.sort(key=lambda e: (e.timestamp, e.id), reverse=not 
self._ascending)
+            self._cache = events[: self._limit] if self._limit is not None 
else events
+        return self._cache
+
+    def __iter__(self):
+        return iter(self._events)
+
+    def __getitem__(self, i):
+        return self._events[i]
+
+    def __len__(self):
+        return len(self._events)
+
+
+def timed_event(i: int, extra: dict[str, Any]) -> SimpleNamespace:
+    return SimpleNamespace(
+        id=i, timestamp=datetime(2026, 10, 1, tzinfo=UTC) + 
timedelta(minutes=i), extra=extra
+    )
+
+
+class _EventsByAssetUri:
+    """
+    Map Asset keys by URI.
+
+    Real ``inlet_events`` / ``outlet_events`` accessors resolve Assets to a 
hashable unique
+    key. Older Airflow versions ship an Asset that is not usable as a plain 
``dict`` key
+    (unhashable Asset on 3.0; hashing fails on the mutable ``extra`` dict on 
3.1), so the
+    test stand-in must not rely on ``Asset.__hash__``.
+    """
+
+    def __init__(self, initial: dict[str, Any] | None = None, *, 
default_factory=None):
+        self._by_uri: dict[str, Any] = dict(initial or {})
+        self._default_factory = default_factory
+
+    @staticmethod
+    def _uri(key: Any) -> str:
+        return key if isinstance(key, str) else key.uri
+
+    def __getitem__(self, key: Any) -> Any:
+        uri = self._uri(key)
+        if uri not in self._by_uri:
+            if self._default_factory is None:
+                raise KeyError(key)
+            self._by_uri[uri] = self._default_factory()
+        return self._by_uri[uri]
+
+    def items(self):
+        return ((SimpleNamespace(uri=uri), value) for uri, value in 
self._by_uri.items())
+
+
+def make_context(events=None, accessor=None) -> Any:
+    """Return a minimal stand-in for the task ``Context`` with only the asset 
event accessors."""
+    inlet = accessor if accessor is not None else list(events or [])
+    return {
+        "inlet_events": _EventsByAssetUri({URI: inlet}),
+        "outlet_events": _EventsByAssetUri(default_factory=lambda: 
SimpleNamespace(extra={})),
+    }
+
+
+def published(context) -> dict[str, Any] | None:
+    for asset, event in context["outlet_events"].items():
+        if asset.uri == URI:
+            return event.extra
+    return None
+
+
[email protected]
+def table():
+    return FakeDeltaTable([(0, "CREATE TABLE"), (1, "WRITE"), (2, "WRITE")])
+
+
[email protected]
+def hook(table):
+    with mock.patch(HOOK_PATH, autospec=True) as hook_cls:
+        instance = hook_cls.return_value
+        type(instance).host = mock.PropertyMock(return_value=HOST)
+        instance.run.side_effect = table.run
+        yield instance
+
+
+def make_operator(**kwargs) -> DatabricksDeltaTableVersionProducerOperator:
+    kwargs.setdefault("checkpoint_key", KEY)
+    return DatabricksDeltaTableVersionProducerOperator(
+        task_id="watch_orders", unity_table=TABLE, 
http_path="/sql/1.0/warehouses/abc", **kwargs
+    )
+
+
+class TestConstruction:
+    def test_defaults(self):
+        op = make_operator(checkpoint_key=None)
+        assert [a.uri for a in op.outlets] == [URI]
+        assert [a.uri for a in op.inlets] == [URI]
+        assert op.max_active_tis_per_dag == 1
+        assert op.ignored_operations == ("OPTIMIZE", "VACUUM START", "VACUUM 
END")
+        assert op.table_name == QUOTED
+
+    def test_keeps_user_inlets_and_outlets(self):
+        other = Asset(uri="s3://bucket/other")
+        op = make_operator(inlets=[other], outlets=[other, TABLE.to_asset()], 
max_active_tis_per_dag=4)
+        assert [a.uri for a in op.inlets] == ["s3://bucket/other", URI]
+        assert [a.uri for a in op.outlets] == ["s3://bucket/other", URI]
+        assert op.max_active_tis_per_dag == 4
+
+    def test_accepts_single_inlet_and_outlet(self):
+        other = Asset(uri="s3://bucket/other")
+        op = make_operator(inlets=other, outlets=TABLE.to_asset())
+        assert [a.uri for a in op.inlets] == ["s3://bucket/other", URI]
+        assert [a.uri for a in op.outlets] == [URI]
+
+    @pytest.mark.parametrize("outlets", [[], ["s3://bucket/other"]])
+    def test_rejects_outlets_without_the_table_asset(self, outlets):
+        with pytest.raises(ValueError, match="outlets must include"):
+            make_operator(outlets=[Asset(uri=uri) for uri in outlets])
+
+    def test_does_not_duplicate_inlet(self):
+        op = make_operator(inlets=[TABLE.to_asset()])
+        assert [a.uri for a in op.inlets] == [URI]
+
+    @pytest.mark.parametrize(
+        ("kwargs", "message"),
+        [
+            ({"baseline_version": -1}, "baseline_version"),
+            ({"baseline_version": "3"}, "baseline_version"),
+            ({"baseline_version": True}, "baseline_version"),
+            ({"max_history_rows": 0}, "max_history_rows"),
+            ({"on_recreation": "ignore"}, "on_recreation"),
+            ({"checkpoint_key": ""}, "checkpoint_key"),
+        ],
+    )
+    def test_rejects_invalid_arguments(self, kwargs, message):
+        with pytest.raises(ValueError, match=message):
+            make_operator(**kwargs)
+
+    def test_quotes_backticks_in_identifiers(self):
+        odd = UnityTableIdentity(host=HOST, catalog="main", schema="sales", 
table="we`ird")
+        op = DatabricksDeltaTableVersionProducerOperator(task_id="t", 
unity_table=odd)
+        assert op.table_name == "`main`.`sales`.`we``ird`"
+
+
+class TestEnrollment:
+    def test_first_run_publishes_enrollment_event(self, hook, table):
+        context = make_context()
+        result = make_operator().execute(context)
+
+        assert published(context) == result
+        assert result == {
+            "version": 2,
+            "observed_version": 2,
+            "previous_version": None,
+            "table_identity": {"host": HOST, "catalog": "main", "schema": 
"sales", "table": "orders"},
+            "operation": "WRITE",
+            "timestamp": "2026-10-01T12:02:00+00:00",
+            "change": "enrollment",
+            "delta_table_id": TABLE_ID,
+            "dedup_id": f"{URI}@{TABLE_ID}:2",
+            CHECKPOINT_KEY_EXTRA: KEY,
+        }
+        assert table.statements == [f"DESCRIBE DETAIL {QUOTED}", f"DESCRIBE 
HISTORY {QUOTED} LIMIT 1"]
+
+    def test_baseline_at_current_version_skips(self, hook):
+        context = make_context()
+        with pytest.raises(AirflowSkipException, match="still at version 2"):
+            make_operator(baseline_version=2).execute(context)
+        assert published(context) is None
+
+    def test_baseline_older_than_current_publishes_new_version(self, hook):
+        context = make_context()
+        result = make_operator(baseline_version=1).execute(context)
+        assert result["change"] == "new_version"
+        assert result["previous_version"] == 1
+        assert result["version"] == 2
+
+    def test_baseline_newer_than_table_fails(self, hook):
+        with pytest.raises(DatabricksDeltaTableError, match="older than 
baseline_version 5"):
+            make_operator(baseline_version=5).execute(make_context())
+
+    def test_checkpoint_wins_over_baseline(self, hook):
+        context = make_context([checkpoint_event(2)])
+        with pytest.raises(AirflowSkipException):
+            make_operator(baseline_version=0).execute(context)
+
+    def test_default_checkpoint_key_is_dag_and_task(self, hook):
+        with DAG("producer", schedule=None):
+            op = make_operator(checkpoint_key=None)
+        result = op.execute(make_context())
+        assert result[CHECKPOINT_KEY_EXTRA] == "producer.watch_orders"
+
+    def test_checkpoint_key_only_on_table_asset_outlet(self, hook, table):
+        other = Asset(uri="s3://bucket/other")
+        op = make_operator(outlets=[other, TABLE.to_asset()])
+        context = make_context()
+        result = op.execute(context)
+
+        table_extra = context["outlet_events"][TABLE.to_asset()].extra
+        other_extra = context["outlet_events"][other].extra
+
+        assert table_extra == result
+        assert table_extra["version"] == other_extra["version"] == 2
+        assert table_extra[CHECKPOINT_KEY_EXTRA] == KEY
+        assert CHECKPOINT_KEY_EXTRA not in other_extra
+        assert other_extra["delta_table_id"] == TABLE_ID
+        assert other_extra["change"] == "enrollment"
+
+
+class TestVersionChanges:
+    def test_unchanged_version_skips_without_event(self, hook):
+        context = make_context([checkpoint_event(2)])
+        with pytest.raises(AirflowSkipException, match="still at version 2"):
+            make_operator().execute(context)
+        assert published(context) is None
+
+    def test_new_versions_are_coalesced_into_one_event_for_the_latest(self, 
hook, table):
+        table.commits += [(3, "WRITE"), (4, "MERGE"), (5, "UPDATE")]
+        context = make_context([checkpoint_event(2)])
+        result = make_operator().execute(context)
+        assert (result["change"], result["previous_version"], 
result["version"]) == ("new_version", 2, 5)
+        assert result["operation"] == "UPDATE"
+        assert published(context) == result
+        # The head commit is a data change, so the operator does not scan 
further history.
+        assert table.statements[-1] == f"DESCRIBE HISTORY {QUOTED} LIMIT 1"
+
+    def test_only_maintenance_commits_skip(self, hook, table):
+        table.commits += [(3, "OPTIMIZE"), (4, "VACUUM START"), (5, "VACUUM 
END")]
+        context = make_context([checkpoint_event(2)])
+        with pytest.raises(AirflowSkipException, match="only has"):
+            make_operator().execute(context)
+        assert published(context) is None
+        assert table.statements[-1] == f"DESCRIBE HISTORY {QUOTED} LIMIT 3"
+
+    @pytest.mark.parametrize(
+        ("commits", "operation"),
+        [
+            ([(3, "WRITE"), (4, "OPTIMIZE"), (5, "VACUUM END")], "WRITE"),
+            # WRITE then MERGE then OPTIMIZE: fails if the window is walked 
oldest-first.
+            ([(3, "WRITE"), (4, "MERGE"), (5, "OPTIMIZE")], "MERGE"),
+        ],
+        ids=["write-behind-optimize-vacuum", "write-merge-optimize"],
+    )
+    def test_data_commit_hidden_behind_maintenance_publishes(self, hook, 
table, commits, operation):
+        table.commits += commits
+        result = make_operator().execute(make_context([checkpoint_event(2)]))
+        assert result["version"] == 5
+        assert result["operation"] == operation
+
+    def test_ignored_operations_are_case_insensitive(self, hook, table):
+        table.commits += [(3, "optimize")]
+        with pytest.raises(AirflowSkipException):
+            make_operator().execute(make_context([checkpoint_event(2)]))
+
+    def test_empty_ignored_operations_publish_for_every_commit(self, hook, 
table):
+        table.commits += [(3, "OPTIMIZE")]
+        result = 
make_operator(ignored_operations=()).execute(make_context([checkpoint_event(2)]))
+        assert result["version"] == 3
+        assert table.statements[-1] == f"DESCRIBE HISTORY {QUOTED} LIMIT 1"
+
+    def test_expired_history_publishes_conservatively(self, hook, table):
+        table.commits += [(3, "WRITE"), (4, "OPTIMIZE"), (5, "OPTIMIZE")]
+        table.retained_from = 4  # Versions 3 and earlier were removed from 
the Delta log.
+        result = make_operator().execute(make_context([checkpoint_event(2)]))
+        assert result["version"] == 5
+        # Incomplete window: operation stays at the head commit.
+        assert result["operation"] == "OPTIMIZE"
+
+    def test_history_beyond_scan_window_publishes_conservatively(self, hook, 
table):
+        table.commits += [(v, "OPTIMIZE") for v in range(3, 13)]
+        result = 
make_operator(max_history_rows=5).execute(make_context([checkpoint_event(2)]))
+        assert result["version"] == 12
+        assert result["operation"] == "OPTIMIZE"
+        # It did not run the large scan at all.
+        assert all("LIMIT 10" not in s for s in table.statements)
+
+
+class TestRecreation:
+    def test_new_table_id_publishes_recreated(self, hook, table):
+        table.table_id = OTHER_TABLE_ID
+        context = make_context([checkpoint_event(2)])
+        result = make_operator().execute(context)
+        assert result["change"] == "recreated"
+        assert result["delta_table_id"] == OTHER_TABLE_ID
+        assert result["dedup_id"] == f"{URI}@{OTHER_TABLE_ID}:2"
+
+    def test_version_moved_backwards_publishes_recreated(self, hook, table):
+        table.commits = [(0, "CREATE OR REPLACE TABLE")]
+        result = make_operator().execute(make_context([checkpoint_event(7, 
table_id=None)]))
+        assert (result["change"], result["previous_version"], 
result["version"]) == ("recreated", 7, 0)
+
+    def test_recreated_with_only_maintenance_still_publishes(self, hook, 
table):
+        table.table_id = OTHER_TABLE_ID
+        table.commits = [(0, "CREATE TABLE"), (1, "OPTIMIZE")]
+        result = make_operator().execute(make_context([checkpoint_event(1)]))
+        assert result["change"] == "recreated"
+
+    def test_on_recreation_fail_raises_and_publishes_nothing(self, hook, 
table):
+        table.table_id = OTHER_TABLE_ID
+        context = make_context([checkpoint_event(2)])
+        with pytest.raises(DatabricksDeltaTableError, match="dropped and 
recreated"):
+            make_operator(on_recreation="fail").execute(context)
+        assert published(context) is None
+
+
+class TestTableErrors:
+    def test_non_delta_table_fails(self, hook, table):
+        table.table_format = "parquet"
+        context = make_context()
+        with pytest.raises(DatabricksDeltaTableError, match="only Delta tables 
have versions"):
+            make_operator().execute(context)
+        assert published(context) is None
+
+    def 
test_missing_or_inaccessible_table_propagates_and_publishes_nothing(self, hook):
+        hook.run.side_effect = 
DatabricksSqlExecutionError("[TABLE_OR_VIEW_NOT_FOUND]")
+        context = make_context([checkpoint_event(2)])
+        with pytest.raises(DatabricksSqlExecutionError):
+            make_operator().execute(context)
+        assert published(context) is None
+
+    def test_empty_history_fails(self, hook, table):
+        table.commits = []
+        with pytest.raises(DatabricksDeltaTableError, match="DESCRIBE HISTORY 
returned no rows"):
+            make_operator().execute(make_context())
+
+    def test_host_mismatch_fails_before_any_query(self, hook):
+        type(hook).host = 
mock.PropertyMock(return_value="other.cloud.databricks.com")
+        with pytest.raises(ValueError, match="does not match unity_table 
host"):
+            make_operator().execute(make_context())
+        hook.run.assert_not_called()
+
+    def test_host_match_is_case_insensitive(self, hook):
+        type(hook).host = mock.PropertyMock(return_value=HOST.upper())
+        assert make_operator().execute(make_context())["version"] == 2
+
+
+class TestCheckpoint:
+    def test_ignores_events_from_other_producers_and_scopes(self, hook):
+        events = [
+            checkpoint_event(1),
+            checkpoint_event(2, key="another.producer"),
+            make_event({"version": 2, "observed_version": 2}),  # Sensor 
outlet event.
+            make_event({}),  # COPY INTO outlet event.
+            make_event(None),
+        ]
+        op = make_operator()
+        assert op.read_checkpoint(make_context(events)) == 
DeltaCheckpoint(version=1, table_id=TABLE_ID)
+
+    def test_uses_newest_matching_event(self, hook):
+        events = [checkpoint_event(1), checkpoint_event(2), make_event({})]
+        assert make_operator().read_checkpoint(make_context(events)).version 
== 2
+
+    def test_no_matching_event_means_no_checkpoint(self, hook):
+        assert make_operator().read_checkpoint(make_context([make_event({})])) 
is None
+
+    def test_uses_server_side_filter_when_available(self, hook):
+        accessor = FilteringAccessor([checkpoint_event(1), checkpoint_event(5, 
key="x"), make_event({})])
+        assert 
make_operator().read_checkpoint(make_context(accessor=accessor)).version == 1
+        assert accessor.calls == [("extra", CHECKPOINT_KEY_EXTRA, KEY), 
("ascending", False), ("limit", 1)]
+
+    def test_pages_newest_first_when_extra_filter_is_unavailable(self, hook):
+        events = [timed_event(0, {"version": 1, "delta_table_id": TABLE_ID, 
CHECKPOINT_KEY_EXTRA: KEY})]
+        events += [timed_event(i, {}) for i in range(1, 250)]
+        accessor = PagingAccessor(events)
+        assert 
make_operator().read_checkpoint(make_context(accessor=accessor)).version == 1
+        assert accessor.requests == 3
+
+    def test_paging_stops_at_the_newest_checkpoint(self, hook):
+        events = [timed_event(i, {}) for i in range(250)]
+        events.append(timed_event(250, {"version": 7, "delta_table_id": 
TABLE_ID, CHECKPOINT_KEY_EXTRA: KEY}))
+        accessor = PagingAccessor(events)
+        assert 
make_operator().read_checkpoint(make_context(accessor=accessor)).version == 7
+        assert accessor.requests == 1
+
+    def test_paging_without_checkpoint_reads_all_pages_once(self, hook):
+        accessor = PagingAccessor([timed_event(i, {}) for i in range(150)])
+        assert 
make_operator().read_checkpoint(make_context(accessor=accessor)) is None
+        assert accessor.requests == 2
+
+    def test_corrupt_checkpoint_fails(self, hook):
+        bad = make_event({CHECKPOINT_KEY_EXTRA: KEY, "version": "2"})
+        with pytest.raises(DatabricksDeltaTableError, match="no integer 
version"):
+            make_operator().read_checkpoint(make_context([bad]))
+
+
+class TestDelivery:
+    """
+    The checkpoint is the published event, so these tests replay the producer 
across runs.
+
+    A run that is killed before the task succeeds leaves no event; Airflow 
records the outlet
+    event only on success. These tests model that by appending the event to 
the asset's
+    history only for runs that complete.
+    """
+
+    def run_once(self, history: list, *, succeed: bool = True) -> dict[str, 
Any] | None:
+        context = make_context(list(history))
+        try:
+            make_operator().execute(context)
+        except AirflowSkipException:
+            return None
+        extra = published(context)
+        if succeed:
+            assert extra is not None
+            history.append(make_event(extra))
+        return extra
+
+    def test_crash_before_publication_redelivers_same_version(self, hook, 
table):
+        history: list = [checkpoint_event(2)]
+        table.commits.append((3, "WRITE"))
+
+        lost = self.run_once(history, succeed=False)
+        retried = self.run_once(history)
+
+        assert lost is not None
+        assert retried == lost
+        assert retried["dedup_id"] == f"{URI}@{TABLE_ID}:3"
+
+    def test_crash_before_publication_coalesces_commits_made_meanwhile(self, 
hook, table):
+        history: list = [checkpoint_event(2)]
+        table.commits.append((3, "WRITE"))
+        self.run_once(history, succeed=False)
+        table.commits.append((4, "WRITE"))
+
+        retried = self.run_once(history)
+        assert (retried["previous_version"], retried["version"]) == (2, 4)
+
+    def test_after_publication_checkpoint_has_advanced(self, hook, table):
+        history: list = [checkpoint_event(2)]
+        table.commits.append((3, "WRITE"))
+        assert self.run_once(history)["version"] == 3
+        assert self.run_once(history) is None
+        table.commits.append((4, "WRITE"))
+        assert self.run_once(history)["previous_version"] == 3
+
+    def test_concurrent_runs_publish_same_dedup_id(self, hook, table):
+        history: list = [checkpoint_event(2)]
+        table.commits.append((3, "WRITE"))
+        # Both runs read the checkpoint before either publishes.
+        first_ctx, second_ctx = make_context(list(history)), 
make_context(list(history))
+        first = make_operator().execute(first_ctx)
+        second = make_operator().execute(second_ctx)
+        assert first["dedup_id"] == second["dedup_id"]
+        history += [make_event(first), make_event(second)]
+        assert self.run_once(history) is None
+
+
[email protected]_test
[email protected](not AIRFLOW_V_3_2_PLUS, reason="Runs the Dag through the 
Airflow 3 execution API")
+class TestProducerDagRuns:
+    """Run a producer Dag against the metadata database, so checkpoints are 
real asset events."""
+
+    @pytest.fixture(autouse=True)
+    def _clean_db(self):
+        clear_db_runs()
+        clear_db_assets()
+        clear_db_dags()
+        yield
+        clear_db_runs()
+        clear_db_assets()
+        clear_db_dags()
+
+    def test_events_and_skips_across_runs(self, hook, table):
+        with DAG("delta_producer", schedule=None) as dag:
+            
DatabricksDeltaTableVersionProducerOperator(task_id="watch_orders", 
unity_table=TABLE)
+        sync_dag_to_db(dag)
+
+        states = []
+        for commit in (None, None, (3, "WRITE"), (4, "OPTIMIZE"), None):
+            if commit:
+                table.commits.append(commit)
+            states.append(dag.test().get_task_instance("watch_orders").state)
+
+        with create_session() as session:
+            events = 
session.scalars(select(AssetEvent).order_by(AssetEvent.id)).all()
+            published = [(e.asset.uri, e.extra["change"], e.extra["version"]) 
for e in events]
+
+        assert states == ["success", "skipped", "success", "skipped", 
"skipped"]
+        assert published == [(URI, "enrollment", 2), (URI, "new_version", 3)]

Reply via email to