zozo123 opened a new issue, #74195: URL: https://github.com/apache/airflow/issues/74195
### Description Add explicit Delta-table version observation, first as a sensor and subsequently as a checkpointed external asset producer using existing Airflow scheduling/event facilities. ### Use case/motivation #74191 addresses Airflow-owned COPY INTO writes. A table can also change through another job or system. Downstream DAGs need a defined way to react to those external commits without depending on metadata updated_at timestamps. ### Delivery and acceptance - Sensor PR: take an explicit table identity and baseline version; wait for a newer committed Delta version. Use deferrable polling where supported, with async-safe authentication and bounded backoff. - External-producer PR: choose and document delivery semantics. Coalescing to the latest observed version is acceptable for an initial implementation; do not imply every intermediate commit is delivered. - Define initial enrollment, checkpoint scope, concurrent pollers, table rename/recreation, missing history and inaccessible/non-Delta tables. - Event provenance should include observed Delta version and table identity. Reuse the provider's existing asset URI conventions and the outcome of #74191; do not silently introduce a second URI scheme. - Prove crash behavior before and after event publication/checkpoint advancement. Where atomicity is unavailable, document at-least-once delivery and provide an explicit deduplication identity rather than claiming exactly-once events. - Test unchanged/new versions, recreation, expired history, restart and concurrent checkpoint updates. - Document that event version N does not automatically pin downstream reads to N; consumers requiring reproducibility must explicitly read that version. ### Related issues Follow-up to #74191, not a blocker. A sensor waits inside an already scheduled task; continuous external ingestion needs a separate recurring producer or supported watcher. No core scheduler redesign is proposed. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected]
