The GitHub Actions job "Tests (AMD)" on airflow.git/fix/asset-triggered-dag-run has failed. Run started by GitHub user dingo4dev (triggered by dingo4dev).
Head commit for run: dff84902c7d9f7501eaf15f7e668fafc236f27d9 / Stanley Law <[email protected]> Resolve race conditions on asset event and asset dag run queue Fixes an issue where the Scheduler misses AssetEvents due to a visibility gap between flush() and commit(). When multiple tasks create events for the same asset, the AssetEvent timestamp is generated during flush, but the record remains invisible to the Scheduler until the final commit. If the AssetDagRunQueue is processed in the interim, the 'late-committing' event is orphaned. The job scheduler will read the AssetDagRunQueue and fetch the created_at column as the triggered_date which will use for fetching the asset event and assign dag `run_after`. This change ensures timestamps are synchronized and the transaction boundary is tightened to prevent data-aware scheduling misses. Add check for asset events before creating asset-triggered DAG runs This is to prevent double triggering when the late-commit asset_dag_run_queue is exist in entry but the asset event is already process in prev dag Fix update statement condition in AssetDagRunQueue to use correct column reference Add tests for concurrent asset events and no asset event scenarios in DAG run creation linting tests make mypy happy Add check for existing asset DAG run queue entry before merge Co-authored-by: Kaxil Naik <[email protected]> add log when no dag run without asset event Support concurrent asset event queuing on MySQL Implement a MySQL-specific optimization using `ON DUPLICATE KEY UPDATE` when queuing asset-triggered DAG runs. This prevents unique constraint violations during concurrent asset events, mirroring the existing PostgreSQL implementation. The concurrency tests are also refactored to verify correct behavior across both database dialects. Refactor asset event handling and improve session management in AssetManager make lint happy Refactor asset event handling to streamline session management in AssetManager Re-attach asset_event to caller's session to prevent DetachedInstanceError fix: prevent race condition causing lost events in asset-triggered DAG run creation - Remove CTE lower-bound filter from asset event query: the bound excluded late-committed events that arrived after a concurrent DagRun was created, causing those events to be permanently lost and no follow-up DagRun created. - Add 'not consumed' filter using association_table to prevent duplicate consumption of asset events across concurrent DagRuns. - Always delete ADRQ rows after processing, even when no new DagRun is created, to prevent stale entries accumulating and causing infinite scheduler loops. - Fix concurrent test to use seen_dr_ids set instead of prev_dr reference, ensuring all DagRuns are counted regardless of creation order. fix: Persist AssetEvent in independent session for immediate visibility Persist AssetEvent using a truly independent session (`scoped=False`) to ensure it's committed and visible to processes like the Scheduler before other transaction operations proceed. This closes a critical visibility gap for Asset-Triggered DagRuns (ADRQ). The event is then explicitly reloaded into the caller's session to prevent DetachedInstanceError and allow subsequent relationship operations to function correctly. make static check happy refactor: Consolidate `AssetEvent` persistence logic Extract the complex logic for persisting `AssetEvent` instances into a new dedicated class method, `create_asset_event`. This centralizes the handling of distinct session management strategies (independent session for non-SQLite, direct flush for SQLite) and ensures immediate visibility of asset events to the scheduler. It also re-attaches the event to the caller's session to prevent `DetachedInstanceError`. fix: Ensure DagRun updates and AssetEvents are visible across transaction boundaries Add `session.flush()` after `DagRun` partition key updates to ensure these changes are persisted within the caller's transaction before an independent session for `AssetEvent` creation begins. This prevents stale `DagRun` data from being read. Change `AssetEvent` creation to use `create_session(scoped=False)`, guaranteeing a truly independent session and immediate commit of the asset event. This ensures the event is visible to processes like the Scheduler for Asset-Triggered DagRuns (ADRQ) without transactional delays. fix: Change log level from warning to info for DagRun creation status apply copilot reviewer Report URL: https://github.com/apache/airflow/actions/runs/26525458289 With regards, GitHub Actions via GitBox --------------------------------------------------------------------- To unsubscribe, e-mail: [email protected] For additional commands, e-mail: [email protected]
