joseotaviorf opened a new pull request, #63316:
URL: https://github.com/apache/airflow/pull/63316
## Issue
When a dataset fires multiple times before its downstream DAG runs,
`DatasetDagRunQueue.created_at` is never refreshed. Both the PostgreSQL path
(`ON CONFLICT DO NOTHING`) and the non-Postgres path (a `session.merge()` with
no explicit `created_at`, which silently fails on conflict due to the
`nullable=False` constraint) leave the original timestamp stale. This means the
scheduler always sees the time of the *first* event rather than the *most
recent* one.
The bug is present across Airflow 2.10.4, 2.11.0, and 2.11.1.
## Changes
**`airflow/datasets/manager.py`**
- **Postgres path**: replaced `ON CONFLICT DO NOTHING` with `ON CONFLICT DO
UPDATE SET created_at = excluded.created_at`, so repeated events refresh the
timestamp.
- **Slow path (SQLite/MySQL)**: pass `created_at=timezone.utcnow()`
explicitly to the `DatasetDagRunQueue` constructor so that `session.merge()`
writes the fresh timestamp to any existing row instead of silently doing
nothing.
**`tests/datasets/test_manager.py`**
- `test_slow_path_queue_dagruns_updates_timestamp`: verifies that calling
`_slow_path_queue_dagruns` on an already-queued record updates `created_at` to
a time ≥ the moment of the call. Runs on all non-Postgres backends.
- `test_postgres_queue_dagruns_updates_timestamp`: same verification against
a real PostgreSQL database, gated with `@pytest.mark.backend("postgres")`.
## Test plan
- [ ] Run `pytest
tests/datasets/test_manager.py::TestDatasetManager::test_slow_path_queue_dagruns_updates_timestamp`
against SQLite
- [ ] Run `pytest
tests/datasets/test_manager.py::TestDatasetManager::test_postgres_queue_dagruns_updates_timestamp
--backend postgres` against PostgreSQL
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: [email protected]
For queries about this service, please contact Infrastructure at:
[email protected]
