developer-rpai opened a new pull request, #74435:
URL: https://github.com/apache/airflow/pull/74435
Fixes #74428.
## What happened
With `use_job_schedule=True`, an `AssetOrTimeSchedule` Dag with
`allowed_run_types=["asset_triggered", "manual"]` is time-due, so
`DagModel.dags_needing_dagruns` selects it for scheduled-run creation.
`SchedulerJobRunner._create_dag_runs` then skips it ("Dag does not allow
scheduled runs; skipping") -- but the Dag has already consumed one of the
`max_dagruns_to_create_per_loop` batch slots. Enough such Dags starve
unrelated Dags of scheduled runs entirely.
## Fix
Exclude Dags that disallow scheduled runs from the time-based leg of the
`dags_needing_dagruns` candidate query, so they never occupy a batch slot.
This uses a new dialect-aware `JsonArrayContains` helper
(`airflow/utils/sqlalchemy.py`): the `?` operator over a JSONB cast on
PostgreSQL, `JSON_CONTAINS` on MySQL, and a `json_each` EXISTS subquery on
SQLite -- all with bound parameters. `allowed_run_types IS NULL` (no
restriction) keeps the previous behavior. The existing skip in
`_create_dag_runs` stays as a safety net.
## Testing
- New regression tests in `test_scheduler_job.py`:
`test_dags_needing_dagruns_excludes_dags_disallowing_scheduled_runs`
(fails without the fix) and a parametrized
`test_dags_needing_dagruns_selects_dags_allowing_scheduled_runs`
covering `["scheduled"]`, `["scheduled", "manual"]`, and `None`.
- Verified the generated SQL and predicate semantics on SQLite directly,
and compilation on PostgreSQL/MySQL dialects. Full CI verification
pending.
---
*AI disclosure: this change was developed with AI assistance and reviewed by
the author before submission.*
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: [email protected]
For queries about this service, please contact Infrastructure at:
[email protected]
