avolant commented on PR #70307:
URL: https://github.com/apache/airflow/pull/70307#issuecomment-5997833474
Thanks for the review. All four points are addressed:
1. Rebased onto `main`. The selected-ID exclusion is now the keyword-only
`excluded_ti_ids` parameter of `_build_schedulable_tis_query`, applied before
the `row_number()` window together with the starvation filters.
2. Refill queries are capped at `MAX_TI_REFILL_QUERIES_PER_LOOP = 3`, an
internal constant next to `MAX_PARTITION_DAG_RUNS_PER_LOOP`. Hitting the cap
logs at debug level and increments
`scheduler.critical_section_refill_limit_reached`, which is added to the
metrics registry. The production logs earlier in this thread only show retries
after the first query, so three leaves headroom. The cap counts only queries
issued after the first TI is selected. Queries before that stay uncapped as on
`main`, because every loop restarts with empty filters, and a cap there could
starve runnable TIs indefinitely behind enough blocked ones.
3. Added one parametrized test over a scenario where each three-row page
holds a single runnable TI until the last page:
- `executor_slots_shared_across_queries` uses two executor slots with
`max_tis=3` and expects exactly two queued TIs. It fails when the slot map is
rebuilt per query.
- `fills_batch_up_to_max_tis` ends on a page with two runnable TIs for
one free position and expects exactly three. It fails without the in-page break.
A separate test patches the cap to 1, expects the batch to stop after one
refill, and checks the counter.
4. Dropped the `session.scalars.call_count` assertion and updated the
description.
One more behaviour change surfaced while writing the tests. `main` also ends
the loop on a short page, but a run's `max_active_tasks` window can return only
its blocked TIs while hiding its runnable ones. The PR keeps querying in that
case, and
`test_select_task_instances_to_queue_finds_runnable_hidden_by_max_active_tasks_window`
covers it.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: [email protected]
For queries about this service, please contact Infrastructure at:
[email protected]
