dabla commented on code in PR #75659:
URL: https://github.com/apache/airflow/pull/75659#discussion_r4626541981


##########
airflow-core/docs/authoring-and-scheduling/deferring.rst:
##########
@@ -301,6 +303,21 @@ If you want to defer your task directly to the triggerer 
without going into the
 * ``next_kwargs``: Additional keyword arguments to pass to the ``next_method`` 
when it is called.
 * ``timeout``: (Optional) A timedelta that specifies a timeout after which 
this deferral will fail, and fail the task instance. Defaults to ``None``, 
which means no timeout.
 
+The scheduler defers these tasks in the same step where it queues other tasks, 
so pool slots,
+``max_active_tasks``, ``max_active_tis_per_dag`` and 
``max_active_tis_per_dagrun`` must allow it.
+They do not use executor slots. A task is deferred one scheduler loop after it 
becomes ``SCHEDULED``,

Review Comment:
   **[nit]** "They do not use executor slots" is only half true now.
   
   These tasks don't take a slot from a specific executor, but the deferral now 
happens inside the critical section, which is gated on executor capacity: it is 
skipped when all executors are full, and `max_tis` is bounded by 
`core.parallelism`. So a saturated executor does delay them (see [3]). Either 
fix the gate or say so here.



##########
airflow-core/src/airflow/jobs/scheduler_job_runner.py:
##########
@@ -1051,6 +1092,9 @@ def _select_task_instances_to_queue(
         stats.gauge("scheduler.tasks.starving", num_starving_tasks_total)
         stats.gauge("scheduler.tasks.executable", len(executable_tis))
 
+        for task_instance in tis_to_defer:
+            task_instance.defer_task(session=session)

Review Comment:
   **[blocker]** `start_from_trigger` tasks are no longer deferred while the 
executors are full.
   
   Before this PR, `schedule_tis()` deferred them whatever the executor's load 
was. Now deferral only happens in `_select_task_instances_to_queue()`, and the 
scheduler never gets there when workers are saturated:
   
   - `_do_scheduling` skips the critical section entirely when 
`sum(executor.slots_available) <= 0` ([line 
2353](https://github.com/apache/airflow/blob/e74b432a3f9056cbdfa00b2a4abfdd7941ca6c19/airflow-core/src/airflow/jobs/scheduler_job_runner.py#L2353),
 "All executors are full, skipping critical section").
   - `_critical_section_enqueue_task_instances` returns 0 when 
`core.parallelism - slots_occupied <= 0` ([line 
1412](https://github.com/apache/airflow/blob/e74b432a3f9056cbdfa00b2a4abfdd7941ca6c19/airflow-core/src/airflow/jobs/scheduler_job_runner.py#L1412)).
   - When it does run, `max_tis` still comes from free executor capacity, and 
deferred TIs now count against it (`len(executable_tis) + len(tis_to_defer) >= 
max_tis`).
   
   Scenario: `parallelism=32`, 32 long-running tasks occupy the workers, and a 
Dag with 100 `DateTimeSensorAsync(start_from_trigger=True)` tasks in a pool 
with free slots. All 100 stay `SCHEDULED` until a worker frees up, and then 
they go out at most `max_tis` per loop. Running without a worker is the whole 
point of `start_from_trigger`, and the docs in this PR say they "do not use 
executor slots".
   
   The deferral needs a budget that doesn't depend on executor capacity: for 
example, let the critical section run when there are `start_from_trigger` 
candidates even with full executors, and keep a separate counter for 
`tis_to_defer` instead of sharing `max_tis`. Please also add a test with 
`slots_available = 0` on the executor.



##########
airflow-core/src/airflow/ti_deps/dependencies_states.py:
##########
@@ -23,17 +23,23 @@
     TaskInstanceState.QUEUED,
 }
 
+# States counted towards max_active_tasks of a Dag run. A deferred task 
instance holds no worker slot but
+# is still in flight, so it takes a max_active_tasks slot until it finishes.
+DAG_RUN_ACTIVE_STATES = {

Review Comment:
   **[warning]** This reverses a deliberate decision for every deferrable task, 
not only `start_from_trigger`, and it's bundled with the pool fix.
   
   #61769 made deferred TIs count towards 
`max_active_tis_per_dag`/`_per_dagrun` and kept them out of `max_active_tasks` 
on purpose. It added 
`test_find_executable_task_instances_deferred_does_not_affect_max_active_tasks`,
 whose docstring reads "max_active_tasks is about worker-level parallelism, 
while deferred tasks don't consume worker slots". This PR renames that test and 
flips its assertion. Every Dag that relies on many deferrable sensors running 
at once (the newsfragment's own example) loses concurrency after upgrading, 
with no setting to get the old behaviour back except raising `max_active_tasks`.
   
   That may well be the right call for #40528, but it's a behaviour change with 
its own trade-off, separate from #75497 (start_from_trigger ignores pools), 
which is a clear bug. I'd suggest splitting it into its own PR, so the pool fix 
can land without waiting on agreement about what `max_active_tasks` means, and 
so the title says what changes for users.



##########
airflow-core/src/airflow/jobs/scheduler_job_runner.py:
##########
@@ -1051,6 +1092,9 @@ def _select_task_instances_to_queue(
         stats.gauge("scheduler.tasks.starving", num_starving_tasks_total)
         stats.gauge("scheduler.tasks.executable", len(executable_tis))
 
+        for task_instance in tis_to_defer:
+            task_instance.defer_task(session=session)

Review Comment:
   **[warning]** One TI that fails `defer_task()` now blocks queuing for every 
Dag.
   
   In `schedule_tis()`, an exception from `defer_task()` was contained per Dag 
run: `_schedule_all_dag_runs` catches `Exception` and logs "Error scheduling 
DAG run" ([line 
3402](https://github.com/apache/airflow/blob/e74b432a3f9056cbdfa00b2a4abfdd7941ca6c19/airflow-core/src/airflow/jobs/scheduler_job_runner.py#L3402)).
 Here it runs inside the critical section with no guard. If it raises (for 
example while `Trigger.encrypt_kwargs` serializes the `trigger_kwargs` with 
serde and Fernet), the exception leaves 
`_critical_section_enqueue_task_instances`, the whole selection for every Dag 
rolls back, and the scheduler loop dies. On restart the same `SCHEDULED` TI is 
picked again, so you get a crash loop that a single Dag causes.
   
   Guard each `defer_task()` call: log it, and fail that TI (or at least starve 
it) so the rest of the selection still gets queued. Per "Errors should never 
pass silently" ([PEP 20](https://peps.python.org/pep-0020/)), the failure 
should land on the TI that caused it, not on the scheduler.
   
   Side note: with `multi_team` on, `defer_task()` runs 
`DagModel.get_team_name` once per TI, inside the lock, even though this method 
already resolved `dag_id_to_team_name` for the batch.



##########
airflow-core/src/airflow/jobs/scheduler_job_runner.py:
##########
@@ -612,6 +609,37 @@ def _debug_dump(self, signum: int, frame: FrameType | 
None) -> None:
             self.log.info("\n\t".join(map(repr, callstack)))
             self.log.info("-" * 80)
 
+    def _set_task_if_started_from_trigger(
+        self,
+        task_instance: TI,
+        *,
+        serialized_dags: dict[tuple[str, str], SerializedDAG | None],
+        session: Session,
+    ) -> bool:
+        """
+        Set the task of ``task_instance`` and return True if the triggerer 
starts it, without the executor.
+
+        :param serialized_dags: the Dags already read, by Dag run: read once 
per Dag run, not per task.
+
+        :meta private:
+        """
+        # A task resumed by its trigger has next_method set: it must run its 
next method on the executor
+        if task_instance.next_method is not None:
+            return False
+        dag_run_key = (task_instance.dag_id, task_instance.run_id)
+        if dag_run_key not in serialized_dags:
+            serialized_dags[dag_run_key] = 
self.scheduler_dag_bag.get_dag_for_run(

Review Comment:
   **[warning]** This loads the serialized Dag for every selected TI inside the 
critical section, and repeats the lookup in 
`_task_concurrency_allows_execution`.
   
   The selection loop deliberately skips that load ([line 
989](https://github.com/apache/airflow/blob/e74b432a3f9056cbdfa00b2a4abfdd7941ca6c19/airflow-core/src/airflow/jobs/scheduler_job_runner.py#L989):
 "Many DAGs do not define task concurrency limits, so avoid loading the 
serialized DAG unless required"). With this change every TI that passes the 
pool and `max_active_tasks` checks pays for it, deferrable or not. The cost is 
real, not just an in-memory cache hit:
   
   - For a run with no `bundle_version` (any non-versioned bundle, e.g. the 
local one), `DBDagBag._version_from_dag_run` runs 
`DagVersion.get_latest_version` on every call. That's one extra query per Dag 
run per selection pass, while the pool rows and the advisory lock are held.
   - On a cache miss the Dag is deserialized inside the lock, and past the 
revalidation window the `dag_hash` check adds another query.
   
   It also duplicates the `get_dag_for_run` + `has_task` + `get_task` sequence 
of `_task_concurrency_allows_execution`, which already ran for the same TI when 
the Dag has task concurrency limits. Two lookups of the same task with 
different error handling (the new one has no `try/except` for a Dag that fails 
to load) will drift apart.
   
   Suggestion: gate on a cheap Dag-level flag the way 
`has_task_concurrency_limits` does (computed at parse time, so only Dags that 
have a `start_from_trigger` task pay for the lookup), and resolve the task once 
per TI for both checks. Numbers for `scheduler.critical_section_duration` 
before and after, on a deployment with many Dag runs, would show what this 
costs.



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: [email protected]

For queries about this service, please contact Infrastructure at:
[email protected]

Reply via email to