ramitkataria commented on code in PR #76303:
URL: https://github.com/apache/airflow/pull/76303#discussion_r4622721345


##########
airflow-core/src/airflow/models/trigger.py:
##########
@@ -908,17 +908,19 @@ def _(event: BaseTaskEndEvent, *, task_instance: 
TaskInstance, session: Session)
     callback_type = event.task_instance_state
     should_retry = False
     dag_bag = DBDagBag()
+    dag = None
 
-    if event.task_instance_state == TaskInstanceState.FAILED:
-        # Load the serialized task so retry eligibility matches the normal 
task path.
+    if event.task_instance_state in (TaskInstanceState.SUCCESS, 
TaskInstanceState.FAILED):

Review Comment:
   This now loads and deserializes the Dag for every trigger-ended SUCCESS too; 
before it was FAILED only. As far as I can tell the success request is dropped 
either way: the sink skips it with routing on, and `_execute_task_callbacks` 
returns early for non-failure requests with routing off.
   
   This runs with the flag off as well. Could we keep the load to FAILED? The 
`relative_fileloc` fallback below already covers the case where `dag` is `None`.



##########
task-sdk/src/airflow/sdk/api/client.py:
##########
@@ -1204,11 +1208,33 @@ def __init__(self, client: Client):
 
     def run(self, callback_id: uuid.UUID) -> None:
         """Exchange the single-use callback token for an execution token."""
-        self.client.patch(f"callbacks/{callback_id}/run")
+        # Retries of this request resend the same attempt ID, so the API 
server accepts a retry whose
+        # first response was lost, but not a second delivery of the callback.
+        body = CallbackRunPayload(hostname=get_hostname(), pid=os.getpid(), 
attempt_id=uuid.uuid4())
+        try:
+            self.client.patch(f"callbacks/{callback_id}/run", 
content=body.model_dump_json())
+        except ServerResponseError as e:
+            if e.response.status_code == HTTPStatus.CONFLICT:

Review Comment:
   Dropping a redelivered callback quietly makes sense when the first worker is 
still alive. I think there's a gap when it died after the token exchange: the 
redelivery is ignored on Celery, and the reaper only looks at `dag_attribute` 
rows, so a deadline (`import_path`) callback stays RUNNING and holds its 
executor slot until the scheduler restarts.
   
   Before this change the redelivery was recorded as a failure, which settled 
the row. This applies with routing off too. Could the reaper's RUNNING check 
cover deadline callbacks as well?



##########
task-sdk/src/airflow/sdk/execution_time/callback_supervisor.py:
##########
@@ -534,22 +707,27 @@ def supervise_callback(
                 exit_code=exit_code,
                 duration=end - start,
             )
-            if exit_code == 0:
-                terminal_state = CallbackTerminalState.SUCCESS
-            else:
+            if process.timed_out:
+                stats.incr("callback.timeouts")
+                terminal_output = (
+                    f"Callback timed out after 
{process._get_callback_execution_timeout()} seconds"
+                )
+            elif exit_code != 0:
                 terminal_output = f"Callback subprocess exited with code 
{exit_code}"
+            else:
+                terminal_state = CallbackTerminalState.SUCCESS
         except Exception as e:
             terminal_output = f"Callback supervisor error: {type(e).__name__}: 
{e}"
             raise
         finally:
-            try:
-                client.callbacks.finish(UUID(id), state=terminal_state, 
output=terminal_output)
-            except Exception:
-                log.exception("Failed to report final callback state", 
workload_id=id)
+            _report_callback_state(client, id, terminal_state, terminal_output)
+            # Report first so a slow log upload cannot use up the scheduler's 
timeout grace period.
+            if process is not None:
+                process._upload_logs()
 
             if log_path and log_file_descriptor:
                 log_file_descriptor.close()
 
-    if exit_code != 0:
-        raise RuntimeError(f"Callback subprocess exited with code {exit_code}")
+    if terminal_state == CallbackTerminalState.FAILED:
+        raise RuntimeError(terminal_output)

Review Comment:
   Potentially out of scope but a failed callback is reported as FAILED and 
then we raise here, so the process exits non-zero. A failed task exits 0 
(`supervise_task` returns its exit code), and the ECS and Batch executors rely 
on that: they read a non-zero exit as a failed launch and resubmit. So a failed 
callback is relaunched until the attempts run out, with each relaunch getting a 
409 from `/run`.
   
   What if we exit cleanly here like tasks do? Or do you have other ideas? I 
can help with a followup PR as well about this



##########
airflow-core/src/airflow/jobs/scheduler_job_runner.py:
##########
@@ -1481,10 +1565,444 @@ def _enqueue_executor_callbacks(self, session: 
Session) -> None:
                     generator=executor.jwt_generator,
                 )
 
-                executor.queue_workload(workload, session=session)
-                callback.state = CallbackState.QUEUED
+                try:
+                    executor.queue_workload(workload, session=session)
+                except NotImplementedError as e:
+                    # Some executors run callbacks without declaring it, so 
ask the executor itself.
+                    self.log.error("Failing callback %s: %s", callback.id, e)
+                    callback.state = CallbackState.FAILED
+                    callback.output = str(e)
+                else:
+                    callback.state = CallbackState.QUEUED
                 session.add(callback)
 
+    def _route_dag_processor_callbacks(self, *, session: Session) -> None:
+        """
+        Queue Dag processor callbacks on executors.
+
+        With routing on, callback sinks store Dag processor callbacks as 
executor callbacks waiting
+        SCHEDULED (see ``ExecutorCallback.from_dag_processor_request``). A 
routed callback is marked
+        QUEUED in the transaction that queues it. One that no executor can run 
is failed with the
+        reason, and one that fails to route is tried again in a later loop, up 
to
+        ``MAX_ROUTED_CALLBACK_TRIES``.
+
+        Each scheduler admits callbacks per loop into at most half its free 
logical slots and half
+        each of its executor instances' free slots, rounded down, with at 
least one when a slot is
+        free. This leaves local capacity for tasks when more than one slot is 
free; it does not
+        reserve shared workers or limit callback concurrency across 
schedulers. Executors send
+        callbacks first, and one left waiting in an executor for a slot would 
age toward its start
+        deadline. Callbacks that could be routed but find no free slot wait 
for a later loop, while
+        those no executor can run are failed without waiting for one.
+        """
+        free_slots = self._parallelism - sum(executor.slots_occupied for 
executor in self.executors)
+        budget = min(self._max_routed_callbacks_per_loop, max(1, free_slots // 
2)) if free_slots > 0 else 0
+        executor_budgets = {
+            executor: max(1, executor.slots_available // 2) if 
executor.slots_available > 0 else 0
+            for executor in self.executors
+        }
+        now = timezone.utcnow()
+        examined: set[UUID] = set()
+        if budget > 0:
+            # Oldest first, so callbacks are routed in the order they were 
requested.
+            oldest = self._select_waiting_callbacks(session=session)
+            for callback in oldest:
+                examined.add(callback.id)
+                if executor := self._settle_waiting_callback(
+                    callback,
+                    now=now,
+                    can_queue=budget > 0,
+                    executor_budgets=executor_budgets,
+                    session=session,
+                ):
+                    budget -= 1
+                    executor_budgets[executor] -= 1
+            if len(oldest) < self._max_routed_callbacks_per_loop:
+                # That was every waiting callback no other scheduler is 
locking.
+                return
+
+        # After a full oldest window, or none for lack of a free slot, the 
next window is checked too,
+        # resuming after the last one checked, so callbacks waiting for a slot 
don't keep the rest from
+        # being routed or failed.
+        swept = self._select_waiting_callbacks(
+            session=session, after=self._callback_sweep_cursor, 
exclude=examined
+        )
+        self._callback_sweep_cursor = (
+            swept[-1].id if len(swept) == self._max_routed_callbacks_per_loop 
else None
+        )
+        for callback in swept:
+            if executor := self._settle_waiting_callback(
+                callback,
+                now=now,
+                can_queue=budget > 0,
+                executor_budgets=executor_budgets,
+                session=session,
+            ):
+                budget -= 1
+                executor_budgets[executor] -= 1
+
+    def _select_waiting_callbacks(
+        self, *, session: Session, after: UUID | None = None, exclude: 
set[UUID] | None = None
+    ) -> Sequence[ExecutorCallback]:
+        query = (
+            select(ExecutorCallback)
+            .where(
+                ExecutorCallback.fetch_method == 
CallbackFetchMethod.DAG_ATTRIBUTE,
+                ExecutorCallback.state == CallbackState.SCHEDULED,
+            )
+            .order_by(ExecutorCallback.id)
+            .limit(self._max_routed_callbacks_per_loop)
+        )
+        if after is not None:
+            query = query.where(ExecutorCallback.id > after)
+        if exclude:
+            query = query.where(ExecutorCallback.id.not_in(exclude))
+        return session.scalars(with_row_locks(query, session, 
of=ExecutorCallback, skip_locked=True)).all()
+
+    def _settle_waiting_callback(
+        self,
+        callback: ExecutorCallback,
+        *,
+        now: datetime,
+        can_queue: bool,
+        executor_budgets: dict[BaseExecutor, int],
+        session: Session,
+    ) -> BaseExecutor | None:
+        """
+        Route a waiting callback, fail it if no executor can run it, or leave 
it waiting.
+
+        A callback left waiting is only looked up again once its executor has 
a free slot: waiting costs
+        no queries, and the callback is queued by its current Dag run and team.
+
+        :return: The executor it was queued on, or ``None`` if it wasn't 
queued.
+        """
+
+        def has_free_slot(executor: BaseExecutor) -> bool:
+            return can_queue and executor_budgets[executor] > 0 and 
executor.slots_available > 0
+
+        try:
+            executor = self._waiting_callback_executors.pop(callback.id, None)
+            if executor is not None and not has_free_slot(executor):
+                self._waiting_callback_executors[callback.id] = executor
+                return None
+            with session.begin_nested():
+                target = self._get_callback_target(callback, session=session)
+                if has_free_slot(target.executor):
+                    self._queue_routed_callback(callback, target, now=now, 
session=session)
+                    return target.executor
+            self._waiting_callback_executors[callback.id] = target.executor
+        except _UnroutableCallbackError as e:
+            self.log.warning("Failing callback %s, which no executor can run: 
%s", callback.id, e)
+            callback.state = CallbackState.FAILED
+            callback.output = str(e)
+        except Exception as e:
+            self.log.exception("Failed to route callback %s", callback.id)
+            try_number = callback.data.get(TRY_NUMBER_KEY, 1)
+            if try_number < MAX_ROUTED_CALLBACK_TRIES:
+                callback.data = {**callback.data, TRY_NUMBER_KEY: try_number + 
1}
+            else:
+                callback.state = CallbackState.FAILED
+                callback.output = f"Could not be routed in {try_number} tries: 
{e}"
+        return None
+
+    def _get_callback_target(self, callback: ExecutorCallback, *, session: 
Session) -> _CallbackTarget:
+        """
+        Return where a waiting Dag processor callback runs.
+
+        :raises _UnroutableCallbackError: If no executor can run it.
+        """
+        request = callback.get_callback_request()
+        if request is None:
+            raise _UnroutableCallbackError("It stores no callback request")
+        if get_claiming_importer(request.filepath, request.bundle_name) is not 
None:
+            raise _UnroutableCallbackError("Callbacks of Lang-SDK Dags are not 
supported")
+
+        executor_name: str | None = None
+        queue: str | None = None
+        executor_config: dict | None = None
+        if isinstance(request, DagCallbackRequest):

Review Comment:
   Dag callbacks always go to the default executor and queue, so those workers 
need to be able to import every Dag file that has callbacks (for example a Dag 
whose tasks run on a queue with extra dependencies). If the parse fails on the 
worker, the callback ends up FAILED with no retry. Could the docs call that out 
as a requirement?



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: [email protected]

For queries about this service, please contact Infrastructure at:
[email protected]

Reply via email to