uranusjr commented on code in PR #71072:
URL: https://github.com/apache/airflow/pull/71072#discussion_r4582164436
##########
airflow-core/src/airflow/jobs/scheduler_job_runner.py:
##########
@@ -400,6 +424,12 @@ def __init__(
self._multi_team = conf.getboolean("core", "multi_team")
self._dag_tags_in_metrics = conf.getboolean("metrics",
"dag_tags_in_metrics", fallback=False)
self._max_partition_dag_runs_per_loop = MAX_PARTITION_DAG_RUNS_PER_LOOP
+ # Edge-triggers the "partition Dag run cap reached" audit Log rows:
the dag_ids whose row
+ # was committed during the current backlog episode. Cleared by
+ # `_report_partition_cap_backlog` when the backlog is at or below the
cap.
+ # Process-local: each scheduler in an HA deployment writes its own
audit row per Dag per
+ # episode; cross-process de-duplication would need a DB read per
callback run.
Review Comment:
Every scheduler runs this callback on its own timer. In an HA deployment
with N schedulers, each affected Dag now always gets N rows per episode. The
comment's reason for not deduplicating is weaker now that this runs once a
minute rather than every loop.
One option is to skip any dag_id that already has a “partition Dag run cap”
reached row within the last few intervals. OK if duplicates are acceptable, but
the comment should say each scheduler writes its own row.
##########
airflow-core/src/airflow/jobs/scheduler_job_runner.py:
##########
@@ -2521,28 +2552,22 @@ def _create_dagruns_for_partitioned_asset_dags(self,
session: Session) -> set[st
meantime, in which case the stale-fingerprint cleanup below drops it
instead.
"""
# Cap per-tick work so the scheduler transaction stays bounded and
other
- # scheduling work isn't starved. Remaining APDRs drain across
subsequent ticks.
- # FIFO is intentional: the oldest pending APDR fires first. A
persistently
- # unsatisfiable APDR at the head (e.g. broken mapper, upstream that
will
- # never arrive) blocks newer ones until an operator removes it or fixes
- # the underlying mapper. We surface the stuck state rather than
silently
- # rotating past it.
- # `with_row_locks(skip_locked=True)` mirrors the sibling ADRQ claim
path:
- # in HA two schedulers can otherwise both grab the same satisfied APDR
- # and race the `created_dag_run_id` UPDATE, orphaning whichever DagRun
- # loses. The `id` tiebreaker on `order_by` keeps LIMIT deterministic
when
- # two APDRs share a `created_at` under bulk asset-event ingestion.
- # SQLite is single-writer and silently drops `FOR UPDATE`, which is
fine.
+ # scheduling work isn't starved; the remainder drains over subsequent
ticks.
+ # FIFO is intentional: a persistently unsatisfiable APDR at the head
(e.g.
+ # broken mapper, upstream that will never arrive) blocks newer ones
until an
+ # operator removes it or fixes the mapper, surfacing the stuck state
rather
+ # than silently rotating past it.
+ # `with_row_locks(skip_locked=True)` mirrors the sibling ADRQ claim
path: in
+ # HA two schedulers can otherwise both grab the same satisfied APDR
and race
+ # the `created_dag_run_id` UPDATE, orphaning whichever DagRun loses.
The `id`
+ # tiebreaker on `order_by` keeps LIMIT deterministic when two APDRs
share a
+ # `created_at` under bulk asset-event ingestion. SQLite is
single-writer and
+ # silently drops `FOR UPDATE`, which is fine.
Review Comment:
This only changes wrapping now. Revert if it’s not intended.
##########
airflow-core/src/airflow/jobs/scheduler_job_runner.py:
##########
@@ -4332,6 +4357,61 @@ def _remove_unreferenced_triggers(self, *, session:
Session = NEW_SESSION) -> No
.execution_options(synchronize_session="fetch")
)
+ def _report_partition_cap_backlog(self) -> None:
+ """Log and audit the Dags in the pending partitioned Dag run backlog
above the per-tick cap."""
+ with create_session() as session:
+ backlogs_per_dag = {
+ dag_id: count
+ for dag_id, count in session.execute(
+ select(AssetPartitionDagRun.target_dag_id, func.count())
+ .select_from(AssetPartitionDagRun)
+ .join(DagModel, DagModel.dag_id ==
AssetPartitionDagRun.target_dag_id)
+ .where(*_build_pending_partitioned_apdr_filters())
+ .group_by(AssetPartitionDagRun.target_dag_id)
+ )
+ }
+ backlog_total = sum(backlogs_per_dag.values())
+ if backlog_total <= self._max_partition_dag_runs_per_loop:
+ self._partition_cap_reported_dag_ids.clear()
+ return
+
+ # No SQL-side ORDER BY: it would have no dag_id tie-break, and
relying on dict
+ # insertion order mirroring DB row order across backends (SQLite
in tests vs
+ # Postgres/MySQL in production) isn't a contract worth trusting —
sort explicitly.
Review Comment:
The "No SQL-side ORDER BY" note just explains the `sorted()` call on the
next line and is very AI-toned. Good candidate for human rewrite.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: [email protected]
For queries about this service, please contact Infrastructure at:
[email protected]