Hi all,
I'd like to discuss a change proposed in PR #65856 [1] that relaxes the
blanket ban on future logical_date for manually triggered and
operator-triggered DAG runs.
Background: PR #46663 removed the allow_trigger_in_future config and
hardcoded a block on any DAG run whose logical_date is in the future. This
affects both the scheduler (_schedule_dag_run) and the RunnableExecDateDep
task dependency. The result: when a user triggers a DAG with a future
logical_date, the DagRun appears as "running" but tasks do not execute
until the logical_date is reached.
This is blocking our migration from Airflow 2 to 3 — we use logical_date to
represent business session dates that don't align with calendar dates, and
these runs need to execute immediately at trigger time.
Given that run_after is used for scheduling and logical_date is meant to
"not contain any semantics, but is simply a value for logical
identification," I believe it makes sense to relax the block on future
logical_date. Please see the PR description for the rationale and use
cases, and more details.
Proposed change: Skip the future logical_date block for MANUAL and
OPERATOR_TRIGGERED run types only. Scheduled runs remain blocked. The
existing run_after <= now() query filter still controls when the scheduler
picks up a run, but dag runs with future logical_date will be able to
execute immediately if run_after is set to None or now(). A "Run
immediately" checkbox in the Trigger DAG UI gives users explicit control.
Also added run_after to TriggerDagRunOperator and TriggerDAGRunPayload so
that users can control this from the operator path as well.
Why I believe this is non-breaking: The default behavior is unchanged. The
run_after field already controls when the scheduler picks up a run. Manual
triggers via the Core API already allow setting run_after=now() with a
future logical_date — this just removes the secondary block that prevents
it from working.
Broader question raised in review: logical_date is still used for TI
prioritization in _executable_task_instances_to_queued and as a scheduling
guard in _schedule_dag_run. Given that logical_date is defined as purely
identificational ("does not contain any semantics") while run_after is
defined as the scheduling control, should we decouple logical_date from
scheduling entirely? I think that's a separate, larger discussion — this PR
intentionally avoids changing TI ordering.
Looking for feedback on whether the approach is reasonable and if the
broader decoupling warrants its own discussion thread.
[1] https://github.com/apache/airflow/pull/65856
Thanks,
Mingjie Zhao
