Hello everyone, I've been looking into the *logical_date* and it seems that there are some inconsistencies in the documentation. In particular,
https://airflow.apache.org/docs/apache-airflow/stable/templates-ref.html#variables *"This value does not contain any semantics, but is simply a value for identification."* https://airflow.apache.org/docs/apache-airflow/stable/core-concepts/dags.html#running-dags *"there is another date called logical date (formally known as execution date), which describes the intended time a Dag run is scheduled or triggered."* One says that it's just an identifier while the other indicates that it's the intended start time. I looked at the code to get a better understanding of the field and I also tried to track down all the changes in the commit history since AIP-39 that introduced the *data_interval* and the *Timetable* interface. The concept doesn't become clear from the code either. It seems that the *logical_date* hasn't been updated entirely and some of the current behavior is derived from the old functionality from when it was still named *execution_date*. The *logical_date* is supposed to be an identifier for Dag runs and for scheduled runs it’s tied to the *data_interval_start*. The *data_interval* shows what date we are processing. Let’s say that we have 2 scenarios with a custom timetable or a manual trigger: 1. A Dag run starts today morning and it's supposed to be doing prep work for tomorrow. The user wants the *data_interval* and the *logical_date* to be tomorrow 2. A Dag run starts today morning and it's supposed to be doing processing for yesterday. The user wants the *data_interval* and the *logical_date* to be yesterday Case 2 works while for case 1, the Dag starts running and then the scheduler logs an error *"Logical date is in future: "* until the future date arrives. There used to be an option to allow a *logical_date* to point to the future but it was removed. The *run_after* field points at “when this should start”, and the scheduler already uses that to make sure that a Dag run isn't in the future. Therefore, the *logical_date* future restriction, which seems to have been a leftover from when it was still named *execution_date*, has become redundant and it could be removed. The scheduler still uses the *logical_date* to order the task instances during scheduling. If the future value constraint is removed, then *run_after* must be used for ordering in the scheduler, otherwise a run with a future logical date starts and then sorts behind every other run whenever slots are contended. The code changes are relatively small and so I've opened a PR but I think this also deserves a discussion for more visibility. PR: https://github.com/apache/airflow/pull/76489 The patch doesn't change how *logical_date* is derived. Furthermore, future logical dates will start only after the *run_after* time. Please let me know what you think. Thanks! Christos
