Ok sounds good. What do you do with a dag that gets predated and with an existing dag run? What happens if the interval changes, i.e. non cron syntax?
(Just thinking out loud) B. Sent from my iPhone > On 7 Mar 2017, at 22:27, Dan Davydov <[email protected]> wrote: > > Sure thing. > > Current Behavior: > - User creates DAG with default_args start date to 2015 > - dagrun gets kicked off for 2015 > - User changes default_args start date to 2016 > - dagruns continue running for 2015 > > New Behavior: > - User creates DAG with default_args start date to 2015 > - dagrun gets kicked off for 2015 > - User changes default_args start date to 2016 > - *dagruns start running for the 2016 start date instead of 2015* > >> On Tue, Mar 7, 2017 at 11:49 AM, Bolke de Bruin <[email protected]> wrote: >> >> Hey Dan, >> >> Im not sure if I am seeing a difference for #1 vs now, except you are >> excluding backfills now from the calculation? Can you provide an example? >> >> Bolke >> >>> On 7 Mar 2017, at 20:38, Dan Davydov <[email protected]> >> wrote: >>> >>> A very common source of confusion for our users is when they specify >>> start_date in default_args but not in their DAG arguments and then try to >>> change this start_date to move the execution of their DAG forward (e.g. >>> from 2015 to 2016). This doesn't work because the logic that is used to >>> calculate the "initial" start date of a dag differs from the logic to >>> calculate subsequent dagrun start dates. >>> >>> Current Airflow Logic: >>> DS to schedule initial dagrun: dag.start_date if it exists, else >> min(start >>> date of tasks_of_dag) >>> DS to schedule subsequent dagruns: last_dagrun + scheduled_interval >>> >>> There are a couple ways of addressing this: >>> 1. Change the definition of start date for subsequent dagruns to match >> the >>> "initial" dagrun start date (calculated from the minimum of task start >>> dates) >>> 2. Force explicit dag start dates >>> >>> I personally like 1. >>> >>> I also propose that we throw errors for DAGs that have tasks that depend >> on >>> other tasks with start dates that occur after theirs (otherwise there >> could >>> be deadlocks). >>> >>> What do people think? >> >>
