Re: Proposal to simplify start/end dates

2017-03-07 Thread Dan Davydov
Those semantics should not change with my specific proposal, but I think logically moving a DAG's start date back should backfill those old dagruns (not the current behavior which continues running from the current date). Interval changes are a bit of a hairy topic, I think those kinds of changes a

Re: Proposal to simplify start/end dates

2017-03-07 Thread Bolke de Bruin
Ok sounds good. What do you do with a dag that gets predated and with an existing dag run? What happens if the interval changes, i.e. non cron syntax? (Just thinking out loud) B. Sent from my iPhone > On 7 Mar 2017, at 22:27, Dan Davydov wrote: > > Sure thing. > > Current Behavior: > - Use

Re: Proposal to simplify start/end dates

2017-03-07 Thread Dan Davydov
Sure thing. Current Behavior: - User creates DAG with default_args start date to 2015 - dagrun gets kicked off for 2015 - User changes default_args start date to 2016 - dagruns continue running for 2015 New Behavior: - User creates DAG with default_args start date to 2015 - dagrun gets kicked off

Re: Proposal to simplify start/end dates

2017-03-07 Thread Bolke de Bruin
Hey Dan, Im not sure if I am seeing a difference for #1 vs now, except you are excluding backfills now from the calculation? Can you provide an example? Bolke > On 7 Mar 2017, at 20:38, Dan Davydov wrote: > > A very common source of confusion for our users is when they specify > start_date in

Proposal to simplify start/end dates

2017-03-07 Thread Dan Davydov
A very common source of confusion for our users is when they specify start_date in default_args but not in their DAG arguments and then try to change this start_date to move the execution of their DAG forward (e.g. from 2015 to 2016). This doesn't work because the logic that is used to calculate th