I'm generally in favor of this idea. Several people on my team have been confused by the different date options and their meaning.
For the dates, I think we should switch to providing alternate representations of dates exclusively via Jinja filters. So instead of "next_execution_date", you'd use "execution_date | delta(1)" or whatever. We have some operators that do this internally and it's generally much clearer to read. For inlets/outlets, surely these should be toggled by a config option as to whether you're using the lineage features, rather than removed wholesale? They're still in development, so it makes sense they'd be unused; but it would be annoying to have to patch Airflow to access the variables. On Mon, Apr 8, 2019 at 8:37 AM Bas Harenslak <basharens...@godatadriven.com> wrote: > Hi all, > > Following Tao Feng’s question to discuss this PR< > https://github.com/apache/airflow/pull/5010> (AIRFLOW-4192< > https://issues.apache.org/jira/browse/AIRFLOW-4192>), please discuss here > if you agree/disagree/would change. > > ----------- > > The summary of the PR: > > I was confused by the task context values and suggest to clean up and > clarify these variables. Some are derivations from other variables, some > are undocumented and unused, some are wrong (name doesn’t match the value). > Please discuss what you think of the removal of these variables: > > > * Removed yesterday_ds, yesterday_ds_nodash, tomorrow_ds, > tomorrow_ds_nodash. IMO the next_* and previous_* variables are useful > since these require complex logic to compute the next execution date, > however would leave computing the yesterday* and tomorrow* variables up to > the user since they are simple one-liners and don't relate to the DAG > interval. > * Removed tables. This is a field in params, and is thus also > accessible by the user ({{ params.tables }}). Also, it was undocumented. > * Removed latest_date. It's the same as ds and was also undocumented. > * Removed inlets and outlets. Also undocumented, and have the > inlets/outlets ever worked/ever been used by anybody? > * Removed end_date and END_DATE. Both have the same value, so it > doesn't make sense to have both variables. Also, the value is ds which > contains the start date of the interval, so the naming didn't make sense to > me. However, if anybody argues in favour of adding "start_date" and > "end_date" to provide the start and end datetime of task instance > intervals, I'd be happy to add them. > > Cheers, > Bas >
