Does anyone actually use end_date (of either spelling) given it's value is
currently the same as `ds`:
https://github.com/apache/airflow/blob/3020d9733cb189f091489a62f58f0f586dc8d4a9/airflow/models/taskinstance.py#L1183-L1184
'END_DATE': ds,
'end_date': ds,
That seems like it is just the wrong value - I could see it being useful as
either the same value as `next_execution_date` or `dag.end_date` but the
current value (from a quick thought) doesn't seem like it is useful?
Given what the value is currently set to I vote for deprecating/removing these
two.
> On 12 Apr 2019, at 09:39, Bolke de Bruin <bdbr...@gmail.com> wrote:
>
> I don’t think guarding inlets and outlets makes sense, cause if they are not
> defined they code won’t be executed anyway. Ie. The logic is already there,
> no need for a config flag.
>
> End-date is actually used by some beyond Airbnb as far as I know. What is the
> reason for deprecating it?
>
> Execution_date in finance is actually very well known. A transaction has an
> execution date it is not the time of the sending of the data. I honestly
> think that a change here would make it a thing that techies understand but
> does not make business sense. What is “today’s data?” it might not be the
> data that was sent today.
>
>
>
> Verstuurd vanaf mijn iPad
>
>> Op 12 apr. 2019 om 08:35 heeft Maxime Beauchemin
>> <maximebeauche...@gmail.com> het volgende geschreven:
>>
>> It should be labelled as a "bridge" or "transition" release somehow, and
>> tell the community to absolutely to go there first and address all
>> deprecation warnings prior to upgrade to 2.0. I wonder if semver has
>> something for that.
>>
>> It might be the time to deprecate more things. Spring cleaning!
>>
>> Should we start another thread labeled "[2.0 spring cleaning] what Airflow
>> features should we deprecate!?"
>>
>> Max
>>
>>> On Thu, Apr 11, 2019 at 11:26 PM Ash Berlin-Taylor <a...@apache.org> wrote:
>>>
>>> I am happy to do another small(!) 1.10.x release.
>>>
>>> (There was a small bug I introduced where I broke the rendering of doc_md
>>> on dags)
>>>
>>> On 11 April 2019 22:52:13 BST, "Driesprong, Fokko" <fo...@driesprong.frl>
>>> wrote:
>>>> I agree with Max here, we should be careful.
>>>>
>>>> Regarding the yesterday_ds, yesterday_ds_nodash, tomorrow_ds,
>>>> tomorrow_ds_nodash. I'm not against having better readable shorthands,
>>>> but
>>>> more about the fact that having a tomorrow_ds doesn't make sense when
>>>> we
>>>> have an hourly (or weekly) job.
>>>>
>>>> I'm against guarding the {in,out}lets. Since the guarding will add
>>>> extra
>>>> logic, and this is not worth the added complexity (and another flag) in
>>>> my
>>>> opinion.
>>>>
>>>> - We start by putting deprecation warnings on tables, latest_date,
>>>> end_date
>>>> and END_DATE and remove them in Airflow 2.0.
>>>>
>>>> This is only possible if there is another version before 2.0.
>>>> Currently,
>>>> we're preparing to move to 2.0
>>>>
>>>> - We add a lineage_enabled config which is false by default and thus
>>>> inlets/outlets aren’t provided, unless set to true.
>>>>
>>>> I'm against guarding the {in,out}lets. Since the guarding will add
>>>> extra
>>>> logic, and this is not worth the added complexity (and another flag) in
>>>> my
>>>> opinion.
>>>>
>>>> - We continue discussion about clarification of execution_date and
>>>> better
>>>> named variables such as something_start and something_end
>>>>
>>>>
>>> https://lists.apache.org/thread.html/31423aa7feba421c53356a1e566f777c7a7973966c3320611286a2fb@%3Cdev.airflow.apache.org%3E
>>>>
>>>> Cheers, Fokko
>>>>
>>>> Op do 11 apr. 2019 om 08:44 schreef Bas Harenslak <
>>>> basharens...@godatadriven.com>:
>>>>
>>>>> Great discussion, let’s stay on track. If I can summarise:
>>>>>
>>>>>
>>>>> * yesterday_ds, yesterday_ds_nodash, tomorrow_ds,
>>>> tomorrow_ds_nodash
>>>>> * Arthur: some users use these for convenience
>>>>> * Bas/Fokko: these are values that can be easily derived in a
>>>>> one-liner
>>>>>
>>>>> * tables
>>>>> * nobody?
>>>>>
>>>>> * latest_date
>>>>> * Arthur: used by internal backfill framework at Airbnb, but
>>>> no
>>>>> issue removing them.
>>>>>
>>>>> * inlets/outlets
>>>>> * Arthur: still in development, could be guarded behind
>>>> feature
>>>>> flag.
>>>>>
>>>>> * end_date/END_DATE
>>>>> * Arthur: used by internal backfill framework at Airbnb, but
>>>> no
>>>>> issue removing them.
>>>>>
>>>>> And to add to the discussion:
>>>>>
>>>>>
>>>>> * execution_date
>>>>> * The meaning of execution_date is ambivalent and super
>>>> confusing
>>>>> to everybody because it’s not the date of execution.
>>>>> * Suggestions to add period_start/period_end, or
>>>>> interval_start/interval_end, or something like that, to provide
>>>> datetime
>>>>> variables for start & end datetime of a DAG run.
>>>>> * Other:
>>>>> * Removal of variables should be done in major version and
>>>>> deprecation warnings should be added.
>>>>>
>>>>> So how about the following:
>>>>> - We start by putting deprecation warnings on tables, latest_date,
>>>>> end_date and END_DATE and remove them in Airflow 2.0.
>>>>> - We add a lineage_enabled config which is false by default and thus
>>>>> inlets/outlets aren’t provided, unless set to true.
>>>>> - We continue discussion about yesterday* and tomorrow*
>>>>> - We continue discussion about clarification of execution_date and
>>>> better
>>>>> named variables such as something_start and something_end
>>>>>
>>>>> Cheers,
>>>>> Bas
>>>>>
>>>>>
>>>>> On 11 Apr 2019, at 04:52, Maxime Beauchemin
>>>> <maximebeauche...@gmail.com
>>>>> <mailto:maximebeauche...@gmail.com>> wrote:
>>>>>
>>>>> Making backwards incompatible changes that require altering the
>>>> thousands
>>>>> (millions?!) of DAGs in the wild will alienate the community and
>>>> prevent
>>>>> many from orchestrating an upgrade. Upgrading hundreds of DAGs and
>>>> Airflow
>>>>> atomically would be hard and dangerous.
>>>>>
>>>>> To mitigate this, changes to the DAG / core API should be backward
>>>>> compatible, at least for a release range, and alert on upcoming
>>>> deprecation
>>>>> (if needed)
>>>>>
>>>>> To be clear if `execution_date` is renamed to `foo`, we should have a
>>>>> version range where both work just as well, but using
>>>> `execution_date`
>>>>> would result is warning messages about its upcoming deprecation,
>>>> maybe
>>>>> something like "It appears you're using the task parameter
>>>> `execution_date`
>>>>> which will be deprecated in favor or `foo` in upcoming release 2.0.0,
>>>> find
>>>>> more info at link.to/foo<http://link.to/foo>"
>>>>>
>>>>> Max
>>>>>
>>>>> On Tue, Apr 9, 2019 at 7:58 AM James Meickle
>>>>>
>>>> <jmeic...@quantopian.com.invalid<mailto:jmeic...@quantopian.com.invalid>>
>>>>> wrote:
>>>>>
>>>>> I agree with Ash here. The naming of "execution_date" is incredibly
>>>>> confusing to people who are new to Airflow, who think it has
>>>> something to
>>>>> do with... execution.
>>>>>
>>>>> However, I think that there's still room for improvement with
>>>>> "period_start" and "period_end". Think about manually triggered tasks
>>>> -
>>>>> they'd have a "period_start", but no useful "period_end" until the
>>>> next
>>>>> trigger occurs. And mid-day ETL tasks that are dated to "today" still
>>>> have
>>>>> to reference "period_end" to get the correct date, even though the
>>>> DAGrun
>>>>> won't be over yet!
>>>>>
>>>>> If we're going to go through and rename "execution_date", I'd rather
>>>> see a
>>>>> wider-ranging rework that understands a mapping from a date to a
>>>> window
>>>>> (like "every day, process the data from one week ago", or "every
>>>> Saturday,
>>>>> process today's data"). Maybe something closer to how Beam does it
>>>>> (but retaining simplicity for the 99% of cases that are just "run
>>>> daily,
>>>>> for the past day").
>>>>>
>>>>>
>>>>>
>>>>
>>> https://beam.apache.org/documentation/programming-guide/#provided-windowing-functions
>>>>>
>>>>> I could see something where there are two default DAG parameters are:
>>>>> schedule="@daily"
>>>>> window=-1 (or some schedule_delta object that provides a windowing
>>>> impl, or
>>>>> None if there is no window)
>>>>>
>>>>> In this world, all DAGRuns would have a "schedule_date" (the date
>>>> this
>>>>> would have normally started at), and an "execution_date" (when it was
>>>>> actually executed). If a given DAGrun has a window (one-off DAGs may
>>>> or may
>>>>> not!), there would be variables for "window_start" (the start of the
>>>>> window) and "window_end" (the end of the window; this would default
>>>> to the
>>>>> same as schedule_date, and to the next DAGrun's window_start).
>>>>>
>>>>> Disconnecting schedule date, execution date, and window/period would
>>>> also
>>>>> open a pathway for more advanced users to implement irregular
>>>> schedules and
>>>>> windows, without having to rely on hacks. For example: a DAG that
>>>> runs at
>>>>> "the end of the week" to produce a end-of-week would normally run on
>>>>> Friday, but on Friday holidays, instead complete a day early (and
>>>> scan one
>>>>> fewer day in its window).
>>>>>
>>>>> I know there's some historical baggage here, but I think the above is
>>>> much
>>>>> more natural to new users than what we're offering today. And I think
>>>> from
>>>>> a current user perspective, there would be breaking variable name
>>>> changes,
>>>>> but no logical differences for the majority of DAGs.
>>>>>
>>>>> On Tue, Apr 9, 2019 at 6:17 AM Ash Berlin-Taylor <a...@apache.org>
>>>> wrote:
>>>>>
>>>>> To (slightly) hijack this thread:
>>>>>
>>>>> On the subject of execuction_date: as I'm sure we're all aware the
>>>>> concept
>>>>> of execution_date is confusing to new-commers to Airflow (there are
>>>> many
>>>>> questions about "why hasn't my DAG run yet"? "Why is my dag a day
>>>>> behind?"
>>>>> etc.) and although we mention this in the docs it's a confusing
>>>> concept.
>>>>>
>>>>> What to people think about adding two new parameters: `period_start`
>>>> and
>>>>> `period_end` and making these the preferred terms in place of
>>>>> execution_date and next_execution_date?
>>>>>
>>>>> This hopefully avoids any ambitious terms like "execution" or "run"
>>>> which
>>>>> is understandably easy to conflate with the time the task is being
>>>> run
>>>>> (i.e. `now()`)
>>>>>
>>>>> If people think this naming is better and less confusing I would
>>>> suggest
>>>>> we update all the docs and examples to use these terms (but still
>>>> mention
>>>>> the old names somewhere, probably in the macros docs)
>>>>>
>>>>> Thoughts?
>>>>>
>>>>> -ash
>>>>>
>>>>>
>>>>> On 8 Apr 2019, at 16:20, Arthur Wiedmer <arthur.wied...@gmail.com>
>>>>> wrote:
>>>>>
>>>>> Hi Bas,
>>>>>
>>>>> 1) I am aware of a few places where those parameters are used in
>>>>> production
>>>>> in a few hundred jobs. I highly recommend we don't deprecate them
>>>>> unless
>>>>> we
>>>>> do it in a major version.
>>>>>
>>>>> 2) As James mentioned, inlets and outlets are a lineage annotation
>>>>> feature
>>>>> which is still under development. Let's leave them in, but we can
>>>> guard
>>>>> them behind a feature flag if you prefer.
>>>>>
>>>>> 3) the yesterday*/tomorrow* params are convenience ones if you use a
>>>>> daily
>>>>> ETL. I agree with you that they are simple to compute, but not
>>>> everyone
>>>>> using Apache Airflow is amazing with Python. Some users are only
>>>> trying
>>>>> to
>>>>> get a query scheduled and rely on a couple of niceties like these to
>>>>> get
>>>>> by.
>>>>>
>>>>> 4) latest_date, end_date (I feel like there used to be start_date,
>>>> but
>>>>> maybe it got lost) were a blend of things which were used by a
>>>> backfill
>>>>> framework used internally at Airbnb. Latest date was used if you
>>>> needed
>>>>> to
>>>>> join to a dimension for which you only wanted the latest version of
>>>> the
>>>>> attributes in you backfill. end_date was used for time ranges where
>>>>> several
>>>>> days were processed together in a range to save on compute. I don't
>>>> see
>>>>> an
>>>>> issue with removing them.
>>>>>
>>>>> Best regards,
>>>>> Arthur
>>>>>
>>>>>
>>>>>
>>>>> On Mon, Apr 8, 2019 at 5:37 AM Bas Harenslak <
>>>>> basharens...@godatadriven.com>
>>>>> wrote:
>>>>>
>>>>> Hi all,
>>>>>
>>>>> Following Tao Feng’s question to discuss this PR<
>>>>> https://github.com/apache/airflow/pull/5010> (AIRFLOW-4192<
>>>>> https://issues.apache.org/jira/browse/AIRFLOW-4192>), please discuss
>>>>> here
>>>>> if you agree/disagree/would change.
>>>>>
>>>>> -----------
>>>>>
>>>>> The summary of the PR:
>>>>>
>>>>> I was confused by the task context values and suggest to clean up and
>>>>> clarify these variables. Some are derivations from other variables,
>>>>> some
>>>>> are undocumented and unused, some are wrong (name doesn’t match the
>>>>> value).
>>>>> Please discuss what you think of the removal of these variables:
>>>>>
>>>>>
>>>>> * Removed yesterday_ds, yesterday_ds_nodash, tomorrow_ds,
>>>>> tomorrow_ds_nodash. IMO the next_* and previous_* variables are
>>>> useful
>>>>> since these require complex logic to compute the next execution date,
>>>>> however would leave computing the yesterday* and tomorrow* variables
>>>>> up
>>>>> to
>>>>> the user since they are simple one-liners and don't relate to the DAG
>>>>> interval.
>>>>> * Removed tables. This is a field in params, and is thus also
>>>>> accessible by the user ({{ params.tables }}). Also, it was
>>>>> undocumented.
>>>>> * Removed latest_date. It's the same as ds and was also
>>>>> undocumented.
>>>>> * Removed inlets and outlets. Also undocumented, and have the
>>>>> inlets/outlets ever worked/ever been used by anybody?
>>>>> * Removed end_date and END_DATE. Both have the same value, so it
>>>>> doesn't make sense to have both variables. Also, the value is ds
>>>> which
>>>>> contains the start date of the interval, so the naming didn't make
>>>>> sense to
>>>>> me. However, if anybody argues in favour of adding "start_date" and
>>>>> "end_date" to provide the start and end datetime of task instance
>>>>> intervals, I'd be happy to add them.
>>>>>
>>>>> Cheers,
>>>>> Bas
>>>>>
>>>>>
>>>>>
>>>>>
>>>>>
>>>>>
>>>
