I am all for the change - it's been a source of confusion.
And yes, automatically rewriting the DAGs and possibly even detection with
error explaining what to do (on Airflow 3) if template string is passed to
"old" template_field - are really a prerequisite for that (and should be
quite possible and fairly reliable, I think).
Also - we should find a good approach for the community (and possibly
3rd-party) providers that might be run potentially with both: Airflow 2 and
Airflow 3. Currently we have 1252 usages of template_fields in the
community providers and if we release providers without them, they will
essentially stop working for Airflow 2 (which is fine at some point of
time, but likely not immediately - even after releasing Airflow 3).
My first thought is that we should keep template_fields, and template_ext
for a while - until we drop Airflow 2 for new provider releases. Possibly
we should move them aside (and keep Airflow 2 compatibility via dynamic
injection) so that they are not cluttering the Provider code. Maybe for
example we can keep them in the "compat" provider and inject them
dynamically on Airflow 2? Such code would not be part of Airflow 3 core and
we could easily remove the initialization code doing the injection in
providers when we decide to stop supporting Airflow 2 in the providers.
That also has the nice side effect that we could possibly raise an error
in Airflow 3 (with very clear description of how to fix it) when someone
sets a string containing "{{" to "formerly templated Airflow 2 field". I
imagine there are many thousands of examples of using a number of community
operators and templated_field in the wild and they will immediately stop
working in Airflow 3, but with such a clear error message we will be able
to guide our users to convert it. Maybe even we could keep such a list of
"old templated fields" in the compat provider forever for that purpose -
that would be a really nice thing for especially new users who will start
Airflow 3 adventure but will have old examples (those examples will be
there and used for many, many years).
J.
On Tue, Jul 16, 2024 at 9:59 AM Ephraim Anierobi <ephraimanier...@gmail.com>
wrote:
> Good proposal. Big +1. I once had trouble with the trailing space issue and
> spent hours debugging it.
>
> Provided the migration tool can rewrite the DAGs, I support it.
>
> On Tue, 16 Jul 2024 at 08:42, Tzu-ping Chung <t...@astronomer.io.invalid>
> wrote:
>
> > Good idea. Personally I think changing the import might be easier to
> read,
> > but having a DAG policy is better if the goal is to rewrite DAG files as
> > little as possible.
> >
> > Not sure if that’s a goal I’d agree with, but that’s not my decision to
> > make 😛
> >
> > I’ll find a way to add it to the document.
> >
> >
> >
> > > On Jul 16, 2024, at 15:07, Abhishek Bhakat
> > <abhishek.bha...@astronomer.io.INVALID> wrote:
> > >
> > > Seems nice. For the 2nd proposed change, I'm thinking another method
> for
> > > compat may be a dag_policy that drop-in replaces the params of any
> > Operator.
> > >
> > > Avi
> >
> >
> > ---------------------------------------------------------------------
> > To unsubscribe, e-mail: dev-unsubscr...@airflow.apache.org
> > For additional commands, e-mail: dev-h...@airflow.apache.org
> >
> >
>
