+1 for this change. Also +1 for airflow_compat. I think it could be very helpful especially for users with a lot of custom operators, and significantly reduces the migration efforts or makes it smoother.
On Thu, Jul 18, 2024 at 4:20 PM Vincent Beck <vincb...@apache.org> wrote: > +1 here as well. Keeping these template_fields in sync with the parameters > has always been a challenge. Happy to no longer check that when doing PR > reviews. > > On 2024/07/18 20:14:13 "Scheffler Jens (XC-AS/EAE-ADA-T)" wrote: > > Bis +1 from me. The existing Jinja Syntax was a pitfall for all firdt > time DAG developers and there are many hard-to-read examples neede for > medium conplex things like a dict with templated fields some comments left > in the doc. Looking forward for an AIP > > > > Sent from Outlook for iOS<https://aka.ms/o0ukef> > > ________________________________ > > From: Jarek Potiuk <ja...@potiuk.com> > > Sent: Tuesday, July 16, 2024 12:52:05 PM > > To: dev@airflow.apache.org <dev@airflow.apache.org> > > Subject: Re: Operator Templating in Airflow 3 > > > > I am all for the change - it's been a source of confusion. > > > > And yes, automatically rewriting the DAGs and possibly even detection > with > > error explaining what to do (on Airflow 3) if template string is passed > to > > "old" template_field - are really a prerequisite for that (and should be > > quite possible and fairly reliable, I think). > > > > Also - we should find a good approach for the community (and possibly > > 3rd-party) providers that might be run potentially with both: Airflow 2 > and > > Airflow 3. Currently we have 1252 usages of template_fields in the > > community providers and if we release providers without them, they will > > essentially stop working for Airflow 2 (which is fine at some point of > > time, but likely not immediately - even after releasing Airflow 3). > > > > My first thought is that we should keep template_fields, and template_ext > > for a while - until we drop Airflow 2 for new provider releases. Possibly > > we should move them aside (and keep Airflow 2 compatibility via dynamic > > injection) so that they are not cluttering the Provider code. Maybe for > > example we can keep them in the "compat" provider and inject them > > dynamically on Airflow 2? Such code would not be part of Airflow 3 core > and > > we could easily remove the initialization code doing the injection in > > providers when we decide to stop supporting Airflow 2 in the providers. > > > > That also has the nice side effect that we could possibly raise an error > > in Airflow 3 (with very clear description of how to fix it) when someone > > sets a string containing "{{" to "formerly templated Airflow 2 field". I > > imagine there are many thousands of examples of using a number of > community > > operators and templated_field in the wild and they will immediately stop > > working in Airflow 3, but with such a clear error message we will be able > > to guide our users to convert it. Maybe even we could keep such a list > of > > "old templated fields" in the compat provider forever for that purpose - > > that would be a really nice thing for especially new users who will start > > Airflow 3 adventure but will have old examples (those examples will be > > there and used for many, many years). > > > > J. > > > > > > On Tue, Jul 16, 2024 at 9:59 AM Ephraim Anierobi < > ephraimanier...@gmail.com> > > wrote: > > > > > Good proposal. Big +1. I once had trouble with the trailing space > issue and > > > spent hours debugging it. > > > > > > Provided the migration tool can rewrite the DAGs, I support it. > > > > > > On Tue, 16 Jul 2024 at 08:42, Tzu-ping Chung <t...@astronomer.io.invalid > > > > > wrote: > > > > > > > Good idea. Personally I think changing the import might be easier to > > > read, > > > > but having a DAG policy is better if the goal is to rewrite DAG > files as > > > > little as possible. > > > > > > > > Not sure if that’s a goal I’d agree with, but that’s not my decision > to > > > > make 😛 > > > > > > > > I’ll find a way to add it to the document. > > > > > > > > > > > > > > > > > On Jul 16, 2024, at 15:07, Abhishek Bhakat > > > > <abhishek.bha...@astronomer.io.INVALID> wrote: > > > > > > > > > > Seems nice. For the 2nd proposed change, I'm thinking another > method > > > for > > > > > compat may be a dag_policy that drop-in replaces the params of any > > > > Operator. > > > > > > > > > > Avi > > > > > > > > > > > > --------------------------------------------------------------------- > > > > To unsubscribe, e-mail: dev-unsubscr...@airflow.apache.org > > > > For additional commands, e-mail: dev-h...@airflow.apache.org > > > > > > > > > > > > > > > --------------------------------------------------------------------- > To unsubscribe, e-mail: dev-unsubscr...@airflow.apache.org > For additional commands, e-mail: dev-h...@airflow.apache.org > >
