+1 here as well. Keeping these template_fields in sync with the parameters has always been a challenge. Happy to no longer check that when doing PR reviews.
On 2024/07/18 20:14:13 "Scheffler Jens (XC-AS/EAE-ADA-T)" wrote: > Bis +1 from me. The existing Jinja Syntax was a pitfall for all firdt time > DAG developers and there are many hard-to-read examples neede for medium > conplex things like a dict with templated fields some comments left in the > doc. Looking forward for an AIP > > Sent from Outlook for iOS<https://aka.ms/o0ukef> > ________________________________ > From: Jarek Potiuk <ja...@potiuk.com> > Sent: Tuesday, July 16, 2024 12:52:05 PM > To: dev@airflow.apache.org <dev@airflow.apache.org> > Subject: Re: Operator Templating in Airflow 3 > > I am all for the change - it's been a source of confusion. > > And yes, automatically rewriting the DAGs and possibly even detection with > error explaining what to do (on Airflow 3) if template string is passed to > "old" template_field - are really a prerequisite for that (and should be > quite possible and fairly reliable, I think). > > Also - we should find a good approach for the community (and possibly > 3rd-party) providers that might be run potentially with both: Airflow 2 and > Airflow 3. Currently we have 1252 usages of template_fields in the > community providers and if we release providers without them, they will > essentially stop working for Airflow 2 (which is fine at some point of > time, but likely not immediately - even after releasing Airflow 3). > > My first thought is that we should keep template_fields, and template_ext > for a while - until we drop Airflow 2 for new provider releases. Possibly > we should move them aside (and keep Airflow 2 compatibility via dynamic > injection) so that they are not cluttering the Provider code. Maybe for > example we can keep them in the "compat" provider and inject them > dynamically on Airflow 2? Such code would not be part of Airflow 3 core and > we could easily remove the initialization code doing the injection in > providers when we decide to stop supporting Airflow 2 in the providers. > > That also has the nice side effect that we could possibly raise an error > in Airflow 3 (with very clear description of how to fix it) when someone > sets a string containing "{{" to "formerly templated Airflow 2 field". I > imagine there are many thousands of examples of using a number of community > operators and templated_field in the wild and they will immediately stop > working in Airflow 3, but with such a clear error message we will be able > to guide our users to convert it. Maybe even we could keep such a list of > "old templated fields" in the compat provider forever for that purpose - > that would be a really nice thing for especially new users who will start > Airflow 3 adventure but will have old examples (those examples will be > there and used for many, many years). > > J. > > > On Tue, Jul 16, 2024 at 9:59 AM Ephraim Anierobi <ephraimanier...@gmail.com> > wrote: > > > Good proposal. Big +1. I once had trouble with the trailing space issue and > > spent hours debugging it. > > > > Provided the migration tool can rewrite the DAGs, I support it. > > > > On Tue, 16 Jul 2024 at 08:42, Tzu-ping Chung <t...@astronomer.io.invalid> > > wrote: > > > > > Good idea. Personally I think changing the import might be easier to > > read, > > > but having a DAG policy is better if the goal is to rewrite DAG files as > > > little as possible. > > > > > > Not sure if that’s a goal I’d agree with, but that’s not my decision to > > > make 😛 > > > > > > I’ll find a way to add it to the document. > > > > > > > > > > > > > On Jul 16, 2024, at 15:07, Abhishek Bhakat > > > <abhishek.bha...@astronomer.io.INVALID> wrote: > > > > > > > > Seems nice. For the 2nd proposed change, I'm thinking another method > > for > > > > compat may be a dag_policy that drop-in replaces the params of any > > > Operator. > > > > > > > > Avi > > > > > > > > > --------------------------------------------------------------------- > > > To unsubscribe, e-mail: dev-unsubscr...@airflow.apache.org > > > For additional commands, e-mail: dev-h...@airflow.apache.org > > > > > > > > > --------------------------------------------------------------------- To unsubscribe, e-mail: dev-unsubscr...@airflow.apache.org For additional commands, e-mail: dev-h...@airflow.apache.org
