Bis +1 from me. The existing Jinja Syntax was a pitfall for all firdt time DAG developers and there are many hard-to-read examples neede for medium conplex things like a dict with templated fields some comments left in the doc. Looking forward for an AIP
Sent from Outlook for iOS<https://aka.ms/o0ukef> ________________________________ From: Jarek Potiuk <ja...@potiuk.com> Sent: Tuesday, July 16, 2024 12:52:05 PM To: dev@airflow.apache.org <dev@airflow.apache.org> Subject: Re: Operator Templating in Airflow 3 I am all for the change - it's been a source of confusion. And yes, automatically rewriting the DAGs and possibly even detection with error explaining what to do (on Airflow 3) if template string is passed to "old" template_field - are really a prerequisite for that (and should be quite possible and fairly reliable, I think). Also - we should find a good approach for the community (and possibly 3rd-party) providers that might be run potentially with both: Airflow 2 and Airflow 3. Currently we have 1252 usages of template_fields in the community providers and if we release providers without them, they will essentially stop working for Airflow 2 (which is fine at some point of time, but likely not immediately - even after releasing Airflow 3). My first thought is that we should keep template_fields, and template_ext for a while - until we drop Airflow 2 for new provider releases. Possibly we should move them aside (and keep Airflow 2 compatibility via dynamic injection) so that they are not cluttering the Provider code. Maybe for example we can keep them in the "compat" provider and inject them dynamically on Airflow 2? Such code would not be part of Airflow 3 core and we could easily remove the initialization code doing the injection in providers when we decide to stop supporting Airflow 2 in the providers. That also has the nice side effect that we could possibly raise an error in Airflow 3 (with very clear description of how to fix it) when someone sets a string containing "{{" to "formerly templated Airflow 2 field". I imagine there are many thousands of examples of using a number of community operators and templated_field in the wild and they will immediately stop working in Airflow 3, but with such a clear error message we will be able to guide our users to convert it. Maybe even we could keep such a list of "old templated fields" in the compat provider forever for that purpose - that would be a really nice thing for especially new users who will start Airflow 3 adventure but will have old examples (those examples will be there and used for many, many years). J. On Tue, Jul 16, 2024 at 9:59 AM Ephraim Anierobi <ephraimanier...@gmail.com> wrote: > Good proposal. Big +1. I once had trouble with the trailing space issue and > spent hours debugging it. > > Provided the migration tool can rewrite the DAGs, I support it. > > On Tue, 16 Jul 2024 at 08:42, Tzu-ping Chung <t...@astronomer.io.invalid> > wrote: > > > Good idea. Personally I think changing the import might be easier to > read, > > but having a DAG policy is better if the goal is to rewrite DAG files as > > little as possible. > > > > Not sure if that’s a goal I’d agree with, but that’s not my decision to > > make 😛 > > > > I’ll find a way to add it to the document. > > > > > > > > > On Jul 16, 2024, at 15:07, Abhishek Bhakat > > <abhishek.bha...@astronomer.io.INVALID> wrote: > > > > > > Seems nice. For the 2nd proposed change, I'm thinking another method > for > > > compat may be a dag_policy that drop-in replaces the params of any > > Operator. > > > > > > Avi > > > > > > --------------------------------------------------------------------- > > To unsubscribe, e-mail: dev-unsubscr...@airflow.apache.org > > For additional commands, e-mail: dev-h...@airflow.apache.org > > > > >
