> On Jul 16, 2024, at 14:38, Bas Harenslak <b...@astronomer.io.INVALID> wrote: > > +1 for this change! I especially had lots of trouble with template files with > templated filenames and this would solve it. > > Few comments/questions: > "improved debuggability when things go wrong": Could you elaborate on how > this proposal improves debuggability?
I was simply referring to the fact that since you need to choose how a field is interpreted explicitly (as a path to file, or a template string), it is easier to understand the error message because you know what logic the argument went through. It’s also possible Airflow may be able to emit better error messages based on the situation since it won’t need to guess whether a value is intended as a template string, a path, or a literal value. I have not thought much into exactly what can be done here, admittedly. > "better discoverability of whether a field is template-able”: > Would be good to mention that this proposal defines template-ability on > argument-level instead of (currently) attribute-level Good idea. The template-ability is still sort of tied to attributes since operators current do not (cannot) distinguish between the two, and I don’t see it valuable breaking the expectation. It is still a good thing to call that out though. > It appears that a code author would still have to read an operator’s source > code to understand which arguments are template-able. How does this proposal > improve discoverability? We should be able to generate operator documentation automatically from the argument types. For example, in the DockerOperator documentation[1], we can add a marker to a templated argument if it accepts a Template subclass. This should be doable with Sphinx. We already mark template fields with a (templated) suffix, as you can see in the current documentation. This is entirely manual, however, and I’ve seen many cases it drifts away from the actual functionality. It’s probably why not many people actually read those documentation. [1]: https://airflow.apache.org/docs/apache-airflow-providers-docker/stable/_api/airflow/providers/docker/operators/docker/index.html > Have you considered implementing template classes for other native types e.g. > BoolTemplate or IntTemplate, or do you intend to use > render_template_as_native_obj for that? Oh, yes, I thought of something a bit different but forgot to put it in, like StringTemplate("...", converter=bool). It is probably better this way since we can render either a bool or a int from either a string or file. I’ll add it to the document. > > Bas > >> On 16 Jul 2024, at 04:09, Tzu-ping Chung <t...@astronomer.io.INVALID >> <mailto:t...@astronomer.io.INVALID>> wrote: >> >> Hi all, >> >> Following previous discussions in Airflow 3 Dev Calls, I’m proposing a new >> templating syntax for Airflow 3 to replace the current field-templating >> behaviour. Link to document is attached below. While this is a big breaking >> change in DAG syntax, I feel the benefits are significant enough to attempt >> for the major version bump. >> >> I want to also use this opportunity to raise a couple of issues related to >> this, and some other proposed changes: >> >> 1. Possibility of a transitional 2.11 to “back-port” some syntaxes backwards >> so users can use them before they upgrade. >> 2. A separate “airflow_compat” package to allow users to minimally change >> the code before upgrading to Airflow 3 to receive the new behaviour, and >> change the DAG syntax later. >> >> The new ideas do not depend on each other; they are means to a similar goal >> approached from different directions. >> >> I’m interested to hearing any thoughts on this. >> >> TP >> >> >> https://docs.google.com/document/d/1R4uSF0MGWifKrWk_pzjDcaYtgwtmJo9q7wg1iFiFKO4/edit?usp=sharing
