> For the first case, the operator does not actually interact with the > templates—only Airflow is (logic in BaseOperator)—and the custom operator > class only cares about what fields are templated, and what the rendered > values are. I’ve decided therefore to make BaseOperator in Airflow 3 still > accept template_fields with a warning. BaseOperator will simply copy the > value to the canonical _template_fields. This should be pretty > straightforward, probably less than 10 lines of code in the meta class. > This change would allow simple custom operators that only do real logic in > execute() to properly function at runtime entirely changed. >
I like the idea and likely even we could set either time or the Airflow version (3.4? 3.5?) where this mechanism will stop working (similarly as in Python). That would require Airflow 3 to relax the "strict semver" approach - but after careful consideration and good, defined process around removing deprecations maybe it's worth it - eventually. Also we should IMHO work out the mechanism where you will be able to write Airflow 3 - style template operators, but by importing the "compat" package, such operators will also work in Airflow 2. I think a number of 3rd-party operator creators will want - similarly to our community providers - to have their operators work for both Airflow 2 and Airflow 3 for extended time. We could actually implement it for our providers, paving the way for all others and showing them how to do it (and test it at the same time). Ideally at some point our "community provider" operators will be defined in purely Airflow 3 style, and compat layer would make them work for Airlfow 2 as well. That will also give such 3rd-party operators a way (and confidence they can do it and lots of examples from our approach) how they can incrementally get rid of warnings in their operators, while getting them still working on Airflow 2 - when they prepare for Airlfow 3 migration. > The second case is more complicated. Although we have always said doing > logic in __init__ is an anti-pattern, I am sure there are still plenty of > users doing this in the wild. I propose adding a pre-processor in the > compat module, to provide a new style argument interface for an operator > defined in the old style. It would work like this: > I am not too worried about that. This mostly does not work the way people think it works - they don't realize the templates will be expanded. I would be willing to break such operators in Airflow 3 and have Airflow 2.11 raising warnings if someone does it - serving Airflow 2.11 as "here you can test your operators and fix all the warnings before you migrate). > TP > --------------------------------------------------------------------- > To unsubscribe, e-mail: dev-unsubscr...@airflow.apache.org > For additional commands, e-mail: dev-h...@airflow.apache.org > >
