I considered breakages that are likely to happen for a third-party operator.
Without actually looking into them, from experience, they generally need to do
one of the following two things (or both) in the task:
1. Access the templated values in execute()
2. Access the arguments containing a template in __init__()
For the first case, the operator does not actually interact with the
templates—only Airflow is (logic in BaseOperator)—and the custom operator class
only cares about what fields are templated, and what the rendered values are.
I’ve decided therefore to make BaseOperator in Airflow 3 still accept
template_fields with a warning. BaseOperator will simply copy the value to the
canonical _template_fields. This should be pretty straightforward, probably
less than 10 lines of code in the meta class. This change would allow simple
custom operators that only do real logic in execute() to properly function at
runtime entirely changed.
The second case is more complicated. Although we have always said doing logic
in __init__ is an anti-pattern, I am sure there are still plenty of users doing
this in the wild. I propose adding a pre-processor in the compat module, to
provide a new style argument interface for an operator defined in the old
style. It would work like this:
from airflow.templates import StringTemplate
from airflow_compat import new_template_interface
from third_party.operators import MyUnpopularOperator
MyUnpopularOperator = new_template_interface(MyUnpopularOperator)
my_task = MyUnpopularOperator(
task_id="my_task",
value=StringTemplate("{{ ti.run_id }}"),
)
The transformed class would try to detect template values in arguments, and
convert them back to the Airflow 2 style. In fact, I think first-class
providers can use this to convert many simple operators trivially as well,
instead of needing a more educated rewrite.
This would still require the DAG author to do something to make a library
usable, but should be sufficient for a lot of the cases. As I mentioned
previously, we should draw a line in the sand somewhere, and I feel this would
eliminate dependencies enough for a lot of users (only need to change DAG code
without needing to wait for third-party library authors), this is a beneficial
enough change to tilt the balance.
I will work on including this and other migration comments made in previous
messages into the AIP document.
TP
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
