Passing post_execute as an argument is somewhat useful for operators that don’t
support assets natively (most of them) but when you want to emit a dynamic
path. For example:
def _send_asset_event(context, result):
# Rendered value!
name = context["task"].output
# Trigger an event against the emitted path.
context["outlet_events"][write_data_outlet].add(Asset(name))
write_data_outlet = AssetAlias("data")
WriteSomeDataOperator(
task_id="write_data",
output="write_data_{{ run_id }}.parquet",
outlets=[write_data_outlet],
post_execute=_send_asset_event,
)
Without the functionality, you’ll have to write a subclass for each operator
you want to do this, which is quite a bit boilerplate.
Arguably this is only needed since we use operators too much. This wouldn’t be
an issue if we rely more on the PythonOperator+hooks approach (like Bolke
discussed at last year’s Airflow Summit), but alas, people don’t like to change
how they do things, and operators are still very popular.
The pre_execute argument *might* also be useful if you want to pre-process some
values. That’s probably a lot less common, so I wouldn’t fret too much if it
goes away. However, since post_execute and pre_execute basically use the same
implementation, just one run right before and one right after execute, they
should probably stay or go together.
I think the ability of overriding pre_execute and post_execute in a subclass
can definitely go away. They are practically useles; you can just put
everything in execute, which always needs to exist in a BaseOperator subclass
anyway.
TP
> On 28 Mar 2025, at 22:12, Kaxil Naik <[email protected]> wrote:
>
> I am in favor of dropping support as they essentially do the same -- and
> setup & teardown is more "native" (first-class UI support)
>
> On Fri, 28 Mar 2025 at 19:41, Kaxil Naik <[email protected]> wrote:
>
>> Hi team,
>>
>> Should we drop support for pre_execute and post_execute for AF 3.0? They
>> are still marked as experimental [1]. They were added [2] in a world
>> without Setup and Teardown tasks.
>>
>> Regards,
>>
>> Kaxil
>>
>>
>> [1]:
>> https://github.com/apache/airflow/blob/7af0319ba16749f4aea78085dfe7823f321d262a/task-sdk/src/airflow/sdk/bases/baseoperator.py#L715-L724
>> [2]: https://github.com/apache/airflow/pull/17576
>>
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
