Hi Ash, I love this idea, especially about the `teardown`, which is way cleaner than defining a task as the downstream task of all other tasks with trigger_rule as `all_done`.
Thanks, Ping On Fri, Dec 16, 2022 at 8:05 AM Ash Berlin-Taylor <[email protected]> wrote: > Hi everyone, > > I'd like to start a discussion about a new feature we'd like to add to > Airflow we call "setup and tear down tasks" > > > https://cwiki.apache.org/confluence/display/AIRFLOW/AIP-52+Automatic+setup+and+teardown+tasks > > Lets start with a code example, that if you are familiar with > unittest.TestCase I hope will be familiar and you can guess what the > effect would be: > > ``` > from airflow import DAG, task, setup, teardown > > > with DAG(dag_id='test'): > @setup > def create_cluster(): > ... > return cluster_id > > @task > def load(ti): > # Example: > cluster_id = ti.xcom_pull(task_id="create_cluster") > > def summarize(): > ... > > @teardown(on_failure_fail_dagrun=False) > def teardown_cluster(): > ... > cluster_id = ti.xcom_pull(task_id="create_cluster") > > create_cluster() > load() >> summarize() > teardown_cluster() > ``` > > (This has been an itch of mine that I've had since I first started using > Airflow back in 2017!) > > We go in to quite a bit of detail about the semantics and behaviours of > these new task types, but the tl;dr (copied from the AIP doc) > > Add a new syntax for marking tasks as setup/teardown that: > > * Teardown tasks will still run even if the upstream tasks have > otherwise failed > * Teardown tasks failing don't always result in the DagRun being marked > as failed (up to DAG author to choose failure mode) > * Automatically clear setup/teardown tasks when clearing a dependent > task > > I'm not sure when we'll start a vote on this due to the holiday season, > but we'd like to start working on this in January. > > -ash >
