dstandish commented on PR #33570:
URL: https://github.com/apache/airflow/pull/33570#issuecomment-1692446901
Cool, so one thing @hussein-awala ... I think in order to ensure consistent
behavior, we must also add something like this to function
`validate_setup_teardown`
```
if task.is_setup:
for down_task in task.downstream_list:
if not down_task.is_teardown and down_task.trigger_rule
not in one_success_rules:
# this is required to ensure consistent clearing
behavior when upstream
raise ValueError(
"Setup tasks must be followed with trigger rule
ALL_SUCCESS or ONE_SUCCESS."
)
```
or perhaps better, do the check when setting upstream / downstream. The
reason is that, what you are doing (and needfully so) is essentially to check
that upstream setups are done and successful before the downstream (which is
not directly connected) may run. BUT, if a user uses a trigger rule such as
`one_failed` following a setup task, then this can create odd results where,
the scenario where the task runs is when the setup fails, but we're clearing it
and running it, and the setup succeeds, so it shouldn't run. It's very odd and
nonsensical, but we should guard against it and for consistency. If it's not
clear I can try and create an example. We could perhaps relax this a bit to
allow other trigger rules for work tasks not inside the scope of the setup.
Other point... on the topic of scope of the setup.... I think your logic
for `get_setups_only` may be incorrect because it gets, i think, all upstream
setups, but note that the scope of the setup can be constrained by adding a
teardown, which means that anything that is not between the setup and its
teardown is not assumed to require that setup. i think the easiest way for you
to obtain just the "relevant" setups is to just use
`get_upstreams_only_setups_and_teardowns` and filter out the teardowns. This
method restricts to just setups and teardowns for which the object task is "in
scope".
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: [email protected]
For queries about this service, please contact Infrastructure at:
[email protected]
