potiuk commented on issue #22816:
URL: https://github.com/apache/airflow/issues/22816#issuecomment-1092202617
> Why not?
No. I specifically ask Why? - because when you want to add something you
have to justify it rather than someone should justify why you should not do it.
That's very simple. Any new stuff adds complexity.
More - any new way of interacting with Airflow adds confusion - shoudl I do
it this or that way. So yes by all means you should make plenty of effort to
justify things and show alternatives that you considered and explain why the
current alternatives are not good.
The Python zen says:
> there should be one — and preferably only one — obvious way to do it
We already can compose tasks - sensor and operators, and dependencies
between them via - yes - task dependencies. Airflow DAG is all about it. Seems
that you want to add "yet another" way of composing tasks at a different level.
That immediately "feels" wrong. I am not saying it is wrong for sure - but
it definitely "feels" wrong immediately.
Putting Two Operators where everyone expects one is totally, and utterly
unexpected and alien to Airflow. You can compose Hooks in Python operators -
this is perfectly fine, but composing two Operators (Sensors are just
specialized operators) is just "don't do it" and it raises immediate questions
about lifecycle:
* when pre/post_installs are running, in which sequence would they be run
* what happens if one of the pre_installs fail - will it fail the other too?
* what happens if one of the executes raisees exception? Will the other fail
or continue running or throw exception?
* what is the retry pattern of such compose operators?
* can you compsoe more than two?
* can you compose them in more complex fashion (two failed/one succeeed?)
And suddenly it looks like you want to develop whole complexity of whole
Airflow DAG as a single composed task. Again.
(And please don't even try answering those questions - they ar rethorical
questions at this stage of discussion).
And this is **just** the beginning. You try to add new abstraction, which
(unlike dependencies, tasks, relations in airflow) have completely undefined
and not hashed out lifecycle, properties, behaviours, people are unfamiliar
with it, etc. etc.
when shoudl people use which ? Why they should choose this composition over
the already familiar one - DAG ? etc. etc.
That's why It's YOU you whe have to explore all the alternatives and prove
that what you want to achieve:
a) is needed
b) you cannot do it using current composition method
c) you cannot modify current composition method to fit the cases you want.
The c) is the most difficult amd requires creativitiy - and before you start
justifying new composition, I'd rather try to come up with few creative ways
how current dependencies and tasks could be modified to achieved what you want
(if you cannot achieve it today).
I wholeheartedly agree with Elad that it feels much better to compose
sensors using already existing compositions.
```
op = Operator(trigger_rule='one_success')
[s1, s2 ] >> op3
```
Is immediately understood by anyone who wrote a single DAG in python and
already has plenty of more functionality for example:
```
[s1, s2 ] >> op3
```
will wait for both sensors (and there are many more cases).
So before you dive into composing sensors - please, please, please. Do not
"anchor" on your "solution". Think about the use case you have in hand. Spend
very good deal of time on thinking whether we really miss some functionality
and if we do - how we can achieve it using current relations and tasks without
adding completely new abstraction.
And yes. This is your task to come and discuss those ideas and propose some
brainstorming before we all conclude that
a) indeed the use case is missing that we want to handle
b) we cannot creatively think of any ideas how we can extend the current
approach rather than add completely new abstraction
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: [email protected]
For queries about this service, please contact Infrastructure at:
[email protected]
