jamescnowell commented on issue #31183:
URL: https://github.com/apache/airflow/issues/31183#issuecomment-1577376322
With regards to SparkKubernetesSensor, the Operator now remains Running for
the duration of the Spark job, which (I think) means some DAG topologies will
no longer work as expected. For example
```
/ ----> some other thing \
Spark Job --> Final Thing
\ ----> sensor ----------- /
```
Will no longer run "some other thing" while the Spark Job is running, but
instead after the job is completed and the DAG will need to be re-arranged.
This will mean either:
1. People will need to update their DAGs when they notice a runtime change or
2. People will not notice the change, but will be running a useless
SparkKubernetesSensor
This is why I would propose making this a separate operator, or behind
behavioral flags on the existing operator to maintain the existing behavior
while documenting an upgrade path.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: [email protected]
For queries about this service, please contact Infrastructure at:
[email protected]