jamescnowell commented on issue #31183:
URL: https://github.com/apache/airflow/issues/31183#issuecomment-1577376322

   With regards to SparkKubernetesSensor, the Operator now  remains Running for 
the duration of the Spark job, which (I think) means some DAG topologies will 
no longer work as expected. For example
   ```
                   / ----> some other thing \
   Spark Job                                          --> Final Thing
                   \ ----> sensor ----------- /
   ```
   Will no longer run "some other thing" while the Spark Job is running, but 
instead after the job is completed and the DAG will need to be re-arranged.
   
   This will mean either:
   1. People will need to update their DAGs when they notice a runtime change or
   2. People will not notice the change, but will be running a useless 
SparkKubernetesSensor
   
   This is why I would propose making this a separate operator, or behind 
behavioral flags on the existing operator to maintain the existing behavior 
while documenting an upgrade path.


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: [email protected]

For queries about this service, please contact Infrastructure at:
[email protected]

Reply via email to