syedatifakhtar commented on issue #15588:
URL: https://github.com/apache/airflow/issues/15588#issuecomment-1529520511

   I apologize for not describing the problem well enough. We saw this issue 
until Airflow 2.5.1.
   
   Like @eladkal called out this is a problem with the way the Databricks 
operator is implemented and not specific to the Airflow version. Basically in 
our case we use airflow in a single docker container bundled together with our 
DAGs due to lack of a good hosting layer on Azure. Every time we do a 
deployment we noticed that the Databricks operator would just schedule the job 
again even if there was already one triggered
   
   I have since moved on from the team where we had this issue but the way we 
resolved it was moving to the deferrable operator which we did not want to move 
to for other reasons.
   
   Maybe i would recommend if we document this as an issue with the 
non-deferrable version of the DatabricksRunNowOperator and the move to using 
deferrable operator as a solution. I would be happy to open a PR!
   


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: [email protected]

For queries about this service, please contact Infrastructure at:
[email protected]

Reply via email to