syedatifakhtar commented on issue #15588: URL: https://github.com/apache/airflow/issues/15588#issuecomment-1529520511
I apologize for not describing the problem well enough. We saw this issue until Airflow 2.5.1. Like @eladkal called out this is a problem with the way the Databricks operator is implemented and not specific to the Airflow version. Basically in our case we use airflow in a single docker container bundled together with our DAGs due to lack of a good hosting layer on Azure. Every time we do a deployment we noticed that the Databricks operator would just schedule the job again even if there was already one triggered I have since moved on from the team where we had this issue but the way we resolved it was moving to the deferrable operator which we did not want to move to for other reasons. Maybe i would recommend if we document this as an issue with the non-deferrable version of the DatabricksRunNowOperator and the move to using deferrable operator as a solution. I would be happy to open a PR! -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected]
