rafael-buttignon opened a new issue, #53904: URL: https://github.com/apache/airflow/issues/53904
### Apache Airflow version Other Airflow 2 version (please specify below) ### If "Other Airflow 2 version" selected, which one? v2.10.4 ### What happened? I'm experiencing an issue with DatabricksSubmitRunOperator in Airflow. When a Databricks job fails very quickly, especially due to a duplicate library installation error (before the notebook starts), the failed or cancelled status is not always correctly reflected in the Airflow DAG. For example, if there are duplicate PyPI libraries, Databricks cancels the job in less than 1 second. In these cases, Airflow often marks the task as successful, even though the job was cancelled or failed in Databricks. On the other hand, if the job takes longer to start (e.g., while the cluster is starting up), the error status is propagated correctly to Airflow. Additional context: I am using deferrable=True and wait_for_termination=True. Tried workarounds like adding delays, but they do not work for errors external to the notebook. The issue only occurs with very fast failures, mainly related to library installation. This behavior can cause inconsistencies in pipeline monitoring, as the DAG may be marked as successful even when the Databricks job actually failed. ### What you think should happen instead? Whenever the Databricks job fails or is cancelled (regardless of execution time or reason, such as library installation errors), Airflow should mark the task as "failed" in the DAG. This ensures the real job status is correctly reflected in monitoring and pipeline dependencies. <img width="856" height="344" alt="Image" src="https://github.com/user-attachments/assets/9e3435d2-6b69-4080-b4b0-59b726322bcd"; /> ### How to reproduce To reproduce, I simply added a problematic library to the job configuration, which causes Databricks to fail immediately (within the first second), before the notebook even starts. This simulates a real user scenario. In this case, Databricks marks the job as failed, but Airflow marks the task as successful. If the process takes longer to execute (for example, if the job cluster takes time to start), Airflow correctly marks the task as failed in the UI. But when the failure is very fast, this does not happen. <img width="342" height="200" alt="Image" src="https://github.com/user-attachments/assets/ab5a054a-8b0f-41f8-a560-8cc25b0f639f"; /> ### Operating System n/a ### Versions of Apache Airflow Providers _No response_ ### Deployment Official Apache Airflow Helm Chart ### Deployment details Chart version: 1.15.0 Airflow Version: 2.10.4 Python 3.10 ### Anything else? _No response_ ### Are you willing to submit PR? - [ ] Yes I am willing to submit a PR! ### Code of Conduct - [x] I agree to follow this project's [Code of Conduct](https://github.com/apache/airflow/blob/main/CODE_OF_CONDUCT.md) -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected]
