rinzool opened a new issue, #34450: URL: https://github.com/apache/airflow/issues/34450
### Apache Airflow version Other Airflow 2 version (please specify below) ### What happened Hi, I noticed an issue with `DatabricksRunNowDeferrableOperator` and Databricks OAuth connecion. I am using Airflow 2.7.0 and databricks provider 4.5.0 (latest). I created a [Databricks connection](https://airflow.apache.org/docs/apache-airflow-providers-databricks/stable/connections/databricks.html) using [Databricks oauth](https://docs.databricks.com/en/dev-tools/authentication-oauth.html) (so with a usename, password + extra `service_principal_oauth: true`) I ran a DAG with a `DatabricksRunNowDeferrableOperator`. My databricks job is started by airflow without any issue, but the task is immediatly in failure with the following stacktrace: ``` [2023-09-18, 09:33:51 UTC] {taskinstance.py:1720} ERROR - Trigger failed: Traceback (most recent call last): File "/home/airflow/.local/lib/python3.11/site-packages/airflow/jobs/triggerer_job_runner.py", line 527, in cleanup_finished_triggers result = details["task"].result() ^^^^^^^^^^^^^^^^^^^^^^^^ File "/home/airflow/.local/lib/python3.11/site-packages/airflow/jobs/triggerer_job_runner.py", line 599, in run_trigger async for event in trigger.run(): File "/home/airflow/.local/lib/python3.11/site-packages/airflow/providers/databricks/triggers/databricks.py", line 83, in run run_state = await self.hook.a_get_run_state(self.run_id) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/home/airflow/.local/lib/python3.11/site-packages/airflow/providers/databricks/hooks/databricks.py", line 341, in a_get_run_state response = await self._a_do_api_call(GET_RUN_ENDPOINT, json) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/home/airflow/.local/lib/python3.11/site-packages/airflow/providers/databricks/hooks/databricks_base.py", line 632, in _a_do_api_call token = await self._a_get_token() ^^^^^^^^^^^^^^^^^^^^^^^^^ File "/home/airflow/.local/lib/python3.11/site-packages/airflow/providers/databricks/hooks/databricks_base.py", line 540, in _a_get_token return await self._a_get_sp_token(OIDC_TOKEN_SERVICE_URL.format(self.databricks_conn.host)) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/home/airflow/.local/lib/python3.11/site-packages/airflow/providers/databricks/hooks/databricks_base.py", line 260, in _a_get_sp_token async for attempt in self._a_get_retry_object(): File "/home/airflow/.local/lib/python3.11/site-packages/tenacity/_asyncio.py", line 71, in __anext__ do = self.iter(retry_state=self._retry_state) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/home/airflow/.local/lib/python3.11/site-packages/tenacity/__init__.py", line 314, in iter return fut.result() ^^^^^^^^^^^^ File "/usr/local/lib/python3.11/concurrent/futures/_base.py", line 449, in result return self.__get_result() ^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/concurrent/futures/_base.py", line 401, in __get_result raise self._exception File "/home/airflow/.local/lib/python3.11/site-packages/airflow/providers/databricks/hooks/databricks_base.py", line 262, in _a_get_sp_token async with self._session.post( File "/home/airflow/.local/lib/python3.11/site-packages/aiohttp/client.py", line 1141, in __aenter__ self._resp = await self._coro ^^^^^^^^^^^^^^^^ File "/home/airflow/.local/lib/python3.11/site-packages/aiohttp/client.py", line 508, in _request req = self._request_class( ^^^^^^^^^^^^^^^^^^^^ File "/home/airflow/.local/lib/python3.11/site-packages/aiohttp/client_reqrep.py", line 310, in __init__ self.update_auth(auth) File "/home/airflow/.local/lib/python3.11/site-packages/aiohttp/client_reqrep.py", line 495, in update_auth raise TypeError("BasicAuth() tuple is required instead") TypeError: BasicAuth() tuple is required instead [2023-09-18, 09:33:51 UTC] {taskinstance.py:1943} ERROR - Task failed with exception airflow.exceptions.TaskDeferralError: Trigger failure ``` I tried the exact same DAG with `DatabricksRunNowOperator` and I have no errors. It seems like the triggerer has issue to create a connection with OAuth. ### What you think should happen instead The `DatabricksRunNowDeferrableOperator` should work with a databricks connection using OAuth. It should work exactly like the `DatabricksRunNowOperator` ### How to reproduce * Create a databricks service principal and create Client ID and client Secret * Create a databricks connection with those ID/Secret + extra `service_principal_oauth: true` * Create a DAG with a `DatabricksRunNowDeferrableOperator` * Run the DAG and you should see the error ### Operating System Debian GNU/Linux 11 (bullseye) ### Versions of Apache Airflow Providers ``` apache-airflow-providers-amazon==8.5.1 apache-airflow-providers-celery==3.3.2 apache-airflow-providers-cncf-kubernetes==7.4.2 apache-airflow-providers-common-sql==1.7.0 apache-airflow-providers-daskexecutor==1.0.0 apache-airflow-providers-databricks==4.5.0 apache-airflow-providers-docker==3.7.3 apache-airflow-providers-elasticsearch==5.0.0 apache-airflow-providers-ftp==3.5.0 apache-airflow-providers-google==10.6.0 apache-airflow-providers-grpc==3.2.1 apache-airflow-providers-hashicorp==3.4.2 apache-airflow-providers-http==4.5.0 apache-airflow-providers-imap==3.3.0 apache-airflow-providers-microsoft-azure==6.2.4 apache-airflow-providers-mysql==5.2.1 apache-airflow-providers-odbc==4.0.0 apache-airflow-providers-openlineage==1.0.1 apache-airflow-providers-postgres==5.6.0 apache-airflow-providers-redis==3.3.1 apache-airflow-providers-sendgrid==3.2.1 apache-airflow-providers-sftp==4.5.0 apache-airflow-providers-slack==7.3.2 apache-airflow-providers-snowflake==4.4.2 apache-airflow-providers-sqlite==3.4.3 apache-airflow-providers-ssh==3.7.1 ``` ### Deployment Official Apache Airflow Helm Chart ### Deployment details _No response_ ### Anything else _No response_ ### Are you willing to submit PR? - [ ] Yes I am willing to submit a PR! ### Code of Conduct - [X] I agree to follow this project's [Code of Conduct](https://github.com/apache/airflow/blob/main/CODE_OF_CONDUCT.md) -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected]
