AlejandroMorgante opened a new pull request, #74386: URL: https://github.com/apache/airflow/pull/74386
Add `DatabricksAgentInvokeOperator` and `DatabricksAgentHook` so Airflow tasks can invoke agents deployed with `DurableAgentServer` on Databricks Apps through `/api/invocations`. The operator supports background submission, normal polling and deferrable waiting. It uses service principal OAuth, preserves the complete invocation response in XCom, and reuses a stable invocation UUID across retries and clears within the same Dag run. Failed invocations fail the task; interrupted invocations return to the Dag for downstream handling. Includes provider metadata, documentation, unit tests and a deterministic agent fixture with a system test covering both normal and deferrable invocation. The fixture requires an existing deployed app and does not require a model endpoint. It uses a single instance with the default in-process Runtime Store, so this test does not cover persistence or recovery across app restarts. Validation: - Complete Databricks unit suite: 1054 passed, 12 skipped in Breeze (including the 55 new hook, operator and trigger tests). - System test against a real Databricks workspace with OAuth using the command below: 1 passed in 31.47 seconds. Both invocations and their output/session assertions passed; the deferrable task paused, completed its trigger and resumed. - Fast static checks and all applicable manual checks passed, including provider-wide mypy. - Databricks documentation build and spell checking passed. The system test can be reproduced from the Airflow checkout after deploying the fixture in `providers/databricks/tests/system/databricks/resources/agent_server` to a Databricks App and waiting until its compute is `ACTIVE` and its app status is `RUNNING`. Grant the caller service principal workspace access and `CAN_USE` on the app. Create a private Airflow connections JSON file at `files/databricks-agent-connections.json` with a `databricks_oauth` connection: workspace URL in `host`, OAuth client ID in `login`, client secret in `password`, and `service_principal_oauth: true` in extras. Create `files/databricks-agent-system-test.env` with the app's base URL and the connection file path as seen inside Breeze (`files/` is mounted at `/files`): ```bash DATABRICKS_AGENT_APP_URL=https://<app-name>.aws.databricksapps.com DATABRICKS_AGENT_CONN_ID=databricks_oauth DATABRICKS_AGENT_CONN_FILE=/files/databricks-agent-connections.json ``` Run the same system-test workflow used in [the GKE operator PR](https://github.com/apache/airflow/pull/72577): ```bash SYSTEM_TESTS_ENV_ID=<unique-id> \ BREEZE_INIT_COMMAND='set -a; . /files/databricks-agent-system-test.env; set +a' \ breeze testing system-tests \ --backend sqlite \ --forward-credentials \ --test-timeout 2400 \ providers/databricks/tests/system/databricks/example_databricks_agent.py \ -q ``` The test verifies normal and deferrable invocation against the deployed app. The app is deployed separately; stop its compute after the run. --- ##### Was generative AI tooling used to co-author this PR? - [X] Yes — Codex (GPT-6) Generated-by: Codex (GPT-6) following [the guidelines](https://github.com/apache/airflow/blob/main/contributing-docs/05_pull_requests.rst#gen-ai-assisted-contributions) -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected]
