kaxil opened a new pull request, #73418: URL: https://github.com/apache/airflow/pull/73418
Adds `apache-airflow-providers-modal` with one connection type, `modal`, and one hook, `ModalHook`, which turns that connection into an authenticated `modal.Client`. Nothing else ships in this PR: no operators, no executor. Today the Modal SDK reads credentials from the worker's environment (`MODAL_TOKEN_ID` / `MODAL_TOKEN_SECRET`) or from `~/.modal.toml`. That works on a laptop and stops working as soon as two deployments need different tokens, a token lives in a secrets backend, or a task should run under a narrower token than the worker has. Every Airflow-side consumer of Modal so far (the Common AI sandbox backend in #72910, and a ModalExecutor prototype) has had to fall back to ambient credentials because nothing owned a `modal` connection type. This provider gives those credentials the same home every other integration has, and one place that resolves them. ## Design rationale **Why a provider for a single hook.** The connection type has to live in a provider to be registered, and putting it in `common.ai` would tie every future Modal consumer (operators, an executor) to an AI provider it has no other reason to depend on. `common.ai`'s Modal sandbox backend will take an optional `modal_conn_id` resolved through this hook in a follow-up; the dependency direction is the same one the Anthropic and OpenAI providers already have. **Connection fields are login, password, and one extra.** Login is the token id, password the token secret, and `extra.environment` the Modal environment. Two things one might expect are deliberately absent: *workspace*, because Modal derives it server-side from the token and the SDK has no input for it, and *profile*, because a profile only selects a section of `~/.modal.toml` and has no meaning once a connection supplies the token directly. **Credential precedence.** Both token fields set: `modal.Client.from_credentials(login, password)`. Neither set, or no `modal_default` connection at all: `modal.Client.from_env()`, so a worker that already ran `modal token new` needs no Airflow configuration (`AnthropicHook` treats an empty API key the same way). Exactly one set: `ModalConnectionError`, since falling back there would hide a typo. A missing connection under a non-default id raises as usual. **The client is cached per hook instance and per PID.** `Client.from_credentials` opens its gRPC transport eagerly, so the hook does not rebuild it per call. The Modal SDK documents its recovery of a client inherited across `fork()` as best-effort and recommends a fresh client instead, so the cache is keyed on `os.getpid()` and a hook populated before a fork rebuilds in the child. **`client_kwargs` carries only `client=`.** `Sandbox.create(environment_name=...)` warns as deprecated in modal 1.5.x (a sandbox inherits its app's environment), so the connection's environment applies through `ModalHook.lookup_app` and nowhere else. **`test_connection` does not leave a client behind.** For explicit credentials it calls `modal.Client.verify`, which opens a throwaway client, sends one `ClientHello`, and closes it; repeated form tests in the API server therefore do not accumulate transports. For the ambient path it calls `hello()` on the SDK's shared `from_env` client, which the hook does not own and does not close. All SDK behaviour above was checked against `modal==1.5.5` source and the published reference (https://modal.com/docs/reference/modal.Client, https://modal.com/docs/guide/troubleshooting#connection-issues-in-forked-processes); the floor is `modal>=1.5.0`. One caveat: `Client.verify` and `Client.from_env` are public classmethods that the reference page hides (`mdmd:hidden`), although the Sandbox reference itself documents every `client=None` as "defaults to `Client.from_env()`". If a future SDK removes `verify`, `test_connection` degrades to building and discarding a client; nothing else depends on it. ## Usage ```bash export AIRFLOW_CONN_MODAL_DEFAULT='{"conn_type": "modal", "login": "ak-...", "password": "as-...", "extra": {"environment": "main"}}' ``` ```python hook = ModalHook() app = hook.lookup_app("airflow-tasks", create_if_missing=True) sandbox = modal.Sandbox.create("python", "-c", "print('hi')", app=app, image=image, **hook.client_kwargs) ``` ## Gotchas Modal also supports OAuth credentials (`Client.from_oauth_credentials`). The connection form does not model them yet; the natural shape is a second pair of extra fields, which can land without changing the fields added here. The provider requires Airflow 3.0+, so it is added to the 2.11.1 `remove-providers` list in the compatibility test matrix, as the Anthropic provider was. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected]
