kaxil opened a new pull request, #72910: URL: https://github.com/apache/airflow/pull/72910
`SandboxToolset` shipped in #68847 with one backend, `sbx`, which needs the `sbx` binary on the worker host, a Docker login, a one-time `sbx policy init`, and KVM on Linux. An unprivileged container cannot provide the last of those, so **every Kubernetes deployment has been unable to turn the toolset on at all**, and its own docs say to run something else in production without offering one. `ModalSandboxBackend` is that something else. Each sandbox is a [Modal sandbox](https://modal.com/docs/guide/sandbox) provisioned over the API: nothing is installed on the worker, model-written code never executes on the worker host, and Modal reclaims a sandbox at its own `sandbox_timeout` whether or not the worker survives, which is the orphan problem `sbx` leaves you with. ```python from airflow.providers.common.ai.operators.agent import AgentOperator from airflow.providers.common.ai.sandbox import ModalSandboxBackend from airflow.providers.common.ai.toolsets import SandboxToolset AgentOperator( task_id="sandboxed_analyst", prompt="Estimate pi with a Monte Carlo simulation of one million points.", llm_conn_id="pydanticai_default", toolsets=[SandboxToolset(ModalSandboxBackend())], ) ``` Credentials are ambient, as they are for Modal's own CLI: `modal token new`, or `MODAL_TOKEN_ID` and `MODAL_TOKEN_SECRET` on the worker. Nothing is read until the first sandbox is created, so a Dag file that constructs the backend parses without them. ## What it looks like The four tools driven end to end in one sandbox, with a deliberate two second timeout in the middle and the file from before it still readable afterwards, which is the behaviour `sbx` cannot offer:  And the boundary itself: the sandbox comes up with `block_network=True`, carries none of Airflow's environment, cannot resolve a hostname, and is destroyed when the task ends.  ## Design rationale **`read_file` deliberately stays on the base class's shell implementation.** Modal's native read takes no length argument, so honoring a read budget through it means `stat` and then read: the TOCTOU window `base.py` documents avoiding, and no protection at all against anything `stat` reports as zero-length. Measured against `modal==1.5.5`, `read_bytes("/dev/zero")` never returned in 195 seconds while `stat` called that device `size=0`, and a 200 MB regular file came back whole, so there is no server-side ceiling either. The inherited implementation caps inside the guest with `head -c`, which cannot be lied to. `write_file` and `list_directory` do use the native API, where it is a clear win. **A `SandboxSpec` naming `allow_egress_to` is refused unless you pass `egress_enforcement="sni"`.** Modal cannot combine an allowlist with `block_network` at all, and its hostname allowlist is matched against the name in the TLS handshake, which means TLS on 443 only, DNS left open for every hostname, and a host sharing a TLS endpoint with an allowed one still reachable. That last one is not theoretical: with `pypi.org` as the only allowed host, a TLS session opened to `files.pythonhosted.org` while presenting `pypi.org` as the handshake name was allowed through and answered. `block_network=True` on its own maps exactly and takes DNS down with everything else, and it is the default. This mirrors `sbx`, which refuses an allowlist until the Deployment Manager declares the host policy. **A command timeout does not cost you the sandbox.** Modal stops the command server-side, so files written by earlier calls survive and the model can inspect them to work out what went wrong. `sbx` has to destroy the sandbox to be certain a command stopped. **A timeout is a returncode rather than an exception, and not always the same one.** `_ContainerProcess.poll` and `.wait` catch `ExecTimeoutError` internally and set `-1`, carrying the SDK's own note that it should probably raise. The same command at the same budget has also been seen returning `137`, at the same elapsed time and from the same SDK version, so classification uses elapsed time rather than the status alone: a deadline lands at or past its budget while a guest killing itself lands nowhere near it. ## Two changes outside the backend `SandboxExecResult` gains an optional `applied_timeout`, and the toolset reports a replaced sandbox after any failure rather than only after a timeout. Both exist because this backend produces states `sbx` never did: it can shorten a command's deadline to fit what is left of a sandbox's life, and it can lose a sandbox under a command that looks like an ordinary non-zero exit. Without them the model is told a budget it did not have, or is handed a fresh empty sandbox with no explanation. Both are additive and `sbx` is unaffected. ## Tradeoffs and limitations **Work does not survive the run.** The sandbox is created on the model's first tool call and destroyed when the run ends, so an artifact the agent built leaves only through the model's context, which is bounded. If a task has to produce a file, drive a backend directly from a `@task` rather than handing it to an agent. This is a property of the toolset rather than of this backend, and the docs now say so plainly. **A default sandbox cannot install packages**, because the default spec denies all egress including DNS. Either bake what the agent needs into the image with a prepared `modal.Image`, which needs no egress, or allow `pypi.org` and `files.pythonhosted.org` and accept the SNI caveats. Both are documented. **One sandbox spans a whole agent run**, so you pay for the model's thinking time and any human review pause, not only for the seconds your commands run. A default sandbox is about two cents an hour; a data-sized one is about two dollars. **`memory` is a request, not a ceiling.** A sandbox created with `memory=512` allocated 1.5 GiB without complaint, so it schedules the sandbox somewhere with room and does not bound what model-written code can take. **Tags identify the Dag, not the run.** They are fixed when the backend is constructed, and `create` receives no task context, so every sandbox in a mapped task carries the same ones. ## Known issues, not addressed here `durable=True` and HITL regeneration both interact badly with any sandbox backend, `sbx` included: durable replays cached tool results describing a filesystem that no longer exists, and a regeneration hands the agent a fresh empty sandbox while its history insists the files are there. Both are documented as unsupported and neither is enforced. They are pre-existing and tracked separately rather than widened into this change. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected]
