kaxil opened a new pull request, #73990:
URL: https://github.com/apache/airflow/pull/73990
A file an agent builds in a `SandboxToolset` sandbox is destroyed with the
sandbox when the run ends. The only way anything leaves today is through the
model's context, which is text-only and capped at 50 KiB per command stream and
5 MiB per `read_file`. So "the agent produces an artifact" (a parquet file, a
chart, a cleaned CSV, a trained model) has had no path out, short of the attach
shape where a separate task provisions the sandbox and reads the file
afterwards.
This adds `exports`: name the files the agent will write and where each
should land, and the toolset copies them to object storage when the run ends,
before it destroys the sandbox.
```python
AgentOperator(
task_id="normalize",
prompt="...write staging.csv...",
toolsets=[
SandboxToolset(
SbxSandboxBackend(host_network_policy="deny-all"),
exports={"staging.csv": "s3://staging/{{ run_id }}/staging.csv"},
export_conn_id="aws_default",
),
],
)
```
Destinations are templated when the toolset goes through `AgentOperator`,
the same way `attach_to` is, and anything `ObjectStoragePath` can open works.
The `run_command` tool description tells the model which files will be
collected, so it writes them where they are expected.
## Design rationale
**Why not raise `max_read_bytes` instead.** Reading a file through the
model's tools costs roughly three times its size in worker memory, and the
point is to move the file, not to show it to the model. The copy here never
goes through the model or XCom.
**A new backend method, with a default so no backend has to change.**
`SandboxBackend.export_file(sandbox, path, dest, *, max_bytes)` writes into a
binary stream without holding the file. The default on the base class reads 4
MiB slices through `run_command`, so a third-party backend gets exports for
free. `sbx` overrides it to stream a single `exec` straight into the
destination, and OpenSandbox uses the SDK's ranged download. Only regular files
are exported, and a file whose size changes during the copy is an error, since
something in the sandbox is still writing it.
**What fails the task and what does not.** A promised file that cannot be
exported (missing, a directory, over `max_export_bytes`, changed mid-copy, or
refused by storage) fails the task, and everything that attempt exported is
removed so a consumer that runs regardless of outcome does not find half a set.
The sandbox is destroyed either way. A `destroy` that fails after a good export
only logs the sandbox name, as teardown failures already do. A run that failed
exports nothing.
**Detecting a failed run.** pydantic-ai exits its toolsets through an exit
stack that passes `None` to `__aexit__` whether or not the run raised, so the
toolset reads the exception being handled at exit instead. An exception that
was already being handled when the run entered is ignored, so an agent started
from inside an `except` block still exports.
**Refused together with `attach_to`.** An attached sandbox belongs to the
task that created it, which already reads out whatever the agent left, and that
shape is also the one to use when files must survive a failed run.
## Evidence
On a local `sbx` microVM, exporting a 200 MB random file took about 2.6 s,
and the sha256 on the worker matched the one computed in the guest. Worker peak
RSS was 259 MB for a 20 MB export and 260 MB for a 400 MB one, so the copy is
bounded by chunk size rather than file size.
## Gotchas
- The bytes pass through the worker. Giving the sandbox storage credentials
so it could upload directly would put a credential inside the boundary this
toolset exists to keep.
- The OpenSandbox override is covered by unit tests against the SDK's
interfaces but has not been run against a live server.
- Seeding input files into the sandbox before the run is not part of this PR.
---
* Read the **[Pull Request
Guidelines](https://github.com/apache/airflow/blob/main/contributing-docs/05_pull_requests.rst#pull-request-guidelines)**
for more information. Note: commit author/co-author name and email in commits
become permanently public when merged.
* For fundamental code changes, an Airflow Improvement Proposal
([AIP](https://cwiki.apache.org/confluence/display/AIRFLOW/Airflow+Improvement+Proposals))
is needed.
* When adding dependency, check compliance with the [ASF 3rd Party License
Policy](https://www.apache.org/legal/resolved.html#category-x).
* For significant user-facing changes create newsfragment:
`{pr_number}.significant.rst`, in
[airflow-core/newsfragments](https://github.com/apache/airflow/tree/main/airflow-core/newsfragments).
You can add this file in a follow-up commit after the PR is created so you
know the PR number.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: [email protected]
For queries about this service, please contact Infrastructure at:
[email protected]
