aicam opened a new pull request, #6897:
URL: https://github.com/apache/texera/pull/6897
### What changes were proposed in this PR?
Let a **Python UDF consume mounted datasets** — the final step of the
dataset-mounting feature (#6606), stacked on the per-computing-unit mounting PR
(#6896).
Each UDF binding maps a dataset version to a Python variable that, at
runtime, holds the **local filesystem path** of the mounted dataset, e.g.
`open(f"{A}/file.csv")`.
- **`PythonUDFOpDescV2`** — a list of `(variableName -> dataset)` bindings,
validated as Python identifiers and resolved via `FileResolver`; the resolved
locators flow through `PhysicalOp` and `WorkerConfig`.
- **`DatasetMountManager`** — on the executor, ensures each bound dataset is
mounted on the computing unit and hands its path to the Python worker, which
injects it as a module-level variable (`MOUNTED_DATASETS`).
- **"Mounted dataset variables" property editor** for the Python UDF.
> **Stacked on #6896** — until that merges, the diff here also shows the
#6896 changes.
### Any related issues, documentation, discussions?
Closes #6895 · part of #6606 · builds on #6896.
### How was this PR tested?
- Scala + Python unit tests: `PythonUDFOpDescV2Spec` (binding validation —
Python identifiers, blank rows, empty dataset), `DatasetVariableMappingSpec`,
`DatasetMountManagerSpec` (locator/identity validation),
`PythonWorkflowWorkerStartupConfigSpec`, and `test_executor_manager` /
`test_run_python_worker` (the `MOUNTED_DATASETS` variable injection).
- Validated end-to-end on a single-node minikube: a Python UDF bound a
mounted ~2 GB PyTorch model to a variable and `torch.load`-ed it from the
propagated path with bit-exact output.
### Was this PR authored or co-authored using generative AI tooling?
Generated-by: Claude Opus 4.8
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: [email protected]
For queries about this service, please contact Infrastructure at:
[email protected]
