GitHub user aicam edited a discussion: Design input: should a Python UDF mount multiple datasets, or just one?
We just added dataset mounting for Python UDFs (#6608, closes #6606): a UDF can pick a versioned dataset and read its files from a local path, streamed on demand via GeeseFS through a JWT-authenticated S3 proxy โ no download step, version-pinned, and the computing-unit pod stays unprivileged. **Today it's one dataset per UDF.** You pick one `dataset@version`, it gets mounted at `/mnt/texera-mounts/<repo>/<commit>`, and your code reads it from `MOUNTED_DATASET_PATH`. Here is the current flow end to end: [Demo Video](https://github.com/user-attachments/assets/15519610-ee71-4bc9-8f98-01c85daf6d67) ## The question Should we keep this to one dataset per UDF, or let a single UDF mount several? A concrete case for multiple: a UDF that loads a model from one dataset **and** reads an evaluation or reference dataset from another. Today that needs two operators or a workaround. **One dataset (what we have)** - Simple picker, a single `MOUNTED_DATASET_PATH`, easy to document - Simplest mount lifecycle to reason about - A UDF that needs two sources cannot do it in one operator **Multiple datasets** - Model + data + config in one UDF, without glue operators - Needs a list picker in the UI, and a way to expose several paths โ named mounts? a `MOUNTED_DATASET_PATHS` map or JSON? - The infrastructure already allows it: the mounter keys each mount by `repo/commit` under the computing unit, so several mounts per pod work today. The open questions are really the UI and how the paths are surfaced to UDF code. If we go with multiple, how would you want the paths exposed to your code โ a JSON map keyed by dataset name, an ordered list, or separate named environment variables? And should there be a sensible upper bound? Context: #6608 ยท #6606 --- _Disclosure: this post was drafted with Claude (Opus 4.8) and reviewed before posting._ GitHub link: https://github.com/apache/texera/discussions/6838 ---- This is an automatically sent email for [email protected]. To unsubscribe, please send an email to: [email protected]
