tanishqgandhi1908 commented on issue #6494:
URL: https://github.com/apache/texera/issues/6494#issuecomment-5028091968
Thanks! Answering both:
1) Why GeeseFS. Model loaders expect a real local directory and open files
themselves — you can't hand them a byte stream. A model version is also many
files (weights, config, tokenizer, shards). GeeseFS mounts the version's
storage prefix as a local folder, so the loader reads files normally, on demand
(no eager full download or full-disk copy). It's tuned for many small files via
parallel prefetch and mounts standard S3, so it works directly against our
MinIO with no enterprise licensing.
2) Why a separate table for models. Models and datasets share the same
storage shape, so we reuse the storage/versioning engine — but their metadata
and lifecycle diverge. A dedicated `model` table (plus its own `model-{mid}`
repo namespace) keeps those fields out of the dataset table, avoids
`type`-branching everywhere, and lets model access-control/versioning evolve on
its own. We considered one shared table with a type discriminator but rejected
it — it conflates two resources and blocks model-specific schema growth. The
path prefix (`/models/…` vs `/datasets/…`) is what routes to the right table.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: [email protected]
For queries about this service, please contact Infrastructure at:
[email protected]
