GitHub user zyratlo added a comment to the discussion: Mounting datasets and models: three open questions
I prefer keeping `DatasetFileDocument` as a fallback. My main concern is that a user shouldn't have to know how Texera was deployed to write a workflow that works. Right now mounting is off by default, so if we move UDFs or native operators over to it, the same workflow could run on one deployment and fail on another. Ideally users wouldn't have to pick between the two at all. Could the library use the mount when one exists and fall back to downloading when it doesn't? Then existing workflows keep working, and we can decide on deprecation once mounting is on everywhere. The same idea applies to native operators. Reading a folder or a set of files matching a pattern is definitely needed, and I don't think it should only work on clusters that have mounting enabled. If listing files can be done without mounting, I'd build the user-facing part on that first and treat mounting as a faster way to read the files when it's available. For which operators come first, I think extending our current scan operators is the best direction. Would we enforce or check the user-defined patterns, or allow them to specify whatever they want? On question 3, one thing I'd like to understand as a user: if someone loses access to a dataset, how long until they actually can't read it anymore? I understand the proxy checks access on every request, but with the disk cache, files that were already read could still be readable afterward. Mounts living until the pod is deleted seems to make that window longer than it needs to be. Is there a reason mounts can't end when the execution ends? GitHub link: https://github.com/apache/texera/discussions/9048#discussioncomment-19083317 ---- This is an automatically sent email for [email protected]. To unsubscribe, please send an email to: [email protected]
