GitHub user xuang7 edited a comment on the discussion: Supporting Directory-Based Dataset Access in UDFs
Recently, I have been working on integrating brain imaging visualization tools into the platform. These tools typically expect a folder containing a specific set of files, such as a manifest, coordinates, cell metadata, and other supporting files. Since we do not currently have a directory-access API, the current workaround is essentially a file-by-file presigned URL approach at the frontend level. For each file, the viewer has to: 1. call `/api/dataset/presign-download?...` to get a presigned URL 2. fetch the actual file content from MinIO using that URL This is effectively similar to Design Choice 1, except that it is currently happening at the frontend level. It works as a workaround since my use case can lazy-load files on demand, but it may become inefficient when a dataset contains a large number of files. A directory-access API would simplify this significantly and would be very helpful for tools that require the full directory. GitHub link: https://github.com/apache/texera/discussions/4352#discussioncomment-16484470 ---- This is an automatically sent email for [email protected]. To unsubscribe, please send an email to: [email protected]
