On Thu, Mar 13, 2025 at 8:35 AM fidel e. <[email protected]> wrote:
> Thank you for your suggestions, > > Hi Brooke, > Yeah, that's my guess as well, that there will always be a copy involved. > For large files, I also don't want to copy everything at one go, I need to > split it in chunks however my C++ code expects it to be done in synchronous > manner but the FileReader API from JS is async. > > Hi Sam, > Could you elaborate more on this, > "Using a JS-backed filesystem, rather than a memory-backed filesystem. I > believe this is possible in both WASMFS and the default FS." ? > Are you aware of the new WASMFS filesystem vs the current default filesystem? See https://emscripten.org/docs/api_reference/Filesystem-API.html#new-file-system-wasmfs for more on that. On both filesystems the default is that files live in linear memory. If the API you are dealing with supports BYOB then you can read your file directly into a linear memory file. However, that only really works if you are reading the whole file into memory at once. Since you say you want to read individual chunks at a time I think you would need to implement a custom filesystem in JS. The way you do this looks a little different for WASMFS vs the current filesystem but it should be possible in both cases. The fact that your file API is async is doing to be an issue. The JS file operations will need to happen on the main thread and they will need to be async so you have a few options for how to deal with that: 1. Mandate that all user code lives on a background thread (which can block) and then proxy all the async work to the main thread 2. Use ASYNCIFY or JSPI to suspend the wasm execution on the main thread while the FS operation is underway. > > > > > > > On Thursday, March 13, 2025 at 1:18:32 AM UTC+8 [email protected] wrote: > >> Are you actually using WASMFS? Or are you using the default emscripten >> filesystem? >> >> If the you want to avoid copies I think you have two choices: >> >> 1. Using a JS-backed filesystem, rather than a memory-backed filesystem. >> I believe this is possible in both WASMFS and the default FS. >> 2. (As Booke suggested) Pre-allocate space using `malloc` and then pass >> the wasm memory buffer (with offset) to the FileReader API. This technique >> is called BYOB (bring your own buffer). I don't if the FileReader API >> supports this. If it doesn't, you can't use this option. >> >> cheers, >> sam >> >> On Tue, Mar 11, 2025 at 10:48 PM fidel e. <[email protected]> wrote: >> >>> Hello, >>> >>> Just wondering if there is a way to access a file from user's local >>> system (accessed via JavaScript) without copying it to the wasm file sytem? >>> >>> My initial plan was to use the FileReader API from JS then on the >>> 'onload' event, retrieve the byteOffset from the array buffer and pass that >>> wasm. But it seems that the array where the file was loaded is located on >>> the RAM. My understanding is we can only pass the address (byteOffset) if >>> the array is created on the heap via malloc. >>> >>> Any ideas? >>> >>> Thank you very much. >>> >>> -- >>> >> You received this message because you are subscribed to the Google Groups >>> "emscripten-discuss" group. >>> To unsubscribe from this group and stop receiving emails from it, send >>> an email to [email protected]. >>> To view this discussion visit >>> https://groups.google.com/d/msgid/emscripten-discuss/6a199ee0-52b7-4d3e-bd49-e55a89771f7dn%40googlegroups.com >>> <https://groups.google.com/d/msgid/emscripten-discuss/6a199ee0-52b7-4d3e-bd49-e55a89771f7dn%40googlegroups.com?utm_medium=email&utm_source=footer> >>> . >>> >> -- > You received this message because you are subscribed to the Google Groups > "emscripten-discuss" group. > To unsubscribe from this group and stop receiving emails from it, send an > email to [email protected]. > To view this discussion visit > https://groups.google.com/d/msgid/emscripten-discuss/17904474-d567-43ef-b662-6f13b3742dben%40googlegroups.com > <https://groups.google.com/d/msgid/emscripten-discuss/17904474-d567-43ef-b662-6f13b3742dben%40googlegroups.com?utm_medium=email&utm_source=footer> > . > -- You received this message because you are subscribed to the Google Groups "emscripten-discuss" group. To unsubscribe from this group and stop receiving emails from it, send an email to [email protected]. To view this discussion visit https://groups.google.com/d/msgid/emscripten-discuss/CAL_va29Y7AjkDtx1qsNBxDVS%2BS5iLFhzBFjhYhcM-Ph8BKoaWA%40mail.gmail.com.
