Thanks for the great writeup! > WasmFS will notify jsdos layer that some file is changed (write/delete etc), so I can schedule update of *second* archive in cloud.
Once the core WasmFS system is stable and documented, I hope it will be straightforward to create a custom virtual backend (i.e. a backend that forwards work to another backend) to fire notifications on file modifications. > At the end of extraction you will consume the same amount of memory as the original game size (e.g. 600Mb for CDROM games). ***Unacceptable for mobiles***. This should be possible to fix with WasmFS by streaming files into WasmFS persistent backends like the OPFS backend or a future IndexedDB backend. These backends will never store entire files in memory, unlike IDBFS in the current system. That streaming should be able to use the browser's built-in decompression schemes as well. > I can switch from zip archives to any "bundle/asset" file format* introduced by WasmFS* I don't think we have plans to introduce a new data bundle format native to WasmFS, but you can implement your own however you'd like. > Initially I can download some metafile that contain information about the whole FS. When the dosbox tries to access some file, I will... This sounds like it could be implemented as a read-only backend that fetches data from the network combined with a layering virtual backend that reads missing data from the network backend and writes modifications to a second backend. Overall your planned changes sound like a good match for WasmFS. The core WasmFS backend API is still in flux, but once it is stabilized, we could certainly use help documenting it, providing utilities to make it easy to write new backends from C, C++, or JS, and writing general-purpose virtual backends for caching, layering, notifying on changes, etc. On Wed, Jul 27, 2022 at 10:17 AM Александр Гурьянов <[email protected]> wrote: > I know that WasmFS is in active development, and I want to switch jsdos to > WasmFS. > So, I want to share my use case and if needed help with development. > > jsdos is based on dosbox and it uses POSIX syscalls to access the file > system. > > *dosbox* has the following scenarios accessing files: > * directly accessing files in FS (read/write) > * mounting iso/cue images (CDROM support) (only read) > > *jsdos layer* has additional features: > * storing FS changes in cloud or idbs > * FS compression (currently I use zip) > > jsdos works with zip archives to populate the filesystem. The emulated > program has 2 zip archives: > ***first*** - is a initial game file system > ***second*** - contains changed files. > Before dosbox actually starts, these two "FS archives" are merged into > actual FS. > > Merge implementation is very simple: > 1. Extract *first* archive ***and store modification time of files in > hashtable***. > 2. Extract *second* archive and *overwrite* files > > When jsdos is exited (closed by user) it iterates over FS and checks if > some file have newer > modification time. If so, this file is added to brand new ***second*** > archive. > > The ***second*** archive can be stored in idbfs, or in cloud. > > *This solution has practical problems:* > *1*. zip is slow when extracting files. Not dramatically, but it will be > much better to use native decoder of the browser (gzip/brotli). > *2*. I can say zip compression rate ***is good enough*** for most games. > But you* can't extract *big games on mobiles. It results in OOM. To > extract you need to download the whole archive (*1st copy*), put it into > WASM memory (*2st copy*) and keep it while it is extracting. > *3*. At the end of extraction you will consume the same amount of memory > as the original game size (e.g. 600Mb for CDROM games). ***Unacceptable > for mobiles***. > *4*. There is no option in emscripten FS to notify jsdos when some file > is changed, because of that storing changes is triggered manually when the > user closes the jsdos window. BUT, most users just ***closes the browser > tab and lost all progress.*** > *5*. The ***second*** archive can't handle deletion of files. If some > file is deleted it will be there on the next run. It's because there is no > marker in the zip to signal that some file needs to be deleted. However, > for jsdos this happens very rarely and can be ignored. > > *WasmFS ideas:* > *1*. WasmFS will notify jsdos layer that some file is changed > (write/delete etc), so I can schedule update of *second* archive in cloud. > *2*. I can switch from zip archives to any "bundle/asset" file format* > introduced by WasmFS*. Initially I can download some metafile that > contain information about the whole FS. When the dosbox tries to access > some file, I will: > > * stop execution with asyncify > * lookup in meta which FS block need to be downloaded > * download it (with progress indication) > * mount it > * resume execution > > I imagine that we can use gzip/brotli *browser decoder* here. > I think this flow is more or less generic and can be implemented on the > WasmFS side, so from jsdos side I only need to build an asyncify white list. > > *3*. Replacement for iso/cue. I can fix dosbox to understand WasmFS file > format form (2) to download only needed blocks. > > > *With switching to WasmFS I want to achieve three key features:* > * Minimize download size of full-CD games/programs (like RedAlert, Full > Throttle, Win95) > * Minimize memory consumption to work on mobiles > * Effectively store changes in cloud > > If you have any question, or need my help, feel free to ask. > > > > > -- > You received this message because you are subscribed to the Google Groups > "emscripten-discuss" group. > To unsubscribe from this group and stop receiving emails from it, send an > email to [email protected]. > To view this discussion on the web visit > https://groups.google.com/d/msgid/emscripten-discuss/CAKOm%3DVHwC_W%2BkScuLm0U0m4UF4vCJ6ZUzF7D1F4AJZosrKx1kQ%40mail.gmail.com > <https://groups.google.com/d/msgid/emscripten-discuss/CAKOm%3DVHwC_W%2BkScuLm0U0m4UF4vCJ6ZUzF7D1F4AJZosrKx1kQ%40mail.gmail.com?utm_medium=email&utm_source=footer> > . > -- You received this message because you are subscribed to the Google Groups "emscripten-discuss" group. To unsubscribe from this group and stop receiving emails from it, send an email to [email protected]. To view this discussion on the web visit https://groups.google.com/d/msgid/emscripten-discuss/CAJZD_EVm7x4v%3Dch%3Dzeg_7YOeEWNBQ8UX9HJ-0JeAm_sePuMmMA%40mail.gmail.com.
