On Thu, Nov 7, 2013 at 12:22 PM, Jon Robson <jdlrob...@gmail.com> wrote:
> I'm not sure why shift reloading would make the cache grow since the > same page should load exactly the same modules - if that's not the > case that points at a bug somewhere. > Seems like a bug, to me. > That said since there are a potentially infinite amount of > modules/gadgets, many of which are rarely used that can be loaded in > localStorage it could be a good idea to control this somehow - e.g. > only cache commonly loaded modules . Ori have you thought much about > this approach? I could imagine two approaches: > > 1) only caching modules loaded by startup module > 2) keeping a counter for each module name to count how many times it > loads and when it hits a certain threshold only then the css/js is > cached. For instance there is not much point in caching a bit of JS > that runs on > 3) a mixture of these 2 approaches > It seems like it would also be useful for gadgets/extensions to be able to interact with the resource loader, to effectively say, "reserve X bytes for me". Maybe this is just having a programmatic way for gadgets/extensions to throw out the LRU bytes in the resource loader cache if they try to save something to local storage and fail. Alternatively: a priority scheme, so gadgets can automatically get prioritized access to the limited amount of localStorage. > I was just reminded also of an old talk I went to [1] which had > awesome ideas for maximising storage that might be worth exploring. > Since JavaScript uses UTF-16 for text encoding the idea was > essentially to convert to UTF-8/ASCII to get up to 50% compression > ratio. There were also ideas about bit shifting base64. Worth > exploring and making this even more of a dangerous experiment?!?! :D > I've played around with some of these ideas in the past. I'm not sure it's worth it, in the end. The spec doesn't actually say how the localstorage accounting works in the browser, so even though strings are "nominally" UTF-16, it's not clear that browsers are using "size in UTF-16 encoding" as their internal quota metric. More likely they are using size in whatever internal representation they have. As far as the spec is concerned, the browser can compress the stored data itself to maximize space available to user scripts... or just increase the quota allowance. I've had some conversations about this with the mobile firefox guys. If you wanted to play around with this, though, https://github.com/cscott/nell-colors/blob/master/src/lzw.js contains an implementation of lzw compression optimized for storage in UTF-16 strings. If you really want to maximize storage, you'd be better off using the IndexedDB API, which allows you to directly store byte arrays. Then you could use all the fancy compression algorithms in https://github.com/cscott/compressjs to really squeeze things down. (Or just use the byte arrays to ensure that the script is stored in UTF8.) It's a tradeoff, though. A bandwidth-limited user might tolerate the slowdown of compression, but if you're trying to use the cache to speed things up for a high-bandwidth desktop user it's hard to make compression pay for itself. --scott > > [1] http://lanyrd.com/2012/londonjs-10/srdzy/ > > On Thu, Nov 7, 2013 at 8:38 AM, Chris McMahon <cmcma...@wikimedia.org> > wrote: > > On Thu, Nov 7, 2013 at 8:05 AM, Jon Robson <jdlrob...@gmail.com> wrote: > > > >> From personal experience don't touch cache manifests with a barge > pole... > >> > >> Bear in mind the majority of browsers provide at least 5mb of local > storage > >> and we are talking about caching a few kB at most of minified > >> JavaScript.... > >> On 7 Nov 2013 00:35, "Daniel Friesen" <dan...@nadir-seen-fire.com> > wrote: > >> > > > > What I'm seeing in Chromium/Chrome is that when I invoke VisualEditor one > > time localStorage hits ~3.5MB immediately. > > > > Hit shift-reload a few times in quick succession and it's pretty easy to > > max out localStorage for Chromium at 5.xMB. Default for Chrome seems to > be > > 10MB. > > _______________________________________________ > > Wikitech-l mailing list > > Wikitech-l@lists.wikimedia.org > > https://lists.wikimedia.org/mailman/listinfo/wikitech-l > > > > -- > Jon Robson > http://jonrobson.me.uk > @rakugojon > > _______________________________________________ > Wikitech-l mailing list > Wikitech-l@lists.wikimedia.org > https://lists.wikimedia.org/mailman/listinfo/wikitech-l > -- (http://cscott.net) _______________________________________________ Wikitech-l mailing list Wikitech-l@lists.wikimedia.org https://lists.wikimedia.org/mailman/listinfo/wikitech-l
