Re: [Wikitech-l] Module storage is coming
On Tue, Dec 3, 2013 at 10:02 PM, Roan Kattouw roan.katt...@gmail.comwrote: * The drop occurred because ResourceLoaderLanguageDataModule had a bug in its mtime computation, causing it to recache all the time; module storage greatly dampened the impact of that bug. This is true, and it was a confounding factor in the analysis. So our initial results were bupkes. Aaron and I decided to repeat the experiment, so we re-ran it for a ten-day period between January 6th and January 16th. The results are more modest but still significant: module storage reduces load time overall by about 100ms. This effect is consistent between mobile and desktop. http://meta.wikimedia.org/wiki/File:Module_storage.experiment_2.load_time.geo_mean.by_group.svg It is now re-enabled across Wikimedia wikis. ___ Wikitech-l mailing list Wikitech-l@lists.wikimedia.org https://lists.wikimedia.org/mailman/listinfo/wikitech-l
Re: [Wikitech-l] Module storage is coming
On 12/08/2013 11:55 PM, Tyler Romeo wrote: I'm sure this has already been taken into consideration, but keep in mind that code that is executed using eval() in Javascript is *not* optimized by the V8 compiler like normal script resources are. Are you sure this is still the case? https://code.google.com/p/v8/issues/detail?id=2315 seems to suggest that this was fixed in V8 last year. Gabriel ___ Wikitech-l mailing list Wikitech-l@lists.wikimedia.org https://lists.wikimedia.org/mailman/listinfo/wikitech-l
Re: [Wikitech-l] Module storage is coming
I am somewhat concerned about the implications for JS debugging here. Debugging JS problems with the live sites is already pretty complicated: 1. debug=true won't reproduce some bugs (usually race condition related) 2. Sometimes old code gets cached with the new cache-busting timestamp (due to a race condition between bits and apache at deployment) 3. Sometimes the cache servers cache different code for the same URL, i.e. one cache server will have one version and another cache server will have another 4. In some cases ResourceLoader doesn't change the cache-busting timestamp (if you just remove files or add older files to a module) 5. You always have to face the 5 Minutes of Doom (how frequently ResourceLoader rebuilds the startup manifest) And that's not even mentioning client-side caching. Shouldn't we work on fixing or mitigating some of these issues before we make JS debugging even more complicated? Ryan Kaldari On Sun, Nov 3, 2013 at 5:27 PM, Ori Livneh o...@wikimedia.org wrote: The roll-out of 1.23wmf2 to your favorite Wikimedia wiki will inaugurate the era of ResourceLoader module storage -- an era which will be marked by terrifying and hilarious new bugs, intermittent client-side failures, and generally inexcusable disruptions to user experience. Sounds exciting? Read on! What is it? --- Module storage refers to the use of localStorage in ResourceLoader as an auxiliary to the browser cache. ResourceLoader, remember, is the MediaWiki subsystem that delivers JavaScript and CSS to your browser. One of its primary functions is to minimize the total number of network requests that your browser needs to make in order to render the page, and to make the remaining requests as slim as possible. One of the ways ResourceLoader does this is by making a list of all the different components your browser needs and then transmitting them in bulk. The downside of this approach is that a small update to MediaWiki's code, touching perhaps one or two components, will often force the browser to throw out (and re-retrieve) a bunch of unrelated modules that did not change and did not ostensibly need to be re-retrieved. This is because the browser cache operates on the level of network requests; it is not intelligent enough to decompose a request into its constituitive parts and to cache these parts separately from one another. Starting with the 1.23wmf2 branch, ResourceLoader will check if your browser implements localStorage, a mechanism for storing structure data. If it does, ResourceLoader will use it as an auxiliary cache, capable of versioning individual components. Why? The primary goals are: * Destabalize MediaWiki's front-end code, by coupling it to an inconsistently-implemented and poorly-understood HTML5 API. * Make debugging fun again, by adding another layer of caching to MediaWiki. Yes!! However, as with all new features, unintended side-effects are possible. Specific concerns for module storage include the risk of making the network footprint of page loads smaller, resulting in improved latency. But seriously, -- * Module storage is enabled only if the underlying browser supports localStorage (~89% of browsers in use, according to http://caniuse.com/#feat=namevalue-storage). Users with older browsers will not see a benefit, but they will not suffer a penalty either. * Module storage may or may not improve site performance. We need to test it against a wide range of browsers and platforms to know for certain. If it doesn't improve performance, we'll blame it on you, your poor choice of browsers, and your uncooperative attitude, but then we'll scrap it. How can I test it? -- Glad you asked! Module storage is enabled by default on the beta cluster, and on test test2 wikis. The simplest way you can help is by being alert to to performance regressions and front-end code breakage. If you know how to use your browser's JavaScript console, you can get stats about module storage usage on the current page by checking the value of 'mw.loader.store.stats'. When the code is rolled out across the cluster, it will be disabled by default, but you will be able to opt-in by manually setting a cookie in your browser. I will follow up with the exact details. If module storage proves stable enough for production use, we'll seek to asses its utility by running a controlled study of some kind. If we can demonstrate that module storage leads to a significant improvement in performance, we'll enable by it default. --- Ori Livneh o...@wikimedia.org ___ Wikitech-l mailing list Wikitech-l@lists.wikimedia.org https://lists.wikimedia.org/mailman/listinfo/wikitech-l ___ Wikitech-l mailing list Wikitech-l@lists.wikimedia.org https://lists.wikimedia.org/mailman/listinfo/wikitech-l
Re: [Wikitech-l] Module storage is coming
On Mon, Dec 9, 2013 at 2:58 PM, Ryan Kaldari rkald...@wikimedia.org wrote: I am somewhat concerned about the implications for JS debugging here. Debugging JS problems with the live sites is already pretty complicated: 1. debug=true won't reproduce some bugs (usually race condition related) Yeah, debug mode sucks. I think we need to think it over. 2. Sometimes old code gets cached with the new cache-busting timestamp (due to a race condition between bits and apache at deployment) Fixed in https://gerrit.wikimedia.org/r/#/c/90541/ and https://gerrit.wikimedia.org/r/#/c/90541/. 3. Sometimes the cache servers cache different code for the same URL, i.e. one cache server will have one version and another cache server will have another I haven't seen this happen. 4. In some cases ResourceLoader doesn't change the cache-busting timestamp (if you just remove files or add older files to a module) Timo fixed this in https://gerrit.wikimedia.org/r/#/c/90541/. 5. You always have to face the 5 Minutes of Doom (how frequently ResourceLoader rebuilds the startup manifest) Well, yes, there's that. And that's not even mentioning client-side caching. Shouldn't we work on fixing or mitigating some of these issues before we make JS debugging even more complicated? I don't entirely agree. Modules in localStorage are versioned in the same way URLs are versioned, and to date there no bugs with module storage cache management have been reported, despite the fact that module storage is now used across all projects. ___ Wikitech-l mailing list Wikitech-l@lists.wikimedia.org https://lists.wikimedia.org/mailman/listinfo/wikitech-l
Re: [Wikitech-l] Module storage is coming
On Mon, Dec 9, 2013 at 5:25 PM, Ori Livneh o...@wikimedia.org wrote: On Mon, Dec 9, 2013 at 2:58 PM, Ryan Kaldari rkald...@wikimedia.orgwrote: I am somewhat concerned about the implications for JS debugging here. Debugging JS problems with the live sites is already pretty complicated: 1. debug=true won't reproduce some bugs (usually race condition related) Yeah, debug mode sucks. I think we need to think it over. 2. Sometimes old code gets cached with the new cache-busting timestamp (due to a race condition between bits and apache at deployment) Fixed in https://gerrit.wikimedia.org/r/#/c/90541/ and https://gerrit.wikimedia.org/r/#/c/90541/. One of these should have been https://gerrit.wikimedia.org/r/#/c/94591/ ___ Wikitech-l mailing list Wikitech-l@lists.wikimedia.org https://lists.wikimedia.org/mailman/listinfo/wikitech-l
Re: [Wikitech-l] Module storage is coming
On Mon, Dec 9, 2013 at 2:15 PM, Gabriel Wicke gwi...@wikimedia.org wrote: Are you sure this is still the case? https://code.google.com/p/v8/issues/detail?id=2315 seems to suggest that this was fixed in V8 last year. Not sure if it's related, but looking at the bleeding edge compiler.cc code directly, it explicitly turns off optimizations inside the eval compiler. [0] On Mon, Dec 9, 2013 at 5:08 PM, Daniel Friesen dan...@nadir-seen-fire.com wrote: Also are we even using eval anyways? The module storage stores whole scripts, it should be able to function by inserting inline script// downloaded text/script elements. It depends. Right now we use $.globalEval(), [1] which will make a script element if the Javascript has use strict at the very very beginning. Otherwise it just uses eval(). [2] [0] https://code.google.com/p/v8/source/browse/branches/bleeding_edge/src/compiler.cc#872 [1] https://doc.wikimedia.org/mediawiki-core/master/js/source/mediawiki.html#mw-loader-method-work [2] https://github.com/jquery/jquery/blob/master/src/core.js#L283 *-- * *Tyler Romeo* Stevens Institute of Technology, Class of 2016 Major in Computer Science ___ Wikitech-l mailing list Wikitech-l@lists.wikimedia.org https://lists.wikimedia.org/mailman/listinfo/wikitech-l
Re: [Wikitech-l] Module storage is coming
On Mon, Dec 9, 2013 at 8:33 PM, Tyler Romeo tylerro...@gmail.com wrote: On Mon, Dec 9, 2013 at 2:15 PM, Gabriel Wicke gwi...@wikimedia.org wrote: Are you sure this is still the case? https://code.google.com/p/v8/issues/detail?id=2315 seems to suggest that this was fixed in V8 last year. Not sure if it's related, but looking at the bleeding edge compiler.cc code directly, it explicitly turns off optimizations inside the eval compiler. [0] Interesting! I see the logic in eschewing eval in favor of Function, because the latter does not create a closure. I hadn't thought of that. It's definitely worth a shot. Let's run another short experiment to compare the two. --- Ori Livneh o...@wikimedia.org ___ Wikitech-l mailing list Wikitech-l@lists.wikimedia.org https://lists.wikimedia.org/mailman/listinfo/wikitech-l
Re: [Wikitech-l] Module storage is coming
On 12/09/2013 08:33 PM, Tyler Romeo wrote: On Mon, Dec 9, 2013 at 2:15 PM, Gabriel Wicke gwi...@wikimedia.org wrote: Are you sure this is still the case? https://code.google.com/p/v8/issues/detail?id=2315 seems to suggest that this was fixed in V8 last year. Not sure if it's related, but looking at the bleeding edge compiler.cc code directly, it explicitly turns off optimizations inside the eval compiler. [0] At least in Chromium 31 and Firefox 25 nested scopes *are* optimized: http://jsperf.com/eval-vs-function-fib In Chromium 31 the self-executing eval'ed code is slightly faster than even the native fib. Without the function wrapper the code is not optimized. Firefox 25 seems to optimize all three versions almost equally well, although my data is a bit noisy as a backup is currently running in the background. Gabriel ___ Wikitech-l mailing list Wikitech-l@lists.wikimedia.org https://lists.wikimedia.org/mailman/listinfo/wikitech-l
Re: [Wikitech-l] Module storage is coming
I'm sure this has already been taken into consideration, but keep in mind that code that is executed using eval() in Javascript is *not* optimized by the V8 compiler like normal script resources are. Considering our scripts do not perform much intensive work AFAIK, it should not be an issue, but any module that is cached in localStorage will no longer benefit from V8's compiler optimizations. It may be useful to have an option that disables the use of localStorage for specific modules. *-- * *Tyler Romeo* Stevens Institute of Technology, Class of 2016 Major in Computer Science On Wed, Dec 4, 2013 at 1:02 AM, Roan Kattouw roan.katt...@gmail.com wrote: On Tue, Dec 3, 2013 at 12:30 AM, Ori Livneh o...@wikimedia.org wrote: We'll gradually enable module storage on all Wikimedia wikis over the course of the next week or two. Ori deployed this to the live site earlier today :) . For reference, the original post about module storage is archived at [1]. I tweeted an impressive graph from Ganglia [2] that Ori shared on IRC a little while after the deployment, and consequently my phone is now blowing up as lots of people are retweeting it and replying to it. Turns out lots of people are interested in Wikipedia :) However, while the graph was impressive, there were some caveats: * It was of internal traffic into the Apache backends serving bits, not the Varnish caches * The Varnish traffic (that actually get all the traffic) dropped suddenly but not very sharply, and the drop was insignificant compared to time-of-day and day-of-week variance * The drop occurred because ResourceLoaderLanguageDataModule had a bug in its mtime computation, causing it to recache all the time; module storage greatly dampened the impact of that bug. The bug was identified later that day and fixed by Krinkle [3], in the commit with probably the highest commit-message-to-diff ratio of all time. Although it wasn't really responsible for the huge drop we saw in the graphs, make no mistake, this is awesome. Thanks Ori, for working on this and putting up with my code review slowness and nitpicking :) One interesting response I got on Twitter [4] said we should avoid localStorage in favor of indexedDB and modern async APIs. I suppose we could look into that :) Roan [1] http://lists.wikimedia.org/pipermail/wikitech-l/2013-November/072839.html [2] https://twitter.com/catrope/status/408018210529615872 [3] https://gerrit.wikimedia.org/r/#/c/99010/ [4] https://twitter.com/andreasgal/status/408108587320623104 ___ Wikitech-l mailing list Wikitech-l@lists.wikimedia.org https://lists.wikimedia.org/mailman/listinfo/wikitech-l ___ Wikitech-l mailing list Wikitech-l@lists.wikimedia.org https://lists.wikimedia.org/mailman/listinfo/wikitech-l
Re: [Wikitech-l] Module storage is coming
On Tue, Nov 5, 2013 at 4:56 PM, Ori Livneh o...@wikimedia.org wrote: On Mon, Nov 4, 2013 at 4:50 AM, Mark A. Hershberger m...@nichework.com wrote: Hrm... I should probably go ask them about that. But I'm curious about your perspective and to see if we have any information on the bandwidth available to various users. There is some, but not enough for us to know in advance what sort of impact this will have. But Aaron Hafaker and I are working on it, and we are going to be rigorous about measuring it, and we will report back. We ran a controlled test and found that module storage reduced page load times by 156 ms, on average. Aaron has some data available at https://meta.wikimedia.org/wiki/Research:Module_storage_performance, but we still need to write several sections. The size of the effect is substantially smaller on mobile, for some reason, which is surprising. We hope to make the dataset public soon. 156ms shaved off of 90% of page views is pretty nice. http://perspectives.mvdirona.com/2009/10/31/TheCostOfLatency.aspx is worth reading for context and scale: This conclusion may be surprising -- people notice a half second delay? -- but we had a similar experience at Amazon.com. In A/B tests, we tried delaying the page in increments of 100 milliseconds and found that even very small delays would result in substantial and costly drops in revenue. We'll gradually enable module storage on all Wikimedia wikis over the course of the next week or two. ___ Wikitech-l mailing list Wikitech-l@lists.wikimedia.org https://lists.wikimedia.org/mailman/listinfo/wikitech-l
Re: [Wikitech-l] Module storage is coming
On 03.12.2013, 12:30 Ori wrote: The size of the effect is substantially smaller on mobile, for some reason, which is surprising. Mobile simply has fewer modules;) -- Best regards, Max Semenik ([[User:MaxSem]]) ___ Wikitech-l mailing list Wikitech-l@lists.wikimedia.org https://lists.wikimedia.org/mailman/listinfo/wikitech-l
Re: [Wikitech-l] Module storage is coming
On Tue, Dec 03, 2013 at 12:30:00AM -0800, Ori Livneh wrote: We ran a controlled test and found that module storage reduced page load times by 156 ms, on average. Aaron has some data available at https://meta.wikimedia.org/wiki/Research:Module_storage_performance, but we still need to write several sections. The size of the effect is substantially smaller on mobile, for some reason, which is surprising. We hope to make the dataset public soon. That sounds great, Ori. Nice work, from both of you :) 156ms shaved off of 90% of page views is pretty nice. http://perspectives.mvdirona.com/2009/10/31/TheCostOfLatency.aspx is worth reading for context and scale: This conclusion may be surprising -- people notice a half second delay? -- but we had a similar experience at Amazon.com. In A/B tests, we tried delaying the page in increments of 100 milliseconds and found that even very small delays would result in substantial and costly drops in revenue. I couldn't agree more. It's widely accepted across the industry that bad site performance/latency is detrimental to user engagement (simply put: speed is a feature). It's exciting to see some much-needed good work in this area. Regards, Faidon ___ Wikitech-l mailing list Wikitech-l@lists.wikimedia.org https://lists.wikimedia.org/mailman/listinfo/wikitech-l
Re: [Wikitech-l] Module storage is coming
To be fair last time I checked there was a lot of dead JS and CSS (e.g. [1]) that should not be loaded in the first place for every page. Reducing this should make things even smoother for users. [1] https://bugzilla.wikimedia.org/show_bug.cgi?id=54604 On Tue, Dec 3, 2013 at 7:04 AM, Faidon Liambotis fai...@wikimedia.org wrote: On Tue, Dec 03, 2013 at 12:30:00AM -0800, Ori Livneh wrote: We ran a controlled test and found that module storage reduced page load times by 156 ms, on average. Aaron has some data available at https://meta.wikimedia.org/wiki/Research:Module_storage_performance, but we still need to write several sections. The size of the effect is substantially smaller on mobile, for some reason, which is surprising. We hope to make the dataset public soon. That sounds great, Ori. Nice work, from both of you :) 156ms shaved off of 90% of page views is pretty nice. http://perspectives.mvdirona.com/2009/10/31/TheCostOfLatency.aspx is worth reading for context and scale: This conclusion may be surprising -- people notice a half second delay? -- but we had a similar experience at Amazon.com. In A/B tests, we tried delaying the page in increments of 100 milliseconds and found that even very small delays would result in substantial and costly drops in revenue. I couldn't agree more. It's widely accepted across the industry that bad site performance/latency is detrimental to user engagement (simply put: speed is a feature). It's exciting to see some much-needed good work in this area. Regards, Faidon ___ Wikitech-l mailing list Wikitech-l@lists.wikimedia.org https://lists.wikimedia.org/mailman/listinfo/wikitech-l -- Jon Robson http://jonrobson.me.uk @rakugojon ___ Wikitech-l mailing list Wikitech-l@lists.wikimedia.org https://lists.wikimedia.org/mailman/listinfo/wikitech-l
Re: [Wikitech-l] Module storage is coming
On Tue, Dec 3, 2013 at 12:30 AM, Ori Livneh o...@wikimedia.org wrote: We'll gradually enable module storage on all Wikimedia wikis over the course of the next week or two. Ori deployed this to the live site earlier today :) . For reference, the original post about module storage is archived at [1]. I tweeted an impressive graph from Ganglia [2] that Ori shared on IRC a little while after the deployment, and consequently my phone is now blowing up as lots of people are retweeting it and replying to it. Turns out lots of people are interested in Wikipedia :) However, while the graph was impressive, there were some caveats: * It was of internal traffic into the Apache backends serving bits, not the Varnish caches * The Varnish traffic (that actually get all the traffic) dropped suddenly but not very sharply, and the drop was insignificant compared to time-of-day and day-of-week variance * The drop occurred because ResourceLoaderLanguageDataModule had a bug in its mtime computation, causing it to recache all the time; module storage greatly dampened the impact of that bug. The bug was identified later that day and fixed by Krinkle [3], in the commit with probably the highest commit-message-to-diff ratio of all time. Although it wasn't really responsible for the huge drop we saw in the graphs, make no mistake, this is awesome. Thanks Ori, for working on this and putting up with my code review slowness and nitpicking :) One interesting response I got on Twitter [4] said we should avoid localStorage in favor of indexedDB and modern async APIs. I suppose we could look into that :) Roan [1] http://lists.wikimedia.org/pipermail/wikitech-l/2013-November/072839.html [2] https://twitter.com/catrope/status/408018210529615872 [3] https://gerrit.wikimedia.org/r/#/c/99010/ [4] https://twitter.com/andreasgal/status/408108587320623104 ___ Wikitech-l mailing list Wikitech-l@lists.wikimedia.org https://lists.wikimedia.org/mailman/listinfo/wikitech-l
Re: [Wikitech-l] Module storage is coming
That is a statement, not an explanation. John On Thu, Nov 7, 2013 at 4:05 PM, Jon Robson jdlrob...@gmail.com wrote: From personal experience don't touch cache manifests with a barge pole... Bear in mind the majority of browsers provide at least 5mb of local storage and we are talking about caching a few kB at most of minified JavaScript On 7 Nov 2013 00:35, Daniel Friesen dan...@nadir-seen-fire.com wrote: Cache manifests are extremely inflexible. The HTTP caching we already have is more flexible than cache manifests. So cache manifests won't help make any improvements. ~Daniel Friesen (Dantman, Nadir-Seen-Fire) [http://danielfriesen.name/] On 2013-11-07 12:19 AM, John Erling Blad wrote: Can you explain why you use LocalStorage for this? It seems to me like this is the wrong solution and you should use cache manifests instead. LocalStorage is a quite limited area for _data_ storage and it will create problems if we start wasting that space for _code_ storage. John ___ Wikitech-l mailing list Wikitech-l@lists.wikimedia.org https://lists.wikimedia.org/mailman/listinfo/wikitech-l ___ Wikitech-l mailing list Wikitech-l@lists.wikimedia.org https://lists.wikimedia.org/mailman/listinfo/wikitech-l ___ Wikitech-l mailing list Wikitech-l@lists.wikimedia.org https://lists.wikimedia.org/mailman/listinfo/wikitech-l
Re: [Wikitech-l] Module storage is coming
That is a statement, not an explanation. Please provide a valid explanation why you want to do this. John On Thu, Nov 7, 2013 at 9:35 AM, Daniel Friesen dan...@nadir-seen-fire.com wrote: Cache manifests are extremely inflexible. The HTTP caching we already have is more flexible than cache manifests. So cache manifests won't help make any improvements. ~Daniel Friesen (Dantman, Nadir-Seen-Fire) [http://danielfriesen.name/] On 2013-11-07 12:19 AM, John Erling Blad wrote: Can you explain why you use LocalStorage for this? It seems to me like this is the wrong solution and you should use cache manifests instead. LocalStorage is a quite limited area for _data_ storage and it will create problems if we start wasting that space for _code_ storage. John ___ Wikitech-l mailing list Wikitech-l@lists.wikimedia.org https://lists.wikimedia.org/mailman/listinfo/wikitech-l ___ Wikitech-l mailing list Wikitech-l@lists.wikimedia.org https://lists.wikimedia.org/mailman/listinfo/wikitech-l
Re: [Wikitech-l] Module storage is coming
I'm not interested in writing an entire explanation for you on how cache manifests work and their faults when you could simply do a web search for one of the many existing tutorials. The issues with using cache manifests to try and do this should be perfectly understandable once someone understands how cache manifests work. They are unusable for this technique. And Ori has already explained the advantage of being able to have modules cached separately. You're basically suggesting we try to make orange juice out of apples instead of oranges. ~Daniel Friesen (Dantman, Nadir-Seen-Fire) [http://danielfriesen.name/] On 2013-11-08 12:46 AM, John Erling Blad wrote: That is a statement, not an explanation. Please provide a valid explanation why you want to do this. John On Thu, Nov 7, 2013 at 9:35 AM, Daniel Friesen dan...@nadir-seen-fire.com wrote: Cache manifests are extremely inflexible. The HTTP caching we already have is more flexible than cache manifests. So cache manifests won't help make any improvements. ~Daniel Friesen (Dantman, Nadir-Seen-Fire) [http://danielfriesen.name/] On 2013-11-07 12:19 AM, John Erling Blad wrote: Can you explain why you use LocalStorage for this? It seems to me like this is the wrong solution and you should use cache manifests instead. LocalStorage is a quite limited area for _data_ storage and it will create problems if we start wasting that space for _code_ storage. John ___ Wikitech-l mailing list Wikitech-l@lists.wikimedia.org https://lists.wikimedia.org/mailman/listinfo/wikitech-l ___ Wikitech-l mailing list Wikitech-l@lists.wikimedia.org https://lists.wikimedia.org/mailman/listinfo/wikitech-l ___ Wikitech-l mailing list Wikitech-l@lists.wikimedia.org https://lists.wikimedia.org/mailman/listinfo/wikitech-l
Re: [Wikitech-l] Module storage is coming
Le 08/11/13 10:33, Daniel Friesen a écrit : I'm not interested in writing an entire explanation for you on how cache manifests work and their faults when you could simply do a web search for one of the many existing tutorials. If you could explain to the newbie there like me what a cache manifest here, that would help the discussion. Not everyone has the time to attempt to figure out every technical matters, much less figuring it wrong. So what is a cache manifest? :D -- Antoine hashar Musso ___ Wikitech-l mailing list Wikitech-l@lists.wikimedia.org https://lists.wikimedia.org/mailman/listinfo/wikitech-l
Re: [Wikitech-l] Module storage is coming
On Fri, Nov 8, 2013 at 9:45 AM, Antoine Musso hashar+...@free.fr wrote: So what is a cache manifest? :D tl;dr - Cache manifests are made for offline web apps, and Wikipedia is not an offline web app. Cache manifests are a new HTML5 feature that is specifically made for single page (or, at the very least, few-paged) offline web apps. You add a special attribute to the html tag of all pages in your application. The value of the attribute is a URL to a manifest file (it has its own mime type and everything). In this file it specifies what pages in your application should be explicitly cached. The difference between cache manifests and normal browser caching is that the browser will never update the cache unless the manifest changes. In other words, if it has an offline copy, it will always serve it unless the manifest file changes. This is useful in cases where you have a web app that is entirely front-end, i.e., once you download the HTML files you don't need to do anything else (think something along the lines of a single player game). That way the files will be permanently cached and the user can view the website even if the site itself is offline. Most apps in the Chrome Web Store will use this technique to have their web app stored. There are multiple reasons it is not used here: 1) Wikipedia is not a single-paged app, it is many, many pages, and every page of the app usually links to the manifest. It would be strange to have any Wikipedia article a user visits permanently stored in the user's browser. (Before somebody says well just don't put articles in the manifest, any page that has the manifest attribute is implicitly cached, regardless of if it's in the manifest.) 2) It doesn't solve the actual problem. The problem here is the issue of combining all JS files into one. We combine all the files using RL in order to reduce round-trip time for first-time visitors, but at the same time it increases what has to be downloaded for previous visitors when updates are made. Cache manifests do not get around the round-trip time issue, so it doesn't allow us to split up JS files. And with the JS files still combined, cache manifests don't have a way to partially update modules. So in the end it is completely useless. See the following links for more information: https://en.wikipedia.org/wiki/The_cache_manifest_in_HTML5 http://www.html5rocks.com/en/tutorials/appcache/beginner/ *-- * *Tyler Romeo* Stevens Institute of Technology, Class of 2016 Major in Computer Science ___ Wikitech-l mailing list Wikitech-l@lists.wikimedia.org https://lists.wikimedia.org/mailman/listinfo/wikitech-l
Re: [Wikitech-l] Module storage is coming
.. 3) it is a nightmare http://alistapart.com/article/application-cache-is-a-douchebag is a good read to anyone who is curious to the why. On 8 Nov 2013 07:06, Tyler Romeo tylerro...@gmail.com wrote: On Fri, Nov 8, 2013 at 9:45 AM, Antoine Musso hashar+...@free.fr wrote: So what is a cache manifest? :D tl;dr - Cache manifests are made for offline web apps, and Wikipedia is not an offline web app. Cache manifests are a new HTML5 feature that is specifically made for single page (or, at the very least, few-paged) offline web apps. You add a special attribute to the html tag of all pages in your application. The value of the attribute is a URL to a manifest file (it has its own mime type and everything). In this file it specifies what pages in your application should be explicitly cached. The difference between cache manifests and normal browser caching is that the browser will never update the cache unless the manifest changes. In other words, if it has an offline copy, it will always serve it unless the manifest file changes. This is useful in cases where you have a web app that is entirely front-end, i.e., once you download the HTML files you don't need to do anything else (think something along the lines of a single player game). That way the files will be permanently cached and the user can view the website even if the site itself is offline. Most apps in the Chrome Web Store will use this technique to have their web app stored. There are multiple reasons it is not used here: 1) Wikipedia is not a single-paged app, it is many, many pages, and every page of the app usually links to the manifest. It would be strange to have any Wikipedia article a user visits permanently stored in the user's browser. (Before somebody says well just don't put articles in the manifest, any page that has the manifest attribute is implicitly cached, regardless of if it's in the manifest.) 2) It doesn't solve the actual problem. The problem here is the issue of combining all JS files into one. We combine all the files using RL in order to reduce round-trip time for first-time visitors, but at the same time it increases what has to be downloaded for previous visitors when updates are made. Cache manifests do not get around the round-trip time issue, so it doesn't allow us to split up JS files. And with the JS files still combined, cache manifests don't have a way to partially update modules. So in the end it is completely useless. See the following links for more information: https://en.wikipedia.org/wiki/The_cache_manifest_in_HTML5 http://www.html5rocks.com/en/tutorials/appcache/beginner/ *-- * *Tyler Romeo* Stevens Institute of Technology, Class of 2016 Major in Computer Science ___ Wikitech-l mailing list Wikitech-l@lists.wikimedia.org https://lists.wikimedia.org/mailman/listinfo/wikitech-l ___ Wikitech-l mailing list Wikitech-l@lists.wikimedia.org https://lists.wikimedia.org/mailman/listinfo/wikitech-l
Re: [Wikitech-l] Module storage is coming
On Fri, Nov 8, 2013 at 11:33 AM, Jon Robson jdlrob...@gmail.com wrote: .. 3) it is a nightmare http://alistapart.com/article/application-cache-is-a-douchebag is a good read to anyone who is curious to the why. I wouldn't go so far to say it is a nightmare. The article you linked blows things way out of proportion. In reality cache manifests are just one of those cool new features that people like to use even if it's not a proper solution for their application. The ApplicationCache behaves in a fairly defined manner, as I explained above. It's just an additional cache on top of normal HTTP caching that permanently caches files based on a manifest. From that article, the only true gotcha I would mention is #5, which explains that files not part of the cache manifest will actually not be loaded, even if you're online. That aspect is a little unintuitive, but once you know about it, it's not really a problem. Even more amusing is the second part of the article that attempts to use ApplicationCache for caching Wikipedia, which, like I just said, is exactly *not* what ApplicationCache was meant for. In the end I can understand the reason cache manifests exist: for explicitly offline applications. If your application is not an offline application, then you should not be using cache manifests in the first place, because that's not what it's meant for. *-- * *Tyler Romeo* Stevens Institute of Technology, Class of 2016 Major in Computer Science ___ Wikitech-l mailing list Wikitech-l@lists.wikimedia.org https://lists.wikimedia.org/mailman/listinfo/wikitech-l
Re: [Wikitech-l] Module storage is coming
I've used cache manifests for offline applications. It works reasonably well in that situation. So I wouldn't dismiss it entirely for all purposes. But it's not the right solution here. --scott ___ Wikitech-l mailing list Wikitech-l@lists.wikimedia.org https://lists.wikimedia.org/mailman/listinfo/wikitech-l
Re: [Wikitech-l] Module storage is coming
Can you explain why you use LocalStorage for this? It seems to me like this is the wrong solution and you should use cache manifests instead. LocalStorage is a quite limited area for _data_ storage and it will create problems if we start wasting that space for _code_ storage. John On Mon, Nov 4, 2013 at 2:27 AM, Ori Livneh o...@wikimedia.org wrote: The roll-out of 1.23wmf2 to your favorite Wikimedia wiki will inaugurate the era of ResourceLoader module storage -- an era which will be marked by terrifying and hilarious new bugs, intermittent client-side failures, and generally inexcusable disruptions to user experience. Sounds exciting? Read on! What is it? --- Module storage refers to the use of localStorage in ResourceLoader as an auxiliary to the browser cache. ResourceLoader, remember, is the MediaWiki subsystem that delivers JavaScript and CSS to your browser. One of its primary functions is to minimize the total number of network requests that your browser needs to make in order to render the page, and to make the remaining requests as slim as possible. One of the ways ResourceLoader does this is by making a list of all the different components your browser needs and then transmitting them in bulk. The downside of this approach is that a small update to MediaWiki's code, touching perhaps one or two components, will often force the browser to throw out (and re-retrieve) a bunch of unrelated modules that did not change and did not ostensibly need to be re-retrieved. This is because the browser cache operates on the level of network requests; it is not intelligent enough to decompose a request into its constituitive parts and to cache these parts separately from one another. Starting with the 1.23wmf2 branch, ResourceLoader will check if your browser implements localStorage, a mechanism for storing structure data. If it does, ResourceLoader will use it as an auxiliary cache, capable of versioning individual components. Why? The primary goals are: * Destabalize MediaWiki's front-end code, by coupling it to an inconsistently-implemented and poorly-understood HTML5 API. * Make debugging fun again, by adding another layer of caching to MediaWiki. Yes!! However, as with all new features, unintended side-effects are possible. Specific concerns for module storage include the risk of making the network footprint of page loads smaller, resulting in improved latency. But seriously, -- * Module storage is enabled only if the underlying browser supports localStorage (~89% of browsers in use, according to http://caniuse.com/#feat=namevalue-storage). Users with older browsers will not see a benefit, but they will not suffer a penalty either. * Module storage may or may not improve site performance. We need to test it against a wide range of browsers and platforms to know for certain. If it doesn't improve performance, we'll blame it on you, your poor choice of browsers, and your uncooperative attitude, but then we'll scrap it. How can I test it? -- Glad you asked! Module storage is enabled by default on the beta cluster, and on test test2 wikis. The simplest way you can help is by being alert to to performance regressions and front-end code breakage. If you know how to use your browser's JavaScript console, you can get stats about module storage usage on the current page by checking the value of 'mw.loader.store.stats'. When the code is rolled out across the cluster, it will be disabled by default, but you will be able to opt-in by manually setting a cookie in your browser. I will follow up with the exact details. If module storage proves stable enough for production use, we'll seek to asses its utility by running a controlled study of some kind. If we can demonstrate that module storage leads to a significant improvement in performance, we'll enable by it default. --- Ori Livneh o...@wikimedia.org ___ Wikitech-l mailing list Wikitech-l@lists.wikimedia.org https://lists.wikimedia.org/mailman/listinfo/wikitech-l ___ Wikitech-l mailing list Wikitech-l@lists.wikimedia.org https://lists.wikimedia.org/mailman/listinfo/wikitech-l
Re: [Wikitech-l] Module storage is coming
Cache manifests are extremely inflexible. The HTTP caching we already have is more flexible than cache manifests. So cache manifests won't help make any improvements. ~Daniel Friesen (Dantman, Nadir-Seen-Fire) [http://danielfriesen.name/] On 2013-11-07 12:19 AM, John Erling Blad wrote: Can you explain why you use LocalStorage for this? It seems to me like this is the wrong solution and you should use cache manifests instead. LocalStorage is a quite limited area for _data_ storage and it will create problems if we start wasting that space for _code_ storage. John ___ Wikitech-l mailing list Wikitech-l@lists.wikimedia.org https://lists.wikimedia.org/mailman/listinfo/wikitech-l
Re: [Wikitech-l] Module storage is coming
From personal experience don't touch cache manifests with a barge pole... Bear in mind the majority of browsers provide at least 5mb of local storage and we are talking about caching a few kB at most of minified JavaScript On 7 Nov 2013 00:35, Daniel Friesen dan...@nadir-seen-fire.com wrote: Cache manifests are extremely inflexible. The HTTP caching we already have is more flexible than cache manifests. So cache manifests won't help make any improvements. ~Daniel Friesen (Dantman, Nadir-Seen-Fire) [http://danielfriesen.name/] On 2013-11-07 12:19 AM, John Erling Blad wrote: Can you explain why you use LocalStorage for this? It seems to me like this is the wrong solution and you should use cache manifests instead. LocalStorage is a quite limited area for _data_ storage and it will create problems if we start wasting that space for _code_ storage. John ___ Wikitech-l mailing list Wikitech-l@lists.wikimedia.org https://lists.wikimedia.org/mailman/listinfo/wikitech-l ___ Wikitech-l mailing list Wikitech-l@lists.wikimedia.org https://lists.wikimedia.org/mailman/listinfo/wikitech-l
Re: [Wikitech-l] Module storage is coming
What should I be aware of as a developer? That is, if I'm running a local MW and hacking on a resources (an extension / JavaScript / etc) will things Just Work? Or should I take active steps to disable Module Storage so that I don't inadvertently run the 'old' version of something I'm hacking on? From what I understand, it's currently disabled by default, etc, so I don't have to worry yet. But I guess I'm asking in advance to ensure that it's well documented by the time we actually turn this on by default and developers have to think about it. --scott On Sun, Nov 3, 2013 at 8:27 PM, Ori Livneh o...@wikimedia.org wrote: The roll-out of 1.23wmf2 to your favorite Wikimedia wiki will inaugurate the era of ResourceLoader module storage -- an era which will be marked by terrifying and hilarious new bugs, intermittent client-side failures, and generally inexcusable disruptions to user experience. Sounds exciting? Read on! What is it? --- Module storage refers to the use of localStorage in ResourceLoader as an auxiliary to the browser cache. ResourceLoader, remember, is the MediaWiki subsystem that delivers JavaScript and CSS to your browser. One of its primary functions is to minimize the total number of network requests that your browser needs to make in order to render the page, and to make the remaining requests as slim as possible. One of the ways ResourceLoader does this is by making a list of all the different components your browser needs and then transmitting them in bulk. The downside of this approach is that a small update to MediaWiki's code, touching perhaps one or two components, will often force the browser to throw out (and re-retrieve) a bunch of unrelated modules that did not change and did not ostensibly need to be re-retrieved. This is because the browser cache operates on the level of network requests; it is not intelligent enough to decompose a request into its constituitive parts and to cache these parts separately from one another. Starting with the 1.23wmf2 branch, ResourceLoader will check if your browser implements localStorage, a mechanism for storing structure data. If it does, ResourceLoader will use it as an auxiliary cache, capable of versioning individual components. Why? The primary goals are: * Destabalize MediaWiki's front-end code, by coupling it to an inconsistently-implemented and poorly-understood HTML5 API. * Make debugging fun again, by adding another layer of caching to MediaWiki. Yes!! However, as with all new features, unintended side-effects are possible. Specific concerns for module storage include the risk of making the network footprint of page loads smaller, resulting in improved latency. But seriously, -- * Module storage is enabled only if the underlying browser supports localStorage (~89% of browsers in use, according to http://caniuse.com/#feat=namevalue-storage). Users with older browsers will not see a benefit, but they will not suffer a penalty either. * Module storage may or may not improve site performance. We need to test it against a wide range of browsers and platforms to know for certain. If it doesn't improve performance, we'll blame it on you, your poor choice of browsers, and your uncooperative attitude, but then we'll scrap it. How can I test it? -- Glad you asked! Module storage is enabled by default on the beta cluster, and on test test2 wikis. The simplest way you can help is by being alert to to performance regressions and front-end code breakage. If you know how to use your browser's JavaScript console, you can get stats about module storage usage on the current page by checking the value of 'mw.loader.store.stats'. When the code is rolled out across the cluster, it will be disabled by default, but you will be able to opt-in by manually setting a cookie in your browser. I will follow up with the exact details. If module storage proves stable enough for production use, we'll seek to asses its utility by running a controlled study of some kind. If we can demonstrate that module storage leads to a significant improvement in performance, we'll enable by it default. --- Ori Livneh o...@wikimedia.org ___ Wikitech-l mailing list Wikitech-l@lists.wikimedia.org https://lists.wikimedia.org/mailman/listinfo/wikitech-l -- (http://cscott.net) ___ Wikitech-l mailing list Wikitech-l@lists.wikimedia.org https://lists.wikimedia.org/mailman/listinfo/wikitech-l
Re: [Wikitech-l] Module storage is coming
On Thu, Nov 7, 2013 at 8:05 AM, Jon Robson jdlrob...@gmail.com wrote: From personal experience don't touch cache manifests with a barge pole... Bear in mind the majority of browsers provide at least 5mb of local storage and we are talking about caching a few kB at most of minified JavaScript On 7 Nov 2013 00:35, Daniel Friesen dan...@nadir-seen-fire.com wrote: What I'm seeing in Chromium/Chrome is that when I invoke VisualEditor one time localStorage hits ~3.5MB immediately. Hit shift-reload a few times in quick succession and it's pretty easy to max out localStorage for Chromium at 5.xMB. Default for Chrome seems to be 10MB. ___ Wikitech-l mailing list Wikitech-l@lists.wikimedia.org https://lists.wikimedia.org/mailman/listinfo/wikitech-l
Re: [Wikitech-l] Module storage is coming
I'm not sure why shift reloading would make the cache grow since the same page should load exactly the same modules - if that's not the case that points at a bug somewhere. That said since there are a potentially infinite amount of modules/gadgets, many of which are rarely used that can be loaded in localStorage it could be a good idea to control this somehow - e.g. only cache commonly loaded modules . Ori have you thought much about this approach? I could imagine two approaches: 1) only caching modules loaded by startup module 2) keeping a counter for each module name to count how many times it loads and when it hits a certain threshold only then the css/js is cached. For instance there is not much point in caching a bit of JS that runs on 3) a mixture of these 2 approaches I was just reminded also of an old talk I went to [1] which had awesome ideas for maximising storage that might be worth exploring. Since JavaScript uses UTF-16 for text encoding the idea was essentially to convert to UTF-8/ASCII to get up to 50% compression ratio. There were also ideas about bit shifting base64. Worth exploring and making this even more of a dangerous experiment?!?! :D [1] http://lanyrd.com/2012/londonjs-10/srdzy/ On Thu, Nov 7, 2013 at 8:38 AM, Chris McMahon cmcma...@wikimedia.org wrote: On Thu, Nov 7, 2013 at 8:05 AM, Jon Robson jdlrob...@gmail.com wrote: From personal experience don't touch cache manifests with a barge pole... Bear in mind the majority of browsers provide at least 5mb of local storage and we are talking about caching a few kB at most of minified JavaScript On 7 Nov 2013 00:35, Daniel Friesen dan...@nadir-seen-fire.com wrote: What I'm seeing in Chromium/Chrome is that when I invoke VisualEditor one time localStorage hits ~3.5MB immediately. Hit shift-reload a few times in quick succession and it's pretty easy to max out localStorage for Chromium at 5.xMB. Default for Chrome seems to be 10MB. ___ Wikitech-l mailing list Wikitech-l@lists.wikimedia.org https://lists.wikimedia.org/mailman/listinfo/wikitech-l -- Jon Robson http://jonrobson.me.uk @rakugojon ___ Wikitech-l mailing list Wikitech-l@lists.wikimedia.org https://lists.wikimedia.org/mailman/listinfo/wikitech-l
Re: [Wikitech-l] Module storage is coming
On Thu, Nov 7, 2013 at 12:22 PM, Jon Robson jdlrob...@gmail.com wrote: I'm not sure why shift reloading would make the cache grow since the same page should load exactly the same modules - if that's not the case that points at a bug somewhere. Seems like a bug, to me. That said since there are a potentially infinite amount of modules/gadgets, many of which are rarely used that can be loaded in localStorage it could be a good idea to control this somehow - e.g. only cache commonly loaded modules . Ori have you thought much about this approach? I could imagine two approaches: 1) only caching modules loaded by startup module 2) keeping a counter for each module name to count how many times it loads and when it hits a certain threshold only then the css/js is cached. For instance there is not much point in caching a bit of JS that runs on 3) a mixture of these 2 approaches It seems like it would also be useful for gadgets/extensions to be able to interact with the resource loader, to effectively say, reserve X bytes for me. Maybe this is just having a programmatic way for gadgets/extensions to throw out the LRU bytes in the resource loader cache if they try to save something to local storage and fail. Alternatively: a priority scheme, so gadgets can automatically get prioritized access to the limited amount of localStorage. I was just reminded also of an old talk I went to [1] which had awesome ideas for maximising storage that might be worth exploring. Since JavaScript uses UTF-16 for text encoding the idea was essentially to convert to UTF-8/ASCII to get up to 50% compression ratio. There were also ideas about bit shifting base64. Worth exploring and making this even more of a dangerous experiment?!?! :D I've played around with some of these ideas in the past. I'm not sure it's worth it, in the end. The spec doesn't actually say how the localstorage accounting works in the browser, so even though strings are nominally UTF-16, it's not clear that browsers are using size in UTF-16 encoding as their internal quota metric. More likely they are using size in whatever internal representation they have. As far as the spec is concerned, the browser can compress the stored data itself to maximize space available to user scripts... or just increase the quota allowance. I've had some conversations about this with the mobile firefox guys. If you wanted to play around with this, though, https://github.com/cscott/nell-colors/blob/master/src/lzw.js contains an implementation of lzw compression optimized for storage in UTF-16 strings. If you really want to maximize storage, you'd be better off using the IndexedDB API, which allows you to directly store byte arrays. Then you could use all the fancy compression algorithms in https://github.com/cscott/compressjs to really squeeze things down. (Or just use the byte arrays to ensure that the script is stored in UTF8.) It's a tradeoff, though. A bandwidth-limited user might tolerate the slowdown of compression, but if you're trying to use the cache to speed things up for a high-bandwidth desktop user it's hard to make compression pay for itself. --scott [1] http://lanyrd.com/2012/londonjs-10/srdzy/ On Thu, Nov 7, 2013 at 8:38 AM, Chris McMahon cmcma...@wikimedia.org wrote: On Thu, Nov 7, 2013 at 8:05 AM, Jon Robson jdlrob...@gmail.com wrote: From personal experience don't touch cache manifests with a barge pole... Bear in mind the majority of browsers provide at least 5mb of local storage and we are talking about caching a few kB at most of minified JavaScript On 7 Nov 2013 00:35, Daniel Friesen dan...@nadir-seen-fire.com wrote: What I'm seeing in Chromium/Chrome is that when I invoke VisualEditor one time localStorage hits ~3.5MB immediately. Hit shift-reload a few times in quick succession and it's pretty easy to max out localStorage for Chromium at 5.xMB. Default for Chrome seems to be 10MB. ___ Wikitech-l mailing list Wikitech-l@lists.wikimedia.org https://lists.wikimedia.org/mailman/listinfo/wikitech-l -- Jon Robson http://jonrobson.me.uk @rakugojon ___ Wikitech-l mailing list Wikitech-l@lists.wikimedia.org https://lists.wikimedia.org/mailman/listinfo/wikitech-l -- (http://cscott.net) ___ Wikitech-l mailing list Wikitech-l@lists.wikimedia.org https://lists.wikimedia.org/mailman/listinfo/wikitech-l
Re: [Wikitech-l] Module storage is coming
We may want to make some considerations for multiple wiki on the same origin though. For example a wiki that uses paths for multiple languages: wiki.example.com/en/Foo wiki.example.com/fr/Foo wiki.example.com/de/Foo ... On a setup like this all the wikis will share the same localStorage origin and a number of them will fill up the localStorage with cache entries for each wiki if a user visits them all. Most of these cache entries will likely be duplicates of the contents of other wiki. So we may want to consider a setup where the current localStorage cache stores hashes and then stores the actual module data inside a different localStorage key which is shared among wiki. ~Daniel Friesen (Dantman, Nadir-Seen-Fire) [http://danielfriesen.name/] On 2013-11-07 7:05 AM, Jon Robson wrote: From personal experience don't touch cache manifests with a barge pole... Bear in mind the majority of browsers provide at least 5mb of local storage and we are talking about caching a few kB at most of minified JavaScript ___ Wikitech-l mailing list Wikitech-l@lists.wikimedia.org https://lists.wikimedia.org/mailman/listinfo/wikitech-l
Re: [Wikitech-l] Module storage is coming
On Sun, Nov 3, 2013 at 5:27 PM, Ori Livneh o...@wikimedia.org wrote: How can I test it? -- Glad you asked! Module storage is enabled by default on the beta cluster, and on test test2 wikis. It's also enabled on MediaWiki.org now, the last such wiki before doing performance testing. I figured it'd be OK because it has been running on beta / test / test2 for over a week with no bugs being reported. Please report back if you notice anything good or bad. ___ Wikitech-l mailing list Wikitech-l@lists.wikimedia.org https://lists.wikimedia.org/mailman/listinfo/wikitech-l
Re: [Wikitech-l] Module storage is coming
I found a usability problem with some versions of the webkit/chrome inspector for localStorage due to this, it is being tracked here: https://bugs.webkit.org/show_bug.cgi?id=123750 On Mon, Nov 4, 2013 at 7:20 PM, Isarra Yos zhoris...@gmail.com wrote: On 04/11/13 15:31, Mark A. Hershberger wrote: On 11/04/2013 10:10 AM, Max Semenik wrote: Most of our mobile traffic is genrated by iOS and Android - both of which support LocalStorage, so it will work for them too. People using 2G/GPRS are not necessarily using a mobile device. I've heard from at least one user (hence https://bugzilla.wikimedia.org/55842) that he is using his cell phone as a modem. I suspect this is not a small percentage of users in less developed areas where cell phone towers are plentiful but cables are not. Non-mobile UAs on mobile IPs is what I'm asking about. Mark. ___ Wikitech-l mailing list Wikitech-l@lists.wikimedia.org https://lists.wikimedia.org/mailman/listinfo/wikitech-l Er, that's a fairly standard speed for low-cost wired plans as well in much of the US. I'd actually be happy to have something that fast, but I couldn't afford it. Well, it's the US's version of 'low cost', anyway. ___ Wikitech-l mailing list Wikitech-l@lists.wikimedia.org https://lists.wikimedia.org/mailman/listinfo/wikitech-l ___ Wikitech-l mailing list Wikitech-l@lists.wikimedia.org https://lists.wikimedia.org/mailman/listinfo/wikitech-l
Re: [Wikitech-l] Module storage is coming
On Mon, Nov 4, 2013 at 4:50 AM, Mark A. Hershberger m...@nichework.com wrote:
Hrm... I should probably go ask them about that. But I'm curious about
your perspective and to see if we have any information on the bandwidth
available to various users.
There is some, but not enough for us to know in advance what sort of
impact this will have. But Aaron Hafaker and I are working on it, and
we are going to be rigorous about measuring it, and we will report
back.
On Mon, Nov 4, 2013 at 9:03 AM, Brad Jorsch (Anomie)
bjor...@wikimedia.org wrote:
On Mon, Nov 4, 2013 at 11:13 AM, Jon Robson jdlrob...@gmail.com wrote:
I'm looking forward to seeing how this plays out. The only downside I
can so far see is that the amount of browser storage available varies
drastically [1] and I wonder whether this will cause upsets for those
browsers with extremely strict limits.
Or, for that matter, if it will fill up the allowed storage so user
scripts and gadgets can't make effective use of it.
You can find out the current byte size of the module store by
evaluating mw.loader.inspect('store') in a JavaScript console on the
wikis on which it is enabled (test, test2, beta cluster, and twn).
testwiki's JavaScript payload is usually a superset of the modules
deployed on various wikis, and the size of a fully warm module cache
is 710 kB. This should leave plenty of room on all but the most
restrictive platforms. However, if it does end up soaking up so much
space that it compromises the functionality of other scripts, I think
we could simply modify the implementation to make it honor some soft
limit, possibly one that is determined in reference to the user agent.
On Tue, Nov 5, 2013 at 1:00 AM, Derk-Jan Hartman
d.j.hartman+wmf...@gmail.com wrote:
I found a usability problem with some versions of the webkit/chrome
inspector for localStorage due to this, it is being tracked here:
https://bugs.webkit.org/show_bug.cgi?id=123750
Thank you for doing that.
---
Ori Livneh
___
Wikitech-l mailing list
Wikitech-l@lists.wikimedia.org
https://lists.wikimedia.org/mailman/listinfo/wikitech-l
Re: [Wikitech-l] Module storage is coming
On Nov 4, 2013 2:28 AM, Ori Livneh o...@wikimedia.org wrote: The roll-out of 1.23wmf2 to your favorite Wikimedia wiki will inaugurate the era of ResourceLoader module storage -- an era which will be marked by terrifying and hilarious new bugs, intermittent client-side failures, and generally inexcusable disruptions to user experience. Sounds exciting? Read on! What is it? --- Module storage refers to the use of localStorage in ResourceLoader as an auxiliary to the browser cache. ResourceLoader, remember, is the MediaWiki subsystem that delivers JavaScript and CSS to your browser. One of its primary functions is to minimize the total number of network requests that your browser needs to make in order to render the page, and to make the remaining requests as slim as possible. One of the ways ResourceLoader does this is by making a list of all the different components your browser needs and then transmitting them in bulk. The downside of this approach is that a small update to MediaWiki's code, touching perhaps one or two components, will often force the browser to throw out (and re-retrieve) a bunch of unrelated modules that did not change and did not ostensibly need to be re-retrieved. This is because the browser cache operates on the level of network requests; it is not intelligent enough to decompose a request into its constituitive parts and to cache these parts separately from one another. Starting with the 1.23wmf2 branch, ResourceLoader will check if your browser implements localStorage, a mechanism for storing structure data. If it does, ResourceLoader will use it as an auxiliary cache, capable of versioning individual components. Why? The primary goals are: * Destabalize MediaWiki's front-end code, by coupling it to an inconsistently-implemented and poorly-understood HTML5 API. * Make debugging fun again, by adding another layer of caching to MediaWiki. Yes!! However, as with all new features, unintended side-effects are possible. Specific concerns for module storage include the risk of making the network footprint of page loads smaller, resulting in improved latency. But seriously, -- * Module storage is enabled only if the underlying browser supports localStorage (~89% of browsers in use, according to http://caniuse.com/#feat=namevalue-storage). Users with older browsers will not see a benefit, but they will not suffer a penalty either. * Module storage may or may not improve site performance. We need to test it against a wide range of browsers and platforms to know for certain. If it doesn't improve performance, we'll blame it on you, your poor choice of browsers, and your uncooperative attitude, but then we'll scrap it. How can I test it? -- Glad you asked! Module storage is enabled by default on the beta cluster, and on test test2 wikis. The simplest way you can help is by being alert to to performance regressions and front-end code breakage. If you know how to use your browser's JavaScript console, you can get stats about module storage usage on the current page by checking the value of 'mw.loader.store.stats'. When the code is rolled out across the cluster, it will be disabled by default, but you will be able to opt-in by manually setting a cookie in your browser. I will follow up with the exact details. If module storage proves stable enough for production use, we'll seek to asses its utility by running a controlled study of some kind. If we can demonstrate that module storage leads to a significant improvement in performance, we'll enable by it default. Sounds awesome. I'll wait patiently for hilarity to ensue. --- Ori Livneh o...@wikimedia.org ___ Wikitech-l mailing list Wikitech-l@lists.wikimedia.org https://lists.wikimedia.org/mailman/listinfo/wikitech-l ___ Wikitech-l mailing list Wikitech-l@lists.wikimedia.org https://lists.wikimedia.org/mailman/listinfo/wikitech-l
Re: [Wikitech-l] Module storage is coming
On 11/03/2013 08:27 PM, Ori Livneh wrote: * Module storage is enabled only if the underlying browser supports localStorage (~89% of browsers in use, according to http://caniuse.com/#feat=namevalue-storage). Users with older browsers will not see a benefit, but they will not suffer a penalty either. Ori, Thank you for this fun-to-read introduction to Module Storage. I wonder if this will have any affect on low bandwidth users. It seems like this would help if someone had only a 2G/GPRS connection, but do a lower percentage of those users have browsers that can use Module Storage? If so, is there anything that we can to do address this issue? I imagine there are things we could do in the software, but those things would take work and time that might not make sense for limited resources. Would it make sense to see if Mozilla has any interest in helping to spread HTML5-capable browsers via USB keychains and/or local Mozillians? Hrm... I should probably go ask them about that. But I'm curious about your perspective and to see if we have any information on the bandwidth available to various users. Mark. ___ Wikitech-l mailing list Wikitech-l@lists.wikimedia.org https://lists.wikimedia.org/mailman/listinfo/wikitech-l
Re: [Wikitech-l] Module storage is coming
On Mon, Nov 4, 2013 at 1:50 PM, Mark A. Hershberger m...@nichework.comwrote: Ori, Thank you for this fun-to-read introduction to Module Storage. +1 Željko ___ Wikitech-l mailing list Wikitech-l@lists.wikimedia.org https://lists.wikimedia.org/mailman/listinfo/wikitech-l
Re: [Wikitech-l] Module storage is coming
On 04.11.2013, 16:50 Mark wrote: Ori, Thank you for this fun-to-read introduction to Module Storage. I wonder if this will have any affect on low bandwidth users. It seems like this would help if someone had only a 2G/GPRS connection, but do a lower percentage of those users have browsers that can use Module Storage? If so, is there anything that we can to do address this issue? I imagine there are things we could do in the software, but those things would take work and time that might not make sense for limited resources. Would it make sense to see if Mozilla has any interest in helping to spread HTML5-capable browsers via USB keychains and/or local Mozillians? Hrm... I should probably go ask them about that. But I'm curious about your perspective and to see if we have any information on the bandwidth available to various users. Most of our mobile traffic is genrated by iOS and Android - both of which support LocalStorage, so it will work for them too. The situation for mobile users in developing countries is less certain - lots of various browsers with wildly varied support of modern features. Still, it's a huge improvement for the majority of users. -- Best regards, Max Semenik ([[User:MaxSem]]) ___ Wikitech-l mailing list Wikitech-l@lists.wikimedia.org https://lists.wikimedia.org/mailman/listinfo/wikitech-l
Re: [Wikitech-l] Module storage is coming
On 11/04/2013 10:10 AM, Max Semenik wrote: Most of our mobile traffic is genrated by iOS and Android - both of which support LocalStorage, so it will work for them too. People using 2G/GPRS are not necessarily using a mobile device. I've heard from at least one user (hence https://bugzilla.wikimedia.org/55842) that he is using his cell phone as a modem. I suspect this is not a small percentage of users in less developed areas where cell phone towers are plentiful but cables are not. Non-mobile UAs on mobile IPs is what I'm asking about. Mark. ___ Wikitech-l mailing list Wikitech-l@lists.wikimedia.org https://lists.wikimedia.org/mailman/listinfo/wikitech-l
Re: [Wikitech-l] Module storage is coming
I'm looking forward to seeing how this plays out. The only downside I can so far see is that the amount of browser storage available varies drastically [1] and I wonder whether this will cause upsets for those browsers with extremely strict limits. I've also been toying with the idea of using localStorage and cache manifests in mobile for offline usage in mobile for some time and this will compete for that space and make that experiment if it ever happens even more interesting. :-) [1] http://dev-test.nemikor.com/web-storage/support-test/ On Mon, Nov 4, 2013 at 7:31 AM, Mark A. Hershberger m...@nichework.com wrote: On 11/04/2013 10:10 AM, Max Semenik wrote: Most of our mobile traffic is genrated by iOS and Android - both of which support LocalStorage, so it will work for them too. People using 2G/GPRS are not necessarily using a mobile device. I've heard from at least one user (hence https://bugzilla.wikimedia.org/55842) that he is using his cell phone as a modem. I suspect this is not a small percentage of users in less developed areas where cell phone towers are plentiful but cables are not. Non-mobile UAs on mobile IPs is what I'm asking about. Mark. ___ Wikitech-l mailing list Wikitech-l@lists.wikimedia.org https://lists.wikimedia.org/mailman/listinfo/wikitech-l -- Jon Robson http://jonrobson.me.uk @rakugojon ___ Wikitech-l mailing list Wikitech-l@lists.wikimedia.org https://lists.wikimedia.org/mailman/listinfo/wikitech-l
Re: [Wikitech-l] Module storage is coming
On Mon, Nov 4, 2013 at 11:13 AM, Jon Robson jdlrob...@gmail.com wrote: I'm looking forward to seeing how this plays out. The only downside I can so far see is that the amount of browser storage available varies drastically [1] and I wonder whether this will cause upsets for those browsers with extremely strict limits. Or, for that matter, if it will fill up the allowed storage so user scripts and gadgets can't make effective use of it. -- Brad Jorsch (Anomie) Software Engineer Wikimedia Foundation ___ Wikitech-l mailing list Wikitech-l@lists.wikimedia.org https://lists.wikimedia.org/mailman/listinfo/wikitech-l
Re: [Wikitech-l] Module storage is coming
On 04/11/13 15:31, Mark A. Hershberger wrote: On 11/04/2013 10:10 AM, Max Semenik wrote: Most of our mobile traffic is genrated by iOS and Android - both of which support LocalStorage, so it will work for them too. People using 2G/GPRS are not necessarily using a mobile device. I've heard from at least one user (hence https://bugzilla.wikimedia.org/55842) that he is using his cell phone as a modem. I suspect this is not a small percentage of users in less developed areas where cell phone towers are plentiful but cables are not. Non-mobile UAs on mobile IPs is what I'm asking about. Mark. ___ Wikitech-l mailing list Wikitech-l@lists.wikimedia.org https://lists.wikimedia.org/mailman/listinfo/wikitech-l Er, that's a fairly standard speed for low-cost wired plans as well in much of the US. I'd actually be happy to have something that fast, but I couldn't afford it. Well, it's the US's version of 'low cost', anyway. ___ Wikitech-l mailing list Wikitech-l@lists.wikimedia.org https://lists.wikimedia.org/mailman/listinfo/wikitech-l
Re: [Wikitech-l] Module storage is coming
MediaWiki: Manually fixing broken browser functionality since 2012. Also I do wonder how useful this actually is. Does site JavaScript really change that often? I suppose we'll find out after testing. *-- * *Tyler Romeo* Stevens Institute of Technology, Class of 2016 Major in Computer Science On Sun, Nov 3, 2013 at 8:27 PM, Ori Livneh o...@wikimedia.org wrote: The roll-out of 1.23wmf2 to your favorite Wikimedia wiki will inaugurate the era of ResourceLoader module storage -- an era which will be marked by terrifying and hilarious new bugs, intermittent client-side failures, and generally inexcusable disruptions to user experience. Sounds exciting? Read on! What is it? --- Module storage refers to the use of localStorage in ResourceLoader as an auxiliary to the browser cache. ResourceLoader, remember, is the MediaWiki subsystem that delivers JavaScript and CSS to your browser. One of its primary functions is to minimize the total number of network requests that your browser needs to make in order to render the page, and to make the remaining requests as slim as possible. One of the ways ResourceLoader does this is by making a list of all the different components your browser needs and then transmitting them in bulk. The downside of this approach is that a small update to MediaWiki's code, touching perhaps one or two components, will often force the browser to throw out (and re-retrieve) a bunch of unrelated modules that did not change and did not ostensibly need to be re-retrieved. This is because the browser cache operates on the level of network requests; it is not intelligent enough to decompose a request into its constituitive parts and to cache these parts separately from one another. Starting with the 1.23wmf2 branch, ResourceLoader will check if your browser implements localStorage, a mechanism for storing structure data. If it does, ResourceLoader will use it as an auxiliary cache, capable of versioning individual components. Why? The primary goals are: * Destabalize MediaWiki's front-end code, by coupling it to an inconsistently-implemented and poorly-understood HTML5 API. * Make debugging fun again, by adding another layer of caching to MediaWiki. Yes!! However, as with all new features, unintended side-effects are possible. Specific concerns for module storage include the risk of making the network footprint of page loads smaller, resulting in improved latency. But seriously, -- * Module storage is enabled only if the underlying browser supports localStorage (~89% of browsers in use, according to http://caniuse.com/#feat=namevalue-storage). Users with older browsers will not see a benefit, but they will not suffer a penalty either. * Module storage may or may not improve site performance. We need to test it against a wide range of browsers and platforms to know for certain. If it doesn't improve performance, we'll blame it on you, your poor choice of browsers, and your uncooperative attitude, but then we'll scrap it. How can I test it? -- Glad you asked! Module storage is enabled by default on the beta cluster, and on test test2 wikis. The simplest way you can help is by being alert to to performance regressions and front-end code breakage. If you know how to use your browser's JavaScript console, you can get stats about module storage usage on the current page by checking the value of 'mw.loader.store.stats'. When the code is rolled out across the cluster, it will be disabled by default, but you will be able to opt-in by manually setting a cookie in your browser. I will follow up with the exact details. If module storage proves stable enough for production use, we'll seek to asses its utility by running a controlled study of some kind. If we can demonstrate that module storage leads to a significant improvement in performance, we'll enable by it default. --- Ori Livneh o...@wikimedia.org ___ Wikitech-l mailing list Wikitech-l@lists.wikimedia.org https://lists.wikimedia.org/mailman/listinfo/wikitech-l ___ Wikitech-l mailing list Wikitech-l@lists.wikimedia.org https://lists.wikimedia.org/mailman/listinfo/wikitech-l
