Re: [b2g] Using blobs to display images
On Fri, Apr 25, 2014 at 4:30 PM, Andrew Sutherland wrote: > (Although we'd still want code to be able to at least createObjectURL things > explicitly into that space for at least Blobs. I think manually requiring > legwork in the non-Blob cases could still be okay.) createObjectURL will continue to work as-is for all Blobs. Including blobs loaded from IDB. / Jonas ___ dev-b2g mailing list dev-b2g@lists.mozilla.org https://lists.mozilla.org/listinfo/dev-b2g
Re: [b2g] Using blobs to display images
On 04/25/2014 06:50 PM, Jonas Sicking wrote: I think very few use cases would get broken by equating "reading from indexeddb://" to "running script". Yes, that formulation sells me on that approach. (Although we'd still want code to be able to at least createObjectURL things explicitly into that space for at least Blobs. I think manually requiring legwork in the non-Blob cases could still be okay.) Andrew ___ dev-b2g mailing list dev-b2g@lists.mozilla.org https://lists.mozilla.org/listinfo/dev-b2g
Re: [b2g] Using blobs to display images
On Wed, Apr 23, 2014 at 10:12 PM, Andrew Sutherland wrote: >> Another solution is to define that sandboxed pages with scripts >> disabled can't load from local urls. At least not unless explicitly >> allowed by some new sandbox flag. > > This seems unintuitive. > > I think the simplest/best option is just to not expose/map the indexedDB URI > scheme unless it is explicitly requested. The problem is that this simply reduces the problem. It doesn't solve it. I.e. if I have a database which I do want to be able to get data from using indexeddb: URLs for my own pages, how do I protect myself from 3rd party content reading from that database? Another thing to keep in mind is that multiple people often work on the same website. Just because one person knows that he needs to access indexeddb: URLs for a particular database, doesn't mean that another person knows that running 3rd party content in-origin now enables that 3rd party content to read from a particular database. I think very few use cases would get broken by equating "reading from indexeddb://" to "running script". / Jonas ___ dev-b2g mailing list dev-b2g@lists.mozilla.org https://lists.mozilla.org/listinfo/dev-b2g
Re: [b2g] Using blobs to display images
Le 24/04/2014 08:34, Jonas Sicking a écrit : > On Sat, Apr 19, 2014 at 6:49 AM, Julien Wajsberg > wrote: >>> At the very least you can revoke the URL when we stop displaying the >>> image. I.e. if we change the background to some other URL, or if that >>> part of the UI is completely hidden >> Yes that's also a possibility I had in mind. Aren't we taking up valuable >> memory if you keep the blob url valid? Or is that memory used anyway, >> because the image is displayed, and as a result it would not take more than >> what's already taken? > It uses valuable memory indeed. However releasing it once the image > stops being displayed is better than not releasing it at all. > > But note that if you get the Blob from indexedDB or from DataStore, > then the Blob and bloburl uses very little memory since it's basically > just a pointer into the database. Okay, good to know, thanks. > >>> (I don't know what piece of UI >>> that we're talking about here). >> We're doing this in 2 parts of the UI: >> >> * displaying contacts pictures. This currently comes from the mozContacts >> API only. But for Facebook contacts this will eventually come from a >> Datastore or a local IDB as well. We might even do a local IDB for all >> contact's source too, in the future. > So there's a risk here that we'll display a large number of images at once > here. > > As long as all those blobs point to data in an IDB database that should be ok. Currently they don't, but they should once we use background-size. > >> * displaying attachments. Until very recently, we were generating thumbnails >> using the canvas API, and export them as data url, and use this as >> background-image. In bug 996516 we'll take advantage of the downsamping url >> feature to use directly the blob url in this. The data comes from the >> mozMobileMessage API, and from activities. > We really should not use data urls for anything. They suck up a lot of > memory. Not only does it keep the whole encoded image in memory. It > likely keeps several copies of it. And each copy is in a very > ineffective format (130-260% of binary size) We actually tried to get rid of it earlier (bug 889902 [1]) but using blobs proved to be worse than with dataurl (see comment 21). Probably we didn't try correctly, but anyway the new try using the samplesize and background-size should be way better. [1] https://bugzilla.mozilla.org/show_bug.cgi?id=889902 > >>> Another solution here would be if we added support for URLs that map >>> directly into indexedDB. See >>> >>> http://lists.w3.org/Archives/Public/public-webapps/2013JulSep/0081.html >> Would that cross process boundaries? I guess not, and so would not be that >> useful for us. > Not sure what you mean by "process boundaries". IndexedDB is always > accessed through the parent process. As is any time that we load > something from the network. > > So in that sense this would indeed work across process boundaries. > > But if you mean being able to load another application's IDB data that > would not work. But that has nothing to do with process boundaries but > rather is by design. oki, that's what I wanted to mean, thanks :) -- Julien signature.asc Description: OpenPGP digital signature ___ dev-b2g mailing list dev-b2g@lists.mozilla.org https://lists.mozilla.org/listinfo/dev-b2g
Re: [b2g] Using blobs to display images
On Sat, Apr 19, 2014 at 6:49 AM, Julien Wajsberg wrote: >> At the very least you can revoke the URL when we stop displaying the >> image. I.e. if we change the background to some other URL, or if that >> part of the UI is completely hidden > > Yes that's also a possibility I had in mind. Aren't we taking up valuable > memory if you keep the blob url valid? Or is that memory used anyway, > because the image is displayed, and as a result it would not take more than > what's already taken? It uses valuable memory indeed. However releasing it once the image stops being displayed is better than not releasing it at all. But note that if you get the Blob from indexedDB or from DataStore, then the Blob and bloburl uses very little memory since it's basically just a pointer into the database. >> (I don't know what piece of UI >> that we're talking about here). > > We're doing this in 2 parts of the UI: > > * displaying contacts pictures. This currently comes from the mozContacts > API only. But for Facebook contacts this will eventually come from a > Datastore or a local IDB as well. We might even do a local IDB for all > contact's source too, in the future. So there's a risk here that we'll display a large number of images at once here. As long as all those blobs point to data in an IDB database that should be ok. > * displaying attachments. Until very recently, we were generating thumbnails > using the canvas API, and export them as data url, and use this as > background-image. In bug 996516 we'll take advantage of the downsamping url > feature to use directly the blob url in this. The data comes from the > mozMobileMessage API, and from activities. We really should not use data urls for anything. They suck up a lot of memory. Not only does it keep the whole encoded image in memory. It likely keeps several copies of it. And each copy is in a very ineffective format (130-260% of binary size) >> However it would also be good to have some sort of event fired once >> layout code has fully loaded any resources that it depends on, such as >> background images. However I just talked to dbaron and he pointed out >> that this would be of pretty narrow functionality since it wouldn't >> necessarily mean that the page is ready to be painted (asynchronous >> image decoding might still need to happen), and it also wouldn't work >> in cases where we avoid doing image loads for images that are outside >> of the current visible part of the page. > > Does that mean that revoking after 30 seconds would not work either in some > situations? In current gecko it should be fine. But there is indeed some risk that it'd break in the future. But more to the point the "fire an event when layout is done" solution wasn't as easy as I had hoped. >> Another solution here would be if we added support for URLs that map >> directly into indexedDB. See >> >> http://lists.w3.org/Archives/Public/public-webapps/2013JulSep/0081.html > > Would that cross process boundaries? I guess not, and so would not be that > useful for us. Not sure what you mean by "process boundaries". IndexedDB is always accessed through the parent process. As is any time that we load something from the network. So in that sense this would indeed work across process boundaries. But if you mean being able to load another application's IDB data that would not work. But that has nothing to do with process boundaries but rather is by design. Using the URLs above you would be able to access the exact same data that you can through the IDB API. It just saves you the work of getting a Blob from the IDB API and calling createObjectURL on that Blob. And because you don't call createObjectURL you also wouldn't have to worry about when to call revokeObjectURL. / Jonas ___ dev-b2g mailing list dev-b2g@lists.mozilla.org https://lists.mozilla.org/listinfo/dev-b2g
Re: [b2g] Using blobs to display images
On 04/24/2014 12:25 AM, Jonas Sicking wrote: Even though you could link to resources in local databases, how would you then extract data and send it to the server? Simply displaying an image from the database could be surprising/scary to the user, but doesn't actually extract any sensitive information. The information leak is akin to CSS :visited attacks a la http://dbaron.org/mozilla/visited-privacy In this case the information channel is either in the timing delta of multiple external resources fetches, missing subsequent external resource fetches, or as you alternately propose, entirely conditional resource fetches. Another solution is to define that sandboxed pages with scripts disabled can't load from local urls. At least not unless explicitly allowed by some new sandbox flag. This seems unintuitive. I think the simplest/best option is just to not expose/map the indexedDB URI scheme unless it is explicitly requested. Say as a parameter for opening the database. An argument against this would be if we thought it would make sense for a URI access to go and open an IndexedDB database on its own. Given the versioned semantics of IndexedDB and general sanity, I don't think we would want that. This addresses my hypothetical example fairly well since it's explicitly opt-in and if the flag is well-named, even someone fairly blithely copying and pasting IndexedDB boilerplate has a fair chance of noticing "exposeDbViaIndexedDBUriScheme: true". (In my example I think I'm assuming a competent but naive/not super-experienced with the web platform and information leakage via side-channels. So something like a magic backdoor URI scheme that exposes the entirety of the database without requiring an opt-in would be a big vulnerability they wouldn't be aware of.) Andrew ___ dev-b2g mailing list dev-b2g@lists.mozilla.org https://lists.mozilla.org/listinfo/dev-b2g
Re: [b2g] Using blobs to display images
On Sat, Apr 19, 2014 at 6:31 AM, Andrew Sutherland wrote: > On 04/18/2014 08:58 PM, Jonas Sicking wrote: >> >> Another solution here would be if we added support for URLs that map >> directly into indexedDB. See >> >> http://lists.w3.org/Archives/Public/public-webapps/2013JulSep/0081.html > > > It seems like this could open new avenues for information leakage unless > URLs require JS code execution to bless them into validity using a variant > on createObjectURL or just the getURL your proposed bug magicked into > createURL/mapURL. For example... > > Hypothetically, let's say I'm an email app with a predictable db schema; > messages are stored by the uniqueish "message-id"[1] header included with > every message and known to every recipient of that message. Let's also say > my authors didn't care about user privacy too much so they display remote > images/resources by default and don't bother sanitizing them for this > reason. I, hypothetical email app, do put that HTML content in an sandboxed > iframe that forbids script from running because I'm not completely reckless. > But I do keep the iframe in my origin because in order for me to display > embedded images in the e-mail which I've saved to IndexedDB as Blobs, well, > that's what I've got to do to surface the Blobs or be able to > manipulate/interact with the iframe's HTML DOM at all after minting the > iframe and its document. Even though you could link to resources in local databases, how would you then extract data and send it to the server? Simply displaying an image from the database could be surprising/scary to the user, but doesn't actually extract any sensitive information. However I think CSS actually would allow using a fallback syntax which would only trigger loads back to a server if a local load failed. I.e. I think something like the following http://snooper.com/...))"> will only send a request to snooper.com if the load from indexeddb failed. There are two ways we could handle this. We could define that doing fallback from a local url like indexeddb:// isn't allowed. This seems unfortunate as it seems like a useful feature. Another solution is to define that sandboxed pages with scripts disabled can't load from local urls. At least not unless explicitly allowed by some new sandbox flag. / Jonas ___ dev-b2g mailing list dev-b2g@lists.mozilla.org https://lists.mozilla.org/listinfo/dev-b2g
Re: [b2g] Using blobs to display images
Julien Wajsberg writes: > Le 16/04/2014 20:24, Kyle Huey a écrit : >> >> The correct way to think of it is that revoking a blob URL only >> affects the network level. If loading/etc is complete it will >> generally have no effect. There are certain caveats to this for >> things that happen lazily (such as CSS background image loading, which >> only happens when there is an element that actually has the image as a >> background). >> > So, does that mean that when we use a blob url, imglib keeps a second > copy of the compressed image data, and we end up with 2 copies of that > data until we revoke the blob url? Or somehow is this data mmap-ed or > something, so that it's shared in a way that it's not freed when we > revoke the blob url? imglib doesn't know from where the image is loaded so it copies everything it received. So yes, you could get 2 copies of that data if it's a memory-backed Blob. ___ dev-b2g mailing list dev-b2g@lists.mozilla.org https://lists.mozilla.org/listinfo/dev-b2g
Re: [b2g] Using blobs to display images
Julien Wajsberg writes: > Le 19/04/2014 16:39, Andrew Sutherland a écrit : >> On 04/19/2014 09:49 AM, Julien Wajsberg wrote: >>> Yes that's also a possibility I had in mind. Aren't we taking up >>> valuable memory if you keep the blob url valid? Or is that memory >>> used anyway, because the image is displayed, and as a result it >>> would not take more than what's already taken? >> >> If it's a memory-backed Blob and that's the only reference to it, I >> believe the answer is yes because you are keeping both the Blob JS >> Object/its backing XPCOM object and its referenced data alive. > > David and Jonas and Kyle were saying that imglib keeps the compressed > data in memory to display it. So my question was more: does it do a > real copy of it, or does it do a fake copy that is just another > reference? What about imglib just keeping a ref to the blob instead of > copying the data? (That's what Thinker is suggesting in [1]). imagelib does keep a real copy of the compressed data in memory and use that to decode when needed. > [1] https://bugzilla.mozilla.org/show_bug.cgi?id=996512#c4 >> If it's backed by a file on disk such as a File/Blob reference >> provided by IndexedDB, you're only keeping the smallish JS >> Object/XPCOM object alive so it's less bad. >> >> In general if you have a memory-backed Blob you really want to: >> - Save it to IndexedDB or DeviceStorage immediately if you are going >> to ever save it to those spots. >> - Get a *new* reference to the Blob and forget all your old >> references to the memory backed Blob. > > > I would think that Gecko could do this for me if it needs the > memory. Like a swap mechanism. Seth and I had discussed about saving memory-backed Blob & other temporary resources to necko cache for just one session. I'm not sure if we could pin a resource in necko already. Note that even file-backed Blob will be copied by imagelib. So the blob:// I mentioned in bug 996512 is actually referring to file-backed Blobs. Kanru ___ dev-b2g mailing list dev-b2g@lists.mozilla.org https://lists.mozilla.org/listinfo/dev-b2g
Re: [b2g] Using blobs to display images
Le 19/04/2014 16:39, Andrew Sutherland a écrit : On 04/19/2014 09:49 AM, Julien Wajsberg wrote: Yes that's also a possibility I had in mind. Aren't we taking up valuable memory if you keep the blob url valid? Or is that memory used anyway, because the image is displayed, and as a result it would not take more than what's already taken? If it's a memory-backed Blob and that's the only reference to it, I believe the answer is yes because you are keeping both the Blob JS Object/its backing XPCOM object and its referenced data alive. David and Jonas and Kyle were saying that imglib keeps the compressed data in memory to display it. So my question was more: does it do a real copy of it, or does it do a fake copy that is just another reference? What about imglib just keeping a ref to the blob instead of copying the data? (That's what Thinker is suggesting in [1]). [1] https://bugzilla.mozilla.org/show_bug.cgi?id=996512#c4 If it's backed by a file on disk such as a File/Blob reference provided by IndexedDB, you're only keeping the smallish JS Object/XPCOM object alive so it's less bad. In general if you have a memory-backed Blob you really want to: - Save it to IndexedDB or DeviceStorage immediately if you are going to ever save it to those spots. - Get a *new* reference to the Blob and forget all your old references to the memory backed Blob. I would think that Gecko could do this for me if it needs the memory. Like a swap mechanism. -- Julien ___ dev-b2g mailing list dev-b2g@lists.mozilla.org https://lists.mozilla.org/listinfo/dev-b2g
Re: [b2g] Using blobs to display images
On 04/19/2014 09:49 AM, Julien Wajsberg wrote: Yes that's also a possibility I had in mind. Aren't we taking up valuable memory if you keep the blob url valid? Or is that memory used anyway, because the image is displayed, and as a result it would not take more than what's already taken? If it's a memory-backed Blob and that's the only reference to it, I believe the answer is yes because you are keeping both the Blob JS Object/its backing XPCOM object and its referenced data alive. If it's backed by a file on disk such as a File/Blob reference provided by IndexedDB, you're only keeping the smallish JS Object/XPCOM object alive so it's less bad. In general if you have a memory-backed Blob you really want to: - Save it to IndexedDB or DeviceStorage immediately if you are going to ever save it to those spots. - Get a *new* reference to the Blob and forget all your old references to the memory backed Blob. To be super-clear, putting memory-backed 'fooBlob' to IndexedDB does not change the nature of 'fooBlob'. It's still the same memory-backed Blob. Andrew ___ dev-b2g mailing list dev-b2g@lists.mozilla.org https://lists.mozilla.org/listinfo/dev-b2g
Re: [b2g] Using blobs to display images
Le 16/04/2014 20:24, Kyle Huey a écrit : The correct way to think of it is that revoking a blob URL only affects the network level. If loading/etc is complete it will generally have no effect. There are certain caveats to this for things that happen lazily (such as CSS background image loading, which only happens when there is an element that actually has the image as a background). So, does that mean that when we use a blob url, imglib keeps a second copy of the compressed image data, and we end up with 2 copies of that data until we revoke the blob url? Or somehow is this data mmap-ed or something, so that it's shared in a way that it's not freed when we revoke the blob url? ___ dev-b2g mailing list dev-b2g@lists.mozilla.org https://lists.mozilla.org/listinfo/dev-b2g
Re: [b2g] Using blobs to display images
Le 19/04/2014 02:58, Jonas Sicking a écrit : On Fri, Apr 18, 2014 at 6:12 AM, Julien Wajsberg wrote: On Wed, Apr 16, 2014 at 10:45 AM, ben turner wrote: -- Forwarded message -- From: Julien Wajsberg Date: Wed, Apr 16, 2014 at 9:30 AM Subject: [b2g] Using blobs to display images To: dev-b2g Hey, Basically, I'd like to know whether it's better to revoke a blob URL just after an image has been loaded, even if it's still displayed, or to keep the blob url valid until the image is hidden. tl;dr: You should revoke it when the onload or onerror events fire. Ok, I see that I forgot one important bit of information: I'm actually concerned about background images. For normal images in it's easy to do in onerror/onload. I tried to do clever things like loading asychronously (after setTimeout) an in-memory image with the same blob url and discarding at onload/onerror but this randomly fails. The "best" solution I have right now is to put a timer of say 30 seconds and revoke the URL then, but I'm afraid to retain memory in case of memory pressure. But maybe that's good enough? Ouch, yeah, this is a bit of a problem. At the very least you can revoke the URL when we stop displaying the image. I.e. if we change the background to some other URL, or if that part of the UI is completely hidden Yes that's also a possibility I had in mind. Aren't we taking up valuable memory if you keep the blob url valid? Or is that memory used anyway, because the image is displayed, and as a result it would not take more than what's already taken? (I don't know what piece of UI that we're talking about here). We're doing this in 2 parts of the UI: * displaying contacts pictures. This currently comes from the mozContacts API only. But for Facebook contacts this will eventually come from a Datastore or a local IDB as well. We might even do a local IDB for all contact's source too, in the future. * displaying attachments. Until very recently, we were generating thumbnails using the canvas API, and export them as data url, and use this as background-image. In bug 996516 we'll take advantage of the downsamping url feature to use directly the blob url in this. The data comes from the mozMobileMessage API, and from activities. In both cases, using background-image brings us the ratio-aware property background-size. It brings us also background-position which we don't use yet but could be useful to properly position a contact's face. Probably if we had the same possibility for usual images we could use them instead (even if in the contact picture case, a background image looks more semantic). However it would also be good to have some sort of event fired once layout code has fully loaded any resources that it depends on, such as background images. However I just talked to dbaron and he pointed out that this would be of pretty narrow functionality since it wouldn't necessarily mean that the page is ready to be painted (asynchronous image decoding might still need to happen), and it also wouldn't work in cases where we avoid doing image loads for images that are outside of the current visible part of the page. Does that mean that revoking after 30 seconds would not work either in some situations? Another solution here would be if we added support for URLs that map directly into indexedDB. See http://lists.w3.org/Archives/Public/public-webapps/2013JulSep/0081.html Would that cross process boundaries? I guess not, and so would not be that useful for us. Thanks ! -- Julien ___ dev-b2g mailing list dev-b2g@lists.mozilla.org https://lists.mozilla.org/listinfo/dev-b2g
Re: [b2g] Using blobs to display images
In the Messages app, we don't have several times the same image displayed in the same time... yet. But this will eventually come if we display the contact's picture in the thread view. So thanks for this bit of advice ! Le 19/04/2014 03:09, L. David Baron a écrit : Another thing that's a bit dangerous about blobs that sicking and khuey and I were discussing in person that I think is worth bringing up: If you create two blob URLs for the same Blob (or, for that matter, for different Blob objects that represent the same data), the rest of the system doesn't know that these are the same. So, for example, if they're images, you'll end up with two (identical) decoded images in the image cache, which uses more memory, since imagelib has no idea that these two blob URLs lead to the same resource. So if you're going want to use the same blob as a URL in multiple places, it's best to use the same URL. If they're stylesheets (though I admit that sounds odd), you'd end up defeating a different pile of sharing optimizations (that propagate from failing to share the sheet structures through, as a result, failing to share computed style data). And probably similar things for other types of resources. (Yes, this API seems less than ideal.) -David ___ dev-b2g mailing list dev-b2g@lists.mozilla.org https://lists.mozilla.org/listinfo/dev-b2g
Re: [b2g] Using blobs to display images
On 04/18/2014 08:58 PM, Jonas Sicking wrote: Another solution here would be if we added support for URLs that map directly into indexedDB. See http://lists.w3.org/Archives/Public/public-webapps/2013JulSep/0081.html It seems like this could open new avenues for information leakage unless URLs require JS code execution to bless them into validity using a variant on createObjectURL or just the getURL your proposed bug magicked into createURL/mapURL. For example... Hypothetically, let's say I'm an email app with a predictable db schema; messages are stored by the uniqueish "message-id"[1] header included with every message and known to every recipient of that message. Let's also say my authors didn't care about user privacy too much so they display remote images/resources by default and don't bother sanitizing them for this reason. I, hypothetical email app, do put that HTML content in an sandboxed iframe that forbids script from running because I'm not completely reckless. But I do keep the iframe in my origin because in order for me to display embedded images in the e-mail which I've saved to IndexedDB as Blobs, well, that's what I've got to do to surface the Blobs or be able to manipulate/interact with the iframe's HTML DOM at all after minting the iframe and its document. And for the payoff: evil bad person formulates an HTML e-mail that checks whether the user has received a message that definitely exists or might exist by including resource references to the IndexedDB database that impacts the timing of other external resources fetched by the e-mail where the attacker controls the resources or can somehow observe the timing that the resources are fetched. If the web platform in question defers fetching resources until they can be displayed, the failure to fetch a resource in a timely fashion can also produce entropy. This probably does require the user to open the email, although if the app tries to do any parsing of the email ahead of time to generate snippets or detect links or phone numbers or package delivery references or the like, no interaction would be required. Such an attack would be a bit easier if the message the attacker is looking for included an embedded Blob so it can be used to conditionally take up document real-estate. But even use of an img tag in a way that's expected to fail to load regardless of the presence of the message would likely result in observable timing differences based on our IndexedDB schema. There may also be a class of vulnerabilities related to the use of elements, but that more definitely involves user interaction. Note that the Gaia email app does not work like this hypothetical email app. Andrew 1: Note that the message-id isn't intended to be unguessable, just not collide with other messages. Many mail clients' generated id's are likely to have much of their entropy come from the time at which they were sent, so the brute forcing space is limited if you know around when the message was sent. ___ dev-b2g mailing list dev-b2g@lists.mozilla.org https://lists.mozilla.org/listinfo/dev-b2g
Re: [b2g] Using blobs to display images
Another thing that's a bit dangerous about blobs that sicking and khuey and I were discussing in person that I think is worth bringing up: If you create two blob URLs for the same Blob (or, for that matter, for different Blob objects that represent the same data), the rest of the system doesn't know that these are the same. So, for example, if they're images, you'll end up with two (identical) decoded images in the image cache, which uses more memory, since imagelib has no idea that these two blob URLs lead to the same resource. So if you're going want to use the same blob as a URL in multiple places, it's best to use the same URL. If they're stylesheets (though I admit that sounds odd), you'd end up defeating a different pile of sharing optimizations (that propagate from failing to share the sheet structures through, as a result, failing to share computed style data). And probably similar things for other types of resources. (Yes, this API seems less than ideal.) -David -- 𝄞 L. David Baron http://dbaron.org/ 𝄂 𝄢 Mozilla https://www.mozilla.org/ 𝄂 Before I built a wall I'd ask to know What I was walling in or walling out, And to whom I was like to give offense. - Robert Frost, Mending Wall (1914) signature.asc Description: Digital signature ___ dev-b2g mailing list dev-b2g@lists.mozilla.org https://lists.mozilla.org/listinfo/dev-b2g
Re: [b2g] Using blobs to display images
On Fri, Apr 18, 2014 at 6:12 AM, Julien Wajsberg wrote: >> On Wed, Apr 16, 2014 at 10:45 AM, ben turner >> wrote: >>> >>> >>> -- Forwarded message -- >>> From: Julien Wajsberg >>> Date: Wed, Apr 16, 2014 at 9:30 AM >>> Subject: [b2g] Using blobs to display images >>> To: dev-b2g >>> >>> >>> Hey, >>> >>> Basically, I'd like to know whether it's better to revoke a blob URL >>> just after an image has been loaded, even if it's still displayed, or to >>> keep the blob url valid until the image is hidden. >> >> tl;dr: You should revoke it when the onload or onerror events fire. > > Ok, I see that I forgot one important bit of information: I'm actually > concerned about background images. For normal images in it's easy to > do in onerror/onload. > > I tried to do clever things like loading asychronously (after setTimeout) an > in-memory image with the same blob url and discarding at onload/onerror but > this randomly fails. > > The "best" solution I have right now is to put a timer of say 30 seconds and > revoke the URL then, but I'm afraid to retain memory in case of memory > pressure. But maybe that's good enough? Ouch, yeah, this is a bit of a problem. At the very least you can revoke the URL when we stop displaying the image. I.e. if we change the background to some other URL, or if that part of the UI is completely hidden (I don't know what piece of UI that we're talking about here). However it would also be good to have some sort of event fired once layout code has fully loaded any resources that it depends on, such as background images. However I just talked to dbaron and he pointed out that this would be of pretty narrow functionality since it wouldn't necessarily mean that the page is ready to be painted (asynchronous image decoding might still need to happen), and it also wouldn't work in cases where we avoid doing image loads for images that are outside of the current visible part of the page. Another solution here would be if we added support for URLs that map directly into indexedDB. See http://lists.w3.org/Archives/Public/public-webapps/2013JulSep/0081.html / Jonas ___ dev-b2g mailing list dev-b2g@lists.mozilla.org https://lists.mozilla.org/listinfo/dev-b2g
Re: [b2g] Using blobs to display images
Thanks Kyle for this valuable information. Some more questions: Le 16/04/2014 20:24, Kyle Huey a écrit : On Wed, Apr 16, 2014 at 10:45 AM, ben turner wrote: -- Forwarded message -- From: Julien Wajsberg Date: Wed, Apr 16, 2014 at 9:30 AM Subject: [b2g] Using blobs to display images To: dev-b2g Hey, Basically, I'd like to know whether it's better to revoke a blob URL just after an image has been loaded, even if it's still displayed, or to keep the blob url valid until the image is hidden. tl;dr: You should revoke it when the onload or onerror events fire. Ok, I see that I forgot one important bit of information: I'm actually concerned about background images. For normal images in it's easy to do in onerror/onload. I tried to do clever things like loading asychronously (after setTimeout) an in-memory image with the same blob url and discarding at onload/onerror but this randomly fails. The "best" solution I have right now is to put a timer of say 30 seconds and revoke the URL then, but I'm afraid to retain memory in case of memory pressure. But maybe that's good enough? * if the source is a http URL or a data URL, then gecko is confident that the source will always be available; No, it's not. HTTP is not idempotent. What happens if you're offline, or your cookies that you need to view the image have expired? I was thinking that the data downloaded using HTTP would be cached in the network cache. But you're right, this was a bad assumption. -- Julien ___ dev-b2g mailing list dev-b2g@lists.mozilla.org https://lists.mozilla.org/listinfo/dev-b2g
Re: [b2g] Using blobs to display images
On Wed, Apr 16, 2014 at 10:45 AM, ben turner wrote: > > > -- Forwarded message -- > From: Julien Wajsberg > Date: Wed, Apr 16, 2014 at 9:30 AM > Subject: [b2g] Using blobs to display images > To: dev-b2g > > > Hey, > > Basically, I'd like to know whether it's better to revoke a blob URL > just after an image has been loaded, even if it's still displayed, or to > keep the blob url valid until the image is hidden. tl;dr: You should revoke it when the onload or onerror events fire. (I'm going to ignore some things like the Necko cache and image scaling for this discussion to avoid making it too complicated.) As far as the image layer of Gecko is concerned images loaded via the blob: protocol are no different than those loaded over http:, file:, data:, etc. Gecko does not in general assume that URL loading is idempotent. For certain protocols it is guaranteed to be (e.g. data) and for others it is not (e.g. http), but we generally assume it is not since a non-idempotent scheme (http) is by far the most dominant protocol on the web. This means that layers that require the original source data to be available must store it themselves. Not every part of Gecko has this requirement: the DOM can discard the source document after parsing because the DOM tree representation becomes the canonical copy of that data. But for images, where the uncompressed copy of the image can be several times the size of the copy we loaded off the network, we want to be able to discard the uncompressed copy to save memory. Thus imagelib must save the original compressed copies of the image itself. Thus, > Here is a rationale I can't remove from my mind: > * the image as a blob will likely be smaller (because it's compressed) > than the image as displayed (because it's uncompressed) That's correct, but this is also true of all non-bitmap images. The original source is essentially always smaller than the uncompressed copy that we need to paint to the screen. > * what should gecko do once the image is displayed? We will eventually > need the uncompressed data to repaint. Gecko attempts to be smart about when it decompresses images and how long it keeps that data around for before throwing it away and decompressing it again when it is needed next. The heuristics for this are certainly not perfect, and they may not be applied at all to certain things such as CSS background images. > * if the source is a http URL or a data URL, then gecko is confident > that the source will always be available; No, it's not. HTTP is not idempotent. What happens if you're offline, or your cookies that you need to view the image have expired? > therefore it could discard any > uncompressed data in case of memory pressure, because it could calculate > it again from the source. Gecko stores the compressed data for everything, so it can always discard uncompressed image data and recalculate it again when necessary. > * if the source is a blob URL and is revoked, then gecko won't be able > to get the uncompressed data once it's discarded, therefore we need to > keep it in memory. Right, but this is true of basically every protocol (except data:, but we don't have special behavior for them). > * if the source is a blob URL and still valid, we can argue both behaviors. > > I don't know _at all_ how much of what I describe here is true/wrong, > and therefore I hope somebody will be able to answer this, not in terms > of "what I think happens" but in terms of "what gecko is doing". The correct way to think of it is that revoking a blob URL only affects the network level. If loading/etc is complete it will generally have no effect. There are certain caveats to this for things that happen lazily (such as CSS background image loading, which only happens when there is an element that actually has the image as a background). - Kyle ___ dev-b2g mailing list dev-b2g@lists.mozilla.org https://lists.mozilla.org/listinfo/dev-b2g
[b2g] Using blobs to display images
Hey, Basically, I'd like to know whether it's better to revoke a blob URL just after an image has been loaded, even if it's still displayed, or to keep the blob url valid until the image is hidden. Here is a rationale I can't remove from my mind: * the image as a blob will likely be smaller (because it's compressed) than the image as displayed (because it's uncompressed) * what should gecko do once the image is displayed? We will eventually need the uncompressed data to repaint. * if the source is a http URL or a data URL, then gecko is confident that the source will always be available; therefore it could discard any uncompressed data in case of memory pressure, because it could calculate it again from the source. * if the source is a blob URL and is revoked, then gecko won't be able to get the uncompressed data once it's discarded, therefore we need to keep it in memory. * if the source is a blob URL and still valid, we can argue both behaviors. I don't know _at all_ how much of what I describe here is true/wrong, and therefore I hope somebody will be able to answer this, not in terms of "what I think happens" but in terms of "what gecko is doing". Thanks a lot in advance! -- Julien signature.asc Description: OpenPGP digital signature ___ dev-b2g mailing list dev-b2g@lists.mozilla.org https://lists.mozilla.org/listinfo/dev-b2g
