Re: [whatwg] ImageBitmap feature requests
On Sun, May 18, 2014 at 11:02 PM, Robert O'Callahan rob...@ocallahan.orgwrote: On Sat, May 17, 2014 at 4:18 AM, Anne van Kesteren ann...@annevk.nlwrote: Maybe we should have img.srcObject similar to what we're doing for media elements. img.src can simply return about:imagebitmap or some such. That way you can also assign a Blob to an img element without having to do the weird createObjectURL() hack that might leak memory if you're not careful. I like this approach, but I think it's simpler to continue to have HTMLImageElement.src reflect the src content attribute. I wonder what kind of broader effect it would have if image content can no longer be uniquely identified or retrieved using a URL. In many places in Blink/WebKit (and presumably other implementations as well) URLs are used as keys and handles for image resources. All of that would have to be refactored.
Re: [whatwg] ImageBitmap feature requests
Well, you could assign a unique sequential identifier or GUID to ImageBitmaps, like object URLs, as long as you remove the lifetime relationship where the object has to be manually freed. That would let you do some of those caching scenarios, the key is that the lifetime is now managed by 'do any elements use this ImageBitmap as a source, or is it retained by user JS', I think? On Tue, May 20, 2014 at 1:01 PM, Justin Novosad ju...@google.com wrote: On Sun, May 18, 2014 at 11:02 PM, Robert O'Callahan rob...@ocallahan.orgwrote: On Sat, May 17, 2014 at 4:18 AM, Anne van Kesteren ann...@annevk.nlwrote: Maybe we should have img.srcObject similar to what we're doing for media elements. img.src can simply return about:imagebitmap or some such. That way you can also assign a Blob to an img element without having to do the weird createObjectURL() hack that might leak memory if you're not careful. I like this approach, but I think it's simpler to continue to have HTMLImageElement.src reflect the src content attribute. I wonder what kind of broader effect it would have if image content can no longer be uniquely identified or retrieved using a URL. In many places in Blink/WebKit (and presumably other implementations as well) URLs are used as keys and handles for image resources. All of that would have to be refactored.
Re: [whatwg] ImageBitmap feature requests
On Sat, May 17, 2014 at 4:18 AM, Anne van Kesteren ann...@annevk.nl wrote: Maybe we should have img.srcObject similar to what we're doing for media elements. img.src can simply return about:imagebitmap or some such. That way you can also assign a Blob to an img element without having to do the weird createObjectURL() hack that might leak memory if you're not careful. I like this approach, but I think it's simpler to continue to have HTMLImageElement.src reflect the src content attribute. Rob -- Jtehsauts tshaei dS,o n Wohfy Mdaon yhoaus eanuttehrotraiitny eovni le atrhtohu gthot sf oirng iyvoeu rs ihnesa.rt sS?o Whhei csha iids teoa stiheer :p atroa lsyazye,d 'mYaonu,r sGients uapr,e tfaokreg iyvoeunr, 'm aotr atnod sgaoy ,h o'mGee.t uTph eann dt hwea lmka'n? gBoutt uIp waanndt wyeonut thoo mken.o w
Re: [whatwg] ImageBitmap feature requests
On Fri, May 16, 2014 at 7:47 PM, Justin Novosad ju...@google.com wrote: On Fri, May 9, 2014 at 5:51 PM, Ian Hickson i...@hixie.ch wrote: This is an interesting idea. I don't know what the state of the other methods discussed here is (see my comment at the top of the e-mail cited above). However, I've filed a bug for this too: https://www.w3.org/Bugs/Public/show_bug.cgi?id=25647 I think what might make the most sense here is to have a way to destructively convert an ImageBitmap into an img, rather than doing anything with a canvas. What would go in the 'src' attribute of the image element? A a data URL? Maybe we should have img.srcObject similar to what we're doing for media elements. img.src can simply return about:imagebitmap or some such. That way you can also assign a Blob to an img element without having to do the weird createObjectURL() hack that might leak memory if you're not careful. -- http://annevankesteren.nl/
Re: [whatwg] ImageBitmap feature requests
On Fri, May 9, 2014 at 5:51 PM, Ian Hickson i...@hixie.ch wrote: On Thu, 18 Jul 2013, Justin Novosad wrote: However, one weakness of the current API is that upon construction of the ImageBitmap, the browser does not know whether the asset will be used with a GPU-accelerated rendering context or not. If this information were available, the asset could be pre-cached on the GPU when appropriate. Maybe something like ctx.prefetch(image) would be appropriate for warming up the caches. Is this a measurable performance problem currently? I'd hate to provide such an API, which could easily be misunderstood or misused, only to find that in practice things already work ok. Agreed. A slower first draw is often tolerable. And the performance hit of a GPU texture upload is usually quite small relative to decode, disk cache access and network access. On Fri, 19 Jul 2013, Justin Novosad wrote: A related issue we have now is with canvas backing stores. It is common for web apps to create temporary canvases to do some offscreen rendering. When the temporary canvas goes out of scope, it continues to consume RAM or GPU memory until it is garbage collected. Occasionally this results in memory-leak-like symptoms. The usual workaround is to use a single persistent global canvas for offscreen work instead of temporary ones (yuck). This could be handled in a cleaner way if there were a .discard() method on canvases elements too. Would setting the canvas dimensions to zero have the same effect? Absolutely. And I have given that tip to devs many times. It is just that finding that trick is non-obvious to a dev who is unaware of how the browser is implemented internally, which is usually the case. We could have a method that just sets the dimensions to zero, if so, and if this is common enough to warrant a convenience method. Yes. A dispose() would feel less hacky and developers would find it more easily. 5) Once a large image is decoded and downsampled into a smaller ImageBitmap, the only thing that we can do with that ImageBitmap is to copy it into a Canvas, either for display to the end user (as an alternative to an img) or for re-encoding with Canvas.toBlob() (when creating thumbnails for large images). The motivation for this downsampling feature is memory use. But having to copy an ImageBitmap into a canvas in order to use it immediately doubles the amount of memory required. So for this reason, I also want to propose that ImageBitmap have a transferToCanvas() method akin to the transferToImageBitmap() and transferToImage() methods proposed at http://wiki.whatwg.org/wiki/WorkerCanvas. transferToCanvas would transfer the image data into a new Canvas object and would neuter the ImageBitmap so that it could no longer be used. This is an interesting idea. I don't know what the state of the other methods discussed here is (see my comment at the top of the e-mail cited above). However, I've filed a bug for this too: https://www.w3.org/Bugs/Public/show_bug.cgi?id=25647 I think what might make the most sense here is to have a way to destructively convert an ImageBitmap into an img, rather than doing anything with a canvas. What would go in the 'src' attribute of the image element? A a data URL? -- Ian Hickson U+1047E)\._.,--,'``.fL http://ln.hixie.ch/ U+263A/, _.. \ _\ ;`._ ,. Things that are impossible just take longer. `._.-(,_..'--(,_..'`-.;.'
[whatwg] ImageBitmap feature requests
On Thu, 18 Jul 2013, Justin Novosad wrote: On Thu, Jul 18, 2013 at 12:50 PM, Ian Hickson i...@hixie.ch wrote: On Wed, 9 Jan 2013, Ashley Gullen wrote: Some developers are starting to design large scale games using our HTML5 game engine, and we're finding we're running in to memory management issues. Consider a device with 50mb of texture memory available. A game might contain 100mb of texture assets, but only use a maximum of 30mb of them at a time (e.g. if there are three levels each using 30mb of different assets, and a menu that uses 10mb of assets). This game ought to fit in memory at all times, but if a user agent is not smart about how image loading is handled, it could run out of memory. The Web API tries to use garbage collection for this; the idea being that you load the images you need when you need them, then discard then when you're done, and the memory gets reclaimed when possible. This is probably an area where most browsers could do a better job. Browsers should be able to handle the texture memory issues automatically without any new APIs, if they can't, then file bug reports. If garbage collection is not kicking-in at the right time, report it to the vendor. ImageBitmap should provide the same kind of pinning semantics as the suggested ctx.load/unload. This is good to know. If you are an author finding these problems, please do file bugs! However, one weakness of the current API is that upon construction of the ImageBitmap, the browser does not know whether the asset will be used with a GPU-accelerated rendering context or not. If this information were available, the asset could be pre-cached on the GPU when appropriate. Maybe something like ctx.prefetch(image) would be appropriate for warming up the caches. Is this a measurable performance problem currently? I'd hate to provide such an API, which could easily be misunderstood or misused, only to find that in practice things already work ok. On Fri, 19 Jul 2013, Ashley Gullen wrote: FWIW, imageBitmap.discard() wouldn't be unprecedented - WebGL allows you to explicitly release memory with deleteTexture() rather than letting the GC collect unused textures. What has implementation experience been with this API? Is it misused much? On Fri, 19 Jul 2013, Justin Novosad wrote: A related issue we have now is with canvas backing stores. It is common for web apps to create temporary canvases to do some offscreen rendering. When the temporary canvas goes out of scope, it continues to consume RAM or GPU memory until it is garbage collected. Occasionally this results in memory-leak-like symptoms. The usual workaround is to use a single persistent global canvas for offscreen work instead of temporary ones (yuck). This could be handled in a cleaner way if there were a .discard() method on canvases elements too. Would setting the canvas dimensions to zero have the same effect? We could have a method that just sets the dimensions to zero, if so, and if this is common enough to warrant a convenience method. On Fri, 19 Jul 2013, K. Gadd wrote: Some of my applications would definitely benefit from this as well. A port of one client's game managed to hit around 1GB of backing store/bitmap data combined when preloading all their image assets using img. Even though browsers then discard the bitmap data, it made it difficult to get things running without killing a tab due to hitting a memory limit temporarily. (The assets were not all in use at once, so the actual usage while playing is fine). Having explicit control over whether bitmaps are resident in memory would be great for this use case since I can preload the actual file over the network, then do the actual async forced decode by creating an ImageBitmap from a Blob, and discard it when the pixel data is no longer needed (the game already has this information since it uses the C# IDisposable pattern, where resources are disposed after use) Well, browsers should be more aggressive about garbage collecting if the lack of having garbage collected is causing performance issues due to lack of RAM, no? Have you filed any bugs on browsers for this? Justin's comments above suggests we should maybe start with that. On Tue, 13 Aug 2013, Kenneth Russell wrote: We could have a constructor for ImageData objects, sure. That would be relatively easy to add, if it's really needed. I don't understand why it's hard to keep track of ImageData objects, though. Can you elaborate? I have in mind new APIs for typed arrays which allow sharding of typed arrays to workers and re-assembly of the component pieces when the work is complete. This would involve multiple manipulations of the ArrayBuffer and its views. It would be most convenient if the result could be wrapped in an ImageData if it's destined to be drawn to a Canvas. Otherwise it's likely that a
