Re: Subject=Re: Async Image - ImageData conversion
Good point - it makes sense to have ImageBitmap as the hub of all conversion. I've drafted a new spec which instead proposes ImageBitmap.toBlob() and ImageBitmap.toImageData(): https://www.scirra.com/labs/specs/imagebitmap-conversion-extensions.html This should cover all conversion cases asynchronously. On 23 June 2015 at 20:34, Justin Novosad ju...@google.com wrote: Based on feedback received from web developers, new APIs that move image data around should also strive to eliminate intermediate copies of the image data to avoid memory bloat. I think your proposal achieves that, but I think memory footprint reduction could be a stated objective of the proposal. For example, the current solution of using a canvas forces the creation of an intermediate copy (the canvas). Also, to avoid the multiplication of APIs for moving image data between various representations, I would like to suggest an alternative: using the ImageBitmap interface as a hub. The creation of ImageBitmaps is already asynchronous (createImageBitmap returns a promise), and it has overloads for acquiring images from img, video, canvas, url, blob, imagedata. All that is missing are a few methods for getting data directly out of an ImageBitmap. So I think adding toBlob and toImageData (both async) to ImageBitmap is a more succinct proposal that would address your use cases, and many additional ones at the same time.
Detecting image encoding/decoding support
While drafting https://www.scirra.com/labs/specs/imagebitmap-conversion-extensions.html I realised that there is no way to tell in JS what image formats the browser can decode (with Image or ImageBitmap objects) or encode (with canvas toDataURL()/toBlob() or the ImageBitmap.toBlob() method I was proposing). HTMLMediaElement.canPlayType() can give an indication if an audio or video format can be decoded. MediaRecorder.canRecordMimeType() can give an indication if an audio or video format can be encoded. Should we not have the same for images? For example there are a number of image formats beyond PNG and JPEG which some browsers support, such as: - JPEG 2000 (Safari) - JPEG XR (IE/Edge) - WebP (Blink) - APNG (Firefox, Safari) - any other future formats Most client-side detection I've seen involves either trying to load a data URI included in the source, or using canvas' toDataURL() and seeing if the resulting data URI includes the expected MIME type. I think there should be methods on some object (be it Image, ImageBitmap or Canvas) for: - canDecodeType(): indicate if an image format can be encoded by Image/ImageBitmap objects - canEncodeType(): indicate if an image format can be encoded by Canvas toDataURL/toBlob (or my proposed ImageBitmap.toBlob()) Thoughts?
Re: Async Image - ImageData conversion
Sorry for the confusion. Yes, the latest URL is: https://www.scirra.com/labs/specs/imagebitmap-conversion-extensions.html I'm new to specs and WebIDL, my intent was to say those are new methods on ImageBitmap. Is partial interface ImageBitmap the correct way to say that? (I updated the URL to say that instead) The wording of undue latency in the ImageBitmap spec does not appear to rule out ImageBitmap storing its encoded image data, and lazily decoding it. However I'm not sure what the point of that would be, since it seems to be how drawing HTMLImageElements to a canvas already works. I see a couple of options: - allow lazy decoding in ImageBitmap, and therefore converting to ImageData is an explicit request to decompress - require ImageBitmap to decompress its contents, and make it available through a read-only Uint8ClampedArray like ImageData has. Therefore, converting to ImageData indicates the intent to modify this content, therefore a copy is warranted. - provide a way to neuter the ImageBitmap when converting it to ImageData, transferring its contents and avoiding any copy, which is useful if the ImageBitmap is a temporary object only being created for the purposes of getting the ImageData. I guess this would be similar to transferrable objects with workers. - add some kind of load method on ImageBitmap. It may store its contents in compressed form, but calling load causes it to be decompressed (or loaded from somewhere else). The ImageBitmap is only guaranteed to be drawn with undue latency after the load call. Therefore for the use case of low-latency rendering load can always be called immediately after creation, but for conversion purposes not calling load avoids duplicating memory. Note for devices with discrete GPUs, it's possible that ImageBitmap stores the decompressed image data in a GPU texture but does not have it in system RAM. In this case making a copy for the ImageData is also warranted. I'm a web developer proposing this because it would be useful for my purposes, I've no idea which of these would be favoured by implementors or the spec process. I'm happy to get involved in spec work though so please let me know if you have any feedback/suggestions on anything I'm doing here. Ashley On 24 June 2015 at 19:12, Boris Zbarsky bzbar...@mit.edu wrote: On 6/19/15 5:43 AM, Ashley Gullen wrote: I've not done this before, so I've no idea if this is the right/useful approach, but I drafted a spec for it here: https://www.scirra.com/labs/specs/imagedata-blob-extensions.html Ashley, We at Mozilla were just discussing this proposal, and we have a concern. With this proposal, going from an img to an ImageData requires conversion of an intermediate ImageBitmap. Unfortunately, an ImageBitmap is specified to be paintable without undue latency, which we interpret to mean it needs to have decoded image data, and the ImageData will end up needing to copy said data in practice (because it needs an editable copy). That creates two copies of the decoded image data in memory, which seems fairly undesirable on memory-constrained devices. -Boris
Re: Async Image - ImageData conversion
On Wed, Jun 24, 2015 at 1:28 PM, Ashley Gullen ash...@scirra.com wrote: Sorry for the confusion. Yes, the latest URL is: https://www.scirra.com/labs/specs/imagebitmap-conversion-extensions.html I'm new to specs and WebIDL, my intent was to say those are new methods on ImageBitmap. Is partial interface ImageBitmap the correct way to say that? (I updated the URL to say that instead) The wording of undue latency in the ImageBitmap spec does not appear to rule out ImageBitmap storing its encoded image data, and lazily decoding it. However I'm not sure what the point of that would be, since it seems to be how drawing HTMLImageElements to a canvas already works. I see a couple of options: - allow lazy decoding in ImageBitmap, and therefore converting to ImageData is an explicit request to decompress - require ImageBitmap to decompress its contents, and make it available through a read-only Uint8ClampedArray like ImageData has. Therefore, converting to ImageData indicates the intent to modify this content, therefore a copy is warranted. - provide a way to neuter the ImageBitmap when converting it to ImageData, transferring its contents and avoiding any copy, which is useful if the ImageBitmap is a temporary object only being created for the purposes of getting the ImageData. I guess this would be similar to transferrable objects with workers. Perhaps this could be done by specifying a transferToImageData method. This proposal makes ImageBitmap neuterable, and there's intent to implement it in order to allow rendering to canvas contexts from worker threads: https://wiki.whatwg.org/wiki/OffscreenCanvas . -Ken - add some kind of load method on ImageBitmap. It may store its contents in compressed form, but calling load causes it to be decompressed (or loaded from somewhere else). The ImageBitmap is only guaranteed to be drawn with undue latency after the load call. Therefore for the use case of low-latency rendering load can always be called immediately after creation, but for conversion purposes not calling load avoids duplicating memory. Note for devices with discrete GPUs, it's possible that ImageBitmap stores the decompressed image data in a GPU texture but does not have it in system RAM. In this case making a copy for the ImageData is also warranted. I'm a web developer proposing this because it would be useful for my purposes, I've no idea which of these would be favoured by implementors or the spec process. I'm happy to get involved in spec work though so please let me know if you have any feedback/suggestions on anything I'm doing here. Ashley On 24 June 2015 at 19:12, Boris Zbarsky bzbar...@mit.edu wrote: On 6/19/15 5:43 AM, Ashley Gullen wrote: I've not done this before, so I've no idea if this is the right/useful approach, but I drafted a spec for it here: https://www.scirra.com/labs/specs/imagedata-blob-extensions.html Ashley, We at Mozilla were just discussing this proposal, and we have a concern. With this proposal, going from an img to an ImageData requires conversion of an intermediate ImageBitmap. Unfortunately, an ImageBitmap is specified to be paintable without undue latency, which we interpret to mean it needs to have decoded image data, and the ImageData will end up needing to copy said data in practice (because it needs an editable copy). That creates two copies of the decoded image data in memory, which seems fairly undesirable on memory-constrained devices. -Boris
Re: Clipboard API: remove dangerous formats from mandatory data types
Hallvord, Yes, content would be limited to providing text, image etc data to the user agent to place on the clipboard, and letting the user agent synthesize whatever formats (JPEG, PNG etc) other apps require. That has the advantage of preventing malicious content using esoteric flags or features to compromise recipients, but conversely means that legitimate content cannot use format-specific features, e.g. content would not be able to write a JPEG containing a comment block, geo tags or timestamp information. Wez On Sat, 13 Jun 2015 at 11:57 Hallvord Reiar Michaelsen Steen hst...@mozilla.com wrote: On Thu, Jun 11, 2015 at 7:51 PM, Wez w...@google.com wrote: Hallvord, The proposal isn't to remove support for copying/pasting images, but to restrict web content from placing compressed image data in one of these formats on the clipboard directly - there's no issue with content pasting raw pixels from a canvas, for example, since scope for abusing that to compromise the recipient is extremely limited by comparison to JPEG, PNG or GIF. Well, but as far as I can tell we don't currently *have* a way JS can place pixels from a canvas on the clipboard (except by putting a piece of data labelled as image/png or similar there). So if you're pushing back against the idea that JS can place random data on the clipboard and label it image/png, how exactly would you propose JS-triggered copy of image data to work? Say, from a CANVAS-based image editor? -Hallvord
Re: Clipboard API: remove dangerous formats from mandatory data types
And how exactly do you intend to support for instance OpenEXR? On Wed, Jun 24, 2015 at 5:44 PM, Wez w...@google.com wrote: Hallvord, Yes, content would be limited to providing text, image etc data to the user agent to place on the clipboard, and letting the user agent synthesize whatever formats (JPEG, PNG etc) other apps require. That has the advantage of preventing malicious content using esoteric flags or features to compromise recipients, but conversely means that legitimate content cannot use format-specific features, e.g. content would not be able to write a JPEG containing a comment block, geo tags or timestamp information. Wez On Sat, 13 Jun 2015 at 11:57 Hallvord Reiar Michaelsen Steen hst...@mozilla.com wrote: On Thu, Jun 11, 2015 at 7:51 PM, Wez w...@google.com wrote: Hallvord, The proposal isn't to remove support for copying/pasting images, but to restrict web content from placing compressed image data in one of these formats on the clipboard directly - there's no issue with content pasting raw pixels from a canvas, for example, since scope for abusing that to compromise the recipient is extremely limited by comparison to JPEG, PNG or GIF. Well, but as far as I can tell we don't currently *have* a way JS can place pixels from a canvas on the clipboard (except by putting a piece of data labelled as image/png or similar there). So if you're pushing back against the idea that JS can place random data on the clipboard and label it image/png, how exactly would you propose JS-triggered copy of image data to work? Say, from a CANVAS-based image editor? -Hallvord
Re: Async Image - ImageData conversion
On 6/19/15 5:43 AM, Ashley Gullen wrote: I've not done this before, so I've no idea if this is the right/useful approach, but I drafted a spec for it here: https://www.scirra.com/labs/specs/imagedata-blob-extensions.html Let me know if you have any feedback on this. Ashley, I assume the relevant link is now https://www.scirra.com/labs/specs/imagebitmap-conversion-extensions.html, right? I'm trying to understand where the methods are being added. The IDL says partial interface ImageBitmapConversion but there is no current ImageBitmapConversion interface I'm aware of. Is this supposed to be partial interface ImageBitmap? -Boris
Re: Clipboard API: remove dangerous formats from mandatory data types
I don't think OpenEXR is one of the formats required by the Clipboard Events spec, is it..? On Wed, Jun 24, 2015, 18:49 Florian Bösch pya...@gmail.com wrote: And how exactly do you intend to support for instance OpenEXR? On Wed, Jun 24, 2015 at 5:44 PM, Wez w...@google.com wrote: Hallvord, Yes, content would be limited to providing text, image etc data to the user agent to place on the clipboard, and letting the user agent synthesize whatever formats (JPEG, PNG etc) other apps require. That has the advantage of preventing malicious content using esoteric flags or features to compromise recipients, but conversely means that legitimate content cannot use format-specific features, e.g. content would not be able to write a JPEG containing a comment block, geo tags or timestamp information. Wez On Sat, 13 Jun 2015 at 11:57 Hallvord Reiar Michaelsen Steen hst...@mozilla.com wrote: On Thu, Jun 11, 2015 at 7:51 PM, Wez w...@google.com wrote: Hallvord, The proposal isn't to remove support for copying/pasting images, but to restrict web content from placing compressed image data in one of these formats on the clipboard directly - there's no issue with content pasting raw pixels from a canvas, for example, since scope for abusing that to compromise the recipient is extremely limited by comparison to JPEG, PNG or GIF. Well, but as far as I can tell we don't currently *have* a way JS can place pixels from a canvas on the clipboard (except by putting a piece of data labelled as image/png or similar there). So if you're pushing back against the idea that JS can place random data on the clipboard and label it image/png, how exactly would you propose JS-triggered copy of image data to work? Say, from a CANVAS-based image editor? -Hallvord
Re: Clipboard API: remove dangerous formats from mandatory data types
No, but the specification doesn't require you to exclude it. So how're applications going to swap OpenEXR if you only let em stick in jpegs, pngs and gifs? On Wed, Jun 24, 2015 at 8:46 PM, Wez w...@google.com wrote: I don't think OpenEXR is one of the formats required by the Clipboard Events spec, is it..? On Wed, Jun 24, 2015, 18:49 Florian Bösch pya...@gmail.com wrote: And how exactly do you intend to support for instance OpenEXR? On Wed, Jun 24, 2015 at 5:44 PM, Wez w...@google.com wrote: Hallvord, Yes, content would be limited to providing text, image etc data to the user agent to place on the clipboard, and letting the user agent synthesize whatever formats (JPEG, PNG etc) other apps require. That has the advantage of preventing malicious content using esoteric flags or features to compromise recipients, but conversely means that legitimate content cannot use format-specific features, e.g. content would not be able to write a JPEG containing a comment block, geo tags or timestamp information. Wez On Sat, 13 Jun 2015 at 11:57 Hallvord Reiar Michaelsen Steen hst...@mozilla.com wrote: On Thu, Jun 11, 2015 at 7:51 PM, Wez w...@google.com wrote: Hallvord, The proposal isn't to remove support for copying/pasting images, but to restrict web content from placing compressed image data in one of these formats on the clipboard directly - there's no issue with content pasting raw pixels from a canvas, for example, since scope for abusing that to compromise the recipient is extremely limited by comparison to JPEG, PNG or GIF. Well, but as far as I can tell we don't currently *have* a way JS can place pixels from a canvas on the clipboard (except by putting a piece of data labelled as image/png or similar there). So if you're pushing back against the idea that JS can place random data on the clipboard and label it image/png, how exactly would you propose JS-triggered copy of image data to work? Say, from a CANVAS-based image editor? -Hallvord
