Re: [whatwg] I believe source rectangles for HTML5 Canvas drawImage are specified incorrectly
On Thu, Dec 20, 2012 at 7:33 AM, Rik Cabanier wrote: > Would it also handled HD backing stores automatically, like drawImage > does? > I think so. Rob -- Jesus called them together and said, “You know that the rulers of the Gentiles lord it over them, and their high officials exercise authority over them. Not so with you. Instead, whoever wants to become great among you must be your servant, and whoever wants to be first must be your slave — just as the Son of Man did not come to be served, but to serve, and to give his life as a ransom for many.” [Matthew 20:25-28]
Re: [whatwg] I believe source rectangles for HTML5 Canvas drawImage are specified incorrectly
On Wed, Dec 19, 2012 at 1:29 AM, Robert O'Callahan wrote: > On Wed, Dec 19, 2012 at 5:39 PM, Rik Cabanier wrote: > >> On Tue, Dec 18, 2012 at 1:14 PM, Robert O'Callahan >> wrote: >> >>> It seems to me a synchronous constructor that creates an ImageBitmap for >>> a subimage of another ImageBitmap (but not other kind of source!) will be >>> needed. >>> >>> >> Did you mean to say a synchronous constructor? >> > > Yes. That's what I said. > > >> Reading the new spec ( >> http://www.whatwg.org/specs/web-apps/current-work/#imagebitmap), I still >> fail to see how this is better than a drawimage with dx=dy=0 and >> dw/dy=canvas width/height >> > > You mean, how is using createImageBitmap better than drawing the input > ImageBitmap into a temporary canvas and then using that canvas instead of > the result of createImageBitmap? > > Basically, createImageBitmap is a more direct approach that's easier to > optimize to avoid copying the image data. Would it also handled HD backing stores automatically, like drawImage does?
Re: [whatwg] I believe source rectangles for HTML5 Canvas drawImage are specified incorrectly
On Wed, Dec 19, 2012 at 5:39 PM, Rik Cabanier wrote: > On Tue, Dec 18, 2012 at 1:14 PM, Robert O'Callahan > wrote: > >> It seems to me a synchronous constructor that creates an ImageBitmap for >> a subimage of another ImageBitmap (but not other kind of source!) will be >> needed. >> >> > Did you mean to say a synchronous constructor? > Yes. That's what I said. > Reading the new spec ( > http://www.whatwg.org/specs/web-apps/current-work/#imagebitmap), I still > fail to see how this is better than a drawimage with dx=dy=0 and > dw/dy=canvas width/height > You mean, how is using createImageBitmap better than drawing the input ImageBitmap into a temporary canvas and then using that canvas instead of the result of createImageBitmap? Basically, createImageBitmap is a more direct approach that's easier to optimize to avoid copying the image data. Rob -- Jesus called them together and said, “You know that the rulers of the Gentiles lord it over them, and their high officials exercise authority over them. Not so with you. Instead, whoever wants to become great among you must be your servant, and whoever wants to be first must be your slave — just as the Son of Man did not come to be served, but to serve, and to give his life as a ransom for many.” [Matthew 20:25-28]
Re: [whatwg] I believe source rectangles for HTML5 Canvas drawImage are specified incorrectly
On Tue, Dec 18, 2012 at 1:14 PM, Robert O'Callahan wrote: > On Wed, Dec 19, 2012 at 4:17 AM, Kevin Gadd wrote: > >> Sorry, upon reading over the ImageBitmap part of the spec again I'm >> confused: Why is constructing an ImageBitmap asynchronous? I thought >> any decoding isn't supposed to happen until drawImage, so I don't >> really understand why this operation involves a callback and a delay. >> Making ImageBitmap creation async means that you *cannot* use this as >> a replacement for drawImage source rectangles unless you know all >> possible source rectangles in advance. This is not possible for many, >> many use cases (scrolling through a bitmap would be one trivial >> example). >> > > I think this is a real problem. > > It seems to me a synchronous constructor that creates an ImageBitmap for a > subimage of another ImageBitmap (but not other kind of source!) will be > needed. > > Did you mean to say a synchronous constructor? Reading the new spec ( http://www.whatwg.org/specs/web-apps/current-work/#imagebitmap), I still fail to see how this is better than a drawimage with dx=dy=0 and dw/dy=canvas width/height
Re: [whatwg] I believe source rectangles for HTML5 Canvas drawImage are specified incorrectly
On Wed, Dec 19, 2012 at 4:17 AM, Kevin Gadd wrote: > Sorry, upon reading over the ImageBitmap part of the spec again I'm > confused: Why is constructing an ImageBitmap asynchronous? I thought > any decoding isn't supposed to happen until drawImage, so I don't > really understand why this operation involves a callback and a delay. > Making ImageBitmap creation async means that you *cannot* use this as > a replacement for drawImage source rectangles unless you know all > possible source rectangles in advance. This is not possible for many, > many use cases (scrolling through a bitmap would be one trivial > example). > I think this is a real problem. It seems to me a synchronous constructor that creates an ImageBitmap for a subimage of another ImageBitmap (but not other kind of source!) will be needed. Rob -- Jesus called them together and said, “You know that the rulers of the Gentiles lord it over them, and their high officials exercise authority over them. Not so with you. Instead, whoever wants to become great among you must be your servant, and whoever wants to be first must be your slave — just as the Son of Man did not come to be served, but to serve, and to give his life as a ransom for many.” [Matthew 20:25-28]
Re: [whatwg] I believe source rectangles for HTML5 Canvas drawImage are specified incorrectly
> > > >> Each time you wait for that callback, won't any pending drawing >> operations to your canvas get flushed, resulting in the scene slowly >> rendering onscreen? >> > > Another great reason to batch the calls, as described above. > Another solution is to use an offscreen canvas. > > >> >> -kg >> >> On Tue, Dec 18, 2012 at 7:39 AM, Justin Novosad >> wrote: >> > On Tue, Dec 18, 2012 at 10:17 AM, Kevin Gadd >> wrote: >> >> >> >> Sorry, upon reading over the ImageBitmap part of the spec again I'm >> >> confused: Why is constructing an ImageBitmap asynchronous? I thought >> >> any decoding isn't supposed to happen until drawImage, so I don't >> >> really understand why this operation involves a callback and a delay. >> > >> > >> > The image may have to be downloaded, which is a good reason for being >> async. >> > >> >> >> >> Making ImageBitmap creation async means that you *cannot* use this as >> >> a replacement for drawImage source rectangles unless you know all >> >> possible source rectangles in advance. This is not possible for many, >> >> many use cases (scrolling through a bitmap would be one trivial >> >> example). >> > >> > >> > I am not sure I understand the problem. If you need synchronous >> behavior, >> > you can just make the app wait for the callback to be called. >> > >> > -Justin >> > >> >> >> >> On Tue, Dec 18, 2012 at 7:12 AM, Kevin Gadd >> wrote: >> >> > Is it possible to expose the width/height of an ImageBitmap, or even >> >> > expose all the rectangle coordinates? Exposing width/height would be >> >> > nice for parity with Image and Canvas when writing functions that >> >> > accept any drawable image source. >> >> > >> >> > Thanks for the prompt action here, this looks like a straightforward >> >> > solution. >> >> > >> >> > -kg >> >> > >> >> > On Tue, Dec 18, 2012 at 7:03 AM, Justin Novosad >> >> > wrote: >> >> >> Thanks Ian. >> >> >> http://html5.org/tools/web-apps-tracker?from=7588&to=7589 >> >> >> >> >> >> On Tue, Dec 18, 2012 at 1:04 AM, Robert O'Callahan >> >> >> >> >> >> wrote: >> >> >>> >> >> >>> On Tue, Dec 18, 2012 at 6:57 PM, Rik Cabanier >> >> >>> wrote: >> >> >> >> Does it matter that this solution is in the image world and not 2d >> >> canvas? It seems that this will have a bigger impact on the >> browser's >> >> codebase since it spans images and canvas. >> >> >>> >> >> >>> >> >> >>> I think that's a good thing --- it's more generally useful. >> >> >>> >> >> >>> Note that Ian has adopted this approach (via an ImageBitmap >> >> >>> constructor). >> >> >>> >> >> >>> >> >> >>> Rob >> >> >>> -- >> >> >>> Jesus called them together and said, “You know that the rulers of >> the >> >> >>> Gentiles lord it over them, and their high officials exercise >> >> >>> authority over >> >> >>> them. Not so with you. Instead, whoever wants to become great among >> >> >>> you must >> >> >>> be your servant, and whoever wants to be first must be your slave — >> >> >>> just as >> >> >>> the Son of Man did not come to be served, but to serve, and to give >> >> >>> his life >> >> >>> as a ransom for many.” [Matthew 20:25-28] >> >> >>> >> >> >> >> >> > >> >> > >> >> > >> >> > -- >> >> > -kg >> >> >> >> >> >> >> >> -- >> >> -kg >> > >> > >> >> >> >> -- >> -kg >> > >
Re: [whatwg] I believe source rectangles for HTML5 Canvas drawImage are specified incorrectly
On Tue, Dec 18, 2012 at 11:03 AM, Kevin Gadd wrote: > How do you wait synchronously for a callback from inside > requestAnimationFrame? If you are inside a rAF callback, you exit the callback, and resume drawing in the createBitmapImage callback. > I'm not aware of a way to do this in the spec. > Furthermore, wouldn't that mean returning once to the event loop for > each individual drawImage call you wish to make using a source > rectangle - so for a single scene containing lots of dynamic source > rectangles you could end up having to wait for dozens of callbacks. > You make all the createBitmapImage calls you need, then exit (return to event loop). In the createBitmapImage callback you track completion of all the createBitmapImage calls. This can be done using a common callback function and a counter. If not all bitmaps are ready, the callback exits immediately. Once you know all the bitmaps are ready, draw. This way, all the createBitmapImage calls run concurrently. This can a big performance win when the source images are remote and network latency is a factor. > Each time you wait for that callback, won't any pending drawing > operations to your canvas get flushed, resulting in the scene slowly > rendering onscreen? > Another great reason to batch the calls, as described above. > > -kg > > On Tue, Dec 18, 2012 at 7:39 AM, Justin Novosad > wrote: > > On Tue, Dec 18, 2012 at 10:17 AM, Kevin Gadd > wrote: > >> > >> Sorry, upon reading over the ImageBitmap part of the spec again I'm > >> confused: Why is constructing an ImageBitmap asynchronous? I thought > >> any decoding isn't supposed to happen until drawImage, so I don't > >> really understand why this operation involves a callback and a delay. > > > > > > The image may have to be downloaded, which is a good reason for being > async. > > > >> > >> Making ImageBitmap creation async means that you *cannot* use this as > >> a replacement for drawImage source rectangles unless you know all > >> possible source rectangles in advance. This is not possible for many, > >> many use cases (scrolling through a bitmap would be one trivial > >> example). > > > > > > I am not sure I understand the problem. If you need synchronous behavior, > > you can just make the app wait for the callback to be called. > > > > -Justin > > > >> > >> On Tue, Dec 18, 2012 at 7:12 AM, Kevin Gadd > wrote: > >> > Is it possible to expose the width/height of an ImageBitmap, or even > >> > expose all the rectangle coordinates? Exposing width/height would be > >> > nice for parity with Image and Canvas when writing functions that > >> > accept any drawable image source. > >> > > >> > Thanks for the prompt action here, this looks like a straightforward > >> > solution. > >> > > >> > -kg > >> > > >> > On Tue, Dec 18, 2012 at 7:03 AM, Justin Novosad > >> > wrote: > >> >> Thanks Ian. > >> >> http://html5.org/tools/web-apps-tracker?from=7588&to=7589 > >> >> > >> >> On Tue, Dec 18, 2012 at 1:04 AM, Robert O'Callahan > >> >> > >> >> wrote: > >> >>> > >> >>> On Tue, Dec 18, 2012 at 6:57 PM, Rik Cabanier > >> >>> wrote: > >> > >> Does it matter that this solution is in the image world and not 2d > >> canvas? It seems that this will have a bigger impact on the > browser's > >> codebase since it spans images and canvas. > >> >>> > >> >>> > >> >>> I think that's a good thing --- it's more generally useful. > >> >>> > >> >>> Note that Ian has adopted this approach (via an ImageBitmap > >> >>> constructor). > >> >>> > >> >>> > >> >>> Rob > >> >>> -- > >> >>> Jesus called them together and said, “You know that the rulers of > the > >> >>> Gentiles lord it over them, and their high officials exercise > >> >>> authority over > >> >>> them. Not so with you. Instead, whoever wants to become great among > >> >>> you must > >> >>> be your servant, and whoever wants to be first must be your slave — > >> >>> just as > >> >>> the Son of Man did not come to be served, but to serve, and to give > >> >>> his life > >> >>> as a ransom for many.” [Matthew 20:25-28] > >> >>> > >> >> > >> > > >> > > >> > > >> > -- > >> > -kg > >> > >> > >> > >> -- > >> -kg > > > > > > > > -- > -kg >
Re: [whatwg] I believe source rectangles for HTML5 Canvas drawImage are specified incorrectly
How do you wait synchronously for a callback from inside requestAnimationFrame? I'm not aware of a way to do this in the spec. Furthermore, wouldn't that mean returning once to the event loop for each individual drawImage call you wish to make using a source rectangle - so for a single scene containing lots of dynamic source rectangles you could end up having to wait for dozens of callbacks. Each time you wait for that callback, won't any pending drawing operations to your canvas get flushed, resulting in the scene slowly rendering onscreen? -kg On Tue, Dec 18, 2012 at 7:39 AM, Justin Novosad wrote: > On Tue, Dec 18, 2012 at 10:17 AM, Kevin Gadd wrote: >> >> Sorry, upon reading over the ImageBitmap part of the spec again I'm >> confused: Why is constructing an ImageBitmap asynchronous? I thought >> any decoding isn't supposed to happen until drawImage, so I don't >> really understand why this operation involves a callback and a delay. > > > The image may have to be downloaded, which is a good reason for being async. > >> >> Making ImageBitmap creation async means that you *cannot* use this as >> a replacement for drawImage source rectangles unless you know all >> possible source rectangles in advance. This is not possible for many, >> many use cases (scrolling through a bitmap would be one trivial >> example). > > > I am not sure I understand the problem. If you need synchronous behavior, > you can just make the app wait for the callback to be called. > > -Justin > >> >> On Tue, Dec 18, 2012 at 7:12 AM, Kevin Gadd wrote: >> > Is it possible to expose the width/height of an ImageBitmap, or even >> > expose all the rectangle coordinates? Exposing width/height would be >> > nice for parity with Image and Canvas when writing functions that >> > accept any drawable image source. >> > >> > Thanks for the prompt action here, this looks like a straightforward >> > solution. >> > >> > -kg >> > >> > On Tue, Dec 18, 2012 at 7:03 AM, Justin Novosad >> > wrote: >> >> Thanks Ian. >> >> http://html5.org/tools/web-apps-tracker?from=7588&to=7589 >> >> >> >> On Tue, Dec 18, 2012 at 1:04 AM, Robert O'Callahan >> >> >> >> wrote: >> >>> >> >>> On Tue, Dec 18, 2012 at 6:57 PM, Rik Cabanier >> >>> wrote: >> >> Does it matter that this solution is in the image world and not 2d >> canvas? It seems that this will have a bigger impact on the browser's >> codebase since it spans images and canvas. >> >>> >> >>> >> >>> I think that's a good thing --- it's more generally useful. >> >>> >> >>> Note that Ian has adopted this approach (via an ImageBitmap >> >>> constructor). >> >>> >> >>> >> >>> Rob >> >>> -- >> >>> Jesus called them together and said, “You know that the rulers of the >> >>> Gentiles lord it over them, and their high officials exercise >> >>> authority over >> >>> them. Not so with you. Instead, whoever wants to become great among >> >>> you must >> >>> be your servant, and whoever wants to be first must be your slave — >> >>> just as >> >>> the Son of Man did not come to be served, but to serve, and to give >> >>> his life >> >>> as a ransom for many.” [Matthew 20:25-28] >> >>> >> >> >> > >> > >> > >> > -- >> > -kg >> >> >> >> -- >> -kg > > -- -kg
Re: [whatwg] I believe source rectangles for HTML5 Canvas drawImage are specified incorrectly
On Tue, Dec 18, 2012 at 10:17 AM, Kevin Gadd wrote: > Sorry, upon reading over the ImageBitmap part of the spec again I'm > confused: Why is constructing an ImageBitmap asynchronous? I thought > any decoding isn't supposed to happen until drawImage, so I don't > really understand why this operation involves a callback and a delay. > The image may have to be downloaded, which is a good reason for being async. > Making ImageBitmap creation async means that you *cannot* use this as > a replacement for drawImage source rectangles unless you know all > possible source rectangles in advance. This is not possible for many, > many use cases (scrolling through a bitmap would be one trivial > example). > I am not sure I understand the problem. If you need synchronous behavior, you can just make the app wait for the callback to be called. -Justin > On Tue, Dec 18, 2012 at 7:12 AM, Kevin Gadd wrote: > > Is it possible to expose the width/height of an ImageBitmap, or even > > expose all the rectangle coordinates? Exposing width/height would be > > nice for parity with Image and Canvas when writing functions that > > accept any drawable image source. > > > > Thanks for the prompt action here, this looks like a straightforward > solution. > > > > -kg > > > > On Tue, Dec 18, 2012 at 7:03 AM, Justin Novosad > wrote: > >> Thanks Ian. > >> http://html5.org/tools/web-apps-tracker?from=7588&to=7589 > >> > >> On Tue, Dec 18, 2012 at 1:04 AM, Robert O'Callahan < > rob...@ocallahan.org> > >> wrote: > >>> > >>> On Tue, Dec 18, 2012 at 6:57 PM, Rik Cabanier > wrote: > > Does it matter that this solution is in the image world and not 2d > canvas? It seems that this will have a bigger impact on the browser's > codebase since it spans images and canvas. > >>> > >>> > >>> I think that's a good thing --- it's more generally useful. > >>> > >>> Note that Ian has adopted this approach (via an ImageBitmap > constructor). > >>> > >>> > >>> Rob > >>> -- > >>> Jesus called them together and said, “You know that the rulers of the > >>> Gentiles lord it over them, and their high officials exercise > authority over > >>> them. Not so with you. Instead, whoever wants to become great among > you must > >>> be your servant, and whoever wants to be first must be your slave — > just as > >>> the Son of Man did not come to be served, but to serve, and to give > his life > >>> as a ransom for many.” [Matthew 20:25-28] > >>> > >> > > > > > > > > -- > > -kg > > > > -- > -kg >
Re: [whatwg] I believe source rectangles for HTML5 Canvas drawImage are specified incorrectly
Sorry, upon reading over the ImageBitmap part of the spec again I'm confused: Why is constructing an ImageBitmap asynchronous? I thought any decoding isn't supposed to happen until drawImage, so I don't really understand why this operation involves a callback and a delay. Making ImageBitmap creation async means that you *cannot* use this as a replacement for drawImage source rectangles unless you know all possible source rectangles in advance. This is not possible for many, many use cases (scrolling through a bitmap would be one trivial example). Is it async because it supports using Video and Blob as the source? I really love the feature set (being able to pass ImageData in is going to be a huge boon - no more temporary canvases just to create images from pixel data!) but if it's async-only I don't know how useful it will be for the issues that led me to starting this discussion thread in the first place. -kg On Tue, Dec 18, 2012 at 7:12 AM, Kevin Gadd wrote: > Is it possible to expose the width/height of an ImageBitmap, or even > expose all the rectangle coordinates? Exposing width/height would be > nice for parity with Image and Canvas when writing functions that > accept any drawable image source. > > Thanks for the prompt action here, this looks like a straightforward solution. > > -kg > > On Tue, Dec 18, 2012 at 7:03 AM, Justin Novosad wrote: >> Thanks Ian. >> http://html5.org/tools/web-apps-tracker?from=7588&to=7589 >> >> On Tue, Dec 18, 2012 at 1:04 AM, Robert O'Callahan >> wrote: >>> >>> On Tue, Dec 18, 2012 at 6:57 PM, Rik Cabanier wrote: Does it matter that this solution is in the image world and not 2d canvas? It seems that this will have a bigger impact on the browser's codebase since it spans images and canvas. >>> >>> >>> I think that's a good thing --- it's more generally useful. >>> >>> Note that Ian has adopted this approach (via an ImageBitmap constructor). >>> >>> >>> Rob >>> -- >>> Jesus called them together and said, “You know that the rulers of the >>> Gentiles lord it over them, and their high officials exercise authority over >>> them. Not so with you. Instead, whoever wants to become great among you must >>> be your servant, and whoever wants to be first must be your slave — just as >>> the Son of Man did not come to be served, but to serve, and to give his life >>> as a ransom for many.” [Matthew 20:25-28] >>> >> > > > > -- > -kg -- -kg
Re: [whatwg] I believe source rectangles for HTML5 Canvas drawImage are specified incorrectly
Is it possible to expose the width/height of an ImageBitmap, or even expose all the rectangle coordinates? Exposing width/height would be nice for parity with Image and Canvas when writing functions that accept any drawable image source. Thanks for the prompt action here, this looks like a straightforward solution. -kg On Tue, Dec 18, 2012 at 7:03 AM, Justin Novosad wrote: > Thanks Ian. > http://html5.org/tools/web-apps-tracker?from=7588&to=7589 > > On Tue, Dec 18, 2012 at 1:04 AM, Robert O'Callahan > wrote: >> >> On Tue, Dec 18, 2012 at 6:57 PM, Rik Cabanier wrote: >>> >>> Does it matter that this solution is in the image world and not 2d >>> canvas? It seems that this will have a bigger impact on the browser's >>> codebase since it spans images and canvas. >> >> >> I think that's a good thing --- it's more generally useful. >> >> Note that Ian has adopted this approach (via an ImageBitmap constructor). >> >> >> Rob >> -- >> Jesus called them together and said, “You know that the rulers of the >> Gentiles lord it over them, and their high officials exercise authority over >> them. Not so with you. Instead, whoever wants to become great among you must >> be your servant, and whoever wants to be first must be your slave — just as >> the Son of Man did not come to be served, but to serve, and to give his life >> as a ransom for many.” [Matthew 20:25-28] >> > -- -kg
Re: [whatwg] I believe source rectangles for HTML5 Canvas drawImage are specified incorrectly
Thanks Ian. http://html5.org/tools/web-apps-tracker?from=7588&to=7589 On Tue, Dec 18, 2012 at 1:04 AM, Robert O'Callahan wrote: > On Tue, Dec 18, 2012 at 6:57 PM, Rik Cabanier wrote: > >> Does it matter that this solution is in the image world and not 2d >> canvas? It seems that this will have a bigger impact on the browser's >> codebase since it spans images and canvas. > > > I think that's a good thing --- it's more generally useful. > > Note that Ian has adopted this approach (via an ImageBitmap constructor). > > > Rob > -- > Jesus called them together and said, “You know that the rulers of the > Gentiles lord it over them, and their high officials exercise authority > over them. Not so with you. Instead, whoever wants to become great among > you must be your servant, and whoever wants to be first must be your > slave — just as the Son of Man did not come to be served, but to serve, > and to give his life as a ransom for many.” [Matthew 20:25-28] > >
Re: [whatwg] I believe source rectangles for HTML5 Canvas drawImage are specified incorrectly
On Tue, Dec 18, 2012 at 6:57 PM, Rik Cabanier wrote: > Does it matter that this solution is in the image world and not 2d canvas? > It seems that this will have a bigger impact on the browser's codebase > since it spans images and canvas. I think that's a good thing --- it's more generally useful. Note that Ian has adopted this approach (via an ImageBitmap constructor). Rob -- Jesus called them together and said, “You know that the rulers of the Gentiles lord it over them, and their high officials exercise authority over them. Not so with you. Instead, whoever wants to become great among you must be your servant, and whoever wants to be first must be your slave — just as the Son of Man did not come to be served, but to serve, and to give his life as a ransom for many.” [Matthew 20:25-28]
Re: [whatwg] I believe source rectangles for HTML5 Canvas drawImage are specified incorrectly
On Mon, Dec 17, 2012 at 2:42 PM, Robert O'Callahan wrote: > On Tue, Dec 18, 2012 at 4:23 AM, Justin Novosad wrote: > >> This gives me another >> idea: we could just have a new Image constructor that creates a new image >> element that is a subregion of another: >> var mySprite = new Image(spriteMap, x, y, w, h); >> This can be implemented in a lightweight way that just references the data >> of the source image. >> > > I like this idea. It means that implementations that need to make a copy > of the data can make a copy and associate it with the new Image object --- > which seems more straightforward than a drawSubimage call, where such > implementations would have to cache subimages internally. > > This approach also avoids having to define how subimages would behave for > video elements, canvases, etc. > > This approach is also more generally useful than a drawSubimage call or a > new drawImage parameter. > This looks like the ideal solution. Does it matter that this solution is in the image world and not 2d canvas? It seems that this will have a bigger impact on the browser's codebase since it spans images and canvas. I still think there's value in being smart when doing drawimage but the benefits might not be worth the effort. > > We would have to define how this subimage constructor works for an > animated image. I suggest it should return a non-animated subimage of the > first frame. > > I think the main risk of this approach is that it requires Web apps to opt > into it instead of changing the spec to match what Chrome is currently > doing, so it doesn't solve the immediate problem of Web apps that currently > work in Chrome not working in other browsers. Chrome changing behavior to > match the spec would solve this. Can that happen quickly, please? :-) > yes!
Re: [whatwg] I believe source rectangles for HTML5 Canvas drawImage are specified incorrectly
On Tue, Dec 18, 2012 at 4:23 AM, Justin Novosad wrote: > This gives me another > idea: we could just have a new Image constructor that creates a new image > element that is a subregion of another: > var mySprite = new Image(spriteMap, x, y, w, h); > This can be implemented in a lightweight way that just references the data > of the source image. > I like this idea. It means that implementations that need to make a copy of the data can make a copy and associate it with the new Image object --- which seems more straightforward than a drawSubimage call, where such implementations would have to cache subimages internally. This approach also avoids having to define how subimages would behave for video elements, canvases, etc. This approach is also more generally useful than a drawSubimage call or a new drawImage parameter. We would have to define how this subimage constructor works for an animated image. I suggest it should return a non-animated subimage of the first frame. I think the main risk of this approach is that it requires Web apps to opt into it instead of changing the spec to match what Chrome is currently doing, so it doesn't solve the immediate problem of Web apps that currently work in Chrome not working in other browsers. Chrome changing behavior to match the spec would solve this. Can that happen quickly, please? :-) Rob -- Jesus called them together and said, “You know that the rulers of the Gentiles lord it over them, and their high officials exercise authority over them. Not so with you. Instead, whoever wants to become great among you must be your servant, and whoever wants to be first must be your slave — just as the Son of Man did not come to be served, but to serve, and to give his life as a ransom for many.” [Matthew 20:25-28]
Re: [whatwg] I believe source rectangles for HTML5 Canvas drawImage are specified incorrectly
On Mon, Dec 17, 2012 at 1:23 PM, Rik Cabanier wrote: > > > On Mon, Dec 17, 2012 at 11:49 AM, Justin Novosad wrote: >> >> >>> Isn't this the same as what Ian suggested: copy it to a temporary canvas >>> and use the temporary canvas scales. >>> It seems that you can optimize that case too. >> >> >> Hmm... It would one of those optimizations that only works if you perform >> the secret handshake just right. >> If I understand correctly, the hypothetical optimization would work >> something like this: When a 2d canvas context is created, do not allocate a >> backing store immediately. If the first draw operation into the canvas is a >> drawImage from an image element and covers the entire canvas, then use a >> shallow copy of the bitmap as the backing store. If anything else is drawn >> into the canvas, then a proper backing store should be allocated and used. >> I guess it could work... but I would fear that this behavior would be >> perceived as idiosyncratic. > > > I don't disagree. However, there probably are other occasions in canvas > today where a common idiom gets optimized. > In addition, developers have proven that they are quite comfortable with > idiosyncrasies. For instance, many sites force their CSS animations to run > in 3D because it tricks the browser into hardware acceleration. > I think I agree in general with your thinking here, but it seems questionable whether it can be applied in this specific case - right now if you were to replace existing uses of drawImage with a temporary canvas, you'd see a dramatic performance hit in most (if not all) canvas implementations. Without a reliable way to detect that an implementation understands your 'secret handshake', you'd basically be murdering the playability of your game (or responsiveness of your application, if it's an app) in the hopes that it will run better in the future. I'd be totally happy with the temporary canvas as a long term solution as long as there is a reasonable way to identify whether it's the right thing from the JS side. I have enough performance issues in my games resulting from bad canvas implementations that I'm not excited about introducing another one. As I've mentioned in previous posts, I have observed temporary canvases causing *significant* performance hits in the wild in otherwise stellar canvas 2d implementations; I've had to blacklist them as a result and that's super fragile. Some thought also needs to go into what the 'right way' to use a temporary canvas is: Should there be one temporary canvas of adequate size, that gets updated before each drawing operation? A fixed-size pool of temporary canvases that get reused? One temporary canvas for each source region of the spritemap that's being used? I could see the correct answer here depending on the implementation. If there's an easy answer, maybe that guidance can go into an addendum to the canvas spec or strongly evangelized page on some developer-visible wiki like MDN. For a highly parallel canvas implementation (one backed by OpenGL or Direct3D) I would expect you would want one temporary canvas per source region, and prefer canvases that never change, so that it's at least possible for the implementation to map them to textures. On the other hand, if canvas is being implemented in software you would probably get the best performance out of a single temporary canvas (cache locality, etc). And for backends like Direct2D, Cairo etc I have no idea what to expect - they tend to defy my expectations since so much interesting stuff has to go on in order to achieve the guarantees they make to end-users. -kg
Re: [whatwg] I believe source rectangles for HTML5 Canvas drawImage are specified incorrectly
On Mon, Dec 17, 2012 at 11:49 AM, Justin Novosad wrote: > > Isn't this the same as what Ian suggested: copy it to a temporary canvas >> and use the temporary canvas scales. >> It seems that you can optimize that case too. >> > > Hmm... It would one of those optimizations that only works if you perform > the secret handshake just right. > If I understand correctly, the hypothetical optimization would work > something like this: When a 2d canvas context is created, do not allocate a > backing store immediately. If the first draw operation into the canvas is a > drawImage from an image element and covers the entire canvas, then use a > shallow copy of the bitmap as the backing store. If anything else is drawn > into the canvas, then a proper backing store should be allocated and used. > I guess it could work... but I would fear that this behavior would be > perceived as idiosyncratic. > I don't disagree. However, there probably are other occasions in canvas today where a common idiom gets optimized. In addition, developers have proven that they are quite comfortable with idiosyncrasies. For instance, many sites force their CSS animations to run in 3D because it tricks the browser into hardware acceleration.
Re: [whatwg] I believe source rectangles for HTML5 Canvas drawImage are specified incorrectly
> Isn't this the same as what Ian suggested: copy it to a temporary canvas > and use the temporary canvas scales. > It seems that you can optimize that case too. > Hmm... It would one of those optimizations that only works if you perform the secret handshake just right. If I understand correctly, the hypothetical optimization would work something like this: When a 2d canvas context is created, do not allocate a backing store immediately. If the first draw operation into the canvas is a drawImage from an image element and covers the entire canvas, then use a shallow copy of the bitmap as the backing store. If anything else is drawn into the canvas, then a proper backing store should be allocated and used. I guess it could work... but I would fear that this behavior would be perceived as idiosyncratic.
Re: [whatwg] I believe source rectangles for HTML5 Canvas drawImage are specified incorrectly
On Mon, Dec 17, 2012 at 7:23 AM, Justin Novosad wrote: > > > On Sun, Dec 16, 2012 at 11:52 PM, Rik Cabanier wrote: > >> >> >> On Wed, Dec 12, 2012 at 10:24 AM, Justin Novosad wrote: >> >>> >>> >>> On Wed, Dec 12, 2012 at 12:39 PM, Rik Cabanier wrote: >>> What would be the next step? Should we define a new version of drawImage with the extra parameter? >>> >>> >>> That is one option, but I think a context attribute (like >>> imageSmoothingEnabled) is also worth considering. >>> Perhaps image smoothing could be an enum rather than boolean with an >>> additional mode that prevents color bleeding. >>> >> >> It seems a bit too expensive to add a variable to the graphics context >> that is checked for just this call. >> Maybe just a new drawImage call (like drawNonSmoothedImage?) is all that >> is needed. >> > > Yes. That sounds quite reasonable to me, but we can find a better name. > The name "drawNonSmoothedImage" suggests that the image won't be smoothed > at all, which is not the case. It's hard to find a name that correctly > describes the right behavior without getting too technical. I am thinking > "drawSubImage", in the sense that the sub region delimited by the source > rectangle is treated as if it were a whole image. > That sounds reasonable. > This gives me another idea: we could just have a new Image constructor > that creates a new image element that is a subregion of another: > var mySprite = new Image(spriteMap, x, y, w, h); > This can be implemented in a lightweight way that just references the data > of the source image. > Isn't this the same as what Ian suggested: copy it to a temporary canvas and use the temporary canvas scales. It seems that you can optimize that case too.
Re: [whatwg] I believe source rectangles for HTML5 Canvas drawImage are specified incorrectly
On Mon, Dec 17, 2012 at 10:33 AM, Kevin Gadd wrote: > A simple way to create an Image that represents a subregion of another > Image or Canvas would be a nice solution here, since for > implementations that need a temporary image anyway this lets them > ensure it's only created once, and it lets you avoid the costs > associated with temporary canvases. A few questions come to mind, > though: > > Will it be possible to accept a canvas as the first argument instead > of an Image? > That is a trickier case. What happens when you draw to the canvas after the creating the Image. Do the draws affect the Image? If it is meant to behave like a snapshot, then pixels we need to be copied so that they can persist. > If you have a few live references to subregions of a larger image, > will that prevent browsers like Chrome from discarding the decoded > pixels? I don't see that being a problem. A subimage could reference the image resource the same way a regular image element does. You can already have multiple image elements referencing the same resource (e.g. two image tags with same "src") > > Will creating the subregion imply a copy and the associated garbage > collection overhead? If so, API consumers would probably end up having > to create some sort of caching library to keep cached copies of > subregions around, and then those cached subregions might leak > forever. I think what you'd want here is for it to be a reference to > the subregion within the existing image, which means if it's a > reference to a subregion of a canvas, when the canvas changes the > subregion changes too. Would it break things for the semantics to work > that way - i.e. it is now possible for the contents of an Image to > change? > The contents of an image can change if you change the value of its "src" attribute, which results in a heavy invalidation that triggers a re-layout of the page. > How would a new overload of the Image constructor be feature-detected in > JS? > It would be possible for the subregion-ness of an Image element to be expressed by attributes on the Image element (subregion x, y, w, h) If we do it that way, one could test for attribute existence. > If this becomes the correct way to solve this problem, what happens to > existing implementations that provided alternative sampling behavior > (like Chrome)? Will they get changed to match the spec, breaking apps > until the new Image constructor is rolled out? > If an app only works in WK browsers, then there is already a problem with it (and with the web platform). Anyways, we can come up with ways to assure a smooth transition, but that is a discussion for a different forum. > > Also, the more I think about it, the more the garbage collection > impact of invoking a constructor for every blit seems like a potential > problem. Even in generational GCs like V8, allocating a bunch of > objects isn't cheap. Given that JS provides no way to hold weak > references, it wouldn't be straightforward to cache and evict Image > objects for each particular source rectangle used when drawing. > Right now, you have to store your sprite maps in globals to avoid this problem, right? I guess it would be the same for sprite (sub image) arrays. I don't think that is much worse that the status quo. > > -kg > > On Mon, Dec 17, 2012 at 7:23 AM, Justin Novosad > wrote: > > > > > > On Sun, Dec 16, 2012 at 11:52 PM, Rik Cabanier > wrote: > >> > >> > >> > >> On Wed, Dec 12, 2012 at 10:24 AM, Justin Novosad > >> wrote: > >>> > >>> > >>> > >>> On Wed, Dec 12, 2012 at 12:39 PM, Rik Cabanier > >>> wrote: > > > What would be the next step? Should we define a new version of > drawImage > with the extra parameter? > >>> > >>> > >>> That is one option, but I think a context attribute (like > >>> imageSmoothingEnabled) is also worth considering. > >>> Perhaps image smoothing could be an enum rather than boolean with an > >>> additional mode that prevents color bleeding. > >> > >> > >> It seems a bit too expensive to add a variable to the graphics context > >> that is checked for just this call. > >> Maybe just a new drawImage call (like drawNonSmoothedImage?) is all that > >> is needed. > > > > > > Yes. That sounds quite reasonable to me, but we can find a better name. > The > > name "drawNonSmoothedImage" suggests that the image won't be smoothed at > > all, which is not the case. It's hard to find a name that correctly > > describes the right behavior without getting too technical. I am > thinking > > "drawSubImage", in the sense that the sub region delimited by the source > > rectangle is treated as if it were a whole image. This gives me another > > idea: we could just have a new Image constructor that creates a new image > > element that is a subregion of another: > > var mySprite = new Image(spriteMap, x, y, w, h); > > This can be implemented in a lightweight way that just references the > data > > of the source image. > > > >> > >> > >>> > >>> > >
Re: [whatwg] I believe source rectangles for HTML5 Canvas drawImage are specified incorrectly
A simple way to create an Image that represents a subregion of another Image or Canvas would be a nice solution here, since for implementations that need a temporary image anyway this lets them ensure it's only created once, and it lets you avoid the costs associated with temporary canvases. A few questions come to mind, though: Will it be possible to accept a canvas as the first argument instead of an Image? If you have a few live references to subregions of a larger image, will that prevent browsers like Chrome from discarding the decoded pixels? The fact that Chrome discards unused decoded image pixels helps keep memory usage low for HTML5 games, so it would suck if this API change prevented it (and other browsers) from doing that effectively. Will creating the subregion imply a copy and the associated garbage collection overhead? If so, API consumers would probably end up having to create some sort of caching library to keep cached copies of subregions around, and then those cached subregions might leak forever. I think what you'd want here is for it to be a reference to the subregion within the existing image, which means if it's a reference to a subregion of a canvas, when the canvas changes the subregion changes too. Would it break things for the semantics to work that way - i.e. it is now possible for the contents of an Image to change? How would a new overload of the Image constructor be feature-detected in JS? If this becomes the correct way to solve this problem, what happens to existing implementations that provided alternative sampling behavior (like Chrome)? Will they get changed to match the spec, breaking apps until the new Image constructor is rolled out? Also, the more I think about it, the more the garbage collection impact of invoking a constructor for every blit seems like a potential problem. Even in generational GCs like V8, allocating a bunch of objects isn't cheap. Given that JS provides no way to hold weak references, it wouldn't be straightforward to cache and evict Image objects for each particular source rectangle used when drawing. -kg On Mon, Dec 17, 2012 at 7:23 AM, Justin Novosad wrote: > > > On Sun, Dec 16, 2012 at 11:52 PM, Rik Cabanier wrote: >> >> >> >> On Wed, Dec 12, 2012 at 10:24 AM, Justin Novosad >> wrote: >>> >>> >>> >>> On Wed, Dec 12, 2012 at 12:39 PM, Rik Cabanier >>> wrote: What would be the next step? Should we define a new version of drawImage with the extra parameter? >>> >>> >>> That is one option, but I think a context attribute (like >>> imageSmoothingEnabled) is also worth considering. >>> Perhaps image smoothing could be an enum rather than boolean with an >>> additional mode that prevents color bleeding. >> >> >> It seems a bit too expensive to add a variable to the graphics context >> that is checked for just this call. >> Maybe just a new drawImage call (like drawNonSmoothedImage?) is all that >> is needed. > > > Yes. That sounds quite reasonable to me, but we can find a better name. The > name "drawNonSmoothedImage" suggests that the image won't be smoothed at > all, which is not the case. It's hard to find a name that correctly > describes the right behavior without getting too technical. I am thinking > "drawSubImage", in the sense that the sub region delimited by the source > rectangle is treated as if it were a whole image. This gives me another > idea: we could just have a new Image constructor that creates a new image > element that is a subregion of another: > var mySprite = new Image(spriteMap, x, y, w, h); > This can be implemented in a lightweight way that just references the data > of the source image. > >> >> >>> >>> If so, we probably want the description of the existing drawImage call to change so it leaves the resampling issue undefined unless WebKit is OK with changing their current behavior. >>> >>> >>> IMHO: Undifined behavior is a spec bug. If we have a problem with the >>> spec, we fix the spec, we don't just each do our own thing. >>> >> > -- -kg
Re: [whatwg] I believe source rectangles for HTML5 Canvas drawImage are specified incorrectly
On Sun, Dec 16, 2012 at 11:52 PM, Rik Cabanier wrote: > > > On Wed, Dec 12, 2012 at 10:24 AM, Justin Novosad wrote: > >> >> >> On Wed, Dec 12, 2012 at 12:39 PM, Rik Cabanier wrote: >> >>> >>> What would be the next step? Should we define a new version of drawImage >>> with the extra parameter? >> >> >> That is one option, but I think a context attribute (like >> imageSmoothingEnabled) is also worth considering. >> Perhaps image smoothing could be an enum rather than boolean with an >> additional mode that prevents color bleeding. >> > > It seems a bit too expensive to add a variable to the graphics context > that is checked for just this call. > Maybe just a new drawImage call (like drawNonSmoothedImage?) is all that > is needed. > Yes. That sounds quite reasonable to me, but we can find a better name. The name "drawNonSmoothedImage" suggests that the image won't be smoothed at all, which is not the case. It's hard to find a name that correctly describes the right behavior without getting too technical. I am thinking "drawSubImage", in the sense that the sub region delimited by the source rectangle is treated as if it were a whole image. This gives me another idea: we could just have a new Image constructor that creates a new image element that is a subregion of another: var mySprite = new Image(spriteMap, x, y, w, h); This can be implemented in a lightweight way that just references the data of the source image. > > >> >> >>> If so, we probably want the description of the existing drawImage call >>> to change so it leaves the resampling issue undefined unless WebKit is OK >>> with changing their current behavior. >>> >> >> IMHO: Undifined behavior is a spec bug. If we have a problem with the >> spec, we fix the spec, we don't just each do our own thing. >> >> >
Re: [whatwg] I believe source rectangles for HTML5 Canvas drawImage are specified incorrectly
On Wed, Dec 12, 2012 at 10:24 AM, Justin Novosad wrote: > > > On Wed, Dec 12, 2012 at 12:39 PM, Rik Cabanier wrote: > >> >> What would be the next step? Should we define a new version of drawImage >> with the extra parameter? > > > That is one option, but I think a context attribute (like > imageSmoothingEnabled) is also worth considering. > Perhaps image smoothing could be an enum rather than boolean with an > additional mode that prevents color bleeding. > It seems a bit too expensive to add a variable to the graphics context that is checked for just this call. Maybe just a new drawImage call (like drawNonSmoothedImage?) is all that is needed. > > >> If so, we probably want the description of the existing drawImage call to >> change so it leaves the resampling issue undefined unless WebKit is OK with >> changing their current behavior. >> > > IMHO: Undifined behavior is a spec bug. If we have a problem with the > spec, we fix the spec, we don't just each do our own thing. > >
Re: [whatwg] I believe source rectangles for HTML5 Canvas drawImage are specified incorrectly
> I agree. What I meant to say that the spec should specify that resampling > behavior is undefined with the existing API. > It would of course be best if WK would fix their code so it follows the > current spec description and instructs developers to use the new parameter. > Yep. That's how I see it: we currently have a compliance bug in our implementation.
Re: [whatwg] I believe source rectangles for HTML5 Canvas drawImage are specified incorrectly
On Wed, Dec 12, 2012 at 12:39 PM, Rik Cabanier wrote: > > What would be the next step? Should we define a new version of drawImage > with the extra parameter? That is one option, but I think a context attribute (like imageSmoothingEnabled) is also worth considering. Perhaps image smoothing could be an enum rather than boolean with an additional mode that prevents color bleeding. > If so, we probably want the description of the existing drawImage call to > change so it leaves the resampling issue undefined unless WebKit is OK with > changing their current behavior. > IMHO: Undifined behavior is a spec bug. If we have a problem with the spec, we fix the spec, we don't just each do our own thing.
Re: [whatwg] I believe source rectangles for HTML5 Canvas drawImage are specified incorrectly
On Wed, Dec 12, 2012 at 10:16 AM, Ian Hickson wrote: > On Wed, 12 Dec 2012, Rik Cabanier wrote: > > > > What would be the next step? Should we define a new version of drawImage > > with the extra parameter? > > At the WHATWG, the process, insofar as there is one, is that the spec's > editor (in this case me) goes through the e-mails on a topic and edits the > spec based on the feedback and based on how much vendor buy-in an issue > has. > > On this particular topic, I've only heard interest in implementing this > from one vendor, Google, so adding support to the spec at this point is > probably premature. If other vendors are interested in implementing > something for this (a feature to make drawImage scale subsets with > clamping at the source rectangle rather than the original image), I > encourage them to indicate this on this thread. > > > If so, we probably want the description of the existing drawImage call > > to change so it leaves the resampling issue undefined unless WebKit is > > OK with changing their current behavior. > > Leaving the spec undefined is not acceptable. :-) > > I agree. What I meant to say that the spec should specify that resampling behavior is undefined with the existing API. It would of course be best if WK would fix their code so it follows the current spec description and instructs developers to use the new parameter.
Re: [whatwg] I believe source rectangles for HTML5 Canvas drawImage are specified incorrectly
On Wed, 12 Dec 2012, Rik Cabanier wrote: > > What would be the next step? Should we define a new version of drawImage > with the extra parameter? At the WHATWG, the process, insofar as there is one, is that the spec's editor (in this case me) goes through the e-mails on a topic and edits the spec based on the feedback and based on how much vendor buy-in an issue has. On this particular topic, I've only heard interest in implementing this from one vendor, Google, so adding support to the spec at this point is probably premature. If other vendors are interested in implementing something for this (a feature to make drawImage scale subsets with clamping at the source rectangle rather than the original image), I encourage them to indicate this on this thread. > If so, we probably want the description of the existing drawImage call > to change so it leaves the resampling issue undefined unless WebKit is > OK with changing their current behavior. Leaving the spec undefined is not acceptable. :-) -- Ian Hickson U+1047E)\._.,--,'``.fL http://ln.hixie.ch/ U+263A/, _.. \ _\ ;`._ ,. Things that are impossible just take longer. `._.-(,_..'--(,_..'`-.;.'
Re: [whatwg] I believe source rectangles for HTML5 Canvas drawImage are specified incorrectly
On Tue, Dec 11, 2012 at 7:56 AM, Justin Novosad wrote: > > > On Mon, Dec 10, 2012 at 6:53 PM, Rik Cabanier wrote: >> >> >> I assume this would only be for the 9 argument version of drawImage? >> > > Yeah, the all dressed version. > > >> >> FWIW >> I was curious why this bug doesn't show up in WebKit so I looked at the >> implementation. >> They actually catch cases where there's scaling of a portion of an image >> and special case it by copying the non-scaled image to a temporary canvas >> first: >> // When the image is scaled using high-quality interpolation, we >> create a temporary CGImage >> // containing only the portion we want to display. We need to do >> this because high-quality >> // interpolation smoothes sharp edges, causing pixels from >> outside the source rect to bleed >> // into the destination rect. See . >> >> It would be nice to see why the Safari folks added this code since it >> makes them inconsistent with the spec. >> >> > For the skia port of WebKit (used by Chrome) clamping to the source region > is built into the blitter functions (sw rendering) and shaders (gpu > rendering) so that color bleeding can be prevented without creating a > temporary copy of the image. The reason it was implemented this way is > that the bleeding was reported as a bug by game developers and the spec was > not as clear then as it is today. > > We were getting more complaints about this issue back when there was color > bleeding, which is the main reason why I have been arguing in favor of > clamping to the bounds of the source rectangle. It seems more web > developers are getting the results they expect from drawImage now that > there is no color bleeding in Chrome, which is unfortunately not spec > compliant behavior. > > What would be the next step? Should we define a new version of drawImage with the extra parameter? If so, we probably want the description of the existing drawImage call to change so it leaves the resampling issue undefined unless WebKit is OK with changing their current behavior.
Re: [whatwg] I believe source rectangles for HTML5 Canvas drawImage are specified incorrectly
On Mon, Dec 10, 2012 at 6:53 PM, Rik Cabanier wrote: > > > I assume this would only be for the 9 argument version of drawImage? > Yeah, the all dressed version. > > FWIW > I was curious why this bug doesn't show up in WebKit so I looked at the > implementation. > They actually catch cases where there's scaling of a portion of an image > and special case it by copying the non-scaled image to a temporary canvas > first: > // When the image is scaled using high-quality interpolation, we > create a temporary CGImage > // containing only the portion we want to display. We need to do > this because high-quality > // interpolation smoothes sharp edges, causing pixels from outside > the source rect to bleed > // into the destination rect. See . > > It would be nice to see why the Safari folks added this code since it > makes them inconsistent with the spec. > > For the skia port of WebKit (used by Chrome) clamping to the source region is built into the blitter functions (sw rendering) and shaders (gpu rendering) so that color bleeding can be prevented without creating a temporary copy of the image. The reason it was implemented this way is that the bleeding was reported as a bug by game developers and the spec was not as clear then as it is today. We were getting more complaints about this issue back when there was color bleeding, which is the main reason why I have been arguing in favor of clamping to the bounds of the source rectangle. It seems more web developers are getting the results they expect from drawImage now that there is no color bleeding in Chrome, which is unfortunately not spec compliant behavior.
Re: [whatwg] I believe source rectangles for HTML5 Canvas drawImage are specified incorrectly
On Mon, Dec 10, 2012 at 1:18 PM, Justin Novosad wrote: > On Mon, Dec 10, 2012 at 4:04 PM, Rik Cabanier wrote: > >> >> >> As Ian said, copying the sprite image to a non-scaled canvas first (which >> most games probably do anyway) works around the issue. >> There is no need for padding in that case. >> >> > Yes, I am sure a lot of games do it already to work around the color > bleeding problem but there is a performance cost to doing that. > > Couldn't we just make everyone happy by making the behavior controllable > through a context attribute or an additional overload of drawImage that > takes an extra argument? > I assume this would only be for the 9 argument version of drawImage? FWIW I was curious why this bug doesn't show up in WebKit so I looked at the implementation. They actually catch cases where there's scaling of a portion of an image and special case it by copying the non-scaled image to a temporary canvas first: // When the image is scaled using high-quality interpolation, we create a temporary CGImage // containing only the portion we want to display. We need to do this because high-quality // interpolation smoothes sharp edges, causing pixels from outside the source rect to bleed // into the destination rect. See . It would be nice to see why the Safari folks added this code since it makes them inconsistent with the spec.
Re: [whatwg] I believe source rectangles for HTML5 Canvas drawImage are specified incorrectly
On Mon, Dec 10, 2012 at 4:04 PM, Rik Cabanier wrote: > > > As Ian said, copying the sprite image to a non-scaled canvas first (which > most games probably do anyway) works around the issue. > There is no need for padding in that case. > > Yes, I am sure a lot of games do it already to work around the color bleeding problem but there is a performance cost to doing that. Couldn't we just make everyone happy by making the behavior controllable through a context attribute or an additional overload of drawImage that takes an extra argument?
Re: [whatwg] I believe source rectangles for HTML5 Canvas drawImage are specified incorrectly
I can't speak for the most common approach in HTML5 games, but using a temporary surface for drawing sprites is definitely not a common approach in non-canvas games. I've never seen it proposed before this thread for canvas dev, either, but I'm not an expert there. This is at least an area where someone who has a passing familiarity with graphics programming is not going to have their expectations met. One reason spritemaps get used is to reduce overhead when deploying game content. In that case, copying the individual images out is not a problem and it's no big deal that canvas is specified this way. Another reason is that in Direct3D and OpenGL, historically state changes have come at a significant cost, so putting all of your individual images into a single texture allows you to draw lots of objects with no state changes. While state changes are dramatically cheaper on modern hardware, they still aren't free. Of course, this doesn't necessarily apply to canvas backends, but that's part of why people do it. If, like me, you're building browser games that use WebGL where available and use canvas for fallback, it is basically impossible to convert your texture atlases/spritemaps into individual images because texture coordinates (and as a result, the canvas source rectangles used in fallback) can map to any part of the texture. Re: the efficiency of clips, I don't think you can state that it's 'much more expensive' unless you've tested it. The kind of clip he describes is absurdly fast on 3D hardware and built into both the Direct3D and OpenGL specs: http://msdn.microsoft.com/en-us/library/windows/desktop/bb147354%28v=vs.85%29.aspx http://www.opengl.org/sdk/docs/man/xhtml/glScissor.xml It should be possible to apply a similar technique to software canvas implementations, but I won't claim to know whether that's done. Canvas clipping in general is certainly more complicated, but what he described is not. I agree that using clips is a reasonable way to express that kind of rendering operation (especially since it simplifies the javascript you have to write) and it allows both of these use cases - spritemaps and subregion updates - to coexist. It's also not a safe assumption that subregion updates are common in games. Certainly any game using Direct3D, OpenGL or WebGL is unlikely to be doing subregion updates since typically your entire framebuffer is discarded by the GPU (or at least not reliable) after you swapbuffers/present. Doing partial updates requires extra effort to set up offscreen render targets and make copies of pixel data. I wouldn't be shocked if there are canvas-based games out there doing subregion updates, but I can't think of more than one or two that I've ever seen. If we're talking about optimizing for one use case over another, there should be data to support that it is the only important use case. Arguably, it should be possible to provide reasonable performance for both use cases here... -kg On Mon, Dec 10, 2012 at 1:04 PM, Rik Cabanier wrote: > > > On Mon, Dec 10, 2012 at 11:03 AM, Justin Novosad wrote: >> >> On Mon, Dec 10, 2012 at 1:24 PM, Ian Hickson wrote: >> >> > >> > The reason to prefer the current behaviour is if you want to just update >> > a >> > small part of an image. For example, if you draw a bit photo, then draw >> > text over it, then want to remove the text by just drawing the photo >> > over >> > where the text was but not redrawing the whole thing. If we clamped to >> > source rectangle, we'd get artefacts in this case that couldn't be >> > worked >> > around (unlike the problems with scaling sprites, which can be worked >> > around, albeit in a suboptimal fashion). >> > >> > >> I disagree. There is an efficient workaround for updating a subregion of >> an >> image without getting filtering artifacts around the edges : use clipping >> to limit the update region > > > How is that an efficient workaround? > It involves setting a clip, calling drawimge with a larger (complete?) image > and then removing the clip. > This seems much more expensive for such as common operation. (ie a game that > wants to remove an animated character) > >> >> On the other hand, the workaround to prevent color bleeding when using >> spritemap is much less efficient since it requires padding, which is >> wasteful by definition, and in cases where the sprite can be downsized, >> the >> padding margins may have to be very large. > > > As Ian said, copying the sprite image to a non-scaled canvas first (which > most games probably do anyway) works around the issue. > There is no need for padding in that case. > > > > -- -kg
Re: [whatwg] I believe source rectangles for HTML5 Canvas drawImage are specified incorrectly
On Mon, Dec 10, 2012 at 11:03 AM, Justin Novosad wrote: > On Mon, Dec 10, 2012 at 1:24 PM, Ian Hickson wrote: > > > > > The reason to prefer the current behaviour is if you want to just update > a > > small part of an image. For example, if you draw a bit photo, then draw > > text over it, then want to remove the text by just drawing the photo over > > where the text was but not redrawing the whole thing. If we clamped to > > source rectangle, we'd get artefacts in this case that couldn't be worked > > around (unlike the problems with scaling sprites, which can be worked > > around, albeit in a suboptimal fashion). > > > > > I disagree. There is an efficient workaround for updating a subregion of an > image without getting filtering artifacts around the edges : use clipping > to limit the update region > How is that an efficient workaround? It involves setting a clip, calling drawimge with a larger (complete?) image and then removing the clip. This seems much more expensive for such as common operation. (ie a game that wants to remove an animated character) > On the other hand, the workaround to prevent color bleeding when using > spritemap is much less efficient since it requires padding, which is > wasteful by definition, and in cases where the sprite can be downsized, the > padding margins may have to be very large. As Ian said, copying the sprite image to a non-scaled canvas first (which most games probably do anyway) works around the issue. There is no need for padding in that case.
Re: [whatwg] I believe source rectangles for HTML5 Canvas drawImage are specified incorrectly
On Mon, Dec 10, 2012 at 1:24 PM, Ian Hickson wrote: > > The reason to prefer the current behaviour is if you want to just update a > small part of an image. For example, if you draw a bit photo, then draw > text over it, then want to remove the text by just drawing the photo over > where the text was but not redrawing the whole thing. If we clamped to > source rectangle, we'd get artefacts in this case that couldn't be worked > around (unlike the problems with scaling sprites, which can be worked > around, albeit in a suboptimal fashion). > > I disagree. There is an efficient workaround for updating a subregion of an image without getting filtering artifacts around the edges : use clipping to limit the update region On the other hand, the workaround to prevent color bleeding when using spritemap is much less efficient since it requires padding, which is wasteful by definition, and in cases where the sprite can be downsized, the padding margins may have to be very large. -Justin
Re: [whatwg] I believe source rectangles for HTML5 Canvas drawImage are specified incorrectly
(Sorry if you get this twice. For some reason my first reply was bounced by the listserv?) On Mon, Dec 10, 2012 at 10:24 AM, Ian Hickson wrote: > There's two ways to do scaled sprites with drawImage(): have a border of > transparent black around each sprite, or copy the data out of the sprite > sheet and into a temporary canvas at its original size, then scaling from > that. How big does the border of transparent black have to be? Maybe I'm reading the spec incorrectly, but given the way it's written, implementations would be free to make use of hardware mip-maps when rendering images, which would mean that when scaling down, values from arbitrarily far outside the source rectangle could get pulled in. Temporary canvases for every blit seems like it would imply a significant performance penalty, as well, but I haven't tested that technique - maybe it's okay. I do know that creating a large number of temporary canvases causes performance issues in IE. > Disabling image smoothing will increase artefacts, that's kind of the > point. :-) Having said that, I don't really see what that test case is > demonstrating. Can you elaborate? If the test case is demonstrating the behavior I argue correct, the drawn images in the canvas at the bottom will not show any red pixels. Another arguably correct approach would be for red pixels only to appear with image smoothing enabled. It doesn't make sense even given the current spec for red pixels to appear when smoothing is disabled - if you're drawing with nearest neighbor sampling, the red pixels that appear in the current test case (in Firefox on Windows, at least) can only be there if partial pixels are being rendered, which shouldn't be possible using nearest neighbor. An acceptable compromise in this case, IMO, would be to at least require that pixels from outside the source rectangle are not read if image smoothing is disabled. > The reason to prefer the current behaviour is if you want to just update a > small part of an image. For example, if you draw a bit photo, then draw > text over it, then want to remove the text by just drawing the photo over > where the text was but not redrawing the whole thing. If we clamped to > source rectangle, we'd get artefacts in this case that couldn't be worked > around (unlike the problems with scaling sprites, which can be worked > around, albeit in a suboptimal fashion). Using a clip seems like the right way to do that. If you want to update a subregion, set a clip to the region to update, and run the exact same drawing code you ran before. That would work regardless of the semantics defined for source rectangles, and be applicable for all drawing primitives, not just drawImage. I can see how the performance characteristics for canvas clips might be too awful for that to be viable, though, but in that case we're now arguing over which applications should get the bigger performance hit. In that scenario, I would strongly suggest that games should be the ones to get privileged performance since they're extremely performance dependent to begin with. I don't know what applications actually use that behavior, though, so maybe they're just as performance focused. -kg
Re: [whatwg] I believe source rectangles for HTML5 Canvas drawImage are specified incorrectly
On Mon, 20 Aug 2012, Kevin Gadd wrote: > > Hi, I've been digging into an inconsistency between various browsers' > Canvas implementations and I think the spec might be allowing > undesirable behavior here. > > The current version of the spec says > (http://www.whatwg.org/specs/web-apps/current-work/multipage/the-canvas-element.html#dom-context-2d-drawimage): > > If the original image data is a bitmap image, the value painted at a > point in the destination rectangle is computed by filtering the original > image data. The user agent may use any filtering algorithm (for example > bilinear interpolation or nearest-neighbor). When the filtering > algorithm requires a pixel value from outside the original image data, > it must instead use the value from the nearest edge pixel. (That is, the > filter uses 'clamp-to-edge' behavior.) > > While clamp-to-edge is desirable, the way this is specified means that > it only ever clamps to the edges of the source bitmap, not to the source > rectangle. That means that attempting to do the equivalent of css > sprites or video game style 'tile sets' - where a single source image > contains many smaller images - is not possible, because the spec allows > implementations to read pixels from outside the source rectangle. There's two ways to do scaled sprites with drawImage(): have a border of transparent black around each sprite, or copy the data out of the sprite sheet and into a temporary canvas at its original size, then scaling from that. > Unfortunately, at present Internet Explorer and Firefox both read pixels > from outside the source rectangle, as demonstrated by this test case: > https://dl.dropbox.com/u/1643240/canvas_artifacts.html > Worse still, in implementations with imageSmoothingEnabled available, > turning off image smoothing is not sufficient to eliminate the > artifacts. Disabling image smoothing will increase artefacts, that's kind of the point. :-) Having said that, I don't really see what that test case is demonstrating. Can you elaborate? > Google Chrome appears to implement this the way you would probably want > it to work - by clamping to the edges of the source rectangle, instead > of the source image. I can't think of a good reason to prefer the > current behavior over what Chrome does, and I haven't been able to find > a reliable way to compensate for the current behavior. The reason to prefer the current behaviour is if you want to just update a small part of an image. For example, if you draw a bit photo, then draw text over it, then want to remove the text by just drawing the photo over where the text was but not redrawing the whole thing. If we clamped to source rectangle, we'd get artefacts in this case that couldn't be worked around (unlike the problems with scaling sprites, which can be worked around, albeit in a suboptimal fashion). On Mon, 20 Aug 2012, Justin Novosad wrote: > > The same artifact use to be present in Chrome not that long ago. When we > fixed it, we chose to interpret "original image data" as meaning the > part of the image data that is within the bounds of the of the source > rectangle. I don't think this reading is consistent with the usage of the term "original image data" in the spec. It's pretty consistently used to mean the whole image. I've clarified the spec to make this even less ambiguous. > Also, it makes more sense to do it that way. I think arguments could be made for both ways, but the problem is there's only a way to work around one of the ways. On Mon, 10 Sep 2012, Vladimir Vukicevic wrote: > > This is pretty tricky to get right -- there's just a general graphics > problem in this case. There are valid use cases for both sampling > outside and not sampling outside the source rectangle, as well as > implementation issues for being able to do source rectangle clamping. > For example, should you be able to take a source image and draw it > scaled up using 4 rectangles (one for each quadrant) and have the result > be equal to just doing it in one draw? Or take any random subimage (for > example, for efficient updates of some destination) and draw it in. Right. > At best, I think a new mode toggle or flag would be needed to allow you > to select. I'd be happy to add a flag if it's something browser implementors want to support. > Additionally, I think there's a related bug filed from a while ago about > defining how to sample pixels that are outside of the source bounds -- > do you clamp to edge, do you sample transparent black, etc. Fixing that bug is how the current text came to be, IIRC. On Mon, 10 Sep 2012, Justin Novosad wrote: > > You are referring to the fact that clamping modes only apply to texture > borders in OpenGL and DirectX? That is not that big of a big deal, you > can implement the clamp in a shader. > > I think Vladimir makes a good point that there are valid use cases for > both ways. To summarize the issues: >
Re: [whatwg] I believe source rectangles for HTML5 Canvas drawImage are specified incorrectly
On 2012-09-10, at 5:28 PM, Justin Novosad wrote: > > On Mon, Sep 10, 2012 at 4:49 PM, Jeff Muizelaar > wrote: > > On 2012-09-10, at 3:43 PM, Vladimir Vukicevic wrote: > > > FWIW, there are also negative performance implications to clamping samples to > the source rect. Many graphics APIs do not support this kind sampling, and > supporting this behaviour on top of those apis requires a temporary copy of > the subimage to be made. > > > You are referring to the fact that clamping modes only apply to texture > borders in OpenGL and DirectX? That is not that big of a big deal, you can > implement the clamp in a shader. No, I was referring to CoreGraphics, Cairo and Direct2D. -Jeff
Re: [whatwg] I believe source rectangles for HTML5 Canvas drawImage are specified incorrectly
On Mon, Sep 10, 2012 at 4:49 PM, Jeff Muizelaar wrote: > > On 2012-09-10, at 3:43 PM, Vladimir Vukicevic wrote: > > > FWIW, there are also negative performance implications to clamping samples > to the source rect. Many graphics APIs do not support this kind sampling, > and supporting this behaviour on top of those apis requires a temporary > copy of the subimage to be made. > > You are referring to the fact that clamping modes only apply to texture borders in OpenGL and DirectX? That is not that big of a big deal, you can implement the clamp in a shader. I think Vladimir makes a good point that there are valid use cases for both ways. To summarize the issues: * clamping to source rect can cause filtering artefacts at boundaries, especially visible when tiling. * clamping to source image bounds can cause color bleeding artefacts when rendering a sprite from a spritemap.
Re: [whatwg] I believe source rectangles for HTML5 Canvas drawImage are specified incorrectly
On 2012-09-10, at 3:43 PM, Vladimir Vukicevic wrote: > This is pretty tricky to get right -- there's just a general graphics > problem in this case. There are valid use cases for both sampling outside > and not sampling outside the source rectangle, as well as implementation > issues for being able to do source rectangle clamping. For example, should > you be able to take a source image and draw it scaled up using 4 rectangles > (one for each quadrant) and have the result be equal to just doing it in > one draw? Or take any random subimage (for example, for efficient updates > of some destination) and draw it in. > > I do agree that the spec needs some clarity here, but I don't think that > just stating that drawImage should always sample in the source is the right > thing. At best, I think a new mode toggle or flag would be needed to allow > you to select. FWIW, there are also negative performance implications to clamping samples to the source rect. Many graphics APIs do not support this kind sampling, and supporting this behaviour on top of those apis requires a temporary copy of the subimage to be made. -Jeff
Re: [whatwg] I believe source rectangles for HTML5 Canvas drawImage are specified incorrectly
This is pretty tricky to get right -- there's just a general graphics problem in this case. There are valid use cases for both sampling outside and not sampling outside the source rectangle, as well as implementation issues for being able to do source rectangle clamping. For example, should you be able to take a source image and draw it scaled up using 4 rectangles (one for each quadrant) and have the result be equal to just doing it in one draw? Or take any random subimage (for example, for efficient updates of some destination) and draw it in. I do agree that the spec needs some clarity here, but I don't think that just stating that drawImage should always sample in the source is the right thing. At best, I think a new mode toggle or flag would be needed to allow you to select. Additionally, I think there's a related bug filed from a while ago about defining how to sample pixels that are outside of the source bounds -- do you clamp to edge, do you sample transparent black, etc. - Vlad On Mon, Aug 20, 2012 at 10:09 AM, Justin Novosad wrote: > Hi Kevin, > > The same artifact use to be present in Chrome not that long ago. When we > fixed it, we chose to interpret "original image data" as meaning the part > of the image data that is within the bounds of the of the source rectangle. > Also, it makes more sense to do it that way. I agree that the spec could > use more clarity here. > I support your case that it is preferable for the filtering algorithm to > clamp to the border of the source rectangle rather than to the border the > border of the source image. This is essential for implementing sprite maps > without having to waste pixels to pad the borders between tiles. > > -Justin Novosad > > On Mon, Aug 20, 2012 at 9:38 AM, Kevin Gadd wrote: > > > Hi, I've been digging into an inconsistency between various browsers' > > Canvas implementations and I think the spec might be allowing > > undesirable behavior here. > > > > The current version of the spec says > > ( > > > http://www.whatwg.org/specs/web-apps/current-work/multipage/the-canvas-element.html#dom-context-2d-drawimage > > ): > > > > If the original image data is a bitmap image, the value painted at a > > point in the destination rectangle is computed by filtering the > > original image data. The user agent may use any filtering algorithm > > (for example bilinear interpolation or nearest-neighbor). When the > > filtering algorithm requires a pixel value from outside the original > > image data, it must instead use the value from the nearest edge pixel. > > (That is, the filter uses 'clamp-to-edge' behavior.) > > > > While clamp-to-edge is desirable, the way this is specified means that > > it only ever clamps to the edges of the source bitmap, not to the > > source rectangle. That means that attempting to do the equivalent of > > css sprites or video game style 'tile sets' - where a single source > > image contains many smaller images - is not possible, because the spec > > allows implementations to read pixels from outside the source > > rectangle. > > > > Unfortunately, at present Internet Explorer and Firefox both read > > pixels from outside the source rectangle, as demonstrated by this test > > case: > > https://dl.dropbox.com/u/1643240/canvas_artifacts.html > > Worse still, in implementations with imageSmoothingEnabled available, > > turning off image smoothing is not sufficient to eliminate the > > artifacts. > > > > Google Chrome appears to implement this the way you would probably > > want it to work - by clamping to the edges of the source rectangle, > > instead of the source image. I can't think of a good reason to prefer > > the current behavior over what Chrome does, and I haven't been able to > > find a reliable way to compensate for the current behavior. > > > > Thanks, > > -kg > > >
Re: [whatwg] I believe source rectangles for HTML5 Canvas drawImage are specified incorrectly
On Mon, 20 Aug 2012, Rik Cabanier wrote: > > can you log a bug on this? > https://www.w3.org/Bugs/Public/enter_bug.cgi once it's in the system, we > can fix the wording in the spec. No need to file a bug; any e-mail sent to this list automatically ends up on the list of issues to fix and will get a response in due course. -- Ian Hickson U+1047E)\._.,--,'``.fL http://ln.hixie.ch/ U+263A/, _.. \ _\ ;`._ ,. Things that are impossible just take longer. `._.-(,_..'--(,_..'`-.;.'
Re: [whatwg] I believe source rectangles for HTML5 Canvas drawImage are specified incorrectly
Hi Kevin, can you log a bug on this? https://www.w3.org/Bugs/Public/enter_bug.cgi once it's in the system, we can fix the wording in the spec. Rik On Mon, Aug 20, 2012 at 6:38 AM, Kevin Gadd wrote: > Hi, I've been digging into an inconsistency between various browsers' > Canvas implementations and I think the spec might be allowing > undesirable behavior here. > > The current version of the spec says > ( > http://www.whatwg.org/specs/web-apps/current-work/multipage/the-canvas-element.html#dom-context-2d-drawimage > ): > > If the original image data is a bitmap image, the value painted at a > point in the destination rectangle is computed by filtering the > original image data. The user agent may use any filtering algorithm > (for example bilinear interpolation or nearest-neighbor). When the > filtering algorithm requires a pixel value from outside the original > image data, it must instead use the value from the nearest edge pixel. > (That is, the filter uses 'clamp-to-edge' behavior.) > > While clamp-to-edge is desirable, the way this is specified means that > it only ever clamps to the edges of the source bitmap, not to the > source rectangle. That means that attempting to do the equivalent of > css sprites or video game style 'tile sets' - where a single source > image contains many smaller images - is not possible, because the spec > allows implementations to read pixels from outside the source > rectangle. > > Unfortunately, at present Internet Explorer and Firefox both read > pixels from outside the source rectangle, as demonstrated by this test > case: > https://dl.dropbox.com/u/1643240/canvas_artifacts.html > Worse still, in implementations with imageSmoothingEnabled available, > turning off image smoothing is not sufficient to eliminate the > artifacts. > > Google Chrome appears to implement this the way you would probably > want it to work - by clamping to the edges of the source rectangle, > instead of the source image. I can't think of a good reason to prefer > the current behavior over what Chrome does, and I haven't been able to > find a reliable way to compensate for the current behavior. > > Thanks, > -kg >
Re: [whatwg] I believe source rectangles for HTML5 Canvas drawImage are specified incorrectly
Hi Kevin, The same artifact use to be present in Chrome not that long ago. When we fixed it, we chose to interpret "original image data" as meaning the part of the image data that is within the bounds of the of the source rectangle. Also, it makes more sense to do it that way. I agree that the spec could use more clarity here. I support your case that it is preferable for the filtering algorithm to clamp to the border of the source rectangle rather than to the border the border of the source image. This is essential for implementing sprite maps without having to waste pixels to pad the borders between tiles. -Justin Novosad On Mon, Aug 20, 2012 at 9:38 AM, Kevin Gadd wrote: > Hi, I've been digging into an inconsistency between various browsers' > Canvas implementations and I think the spec might be allowing > undesirable behavior here. > > The current version of the spec says > ( > http://www.whatwg.org/specs/web-apps/current-work/multipage/the-canvas-element.html#dom-context-2d-drawimage > ): > > If the original image data is a bitmap image, the value painted at a > point in the destination rectangle is computed by filtering the > original image data. The user agent may use any filtering algorithm > (for example bilinear interpolation or nearest-neighbor). When the > filtering algorithm requires a pixel value from outside the original > image data, it must instead use the value from the nearest edge pixel. > (That is, the filter uses 'clamp-to-edge' behavior.) > > While clamp-to-edge is desirable, the way this is specified means that > it only ever clamps to the edges of the source bitmap, not to the > source rectangle. That means that attempting to do the equivalent of > css sprites or video game style 'tile sets' - where a single source > image contains many smaller images - is not possible, because the spec > allows implementations to read pixels from outside the source > rectangle. > > Unfortunately, at present Internet Explorer and Firefox both read > pixels from outside the source rectangle, as demonstrated by this test > case: > https://dl.dropbox.com/u/1643240/canvas_artifacts.html > Worse still, in implementations with imageSmoothingEnabled available, > turning off image smoothing is not sufficient to eliminate the > artifacts. > > Google Chrome appears to implement this the way you would probably > want it to work - by clamping to the edges of the source rectangle, > instead of the source image. I can't think of a good reason to prefer > the current behavior over what Chrome does, and I haven't been able to > find a reliable way to compensate for the current behavior. > > Thanks, > -kg >
