Re: [whatwg] Adding features needed for WebGL to ImageBitmap

[Justin Novosad]

To help us iterate further, I've attempted to capture the essence of this
thread on the whatwg wiki, using the problem solving template.
I tried to capture the main ideas that we seem to agree on so far and I
started to think about how to handle special cases.

http://wiki.whatwg.org/wiki/ImageBitmap_Options

-Justin




On Thu, Jul 18, 2013 at 1:22 PM, Justin Novosad  wrote:

>
>
>
> On Thu, Jul 18, 2013 at 5:45 AM, Mark Callow 
> wrote:
>
>> On 2013/07/18 16:34, K. Gadd wrote:
>> >
>> > I understand the rationale behind gregg's suggestion for flipY, but
>> > ultimately don't know if that one makes any sense in a HTML5 context. It
>> > basically only exists because of the annoying disagreement between APIs
>> > like OpenGL and other APIs like HTML5 Canvas or Direct3D, specifically
>> > about which direction the Y axis goes.
>> It exists because of the annoying disagreement between the orientation
>> of the data in most image file formats and the default orientation for
>> textures images in OpenGL. There are a some image file formats that have
>> a bottom left orientation and there is one, extremely common, format,
>> EXIF, that includes metadata giving the visual orientation of the image.
>> The flipY item in the proposed dictionary could be handily extended to
>> an enum. E.g.,
>>
>>   * "none" - leave orientation alone
>>   * "flipY" - ignore the EXIF orientation, just flip in Y
>>   * "topLeftEXIF" - identify visual orientation from EXIF data and
>> re-order data so top-down, left-to-right processing for display
>> results in correct visual orientation
>>   * "bottomRightEXIF" - as above but ordered for bottom-up,
>> left-to-right processing
>>
>> Regards
>>
>> -Mark
>>
>
> EXIF, may be a defacto standard in some image formats we use today, but I
> think we should probably be more general and just refer to this as image
> media meta-data.  What should the default be? I hesitate between 'topLeft'
> and 'none'.
>
>
>

Re: [whatwg] Adding features needed for WebGL to ImageBitmap

[Justin Novosad]

On Thu, Jul 18, 2013 at 5:45 AM, Mark Callow wrote:

> On 2013/07/18 16:34, K. Gadd wrote:
> >
> > I understand the rationale behind gregg's suggestion for flipY, but
> > ultimately don't know if that one makes any sense in a HTML5 context. It
> > basically only exists because of the annoying disagreement between APIs
> > like OpenGL and other APIs like HTML5 Canvas or Direct3D, specifically
> > about which direction the Y axis goes.
> It exists because of the annoying disagreement between the orientation
> of the data in most image file formats and the default orientation for
> textures images in OpenGL. There are a some image file formats that have
> a bottom left orientation and there is one, extremely common, format,
> EXIF, that includes metadata giving the visual orientation of the image.
> The flipY item in the proposed dictionary could be handily extended to
> an enum. E.g.,
>
>   * "none" - leave orientation alone
>   * "flipY" - ignore the EXIF orientation, just flip in Y
>   * "topLeftEXIF" - identify visual orientation from EXIF data and
> re-order data so top-down, left-to-right processing for display
> results in correct visual orientation
>   * "bottomRightEXIF" - as above but ordered for bottom-up,
> left-to-right processing
>
> Regards
>
> -Mark
>

EXIF, may be a defacto standard in some image formats we use today, but I
think we should probably be more general and just refer to this as image
media meta-data.  What should the default be? I hesitate between 'topLeft'
and 'none'.

Re: [whatwg] Adding features needed for WebGL to ImageBitmap

[Mark Callow]

On 2013/07/18 16:34, K. Gadd wrote:
>
> I understand the rationale behind gregg's suggestion for flipY, but
> ultimately don't know if that one makes any sense in a HTML5 context. It
> basically only exists because of the annoying disagreement between APIs
> like OpenGL and other APIs like HTML5 Canvas or Direct3D, specifically
> about which direction the Y axis goes. 
It exists because of the annoying disagreement between the orientation
of the data in most image file formats and the default orientation for
textures images in OpenGL. There are a some image file formats that have
a bottom left orientation and there is one, extremely common, format,
EXIF, that includes metadata giving the visual orientation of the image.
The flipY item in the proposed dictionary could be handily extended to
an enum. E.g.,

  * "none" - leave orientation alone
  * "flipY" - ignore the EXIF orientation, just flip in Y
  * "topLeftEXIF" - identify visual orientation from EXIF data and
re-order data so top-down, left-to-right processing for display
results in correct visual orientation
  * "bottomRightEXIF" - as above but ordered for bottom-up,
left-to-right processing

Regards

-Mark

-- 
注意：この電子メールには、株式会社エイチアイの機密情報が含まれている場合
が有ります。正式なメール受信者では無い場合はメール複製、 再配信または情
報の使用を固く禁じております。エラー、手違いでこのメールを受け取られまし
たら削除を行い配信者にご連絡をお願いいたし ます.

NOTE: This electronic mail message may contain confidential and
privileged information from HI Corporation. If you are not the intended
recipient, any disclosure, photocopying, distribution or use of the
contents of the received information is prohibited. If you have received
this e-mail in error, please notify the sender immediately and
permanently delete this message and all related copies.

Re: [whatwg] Adding features needed for WebGL to ImageBitmap

[K. Gadd]

To respond on the topic of WebGL/ImageBitmap integration - and in
particular some of the features requested earlier in the thread. Apologies
if I missed a post where this stuff was already addressed directly; I
couldn't follow this thread easily because of how much context was stripped
out of replies:

Having control over when or where colorspace conversion occurs would be
tremendously valuable. Right now the only place where you have control over
this is in WebGL, and when it comes to canvas each browser seems to
implement it differently. This is already a problem for people trying to do
image processing in JavaScript; an end-user of my compiler ran into this by
writing a simple app that read pixel data out of PNGs and then discovered
that every browser had its own unique interpretation of what a simple
image's data should look like when using getImageData:

https://bugzilla.mozilla.org/show_bug.cgi?id=867594

Ultimately the core here is that without control over colorspace
conversion, any sort of deterministic image processing in HTML5 is off the
table, and you have to write your own image decoders, encoders, and
manipulation routines in JavaScript using raw typed arrays. Maybe that's
how it has to be, but it would be cool to at least support basic variations
of these use cases in Canvas since getImageData/putImageData already exist
and are fairly well-specified (other than this problem, and some nits
around source rectangles and alpha transparency).

Out of the features suggested previously in the thread, I would immediately
be able to make use of control over colorspace conversion and an ability to
opt into premultiplied alpha. Not getting premultiplied alpha, as is the
case in virtually every canvas implementation I've tried, has visible
negative consequences for image quality and also reduces the performance of
some use cases where bitmap manipulation needs to happen, due to the fact
that premultiplied alpha is the 'preferred' form for certain types of
rendering and the math works out better. I think the upsides to getting
premultiplication are the same here as they are in WebGL: faster
uploads/downloads, better results, etc.

I understand the rationale behind gregg's suggestion for flipY, but
ultimately don't know if that one makes any sense in a HTML5 context. It
basically only exists because of the annoying disagreement between APIs
like OpenGL and other APIs like HTML5 Canvas or Direct3D, specifically
about which direction the Y axis goes. Normally one would assume that you
can correct this by simply inverting heights/y coordinates in the correct
places, but when you're rendering to offscreen surfaces, the confusion over
the Y axis ends up causing you to have to do a bunch of weird things to
coordinates and sampling in order to get correct results, because your
offscreen surfaces are *actually* upside down. It's gross.

To clearly state what would make ImageBitmap useful for the use cases I
encounter and my end-users encounter:
ImageBitmap should be a canonical representation of a 2D bitmap, with a
known color space, known pixel format, known alpha representation
(premultiplied/not premultiplied), and ready for immediate rendering or
pixel data access. It's okay if it's immutable, and it's okay if
constructing one from an  or a Blob takes time, as long as once I have
an ImageBitmap I can use it to render and use it to extract pixel data
without user configuration/hardware producing unpredictable results.

Colorspace conversion would allow me to address outstanding bugs that
currently require my end users to manually strip color profiles and gamma
from their image files, and premultiplied alpha would dramatically improve
the performance of some test cases and shipped games out there based on my
compiler. (Naturally, this all requires browser vendors to implement this
stuff, so I understand that these gains would probably not manifest for
years.)

-kg

Re: [whatwg] Adding features needed for WebGL to ImageBitmap

[Kenneth Russell]

On Tue, Jul 16, 2013 at 7:16 AM, Justin Novosad  wrote:
>
>
>
> On Tue, Jul 16, 2013 at 12:25 AM, Mark Callow 
> wrote:
>>
>> On 2013/07/15 10:46, Justin Novosad wrote:
>>
>> But to circle back to your point, I agree that an exception is a good idea
>> to avoid having to hold a triplicate copy in RAM, or having to redecode
>> all
>> the time. Better to force the dev to make additional copies explicitly if
>> needed than to make a potentially uselessly costly implementation.mf
>>
>> Maybe I am misunderstanding but the only reason I can see for 3 copies (2
>> if you ignore the undecoded copy) is if you propose to ignore the specified
>> parameters when drawing the image to a 2D canvas.
>
>
> Yes, that is what I was referring to because that was suggested earlier on
> this thread. But I think it is becoming clearer that that is not the right
> thing to do.
>
>>
>> I would expect to always draw the image decoded as indicated by the
>> proposed parameters so no additional copy would be necessary. Sure the image
>> might not be correct (colors off or image upside down) but that would be a
>> programmer error.
>
>
> Exactly.  That is what I am suggesting.  If the programmer wants several
> copies of the same image with different baked-in transformations, then the
> programmer should create several ImageBitmaps explicitly.  No under the hood
> magic. It's clearer that way.

This sounds good. Additionally, the WebGL spec can be updated to state
that the parameters UNPACK_FLIP_Y_WEBGL, etc. don't apply to
ImageBitmap, so the only way to affect the decoding is with the
dictionary of options.


>> I would like to see ImageBitmap fully support WebGL so WebGL apps can use
>> a Browser's built-in image decoders. And, if the rumors are true, come IE11,
>> WebGL will be supported by all major browsers so it should be treated as a
>> first-class citizen.
>
>
> Yes!

Agree with this sentiment also.

-Ken


>>
>> Regards
>>
>> -Mark
>>
>> --
>> 注意：この電子メールには、株式会社エイチアイの機密情報が含まれている場合が有ります。正式なメール受信者では無い場合はメール複製、
>> 再配信または情報の使用を固く禁じております。エラー、手違いでこのメールを受け取られましたら削除を行い配信者にご連絡をお願いいたし ます.
>>
>> NOTE: This electronic mail message may contain confidential and privileged
>> information from HI Corporation. If you are not the intended recipient, any
>> disclosure, photocopying, distribution or use of the contents of the
>> received information is prohibited. If you have received this e-mail in
>> error, please notify the sender immediately and permanently delete this
>> message and all related copies.
>
>

Re: [whatwg] Adding features needed for WebGL to ImageBitmap

[Justin Novosad]

On Tue, Jul 16, 2013 at 12:25 AM, Mark Callow wrote:

>  On 2013/07/15 10:46, Justin Novosad wrote:
>
> But to circle back to your point, I agree that an exception is a good idea
> to avoid having to hold a triplicate copy in RAM, or having to redecode all
> the time. Better to force the dev to make additional copies explicitly if
> needed than to make a potentially uselessly costly implementation.mf
>
>  Maybe I am misunderstanding but the only reason I can see for 3 copies (2
> if you ignore the undecoded copy) is if you propose to ignore the specified
> parameters when drawing the image to a 2D canvas.
>

Yes, that is what I was referring to because that was suggested earlier on
this thread. But I think it is becoming clearer that that is not the right
thing to do.


> I would expect to always draw the image decoded as indicated by the
> proposed parameters so no additional copy would be necessary. Sure the
> image might not be correct (colors off or image upside down) but that would
> be a programmer error.
>

Exactly.  That is what I am suggesting.  If the programmer wants several
copies of the same image with different baked-in transformations, then the
programmer should create several ImageBitmaps explicitly.  No under the
hood magic. It's clearer that way.


>
> I would like to see ImageBitmap fully support WebGL so WebGL apps can use
> a Browser's built-in image decoders. And, if the rumors are true, come
> IE11, WebGL will be supported by all major browsers so it should be treated
> as a first-class citizen.
>

Yes!

> Regards
>
> -Mark
> --
>  注意：この電子メールには、株式会社エイチアイの機密情報が含まれている場合が有ります。正式なメール受信者では無い場合はメール複製、
> 再配信または情報の使用を固く禁じております。エラー、手違いでこのメールを受け取られましたら削除を行い配信者にご連絡をお願いいたし ます.
>
> NOTE: This electronic mail message may contain confidential and privileged
> information from HI Corporation. If you are not the intended recipient, any
> disclosure, photocopying, distribution or use of the contents of the
> received information is prohibited. If you have received this e-mail in
> error, please notify the sender immediately and permanently delete this
> message and all related copies.
>

Re: [whatwg] Adding features needed for WebGL to ImageBitmap

[Mark Callow]

On 2013/07/15 10:46, Justin Novosad wrote:
> But to circle back to your point, I agree that an exception is a good idea
> to avoid having to hold a triplicate copy in RAM, or having to redecode all
> the time. Better to force the dev to make additional copies explicitly if
> needed than to make a potentially uselessly costly implementation.mf
Maybe I am misunderstanding but the only reason I can see for 3 copies
(2 if you ignore the undecoded copy) is if you propose to ignore the
specified parameters when drawing the image to a 2D canvas. I would
expect to always draw the image decoded as indicated by the proposed
parameters so no additional copy would be necessary. Sure the image
might not be correct (colors off or image upside down) but that would be
a programmer error.

I would like to see ImageBitmap fully support WebGL so WebGL apps can
use a Browser's built-in image decoders. And, if the rumors are true,
come IE11, WebGL will be supported by all major browsers so it should be
treated as a first-class citizen.

Regards

-Mark

-- 
注意：この電子メールには、株式会社エイチアイの機密情報が含まれている場合
が有ります。正式なメール受信者では無い場合はメール複製、 再配信または情
報の使用を固く禁じております。エラー、手違いでこのメールを受け取られまし
たら削除を行い配信者にご連絡をお願いいたし ます.

NOTE: This electronic mail message may contain confidential and
privileged information from HI Corporation. If you are not the intended
recipient, any disclosure, photocopying, distribution or use of the
contents of the received information is prohibited. If you have received
this e-mail in error, please notify the sender immediately and
permanently delete this message and all related copies.

Re: [whatwg] Adding features needed for WebGL to ImageBitmap

[Justin Novosad]

On Sun, Jul 14, 2013 at 4:34 PM, Rik Cabanier  wrote:

>
>
> On Fri, Jul 12, 2013 at 7:18 AM, Justin Novosad  wrote:
>
>> Thanks Ken, that makes it much clearer to me.
>>
>> The main concern I have with all this is the potential for OOM crashes.
>>  I'm happy as long as the spec remains vague about what "undue latency"
>> means, so we still have the possibility of gracefully degrading performance
>> in low memory conditions by evicting prepared/decoded buffers when
>> necessary, to only hang on to compressed copies in the in-memory resource
>> cache.  As far as the decode cache is concerned, the idea I have is to give
>> priority to pixel buffers that are held by ImageBitmap objects, and only
>> evict them as a last resort.
>>
>
> So, you are OK with double storing of the image data?
> This will make Canvas 2D a lot slower since it didn't have to care about
> this before so ImageBitmap didn't have to store the original data. I'd
> rather see an exception if the options are not canvas 2d-compatible than
> having to double the memory and introduce extra processing.
>

Actually, in that comment I was referring to the double copies in RAM that
we already have today in WebKit/Blink: The image data associated with an
image element may be in RAM twice: in compressed form in the resource
cache, and in uncompressed form in the decoded image cache.

But to circle back to your point, I agree that an exception is a good idea
to avoid having to hold a triplicate copy in RAM, or having to redecode all
the time.  Better to force the dev to make additional copies explicitly if
needed than to make a potentially uselessly costly implementation.


>
>
>>
>> On Thu, Jul 11, 2013 at 4:24 PM, Kenneth Russell  wrote:
>>
>>> On Thu, Jul 11, 2013 at 8:29 AM, Justin Novosad 
>>> wrote:
>>> >
>>> >
>>> > On Wed, Jul 10, 2013 at 9:37 PM, Rik Cabanier 
>>> wrote:
>>> >>
>>> >> On Wed, Jul 10, 2013 at 5:07 PM, Ian Hickson  wrote:
>>> >>
>>> >> > On Wed, 10 Jul 2013, Kenneth Russell wrote:
>>> >> > >
>>> >> > > ImageBitmap can cleanly address all of the desired use cases
>>> simply by
>>> >> > > adding an optional dictionary of options.
>>> >> >
>>> >> > I don't think that's true. The options only make sense for WebGL --
>>> >> > flipping which pixel is the first pixel, for example, doesn't do
>>> >> > anything
>>> >> > to 2D canvas, which works at a higher level.
>>> >> >
>>> >> > (The other two options don't make much sense to me even for GL. If
>>> you
>>> >> > don't want a color space, don't set one. If you don't want an alpha
>>> >> > channel, don't set one. You control the image, after all.)
>>> >> >
>>> >> >
>>> >> > > I suspect that in the future some options will be desired even
>>> for the
>>> >> > > 2D canvas use case, and having the dictionary already specified
>>> will
>>> >> > > make that easier. There is no need to invent a new primitive and
>>> means
>>> >> > > of loading it.
>>> >> >
>>> >> > If options make sense for 2D canvas, then having ImageBitmap options
>>> >> > would
>>> >> > make sense, sure.
>>> >> >
>>> >> >
>>> >> yeah, these options seem a bit puzzling.
>>> >> From the spec:
>>> >>
>>> >> An ImageBitmap object represents a bitmap image that can be painted
>>> to a
>>> >> canvas without undue latency.
>>> >>
>>> >> note: The exact judgement of what is undue latency of this is left up
>>> to
>>> >> the implementer, but in general if making use of the bitmap requires
>>> >> network I/O, or even local disk I/O, then the latency is probably
>>> undue;
>>> >> whereas if it only requires a blocking read from a GPU or system RAM,
>>> the
>>> >> latency is probably acceptable.
>>> >>
>>> >> It seems that people see the imageBitmap as something that doesn't
>>> just
>>> >> represent in-memory pixels but that those pixels are also
>>> preprocessed so
>>> >> they can be drawn quickly. The latter is not in the spec.
>>> >>
>>> >> I think authors will be very confused by these options. What would it
>>> mean
>>> >> to pass a non-premultiplied ImageBitmap to a canvas object? Would the
>>> >> browser have to add code to support it or is it illegal?
>>> >> Maybe it's easier to add an optional parameter to createImageBitmap to
>>> >> signal if the ImageBitmap is for WebGL or for Canvas and disallow a
>>> Canvas
>>> >> ImageBitmap in WebGL and vice versa.
>>> >
>>> >
>>> > You are implying a pretty heavy imposition as to what constitutes undue
>>> > latency.
>>> > I think the spec should stay away from forcing implementations to pin
>>> > decoded image buffers in RAM (or on the GPU), so that the browser may
>>> have
>>> > some latitude in preventing out of memory exceptions. In its current
>>> form,
>>> > the spec implies that it would be acceptable for an implementation to
>>> > discard the decoded buffer and only retain the resource in encoded
>>> form in
>>> > RAM.  Do we really need to make further optimizations explicit? For
>>> example,
>>> > an implementation could prepare the image data for use with WebGL the
>>> firs

Re: [whatwg] Adding features needed for WebGL to ImageBitmap

[Rik Cabanier]

On Fri, Jul 12, 2013 at 7:18 AM, Justin Novosad  wrote:

> Thanks Ken, that makes it much clearer to me.
>
> The main concern I have with all this is the potential for OOM crashes.
>  I'm happy as long as the spec remains vague about what "undue latency"
> means, so we still have the possibility of gracefully degrading performance
> in low memory conditions by evicting prepared/decoded buffers when
> necessary, to only hang on to compressed copies in the in-memory resource
> cache.  As far as the decode cache is concerned, the idea I have is to give
> priority to pixel buffers that are held by ImageBitmap objects, and only
> evict them as a last resort.
>

So, you are OK with double storing of the image data?
This will make Canvas 2D a lot slower since it didn't have to care about
this before so ImageBitmap didn't have to store the original data. I'd
rather see an exception if the options are not canvas 2d-compatible than
having to double the memory and introduce extra processing.



>
> On Thu, Jul 11, 2013 at 4:24 PM, Kenneth Russell  wrote:
>
>> On Thu, Jul 11, 2013 at 8:29 AM, Justin Novosad  wrote:
>> >
>> >
>> > On Wed, Jul 10, 2013 at 9:37 PM, Rik Cabanier 
>> wrote:
>> >>
>> >> On Wed, Jul 10, 2013 at 5:07 PM, Ian Hickson  wrote:
>> >>
>> >> > On Wed, 10 Jul 2013, Kenneth Russell wrote:
>> >> > >
>> >> > > ImageBitmap can cleanly address all of the desired use cases
>> simply by
>> >> > > adding an optional dictionary of options.
>> >> >
>> >> > I don't think that's true. The options only make sense for WebGL --
>> >> > flipping which pixel is the first pixel, for example, doesn't do
>> >> > anything
>> >> > to 2D canvas, which works at a higher level.
>> >> >
>> >> > (The other two options don't make much sense to me even for GL. If
>> you
>> >> > don't want a color space, don't set one. If you don't want an alpha
>> >> > channel, don't set one. You control the image, after all.)
>> >> >
>> >> >
>> >> > > I suspect that in the future some options will be desired even for
>> the
>> >> > > 2D canvas use case, and having the dictionary already specified
>> will
>> >> > > make that easier. There is no need to invent a new primitive and
>> means
>> >> > > of loading it.
>> >> >
>> >> > If options make sense for 2D canvas, then having ImageBitmap options
>> >> > would
>> >> > make sense, sure.
>> >> >
>> >> >
>> >> yeah, these options seem a bit puzzling.
>> >> From the spec:
>> >>
>> >> An ImageBitmap object represents a bitmap image that can be painted to
>> a
>> >> canvas without undue latency.
>> >>
>> >> note: The exact judgement of what is undue latency of this is left up
>> to
>> >> the implementer, but in general if making use of the bitmap requires
>> >> network I/O, or even local disk I/O, then the latency is probably
>> undue;
>> >> whereas if it only requires a blocking read from a GPU or system RAM,
>> the
>> >> latency is probably acceptable.
>> >>
>> >> It seems that people see the imageBitmap as something that doesn't just
>> >> represent in-memory pixels but that those pixels are also preprocessed
>> so
>> >> they can be drawn quickly. The latter is not in the spec.
>> >>
>> >> I think authors will be very confused by these options. What would it
>> mean
>> >> to pass a non-premultiplied ImageBitmap to a canvas object? Would the
>> >> browser have to add code to support it or is it illegal?
>> >> Maybe it's easier to add an optional parameter to createImageBitmap to
>> >> signal if the ImageBitmap is for WebGL or for Canvas and disallow a
>> Canvas
>> >> ImageBitmap in WebGL and vice versa.
>> >
>> >
>> > You are implying a pretty heavy imposition as to what constitutes undue
>> > latency.
>> > I think the spec should stay away from forcing implementations to pin
>> > decoded image buffers in RAM (or on the GPU), so that the browser may
>> have
>> > some latitude in preventing out of memory exceptions. In its current
>> form,
>> > the spec implies that it would be acceptable for an implementation to
>> > discard the decoded buffer and only retain the resource in encoded form
>> in
>> > RAM.  Do we really need to make further optimizations explicit? For
>> example,
>> > an implementation could prepare the image data for use with WebGL the
>> first
>> > time it is drawn to WebGL, and keep it cached in that state. If the same
>> > ImageBitmap is subsequently drawn to a 2D canvas, then it would use the
>> > non-WebGLified copy, which may be cached, or may require re-decoding the
>> > image. No big deal.
>>
>> The step of preparing the image for use, either with WebGL or 2D
>> canvas, is expensive. Today, this step is necessarily done
>> synchronously when an HTMLImageElement is uploaded to WebGL. The
>> current ImageBitmap proposal would still require this synchronous
>> step, so for WebGL at least, it provides no improvement over the
>> current HTML5 APIs. A major goal of ImageBitmap was to allow Web
>> Workers to load them, and even this ability currently provides no
>> advantage over HTM

Re: [whatwg] Adding features needed for WebGL to ImageBitmap

[Rik Cabanier]

On Thu, Jul 11, 2013 at 1:24 PM, Kenneth Russell  wrote:

> On Thu, Jul 11, 2013 at 8:29 AM, Justin Novosad  wrote:
> >
> >
> > On Wed, Jul 10, 2013 at 9:37 PM, Rik Cabanier 
> wrote:
> >>
> >> On Wed, Jul 10, 2013 at 5:07 PM, Ian Hickson  wrote:
> >>
> >> > On Wed, 10 Jul 2013, Kenneth Russell wrote:
> >> > >
> >> > > ImageBitmap can cleanly address all of the desired use cases simply
> by
> >> > > adding an optional dictionary of options.
> >> >
> >> > I don't think that's true. The options only make sense for WebGL --
> >> > flipping which pixel is the first pixel, for example, doesn't do
> >> > anything
> >> > to 2D canvas, which works at a higher level.
> >> >
> >> > (The other two options don't make much sense to me even for GL. If you
> >> > don't want a color space, don't set one. If you don't want an alpha
> >> > channel, don't set one. You control the image, after all.)
> >> >
> >> >
> >> > > I suspect that in the future some options will be desired even for
> the
> >> > > 2D canvas use case, and having the dictionary already specified will
> >> > > make that easier. There is no need to invent a new primitive and
> means
> >> > > of loading it.
> >> >
> >> > If options make sense for 2D canvas, then having ImageBitmap options
> >> > would
> >> > make sense, sure.
> >> >
> >> >
> >> yeah, these options seem a bit puzzling.
> >> From the spec:
> >>
> >> An ImageBitmap object represents a bitmap image that can be painted to a
> >> canvas without undue latency.
> >>
> >> note: The exact judgement of what is undue latency of this is left up to
> >> the implementer, but in general if making use of the bitmap requires
> >> network I/O, or even local disk I/O, then the latency is probably undue;
> >> whereas if it only requires a blocking read from a GPU or system RAM,
> the
> >> latency is probably acceptable.
> >>
> >> It seems that people see the imageBitmap as something that doesn't just
> >> represent in-memory pixels but that those pixels are also preprocessed
> so
> >> they can be drawn quickly. The latter is not in the spec.
> >>
> >> I think authors will be very confused by these options. What would it
> mean
> >> to pass a non-premultiplied ImageBitmap to a canvas object? Would the
> >> browser have to add code to support it or is it illegal?
> >> Maybe it's easier to add an optional parameter to createImageBitmap to
> >> signal if the ImageBitmap is for WebGL or for Canvas and disallow a
> Canvas
> >> ImageBitmap in WebGL and vice versa.
> >
> >
> > You are implying a pretty heavy imposition as to what constitutes undue
> > latency.
> > I think the spec should stay away from forcing implementations to pin
> > decoded image buffers in RAM (or on the GPU), so that the browser may
> have
> > some latitude in preventing out of memory exceptions. In its current
> form,
> > the spec implies that it would be acceptable for an implementation to
> > discard the decoded buffer and only retain the resource in encoded form
> in
> > RAM.  Do we really need to make further optimizations explicit? For
> example,
> > an implementation could prepare the image data for use with WebGL the
> first
> > time it is drawn to WebGL, and keep it cached in that state. If the same
> > ImageBitmap is subsequently drawn to a 2D canvas, then it would use the
> > non-WebGLified copy, which may be cached, or may require re-decoding the
> > image. No big deal.
>
> The step of preparing the image for use, either with WebGL or 2D
> canvas, is expensive. Today, this step is necessarily done
> synchronously when an HTMLImageElement is uploaded to WebGL. The
> current ImageBitmap proposal would still require this synchronous
> step, so for WebGL at least, it provides no improvement over the
> current HTML5 APIs. A major goal of ImageBitmap was to allow Web
> Workers to load them, and even this ability currently provides no
> advantage over HTMLImageElement.
>
> > Fundamental question: Do we really need the caller to be able to specify
> > what treatments need to be applied to prepare an image for WebGL, or is
> it
> > always possible to figure that out automatically?
>
> It is never possible to figure out automatically how the image needs
> to be treated when preparing it for use with WebGL. I'm not sure where
> that idea came from.


Gregg's email says that WebGL almost always has the opposite options of
Canvas.
I was thinking that maybe it's acceptable to just make it a switch between
Canvas 2D and WebGL.

On the contrary, there are eight possibilities
> (2^3), and different applications require different combinations.
>
>

Re: [whatwg] Adding features needed for WebGL to ImageBitmap

[Justin Novosad]

Thanks Ken, that makes it much clearer to me.

The main concern I have with all this is the potential for OOM crashes.
 I'm happy as long as the spec remains vague about what "undue latency"
means, so we still have the possibility of gracefully degrading performance
in low memory conditions by evicting prepared/decoded buffers when
necessary, to only hang on to compressed copies in the in-memory resource
cache.  As far as the decode cache is concerned, the idea I have is to give
priority to pixel buffers that are held by ImageBitmap objects, and only
evict them as a last resort.


On Thu, Jul 11, 2013 at 4:24 PM, Kenneth Russell  wrote:

> On Thu, Jul 11, 2013 at 8:29 AM, Justin Novosad  wrote:
> >
> >
> > On Wed, Jul 10, 2013 at 9:37 PM, Rik Cabanier 
> wrote:
> >>
> >> On Wed, Jul 10, 2013 at 5:07 PM, Ian Hickson  wrote:
> >>
> >> > On Wed, 10 Jul 2013, Kenneth Russell wrote:
> >> > >
> >> > > ImageBitmap can cleanly address all of the desired use cases simply
> by
> >> > > adding an optional dictionary of options.
> >> >
> >> > I don't think that's true. The options only make sense for WebGL --
> >> > flipping which pixel is the first pixel, for example, doesn't do
> >> > anything
> >> > to 2D canvas, which works at a higher level.
> >> >
> >> > (The other two options don't make much sense to me even for GL. If you
> >> > don't want a color space, don't set one. If you don't want an alpha
> >> > channel, don't set one. You control the image, after all.)
> >> >
> >> >
> >> > > I suspect that in the future some options will be desired even for
> the
> >> > > 2D canvas use case, and having the dictionary already specified will
> >> > > make that easier. There is no need to invent a new primitive and
> means
> >> > > of loading it.
> >> >
> >> > If options make sense for 2D canvas, then having ImageBitmap options
> >> > would
> >> > make sense, sure.
> >> >
> >> >
> >> yeah, these options seem a bit puzzling.
> >> From the spec:
> >>
> >> An ImageBitmap object represents a bitmap image that can be painted to a
> >> canvas without undue latency.
> >>
> >> note: The exact judgement of what is undue latency of this is left up to
> >> the implementer, but in general if making use of the bitmap requires
> >> network I/O, or even local disk I/O, then the latency is probably undue;
> >> whereas if it only requires a blocking read from a GPU or system RAM,
> the
> >> latency is probably acceptable.
> >>
> >> It seems that people see the imageBitmap as something that doesn't just
> >> represent in-memory pixels but that those pixels are also preprocessed
> so
> >> they can be drawn quickly. The latter is not in the spec.
> >>
> >> I think authors will be very confused by these options. What would it
> mean
> >> to pass a non-premultiplied ImageBitmap to a canvas object? Would the
> >> browser have to add code to support it or is it illegal?
> >> Maybe it's easier to add an optional parameter to createImageBitmap to
> >> signal if the ImageBitmap is for WebGL or for Canvas and disallow a
> Canvas
> >> ImageBitmap in WebGL and vice versa.
> >
> >
> > You are implying a pretty heavy imposition as to what constitutes undue
> > latency.
> > I think the spec should stay away from forcing implementations to pin
> > decoded image buffers in RAM (or on the GPU), so that the browser may
> have
> > some latitude in preventing out of memory exceptions. In its current
> form,
> > the spec implies that it would be acceptable for an implementation to
> > discard the decoded buffer and only retain the resource in encoded form
> in
> > RAM.  Do we really need to make further optimizations explicit? For
> example,
> > an implementation could prepare the image data for use with WebGL the
> first
> > time it is drawn to WebGL, and keep it cached in that state. If the same
> > ImageBitmap is subsequently drawn to a 2D canvas, then it would use the
> > non-WebGLified copy, which may be cached, or may require re-decoding the
> > image. No big deal.
>
> The step of preparing the image for use, either with WebGL or 2D
> canvas, is expensive. Today, this step is necessarily done
> synchronously when an HTMLImageElement is uploaded to WebGL. The
> current ImageBitmap proposal would still require this synchronous
> step, so for WebGL at least, it provides no improvement over the
> current HTML5 APIs. A major goal of ImageBitmap was to allow Web
> Workers to load them, and even this ability currently provides no
> advantage over HTMLImageElement.


> > Fundamental question: Do we really need the caller to be able to specify
> > what treatments need to be applied to prepare an image for WebGL, or is
> it
> > always possible to figure that out automatically?
>
> It is never possible to figure out automatically how the image needs
> to be treated when preparing it for use with WebGL. I'm not sure where
> that idea came from. On the contrary, there are eight possibilities
> (2^3), and different applications require different combinations.
>
> -Ken

Re: [whatwg] Adding features needed for WebGL to ImageBitmap

[Kenneth Russell]

On Thu, Jul 11, 2013 at 8:29 AM, Justin Novosad  wrote:
>
>
> On Wed, Jul 10, 2013 at 9:37 PM, Rik Cabanier  wrote:
>>
>> On Wed, Jul 10, 2013 at 5:07 PM, Ian Hickson  wrote:
>>
>> > On Wed, 10 Jul 2013, Kenneth Russell wrote:
>> > >
>> > > ImageBitmap can cleanly address all of the desired use cases simply by
>> > > adding an optional dictionary of options.
>> >
>> > I don't think that's true. The options only make sense for WebGL --
>> > flipping which pixel is the first pixel, for example, doesn't do
>> > anything
>> > to 2D canvas, which works at a higher level.
>> >
>> > (The other two options don't make much sense to me even for GL. If you
>> > don't want a color space, don't set one. If you don't want an alpha
>> > channel, don't set one. You control the image, after all.)
>> >
>> >
>> > > I suspect that in the future some options will be desired even for the
>> > > 2D canvas use case, and having the dictionary already specified will
>> > > make that easier. There is no need to invent a new primitive and means
>> > > of loading it.
>> >
>> > If options make sense for 2D canvas, then having ImageBitmap options
>> > would
>> > make sense, sure.
>> >
>> >
>> yeah, these options seem a bit puzzling.
>> From the spec:
>>
>> An ImageBitmap object represents a bitmap image that can be painted to a
>> canvas without undue latency.
>>
>> note: The exact judgement of what is undue latency of this is left up to
>> the implementer, but in general if making use of the bitmap requires
>> network I/O, or even local disk I/O, then the latency is probably undue;
>> whereas if it only requires a blocking read from a GPU or system RAM, the
>> latency is probably acceptable.
>>
>> It seems that people see the imageBitmap as something that doesn't just
>> represent in-memory pixels but that those pixels are also preprocessed so
>> they can be drawn quickly. The latter is not in the spec.
>>
>> I think authors will be very confused by these options. What would it mean
>> to pass a non-premultiplied ImageBitmap to a canvas object? Would the
>> browser have to add code to support it or is it illegal?
>> Maybe it's easier to add an optional parameter to createImageBitmap to
>> signal if the ImageBitmap is for WebGL or for Canvas and disallow a Canvas
>> ImageBitmap in WebGL and vice versa.
>
>
> You are implying a pretty heavy imposition as to what constitutes undue
> latency.
> I think the spec should stay away from forcing implementations to pin
> decoded image buffers in RAM (or on the GPU), so that the browser may have
> some latitude in preventing out of memory exceptions. In its current form,
> the spec implies that it would be acceptable for an implementation to
> discard the decoded buffer and only retain the resource in encoded form in
> RAM.  Do we really need to make further optimizations explicit? For example,
> an implementation could prepare the image data for use with WebGL the first
> time it is drawn to WebGL, and keep it cached in that state. If the same
> ImageBitmap is subsequently drawn to a 2D canvas, then it would use the
> non-WebGLified copy, which may be cached, or may require re-decoding the
> image. No big deal.

The step of preparing the image for use, either with WebGL or 2D
canvas, is expensive. Today, this step is necessarily done
synchronously when an HTMLImageElement is uploaded to WebGL. The
current ImageBitmap proposal would still require this synchronous
step, so for WebGL at least, it provides no improvement over the
current HTML5 APIs. A major goal of ImageBitmap was to allow Web
Workers to load them, and even this ability currently provides no
advantage over HTMLImageElement.

> Fundamental question: Do we really need the caller to be able to specify
> what treatments need to be applied to prepare an image for WebGL, or is it
> always possible to figure that out automatically?

It is never possible to figure out automatically how the image needs
to be treated when preparing it for use with WebGL. I'm not sure where
that idea came from. On the contrary, there are eight possibilities
(2^3), and different applications require different combinations.

-Ken

Re: [whatwg] Adding features needed for WebGL to ImageBitmap

[Rik Cabanier]

On Thu, Jul 11, 2013 at 8:29 AM, Justin Novosad  wrote:

>
>
> On Wed, Jul 10, 2013 at 9:37 PM, Rik Cabanier  wrote:
>
>> On Wed, Jul 10, 2013 at 5:07 PM, Ian Hickson  wrote:
>>
>> > On Wed, 10 Jul 2013, Kenneth Russell wrote:
>> > >
>> > > ImageBitmap can cleanly address all of the desired use cases simply by
>> > > adding an optional dictionary of options.
>> >
>> > I don't think that's true. The options only make sense for WebGL --
>> > flipping which pixel is the first pixel, for example, doesn't do
>> anything
>> > to 2D canvas, which works at a higher level.
>> >
>> > (The other two options don't make much sense to me even for GL. If you
>> > don't want a color space, don't set one. If you don't want an alpha
>> > channel, don't set one. You control the image, after all.)
>> >
>> >
>> > > I suspect that in the future some options will be desired even for the
>> > > 2D canvas use case, and having the dictionary already specified will
>> > > make that easier. There is no need to invent a new primitive and means
>> > > of loading it.
>> >
>> > If options make sense for 2D canvas, then having ImageBitmap options
>> would
>> > make sense, sure.
>> >
>> >
>> yeah, these options seem a bit puzzling.
>> From the spec:
>>
>> An ImageBitmap object represents a bitmap image that can be painted to a
>> canvas without undue latency.
>>
>> note: The exact judgement of what is undue latency of this is left up to
>> the implementer, but in general if making use of the bitmap requires
>> network I/O, or even local disk I/O, then the latency is probably undue;
>> whereas if it only requires a blocking read from a GPU or system RAM, the
>> latency is probably acceptable.
>>
>> It seems that people see the imageBitmap as something that doesn't just
>> represent in-memory pixels but that those pixels are also preprocessed so
>> they can be drawn quickly. The latter is not in the spec.
>>
>> I think authors will be very confused by these options. What would it mean
>> to pass a non-premultiplied ImageBitmap to a canvas object? Would the
>> browser have to add code to support it or is it illegal?
>> Maybe it's easier to add an optional parameter to createImageBitmap to
>> signal if the ImageBitmap is for WebGL or for Canvas and disallow a Canvas
>> ImageBitmap in WebGL and vice versa.
>>
>
> You are implying a pretty heavy imposition as to what constitutes undue
> latency.
> I think the spec should stay away from forcing implementations to pin
> decoded image buffers in RAM (or on the GPU), so that the browser may have
> some latitude in preventing out of memory exceptions. In its current form,
> the spec implies that it would be acceptable for an implementation to
> discard the decoded buffer and only retain the resource in encoded form in
> RAM.  Do we really need to make further optimizations explicit? For
> example, an implementation could prepare the image data for use with WebGL
> the first time it is drawn to WebGL, and keep it cached in that state. If
> the same ImageBitmap is subsequently drawn to a 2D canvas, then it would
> use the non-WebGLified copy, which may be cached, or may require
> re-decoding the image. No big deal.
>

Doesn't that double the memory requirement?
Also, since bitmaps will need to be prepared for either WebGL or Canvas 2d,
won't that introduce "undue latency"?


> Fundamental question: Do we really need the caller to be able to specify
> what treatments need to be applied to prepare an image for WebGL, or is it
> always possible to figure that out automatically? As long as there is a way
> to do it automatically, shouldn't we avoid adding complexity to the API?
>

I agree.  However, since Gregg was asking for this features, he must
believe there's a need for this feature.

Re: [whatwg] Adding features needed for WebGL to ImageBitmap

[Justin Novosad]

On Wed, Jul 10, 2013 at 9:37 PM, Rik Cabanier  wrote:

> On Wed, Jul 10, 2013 at 5:07 PM, Ian Hickson  wrote:
>
> > On Wed, 10 Jul 2013, Kenneth Russell wrote:
> > >
> > > ImageBitmap can cleanly address all of the desired use cases simply by
> > > adding an optional dictionary of options.
> >
> > I don't think that's true. The options only make sense for WebGL --
> > flipping which pixel is the first pixel, for example, doesn't do anything
> > to 2D canvas, which works at a higher level.
> >
> > (The other two options don't make much sense to me even for GL. If you
> > don't want a color space, don't set one. If you don't want an alpha
> > channel, don't set one. You control the image, after all.)
> >
> >
> > > I suspect that in the future some options will be desired even for the
> > > 2D canvas use case, and having the dictionary already specified will
> > > make that easier. There is no need to invent a new primitive and means
> > > of loading it.
> >
> > If options make sense for 2D canvas, then having ImageBitmap options
> would
> > make sense, sure.
> >
> >
> yeah, these options seem a bit puzzling.
> From the spec:
>
> An ImageBitmap object represents a bitmap image that can be painted to a
> canvas without undue latency.
>
> note: The exact judgement of what is undue latency of this is left up to
> the implementer, but in general if making use of the bitmap requires
> network I/O, or even local disk I/O, then the latency is probably undue;
> whereas if it only requires a blocking read from a GPU or system RAM, the
> latency is probably acceptable.
>
> It seems that people see the imageBitmap as something that doesn't just
> represent in-memory pixels but that those pixels are also preprocessed so
> they can be drawn quickly. The latter is not in the spec.
>
> I think authors will be very confused by these options. What would it mean
> to pass a non-premultiplied ImageBitmap to a canvas object? Would the
> browser have to add code to support it or is it illegal?
> Maybe it's easier to add an optional parameter to createImageBitmap to
> signal if the ImageBitmap is for WebGL or for Canvas and disallow a Canvas
> ImageBitmap in WebGL and vice versa.
>

You are implying a pretty heavy imposition as to what constitutes undue
latency.
I think the spec should stay away from forcing implementations to pin
decoded image buffers in RAM (or on the GPU), so that the browser may have
some latitude in preventing out of memory exceptions. In its current form,
the spec implies that it would be acceptable for an implementation to
discard the decoded buffer and only retain the resource in encoded form in
RAM.  Do we really need to make further optimizations explicit? For
example, an implementation could prepare the image data for use with WebGL
the first time it is drawn to WebGL, and keep it cached in that state. If
the same ImageBitmap is subsequently drawn to a 2D canvas, then it would
use the non-WebGLified copy, which may be cached, or may require
re-decoding the image. No big deal.

Fundamental question: Do we really need the caller to be able to specify
what treatments need to be applied to prepare an image for WebGL, or is it
always possible to figure that out automatically? As long as there is a way
to do it automatically, shouldn't we avoid adding complexity to the API?

Re: [whatwg] Adding features needed for WebGL to ImageBitmap

[Rik Cabanier]

On Wed, Jul 10, 2013 at 5:07 PM, Ian Hickson  wrote:

> On Wed, 10 Jul 2013, Kenneth Russell wrote:
> >
> > ImageBitmap can cleanly address all of the desired use cases simply by
> > adding an optional dictionary of options.
>
> I don't think that's true. The options only make sense for WebGL --
> flipping which pixel is the first pixel, for example, doesn't do anything
> to 2D canvas, which works at a higher level.
>
> (The other two options don't make much sense to me even for GL. If you
> don't want a color space, don't set one. If you don't want an alpha
> channel, don't set one. You control the image, after all.)
>
>
> > I suspect that in the future some options will be desired even for the
> > 2D canvas use case, and having the dictionary already specified will
> > make that easier. There is no need to invent a new primitive and means
> > of loading it.
>
> If options make sense for 2D canvas, then having ImageBitmap options would
> make sense, sure.
>
>
yeah, these options seem a bit puzzling.
>From the spec:

An ImageBitmap object represents a bitmap image that can be painted to a
canvas without undue latency.

note: The exact judgement of what is undue latency of this is left up to
the implementer, but in general if making use of the bitmap requires
network I/O, or even local disk I/O, then the latency is probably undue;
whereas if it only requires a blocking read from a GPU or system RAM, the
latency is probably acceptable.

It seems that people see the imageBitmap as something that doesn't just
represent in-memory pixels but that those pixels are also preprocessed so
they can be drawn quickly. The latter is not in the spec.

I think authors will be very confused by these options. What would it mean
to pass a non-premultiplied ImageBitmap to a canvas object? Would the
browser have to add code to support it or is it illegal?
Maybe it's easier to add an optional parameter to createImageBitmap to
signal if the ImageBitmap is for WebGL or for Canvas and disallow a Canvas
ImageBitmap in WebGL and vice versa.

Re: [whatwg] Adding features needed for WebGL to ImageBitmap

[Kenneth Russell]

On Wed, Jul 10, 2013 at 5:13 PM, Peter Kasting  wrote:
> On Wed, Jul 10, 2013 at 5:07 PM, Ian Hickson  wrote:
>>
>> (The other two options don't make much sense to me even for GL. If you
>> don't want a color space, don't set one. If you don't want an alpha
>> channel, don't set one. You control the image, after all.)
>
>
> I only have a small amount of graphics experience, but I don't think that
> latter comment is right, at least.
>
> At least for the alpha channel, as Gregg already wrote, a lot of GL
> algorithms use that data for something per-pixel that's not "alpha"
> (generally some other kind of per-pixel map).  It's not appropriate for the
> browser to assume that it's safe to muck with the values there.  Fixing this
> by instead trying to pass these values separate from the rest of the pixel
> data is inefficient as well as just weird from the perspective of anyone
> with significant experience in using these sorts of algorithms.

This is correct. Further, even if an image doesn't contain any color
space information, the browser may still incorrectly decide to adjust
the colorspace of decoded image data based on the client machine's
settings. It's required to be able to tell the browser to not do this.

I would find it really discouraging if the WebGL spec had to subsume
image loading functionality. It's a statement of fact that with a
dictionary of options, ImageBitmap can work as efficiently for the
WebGL canvas context as it's intended to for the 2D canvas context.

-Ken

Re: [whatwg] Adding features needed for WebGL to ImageBitmap

[Peter Kasting]

On Wed, Jul 10, 2013 at 5:07 PM, Ian Hickson  wrote:

> (The other two options don't make much sense to me even for GL. If you
> don't want a color space, don't set one. If you don't want an alpha
> channel, don't set one. You control the image, after all.)


I only have a small amount of graphics experience, but I don't think that
latter comment is right, at least.

At least for the alpha channel, as Gregg already wrote, a lot of GL
algorithms use that data for something per-pixel that's not "alpha"
(generally some other kind of per-pixel map).  It's not appropriate for the
browser to assume that it's safe to muck with the values there.  Fixing
this by instead trying to pass these values separate from the rest of the
pixel data is inefficient as well as just weird from the perspective of
anyone with significant experience in using these sorts of algorithms.

PK

Re: [whatwg] Adding features needed for WebGL to ImageBitmap

[Ian Hickson]

On Wed, 10 Jul 2013, Kenneth Russell wrote:
> 
> ImageBitmap can cleanly address all of the desired use cases simply by 
> adding an optional dictionary of options.

I don't think that's true. The options only make sense for WebGL -- 
flipping which pixel is the first pixel, for example, doesn't do anything 
to 2D canvas, which works at a higher level.

(The other two options don't make much sense to me even for GL. If you 
don't want a color space, don't set one. If you don't want an alpha 
channel, don't set one. You control the image, after all.)


> I suspect that in the future some options will be desired even for the 
> 2D canvas use case, and having the dictionary already specified will 
> make that easier. There is no need to invent a new primitive and means 
> of loading it.

If options make sense for 2D canvas, then having ImageBitmap options would 
make sense, sure.

-- 
Ian Hickson   U+1047E)\._.,--,'``.fL
http://ln.hixie.ch/   U+263A/,   _.. \   _\  ;`._ ,.
Things that are impossible just take longer.   `._.-(,_..'--(,_..'`-.;.'

Re: [whatwg] Adding features needed for WebGL to ImageBitmap

[Kenneth Russell]

On Wed, Jul 10, 2013 at 4:37 PM, Ian Hickson  wrote:
> On Wed, 10 Jul 2013, Kenneth Russell wrote:
>>
>> Some background: when uploading HTMLImageElements to WebGL it's required
>> to be able to specify certain options, such as whether to premultiply
>> the alpha channel, or perform colorspace conversion. Because it seemed
>> infeasible at the time to modify the HTML spec, these options are set
>> via the WebGL API. If they're set differently from the browser's
>> defaults (which are generally to do premultiplication, and do colorspace
>> conversion), then the WebGL implementation has to re-decode the image
>> when it's uploaded to a WebGL texture. (There's no way to know in
>> advance whether a given image is intended for upload to WebGL as opposed
>> to insertion into the document, and making image decoding lazier than it
>> currently is would introduce bad hiccups while scrolling.)
>
> It seems like the right solution is to create a primitive for WebGL that
> represents images that are going to be used in WebGL calls. Such a
> primitive could use the same sources for images as ImageBitmap, but would
> be specifically for use with WebGL, in the same way that ImageBitmap is
> used just by the 2D Canvas API.

That sounds like the wrong solution to me. The goal of HTML5 should be
good integration of all of the component APIs, not to treat some, like
WebGL, as bolt-on mechanisms.

ImageBitmap can cleanly address all of the desired use cases simply by
adding an optional dictionary of options. I suspect that in the future
some options will be desired even for the 2D canvas use case, and
having the dictionary already specified will make that easier. There
is no need to invent a new primitive and means of loading it.

-Ken


> --
> Ian Hickson   U+1047E)\._.,--,'``.fL
> http://ln.hixie.ch/   U+263A/,   _.. \   _\  ;`._ ,.
> Things that are impossible just take longer.   `._.-(,_..'--(,_..'`-.;.'

Re: [whatwg] Adding features needed for WebGL to ImageBitmap

[Ian Hickson]

On Wed, 10 Jul 2013, Kenneth Russell wrote:
> 
> Some background: when uploading HTMLImageElements to WebGL it's required 
> to be able to specify certain options, such as whether to premultiply 
> the alpha channel, or perform colorspace conversion. Because it seemed 
> infeasible at the time to modify the HTML spec, these options are set 
> via the WebGL API. If they're set differently from the browser's 
> defaults (which are generally to do premultiplication, and do colorspace 
> conversion), then the WebGL implementation has to re-decode the image 
> when it's uploaded to a WebGL texture. (There's no way to know in 
> advance whether a given image is intended for upload to WebGL as opposed 
> to insertion into the document, and making image decoding lazier than it 
> currently is would introduce bad hiccups while scrolling.)

It seems like the right solution is to create a primitive for WebGL that 
represents images that are going to be used in WebGL calls. Such a 
primitive could use the same sources for images as ImageBitmap, but would 
be specifically for use with WebGL, in the same way that ImageBitmap is 
used just by the 2D Canvas API.

-- 
Ian Hickson   U+1047E)\._.,--,'``.fL
http://ln.hixie.ch/   U+263A/,   _.. \   _\  ;`._ ,.
Things that are impossible just take longer.   `._.-(,_..'--(,_..'`-.;.'

Re: [whatwg] Adding features needed for WebGL to ImageBitmap

[Kenneth Russell]

(Replying on behalf of Gregg, who unfortunately isn't at Google any more)

On Wed, Jul 10, 2013 at 3:17 PM, Ian Hickson  wrote:
> On Wed, 19 Jun 2013, Gregg Tavares wrote:
>>
>> In order for ImageBitmap to be useful for WebGL we need more options
>
> ImageBitmap is trying to just be a generic HTMLImageElement, that is, a
> bitmap image. It's not trying to be anything more than that.
>
> Based on some of these questions, though, maybe you mean ImageData?

Gregg meant ImageBitmap.

Some background: when uploading HTMLImageElements to WebGL it's
required to be able to specify certain options, such as whether to
premultiply the alpha channel, or perform colorspace conversion.
Because it seemed infeasible at the time to modify the HTML spec,
these options are set via the WebGL API. If they're set differently
from the browser's defaults (which are generally to do
premultiplication, and do colorspace conversion), then the WebGL
implementation has to re-decode the image when it's uploaded to a
WebGL texture. (There's no way to know in advance whether a given
image is intended for upload to WebGL as opposed to insertion into the
document, and making image decoding lazier than it currently is would
introduce bad hiccups while scrolling.)

We'd like to avoid the same problems with the new ImageBitmap concept.

The current ImageBitmap draft has the problem that when the callback
is called, image decoding will already have been done, just like
HTMLImageElement -- at least, this is almost surely how it'll be
implemented, in order to obey the rule "An ImageBitmap object
represents a bitmap image that can be painted to a canvas without
undue latency". Just like HTMLImageElement, these options need to be
set before decoding occurs, to avoid redundant work and rendering
pauses which would happen if operations like colorspace conversion
were done lazily. (By the way, colorspace conversion is typically
implemented inside the image decoder itself, and it would be a lot of
work to factor it out into code which can be applied to a
previously-decoded image. In fact from looking again at the code in
Blink which does this I'd say it's completely infeasible.)


>> premultipliedAlpha: true/false (default true)
>> Nearly all GL games use non-premultipiled alpha textures. So all those
>> games people want to port to WebGL will require non-premultipied textures.
>> Often in games the alpha might not even be used for alpha but rather for
>> glow maps or specular maps or the other kinds of data.
>
> How do you do this with  today?

Per above, by specifying the option via the WebGL API, and performing
a synchronous image re-decode. This re-decode is really expensive, and
a major pain point for WebGL developers. It's so bad that developers
are using pure JavaScript decoders for PNG and JPG formats just so
that they can do this on a worker thread.


>> flipY: true/false (default false)
>> Nearly all 3D modeling apps expect the bottom left pixel to be the first
>> pixel in a texture so many 3D engines flip the textures on load. WebGL
>> provides this option but it takes time and memory to flip a large image
>> therefore it would be nice if that flip happened before the callback
>> from ImageBitmap
>
> No pixel is the first pixel in an ImageBitmap. I don't really understand
> what this means.

There's a longstanding difference between the coordinate systems used
by most 2D libraries, and 3D APIs. OpenGL in particular long ago
adopted the convention that the origin of a texture is its lower-left
corner, with the Y axis pointing up.

Every image loading library ever created for OpenGL has had an option
to flip (or not) loaded textures along the Y axis; the option is
required to support pipelines for loading artists' work.

The WebGL spec offers this option via the UNPACK_FLIP_Y_WEBGL state.
http://www.khronos.org/registry/webgl/specs/latest/#TEXIMAGE2D_HTML
defines that the upper left pixel of images is by default the first
pixel transferred to the GPU.

Flipping large images vertically is expensive, taking a significant
percentage of frame time. As with premultiplication of alpha, we want
to avoid doing it unnecessarily, redundantly, or synchronously with
respect to the application. For this reason we want to make it an
option on createImageBitmap so when the callback is called, the
decoded image data is already oriented properly for upload to the GPU.


>> colorspaceConversion: true/false (default true)
>> Some browsers apply color space conversion to match monitor settings.
>> That's fine for images with color but WebGL apps often load heightmaps,
>> normalmaps, lightmaps, global illumination maps and many other kinds of
>> data through images. If the browser applies a colorspace conversion the
>> data is not longer suitable for it's intended purpose therefore many WebGL
>> apps turn off color conversions. As it is now, when an image is uploaded to
>> WebGL, if colorspace conversion is
>> off

Re: [whatwg] Adding features needed for WebGL to ImageBitmap

[Ian Hickson]

On Wed, 19 Jun 2013, Gregg Tavares wrote:
>
> In order for ImageBitmap to be useful for WebGL we need more options

ImageBitmap is trying to just be a generic HTMLImageElement, that is, a 
bitmap image. It's not trying to be anything more than that.

Based on some of these questions, though, maybe you mean ImageData?


> premultipliedAlpha: true/false (default true)
> Nearly all GL games use non-premultipiled alpha textures. So all those
> games people want to port to WebGL will require non-premultipied textures.
> Often in games the alpha might not even be used for alpha but rather for
> glow maps or specular maps or the other kinds of data.

How do you do this with  today?


> flipY: true/false (default false)
> Nearly all 3D modeling apps expect the bottom left pixel to be the first 
> pixel in a texture so many 3D engines flip the textures on load. WebGL 
> provides this option but it takes time and memory to flip a large image 
> therefore it would be nice if that flip happened before the callback 
> from ImageBitmap

No pixel is the first pixel in an ImageBitmap. I don't really understand 
what this means.


> colorspaceConversion: true/false (default true)
> Some browsers apply color space conversion to match monitor settings.
> That's fine for images with color but WebGL apps often load heightmaps,
> normalmaps, lightmaps, global illumination maps and many other kinds of
> data through images. If the browser applies a colorspace conversion the
> data is not longer suitable for it's intended purpose therefore many WebGL
> apps turn off color conversions. As it is now, when an image is uploaded to
> WebGL, if colorspace conversion is
> off,
> WebGL has to synchronously re-decode the image. It would be nice if
> ImageBitmap could handle this case so it can decode the image without
> applying any colorspace manipulations.

ImageBitmap doesn't apply any colour space manipulation. That's only done 
when drawing, according to the spec.


On Wed, 19 Jun 2013, Gregg Tavares wrote:
> 
> colorspaceConversion: true   = browser does whatever it thinks is best for
> color images.
> colorspaceConversion: false  = give me the bits in the image file. Don't
> manipulate them with either embedded color data or local machine gamma
> corrections or anything else.

This seems like something that should apply when _using_ the image, not in 
the API that just represents the raw image data.

We could provide a way to say "strip color space information from any 
images loaded this way", but I don't understand why you'd include color 
space information that was wrong in the first place.


> c = document.createElement("canvas");
> ctx = c.getContext("2d");
> i = ctx.getImageData(0, 0, 1, 1);
> i.data[0] = 255;
> ctx.putImageData(i, 0, 0);
> i2 = ctx.getImageData(0, 0, 1, 1);
> console.log(i2.data[0])  // prints 0 on both FF and Chrome

This is using ImageData, not ImageBitmap. Are we talking about the same 
thing here? I'm confused.

-- 
Ian Hickson   U+1047E)\._.,--,'``.fL
http://ln.hixie.ch/   U+263A/,   _.. \   _\  ;`._ ,.
Things that are impossible just take longer.   `._.-(,_..'--(,_..'`-.;.'

Re: [whatwg] Adding features needed for WebGL to ImageBitmap

[Rik Cabanier]

On Wed, Jun 19, 2013 at 4:47 PM, Gregg Tavares  wrote:

>
>
>
> On Wed, Jun 19, 2013 at 4:03 PM, Rik Cabanier  wrote:
>
>>
>>
>> On Wed, Jun 19, 2013 at 3:24 PM, Gregg Tavares  wrote:
>>
>>>
>>>
>>>
>>> On Wed, Jun 19, 2013 at 3:13 PM, Rik Cabanier wrote:
>>>

 On Wed, Jun 19, 2013 at 2:47 PM, Gregg Tavares  wrote:

> In order for ImageBitmap to be useful for WebGL we need more options
>
> Specifically
>
> premultipliedAlpha: true/false (default true)
> Nearly all GL games use non-premultipiled alpha textures. So all those
> games people want to port to WebGL will require non-premultipied
> textures.
> Often in games the alpha might not even be used for alpha but rather
> for
> glow maps or specular maps or the other kinds of data.
>

 When would premultipliedAlpha ever be true?
 2D Canvas always works with non-premultiplied alpha in its APIs.

>>>
>>> AFAIK the canvas API expects all images to be premultiplied. Certainly
>>> in WebKit and Blink images used in the canvas and displayed in the img tag
>>> are loaded premultiplied which is why we had to add the option on WebGL
>>> since we needed those images before they had lost data.
>>>
>>>


>
> flipY: true/false (default false)
> Nearly all 3D modeling apps expect the bottom left pixel to be the
> first
> pixel in a texture so many 3D engines flip the textures on load. WebGL
> provides this option but it takes time and memory to flip a large image
> therefore it would be nice if that flip happened before the callback
> from
> ImageBitmap
>

 Couldn't you just draw upside down?

>>>
>>> No, games often animate texture coordinates and other things making it
>>> far more complicated. There are ways to work around this issue in code yes,
>>> they often require a ton of work.
>>>
>>> Most professional game engines pre-process the textures and flip them
>>> offline but that doesn't help when you're downloading models off say
>>> http://sketchup.google.com/3dwarehouse/
>>>
>>>


>
> colorspaceConversion: true/false (default true)
> Some browsers apply color space conversion to match monitor settings.
> That's fine for images with color but WebGL apps often load heightmaps,
> normalmaps, lightmaps, global illumination maps and many other kinds of
> data through images. If the browser applies a colorspace conversion the
> data is not longer suitable for it's intended purpose therefore many
> WebGL
> apps turn off color conversions. As it is now, when an image is
> uploaded to
> WebGL, if colorspace conversion is
> off<
> http://www.khronos.org/registry/webgl/specs/latest/#PIXEL_STORAGE_PARAMETERS
> >,


>> OK, I see what you're trying to accomplish. You want to pass
>> non-premultiplied data and color converted (from sRGB to monitor) pixels to
>> WebGL
>> I think your API looks fine, except that the defaults should all be
>> false...
>>
>
> Yes, that's what I meant. I think I choose bad labels. the intent of the
> colorspaceConversion flag is
>
> colorspaceConversion: true   = browser does whatever it thinks is best for
> color images.
> colorspaceConversion: false  = give me the bits in the image file. Don't
> manipulate them with either embedded color data or local machine gamma
> corrections or anything else.
>
> So maybe a better name?
>

yeah. I'm still fuzzy about what exactly you're trying to do. Is this
specified anywhere?


>
> for premultipiedAlpha again, maybe there are 3 options needed
>
> 1) do whatever is best for drawing with drawImage for perf
> 2) give me the data with premutlipied alpha
> 3) give me the data with non-premultied alpha.
>

I think that sounds good. That way implementations aren't forced to
premultiply the alpha.
Maybe not passing premultipiedAlpha in the dictionary (= leave it
undefined) will signal this.


>
> It's possible that #1 is not needed as maybe GPU code can use different
> blend modes for drawImage with non-premultpiled alpha. It's just my
> understanding that at least in Chrome all images are loaded premultiplied.
> In fact I don't think you can get non-premultipied data from canvas. At
> least this does not make it appear that way
>
> c = document.createElement("canvas");
> ctx = c.getContext("2d");
> i = ctx.getImageData(0, 0, 1, 1);
> i.data[0] = 255;
> ctx.putImageData(i, 0, 0);
> i2 = ctx.getImageData(0, 0, 1, 1);
> console.log(i2.data[0])  // prints 0 on both FF and Chrome
>
> I mean I know you get unpremultiplied data from getIamgeData but the data
> in the canvas is premultipied which means if alpha is zero you lose the RGB
> data. In other words a round trip of putImageData, getImageData is lossy
> for any non 255 alpha.  Change the above to
>
> c = document.createElement("canvas");
> ctx = c.getContext("2d");
> i = ctx.getImageData(0, 0, 1, 1);
> i.data[0] = 128

Re: [whatwg] Adding features needed for WebGL to ImageBitmap

[Gregg Tavares]

On Wed, Jun 19, 2013 at 4:03 PM, Rik Cabanier  wrote:

>
>
> On Wed, Jun 19, 2013 at 3:24 PM, Gregg Tavares  wrote:
>
>>
>>
>>
>> On Wed, Jun 19, 2013 at 3:13 PM, Rik Cabanier  wrote:
>>
>>>
>>> On Wed, Jun 19, 2013 at 2:47 PM, Gregg Tavares  wrote:
>>>
 In order for ImageBitmap to be useful for WebGL we need more options

 Specifically

 premultipliedAlpha: true/false (default true)
 Nearly all GL games use non-premultipiled alpha textures. So all those
 games people want to port to WebGL will require non-premultipied
 textures.
 Often in games the alpha might not even be used for alpha but rather for
 glow maps or specular maps or the other kinds of data.

>>>
>>> When would premultipliedAlpha ever be true?
>>> 2D Canvas always works with non-premultiplied alpha in its APIs.
>>>
>>
>> AFAIK the canvas API expects all images to be premultiplied. Certainly in
>> WebKit and Blink images used in the canvas and displayed in the img tag are
>> loaded premultiplied which is why we had to add the option on WebGL since
>> we needed those images before they had lost data.
>>
>>
>>>
>>>

 flipY: true/false (default false)
 Nearly all 3D modeling apps expect the bottom left pixel to be the first
 pixel in a texture so many 3D engines flip the textures on load. WebGL
 provides this option but it takes time and memory to flip a large image
 therefore it would be nice if that flip happened before the callback
 from
 ImageBitmap

>>>
>>> Couldn't you just draw upside down?
>>>
>>
>> No, games often animate texture coordinates and other things making it
>> far more complicated. There are ways to work around this issue in code yes,
>> they often require a ton of work.
>>
>> Most professional game engines pre-process the textures and flip them
>> offline but that doesn't help when you're downloading models off say
>> http://sketchup.google.com/3dwarehouse/
>>
>>
>>>
>>>

 colorspaceConversion: true/false (default true)
 Some browsers apply color space conversion to match monitor settings.
 That's fine for images with color but WebGL apps often load heightmaps,
 normalmaps, lightmaps, global illumination maps and many other kinds of
 data through images. If the browser applies a colorspace conversion the
 data is not longer suitable for it's intended purpose therefore many
 WebGL
 apps turn off color conversions. As it is now, when an image is
 uploaded to
 WebGL, if colorspace conversion is
 off<
 http://www.khronos.org/registry/webgl/specs/latest/#PIXEL_STORAGE_PARAMETERS
 >,
>>>
>>>
> OK, I see what you're trying to accomplish. You want to pass
> non-premultiplied data and color converted (from sRGB to monitor) pixels to
> WebGL
> I think your API looks fine, except that the defaults should all be
> false...
>

Yes, that's what I meant. I think I choose bad labels. the intent of the
colorspaceConversion flag is

colorspaceConversion: true   = browser does whatever it thinks is best for
color images.
colorspaceConversion: false  = give me the bits in the image file. Don't
manipulate them with either embedded color data or local machine gamma
corrections or anything else.

So maybe a better name?

for premultipiedAlpha again, maybe there are 3 options needed

1) do whatever is best for drawing with drawImage for perf
2) give me the data with premutlipied alpha
3) give me the data with non-premultied alpha.

It's possible that #1 is not needed as maybe GPU code can use different
blend modes for drawImage with non-premultpiled alpha. It's just my
understanding that at least in Chrome all images are loaded premultiplied.
In fact I don't think you can get non-premultipied data from canvas. At
least this does not make it appear that way

c = document.createElement("canvas");
ctx = c.getContext("2d");
i = ctx.getImageData(0, 0, 1, 1);
i.data[0] = 255;
ctx.putImageData(i, 0, 0);
i2 = ctx.getImageData(0, 0, 1, 1);
console.log(i2.data[0])  // prints 0 on both FF and Chrome

I mean I know you get unpremultiplied data from getIamgeData but the data
in the canvas is premultipied which means if alpha is zero you lose the RGB
data. In other words a round trip of putImageData, getImageData is lossy
for any non 255 alpha.  Change the above to

c = document.createElement("canvas");
ctx = c.getContext("2d");
i = ctx.getImageData(0, 0, 1, 1);
i.data[0] = 128;
i.data[3] = 1;
ctx.putImageData(i, 0, 0);
i2 = ctx.getImageData(0, 0, 1, 1);
console.log(i2.data[0])  // prints 255.

This is because the data in the canvas is premutliplied and it's being
un-premultiplied when calling getImageData











>
>
>>
 WebGL has to synchronously re-decode the image. It would be nice if
 ImageBitmap could handle this case so it can decode the image without
 applying any colorspace manipulations.

>>>
>>> Shouldn't the color space conversion happen 

Re: [whatwg] Adding features needed for WebGL to ImageBitmap

[Rik Cabanier]

On Wed, Jun 19, 2013 at 3:24 PM, Gregg Tavares  wrote:

>
>
>
> On Wed, Jun 19, 2013 at 3:13 PM, Rik Cabanier  wrote:
>
>>
>> On Wed, Jun 19, 2013 at 2:47 PM, Gregg Tavares  wrote:
>>
>>> In order for ImageBitmap to be useful for WebGL we need more options
>>>
>>> Specifically
>>>
>>> premultipliedAlpha: true/false (default true)
>>> Nearly all GL games use non-premultipiled alpha textures. So all those
>>> games people want to port to WebGL will require non-premultipied
>>> textures.
>>> Often in games the alpha might not even be used for alpha but rather for
>>> glow maps or specular maps or the other kinds of data.
>>>
>>
>> When would premultipliedAlpha ever be true?
>> 2D Canvas always works with non-premultiplied alpha in its APIs.
>>
>
> AFAIK the canvas API expects all images to be premultiplied. Certainly in
> WebKit and Blink images used in the canvas and displayed in the img tag are
> loaded premultiplied which is why we had to add the option on WebGL since
> we needed those images before they had lost data.
>
>
>>
>>
>>>
>>> flipY: true/false (default false)
>>> Nearly all 3D modeling apps expect the bottom left pixel to be the first
>>> pixel in a texture so many 3D engines flip the textures on load. WebGL
>>> provides this option but it takes time and memory to flip a large image
>>> therefore it would be nice if that flip happened before the callback from
>>> ImageBitmap
>>>
>>
>> Couldn't you just draw upside down?
>>
>
> No, games often animate texture coordinates and other things making it far
> more complicated. There are ways to work around this issue in code yes,
> they often require a ton of work.
>
> Most professional game engines pre-process the textures and flip them
> offline but that doesn't help when you're downloading models off say
> http://sketchup.google.com/3dwarehouse/
>
>
>>
>>
>>>
>>> colorspaceConversion: true/false (default true)
>>> Some browsers apply color space conversion to match monitor settings.
>>> That's fine for images with color but WebGL apps often load heightmaps,
>>> normalmaps, lightmaps, global illumination maps and many other kinds of
>>> data through images. If the browser applies a colorspace conversion the
>>> data is not longer suitable for it's intended purpose therefore many
>>> WebGL
>>> apps turn off color conversions. As it is now, when an image is uploaded
>>> to
>>> WebGL, if colorspace conversion is
>>> off<
>>> http://www.khronos.org/registry/webgl/specs/latest/#PIXEL_STORAGE_PARAMETERS
>>> >,
>>
>>
OK, I see what you're trying to accomplish. You want to pass
non-premultiplied data and color converted (from sRGB to monitor) pixels to
WebGL
I think your API looks fine, except that the defaults should all be false...


>
>>> WebGL has to synchronously re-decode the image. It would be nice if
>>> ImageBitmap could handle this case so it can decode the image without
>>> applying any colorspace manipulations.
>>>
>>
>> Shouldn't the color space conversion happen when the final canvas bit are
>> blitted to the screen?
>> It seems like you should never do it during compositing since you could
>> get double conversions.
>>
>
> Maybe but that's not relevant to ImageBitmap is it? The point here is we
> want the ImageBitmap to give us the data in the format we need. It's
> designed to be async so it can do this but it we can't prevent it from
> applying colorspace conversions.
> Some browsers did that for regular img tags which pointed out the original
> problem. The browser can't guess how the image is going to be used and
> since it's a lot of work to decode an image you'd like to be able to tell
> it what you really need before it guesses wrong.
>
>
>>
>>
>>>
>>> If it was up to me I'd make createImageBitmap take on object with
>>> properties so that new options can be added later as in
>>>
>>> createImageBitmap(src, callback, {
>>>premultipliedAlpha: false,
>>>colorspaceConversion: false,
>>>x: 123,
>>> });
>>>
>>> But I'm not familiar if there is a common way to make APIs take a options
>>> like this except for the XHR way which is to create a request, set
>>> properties on the request, and finally execute the request.
>>
>>
>>  Like Tab said, it's fine to implement it that way.
>> Be aware that you might have to do some work in your idl compiler since I
>> *think* there are no other APIs (in Blink) that take a dictionary.
>>
>>
>

Re: [whatwg] Adding features needed for WebGL to ImageBitmap

[Rik Cabanier]

On Wed, Jun 19, 2013 at 3:24 PM, Gregg Tavares  wrote:

>
>
>
> On Wed, Jun 19, 2013 at 3:13 PM, Rik Cabanier  wrote:
>
>>
>> On Wed, Jun 19, 2013 at 2:47 PM, Gregg Tavares  wrote:
>>
>>> In order for ImageBitmap to be useful for WebGL we need more options
>>>
>>> Specifically
>>>
>>> premultipliedAlpha: true/false (default true)
>>> Nearly all GL games use non-premultipiled alpha textures. So all those
>>> games people want to port to WebGL will require non-premultipied
>>> textures.
>>> Often in games the alpha might not even be used for alpha but rather for
>>> glow maps or specular maps or the other kinds of data.
>>>
>>
>> When would premultipliedAlpha ever be true?
>> 2D Canvas always works with non-premultiplied alpha in its APIs.
>>
>
> AFAIK the canvas API expects all images to be premultiplied. Certainly in
> WebKit and Blink images used in the canvas and displayed in the img tag are
> loaded premultiplied which is why we had to add the option on WebGL since
> we needed those images before they had lost data.
>

You might have implemented it this way under the hood, but the Canvas spec
doesn't say anything about this.
It just says that data needs to be in non-premultiplied alpha when you read
or write it.


>
>
>>
>>
>>>
>>> flipY: true/false (default false)
>>> Nearly all 3D modeling apps expect the bottom left pixel to be the first
>>> pixel in a texture so many 3D engines flip the textures on load. WebGL
>>> provides this option but it takes time and memory to flip a large image
>>> therefore it would be nice if that flip happened before the callback from
>>> ImageBitmap
>>>
>>
>> Couldn't you just draw upside down?
>>
>
> No, games often animate texture coordinates and other things making it far
> more complicated. There are ways to work around this issue in code yes,
> they often require a ton of work.
>
> Most professional game engines pre-process the textures and flip them
> offline but that doesn't help when you're downloading models off say
> http://sketchup.google.com/3dwarehouse/
>
>
>>
>>
>>>
>>> colorspaceConversion: true/false (default true)
>>> Some browsers apply color space conversion to match monitor settings.
>>> That's fine for images with color but WebGL apps often load heightmaps,
>>> normalmaps, lightmaps, global illumination maps and many other kinds of
>>> data through images. If the browser applies a colorspace conversion the
>>> data is not longer suitable for it's intended purpose therefore many
>>> WebGL
>>> apps turn off color conversions. As it is now, when an image is uploaded
>>> to
>>> WebGL, if colorspace conversion is
>>> off<
>>> http://www.khronos.org/registry/webgl/specs/latest/#PIXEL_STORAGE_PARAMETERS
>>> >,
>>>
>>> WebGL has to synchronously re-decode the image. It would be nice if
>>> ImageBitmap could handle this case so it can decode the image without
>>> applying any colorspace manipulations.
>>>
>>
>> Shouldn't the color space conversion happen when the final canvas bit are
>> blitted to the screen?
>> It seems like you should never do it during compositing since you could
>> get double conversions.
>>
>
> Maybe but that's not relevant to ImageBitmap is it? The point here is we
> want the ImageBitmap to give us the data in the format we need. It's
> designed to be async so it can do this but it we can't prevent it from
> applying colorspace conversions.
> Some browsers did that for regular img tags which pointed out the original
> problem. The browser can't guess how the image is going to be used and
> since it's a lot of work to decode an image you'd like to be able to tell
> it what you really need before it guesses wrong.
>

I'm unsure if I follow. Some browsers will convert images to sRGB if they
have embedded profiles. However, if the images is sRGB already, what other
color conversions do you need? All compositing should happen in sRGB...


>
>
>>
>>
>>>
>>> If it was up to me I'd make createImageBitmap take on object with
>>> properties so that new options can be added later as in
>>>
>>> createImageBitmap(src, callback, {
>>>premultipliedAlpha: false,
>>>colorspaceConversion: false,
>>>x: 123,
>>> });
>>>
>>> But I'm not familiar if there is a common way to make APIs take a options
>>> like this except for the XHR way which is to create a request, set
>>> properties on the request, and finally execute the request.
>>
>>
>>  Like Tab said, it's fine to implement it that way.
>> Be aware that you might have to do some work in your idl compiler since I
>> *think* there are no other APIs (in Blink) that take a dictionary.
>>
>>
>

Re: [whatwg] Adding features needed for WebGL to ImageBitmap

[Gregg Tavares]

On Wed, Jun 19, 2013 at 3:13 PM, Rik Cabanier  wrote:

>
> On Wed, Jun 19, 2013 at 2:47 PM, Gregg Tavares  wrote:
>
>> In order for ImageBitmap to be useful for WebGL we need more options
>>
>> Specifically
>>
>> premultipliedAlpha: true/false (default true)
>> Nearly all GL games use non-premultipiled alpha textures. So all those
>> games people want to port to WebGL will require non-premultipied textures.
>> Often in games the alpha might not even be used for alpha but rather for
>> glow maps or specular maps or the other kinds of data.
>>
>
> When would premultipliedAlpha ever be true?
> 2D Canvas always works with non-premultiplied alpha in its APIs.
>

AFAIK the canvas API expects all images to be premultiplied. Certainly in
WebKit and Blink images used in the canvas and displayed in the img tag are
loaded premultiplied which is why we had to add the option on WebGL since
we needed those images before they had lost data.


>
>
>>
>> flipY: true/false (default false)
>> Nearly all 3D modeling apps expect the bottom left pixel to be the first
>> pixel in a texture so many 3D engines flip the textures on load. WebGL
>> provides this option but it takes time and memory to flip a large image
>> therefore it would be nice if that flip happened before the callback from
>> ImageBitmap
>>
>
> Couldn't you just draw upside down?
>

No, games often animate texture coordinates and other things making it far
more complicated. There are ways to work around this issue in code yes,
they often require a ton of work.

Most professional game engines pre-process the textures and flip them
offline but that doesn't help when you're downloading models off say
http://sketchup.google.com/3dwarehouse/


>
>
>>
>> colorspaceConversion: true/false (default true)
>> Some browsers apply color space conversion to match monitor settings.
>> That's fine for images with color but WebGL apps often load heightmaps,
>> normalmaps, lightmaps, global illumination maps and many other kinds of
>> data through images. If the browser applies a colorspace conversion the
>> data is not longer suitable for it's intended purpose therefore many WebGL
>> apps turn off color conversions. As it is now, when an image is uploaded
>> to
>> WebGL, if colorspace conversion is
>> off<
>> http://www.khronos.org/registry/webgl/specs/latest/#PIXEL_STORAGE_PARAMETERS
>> >,
>>
>> WebGL has to synchronously re-decode the image. It would be nice if
>> ImageBitmap could handle this case so it can decode the image without
>> applying any colorspace manipulations.
>>
>
> Shouldn't the color space conversion happen when the final canvas bit are
> blitted to the screen?
> It seems like you should never do it during compositing since you could
> get double conversions.
>

Maybe but that's not relevant to ImageBitmap is it? The point here is we
want the ImageBitmap to give us the data in the format we need. It's
designed to be async so it can do this but it we can't prevent it from
applying colorspace conversions.
Some browsers did that for regular img tags which pointed out the original
problem. The browser can't guess how the image is going to be used and
since it's a lot of work to decode an image you'd like to be able to tell
it what you really need before it guesses wrong.


>
>
>>
>> If it was up to me I'd make createImageBitmap take on object with
>> properties so that new options can be added later as in
>>
>> createImageBitmap(src, callback, {
>>premultipliedAlpha: false,
>>colorspaceConversion: false,
>>x: 123,
>> });
>>
>> But I'm not familiar if there is a common way to make APIs take a options
>> like this except for the XHR way which is to create a request, set
>> properties on the request, and finally execute the request.
>
>
>  Like Tab said, it's fine to implement it that way.
> Be aware that you might have to do some work in your idl compiler since I
> *think* there are no other APIs (in Blink) that take a dictionary.
>
>

Re: [whatwg] Adding features needed for WebGL to ImageBitmap

[Rik Cabanier]

On Wed, Jun 19, 2013 at 2:47 PM, Gregg Tavares  wrote:

> In order for ImageBitmap to be useful for WebGL we need more options
>
> Specifically
>
> premultipliedAlpha: true/false (default true)
> Nearly all GL games use non-premultipiled alpha textures. So all those
> games people want to port to WebGL will require non-premultipied textures.
> Often in games the alpha might not even be used for alpha but rather for
> glow maps or specular maps or the other kinds of data.
>

When would premultipliedAlpha ever be true?
2D Canvas always works with non-premultiplied alpha in its APIs.


>
> flipY: true/false (default false)
> Nearly all 3D modeling apps expect the bottom left pixel to be the first
> pixel in a texture so many 3D engines flip the textures on load. WebGL
> provides this option but it takes time and memory to flip a large image
> therefore it would be nice if that flip happened before the callback from
> ImageBitmap
>

Couldn't you just draw upside down?


>
> colorspaceConversion: true/false (default true)
> Some browsers apply color space conversion to match monitor settings.
> That's fine for images with color but WebGL apps often load heightmaps,
> normalmaps, lightmaps, global illumination maps and many other kinds of
> data through images. If the browser applies a colorspace conversion the
> data is not longer suitable for it's intended purpose therefore many WebGL
> apps turn off color conversions. As it is now, when an image is uploaded to
> WebGL, if colorspace conversion is
> off<
> http://www.khronos.org/registry/webgl/specs/latest/#PIXEL_STORAGE_PARAMETERS
> >,
> WebGL has to synchronously re-decode the image. It would be nice if
> ImageBitmap could handle this case so it can decode the image without
> applying any colorspace manipulations.
>

Shouldn't the color space conversion happen when the final canvas bit are
blitted to the screen?
It seems like you should never do it during compositing since you could get
double conversions.


>
> If it was up to me I'd make createImageBitmap take on object with
> properties so that new options can be added later as in
>
> createImageBitmap(src, callback, {
>premultipliedAlpha: false,
>colorspaceConversion: false,
>x: 123,
> });
>
> But I'm not familiar if there is a common way to make APIs take a options
> like this except for the XHR way which is to create a request, set
> properties on the request, and finally execute the request.


 Like Tab said, it's fine to implement it that way.
Be aware that you might have to do some work in your idl compiler since I
*think* there are no other APIs (in Blink) that take a dictionary.

Re: [whatwg] Adding features needed for WebGL to ImageBitmap

[Tab Atkins Jr.]

On Wed, Jun 19, 2013 at 2:47 PM, Gregg Tavares  wrote:
> If it was up to me I'd make createImageBitmap take on object with
> properties so that new options can be added later as in
>
> createImageBitmap(src, callback, {
>premultipliedAlpha: false,
>colorspaceConversion: false,
>x: 123,
> });
>
> But I'm not familiar if there is a common way to make APIs take a options
> like this except for the XHR way which is to create a request, set
> properties on the request, and finally execute the request.

Passing an options object is the standard way.  Don't look to XHR for
*anything* about precedents.

~TJ