Re: [whatwg] Canvas in workers

[James Robinson]

On Thu, Oct 24, 2013 at 6:59 AM, Glenn Maynard  wrote:

> >
> > - Original Message -
> > From: "Robert O'Callahan" 
> >
> > We talked through this proposal with a lot of Mozilla people in a meeting
> > and collectively decided that we don't care about the case of workers
> that
> > commit multiple frames to a canvas without yielding --- at least for now.
> > So we want to remove commit() and copy the main-thread semantics that a
> > canvas frame is eligible for presentation whenever script is not running
> in
> > the worker.
> >
>
> On Thu, Oct 24, 2013 at 7:25 AM, Jeff Gilbert 
> wrote:
>
> > This is not the current WebGL semantics:
> > WebGL presents its drawing buffer to the HTML page compositor
> > immediately before a compositing operation[...]
> >
>
> (Can you please quote correctly?  Having one person top-quoting makes a
> mess of the whole thread, and it looked like you were saying that the WebGL
> spec language you were quoting was incorrect.)
>
> The assumption WebGL is making here is that compositing is a synchronous
> task in the event loop, which happens while no script is running.  That is,
> the semantics Robert describes are the same as what the WebGL spec is
> trying to say.  That's not necessarily how compositing actually works,
> though, and that language also won't make sense with threaded rendering.
> It might be better for WebGL to define this using the "global script
> clean-up jobs" task that HTML now defines.
>
> http://www.whatwg.org/specs/web-apps/current-work/#run-the-global-script-clean-up-jobs
> I'd recommend spinning off a separate thread if we want to go into this
> further.
>

The time that compositing occurs is already specified by the HTML event
loop processing model (7.1.4.2):

http://www.whatwg.org/specs/web-apps/current-work/multipage/webappapis.html#processing-model-4

An event 
loop
must
continually run through the following steps for as long as it exists:

   1.

   Run the oldest
task
on
   one of the event
loop
   's task 
queues,
   if any, ignoring tasks whose associated
Documents
   are not fully
active.
   The user agent may pick any task
queue
   .
   2.

   If the storage
mutex
is
   now owned by the event
loop,
   release it so that it is once again free.
   3.

   If a task was run in the first step above, remove that task from its task
   
queue
   .
   4.

   If this event
loop
is
   not a worker's event
loop,
   run these substeps:
   1.

  Perform a microtask
checkpoint
  .
  2.

  Provide a stable
state
  .
  3.

  If necessary, update the rendering or user interface of any
Document
   or browsing
context
to
  reflect the current state.
  5.

   Otherwise, if this event
loop
is
   running for a
WorkerGlobalScope,
   but there are no events in the event
loop
   's task 
queues
and
   the 
WorkerGlobalScope
object's 
closing
flag
   is true, then destroy the event
loop

Re: [whatwg] Canvas in workers

[Glenn Maynard]

>
> - Original Message -
> From: "Robert O'Callahan" 
>
> We talked through this proposal with a lot of Mozilla people in a meeting
> and collectively decided that we don't care about the case of workers that
> commit multiple frames to a canvas without yielding --- at least for now.
> So we want to remove commit() and copy the main-thread semantics that a
> canvas frame is eligible for presentation whenever script is not running in
> the worker.
>

On Thu, Oct 24, 2013 at 7:25 AM, Jeff Gilbert  wrote:

> This is not the current WebGL semantics:
> WebGL presents its drawing buffer to the HTML page compositor
> immediately before a compositing operation[...]
>

(Can you please quote correctly?  Having one person top-quoting makes a
mess of the whole thread, and it looked like you were saying that the WebGL
spec language you were quoting was incorrect.)

The assumption WebGL is making here is that compositing is a synchronous
task in the event loop, which happens while no script is running.  That is,
the semantics Robert describes are the same as what the WebGL spec is
trying to say.  That's not necessarily how compositing actually works,
though, and that language also won't make sense with threaded rendering.
It might be better for WebGL to define this using the "global script
clean-up jobs" task that HTML now defines.
http://www.whatwg.org/specs/web-apps/current-work/#run-the-global-script-clean-up-jobs
I'd recommend spinning off a separate thread if we want to go into this
further.

-- 
Glenn Maynard

Re: [whatwg] Canvas in workers

[Jeff Gilbert]

This is not the current WebGL semantics:
WebGL presents its drawing buffer to the HTML page compositor immediately 
before a compositing operation[...]

I can't find where this is specified for canvas, though I assume it's all 
implied.

-Jeff

- Original Message -
From: "Robert O'Callahan" 
To: "Justin Novosad" 
Cc: "WHAT Working Group" , "Glenn Maynard" , 
"Kenneth Russell" , "Kyle Huey" 
Sent: Thursday, October 24, 2013 2:24:27 AM
Subject: Re: [whatwg] Canvas in workers

We talked through this proposal with a lot of Mozilla people in a meeting
and collectively decided that we don't care about the case of workers that
commit multiple frames to a canvas without yielding --- at least for now.
So we want to remove commit() and copy the main-thread semantics that a
canvas frame is eligible for presentation whenever script is not running in
the worker.

Rob
-- 
Jtehsauts  tshaei dS,o n" Wohfy  Mdaon  yhoaus  eanuttehrotraiitny  eovni
le atrhtohu gthot sf oirng iyvoeu rs ihnesa.r"t sS?o  Whhei csha iids  teoa
stiheer :p atroa lsyazye,d  'mYaonu,r  "sGients  uapr,e  tfaokreg iyvoeunr,
'm aotr  atnod  sgaoy ,h o'mGee.t"  uTph eann dt hwea lmka'n?  gBoutt  uIp
waanndt  wyeonut  thoo mken.o w  *
*

Re: [whatwg] Canvas in workers

[Robert O'Callahan]

We talked through this proposal with a lot of Mozilla people in a meeting
and collectively decided that we don't care about the case of workers that
commit multiple frames to a canvas without yielding --- at least for now.
So we want to remove commit() and copy the main-thread semantics that a
canvas frame is eligible for presentation whenever script is not running in
the worker.

Rob
-- 
Jtehsauts  tshaei dS,o n" Wohfy  Mdaon  yhoaus  eanuttehrotraiitny  eovni
le atrhtohu gthot sf oirng iyvoeu rs ihnesa.r"t sS?o  Whhei csha iids  teoa
stiheer :p atroa lsyazye,d  'mYaonu,r  "sGients  uapr,e  tfaokreg iyvoeunr,
'm aotr  atnod  sgaoy ,h o'mGee.t"  uTph eann dt hwea lmka'n?  gBoutt  uIp
waanndt  wyeonut  thoo mken.o w  *
*

Re: [whatwg] Canvas in workers

[Justin Novosad]

On Wed, Oct 23, 2013 at 11:46 AM, Justin Novosad  wrote:

>
>
> Interface Transferable is not defined.
>
> Never mind. :-)

Re: [whatwg] Canvas in workers

[Justin Novosad]

On Tue, Oct 22, 2013 at 5:59 PM, Robert O'Callahan wrote:

> On Tue, Oct 22, 2013 at 10:44 PM, Robert O'Callahan  >wrote:
>
> > No problem at all. Can you do it? I need to get a WHATWG account :-).
> >
>
> OK, I added the proposal here:
> http://wiki.whatwg.org/wiki/WorkerCanvas
> A couple of changes from the previous version:
> -- Added ImageBitmap.transferToImage.
> -- Removed HTMLImageElement.srcObject (though it may come back in other
> contexts)
> -- Made ImageBitmap Transferable. This makes it possible to ensure the
> worker that generated an ImageBitmap does not keep alive a reference to the
> buffer.
> -- Removed ImageBitmaps.close() since the use-cases driving this proposal
> probably don't need it anymore.


Interface Transferable is not defined.


>

Re: [whatwg] Canvas in workers

[Robert O'Callahan]

On Tue, Oct 22, 2013 at 10:44 PM, Robert O'Callahan wrote:

> No problem at all. Can you do it? I need to get a WHATWG account :-).
>

OK, I added the proposal here:
http://wiki.whatwg.org/wiki/WorkerCanvas
A couple of changes from the previous version:
-- Added ImageBitmap.transferToImage.
-- Removed HTMLImageElement.srcObject (though it may come back in other
contexts)
-- Made ImageBitmap Transferable. This makes it possible to ensure the
worker that generated an ImageBitmap does not keep alive a reference to the
buffer.
-- Removed ImageBitmaps.close() since the use-cases driving this proposal
probably don't need it anymore.

Rob
-- 
Jtehsauts  tshaei dS,o n" Wohfy  Mdaon  yhoaus  eanuttehrotraiitny  eovni
le atrhtohu gthot sf oirng iyvoeu rs ihnesa.r"t sS?o  Whhei csha iids  teoa
stiheer :p atroa lsyazye,d  'mYaonu,r  "sGients  uapr,e  tfaokreg iyvoeunr,
'm aotr  atnod  sgaoy ,h o'mGee.t"  uTph eann dt hwea lmka'n?  gBoutt  uIp
waanndt  wyeonut  thoo mken.o w  *
*

Re: [whatwg] Canvas in workers

[Kenneth Russell]

Great.


On Tue, Oct 22, 2013 at 2:54 PM, Robert O'Callahan  wrote:
> I got an account and I'm uploading the proposal now.
>
>
> Rob
> --
> Jtehsauts  tshaei dS,o n" Wohfy  Mdaon  yhoaus  eanuttehrotraiitny  eovni le
> atrhtohu gthot sf oirng iyvoeu rs ihnesa.r"t sS?o  Whhei csha iids  teoa
> stiheer :p atroa lsyazye,d  'mYaonu,r  "sGients  uapr,e  tfaokreg iyvoeunr,
> 'm aotr  atnod  sgaoy ,h o'mGee.t"  uTph eann dt hwea lmka'n?  gBoutt  uIp
> waanndt  wyeonut  thoo mken.o w

Re: [whatwg] Canvas in workers

[Robert O'Callahan]

I got an account and I'm uploading the proposal now.

Rob
-- 
Jtehsauts  tshaei dS,o n" Wohfy  Mdaon  yhoaus  eanuttehrotraiitny  eovni
le atrhtohu gthot sf oirng iyvoeu rs ihnesa.r"t sS?o  Whhei csha iids  teoa
stiheer :p atroa lsyazye,d  'mYaonu,r  "sGients  uapr,e  tfaokreg iyvoeunr,
'm aotr  atnod  sgaoy ,h o'mGee.t"  uTph eann dt hwea lmka'n?  gBoutt  uIp
waanndt  wyeonut  thoo mken.o w  *
*

Re: [whatwg] Canvas in workers

[Kenneth Russell]

On Tue, Oct 22, 2013 at 1:44 PM, Robert O'Callahan  wrote:
> On Tue, Oct 22, 2013 at 7:31 PM, Kenneth Russell  wrote:
>>
>> Robert, please don't remove those APIs from your proposal. They're
>> needed in order to address known use cases, and splitting them off
>> will make it difficult to understand how they interact with
>> WorkerCanvas later.
>
>
> Yes, I think it's a good idea to specify a complete set of APIs that fit
> together logically and if there are some we don't need, we can remove them
> or just delay implementing them until they're needed.
>
>> I would like to suggest changing the 'srcObject' property on
>> HTMLImageElement into some sort of method taking ImageBitmap as
>> argument. If an ImageBitmap had been previously set against the
>> HTMLImageElement, the method would automatically call 'close()'
>> against it. Fundamentally there should be some easy way to repeatedly
>> update the HTMLImageElement without having to query its previous
>> ImageBitmap and call close() against it before setting srcObject.
>
>
> Hmm. I'm not sure how this should work.
>
> Maybe instead we should use canvas elements and define
> ImageBitmap.transferToCanvas(HTMLCanvasElement). That would mitigate Glenn's
> argument about having to change element types too.

Using a Canvas as the target for displaying an ImageBitmap is fraught
with problems. Because Canvases are rendering targets themselves,
transferring in an ImageBitmap has to interoperate somehow with the
canvas's rendering context as well as other APIs like toDataURL. The
DrawingBuffer proposal in http://wiki.whatwg.org/wiki/CanvasInWorkers
was admittedly complex, but did properly handle fully detaching a
canvas's framebuffer and attaching it to another one. ImageBitmap,
being semantically read-only, isn't well suited for this task.

The idea to use an HTMLImageElement to display the ImageBitmaps is
elegant and I would like to see it explored, including prototyping it
behind an experimental flag and seeing how well it works.

>> Would you consider copying
>> https://wiki.mozilla.org/User:Roc/WorkerCanvasProposal to
>> http://wiki.whatwg.org/wiki/Category:Proposals so that it's easier to
>> collaborate on it?
>
>
> No problem at all. Can you do it? I need to get a WHATWG account :-).

Yes, I'll take care of the initial draft. The whatwg was pretty
responsive when I got my account though.

-Ken


> Rob
> --
> Jtehsauts  tshaei dS,o n" Wohfy  Mdaon  yhoaus  eanuttehrotraiitny  eovni le
> atrhtohu gthot sf oirng iyvoeu rs ihnesa.r"t sS?o  Whhei csha iids  teoa
> stiheer :p atroa lsyazye,d  'mYaonu,r  "sGients  uapr,e  tfaokreg iyvoeunr,
> 'm aotr  atnod  sgaoy ,h o'mGee.t"  uTph eann dt hwea lmka'n?  gBoutt  uIp
> waanndt  wyeonut  thoo mken.o w

Re: [whatwg] Canvas in workers

[Robert O'Callahan]

On Tue, Oct 22, 2013 at 7:31 PM, Kenneth Russell  wrote:

> Robert, please don't remove those APIs from your proposal. They're
> needed in order to address known use cases, and splitting them off
> will make it difficult to understand how they interact with
> WorkerCanvas later.
>

Yes, I think it's a good idea to specify a complete set of APIs that fit
together logically and if there are some we don't need, we can remove them
or just delay implementing them until they're needed.

I would like to suggest changing the 'srcObject' property on
> HTMLImageElement into some sort of method taking ImageBitmap as
> argument. If an ImageBitmap had been previously set against the
> HTMLImageElement, the method would automatically call 'close()'
> against it. Fundamentally there should be some easy way to repeatedly
> update the HTMLImageElement without having to query its previous
> ImageBitmap and call close() against it before setting srcObject.
>

Hmm. I'm not sure how this should work.

Maybe instead we should use canvas elements and define
ImageBitmap.transferToCanvas(HTMLCanvasElement). That would mitigate
Glenn's argument about having to change element types too.

Would you consider copying
> https://wiki.mozilla.org/User:Roc/WorkerCanvasProposal to
> http://wiki.whatwg.org/wiki/Category:Proposals so that it's easier to
> collaborate on it?
>

No problem at all. Can you do it? I need to get a WHATWG account :-).

Rob
-- 
Jtehsauts  tshaei dS,o n" Wohfy  Mdaon  yhoaus  eanuttehrotraiitny  eovni
le atrhtohu gthot sf oirng iyvoeu rs ihnesa.r"t sS?o  Whhei csha iids  teoa
stiheer :p atroa lsyazye,d  'mYaonu,r  "sGients  uapr,e  tfaokreg iyvoeunr,
'm aotr  atnod  sgaoy ,h o'mGee.t"  uTph eann dt hwea lmka'n?  gBoutt  uIp
waanndt  wyeonut  thoo mken.o w  *
*

Re: [whatwg] Canvas in workers

[Kenneth Russell]

On Mon, Oct 21, 2013 at 8:03 AM, Glenn Maynard  wrote:
> On Sun, Oct 20, 2013 at 11:53 PM, Robert O'Callahan 
> wrote:
>>
>> Glenn, taking a step back for a bit, is there anything in
>> https://wiki.mozilla.org/User:Roc/WorkerCanvasProposal that you would
>> actually object to? IOW, is there anything there that you would think is
>> completely superfluous to the platform if all your proposals were to be
>> adopted as well?
>
>
> I have no objection to the overall change from CanvasProxy to WorkerCanvas,
> eg. the stuff in Kyle's original mail to the thread.  (Being able to settle
> on that is one of the reasons I've tried to detach discussion for the other
> use cases.)
>
> I'd only recommend leaving out transferToImageBitmap, srcObject and
> ImageBitmap.close() parts.  I do think that would be redundant with with
> "present" proposal.  They can always be added later, and leaving them out
> keeps the WorkerCanvas proposal itself focused.

Robert, please don't remove those APIs from your proposal. They're
needed in order to address known use cases, and splitting them off
will make it difficult to understand how they interact with
WorkerCanvas later.

I would like to suggest changing the 'srcObject' property on
HTMLImageElement into some sort of method taking ImageBitmap as
argument. If an ImageBitmap had been previously set against the
HTMLImageElement, the method would automatically call 'close()'
against it. Fundamentally there should be some easy way to repeatedly
update the HTMLImageElement without having to query its previous
ImageBitmap and call close() against it before setting srcObject.

Would you consider copying
https://wiki.mozilla.org/User:Roc/WorkerCanvasProposal to
http://wiki.whatwg.org/wiki/Category:Proposals so that it's easier to
collaborate on it?

Re: [whatwg] Canvas in workers

[Glenn Maynard]

On Sun, Oct 20, 2013 at 11:47 PM, Robert O'Callahan
wrote:

> If you think that a single context outputting to multiple canvases
>> fundamentally won't work well with canvases of different sizes, then
>> forget
>> about the feature.
>>
>> When you attachToCanvas, you're attaching that canvas's rendering buffer,
>> not just its color buffer.  In WebGL terms, each canvas is a Framebuffer,
>> with all its associated Renderbuffers.  Attaching the context to a canvas
>> is like using bindFramebuffer to render to its backing store.
>>
>
> This is actually a disadvantage for attachToCanvas in some situations. If
> I just want to generate N different rendered images (of the same size)
> (e.g. http://www.turbosquid.com/Search/3D-Models/Vehicle/Car), the only
> thing I want N of is the final color buffer.
>

Note that it should be possible to reuse buffers as an optimization.  For
example, if you render to a canvas and commit(), then attach to a different
canvas of the same size, you could reuse the backbuffer you're already on
instead of creating a new one.  (For WebGL this requires a
preserveDrawingBuffer of false, which I think I specified as required in
the attachToCanvas proposal: attaching to a canvas always clears it.)  The
only thing you actually need to keep around in that case is the current
front color buffer that's being displayed.

That said, everyone prefers approaches that don't hinge on "this works fine
as long as you do this tricky optimization". If you have other use cases
for it, then having an API to retrieve an ImageBitmap is fine.  It just
doesn't seem like the right solution for rendering to multiple things in
the document.

-- 
Glenn Maynard

Re: [whatwg] Canvas in workers

[Glenn Maynard]

On Sun, Oct 20, 2013 at 11:53 PM, Robert O'Callahan wrote:

> Glenn, taking a step back for a bit, is there anything in
> https://wiki.mozilla.org/User:Roc/WorkerCanvasProposal that you would
> actually object to? IOW, is there anything there that you would think is
> completely superfluous to the platform if all your proposals were to be
> adopted as well?
>

I have no objection to the overall change from CanvasProxy to WorkerCanvas,
eg. the stuff in Kyle's original mail to the thread.  (Being able to settle
on that is one of the reasons I've tried to detach discussion for the other
use cases.)

I'd only recommend leaving out transferToImageBitmap, srcObject and
ImageBitmap.close() parts.  I do think that would be redundant with with
"present" proposal.  They can always be added later, and leaving them out
keeps the WorkerCanvas proposal itself focused.

-- 
Glenn Maynard

Re: [whatwg] Canvas in workers

[Robert O'Callahan]

Glenn, taking a step back for a bit, is there anything in
https://wiki.mozilla.org/User:Roc/WorkerCanvasProposal that you would
actually object to? IOW, is there anything there that you would think is
completely superfluous to the platform if all your proposals were to be
adopted as well?

Rob
-- 
Jtehsauts  tshaei dS,o n" Wohfy  Mdaon  yhoaus  eanuttehrotraiitny  eovni
le atrhtohu gthot sf oirng iyvoeu rs ihnesa.r"t sS?o  Whhei csha iids  teoa
stiheer :p atroa lsyazye,d  'mYaonu,r  "sGients  uapr,e  tfaokreg iyvoeunr,
'm aotr  atnod  sgaoy ,h o'mGee.t"  uTph eann dt hwea lmka'n?  gBoutt  uIp
waanndt  wyeonut  thoo mken.o w  *
*

Re: [whatwg] Canvas in workers

[Robert O'Callahan]

On Sun, Oct 20, 2013 at 5:29 PM, Glenn Maynard  wrote:

> That's not the problem attachToCanvas tries to solve.  It tries to solve
> rendering to multiple canvases, without requiring synchronization with the
> UI thread.  I have a proposal for handling synchronization with DOM
> updates, but I'll post it in a separate thread.
>

OK.


>   - If you're rendering in a worker and the eventual target is in the main
> >> thread, the worker needs to be careful to not start rendering again
> until
> >> the main thread has assigned the ImageBitmap to where it wants it, and
> >> called .close().  You'd need to send a message back to the worker going
> >> "okay, you can continue now".  Otherwise, you'd start rendering before a
> >> buffer has been freed up for reuse, and end up creating more backbuffers
> >> than you intended (which matters for large screens).  This seems easy to
> >> get wrong, and attachToCanvas doesn't have this problem.
> >>
> >
> > Not if you use transferToImageBitmap.
>
> transferToImageBitmap does have this problem.  If you transferToImageBitmap
> to detach your backing store to display it somewhere, then start rendering
> the next frame without waiting for the ImageBitmap to be given to the
> target, then as soon as you start rendering you'll end up creating a 3rd
> rendering buffer.
>

OK.

If you think that a single context outputting to multiple canvases
> fundamentally won't work well with canvases of different sizes, then forget
> about the feature.
>
> When you attachToCanvas, you're attaching that canvas's rendering buffer,
> not just its color buffer.  In WebGL terms, each canvas is a Framebuffer,
> with all its associated Renderbuffers.  Attaching the context to a canvas
> is like using bindFramebuffer to render to its backing store.
>

This is actually a disadvantage for attachToCanvas in some situations. If I
just want to generate N different rendered images (of the same size) (e.g.
http://www.turbosquid.com/Search/3D-Models/Vehicle/Car), the only thing I
want N of is the final color buffer.

Rob
-- 
Jtehsauts  tshaei dS,o n" Wohfy  Mdaon  yhoaus  eanuttehrotraiitny  eovni
le atrhtohu gthot sf oirng iyvoeu rs ihnesa.r"t sS?o  Whhei csha iids  teoa
stiheer :p atroa lsyazye,d  'mYaonu,r  "sGients  uapr,e  tfaokreg iyvoeunr,
'm aotr  atnod  sgaoy ,h o'mGee.t"  uTph eann dt hwea lmka'n?  gBoutt  uIp
waanndt  wyeonut  thoo mken.o w  *
*

Re: [whatwg] Canvas in workers

[Glenn Maynard]

On Sat, Oct 19, 2013 at 10:11 AM, Robert O'Callahan wrote:

> It's not clear to me how attachToCanvas works. An application like Google
> Maps wants to draw to multiple canvases from a worker and then render the
> updated canvas contents all at once, in synchrony with changes to the DOM
> made by the main thread. How would you do that with attachToCanvas?
>

That's not the problem attachToCanvas tries to solve.  It tries to solve
rendering to multiple canvases, without requiring synchronization with the
UI thread.  I have a proposal for handling synchronization with DOM
updates, but I'll post it in a separate thread.

(To clarify, this thread is talking about three different things: rendering
from a worker to the UI thread, rendering to multiple canvases, and
synchronizing rendering in a worker to DOM updates in the main thread.  The
only reason they're in the same thread is because some proposals are trying
to handle two or all three of them together.  Trying to do that is leading
to unwanted limitations, such as forcing synchronization when you don't
want it, and it's making the conversation hard to follow.  Since my
proposal is orthogonal to the rest--it's separate and compatible with both
WorkerCanvas and attachToCanvas--and this thread is discussing too many
things at once, I'll move the third to a separate thread.)


 - If you're rendering in a worker and the eventual target is in the main
>> thread, the worker needs to be careful to not start rendering again until
>> the main thread has assigned the ImageBitmap to where it wants it, and
>> called .close().  You'd need to send a message back to the worker going
>> "okay, you can continue now".  Otherwise, you'd start rendering before a
>> buffer has been freed up for reuse, and end up creating more backbuffers
>> than you intended (which matters for large screens).  This seems easy to
>> get wrong, and attachToCanvas doesn't have this problem.
>>
>
> Not if you use transferToImageBitmap.
>

transferToImageBitmap does have this problem.  If you transferToImageBitmap
to detach your backing store to display it somewhere, then start rendering
the next frame without waiting for the ImageBitmap to be given to the
target, then as soon as you start rendering you'll end up creating a 3rd
rendering buffer.

(The present() proposal also has this problem, but users are only affected
by it if they're actually synchronizing to DOM updates.)


> With attachToCanvas, you just size both canvases normally
>> once, and switch between them with a single function call.
>>
>
> I'm not sure how helpful this is. In the case of WebGL, the rendering
> context has resources that need to be sized the same as the destination
> color buffer, so they'll need to be resized anyway if you're actually using
> a single context to render to canvases of different sizes. My guess is that
> the advice will always be "don't do that".
>

If you think that a single context outputting to multiple canvases
fundamentally won't work well with canvases of different sizes, then forget
about the feature.

When you attachToCanvas, you're attaching that canvas's rendering buffer,
not just its color buffer.  In WebGL terms, each canvas is a Framebuffer,
with all its associated Renderbuffers.  Attaching the context to a canvas
is like using bindFramebuffer to render to its backing store.


> I believe these are minor changes, especially compared to moving drawing
> to a worker.
>

Moving drawing to a worker is unrelated to drawing to multiple canvases.  I
don't agree that it's a minor change: today you can create a black-box API
that renders to a context, without caring about the DOM at all.  If you
have to move the image to an  to display it, suddenly you have to care
about the DOM to render.  It breaks the basic model.

-- 
Glenn Maynard

Re: [whatwg] Canvas in workers

[Robert O'Callahan]

On Sat, Oct 19, 2013 at 1:28 AM, Glenn Maynard  wrote:

> I'd like to hear thoughts on the "context.attachToCanvas" approach.  I
> think it has important advantages over ImageBitmap:
>
> - ImageBitmap requires the user to call close().  If the user forgets, or
> doesn't know, or misses it in some code paths, the problems caused aren't
> obvious.  Worse, they may only appear in some implementations and not
> others, depending on GC strategies.  attachToCanvas doesn't need cleanup in
> the first place, which is a nicer solution--there's nothing for the user to
> get wrong.
>

It's not clear to me how attachToCanvas works. An application like Google
Maps wants to draw to multiple canvases from a worker and then render the
updated canvas contents all at once, in synchrony with changes to the DOM
made by the main thread. How would you do that with attachToCanvas?


> - If you're rendering in a worker and the eventual target is in the main
> thread, the worker needs to be careful to not start rendering again until
> the main thread has assigned the ImageBitmap to where it wants it, and
> called .close().  You'd need to send a message back to the worker going
> "okay, you can continue now".  Otherwise, you'd start rendering before a
> buffer has been freed up for reuse, and end up creating more backbuffers
> than you intended (which matters for large screens).  This seems easy to
> get wrong, and attachToCanvas doesn't have this problem.
>

Not if you use transferToImageBitmap.


> - With ImageBitmap, you need to create a helper canvas, then each time you
> render to a new target, you need to resize the canvas to match where it'll
> eventually go, so the resulting ImageBitmap is the size of its
> destination.  (This may also need to be carefully optimized, so the
> implementation doesn't actually resize the backing store every time its
> size changes.)  With attachToCanvas, you just size both canvases normally
> once, and switch between them with a single function call.
>

I'm not sure how helpful this is. In the case of WebGL, the rendering
context has resources that need to be sized the same as the destination
color buffer, so they'll need to be resized anyway if you're actually using
a single context to render to canvases of different sizes. My guess is that
the advice will always be "don't do that".

- attachToCanvas matches the way Canvas works today: you create a Canvas,
> put it in the document (if it's for display), and render to it.  For two
> canvases, you'd just add a second Canvas, and toggle as needed.  With
> ImageBitmap, you have to restructure everything as soon as you want a
> second canvas, since you'd want to have a single offscreen Canvas for
> rendering, and to have  elements in the document instead of canvases.
>

I believe these are minor changes, especially compared to moving drawing to
a worker.

Rob
-- 
Jtehsauts  tshaei dS,o n" Wohfy  Mdaon  yhoaus  eanuttehrotraiitny  eovni
le atrhtohu gthot sf oirng iyvoeu rs ihnesa.r"t sS?o  Whhei csha iids  teoa
stiheer :p atroa lsyazye,d  'mYaonu,r  "sGients  uapr,e  tfaokreg iyvoeunr,
'm aotr  atnod  sgaoy ,h o'mGee.t"  uTph eann dt hwea lmka'n?  gBoutt  uIp
waanndt  wyeonut  thoo mken.o w  *
*

Re: [whatwg] Canvas in workers

[Glenn Maynard]

On Fri, Oct 18, 2013 at 2:06 PM, Kenneth Russell  wrote:

> Capturing Glenn Maynard's feedback from the other thread started by
> Rik Cabanier, Glenn made a good point that there needs to be a way to
> explicitly deallocate the ImageBitmap. Otherwise, the JavaScript
> objects will have to be garbage collected before the GPU resource
> (texture) it references can be freed, and that will not work -- GPU
> resources will quickly pile up.
>

I'd like to hear thoughts on the "context.attachToCanvas" approach.  I
think it has important advantages over ImageBitmap:

- ImageBitmap requires the user to call close().  If the user forgets, or
doesn't know, or misses it in some code paths, the problems caused aren't
obvious.  Worse, they may only appear in some implementations and not
others, depending on GC strategies.  attachToCanvas doesn't need cleanup in
the first place, which is a nicer solution--there's nothing for the user to
get wrong.
- If you're rendering in a worker and the eventual target is in the main
thread, the worker needs to be careful to not start rendering again until
the main thread has assigned the ImageBitmap to where it wants it, and
called .close().  You'd need to send a message back to the worker going
"okay, you can continue now".  Otherwise, you'd start rendering before a
buffer has been freed up for reuse, and end up creating more backbuffers
than you intended (which matters for large screens).  This seems easy to
get wrong, and attachToCanvas doesn't have this problem.
- With ImageBitmap, you need to create a helper canvas, then each time you
render to a new target, you need to resize the canvas to match where it'll
eventually go, so the resulting ImageBitmap is the size of its
destination.  (This may also need to be carefully optimized, so the
implementation doesn't actually resize the backing store every time its
size changes.)  With attachToCanvas, you just size both canvases normally
once, and switch between them with a single function call.
- attachToCanvas matches the way Canvas works today: you create a Canvas,
put it in the document (if it's for display), and render to it.  For two
canvases, you'd just add a second Canvas, and toggle as needed.  With
ImageBitmap, you have to restructure everything as soon as you want a
second canvas, since you'd want to have a single offscreen Canvas for
rendering, and to have  elements in the document instead of canvases.

Here's the example from the other thread to consolidate the discussion.  If
you're in a worker, canvas and canvas2 can both be WorkerCanvases posted
from the main thread or created directly:

var canvas = document.querySelector(".canvas1");
var gl = canvas.getContext("webgl");
loadExpensiveResources(gl);
drawStuff(gl);
var canvas2 = document.querySelector(".canvas2");
gl.attachToCanvas(canvas2);
drawStuff(gl); // don't need to loadExpensiveResources again

-- 
Glenn Maynard

Re: [whatwg] Canvas in workers

[Kenneth Russell]

On Tue, Oct 15, 2013 at 5:30 PM, Kenneth Russell  wrote:
> On Tue, Oct 15, 2013 at 4:41 PM, Robert O'Callahan  
> wrote:
>> On Wed, Oct 16, 2013 at 11:55 AM, Kenneth Russell  wrote:
>>>
>>> On Mon, Oct 14, 2013 at 1:34 PM, Robert O'Callahan 
>>> wrote:
>>> > On Mon, Oct 14, 2013 at 2:20 PM, Kenneth Russell  wrote:
>>> >>
>>> >> Would you mind looking at the proposal
>>> >> http://wiki.whatwg.org/wiki/CanvasInWorkers and commenting on it?
>>> >
>>> >
>>> > Sure. Kyle and I looked at it while we were working on our proposal. The
>>> > main issues I have with it are that rearchitecting  to introduce
>>> > the
>>> > DrawingBuffer layer of abstraction seems unnecessarily complex, and it
>>> > doesn't handle direct presentation of frames from the worker, bypassing
>>> > the
>>> > main thread.
>>>
>>> Note that the CanvasInWorkers draft solves some other longstanding
>>> issues not addressed by the WorkerCanvas proposal. It provides the
>>> ability to render to multiple canvases from a single context, whether
>>> workers are involved or not.
>>
>>
>> That may be a useful feature, but I'd like to see it justified in its own
>> right.
>
> There has been a lot of developer feedback on the WebGL mailing lists
> over the past couple of years about exactly this feature. Web sites
> like Turbosquid want to present lots of little thumbnails of models --
> see for example http://www.turbosquid.com/Search/3D-Models/Vehicle/Car
> -- and have them be interactive. It's too resource-intensive to create
> a separate WebGL context for each. The most direct solution is to
> allow one context to render to multiple canvases.
>
>
>>> It achieves ideal memory utilization by
>>> being very explicit in the API, without the need for extensive and
>>> subtle optimizations behind the scenes.
>>
>>
>> We can be more explicit with ImageBitmaps. We could provide
>> WorkerCanvas.transferToImageBitmap which transfers the current canvas
>> contents to an ImageBitmap and clears the canvas. (Canvas implementations
>> are already optimized to support a zero-cost "cleared" state, because
>> existing benchmarks require it.) Sharing ImageBitmap contents across threads
>> during structured clone is not subtle. We can add an
>> HTMLImageElement.srcObject attribute which could take a Blob or an
>> ImageBitmap to enable explicit zero-copy rendering of ImageBitmaps. Would
>> that be explicit enough for you?
>
> Yes, that generally sounds good.
>
>
>> Personally I think high-performance manipulation of ImageBitmaps would be
>> more generally useful than detachable DrawingBuffers, and would be easier
>> for authors to understand.
>>
>> If you squint, WorkerCanvas.transferToImageBitmap is similar to detaching a
>> DrawingBuffer. But I don't see a need to reattach a buffer to a canvas for
>> further drawing. Do you?
>
> Not immediately. The ability to transfer out the canvas's contents,
> and render them in an HTMLImageElement without incurring an extra
> blit, should address the Maps team's requirements.
>
> Actually, adding transferToImageBitmap to HTMLCanvasElement as well
> would address the use case of rendering to multiple targets using one
> context. Instead of using multiple canvases as the targets, one would
> simply use multiple images. That sounds appealing.
>
> If WorkerCanvas is changed so that its width and height are mutable
> within the worker as you mentioned above, it sounds like it's
> addressing the known use cases.

Capturing Glenn Maynard's feedback from the other thread started by
Rik Cabanier, Glenn made a good point that there needs to be a way to
explicitly deallocate the ImageBitmap. Otherwise, the JavaScript
objects will have to be garbage collected before the GPU resource
(texture) it references can be freed, and that will not work -- GPU
resources will quickly pile up.

Would it be acceptable to say that setting the 'src' property of an
HTMLImageElement to an ImageBitmap neuters the incoming ImageBitmap?
If not, would it be feasible to add another method to HTMLImageElement
like setToImageBitmap which does this?

-Ken


>>> It's worth considering whether a change to the CanvasInWorkers
>>> proposal could support presenting frames directly from the worker.
>>
>>
>> Sure, by adding a commit() method to DrawingBuffer. Right?
>
> I'm not exactly sure how it would be done. In the proposal as written,
> the DrawingBuffer's not shared between threads, only transferred.
>
> -Ken
>
>
>> Rob
>> --
>> Jtehsauts  tshaei dS,o n" Wohfy  Mdaon  yhoaus  eanuttehrotraiitny  eovni le
>> atrhtohu gthot sf oirng iyvoeu rs ihnesa.r"t sS?o  Whhei csha iids  teoa
>> stiheer :p atroa lsyazye,d  'mYaonu,r  "sGients  uapr,e  tfaokreg iyvoeunr,
>> 'm aotr  atnod  sgaoy ,h o'mGee.t"  uTph eann dt hwea lmka'n?  gBoutt  uIp
>> waanndt  wyeonut  thoo mken.o w

Re: [whatwg] Canvas in workers

[Justin Novosad]

On Wed, Oct 16, 2013 at 5:29 PM, Domenic Denicola <
dome...@domenicdenicola.com> wrote:

> From: whatwg-boun...@lists.whatwg.org [whatwg-boun...@lists.whatwg.org]
> on behalf of Anne van Kesteren [ann...@annevk.nl]
>
> > On Wed, Oct 16, 2013 at 9:23 PM, Kenneth Russell  wrote:
> >> While the Promise returned from createImageBitmap(HTMLCanvasElement)
> can be fulfilled immediately, is it worth introducing a special overload
> with a different return type?
> >
> > Well if you want a synchronous method a promise is not going to cut it.
> However, please do not overload the existing method and actually provide a
> new one. Having one method that can be used both synchronous and
> asynchronous depending on its argument would be terrible.
>
> It's worth noting that immediately-fulfilled promises are pretty fast,
> since promises use microtasks.
>

Agreed, but even if the returned promise is immediately resolved, the
resolution notification (and the associated state transition) will be async
and will not happen at least until the current task (JS execution) ends,
which is what we would like to avoid by providing a synchronous API.  The
objective is to remove the complexity of having to use an async API in
cases where there is no benefit from being asynchronous.

Re: [whatwg] Canvas in workers

[Domenic Denicola]

From: whatwg-boun...@lists.whatwg.org [whatwg-boun...@lists.whatwg.org] on 
behalf of Anne van Kesteren [ann...@annevk.nl]

> On Wed, Oct 16, 2013 at 9:23 PM, Kenneth Russell  wrote:
>> While the Promise returned from createImageBitmap(HTMLCanvasElement) can be 
>> fulfilled immediately, is it worth introducing a special overload with a 
>> different return type?
>
> Well if you want a synchronous method a promise is not going to cut it. 
> However, please do not overload the existing method and actually provide a 
> new one. Having one method that can be used both synchronous and asynchronous 
> depending on its argument would be terrible.

It's worth noting that immediately-fulfilled promises are pretty fast, since 
promises use microtasks.

Re: [whatwg] Canvas in workers

[Anne van Kesteren]

On Wed, Oct 16, 2013 at 9:23 PM, Kenneth Russell  wrote:
> While the Promise returned from createImageBitmap(HTMLCanvasElement)
> can be fulfilled immediately, is it worth introducing a special
> overload with a different return type?

Well if you want a synchronous method a promise is not going to cut
it. However, please do not overload the existing method and actually
provide a new one. Having one method that can be used both synchronous
and asynchronous depending on its argument would be terrible.


-- 
http://annevankesteren.nl/

Re: [whatwg] Canvas in workers

[Kenneth Russell]

On Wed, Oct 16, 2013 at 1:13 PM, Justin Novosad  wrote:
>
>
>
> On Wed, Oct 16, 2013 at 3:53 PM, Robert O'Callahan 
> wrote:
>>
>> On Thu, Oct 17, 2013 at 6:35 AM, Kenneth Russell  wrote:
>>>
>>> Yes, right. That factory method is already spec'ed on the
>>> WorkerGlobalScope [1]. It actually returns a Promise, so presumably
>>> transferToImageBitmap would have to as well.
>>
>>
>> The whole point of transferToImageBitmap is that it's really fast, so I
>> don't see why it has to be async.
>
>
> True. I also wonder why all of the currently spec'ed ImageBitmap creation
> methods are async. I think asynchrony makes sense when creating ImageBitmaps
> from blobs, image elements and video elements, which may not be in an
> immediately accessible state, but creating an ImageBitmap from a Canvas or
> canvas context (or a WorkerCanvas) could be immediate.

It would be fine in my opinion for transferToImageBitmap to simply
return an ImageBitmap. The suggestion to return a Promise was merely
for symmetry with createImageBitmap.

While the Promise returned from createImageBitmap(HTMLCanvasElement)
can be fulfilled immediately, is it worth introducing a special
overload with a different return type?

-Ken

Re: [whatwg] Canvas in workers

[Justin Novosad]

On Wed, Oct 16, 2013 at 3:53 PM, Robert O'Callahan wrote:

> On Thu, Oct 17, 2013 at 6:35 AM, Kenneth Russell  wrote:
>
>> Yes, right. That factory method is already spec'ed on the
>> WorkerGlobalScope [1]. It actually returns a Promise, so presumably
>> transferToImageBitmap would have to as well.
>>
>
> The whole point of transferToImageBitmap is that it's really fast, so I
> don't see why it has to be async.
>

True. I also wonder why all of the currently spec'ed ImageBitmap creation
methods are async. I think asynchrony makes sense when creating
ImageBitmaps from blobs, image elements and video elements, which may not
be in an immediately accessible state, but creating an ImageBitmap from a
Canvas or canvas context (or a WorkerCanvas) could be immediate.

Re: [whatwg] Canvas in workers

[Robert O'Callahan]

On Thu, Oct 17, 2013 at 6:35 AM, Kenneth Russell  wrote:

> Yes, right. That factory method is already spec'ed on the
> WorkerGlobalScope [1]. It actually returns a Promise, so presumably
> transferToImageBitmap would have to as well.
>

The whole point of transferToImageBitmap is that it's really fast, so I
don't see why it has to be async.

Rob
-- 
Jtehsauts  tshaei dS,o n" Wohfy  Mdaon  yhoaus  eanuttehrotraiitny  eovni
le atrhtohu gthot sf oirng iyvoeu rs ihnesa.r"t sS?o  Whhei csha iids  teoa
stiheer :p atroa lsyazye,d  'mYaonu,r  "sGients  uapr,e  tfaokreg iyvoeunr,
'm aotr  atnod  sgaoy ,h o'mGee.t"  uTph eann dt hwea lmka'n?  gBoutt  uIp
waanndt  wyeonut  thoo mken.o w  *
*

Re: [whatwg] Canvas in workers

[Kenneth Russell]

On Wed, Oct 16, 2013 at 5:39 AM, Justin Novosad  wrote:
> On Tue, Oct 15, 2013 at 8:30 PM, Kenneth Russell  wrote:
>>
>> On Tue, Oct 15, 2013 at 4:41 PM, Robert O'Callahan 
>> wrote:
>> > If you squint, WorkerCanvas.transferToImageBitmap is similar to
>> > detaching a
>> > DrawingBuffer. But I don't see a need to reattach a buffer to a canvas
>> > for
>> > further drawing. Do you?
>>
>> Not immediately. The ability to transfer out the canvas's contents,
>> and render them in an HTMLImageElement without incurring an extra
>> blit, should address the Maps team's requirements.
>
>
> WorkerCanvas.copyToImageBitmap could be just as effective with a proper lazy
> copy-on-write mechanism. It would offer the same performance in cases where
> you would just need to transfer (as opposed to copy) the buffer, with the
> added flexibility that it reattaches a new buffer to the canvas, only if
> needed (at next draw).  Also the lazy copy can be skipped if the next draw
> operation to the canvas context is a clear, in which case the UA only needs
> to attach an uninitialized buffer.

I'm assuming that transferToImageBitmap will attach a new buffer to
the canvas as well. The semantic would be that the color buffer of the
canvas gets transferred to the ImageBitmap, and the canvas gets a new,
blank color buffer. (Any auxiliary buffers, like a depth buffer, would
also be implicitly cleared.)

It's easier to understand how to make the transfer operation efficient
than how to optimize the copy-on-write, which requires deeper analysis
of the calls made against the canvas and context in order to get peak
performance. If both are present then both the createImageBitmap and
transfer implementations can be really simple. Do you have a
particular objection to including the transfer API?

> Also, because ImageBitmaps are immutable objects, the API should probably be
> more like var imageBitmap = createImageBitmap(myWorkerCanvas);

Yes, right. That factory method is already spec'ed on the
WorkerGlobalScope [1]. It actually returns a Promise, so presumably
transferToImageBitmap would have to as well.

[1] 
http://www.whatwg.org/specs/web-apps/current-work/multipage/timers.html#imagebitmapfactories

-Ken

Re: [whatwg] Canvas in workers

[Glenn Maynard]

On Wed, Oct 16, 2013 at 8:01 AM, Justin Novosad  wrote:

> > ... oh... so the UI could be updated even if JS is blocking... the future
>
> is bright :-)
> >
>
> If the UI is all painted in a canvas, then yes.
>
> Let's not get ahead of ourselves though. Browsers that have a compositor in
> a separate thread can present frames without synchronizing with the main
> thread, but updating a regular (DOM-based) UI would likely require style
> and layout calculations to be propagated from the main thread to the
> compositor.


Right, that's the only sort of async update we're talking about here:
changing what gets composited, and not anything that affects layout or is
detectable by scripts.

But actually, there's no disagreement.  I misread Kenneth's mail as saying
"your proposal requires synchronization with the main thread", but he
actually said "this other proposal requires synchronization with the main
thread (but has other benefits)".

I'm not sure how big a problem that synchronization will be.  Posting
messages should be extremely cheap: essentially free, when you're just
transferring an object to a thread in the same process.  The UI thread
should also not be heavily loaded: you shouldn't have to wait long for the
script to receive the object and push it into the canvas.  But, even a cost
of 2ms would be a massive hit, since that's 12% of the time you have
available when rendering at 60 FPS.  In practice it might make it harder to
sustain smooth 60 FPS animation.  For example, if the main thread
occasionally spends 4ms doing GC work, you may have a rendering hitch that
you wouldn't have otherwise.

-- 
Glenn Maynard

Re: [whatwg] Canvas in workers

[Justin Novosad]

On Wed, Oct 16, 2013 at 4:32 AM, David Bruant  wrote:

>  Le 16/10/2013 01:26, Robert O'Callahan a écrit :
>
> On Wed, Oct 16, 2013 at 11:55 AM, David Bruant  wrote:
>
>> If the main thread is blocked, the app drops frames anyway, no?
>>
>
>  Not necessarily. We can allow workers to present frames to the
> compositor without synchronizing with the main thread.
>
> ... oh... so the UI could be updated even if JS is blocking... the future
> is bright :-)
>

If the UI is all painted in a canvas, then yes.

Let's not get ahead of ourselves though. Browsers that have a compositor in
a separate thread can present frames without synchronizing with the main
thread, but updating a regular (DOM-based) UI would likely require style
and layout calculations to be propagated from the main thread to the
compositor. So we are still gated by the main thread for a lot of things.
That will most like remain an issue for the foreseeable future. Presenting
frames directly from a worker can be trivially achieved in cases that do
not require updates to the DOM (like canvas contents changed). Some simple
types of changes to the DOM may also be represented directly in the
compositor, thus making it possible to short-circuit the main thread, but
these are a very limited subset of what a typical app might use in its UI.
 The saner approach is still for app devs is to move whatever blocking JS
they have off of the main thread.

   -Justin

Re: [whatwg] Canvas in workers

[Justin Novosad]

On Tue, Oct 15, 2013 at 8:30 PM, Kenneth Russell  wrote:

> On Tue, Oct 15, 2013 at 4:41 PM, Robert O'Callahan 
> wrote:
> > If you squint, WorkerCanvas.transferToImageBitmap is similar to
> detaching a
> > DrawingBuffer. But I don't see a need to reattach a buffer to a canvas
> for
> > further drawing. Do you?
>
> Not immediately. The ability to transfer out the canvas's contents,
> and render them in an HTMLImageElement without incurring an extra
> blit, should address the Maps team's requirements.
>

WorkerCanvas.copyToImageBitmap could be just as effective with a proper
lazy copy-on-write mechanism. It would offer the same performance in cases
where you would just need to transfer (as opposed to copy) the buffer, with
the added flexibility that it reattaches a new buffer to the canvas, only
if needed (at next draw).  Also the lazy copy can be skipped if the next
draw operation to the canvas context is a clear, in which case the UA only
needs to attach an uninitialized buffer.

Also, because ImageBitmaps are immutable objects, the API should probably
be more like var imageBitmap = createImageBitmap(myWorkerCanvas);

Re: [whatwg] Canvas in workers

[David Bruant]

Le 16/10/2013 01:26, Robert O'Callahan a écrit :
On Wed, Oct 16, 2013 at 11:55 AM, David Bruant > wrote:


If the main thread is blocked, the app drops frames anyway, no?


Not necessarily. We can allow workers to present frames to the 
compositor without synchronizing with the main thread.
... oh... so the UI could be updated even if JS is blocking... the 
future is bright :-)


David

Re: [whatwg] Canvas in workers

[Kenneth Russell]

On Tue, Oct 15, 2013 at 4:41 PM, Robert O'Callahan  wrote:
> On Wed, Oct 16, 2013 at 11:55 AM, Kenneth Russell  wrote:
>>
>> On Mon, Oct 14, 2013 at 1:34 PM, Robert O'Callahan 
>> wrote:
>> > On Mon, Oct 14, 2013 at 2:20 PM, Kenneth Russell  wrote:
>> >>
>> >> Would you mind looking at the proposal
>> >> http://wiki.whatwg.org/wiki/CanvasInWorkers and commenting on it?
>> >
>> >
>> > Sure. Kyle and I looked at it while we were working on our proposal. The
>> > main issues I have with it are that rearchitecting  to introduce
>> > the
>> > DrawingBuffer layer of abstraction seems unnecessarily complex, and it
>> > doesn't handle direct presentation of frames from the worker, bypassing
>> > the
>> > main thread.
>>
>> Note that the CanvasInWorkers draft solves some other longstanding
>> issues not addressed by the WorkerCanvas proposal. It provides the
>> ability to render to multiple canvases from a single context, whether
>> workers are involved or not.
>
>
> That may be a useful feature, but I'd like to see it justified in its own
> right.

There has been a lot of developer feedback on the WebGL mailing lists
over the past couple of years about exactly this feature. Web sites
like Turbosquid want to present lots of little thumbnails of models --
see for example http://www.turbosquid.com/Search/3D-Models/Vehicle/Car
-- and have them be interactive. It's too resource-intensive to create
a separate WebGL context for each. The most direct solution is to
allow one context to render to multiple canvases.


>> It achieves ideal memory utilization by
>> being very explicit in the API, without the need for extensive and
>> subtle optimizations behind the scenes.
>
>
> We can be more explicit with ImageBitmaps. We could provide
> WorkerCanvas.transferToImageBitmap which transfers the current canvas
> contents to an ImageBitmap and clears the canvas. (Canvas implementations
> are already optimized to support a zero-cost "cleared" state, because
> existing benchmarks require it.) Sharing ImageBitmap contents across threads
> during structured clone is not subtle. We can add an
> HTMLImageElement.srcObject attribute which could take a Blob or an
> ImageBitmap to enable explicit zero-copy rendering of ImageBitmaps. Would
> that be explicit enough for you?

Yes, that generally sounds good.


> Personally I think high-performance manipulation of ImageBitmaps would be
> more generally useful than detachable DrawingBuffers, and would be easier
> for authors to understand.
>
> If you squint, WorkerCanvas.transferToImageBitmap is similar to detaching a
> DrawingBuffer. But I don't see a need to reattach a buffer to a canvas for
> further drawing. Do you?

Not immediately. The ability to transfer out the canvas's contents,
and render them in an HTMLImageElement without incurring an extra
blit, should address the Maps team's requirements.

Actually, adding transferToImageBitmap to HTMLCanvasElement as well
would address the use case of rendering to multiple targets using one
context. Instead of using multiple canvases as the targets, one would
simply use multiple images. That sounds appealing.

If WorkerCanvas is changed so that its width and height are mutable
within the worker as you mentioned above, it sounds like it's
addressing the known use cases.


>> It's worth considering whether a change to the CanvasInWorkers
>> proposal could support presenting frames directly from the worker.
>
>
> Sure, by adding a commit() method to DrawingBuffer. Right?

I'm not exactly sure how it would be done. In the proposal as written,
the DrawingBuffer's not shared between threads, only transferred.

-Ken


> Rob
> --
> Jtehsauts  tshaei dS,o n" Wohfy  Mdaon  yhoaus  eanuttehrotraiitny  eovni le
> atrhtohu gthot sf oirng iyvoeu rs ihnesa.r"t sS?o  Whhei csha iids  teoa
> stiheer :p atroa lsyazye,d  'mYaonu,r  "sGients  uapr,e  tfaokreg iyvoeunr,
> 'm aotr  atnod  sgaoy ,h o'mGee.t"  uTph eann dt hwea lmka'n?  gBoutt  uIp
> waanndt  wyeonut  thoo mken.o w

Re: [whatwg] Canvas in workers

[Robert O'Callahan]

On Wed, Oct 16, 2013 at 11:55 AM, Kenneth Russell  wrote:

> On Mon, Oct 14, 2013 at 1:34 PM, Robert O'Callahan 
> wrote:
> > On Mon, Oct 14, 2013 at 2:20 PM, Kenneth Russell  wrote:
> >>
> >> Would you mind looking at the proposal
> >> http://wiki.whatwg.org/wiki/CanvasInWorkers and commenting on it?
> >
> >
> > Sure. Kyle and I looked at it while we were working on our proposal. The
> > main issues I have with it are that rearchitecting  to introduce
> the
> > DrawingBuffer layer of abstraction seems unnecessarily complex, and it
> > doesn't handle direct presentation of frames from the worker, bypassing
> the
> > main thread.
>
> Note that the CanvasInWorkers draft solves some other longstanding
> issues not addressed by the WorkerCanvas proposal. It provides the
> ability to render to multiple canvases from a single context, whether
> workers are involved or not.


That may be a useful feature, but I'd like to see it justified in its own
right.


> It achieves ideal memory utilization by
> being very explicit in the API, without the need for extensive and
> subtle optimizations behind the scenes.
>

We can be more explicit with ImageBitmaps. We could provide
WorkerCanvas.transferToImageBitmap which transfers the current canvas
contents to an ImageBitmap and clears the canvas. (Canvas implementations
are already optimized to support a zero-cost "cleared" state, because
existing benchmarks require it.) Sharing ImageBitmap contents across
threads during structured clone is not subtle. We can add an
HTMLImageElement.srcObject attribute which could take a Blob or an
ImageBitmap to enable explicit zero-copy rendering of ImageBitmaps. Would
that be explicit enough for you?

Personally I think high-performance manipulation of ImageBitmaps would be
more generally useful than detachable DrawingBuffers, and would be easier
for authors to understand.

If you squint, WorkerCanvas.transferToImageBitmap is similar to detaching a
DrawingBuffer. But I don't see a need to reattach a buffer to a canvas for
further drawing. Do you?

It's worth considering whether a change to the CanvasInWorkers
> proposal could support presenting frames directly from the worker.
>

Sure, by adding a commit() method to DrawingBuffer. Right?

Rob
-- 
Jtehsauts  tshaei dS,o n" Wohfy  Mdaon  yhoaus  eanuttehrotraiitny  eovni
le atrhtohu gthot sf oirng iyvoeu rs ihnesa.r"t sS?o  Whhei csha iids  teoa
stiheer :p atroa lsyazye,d  'mYaonu,r  "sGients  uapr,e  tfaokreg iyvoeunr,
'm aotr  atnod  sgaoy ,h o'mGee.t"  uTph eann dt hwea lmka'n?  gBoutt  uIp
waanndt  wyeonut  thoo mken.o w  *
*

Re: [whatwg] Canvas in workers

[Robert O'Callahan]

On Wed, Oct 16, 2013 at 11:55 AM, David Bruant  wrote:

> If the main thread is blocked, the app drops frames anyway, no?
>

Not necessarily. We can allow workers to present frames to the compositor
without synchronizing with the main thread.

Rob
-- 
Jtehsauts  tshaei dS,o n" Wohfy  Mdaon  yhoaus  eanuttehrotraiitny  eovni
le atrhtohu gthot sf oirng iyvoeu rs ihnesa.r"t sS?o  Whhei csha iids  teoa
stiheer :p atroa lsyazye,d  'mYaonu,r  "sGients  uapr,e  tfaokreg iyvoeunr,
'm aotr  atnod  sgaoy ,h o'mGee.t"  uTph eann dt hwea lmka'n?  gBoutt  uIp
waanndt  wyeonut  thoo mken.o w  *
*

Re: [whatwg] Canvas in workers

[Glenn Maynard]

On Mon, Oct 14, 2013 at 1:20 PM, Kenneth Russell  wrote:

>   1) Rendering from a worker and displaying on the main thread with no
> extra blits of the rendering results
>   2) Allows one context to render to multiple canvases
>   3) Supports resizing of the drawing buffer
>

The WorkerCanvas proposal should allow #1 and #3.  (It doesn't support #3
for purely offscreen worker canvases, but that'd be easy to add.)  #2 would
be nice with WebGL, where setting up extra contexts can be expensive, and
it may be simpler to do at the Canvas level than by mimicing OpenGL (eg.
shared resources across contexts).

There's been some recent discussion in the WebGL WG on this topic and

concerns were raised from other parties at Mozilla about the
> DrawingBuffer proposal above, including that it isn't possible to
> render from a worker without synchronizing with the main thread.
>

Your proposal does seem to require synchronization with the main thread, at
least with double-buffering.  You postMessage the DrawingBuffer to the main
thread to ask it to be displayed.  The worker can't start drawing the next
frame until it knows that the drawing buffers have been flipped; the buffer
flip happens in the main thread, when transferDrawingBufferToCanvas is
called.

WorkerCanvas performs the flip itself in the worker, when .commit() is
called (and possibly also when the script returns).  Even if the main
thread is busy, the worker should be able to do this immediately.  It does
need to be a thread-safe operation, but it doesn't need to block if the UI
thread is busy.

This proposal doesn't handle synchronizing DOM updates with threaded canvas
updates, but it seems like that inherently requires synchronization...

-- 
Glenn Maynard

Re: [whatwg] Canvas in workers

[Kenneth Russell]

On Mon, Oct 14, 2013 at 1:34 PM, Robert O'Callahan  wrote:
> On Mon, Oct 14, 2013 at 2:20 PM, Kenneth Russell  wrote:
>>
>> Would you mind looking at the proposal
>> http://wiki.whatwg.org/wiki/CanvasInWorkers and commenting on it?
>
>
> Sure. Kyle and I looked at it while we were working on our proposal. The
> main issues I have with it are that rearchitecting  to introduce the
> DrawingBuffer layer of abstraction seems unnecessarily complex, and it
> doesn't handle direct presentation of frames from the worker, bypassing the
> main thread.

Note that the CanvasInWorkers draft solves some other longstanding
issues not addressed by the WorkerCanvas proposal. It provides the
ability to render to multiple canvases from a single context, whether
workers are involved or not. It achieves ideal memory utilization by
being very explicit in the API, without the need for extensive and
subtle optimizations behind the scenes.

It's worth considering whether a change to the CanvasInWorkers
proposal could support presenting frames directly from the worker.


>> There's been some recent discussion in the WebGL WG on this topic and
>> concerns were raised from other parties at Mozilla about the
>> DrawingBuffer proposal above, including that it isn't possible to
>> render from a worker without synchronizing with the main thread.
>
>
> This is a huge limitation. For games and other animated content, achieving a
> stable frame rate is super important and a key motivation for adding canvas
> support to workers.
>
> My vision for handling the Maps use-cases based on our proposal is this: the
> worker renders to one or more WorkerCanvases, then does
> createImageBitmap(canvasContext), then ships the ImageBitmap(s) to the main
> thread using postMessage, and then renders those ImageBitmaps either by
> drawing them to a canvas or in some more direct way.
>
> This can all be implemented in a zero-copy way with the APIs currently in
> the spec, though it's not trivial. You'd need to do a few optimizations:
> -- createImageBitmap(canvasContext) would take a lazy snapshot of the canvas
> contents; i.e., further modifications to the canvas would trigger a copy (on
> the worker), but if the canvas is untouched no copy is required.
> -- structured clone of the ImageBitmap would not copy. This may actually
> require a small spec change.
> -- drawing an ImageBitmap to a 2D , if it covers the whole canvas,
> would simply replace the canvas buffer with the ImageBitmap contents (and
> perform copy-on-write if the script later makes changes to that canvas).
> These optimizations would all be useful in other contexts too. Whatever
> happens with canvas-in-a-worker, I bet we'll end up doing these
> optimizations for other reasons.

I agree that it's probably possible to make this work. It is still
worth considering whether these optimizations are going to be fragile,
and whether developers will fall off a performance cliff if something
subtle changes in their code or in the browser's behavior in the
future.


> It might make sense to create an API that renders an ImageBitmap more
> directly than drawing it to a canvas. For example we could create an API
> that allows an  element to render an ImageBitmap. It would be a bit
> simpler for authors and perhaps a bit simpler to implement than the final
> optimization in my list above.

That's an interesting possibility. It would probably be quite a bit
simpler to both specify and implement rather than using a canvas on
the receiving end.

-Ken


> Rob
> --
> Jtehsauts  tshaei dS,o n" Wohfy  Mdaon  yhoaus  eanuttehrotraiitny  eovni le
> atrhtohu gthot sf oirng iyvoeu rs ihnesa.r"t sS?o  Whhei csha iids  teoa
> stiheer :p atroa lsyazye,d  'mYaonu,r  "sGients  uapr,e  tfaokreg iyvoeunr,
> 'm aotr  atnod  sgaoy ,h o'mGee.t"  uTph eann dt hwea lmka'n?  gBoutt  uIp
> waanndt  wyeonut  thoo mken.o w

Re: [whatwg] Canvas in workers

[David Bruant]

Le 16/10/2013 00:12, Robert O'Callahan a écrit :
On Wed, Oct 16, 2013 at 10:19 AM, Justin Novosad > wrote:


rAF doesn't work in a worker, but that is easy to work around. You can
have a rAF handler on the main thread that propagates the signal
to your
workers by posting a "rAF" message to them.


That isn't a good solution since it forces the app to drop frames if 
the main thread is blocked.

If the main thread is blocked, the app drops frames anyway, no?
Anyway, on not-so-good hardware (most mobiles?), workers communication 
cost is non-trivial and eats a bit of the 16,6ms budget.


A friend told me recently that in AS3, objects on the screen can all 
listen to a "beforeframe" event (or something like that). In web 
platform equivalents, they have a per-element requestAnimationFrame 
(instead of only a global one).
I found this idea interesting. Among other things, it allows 
implementors to not emit the event if the object isn't on the screen. It 
feels like it encourages good practices too; if my  element is 
too low in the document to be on screen, maybe I don't want to draw on 
it... I wonder how many people check if the thing they want to update is 
actually on screen...


In the case of canvas, the UA not only could decide to not dispatch 
events, but could also tell precisely which part of the canvas is 
visible so that if the app cares or is capable or partial canvas paints, 
it can decide to do so.


In case the main thread is blocked, events can be saved in the worker 
and save processing resource/battery.


David

Re: [whatwg] Canvas in workers

[Robert O'Callahan]

On Wed, Oct 16, 2013 at 10:19 AM, Justin Novosad  wrote:

> rAF doesn't work in a worker, but that is easy to work around. You can
> have a rAF handler on the main thread that propagates the signal to your
> workers by posting a "rAF" message to them.
>

That isn't a good solution since it forces the app to drop frames if the
main thread is blocked.

Rob
-- 
Jtehsauts  tshaei dS,o n" Wohfy  Mdaon  yhoaus  eanuttehrotraiitny  eovni
le atrhtohu gthot sf oirng iyvoeu rs ihnesa.r"t sS?o  Whhei csha iids  teoa
stiheer :p atroa lsyazye,d  'mYaonu,r  "sGients  uapr,e  tfaokreg iyvoeunr,
'm aotr  atnod  sgaoy ,h o'mGee.t"  uTph eann dt hwea lmka'n?  gBoutt  uIp
waanndt  wyeonut  thoo mken.o w  *
*

Re: [whatwg] Canvas in workers

[Robert O'Callahan]

On Wed, Oct 16, 2013 at 10:06 AM, Justin Novosad  wrote:

> If we put that idea into the WorkerCanvas proposal, here is an idea of how
> commits could handle resizes asynchronously:
>
> 1) Main thread executes some JS that changes the size of the canvas, and
> posts an event to the Worker in order to propagate the change to the
> WorkerCanvas.
> 2) On the worker thread, the WorkerCanvas stores the new size, but does not
> apply it right away to avoid resetting canvas contents mid frame.
> 3) commit() is called on the WorkerCanvas, causing the current canvas
> contents to be sent to the main thread for display, and the new canvas size
> comes into effect (lazily?)
> 4) The main thread receives the committed pixels. N.B.: The size of the
> received buffer does not match canvas element's intrinsic size
> 5) At paint time, the canvas contents get displayed respecting the canvas
> element's current size. The mismatched pixel buffer is either
> streched/squashed or clipped/padded to fit.
>

Step 5 has an issue that different apps might want to make different
choices about whether and how to scale/pad/crop. But I think here we can
leverage CSS3 object-fit and object-position properties:
http://dev.w3.org/csswg/css-images-3/#object-fit
Let's make those apply to , where the "replaced content" is the
canvas backing store!

Then we would modify your proposal a little bit. Let's say that the main
thread cannot modify the canvas buffer size after the canvas has been
transferred. If you want to change the buffer size, you'll have to
postMessage your worker and have it update the buffer size. When the worker
commits a canvas buffer with a size that doesn't match the element size,
object-fit and object-position are consulted to determine what gets
rendered. Those values can be sent to the compositor, so we can always
render the right thing.

This means a worker constantly producing frames without yielding can't be
notified of a buffer size change, but I don't think that's a big deal.

Rob
-- 
Jtehsauts  tshaei dS,o n" Wohfy  Mdaon  yhoaus  eanuttehrotraiitny  eovni
le atrhtohu gthot sf oirng iyvoeu rs ihnesa.r"t sS?o  Whhei csha iids  teoa
stiheer :p atroa lsyazye,d  'mYaonu,r  "sGients  uapr,e  tfaokreg iyvoeunr,
'm aotr  atnod  sgaoy ,h o'mGee.t"  uTph eann dt hwea lmka'n?  gBoutt  uIp
waanndt  wyeonut  thoo mken.o w  *
*

Re: [whatwg] Canvas in workers

[Justin Novosad]

On Sun, Oct 13, 2013 at 7:55 PM, David Bruant  wrote:

>
> Not directly related, but I imagine requestAnimationFrame doesn't work in
> a worker. How can a script know when to stop drawing on a given canvas?
>
> rAF doesn't work in a worker, but that is easy to work around. You can
have a rAF handler on the main thread that propagates the signal to your
workers by posting a "rAF" message to them.

Re: [whatwg] Canvas in workers

[Justin Novosad]

Robert, I think your argument makes sense. The DrawingBuffer mechanism
described in  http://wiki.whatwg.org/wiki/CanvasInWorkers can be made
implicit and hidden under the hood in a way that is just as memory
efficient by using the right combination of read locks and copy-on-write.
However, I have concerns about how resizing would be handled by the
proposed commit mechanism. I think there is no perfect solution, but
whatever we do, I think animation smoothness should be a primary concern,
so we should avoid blocking any threads and we should avoid dropping frames
as much as possible.  I like the concept of handling canvas size mismatches
that is brought forward in the CanvasInWorkers proposal.

If we put that idea into the WorkerCanvas proposal, here is an idea of how
commits could handle resizes asynchronously:

1) Main thread executes some JS that changes the size of the canvas, and
posts an event to the Worker in order to propagate the change to the
WorkerCanvas.
2) On the worker thread, the WorkerCanvas stores the new size, but does not
apply it right away to avoid resetting canvas contents mid frame.
3) commit() is called on the WorkerCanvas, causing the current canvas
contents to be sent to the main thread for display, and the new canvas size
comes into effect (lazily?)
4) The main thread receives the committed pixels. N.B.: The size of the
received buffer does not match canvas element's intrinsic size
5) At paint time, the canvas contents get displayed respecting the canvas
element's current size. The mismatched pixel buffer is either
streched/squashed or clipped/padded to fit.

An alternative that was proposed earlier in this (e-mail) thread was for
the main thread to reject commits when there is a size mismatch. That does
respect the desire to be asynchronous, and it avoid the inconvenience of
resizing contents, but I don't like the idea because it means there can be
cases where we may have a large number of consecutive dropped frames. Think
of drag-resizing for example. We should avoid making that common use case
behave poorly.





On Mon, Oct 14, 2013 at 4:34 PM, Robert O'Callahan wrote:

> On Mon, Oct 14, 2013 at 2:20 PM, Kenneth Russell  wrote:
>
> > Would you mind looking at the proposal
> > http://wiki.whatwg.org/wiki/CanvasInWorkers and commenting on it?
>
>
> Sure. Kyle and I looked at it while we were working on our proposal. The
> main issues I have with it are that rearchitecting  to introduce
> the DrawingBuffer layer of abstraction seems unnecessarily complex, and it
> doesn't handle direct presentation of frames from the worker, bypassing the
> main thread.
>
> There's been some recent discussion in the WebGL WG on this topic and
> > concerns were raised from other parties at Mozilla about the
> > DrawingBuffer proposal above, including that it isn't possible to
> > render from a worker without synchronizing with the main thread.
> >
>
> This is a huge limitation. For games and other animated content, achieving
> a stable frame rate is super important and a key motivation for adding
> canvas support to workers.
>
> My vision for handling the Maps use-cases based on our proposal is this:
> the worker renders to one or more WorkerCanvases, then does
> createImageBitmap(canvasContext), then ships the ImageBitmap(s) to the main
> thread using postMessage, and then renders those ImageBitmaps either by
> drawing them to a canvas or in some more direct way.
>
> This can all be implemented in a zero-copy way with the APIs currently in
> the spec, though it's not trivial. You'd need to do a few optimizations:
> -- createImageBitmap(canvasContext) would take a lazy snapshot of the
> canvas contents; i.e., further modifications to the canvas would trigger a
> copy (on the worker), but if the canvas is untouched no copy is required.
> -- structured clone of the ImageBitmap would not copy. This may actually
> require a small spec change.
> -- drawing an ImageBitmap to a 2D , if it covers the whole canvas,
> would simply replace the canvas buffer with the ImageBitmap contents (and
> perform copy-on-write if the script later makes changes to that canvas).
> These optimizations would all be useful in other contexts too. Whatever
> happens with canvas-in-a-worker, I bet we'll end up doing these
> optimizations for other reasons.
>
> It might make sense to create an API that renders an ImageBitmap more
> directly than drawing it to a canvas. For example we could create an API
> that allows an  element to render an ImageBitmap. It would be a bit
> simpler for authors and perhaps a bit simpler to implement than the final
> optimization in my list above.
>
> Rob
> --
> Jtehsauts  tshaei dS,o n" Wohfy  Mdaon  yhoaus  eanuttehrotraiitny  eovni
> le atrhtohu gthot sf oirng iyvoeu rs ihnesa.r"t sS?o  Whhei csha iids  teoa
> stiheer :p atroa lsyazye,d  'mYaonu,r  "sGients  uapr,e  tfaokreg iyvoeunr,
> 'm aotr  atnod  sgaoy ,h o'mGee.t"  uTph eann dt hwea lmka'n?  gBoutt  uIp
> waanndt  wyeonut  thoo mken.o w 

Re: [whatwg] Canvas in workers

[Robert O'Callahan]

On Mon, Oct 14, 2013 at 2:20 PM, Kenneth Russell  wrote:

> Would you mind looking at the proposal
> http://wiki.whatwg.org/wiki/CanvasInWorkers and commenting on it?


Sure. Kyle and I looked at it while we were working on our proposal. The
main issues I have with it are that rearchitecting  to introduce
the DrawingBuffer layer of abstraction seems unnecessarily complex, and it
doesn't handle direct presentation of frames from the worker, bypassing the
main thread.

There's been some recent discussion in the WebGL WG on this topic and
> concerns were raised from other parties at Mozilla about the
> DrawingBuffer proposal above, including that it isn't possible to
> render from a worker without synchronizing with the main thread.
>

This is a huge limitation. For games and other animated content, achieving
a stable frame rate is super important and a key motivation for adding
canvas support to workers.

My vision for handling the Maps use-cases based on our proposal is this:
the worker renders to one or more WorkerCanvases, then does
createImageBitmap(canvasContext), then ships the ImageBitmap(s) to the main
thread using postMessage, and then renders those ImageBitmaps either by
drawing them to a canvas or in some more direct way.

This can all be implemented in a zero-copy way with the APIs currently in
the spec, though it's not trivial. You'd need to do a few optimizations:
-- createImageBitmap(canvasContext) would take a lazy snapshot of the
canvas contents; i.e., further modifications to the canvas would trigger a
copy (on the worker), but if the canvas is untouched no copy is required.
-- structured clone of the ImageBitmap would not copy. This may actually
require a small spec change.
-- drawing an ImageBitmap to a 2D , if it covers the whole canvas,
would simply replace the canvas buffer with the ImageBitmap contents (and
perform copy-on-write if the script later makes changes to that canvas).
These optimizations would all be useful in other contexts too. Whatever
happens with canvas-in-a-worker, I bet we'll end up doing these
optimizations for other reasons.

It might make sense to create an API that renders an ImageBitmap more
directly than drawing it to a canvas. For example we could create an API
that allows an  element to render an ImageBitmap. It would be a bit
simpler for authors and perhaps a bit simpler to implement than the final
optimization in my list above.

Rob
-- 
Jtehsauts  tshaei dS,o n" Wohfy  Mdaon  yhoaus  eanuttehrotraiitny  eovni
le atrhtohu gthot sf oirng iyvoeu rs ihnesa.r"t sS?o  Whhei csha iids  teoa
stiheer :p atroa lsyazye,d  'mYaonu,r  "sGients  uapr,e  tfaokreg iyvoeunr,
'm aotr  atnod  sgaoy ,h o'mGee.t"  uTph eann dt hwea lmka'n?  gBoutt  uIp
waanndt  wyeonut  thoo mken.o w  *
*

Re: [whatwg] Canvas in workers

[Kenneth Russell]

Would you mind looking at the proposal
http://wiki.whatwg.org/wiki/CanvasInWorkers and commenting on it? This
was arrived at after extensive discussions with the Google Maps team,
and addresses their key use cases. Compared to the one below, it
solves the following problems:

  1) Rendering from a worker and displaying on the main thread with no
extra blits of the rendering results
  2) Allows one context to render to multiple canvases
  3) Supports resizing of the drawing buffer

The main difference in my mind is that in the DrawingBuffer proposal,
the back buffer of the canvas can be "detached", transferred via
postMessage to the other side, and attached to a Canvas, replacing its
previous back buffer. The semantics are simple, clear, avoid extra
blits of the rendered content, and support rendering into multiple
canvases from one context.

There's been some recent discussion in the WebGL WG on this topic and
concerns were raised from other parties at Mozilla about the
DrawingBuffer proposal above, including that it isn't possible to
render from a worker without synchronizing with the main thread.
Still, it seems to me it's worth considering all aspects of the
proposal, because it was motivated by a major existing web app which
is both using Canvas 2D and WebGL extensively, and desires to use
workers more heavily in its rendering pipeline.

-Ken



On Sat, Oct 12, 2013 at 9:12 PM, Kyle Huey  wrote:
> I talked at length with Robert O'Callahan about what the DOM API for
> supporting  in web workers should look like and we came up with the
> following modifications to the spec.
>
>1. Rename CanvasProxy to WorkerCanvas and only allow it to be
>transferred to workers.  I don't think we're interested in supporting
>cross-origin  via CanvasProxy (I would be curious to hear more
>about what the use cases are).
>2. Add a worker-only WorkerCanvas constructor that takes the desired
>width/height of the drawing surface.
>3. Remove the rendering context constructors and the setContext method
>on WorkerCanvas (née CanvasProxy).
>4. Copy all of the sensible non-node related things from
>HTMLCanvasElement to WorkerCanvas.  This would include
>- width and height as readonly attributes
>   - getContext (to replace what we removed in step 3).  roc prefers to
>   have getContext2D and getContextWebGL, and dispense with the string
>   argument version entirely, but I don't have strong feelings.
>   - toBlob.  We do not intend to implement toDataURL here.
>5. Add a "commit" method to WorkerCanvas.  For a WorkerCanvas obtained
>from a main thread  element, this would cause the buffer displayed
>on screen to swap.  For a WorkerCanvas created *de novo* on a worker
>thread, it would do nothing.  This commit method would also commit a minor
>violation of run-to-completion semantics, described below.
>6. We would rely on extracting ImageBitmaps from the WorkerCanvas and
>shipping them to the main thread via postMessage to allow synchronizing
>canvas updates with DOM updates.  We explored a couple other options but we
>didn't come up with anything else that allows synchronizing updates to
>multiple canvases from a worker.  This isn't really sketched out here.
>
> So the IDL would look something like:
>
>> [Constructor(unsigned long width, unsigned long height)]
>>
>> interface WorkerCanvas {
>>
>>   readonly attribute unsigned long width;
>>
>>   readonly attribute unsigned long height;
>>
>>
>>   CanvasRenderingContext2D? getContext2D(any... args);
>>
>>   WebGLRenderingContext? getContextWebGL(any... args);
>>
>>
>>   void toBlob(FileCallback? _callback, optional DOMString type, any...
>> arguments);
>>
>>
>>   bool commit();
>>
>> };
>>
>> WorkerCanvas implements Transferable;
>>
> Everything would be behave pretty much as one would expect, except perhaps
> for the commit method.  The width and height of the canvas can be modified
> on the main thread while the worker is drawing.  This would fire an event
> off to the worker to update the WorkerCanvas's dimensions that would be
> scheduled as if the main thread had postMessage()d something to the
> worker.  But it's possible that the worker would attempt to draw to the
>  before that update runs.  It's also possible that the worker would
> simply draw in a loop without yielding.  To solve this, if commit is called
> and the current dimensions on the main thread don't match the dimensions of
> the WorkerCanvas it would fail (return false) and update the dimensions of
> the WorkerCanvas before returning.  This is technically a violation of
> run-to-completion semantics, but is needed to support workers that do not
> yield.
>
> Thoughts?
>
> - Kyle

Re: [whatwg] Canvas in workers

[Glenn Maynard]

On Sun, Oct 13, 2013 at 11:22 AM, David Bruant  wrote:

>  bool commit();
>>>
 Boolean as return value for success? :-s

>>> A promise instead maybe? throw instead of false at least?
>>> In any case, it looks like commit could be a long operation (tell me if
>>> I'm wrong here. Do you have numbers on how long it takes/would take?),
>>> having it async sounds reasonable.
>>>
>>
>> This should be synchronous and never block.
>>
> I might have misused the word "async" again, sorry about that. I think we
> agree.


Sorry, I also stated this imprecisely ("block" is somewhat overloaded).
Specifically, this function never has to sit around and wait for the
corresponding Canvas thread to become available; it can perform the flip
while the other thread is working.  And if it really did need to
block--we're in a worker anyway, so that's OK.


>  Even if the Canvas is in a
>> different process, it should be possible to do this with IPC without
>> waiting for the other side to process the change.
>>
> How does a worker know when the changes on the screen happened? I imagine
> a worker would want to know that before performing other changes to the
> canvas.
>

If the Canvas wants to change its size, then it should ask the owner of the
WorkerCanvas to make the change, which means the canvas size will never
change while the worker is drawing.  The WorkerCanvas would never change
size while the worker is rendering, and it would know that the canvas
changed size when it receives an onresize event.


On Sun, Oct 13, 2013 at 4:42 PM, Robert O'Callahan wrote:

> >1. Rename CanvasProxy to WorkerCanvas and only allow it to be
>> >transferred to workers.  I don't think we're interested in supporting
>> >cross-origin  via CanvasProxy (I would be curious to hear
>> more
>> >about what the use cases are).
>>
>

>
> Basically it's simpler to have CanvasProxy/WorkerCanvas only supported on
> workers. Cross-origin isn't itself a concern.
>

I think restricting this to workers is fine.

A simple way is for calls to any methods throw an exception if you're not
in a worker.  This keeps structured clone and transfer simple: the rules
for passing it around are the same as anything else, you just can't use it
if you're not a Worker.  If
https://www.w3.org/Bugs/Public/show_bug.cgi?id=23358 is implemented, that'd
give an easy way to define this.



> 2. Add a worker-only WorkerCanvas constructor that takes the desired
>> >width/height of the drawing surface.
>> >
>>
>> This looks like it's trying to allow entirely off-screen rendering within
>> a
>> Worker, which is fine, but there's no way to resize the backing store in
>> this mode.
>>
>
> We don't have a use-case for resizing the backing store of a
> worker-created canvas.
>

I suspect this will come up with WebGL, since recreating the context from
scratch can be a lot more expensive there.  We should at least make sure
this is possible later.

Actually, there is a way: change the width and height on the
CanvasRenderingContext2D you get from getContext, which isn't readonly.
I'm guessing that would actually want to be read-only in this proposal,
since it allows making changes that are visible to scripts in the UI thread.


> There is the slight problem that changing both width and height would fire
> two events.
>

Changes to width and height should probably only be applied when the script
returns to its event loop.


> A bigger problem is that your approach isn't compatible with a worker that
> draws frames in a loop without yielding. I'm uncertain how important that
> is, so I'll wait for Kyle to address that.
>

OK.  What are the use cases for doing that?  Being able to do complex work
in a worker in a linear, non-event-based manner is an important use case
for workers in general, but I can't think of any way this applies to
drawing successive frames to a canvas.

-- 
Glenn Maynard

Re: [whatwg] Canvas in workers

[David Bruant]

Le 14/10/2013 01:27, Robert O'Callahan a écrit :
On Sun, Oct 13, 2013 at 6:07 PM, David Bruant > wrote:


Le 13/10/2013 23:33, Robert O'Callahan a écrit :



   bool commit();

Boolean as return value for success? :-s
A promise instead maybe? throw instead of false at least?
In any case, it looks like commit could be a long
operation (tell
me if I'm wrong here. Do you have numbers on how long it
takes/would take?), having it async sounds reasonable.


"commit" does not need to be async, it doesn't have to block
in practice.

Just so that I better understand how commit works, what happens in
this case:

   // draw things
  wc.commit();
  // draw other things
  wc.commit();


Two frames get queued as eligible for presentation. Which ones reach 
the screen is unspecified of course.
Is there a way to know if/when a frame reaches the screen? Wouldn't this 
information be useful at least to know when it's safe to commit again?


Not directly related, but I imagine requestAnimationFrame doesn't work 
in a worker. How can a script know when to stop drawing on a given canvas?





Personally I don't see this as a violation of
run-to-completion semantics. The important thing is that the
dimensions of the WorkerCanvas (and any other state observable
by the worker) do not change except during the call to
commit(). Effectively commit() itself performs the change.

Hmm... Interesting point of view...
That's the only way to learn that the dimensions have changed? Via
commit() throwing an error/returning false? It feels very ad-hoc.
It doesn't seem to scale very well to other reasons why commit()
might fail.


What other reasons are there?

Won't there ever be?



I also imagine that the normal pattern will be "draw then commit",
so it means that you always spend efforts drawing without ever
knowing if what you're drawing is ever going to end up on screen;
which feels like a waste of resources.


I don't see this as being a problem, or at least no more of a problem 
than it already is.
What do you mean by "than it already is"? Currently, if one calls a 
method on a context2d, the result is guaranteed to appear on screen 
eventually, no? It never happens that you start drawing and all of it 
gets dropped because of external conditions, right?





Of course it's OK for DOM API implementations to change state :-).

I don't understand what you mean here.


It's OK for calling DOM methods to change state visible to script.
That's not the problem. The semantics of commit is "send this frame and 
maybe change dimensions". It's an awkward semantics mixing largely 
unrelated things.
It's like if the send() method of a worker's XHR didn't work when the 
 changed in the parent page. It's also very dependent on timing 
and may cause bugs that are very hard to debug.


David

Re: [whatwg] Canvas in workers

[Robert O'Callahan]

On Sun, Oct 13, 2013 at 6:07 PM, David Bruant  wrote:

> Le 13/10/2013 23:33, Robert O'Callahan a écrit :
>
>
>>
>>bool commit();
>>
>> Boolean as return value for success? :-s
>> A promise instead maybe? throw instead of false at least?
>> In any case, it looks like commit could be a long operation (tell
>> me if I'm wrong here. Do you have numbers on how long it
>> takes/would take?), having it async sounds reasonable.
>>
>>
>> "commit" does not need to be async, it doesn't have to block in practice.
>>
> Just so that I better understand how commit works, what happens in this
> case:
>
>// draw things
>   wc.commit();
>   // draw other things
>   wc.commit();


Two frames get queued as eligible for presentation. Which ones reach the
screen is unspecified of course.


>  Personally I don't see this as a violation of run-to-completion
>> semantics. The important thing is that the dimensions of the WorkerCanvas
>> (and any other state observable by the worker) do not change except during
>> the call to commit(). Effectively commit() itself performs the change.
>>
> Hmm... Interesting point of view...
> That's the only way to learn that the dimensions have changed? Via
> commit() throwing an error/returning false? It feels very ad-hoc. It
> doesn't seem to scale very well to other reasons why commit() might fail.
>

What other reasons are there?

I also imagine that the normal pattern will be "draw then commit", so it
> means that you always spend efforts drawing without ever knowing if what
> you're drawing is ever going to end up on screen; which feels like a waste
> of resources.


I don't see this as being a problem, or at least no more of a problem than
it already is.


>  Of course it's OK for DOM API implementations to change state :-).
>>
> I don't understand what you mean here.
>

It's OK for calling DOM methods to change state visible to script.

What happens to WorkerCanvas in the canvas element is removed from the
> document or display:hidden-ed?
>

It's unaffected.

Rob
-- 
Jtehsauts  tshaei dS,o n" Wohfy  Mdaon  yhoaus  eanuttehrotraiitny  eovni
le atrhtohu gthot sf oirng iyvoeu rs ihnesa.r"t sS?o  Whhei csha iids  teoa
stiheer :p atroa lsyazye,d  'mYaonu,r  "sGients  uapr,e  tfaokreg iyvoeunr,
'm aotr  atnod  sgaoy ,h o'mGee.t"  uTph eann dt hwea lmka'n?  gBoutt  uIp
waanndt  wyeonut  thoo mken.o w  *
*

Re: [whatwg] Canvas in workers

[David Bruant]

Le 13/10/2013 23:33, Robert O'Callahan a écrit :



   bool commit();

Boolean as return value for success? :-s
A promise instead maybe? throw instead of false at least?
In any case, it looks like commit could be a long operation (tell
me if I'm wrong here. Do you have numbers on how long it
takes/would take?), having it async sounds reasonable.


"commit" does not need to be async, it doesn't have to block in practice.
Just so that I better understand how commit works, what happens in this 
case:


   // draw things
  wc.commit();
  // draw other things
  wc.commit();

?




I feel fairly strongly against the run-to-completion violation as
it's a foundation of how JavaScript works, how people reason about
programs. Also, TC39 is working hard to close the gap between what
can be expressed in pure ECMAScript and what the web platform does
express, it'd be inappropriate to add things that widen this gap
in my opinion.


Personally I don't see this as a violation of run-to-completion 
semantics. The important thing is that the dimensions of the 
WorkerCanvas (and any other state observable by the worker) do not 
change except during the call to commit(). Effectively commit() itself 
performs the change.

Hmm... Interesting point of view...
That's the only way to learn that the dimensions have changed? Via 
commit() throwing an error/returning false? It feels very ad-hoc. It 
doesn't seem to scale very well to other reasons why commit() might fail.
I also imagine that the normal pattern will be "draw then commit", so it 
means that you always spend efforts drawing without ever knowing if what 
you're drawing is ever going to end up on screen; which feels like a 
waste of resources.



Of course it's OK for DOM API implementations to change state :-).

I don't understand what you mean here.

What happens to WorkerCanvas in the canvas element is removed from the 
document or display:hidden-ed?


David

Re: [whatwg] Canvas in workers

[Robert O'Callahan]

On Sun, Oct 13, 2013 at 12:12 PM, Glenn Maynard  wrote:

>  On Sat, Oct 12, 2013 at 11:12 PM, Kyle Huey  wrote:
>
> >1. Rename CanvasProxy to WorkerCanvas and only allow it to be
> >transferred to workers.  I don't think we're interested in supporting
> >cross-origin  via CanvasProxy (I would be curious to hear more
> >about what the use cases are).
> >
>
> You can transfer data to a worker that's cross-origin, if you have a
> MessagePort where the other side goes to another origin's worker (possibly
> given to you via eg. window.postMessage).
>
> Is the real goal here trying to limit this to threads and avoid IPC, or is
> this actually a cross-origin concern?
>

Basically it's simpler to have CanvasProxy/WorkerCanvas only supported on
workers. Cross-origin isn't itself a concern.


>2. Add a worker-only WorkerCanvas constructor that takes the desired
> >width/height of the drawing surface.
> >
>
> This looks like it's trying to allow entirely off-screen rendering within a
> Worker, which is fine, but there's no way to resize the backing store in
> this mode.
>

We don't have a use-case for resizing the backing store of a worker-created
canvas.

> simply draw in a loop without yielding.  To solve this, if commit is
> called
> > and the current dimensions on the main thread don't match the dimensions
> of
> > the WorkerCanvas it would fail (return false) and update the dimensions
> of
> > the WorkerCanvas before returning.  This is technically a violation of
> > run-to-completion semantics, but is needed to support workers that do not
> > yield.
> >
>
> This sounds like it's easy to get wrong, since it'll probably be rare.  An
> exception might be better, so if you don't handle it you at least get an
> error logged to the console.
>

Sure, that's probably better.

There will be flicker issues with this.  The canvas is cleared when you
> change width or height.  In the UI thread that's OK, since the author can
> synchronously redraw immediately after changing the size.  Here, it's
> likely it won't be redrawn in time, so it'll flicker whenever the size
> changes, especially if it's being changed smoothly.  Here's a suggestion to
> fix this:
>
> - When the UI thread changes Canvas.width or Canvas.height, it doesn't
> actually resize buffers.  Instead, it sends a message to the WorkerCanvas
> asking for the change.  Until the change actually happens, the Canvas
> continues to be composited as before.  (However, the change to .width and
> .height is visible on the object immediately.)
> - When the WorkerCanvas's event loop receives a message asking for a size
> change:
>   - Change the size of the back-buffer, and update WorkerCanvas.width and
> WorkerCanvas.height accordingly.
>   - Fire onresize on the WorkerCanvas.  The worker is expected to redraw
> here.  (This is where the "implicit commit" matters: we want to guarantee a
> commit here.)
>   - Only when the newly-redrawn buffer is committed does the front buffer's
> size get updated to match the back-buffer.
>
> In other words, when you change the size in the UI thread, it continues to
> composite the same image (possibly not filling the whole element, or being
> stretched) until the worker actually gets the resize and has a chance to
> redraw it.
>
> This also means the idea of not being able to commit because of a resize
> can go away, and commit() can be void, since the back-buffer size never
> actually changes while the worker is drawing.
>

There is the slight problem that changing both width and height would fire
two events.

A bigger problem is that your approach isn't compatible with a worker that
draws frames in a loop without yielding. I'm uncertain how important that
is, so I'll wait for Kyle to address that.

Rob
-- 
Jtehsauts  tshaei dS,o n" Wohfy  Mdaon  yhoaus  eanuttehrotraiitny  eovni
le atrhtohu gthot sf oirng iyvoeu rs ihnesa.r"t sS?o  Whhei csha iids  teoa
stiheer :p atroa lsyazye,d  'mYaonu,r  "sGients  uapr,e  tfaokreg iyvoeunr,
'm aotr  atnod  sgaoy ,h o'mGee.t"  uTph eann dt hwea lmka'n?  gBoutt  uIp
waanndt  wyeonut  thoo mken.o w  *
*

Re: [whatwg] Canvas in workers

[Robert O'Callahan]

On Sun, Oct 13, 2013 at 12:01 PM, David Bruant  wrote:

> Le 13/10/2013 06:12, Kyle Huey a écrit :
>
>  2. Add a worker-only WorkerCanvas constructor that takes the desired
>> width/height of the drawing surface.
>>
> What is the use case for this constructor? Draw something in worker,
> output an image to be sent to main thread?


When drawing complex stuff to a canvas (e.g. opacity groups) you often need
to create a temporary canvas to hold some subset of the content and then
composite that temporary canvas to the main canvas.


>
>
>  3. Remove the rendering context constructors and the setContext method
>> on WorkerCanvas (née CanvasProxy).
>> 4. Copy all of the sensible non-node related things from
>> HTMLCanvasElement to WorkerCanvas.  This would include
>> - width and height as readonly attributes
>>- getContext (to replace what we removed in step 3).  roc prefers
>> to
>>have getContext2D and getContextWebGL, and dispense with the string
>>argument version entirely, but I don't have strong feelings.
>>
> For the sake of writing interoperable code in main thread and worker, I
> tend to be against this sort of change. I'm aware of the ugliness of some
> APIs, but consistent ugly APIs beats a mix of beautiful and ugly API.


Fine, let's consider that suggestion dropped.


>
>
> - toBlob.  We do not intend to implement toDataURL here.
>> 5. Add a "commit" method to WorkerCanvas.  For a WorkerCanvas obtained
>> from a main thread  element, this would cause the buffer
>> displayed
>> on screen to swap.  For a WorkerCanvas created *de novo* on a worker
>> thread, it would do nothing.
>>
> Let's have this method optional, then? Or create 2 interfaces? I'm not
> sure of what can be expressed in WebIDL to solve this, but useless methods
> aren't a good idea. If I can't do anything, don't give me the method. It's
> like in UI. If I can't click a button, just don't show me the button or at
> least grey it out.


We could put it in a separate interface. I'm not sure that's worth it, but
whatever, that's a trivial issue.


>
>>bool commit();
>>>
>> Boolean as return value for success? :-s
> A promise instead maybe? throw instead of false at least?
> In any case, it looks like commit could be a long operation (tell me if
> I'm wrong here. Do you have numbers on how long it takes/would take?),
> having it async sounds reasonable.


"commit" does not need to be async, it doesn't have to block in practice.

I feel fairly strongly against the run-to-completion violation as it's a
> foundation of how JavaScript works, how people reason about programs. Also,
> TC39 is working hard to close the gap between what can be expressed in pure
> ECMAScript and what the web platform does express, it'd be inappropriate to
> add things that widen this gap in my opinion.
>

Personally I don't see this as a violation of run-to-completion semantics.
The important thing is that the dimensions of the WorkerCanvas (and any
other state observable by the worker) do not change except during the call
to commit(). Effectively commit() itself performs the change. Of course
it's OK for DOM API implementations to change state :-).

Rob
-- 
Jtehsauts  tshaei dS,o n" Wohfy  Mdaon  yhoaus  eanuttehrotraiitny  eovni
le atrhtohu gthot sf oirng iyvoeu rs ihnesa.r"t sS?o  Whhei csha iids  teoa
stiheer :p atroa lsyazye,d  'mYaonu,r  "sGients  uapr,e  tfaokreg iyvoeunr,
'm aotr  atnod  sgaoy ,h o'mGee.t"  uTph eann dt hwea lmka'n?  gBoutt  uIp
waanndt  wyeonut  thoo mken.o w  *
*

Re: [whatwg] Canvas in workers

[David Bruant]

Le 13/10/2013 18:12, Glenn Maynard a écrit :
On Sun, Oct 13, 2013 at 11:01 AM, David Bruant  
wrote:

bool commit();

Boolean as return value for success? :-s

A promise instead maybe? throw instead of false at least?
In any case, it looks like commit could be a long operation (tell me if
I'm wrong here. Do you have numbers on how long it takes/would take?),
having it async sounds reasonable.


This should be synchronous and never block.
I might have misused the word "async" again, sorry about that. I think 
we agree.



Even if the Canvas is in a
different process, it should be possible to do this with IPC without
waiting for the other side to process the change.
How does a worker know when the changes on the screen happened? I 
imagine a worker would want to know that before performing other changes 
to the canvas.


David

Re: [whatwg] Canvas in workers

[Glenn Maynard]

 On Sat, Oct 12, 2013 at 11:12 PM, Kyle Huey  wrote:

>1. Rename CanvasProxy to WorkerCanvas and only allow it to be
>transferred to workers.  I don't think we're interested in supporting
>cross-origin  via CanvasProxy (I would be curious to hear more
>about what the use cases are).
>

You can transfer data to a worker that's cross-origin, if you have a
MessagePort where the other side goes to another origin's worker (possibly
given to you via eg. window.postMessage).

Is the real goal here trying to limit this to threads and avoid IPC, or is
this actually a cross-origin concern?

   2. Add a worker-only WorkerCanvas constructor that takes the desired
>width/height of the drawing surface.
>

This looks like it's trying to allow entirely off-screen rendering within a
Worker, which is fine, but there's no way to resize the backing store in
this mode.  I don't know if that would need a separate subclass of
WorkerCanvas to allow making width/height writable.


>   - getContext (to replace what we removed in step 3).  roc prefers to
>   have getContext2D and getContextWebGL, and dispense with the string
>   argument version entirely, but I don't have strong feelings.
>


> >   CanvasRenderingContext2D? getContext2D(any... args);
> >
> >   WebGLRenderingContext? getContextWebGL(any... args);
>

This is crazy.  The platform is inconsistent enough.  We have an API for
this already, getContext(); don't add a different API for the exact same
thing.

   5. Add a "commit" method to WorkerCanvas.  For a WorkerCanvas obtained
>

I'm not sure what this is for.  If you draw in a worker and return without
calling .commit(), is the commit implicit when you return to the event
loop?  (See below for where this matters.)


> simply draw in a loop without yielding.  To solve this, if commit is called
> and the current dimensions on the main thread don't match the dimensions of
> the WorkerCanvas it would fail (return false) and update the dimensions of
> the WorkerCanvas before returning.  This is technically a violation of
> run-to-completion semantics, but is needed to support workers that do not
> yield.
>

This sounds like it's easy to get wrong, since it'll probably be rare.  An
exception might be better, so if you don't handle it you at least get an
error logged to the console.

There will be flicker issues with this.  The canvas is cleared when you
change width or height.  In the UI thread that's OK, since the author can
synchronously redraw immediately after changing the size.  Here, it's
likely it won't be redrawn in time, so it'll flicker whenever the size
changes, especially if it's being changed smoothly.  Here's a suggestion to
fix this:

- When the UI thread changes Canvas.width or Canvas.height, it doesn't
actually resize buffers.  Instead, it sends a message to the WorkerCanvas
asking for the change.  Until the change actually happens, the Canvas
continues to be composited as before.  (However, the change to .width and
.height is visible on the object immediately.)
- When the WorkerCanvas's event loop receives a message asking for a size
change:
  - Change the size of the back-buffer, and update WorkerCanvas.width and
WorkerCanvas.height accordingly.
  - Fire onresize on the WorkerCanvas.  The worker is expected to redraw
here.  (This is where the "implicit commit" matters: we want to guarantee a
commit here.)
  - Only when the newly-redrawn buffer is committed does the front buffer's
size get updated to match the back-buffer.

In other words, when you change the size in the UI thread, it continues to
composite the same image (possibly not filling the whole element, or being
stretched) until the worker actually gets the resize and has a chance to
redraw it.

This also means the idea of not being able to commit because of a resize
can go away, and commit() can be void, since the back-buffer size never
actually changes while the worker is drawing.



On Sun, Oct 13, 2013 at 11:01 AM, David Bruant  wrote:

>bool commit();
>>>
>> Boolean as return value for success? :-s
> A promise instead maybe? throw instead of false at least?
> In any case, it looks like commit could be a long operation (tell me if
> I'm wrong here. Do you have numbers on how long it takes/would take?),
> having it async sounds reasonable.


This should be synchronous and never block.  Even if the Canvas is in a
different process, it should be possible to do this with IPC without
waiting for the other side to process the change.

-- 
Glenn Maynard

Re: [whatwg] Canvas in workers

[David Bruant]

Le 13/10/2013 06:12, Kyle Huey a écrit :

I talked at length with Robert O'Callahan about what the DOM API for
supporting  in web workers should look like and we came up with the
following modifications to the spec.

1. Rename CanvasProxy to WorkerCanvas and only allow it to be
transferred to workers.  I don't think we're interested in supporting
cross-origin  via CanvasProxy (I would be curious to hear more
about what the use cases are).
2. Add a worker-only WorkerCanvas constructor that takes the desired
width/height of the drawing surface.
What is the use case for this constructor? Draw something in worker, 
output an image to be sent to main thread?



3. Remove the rendering context constructors and the setContext method
on WorkerCanvas (née CanvasProxy).
4. Copy all of the sensible non-node related things from
HTMLCanvasElement to WorkerCanvas.  This would include
- width and height as readonly attributes
   - getContext (to replace what we removed in step 3).  roc prefers to
   have getContext2D and getContextWebGL, and dispense with the string
   argument version entirely, but I don't have strong feelings.
For the sake of writing interoperable code in main thread and worker, I 
tend to be against this sort of change. I'm aware of the ugliness of 
some APIs, but consistent ugly APIs beats a mix of beautiful and ugly API.



   - toBlob.  We do not intend to implement toDataURL here.
5. Add a "commit" method to WorkerCanvas.  For a WorkerCanvas obtained
from a main thread  element, this would cause the buffer displayed
on screen to swap.  For a WorkerCanvas created *de novo* on a worker
thread, it would do nothing.
Let's have this method optional, then? Or create 2 interfaces? I'm not 
sure of what can be expressed in WebIDL to solve this, but useless 
methods aren't a good idea. If I can't do anything, don't give me the 
method. It's like in UI. If I can't click a button, just don't show me 
the button or at least grey it out.




This commit method would also commit a minor
violation of run-to-completion semantics, described below.
6. We would rely on extracting ImageBitmaps from the WorkerCanvas and
shipping them to the main thread via postMessage to allow synchronizing
canvas updates with DOM updates.  We explored a couple other options but we
didn't come up with anything else that allows synchronizing updates to
multiple canvases from a worker.  This isn't really sketched out here.

So the IDL would look something like:


[Constructor(unsigned long width, unsigned long height)]

interface WorkerCanvas {

   readonly attribute unsigned long width;

   readonly attribute unsigned long height;


   CanvasRenderingContext2D? getContext2D(any... args);

   WebGLRenderingContext? getContextWebGL(any... args);


   void toBlob(FileCallback? _callback, optional DOMString type, any...
arguments);


   bool commit();

Boolean as return value for success? :-s
A promise instead maybe? throw instead of false at least?
In any case, it looks like commit could be a long operation (tell me if 
I'm wrong here. Do you have numbers on how long it takes/would take?), 
having it async sounds reasonable.




};

WorkerCanvas implements Transferable;


Everything would be behave pretty much as one would expect, except perhaps
for the commit method.  The width and height of the canvas can be modified
on the main thread while the worker is drawing.  This would fire an event
off to the worker to update the WorkerCanvas's dimensions that would be
scheduled as if the main thread had postMessage()d something to the
worker.  But it's possible that the worker would attempt to draw to the
 before that update runs.  It's also possible that the worker would
simply draw in a loop without yielding.  To solve this, if commit is called
and the current dimensions on the main thread don't match the dimensions of
the WorkerCanvas it would fail (return false) and update the dimensions of
the WorkerCanvas before returning.  This is technically a violation of
run-to-completion semantics, but is needed to support workers that do not
yield.
I feel fairly strongly against the run-to-completion violation as it's a 
foundation of how JavaScript works, how people reason about programs. 
Also, TC39 is working hard to close the gap between what can be 
expressed in pure ECMAScript and what the web platform does express, 
it'd be inappropriate to add things that widen this gap in my opinion.


I would be much more in favor of a solution like how IndexedDB handles 
transactions. Here is how it could work:
For a canvas which WorkerCanvas has been transfered, when trying to 
change width/height:

*1 Store the values, but don't do it yet
*2 Send a message to the worker to re-dimension,
#1 in the worker: wait for the current message to be taken care of
#2 dispatch a "dimension changed" event (or equivalent)
#3 send a confirmation to main thread
#4 ru

Re: [whatwg] Canvas in workers

[Kyle Huey]

On Sun, Oct 13, 2013 at 12:50 PM, Michael Norton  wrote:

> Thank you!  Read over the 1st link you provided,  interesting.  A worker
> then seems similar to a runtime process in an os environment - is it a
> derivation of RPC?
>

In a manner of speaking.  Workers are an abstraction allowing for parallel
computing on the web without any shared mutable state or locking.  They
communicate via message passing, which can be thought of as an asynchronous
RPC mechanism.


>  Also, "Delegation" [Section 9.1.2.5] sounds like that SETI screensaver
> project years ago, for example.  Is that an accurate description of that
> section's use?
>

I am not familiar with this project.

As for , in Firefox I don't think you need it if you utilize XUL
> which has similar elements essentially describing the same process(es), for
> example employing  and  with various calls to and fro 
> and DOM elements via event listeners, etc.
>

This mailing list is for discussing standards track features for the web,
so the capabilities of proprietary Mozilla features such as XUL have little
relevance here, except perhaps as sources to draw inspiration from.

- Kyle

Re: [whatwg] Canvas in workers

[Kyle Huey]

I'm referring to Web Workers.

http://www.whatwg.org/specs/web-apps/current-work/multipage/workers.html
https://developer.mozilla.org/en-US/docs/Web/Guide/Performance/Using_web_workers

- Kyle


On Sun, Oct 13, 2013 at 12:20 PM, Michael Norton  wrote:

> Hi Kyle,
>
> I have read the editor's draft of canvas 2d context and I have some
> familiarity with DOM.  But I do not understand what you are referencing to
> "worker", missing some lingo here.  Can you expand on that?
>
> Cheers,
>
> Mike
>
> Sent by the hope boat.
>
> On Oct 13, 2013, at 12:12 AM, Kyle Huey  wrote:
>
> > I talked at length with Robert O'Callahan about what the DOM API for
> > supporting  in web workers should look like and we came up with
> the
> > following modifications to the spec.
> >
> >   1. Rename CanvasProxy to WorkerCanvas and only allow it to be
> >   transferred to workers.  I don't think we're interested in supporting
> >   cross-origin  via CanvasProxy (I would be curious to hear more
> >   about what the use cases are).
> >   2. Add a worker-only WorkerCanvas constructor that takes the desired
> >   width/height of the drawing surface.
> >   3. Remove the rendering context constructors and the setContext method
> >   on WorkerCanvas (née CanvasProxy).
> >   4. Copy all of the sensible non-node related things from
> >   HTMLCanvasElement to WorkerCanvas.  This would include
> >   - width and height as readonly attributes
> >  - getContext (to replace what we removed in step 3).  roc prefers to
> >  have getContext2D and getContextWebGL, and dispense with the string
> >  argument version entirely, but I don't have strong feelings.
> >  - toBlob.  We do not intend to implement toDataURL here.
> >   5. Add a "commit" method to WorkerCanvas.  For a WorkerCanvas obtained
> >   from a main thread  element, this would cause the buffer
> displayed
> >   on screen to swap.  For a WorkerCanvas created *de novo* on a worker
> >   thread, it would do nothing.  This commit method would also commit a
> minor
> >   violation of run-to-completion semantics, described below.
> >   6. We would rely on extracting ImageBitmaps from the WorkerCanvas and
> >   shipping them to the main thread via postMessage to allow synchronizing
> >   canvas updates with DOM updates.  We explored a couple other options
> but we
> >   didn't come up with anything else that allows synchronizing updates to
> >   multiple canvases from a worker.  This isn't really sketched out here.
> >
> > So the IDL would look something like:
> >
> >> [Constructor(unsigned long width, unsigned long height)]
> >>
> >> interface WorkerCanvas {
> >>
> >>  readonly attribute unsigned long width;
> >>
> >>  readonly attribute unsigned long height;
> >>
> >>
> >>  CanvasRenderingContext2D? getContext2D(any... args);
> >>
> >>  WebGLRenderingContext? getContextWebGL(any... args);
> >>
> >>
> >>  void toBlob(FileCallback? _callback, optional DOMString type, any...
> >> arguments);
> >>
> >>
> >>  bool commit();
> >>
> >> };
> >>
> >> WorkerCanvas implements Transferable;
> > Everything would be behave pretty much as one would expect, except
> perhaps
> > for the commit method.  The width and height of the canvas can be
> modified
> > on the main thread while the worker is drawing.  This would fire an event
> > off to the worker to update the WorkerCanvas's dimensions that would be
> > scheduled as if the main thread had postMessage()d something to the
> > worker.  But it's possible that the worker would attempt to draw to the
> >  before that update runs.  It's also possible that the worker
> would
> > simply draw in a loop without yielding.  To solve this, if commit is
> called
> > and the current dimensions on the main thread don't match the dimensions
> of
> > the WorkerCanvas it would fail (return false) and update the dimensions
> of
> > the WorkerCanvas before returning.  This is technically a violation of
> > run-to-completion semantics, but is needed to support workers that do not
> > yield.
> >
> > Thoughts?
> >
> > - Kyle
>

[whatwg] Canvas in workers

[Kyle Huey]

I talked at length with Robert O'Callahan about what the DOM API for
supporting  in web workers should look like and we came up with the
following modifications to the spec.

   1. Rename CanvasProxy to WorkerCanvas and only allow it to be
   transferred to workers.  I don't think we're interested in supporting
   cross-origin  via CanvasProxy (I would be curious to hear more
   about what the use cases are).
   2. Add a worker-only WorkerCanvas constructor that takes the desired
   width/height of the drawing surface.
   3. Remove the rendering context constructors and the setContext method
   on WorkerCanvas (née CanvasProxy).
   4. Copy all of the sensible non-node related things from
   HTMLCanvasElement to WorkerCanvas.  This would include
   - width and height as readonly attributes
  - getContext (to replace what we removed in step 3).  roc prefers to
  have getContext2D and getContextWebGL, and dispense with the string
  argument version entirely, but I don't have strong feelings.
  - toBlob.  We do not intend to implement toDataURL here.
   5. Add a "commit" method to WorkerCanvas.  For a WorkerCanvas obtained
   from a main thread  element, this would cause the buffer displayed
   on screen to swap.  For a WorkerCanvas created *de novo* on a worker
   thread, it would do nothing.  This commit method would also commit a minor
   violation of run-to-completion semantics, described below.
   6. We would rely on extracting ImageBitmaps from the WorkerCanvas and
   shipping them to the main thread via postMessage to allow synchronizing
   canvas updates with DOM updates.  We explored a couple other options but we
   didn't come up with anything else that allows synchronizing updates to
   multiple canvases from a worker.  This isn't really sketched out here.

So the IDL would look something like:

> [Constructor(unsigned long width, unsigned long height)]
>
> interface WorkerCanvas {
>
>   readonly attribute unsigned long width;
>
>   readonly attribute unsigned long height;
>
>
>   CanvasRenderingContext2D? getContext2D(any... args);
>
>   WebGLRenderingContext? getContextWebGL(any... args);
>
>
>   void toBlob(FileCallback? _callback, optional DOMString type, any...
> arguments);
>
>
>   bool commit();
>
> };
>
> WorkerCanvas implements Transferable;
>
Everything would be behave pretty much as one would expect, except perhaps
for the commit method.  The width and height of the canvas can be modified
on the main thread while the worker is drawing.  This would fire an event
off to the worker to update the WorkerCanvas's dimensions that would be
scheduled as if the main thread had postMessage()d something to the
worker.  But it's possible that the worker would attempt to draw to the
 before that update runs.  It's also possible that the worker would
simply draw in a loop without yielding.  To solve this, if commit is called
and the current dimensions on the main thread don't match the dimensions of
the WorkerCanvas it would fail (return false) and update the dimensions of
the WorkerCanvas before returning.  This is technically a violation of
run-to-completion semantics, but is needed to support workers that do not
yield.

Thoughts?

- Kyle

Re: [whatwg] Canvas in Workers

[Ian Hickson]

On Wed, 9 Jan 2013, James Robinson wrote:
> On Wed, Jan 9, 2013 at 11:59 AM, Ian Hickson  wrote:
> > On Wed, 9 Jan 2013, Stephen White wrote:
> > > >
> > > > Right now by 2d canvases are effectively single buffered. At the 
> > > > appropriate time a copy of the canvas is made and passed to the 
> > > > compositor. This copy is slow, especially on mobile.
> > >
> > > Currently, to lower the VRAM footprint and improve performance, we 
> > > don't do a copy in 2d canvas.  We temporarily transfer ownership of 
> > > the texture to the compositor at commit time, and block the renderer 
> > > until the composite is complete.  That may change, however.
> >
> > Good to know, thanks. Should this change, or should there be a desire 
> > for a mode where it looks like after the commit the buffer is cleared, 
> > please do let me know.
> 
> We (chromium) would really appreciate a way for the author to express 
> clear-after-commit.  I think this would match up with what authors 
> frequently want, since many canvas pages clear the canvas at the start 
> of the next frame anyway, but allow for more flexibility and 
> optimizations in our implementation.

Noted: 
http://wiki.whatwg.org/wiki/New_Features_Awaiting_Implementation_Interest

-- 
Ian Hickson   U+1047E)\._.,--,'``.fL
http://ln.hixie.ch/   U+263A/,   _.. \   _\  ;`._ ,.
Things that are impossible just take longer.   `._.-(,_..'--(,_..'`-.;.'

Re: [whatwg] Canvas in Workers

[James Robinson]

On Wed, Jan 9, 2013 at 11:59 AM, Ian Hickson  wrote:

> On Wed, 9 Jan 2013, Stephen White wrote:
> > >
> > > Right now by 2d canvases are effectively single buffered. At the
> > > appropriate time a copy of the canvas is made and passed to the
> > > compositor. This copy is slow, especially on mobile.
> >
> > Currently, to lower the VRAM footprint and improve performance, we don't
> > do a copy in 2d canvas.  We temporarily transfer ownership of the
> > texture to the compositor at commit time, and block the renderer until
> > the composite is complete.  That may change, however.
>
> Good to know, thanks. Should this change, or should there be a desire for
> a mode where it looks like after the commit the buffer is cleared, please
> do let me know.
>

We (chromium) would really appreciate a way for the author to express
clear-after-commit.  I think this would match up with what authors
frequently want, since many canvas pages clear the canvas at the start of
the next frame anyway, but allow for more flexibility and optimizations in
our implementation.

- James

>
> Would still love input from any other vendors, too.
>
> --
> Ian Hickson   U+1047E)\._.,--,'``.fL
> http://ln.hixie.ch/   U+263A/,   _.. \   _\  ;`._ ,.
> Things that are impossible just take longer.   `._.-(,_..'--(,_..'`-.;.'
>

Re: [whatwg] Canvas in Workers

[Justin Novosad]

On Wed, Jan 9, 2013 at 2:50 PM, Stephen White wrote:

>
>
> Currently, to lower the VRAM footprint and improve performance, we don't do
> a copy in 2d canvas.  We temporarily transfer ownership of the texture to
> the compositor at commit time, and block the renderer until the composite
> is complete.  That may change, however.
>
> Stephen
>
>
To be more accurate, the renderer only blocks if it needs to access the
texture, for example if getImageData is called before the composite is
complete. All write operations are non-blocking because they can be
recorded an deferred until the composite is complete.

>
>

Re: [whatwg] Canvas in Workers

[Ian Hickson]

On Wed, 9 Jan 2013, Stephen White wrote:
> >
> > Right now by 2d canvases are effectively single buffered. At the 
> > appropriate time a copy of the canvas is made and passed to the 
> > compositor. This copy is slow, especially on mobile.
> 
> Currently, to lower the VRAM footprint and improve performance, we don't 
> do a copy in 2d canvas.  We temporarily transfer ownership of the 
> texture to the compositor at commit time, and block the renderer until 
> the composite is complete.  That may change, however.

Good to know, thanks. Should this change, or should there be a desire for 
a mode where it looks like after the commit the buffer is cleared, please 
do let me know.

Would still love input from any other vendors, too.

-- 
Ian Hickson   U+1047E)\._.,--,'``.fL
http://ln.hixie.ch/   U+263A/,   _.. \   _\  ;`._ ,.
Things that are impossible just take longer.   `._.-(,_..'--(,_..'`-.;.'

Re: [whatwg] Canvas in Workers

[Stephen White]

On Thu, Jan 3, 2013 at 7:46 PM, Gregg Tavares  wrote:

> On Tue, Dec 11, 2012 at 9:04 AM, Ian Hickson  wrote:
>
> > On Tue, 11 Dec 2012, Gregg Tavares (社ç~T¨) wrote:
> > >
> > > discussion seems to have died down here but I'd like to bring up
> another
> > > issue
> > >
> > > In WebGL land we have creation attributes on the drawingbuffer made
> for a
> > > canvas. Example
> > >
> > > gl = canvas.getContext("webgl", { preserveDrawingBuffer: false });
> > >
> > > We're working out the details on how to set those options for the case
> > > where we have 1 context and multiple canvases.
> > >
> > > The particular option above would apparently be a huge perf win for
> > > canvas 2d for mobile. Which suggests that whatever API is decided on it
> > > would be nice if it worked for both APIs the same.
> >
> > What does it do?
> >
>
> Effectively it makes the canvas double buffered.
>
> Right now by 2d canvases are effectively single buffered. At the
> appropriate time a copy
> of the canvas is made and passed to the compositor. This copy is slow,
> especially on
> mobile.
>

Currently, to lower the VRAM footprint and improve performance, we don't do
a copy in 2d canvas.  We temporarily transfer ownership of the texture to
the compositor at commit time, and block the renderer until the composite
is complete.  That may change, however.

Stephen


>
> Apple requested that for WebGL the default be for double buffering. When
> double buffered, when the canvas is composited (when the current JavaScript
> event exits)
> the canvas's buffer is given to the compositor and the canvas is given a
> new buffer (or
> an old one). That new buffer is cleared, meaning the contents is gone. It's
> up to the app to
> draw stuff into again. If nothing is drawn the compositor will continue to
> use the
> buffer it acquired earlier.
>
> In WebGL you can opt into the slower "copy" path. For Canvas 2D while the
> default
> has to remain the slow "copy" path it would be nice to be able to opt into
> the faster
> "swap" double buffered path.
>
>
>
>
>
>
>
> >
> > In the 2D canvas, whenever you bind to a new canvas, the context is reset
> > to its default state, the context's hit region list is reset, and the
> > context's bitmap is reset. The next time the context is flushed, the
> > canvas itself is always reset (since flushing the context causes the
> > bitmap and hit region list to be pushed to the canvas, replacing whatever
> > was there before).
> >
> > --
> > Ian Hickson   U+1047E)\._.,--,'``.fL
> > http://ln.hixie.ch/   U+263A/,   _.. \   _\  ;`._ ,.
> > Things that are impossible just take longer.   `._.-(,_..'--(,_..'`-.;.'
> >
>

Re: [whatwg] Canvas in Workers

[Mark Callow]

On 2013/01/09 12:37, Ian Hickson wrote:
> On Wed, 9 Jan 2013, Mark Callow wrote:
>> I would expect these wins to be equally important when doing animation 
>> in Canvas2D as I would double buffering to to avoid flicker.
> Are implementors interested in a way to make 2D rendering contexts use 
> page flipping instead of double buffering?
>
I don't understand what you mean by page flipping so, even if I were a
browser implementer, I couldn't answer.

Regards

-Mark

-- 
注意：この電子メールには、株式会社エイチアイの機密情報が含まれている場合
が有ります。正式なメール受信者では無い場合はメール複製、 再配信または情
報の使用を固く禁じております。エラー、手違いでこのメールを受け取られまし
たら削除を行い配信者にご連絡をお願いいたし ます.

NOTE: This electronic mail message may contain confidential and
privileged information from HI Corporation. If you are not the intended
recipient, any disclosure, photocopying, distribution or use of the
contents of the received information is prohibited. If you have received
this e-mail in error, please notify the sender immediately and
permanently delete this message and all related copies.

Re: [whatwg] Canvas in Workers

[Ian Hickson]

On Wed, 9 Jan 2013, Mark Callow wrote:
> 
> I would expect these wins to be equally important when doing animation 
> in Canvas2D as I would double buffering to to avoid flicker.

Are implementors interested in a way to make 2D rendering contexts use 
page flipping instead of double buffering?

-- 
Ian Hickson   U+1047E)\._.,--,'``.fL
http://ln.hixie.ch/   U+263A/,   _.. \   _\  ;`._ ,.
Things that are impossible just take longer.   `._.-(,_..'--(,_..'`-.;.'

Re: [whatwg] Canvas in Workers

[Mark Callow]

On 2013/01/04 9:46, Gregg Tavares wrote:
> On Tue, Dec 11, 2012 at 9:04 AM, Ian Hickson  wrote:
>
>> On Tue, 11 Dec 2012, Gregg Tavares (社ç~T¨) wrote:
>>> In WebGL land we have creation attributes on the drawingbuffer made for a
>>> canvas. Example
>>>
>>> gl = canvas.getContext("webgl", { preserveDrawingBuffer: false });
>>>
>>> We're working out the details on how to set those options for the case
>>> where we have 1 context and multiple canvases.
>>>
>>> The particular option above would apparently be a huge perf win for
>>> canvas 2d for mobile. Which suggests that whatever API is decided on it
>>> would be nice if it worked for both APIs the same.
>> What does it do?
>>
> Effectively it makes the canvas double buffered.

The reason for this attribute and defaulting it to false, is so that
tiled renderers can avoid copying the area of the previously-rendered
drawing buffer corresponding to the current tile, into tile memory prior
to rendering. Avoiding this copy is a big performance and power win for
tiled renderers. As far as I am aware every mobile GPU, with the
exception of NVIDIA's Tegra, is a tiled renderer so supporting them is
very important.

I would expect these wins to be equally important when doing animation
in Canvas2D as I would double buffering to to avoid flicker.

Regards

-Mark

-- 
注意：この電子メールには、株式会社エイチアイの機密情報が含まれている場合
が有ります。正式なメール受信者では無い場合はメール複製、 再配信または情
報の使用を固く禁じております。エラー、手違いでこのメールを受け取られまし
たら削除を行い配信者にご連絡をお願いいたし ます.

NOTE: This electronic mail message may contain confidential and
privileged information from HI Corporation. If you are not the intended
recipient, any disclosure, photocopying, distribution or use of the
contents of the received information is prohibited. If you have received
this e-mail in error, please notify the sender immediately and
permanently delete this message and all related copies.

Re: [whatwg] Canvas in Workers

[Ian Hickson]

On Tue, 8 Jan 2013, Gregg Tavares wrote:
> >
> > Supporting page flipping in 2D canvas would be fine too, but I don't 
> > see why it would need a change to the API... you would just make 
> > "commit()" flip which page was active for the context API and clear 
> > the newly active page in one operation.
> 
> How would you choose flip vs copy with just commit?

An argument to the context constructor, probably.


> Just to be clear we're on the same page. I want to be able to do this 
> (not related to workers)
> 
>// create a 2d context that flips buffers instead of copies them
>var ctx = canvas.getContext("2d", { preserveDrawingBuffer: false });

I would do it like this:

   var ctx = new CanvasRenderingContext2D(100, 100, { mode: 'pageflip' });
   canvas.setContext(ctx);

   ctx.fillRect(10,10,10,10);
   ctx.commit(); // implies clearRect(0,0,100,100);

   // later
   ctx.fillRect(11,11,10,10);
   ctx.commit(); // implies clearRect(0,0,100,100);



> But, related to workers, if CanvasProxy is truly a proxy for the canvas 
> then I could do this
> 
>// create a 2d context that flips buffers instead of copies them
>var ctx = canvasProxy.getContext("2d", { preserveDrawingBuffer: false });

CanvasProxy doesn't have getContext(), because the direct mode doesn't 
make sense cross-process. You would do:

   var ctx = new CanvasRenderingContext2D(100, 100, { mode: 'pageflip' });
   canvasProxy.setContext(ctx);

   // draw...


> Agreed but that's a separate problem

These are all related problems that contribute to the design of the same 
API.


> Problem #1) Allow a worker to render to a canvas
> Problem #2) Allow a worker to render offscreen (without communicating with
> the main page)
> 
> I'm suggesting we only solve problem #1 for now. To do that, all we need 
> is CanvasProxy to truly be "a proxy for the canvas".

I'm solving #2 now too. (Actually it's already solved in the spec.) I'm 
happy to only discuss #1 if you want, but #2 influences the design of the 
API for #1 so the conversation will be weird if we ignore #2.


> > setContext() is only needed so that you can use one context with 
> > multiple canvases, which is primarily intended to address the WebGL 
> > case of having one context used to render to multiple views with 
> > different settings (the settings being themselves set on the canvas or 
> > canvas proxy).
>
> Right, but since it doesn't doesn't seem to work for WebGL's needs why 
> spec it now when we can solve problem #1 today and worry about the other 
> problems later?

It seems to work fine as far as I can tell, as shown by the examples in my 
recent e-mail.


> > > Is there some reason that won't work?
> >
> > Well I'd rather not design something that doesn't address a known 
> > issue and then find we have painted ourselves into a corner with 
> > respect to that other issue. Hence trying to solve all the issues at 
> > once, or at least solve them in a way that is compatible with future 
> > solutions.
> 
> Agreed. I'm just trying to make forward progress.

At this point I'm not aware of any problems with the current proposal 
(what's in the HTML spec plus adding setSettings() in WebGL, making the 
WebGLRenderingContext constructable, and providing an in-worker 
alternative destination for WebGL for when you don't want to draw to a 
canvas, as discussed in my earlier e-mail today).


> I'm wondering if we can separate the 2 issues by just making CanvasProxy 
> be "a proxy for the canvas" with the same API as Canvas.

That doesn't work for 2D. The original API (getContext) relies on implicit 
commits, and we can't do implicit commits sanely in a worker. That's the 
main reason for having a different mechanism (setContext).

Anyway, I don't really understand what's wrong with the proposal that 
addresses all the problems that have been raised at once. I don't see a 
need to do this bit by bit when we've already got a full solution.

-- 
Ian Hickson   U+1047E)\._.,--,'``.fL
http://ln.hixie.ch/   U+263A/,   _.. \   _\  ;`._ ,.
Things that are impossible just take longer.   `._.-(,_..'--(,_..'`-.;.'

Re: [whatwg] Canvas in Workers

[Gregg Tavares]

On Tue, Jan 8, 2013 at 11:12 AM, Ian Hickson  wrote:

> On Thu, 3 Jan 2013, Gregg Tavares wrote:
> > On Tue, Dec 11, 2012 at 9:04 AM, Ian Hickson  wrote:
> > > On Tue, 11 Dec 2012, Gregg Tavares (社ç~T¨) wrote:
> > > >
> > > > discussion seems to have died down here but I'd like to bring up
> > > > another issue
> > > >
> > > > In WebGL land we have creation attributes on the drawingbuffer made
> > > > for a canvas. Example
> > > >
> > > > gl = canvas.getContext("webgl", { preserveDrawingBuffer: false
> });
> > > >
> > > > We're working out the details on how to set those options for the
> > > > case where we have 1 context and multiple canvases.
> > > >
> > > > The particular option above would apparently be a huge perf win for
> > > > canvas 2d for mobile. Which suggests that whatever API is decided on
> > > > it would be nice if it worked for both APIs the same.
> > >
> > > What does it do?
> >
> > Effectively it makes the canvas double buffered.
> >
> > Right now by 2d canvases are effectively single buffered. At the
> > appropriate time a copy of the canvas is made and passed to the
> > compositor. This copy is slow, especially on mobile.
> >
> > Apple requested that for WebGL the default be for double buffering. When
> > double buffered, when the canvas is composited (when the current
> > JavaScript event exits) the canvas's buffer is given to the compositor
> > and the canvas is given a new buffer (or an old one). That new buffer is
> > cleared, meaning the contents is gone. It's up to the app to draw stuff
> > into again. If nothing is drawn the compositor will continue to use the
> > buffer it acquired earlier.
>
> I think you mean page flipping, not double buffering.
>
> Supporting page flipping in 2D canvas would be fine too, but I don't see
> why it would need a change to the API... you would just make "commit()"
> flip which page was active for the context API and clear the newly active
> page in one operation.
>

How would you choose flip vs copy with just commit?

Just to be clear we're on the same page. I want to be able to do this (not
related to workers)

   // create a 2d context that flips buffers instead of copies them
   var ctx = canvas.getContext("2d", { preserveDrawingBuffer: false });

But, related to workers, if CanvasProxy is truly a proxy for the canvas
then I could do this

   // create a 2d context that flips buffers instead of copies them
   var ctx = canvasProxy.getContext("2d", { preserveDrawingBuffer: false });



>
> On Thu, 3 Jan 2013, Gregg Tavares wrote:
> >
> > So I've been thinking more about this and I'm wondering, for the time
> > being, why have canvas.setContext and why expose the
> > Canvas2DRenderingContext constructor?
>
> Well the constructor is needed so that there's a way to do an entirely
> off-screen bitmap, for when you just want to do some image work that isn't
> going to be displayed.
>

Agreed but that's a separate problem

Problem #1) Allow a worker to render to a canvas
Problem #2) Allow a worker to render offscreen (without communicating with
the main page)

I'm suggesting we only solve problem #1 for now. To do that, all we need is
CanvasProxy to truly be "a proxy for the canvas".


>
> setContext() is only needed so that you can use one context with multiple
> canvases, which is primarily intended to address the WebGL case of having
> one context used to render to multiple views with different settings (the
> settings being themselves set on the canvas or canvas proxy).
>
>
Right, but since it doesn't doesn't seem to work for WebGL's needs why spec
it now when we can solve problem #1 today and worry about the other
problems later?


> > That means we can solve the 1 context multiple canvases issue later
> > making this a minimal api change?
>
> I thought the "1 context multiple canvases issue" was a higher priority
> than the "canvas on workers" issue. Is this wrong?
>

I don't know if it's higher priority. It seemed to inform the worker
related design so it was important to look at.


>
>
> > Is there some reason that won't work?
>
> Well I'd rather not design something that doesn't address a known issue
> and then find we have painted ourselves into a corner with respect to that
> other issue. Hence trying to solve all the issues at once, or at least
> solve them in a way that is compatible with future solutions.
>

Agreed. I'm just trying to make forward progress. The whole setContext vs
DrawingBuffer etc stuff seems possibly orthogonal to the issue of rendering
to a canvas from a worker. I'm wondering if we can separate the 2 issues by
just making CanvasProxy be "a proxy for the canvas" with the same API as
Canvas. Since Canvas already is what it is, any solution for single context
multiple canvases has to work with Canvas. So adding a new object,
CanvasProxy, which is really just a representation for Canvas with the same
API doesn't seem like it adds any new issues.



>
> --
> Ian Hickson   U+1047E)\._.,--...

Re: [whatwg] Canvas in Workers

[Ian Hickson]

On Thu, 3 Jan 2013, Gregg Tavares wrote:
> On Tue, Dec 11, 2012 at 9:04 AM, Ian Hickson  wrote:
> > On Tue, 11 Dec 2012, Gregg Tavares (社ç~T¨) wrote:
> > >
> > > discussion seems to have died down here but I'd like to bring up 
> > > another issue
> > >
> > > In WebGL land we have creation attributes on the drawingbuffer made 
> > > for a canvas. Example
> > >
> > > gl = canvas.getContext("webgl", { preserveDrawingBuffer: false });
> > >
> > > We're working out the details on how to set those options for the 
> > > case where we have 1 context and multiple canvases.
> > >
> > > The particular option above would apparently be a huge perf win for 
> > > canvas 2d for mobile. Which suggests that whatever API is decided on 
> > > it would be nice if it worked for both APIs the same.
> >
> > What does it do?
> 
> Effectively it makes the canvas double buffered.
>
> Right now by 2d canvases are effectively single buffered. At the 
> appropriate time a copy of the canvas is made and passed to the 
> compositor. This copy is slow, especially on mobile.
>
> Apple requested that for WebGL the default be for double buffering. When 
> double buffered, when the canvas is composited (when the current 
> JavaScript event exits) the canvas's buffer is given to the compositor 
> and the canvas is given a new buffer (or an old one). That new buffer is 
> cleared, meaning the contents is gone. It's up to the app to draw stuff 
> into again. If nothing is drawn the compositor will continue to use the 
> buffer it acquired earlier.

I think you mean page flipping, not double buffering.

Supporting page flipping in 2D canvas would be fine too, but I don't see 
why it would need a change to the API... you would just make "commit()" 
flip which page was active for the context API and clear the newly active 
page in one operation.


On Thu, 3 Jan 2013, Gregg Tavares wrote:
>
> So I've been thinking more about this and I'm wondering, for the time 
> being, why have canvas.setContext and why expose the 
> Canvas2DRenderingContext constructor?

Well the constructor is needed so that there's a way to do an entirely 
off-screen bitmap, for when you just want to do some image work that isn't 
going to be displayed.

setContext() is only needed so that you can use one context with multiple 
canvases, which is primarily intended to address the WebGL case of having 
one context used to render to multiple views with different settings (the 
settings being themselves set on the canvas or canvas proxy).


> That means we can solve the 1 context multiple canvases issue later 
> making this a minimal api change?

I thought the "1 context multiple canvases issue" was a higher priority 
than the "canvas on workers" issue. Is this wrong?


> Is there some reason that won't work?

Well I'd rather not design something that doesn't address a known issue 
and then find we have painted ourselves into a corner with respect to that 
other issue. Hence trying to solve all the issues at once, or at least 
solve them in a way that is compatible with future solutions.

-- 
Ian Hickson   U+1047E)\._.,--,'``.fL
http://ln.hixie.ch/   U+263A/,   _.. \   _\  ;`._ ,.
Things that are impossible just take longer.   `._.-(,_..'--(,_..'`-.;.'

Re: [whatwg] Canvas in Workers

[Gregg Tavares]

On Tue, Dec 11, 2012 at 9:04 AM, Ian Hickson  wrote:

> On Tue, 11 Dec 2012, Gregg Tavares (社ç~T¨) wrote:
> >
> > discussion seems to have died down here but I'd like to bring up another
> > issue
> >
> > In WebGL land we have creation attributes on the drawingbuffer made for a
> > canvas. Example
> >
> > gl = canvas.getContext("webgl", { preserveDrawingBuffer: false });
> >
> > We're working out the details on how to set those options for the case
> > where we have 1 context and multiple canvases.
> >
> > The particular option above would apparently be a huge perf win for
> > canvas 2d for mobile. Which suggests that whatever API is decided on it
> > would be nice if it worked for both APIs the same.
>
> What does it do?
>

Effectively it makes the canvas double buffered.

Right now by 2d canvases are effectively single buffered. At the
appropriate time a copy
of the canvas is made and passed to the compositor. This copy is slow,
especially on
mobile.

Apple requested that for WebGL the default be for double buffering. When
double buffered, when the canvas is composited (when the current JavaScript
event exits)
the canvas's buffer is given to the compositor and the canvas is given a
new buffer (or
an old one). That new buffer is cleared, meaning the contents is gone. It's
up to the app to
draw stuff into again. If nothing is drawn the compositor will continue to
use the
buffer it acquired earlier.

In WebGL you can opt into the slower "copy" path. For Canvas 2D while the
default
has to remain the slow "copy" path it would be nice to be able to opt into
the faster
"swap" double buffered path.







>
> In the 2D canvas, whenever you bind to a new canvas, the context is reset
> to its default state, the context's hit region list is reset, and the
> context's bitmap is reset. The next time the context is flushed, the
> canvas itself is always reset (since flushing the context causes the
> bitmap and hit region list to be pushed to the canvas, replacing whatever
> was there before).
>
> --
> Ian Hickson   U+1047E)\._.,--,'``.fL
> http://ln.hixie.ch/   U+263A/,   _.. \   _\  ;`._ ,.
> Things that are impossible just take longer.   `._.-(,_..'--(,_..'`-.;.'
>

Re: [whatwg] Canvas in Workers

[Ian Hickson]

On Tue, 11 Dec 2012, Gregg Tavares (社�~T�) wrote:
>
> discussion seems to have died down here but I'd like to bring up another
> issue
> 
> In WebGL land we have creation attributes on the drawingbuffer made for a
> canvas. Example
> 
> gl = canvas.getContext("webgl", { preserveDrawingBuffer: false });
> 
> We're working out the details on how to set those options for the case
> where we have 1 context and multiple canvases.
> 
> The particular option above would apparently be a huge perf win for 
> canvas 2d for mobile. Which suggests that whatever API is decided on it 
> would be nice if it worked for both APIs the same.

What does it do?

In the 2D canvas, whenever you bind to a new canvas, the context is reset 
to its default state, the context's hit region list is reset, and the 
context's bitmap is reset. The next time the context is flushed, the 
canvas itself is always reset (since flushing the context causes the 
bitmap and hit region list to be pushed to the canvas, replacing whatever 
was there before).

-- 
Ian Hickson   U+1047E)\._.,--,'``.fL
http://ln.hixie.ch/   U+263A/,   _.. \   _\  ;`._ ,.
Things that are impossible just take longer.   `._.-(,_..'--(,_..'`-.;.'

Re: [whatwg] Canvas in Workers

[社用]

discussion seems to have died down here but I'd like to bring up another
issue

In WebGL land we have creation attributes on the drawingbuffer made for a
canvas. Example

gl = canvas.getContext("webgl", { preserveDrawingBuffer: false });

We're working out the details on how to set those options for the case
where we have 1 context and multiple canvases.

The particular option above would apparently be a huge perf win for canvas
2d for mobile. Which suggests that whatever API is decided on it would be
nice if it worked for both APIs the same.

Re: [whatwg] Canvas in Workers

[社用]

On Sat, Dec 1, 2012 at 2:44 AM, Ian Hickson  wrote:

> On Fri, 30 Nov 2012, Gregg Tavares (社ç~T¨) wrote:
> >
> > on ImageBitmap should zero size canvases just work (and create a 0 sized
> > ImageBitmap)?
> >
> > My personal preference is for APIs that just work with zero sizes so I
> > don't have to write lots of special cases for handling zero.
> >
> > For example [1,2,3].slice(0,0) returns []. It doesn't throw.
> > "abc".substring(0,0) returns "" it doesn't throw. fillRect(x, y, 0, 0)
> > doesn't throw. etc...
> >
> > It just makes life a lot easier
>
> The main reason 0-sized canvases have always thrown in drawImage() is that
> I couldn't work out what you would paint, nor why you'd have a zero-sized
> canvas, and throwing seemed like it'd be the best way to help the author
> figure out where the problem was, rather than just ignoring the call and
> having the author scratch their head about why nothing was happening.
>
> If there's cases where you would legitimately end up with zero-sized
> canvases that you'd try to draw from, though, I'm happy to change it.
>

I don't see how zero sized canvases are any different than zero sized
arrays or empty strings. It's not a matter of use case. It's a matter of
not having to write checks everywhere for 0.  If I'm writing some app that
takes a user supplied size (say a photo editing app where the user can
select a rectangle and copy and paste), why do I want to have to check for
zero?

var x = Math.min(x1, x2);
var y = Math.min(y1, y2);
var width = Math.abs(x1 - x2);
var height = Math.abs(y1 - y2);

// Do something with rect defined by x,y,width,height

This seems no different from malloc(0) in C or the other cases I've
mentioned (array size 0 and empty string).  Lots of programming becomes
easier when size = 0 works. Maybe I'm animating

   function draw() {
  var scale = Math.sin() * 0.5 + 0.5;
  var width = realWidth * scale;
  var height = realHeight * scale;

  // do something with width, height
   }

Why do I want to have to check for zero and special case it?

You could argue that I'd have to check of negative values but that's still
nicer than checking for 0

   function draw() {
  var scale = Math.sin() * 0.5 + 0.5;
  var width = Math.max(0, realWidth * scale);
  var height = Math.max(0, realHeight * scale);

  // do something with width, height
   }

vs

   function draw() {
  var scale = Math.sin() * 0.5 + 0.5;
  var width = realWidth * scale;
  var height = realHeight * scale;

  if (width <= 0 || height <= 0) {
 // skip this step
  } else {
// do something with width, height
  }
   }

I'm just making the case it seems like 0 should always work. That includes
ImageBitmap, Canvas and ImageData



>
> --
> Ian Hickson   U+1047E)\._.,--,'``.fL
> http://ln.hixie.ch/   U+263A/,   _.. \   _\  ;`._ ,.
> Things that are impossible just take longer.   `._.-(,_..'--(,_..'`-.;.'
>

Re: [whatwg] Canvas in Workers

[Ian Hickson]

On Fri, 30 Nov 2012, Gregg Tavares (社�~T�) wrote:
>
> on ImageBitmap should zero size canvases just work (and create a 0 sized
> ImageBitmap)?
> 
> My personal preference is for APIs that just work with zero sizes so I
> don't have to write lots of special cases for handling zero.
> 
> For example [1,2,3].slice(0,0) returns []. It doesn't throw.
> "abc".substring(0,0) returns "" it doesn't throw. fillRect(x, y, 0, 0)
> doesn't throw. etc...
> 
> It just makes life a lot easier

The main reason 0-sized canvases have always thrown in drawImage() is that 
I couldn't work out what you would paint, nor why you'd have a zero-sized 
canvas, and throwing seemed like it'd be the best way to help the author 
figure out where the problem was, rather than just ignoring the call and 
having the author scratch their head about why nothing was happening.

If there's cases where you would legitimately end up with zero-sized 
canvases that you'd try to draw from, though, I'm happy to change it.

-- 
Ian Hickson   U+1047E)\._.,--,'``.fL
http://ln.hixie.ch/   U+263A/,   _.. \   _\  ;`._ ,.
Things that are impossible just take longer.   `._.-(,_..'--(,_..'`-.;.'

Re: [whatwg] Canvas in Workers

[Ian Hickson]

On Mon, 19 Nov 2012, Gregg Tavares (社�~T�) wrote:
>
> Sorry if this is clear in the specs but can you explain how sizing the 
> canvas works?

There's two sizes involved in a canvas: the size of the bitmap, and the 
size of the rendering. Both default to 300x150 in the absence of explicit 
sizing.

The size of the bitmap, for getContext()-based canvases, is given by the 
 element's width and height attributes. For rendering contexts 
created by their constructor, it's initially set by the arguments to the 
constructor. It can be changed using the width and height attributes on 
the rendering context. When a context is bound to a canvas, it takes on 
the dimensions of the canvas bitmap. To change the canvas bitmap 
dimensions before calling setContext() or transferControlToProxy(), you 
use the width and height attributes as before. Once you've called 
setContext() or transferControlToProxy(), though, to avoid race conditions 
with workers, only the width and height attributes on the rendering 
context affect the canvas, so you have to first bind the rendering context 
and then set the size if you need the size changed.

The size of the rendering is set by CSS, and defaults to whatever the 
width and height attributes of the canvas element are. Note that if you 
are drawing to the canvas from a worker and you have changed the 
dimensions of the bitmap after you called setContext() or 
transferControlToProxy(), you will have to explicitly set the dimensions 
of the canvas using the element's attributes or CSS, otherwise the canvas 
bitmap will be resized for rendering.


>// main.html
>

Here the bitmap and rendering are 300x150.

>
> var canvas = document.getElementsByTagName('canvas')[0];
> var worker = new Worker('clock.js');
> var proxy = canvas.transferControlToProxy());

At this point, canvas.width/.height no longer affect the bitmap size.

> worker.postMessage(proxy, [proxy]);
> 
> setTimeout(function() {
> canvas.width = 200;   // does this work? What happens?

This just changes the rendered width.

> }, 4000);
>
> 
>// clock.js worker
>onmessage = function (event) {
>  var context = new CanvasRenderingContext2d();

The constructor could be called with arguments, as in new 
   CanvasRenderingContext2d(200, 150);

That size will have no effect though, since the context is immediately 
bound to a canvas bitmap and takes the bitmap's size:

>  event.data.setContext(context);
>  setInterval(function () {
>context.width = 400;  // Can I do this? What happens when I do?

This changes the canvas bitmap's dimensions (and resets all state of the 
rendering context). It doesn't change the canvas element's rendered 
dimensions, however (we could push the state out like the bitmap itself, I 
guess, but that would make the DOM change out of the blue which is weird).

>context.clearRect(0, 0, context.width, context.height);
>context.fillText(0, 100, new Date());
>context.commit();
>  }, 1000);
>};

HTH,
-- 
Ian Hickson   U+1047E)\._.,--,'``.fL
http://ln.hixie.ch/   U+263A/,   _.. \   _\  ;`._ ,.
Things that are impossible just take longer.   `._.-(,_..'--(,_..'`-.;.'

Re: [whatwg] Canvas in Workers

[Rick Waldron]

On Fri, Nov 16, 2012 at 4:48 PM, Ian Hickson  wrote:

> On Fri, 16 Nov 2012, Rick Waldron wrote:
> > On Fri, Nov 16, 2012 at 4:11 PM, Ian Hickson  wrote:
> > > On Fri, 16 Nov 2012, Rick Waldron wrote:
> > > > On Fri, Nov 16, 2012 at 3:25 PM, Ian Hickson  wrote:
> > > > >
> > > > > var proxy = canvas.transferControlToProxy());
> > > >
> > > > 
> > > >
> > > > Proxy with a capital P is an API in ES6 for creating Proxy objects.
> > > > While I recognize that this is obviously just a conceptual sketch of
> > > > your idea and that you're merely following the camel case tradition,
> now
> > > > is probably the best time to be informed of potential naming
> conflicts.
> > >
> > > The word Proxy is used all over the place on the Web platform, I'm not
> too
> > > worried about that. Right now the object returned above is
> "CanvasProxy",
> > > which is the kind of proxy that it refers to. I'm certainly open to
> > > another name, any suggestions?
> >
> > CanvasProxy is a fine disambiguation.
>
> Do you think we should rename the method above to say that too?
>
> "canvas.transferControlToCanvasProxy()" is a bit verbose. :-)
>
> I had shortened it to just "canvas.transferControlToProxy" on the
>

Is there any other thing that control can be transferred to?
canvas.transferControl() would be sufficient...

Rick


> assumption that the fact that it was called on a canvas would disambiguate
> that it was a canvas proxy, but I'm certainly open to other suggestions.
>
> --
> Ian Hickson   U+1047E)\._.,--,'``.fL
> http://ln.hixie.ch/   U+263A/,   _.. \   _\  ;`._ ,.
> Things that are impossible just take longer.   `._.-(,_..'--(,_..'`-.;.'
>

Re: [whatwg] Canvas in Workers

[Ian Hickson]

On Fri, 16 Nov 2012, Rick Waldron wrote:
> On Fri, Nov 16, 2012 at 4:11 PM, Ian Hickson  wrote:
> > On Fri, 16 Nov 2012, Rick Waldron wrote:
> > > On Fri, Nov 16, 2012 at 3:25 PM, Ian Hickson  wrote:
> > > >
> > > > var proxy = canvas.transferControlToProxy());
> > >
> > > 
> > >
> > > Proxy with a capital P is an API in ES6 for creating Proxy objects.
> > > While I recognize that this is obviously just a conceptual sketch of
> > > your idea and that you're merely following the camel case tradition, now
> > > is probably the best time to be informed of potential naming conflicts.
> >
> > The word Proxy is used all over the place on the Web platform, I'm not too
> > worried about that. Right now the object returned above is "CanvasProxy",
> > which is the kind of proxy that it refers to. I'm certainly open to
> > another name, any suggestions?
>
> CanvasProxy is a fine disambiguation.

Do you think we should rename the method above to say that too?

"canvas.transferControlToCanvasProxy()" is a bit verbose. :-)

I had shortened it to just "canvas.transferControlToProxy" on the 
assumption that the fact that it was called on a canvas would disambiguate 
that it was a canvas proxy, but I'm certainly open to other suggestions.

-- 
Ian Hickson   U+1047E)\._.,--,'``.fL
http://ln.hixie.ch/   U+263A/,   _.. \   _\  ;`._ ,.
Things that are impossible just take longer.   `._.-(,_..'--(,_..'`-.;.'

Re: [whatwg] Canvas in Workers

[Rick Waldron]

CanvasProxy is a fine disambiguation.

Rick


On Fri, Nov 16, 2012 at 4:11 PM, Ian Hickson  wrote:

> On Fri, 16 Nov 2012, Rick Waldron wrote:
> > On Fri, Nov 16, 2012 at 3:25 PM, Ian Hickson  wrote:
> > >
> > > var proxy = canvas.transferControlToProxy());
> >
> > 
> >
> > Proxy with a capital P is an API in ES6 for creating Proxy objects.
> > While I recognize that this is obviously just a conceptual sketch of
> > your idea and that you're merely following the camel case tradition, now
> > is probably the best time to be informed of potential naming conflicts.
>
> The word Proxy is used all over the place on the Web platform, I'm not too
> worried about that. Right now the object returned above is "CanvasProxy",
> which is the kind of proxy that it refers to. I'm certainly open to
> another name, any suggestions?
>
> --
> Ian Hickson   U+1047E)\._.,--,'``.fL
> http://ln.hixie.ch/   U+263A/,   _.. \   _\  ;`._ ,.
> Things that are impossible just take longer.   `._.-(,_..'--(,_..'`-.;.'
>

Re: [whatwg] Canvas in Workers

[Ian Hickson]

On Fri, 16 Nov 2012, Oliver Hunt wrote:
>
> postMessage already has an ownership transfer list, we could always 
> extend the set of transferable types to include a canvas like thing (the 
> context or some such).

That's what CanvasProxy is.


> IIRC the last time I suggested having canvas (or at least canvas-like) 
> APIs available in workers it turned into a spec for rotating and scaling 
> images (which i still don't fully understand).

Yeah, this is intended to make that kind of thing unnecessary.

-- 
Ian Hickson   U+1047E)\._.,--,'``.fL
http://ln.hixie.ch/   U+263A/,   _.. \   _\  ;`._ ,.
Things that are impossible just take longer.   `._.-(,_..'--(,_..'`-.;.'

Re: [whatwg] Canvas in Workers

[Ian Hickson]

On Fri, 16 Nov 2012, Rick Waldron wrote:
> On Fri, Nov 16, 2012 at 3:25 PM, Ian Hickson  wrote:
> >
> > var proxy = canvas.transferControlToProxy());
> 
> 
> 
> Proxy with a capital P is an API in ES6 for creating Proxy objects. 
> While I recognize that this is obviously just a conceptual sketch of 
> your idea and that you're merely following the camel case tradition, now 
> is probably the best time to be informed of potential naming conflicts.

The word Proxy is used all over the place on the Web platform, I'm not too 
worried about that. Right now the object returned above is "CanvasProxy", 
which is the kind of proxy that it refers to. I'm certainly open to 
another name, any suggestions?

-- 
Ian Hickson   U+1047E)\._.,--,'``.fL
http://ln.hixie.ch/   U+263A/,   _.. \   _\  ;`._ ,.
Things that are impossible just take longer.   `._.-(,_..'--(,_..'`-.;.'

Re: [whatwg] Canvas in Workers

[Tab Atkins Jr.]

On Fri, Nov 16, 2012 at 1:03 PM, Oliver Hunt  wrote:
> postMessage already has an ownership transfer list, we could always extend 
> the set of transferable types to include a canvas like thing (the context or 
> some such).

Transferring contexts is tolerable for 2d, but a non-starter for webgl
- there's too much stuff that can be loaded into a context that would
have to be transferred cross-process.  Hixie's approach uses a
lightweight canvas proxy instead.

> IIRC the last time I suggested having canvas (or at least canvas-like) APIs 
> available in workers it turned into a spec for rotating and scaling images 
> (which i still don't fully understand).

That was an attempt to solve the use-cases without doing the work to
actually port  to the worker.  Hixie's now done the latter.

~TJ

Re: [whatwg] Canvas in Workers

[Oliver Hunt]

postMessage already has an ownership transfer list, we could always extend the 
set of transferable types to include a canvas like thing (the context or some 
such).

IIRC the last time I suggested having canvas (or at least canvas-like) APIs 
available in workers it turned into a spec for rotating and scaling images 
(which i still don't fully understand).

--Oliver

On Nov 16, 2012, at 12:58 PM, Rick Waldron  wrote:

> On Fri, Nov 16, 2012 at 3:25 PM, Ian Hickson  wrote:
> 
>> 
>> 
>>var proxy = canvas.transferControlToProxy());
>> 
> 
> 
> 
> Proxy with a capital P is an API in ES6 for creating Proxy objects. While I
> recognize that this is obviously just a conceptual sketch of your idea and
> that you're merely following the camel case tradition, now is probably the
> best time to be informed of potential naming conflicts.
> 
> 
> Rick

Re: [whatwg] Canvas in Workers

[Rick Waldron]

On Fri, Nov 16, 2012 at 3:25 PM, Ian Hickson  wrote:

>
>
> var proxy = canvas.transferControlToProxy());
>



Proxy with a capital P is an API in ES6 for creating Proxy objects. While I
recognize that this is obviously just a conceptual sketch of your idea and
that you're merely following the camel case tradition, now is probably the
best time to be informed of potential naming conflicts.


Rick

[whatwg] Canvas in Workers

[Ian Hickson]

Over the years, there have been lots of requests for asynchronous image 
manipulation APIs, and each time we discuss it we end up concluding that 
what we should really do is just put canvas on workers.

I've now specced a concrete proposal for this.

Since we can't literally put  in a worker since workers don't have 
DOMs, I was faced with two options:

Option A: Provide an API for off-screen graphics in workers, requiring 
that every frame you package the whole thing up, send it over to the main 
thread, and blt it there.

Option B: Provide a mechanism by which a worker can actually paint 
directly onto a main thread  element.

I ended up specifying something that lets you do both. You can instantiate 
a CanvasRenderingContext2D object anywhere, including a worker; when a 
rendering context is not bound to a specific canvas, it can be used as an 
off-screen drawing surface. You can then bind such a rendering context to 
a canvas; when you do this, there is a commit() method that you can use 
which tells the user agent to push the bitmap drawn on the 
CanvasRenderingContext2D object to the canvas bitmap / screen.

A quick example of how this can work:

   // main.html
   
   
var canvas = document.getElementsByTagName('canvas')[0];
var worker = new Worker('clock.js');
var proxy = canvas.transferControlToProxy());
worker.postMessage(proxy, [proxy]);
   

   // clock.js worker
   onmessage = function (event) {
 var context = new CanvasRenderingContext2d();
 event.data.setContext(context);
 setInterval(function () {
   context.clearRect(0, 0, context.width, context.height);
   context.fillText(0, 100, new Date());
   context.commit();
 }, 1000);
   };

There are several difficulties in a canvas-in-worker solution that I'd 
like to mention here, with a brief pointer to the solutions I used:

1. Workers had no way to represent images (e.g. sprites) before, which is 
rather limiting for a bitmap canvas API. I have introduced a new object 
called ImageBitmap that can be constructed from a Blob obtained from XHR 
in a worker, or that can be created from , , and  
elements and transfered over to a worker. This can then be used to render 
images in workers.

2. The 2D rendering context API has several features that actually 
interact with the page, e.g. scrollIntoView(). I've tried to make these 
work too; there's a mechanism by which these actions are queued up and 
then when you call commit() they're run as a task on the main thread.

3. Since the main thread paints on its own schedule, and the worker might 
be half-way through its drawing when the main thread wants to paint, I've 
had to introduce an explicit commit() that tells the UA to push the data 
from the worker to the rendering.

4. Since you can read the image from the main thread (for example, using 
the context.drawImage(canvas, ...) method, or even, now that there's the 
ImageBitmap object, using "new ImageBitmap(canvas)"), as well as read the 
image from the worker (using the same mechanisms, but using the context as 
the source instead of the canvas -- we have to support this, because it's 
common to do things like fade out the current scene, or shift it a bit, or 
otherwise manipulate the image itself), and given that the rendered bitmap 
is not the same as the currently drawing one (since we have to commit, see 
above), I've ended up having to specify that when you're on a worker, a 
canvas really has two bitmaps, the scratch bitmap and the output bitmap. 
This does, if the author actually tries to read the scratch bitmap, mean 
that there's a greater memory requirement when working on a worker, but I 
really don't see a way around it.

5. We don't yet have a way to access Web Fonts in a worker. I'm relying on 
the CSS Fonts module's FontLoader proposal for this.

At the same time, the WebGL community has been looking for a way to bind a 
single rendering context to a several rendering contexts in sequence, 
allowing the same scene to be painted from different angles. I also tried 
to support this model; it will require some additions to WebGL to use the 
hooks that have been provided. Specifically:

 - define the WebGLRenderingContext constructor.

 - define the "binding steps" and the "unbinding steps".

 - define a way to specify the settings for how to render to bitmaps, for 
   example whether to anti-alias or not. If you want to have this 
   information be on the canvas elements themselves, please let me know; 
   you'll need to come up with a mechanism for how to do this cross-thread
   without race conditions.

 - define when pixels are pushed to "the canvas's bitmap". In particular, 
   this needs to handle how to do it when the canvas is in a different 
   thread, such that it is well-defined what bitmap an author gets if they
   use the HTMLCanvasElement itself in a drawImage() call. (See the 
   definition of commit() and the "commit the scratch bitmap" algorithm 
   for how the 2D case do