Re: exposing CANVAS or something like it to Web Workers
So how to take this forward? My #1 priority is to get WebGL in workers. Lots of developers have expressed a need for this from decoding compressed textures in a worker to offloading thousands of draw calls per frame to a worker. WebGL already defines sharing mechanisms so at least for the WebGL case I don't need to solve how to display anything back in the main worker. The worker can draw to a texture. The main page can use that texture. WebGL/OpenGL already define that relationship. But, to get WebGL in a worker I need a way to get a WebGLRenderingContext into a worker. Some Ideas **) Create context in main page, pass to worker* Pros: . No new APIs or objects. Cons: . Transfer is messy. What happens to all the entry points and properties to the context object left in the main page? What happens to the cavnas parameter on the transferred context? How do you synchronize the main page setting canvas.width or canvas.height with the worker trying to render to it? **) Create a context directly in a Worker (or anywhere for that matter)* As in var gl = new WebGLRenderingContext Pros: . ??? Cons: . requires defining how the backbuffer size is set. Maybe there is no backbuffer for a directly created WebGLRenderingContext? . if there is no backbuffer then using one of these contexts to draw into a texture or 2d context is problematic **) Create an offscreen canvas like object minus HTMLElement parts* * * Pros: . same or nearly the API as already exists for canvas . flexible. getContext can return different kinds of context. Maybe only webgl for now? Cons: . ???
Re: exposing CANVAS or something like it to Web Workers
On Wed, May 16, 2012 at 12:42 PM, Kenneth Russell k...@google.com wrote: On Wed, May 16, 2012 at 12:30 PM, Gregg Tavares (勤) g...@google.com wrote: So how to take this forward? My #1 priority is to get WebGL in workers. Lots of developers have expressed a need for this from decoding compressed textures in a worker to offloading thousands of draw calls per frame to a worker. WebGL already defines sharing mechanisms so at least for the WebGL case I don't need to solve how to display anything back in the main worker. The worker can draw to a texture. The main page can use that texture. WebGL/OpenGL already define that relationship. But, to get WebGL in a worker I need a way to get a WebGLRenderingContext into a worker. Some Ideas *) Create context in main page, pass to worker Pros: . No new APIs or objects. Cons: . Transfer is messy. What happens to all the entry points and properties to the context object left in the main page? What happens to the cavnas parameter on the transferred context? How do you synchronize the main page setting canvas.width or canvas.height with the worker trying to render to it? *) Create a context directly in a Worker (or anywhere for that matter) As in var gl = new WebGLRenderingContext Pros: . ??? Cons: . requires defining how the backbuffer size is set. Maybe there is no backbuffer for a directly created WebGLRenderingContext? . if there is no backbuffer then using one of these contexts to draw into a texture or 2d context is problematic This alternative seems like the one that can be moved forward most easily, since essentially all of the changes would be within the WebGL spec. - Extend the spec to support context sharing. (Each WebGLRenderingContext points to an opaque WebGLContextGroup object.) - Define structured cloning semantics for WebGLContextGroup. - Add a constructor or factory method to WebGLRenderingContext allowing creation with a WebGLContextGroup. Contexts created in this manner would have no back buffer by default. (FBOs could still be used to do rendering with the context.) - Allow some or all of the WebGLObject types (textures, etc.) to be either copied during structured cloning or transferred. Then the worker can at least upload textures to the GPU completely asynchronously from the main thread, and inform the main thread via postMessage when they have finished uploading. That seems to be a tractable first step and one that would already have immediate benefits for developers. That problem I have with this path is the cons mentioned above. It's a dead end. With real contexts you can do this webglcontext1.texImage2D(..., webglcontext2.canvas); and this context2d.drawImage(webgl.canvas, ...) But with a context created with new WebGLRenderingContext you can't do anything like that because size of the backbuffer is not defined. Eventually we'll want to support operation like that and we'll likely end up with something like DrawingSurface or CanvasSurface or OffscreenCanvas. At that point this ability to go new WebGLRenderingContext will just be left over cruft. Also we'll have to define how to addEventListner for WebGLRenderContext for listening for lost context or for async context creation (yea, less important on workers). That stuff is currently not on WebGLRenderingContext. It seems like it should stay off if it. -Ken *) Create an offscreen canvas like object minus HTMLElement parts Pros: . same or nearly the API as already exists for canvas . flexible. getContext can return different kinds of context. Maybe only webgl for now? Cons: . ???
exposing CANVAS or something like it to Web Workers
I'd like to work on exposing something like CANVAS to web workers. Ideally how over it works I'd like to be able to *) get a 2d context in a web worker *) get a WebGL context in a web worker *) download images in a web worker and the images with both 2d contexts and WebGL contexts Any thoughts?
Re: exposing CANVAS or something like it to Web Workers
On Mon, May 14, 2012 at 4:42 PM, Jonas Sicking jo...@sicking.cc wrote:
On Mon, May 14, 2012 at 3:28 PM, Glenn Maynard gl...@zewt.org wrote:
On Mon, May 14, 2012 at 3:01 PM, Gregg Tavares (勤) g...@google.com
wrote:
I'd like to work on exposing something like CANVAS to web workers.
Ideally how over it works I'd like to be able to
*) get a 2d context in a web worker
I'd recommend not trying to tackle 2d and 3d contexts at once, and only
worrying about WebGL to start.
Another issue: rendering in a worker thread onto a canvas which is
displayed
in the main thread. This needs to be solved in a way that doesn't cause
the
asynchronous nature of what's happening to be visible to scripts.
toDataURL
and toBlob would probably need to be prohibited on the canvas element.
I'm
not sure what the actual API would look like.
If/when we do this, I think it should be done in such a way that the
main window can't access the canvas object at all. Similar to what
happens when an ArrayBuffer is transferred to a Worker using
structured cloning. Once a canvas is transferred to a Worker, any
access to it should throw or return null/0/. If you want to transfer
pixel data to the main thread, it seems less racy to do that by
getting the pixel data in the Worker which owns the canvas and then
transfer that to the main thread using postMessage.
How about separating the canvasy parts of canvas from canvas and the imagy
parts of image from image.
In other words, Imagine canvas is implemented like this
class Canvas : public HTMLElement {
private:
CanvasSurface* m_surface; // everything about canvas that is not
HTMLElement
};
And that Image is similarly implemented as
class Image : public HTMLElement {
private:
Picture* m_picture; // everything about Image that is not HTMLElement
}
now imagine you can instantiate inner implementation of these things. The
parts that are not HTMLElement
var canvasSurface = new CanvasSurface();
var ctx = canvasSurface.getContext(2d);
var pic = new Picture;
pic.src = http://someplace.com/someimage.jpg;;
pic.onload = function() {
ctx.drawImage(pic, 0, );
}
Let's assume you can instantiate these things in either the page or a
worker. Nether can be transfered.
Would that work? What problems would that have?
This would also require some
equivalent to requestAnimationFrame in the worker thread.
Agreed!
/ Jonas
Re: exposing CANVAS or something like it to Web Workers
On Mon, May 14, 2012 at 5:12 PM, Boris Zbarsky bzbar...@mit.edu wrote:
On 5/14/12 8:03 PM, Gregg Tavares (勤) wrote:
var canvasSurface = new CanvasSurface();
var ctx = canvasSurface.getContext(2d)**;
var pic = new Picture;
pic.src =
http://someplace.com/**someimage.jpghttp://someplace.com/someimage.jpg
;
pic.onload = function() {
ctx.drawImage(pic, 0, );
}
Let's assume you can instantiate these things in either the page or a
worker. Nether can be transfered.
Would that work? What problems would that have?
Two things that come to mind immediately:
1) Various canvas 2d context methods depend on the styles of the canvas
to define how they actually behave. Clearly this would need some sort of
changes for Workers anyway; the question is what those changes would need
to be.
Which methods are these?
2) How (or maybe whether) this should work with pic.src =
http://someplace.com/**someimage.svg http://someplace.com/someimage.svg
is an interesting question.
3) This leaves open the question of how to get data from the worker back
to the main thread to paint there.
I feel that's a somewhat orthogonal issue to getting a Canvas or
CanvasSurface in a worker. At least for the WebGL case, WebGL already
defines ways to share data safely between contexts so that use case is
already covered.
For a 2d context I can certainly imagine lots of ways. Some fast, some slow
slow) pass the result of toDataURL back
slow) pass an ImageData.data back (transfer of ownership)
fast) some how use a CanvasSurface as the source of a video tag
maybe there are some other ideas? But, even if we didn't resolve this, it's
still massively useful, at least for WebGL, to be able to instantiate a
WebGL context in a worker.
#1 is an issue with any proposal that puts a 2d context in a worker.
#2 is only an issue if we do the image loads in workers somehow; if we
wanted to just rasterize the SVG and pass the resulting data to the worker
there would be no problem, obviously.
For #3, being able to transfer a canvas context from the page to a worker
but leave it painting to the canvas may still be desirable...
-Boris
Re: exposing CANVAS or something like it to Web Workers
On Mon, May 14, 2012 at 6:07 PM, Boris Zbarsky bzbar...@mit.edu wrote: On 5/14/12 8:55 PM, Gregg Tavares (勤) wrote: 1) Various canvas 2d context methods depend on the styles of the canvas to define how they actually behave. Clearly this would need some sort of changes for Workers anyway; the question is what those changes would need to be. Which methods are these? Anything involving setting color (e.g. the strokeStyle setter, the fillStyle setter), due to currentColor. Anything involving text because font styles come from the element or document. Good to know. That doesn't sound like a showstopper though. If a canvas/CanvasSurface is available in workers the simplest solution would just be that currentColor defaults to something black? or nothing . Pick one. Those are the ones that come to mind offhand, but I haven't looked at the various recent additions to the 2d context closely. -Boris
Re: exposing CANVAS or something like it to Web Workers
On Mon, May 14, 2012 at 6:23 PM, Jonas Sicking jo...@sicking.cc wrote: On Mon, May 14, 2012 at 5:10 PM, Charles Pritchard ch...@jumis.com wrote: On May 14, 2012, at 4:42 PM, Jonas Sicking jo...@sicking.cc wrote: On Mon, May 14, 2012 at 3:28 PM, Glenn Maynard gl...@zewt.org wrote: On Mon, May 14, 2012 at 3:01 PM, Gregg Tavares (勤) g...@google.com wrote: I'd like to work on exposing something like CANVAS to web workers. Ideally how over it works I'd like to be able to *) get a 2d context in a web worker I'd recommend not trying to tackle 2d and 3d contexts at once, and only worrying about WebGL to start. Another issue: rendering in a worker thread onto a canvas which is displayed in the main thread. This needs to be solved in a way that doesn't cause the asynchronous nature of what's happening to be visible to scripts. toDataURL and toBlob would probably need to be prohibited on the canvas element. I'm not sure what the actual API would look like. If/when we do this, I think it should be done in such a way that the main window can't access the canvas object at all. Similar to what happens when an ArrayBuffer is transferred to a Worker using structured cloning. Once a canvas is transferred to a Worker, any access to it should throw or return null/0/. If you want to transfer pixel data to the main thread, it seems less racy to do that by getting the pixel data in the Worker which owns the canvas and then transfer that to the main thread using postMessage. This would also require some equivalent to requestAnimationFrame in the worker thread. Agreed! / Jonas I'm a bit lost-- wouldn't we just postMessage from the document over to the web worker when we want a refresh? I agree that we ought to be transferring pixel data not Canvas contexts; with the possible exception of CSS context. We could just create new canvas instances inside the worker thread. I'd still prefer to clone CanvasPattern as a means of transferring paint over to the worker, though sending pixel data would work too. I heard Picture come up-- it seems like that object might have additional semantics for high resolution alternatives that may need to be considered. I was saying that I think we should transfer the context from the main thread to the worker thread. The worker thread should then be able to use the context to draw directly to the screen without interacting with the main thread. That's a great goal but is transferring the right way to do it? context.canvas is currently a reference to the canvas from which the context as created. Will that just go NULL on transfer? Will it change to some other object that is not a Canvas but acts similarly? What happens to the context on the main page? All of it's methods become no-ops and it's properties become non-live? How will things synchronize if I call canvas.toDataURL after passing the context to a work? Exception? nothing? It seems hard to define how that would synchronize If there was a way to allow a worker to generate a media stream and one of those ways involves an in worker canvas/canvassurface, then it seems like none of the above questions have to be answered. Instead the worker creates the surface, and then through some other API attached it to a video tag. You can call video.toDataURL, that's well defined (I assume) Synchronization is also somewhat defined in that direction. Doesn't that seem like the path of least resistance? You should generally not need to transfer pixel data. But I think it should be possible to grab pixel data on whichever thread is currently owning a context. This will implicitly make it possible to transfer pixel data if anyone wants to do it. / Jonas
Adding a paint event to HTMLElement to support Web Components / Shadow DOM
Imagine you want to make a Web Component that draws a graph Imagine that you'd like the graph to show more data if it's larger and less if it's smaller. In other words, you don't want it to scale the data. Imagine you set that component's size to width: 100%; height: 100% to scale to its container Question: How does the Web Component know when to repaint its content? AFAICT the resize event only fires on window. If the point of Web Components is to allow people to make self contained components it seems like there might be the need for some way of letting the component know it needs to repaint. Resize is just one example. Imagine a relatively heavy to repaint WebComponent like one that draws an representation of an audio wave. If that component is hidden behind some other component it would be nice if it didn't re-draw itself. When it becomes visible thought needs to know it should redraw itself. Does a 'paint' event make sense?
Re: Adding a paint event to HTMLElement to support Web Components / Shadow DOM
On Tue, May 8, 2012 at 5:56 PM, Boris Zbarsky bzbar...@mit.edu wrote: On 5/8/12 8:30 PM, Gregg Tavares (勤) wrote: AFAICT the resize event only fires on window. There have been proposals over the years to change that. Imagine a relatively heavy to repaint WebComponent like one that draws an representation of an audio wave. If that component is hidden behind some other component it would be nice if it didn't re-draw itself. This seems like a use case for requestAnimationFrame's second argument, and browser quality-of-implementation issues. I don't think I understand how requestAnimationFrame would work here. Maybe my example was poor. I'm not suggesting a live constantly updating audio wave. Instead I'm suggesting a static WebComponent that is heavy to render. For example the wave display in an audio editing tool. Let's say I had an app that displayed several audio tracks, some of those tracks are scrolled off the page. (like this http://www.newfreedownloads.com/imgs/17189-w520.jpg) The tracks components are set to 100% width so that sizing the window ends up re-sizing all the components, even the components off the screen. Ideally only those components that are on the screen will get a paint event since if say 80% of the components are off the screen and they are heavy to update it would be nice if they weren't updated until the user brings them on the screen by scrolling them on. Does a 'paint' event make sense? Possibly, depending on how it's defined; actually running arbitrary script during painting is pretty undesirable (e.g. ideally painting happens on a totally separate thread from script, and having to synchronize them is not particularly ideal). -Boris
