Re: Layer Construction for Off Main Thread canvas
On Thu, Oct 31, 2013 at 12:36 PM, Robert O'Callahan rob...@ocallahan.orgwrote: On Thu, Oct 31, 2013 at 5:28 PM, Kyle Huey m...@kylehuey.com wrote: One thing that's come up that we're not quite how to deal with for OMTcanvas is how to modify GetCanvasLayer. Our problem here is that the context here lives on the worker thread, and presumably we need to construct the layer on the main thread, but creating that layer requires data that also lives on the worker thread. We obviously can't block the main thread on a synchronous call to the worker, and we can't just lock around the values because that will break run to completion semantics on the worker (e.g. if the worker never yields, changes in the canvas size shouldn't take effect). I think the right thing to do is probably to ship the values from the worker to the main thread asynchronously as they are updated on the worker (once the worker yields/etc) and create the layer using those values. How often do we create layers? It would be a shame if rendering straight from the worker to the compositor were blocked by the main thread due to layer creation. Anything else we should be thinking about here? What values do you need from the worker in order to create the layer? Well, at the very least we need the dimensions of the canvas buffer. We need to know if the context is lost too, but that can already happen anytime so that's not too bad. The dimensions live on the worker thread (which can update them) so we need to either lock and read them from the main thread or maintain a separate copy of the dimensions on the main thread and post messages to update them. If we do the first then it's possible for the dimensions the compositor sees to be newer than the last data we pushed to it. If we do the latter then the opposite is possible. Does the compositor care if the layer size and the data we push to it are not in sync? The CanvasLayer also wants the GLContext to do stuff with it. Not sure what we're going to do with that ... It seems to me that we should be able to create a CanvasLayer on the main thread that is sized to the canvas element. Then from the CanvasLayer we get some kind of thread-safe object (analogous to the ImageContainer of an ImageLayer) which can be handed to the worker in the WorkerCanvas. This object would support being one end of the surface stream for WebGL. You would feed a series of surfaces into it which could have different sizes and the compositor will size them to the layer. That sounds reasonable ... it seems like just getting everything set up is the hard part. - Kyle ___ dev-platform mailing list dev-platform@lists.mozilla.org https://lists.mozilla.org/listinfo/dev-platform
Re: Layer Construction for Off Main Thread canvas
With OMTC these days the Layer classes don't hold any logic. All the fancy stuff goes into classes inheriting from CompositableClient (and CompositableHost on the compositor side). While Layer classes can only be manipulated from the main thread and use the PLayerTransaction IPDL protocol to sync with the compositor, Compositable classes can be used by either the PLayerTransaction or the PImageBridge protocol. The logic to attach a Compositable using PImageBridge to a Layer using PLayerTransaction is rather simple (look at ImageClientBridge and ImageContainer) and mostly implemented in the CompositableClient/Host abstraction so you don't have to redo most of it. As Nick said you could have on the main thread a dummy CanvasClient analogous to ImageClientBridge which just forward an asyncID which is the ID used to connect the layer and the compositable on the compositor side, and on the ImageBridge thread you would have the usual CanvasClient. Then you'd need the equivalent of ImageContainer that would interface with the WebGLContext and forward frames to the ImageClient on the ImageBridge thread. This is the way we do with async video, which works well. This email is probably a bit confusing if you haven't looked at the Compositable stuff yet, so don't hesitate to ping me and ask questions about the compositor's IPC stuff and async updates. Cheers, Nical On Thu, Oct 31, 2013 at 7:03 AM, Kyle Huey m...@kylehuey.com wrote: On Thu, Oct 31, 2013 at 12:36 PM, Robert O'Callahan rob...@ocallahan.org wrote: On Thu, Oct 31, 2013 at 5:28 PM, Kyle Huey m...@kylehuey.com wrote: One thing that's come up that we're not quite how to deal with for OMTcanvas is how to modify GetCanvasLayer. Our problem here is that the context here lives on the worker thread, and presumably we need to construct the layer on the main thread, but creating that layer requires data that also lives on the worker thread. We obviously can't block the main thread on a synchronous call to the worker, and we can't just lock around the values because that will break run to completion semantics on the worker (e.g. if the worker never yields, changes in the canvas size shouldn't take effect). I think the right thing to do is probably to ship the values from the worker to the main thread asynchronously as they are updated on the worker (once the worker yields/etc) and create the layer using those values. How often do we create layers? It would be a shame if rendering straight from the worker to the compositor were blocked by the main thread due to layer creation. Anything else we should be thinking about here? What values do you need from the worker in order to create the layer? Well, at the very least we need the dimensions of the canvas buffer. We need to know if the context is lost too, but that can already happen anytime so that's not too bad. The dimensions live on the worker thread (which can update them) so we need to either lock and read them from the main thread or maintain a separate copy of the dimensions on the main thread and post messages to update them. If we do the first then it's possible for the dimensions the compositor sees to be newer than the last data we pushed to it. If we do the latter then the opposite is possible. Does the compositor care if the layer size and the data we push to it are not in sync? The CanvasLayer also wants the GLContext to do stuff with it. Not sure what we're going to do with that ... It seems to me that we should be able to create a CanvasLayer on the main thread that is sized to the canvas element. Then from the CanvasLayer we get some kind of thread-safe object (analogous to the ImageContainer of an ImageLayer) which can be handed to the worker in the WorkerCanvas. This object would support being one end of the surface stream for WebGL. You would feed a series of surfaces into it which could have different sizes and the compositor will size them to the layer. That sounds reasonable ... it seems like just getting everything set up is the hard part. - Kyle ___ dev-platform mailing list dev-platform@lists.mozilla.org https://lists.mozilla.org/listinfo/dev-platform ___ dev-platform mailing list dev-platform@lists.mozilla.org https://lists.mozilla.org/listinfo/dev-platform
Re: Layer Construction for Off Main Thread canvas
On Thu, Oct 31, 2013 at 11:19 PM, Nicolas Silva nical.si...@gmail.comwrote: With OMTC these days the Layer classes don't hold any logic. All the fancy stuff goes into classes inheriting from CompositableClient (and CompositableHost on the compositor side). While Layer classes can only be manipulated from the main thread and use the PLayerTransaction IPDL protocol to sync with the compositor, Compositable classes can be used by either the PLayerTransaction or the PImageBridge protocol. The logic to attach a Compositable using PImageBridge to a Layer using PLayerTransaction is rather simple (look at ImageClientBridge and ImageContainer) and mostly implemented in the CompositableClient/Host abstraction so you don't have to redo most of it. As Nick said you could have on the main thread a dummy CanvasClient analogous to ImageClientBridge which just forward an asyncID which is the ID used to connect the layer and the compositable on the compositor side, and on the ImageBridge thread you would have the usual CanvasClient. Then you'd need the equivalent of ImageContainer that would interface with the WebGLContext and forward frames to the ImageClient on the ImageBridge thread. This last point is very important. It's very important that CanvasClient and ImageBridge and stuff like that not exposed outside the layers system. I think someone from gfx really needs to help with this part. Rob -- Jtehsauts tshaei dS,o n Wohfy Mdaon yhoaus eanuttehrotraiitny eovni le atrhtohu gthot sf oirng iyvoeu rs ihnesa.rt sS?o Whhei csha iids teoa stiheer :p atroa lsyazye,d 'mYaonu,r sGients uapr,e tfaokreg iyvoeunr, 'm aotr atnod sgaoy ,h o'mGee.t uTph eann dt hwea lmka'n? gBoutt uIp waanndt wyeonut thoo mken.o w * * ___ dev-platform mailing list dev-platform@lists.mozilla.org https://lists.mozilla.org/listinfo/dev-platform
Re: Layer Construction for Off Main Thread canvas
I have no idea about construction and resizing, but webgl's frame flinging (and by extension skia-gl's) should Just Work already. WebGL content blithely calls SurfaceStream::SwapProducer, and somewhere in Layers/TextureClient/Host code there's a call to SurfaceStream::SwapConsumer. The only thing WebGL needs moved between the client and host size of things is the SurfaceStream pointer. -Jeff - Original Message - From: Robert O'Callahan rob...@ocallahan.org To: Nicolas Silva nical.si...@gmail.com Cc: Kyle Huey m...@kylehuey.com, Morris Tseng mts...@mozilla.com, Jeff Gilbert jgilb...@mozilla.com, Benoit Jacob bja...@mozilla.com, Milan Sreckovic msrecko...@mozilla.com, Matt Woodrow mwood...@mozilla.com, Nicholas Cameron n...@mozilla.com, dev-platform dev-platform@lists.mozilla.org Sent: Thursday, October 31, 2013 12:54:42 PM Subject: Re: Layer Construction for Off Main Thread canvas On Thu, Oct 31, 2013 at 11:19 PM, Nicolas Silva nical.si...@gmail.comwrote: With OMTC these days the Layer classes don't hold any logic. All the fancy stuff goes into classes inheriting from CompositableClient (and CompositableHost on the compositor side). While Layer classes can only be manipulated from the main thread and use the PLayerTransaction IPDL protocol to sync with the compositor, Compositable classes can be used by either the PLayerTransaction or the PImageBridge protocol. The logic to attach a Compositable using PImageBridge to a Layer using PLayerTransaction is rather simple (look at ImageClientBridge and ImageContainer) and mostly implemented in the CompositableClient/Host abstraction so you don't have to redo most of it. As Nick said you could have on the main thread a dummy CanvasClient analogous to ImageClientBridge which just forward an asyncID which is the ID used to connect the layer and the compositable on the compositor side, and on the ImageBridge thread you would have the usual CanvasClient. Then you'd need the equivalent of ImageContainer that would interface with the WebGLContext and forward frames to the ImageClient on the ImageBridge thread. This last point is very important. It's very important that CanvasClient and ImageBridge and stuff like that not exposed outside the layers system. I think someone from gfx really needs to help with this part. Rob -- Jtehsauts tshaei dS,o n Wohfy Mdaon yhoaus eanuttehrotraiitny eovni le atrhtohu gthot sf oirng iyvoeu rs ihnesa.rt sS?o Whhei csha iids teoa stiheer :p atroa lsyazye,d 'mYaonu,r sGients uapr,e tfaokreg iyvoeunr, 'm aotr atnod sgaoy ,h o'mGee.t uTph eann dt hwea lmka'n? gBoutt uIp waanndt wyeonut thoo mken.o w * * ___ dev-platform mailing list dev-platform@lists.mozilla.org https://lists.mozilla.org/listinfo/dev-platform
Layer Construction for Off Main Thread canvas
One thing that's come up that we're not quite how to deal with for OMTcanvas is how to modify GetCanvasLayer. Our problem here is that the context here lives on the worker thread, and presumably we need to construct the layer on the main thread, but creating that layer requires data that also lives on the worker thread. We obviously can't block the main thread on a synchronous call to the worker, and we can't just lock around the values because that will break run to completion semantics on the worker (e.g. if the worker never yields, changes in the canvas size shouldn't take effect). I think the right thing to do is probably to ship the values from the worker to the main thread asynchronously as they are updated on the worker (once the worker yields/etc) and create the layer using those values. How often do we create layers? It would be a shame if rendering straight from the worker to the compositor were blocked by the main thread due to layer creation. Anything else we should be thinking about here? - Kyle ___ dev-platform mailing list dev-platform@lists.mozilla.org https://lists.mozilla.org/listinfo/dev-platform
Re: Layer Construction for Off Main Thread canvas
You probably don't want to draw into the layer at all. So the main thread should have a canvas layer and the worker thread should draw into a CanvasClient (I think, the canvas layer will have a dummy canvas client too which doesn't do much). This is similar to how async video works where it is called the ImageBridge protocol. You should extend ImageBridge to work with canvas, I think. This would mean the main thread is not involved with rendering at all. Nical is the ImageBridge/async video expert and he can tell you more and correct my errors. Nick On 31/10/2013 5:28 p.m., Kyle Huey wrote: One thing that's come up that we're not quite how to deal with for OMTcanvas is how to modify GetCanvasLayer. Our problem here is that the context here lives on the worker thread, and presumably we need to construct the layer on the main thread, but creating that layer requires data that also lives on the worker thread. We obviously can't block the main thread on a synchronous call to the worker, and we can't just lock around the values because that will break run to completion semantics on the worker (e.g. if the worker never yields, changes in the canvas size shouldn't take effect). I think the right thing to do is probably to ship the values from the worker to the main thread asynchronously as they are updated on the worker (once the worker yields/etc) and create the layer using those values. How often do we create layers? It would be a shame if rendering straight from the worker to the compositor were blocked by the main thread due to layer creation. Anything else we should be thinking about here? - Kyle ___ dev-platform mailing list dev-platform@lists.mozilla.org https://lists.mozilla.org/listinfo/dev-platform
Re: Layer Construction for Off Main Thread canvas
On Thu, Oct 31, 2013 at 5:28 PM, Kyle Huey m...@kylehuey.com wrote: One thing that's come up that we're not quite how to deal with for OMTcanvas is how to modify GetCanvasLayer. Our problem here is that the context here lives on the worker thread, and presumably we need to construct the layer on the main thread, but creating that layer requires data that also lives on the worker thread. We obviously can't block the main thread on a synchronous call to the worker, and we can't just lock around the values because that will break run to completion semantics on the worker (e.g. if the worker never yields, changes in the canvas size shouldn't take effect). I think the right thing to do is probably to ship the values from the worker to the main thread asynchronously as they are updated on the worker (once the worker yields/etc) and create the layer using those values. How often do we create layers? It would be a shame if rendering straight from the worker to the compositor were blocked by the main thread due to layer creation. Anything else we should be thinking about here? What values do you need from the worker in order to create the layer? It seems to me that we should be able to create a CanvasLayer on the main thread that is sized to the canvas element. Then from the CanvasLayer we get some kind of thread-safe object (analogous to the ImageContainer of an ImageLayer) which can be handed to the worker in the WorkerCanvas. This object would support being one end of the surface stream for WebGL. You would feed a series of surfaces into it which could have different sizes and the compositor will size them to the layer. Rob -- Jtehsauts tshaei dS,o n Wohfy Mdaon yhoaus eanuttehrotraiitny eovni le atrhtohu gthot sf oirng iyvoeu rs ihnesa.rt sS?o Whhei csha iids teoa stiheer :p atroa lsyazye,d 'mYaonu,r sGients uapr,e tfaokreg iyvoeunr, 'm aotr atnod sgaoy ,h o'mGee.t uTph eann dt hwea lmka'n? gBoutt uIp waanndt wyeonut thoo mken.o w * * ___ dev-platform mailing list dev-platform@lists.mozilla.org https://lists.mozilla.org/listinfo/dev-platform
